Mgiyuk

Pandas Convert float column containing nan values to int for merge operation Attempt #1 s["order_id"].apply(lambda x: int(x) if pd.notnull(x) else np.nan) Attempt #2 def to_int(x): if(pd.notnull(x)): return int(x) Attempt #3 s["order_id"] = s.loc[pd.notnull(s["order_id"]),"order_id].astype(int) All of these return a series where the values are still formatted as floats. I'm wondering if I could use the update function or take advantage of reindexing. Leveraging Indexing solution attempt: null = np.nan data = {"time":{"0":1528971021539,"1":1529289904697,"2":1529572773525,"3":1529892602301,"4":1530082881098,"5":1530069453264,"6":1528985491630,"7":1529236762719,"8":1529475504491,"9":1529814085541,"10":1529906568681,"11":1530160346468,"12":1529833559160,"13":1530051985183,"14":153024...

How to assign a unique ID to detect repeated rows in a pandas dataframe? I am working with a large pandas dataframe, with several columns pretty much like this: A B C D John Tom 0 1 Homer Bart 2 3 Tom Maggie 1 4 Lisa John 5 0 Homer Bart 2 3 Lisa John 5 0 Homer Bart 2 3 Homer Bart 2 3 Tom Maggie 1 4 How can I assign an unique id to each repeated row? For example: A B C D new_id John Tom 0 1.2 1 Homer Bart 2 3.0 2 Tom Maggie 1 4.2 3 Lisa John 5 0 4 Homer Bart 2 3 5 Lisa John 5 0 4 Homer Bart 2 3.0 2 Homer Bart 2 3.0 2 Tom Maggie 1 4.1 6 I know that I can use duplicate to detect the duplicated rows, however I can not visualize were are reapeting those rows. I tried to: duplicate df.assign(id=(df.columns).astype('category').c...

Proper way to extend Python class I'm looking to extend a Panda's DataFrame, creating an object where all of the original DataFrame attributes/methods are in tact, while making a few new attributes/methods available. I also need the ability to convert (or copy) objects that are already DataFrames to my new class. What I have seems to work, but I feel like I might have violated some fundamental convention. Is this the proper way of doing this, or should I even be doing it in the first place? import pandas as pd class DataFrame(pd.DataFrame): def __init__(self, df): df.__class__ = DataFrame # effectively 'cast' Pandas DataFrame as my own the idea being I could then initialize it directly from a Pandas DataFrame, e.g.: df = DataFrame(pd.read_csv(path)) You're mixing up inheritance and composition. Your DataFrame class both "has a" and "is a" pd.DataFrame . – mypetlion Jun 2...

Load entire dict into a particular dataframe cell df = columnA columnAB row1 xxx row2 yyy row3 zzz row4 xyx expected df = columnA columnAB columnB row1 xxx [('A1(80)', ['BB11', 'A11', 'A21']), ('B1(70)', ['CC55', 'HH21']), ('C1(60)', ['KK88'])] row2 yyy row3 zzz row4 xyx from collections import defaultdict d1, d2 = defaultdict(int), defaultdict(list) d = {'A1BB11': 10, 'B1CC55': 20, 'A1A11': 30, 'A1A21': 40, 'B1HH21': 50, 'C1KK88': 60 } for k, v in d.items(): prefix = k[:2] d1[prefix] += v d2[prefix].append(k[2:]) final = {'{}({})'.format(k, d1[k]): v for k, v in d2.items()} print(final) # {'A1(80)': ['BB11', 'A11', 'A21'], # 'B1(70)': ['CC55', 'HH21'], # 'C1(60)': ['KK88']} Now ...

Read CSV file with features and labels in the same row in Tensorflow I have a .csv file with around 5000 rows and 3757 columns. The first 3751 columns of each row are the features and the last 6 columns are the labels. Each row is a set of features-labels pair. I'd like to know if there are built-in functions or any fast ways that I can: Basically I want to train a DNN model with 3751 features and 1 label and I'd like the output of the parsing function be fed into the following function for training: train_input_fn = tf.estimator.inputs.numpy_input_fn( x={"x": np.array(training_set.data)}, y=np.array(training_set.target), num_epochs=None, shuffle=True) I know some functions like "tf.contrib.learn.datasets.base.load_csv_without_header" can do similar things but it is already deprecated. 2 Answers 2 You could look into tf.data.Dataset 's input pi...

PyQt - Column of Checkboxes in a QTableView I am dynamically creating a QTableView from a Pandas dataframe. I have example code here. I can create the table, with the checkboxes but I cannot get the checkboxes to reflect the model data, or even to change at all to being unchecked. I am following example code from this previous question and taking @raorao answer as a guide. This will display the boxes in the table, but non of the functionality is working. Can anyone suggest any changes, or what is wrong with this code. Why is it not reflecting the model, and why can it not change? Do check out my full example code here. Edit one : Update after comment from Frodon : corrected string cast to bool with a comparison xxx == 'True' class CheckBoxDelegate(QtGui.QStyledItemDelegate): """ A delegate that places a fully functioning QCheckBox in every cell of the column to which it's applied """ def __init__(self, parent): QtGu...

IndexError: single positional indexer is out-of-bounds for my Pandas Dataframe I am trying to perform certain addition operation to the columns in my dataframe and then store the values within the same dataframe. My DataFrame has 3796 rows and 11 columns containing data from the year 1928 till 2000, with weekly data. I want to take a weighted average of the data points(depending upon their dates) and store the values in the same dataframe. My code is as follows: import pandas as pd import numpy as np df=pd.read_excel("D:SUMMER INTERNSHIP CONCORDIAUpdated Work hereKitsim Reservoir (Number 220)Book3_Final.xlsx") a=0 b=0 k=20 p=3 for z in range(3,11): a=0 b=0 for x in range(1928,2000): for y in range(0,12): if df.iloc[a,1]==1: df.iloc[b,k]=(7*df.iloc[a,z]+7*df.iloc[a+1,z]+7*df.iloc[a+2,z]+7*df.iloc[a+3,z]+3*df.iloc[a+4,z])/31 a=a+4 b=b+1 if df.iloc[a,1]==2: ...

Remove categories having count as 0 in pandas groupby I want to remove categories having count as 0 after pandas value_counts function() My data is as follows: categories: Index(['Average', 'Good', 'Poor', ,'VeryGood', 'VeryPoor'], dtype='object') Output of value counts: score Frequency VG 21 G 15 A 63 P 27 VP 0 My result should be as score Frequency VG 21 G 15 A 63 P 27 I want to store this in a dataframe and plot bargraph of this. I don't want to show VP in graph as it's count is 0 and hence eliminate that category My code: quality_scores=quality.SCORE.value_counts() quality_scores=pd.Series.to_frame(quality_scores) quality_scores=quality_scores.rename(columns={'SCORE': 'Frequency'}) quality_scores['Score']=quality_scores.index qualit...

Ordering and Formatting Dates on X-Axis in Seaborn Bar Plot This seems so simple, but for the life of me I can't figure it out. I am new to Python and Seaborn, and I am doing all this online at PythonAnywhere. All I am trying to do is create a simple barplot in seaborn, with dates ordered properly (that is, ascending from left to right), on the x-axis. When I try this: import matplotlib.pyplot as plt import matplotlib.dates as mdates import datetime import pandas as pd import seaborn as sns emp = pd.DataFrame([[32, "5/31/2018"], [3, "2/28/2018"], [40, "11/30/2017"], [50, "8/31/2017"], [51, "5/31/2017"]], columns=["jobs", "12monthsEnding"]) fig = plt.figure(figsize = (10,7)) sns.barplot(x = "12monthsEnding", y = "uniqueClientExits", data = emp, estimator = sum, ci = None) fig.autofmt_xdate() plt.show() I get this: Nice looking bar graph but with the dates ordered descending from ...

Is there a better method than mapping str to float then mapping to int? I need to merge two data frames. In df_A the key is an int. In df_B the key is a string ending in .0 e.g. '10003.0'. I would like to convert the string in df_B to an int for merging. Is there a better way than mapping twice as seen below? df_B['key'].map(float).map(int) The syntax seems awkward to me. Is there a better solution? def floatint(x): return int(float(x)) .. .map(floatint) ? – user2864740 Jun 29 at 17:03 def floatint(x): return int(float(x)) .map(floatint) 1 Answer 1 You can using to_numeric to_numeric pd.to_numeric(df_B['key'],downcast='integer') It does work, but I actually timed slightl...

select index value from groupby on a pandas dataframe in python I have the following dataframe: df = pd.DataFrame({'place' : ['A', 'B', 'C', 'D', 'E', 'F'], 'population': [10 , 20, 30, 15, 25, 35], 'region': ['I', 'II', 'III', 'I', 'II', 'III']}) And it looks like this: place population region 0 A 10 I 1 B 20 II 2 C 30 III 3 D 15 I 4 E 25 II 5 F 35 III I would like to select the place with the smallest population from the region with the highest population. df.groupby('region').population.sum() Returns: region I 25 II 45 III 65 Name: population, dtype: int64 But I have no clue how to proceed from here (using .groupby / .loc / .iloc) Any suggestion? 2 Answers ...

Applying lambda on Python DataFrame I am having a Python Pandas DataFrame like >>> df classification like 0 flower 1 1 flower 0 2 flower 0 3 adventure 1 4 adventure 1 I want to create an output DataFrame like >>> df classification like liked 0 flower 1 True 1 flower 0 False 2 flower 0 False 3 adventure 1 True 4 adventure 1 True I am "apply"ing the Python lambda function on the input DataFrame as follows: >>> df['like'].apply(lambda x: x == 1) But I am getting all 'False' under the 'liked' column >>> df classification like liked 0 flower 1 False 1 flower 0 False 2 flower 0 False 3 adventure 1 False 4 adventure 1 False Any quick suggestions will be helpful. >>> df['like'].astype(int) 0 1 1 0 2 0 3 1 4 1 Name: like, dtype: int...

Bar plot with x and y axis Now i am getting the bar chart as i attached above, kindly let me know why i am getting range in x-axis. Where is the mistake This is sample dataset. Plant Country Fault Level Type Fault Location Fault_loss 0 001 ESPP1 TH All Plant Internal PV Plant Incidents NaN 2.5 1 001 ESPP1 TH All Plant External Grid Forced Outage NaN 1.3 2 001 ESPP1 TH All Plant External Grid Forced Outage NaN 0.3 3 001 ESPP1 TH All Plant External Grid Forced Outage NaN 31.9 4 001 ESPP1 TH All Plant External Grid Forced Outage NaN 0.3 5 001 ESPP1 TH All Plant External Grid Forced Outage NaN 0.1 6 001 ESPP1 TH All Plant External Grid Forced Outage NaN 0.1 7 001 ESPP1 TH All Plant External Grid Forced Outage ...

Creating python function to create categorical bins in pandas I'm trying to create a reusable function in python 2.7(pandas) to form categorical bins, i.e. group less-value categories as 'other'. Can someone help me to create a function for the below: col1, col2, etc. are different categorical variable columns. ##Reducing categories by binning categorical variables - column1 a = df.col1.value_counts() #get top 5 values of index vals = a[:5].index df['col1_new'] = df.col1.where(df.col1.isin(vals), 'other') df = df.drop(['col1'],axis=1) ##Reducing categories by binning categorical variables - column2 a = df.col2.value_counts() #get top 6 values of index vals = a[:6].index df['col2_new'] = df.col2.where(df.col2.isin(vals), 'other') df = df.drop(['col2'],axis=1) 1 Answer 1 You can use: df = pd.DataFrame({'A':list('abcdefabcdefabffeg...