Chapter -2
Python Pandas
Introduction :
Pandas is one of the most preferred and widely used data
science libraries. It offer efficient data structure which are not
only powerful , but also very convenient and flexible.
In this chapter we will learn :
pivoting, sorting and aggregation.
Descriptive statistics – histogram and quantiles, various
functions, reindexing and altering labels.
Pandas provide two basic data structures – Series and
DataFrame.
Series – It represents a one-dimensional array of indexed
data.
DataFrame – It store data in two-dimensional way.
Some attributes of DataFrame are :
1. index – to display index (row labels) of the DataFrame.
2. columns – to display column labels of the DataFrame.
3. axes – Return a list representing both the axes – row and
columns.
4. size – Return number of elements in DataFame
5. shape – Return dimension of DataFrame
6. values – Return DataFrame in the form of Numpy Array.
Selecting/Accessing a Column :
Selecting/Accessing a Subset from a DataFrame using
Row/Column Names :
Syntax :
<DataFrameObject>.loc[<startrow>:<endrow>]
To access selective columns use :
<DF object>.loc[:,<start column>:<end column>]
To access range of columns from a range of rows , use :
<DataFrameObject>.loc[<startrow>:<endrow>,
<startcolumn>:<endcolumn>]
Obtaining a Subset/Slice from a DataFrame using
Row/Columns Numeric Index/Position
<DF object>.iloc[<start row index>: <end row index>,
<start col index> : <end column index>]
Selecting/Accessing Individual Values :
<DF object>.<column>[<row name or row numeric index>]
Adding and Deleting Columns in DataFrames
Deleting Columns : use del statement to delete a column
Del<DF object>[<column name>]
Descriptive Statistics with Pandas :
Pandas include many useful statistical functions.
Reference dataframe namely sal_df
Functions min() and max() : The min() and max() functions
find out the minimum or maximum values respectively form a
given set of data.
Parameters :
axis : (0 or 1) by default, minimum and maximum is
calculated along axis 0.
Functions mode() , mean(), median() :
mode() – Returns the mode value (i.e., the value that appears
most often) from a set of values.
Parameters :
axis : axis 0 or ‘index’ get mode of each column
axis 1 or ‘columns’ : get mode of each row
mean() – Returns the computed mean(average) form a set of
values.
median() – returns the middle number form a set of numbers.
(2000 10000 12000 13000)
(10000+12000) =22000, 22000/2 = 11000
(2000 6000 7000 7000)
(6000 + 7000) = 13000, 13000/2 = 6500
Functions count( ) and sum( )
count( ) – counts the non-NA entries for each row or column.
sum( ) – returns the sum of values for the requested axis.
Applying Functions on a Subset of Dataframe :
Sometimes , we need to apply a function on a selective column
or row or a subset of the data frame.
Applying Function on a Column of a DataFrame:
To apply a function on a column, write –
<dataframe>[<column name>]
Applying function on Multiple Columns of a DataFrame :
<dataframe>[[<column name>, <column name>,….]]
Applying Function on a row of a DataFrame :
<dateframe>.loc[<row index>, :]
Applying Functions on a range of rows of a DataFrame:
<dataframe>.loc[<start row> : <end row>, :]
Applying functions to a subset of the DataFrame :
<dataframe>.loc[<start row>:<end row>, :
<start column> : <end column>]
Advanced Operations on DataFrame :
Three advanced operations are :
1. pivoting
2. sorting
3. aggregation
1. Pivoting : Pivoting is actually a summary technique that
works on tabular data (i.e., data in rows and columns).
Pivoting technique rearranges the data from rows and columns,
by possibly aggregating data from multiple sources, in a report
form (with rows transferred to columns) so that data can be
viewed in a different perspective.
Real life example :
An online tutoring company maintains its data about tutors and
online classes in the following table.
Using pivot Function :
Cells in the pivoted table which do not have a matching entry
in the original one are set with NaN.
Now change the rows and columns , i.e. the index and columns
arguments
We can skip the values argument:
The above data is for one quarter only. The online tutoring
company has data for the entire year as shown below :
The index i.e. , the rows are specified as ‘Tutor’ and the
columns as ‘Country’. There are multiple entries of tutor which
are very much different for same country.
Consider Tahira’s entries
Tutor Classes Quarter Country
Tahira 28 1 USA
Tahira 36 2 USA
Tahira 24 3 Barzil
Tahira 36 4 Japan
Try to create a row for tutor Tahira from the above data with
columns as Country.
USA Brazil Japan
Tahira 24 36
Using pivot_table( ) function :
Example-1 Considering the tutoring company data, compute
total classes per tutor.
Example-2 considering the tutoring company data, compute
number of countries (count) per tutor.
Example -3 Considering the tutoring company data, compute
total classes by country.
Example-4 Considering the tutoring company data, compute
total classes on two fields, tutor and country wise.
Example 2.5 Considering the tutoring company data, compute
average classes on two fields
Sorting : Sorting refers to arranging values in a particular
order.
sort_values( ) – function arrange the values in ascending or
descending order.
Creating Histogram :
Histogram – A histogram is a plot that lets you discover, and
show the underlying frequency distribution(shape) of a set of
continuous data.
hist( ) function of pandas is used to create histogram.
Consider the following histogram that has been computed using
the following datasets containing age of 20 people.
37 28 38 44 53 69 74 53 35 38 66 46 24 45 92 48 51 62 58 57
Bin Frequency Age included in Bin
20-30 2 28,24
30-40 4 37,38,35,38
40-50 4 44,46,45,48
50-60 5 53,53,51,56,57
60-70 3 69,66,92
70-80 1 74
80-90 0 --
90-100 1 92
Function Application : It means that a function (a library
function or user defined function ) may be applied on a
dataframe in multiple ways:
(a) on the whole dataframe
(b) row-wise or column-wise
(c) on individual elements, i.e. element-wise
for the above mentioned three types of function application,
Pandas offers following three functions :
(a) pipe() – dataframe wise function application.
(b) apply() – row-wise/column-wise function application.
(c) applymap() – individual elements-wise function application.
(a) pipe( ) function : The piping of functions through pipe( )
basically means the chaining of functions in the order they are
executed.
pipe() Example 1 Function add( ) followed by multiply( )
applied on a dataframe.
The apply and applymap() functions :
apply () – apply is a series function, so it applies the given
function to one row or one column of the dataframe (as single
row/columns of a dataframe is equivalent to a series).
Syntax - <dataframe>.apply(<funcname>,axis = 0)
axis 0 or 1 default 0 , axis along with the function is applied.
If axis is 0 or ‘index’ : function is applied on each column
If axis is 1 or ‘columns’ : function is applied on each row.
applymap() – is an element function, so it applies the
given function to each individual elements, separately.
Syntax - <dataframe>.applymap(<funcname>)
To apply, apply() row-wise write :
<dataframe>.apply(<func>, axis = 1)
NOTE – The apply() will apply the function on individual
columns/rows, only if the passed function name is a Series
function. If you pass a single value function, then apply() will
behave like applymap()
Function groupby( ) :
The duplicate values in the same filed are grouped together to
form groups, e.g. for creating Tutor wise groups :
All the rows having Tutor as Tahira will be clubbed to form
Tahira group.
groupby() function - is used to create group for the
duplicate values in the same filed.
The groupby() creates the group internally and does not
display the grouped data by default.
Grouping on Multiple Columns : Create a group for Tutor
and for each tutor group, a Country-wise subgroup :
Example : to get a group having tutor name as ‘Anusha’ and
Country as ‘UK’ write :
Aggregation via groupby () :
agg( ) method – aggregates the data of the dataframe using
one or more operations over the specified axis.
Syntax - <dataframe>.agg(func, axis = 0)
mean :-
36,40,30,32
(36+40+30+32)/4 =
34.5
median :-
30 32 36 40
n/2 = 4/2 = 2
(n/2) + 1 = (4/2) + 1
=3
(32 + 36) / 2 = 34
Sum : (36+40+30+32)
= 138
We may combine the groupby( ) and agg( ) in single command
:
The transform( ) function : This function transforms the
aggregate data by repeating the summary result for each row
of the group and make the result have the same shape as
original data.
Q- What if we want to add this aggregate data to the
dataframe itself?
Reindexing and Altering Labels : The methods provided by
Pandas for reindexing and relabeling are :
1. rename( ) – simply rename the index and/or column labels
in a dataframe.
(ii) reindex( ) – specify the new order of existing indexes
and column labels, and/or also create new indexes/column
labels.
(a) Reordering the existing indexes using reindex( )
Adding indexes :
(iii) reindex_like() – for creating indexes/column-labels
based on other dataframe object.
<dataframe>.reindex_like(other)
Solved Problems :
1. Consider the following code to create two dataframes with
similar values. What will be printed by the code given below ?
Justify your answer.
import pandas as pd
df1 = pd.DataFrame([1,2,3])
df2 = pd.DataFrame([[1,2,3]])
print("df1")
print(df1)
print("df2")
print(df2)
Ans :
