Series

Series are:

one-dimensional
labeled arrays
of any data type
Or, said differently...

a sequence of values
with associated labels
 dtype('O')

numpy expects homogenous ("same type") data:

...but strings are variable-length
SOLUTION:
 Boolean Masks
used to index select items at scale
work with [] and .loc
need to be same length as series

pd.Series(['A', 'B', 'C'])[[True, False, True]]

0 A
2 C
dtype: object
List argument:

pd.Series(data=['this', 'is', 'fun'])

Dict argument:

pd.Series(data={0: 'this', 1: 'is', 2: 'fun'})

Also valid series:

pd.Series(data=0) pd.Series(data='weather')
 argument

pd.Series(data=students)

parameter
 Indexing With Callables

used for highly customizable indexing

work with [], .loc and .iloc
a single-argument function that returns indexing output

a list of labels no slice or boolean

mask support
list of booleans
a slice, etc
Methods vs Attributes

Method is a function bound to the object.

Attribute is a variable bound to the object.

 Selection By Label
Approach Example Comment

slices, callables,
[] idx'ing series['label'] boolean masks

slices, callables,
.loc[] series.loc['label'] boolean masks

no slice or boolean
dot access series.label mask support

no slice support;
.get() series.get('label') provides default;
forgiving
 Selection By Position
Approach Example Comment

slices, callables,
[] idx'ing series[0] boolean masks

slices, callables,
.iloc[] series.iloc[0] boolean masks

no slice or boolean
dot access series.0 mask support

no slice support;
.get() series.get(0) provides default;
forgiving
 What is a csv?
COMMA-SEPARATED VALUES (.CSV) FILE

a type of text file containing values delimited by comma

 Bools As Ints

True 1

False 0

The bool type inherits from (is a subclass of) int

bool -> int -> object

 Median
The middlemost element in a sorted list of numbers.

10, 11, 12, 13, 14, 15, 16

10, 11, 12, 13, 14, 15, 16, 17

(13+14) / 2 = 13.5
 diff()
the first discrete element-wise difference in a series

ser.diff(periods=1)
 Dropping Or Filling NAs
.dropna(): excludes NAs from the series

.fillna(): replaces NAs with something else

Note: both methods return a copy of the series

unless

ser.fillna('new value', inplace=True)

 Index by min/max
idxmin(): returns the label of the row with minimum value

idxmax(): returns the label of the row with maximum value

Note: if multiple min/max values,

only the first label is returned
 Sequential vs Vectorized Ops
vectorization: running operations on entire arrays

func()
sequential

vectorized
func() func()

func()
 Series Accounting
.size: number of elements in the series
series.size # 193

.count(): number of non-null elements

series.coun() # 162

.isna().sum(): number of null elements

series.isna().sum() # 31
 Size and Shape
.size: number of elements in the series
series.size # 193
.shape: tuple of the dimensions
for a series: (1D) shape, i.e. length for series
series.shape # (193, )
len(): python built-in function
len(series) # 193
sort_values() & sort_index()
sort_values(): returns a new series, sorted by values

sort_index(): returns a new series, sorted by index labels

Default Params: asending=True,

inplace=False
na_position='last'
kind='quicksort'
 Transforms
update(): modifies series values in place using another series
ser.update(other_series)
apply(): applies function (or ufunc) on each series value
most
ser.apply(np.sqrt) flexible

map(): subs series values with others from a function, series,

or dict more input
types
ser.map({'old_value' : 'new_value'})
 value_counts()
a sorted series containing unique values and their counts

ser.value_counts( sort=True,
ascending=False,
dropna=True,
normalize=False )
 Variance
the average of squared differences from the mean

mean

sum of
 DATAFRAMES
FIRST KEY CONCEPT

dataframes have two

dimensions: labeled
indices and columns
 DATAFRAMES
SECOND KEY CONCEPT

each column in a
dataframe is a series
 DATAFRAMES
THIRD KEY CONCEPT

unlike series,
dataframes could be
heterogenous

dtype object int64 bool

 Our Data Prep Process

COLLECT CREATE RENAME REPLACE &

UNITS MAPPER DF CONVERT
isolate the units create a dictionary of rename the column replace all the units
from each nutrition key:value pairs labels of nutrition from the dataframe
column label containing the old datafram values and convert
labels and the new values to floats
 Our Data Prep Process

COLLECT CREATE RENAME REPLACE &

UNITS MAPPER DF CONVERT
isolate the units create a dictionary of rename the column replace all the units
from each nutrition key:value pairs labels of nutrition from the dataframe
column label containing the old datafram values and convert
labels and the new values to floats
 SINGLE-PURPOSE
unlike .loc or .iloc, .at and .iat are only
used for accessing single values
Why use
.at or .iat? FASTER
because of the lack of overhead, they are
much more performant for their isolated
use-case
 dropna() with subset
df.dropna(axis=0, subset=['gender'])
but only look at gender

drop DF.DROPNA()
rows
removes columns or rows with
missing values

SUBSET
restricts or localizes the method
application to specific
orthogonal labels
 MORE WAYS TO DATAFRAME

dict of tuples dict of dicts

like dict of lists, but with tuples key:value pairs with column names as
keys and index-labeled key:value pairs
column-wise containing values

column-wise

dict of series list of dicts

a continuation of key concept #2 list of key:value pairs containing colum
labels and values
column-wise
row-wise
 RANGE VS INT64INDEX

RangeIndex is a special case of Int64Index

both are immutable, sequences of numbers

RangeIndex is an optimized alternative

pd.RangeIndex(start=0, stop=8789, step=1)

APPLY MNEMONIC applies a function to a dataframe

DF.APPLY()

Is aggregation required?

yes no

DF.AGG() DF.TRANSFORM()
 Binary (or bitwise) Operators
OPERATOR WHAT IS EXAMPLE

| or True | False -> True

& and True & False -> False

^ xor True ^ False -> True

complement
~ ~True -> -2
Comparators
COMPARISON OPERATOR PANDAS METHOD

< .lt() SUPPORT

FI LL_VALUE

≤ .le()
> .gt()
≥ .ge()

== .eq()
 players.duplicated( WHAT COUNTS AS A DUPLICATE?

subset=['name', 'age'],
DEFAULT CUSTOM

keep='first') records with repeating values could be changed to a smaller

across all columns group of attributes using the

)(DETACILPUD

subset paramter

WHICH IS THE ORIGINAL?

DEFAULT CUSTOM

the first occurrence is could be changed to "first",

marked as the original "last" or "neither" using the

keep parmeter
 fillna() axes and methods
FILL DIRECTIONS
AXIS=1
METHOD=FFILL

AXIS=0 AXIS=0
METHOD=FFILL METHOD=BFILL

AXIS=1
METHOD=BFILL
lookup(): another way to fancy index

players.lookup([450], ['age'])

array([30])
pandas

memory FLOATS INTS OBJECT

layout 3 cols
9 cols 7 cols
to pop() or not...
players.pop('age')

3 POINTS TO CONSIDER

pop() works on a single column at a time

pop() returns the removed ('popped') column as a series

pop() modifies the underlying dataframe (operates inplace)

Selection Terminology Recap
OPERATOR WHAT IS

players.loc[0:2] slicing

players.loc[players.age > 37] boolean masking

players.loc[132, 'name'] basic (label-based) indexing

fancy
players.loc[[0, 132], ['name', 'market_value']]
indexing
 Two's Complement
VER
Y IM
P ORT
ANT

BIT INTEGER A NUMBER FORMAT

00000000 0
A system for representing signed
00000001 1
integers in computers. Using x bits we
00000010 2 could represent 2^x numbers.

11111111 -1
11111110 -2 For example, 32 bits represent

11111101 -3 4294967296 numbers, 64 bits

18446744073709551616, and so on.

 Two's Complement
operator
BIT INTEGER

^
00000000 0
00000001 1 inverts the bits
00000010 2

11111111 -1
11111110
11111110 -2
11111101 -3
VECTORIZATION
o p e ra t io n
c o m p le t e
in 2 c y c le s
s , in s t e a d
of 6

CPU
GPU

Made possibly by SIMD at the processor-level

Results in operations that are multiple times faster!
Supported by NumPy, and by extension, pandas
copy view
A "COPY" OF THE DATA A "WINDOW" INTO THE DATA
DEUNITNOC YPOC SV WEIV
HOW DO WE TELL?
2-POINT RULE:

pandas loves to give us copies, but

if we use loc/iloc or at/iat, we are guaranteed to

get a view
what's the
difference? ...almost identical, but:

df.append()
.append() is a DataFrame
instance method

pd.concat()
.append() only operates along
the index axis
 concat()

+ =

glues data sets together

a structure-focused operation
 merge()

1 a v 7 1 a 9
3
9
b
c
+ r
a
4
9
= 3
9
b
c
1
2
b 1
c 2
k 3

combines data sets together based on the content they share

much more flexible than .concat()!
 pd merge

how='inner' how='outer'

+ = + =

only the common keys are selected all keys are selected
similar to set intersection similar to set union
 Join Cardinalities

1-1 1-M M-M

eg: person <-> DNA book - pages book - author

Dual-sided uniques One-sided uniques Dual-sided non-uniques

one of the merge objects both merge objects contain non-
both merge objects contain
contain non-unique values unique values
unique values in the respective
key
in the resulting pd.merge() the in the resulting pd.merge() the
records are repeated M times records are repeated M x M times
 pd merge

how='left' how='right'

+ = + =

left keys are selected right keys are selected

 Always consider sorting
the index Advantages
:NOITADNEMMOCER

improves retrieval performance, which

becomes significant
- for large dataframes, or
- frequent retrieval

enables slicing syntax

overall a good practice when working with

tabular data representations, including
pandas, Excel, SQL, etc
 some of the components that

MULTIINDEX INTERNALS
make up MultiIndex objects,
also known as hierarchical
indices in pands

LEVELS NAMES LEVSHAPE VALUES

a list of lists containing a list containing the names a tuple containing the Use visual charts to
each label value for each of each level length of each level communicate info more
of the levels in the effectively.
MultiIndex

L0 L1
PANEL
deprecated since pandas v0.22
VS
- prefer df.MultiIndex for new projects
MULTIINDEX DF
- many of the same pandas concepts apply
for representing
hierarchical data - older docs still available online for panel
 split data into groups
Split

Apply

Combine
 apply .sum()
Split

Apply

NA_Sales NA_Sales
Combine
0.75 0.80
 combine the output
Split
NA_Sales NA_Sales
0.75 0.80

Apply

Combine