The UOB Python Lectures:

Part 3 - Python for Data

Analysis
Hesham al-Ammal
University of Bahrain

Thursday, April 4, 13
 Small Data
BIG Data

Thursday, April 4, 13
 Data Scientist’s Tasks
Interacting with the outside Preparation
world Cleaning, munging, combining, normalizing,
Reading and writing with a variety of file reshaping, slicing and dicing, and
formats and databases. transforming data for analysis.

Modeling and computation Transformation

Connecting your data to statistical models, Applying mathematical and statistical
machine learning algorithms, or other operations to groups of data sets to derive
computational tools new data sets. For example, aggregating a
large table by group variables.

Presentation
Creating interactive or static graphical
visualizations or textual summaries

Thursday, April 4, 13
 Example 1: .usa.gov data
from bit.ly
JSON: JavaScript Object Notation

Python has many JSON libraries

In [15]: path = 'ch02/
usagov_bitly_data2012-03-16-1331923249.txt'
In [16]: open(path).readline()

We’ll use list comprehension to put the data in

a dictionary
import json
path = 'usagov_bitly_data2012-03-16-1331923249.txt'
records = [json.loads(line) for line in open(path)]
record[0]

Thursday, April 4, 13
 In [19]: records[0]['tz'] Unicode strings and notice how
Out[19]: u'America/New_York' dictionaries work

In [20]: print records[0]['tz']

America/New_York

Counting timezones in Python

Let’s start by using Pyhton only and list comprehension
In [6]: time_zones = [rec['tz'] for rec in records]
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-6-db4fbd348da9> in <module>()
----> 1 time_zones = [rec['tz'] for rec in records]

KeyError: 'tz'
You’ll get an error because not all
records have time zones
Thursday, April 4, 13
 Counting timezones in Python
Solve the problem of missing tz by using an if
In [26]: time_zones = [rec['tz'] for rec in records if 'tz' in rec]
In [27]: time_zones[:10]
Out[27]:
Define a function to def get_counts(sequence):
[u'America/New_York',
count occurrences in a counts = {}
u'America/Denver',
sequence using a for x in sequence:
u'America/New_York',
if x in counts:
u'America/Sao_Paulo', dictionary
counts[x] += 1
u'America/New_York',
else:
u'America/New_York',
counts[x] = 1
u'Europe/Warsaw',
return counts
u'',
u'',
u''] Then just pass the time In [31]: counts = get_counts(time_zones)
zones list In [32]: counts['America/New_York']
Out[32]: 1251
In [33]: len(time_zones)
Out[33]: 3440

Thursday, April 4, 13
 Finding the top 10 timezones
We have to manipulate the dictionary by sorting
def top_counts(count_dict, n=10):
value_key_pairs = [(count, tz) for tz, count in
count_dict.items()]
value_key_pairs.sort()
return value_key_pairs[-n:]

In [35]: top_counts(counts)
Out[35]:
[(33, u'America/Sao_Paulo'),
(35, u'Europe/Madrid'),
(36, u'Pacific/Honolulu'),
(37, u'Asia/Tokyo'),
Then we will have (74, u'Europe/London'),
(191, u'America/Denver'),
(382, u'America/Los_Angeles'),
(400, u'America/Chicago'),
(521, u''),
(1251, u'America/New_York')]

Thursday, April 4, 13
 Let’s do the same thing
in pandas
In [289]: from pandas import DataFrame, Series In [293]: frame['tz'][:10]
In [290]: import pandas as pd Out[293]:
In [291]: frame = DataFrame(records) 0 America/New_York
In [292]: frame 1 America/Denver
2 America/New_York
3 America/Sao_Paulo
4 America/New_York
5 America/New_York
6 Europe/Warsaw
7
8
9
Name: tz

Thursday, April 4, 13
 What is pandas?

pandas : Python Data Analysis Library

an open source, BSD-licensed library providing high-
performance, easy-to-use data structures and data analysis
tools for the Python programming language.

Features:
Effcient Dataframes data structure

Tools for data reading, munging, cleaning, etc.

Thursday, April 4, 13
 To get the counts

In [294]: tz_counts = frame['tz'].value_counts()

In [295]: tz_counts[:10]
Out[295]:
America/New_York 1251
521
America/Chicago 400
America/Los_Angeles 382 To clean missing values
America/Denver 191
Europe/London 74 In [296]: clean_tz = frame['tz'].fillna('Missing')
Asia/Tokyo 37 In [297]: clean_tz[clean_tz == ''] = 'Unknown'
Pacific/Honolulu 36 In [298]: tz_counts = clean_tz.value_counts()
Europe/Madrid 35 In [299]: tz_counts[:10]
America/Sao_Paulo 33 Out[299]:
America/New_York 1251
Unknown 521
America/Chicago 400
America/Los_Angeles 382
America/Denver 191
Missing 120 Remember “data
cleaning”

Thursday, April 4, 13
 To plot the results (presentation)

In [301]: tz_counts[:10].plot(kind='barh', rot=0)

import matplotlib.pyplot as plt

plt.show

Thursday, April 4, 13
 Example 2: Movie Lens
1M Dataset

GroupLens Research (http://

www.grouplens.org/node/73)

Ratings for movies 1990s+2000s

Three tables: 1 million ratings, 6000 users,

4000 movies

Thursday, April 4, 13
 Interacting with the outside Preparation
world Cleaning, munging, combining, normalizing,
Reading and writing with a variety of file reshaping, slicing and dicing, and
formats and databases. transforming data for analysis.

Extract the data from a zip file and load it into

pansdas DataFrames

import pandas as pd

unames = ['user_id', 'gender', 'age', 'occupation', 'zip']

users = pd.read_table('ml-1m/users.dat', sep='::', header=None,names=unames)

rnames = ['user_id', 'movie_id', 'rating', 'timestamp']

ratings = pd.read_table('ml-1m/ratings.dat', sep='::', header=None,names=rnames)

mnames = ['movie_id', 'title', 'genres']

movies = pd.read_table('ml-1m/movies.dat', sep='::', header=None,names=mnames)

Thursday, April 4, 13
 Verify
Preparation
Cleaning, munging, combining, normalizing,
reshaping, slicing and dicing, and
transforming data for analysis.

In [334]: users[:5]

In [335]: ratings[:5]

In [336]: movies[:5]

Thursday, April 4, 13
 Merge Preparation
Cleaning, munging, combining, normalizing,
reshaping, slicing and dicing, and
transforming data for analysis.

Using pandas’s merge function, we first

merge ratings with users then merging that
result with the movies data. pandas infers
which columns to use as the merge (or join)
keys based on overlapping names

Thursday, April 4, 13
 Merge results
In [338]: data = pd.merge(pd.merge(ratings, users), movies)
In [339]: data
Out[339]:
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1000209 entries, 0 to 1000208
Data columns:
user_id 1000209 non-null values
movie_id 1000209 non-null values
rating 1000209 non-null values
timestamp 1000209 non-null values
gender 1000209 non-null values
age 1000209 non-null values
occupation 1000209 non-null values
zip 1000209 non-null values
title 1000209 non-null values
genres 1000209 non-null values
dtypes: int64(6), object(4)

Thursday, April 4, 13
 Example 3: US Baby
Names 1880-2010
United States Social Security Administration
(SSA) http://www.ssa.gov/oact/babynames/
limits.html

Visualize the proportion of babies given a

particular name

Determine the most popular names in each

year or the names with largest increases
or decreases

Thursday, April 4, 13
Thursday, April 4, 13
Thursday, April 4, 13
Thursday, April 4, 13
Thursday, April 4, 13
 Lets do some more
investigations

names[names.name=='Mohammad']

names[names.name=='Fatima']

Thursday, April 4, 13