Live training:
Cleaning Data in Python
About me
ADEL NEHME
Content Developer
The data science workflow
Report or
Dashboard
Model in
Production
Access Explore and Extract
Data Process Data Insights
The data science workflow
Report or
Dashboard
Model in
Production
Access Explore and Extract
Data Process Data Insights
Why do we have dirty data?
Human error Technical error
The dataset
Airbnb data
listing_id: Unique identifier for a listing
Name: Description used for a listing
Host_id: Unique identifier for each host
Host_name: Name of host
Airbnb data featuring listings in New York
Neighbourhood_full: Burrough and neighbourhood
Coordinates: Latitude, Longitude
Room_type: Type of room
Price: Price per night
Number of reviews: Number of reviews so far
Last_review: Date of last review
Reviews_per_month: # of reviews per month
Availability_365: Days available per year
Rating: Average rating (0 to 5)
Number_of_stays: Number of stays so far
5_stars: Percentage of ratings that is 5_stars
Listing_added: Date listing added to site
The end result
Technologies
Popular open source data analysis tool for tabular data
Open source plotting library for 2-D visualizations
Open source plotting library built on top of matplotlib
Technologies
Popular open source computing tool for arrays
missingno Open source plotting library for missing data
datetime Package for easy date data manipulation
Technologies
❗❗Requires a gmail account to edit ❗❗
Session outline
1 Introduction
2 Importing our dataset
3 Diagnosing our data problems
4 Q&A
5 Our to do list
6 Data cleaning
7 Q&A
8 Recap & closing notes
9 Take home question
Notebook
Notebook
Session outline
1 Introduction
2 Importing our dataset
3 Diagnosing our data problems
4 Q&A
5 Our to do list
6 Data cleaning
7 Q&A
8 Recap & closing notes
9 Take home question
The data science workflow - revisited
Report or
Dashboard
Model in
Production
Access Explore and Extract
Data Process Data Insights
The data science workflow - revisited
Report or
Dashboard
Model in
Production
Access Explore and Extract
Data Process Data Insights
The data science workflow - revisited
Report or
Dashboard
Model in
Production
Access Explore and Extract
Data Process Data Insights
The data science workflow - revisited
Report or
Dashboard
Model in
Production
Access Explore and Extract
Data Process Data Insights
Coming soon
Check out our upcoming webinars!
Live training: Data Viz with ggplot2 📈
Register here
DCVirtual: Webinar week 🎉
DataCamp for Enterprise: What’s New in Q2 2020 🌅
Register here
Take home question
Pick one of the following:
1) What is the average price of listings by borough? Visualize your results with a bar plot!
2) What is the average availability in days of listings by borough? Visualize your results with a bar plot!
3) What is the median price per room type in each borough? Visualize your results with a bar plot!
4) Visualize the number of listings over time.
Functions that should/could be used:
● .groupby() and .agg()
● sns.barplot(x = , y = , hue = , data = )
● sns.lineplot(x = , y = , data = )
● .dt.strftime() for extracting specific dates from a datetime column
Bonus points if you finish more than one question
Submission details:
● Share with us a code snippet with your output on LinkedIn, Twitter or Facebook
● Tag us on `@DataCamp` with the hashtag `#datacamplive`
Recap of the functions used
Diagnosis functions Description Treatment functions Description
import pandas as pd Imports the pandas package with the alias pd Replaces one string with another for each row of a str
.str.replace(“”, “”) column
.head() Prints the header of a DataFrame
.str.split(“”, expand = True) Splits a string column into two based on input
.dtypes Gets the data types of each column in a
DataFrame .astype() Converts a column to a datatype of choice
.info() Returns a # observations, data types and missing pd.to_datetime() Converts a date column to datetime
values per column
.str.lower() Lowercases each row in a str column
.describe() Returns statistical distribution of numeric value
in a DataFrame
.str.strip(“”) Removes a pattern from each row of an str column
.isna().sum() Returns # of missing values per column
.replace() Replace values for others in a column
sns.distplot() Plots distribution of one variable
.fillna() Fills missing values of a column with a value of your
choice
msno.matrix() Visualizes missingness matrix
.drop_duplicates() Drops duplicates
msno.barplot() Visualizes missingness barplot
.duplicated(subset = , keep = ) Lets you find duplicates in a DataFrame based on
all or subset of columns
