Data Preprocessing
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
Why Data Preprocessing?
Data in the real world is soiled.
– incomplete: lacking attribute values, lacking certain
attributes of interest, or containing only aggregate
data
e.g., occupation=“ ”
– noisy: containing errors or outliers
e.g., Salary=“-10”
– inconsistent: containing discrepancies in codes or
names
e.g., Age=“42” Birthday=“03/07/1997”
e.g., Was rating “1,2,3”, now rating “A, B, C”
e.g., discrepancy between duplicate records
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
What is Data?
Collection of data objects and Attributes
their attributes
Tid Refund Marital Taxable
An attribute is a property or Status Income Cheat
characteristic of an object 1 Yes Single 125K No
– Examples: eye color of a person, 2 No Married 100K No
temperature, etc. 3 No Single 70K No
– Attribute is also known as variable, 4 Yes Married 120K No
field, characteristic, or feature
Objects
5 No Divorced 95K Yes
6 No Married 60K No
A collection of attributes describe 7 Yes Divorced 220K No
an object 8 No Single 85K Yes
– Object is also known as record, 9 No Married 75K No
point, case, sample, entity, or 10 No Single 90K Yes
instance
10
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
Data Quality
What kinds of data quality problems?
How can we detect problems with the data?
What can we do about these problems?
Examples of data quality problems:
– Noise and outliers
– missing values
– duplicate data
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
Noise
Noise refers to modification of original values
– Examples: distortion of a person’s voice when talking on a poor
phone and “snow” on television screen
Two Sine Waves Two Sine Waves + Noise
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
Outliers
Outliers are data objects with characteristics that are
considerably different than most of the other data objects
in the data set
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
Missing Values
Reasons for missing values
– Information is not collected
(e.g., people decline to give their age and weight)
– Attributes may not be applicable to all cases
(e.g., annual income is not applicable to children)
Handling missing values
– Eliminate Data Objects
– Estimate Missing Values
– Ignore the Missing Value During Analysis
– Replace with all possible values (weighted by their probabilities)
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
Duplicate Data
Data set may include data objects that are duplicates, or
almost duplicates of one another
– Major issue when merging data from heterogeneous sources
Examples:
– Same person with multiple email addresses
Data cleaning
– Process of dealing with duplicate data issues
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
-- … Why Preprocess the Data
Reason for data cleaning
– Incomplete data (missing data)
– Noisy data (contains errors)
– Inconsistent data (containing discrepancies)
Reasons for data integration
– Data comes from multiple sources
Reason for data transformation
– Some data must be transformed to be used for mining
Reasons for data reduction
– Performance
No quality data no quality mining results!
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
Major Tasks in Data Preprocessing
1.Data cleaning
– Fill in missing values, smooth noisy data, identify or remove
outliers, and resolve inconsistencies
2.Data integration
– Integration of multiple databases, data cubes, or files
3.Data transformation
– Normalization and aggregation
4.Data reduction (Sampling, dimensionality reduction,
feature subset selection)
– Obtains reduced representation in volume but produces the same
or similar analytical results
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
5.Data discretization
– For classification algorithms sometimes it is required that data
should be in the form of categorical attributes
– Algo. That find association patterns require that the data be in the
form of binary attributes.
– Thus it is required to transform a continuous attribute to a
categorical attribute( discretization).
– Part of data reduction but with particular importance, especially for
numerical data
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
Forms of Data Preprocessing
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
1.Data Cleaning
Data cleaning tasks
– Fill in missing values
– Identify outliers and smooth out noisy data
– Correct inconsistent data
– Resolve redundancy caused by data integration
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
1.Data Cleaning
: How to Handle Missing Data?
Ignore the tuple: usually done when class label is missing
(assuming the tasks in classification)—not effective
unless the tuple contains several attributes with the
missing values
Fill in the missing value manually- not feasible for large
datasets and time- consuming
Fill in it automatically with
– a global constant : e.g., “unknown”, a new class?!
– the attribute mean
– the most probable value: inference-based such as Bayesian
formula or regression or decision tree induction
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
1.Data Cleaning : How to Handle Noisy Data?
Noise- a random error or variance in a measured variable.
Incorrect attribute values may due to
– faulty data collection
– data entry problems
– data transmission problems
– data conversion errors
– Data decay problems
– technology limitations, e.g. buffer overflow or field size limits
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
1.Data Cleaning : How to Handle Noisy Data?
Methods
Binning
– first sort data and partition into (equal-frequency) bins
– then one can smooth by bin means, smooth by bin median,
smooth by bin boundaries, etc.
Regression
– smooth by fitting the data into regression functions
Clustering
– detect and remove outliers
Combined computer and human inspection
– detect suspicious values and check by human (e.g., deal with
possible outliers)
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
1.Data Cleaning : Regression
•Data can be smoothed by y
fitting the data to a
function such as with
regression. Y1
•Linear regression involves
finding the ‘best’ line to fit
Y1’ y=x+1
2 variables.
X1 x
•Also, it is possible to
predict one variable using
the other variable.
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
1.Data Cleaning : Cluster Analysis
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
2. Data Integration
Data integration:
– Combines data from multiple sources into a coherent store
Schema integration: e.g., A.cust-id B.cust-#
– Integrate metadata from different sources
Entity identification problem:
– Identify real world entities from multiple data sources, e.g., Bill
Clinton = William Clinton
Detecting and resolving data value conflicts
– For the same real world entity, attribute values from different
sources are different
– Possible reasons: different representations, different scales
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
Data Integration
: Handling Redundancy in Data Integration
Redundant data occur often when integration of multiple
databases
– Object identification: The same attribute or object may have
different names in different databases
– Derivable data: One attribute may be a “derived” attribute in
another table, e.g., annual revenue
Redundant attributes may be able to be detected by
correlation analysis
Careful integration of the data from multiple sources may
help reduce/avoid redundancies and inconsistencies and
improve mining speed and quality
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
Data Transformation
Smoothing: remove noise from data
Aggregation: summarization, data cube construction
Generalization: concept hierarchy climbing
Normalization: scaled to fall within a small, specified
range
– min-max normalization
– z-score normalization
– normalization by decimal scaling
Attribute/feature construction
– New attributes constructed from the given ones
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
Data Reduction Strategies
Why data reduction?
– A database/data warehouse may store terabytes of data
– Complex data analysis/mining may take a very long time to run on the
complete data set
Data reduction
– Obtain a reduced representation of the data set that is much smaller in
volume but yet produce the same (or almost the same) analytical
results
Data reduction strategies
– Aggregation
– Sampling
– Dimensionality Reduction
– Feature subset selection
– Feature creation
– Discretization and Binarization
– Attribute Transformation
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
Data Reduction : Aggregation
Combining two or more attributes (or objects) into a single
attribute (or object)
Purpose
– Data reduction
Reduce the number of attributes or objects
– Change of scale
Cities aggregated into regions, states, countries, etc
– More “stable” data
Aggregated data tends to have less variability
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
Data Reduction : Sampling
Sampling is the main technique employed for data selection.
– It is often used for both the preliminary investigation of the data
and the final data analysis.
Statisticians sample because obtaining the entire set of data
of interest is too expensive or time consuming.
Sampling is used in data mining because processing the
entire set of data of interest is too expensive or time
consuming.
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
Thank You
PES Modern college of Engineering, MCA Department Prepared by : Dr.Prakash Kene
