Data Preprocessing
Data is often called the “new oil,” but just like crude oil, raw data is rarely useful in its original
state. It is often noisy, incomplete, inconsistent, or simply too large to be processed directly.
To get accurate and meaningful results from data mining, we must first prepare the data
properly. This preparation process is called data preprocessing.
Data preprocessing plays a critical role in ensuring that the knowledge discovered from data
is reliable and valid. If we try to apply data mining techniques on raw data without cleaning
and transforming it, we may end up with wrong conclusions. For example, imagine a retail
database where some customers have missing addresses, some transactions have been
recorded twice, and some prices are wrongly entered. Mining such data would give
misleading results about customer behavior or sales patterns. Hence, preprocessing is the
foundation for successful data mining.
1. Importance of Data Preprocessing
Before we go into the steps, it is important to understand why preprocessing is required:
● Data quality issues – Real-world data may have errors, duplicate entries, missing
values, and inconsistencies.
● Large volume – Data warehouses and big databases often contain terabytes or
petabytes of records, which are too big for direct mining.
● Heterogeneity – Data may come from different sources such as spreadsheets,
databases, sensors, or the web, each with different formats.
● Improving accuracy – Well-prepared data improves the performance of mining
algorithms, leading to better classification, clustering, or prediction results.
●
● Efficiency – Preprocessing reduces data size and complexity, so mining becomes
faster and more scalable.
Thus, preprocessing acts like the cleaning and organizing stage before analysis, making
sure the data is consistent, accurate, and ready for knowledge discovery.
2. Data Cleaning
The first major task in preprocessing is data cleaning, which deals with missing values, noisy
values, and inconsistencies.
(a) Missing Values
Data often has blank or unknown fields. For example, a customer’s age might not be
recorded, or income data may be absent. Methods to handle missing values include:
● Ignoring the record (only if the dataset is large enough).
● Filling with a global constant (like “Unknown”).
● Replacing with the mean, median, or mode of the attribute.
● Using predictive models (regression, decision trees, or k-nearest neighbor) to
estimate the missing value.
(b) Noisy Data
Noise refers to random errors or variations in data. Example: typing mistakes, wrong
measurements, or outliers. Methods to reduce noise include:
● Binning: Sorting data into bins and replacing values by bin mean, median, or
boundary.
● Regression: Fitting data into a regression function and smoothing values.
● Clustering: Grouping similar records and identifying outliers as noise.
(c) Inconsistencies
Data from different sources may have different formats or spellings. For example,
“Male/Female” vs. “M/F,” or different currency notations. Cleaning detects and corrects such
conflicts to maintain consistency.
3. Data Integration
The second task is data integration, which combines data from multiple sources like
databases, flat files, or data cubes.
Key problems in integration include:
● Entity identification – Matching records that refer to the same entity but have different
names (e.g., “cust_ID” vs. “customer_no”).
● Schema integration – Combining attributes with different names but same meaning.
● Redundancy removal – If the same information is stored in two sources, it must be
detected and merged. Correlation analysis is often used to detect redundancy.
● Value conflict resolution – If two sources record different values for the same
attribute, a strategy is needed to resolve conflicts.
Integration provides a unified view of data, which is essential for data mining in large
organizations where data comes from many departments and systems.
4. Data Reduction
Since real-world data is often huge, data reduction techniques are applied to make the
dataset smaller but still representative of the original. The goal is to reduce the volume while
preserving essential patterns.
Methods of Data Reduction:
1. Data Cube Aggregation – Summarizing data at higher abstraction levels. For example,
instead of storing daily sales, we can keep monthly or yearly totals.
2. Dimensionality Reduction – Reducing the number of attributes using techniques such as
Principal Component Analysis (PCA) or Singular Value Decomposition (SVD).
3. Attribute Subset Selection – Choosing only the most relevant attributes while discarding
irrelevant or redundant ones. Techniques like decision tree induction and stepwise
regression help in this.
4. Numerosity Reduction – Replacing large datasets with models. Example: using regression
equations or clusters to represent the data.
5. Sampling – Selecting a representative sample of the data for analysis instead of the entire
dataset.
6. Data Compression – Using encoding schemes like wavelet transforms to store data in a
compact format.
Reduction improves efficiency, saves storage, and speeds up mining algorithms without
losing much accuracy.
5. Data Transformation
Sometimes, even after cleaning, the data may not be in the right format for mining. Data
transformation modifies data into suitable forms.
● Normalization – Scaling data into a specific range.
● Min-Max Normalization: scales values between 0 and 1.
● Z-score Normalization: rescales using mean and standard deviation.
● Decimal Scaling: moves decimal point based on maximum absolute value.
● Smoothing – Removing noise using techniques like regression or binning.
● Aggregation – Summarizing data (e.g., converting transaction-level data into
customer-level totals).
● Generalization – Replacing low-level data with higher-level concepts. For example,
“23 years” can be generalized to “young.”
Transformation ensures that attributes are comparable and suitable for mining tasks such as
clustering or classification.
6. Data Discretization
Some data mining algorithms work better on categorical data instead of continuous values.
Data discretization converts continuous attributes into discrete intervals.
Methods include:
● Binning – Dividing values into equal-width or equal-frequency intervals.
● Histogram analysis – Using data distribution to define bins.
● Clustering – Grouping values and treating each cluster as a category.
● Decision tree analysis – Splitting data into intervals based on class labels.
● Concept hierarchy generation – Organizing data into multiple levels of abstraction
(e.g., city → state → country).
Discretization helps reduce data size, improve interpretability, and enhance mining accuracy.
7. Conclusion
To summarize, data preprocessing is the foundation of data mining. The five major
tasks—cleaning, integration, reduction, transformation, and discretization—together ensure
that raw data is converted into a high-quality dataset ready for analysis.
● Data cleaning removes noise and handles missing values.
● Data integration combines multiple sources into a unified view.
● Data reduction simplifies data while preserving patterns.
● Data transformation prepares data into suitable formats.
● Data discretization converts continuous data into meaningful categories.
Without preprocessing, data mining results may be misleading or inaccurate. With proper
preprocessing, we can ensure that mining techniques work efficiently and produce reliable
knowledge.
In short, preprocessing is not just the first step, but also the most important step in the entire
data mining process. It transforms raw data into a treasure of meaningful insights.
