1

Q.1 Explain process of working with data from files in Data Science.
Working with data from files is a crucial step in the data science workflow.
Here's an overview of the process:
Step 1: Data Ingestion
- Collect and gather data from various file sources (e.g., CSV, Excel, JSON,
text files).
- Use programming languages like Python, R, or SQL to read and import data
from files.
Step 2: Data Inspection
- Examine the data to understand its structure, quality, and content.
- Use summary statistics, data visualization, and data profiling techniques to
identify patterns, outliers, and missing values.
Step 3: Data Cleaning
- Handle missing values, duplicates, and inconsistent data entries.
- Perform data normalization, feature scaling, and data transformation as
needed.
Step 4: Data Transformation
- Convert data types, perform data aggregation, and create new features.
- Use data manipulation techniques, such as pivoting, melting, and merging.
Step 5: Data Storage
- Store cleaned and transformed data in a suitable format (e.g., Pandas
DataFrame, NumPy array, SQL database).
- Consider using data storage solutions like data warehouses, data lakes, or
cloud storage.
Step 6: Data Analysis
- Apply statistical and machine learning techniques to extract insights and
meaning from the data.
- Use data visualization tools to communicate findings and results.
Step 7: Data Visualization and Communication
- Present findings and insights to stakeholders using clear and effective
visualizations.
- Use storytelling techniques to convey the significance and impact of the
results.
 2

Some popular tools and technologies used in this process include:

- Pandas and NumPy for data manipulation and analysis
- Matplotlib and Seaborn for data visualization
- Scikit-learn and TensorFlow for machine learning
- SQL and NoSQL databases for data storage
- Jupyter Notebooks and R Studio for interactive data exploration and analysis
Q2. Explain use of NumPy arrays for efficient data manipulation.
NumPy (Numerical Python) arrays are a fundamental data structure in
scientific computing and data analysis. They provide an efficient way to store
and manipulate large datasets. Here's how NumPy arrays support efficient data
manipulation:
Advantages of NumPy Arrays
1. Vectorized Operations: NumPy arrays enable vectorized operations, which
allow you to perform operations on entire arrays at once. This eliminates the
need for loops, making your code faster and more concise.
2. Memory Efficiency: NumPy arrays store data in a contiguous block of
memory, which reduces memory usage and improves cache locality. This leads
to faster data access and manipulation.
3. Broadcasting: NumPy arrays support broadcasting, which allows you to
perform operations on arrays with dif…
Q3 Explain structure of data in Pandas and its importance in large datasets
Pandas is a powerful Python library used for data manipulation and analysis.
The structure of data in Pandas is based on two primary data structures: Series
(1-dimensional labeled array) and DataFrame (2-dimensional labeled data
structure with columns of potentially different types).
Pandas Data Structures:
1. Series: A Series is a one-dimensional labeled array of values. It's similar to
a column in a spreadsheet or a column in a relational database. Each value in the
Series is associated with a unique index label.
Example:
import pandas as pd
# Create a Series
series = pd.Series([1, 2, 3, 4, 5], index=['a', 'b', 'c', 'd', 'e'])
 3

print(series)
Output:
a 1
b 2
c 3
d 4
e 5
dtype: int64
1. DataFrame: A DataFrame is a two-dimensional labeled data structure with
columns of potentially different types. It's similar to an Excel spreadsheet or a
table in a relational database. Each column in the DataFrame is a Series, and
each row is identified by a unique index label.
Example:
import pandas as pd
# Create a DataFrame
data = {'Name': ['John', 'Anna', 'Peter', 'Linda'],
'Age': [28, 24, 35, 32],
'Country': ['USA', 'UK', 'Australia', 'Germany']}
df = pd.DataFrame(data)
print(df)
Output:
Name Age Country
0 John 28 USA
1 Anna 24 UK
2 Peter 35 Australia
3 Linda 32 Germany
Importance of Pandas Data Structures in Large Datasets:
1. Efficient Data Storage: Pandas DataFrames and Series provide efficient
data storage, allowing you to store large datasets in memory.
2. Fast Data Manipulation: Pandas provides various methods for fast data
manipulation, such as filtering, sorting, grouping, and merging.
 4

3. Data Alignment: Pandas DataFrames and Series provide data alignment,

which ensures that data is properly aligned and indexed, making it easier to
perform data analysis.
4. Missing Data Handling: Pandas provides built-in support for missing data
handling, allowing you to easily detect and handle missing values in your
dataset.
5. Integration with Other Libraries: Pandas integrates well with other popular
data science libraries, such as NumPy, Matplotlib, Scikit-learn, and Statsmodels.
In summary, Pandas data structures provide an efficient and flexible way to
store and manipulate large datasets, making it an essential library for data
science tasks.
Q4. Explain different data loading and storage formats for Data Science projects.
In Data Science projects, data loading and storage formats play a crucial role
in efficient data processing, analysis, and modeling. Here are different data
loading and storage formats commonly used in Data Science:
1. CSV (Comma Separated Values)
- A plain text file format used for tabular data.
- Widely supported by most data science tools and libraries.
- Easy to read and write, but can be slow for large datasets.
2. JSON (JavaScript Object Notation)
- A lightweight, human-readable data interchange format.
- Suitable for semi-structured data, such as web scraping or API data.
- Can be slower to read and write compared to binary formats.
3. HDF5 (Hierarchical Data Format 5)
- A binary format designed for large, complex datasets.
- Supports hierarchical data structures and efficient data compression.
- Widely used in scientific computing, but may require additional libraries.
4. Apache Parquet
- A columnar storage format designed for big data analytics.
- Optimized for querying and processing large datasets.
- Supported by many big data technologies, including Hadoop and Spark.
5. Apache Arrow
- A cross-language, columnar memory format for big data analytics.
 5

- Designed for high-performance data processing and interchange.

- Supported by many big data technologies, including Pandas, NumPy, and
Spark.
6. Pickle
- A Python-specific binary format for serializing and deserializing data.
- Fast and efficient, but specific to Python and may not be compatible with
other languages.
7. SQL Databases
- Relational databases that store data in tables with defined schemas.
- Suitable for structured data and support SQL queries for data analysis.
- Examples include MySQL, PostgreSQL, and SQLite.
8. NoSQL Databases
- Non-relational databases that store data in flexible, dynamic schemas.
- Suitable for semi-structured or unstructured data, such as documents or
graphs.
- Examples include MongoDB, Cassandra, and Neo4j.
When choosing a data loading and storage format, consider factors such as:
- Data size and complexity
- Performance requirements
- Compatibility with tools and libraries
- Data structure and schema
- Security and data governance requirements
Q5. Explain the process of reshaping and pivoting data for effective
analysis.
Reshaping and pivoting data are essential steps in data preparation for
effective analysis. Here's a step-by-step guide on how to reshape and pivot data:
Reshaping Data
Reshaping data involves transforming data from a wide format to a long
format or vice versa.
1. Wide Format: In a wide format, each row represents a single observation,
and each column represents a variable.
2. Long Format: In a long format, each row represents a single observation-
variable pair.
 6

Tools for Reshaping Data

1. Pandas melt() function: Use the melt() function to transform data from
wide to long format.
2. Pandas pivot() function: Use the pivot() function to transform data from
long to wide format.
Pivoting Data
Pivoting data involves rotating data from a state of rows to columns or vice
versa, creating a spreadsheet-style pivot table.
Types of Pivoting
1. Simple Pivoting: Rotate data from rows to columns.
2. Aggregated Pivoting: Rotate data from rows to columns and perform
aggregation operations (e.g., sum, mean, count).
Tools for Pivoting Data
1. Pandas pivot_table() function: Use the pivot_table() function to create a
pivot table.
2. Pandas pivot() function: Use the pivot() function to perform simple
pivoting.
Example Use Case
Suppose we have a dataset containing sales data for different products across
various regions.
| Region | Product A | Product B | Product C |
| --- | --- | --- | --- |
| North | 100 | 200 | 300 |
| South | 150 | 250 | 350 |
| East | 200 | 300 | 400 |
| West | 250 | 350 | 450 |

To analyze sales data by product and region, we can pivot the data using the
pivot_table() function.
import pandas as pd
# Create a sample dataset
data = {'Region': ['North', 'South', 'East', 'West'],
'Product A': [100, 150, 200, 250],
 7

'Product B': [200, 250, 300, 350],

'Product C': [300, 350, 400, 450]}
df = pd.DataFrame(data)
# Pivot the data
pivoted_df = pd.pivot_table(df, values=['Product A', 'Product B', 'Product C'],
index='Region', aggfunc='sum')
print(pivoted_df)
Output:
| Region | Product A | Product B | Product C |
| --- | --- | --- | --- |
| East | 200 | 300 | 400 |
| North | 100 | 200 | 300 |
| South | 150 | 250 | 350 |
| West | 250 | 350 | 450 |
By pivoting the data, we can easily analyze sales data by product and
region.
Data exploration is a crucial step in Data Science projects that involves
visually and statistically examining the data to understand its underlying
structure, patterns, and relationships. The primary goal of data exploration is to
gain insights into the data, identify potential issues, and inform the subsequent
steps of the project.
Key Objectives of Data Exploration:
1. Understand the data distribution: Examine the distribution of values in each
variable, including central tendency, dispersion, and skewness.
2. Identify patterns and relationships: Look for correlations, trends, and
relationships between variables.
3. Detect outliers and anomalies: Identify data points that are significantly
different from the rest of the data.
4. Assess data quality: Check for missing values, duplicates, and
inconsistencies in the data.
5. Inform feature engineering: Use insights gained during exploration to
inform the creation of new features or transformation of existing ones.
Techniques Used in Data Exploration:
 8

1. Summary statistics: Calculate mean, median, mode, standard deviation, and

variance for numerical variables.
2. Data visualization: Use plots, charts, and heatmaps to visualize the data
distribution, patterns, and relationships.
3. Correlation analysis: Examine the correlation between numerical variables
using correlation coefficients (e.g., Pearson's r).
4. Scatter plots: Visualize the relationship between two numerical variables.
5. Box plots: Compare the distribution of numerical variables across different
categories.
Tools Used in Data Exploration:
1. Pandas: A popular Python library for data manipulation and analysis.
2. Matplotlib: A Python library for creating static, animated, and interactive
visualizations.
3. Seaborn: A Python library built on top of Matplotlib for creating
informative and attractive statistical graphics.
4. Plotly: A Python library for creating interactive, web-based visualizations.
5. Jupyter Notebook: A web-based interactive environment for working with
data and visualizing results.
Best Practices for Data Exploration:
1. Start with a clear question or objective: Focus your exploration on a
specific question or hypothesis.
2. Use a combination of techniques: Employ multiple techniques, such as
summary statistics, visualization, and correlation analysis, to gain a
comprehensive understanding of the data.
3. Be iterative: Refine your exploration as you gain insights and identify new
questions or areas of interest.
4. Document your findings: Record your observations, insights, and
conclusions to inform subsequent steps in the project.

Q6. Explain role of data exploration in Data Science projects

Data exploration is a crucial step in Data Science projects that involves
visually and statistically examining the data to understand its underlying
 9

structure, patterns, and relationships. The primary goal of data exploration is to

gain insights into the data, identify potential issues, and inform the subsequent
steps of the project.
Key Objectives of Data Exploration:
1. Understand the data distribution: Examine the distribution of values in each
variable, including central tendency, dispersion, and skewness.
2. Identify patterns and relationships: Look for correlations, trends, and
relationships between variables.
3. Detect outliers and anomalies: Identify data points that are significantly
different from the rest of the data.
4. Assess data quality: Check for missing values, duplicates, and
inconsistencies in the data.
5. Inform feature engineering: Use insights gained during exploration to
inform the creation of new features or transformation of existing ones.

Techniques Used in Data Exploration:

1. Summary statistics: Calculate mean, median, mode, standard deviation, and
variance for numerical variables.
2. Data visualization: Use plots, charts, and heatmaps to visualize the data
distribution, patterns, and relationships.
3. Correlation analysis: Examine the correlation between numerical variables
using correlation coefficients (e.g., Pearson's r).
4. Scatter plots: Visualize the relationship between two numerical variables.
5. Box plots: Compare the distribution of numerical variables across different
categories.
Tools Used in Data Exploration:
1. Pandas: A popular Python library for data manipulation and analysis.
2. Matplotlib: A Python library for creating static, animated, and interactive
visualizations.
3. Seaborn: A Python library built on top of Matplotlib for creating
informative and attractive statistical graphics.
4. Plotly: A Python library for creating interactive, web-based visualizations.
 10

5. Jupyter Notebook: A web-based interactive environment for working with

data and visualizing results.
Best Practices for Data Exploration:
1. Start with a clear question or objective: Focus your exploration on a
specific question or hypothesis.
2. Use a combination of techniques: Employ multiple techniques, such as
summary statistics, visualization, and correlation analysis, to gain a
comprehensive understanding of the data.
3. Be iterative: Refine your exploration as you gain insights and identify new
questions or areas of interest.
4. Document your findings: Record your observations, insights, and
conclusions to inform subsequent steps in the project.
Q7. Explain process of data cleaning and sampling in a data science
project.
Data cleaning and sampling are crucial steps in a data science project that
ensure the quality and reliability of the data. Here's a step-by-step guide on the
process of data cleaning and sampling:
Data Cleaning
Data cleaning involves identifying and correcting errors, inconsistencies, and
inaccuracies in the data.
Steps in Data Cleaning
1. Data Inspection: Examine the data to identify errors, inconsistencies, and
inaccuracies.
2. Handling Missing Values: Decide on a strategy to handle missing values,
such as imputation, interpolation, or deletion.
3. Data Normalization: Normalize data to ensure consistency in formatting
and scaling.
4. Data Transformation: Transform data to ensure it meets the requirements of
the analysis or model.
5. Data Quality Check: Perform a final quality check to ensure the data is
accurate, complete, and consistent.
Data Sampling
 11

Data sampling involves selecting a subset of data from the original dataset to
reduce the size of the data while maintaining its representativeness.
Types of Data Sampling
1. Random Sampling: Select a random subset of data from the original
dataset.
2. Stratified Sampling: Divide the data into subgroups based on relevant
characteristics and select a random subset from each subgroup.
3. Cluster Sampling: Divide the data into clusters based on relevant
characteristics and select a random subset of clusters.
Steps in Data Sampling
1. Determine the Sampling Method: Choose a suitable sampling method
based on the characteristics of the data and the goals of the analysis.
2. Determine the Sample Size: Calculate the required sample size based on
the desired level of precision, confidence, and power.
3. Select the Sample: Use the chosen sampling method to select the sample
from the original dataset.
4. Evaluate the Sample: Assess the representativeness of the sample and its
suitability for the analysis or model.
Tools and Techniques for Data Cleaning and Sampling
1. Pandas: A popular Python library for data manipulation and analysis.
2. NumPy: A library for efficient numerical computation in Python.
3. Matplotlib: A plotting library for creating static, animated, and interactive
visualizations in Python.
4. Scikit-learn: A machine learning library for Python that includes tools for
data preprocessing, feature selection, and model evaluation.
By following these steps and using these tools and techniques, you can ensure
that your data is clean, reliable, and representative, which is essential for
accurate analysis and modeling in data science projects.
Q8. Explain the concept of broadcasting in NumPy. How does it help in
data processing?
Broadcasting is a powerful feature in NumPy that allows you to perform
operations on arrays with different shapes and sizes. It enables you to write
concise and efficient code for various data processing tasks.
 12

What is Broadcasting?
Broadcasting is the process of aligning arrays with different shapes and sizes
to perform element-wise operations. When operating on two arrays, NumPy
compares their shapes element-wise. It starts with the trailing dimensions, and
works its way forward. Two dimensions are compatible when:
1. They are equal.
2. One of them is 1.
If these conditions are not met, a ValueError is raised.
How Broadcasting Works
Here's an example to illustrate broadcasting:
import numpy as np
# Create two arrays
a = np.array([1, 2, 3]) # shape: (3,)
b = np.array([4]) # shape: (1,)
# Perform element-wise addition
result = a + b
print(result) # Output: [5 6 7]
In this example, a has shape (3,) and b has shape (1,). To perform the
addition, NumPy broadcasts b to match the shape of a. The resulting array has
shape (3,).
Benefits of Broadcasting
Broadcasting provides several benefits in data processing:
1. Concise Code: Broadcasting allows you to write concise and expressive
code, reducing the need for explicit loops.
2. Efficient Computation: By avoiding explicit loops, broadcasting enables
efficient computation and reduces overhead.
3. Flexibility: Broadcasting supports operations on arrays with different
shapes and sizes, making it a versatile tool for data processing.
Common Use Cases for Broadcasting
1. Element-wise Operations: Broadcasting is commonly used for element-
wise operations like addition, subtraction, multiplication, and division.
2. Array Multiplication: Broadcasting is useful for multiplying arrays with
different shapes, such as matrix multiplication.
 13

3. Data Transformation: Broadcasting can be used to transform data by

applying element-wise functions, such as scaling, normalization, or feature
extraction.
In summary, broadcasting is a powerful feature in NumPy that enables
efficient and concise data processing. By understanding how broadcasting
works, you can leverage its benefits to simplify your code and improve
performance.
Q9. Explain essential functionalities of Pandas for data analysis?
Pandas is a powerful Python library for data analysis that provides data
structures and functions to efficiently handle structured data, including tabular
data such as spreadsheets and SQL tables.
Essential Functionalities of Pandas:
1. Data Structures: Pandas provides two primary data structures:
- Series (1-dimensional labeled array of values)
- DataFrame (2-dimensional labeled data structure with columns of
potentially different types)
2. Data Manipulation: Pandas offers various methods for data manipulation,
including:
- Filtering: Selecting specific rows or columns based on conditions
- Sorting: Sorting data by one or more columns
- Grouping: Grouping data by one or more columns and performing
aggregation operations
- Merging: Combining data from multiple sources based on common
columns
- Reshaping: Transforming data from wide to long format or vice versa
3. Data Analysis: Pandas provides various methods for data analysis,
including:
- Summary Statistics: Calculating mean, median, mode, standard deviation,
and variance
- Correlation Analysis: Calculating correlation coefficients between
columns
- Data Visualization: Integrating with visualization libraries like Matplotlib
and Seaborn to create plots and charts
 14

4. Data Input/Output: Pandas supports various data input/output formats,

including:
- CSV: Reading and writing comma-separated values files
- Excel: Reading and writing Excel files
- JSON: Reading and writing JSON files
- SQL: Reading and writing data from SQL databases
5. Data Cleaning: Pandas provides methods for data cleaning, including:
- Handling Missing Values: Detecting and filling missing values
- Data Normalization: Normalizing data to ensure consistency in formatting
and scaling
- Data Transformation: Transforming data to ensure it meets the
requirements of the analysis
Key Benefits of Using Pandas:
1. Efficient Data Manipulation: Pandas provides fast and efficient data
manipulation capabilities.
2. Flexible Data Structures: Pandas offers flexible data structures that can
handle a wide range of data types and formats.
3. Easy Data Analysis: Pandas provides a simple and intuitive API for data
analysis, making it easy to perform common data analysis tasks.
4. Integration with Other Libraries: Pandas integrates well with other popular
data science libraries, including NumPy, Matplotlib, and Scikit-learn.
Q10. Explain how data is loaded, stored, and formatted in different file
types for analysis.
Here's an overview of how data is loaded, stored, and formatted in different
file types for analysis:
Text Files (.txt, .csv)
1. Loading: Text files can be loaded using programming languages like
Python, R, or SQL.
2. Storage: Text files store data in plain text format, with each row
representing a single observation and each column representing a variable.
3. Formatting: Text files typically use commas (CSV) or tabs (TSV) to
separate columns, and newlines to separate rows.
Comma Separated Values (.csv)
 15

1. Loading: CSV files can be loaded using programming languages like

Python, R, or SQL.
2. Storage: CSV files store data in plain text format, with each row
representing a single observation and each column representing a variable.
3. Formatting: CSV files use commas to separate columns and newlines to
separate rows.
Excel Files (.xls, .xlsx)
1. Loading: Excel files can be loaded using programming languages like
Python, R, or SQL.
2. Storage: Excel files store data in a binary format, with each row
representing a single observation and each column representing a variable.
3. Formatting: Excel files use a proprietary format to store data, with support
for formatting, formulas, and charts.
JSON Files (.json)
1. Loading: JSON files can be loaded using programming languages like
Python, R, or SQL.
2. Storage: JSON files store data in a lightweight, human-readable format,
with each object representing a single observation and each key representing a
variable.
3. Formatting: JSON files use key-value pairs to represent data, with support
for nested objects and arrays.
HDF5 Files (.h5)
1. Loading: HDF5 files can be loaded using programming languages like
Python, R, or SQL.
2. Storage: HDF5 files store data in a binary format, with support for large
datasets and high-performance I/O.
3. Formatting: HDF5 files use a hierarchical format to store data, with support
for groups, datasets, and attributes.
Relational Databases (e.g., MySQL, PostgreSQL)
1. Loading: Relational databases can be loaded using SQL queries.
2. Storage: Relational databases store data in tables, with each row
representing a single observation and each column representing a variable.
 16

3. Formatting: Relational databases use a structured format to store data, with

support for data types, constraints, and relationships between tables.
In summary, different file types and databases have their own strengths and
weaknesses when it comes to loading, storing, and formatting data for analysis.
Q11. What is data science ?
Data science is an interdisciplinary field that uses scientific methods,
processes, algorithms, and systems to extract knowledge and insights from
structured and unstructured data. It involves using various techniques from
computer science, statistics, and domain-specific knowledge to turn data into
actionable insights.
Data science encompasses a range of activities, including:
1. Data collection: Gathering data from various sources, such as databases,
APIs, files, and sensors.
2. Data cleaning: Preprocessing data to remove errors, inconsistencies, and
missing values.
3. Data transformation: Converting data into a suitable format for analysis.
4. Data visualization: Using plots, charts, and other visualizations to
communicate insights and patterns in the data.
5. Machine learning: Using algorithms to train models that can make
predictions or classify data.
6. Statistical analysis: Applying statistical techniques to identify trends,
patterns, and correlations in the data.
7. Insight generation: Interpreting the results of the analysis to extract
meaningful insights and recommendations.
Data science has many applications across various industries, including:
1. Business: Customer segmentation, market analysis, and predictive
modeling.
2. Healthcare: Disease diagnosis, patient outcome prediction, and
personalized medicine.
3. Finance: Risk analysis, portfolio optimization, and predictive modeling.
4. Marketing: Customer behavior analysis, campaign optimization, and social
media monitoring.
 17

5. Environmental science: Climate modeling, air quality monitoring, and

natural disaster prediction.
The data science process typically involves the following steps:
1. Problem formulation: Defining the problem or question to be addressed.
2. Data collection: Gathering relevant data from various sources.
3. Data analysis: Applying various techniques to extract insights from the
data.
4. Insight generation: Interpreting the results of the analysis to extract
meaningful insights.
5. Communication: Presenting the insights and recommendations to
stakeholders.
6. Deployment: Implementing the insights and recommendations into
production.
Data science requires a combination of technical skills, including:
1. Programming: Proficiency in languages such as Python, R, or SQL.
2. Data analysis: Knowledge of statistical techniques, machine learning
algorithms, and data visualization tools.
3. Data management: Familiarity with data storage solutions, data
governance, and data quality.
4. Communication: Ability to effectively communicate insights and
recommendations to stakeholders.
Overall, data science is a rapidly evolving field that requires a unique blend of
technical, business, and communication skills to extract insights from data and
drive business value.
Data Science Process Life Cycle
Some steps are necessary for any of the tasks that are being done in the field of
data science to derive any fruitful results from the data at hand.
 Data Collection – After formulating any problem statement the main task is

to calculate data that can help us in our analysis and manipulation.

Sometimes data is collected by performing some kind of survey and there are
times when it is done by performing scrapping.
 18

 Data Cleaning – Most of the real-world data is not structured and requires
cleaning and conversion into structured data before it can be used for any
analysis or modeling.
 Exploratory Data Analysis – This is the step in which we try to find the

hidden patterns in the data at hand. Also, we try to analyze different factors
which affect the target variable and the extent to which it does so. How the
independent features are related to each other and what can be done to
achieve the desired results all these answers can be extracted from this
process as well. This also gives us a direction in which we should work to
get started with the modeling process.
 Model Building – Different types of machine learning algorithms as well as

techniques have been developed which can easily identify complex patterns
in the data which will be a very tedious task to be done by a human.
 Model Deployment – After a model is developed and gives better results on

the holdout or the real-world dataset then we deploy it and monitor its
performance. This is the main part where we use our learning from the data
to be applied in real-world applications and use cases.
Key Components of Data Science Process
 Data Analysis – There are times when there is no need to apply advanced

deep learning and complex methods to the data at hand to derive some
patterns from it. Due to this before moving on to the modeling part, we first
perform an exploratory data analysis to get a basic idea of the data and
patterns which are available in it this gives us a direction to work on if we
want to apply some complex analysis methods on our data.
 Statistics – It is a natural phenomenon that many real-life datasets follow a

normal distribution. And when we already know that a particular dataset

follows some known distribution then most of its properties can be analyzed
at once. Also, descriptive statistics and correlation and covariances between
two features of the dataset help us get a better understanding of how one
factor is related to the other in our dataset.
 Data Engineering – When we deal with a large amount of data then we

have to make sure that the data is kept safe from any online threats also it is
 19

easy to retrieve and make changes in the data as well. To ensure that the data
is used efficiently Data Engineers play a crucial role.
 Advanced Computing

o Machine Learning – Machine Learning has opened new horizons

which had helped us to build different advanced applications and

methodologies so, that the machines become more efficient and
provide a personalized experience to each individual and perform
tasks in a snap of the hand earlier which requires heavy human labor
and time intense.
o Deep Learning – This is also a part of Artificial Intelligence and

Machine Learning but it is a bit more advanced than machine

learning itself. High computing power and a huge corpus of data
have led to the emergence of this field in data science.
Knowledge and Skills for Data Science Professionals
Becoming proficient in Data Science requires a combination of skills,
including:
 Statistics: Wikipedia defines it as the study of the collection, analysis,

interpretation, presentation, and organization of data. Therefore, it shouldn’t

be a surprise that data scientists need to know statistics.
 Programming Language R/ Python: Python and R are one of the most

widely used languages by Data Scientists. The primary reason is the number
of packages available for Numeric and Scientific computing.
 Data Extraction, Transformation, and Loading: Suppose we have

multiple data sources like MySQL DB, MongoDB, Google Analytics. You
have to Extract data from such sources, and then transform it for storing in a
proper format or structure for the purposes of querying and analysis. Finally,
you have to load the data in the Data Warehouse, where you will analyze the
data. So, for people from ETL (Extract Transform and Load) background
Data Science can be a good career option.
Steps for Data Science Processes:
Step 1: Define the Problem and Create a Project Charter
 20

Clearly defining the research goals is the first step in the Data Science
Process. A project charter outlines the objectives, resources, deliverables, and
timeline, ensuring that all stakeholders are aligned.
Step 2: Retrieve Data
Data can be stored in databases, data warehouses, or data lakes within an
organization. Accessing this data often involves navigating company policies
and requesting permissions.
Step 3: Data Cleansing, Integration, and Transformation
Data cleaning ensures that errors, inconsistencies, and outliers are
removed. Data integration combines datasets from different sources,
while data transformation prepares the data for modeling by reshaping
variables or creating new features.
Step 4: Exploratory Data Analysis (EDA)
During EDA, various graphical techniques like scatter plots, histograms, and
box plots are used to visualize data and identify trends. This phase helps in
selecting the right modeling techniques.
Step 5: Build Models
In this step, machine learning or deep learning models are built to make
predictions or classifications based on the data. The choice of algorithm
depends on the complexity of the problem and the type of data.
Step 6: Present Findings and Deploy Models
Once the analysis is complete, results are presented to stakeholders. Models are
deployed into production systems to automate decision-making or support
ongoing analysis.
Benefits and uses of data science and big data
 Governmental organizations are also aware of data’s value. A data scientist in a

governmental organization gets to work on diverse projects such as detecting fraud

and other criminal activity or optimizing project funding.
 Nongovernmental organizations (NGOs) are also no strangers to using data. They

use it to raise money and defend their causes. The World Wildlife Fund (WWF), for
instance, employs data scientists to increase the effectiveness of their fundraising
efforts.
 Universities use data science in their research but also to enhance the study

experience of their students. • Ex: MOOC’s- Massive open online courses.