00:00:00 Data science is not about making complicated models.

It's not about making

awesome visualizations It's not about writing code data science is about using data to create
as much impact as possible for your company Now impact can be in the form of multiple
things It could be in the form of insights in the form of data products or in the form of product
recommendations for a company Now to do those things, then you need tools like making
complicated models or data visualizations or writing code But essentially as a data scientist

00:00:33 your job is to solve real company problems using data and what kind of tools
you use we don't care Now there's a lot of misconception about data science, especially on
YouTube and I think the reason for this is because there's a huge misalignment between
what's popular to talk about and what's needed in the industry. So because of that I want to
make things clear. I am a data scientist working for a GAFA company and those companies
really emphasize on using data to improve their products So this is my take on what is data
science

00:01:21 Before data science, we popularized the term data mining in an article called
from data mining to knowledge discovery in databases in 1996 in which it referred to the
overall process of discovering useful information from data In 2001, William S. Cleveland
wanted to bring data mining to another level He did that by combining computer science with
data mining Basically He made statistics a lot more technical which he believed would
expand the possibilities of data mining and produce a powerful force for innovation

00:01:53 Now you can take advantage of compute power for statistics and he called
this combo data science. Around this time this is also when web 2.0 emerged where
websites are no longer just a digital pamphlet, but a medium for a shared experience
amongst millions and millions of users These are web sites like MySpace in 2003 Facebook
in 2004 and YouTube in 2005. We can now interact with these web sites meaning we can
contribute post comment like upload share leaving our footprint in the digital landscape we
call Internet and help create and shape the ecosystem

00:02:30 we now know and love today. And guess what? That's a lot of data so much
data, it became too much to handle using traditional technologies. So we call this Big Data.
That opened a world of possibilities in finding insights using data But it also meant that the
simplest questions require sophisticated data infrastructure just to support the handling of
the data We needed parallel computing technology like MapReduce, Hadoop, and Spark so
the rise of big data in 2010 sparked the rise of data science to support the needs of the
businesses to draw insights from their massive unstructured data sets

00:03:07 So then the journal of data science described data science as almost
everything that has something to do with data Collecting analyzing modeling. Yet the most
important part is its applications. All sorts of applications. Yes, all sorts of applications like
machine learning So in 2010 with the new abundance of data it made it possible to train
machines with a data-driven approach rather than a knowledge driven approach. All the
theoretical papers about recurring neural networks support vector machines became feasible

00:03:40 Something that can change the way we live and how we experience things in
the world Deep learning is no longer an academic concept in these thesis paper It became a
tangible useful class of machine learning that would affect our everyday lives So machine
learning and AI dominated the media overshadowing every other aspect of data science like
exploratory analysis, experimentation, ... And skills we traditionally called business
intelligence So now the general public think of data science as researchers focused on
machine learning and AI but the industry is hiring data scientists as analysts

00:04:17 So there's a misalignment there The reason for the misalignment is that yes,
most of these data scientists can probably work on more technical problems but big
companies like Google Facebook Netflix have so many low-hanging fruits to improve their
products that they don't require any advanced machine learning or statistical knowledge to
find these impacts in their analysis Being a good data scientist isn't about how advanced
your models are It's about how much impact you can have with your work. You're not a data
cruncher. You're a problem solver

00:04:50 You're strategists. Companies will give you the most ambiguous and hard
problems. And we expect you to guide the company to the right direction Ok, now I want to
conclude with real-life examples of data science jobs in Silicon Valley But first I have to print
some charts. So let's go do that So this is a very useful chart that tells you the needs of data
science. Now, it's pretty obvious but sometimes we kind of forget about it now At the bottom
of the pyramid we have collect you obviously have to collect some sort of data to be able to
use that data

00:05:42 So collect storing transforming all of these data engineering effort is pretty
important and it's actu- It's actually quite captured pretty well in media because of big data
we talked about how difficult it is to manage all this data We talked about parallel computing
which means like Hadoop and Spark Stuff like that. We know about this. Now the thing that's
less known is the stuff in between which is right here everything that's here and Surprisingly
this is actually one of the most important things for companies because you're trying to tell
the company

00:06:15 what to do with your product. So what do I mean by that? So I'm an analytics
that tells you using the data what kind of insights can tell me what are happening to my
users and then metrics this is important because what's going on with my product? You
know, these metrics will tell you if you're successful or not. And then also, you know a be
testing of course Experimentation that allows you to know, which product versions are the
best So these things are actually really important but they're not so covered in media. What's
covered in media

00:06:46 is this part. AI, deep learning. We've heard it on and on about it, you know But
when you think about it for a company, for the industry, It's actually not the highest priority or
at least it's not the thing that yields the most result for the lowest amount of effort That's why
AI deep learning is on top of the hierarchy of needs and these things may be testing
analytics they're actually way more important for industry so that's why we're hiring a lot of
data scientists that does that. So what do data scientists actually do?

00:07:17 Well that depends on the company because of them as of the size So for a
start-up you kind of lack resources So you can only kind of have one DS. So that one data
scientist he has to do everything. So you might be seeing all all this being data scientists.
Maybe you won't be doing AI or deep learning because that's not a priority right now But you
might be doing all of these. You have to set up the whole data infrastructure You might even
have to write some software code to add logging and then you have to do the analytics

00:07:45 yourself, then you have to build the metrics yourself, and you have to do A/B
testing yourself. That's why for startups if they need a data scientist this whole thing is data
science, so that means you have to do everything. But let's look at medium-sized
companies. Now, finally they have a lot more resources. They can separate the data
engineers and the data scientists So usually in collection, this is probably software
engineering. And then here, you're gonna have data engineers doing this. And then
depending if you're medium-sized company does a lot of

00:08:20 recommendation models or stuff that requires AI, then DS will do all these
Right. So as a data scientist, you have to be a lot more technical That's why they only hire
people with PhDs or masters because they want you to be able to do the more complicated
things So let's talk about large company now Because you're getting a lot bigger you
probably have a lot more money and then you can spend it more on employees So you can
have a lot of different employees working on different things. That way the employee does
not need to think about this stuff that they don't want to do and they could focus on the
things that they're

00:08:52 best at. For example, me and my untitled large company I would be in
analytics so I could just focus my work on analytics and metrics and stuff like that So I don't
need to worry about data engineering or AI deep learning stuff So here's how it looks for a
large company Instrumental logging sensors. This is all handled by software engineers
Right? And then here, cleaning and building data pipelines This is for data engineers. Now
here, between these two things, we have Data Science Analytics. That's what it's called

00:09:29 But then once we go to the AI and deep learning, this is where we have
research scientists or we call it data science core and they are backed by and now
engineers which are machine learning engineers. Yeah Anyways, so in summary, as you
can see, data science can be all of this and it depends what company you are in And the
definition will vary. So please let me know what you would like to learn more about AI deep
learning, or A/B testing, experimentation,... Depending on what you want to learn about

Real world application related to

data science
Data science has a wide range of real-world applications across various industries. Here are
some notable examples:

### 1. **Healthcare**
- **Predictive Analytics:** Predicting disease outbreaks, patient readmissions, and
potential health risks using historical data.
 - **Medical Imaging:** Using machine learning to analyze medical images (e.g., X-rays,
MRIs) for early detection of diseases like cancer.
- **Personalized Medicine:** Tailoring treatments based on genetic information and patient
history.
- **Drug Discovery:** Accelerating the development of new drugs by analyzing biological
data.

### 2. **Finance**
- **Fraud Detection:** Identifying fraudulent transactions by analyzing patterns and
anomalies in financial data.
- **Algorithmic Trading:** Using predictive models to make high-frequency trading
decisions.
- **Risk Management:** Assessing and mitigating risks by analyzing market trends and
customer behavior.
- **Credit Scoring:** Evaluating the creditworthiness of individuals and businesses using
historical data.

### 3. **Retail**
- **Customer Segmentation:** Grouping customers based on purchasing behavior to tailor
marketing strategies.
- **Inventory Management:** Optimizing stock levels using demand forecasting models.
- **Recommendation Systems:** Suggesting products to customers based on their
browsing and purchase history (e.g., Amazon, Netflix).
- **Price Optimization:** Dynamically adjusting prices based on demand, competition, and
other factors.

### 4. **Transportation**
- **Route Optimization:** Finding the most efficient routes for delivery and logistics.
- **Autonomous Vehicles:** Using machine learning and computer vision for self-driving
cars.
- **Traffic Management:** Analyzing traffic patterns to reduce congestion and improve
urban planning.
- **Predictive Maintenance:** Monitoring vehicle health to predict and prevent mechanical
failures.

### 5. **Telecommunications**
- **Network Optimization:** Improving network performance by analyzing usage patterns
and predicting congestion.
- **Customer Churn Prediction:** Identifying customers who are likely to switch to
competitors and taking proactive measures to retain them.
- **Sentiment Analysis:** Analyzing customer feedback and social media data to gauge
public sentiment and improve services.

### 6. **Energy**
- **Smart Grids:** Optimizing the distribution and consumption of electricity using real-time
data.
- **Predictive Maintenance:** Monitoring equipment to predict failures and schedule
maintenance.
 - **Energy Consumption Forecasting:** Predicting energy demand to optimize production
and reduce waste.

### 7. **Marketing**
- **Campaign Optimization:** Analyzing the effectiveness of marketing campaigns and
adjusting strategies in real-time.
- **Customer Lifetime Value Prediction:** Estimating the long-term value of customers to
prioritize marketing efforts.
- **Sentiment Analysis:** Monitoring social media and customer reviews to understand
public perception of a brand.

### 8. **Sports**
- **Performance Analysis:** Using data to analyze player performance and develop
training programs.
- **Injury Prediction:** Identifying players at risk of injury based on physical data and
playing conditions.
- **Game Strategy:** Analyzing opponent data to develop game strategies and improve
team performance.

### 9. **Government**
- **Public Health:** Tracking and predicting the spread of diseases to implement timely
interventions.
- **Crime Prediction:** Using data to predict crime hotspots and allocate resources
effectively.
- **Urban Planning:** Analyzing data to improve infrastructure, transportation, and public
services.

### 10. **Entertainment**

- **Content Recommendation:** Suggesting movies, shows, or music based on user
preferences (e.g., Netflix, Spotify).
- **Audience Analysis:** Understanding audience demographics and preferences to tailor
content.
- **Box Office Prediction:** Forecasting the success of movies based on historical data and
market trends.

### 11. **Manufacturing**

- **Quality Control:** Using machine learning to detect defects in products during the
manufacturing process.
- **Supply Chain Optimization:** Improving the efficiency of supply chains by analyzing
data on suppliers, logistics, and demand.
- **Predictive Maintenance:** Monitoring machinery to predict failures and reduce
downtime.

### 12. **Education**

- **Personalized Learning:** Tailoring educational content to individual students based on
their learning patterns.
- **Student Performance Prediction:** Identifying students at risk of underperforming and
providing targeted interventions.
 - **Resource Allocation:** Optimizing the allocation of resources like teachers, classrooms,
and materials.

### 13. **Agriculture**

- **Precision Farming:** Using data from sensors and satellites to optimize planting,
watering, and harvesting.
- **Crop Prediction:** Predicting crop yields based on weather data, soil conditions, and
historical trends.
- **Pest Control:** Identifying and predicting pest outbreaks to minimize crop damage.

### 14. **Environmental Science**

- **Climate Modeling:** Using data to predict climate change and its impact on
ecosystems.
- **Wildlife Conservation:** Tracking animal populations and predicting threats to
biodiversity.
- **Disaster Prediction and Management:** Predicting natural disasters like earthquakes,
floods, and hurricanes to mitigate their impact.

### 15. **Human Resources**

- **Talent Acquisition:** Using data to identify the best candidates for job openings.
- **Employee Retention:** Predicting which employees are likely to leave and
implementing strategies to retain them.
- **Performance Analysis:** Evaluating employee performance and identifying areas for
improvement.

These applications demonstrate the versatility and impact of data science in solving complex
problems and driving innovation across various sectors.

Here’s a brief summary of each topic:

1. **Need for Data Science - What is Data Science**:

Data Science is an interdisciplinary field that uses scientific methods, algorithms, and
systems to extract insights and knowledge from structured and unstructured data. It is
essential for decision-making, predictive analysis, and solving complex problems across
industries.

2. **Data Science Process**:

The Data Science process involves steps like problem definition, data collection, data
cleaning, exploratory data analysis (EDA), model building, evaluation, and deployment. It is
iterative and focuses on deriving actionable insights.

3. **Business Intelligence and Data Science**:

Business Intelligence (BI) focuses on analyzing historical data to provide insights for
decision-making, while Data Science uses advanced analytics, machine learning, and
predictive modeling to uncover future trends and patterns.

4. **Prerequisites for a Data Scientist**:

 A Data Scientist typically needs a strong foundation in mathematics, statistics,
programming, and domain knowledge. Analytical thinking, problem-solving skills, and
familiarity with data tools are also crucial.

5. **Tools and Skills Required for Data Scientists**:

Key tools include Python, R, SQL, and libraries like Pandas, NumPy, and Scikit-learn.
Skills include data wrangling, machine learning, data visualization, and communication.

6. **Structured Query Language (SQL)**:

SQL is a programming language used to manage and query relational databases. It is
essential for data extraction, manipulation, and analysis.

7. **Basic Statistics**:
Statistics is the foundation of Data Science, covering concepts like probability,
distributions, hypothesis testing, and regression analysis, which are used to interpret data.

8. **Data Munging**:
Data munging (or wrangling) involves cleaning, transforming, and preparing raw data into a
usable format for analysis.

9. **Filtering**:
Filtering is the process of selecting specific subsets of data based on certain conditions or
criteria.

10. **Joins, Aggregation**:

Joins combine data from multiple tables, while aggregation summarizes data using
functions like SUM, AVG, COUNT, etc.

11. **Window Functions, Ordered Data Preparing**:

Window functions perform calculations across a set of table rows related to the current
row. Ordered data preparing involves sorting and organizing data for analysis.

12. **No-SQL: Document Databases**:

No-SQL databases like MongoDB store data in flexible, JSON-like documents, making
them suitable for unstructured or semi-structured data.

13. **Wide-column Databases and Graphical Databases**:

Wide-column databases (e.g., Cassandra) store data in columns rather than rows, ideal
for large-scale data. Graphical databases (e.g., Neo4j) focus on relationships between
entities, useful for network analysis.

Let me know if you need further clarification on any topic!

The prerequisites for **Data Science** encompass a combination of **technical skills**,

**mathematical knowledge**, and **soft skills**. Here’s a breakdown:

---
### **1. Technical Skills**:
- **Programming Languages**:
- **Python** or **R** (most commonly used in Data Science).
- Knowledge of libraries like Pandas, NumPy, Scikit-learn, Matplotlib, and TensorFlow (for
Python).
- **SQL**:
- Essential for querying and managing relational databases.
- **Data Wrangling**:
- Cleaning, transforming, and preparing raw data for analysis using tools like Pandas or
dplyr (in R).
- **Data Visualization**:
- Tools like Tableau, Power BI, or libraries like Matplotlib, Seaborn, and Plotly.
- **Big Data Tools**:
- Familiarity with tools like Hadoop, Spark, or cloud platforms (AWS, Google Cloud, Azure).

---

### **2. Mathematical and Statistical Knowledge**:

- **Statistics**:
- Probability, distributions, hypothesis testing, regression analysis, and Bayesian thinking.
- **Linear Algebra**:
- Vectors, matrices, and operations used in machine learning algorithms.
- **Calculus**:
- Understanding derivatives, integrals, and gradients for optimization in machine learning.
- **Probability**:
- Key for understanding algorithms like Naive Bayes, Markov Models, and more.

---

### **3. Machine Learning**:

- Understanding of supervised and unsupervised learning algorithms (e.g., linear regression,
decision trees, clustering, etc.).
- Model evaluation techniques like cross-validation, precision, recall, and F1-score.
- Familiarity with deep learning frameworks like TensorFlow or PyTorch (for advanced roles).

---

### **4. Domain Knowledge**:

- Understanding the industry or domain you’re working in (e.g., healthcare, finance, e-
commerce) to ask the right questions and interpret data effectively.

---

### **5. Soft Skills**:

- **Problem-Solving**: Ability to break down complex problems into manageable parts.
- **Communication**: Explaining technical insights to non-technical stakeholders.
- **Curiosity**: A strong desire to explore data and uncover hidden patterns.
- **Critical Thinking**: Evaluating data and models objectively.
---

### **6. Tools and Technologies**:

- **Version Control**: Git and GitHub for collaboration and code management.
- **Jupyter Notebooks**: For interactive coding and documentation.
- **Databases**: Knowledge of SQL and NoSQL databases (e.g., MongoDB, Cassandra).

---

### **7. Educational Background**:

- A degree in **Computer Science**, **Mathematics**, **Statistics**, **Engineering**, or a
related field is often preferred, but not mandatory. Many Data Scientists come from diverse
backgrounds.

---

### **8. Practical Experience**:

- Working on real-world projects, participating in Kaggle competitions, or contributing to
open-source projects to build a portfolio.

---

By mastering these prerequisites, you can build a strong foundation for a career in Data
Science. Let me know if you need more details on any specific area!

A **real-world example of Business Intelligence (BI)** in action is its use by **Starbucks** to

optimize store locations and improve customer experience. Here's how Starbucks leverages
BI:

---

### **Problem**:
Starbucks wanted to expand its store locations globally while ensuring profitability and
customer satisfaction. They needed to identify the best locations for new stores and
understand customer preferences.

---

### **Solution Using Business Intelligence**:

1. **Data Collection**:
- Starbucks gathered data from various sources, including:
- Point-of-Sale (POS) systems (sales data).
- Customer loyalty programs (purchase history, preferences).
- Demographic data (population density, income levels).
- Geographic data (foot traffic, nearby competitors).

2. **Data Analysis**:
 - Using BI tools like **Tableau** and **Microsoft Power BI**, Starbucks analyzed the data
to:
- Identify high-traffic areas with potential for new stores.
- Understand customer preferences (e.g., popular drinks, peak hours).
- Predict sales performance for new locations.

3. **Visualization and Reporting**:

- Dashboards were created to visualize key metrics like:
- Sales trends.
- Customer demographics.
- Store performance comparisons.
- These dashboards helped executives make data-driven decisions.

4. **Decision-Making**:
- Based on the insights, Starbucks:
- Opened new stores in high-potential locations.
- Customized menus to match local preferences.
- Optimized store layouts and staffing based on peak hours.

---

### **Outcome**:
- Starbucks successfully expanded its global presence while maximizing profitability.
- Improved customer satisfaction by tailoring offerings to local tastes.
- Enhanced operational efficiency through data-driven decisions.

---

This example demonstrates how **Business Intelligence** transforms raw data into
actionable insights, enabling companies like Starbucks to make informed decisions and gain
a competitive edge. Let me know if you'd like more examples!

**Data Munging** (also known as **Data Wrangling**) is the process of cleaning,

transforming, and preparing raw data into a usable format for analysis. It is a critical step in
the data science workflow because real-world data is often messy, incomplete, or
inconsistent. Here's a breakdown of what data munging involves:

---

### **Key Steps in Data Munging**:

1. **Data Cleaning**:
- Handling missing values (e.g., filling them with averages or removing rows/columns).
- Removing duplicates or irrelevant data.
- Correcting errors or inconsistencies in the data (e.g., typos, formatting issues).

2. **Data Transformation**:
- Converting data types (e.g., strings to numbers, dates to a consistent format).
- Normalizing or scaling data (e.g., converting values to a standard range).
 - Encoding categorical variables (e.g., one-hot encoding).

3. **Data Integration**:
- Combining data from multiple sources (e.g., merging datasets).
- Resolving conflicts or mismatches in data formats.

4. **Data Reduction**:
- Removing unnecessary columns or rows.
- Aggregating data (e.g., summarizing sales data by month).

5. **Feature Engineering**:
- Creating new features from existing data (e.g., calculating age from a birthdate).
- Extracting useful information (e.g., splitting addresses into city, state, and zip code).

---

### **Tools for Data Munging**:

- **Python Libraries**:
- **Pandas**: For data manipulation and cleaning.
- **NumPy**: For numerical operations.
- **OpenPyXL**: For working with Excel files.
- **R Libraries**:
- **dplyr**: For data manipulation.
- **tidyr**: For data cleaning.
- **SQL**: For querying and transforming data in databases.
- **ETL Tools**: Tools like **Apache NiFi** or **Talend** for automating data wrangling
workflows.

---

### **Example of Data Munging**:

Imagine you have a dataset of customer orders with the following issues:
- Missing values in the "Age" column.
- Inconsistent date formats (e.g., "01/12/2023" vs. "2023-12-01").
- Duplicate rows.
- A "Price" column with currency symbols (e.g., "$100").

**Steps to Munge the Data**:

1. Fill missing "Age" values with the median age.
2. Standardize the date format.
3. Remove duplicate rows.
4. Strip currency symbols from the "Price" column and convert it to a numeric type.

---

### **Why Data Munging is Important**:

- Ensures data quality and accuracy.
- Makes data suitable for analysis and modeling.
- Saves time and effort in later stages of the data science process.

---

Let me know if you'd like a practical example or code snippet for data munging!

The **5 V's of Data Science** are key characteristics that define the complexity and
challenges of working with data in the modern world. They are:

---

### **1. Volume**:

- Refers to the **amount of data** generated and collected.
- With the rise of IoT, social media, and digital transactions, organizations deal with massive
datasets, often in terabytes or petabytes.
- Example: Facebook processes over 500 terabytes of data daily.

---

### **2. Velocity**:

- Refers to the **speed at which data is generated and processed**.
- Real-time data streams (e.g., stock market data, sensor data) require fast processing to
derive timely insights.
- Example: Twitter processes around 6,000 tweets per second.

---

### **3. Variety**:

- Refers to the **different types of data** available.
- Data can be structured (e.g., databases), semi-structured (e.g., JSON, XML), or
unstructured (e.g., text, images, videos).
- Example: A retail company might analyze structured sales data, customer reviews (text),
and surveillance footage (video).

---

### **4. Veracity**:

- Refers to the **quality and reliability of data**.
- Data can be incomplete, inconsistent, or noisy, making it challenging to derive accurate
insights.
- Example: Sensor data from IoT devices may contain errors due to hardware malfunctions.

---

### **5. Value**:

- Refers to the **usefulness of data** in generating insights and driving decisions.
- The ultimate goal of data science is to extract value from data, whether through predictive
analytics, optimization, or innovation.
- Example: Netflix uses data to recommend personalized content, increasing user
engagement and retention.

---

### **Why the 5 V's Matter**:

- They highlight the challenges and opportunities in managing and analyzing modern
datasets.
- Understanding these dimensions helps organizations design better data strategies and
tools.

---

Let me know if you'd like examples or further details on any of the 5 V's!