The document discusses key concepts in machine learning, focusing on bias, variance, and their tradeoff, which are crucial for model performance. It covers various regression techniques, including linear regression, ridge, lasso, and elastic net, as well as dimensionality reduction through PCA and classification methods like Naive Bayes and SVM. Additionally, it highlights the importance of regularization to prevent overfitting and improve model accuracy.
The document discusses key concepts in machine learning, focusing on bias, variance, and their tradeoff, which are crucial for model performance. It covers various regression techniques, including linear regression, ridge, lasso, and elastic net, as well as dimensionality reduction through PCA and classification methods like Naive Bayes and SVM. Additionally, it highlights the importance of regularization to prevent overfitting and improve model accuracy.
The document discusses key concepts in machine learning, focusing on bias, variance, and their tradeoff, which are crucial for model performance. It covers various regression techniques, including linear regression, ridge, lasso, and elastic net, as well as dimensionality reduction through PCA and classification methods like Naive Bayes and SVM. Additionally, it highlights the importance of regularization to prevent overfitting and improve model accuracy.
The document discusses key concepts in machine learning, focusing on bias, variance, and their tradeoff, which are crucial for model performance. It covers various regression techniques, including linear regression, ridge, lasso, and elastic net, as well as dimensionality reduction through PCA and classification methods like Naive Bayes and SVM. Additionally, it highlights the importance of regularization to prevent overfitting and improve model accuracy.
Definition: Bias refers to the error introduced when a model makes overly simplistic assumptions about the data. It prevents the model from capturing the true relationships within the data. Impact: High bias often leads to underfitting, where the model performs poorly on both training and test data. Example: Using a straight line to fit a complex curve. Key Point: Bias measures how far off the model’s predictions are from the actual data.
2. High Variance in a Model
Definition: Variance measures the sensitivity of a model to changes in the training data. High variance often leads to overfitting, where the model performs well on training data but poorly on new, unseen data. Impact: Models with high variance are too complex and memorize training data instead of generalizing. Key Point: High variance indicates overfitting.
3. High Bias and Low Variance
Scenario: When a model is too simple (high bias) and has low variance, it fails to learn the underlying patterns in the data. Outcome: This situation is referred to as underfitting. Key Point: Balance between bias and variance is necessary for good model performance.
4. Bias-Variance Tradeoff Goal: Achieve a balance between bias and variance to minimize total error. Key Concept: A good model has neither too much bias (underfitting) nor too much variance (overfitting).
1 5. Reducing Variance Methods: • Simplifying the model (e.g., reducing complexity). • Collecting more training data. • Using techniques like regularization. Key Point: Increasing the dataset size can help reduce variance.
6. Simple Linear Regression Formula
Formula: y = mx + c • y: Target variable. • x: Predictor variable. • m: Slope of the line. • c: Intercept. Purpose: Predicts the relationship between one independent variable and a dependent variable.
7. R-Squared Value in Linear Regression
Definition: The R-squared value represents the proportion of variance in the dependent variable that is explained by the independent variables. Range: Between 0 and 1. A higher R-squared value indicates a better fit. Key Point: Measures how well the model fits the data.
8. Assumptions in Linear Regression
Linear regression requires: • A linear relationship between variables. • Independence of residuals. • Homoscedasticity (constant variance of residuals). Not Required: Predictors don’t need to follow a normal distribution.
2 9. Multicollinearity in Linear Regression Definition: Multicollinearity occurs when independent variables are highly correlated with each other. Impact: It can make the coefficient estimates unstable and unreliable. Key Point: Address multicollinearity using techniques like ridge regression.
10. Evaluating Regression Accuracy
Metric: Mean Squared Error (MSE).
• Measures the average squared difference between actual and predicted
values.
• Lower MSE indicates better model performance.
11. Ridge Regression
Penalty Term: Adds λ w2 to the loss function, where w are the model’s P coefficients. Use Case: Addresses multicollinearity by shrinking coefficients. Regularization Type: L2 regularization.
12. Lasso Regression
P Penalty Term: Adds λ |w| to the loss function. Key Feature: Performs feature selection by shrinking some coefficients to zero. Use Case: Preferred when feature selection is required.
13. Elastic Net
Definition: Combines L1 (lasso) and L2 (ridge) regularization, balancing feature selection and regularization. Advantage: Effective when features are highly correlated.
3 14. Principal Component Analysis (PCA) Purpose: Reduces the number of features (dimensionality reduction) while retaining the most variance in the data. Steps: • Scale the features (essential).
• Calculate the principal components.
• Retain components based on explained variance.
15. Naive Bayes
Based On: Bayes’ theorem. Assumption: Features are conditionally independent. Variants: • Gaussian Naive Bayes: Handles continuous data with normal dis- tribution.
• Multinomial Naive Bayes: Used for text classification.
16. Decision Tree
Leaf Node: A terminal node where no further splits occur; represents a prediction. Gini Impurity: Measures the homogeneity of a dataset. Lower values indicate purer nodes. Pruning: Reduces tree depth to prevent overfitting and reduce variance.
17. Support Vector Machines (SVM)
Parameters: • C: Controls the tradeoff between bias and variance.
• Gamma: Determines the influence of individual data points.
Kernels: • Linear Kernel: Suitable for linearly separable data.
4 • RBF Kernel: Effective for non-linear data by mapping to higher di- mensions.
18. Regularization Purpose: Reduces overfitting by adding a penalty term to the model’s loss function. Types:
• L1 (Lasso): Shrinks some coefficients to zero (feature selection).
• L2 (Ridge): Shrinks coefficients without setting them to zero.
