21CS743 | DEEP LEARNING | SEARCH CREATORS.
Module-02
Feedforward Networks and Deep Learning
Introduction to Feedforward Neural Networks
1.1 Basic Concepts
• A feedforward neural network is the simplest form of artificial neural network (ANN)
• Information moves in only one direction: forward, from input nodes through hidden nodes
to output nodes
• No cycles or loops exist in the network structure
1.2 Historical Context
1. Origins
o Inspired by biological neural networks
o First proposed by Warren McCulloch and Walter Pitts (1943)
o Significant advancement with perceptron by Frank Rosenblatt (1958)
2. Evolution
o Single-layer to multi-layer networks
o Development of backpropagation in 1986
o Modern deep learning revolution (2012-present)
Search Creators... Page 1
21CS743 | DEEP LEARNING | SEARCH CREATORS.
1.3 Network Architecture
1. Input Layer
o Receives raw input data
o No computation performed
o Number of neurons equals number of input features
o Standardization/normalization often applied here
2. Hidden Layers
o Performs intermediate computations
o Can have multiple hidden layers
o Each neuron connected to all neurons in previous layer
Search Creators... Page 2
21CS743 | DEEP LEARNING | SEARCH CREATORS.
o Feature extraction and transformation occur here
3. Output Layer
o Produces final network output
o Number of neurons depends on problem type
o Classification: typically one neuron per class
o Regression: usually one neuron
1.4 Activation Functions
1. Sigmoid (Logistic)
o Formula: σ(x) = 1/(1 + e^(-x))
o Range: [0,1]
o Used in binary classification
o Properties:
▪ Smooth gradient
▪ Clear prediction probability
▪ Suffers from vanishing gradient
2. Hyperbolic Tangent (tanh)
o Formula: tanh(x) = (e^x - e^(-x))/(e^x + e^(-x))
o Range: [-1,1]
o Often performs better than sigmoid
o Properties:
▪ Zero-centered
Search Creators... Page 3
21CS743 | DEEP LEARNING | SEARCH CREATORS.
▪ Stronger gradients
▪ Still has vanishing gradient issue
3. ReLU (Rectified Linear Unit)
o Formula: f(x) = max(0,x)
o Most commonly used
o Helps solve vanishing gradient problem
o Properties:
▪ Computationally efficient
▪ No saturation in positive region
▪ Dying ReLU problem
4. Leaky ReLU
o Formula: f(x) = max(0.01x, x)
o Addresses dying ReLU problem
o Small negative slope
o Properties:
▪ Never completely dies
▪ Allows for negative values
▪ More robust than standard ReLU
2. Gradient-Based Learning
2.1 Understanding Gradients
1. Definition
Search Creators... Page 4
21CS743 | DEEP LEARNING | SEARCH CREATORS.
o Gradient is a vector of partial derivatives
o Points in direction of steepest increase
o Used to minimize loss function
2. Properties
o Direction indicates fastest increase
o Magnitude indicates steepness
o Negative gradient used for minimization
2.2 Cost Functions
1. Mean Squared Error (MSE)
o Used for regression problems
o Formula: MSE = (1/n)Σ(y_true - y_pred)²
o Properties:
▪ Always positive
▪ Penalizes larger errors more
▪ Differentiable
2. Cross-Entropy Loss
o Used for classification problems
o Formula: -Σ(y_true * log(y_pred))
o Properties:
▪ Measures probability distribution difference
▪ Better for classification than MSE
Search Creators... Page 5
21CS743 | DEEP LEARNING | SEARCH CREATORS.
▪ Provides stronger gradients
3. Huber Loss
o Combines MSE and MAE
o Less sensitive to outliers
o Formula:
▪ L = 0.5(y - f(x))² if |y - f(x)| ≤ δ
▪ L = δ|y - f(x)| - 0.5δ² otherwise
2.3 Gradient Descent Types
1. Batch Gradient Descent
o Uses entire dataset for each update
o More stable but slower
o Formula: θ = θ - α∇J(θ)
o Memory intensive for large datasets
2. Stochastic Gradient Descent (SGD)
o Updates parameters after each sample
o Faster but less stable
o Better for large datasets
o High variance in parameter updates
3. Mini-batch Gradient Descent
o Compromise between batch and SGD
o Updates parameters after small batches
Search Creators... Page 6
21CS743 | DEEP LEARNING | SEARCH CREATORS.
o Most commonly used in practice
o Typical batch sizes: 32, 64, 128
4. Advanced Optimizers a) Adam (Adaptive Moment Estimation)
o Combines momentum and RMSprop
o Adaptive learning rates
o Formula includes first and second moments
b) RMSprop
o Adaptive learning rates
o Divides by running average of gradient magnitudes
c) Momentum
o Adds fraction of previous update
o Helps escape local minima
o Reduces oscillation
3. Backpropagation and Chain Rule
3.1 Chain Rule Fundamentals
1. Mathematical Basis
o df/dx = df/dy * dy/dx
o Allows computation of composite function derivatives
o Essential for neural network training
2. Application in Neural Networks
o Computes gradients layer by layer
Search Creators... Page 7
21CS743 | DEEP LEARNING | SEARCH CREATORS.
o Propagates error backwards
o Updates weights based on contribution to error
3.2 Forward Pass
1. Input Processing
o Data normalization
o Weight initialization
o Bias addition
2. Layer Computation
python
Copy
# Pseudo-code for forward pass
for layer in network:
Z = W * A + b # Linear transformation
A = activation(Z) # Apply activation function
3. Output Generation
o Final layer activation
o Prediction computation
o Error calculation
3.3 Backward Pass
1. Error Calculation
o Compare output with target
Search Creators... Page 8
21CS743 | DEEP LEARNING | SEARCH CREATORS.
o Calculate loss using cost function
o Initialize gradient computation
2. Weight Updates
o Calculate gradients using chain rule
o Update weights: w_new = w_old - learning_rate * gradient
o Update biases similarly
3. Detailed Steps
python
Copy
# Pseudo-code for backward pass
# Output layer
dZ = A - Y # For MSE
dW = (1/m) * dZ * A_prev.T
db = (1/m) * sum(dZ)
# Hidden layers
dZ = dA * activation_derivative(Z)
dW = (1/m) * dZ * A_prev.T
db = (1/m) * sum(dZ)
4. Regularization for Deep Learning
4.1 L1 Regularization
Search Creators... Page 9
21CS743 | DEEP LEARNING | SEARCH CREATORS.
1. Mathematical Form
o Adds absolute value of weights to loss
o Formula: L1 = λΣ|w|
o Promotes sparsity
2. Properties
o Feature selection capability
o Produces sparse models
o Less sensitive to outliers
4.2 L2 Regularization
1. Mathematical Form
o Adds squared weights to loss
o Formula: L2 = λΣw²
o Prevents large weights
2. Properties
o Smooth weight decay
o No sparse solutions
o More stable training
4.3 Dropout
1. Basic Concept
o Randomly deactivate neurons
o Probability p of keeping neurons
Search Creators... Page 10
21CS743 | DEEP LEARNING | SEARCH CREATORS.
o Different network for each training batch
2. Implementation Details
python
Copy
# Pseudo-code for dropout
mask = np.random.binomial(1, p, size=layer_size)
A = A * mask
A = A / p # Scale to maintain expected value
3. Training vs. Testing
o Used only during training
o Scaled appropriately during inference
o Acts as model ensemble
4.4 Early Stopping
1. Implementation
o Monitor validation error
o Save best model
o Stop when validation error increases
2. Benefits
o Prevents overfitting
o Reduces training time
o Automatic model selection
Search Creators... Page 11
21CS743 | DEEP LEARNING | SEARCH CREATORS.
5. Advanced Concepts
5.1 Batch Normalization
1. Purpose
o Normalizes layer inputs
o Reduces internal covariate shift
o Speeds up training
2. Algorithm
python
Copy
# Pseudo-code for batch normalization
mean = np.mean(x, axis=0)
var = np.var(x, axis=0)
x_norm = (x - mean) / np.sqrt(var + ε)
out = gamma * x_norm + beta
5.2 Weight Initialization
1. Xavier/Glorot Initialization
o Variance = 2/(nin + nout)
o Suitable for tanh activation
2. He Initialization
o Variance = 2/nin
o Better for ReLU activation
Search Creators... Page 12
21CS743 | DEEP LEARNING | SEARCH CREATORS.
6. Practical Implementation
6.1 Network Design Considerations
1. Architecture Choices
o Number of layers
o Neurons per layer
o Activation functions
2. Hyperparameter Selection
o Learning rate
o Batch size
o Regularization strength
6.2 Training Process
1. Data Preparation
o Splitting data
o Normalization
o Augmentation
2. Training Loop
o Forward pass
o Loss computation
o Backward pass
o Parameter updates
Practice Problems and Exercises
Search Creators... Page 13
21CS743 | DEEP LEARNING | SEARCH CREATORS.
1. Basic Concepts
o Explain the role of activation functions in neural networks
o Compare and contrast different types of gradient descent
o Describe the vanishing gradient problem
2. Mathematical Problems
o Calculate gradients for a simple 2-layer network
o Implement batch normalization equations
o Compute different loss functions
3. Implementation Challenges
o Design a network for MNIST classification
o Implement dropout in Python
o Create a custom loss function
Key Formulas Reference Sheet
1. Activation Functions
o Sigmoid: σ(x) = 1/(1 + e^(-x))
o tanh(x) = (e^x - e^(-x))/(e^x + e^(-x))
o ReLU: f(x) = max(0,x)
2. Loss Functions
o MSE = (1/n)Σ(y_true - y_pred)²
o Cross-Entropy = -Σ(y_true * log(y_pred))
3. Regularization
Search Creators... Page 14
21CS743 | DEEP LEARNING | SEARCH CREATORS.
o L1 = λΣ|w|
o L2 = λΣw²
4. Gradient Descent
o Update: w = w - α∇J(w)
o Momentum: v = βv - α∇J(w)
Common Issues and Solutions
1. Vanishing Gradients
o Use ReLU activation
o Implement batch normalization
o Try residual connections
2. Overfitting
o Add dropout
o Use regularization
o Implement early stopping
3. Poor Convergence
o Adjust learning rate
o Try different optimizers
o Check data normalization
Search Creators... Page 15
