AI VIETNAM

All-in-One Course

Module 10 - Project

Multi-Task Learning

AI VIET NAM
Nguyen Quoc Thai

1
Year 2023
 Objectives
! Multi-task Learning for Computer Vision

Task 1 Training Data Model

Generalization

Feature-based MTL
Task 2 Training Data Model
Parameter-based MTL Generalization

Task 3 Training Data Model

Generalization
2
 Outline
Ø Introduction
Ø Deep Multi-Task Architectures
Ø Optimization Strategy
Ø Experiment

3
 Introduction
! Single-Task Learning

Ø Image Classification

MODEL Class: CAT

(LeNet, ResNet,…)

4
 Introduction
! Single-Task Learning

Ø Image Segmentation

0 0 0 0 0 0 0 0
0 1 1 0 0 0 0 0
0 1 1 1 0 2 2 0

MODEL 0 1 1 1 0 2 2 0

(UNet) 0 1 1 1 2 2 2 0
0 1 1 1 1 2 2 0
1 1 1 1 1 2 2 0
0 0 0 0 0 0 0 0

DOG CAT
5
 Introduction
! Single-Task Learning

Ø Object Detection

DOG – 0.98 CAT – 0.87

MODEL
(UNet)

Assign labels, bounding boxes

to objects in the image
6
 Introduction
! Single-Task Learning

Task 1 Training Data Model

Training Generalization

Task 2 Training Data Model

Training Generalization

Task 3 Training Data Model

Training Generalization
7
 Introduction
! Multi-Task Learning

Task 1 Training Data Model

Generalization

Task 2 Training Data Model

Training Generalization

Task 3 Training Data Model

Generalization
8
 Introduction
! Motivation

Ø Learning multiple tasks jointly with the aim of mutual benefit

Ø Improves generalization on other tasks
Caused by the inductive bias provided by the auxiliary task

9
 Introduction
! Multi-Task Learning

Task 1 Training Data Model

Generalization

What to Share?
Task 2 Training Data Model
Generalization
How to Share?

Task 3 Training Data Model

Generalization
10
 Introduction
! MTL Methods (based on what to share?)

Ø Feature-based MTL
o Aims to learn common features among different tasks
Ø Parameter-based MTL
o Learns model parameters to help learn parameters for other tasks
Ø Instance-based MTL
o Identify useful data instances in a task for other task

11
 Introduction
! MTL Methods (based on how to share?)

Ø Feature-based MTL
o Feature learning approach
o Deep learning approach
Ø Parameter-based MTL
o Low-Rank approach

12
 Introduction
! Feature Learning Approach

Ø Why need to learn common feature representations?

o Original features may not have enough expressive power
Ø Two sub-categories
o Feature transformation approach
o Feature selection approach

13
 Introduction
! Feature Learning Approach

Ø Feature transformation approach

o The learned features are a linear or nonlinear transformation of the original
feature representation
o Multi-task feedforward NN

Input 1 Output for task 1

Input d Output for task 2

14
 Introduction
! Feature Learning Approach

Ø Feature selection approach

o Select a subset of the original features as the learned representation
o Eliminates useless features based on different criteria

15
 Introduction
! Low-Rank Approach

Ø Assumes the model parameters of different

tasks share a low-rank subspace

16
 Introduction
! Deep Learning Approach

Ø Deep Multi-Task Architectures

o Encoder-Focused
o Decoder-Focused
Ø Optimization Strategy Methods
o Task Balancing
o Other: Heuristics, Gradient Sign Dropout

17
 Outline
Ø Introduction
Ø Deep Multi-Task Architectures
Ø Optimization Strategy
Ø Experiment

18
 Deep Multi-Task Architectures
! Deep Multi-Task Architectures used in Computer Vision

Deep Multi-Task
Architectures

Encoder-Focused Decoder-Focused Other

MTL Baseline PAD-Net

Cross-Stitch Networks ASTMT
PAP-Net
NDDR-CNN MTI-Net
MTAN

19
 Deep Multi-Task Architectures
! Encoder-Focused

Ø Share the task features in the encoding stage

Task A Task B Task C

Task specific

Shared Encoder
(Soft/Hard)

20
 Deep Multi-Task Architectures
! Encoder-Focused

Ø Hard Parameter Sharing

o Generally applied by sharing the hidden layers between all tasks
o Keep several task-specific output layers

Task A Task B Task C

Task specific

21
 Deep Multi-Task Architectures
! Encoder-Focused

Ø Soft Parameter Sharing

o Each task has its own model with its own parameters
o Uses a linear combination in every layer of the task-specific networks

Task A Task B Task C

Task specific

22
 Deep Multi-Task Architectures
! Encoder-Focused

Ø Cross-Stitch Networks
o Shared the activations amongst all single-task networks in the encoder

Task A Task B Task A Task B

+ 𝛼 𝛼 +

Share Parameters

23
 Deep Multi-Task Architectures
! Encoder-Focused

Ø Cross-Stitch Networks
o Shared the activations amongst all single-task networks in the encoder
o Cross connection

Task A Task B Task A Task B

+ 𝛼 𝛼 + + Conv Conv +

Conv Conv Conv Conv

24
 Deep Multi-Task Architectures
! Encoder-Focused

Ø Multi-Task Attention Networks

o Used a shared backbone network in conjunction with task-specific attention
modules in the encoder
Task B Task C

Task specific

Shared Encoder Attention Module Attention Module

Attention Module Attention Module

25
 Deep Multi-Task Architectures
! Decoder-Focused

Task A Task B Task C

Task A Task B Task C Task specific

Shared Encoder
(Soft/Hard)

26
 Deep Multi-Task Architectures
! Decoder-Focused

Ø PAD-Net
o Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous
Depth Estimation and Scene Parsing

27
 Deep Multi-Task Architectures
! Decoder-Focused

Ø PAD-Net
o Deep Multimodal Distillation

28
 Outline
Ø Introduction
Ø Deep Multi-Task Architectures
Ø Optimization Strategy
Ø Experiment

29
 Optimization Strategy
! Task Balancing Approaches

Ø Set a unique weight for each task

ℒ!"# = # 𝑤$ . ℒ$
$

Ø Use SGD to minimize the objective

𝜕ℒ$
𝑊%&'()* = 𝑊%&'()* − 𝛾 # 𝑤$
𝜕𝑊%&'()*
$

30
 Optimization Strategy
! Uncertainty Weighting

Ø Use the homoscedastic uncertainty to balance the single-task losses

Ø Optimize the model weights W and noise parameters

1 1
ℒ W, σ+ , 𝜎, = , ℒ+ 𝑊 + , ℒ, 𝑊 + log 𝜎+ 𝜎,
2𝜎+ 2𝜎,

31
 Optimization Strategy
! Dynamic Weight Averaging (DWA)

Ø Learns to average task weighting over time by considering the rate of change of loss
for each task
Training Time Relative Loss Change

r- t − 1
N exp T L.(t − 1)
w- t = , r. t − 1 =
r t−1 L.(t − 2)
∑. exp .
T

Temperature
(Softness of Task Weighting)

32
 Optimization Strategy
! Other methods

Ø Gradient Normalization
Ø Dynamic Task Prioritization

33
Quiz

34
 Outline
Ø Introduction
Ø Deep Multi-Task Architectures
Ø Optimization Strategy
Ø Experiment

35
 Experiment
! NYUD-v2 Dataset

36
 Experiment
! Model

Task A Task B Task C Task A Task B Task C

Hard Parameter Sharing Soft Parameter Sharing

37
 Experiment
! Code

38
 Summary

Deep Multi-Task
Optimization Strategy
Architectures

Encoder-Focused Decoder-Focused Other Task Balancing

MTL Baseline PAD-Net Uncertainty Weighting

Cross-Stitch Networks ASTMT Gradient Normalization
PAP-Net
NDDR-CNN MTI-Net DWA
MTAN DTP

39
Thanks!
Any questions?

40