Introduction to

Face Processing with Computer Vision

Gabriel Bianconi
Founder, Scalar Research
AI & Data Science Consulting Firm

Previously at the Stanford AI Lab

Agenda
• Theory
• Detection
• Recognition
• Other Tasks

• Practice
• Rapid Prototyping
• Scaling

3
Theory
4
Face Detection

5
Haar-Like Features
• Summarize image based on simple color patterns
• Manually determined feature extractors (kernels)

• Leveraged for first real-time face detector (2001)

Ref: Viola & Jones (2001). Image: Wikimedia 6

7
8
Histogram of Oriented Gradients (HOG)
• Summarize image by distribution of color gradients
• Gradient intensities and orientations represent edges, etc.
• Captures more information than simple Haar-like features

Ref: Shu et al. (2011). 9

Ref: Shu et al. (2011) 10
Ref: Shu et al. (2011) 11
Ref: Rojas et al. (2011) 12
Ref: Rojas et al. (2011) 13
R-CNN
• Introduces CNNs for object detection
• CNNs learn how to extract features from data
• Breakthrough in performance
• Beats previous SOTA methods by huge margin
• However, detection is extremely slow

Ref: Girshick et al. (2014). 14

CNN Features

Ref: Lee et al. (2009). 15

CNN Features

Ref: Lee et al. (2009). 16

CNN Features

Ref: Lee et al. (2009). 17

CNN Features

Ref: Lee et al. (2009). 18

R-CNN

Ref: Girshick et al. (2014). 19

Fast R-CNN
• Improvement to R-CNN that leverages CNN for
classification and regression
• Other than proposing regions, system is now end-to-end vs. three
components trained greedily.
• Predictions are 200x+ faster with better performance
• Region proposals still are a bottleneck; total inference time is ~2s.

Ref: Girshick (2015). 20

Fast R-CNN

Ref: Girshick (2015). 21

Faster R-CNN
• Leverages CNN for region proposals as well
• “Region Proposal Network”
• Finally an end-to-end system with deep learning
• About 10x faster than Fast R-CNN, with better performance
• Total inference time is ~0.2s

Ref: Ren et al. (2016). 22

Faster R-CNN

Ref: Ren et al. (2016). 23

MTCNN
• Many model for face detection draw heavily from
the generalized object detection methods.
• MTCNN, for example, trains a multi-task system for
detection and alignment.

Ref: Zhang et al. (2015). 24

MTCNN

Ref: Zhang et al. (2015). 25

RetinaFace
• The current SOTA method combines many
techniques such as multi-task learning
• R-CNN family uses a two-stage approach
(proposals → refinement)
• RetinaFace uses a single-stage approach (faster,
higher recall, more false positives)

Ref: Deng et al. (2019). 26

RetinaFace

Ref: Deng et al. (2019). 27

Are we there yet?

WIDER Face (Easy) WIDER Face (Medium) WIDER Face (Hard)

~97% AP ~96% AP ~92% AP

Ref: Yang (2016). 28

Facial Recognition

29
Facial Recognition
• Facial recognition actually corresponds to group of
different tasks.
• Verification vs. Identification vs. Grouping vs. …
• Closed-Set vs. Open-Set

30
Closed-Set Recognition
• Every identity appears in training set
• Example: recognizing celebrities
• Effectively a classification problem
• Model aims to learn separable features

31
Closed-Set Identification

Test Sample Model Label Confidences

Label 0 Label 1 …

… …
Images: Wikimedia 32
Closed-Set Verification

Test Sample A Label Confidences

Model

Test Sample B Label Confidences

Images: Wikimedia 33
Open-Set Recognition
• Not every identity appears in training set
• Example: Facebook Photos
• Effectively a metric learning problem
• Model aims to learn large-margin features (embeddings)

34
Embeddings
• Map each sample to a vector (coordinate system)
• Used for words, graphs, faces, etc.
• Embeddings preserve similarity
• Similar samples close to each other
• Dissimilar samples far from each other

35
Images: Wikimedia 36
Embeddings
• “Similar” depends on the training data
• Same person, physical characteristic, etc.
• Embeddings represent latent information
• High-dimensional embeddings trained on large datasets
learn to represent latent information about the person (e.g.
physical characteristics)

37
Open-Set Identification

Test Sample Model Embedding + Distance

Emb. 0 Emb. 1 Emb. 2 …

Images: Wikimedia 38
Open-Set Verification

Test Sample A Embedding A

Distance
Model vs.
Threshold
Test Sample B Embedding B

Images: Wikimedia 39
Metric Learning

Ref: Liu et al. (2018) 40

Are we there yet?

LFW (Labeled Faces in the Wild)

Verification

99.85%+ accuracy

Ref: Yan et al. (2019); Learned-Miller et al. (2016) 41

 Cross-Factor
Facial Recognition

42
Cross-Age

Ref: Zheng et al. (2017) 43

Cross-Pose

Ref: Li et al. (2011) 44

Cross-Makeup

Ref: Chen et al. (2013) 45

Further Research

46
Security
• How do we deal with adversarial users?
• Real face goes undetected or misclassified
• Fake face gets recognized
• Private data is extracted from model
•…

47
Security

Ref: Grigory Bakunov (2017) 48

Biometrics & Multi-Modal Data
• How do we deal with…
• Identical twins?
• Plastic surgery?
• ...

Ref: Singh et al. (2010) 49

Ref: Singh et al. (2010) 50
Biometrics & Multi-Modal Data
• Combine with other biometric data
• Biometric traits (e.g. hand)
• Multiple sensors (e.g. 2D + 3D)
• Multiple pictures (e.g. viewpoints, sequences)
•…

Ref: Singh et al. (2010); Ross & Jain (2004); Ross & Govindarajan(2005) 51
Ref: Apple 52
Privacy
• How do we deal with…
• Models that can predict gender, race, …?
• Models that leak the data?
• Predictions without sharing the raw data?
•…

Ref: Singh et al. (2010) 53

Other Tasks

54
Alignment & Pose Estimation

Ref: Ruiz et al. (2018) 55

Face Landmarks

56
Classification

Neutral

Happy
Happy

57
3D Reconstruction

Ref: Sela et al. (2017) 58

Practice
59
Rapid Prototyping

60
 61
accuracy
…

ac
e
a ce
aF tF
in gh
t si
Re In
N
N et
N
TC ce
M Fa
n
tio
ni
Dozens of Tools

V
nC c og
pe re
O
c e_
fa
…

…
simplicity
APIs
• There are dozens of APIs providing low-cost face
processing at scale
• Most services charge less than $1 per 1000 images
• Depending on the use case, might be cheaper than provisioning GPUs
and deploying your own models (esp. if considering developer time)

• Often these APIs can achieve performance that’s

close to state-of-the-art

62
APIs – Example: Azure
• Detection
• Classification
• Gender, age, emotion, hair, smile, eyes, glasses, makeup, …
• Landmarks
• Pose Estimation
• Recognition
• Verification, identification, grouping, similarity search, …

63
Embeddings
• Face embeddings are typically used for open-set
recognition systems
• They can be leveraged to quickly train models for
downstream tasks (e.g. classification)
• Tools
• face_recognition (Github): extremely fast, reliable for frontal
• FaceNet: based on deep learning, strong across the board
64
Example – Facebook Photos
• Task: open-set face identification
• Strategy:
1. Detect faces and compute embeddings for known photos
of users; store for future use.
2. Whenever a photo is uploaded, do the same and compare
against known set.

65
Example – Detection

import face_recognition as fr

image = fr.load_image_file("file.jpg")

face_locations = fr.face_locations(image)

Ref: github.com/ageitgey/face_recognition 66
Example – Embedding

image = fr.load_image_file("file.jpg")

face_embedding = fr.face_encodings(image)[0]

Ref: github.com/ageitgey/face_recognition 67
Example – L2 Distance

- 0.31 0.59 0.69

0.31 - 0.52 0.63

0.59 0.52 - 0.50

0.69 0.63 0.50 -

Images: WikiMedia 68
Face Landmarks
• Face landmarks can also be quickly extracted with
pretrained models and used for a number of
downstream tasks.

69
Example – Face Landmarks

face_landmarks = fr.face_landmarks(image)[0]

print(face_landmarks.keys())
# left_eyebrow, right_eyebrow, lower_lip, top_lip, …

Ref: github.com/ageitgey/face_recognition 70
71
Example – Snapchat Filters
• Task: face manipulation
• Strategy:
1. Detect face and localize landmarks in image
2. Add objects, reshape image, etc. based on landmarks

72
Example – Snapchat Filters
from PIL import Image, ImageDraw
…
pil_image = Image.fromarray(image)
d = ImageDraw.Draw(pil_image, 'RGBA’)
lip_fill = (150, 0, 0, 128) # shade of red, 50% alpha
d.polygon(face_landmarks['top_lip'], fill=lip_fill)
d.polygon(face_landmarks['bottom_lip'], fill=lip_fill)

73
Scaling

75
Bias
• People & Demographics
• Is your training set… Coworkers? Single location?
• Environment
• Does it cover… Day and night? Seasons? Lighting
conditions? Backgrounds?
• Sensors
• Did you consider… Diverse hardware? Calibration?
Viewpoint (angle)? Resolution? Occlusion?
76
Optimizations
• It is often easier to simplify the real-world task than
drastically improve ML models.

77
Optimizations

Multiple model optimizations

($$$ in developer time, etc.)
Performance

Time (weeks)

78
Optimizations

Install a new light

($)
Performance

Time (weeks)

79
Risks
• What happens when your model makes a mistake?
• How can you deal with adversarial users?
• What is your threat model?

80
Other Considerations
• How do you handle…
• Model getting stale over time?
• Growing search space?
• Large amounts of real-time data?
• Detecting or tracking people vs. faces?
• Speed vs. cost vs. performance trade-offs?

81
 Thank you.
gabriel@scalarresearch.com

82