TAUCHI – Tampere Unit for Computer-Human Interaction

Automated recognition of facial

expressi ns and identity
2003 UCIT Progress Report

Ioulia Guizatdinova
Research Group for Emotions, Sociality, and Computing
University of Tampere
10.01.2004
 TAUCHI – Tampere Unit for Computer-Human Interaction

Contents

• Research problems

• Aims and Tasks

• Facial landmark extraction : Methods

• Facial landmark extraction : Results

• Future Steps
 TAUCHI – Tampere Unit for Computer-Human Interaction

Research Problems

“Automated recognition of facial expressions and identity”

• Face identification
• Recognition of facial expressions
 TAUCHI – Tampere Unit for Computer-Human Interaction

Research Problems
Face identification
• Classification of input face to one of existing face classes stored in
database

• Rejection of input face as unrecognized/unknown face

Face database
Input face Face classes
still image/video signal

classification 1 Person A
Recognition
system
… … … … …
rejection
N Person Z

Unrecognized face
 TAUCHI – Tampere Unit for Computer-Human Interaction

Research Problems
Recognition of facial expressions

• Facial expressions affect face recognition because a variability of

facial landmarks in their appearance is high
• Humans are good in recognizing facial identity regardless of changes
in facial expressions
• Computer-aided systems of face recognition are dramatically
compromised by changes in facial expressions
 TAUCHI – Tampere Unit for Computer-Human Interaction

Aim of research

• The primary aim of this research is to do theoretical and experimental

investigation on the possibilities to automatically recognize facial
identity independent of changes in facial expressions

• For that purpose two 2D recognition systems will be developed

- Recognition system of facial expressions

- Expression-invariant system of facial identity recognition
 TAUCHI – Tampere Unit for Computer-Human Interaction

Tasks

• Extraction and classification of facial landmarks, namely, regions of

eyes/eye-brows, nose, and mouth from still images

• Detection and recognition of facial expressions - how facial muscle

activations can change appearance of a face during emotional
reactions?

• Expression-invariant recognition of facial identity

 TAUCHI – Tampere Unit for Computer-Human Interaction

Methods of landmark extraction

Feature-b ased method
• Uses knowledge on geometrical structure of human faces

• Based on geometrical features of facial landmarks, such as position

of eyes/eye-brows, nose, and mouth

• Four regions of interest have been selected

as most informative for further recognition
steps

– right eye-brow / eye

– left eye-brow / eye
– nose
– mouth
 TAUCHI – Tampere Unit for Computer-Human Interaction

Methods of landmark extraction

15 0
1
Template-b ase method 14 2
13 5 3
22.
12 ° 4
• Represents a face as a feature 11 5
map/template of original facial image
10 6
9 8 7
• Local oriented edges are used to construct a
feature map of the facial image

• Orientation of edges has been determined

with step of 22.5° and encoded as 0,1,…..15

Oriented edges extracted

in left eye region
 TAUCHI – Tampere Unit for Computer-Human Interaction

Methods of landmark extraction

Database

• Tests were performed using Pictures of Facial Affect [1]

• 110 images with 7 basic facial displays: happiness, surprise, fear,

anger, disgust and neutral expression

• Images were first normalized to three pre-set sizes 100X150,

200X300 and 300X400 in order to test the effect of image size to the
operation of the algorithms

• In sum 110 x 3 = 330 images were used for algorithm testing

[1] Ekman, P., Friesen, W. V., & Hager, J.C. (2002) Facial Action Coding System (FACS).
Published by A Human Face, Salt Lake City, UTAH: USA
 TAUCHI – Tampere Unit for Computer-Human Interaction

Facial landmark extraction

Pre-processing algorithms

• RGB – grey-level transformation

• Multiresolution image representation was performed using a
recursive Gauss transformation

Different resolution levels

Resolution level 2 Resolution level 0

Transformation

Input face (RGB)

Normalization (grey-
level scale)
 TAUCHI – Tampere Unit for Computer-Human Interaction

Facial landmark extraction

• Final feature map has been constructed on base of local oriented

edges extracted in each point of grey-level image at each resolution
level with exception of points which had the contrast values less than
threshold
• Extraction of local edges has been performed by calculation of
difference between two oriented Gaussians with shifted kernels, which
allows determining both orientation and contrast of local edge

Points of interest have been grouped

- if the distance between points of
interest was less than the threshold
the points were grouped - otherwise
ignored

Map of detected points of interest Final feature map

 TAUCHI – Tampere Unit for Computer-Human Interaction

Facial landmark extraction

Orientation portraits of the facial

• Detected regions of interest have been landmarks

compared with orientation portraits of facial 500

400
Right Eye
landmarks I have constructed earlier 300
200

• Regions which did not correspond to the

Number of points of interest

100
0
portraits have been ignored 400
300 Left Eye
200
100

• Pre-knowledge about facial structure have 0

400
been used 300 Nose
200
100
0
400
Mouth
300
matching 200
100
0
2 3 4 5 6 10 11 12 13 14
Edge orientation
 TAUCHI – Tampere Unit for Computer-Human Interaction

Facial landmark extraction

Finally, facial landmarks have been detected!

neutral disgust fear

Examples of feature maps of high-contrast oriented edges detected from the expressive images
 TAUCHI – Tampere Unit for Computer-Human Interaction

Results

100X150 200X300 300X450

100
Performance (%)

90

80

70
L eye R eye Nose Mouth

Performance of the facial landmark detection algorithm averaged by all

expressions for three image sizes
 TAUCHI – Tampere Unit for Computer-Human Interaction

Results

100X150 200X300 300X450

100
Performance (%)

90

80

70
L eye R eye Nose Mouth

Performance of the facial landmark detection algorithm averaged by all

expressions for three image sizes
 TAUCHI – Tampere Unit for Computer-Human Interaction

Results

(a) (b)

Right Eye Left Eye

(c) (d)

Nose Mouth

Performance of the feature detection system for three image sizes.

N-neutral; H-happiness; Sd-sadness; F-fear; A-anger; Sr-surprise; D-disgust
 TAUCHI – Tampere Unit for Computer-Human Interaction

Results

(a) (b)

Right Eye Left Eye

(c) (d)

Nose Mouth

Performance of the feature detection system for three image sizes.

N-neutral; H-happiness; Sd-sadness; F-fear; A-anger; Sr-surprise; D-disgust
 TAUCHI – Tampere Unit for Computer-Human Interaction

Results
Problems
• Algorithms are slow – about few seconds
• Errors in groupping points of interest (red rectangles a, b, c)
• Some landmarks are undetectable (d)

(a) (b) (c) (d)

 TAUCHI – Tampere Unit for Computer-Human Interaction

Results
Recommendations

• To improve detection of nose and mouth regions two alternatives

are proposed

- The first one is selection of different thresholds for detection and

groupping of points of interest for different resolution levels

- The second alternative requires more careful processing of detected

regions and searching different landmark parts such as eye and mouth
corners and nostrils.
 TAUCHI – Tampere Unit for Computer-Human Interaction

Future steps

• Full article about automated expression-invariant detection of facial

landmarks; short article about how emotions affect cognitive
functioning and how this knowledge might be implicated for HCI

• To improve landmark detection; implement prototype of 2D

recognition system of facial expressions (iExpRec)

• To implement and test iExpRec

• To implement of expression-invariant 2D facial identity recognition

system (iFaceRec).
 TAUCHI – Tampere Unit for Computer-Human Interaction

Thank you f r your attention!