6.2 Facial Action Coding System (FACS) Theory
The AI used to examine the facial expression is developed on the basis of the Facial
Action Coding System (FACS). Theoretically, FACS is an analysis tool widely used
in research relating to the face, including facial expressions (Ekman & Friesen,
1978). In FACS theory, an action unit (AU) is used as one unit of facial movement.
A facial expression is expressed by a combination of several movements on the face,
but an AU corresponds to the smallest unit constituting the expression. The parts a
face is composed of—the eyebrows, eyelids, cheeks, lips, nose, mouth, chin, etc.—
are each further subcategorized and have their specific movements assigned an AU
number. For example, the “inner brow raiser” (AU1) and “outer brow raiser” (AU2)
have distinct AU numbers, despite being movements for the same general facial
feature (the eyebrow), because they account for different parts of it and have
different movements. AUs independently do not signify facial expressions; facial
expressions are expressed depending on a combination of AUs. For instance, the
“cheek raiser (AU6)” and “lip corner puller (AU12)” are the bases of the expression
of “happiness” (Kohler et al., 2004) Analyzing facial expressions with FACS
requires sufficient proficiency with the FACS theory, and precise analysis is necessary for even slight facial movements. As such, analysis can be a labor-intensive
process even with still images. Naturally, analyzing moving images frame by frame
is even more difficult.
6.3 Facial Expression Analysis Using Emotion
Recognition AI
Having AI perform automatic analysis is useful in facial expression analysis for data
like those from videos. CAC Corporation’s Kokoro Sensor is an emotion recognition
software that uses AFFDEX to perform automatic analysis on facial expressions and
moving parts of faces in videos and images (McDuff et al., 2016). AFFDEX is an
emotion recognition AI based on FACS theory and developed by a North American
company, Affectiva. AFFDEX detects key landmarks on the face to identify facial
features and, after extracting a texture model of the skin surface, classifies the surface
movements of the face by their AU. These AUs are then used while analyzing the
extracted facial model to deduce facial expressions (McDuff et al., 2013, 2016;
Senechal, McDuff, & Kaliouby, 2015). AFFDEX creates the output of two types of
information from this analysis: the emotion expressed and the facial movements
involved. There are seven possible categories of emotions: joy, sadness, anger,
surprise, fear, disgust, and disdain. Each of these emotions is given a value of
expressiveness from 0 (not expressed at all) to 100 (expressed). Analysis of facial
movements takes parts of the face, such as the eyebrows, eyes, nose, mouth, and
chin, and further subcategorizes them (e.g., outer part vs. inner part); the output is a
value in the range of 0–100, similar to the emotion values. The facial features and
46
Y. Ikeda et al.
The AI used to examine the facial expression is developed on the basis of the Facial
Action Coding System (FACS). Theoretically, FACS is an analysis tool widely used
in research relating to the face, including facial expressions (Ekman & Friesen,
1978). In FACS theory, an action unit (AU) is used as one unit of facial movement.
A facial expression is expressed by a combination of several movements on the face,
but an AU corresponds to the smallest unit constituting the expression. The parts a
face is composed of—the eyebrows, eyelids, cheeks, lips, nose, mouth, chin, etc.—
are each further subcategorized and have their specific movements assigned an AU
number. For example, the “inner brow raiser” (AU1) and “outer brow raiser” (AU2)
have distinct AU numbers, despite being movements for the same general facial
feature (the eyebrow), because they account for different parts of it and have
different movements. AUs independently do not signify facial expressions; facial
expressions are expressed depending on a combination of AUs. For instance, the
“cheek raiser (AU6)” and “lip corner puller (AU12)” are the bases of the expression
of “happiness” (Kohler et al., 2004) Analyzing facial expressions with FACS
requires sufficient proficiency with the FACS theory, and precise analysis is necessary for even slight facial movements. As such, analysis can be a labor-intensive
process even with still images. Naturally, analyzing moving images frame by frame
is even more difficult.
6.3 Facial Expression Analysis Using Emotion
Recognition AI
Having AI perform automatic analysis is useful in facial expression analysis for data
like those from videos. CAC Corporation’s Kokoro Sensor is an emotion recognition
software that uses AFFDEX to perform automatic analysis on facial expressions and
moving parts of faces in videos and images (McDuff et al., 2016). AFFDEX is an
emotion recognition AI based on FACS theory and developed by a North American
company, Affectiva. AFFDEX detects key landmarks on the face to identify facial
features and, after extracting a texture model of the skin surface, classifies the surface
movements of the face by their AU. These AUs are then used while analyzing the
extracted facial model to deduce facial expressions (McDuff et al., 2013, 2016;
Senechal, McDuff, & Kaliouby, 2015). AFFDEX creates the output of two types of
information from this analysis: the emotion expressed and the facial movements
involved. There are seven possible categories of emotions: joy, sadness, anger,
surprise, fear, disgust, and disdain. Each of these emotions is given a value of
expressiveness from 0 (not expressed at all) to 100 (expressed). Analysis of facial
movements takes parts of the face, such as the eyebrows, eyes, nose, mouth, and
chin, and further subcategorizes them (e.g., outer part vs. inner part); the output is a
value in the range of 0–100, similar to the emotion values. The facial features and
46
Y. Ikeda et al.