[Defense] A Mixed Emotions Framework and Its Applications in Affective Computing
Wednesday, April 17, 2024
10:30 am - 12:30 pm
In
Partial
Fulfillment
of
the
Requirements
for
the
Degree
of
Doctor
of
Philosophy
Amanveer
Wesley
will
defend
his
dissertation
A
Mixed
Emotions
Framework
and
Its
Applications
in
Affective
Computing
Abstract
Emotions play a crucial role in human behavior and there has been a long-standing effort in affective computing to identify displayed emotions through image analysis of facial expressions. Traditionally, such identification has been implemented through deep learning, with the neural networks returning for each facial image a probabilistic vector corresponding to the seven basic emotions: sadness, happiness, anger, fear, disgust, surprise, and neutral. We find this approach highly restrictive, as human emotions do not always fall neatly into these seven categories but many times cross categorical borders, forming interesting mixtures. To address this problem, we developed a methodology based on two components: 1) CNN for simple emotions: A convolutional neural network (CNN) that identifies the said seven emotions. We validated this CNN on RAVDESS, a well-known dataset of facial videos of people expressing distinct emotions while talking. This validation on dynamic imagery differentiates our method from other approaches that validate their neural networks on static facial images only. Such static imagery validation does not provide sufficient confidence for using the neural network in naturalistic experiments, where evolving behaviors are recorded continuously. 2) Co-occurrence matrix for mixed emotions: A post-hoc mixed emotion generation method, where the original seven-emotion probabilistic vector output by the CNN is used in an outer-product with itself to produce a co-occurrence matrix. The diagonal of this co-occurrence matrix holds the adjusted probabilities of the seven emotions, while the upper and lower triangles hold all their pair-wise combinations, which contain the corresponding probabilities of mixed emotions. In essence, the said methodology takes away probabilistic strength from the original seven-emotion vector and allocates it to mixed emotions. We tested the novel mixed-emotion vs. the conventional seven-emotion methodologies on two naturalistic experiments: First, in an experiment focusing on the emotional effects of email interruptions during cognitive work, and second, in an experiment focusing on the emotional effects of online public speaking. In the email interruption experiment, the mixed-emotion methodology uniquely determined that unlike knowledge workers who work uninterrupted, knowledge workers who are frequently interrupted by emails tend to display a mixture of sadness and fear on their faces. The latter is presumably the result of re-occurring negative stimuli in the form of random email arrivals. In the online public speaking experiment, the mixed-emotion methodology uniquely determined that more conscientious participants displayed a preponderance of mixed emotions with respect to less conscientious participants. As mixed emotions represent a moderation of pure emotions, like angry-neutral vs. totally angry, they constitute a more acceptable form of visual communication between the speaker and the audience. It makes sense that conscientious people adhere to this social etiquette, and the mixed-emotion methodology captures exactly this. Altogether, the mixed-emotion methodology significantly enhances analytical insight across different experimental settings, emerging as a valuable addition to the conventional seven-emotion methodology used in affective computing.
Wednesday,
April
17,
2024
10:30
AM
-
12:30
PM
CT
HBS, Room 302 and
Dr. Ioannis Pavlidis, dissertation advisor
Faculty, students, and the general public are invited.
