Powder Teams Up with Academia to Teach AI How to Read Emotions
Powder’s AI Research Team, headed by Liam Schoneveld, published two academic papers in collaboration with Dr. Alice Othmani, associate AI professor at Université Paris-Est specialized in emotion recognition. Together, they worked on an AI solution that would allow for improved detection of human emotions. Today, this technology is a core component of the Powder app. Here’s a story of how Powder’s AI Research Team took on the task of making AI ‘emotionally intelligent’.
Our work on emotion recognition began with the predecessor of Powder: Gust — an app that enabled users to react to stories using their facial expressions. While the task of ‘reading’ human emotions is intuitive to people, we learned firsthand that the technology required to succeed at emotional intelligence is challenging to perfect.
The human face conveys a significant amount of information. Through facial expressions, our faces are able to communicate numerous nuanced sentiments without the need for words. When it comes to training AI to read these sentiments, researchers have encountered a persistent problem — subjectivity of emotions. That is, humans interpret each other’s facial expressions in different ways. So when it comes to labelling datasets, which serve to teach the AI to distinguish between, say, a sad or a happy face, the final algorithms end up being biased in the same way that the people who ‘taught’ them were.
Liam and Alice were confronted with the very same challenge. Leveraging their expertise in applied and academic AI research respectively, they joined forces to create a model that would achieve a more precise reading of emotions by overcoming the subjectivity bias. This is how DeepFEVER, their facial expression recognising ML model, was born.
Instead of training on just one dataset of face images — as it is usually done in this kind of research — DeepFEVER was trained simultaneously on three of them: AffectNet, Google Facial Expression Comparison (FEC), as well as an internal dataset developed at Powder.
DeepFEVER proved to be more ‘impartial’ in reading emotions than analogous models, demonstrating superior performance in identifying and categorizing facial expressions across all three datasets (as well as a fourth one, introduced post-training to validate the results). Moreover, thanks to its limitless learning potential, a ML model like DeepFEVER is flexible by definition and can be easily repurposed for any task requiring facial expression understanding. While initially trained to distinguish between some basic emotions, with relatively little data it could eventually be taught to, for example, detect more fine-tuned sentiments (e.g. ‘If excited, then in what way? — see image below).
Today, DeepFEVER is the key component of our recent app feature, the Twitch Highlight Generator, which allows to detect the best moments of a Twitch stream based on the streamer’s emotional reactions. In this way, AI / ML become very useful it comes to extracting the most exciting short highlights from hours of stream, ready to be edited and shared on Powder.
Taking stock, by leveraging the research and continuously iterating on the model, we’ve been learning a great deal about what constitutes a streaming highlight worth sharing. And while with the Twitch Highlights Generation we’re not yet able to predict human emotions, the ambition to go further is there. The research behind DeepFEVER was an important component to delivering some really cool features to the community and ultimately taking us closer to our ultimate goal of becoming the camera of the metaverse.
—
Latest publication ‘Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition’, to be found here. It will be presented by Liam at the upcoming 2021 IEEE International Conference on Image Processing.
To learn more about Powder, check out our website.
