MediaPipe tutorial: Find memes that match your facial expression 😮

A post by Pierre Fenoll, Senior Lead Back-End Engineer at Powder.gg

Introduction

Powder.gg is a startup located in the beautiful district of Marais in Paris. We are working hard to help gamers edit their highlights with machine learning on mobile. AI, memes, sound and visual effects make up the toolkit we aim to provide to people all around the world to help them understand each other better. Our team has been using MediaPipe extensively to create our machine learning enabled video editing app for gamers.We created this tutorial to show how you can use MediaPipe to create your next ML enabled app.

Goals and requirements

Say you’re chatting on your phone with some friends. Your friend, Astrid wrote something amusing and you wanted to post a funny image related to hers . There’s this picture you really want to post but it’s not on your phone and now you’re just trying different words in your search engine of choice without any success. This image is so clear in your mind, the words to look it up just don’t come out right somehow. If only you could just make the same hilarious face to your phone and it would find it for you!

Let’s build a computer vision pipeline that compares a face from a phone’s front camera (for instance) with a collection of images containing at least a face: a collection of Internet memes.

Here’s the app we’re building.

Machine Learning model for our pipeline

We need a machine learning model that can determine how similar 2 images with facial expressions are. The emotions/facial expression recognition module we use is based on a 2019 paper published by Raviteja Vemulapalli and Aseem Agarwala, from Google AI, titled A Compact Embedding for Facial Expression Similarity. At Powder, we re-implemented the approach described in that paper. We then improved on this approach by using knowledge distillation, a technique whereby an often smaller student network is trained to mimic the predictions made by a teacher network. We have found that using knowledge distillation leads to an…