With MimiQgame, we developed a technical proof of concept (TPOC) of a facial retraining game as a support in the rehabilitation of individuals with facial palsy, as there is an international demand for an innovative format of classical movement training of the facial muscles in cases of peripheral facial paralysis (PFP). Daily independent practice with sufficient variation is crucial; however, existing methods are poorly motivating, insufficiently adaptable to individual needs, and provide little targeted feedback. This results in time-intensive preparation and treatment. Practicing in front of a mirror—at home the only alternative to therapist feedback—is experienced by patients as confrontational and often encourages forceful or exaggerated movements, which are counterproductive to recovery.
Despite successful implementation within movement rehabilitation, gamification has not yet been applied in facial retraining. MimiQgame offers therapists a tool that uses gamification to provide varied, tailored exercises at an individually adjustable level (e.g. a small, symmetrical smile). As an alternative to the mirror, patients are presented with a series of exercises in the form of a game on an iPad. The game is controlled by the patient’s facial expressions. In this way, MimiQgame can increase patient adherence while allowing therapists to work far more time-efficiently and cost-effectively.
MimiQgame is developed using facial motion capture and represents a technical exploration of what a gamified exercise for people with PFP could look like. Within this project, two concepts were developed into Technical Proofs of Concept (TPOCs), each with its own focus. The first game emphasizes the therapeutic aspect of the exercises, specifically the repetition and correct execution of a symmetrical smile. The second game focuses more on the “fun” element in order to break the repetitive nature of the exercises. In this exercise, the emphasis lies on training different facial expressions, with a focus on unpredictability—namely, which expressions follow one another.
Both TPOC mini-games incorporate various aspects of rehabilitation such as repetition, functionality, facial expression, variation, and unexpected triggers. These mini-games were presented within a playful theme to counteract the repetitive and discouraging nature of therapeutic exercises.
The overall story or theme of the mini-game forms only a small part of the game and serves primarily as visual and narrative dressing, without further functional weight. The general theme is “floating islands.” Within this context, the patient takes on the role of an air mail carrier, facing challenges encountered by the postman and his airship. This theme was chosen because it appeals to a broad age range, from 1 to 99 years.
This project is the result of a collaboration between Artevelde University of Applied Sciences and DAE Research.
Game concept 01 - Floating islands
The goal of this exercise is to practice symmetrical smiling and maintaining the smile using visual triggers. This game focuses on imitation, repetition, symmetrical smiling (without synkinesis), and live feedback.
The patient is seated in an airship and must reach the floating island in front of them by flying past all the islands and delivering their mail. The player moves the ship forward by raising the wings of the airship. The patient does this by smiling symmetrically, which automatically raises the wings through the facial motion capture system that tracks the patient’s mouth corners. The wings visualize the movement of the two mouth corners; in other words, the movement of the left wing is driven by the left mouth corner. If the left corner moves upward, the left wing of the ship moves upward. The ship only moves forward when both corners are raised symmetrically. If the movement is not symmetrical, the vehicle will not advance.
After correctly performing a number of symmetrical smile movements, the patient reaches the other side. The impulse to smile symmetrically occurs spontaneously, so as not to rush the patient. After each smile, there is a short “cooldown” period during which the patient must relax their mouth. Mouth movements during this cooldown period are not registered. Each successful execution of the movement (with an accepted margin of error) counts as one step forward. The length of this step is determined by the quality of the smile and calculated accordingly. This is then visualized by the airship floating closer to the island ahead. The player can track their progress via a progress bar.
Before the exercise begins, the patient is shown instructions to explain the exercise: which movement is being trained, the goal of the game, and how to play. These instructions are illustrated by a mentor character who demonstrates the symmetrical smile. The visual style of the game is also shown so the patient knows what to expect and how the game will respond. During the tutorial, the patient can imitate the mentor’s symmetrical smile while simultaneously seeing the live feedback of the game and the airship on the screen
GAME CONCEPT 02 – Tetris
The purpose of this game is to repeat exercises learned during therapy in a playful manner. The main focus is on fun for both young and old. The game concept is very similar to the classic version of Tetris. The player must arrange Tetris blocks to form lines in order to score points.
However, this version introduces variation in how blocks are obtained. Instead of receiving blocks at random, the player must form the facial expression shown on the screen within a predefined time window. These expressions may vary between lip pursing, smiling, and pulling the mouth corners downward. Depending on the patient’s abilities, these facial expressions can be adjusted in the parameter menu.
If the movement is performed correctly within the time frame, the block can be played. If not, the patient receives a “penalty block.” This block is randomly generated and colored red. A mistake counter keeps track of errors, providing the patient with insight into their unsuccessful attempts.
After selecting a block using the required facial expression, the patient can control the block using on-screen buttons: one long button in the center for rotating the blocks, and one button on each side of the screen for moving left and right. Rest moments are integrated both during the falling of the blocks and through the option to pause the game.
This game also incorporates the narrative theme of the “air mail carrier.” The postman must still unlock his mail, which can be achieved by forming lines with the blocks. As a result, both the score and the mail counter increase.
