NeuroPlay
Rehabilitation exercises that answer to movement.
- Services
- Interaction designPrototyping in codeComputer vision and sensors
- Sector
- Healthcare
- Touchpoint
- Web appCameraScreen
- Year
- 2026
- Built with
- MediaPipeGoogle AI Studio
Made with
Turning movement into interaction.
NeuroPlay is a system for motor and facial rehabilitation that turns the exercises prescribed by the therapist into short interactive games.
I designed the interaction around one precise constraint: the game only works when the patient performs the movement being trained.
The camera tracks hands, face and eyes in real time through MediaPipe and uses that data to drive the experience. The movement of the patient becomes the real controller, with no additional devices.
At the end of the session, the data collected is used to generate a performance summary and a PDF report automatically.
Process
The problem was not simply to make the exercises more fun.
A game can reward a behaviour other than the one the exercise intends, and in that case the interaction risks interfering with the therapy.
So I designed every experience around a single target movement, avoiding alternative mechanics that would let the same result be reached with an easier one.
Three design decisions came out of that principle: short games that can be composed, interaction based on movement alone, and immediate feedback.
Solutions
Five games, one movement each
Every minigame trains one specific movement.
Hand Flower opens and closes a sunflower following the opening and closing of the hand, working on mobility, flexion and extension.
Digital Musical Scale assigns a note to each finger opposed to the thumb. The musical sequence completes as the patient performs the movement correctly, training fine motor control and sequencing.
Power Brows uses the raising of the eyebrows to move virtual dumbbells and work on control of the frontalis muscle.
Smile Shot takes a photograph when the patient holds the smile for the time required, turning facial activation into an immediately recognisable action.
Blink Test uses a visual traffic light to measure the speed of eyelid closing and the response time.
The games last about a minute and can be combined differently within the same session. The therapist builds a session out of the exercises that are needed, without tying the whole course to a single experience.
The camera becomes the controller
Hands, face and eyes are detected directly from the video stream through MediaPipe.
No controllers, no wearable sensors and no complex calibration procedure: the patient takes position in front of the screen and interacts through their own body.
From an interaction design point of view, this keeps the therapeutic movement at the centre of the experience. The system does not simply recognise that the patient is playing: it verifies that they are performing the intended movement.
Feedback while the movement happens
The feedback is part of the exercise, not something that arrives at the end.
When the hand opens, the petals of the flower open. When the fingers reach the thumb, a note plays. When the smile is held, the photograph is taken.
The interaction makes what the patient is doing visible and audible, so they can correct the movement while performing it.
Outcome
From the session to the data
At the end of the session the system collects performance across the different exercises on its own and compares it with the previous ones.
The results are processed through Google AI Studio to generate a summary of the session and to propose a possible configuration for the next one, based on the performance recorded.
The system also produces an exportable PDF report, meant as documentation of the session and made to be shared with the professional.
