Functional and neurological rehabilitation is at a turning point. On the one hand, the need for continuous monitoring and personalization is growing; on the other, clinical teams work with tight schedules and a heavy record-keeping load that doesn't always leave room to delve into what's important: how each patient is doing in each session and what adjustments can improve their progress.
In this scenario, the combination of artificial intelligence (AI), multimodality e Internet of Things (IoT) opens a clear path: more natural interactions, more adaptive sessions y objective data that complement clinical observation.
What is mHRI and why does it matter in rehabilitation?
mHRI means Multimodal Human-Robot InteractionInstead of relying on a single channel (screen only, buttons only, or voice only), an mHRI system integrates multiple signals to better understand what happens during the session and respond more appropriately.
In this project, multimodality is based especially on three pillars:
-
Natural language (voice/text): to interact more intuitively.
-
Visual signals (image/video): to capture contextual and behavioral information.
-
IoT and physiological data: to incorporate objective metrics of the patient's condition.
The goal is not to "technify" rehabilitation, but to make it more sensitive to each person's real context.
The project: an mHRI system that combines AI and IoT
The project focuses on designing and integrating a rehabilitation-oriented mHRI system, combining:
-
AI applied to language (PLN) for voice or text interaction.
-
computer vision to extract relevant signals from context and behavior.
-
Capture of physiological data using IoTincluding compatible wearables and sensors.
-
A recommendation system, capable of converting heterogeneous signals into useful support for personalization.
At a technical level, the key challenge is to make all of this work as a whole: to reliably capture, process, store and exploit multimodal information in real environments (center and home).
1) Conversational AI: when technology communicates “like a person”
Natural interaction makes all the difference in rehabilitation: it reduces friction, improves task understanding, and increases adherence.
Therefore, an essential part of the system is based on natural language processing (NLP) to allow the patient to:
-
Receive clear and guided instructions during the exercises.
-
Ask for help or repeat instructions without interrupting the flow.
-
Maintain simple communication that fosters motivation.
Furthermore, the system seeks to adapt the interaction to the clinical context, avoiding generic responses and guiding the conversation towards specific therapeutic goals.
2) Computer vision: signals that help to understand how the session is going
Computer vision adds a valuable layer: it allows capturing signals from the environment and behavior that, without replacing clinical judgment, can enrich the interpretation of the session.
In rehabilitation, this can help to:
-
Detect if the person is present and ready to begin.
-
Identify moments of pause, doubt, or disengagement.
-
Provide context about the interaction (e.g., attention, orientation, or response to stimuli).
The challenge is to do it in a robust way: with variable lighting, occlusions and realistic conditions typical of centers and homes.
3) IoT and physiological data: objectivity for better personalization and monitoring
The capture of physiological data through IoT provides something crucial: objective measurementVariables such as heart rate, activity or movement (depending on the available sensors) allow for a better understanding of the patient's response in real time and over time.
Main benefits of incorporating physiological data
-
A better reading of effort and tolerance: It helps to interpret fatigue, activation, and response during the session.
-
Dynamic Personalization: It allows you to adjust the pace, intensity, or type of guidance based on observable signals.
-
Longitudinal follow-up: It makes it easier to compare sessions, identify trends, and detect relevant changes.
-
Support for the professional: It reduces dependence on manual recording and improves the consistency of available information.
The key is to integrate this data with the interaction (language and vision) so that the system not only “measures”, but also understand the context.
From data to decisions: the value of the recommendation system
Multimodality generates rich information, but its true impact emerges when that information is translated into useful actions.
Therefore, the system incorporates a recommendation engine oriented to:
-
Suggest adjustments to the session (for example, changes in pace or level of guidance).
-
Prioritize content or exercises based on the patient's profile and response.
-
Generate tracking signals to aid in decision-making.
This approach seeks to make technology a real supportnot an “extra”: concrete recommendations, interpretable and aligned with therapeutic objectives.
Infrastructure: data storage, integration, and governance
Multimodal capture requires an architecture capable of:
-
Manage heterogeneous data (text/voice, image/video, sensors).
-
Almato have dinner information safely and with the appropriate permits.
-
Facilitate consultation and subsequent exploitation (monitoring, reports and recommendations).
-
Maintain performance and stability in real-world usage scenarios.
In rehabilitation, where data is sensitive, this point is as important as AI itself.
Conclusion: a more humane rehabilitation… thanks to multimodality
The “new era” of rehabilitation is not about adding technology for the sake of adding it. It's about creating systems that:
-
Interact better (natural language).
-
Understand more context (vision).
-
Incorporate objective measurements (Physiological IoT).
-
Help personalize without increasing the professional's workload (recommendation).
That is what this mHRI approach within PERTE Health proposes: a more adaptive rehabilitation, with better follow-up and with an interaction that naturally accompanies the therapeutic process.
