Researchers funded by the National Institutes of Health (NIH) have demonstrated a brain-computer interface that can translate brain activity into both speech and upper-body gestures at the same time.

The research team at the University of California, San Francisco used brain signals from people with severe paralysis to control a personalized full-body virtual avatar. The system was designed to reproduce more natural communication rather than relying only on text-to-speech.

Brain Signals Control a Digital Avatar

Researchers used thin sensor arrays placed on the motor cortex to record neural activity. Machine-learning systems then translated those signals into commands controlling the avatar.

In testing, participants attempted speech and gestures such as a hand wave or thumbs-up. The researchers found that training the system with simultaneous speech-and-gesture activity improved its ability to decode these combined expressions.

The findings are particularly relevant to people with conditions such as ALS or paralysis caused by brainstem strokes, which can severely restrict both verbal and non-verbal communication.

Researchers Plan Wireless Version

The current system uses implanted sensors connected to external processing equipment. Researchers say their next step is to test a fully implantable, wireless version, which could potentially make the technology more practical for longer-term use.

The research, published in Nature Neuroscience, highlights growing efforts to use brain-computer interfaces, artificial intelligence and neural decoding to restore communication abilities for people with severe neurological disabilities.