A brain implant has allowed two people with severe paralysis to speak and gesture simultaneously, a first for brain-computer interfaces that could restore more natural communication. The device, roughly the size of an iPhone, was placed over a large portion of the sensorimotor cortex — a brain region involved in both speech and movement — and decoded intended words and gestures in real time.
Researchers at the University of California, San Francisco, led by Samantha Brosler, tested the implant in two participants. The first, identified as Bravo-1r, became paralysed after a stroke. The second, Bravo-6, had amyotrophic lateral sclerosis, the most common form of motor neuron disease. Both had lost nearly all limb movement and could only produce unintelligible sounds.
In the study, the team recorded brain activity while participants repeatedly attempted to say ten phrases, such as «hello» and «nice to meet you», or perform ten gestures, including waving, clapping and shaking their head. Sometimes they tried speech and gesture separately; other times they attempted both at once. Customised machine learning models were then trained on each person’s brain recordings to predict their intended phrases and gestures.
To test the models, participants attempted pairs of phrases and gestures displayed on a screen that the models had not seen before. The predictions controlled an avatar resembling each participant, while the decoded speech appeared on screen. The team chose an avatar rather than trying to move the participants’ own bodies because it offered a straightforward way to prove that both actions could be decoded simultaneously from brain activity, Brosler said.
Accuracy varied between participants. Bravo-1r’s intended gestures and speech were predicted with 88 per cent and 84 per cent accuracy, respectively. Bravo-6’s were predicted with 66 per cent and 70 per cent accuracy. «If those models were operating on complete chance, the accuracy would be like 9 per cent,» Brosler said. «It was really exciting.»
Henri Lorach at the University of Lausanne in Switzerland called it «a very nice piece of work, with robust results». But he noted that it is based on a very limited set of phrases and gestures, and these still need improving. «For daily use without frustration, there needs to be higher accuracy,» he said.
Brosler said the team is working to address this by further training the models and tweaking the algorithms. She also said future models could move a person’s own body — if their joints and muscles are in good enough condition — and produce a synthetic voice, rather than relying on an avatar.
The research, published in Nature Neuroscience, represents a step toward restoring nuanced communication for people with paralysis. Gestures play an important role in everyday interaction, Brosler noted: «Saying ‘maybe’ and nodding your head has a very different connotation than if you shake your head.»
