Researchers at the University of California, Davis (UC Davis) have made a groundbreaking advancement in brain-computer interface (BCI) technology, enabling a paralyzed man to communicate and ‘sing’ through a computer with minimal delay.
The investigational BCI system was tested on a participant suffering from amyotrophic lateral sclerosis (ALS). The technology captures raw neural signals via four microelectrode arrays surgically implanted into the brain region responsible for the physical production of speech. By combining low-latency processing with an AI-driven decoding model, the system synthesizes the participant’s speech in real-time. Crucially, the system does not read thoughts but translates the brain signals produced when he attempts to use his muscles to speak.
The synthesized speech closely resembles the participant’s natural voice, thanks to a voice cloning algorithm trained on audio samples recorded before the onset of ALS. The entire process, from signal acquisition to speech generation, occurs within an impressive 10 milliseconds, enabling near-instantaneous communication. This allows the participant to engage in conversations more naturally, as the delay is minimal.
The BCI also demonstrated the ability to recognize when the participant was attempting to sing, identifying one of three intended pitches and modulating his voice to synthesize vocal melodies. The technology can reproduce attempts to interject with sounds like ‘aah,’ ‘ooh,’ and ‘hmm,’ determine whether a sentence is a question or a statement, and identify when certain words are stressed. The research team reported that the system could successfully reproduce made-up words not included in the AI decoder’s training data, contributing to a far more expressive synthesized speech output compared to previous BCI systems.
Sergey Stavisky, the senior author of the research paper, highlighted the significance, stating that with instantaneous voice synthesis, users can be more included in conversations. Christian Herff, a computational neuroscientist not involved in the study, described it as “the holy grail in speech BCIs.” The UC Davis researchers plan to expand their study, replicating these results with more subjects experiencing speech loss due to various conditions, offering a pathway to enhanced communication and quality of life for individuals with paralysis.
