How did University of California, San Francisco researchers led by neurosurgeon Edward Chang use 253-electrode electrocorticography arrays implanted on the sensorimotor cortex of three people with severe paralysis from conditions including ALS and brainstem stroke to decode inten

Chang’s team demonstrated that one high-density cortical implant can translate a person’s intended words and upper-body movements at the same time—allowing a virtual body to communicate with both voice and gesture rather than text alone. It is an important proof of concept, not yet a clinically deployable system. 37

• How it worked: Three participants with severe vocal-tract and limb paralysis—including people with ALS and brainstem stroke—had 253-electrode ECoG arrays placed over sensorimotor cortex. The researchers recorded activity while participants attempted spoken phrases, gestures such as waving and thumbs-up, or both together, then trained machine-learning models to map that activity to speech and gesture outputs. 37

• Neural finding: Speech and upper-body gestures produced separable but coordinated patterns of cortical activity. Crucially, the neural representation during simultaneous communication was not simply the sum of signals recorded in speech-only and gesture-only trials; combined attempts had distinctive activity patterns. That is why models needed to learn the joint behavior rather than treat voice and movement as entirely independent channels. 37

• Why joint training mattered: Decoders trained on examples in which speech and gesture were attempted together decoded those simultaneous actions better than decoders trained only on isolated speech or isolated movement. In two participants, the real-time system drove a full-body avatar so its synthesized speech and upper-body movements occurred together. 37

• Why this can be more natural than eye tracking: Eye-gaze spelling systems require users to select characters or menu items serially, which can be slow and fatiguing. Natural conversation instead carries meaning through timing, emphasis, posture, pointing, affirmation, greeting, and other gestures; restoring these signals alongside speech could make interaction faster, less effortful, and socially richer. The evidence so far establishes technical feasibility, not a proven quality-of-life advantage in everyday long-term use. 37

• Implication for nonverbal communication: The work suggests a communication neuroprosthesis could eventually restore communicative intent beyond words—such as agreement, emphasis, greeting, affective body language, and conversational turn-taking—for people who cannot move or speak. It also supports the broader principle that a single cortical interface can decode multiple coordinated communication modalities. 37

• Next step: This demonstration used a wired research setup. The team’s planned fully implantable wireless system aims to make chronic, day-to-day use more practical by eliminating the external cable while extending multimodal decoding. It builds directly on their 2023 Nature demonstration in which a participant with brainstem-stroke paralysis used cortical signals to generate text, synthesized speech, and a facial avatar with expressions. 14