Researchers Use ‘Mind-Captioning’ to Generate Text from Brain Activity
Reading brain activity with advanced technologies is not a new concept. However, most techniques have focused on identifying single words associated with an object or action a person is seeing or thinking of, or matching up brain signals that correspond to spoken words. Some methods used caption databases or deep neural networks, but these approaches were limited by database word coverage or introduced information not present in the brain. Generating detailed, structured descriptions of complex visual perceptions or thoughts remains challenging.
A study,recently published in Science Advances,takes a new approach. Researchers involved in the study have developed what they refer to as a “mind-captioning” technique that uses an iterative optimization process, where a masked language model (MLM) generates text descriptions by aligning text features with brain-decoded features.
The technique also incorporates linear models trained to decode semantic features from a deep language model using brain activity from functional magnetic resonance imaging (fMRI). The result is a detailed text description of what a participant is seeing in their brain.
Generating video captions from human perception
For the first part of the experiment, six people watched 2,196 short videos while their brain activity was scanned with fMRI. The videos featured various random objects, scenes, actions, and events, and the six subjects were native Japanese speakers and non-native English speakers.
The same videos previously underwent a kind of crowdsourced text captioning by other viewers, which was processed by a pretrained LM, called DeBERTa-large that extracted particular features. These features were matched to brain activity and text was generated through an iterative process by the MLM model,called RoBERTa-large.
“Initially,the descriptions were fragmented and lacked clear meaning. However, through iterative optimization, these descriptions became more coherent and accurate,” explained the researchers.
Scientists decode brain activity into understandable sentences, a step toward ‘mind reading’
Researchers have successfully translated brain activity into coherent sentences describing what a person is thinking, marking a significant leap toward the long-held dream of “mind reading.” The study, published in Science Advances, demonstrates a system capable of reconstructing the semantic content of thoughts – essentially, turning brain signals into understandable text.
The team,led by Tomoyasu horikawa at the University of Texas at Austin,used functional magnetic resonance imaging (fMRI) to monitor brain activity while participants listened to podcasts and audiobooks. fMRI detects changes in blood flow related to neural activity, providing a detailed map of which brain regions are engaged during thought.
Crucially, the researchers didn’t focus on identifying what specific words a person was thinking, but rather the underlying meaning and concepts. They trained a large language model (LLM), similar to those powering chatbots, to associate patterns of brain activity with semantic categories – broad concepts like “animal,” “food,” or “action.”
This approach allowed the system to generate descriptive sentences summarizing the content a participant was experiencing, even when the exact words weren’t explicitly represented in the brain data. For example, when a participant was thinking about a dog, the system might generate a sentence like “a furry animal is running in the park.”
“We were surprised that we could reconstruct the semantic content of someone’s thoughts with this level of accuracy,” said Horikawa. “This is a big step forward in understanding how the brain represents meaning.”
While the current system isn’t capable of reading minds in the science fiction sense – it requires extensive training data from each individual and can only reconstruct relatively simple thoughts – it represents a crucial proof-of-concept.
The potential applications are far-reaching. The technology could one day assist individuals who have lost the ability to speak, allowing them to communicate their thoughts directly. It could also provide insights into consciousness, mental illness, and the very nature of thought itself.
Though, the researchers acknowledge the ethical implications of such technology. Safeguards will be needed to protect privacy and prevent misuse as the field advances.
More information:
Tomoyasu Horikawa, Mind captioning: Evolving descriptive text of mental content from human brain activity, Science Advances (2025).DOI: 10.1126/sciadv.adw1464
Keep reading