Watching a Film Reveals How the Brain Balances Eyes and Ears

For most of us, watching a movie feels effortless. We follow dialogue, read facial expressions, notice music cues and shifting scenery, and somehow fuse it all into a coherent story. But beneath that smooth experience, the brain is constantly deciding which sensory stream matters most in each moment.

A new study suggests that the frontal cortex, a region associated with planning and higher cognition, may act as a kind of traffic controller for this process — dynamically shifting attention between what we hear and what we see as a story unfolds.

To investigate, neuroscientists recorded brain activity directly from 19 epilepsy patients who had temporarily implanted electrodes for clinical monitoring. While in the hospital, participants watched a 12-minute multilingual short film* containing scenes in English, Greek, German, and French. Some foreign-language scenes included English subtitles, creating a natural test of how the brain handles changing audiovisual demands. Because the electrodes sat on or inside the brain, the researchers could track neural responses with millisecond precision, far faster than MRI scans allow.

They found that the frontal cortex was not processing all sensory information equally. Instead, it showed a striking internal division. Ventral, or lower, frontal regions responded more strongly to auditory information, while dorsal, or upper, frontal regions were more tuned to visual input.

“This suggests the frontal cortex has an organized map for handling different kinds of information during real-world experiences,” said first author Faxin Zhou, a Ph.D. candidate in the NYU Tandon Biomedical Engineering Department. “It is not just a general control center, it appears to separate sound and sight in a structured way.”

The pattern became even more interesting when the language changed. During English-language scenes, when listeners could understand speech directly, frontal brain areas leaned more heavily on auditory processing. But during scenes in unfamiliar languages, activity shifted toward visual regions, suggesting viewers relied more on facial expressions, gestures, and subtitles to follow the plot.

To confirm that interpretation, the team recruited online volunteers to rate short clips from the film. Participants judged which moments were most important to understanding the story and whether audio or visual cues were more useful in each scene. Those ratings closely matched the neural data: spoken English favored sound, while foreign-language scenes favored visual cues. In other words, the brain appears to reweight its sensory priorities on the fly.

“When comprehension through speech becomes harder, the brain flexibly reallocates resources toward visual signals,” said senior author Adeen Flinker, Associate Professor of Biomedical Engineering at NYU Tandon and Neurology at NYU Grossman School of Medicine. “That adaptability may be essential for navigating everyday environments filled with competing information.”

The findings help illuminate a long-standing question in neuroscience: how the brain merges multiple senses in realistic settings. Much prior research has relied on simplified laboratory tasks. Movies, by contrast, more closely resemble real life, where sensory cues arrive continuously and unpredictably.

The study also hints that the frontal cortex may do more than merge information after the fact. It may actively decide which stream — sound or sight — deserves priority before conscious understanding emerges.

That insight could have practical implications. Better understanding how the brain reallocates sensory attention may help researchers design therapies for people with language disorders, autism, attention deficits, or hearing loss. It could also inspire more adaptive artificial intelligence systems that shift between audio and visual inputs depending on context.

The work has limitations. Because the participants were hospital patients with epilepsy, they may not perfectly represent the general population. Electrode placement was determined by medical need, not experimental design, leaving some brain areas less sampled than others. Still, the precision of direct neural recording offers a rare glimpse into how the living human brain manages everyday perception.

*"Foreign Language," Adam Kelly Morton, Ack, No Ledge Creative