Mapping concept and relational semantic representation in the brain using large language models.

How the brain organizes semantic information is one of the most challenging and expansive questions in cognitive neuroscience. Prior studies have shed light on how the brain represents concepts, but where relations are represented, independently of concept information, remains unclear. To examine this issue, we submitted short texts to a contemporary large language model and extracted relational embeddings for scene-object pairs based on the model's residual stream activity. Using behavioral data (n = 636), we found that these embeddings capture independent information about scenes and objects, along with relational information on their semantic links. Turning to fMRI data (n = 60), we leveraged these embeddings for representational similarity analysis: the occipital and inferior temporal lobes primarily represent concepts in isolation. Medial and lateral parietal structures not only represent concepts but also strongly encode relational information. Interestingly, we also identified areas that minimally represent concepts but robustly encode relational information: the dorsolateral prefrontal cortex and basal ganglia. Beyond representing relational semantic information, the strength of relational representation in parietal, prefrontal, and striatal areas also tracks how participants reason about scenes and objects. Altogether, this research maps where concept and relation information is represented and begins to shed light on the links between this information coding and behavior.

Duke Scholars

Author Roberto Cabeza Psychology & Neuroscience

Author Simon Wilton Davis Neurology, Translational Brain Sciences