Neural and Computational Principles of Action and Social Processing

Abstract:

Recent work has shown that the attention maps of the widely popular DINOv2 model exhibit artifacts, which hurt both model interpretability and performance on dense image tasks. These artifacts emerge due to the model repurposing patch tokens with redundant local information for the storage of global image information. To address this problem, additional register tokens have been incorporated in which the model can store such information instead. We carefully examine the influence of these register tokens on the relationship between global and local image features, showing that while register tokens yield cleaner attention maps, these maps do not accurately reflect the integration of local image information in large models. Instead, global information is dominated by information extracted from register tokens, leading to a disconnect between local and global features. Inspired by these findings, we show that the [CLS] token itself leads to a very similar phenomenon in models without explicit register tokens. Our work shows that care must be taken when interpreting attention maps of large ViTs. Further, by clearly attributing the faulty behavior to register and [CLS] tokens, we show a path towards more interpretable vision models.

Authors: Lappe, Alexander Giese, Martin

Research Areas: Neural and Computational Principles of Action and Social Processing

Type of Publication: Article

Abstract:

Social interactions are essential for survival in primates, relying on both facial expressions and body signals. The accurate characterization of these signals is critical for understanding the neurocomputational mechanisms underlying social communication. While previous work has focused on recognizing monkey behavior, a causal and direct manipulation of individual cues strongly benefits from believable, dynamic body avatars—analogous to those successfully developed for faces. Creating lifelike monkey avatars with realistic body motion, however, is challenging. Acquiring sufficiently accurate movement data for animation with marker-based motion capture is impractical, and markerless tracking methods require extensive manual labeling. To address this, we developed MacAction, a realistic macaque body avatar animated from multi-camera markerless tracking data. Our method reconstructs accurate trajectories for a large number of keypoints, as required for the 3D animation of realistic body models. The entire time course of individual actions is captured using only two labeled keyframes per second, with performance further validated on a large-scale human multi-view dataset. We assessed the animation quality of our dynamic avatar in a free-viewing experiment with eight macaque observers for single macaque actions, where fixation behavior was indistinguishable between our animations and matched real videos. Moreover, by systematically varying the realism of the avatar, we found an uncanny valley effect in macaque body perception, similar to that previously described in both humans and macaque faces. These findings support the commonalities of social vision across primate species, providing a foundation for controlled experiments aimed at clarifying the detailed neurocomputational mechanisms of social body perception in primates.

Authors: Martini, Lucas M. Bognár, Anna Vogels, Rufin Giese, Martin A.

Research Areas: Neural and Computational Principles of Action and Social Processing

Type of Publication: Article

Abstract:

Vision foundation models achieve strong performance on both global and locally dense downstream tasks. Pretrained on large images, the recent DINOv3 model family is able to produce very fine-grained dense feature maps, enabling state-of-the-art performance. However, computing these feature maps requires the input image to be available at very high resolution, as well as large amounts of compute due to the squared complexity of the transformer architecture. To address these issues, we propose BRIXEL, a simple knowl- edge distillation approach that has the student learn to re- produce its own feature maps at higher resolution. Despite its simplicity, BRIXEL outperforms the baseline DINOv3 models by large margins on downstream tasks when the res- olution is kept fixed. Moreover, it is able to produce feature maps that are very similar to those of the teacher at a frac- tion of the computational cost

Authors: Lappe, Alexander; Giese, Martin A.

Research Areas: Neural and Computational Principles of Action and Social Processing

Type of Publication: Article

Abstract:

Understanding how the human brain processes body movements is essential for clarifying the mechanisms underlying social cognition and interaction. This study investigates the encoding of biomechanically possible and impossible body movements in occipitotemporal cortex using ultra-high field 7Tesla fMRI. By predicting the response of single voxels to impossible/possible movements using a computational modelling approach, our findings demonstrate that a combination of postural, biomechanical, and categorical features significantly predicts neural responses in the ventral visual cortex, particularly within the extrastriate body area (EBA), underscoring the brain{\textquoteright}s sensitivity to biomechanical plausibility. Lastly, these findings highlight the functional heterogeneity of EBA, with specific regions (middle/superior occipital gyri) focusing on detailed biomechanical features and anterior regions (lateral occipital sulcus and inferior temporal gyrus) integrating more abstract, categorical information.Competing Interest StatementThe authors have declared no competing interest.

Authors: Raman, Rajani Bognár, Anna Nejad, Ghazaleh Ghamkhari Mukovskiy, Albert; Martini, Lucas M.; Giese, Martin A.; Vogels, Rufin

Research Areas: Neural and Computational Principles of Action and Social Processing

Type of Publication: Article