The fundamental objective of visual processing is to guide motor output appropriate for the visual stimulus. The mirror neuron system within premotor cortex plays a dual role in high-level visual processing of bodies and controlling motor behavior. We are interested in the computational properties of these neurons when dynamic visual body stimuli need to be used as a cue to determine which motor action the agent has to perform. In other words, we are studying how mirror neuron populations translate social visual input into correct motor output.

A monkey observes a video of another monkey performing an action. After pausing briefly, the monkey is tasked with performing either the observed action, or a different one.

To this end, we analyze data from a highly sophisticated experimental setup in which a monkey is tasked with either repeating an observed action or performing a different learned action. We develop methods for dimensionality reduction to gain insights into the intricate relationship between action observation and execution that drives neural responses. The method reveals that the population predominantly stores the necessary information in terms of the self-action and not the visually observed action. Even during video observation, the population is driven more strongly by the self-action cued by the video than the video itself.

We developed a variant of PCA such that each principal component is attributed to one of the experimental factors.

Given the traditional belief that mirror neurons primarily fire for visual observation and execution of the same action, we also investigate whether the population shows more activity when replicating actions than when performing non-observed actions. Indeed, we identify neurons that are tuned to the type of instruction. However, the mirror neuron population is not biased towards ‘mirroring’.

Further, we are studying how well purely task-optimized machine learning models can explain neural responses to body stimuli. The advantage of task-optimized models is that they can be trained on large data sets, allowing for the representation of very general spatio-temporal visual features. We are specifically interested in the effect on goodness-of-fit of different mechanisms of temporal integration like recurrence, temporal convolutions or explicit attention, and the effect of the training objective like action recognition or predictive-coding. Moreover, we will attempt to make sense of the exact shortcomings of artificial-neural-network models for neural processing of bodies. This approach will extend the current literature on artificial-neural-network models for object representations to the time domain.