Non-reviewed Conference Papers and Abstracts

Neural model for the visual tuning properties of action-selective neurons in premotor cortex The visual recognition of goal-directed movements is crucial for the learning of actions, and possibly for the understanding of the intentions and goals of others. The discovery of mirror neurons has stimulated a vast amount of research investigating possible links between action perception and action execution [1,2]. However, it remains largely unknown what the precise nature of this putative visuo-motor interaction is, and which relevant computational functions can be accomplished by purely visual processing. Here, we present a neurophysiologically inspired model for the visual recognition of grasping movements from videos. The model shows that the recognition of functional actions can be accounted for to a substantial degree by the analysis of spatio-temporal visual features using well-established simple neural circuits. The model integrates a hierarchical neural architecture that extracts form information in a view-dependent way accomplishing partial position and scale invariance [3,4,5]. It includes physiologically plausible recurrent neural circuits that result in temporal sequence selectivity [6,7,8]. As a novel computational step, the model proposes a simple neural mechanism that accounts for the selective matching between the spatial properties of goal objects and the specific posture, position and orientation of the effector (hand). Opposed to other models that assume a complete reconstruction of the 3D effector and object shape our model is consistent with the fact that almost 90 % of mirror neurons in premotor cortex show view-tuning. We demonstrate that the model is sufficiently powerful for recognizing goal-directed actions from real video sequences. In addition, it correctly predicts several key properties of the visual tuning of neurons in premotor cortex. We conclude that the recognition of functional actions can be accomplished by simple physiologically plausible mechanisms, without the explicit reconstruction of the 3D structures of objects and effector. Instead, prediction over time can be accomplished by the learning of spatio-temporal visual pattern sequences. This ‘bottom-up’ view of action recognition complements existing models for the mirror neuron system [9] and motivates a more detailed analysis of the complementary contributions of visual pattern analysis and motor representations on the visual recognition of imitable actions. References [1] Di Pellegrino, G. et al. (1992): Exp. Brain Res. 91, 176-180. [2] Rizzolatti, G. and Craighero, L. (2004): Annu. Rev. Neurosci. 27, 169-192. [3] Riesenhuber, M. and Poggio, T. (1999): Nat. Neurosci. 2, 1019-1025. [4] Giese, A.M. and Poggio, T. (2003): Nat. Rev. Neurosci. 4, 179-192. [5] Serre, T. et al. (2007): IEEE Pattern Anal. Mach. Int. 29, 411-426. [6] Zhang, K. (1996): J. Neurosci. 16, 2112-2126. [7] Hopfield, J. and Brody, D. (2000): Proc Natl Acad Sci USA 97, 13919-13924. [8] Xie, X. and Giese, M. (2002): Phys Rev E Stat Nonlin Soft Matter Phys 65, 051904. [9] Oztop, E. et al. (2006): Neural Netw. 19, 254-271.

Neural model of action-selective neurons in STS and area F5 The visual recognition of goal-directed movements is crucial for the understanding of intentions and goals of others as well as for imitation learning. So far, it is largely unknown how visual information about effectors and goal objects of actions is integrated in the brain. Specifically, it is unclear whether a robust recognition of goal-directed actions can be accomplished by purely visual processing or if it requires a reconstruction of the three-dimensional structure of object and effector geometry. We present a neurophysiologically inspired model for the recognition of goal-directed grasping movements from real video sequences. The model integrates several physiologically plausible mechanisms in order to realize the integration of information about goal objects and the effector and its movement: (1) A hierarchical neural architecture for the recognition of hand and object shapes, which realizes position and scale-invariant recognition by subsequent increase of feature complexity and invariance along the hierarchy based on learned example views [1,2,3]. However, in contrast to standard models for visual object recognition this invariance is incomplete, so that the retinal positions of goal object and effector can be extracted by a population code. (2) Simple recurrent neural circuits for the realization of temporal sequence selectivity [4,5,6]. (3) A novel mechanism combines information about object shape and affordance and about effector (hand) posture and position in an object-centered frame of reference. This mechanism exploits gain fields in order to implement the relevant coordinate transformation [7,8]. The model shows that a robust integration of effector and object information can be accomplished by well-established physiologically plausible principles. Specifically, the proposed model does not contain explicit 3D representations of objects and the effector movement. Instead, it realizes predictions over time based on learned view-dependent representation of the visual input. Our results complement those of existing models of action recognition [8] and motivate a more detailed analysis of the complementary contributions of visual pattern analysis and motor representations on the visual recognition of imitable actions. References [1] Riesenhuber, M. and Poggio, T. (1999): Nat. Neurosci. 2, 1019-1025. [2] Giese, A.M. and Poggio, T. (2003): Nat. Rev. Neurosci. 4, 179-192. [3] Serre, T. et al. (2007): IEEE Pattern Anal. Mach. Int. 29, 411-426. [4] Zhang, K. (1996): J. Neurosci. 16, 2112-2126. [5] Hopfield, J. and Brody, D. (2000): Proc Natl Acad Sci USA 97, 13919-13924. [6] Xie, X. and Giese, M. (2002): Phys Rev E Stat Nonlin Soft Matter Phys 65, 051904. [7] Salinas, E. and Abbott, L. (1995): J. Neurosci. 75, 6461-6474. [8] Pouget, A. and Sejnowski, T. (1997): J. Cogn. Neurosci. 9, 222-237. [9] Oztop, E. et al. (2006): Neural Netw. 19, 254-271.

A neural model of the visual tuning properties of action-selective neurons in STS and area F5 The visual recognition of goal-directed movements is crucial for the understanding of intentions and goals of others as well as for imitation learning. So far, it is largely unknown how visual information about effectors and goal objects of actions is integrated in the brain. Specifically, it is unclear whether a robust recognition of goal-directed actions can be accomplished by purely visual processing or if it requires a reconstruction of the three-dimensional structure of object and effector geometry. We present a neurophysiologically inspired model for the recognition of goal-directed grasping movements. The model reproduces fundamental properties of action-selective neurons in STS and area F5. The model is based on a hierarchical architecture with neural detectors that reproduce the properties of cells in visual cortex. It contains a novel physiologically plausible mechanism that combines information on object shape and effector (hand) shape and movement, implementing the necessary coordinate transformations from retinal to an object centered frame of reference. The model was evaluated with real video sequences of human grasping movements, using a separate training and test set. The model reproduces a variety of tuning properties that have been observed in electrophysiological experiments for action-selective neurons in STS and area F5. The model shows that the integration of effector and object information can be accomplished by well-established physiologically plausible principles. Specifically, the proposed model does not compute explicit 3D representations of objects and the action. Instead, it realizes predictions over time based on learned view-dependent representations for sequences of hand shapes. Our results complement those of existing models for the recognition of goal-directed actions and motivate a more detailed analysis of the complementary contributions of visual pattern analysis and motor representations on the visual recognition of imitable actions.

Visual encoding of goal-directed movements: a physiologically plausible neural model Visual responses of action-selective neurons, e.g. in premotor cortex and the superior temporal sulcus of the macaque monkey, are characterized by a remarkable combination of selectivity and invariance. On the one hand, the responses of such neurons show high selectivity for details about the grip and the spatial relationship between effector and object. At the same time, these responses show substantial invariance against the retinal stimulus position. While numerous models for the mirror neuron system have been proposed in robotics and neuroscience, almost none of them accounts for the visual tuning properties of action-selective neurons exploiting physiologically plausible neural mechanisms. In addition, many existing models assume that action encoding is based on a full reconstruction of the 3D geometry of effector and object. This contradicts recent electrophysiological results showing view-dependence of the majority of action-selective neurons, e.g. in premotor cortex. We present a neurophysiologically plausible model for the visual recognition of grasping movements from real videos. The model is based on simple well-established neural circuits. Recognition of effector and goal object is accomplished by a hierarchical neural architecture, where scale and position invariance are accomplished by nonlinear pooling along the hierarchy, consistent with many established models from object recognition. Effector recognition includes a simple predictive neural circuit that results in temporal sequence selectivity. Effector and goal position are encoded within the neural hierarchy in terms of population codes, which can be processed by a simple gain field-like mechanism in order to compute the relative position of effector and object in a retinal frame of reference. Based on this signal, and object and effector shape, the highest hierarchy level accomplishes a distinction between functional (hand matches object shape and position) and dysfunctional (no match between hand and object shape or position) grips, at the same time being invariant against strong changes of the stimulus position. The model was tested with several stimuli from the neurophysiological literature and reproduces, partially even quantitatively, results about action-selective neurons in the STS and premotor cortex. Specifically, the model reproduces visual tuning properties and the view-dependence of mirror neurons in premotor cortex and makes additional predictions, which can be easily tested in electrophysiological experiments.

Invariant recognition of goal-directed hand actions: a physiologically plausible neural model The recognition of transitive, goal-directed actions requires highly selective processing of shape details of effector and goal object, and high robustness with respect to image transformations at the same time. The neural mechanisms required for solving this challenging recognition task remain largely unknown. We propose a neurophysiologically-inspired model for the recognition of transitive grasping actions, which combines high selectivity for different grips with strong position invariance. The model is based on well-established physiologically plausible simple neural mechanisms. Invariance is accomplished by combining nonlinear pooling (by maximum operations) and a specific neural representation of the relative position of object and effector based on a gain-field like mechanism. The proposed architecture accomplishes accurate recognition of different grip types on real video data and reproduces correctly several properties of action-selective neurons in occipital, parietal and premotor areas. In addition, the model shows that the accurate recognition of goal-directed actions can be accomplished without an explicit reconstruction of the 3-D structure of effectors and objects, as assumed in many technical systems for the recognitions of hand actions.