Humans recognize social interactions easily from visual stimuli showing interacting bodies, as well as from abstract stimuli such as moving geometric figures (Heider & Simmel, 1944). Whether social interaction perception from such different stimuli is accomplished by the same neural machinery remains unclear. We present a physiologically plausible neural model that achieves interaction recognition from abstract stimuli as well as from fully articulating bodies of animals in natural environments. METHODS: Exploiting a novel algorithm for the generation of interaction behavior that is based on dynamical navigation models for humans (Hovaidi-Ardestani, et al., 2018, Warren, 2006), we created matched pairs of abstract and naturalistic interaction stimuli (geometric shapes vs. articulating animals), balancing interaction-relevant visual features. Our neural recognition model combines a Deep Neural Network in shape-recognition pathway (VGG16) with top-level neural networks (Radial Basis Function network followed by recurrent Dynamic Neural Fields and Motion Energy detectors), which implement robust tracking of the interacting agents and the extraction of interaction-specific visual features (relative instantaneous position, speed, acceleration and orientation by means of a gain-field mechanism). RESULTS: The combination of these features by a simple neural classifier permits to reproduce psychophysical results on interaction recognition from abstract stimuli (Salatiello, et al. 2021) as well as recognition from interactions in naturalistic videos. CONCLUSION: The model predicts that recognition of social interactions can be achieved by simple physiologically plausible neural mechanisms. It makes concrete predictions about the behavior of neurons in relevant brain areas for the two stimulus classes, at the single cell as well as at the population level.