Description
Currently, facial and body movements are better controlled and categorized by neural systems than any existing technical framework.
Our lab studies the computational principles underlying the control and categorization of body movements in biological systems and transfers them to technical applications, e.g., in computer graphics, humanoid robotics, or biomedical engineering and rehabilitation, where modeling human movements is becoming increasingly important for technical success.
Researchers
Current Projects
Modeling of human robot interaction and use of humanoid robots for rehabilitation training
The control compliant robots in interactive tasks, or even in joint tasks with multiple interacting humans and robots is a challenging problem. It is adressed in the EC H2020 project COGIMON by a highly interdisciplinary approach, combining the expertise form groups in neuroscience and robotics engineering. One interaction scenario of this type is also the training of patients by playing an interactive ball game with a humanoid robot. We used biologically inspired algorithms for movement syntheses to control humanoid robots and for training of coordination skills in ataxia.
Read more
Deep Gaussian Process models for Real-Time Blendshape Prediction on GPU
Modeling virtual character blendshapes via approximate inference in real-time on GPU. This implementation is realized as plugin for Autodesk Maya and Unreal Engine 5.
Read more
Probabilistic models for the online-synthesis of emotional and interactive full-body motion
Generative probabilistic models of interactive and stylized human motion are applicable in a variety of fields. On the technical side, such models are useful in computer animation, or motion recognition and emotional feature analysis. This work was done partly as parts of the EC FP7 projects TANGO.
Read more
Online Controllable Models of Complex Body Movements for Biomedical Applications
Robots driven by signals from the nervous system represent a promising approach to improving the autonomy and the quality of life of people with disabilities, following, for instance, stroke or spinal cord injury. The aim of the KONSENS-NHE project is the development of a non-invasive, neurally-controlled exoskeleton that may be used in everyday life to compensate for the loss of hand function in people with disabilities.
Read moreFinished Projects
AMARSi (Adaptive Modular Architectures for Rich Motor Skills)
Motor skills of humans and animals are still utterly astonishing when compared to robots. AMARSi aims at a qualitative jump in robotic motor skills towards biological richness.
Read more
Component-based Trajectory Models for Human Character Animation
The efficient parameterization of complex human movements is a core problem of modern computer animation. For the synthesis of animations with a high degree of realism learning-based approaches have become increasingly popular.
Read more
Interaction between periodic and non-periodic kinematic motion primitives
In order to provide a highly controlled setup for the recording of arm movements that are coordinated with walking movements, we have developed a novel virtual reality setup combining motion capture using a VICON system and stereoscopic presentation using a setup with Dolby 3D stereo projectors.
Read more
Online motion synthesis by networks of learned dynamic primitives for humanoid robots
Sequential goal-directed full-body motion is a challenging task for humanoid robots. An example is the coordination of bipedal walking with fast upper body movements.
Read more
Synthesis of complex locomotion behavior for humanoid robots based on biological principles
Locomotion in complex situations is a difficult problem in motor control that is unresolved for humanoid robots. We investigate and model how the locomotion behavior of humans is organized for complex locomotion tasks and try to transfer relevant control principles, especially at the level of cognitive control, to humaoid robots. This work is realized within the EC FP7 project KOIROBOT.
Read more
Learning Hierarchical Models for Motor Control
There is strong evidence that the animal Motor Control System is hierarchically organized into highly-interacting specialized subnetworks. In our lab, we combine methods from System Identification Theory and Machine Learning to automatically identify such modules.
Read morePublications
Barliya, A., Krausz, N., Naaman, H., Chiovetto, E., Giese, M. A. & Flash, T. (2024). Human arm redundancy: a new approach for the inverse kinematics problem. Royal Society Open Science, 11.
More
Human arm redundancy: a new approach for the inverse kinematics problem
Abstract:
The inverse kinematics (IK) problem addresses how both humans and robotic systems coordinate movement to resolve redundancy, as in the case of arm reaching where more degrees of freedom are available at the joint versus hand level. This work focuses on which coordinate frames best represent human movements, enabling the motor system to solve the IK problem in the presence of kinematic redundancies. We used a multi-dimensional sparse source separation method to derive sets of basis (or source) functions for both the task and joint spaces, with joint space represented by either absolute or anatomical joint angles. We assessed the similarities between joint and task sources in each of these joint representations, finding that the time-dependent profiles of the absolute reference frame’s sources show greater similarity to corresponding sources in the task space. This result was found to be statistically significant. Our analysis suggests that the nervous system represents multi-joint arm movements using a limited number of basis functions, allowing for simple transformations between task and joint spaces. Additionally, joint space seems to be represented in an absolute reference frame to simplify the IK transformations, given redundancies. Further studies will assess this finding’s generalizability and implications for neural control of movement.
Authors:
Barliya, Avi Krausz, Nili Naaman, Hila Chiovetto, Enrico Giese, Martin A.; Flash, Tamar
Research Areas:
Biomedical and Biologically Motivated Technical Applications
Type of Publication:
Article
Journal:
Royal Society Open Science
Volume:
11
Year:
2024
Month:
02
Lappe, A., Bognár, A., Nejad, G. G., Mukovskiy, A., Martini, L. M., Giese, M. A. et al. (2024). Parallel Backpropagation for Shared-Feature Visualization. Advances in Neural Information Processing Systems(37), 22993-23012.
More
Parallel Backpropagation for Shared-Feature Visualization
Authors:
Lappe, Alexander; Bognár, Anna Nejad, Ghazaleh Ghamkhari Mukovskiy, Albert; Martini, Lucas M.; Giese, Martin A.; Vogels, Rufin
Research Areas:
Biomedical and Biologically Motivated Technical Applications
Type of Publication:
Article
Full text: PDF | Online version
Martini, L. M., Bognár, A., Vogels, R. & Giese, M. A. (2024). MacAction: Realistic 3D macaque body animation based on multi-camera markerless motion capture. bioRxiv.
More
MacAction: Realistic 3D macaque body animation based on multi-camera markerless motion capture
Abstract:
Social interaction is crucial for survival in primates. For the study of social vision in monkeys, highly controllable macaque face avatars have recently been developed, while body avatars with realistic motion do not yet exist. Addressing this gap, we developed a pipeline for three-dimensional motion tracking based on synchronized multi-view video recordings, achieving sufficient accuracy for life-like full-body animation. By exploiting data-driven pose estimation models, we track the complete time course of individual actions using a minimal set of hand-labeled keyframes. Our approach tracks single actions more accurately than existing pose estimation pipelines for behavioral tracking of non-human primates, requiring less data and fewer cameras. This efficiency is also confirmed for a state-of-the-art human benchmark dataset. A behavioral experiment with real macaque monkeys demonstrates that animals perceive the generated animations as similar to genuine videos, and establishes an uncanny valley effect for bodies in monkeys.Competing Interest StatementThe authors have declared no competing interest.
Authors:
Martini, Lucas M.; Bognár, Anna Vogels, Rufin Giese, Martin A.
Research Areas:
Biomedical and Biologically Motivated Technical Applications
Type of Publication:
Article
Full text: Online version
St-Amand, J. & Giese, M. A. (2023). Variable Selection in GPDMs Using the Information Bottleneck Method. 37th Conference on Neural Information Processing Systems (NeurIPS 2023)..
More
Variable Selection in GPDMs Using the Information Bottleneck Method
Abstract:
Accurate real-time models of human motion are important for applications in areas such as cognitive science and robotics. Neural networks are often the favored choice, yet their generalization properties are limited, particularly on small data sets. This paper utilizes the Gaussian process dynamical model (GPDM) as an alternative. Despite their successes in various motion tasks, GPDMs face challenges like high computational complexity and the need for many hyperparameters. This work addresses these issues by integrating the information bottleneck (IB) framework with GPDMs. The IB approach aims to optimally balance data fit and generalization through measures of mutual information. Our technique uses IB variable selection as a component of GPLVM back-constraints to reduce parameter count and to select features for latent space optimization, resulting in improved model accuracy.
Authors:
St-Amand, Jesse; Giese, Martin A.
Research Areas:
Biomedical and Biologically Motivated Technical Applications
Type of Publication:
Article
Full text: PDF | Online version
Bognár, A., Mukovskiy, A., Nejad, G. G., Taubert, N., Stettler, M., Martini, L. M. et al (2023). Simultaneous recordings from posterior and anterior body responsive regions in the macaque Superior Temporal Sulcus . VSS 2023, May 19-24 2023, St. Pete Beach, Florida.
More
Simultaneous recordings from posterior and anterior body responsive regions in the macaque Superior Temporal Sulcus
Authors:
Bognár, A. Mukovskiy, Albert; Nejad, G. Ghamkhari Taubert, Nick; Stettler, Michael; Martini, Lucas M.; Raman, R. Giese, Martin A.; Vogels, R.
Research Areas:
Biomedical and Biologically Motivated Technical Applications
Type of Publication:
In Collection
Publisher:
VSS 2023, May 19-24 2023, St. Pete Beach, Florida
Bognár, A., Mukovskiy, A., Nejad, G. G., Taubert, N., Stettler, M., Martini, L. M. et al (2023). Feature selectivity of body-patch neurons assessed with a large set of monkey avatars . 13th Annual Meeting on PrimateNeurobiology, Apr.26-28 2023, Göttingen Primate Center..
More
Feature selectivity of body-patch neurons assessed with a large set of monkey avatars
Authors:
Bognár, A. Mukovskiy, Albert; Nejad, G. Ghamkhari Taubert, Nick; Stettler, Michael; Martini, Lucas M.; Raman, R. Giese, Martin A.; Vogels, R.
Research Areas:
Biomedical and Biologically Motivated Technical Applications
Type of Publication:
In Collection
Bognár, A., Raman, R., Taubert, N., Li, B., Zafirova, Y., Giese, M. A. et al. (2023). The contribution of dynamics to macaque body and face patch responses. NeuroImage, 269.
More
The contribution of dynamics to macaque body and face patch responses
Abstract:
Previous functional imaging studies demonstrated body-selective patches in the primate visual temporal cortex, comparing activations to static bodies and static images of other categories. However, the use of static instead of dynamic displays of moving bodies may have underestimated the extent of the body patch network. Indeed, body dynamics provide information about action and emotion and may be processed in patches not activated by static images. Thus, to map with fMRI the full extent of the macaque body patch system in the visual temporal cortex, we employed dynamic displays of natural-acting monkey bodies, dynamic monkey faces, objects, and scrambled versions of these videos, all presented during fixation. We found nine body patches in the visual temporal cortex, starting posteriorly in the superior temporal sulcus (STS) and ending anteriorly in the temporal pole. Unlike for static images, body patches were present consistently in both the lower and upper banks of the STS. Overall, body patches showed a higher activation by dynamic displays than by matched static images, which, for identical stimulus displays, was less the case for the neighboring face patches. These data provide the groundwork for future single-unit recording studies to reveal the spatiotemporal features the neurons of these body patches encode. These fMRI findings suggest that dynamics have a stronger contribution to population responses in body than face patches.
Authors:
Bognár, A. Raman, R. Taubert, Nick; Li, B Zafirova, Y Giese, Martin A.; Gelder, B. De Vogels, R.
Research Areas:
Biomedical and Biologically Motivated Technical Applications
Type of Publication:
Article
St-Amand, J., Taubert, N., Gizzi, L. & Giese, M. A (2022). A Hierarchical Gaussian Process Control Algorithm for Bimanual Coordination with Hand Rehabilitation Devices .
More
A Hierarchical Gaussian Process Control Algorithm for Bimanual Coordination with Hand Rehabilitation Devices
Research Areas:
Biomedical and Biologically Motivated Technical Applications
Type of Publication:
In Collection
Siebert, R., Stettler, M., Taubert, N., Dicke, P., Giese, M. A. & Thier, P (2022). Encoding of dynamic facial expressions in the macaque superior temporal sulcus . Society for Neuroscience.
More
Encoding of dynamic facial expressions in the macaque superior temporal sulcus
Authors:
Siebert, Ramona Stettler, Michael; Taubert, Nick; Dicke, Peter Giese, Martin A.; Thier, Peter
Research Areas:
Biomedical and Biologically Motivated Technical Applications
Type of Publication:
In Collection
Mukovskiy, A., Hovaidi-Ardestani, M., Salatiello, A., Stettler, M., Vogels, R. & Giese, M. A (2022). Physiologically-inspired neural model for social interaction recognition from abstract and naturalistic videos . VSS Annual Meeting 2022.
More
Physiologically-inspired neural model for social interaction recognition from abstract and naturalistic videos
Authors:
Mukovskiy, Albert; Hovaidi-Ardestani, Mohammad Salatiello, Alessandro; Stettler, Michael; Vogels, Rufin Giese, Martin A.
Research Areas:
Biomedical and Biologically Motivated Technical Applications
Type of Publication:
In Collection
Chiovetto, E., Salatiello, A., D'Avella, A. & Giese, M. A. (2022). Toward a unifying framework for the modeling and identification of motor primitives. Frontiers in computational neuroscience, 16 926345.
More
Toward a unifying framework for the modeling and identification of motor primitives
Abstract:
A large body of evidence suggests that human and animal movements, despite their apparent complexity and flexibility, are remarkably structured. Quantitative analyses of various classes of motor behaviors consistently identify spatial and temporal features that are invariant across movements. Such invariant features have been observed at different levels of organization in the motor system, including the electromyographic, kinematic, and kinetic levels, and are thought to reflect fixed modules-named motor primitives-that the brain uses to simplify the construction of movement. However, motor primitives across space, time, and organization levels are often described with ad-hoc mathematical models that tend to be domain-specific. This, in turn, generates the need to use model-specific algorithms for the identification of both the motor primitives and additional model parameters. The lack of a comprehensive framework complicates the comparison and interpretation of the results obtained across different domains and studies. In this work, we take the first steps toward addressing these issues, by introducing a unifying framework for the modeling and identification of qualitatively different classes of motor primitives. Specifically, we show that a single model, the anechoic mixture model, subsumes many popular classes of motor primitive models. Moreover, we exploit the flexibility of the anechoic mixture model to develop a new class of identification algorithms based on the Fourier-based Anechoic Demixing Algorithm (FADA). We validate our framework by identifying eight qualitatively different classes of motor primitives from both simulated and experimental data. We show that, compared to established model-specific algorithms for the identification of motor primitives, our flexible framework reaches overall comparable and sometimes superior reconstruction performance. The identification framework is publicly released as a MATLAB toolbox (FADA-T, https://tinyurl.com/compsens) to facilitate the identification and comparison of different motor primitive models.
Authors:
Chiovetto, Enrico Salatiello, Alessandro; d'Avella, Andrea Giese, Martin A.
Research Areas:
Biomedical and Biologically Motivated Technical Applications
Type of Publication:
Article
Taubert, N. & Giese, M. A. (2021). Hierarchical Deep Gaussian Processes Latent Variable Model via Expectation Propagation. Artificial Neural Networks and Machine Learning – ICANN 2021 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 14-17, 2021, Proceedings, Part I. Springer, Berlin.
More
Hierarchical Deep Gaussian Processes Latent Variable Model via Expectation Propagation
Authors:
Taubert, Nick; Giese, Martin A.
Research Areas:
Biomedical and Biologically Motivated Technical Applications
Type of Publication:
Article
Mohammadi, P., Hoffman, E. M., Dehio, N., Malekzadeh, M. S., Giese, M. A., Tsagarakis, N. G. et al. (2019). Compliant humanoids moving toward rehabilitation applications: Transparent integration of real-time control, whole-body motion generation, and virtual reality. IEEE Robotics & Automation Magazine, 26(4), 83-93.
More
Compliant humanoids moving toward rehabilitation applications: Transparent integration of real-time control, whole-body motion generation, and virtual reality
Authors:
Mohammadi, Pouya Hoffman, Enrico Mingo Dehio, Niels Malekzadeh, Milad S Giese, Martin A.; Tsagarakis, Nikos G Steil, Jochen J
Research Areas:
Biomedical and Biologically Motivated Technical Applications
Type of Publication:
Article
Lee, J., Huber, M. E., Chiovetto, E., Giese, M. A., Sternad, D. & Hogan, N. (2019). Human-inspired balance model to account for foot-beam interaction mechanics. 2019 International Conference on Robotics and Automation (ICRA) Palais des congres de Montreal, Montreal, Canada, May 20-24, 2019, 1970-1974.
More
Human-inspired balance model to account for foot-beam interaction mechanics
Authors:
Lee, Jongwoo Huber, Meghan E. Chiovetto, Enrico Giese, Martin A.; Sternad, Dagmar Hogan, Neville
Research Areas:
Biomedical and Biologically Motivated Technical Applications
Type of Publication:
Article
Mohammadi, P., Malekzadeh, M. S., Kodl, J., Mukovskiy, A., Wigand, D., Giese, M. A. et al. (2018). Real-Time Control of Whole-Body Robot Motion and Trajectory Generation for Physiotherapeutic Juggling in VR. IROS2018, 270-277.
More
Real-Time Control of Whole-Body Robot Motion and Trajectory Generation for Physiotherapeutic Juggling in VR
Authors:
Mohammadi, P. Malekzadeh, M. S. Kodl, Jindrich Mukovskiy, Albert; Wigand, D. Giese, Martin A.; Steil, Jochen
Research Areas:
Biomedical and Biologically Motivated Technical Applications
Type of Publication:
Article
Full text: PDF | Online version
Ilg, W., Synofzik, M., Broetz, D., Burkard, S., Giese, M. A. & Schöls, L. (2009). Intensive coordinative training improves motor performance in degenerative disease. Neurology 2009, 73, 1823-1830.
More
Intensive coordinative training improves motor performance in degenerative disease
Abstract:
Objectives: The cerebellum is known to play a strong functional role in both motor control and motor learning. Hence, the benefit of physiotherapeutic training remains controversial for patients with cerebellar degeneration. In this study, we examined the effectiveness of a 4-week intensive coordinative training for 16 patients with progressive ataxia due to cerebellar degeneration (n 10) or degeneration of afferent pathways (n 6). Methods: Effects were assessed by clinical ataxia rating scales, individual goal attainment scores, and quantitative movement analysis. Four assessments were performed: 8 weeks before, immediately before, directly after, and 8 weeks after training. To control for variability in disease progression, we used an intraindividual control design, where performance changes with and without training were compared. Results: Significant improvements in motor performance and reduction of ataxia symptoms were observed in clinical scores after training and were sustained at follow-up assessment. Patients with predominant cerebellar ataxia revealed more distinct improvement than patients with afferent ataxia in several aspects of gait like velocity, lateral sway, and intralimb coordination. Consistently, in patients with cerebellar but without afferent ataxia, the regulation of balance in static and dynamic balance tasks improved significantly. Conclusion: In patients with cerebellar ataxia, coordinative training improves motor performance and reduces ataxia symptoms, enabling them to achieve personally meaningful goals in everyday life. Training effects were more distinct for patients whose afferent pathways were not affected. For both groups, continuous training seems crucial for stabilizing improvements and should become standard of care. Level of evidence: This study provides Class III evidence that coordinative training improves motor performance and reduces ataxia symptoms in patients with progressive cerebellar ataxia
Authors:
Ilg, Winfried; Synofzik, Matthis Broetz, D. Burkard, Susanne Giese, Martin A.; Schöls, L.
Research Areas:
Biomedical and Biologically Motivated Technical Applications
Type of Publication:
Article
Full text: PDF | Online version