Human motor control is compliant, i.e. the musculoskeletal system is elastic, and this elasticity is exploited during control. Compliant control is a complex problem for present humanoid robots, and it is critical for a lot of actions with high relevant for everyday life, such as soft catching, soft reception when falling, or sliding and pushing large objects as well as joint actions performed in teams such as manipulation of large scale objects by multiple humans. The overarching objective of the CogIMon project is to advance key technologies that lead to a step-change in cognitive compliant interaction in human-robot teams, integrating physical human-robot interaction, visually guided manipulation and safety integrated design in a systematic way.

Our group focuses in this project on deciphering human interpersonal sensorimotor strategies, and how they accomplish predictive control in interactive tasks. This requires a better understanding of the cognitive aspects that are essential for the control of compliant motion, i.e. extraction and prediction of task-relevant information from the motion (kinematics) of the interaction partner and the generation of predictive models of partner’s motions using kinematic motion tracking data.

For this purpose, we study extensively catching throwing scenarios and their adaptation to rehabilitation training, especially of ataxia patients. Exploiting models for the online synthesis of full-body motion, and for integrating them with balance control of humanoid robots, we are implementing a catching throwing game for the humanoid robot CoMan.

As intermediate step, we have integrated the CoMan robot simulator in a UNITY based virtual environment. This allows us to test and optimize controllers, including a full simulation of the physics of the robot, in a virtual environment before we test the real system. This is critical since patients are not tolerating large amounts of failure of  the technical system in rehabilitation scenarios. On addition, this allows us to evaluate the benefits of  interactive robotics-based training partially already before the real robot system becomes available.

Video: https://www.youtube.com/watch?v=tTtS92e1q4c

System architecture:   In collaboration with Techn. Univ. Braunschweig the overall system was implemented in a unified software framework that integrates real-time control (OROCOS), the UNITY game engine, and the Gazebo physics simulator of the COMAN robot. A Virtual Reality (VR) environment was implemented using the HTC Vive system, which combined a Head Mounted Display (HMD) with two motion trackers with a sampling frequency of 90 Hz. The system update time via the RSB communicator was about 1 ms and the controller running with an update time of 1 ms. 

Figure: System architecture embedded using unifying OROCOS real-time software environment.

Training scenario: Cerebellar ataxia patients benefit substantially from coordinative motor training in everyday-related situations. Administering such training with sufficient intensity is a substantial problem for cost-efficient rehabilitation. One effective exercise for coordination training is throwing and catching of balls (juggling with two people). In our scenario, one person was replaced by an avatar. Participants had to catch ball thrown by the robot. Throwing style and variability was adjusted in a manner that was suited for the skill level of the individual patients.

Video: VR training for rehabilitation. Ball throwing-catching game.

Figure: The learning of the task across sessions plotted for 2 cerebellar ataxia patients: DS and OA.

Video: Ball throwing-catching game in VR. The general view and the patient view (inset).