Non-reviewed Conference Papers and Abstracts

Abstract:

Previous work has shown that neurons from body patches in macaque superior temporal sulcus (STS) respond selectively to images of bodies. However, the visual features leading to this body selectivity remain unclear. METHODS: We conducted experiments using 720 stimuli presenting a monkey avatar in various poses and viewpoints. Spiking activity was recorded from mid-STS (MSB) and anterior-STS (ASB) body patches, previously identified using fMRI. To identify visual features driving the neural responses, we used a model with a deep network as frontend and a linear readout model that was fitted to predict the neuron activities. Computing the gradients of the outputs backwards along the neural network, we identified the image regions that were most influential for the model neuron output. Since previous work suggests that neurons from this area also respond to some extent to images of objects, we used a similar approach to visualize object parts eliciting responses from the model neurons. Based on an object dataset, we identified the shapes that activate each model unit maximally. Computing and combining the pixel-wise gradients of model activations from object and body processing, we were able to identify common visual features driving neural activity in the model. RESULTS: Linear models fit the data well, with mean noise-corrected correlations with neural data of 0.8 in ASB and 0.94 in MSB. Gradient analysis on the body stimuli did not reveal clear preferences of certain body parts and were difficult to interpret visually. However, the joint gradients between objects and bodies traced visually similar features in both images. CONCLUSION: Deep neural networks model STS data well, even though for all tested models, explained variance was substantially lower in the more anterior region. Further work will test if the features that the deep network relies on are also used by body patch neurons.

Authors: Lappe, Alexander; Bognár, Anna Nejad, Ghazaleh Ghamkhari Raman, Rajani Mukovskiy, Albert; Martini, Lucas M.; Vogels, Rufin Giese, Martin A.

Research Areas: Neural and Computational Principles of Action and Social Processing

Type of Publication: In Collection

Abstract:

Previous work has shown that neurons from body patches in macaque superior temporal sulcus (STS) respond selectively to images of bodies. However, the visual features leading to this body selectivity remain unclear. METHODS: We conducted experiments using 720 stimuli presenting a monkey avatar in various poses and viewpoints. Spiking activity was recorded from mid-STS (MSB) and anterior-STS (ASB) body patches, previously identified using fMRI. To identify visual features driving the neural responses, we used a model with a deep network as frontend and a linear readout model that was fitted to predict the neuron activities. Computing the gradients of the outputs backwards along the neural network, we identified the image regions that were most influential for the model neuron output. Since previous work suggests that neurons from this area also respond to some extent to images of objects, we used a similar approach to visualize object parts eliciting responses from the model neurons. Based on an object dataset, we identified the shapes that activate each model unit maximally. Computing and combining the pixel-wise gradients of model activations from object and body processing, we were able to identify common visual features driving neural activity in the model. RESULTS: Linear models fit the data well, with mean noise-corrected correlations with neural data of 0.8 in ASB and 0.94 in MSB. Gradient analysis on the body stimuli did not reveal clear preferences of certain body parts and were difficult to interpret visually. However, the joint gradients between objects and bodies traced visually similar features in both images. CONCLUSION: Deep neural networks model STS data well, even though for all tested models, explained variance was substantially lower in the more anterior region. Further work will test if the features that the deep network relies on are also used by body patch neurons.

Authors: Martini, Lucas M.; Bognár, Anna Vogels, Rufin Giese, Martin A.

Research Areas: Neural and Computational Principles of Action and Social Processing

Type of Publication: In Collection

Abstract:

Impairments in gait– with a key component of dynamic balance– and stance– with a key component of static balance- represent the key hallmarks of ataxia; not only in clinical assessments and clinician-reported outcomes; but also in patients’ voice burden of disease severity and patient-reported outcomes. While it is obvious that both features are not independent from each other, their interplay in ataxia – in terms of underlying control mechanisms- remains unknown. Here we aimed to assess the interaction be-tween dynamic balance (gait) and static balance (stance) in response to longitudinal changes in cerebellar ataxia using wearable sensors. We assessed cross-sectional and longitudinal balance of subjects with degenerative cerebellar disease (SARA:7.5±5.14) at baseline and 1-year follow-up (n=60) by 3 body-worn inertial sensors in two conditions: (1) stance with feet together (30 seconds), (2) straight walking (2 minutes). Based on the hip sensor, sway path length was calculated as a measure of static balance during stance using both directions of sway (PLtotal), as well as exclusively anterior-posterior (PLap) and medial-lateral (PLml) direction. Gait analysis focussed on ataxic-sensitive measures of spatio-temporal variability: stride length variability (SLCV) in gait direction and lateral step deviation (LSD) as well as upper body range of motion during gait in respective directions (ROMap, ROMml). Cross-sectional analyses revealed significant correlations between PLtotal and LSD as well as ROMml (r {\textgreater}0.6), and between PLtotal and SLCV and ROMap (r{\textgreater}0.4). Matching directions of sway showed a mildly increased effect (e.g. LSD{\textbackslash}PLap:r\_total=0.61,r\_ap= 0.63). Corresponding stance and gait measures showed similar correlations to patient-reported balance confidence (ABC-score;PLtotal:0.65,LSD:0.69). Longitudinal changes in static balance were correlated with changes in dynamic bal-ance specifically in the corresponding direction (e.g. deltaLSD{\textbackslash}deltaPLml:r=0.40). We were able to identify specific influences of the static balance mechanism on gait, demonstrating the patient's relevance of static stance testing and related balance exer-cises in rehabilitation.

Authors: Bohn, Kristina; Seemann, Jens; Giese, Martin A.; Synofzik, Matthis Ilg, Winfried

Research Areas: Clinical Movement Control and real-life Behavior Analysis for Assistive Systems

Type of Publication: In Collection

Abstract:

Objectives: In this observational study, we aim to unravel performance markers of ataxic gait for upcoming therapy trials using wearable sensors. We hypothesize that in short, trial-like time-frames gait measures captured in complex real-life settings of patients are more sensitive to natural disease progression compared to lab-based gait assessments and clinical rating scales. Methods: We assessed longitudinal gait changes of 24 subjects with spinocerebellar ataxia (SCA types: 1, 2, 3, 6) at baseline (SARA:9.4±4.1), 1-year and 2-years follow-up assessment by three body-worn inertial sensors in two conditions: (1) laboratory-based walking; (2) real-life walking in everyday environment. In the real-life walking condition, a context-sensitive analysis was performed by selecting comparable walking bouts according to bout length and number of performed turns. Movement analysis focussed on measures of spatio-temporal variability, in particular lateral step deviation (LSD) and a compound measure of spatial variability (SPcmp). Results: Cross-sectional analyses revealed high correlation to ataxia severity (SARA) and patients subjective balance confidence (ABC-Scale) in both conditions (r>0.7). While clinical ataxia score and gait measure in lab-based gait assessments identified changes after two years only (SARA: rprb=0.71; LSD: rprb=0.67), real life assessment of lateral step deviation and a compound measure of spatial step variability identified changes already after one year, with high effect sizes (LSD: rprb=0.66; SPcmp: rprb=0.68) and additionally increased effect sizes after two years (LSD: rprb=0.77; SPcmp: rprb=0.82). Discussion: Utilizing a context-sensitive matching procedure with high robustness to disease-independent changes of environment, real-life gait measures capture longitudinal change within one year with high effect size. In contrast, clinical scores like the SARA or lab-based gait measures show longitudinal change only after two years. Conclusions: Features of real-life gait constitute promising performance markers for upcoming therapy trials, yielding ecologically validity, earlier sensitivity and increased effect sizes in comparison with clinical scores and lab-based gait assessment.

Authors: Seemann, Jens; Giese, Martin A.; Synofzik, Matthis Ilg, Winfried

Research Areas: Uncategorized

Type of Publication: In Collection

Authors: Smekal, Vojtěch Solanas, Tamás Szűcs Marta Poyo Lappe, Alexander; Giese, Martin A.; de Gelder, Beatrice

Research Areas: Neural and Computational Principles of Action and Social Processing

Type of Publication: In Collection

Abstract:

The primate visual system has evolved subareas in which neurons appear to respond more strongly to images of a specific semantic category, like faces or bodies. The computational processes underlying these regions remain unclear, and there is debate on whether this effect is in fact driven by semantics or rather by visual features that occur more often among images from the specific category. Recent works tackling the question of whether the same visual features drive responses of face-selective cells to face images and non-face images have yielded mixed results. Here, we report findings on shared encoding of body and object images in body-selective neurons in macaque superior temporal sulcus. We targeted two fMRI-defined regions, anterior and posterior body patches in two awake macaques using V probes, recording multi-unit activity in and around these patches. In a first phase, we recorded responses to a set of 475 images of a macaque avatar in various poses. We then trained a deep-neural-network based model to predict responses to these images, and subsequently evaluated the model on two sets of object and body stimuli consisting of 6857 and 2068 images, respectively. These images comprised a variety of object types and animal species. After the inference process, we selected the highest and lowest predicted activator for each recording channel from both object and body images. In a second phase, we recorded responses of the same multi-units to these stimuli. For analysis, we only kept those multi-unit sites with high test/retest reliability. Also, we only considered multi-unit sites for which the selected bodies elicited a significantly higher response than the selected objects. We then tested whether the high-predicted objects/bodies indeed lead to higher responses at the corresponding electrode than the low-predicted ones. Across neurons, we found a significant preference of the high-predicted stimulus for both objects and bodies. The highly-activating objects consisted of a variety of everyday objects and did not necessarily globally resemble a body. Furthermore, the correlations between predicted and recorded responses to the objects were consistently positive for both monkeys and recording areas, meaning that the model was able to predict responses to objects after having only been trained on images of a macaque avatar. Our results show that the feature preferences of body-selective neurons are at least partially shared between bodies and objects. On a larger scope, we provide further evidence that category selectivity arises due to highly shared visual features among category instances, rather than semantics.

Authors: Lappe, Alexander; Bognár, A. Nejad, G. G. Mukovskiy, Albert; Giese, Martin A.; Vogels, Rufin

Research Areas: Uncategorized

Type of Publication: In Collection

Abstract:

Non-verbal social communication relies on the interpretation of visual cues from the body. fMRI studies in macaques have identified regions within the inferotemporal (IT) cortex that exhibit heightened activation to bodies compared to faces and objects. Among these regions, the ventral bank Superior Temporal Sulcus (STS) patches, i.e. the mid STS (MSB) and anterior STS body patch (ASB), show selectivity for static (and dynamic) bodies. However, the body features that drive the response of these neurons, in particular their representation of body posture, within these two levels of processing are unclear. To investigate this, we recorded multi- unit responses, using 16-channel V-probes, within and around MSB and ASB in two monkeys, employing a stimulus set comprising 720 stimuli featuring a monkey avatar in 45 body postures, rendered from 16 viewing angles. The static stimuli were presented during passive fixation. We employed principal component regression to model the response of the neurons based on the 10 principal components of 22 2D body keypoints extracted from the stimuli, which explained about 90% of the stimulus variance. Of the body-category selective neurons (at least twofold higher response to dynamic bodies compared to dynamic faces and objects), the 2D key-point-based model explained the selectivity for body posture and view with a median reliability-corrected coefficient of determination of 0.42 and 0.20 in the MSB and ASB regions, respectively. Inclusion of the depth dimension increased the model fit significantly for ASB but not MSB. When comparing with a convolutional neural network (CNN; ResNet50-robust; regression on 50 PCs) feature-based approach, the keypoint-based model exhibited slightly inferior performance, particularly in ASB, when focusing on higher-layer features but remained superior to the lower- layer features-based CNN model. Inverting the keypoint models allowed visualization of the body features that drove the posture selectivity of the neurons. We found that these body features ranged from local body features like the upper limbs or tail to combinations of them, but rarely the entire body. Some neurons, even in the mid STS region, tolerated changes in the view of the preferred body parts. The view tolerance was significantly greater in ASB compared to MSB. Our study shows that a body keypoint representation explains a sizable proportion of the selectivity to body posture and view of macaque visual cortical neurons, especially in the mid STS. Furthermore, the modeling suggests that 3D cues contribute to the body selectivity of anterior but not posterior IT neurons.

Authors: Vogels, Rufin Raman, R. Nejad, G. G. Mukovskiy, Albert; Lappe, Alexander; Giese, Martin A.; Martini, Lucas M.; Bognár, A.

Research Areas: Uncategorized

Type of Publication: In Collection

Abstract:

In this fMRI study we investigated the role played by biomechanical plausibility in the representation of bodies in EBA. The extrastriate body area (EBA) (Downing et al. 2001, Peelen and Downing, 2005) is currently considered to be a ventral cortex object category area, selective for body stimuli but little yet is understood about its computational functions. In a previous study we showed that the EBA is sensitive to joints position in still body stimuli (Marrazzo et al. 2023). Here, we used video images to investigate whether disrupting joints configuration affects the representation of bodies in EBA. Stimuli depicted artificial whole-body movements and were generated from the MoVI dataset. We selected 60 trials of naturalistic body movement and created 60 (possible) videos. Additionally, these stimuli underwent further processing where elbows and knees position/angle were manually modified, to create (from possible stimuli) biomechanically impossible stimuli. Therefore, the stimuli set included 120 videos (60 possible, 60 impossible). 12 participants were scanned using a 7T (T2*-weighted Multi-Band accelerated EPI 2D BOLD sequence, MB = 2, voxel size = 0.8 mm3, TR = 2300 ms, TE = 27 ms) in a fast event-related design over 12 separate runs. Each run consisted of 20 unique stimuli (10 possible, 10 impossible repeated 6 times across the 12 runs) which appeared on the screen for 2-3 s. Participants were asked to fixate and attention was controlled using catch trials (fixation shape change). The fMRI response was modeled using several features extracted from the stimuli: 3D coordinates and rotation matrices of key joints (kp/rot) and a model which represents within\between distance between joints for each video as a mean to encode biomechanical information (simdist). The fMRI predicted responses from each model were generated via banded ridge regression (Nunez-Elizalde et al. 2019, Dupré La Tour et al. 2022) using crossvalidation. Results show a pattern of responses across visual cortex with simdist and kp model best predicting responses to our stimuli. Specifically, the simdist representation shows higher prediction accuracy in in high-level temporal areas such as EBA outperforming the kp model. These findings expand on previous research showing that EBA codes for specific features of the body, which in the case of kp model, are the joints position (Marrazzo et al. 2023). Additionally, EBA shows high degree of sensitivity for joints configuration to the point that biomechanically possible/impossible bodies appear to be differentially encoded. Acknowledgments: This work was supported by ERC 2019-SyG-RELEVANCE-856495

Authors: Marrazzo, G. Martino, F. De Mukovskiy, Albert; Giese, Martin A.; de Gelder, Beatrice

Research Areas: Uncategorized

Type of Publication: In Collection

Abstract:

Background and Objective: With treatment trials on the horizon, sensitive outcome measures are highly needed for the >100 spastic ataxias. Digital-motor gait measures, assessed by wearable sensors, are considered prime outcome candidates for spastic ataxias and have shown favourable cross-sectional properties in spastic paraplegia type 7 (SPG7). However, their longitudinal sensitivity to change is yet unknown. This study aimed to assess 1-year progression of digital gait measures in patients with SPG7. Methods: Longitudinal multi-center study (7 centers, 6 countries), assessments at baseline and after 1 year. Gait was analysed in 49 SPG7 patients (baseline, median [min-max]: age=52 [22-69], SARA=9.0 [3.5-17.0], SPRS=14 [3-28]) using 3 wearable motion sensors (Opal APDM) during laboratory-based walking and ‘supervised free walking’, resembling real-life walking. Assessments included rating of the Scale for the assessment and rating of ataxia (SARA) and the Spastic paraplegia rating scale (SPRS). Effect size and significance of 1-year changes were assessed using non-parametric matched-pairs rank biserial correlation (rprb) and Wilcoxon signed-rank test, respectively. Results: In laboratory-based walking, 1-year progression was observed for measures of trunk range of motion variability (CoronalRoM_CV: rprb=0.46, p=0.0051), of gait smoothness (harmonicRatioML: rprb=-0.40, p=0.015) and of spatiotemporal stride variability (e.g. DoubleSupport_MADN: rprb=0.31-0.37). In the trial-relevant subcohort of mildly affected patients (SPRS items 1-6≤9; n=34), CoronalRoM_CV (rprb=0.59, p=0.0027) exhibited larger effect size than clinician-reported outcomes like SARA (rprb=0.53, p=0.0055) or SPRS (rprb=0.30, p=0.087). In supervised free walking, progression was observed for measures of gait smoothness and temporal variability (e.g. harmonicRatioML, DoubleSupport_MADN: |rprb|=0.28-0.44). Discussion and Conclusion: In this first longitudinal multi-center study of digital gait measures in SPG7, 1-year progression was captured for several gait measures, with effect sizes partly exceeding those of key clinician-reported outcomes (SARA, SPRS). These gait measures could thus improve sensitivity to treatment effects in future clinical trials in SPG7 and possibly also other spastic ataxias.

Authors: Beichert, Lukas Seemann, Jens; Kessler, Christoph Traschütz, Andreas Ricca, Ivana Satolli, Sara Başak, Ayşe Nazli Coarelli, Giulia Timmann, Dagmar Gagnon, Cynthia van de Warrenburg, Bart P. consortium, PROSPAX Ilg, Winfried; Synofzik, Matthis Schüle, Rebecca

Research Areas: Uncategorized

Type of Publication: In Collection

Abstract:

Introduction: Monitoring gait with wearable sensors provides an opportunity for improving clinical management and evaluating therapeutic interventions in patients with degenerative cerebellar ataxia (DCA). While multi-sensor configurations are recommended for robust gait evaluation, using a single sensor offers several advantages including reduced data footprint, minimized patient burden, and extended battery life. Methods: 96 participants (control: n=42; preclinical DCA: n=19; clinical DCA: n=35) completed two 25m straight walks at their self-selected preferred pace in a laboratory setting. A wearable sensor (APDM, Opal 128Hz) was affixed to the lower back and the dorsum of both feet. Gait events (initial contact-IC, final contact-FC) were detected using a single sensor and multiple sensors (reference system). Agreement between the single and multi-sensor configurations (bias, limits of agreement, intraclass correlation coefficient) and accuracy (Positive predictive value; PPV, median absolute error; MAE) were quantified. Relationships between event detection accuracy and gait outcomes derived by the reference system were explored. Results: A total of 8473 steps were included in the analyses. Accuracy was high for identification of IC in controls (PPV=97%), preclinical DCA (PPV=96%) and clinical DCA (PPV=86%). Accuracy was lower for FC compared to IC for controls (PPV=88%), preclinical DCA (PPV=90%) and clinical DCA (PPV=82%). The MAE was low for all groups (<0.12s). Significant correlations were observed indicating that gait events were detected less accurately for individuals walking with a reduced cadence, longer stride duration, and increased gait variability (gait speed, stride length and duration). Discussion and Conclusion: Accuracy for the single sensor approach was high and exceeded the threshold of 80% indicating that this approach may be used with confidence. Noticeable differences were observed in FC identification for clinical DCA, which may impact the calculation of gait outcomes. Additional refinements to optimize the algorithm should be considered to improve gait event detection accuracy.

Authors: Sarvestan, Javad Seemann, Jens; Din, Silvia Del Synofzik, Matthis Ilg, Winfried; Alcock, Lisa

Research Areas: Uncategorized

Type of Publication: In Collection

Authors: Lappe, Alexander; Bognár, Anna Nejad, Ghazal Ghamkhari Mukovskiy, Albert; Giese, Martin A.; Vogels, Rufin

Research Areas: Uncategorized

Type of Publication: In Collection

Authors: Kumar, Prerana; Raman, Rajani Bognár, Anna Taubert, Nick; Nejad, Ghazal Ghamkhari Vogels, Rufin Giese, Martin A.

Research Areas: Uncategorized

Type of Publication: In Collection

Abstract:

The neural and computational mechanisms of the visual encoding of body pose and motion remain poorly understood. One important obstacle in their investigation is the generation of highly controlled stimuli with exactly specified form and motion parameters. Avatars are ideal for this purpose, but for nonhuman species the generation of appropriate motion and shape data is extremely costly, where video-based methods often are not accurate enough to generate convincing 3D animations with highly specified parameters. METHODS: Based on a photorealistic 3D model for macaque monkeys, which we have developed recently, we propose a method that adjusts this model automatically to other nonhuman primate shapes, requiring only a small number of photographs and hand-labeled keypoints for that species. The resulting 3D model allows to generate highly realistic animations with different primate species, combining the same motion with different body shapes. Our method is based on an algorithm that deforms a polygon mesh of a macaque model with 10,632 vertices with an underlying rig of 115 joints automatically, matching the silhouettes of the animals and a small number of specified key points in the example pictures. Optimization is based on a composite error function that integrates terms for matching quality of the silhouettes, keypoints, and bone length, and for minimizing local surface deformation. RRSULTS: We demonstrate the efficiency of the method for several monkey and ape species. In addition, we are presently investigating in a psychophysical experiment how the body shape of different primate species interacts with the categorization of body movements of humans and non-human primates in human perception. CONCLUSION: Using modern computer graphics methods, highly realistic and well-controlled body motion stimuli can be generated from small numbers of photographs, allowing to study how species-specific motion and body shape interact in visual body motion perception.

Authors: Martini, Lucas M.; Lappe, Alexander; Giese, Martin A.

Research Areas: Uncategorized

Type of Publication: In Collection