Passing a non-verbal turing test: Evaluating gesture animations generated from speech

Manuel Rebol, Christian Gütl, Krzysztof Pietroszek

Research output: Chapter in Book/Report/Conference proceedingConference paperpeer-review


In real life, people communicate using both speech and non-verbal signals such as gestures, face expression or body pose. Non-verbal signals impact the meaning of the spoken utterance in an abundance of ways. An absence of non-verbal signals impoverishes the process of communication. Yet, when users are represented as avatars, it is difficult to translate non-verbal signals along with the speech into the virtual world without specialized motion-capture hardware. In this paper, we propose a novel, data-driven technique for generating gestures directly from speech. Our approach is based on the application of Generative Adversarial Neural Networks (GANs) to model the correlation rather than causation between speech and gestures. This approach approximates neuroscience findings on how non-verbal communication and speech are correlated. We create a large dataset which consists of speech and corresponding gestures in a 3D human pose format from which our model learns the speaker-specific correlation. We evaluate the proposed technique in a user study that is inspired by the Turing test. For the study, we animate the generated gestures on a virtual character. We find that users are not able to distinguish between the generated and the recorded gestures. Moreover, users are able to identify our synthesized gestures as related or not related to a given utterance.
Original languageEnglish
Title of host publicationProceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2021
PublisherIEEE Publications
Number of pages9
ISBN (Electronic)9780738125565
ISBN (Print)978-1-6654-3068-5
Publication statusPublished - Mar 2021
Event28th IEEE Conference on Virtual Reality and 3D User Interfaces: IEEE VR 2021 - online, Virtual, Lisboa, Portugal
Duration: 27 Mar 20213 Apr 2021


Conference28th IEEE Conference on Virtual Reality and 3D User Interfaces
Abbreviated titleIEEE VR'21
CityVirtual, Lisboa
Internet address


  • Solid modeling
  • Three-dimensional displays
  • Correlation
  • Neuroscience
  • Pose estimation
  • Pipelines
  • Predictive models
  • Gesture Animation
  • 3D Human Pose Estimation
  • GAN
  • Human Body Language
  • VR

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Media Technology

Fields of Expertise

  • Information, Communication & Computing


Dive into the research topics of 'Passing a non-verbal turing test: Evaluating gesture animations generated from speech'. Together they form a unique fingerprint.

Cite this