Passing a non-verbal turing test: Evaluating gesture animations generated from speech

Manuel Rebol, Christian Gütl, Krzysztof Pietroszek

Publikation: Beitrag in Buch/Bericht/KonferenzbandBeitrag in einem KonferenzbandBegutachtung


In real life, people communicate using both speech and non-verbal signals such as gestures, face expression or body pose. Non-verbal signals impact the meaning of the spoken utterance in an abundance of ways. An absence of non-verbal signals impoverishes the process of communication. Yet, when users are represented as avatars, it is difficult to translate non-verbal signals along with the speech into the virtual world without specialized motion-capture hardware. In this paper, we propose a novel, data-driven technique for generating gestures directly from speech. Our approach is based on the application of Generative Adversarial Neural Networks (GANs) to model the correlation rather than causation between speech and gestures. This approach approximates neuroscience findings on how non-verbal communication and speech are correlated. We create a large dataset which consists of speech and corresponding gestures in a 3D human pose format from which our model learns the speaker-specific correlation. We evaluate the proposed technique in a user study that is inspired by the Turing test. For the study, we animate the generated gestures on a virtual character. We find that users are not able to distinguish between the generated and the recorded gestures. Moreover, users are able to identify our synthesized gestures as related or not related to a given utterance.
TitelProceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2021
Herausgeber (Verlag)IEEE Publications
ISBN (elektronisch)9780738125565
ISBN (Print)978-1-6654-3068-5
PublikationsstatusVeröffentlicht - März 2021
Veranstaltung28th IEEE Conference on Virtual Reality and 3D User Interfaces: IEEE VR 2021 - online, Virtual, Lisboa, Portugal
Dauer: 27 März 20213 Apr. 2021


Konferenz28th IEEE Conference on Virtual Reality and 3D User Interfaces
KurztitelIEEE VR'21
OrtVirtual, Lisboa

ASJC Scopus subject areas

  • Human-computer interaction
  • Medientechnik

Fields of Expertise

  • Information, Communication & Computing


Untersuchen Sie die Forschungsthemen von „Passing a non-verbal turing test: Evaluating gesture animations generated from speech“. Zusammen bilden sie einen einzigartigen Fingerprint.

Dieses zitieren