Abstract
In real life, people communicate using both speech and non-verbal signals such as gestures, face expression or body pose. Non-verbal signals impact the meaning of the spoken utterance in an abundance of ways. An absence of non-verbal signals impoverishes the process of communication. Yet, when users are represented as avatars, it is difficult to translate non-verbal signals along with the speech into the virtual world without specialized motion-capture hardware. In this paper, we propose a novel, data-driven technique for generating gestures directly from speech. Our approach is based on the application of Generative Adversarial Neural Networks (GANs) to model the correlation rather than causation between speech and gestures. This approach approximates neuroscience findings on how non-verbal communication and speech are correlated. We create a large dataset which consists of speech and corresponding gestures in a 3D human pose format from which our model learns the speaker-specific correlation. We evaluate the proposed technique in a user study that is inspired by the Turing test. For the study, we animate the generated gestures on a virtual character. We find that users are not able to distinguish between the generated and the recorded gestures. Moreover, users are able to identify our synthesized gestures as related or not related to a given utterance.
Originalsprache | englisch |
---|---|
Titel | Proceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2021 |
Herausgeber (Verlag) | IEEE Publications |
Seiten | 573-581 |
Seitenumfang | 9 |
ISBN (elektronisch) | 9780738125565 |
ISBN (Print) | 978-1-6654-3068-5 |
DOIs | |
Publikationsstatus | Veröffentlicht - März 2021 |
Veranstaltung | 28th IEEE Conference on Virtual Reality and 3D User Interfaces: IEEE VR 2021 - online, Virtual, Lisboa, Portugal Dauer: 27 März 2021 → 3 Apr. 2021 http://ieeevr.org/2021/ |
Konferenz
Konferenz | 28th IEEE Conference on Virtual Reality and 3D User Interfaces |
---|---|
Kurztitel | IEEE VR'21 |
Land/Gebiet | Portugal |
Ort | Virtual, Lisboa |
Zeitraum | 27/03/21 → 3/04/21 |
Internetadresse |
ASJC Scopus subject areas
- Human-computer interaction
- Medientechnik
Fields of Expertise
- Information, Communication & Computing