Abstract
We propose a robust and accurate method for estimating the 3D poses of two hands in close interaction from a single color image. This is a very challenging problem, as large occlusions and many confusions between the joints may happen. State-of-the-art methods solve this problem by regressing a heatmap for each joint, which requires solving two problems simultaneously: localizing the joints and recognizing them. In this work, we propose to separate these tasks by relying on a CNN to first localize joints as 2D keypoints, and on self-attention between the CNN features at these keypoints to associate them with the corresponding hand joint. The resulting architecture, which we call 'Keypoint Transformer', is highly efficient as it achieves state-of-the-art performance with roughly half the number of model parameters on the InterHand2.6M dataset. We also show it can be easily extended to estimate the 3D pose of an object manipulated by one or two hands with high performance. Moreover, we created a new dataset of more than 75,000 images of two hands manipulating an object fully annotated in 3D and will make it publicly available.
Original language | English |
---|---|
Title of host publication | Proceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022 |
Publisher | IEEE Computer Society Publications |
Pages | 11080-11090 |
Number of pages | 11 |
ISBN (Electronic) | 9781665469463 |
DOIs | |
Publication status | Published - 2022 |
Event | 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition: CVPR 2022 - New Orleans, United States Duration: 19 Jun 2022 → 24 Jun 2022 |
Conference
Conference | 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition |
---|---|
Abbreviated title | CVPR 2022 |
Country/Territory | United States |
City | New Orleans |
Period | 19/06/22 → 24/06/22 |
Keywords
- 3D from single images
- Datasets and evaluation
- Deep learning architectures and techniques
- Pose estimation and tracking
ASJC Scopus subject areas
- Software
- Computer Vision and Pattern Recognition