Markerless Image-to-Face Registration for Untethered Augmented Reality in Head and Neck Surgery

In the treatment of head and neck cancer, physicians can benefit from augmented reality in preparing and executing treatment. We present a system allowing a physician wearing an untethered augmented reality headset to see medical visualizations precisely overlaid onto the patient. Our main contribution is a strategy for markerless registration of 3D imaging to the patient’s face. We use a neural network to detect the face using the headset’s depth sensor and register it to computed tomography data. The face registration is seamlessly combined with the headset’s continuous self-localization. We report on registration error and compare our approach to an external, high-precision tracking system.