Inside-Out Instrument Tracking for Surgical Navigation in Augmented Reality

Christina Gsaxner; Jianning Li; Antonio Pepe; Dieter Schmalstieg; Jan Egger

doi:10.1145/3489849.3489863

Inside-Out Instrument Tracking for Surgical Navigation in Augmented Reality

Christina Gsaxner^*, Jianning Li, Antonio Pepe, Dieter Schmalstieg, Jan Egger

^*Corresponding author for this work

Institute of Computer Graphics and Vision (7100)

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

Abstract

Surgical navigation requires tracking of instruments with respect to the patient. Conventionally, tracking is done with stationary cameras, and the navigation information is displayed on a stationary display. In contrast, an augmented reality (AR) headset can superimpose surgical navigation information directly in the surgeon’s view. However, AR needs to track the headset, the instruments and the patient, often by relying on stationary infrastructure. We show that 6DOF tracking can be obtained without any stationary, external system by purely utilizing the on-board stereo cameras of a HoloLens 2 to track the same retro-reflective marker spheres used by current optical navigation systems. Our implementation is based on two tracking pipelines complementing each other, one using conventional stereo vision techniques, the other relying on a single-constraint-at-a-time extended Kalman filter. In a technical evaluation of our tracking approach, we show that clinically relevant accuracy of 1.70 mm/1.11° and real-time performance is achievable. We further describe an example application of our system for untethered end-to-end surgical navigation.

Original language	English
Title of host publication	VRST '21
Subtitle of host publication	Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology
Editors	Stephen N. Spencer
Publisher	Association of Computing Machinery
ISBN (Electronic)	9781450390927
DOIs	https://doi.org/10.1145/3489849.3489863
Publication status	Published - 8 Dec 2021
Event	27th ACM Symposium on Virtual Reality Software and Technology: VRST 2021 - Virtuell, Austria Duration: 8 Dec 2021 → 10 Dec 2021

Publication series

Name	Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST

Conference

Conference	27th ACM Symposium on Virtual Reality Software and Technology
Abbreviated title	VRST 2021
Country/Territory	Austria
City	Virtuell
Period	8/12/21 → 10/12/21

Keywords

Augmented Reality
HoloLens 2
Surgical Navigation
Tracking

ASJC Scopus subject areas

Software

Access to Document

10.1145/3489849.3489863

Cite this

Gsaxner, C., Li, J., Pepe, A., Schmalstieg, D., & Egger, J. (2021). Inside-Out Instrument Tracking for Surgical Navigation in Augmented Reality. In S. N. Spencer (Ed.), VRST '21 : Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology Article 4 (Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST). Association of Computing Machinery. https://doi.org/10.1145/3489849.3489863

Inside-Out Instrument Tracking for Surgical Navigation in Augmented Reality. / Gsaxner, Christina; Li, Jianning; Pepe, Antonio et al.
VRST '21 : Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology. ed. / Stephen N. Spencer. Association of Computing Machinery, 2021. 4 (Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST).

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

Gsaxner, C, Li, J, Pepe, A, Schmalstieg, D & Egger, J 2021, Inside-Out Instrument Tracking for Surgical Navigation in Augmented Reality. in SN Spencer (ed.), VRST '21 : Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology., 4, Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST, Association of Computing Machinery, 27th ACM Symposium on Virtual Reality Software and Technology, Virtuell, Austria, 8/12/21. https://doi.org/10.1145/3489849.3489863

Gsaxner C, Li J, Pepe A, Schmalstieg D , Egger J. Inside-Out Instrument Tracking for Surgical Navigation in Augmented Reality. In Spencer SN, editor, VRST '21 : Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology. Association of Computing Machinery. 2021. 4. (Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST). doi: 10.1145/3489849.3489863

Gsaxner, Christina ; Li, Jianning ; Pepe, Antonio et al. / Inside-Out Instrument Tracking for Surgical Navigation in Augmented Reality. VRST '21 : Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology. editor / Stephen N. Spencer. Association of Computing Machinery, 2021. (Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST).

@inproceedings{50f8069d53cb4869a3480366f5269c77,

title = "Inside-Out Instrument Tracking for Surgical Navigation in Augmented Reality",

abstract = "Surgical navigation requires tracking of instruments with respect to the patient. Conventionally, tracking is done with stationary cameras, and the navigation information is displayed on a stationary display. In contrast, an augmented reality (AR) headset can superimpose surgical navigation information directly in the surgeon{\textquoteright}s view. However, AR needs to track the headset, the instruments and the patient, often by relying on stationary infrastructure. We show that 6DOF tracking can be obtained without any stationary, external system by purely utilizing the on-board stereo cameras of a HoloLens 2 to track the same retro-reflective marker spheres used by current optical navigation systems. Our implementation is based on two tracking pipelines complementing each other, one using conventional stereo vision techniques, the other relying on a single-constraint-at-a-time extended Kalman filter. In a technical evaluation of our tracking approach, we show that clinically relevant accuracy of 1.70 mm/1.11° and real-time performance is achievable. We further describe an example application of our system for untethered end-to-end surgical navigation.",

keywords = "Augmented Reality, HoloLens 2, Surgical Navigation, Tracking",

author = "Christina Gsaxner and Jianning Li and Antonio Pepe and Dieter Schmalstieg and Jan Egger",

note = "Publisher Copyright: {\textcopyright} 2021 Association for Computing Machinery.; 27th ACM Symposium on Virtual Reality Software and Technology : VRST 2021, VRST 2021 ; Conference date: 08-12-2021 Through 10-12-2021",

year = "2021",

month = dec,

day = "8",

doi = "10.1145/3489849.3489863",

language = "English",

series = "Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST",

publisher = "Association of Computing Machinery",

editor = "Spencer, {Stephen N.}",

booktitle = "VRST '21",

address = "United States",

}

TY - GEN

T1 - Inside-Out Instrument Tracking for Surgical Navigation in Augmented Reality

AU - Gsaxner, Christina

AU - Li, Jianning

AU - Pepe, Antonio

AU - Schmalstieg, Dieter

AU - Egger, Jan

PY - 2021/12/8

Y1 - 2021/12/8

N2 - Surgical navigation requires tracking of instruments with respect to the patient. Conventionally, tracking is done with stationary cameras, and the navigation information is displayed on a stationary display. In contrast, an augmented reality (AR) headset can superimpose surgical navigation information directly in the surgeon’s view. However, AR needs to track the headset, the instruments and the patient, often by relying on stationary infrastructure. We show that 6DOF tracking can be obtained without any stationary, external system by purely utilizing the on-board stereo cameras of a HoloLens 2 to track the same retro-reflective marker spheres used by current optical navigation systems. Our implementation is based on two tracking pipelines complementing each other, one using conventional stereo vision techniques, the other relying on a single-constraint-at-a-time extended Kalman filter. In a technical evaluation of our tracking approach, we show that clinically relevant accuracy of 1.70 mm/1.11° and real-time performance is achievable. We further describe an example application of our system for untethered end-to-end surgical navigation.

AB - Surgical navigation requires tracking of instruments with respect to the patient. Conventionally, tracking is done with stationary cameras, and the navigation information is displayed on a stationary display. In contrast, an augmented reality (AR) headset can superimpose surgical navigation information directly in the surgeon’s view. However, AR needs to track the headset, the instruments and the patient, often by relying on stationary infrastructure. We show that 6DOF tracking can be obtained without any stationary, external system by purely utilizing the on-board stereo cameras of a HoloLens 2 to track the same retro-reflective marker spheres used by current optical navigation systems. Our implementation is based on two tracking pipelines complementing each other, one using conventional stereo vision techniques, the other relying on a single-constraint-at-a-time extended Kalman filter. In a technical evaluation of our tracking approach, we show that clinically relevant accuracy of 1.70 mm/1.11° and real-time performance is achievable. We further describe an example application of our system for untethered end-to-end surgical navigation.

KW - Augmented Reality

KW - HoloLens 2

KW - Surgical Navigation

KW - Tracking

UR - http://www.scopus.com/inward/record.url?scp=85121437412&partnerID=8YFLogxK

U2 - 10.1145/3489849.3489863

DO - 10.1145/3489849.3489863

M3 - Conference paper

AN - SCOPUS:85121437412

T3 - Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST

BT - VRST '21

A2 - Spencer, Stephen N.

PB - Association of Computing Machinery

T2 - 27th ACM Symposium on Virtual Reality Software and Technology

Y2 - 8 December 2021 through 10 December 2021

ER -

Inside-Out Instrument Tracking for Surgical Navigation in Augmented Reality

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint