Deep Reinforcement Learning Based System for Intraoperative Hyperspectral Video Autofocusing

Charlie Budd; Jianrong Qiu; Oscar MacCormac; Martin Huber; Christopher Mower; Mirek Janatka; Théo Trotouin; Jonathan Shapey; Mads S. Bergholt; Tom Vercauteren

doi:10.1007/978-3-031-43996-4_63

Deep Reinforcement Learning Based System for Intraoperative Hyperspectral Video Autofocusing

Charlie Budd^*, Jianrong Qiu, Oscar MacCormac, Martin Huber, Christopher Mower, Mirek Janatka, Théo Trotouin, Jonathan Shapey, Mads S. Bergholt, Tom Vercauteren

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

2 Citations (Scopus)

Abstract

Hyperspectral imaging (HSI) captures a greater level of spectral detail than traditional optical imaging, making it a potentially valuable intraoperative tool when precise tissue differentiation is essential. Hardware limitations of current optical systems used for handheld real-time video HSI result in a limited focal depth, thereby posing usability issues for integration of the technology into the operating room. This work integrates a focus-tunable liquid lens into a video HSI exoscope, and proposes novel video autofocusing methods based on deep reinforcement learning. A first-of-its-kind robotic focal-time scan was performed to create a realistic and reproducible testing dataset. We benchmarked our proposed autofocus algorithm against traditional policies, and found our novel approach to perform significantly (p< 0.05 ) better than traditional techniques (0.070 ±. 098 mean absolute focal error compared to 0.146 ±. 148 ). In addition, we performed a blinded usability trial by having two neurosurgeons compare the system with different autofocus policies, and found our novel approach to be the most favourable, making our system a desirable addition for intraoperative HSI.

Original language	English
Title of host publication	Medical Image Computing and Computer Assisted Intervention – MICCAI 2023
Subtitle of host publication	26th International Conference, Vancouver, BC, Canada, October 8–12, 2023, Proceedings, Part IX
Editors	Hayit Greenspan, Anant Madabhushi, Parvin Mousavi, Septimiu Salcudean, James Duncan, Tanveer Syeda-Mahmood, Russell Taylor
Place of Publication	Cham
Publisher	Springer Nature
Pages	658-667
Number of pages	10
ISBN (Electronic)	9783031439964
ISBN (Print)	9783031439957
DOIs	https://doi.org/10.1007/978-3-031-43996-4_63
Publication status	Published - 2 Oct 2023
Event	26th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2023 - Vancouver, Canada Duration: 8 Oct 2023 → 12 Oct 2023

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	14228 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	26th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2023
Country/Territory	Canada
City	Vancouver
Period	8/10/2023 → 12/10/2023

Keywords

Autofocus
Computer Assisted Intervention
Deep Reinforcement Learning
Hyperspectral Imaging

Access to Document

10.1007/978-3-031-43996-4_63

Cite this

Budd, C., Qiu, J., MacCormac, O., Huber, M., Mower, C., Janatka, M., Trotouin, T., Shapey, J., Bergholt, M. S., & Vercauteren, T. (2023). Deep Reinforcement Learning Based System for Intraoperative Hyperspectral Video Autofocusing. In H. Greenspan, A. Madabhushi, P. Mousavi, S. Salcudean, J. Duncan, T. Syeda-Mahmood, & R. Taylor (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2023: 26th International Conference, Vancouver, BC, Canada, October 8–12, 2023, Proceedings, Part IX (pp. 658-667). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14228 LNCS). Springer Nature. https://doi.org/10.1007/978-3-031-43996-4_63

Budd, Charlie ; Qiu, Jianrong ; MacCormac, Oscar et al. / Deep Reinforcement Learning Based System for Intraoperative Hyperspectral Video Autofocusing. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023: 26th International Conference, Vancouver, BC, Canada, October 8–12, 2023, Proceedings, Part IX. editor / Hayit Greenspan ; Anant Madabhushi ; Parvin Mousavi ; Septimiu Salcudean ; James Duncan ; Tanveer Syeda-Mahmood ; Russell Taylor. Cham : Springer Nature, 2023. pp. 658-667 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inbook{a2cad928465b4247a4f4ced0328e95eb,

title = "Deep Reinforcement Learning Based System for Intraoperative Hyperspectral Video Autofocusing",

abstract = "Hyperspectral imaging (HSI) captures a greater level of spectral detail than traditional optical imaging, making it a potentially valuable intraoperative tool when precise tissue differentiation is essential. Hardware limitations of current optical systems used for handheld real-time video HSI result in a limited focal depth, thereby posing usability issues for integration of the technology into the operating room. This work integrates a focus-tunable liquid lens into a video HSI exoscope, and proposes novel video autofocusing methods based on deep reinforcement learning. A first-of-its-kind robotic focal-time scan was performed to create a realistic and reproducible testing dataset. We benchmarked our proposed autofocus algorithm against traditional policies, and found our novel approach to perform significantly (p< 0.05 ) better than traditional techniques (0.070 ±. 098 mean absolute focal error compared to 0.146 ±. 148 ). In addition, we performed a blinded usability trial by having two neurosurgeons compare the system with different autofocus policies, and found our novel approach to be the most favourable, making our system a desirable addition for intraoperative HSI.",

keywords = "Autofocus, Computer Assisted Intervention, Deep Reinforcement Learning, Hyperspectral Imaging",

author = "Charlie Budd and Jianrong Qiu and Oscar MacCormac and Martin Huber and Christopher Mower and Mirek Janatka and Th{\'e}o Trotouin and Jonathan Shapey and Bergholt, {Mads S.} and Tom Vercauteren",

note = "Funding Information: This study/project is funded by the NIHR [NIHR202114]. This work was supported by core funding from the Wellcome/EPSRC [WT203148/Z/16/Z; NS/A000049/1]. This project has received funding from the European Union{\textquoteright}s Horizon 2020 research and innovation programme under grant agreement No 101016985 (FAROS project). TV is supported by a Medtronic/RAEng Research Chair [RCSRF1819\7\34]. For the purpose of open access, the authors have applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission. TV is a co-founder and shareholder of Hypervision Surgical. Funding Information: Acknowledgements. This study/project is funded by the NIHR [NIHR202114]. This Funding Information: work was supported by core funding from the Wellcome/EPSRC [WT203148/Z/16/Z; NS/A000049/1]. This project has received funding from the European Union{\textquoteright}s Horizon 2020 research and innovation programme under grant agreement No 101016985 (FAROS project). TV is supported by a Medtronic/RAEng Research Chair [RCSRF1819\7\34]. For the purpose of open access, the authors have applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission. TV is a co-founder and shareholder of Hypervision Surgical. Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2023.; 26th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2023 ; Conference date: 08-10-2023 Through 12-10-2023",

year = "2023",

month = oct,

day = "2",

doi = "10.1007/978-3-031-43996-4_63",

language = "English",

isbn = "9783031439957",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Nature",

pages = "658--667",

editor = "Hayit Greenspan and Anant Madabhushi and Parvin Mousavi and Septimiu Salcudean and James Duncan and Tanveer Syeda-Mahmood and Russell Taylor",

booktitle = "Medical Image Computing and Computer Assisted Intervention – MICCAI 2023",

}

Budd, C, Qiu, J, MacCormac, O, Huber, M , Mower, C, Janatka, M, Trotouin, T, Shapey, J , Bergholt, MS & Vercauteren, T 2023, Deep Reinforcement Learning Based System for Intraoperative Hyperspectral Video Autofocusing. in H Greenspan, A Madabhushi, P Mousavi, S Salcudean, J Duncan, T Syeda-Mahmood & R Taylor (eds), Medical Image Computing and Computer Assisted Intervention – MICCAI 2023: 26th International Conference, Vancouver, BC, Canada, October 8–12, 2023, Proceedings, Part IX. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14228 LNCS, Springer Nature, Cham, pp. 658-667, 26th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2023, Vancouver, Canada, 8/10/2023. https://doi.org/10.1007/978-3-031-43996-4_63

Deep Reinforcement Learning Based System for Intraoperative Hyperspectral Video Autofocusing. / Budd, Charlie; Qiu, Jianrong; MacCormac, Oscar et al.
Medical Image Computing and Computer Assisted Intervention – MICCAI 2023: 26th International Conference, Vancouver, BC, Canada, October 8–12, 2023, Proceedings, Part IX. ed. / Hayit Greenspan; Anant Madabhushi; Parvin Mousavi; Septimiu Salcudean; James Duncan; Tanveer Syeda-Mahmood; Russell Taylor. Cham: Springer Nature, 2023. p. 658-667 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14228 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

TY - CHAP

T1 - Deep Reinforcement Learning Based System for Intraoperative Hyperspectral Video Autofocusing

AU - Budd, Charlie

AU - Qiu, Jianrong

AU - MacCormac, Oscar

AU - Huber, Martin

AU - Mower, Christopher

AU - Janatka, Mirek

AU - Trotouin, Théo

AU - Shapey, Jonathan

AU - Bergholt, Mads S.

AU - Vercauteren, Tom

N1 - Funding Information: This study/project is funded by the NIHR [NIHR202114]. This work was supported by core funding from the Wellcome/EPSRC [WT203148/Z/16/Z; NS/A000049/1]. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101016985 (FAROS project). TV is supported by a Medtronic/RAEng Research Chair [RCSRF1819\7\34]. For the purpose of open access, the authors have applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission. TV is a co-founder and shareholder of Hypervision Surgical. Funding Information: Acknowledgements. This study/project is funded by the NIHR [NIHR202114]. This Funding Information: work was supported by core funding from the Wellcome/EPSRC [WT203148/Z/16/Z; NS/A000049/1]. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101016985 (FAROS project). TV is supported by a Medtronic/RAEng Research Chair [RCSRF1819\7\34]. For the purpose of open access, the authors have applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission. TV is a co-founder and shareholder of Hypervision Surgical. Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023.

PY - 2023/10/2

Y1 - 2023/10/2

N2 - Hyperspectral imaging (HSI) captures a greater level of spectral detail than traditional optical imaging, making it a potentially valuable intraoperative tool when precise tissue differentiation is essential. Hardware limitations of current optical systems used for handheld real-time video HSI result in a limited focal depth, thereby posing usability issues for integration of the technology into the operating room. This work integrates a focus-tunable liquid lens into a video HSI exoscope, and proposes novel video autofocusing methods based on deep reinforcement learning. A first-of-its-kind robotic focal-time scan was performed to create a realistic and reproducible testing dataset. We benchmarked our proposed autofocus algorithm against traditional policies, and found our novel approach to perform significantly (p< 0.05 ) better than traditional techniques (0.070 ±. 098 mean absolute focal error compared to 0.146 ±. 148 ). In addition, we performed a blinded usability trial by having two neurosurgeons compare the system with different autofocus policies, and found our novel approach to be the most favourable, making our system a desirable addition for intraoperative HSI.

AB - Hyperspectral imaging (HSI) captures a greater level of spectral detail than traditional optical imaging, making it a potentially valuable intraoperative tool when precise tissue differentiation is essential. Hardware limitations of current optical systems used for handheld real-time video HSI result in a limited focal depth, thereby posing usability issues for integration of the technology into the operating room. This work integrates a focus-tunable liquid lens into a video HSI exoscope, and proposes novel video autofocusing methods based on deep reinforcement learning. A first-of-its-kind robotic focal-time scan was performed to create a realistic and reproducible testing dataset. We benchmarked our proposed autofocus algorithm against traditional policies, and found our novel approach to perform significantly (p< 0.05 ) better than traditional techniques (0.070 ±. 098 mean absolute focal error compared to 0.146 ±. 148 ). In addition, we performed a blinded usability trial by having two neurosurgeons compare the system with different autofocus policies, and found our novel approach to be the most favourable, making our system a desirable addition for intraoperative HSI.

KW - Autofocus

KW - Computer Assisted Intervention

KW - Deep Reinforcement Learning

KW - Hyperspectral Imaging

UR - http://www.scopus.com/inward/record.url?scp=85174707710&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-43996-4_63

DO - 10.1007/978-3-031-43996-4_63

M3 - Conference paper

AN - SCOPUS:85174707710

SN - 9783031439957

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 658

EP - 667

BT - Medical Image Computing and Computer Assisted Intervention – MICCAI 2023

A2 - Greenspan, Hayit

A2 - Madabhushi, Anant

A2 - Mousavi, Parvin

A2 - Salcudean, Septimiu

A2 - Duncan, James

A2 - Syeda-Mahmood, Tanveer

A2 - Taylor, Russell

PB - Springer Nature

CY - Cham

T2 - 26th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2023

Y2 - 8 October 2023 through 12 October 2023

ER -

Budd C, Qiu J, MacCormac O, Huber M , Mower C, Janatka M et al. Deep Reinforcement Learning Based System for Intraoperative Hyperspectral Video Autofocusing. In Greenspan H, Madabhushi A, Mousavi P, Salcudean S, Duncan J, Syeda-Mahmood T, Taylor R, editors, Medical Image Computing and Computer Assisted Intervention – MICCAI 2023: 26th International Conference, Vancouver, BC, Canada, October 8–12, 2023, Proceedings, Part IX. Cham: Springer Nature. 2023. p. 658-667. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). Epub 2023 Oct 1. doi: 10.1007/978-3-031-43996-4_63