Privileged Anatomical and Protocol Discrimination in Trackerless 3D Ultrasound Reconstruction

Qi Li; Ziyi Shen; Qian Li; Dean C. Barratt; Thomas Dowrick; Matthew J. Clarkson; Tom Vercauteren; Yipeng Hu

doi:10.1007/978-3-031-44521-7_14

Privileged Anatomical and Protocol Discrimination in Trackerless 3D Ultrasound Reconstruction

Qi Li^*, Ziyi Shen, Qian Li, Dean C. Barratt, Thomas Dowrick, Matthew J. Clarkson, Tom Vercauteren, Yipeng Hu

^*Corresponding author for this work

Surgical & Interventional Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

1 Citation (Scopus)

Abstract

Three-dimensional (3D) freehand ultrasound (US) reconstruction without using any additional external tracking device has seen recent advances with deep neural networks (DNNs). In this paper, we first investigated two identified contributing factors of the learned inter-frame correlation that enable the DNN-based reconstruction: anatomy and protocol. We propose to incorporate the ability to represent these two factors - readily available during training - as the privileged information to improve existing DNN-based methods. This is implemented in a new multi-task method, where the anatomical and protocol discrimination are used as auxiliary tasks. We further develop a differentiable network architecture to optimise the branching location of these auxiliary tasks, which controls the ratio between shared and task-specific network parameters, for maximising the benefits from the two auxiliary tasks. Experimental results, on a dataset with 38 forearms of 19 volunteers acquired with 6 different scanning protocols, show that 1) both anatomical and protocol variances are enabling factors for DNN-based US reconstruction; 2) learning how to discriminate different subjects (anatomical variance) and predefined types of scanning paths (protocol variance) both significantly improve frame prediction accuracy, volume reconstruction overlap, accumulated tracking error and final drift, using the proposed algorithm.

Original language	English
Title of host publication	Simplifying Medical Ultrasound
Subtitle of host publication	4th International Workshop, ASMUS 2023, Held in Conjunction with MICCAI 2023, Vancouver, BC, Canada, October 8, 2023, Proceedings
Editors	Bernhard Kainz, Alison Noble, Julia Schnabel, Bishesh Khanal, Johanna Paula Müller, Thomas Day
Place of Publication	Cham
Publisher	Springer Nature
Pages	142-151
Number of pages	10
ISBN (Electronic)	9783031445217
ISBN (Print)	9783031445200
DOIs	https://doi.org/10.1007/978-3-031-44521-7_14
Publication status	Published - 2 Oct 2023
Event	4th International Workshop of Advances in Simplifying Medical Ultrasound, ASMUS 2023 - Vancouver, Canada Duration: 8 Oct 2023 → 8 Oct 2023

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	14337 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	4th International Workshop of Advances in Simplifying Medical Ultrasound, ASMUS 2023
Country/Territory	Canada
City	Vancouver
Period	8/10/2023 → 8/10/2023

Keywords

Freehand ultrasound
Multi-task learning
Privileged information

Access to Document

10.1007/978-3-031-44521-7_14

Cite this

Li, Q., Shen, Z., Li, Q., Barratt, D. C., Dowrick, T., Clarkson, M. J., Vercauteren, T., & Hu, Y. (2023). Privileged Anatomical and Protocol Discrimination in Trackerless 3D Ultrasound Reconstruction. In B. Kainz, A. Noble, J. Schnabel, B. Khanal, J. P. Müller, & T. Day (Eds.), Simplifying Medical Ultrasound: 4th International Workshop, ASMUS 2023, Held in Conjunction with MICCAI 2023, Vancouver, BC, Canada, October 8, 2023, Proceedings (pp. 142-151). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14337 LNCS). Springer Nature. https://doi.org/10.1007/978-3-031-44521-7_14

Li, Qi ; Shen, Ziyi ; Li, Qian et al. / Privileged Anatomical and Protocol Discrimination in Trackerless 3D Ultrasound Reconstruction. Simplifying Medical Ultrasound: 4th International Workshop, ASMUS 2023, Held in Conjunction with MICCAI 2023, Vancouver, BC, Canada, October 8, 2023, Proceedings. editor / Bernhard Kainz ; Alison Noble ; Julia Schnabel ; Bishesh Khanal ; Johanna Paula Müller ; Thomas Day. Cham : Springer Nature, 2023. pp. 142-151 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inbook{c2fd2cb5db8044a5b67f5af862209974,

title = "Privileged Anatomical and Protocol Discrimination in Trackerless 3D Ultrasound Reconstruction",

abstract = "Three-dimensional (3D) freehand ultrasound (US) reconstruction without using any additional external tracking device has seen recent advances with deep neural networks (DNNs). In this paper, we first investigated two identified contributing factors of the learned inter-frame correlation that enable the DNN-based reconstruction: anatomy and protocol. We propose to incorporate the ability to represent these two factors - readily available during training - as the privileged information to improve existing DNN-based methods. This is implemented in a new multi-task method, where the anatomical and protocol discrimination are used as auxiliary tasks. We further develop a differentiable network architecture to optimise the branching location of these auxiliary tasks, which controls the ratio between shared and task-specific network parameters, for maximising the benefits from the two auxiliary tasks. Experimental results, on a dataset with 38 forearms of 19 volunteers acquired with 6 different scanning protocols, show that 1) both anatomical and protocol variances are enabling factors for DNN-based US reconstruction; 2) learning how to discriminate different subjects (anatomical variance) and predefined types of scanning paths (protocol variance) both significantly improve frame prediction accuracy, volume reconstruction overlap, accumulated tracking error and final drift, using the proposed algorithm.",

keywords = "Freehand ultrasound, Multi-task learning, Privileged information",

author = "Qi Li and Ziyi Shen and Qian Li and Barratt, {Dean C.} and Thomas Dowrick and Clarkson, {Matthew J.} and Tom Vercauteren and Yipeng Hu",

note = "Funding Information: Declarations. This work was supported by the EPSRC [EP/T029404/1], a Royal Academy of Engineering / Medtronic Research Chair [RCSRF1819\7\734] (TV), Wellcome/EPSRC Centre for Interventional and Surgical Sciences [203145Z/16/Z], and the International Alliance for Cancer Early Detection, an alliance between Cancer Research UK [C28070/A30912; C73666/A31378], Canary Center at Stanford University, the University of Cambridge, OHSU Knight Cancer Institute, University College London and the University of Manchester. TV is co-founder and shareholder of Hypervision Surgical. Qi Li was supported by the University College London Overseas and Graduate Research Scholarships. For the purpose of open access, the authors have applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission. This study was performed in accordance with the ethical standards in the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards. Approval was granted by the Ethics Committee of local institution (UCL Department of Medical Physics and Biomedical Engineering) on 20th Jan. 2023 [24055/001]. Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 4th International Workshop of Advances in Simplifying Medical Ultrasound, ASMUS 2023 ; Conference date: 08-10-2023 Through 08-10-2023",

year = "2023",

month = oct,

day = "2",

doi = "10.1007/978-3-031-44521-7_14",

language = "English",

isbn = "9783031445200",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Nature",

pages = "142--151",

editor = "Bernhard Kainz and Alison Noble and Julia Schnabel and Bishesh Khanal and M{\"u}ller, {Johanna Paula} and Thomas Day",

booktitle = "Simplifying Medical Ultrasound",

}

Li, Q, Shen, Z, Li, Q, Barratt, DC, Dowrick, T, Clarkson, MJ, Vercauteren, T & Hu, Y 2023, Privileged Anatomical and Protocol Discrimination in Trackerless 3D Ultrasound Reconstruction. in B Kainz, A Noble, J Schnabel, B Khanal, JP Müller & T Day (eds), Simplifying Medical Ultrasound: 4th International Workshop, ASMUS 2023, Held in Conjunction with MICCAI 2023, Vancouver, BC, Canada, October 8, 2023, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14337 LNCS, Springer Nature, Cham, pp. 142-151, 4th International Workshop of Advances in Simplifying Medical Ultrasound, ASMUS 2023, Vancouver, Canada, 8/10/2023. https://doi.org/10.1007/978-3-031-44521-7_14

Privileged Anatomical and Protocol Discrimination in Trackerless 3D Ultrasound Reconstruction. / Li, Qi; Shen, Ziyi; Li, Qian et al.
Simplifying Medical Ultrasound: 4th International Workshop, ASMUS 2023, Held in Conjunction with MICCAI 2023, Vancouver, BC, Canada, October 8, 2023, Proceedings. ed. / Bernhard Kainz; Alison Noble; Julia Schnabel; Bishesh Khanal; Johanna Paula Müller; Thomas Day. Cham: Springer Nature, 2023. p. 142-151 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14337 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

TY - CHAP

T1 - Privileged Anatomical and Protocol Discrimination in Trackerless 3D Ultrasound Reconstruction

AU - Li, Qi

AU - Shen, Ziyi

AU - Li, Qian

AU - Barratt, Dean C.

AU - Dowrick, Thomas

AU - Clarkson, Matthew J.

AU - Vercauteren, Tom

AU - Hu, Yipeng

N1 - Funding Information: Declarations. This work was supported by the EPSRC [EP/T029404/1], a Royal Academy of Engineering / Medtronic Research Chair [RCSRF1819\7\734] (TV), Wellcome/EPSRC Centre for Interventional and Surgical Sciences [203145Z/16/Z], and the International Alliance for Cancer Early Detection, an alliance between Cancer Research UK [C28070/A30912; C73666/A31378], Canary Center at Stanford University, the University of Cambridge, OHSU Knight Cancer Institute, University College London and the University of Manchester. TV is co-founder and shareholder of Hypervision Surgical. Qi Li was supported by the University College London Overseas and Graduate Research Scholarships. For the purpose of open access, the authors have applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission. This study was performed in accordance with the ethical standards in the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards. Approval was granted by the Ethics Committee of local institution (UCL Department of Medical Physics and Biomedical Engineering) on 20th Jan. 2023 [24055/001]. Publisher Copyright: © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

PY - 2023/10/2

Y1 - 2023/10/2

N2 - Three-dimensional (3D) freehand ultrasound (US) reconstruction without using any additional external tracking device has seen recent advances with deep neural networks (DNNs). In this paper, we first investigated two identified contributing factors of the learned inter-frame correlation that enable the DNN-based reconstruction: anatomy and protocol. We propose to incorporate the ability to represent these two factors - readily available during training - as the privileged information to improve existing DNN-based methods. This is implemented in a new multi-task method, where the anatomical and protocol discrimination are used as auxiliary tasks. We further develop a differentiable network architecture to optimise the branching location of these auxiliary tasks, which controls the ratio between shared and task-specific network parameters, for maximising the benefits from the two auxiliary tasks. Experimental results, on a dataset with 38 forearms of 19 volunteers acquired with 6 different scanning protocols, show that 1) both anatomical and protocol variances are enabling factors for DNN-based US reconstruction; 2) learning how to discriminate different subjects (anatomical variance) and predefined types of scanning paths (protocol variance) both significantly improve frame prediction accuracy, volume reconstruction overlap, accumulated tracking error and final drift, using the proposed algorithm.

AB - Three-dimensional (3D) freehand ultrasound (US) reconstruction without using any additional external tracking device has seen recent advances with deep neural networks (DNNs). In this paper, we first investigated two identified contributing factors of the learned inter-frame correlation that enable the DNN-based reconstruction: anatomy and protocol. We propose to incorporate the ability to represent these two factors - readily available during training - as the privileged information to improve existing DNN-based methods. This is implemented in a new multi-task method, where the anatomical and protocol discrimination are used as auxiliary tasks. We further develop a differentiable network architecture to optimise the branching location of these auxiliary tasks, which controls the ratio between shared and task-specific network parameters, for maximising the benefits from the two auxiliary tasks. Experimental results, on a dataset with 38 forearms of 19 volunteers acquired with 6 different scanning protocols, show that 1) both anatomical and protocol variances are enabling factors for DNN-based US reconstruction; 2) learning how to discriminate different subjects (anatomical variance) and predefined types of scanning paths (protocol variance) both significantly improve frame prediction accuracy, volume reconstruction overlap, accumulated tracking error and final drift, using the proposed algorithm.

KW - Freehand ultrasound

KW - Multi-task learning

KW - Privileged information

UR - http://www.scopus.com/inward/record.url?scp=85174724468&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-44521-7_14

DO - 10.1007/978-3-031-44521-7_14

M3 - Conference paper

AN - SCOPUS:85174724468

SN - 9783031445200

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 142

EP - 151

BT - Simplifying Medical Ultrasound

A2 - Kainz, Bernhard

A2 - Noble, Alison

A2 - Schnabel, Julia

A2 - Khanal, Bishesh

A2 - Müller, Johanna Paula

A2 - Day, Thomas

PB - Springer Nature

CY - Cham

T2 - 4th International Workshop of Advances in Simplifying Medical Ultrasound, ASMUS 2023

Y2 - 8 October 2023 through 8 October 2023

ER -

Li Q, Shen Z, Li Q, Barratt DC, Dowrick T, Clarkson MJ et al. Privileged Anatomical and Protocol Discrimination in Trackerless 3D Ultrasound Reconstruction. In Kainz B, Noble A, Schnabel J, Khanal B, Müller JP, Day T, editors, Simplifying Medical Ultrasound: 4th International Workshop, ASMUS 2023, Held in Conjunction with MICCAI 2023, Vancouver, BC, Canada, October 8, 2023, Proceedings. Cham: Springer Nature. 2023. p. 142-151. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). Epub 2023 Oct 1. doi: 10.1007/978-3-031-44521-7_14