Zero-shot super-resolution with a physically-motivated downsampling kernel for endomicroscopy.

Agnieszka Barbara Szczotka; Dzhoshkun Ismail Shakir; Matthew J. Clarkson; Stephen P. Pereira; Tom Vercauteren

doi:10.1109/TMI.2021.3067512

Zero-shot super-resolution with a physically-motivated downsampling kernel for endomicroscopy.

Agnieszka Barbara Szczotka, Dzhoshkun Ismail Shakir, Matthew J. Clarkson, Stephen P. Pereira, Tom Vercauteren

Research output: Contribution to journal › Article › peer-review

13 Citations (Scopus)

59 Downloads (Pure)

Abstract

Super-resolution (SR) methods have seen significant advances thanks to the development of convolutional neural networks (CNNs). CNNs have been successfully employed to improve the quality of endomicroscopy imaging. Yet, the inherent limitation of research on SR in endomicroscopy remains the lack of ground truth high-resolution (HR) images, commonly used for both supervised training and reference-based image quality assessment (IQA). Therefore, alternative methods, such as unsupervised SR are being explored. To address the need for non-reference image quality improvement, we designed a novel zero-shot super-resolution (ZSSR) approach that relies only on the endomicroscopy data to be processed in a self-supervised manner without the need for ground-truth HR images. We tailored the proposed pipeline to the idiosyncrasies of endomicroscopy by introducing both: a physically-motivated Voronoi downscaling kernel accounting for the endomicroscope's irregular fibre-based sampling pattern, and realistic noise patterns. We also took advantage of video sequences to exploit a sequence of images for self-supervised zero-shot image quality improvement. We run ablation studies to assess our contribution in regards to the downscaling kernel and noise simulation. We validate our methodology on both synthetic and original data. Synthetic experiments were assessed with reference-based IQA, while our results for original images were evaluated in a user study conducted with both expert and non-expert observers. The results demonstrated superior performance in image quality of ZSSR reconstructions in comparison to the baseline method. The ZSSR is also competitive when compared to supervised single-image SR, especially being the preferred reconstruction technique by experts.

Original language	English
Article number	9381880
Pages (from-to)	1863-1874
Number of pages	12
Journal	IEEE Transactions on Medical Imaging
Volume	40
Issue number	7
Early online date	19 Mar 2021
DOIs	https://doi.org/10.1109/TMI.2021.3067512
Publication status	Published - Jul 2021

Access to Document

10.1109/TMI.2021.3067512

2021 - Szczotka et al. - IEEE Transactions on Medical Imaging - Zero-shot super-resolution with a physically-motivated downsampling kernAccepted author manuscript, 529 KB

https://ieeexplore.ieee.org/document/9381880/

Cite this

@article{a8f336032da94cf2881450f781be943c,

title = "Zero-shot super-resolution with a physically-motivated downsampling kernel for endomicroscopy.",

abstract = "Super-resolution (SR) methods have seen significant advances thanks to the development of convolutional neural networks (CNNs). CNNs have been successfully employed to improve the quality of endomicroscopy imaging. Yet, the inherent limitation of research on SR in endomicroscopy remains the lack of ground truth high-resolution (HR) images, commonly used for both supervised training and reference-based image quality assessment (IQA). Therefore, alternative methods, such as unsupervised SR are being explored. To address the need for non-reference image quality improvement, we designed a novel zero-shot super-resolution (ZSSR) approach that relies only on the endomicroscopy data to be processed in a self-supervised manner without the need for ground-truth HR images. We tailored the proposed pipeline to the idiosyncrasies of endomicroscopy by introducing both: a physically-motivated Voronoi downscaling kernel accounting for the endomicroscope's irregular fibre-based sampling pattern, and realistic noise patterns. We also took advantage of video sequences to exploit a sequence of images for self-supervised zero-shot image quality improvement. We run ablation studies to assess our contribution in regards to the downscaling kernel and noise simulation. We validate our methodology on both synthetic and original data. Synthetic experiments were assessed with reference-based IQA, while our results for original images were evaluated in a user study conducted with both expert and non-expert observers. The results demonstrated superior performance in image quality of ZSSR reconstructions in comparison to the baseline method. The ZSSR is also competitive when compared to supervised single-image SR, especially being the preferred reconstruction technique by experts.",

author = "Szczotka, {Agnieszka Barbara} and Shakir, {Dzhoshkun Ismail} and Clarkson, {Matthew J.} and Pereira, {Stephen P.} and Tom Vercauteren",

note = "Funding Information: Manuscript received February 1, 2021; revised March 12, 2021; accepted March 14, 2021. Date of publication March 19, 2021; date of current version June 30, 2021. This work was supported in part by the Wellcome Trust under Grant 203145Z/16/Z and Grant 203148/Z/16/Z; in part by the Engineering and Physical Sciences Research Council (EPSRC) under Grant NS/A000050/1, Grant NS/A000049/1, and Grant NS/A000027/1; and in part by the Department of Health (DoH) National Institute for Health Research (NIHR) University College London Hospitals (UCLH) Biomedical Research Centre (BRC) Funding Scheme through University College London (UCL) and UCLH. The work of Agnieszka Barbara Szczotka was supported by Mauna Kea Technologies, Paris, France. The work of Tom Vercauteren was supported by the Medtronic/Royal Academy of Engineering Research Chair under Grant RCSRF1819/7/34. (Corresponding author: Agnieszka Barbara Szczotka.) Agnieszka Barbara Szczotka and Matthew J. Clarkson are with the Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, WC1E 6BT London, U.K. (e-mail: [email protected]; [email protected]). Publisher Copyright: {\textcopyright} 1982-2012 IEEE. Copyright: Copyright 2021 Elsevier B.V., All rights reserved.",

year = "2021",

month = jul,

doi = "10.1109/TMI.2021.3067512",

language = "English",

volume = "40",

pages = "1863--1874",

journal = "IEEE Transactions on Medical Imaging",

issn = "0278-0062",

publisher = "I E E E",

number = "7",

}

TY - JOUR

T1 - Zero-shot super-resolution with a physically-motivated downsampling kernel for endomicroscopy.

AU - Szczotka, Agnieszka Barbara

AU - Shakir, Dzhoshkun Ismail

AU - Clarkson, Matthew J.

AU - Pereira, Stephen P.

AU - Vercauteren, Tom

N1 - Funding Information: Manuscript received February 1, 2021; revised March 12, 2021; accepted March 14, 2021. Date of publication March 19, 2021; date of current version June 30, 2021. This work was supported in part by the Wellcome Trust under Grant 203145Z/16/Z and Grant 203148/Z/16/Z; in part by the Engineering and Physical Sciences Research Council (EPSRC) under Grant NS/A000050/1, Grant NS/A000049/1, and Grant NS/A000027/1; and in part by the Department of Health (DoH) National Institute for Health Research (NIHR) University College London Hospitals (UCLH) Biomedical Research Centre (BRC) Funding Scheme through University College London (UCL) and UCLH. The work of Agnieszka Barbara Szczotka was supported by Mauna Kea Technologies, Paris, France. The work of Tom Vercauteren was supported by the Medtronic/Royal Academy of Engineering Research Chair under Grant RCSRF1819/7/34. (Corresponding author: Agnieszka Barbara Szczotka.) Agnieszka Barbara Szczotka and Matthew J. Clarkson are with the Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, WC1E 6BT London, U.K. (e-mail: [email protected]; [email protected]). Publisher Copyright: © 1982-2012 IEEE. Copyright: Copyright 2021 Elsevier B.V., All rights reserved.

PY - 2021/7

Y1 - 2021/7

N2 - Super-resolution (SR) methods have seen significant advances thanks to the development of convolutional neural networks (CNNs). CNNs have been successfully employed to improve the quality of endomicroscopy imaging. Yet, the inherent limitation of research on SR in endomicroscopy remains the lack of ground truth high-resolution (HR) images, commonly used for both supervised training and reference-based image quality assessment (IQA). Therefore, alternative methods, such as unsupervised SR are being explored. To address the need for non-reference image quality improvement, we designed a novel zero-shot super-resolution (ZSSR) approach that relies only on the endomicroscopy data to be processed in a self-supervised manner without the need for ground-truth HR images. We tailored the proposed pipeline to the idiosyncrasies of endomicroscopy by introducing both: a physically-motivated Voronoi downscaling kernel accounting for the endomicroscope's irregular fibre-based sampling pattern, and realistic noise patterns. We also took advantage of video sequences to exploit a sequence of images for self-supervised zero-shot image quality improvement. We run ablation studies to assess our contribution in regards to the downscaling kernel and noise simulation. We validate our methodology on both synthetic and original data. Synthetic experiments were assessed with reference-based IQA, while our results for original images were evaluated in a user study conducted with both expert and non-expert observers. The results demonstrated superior performance in image quality of ZSSR reconstructions in comparison to the baseline method. The ZSSR is also competitive when compared to supervised single-image SR, especially being the preferred reconstruction technique by experts.

AB - Super-resolution (SR) methods have seen significant advances thanks to the development of convolutional neural networks (CNNs). CNNs have been successfully employed to improve the quality of endomicroscopy imaging. Yet, the inherent limitation of research on SR in endomicroscopy remains the lack of ground truth high-resolution (HR) images, commonly used for both supervised training and reference-based image quality assessment (IQA). Therefore, alternative methods, such as unsupervised SR are being explored. To address the need for non-reference image quality improvement, we designed a novel zero-shot super-resolution (ZSSR) approach that relies only on the endomicroscopy data to be processed in a self-supervised manner without the need for ground-truth HR images. We tailored the proposed pipeline to the idiosyncrasies of endomicroscopy by introducing both: a physically-motivated Voronoi downscaling kernel accounting for the endomicroscope's irregular fibre-based sampling pattern, and realistic noise patterns. We also took advantage of video sequences to exploit a sequence of images for self-supervised zero-shot image quality improvement. We run ablation studies to assess our contribution in regards to the downscaling kernel and noise simulation. We validate our methodology on both synthetic and original data. Synthetic experiments were assessed with reference-based IQA, while our results for original images were evaluated in a user study conducted with both expert and non-expert observers. The results demonstrated superior performance in image quality of ZSSR reconstructions in comparison to the baseline method. The ZSSR is also competitive when compared to supervised single-image SR, especially being the preferred reconstruction technique by experts.

UR - http://www.scopus.com/inward/record.url?scp=85103303786&partnerID=8YFLogxK

U2 - 10.1109/TMI.2021.3067512

DO - 10.1109/TMI.2021.3067512

M3 - Article

SN - 0278-0062

VL - 40

SP - 1863

EP - 1874

JO - IEEE Transactions on Medical Imaging

JF - IEEE Transactions on Medical Imaging

IS - 7

M1 - 9381880

ER -

Zero-shot super-resolution with a physically-motivated downsampling kernel for endomicroscopy.

Abstract

Access to Document

Other files and links

Fingerprint

Cite this