Language Models Meet Anomaly Detection for Better Interpretability and Generalizability

Jun Li; Su Hwan Kim; Philip Müller; Lina Felsner; Daniel Rueckert; Benedikt Wiestler; Julia A. Schnabel; Cosmin I. Bercea

doi:10.1007/978-3-031-84525-3_10

Language Models Meet Anomaly Detection for Better Interpretability and Generalizability

Jun Li, Su Hwan Kim, Philip Müller, Lina Felsner, Daniel Rueckert, Benedikt Wiestler, Julia A. Schnabel^*, Cosmin I. Bercea^*

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

Abstract

This research explores the integration of language models and unsupervised anomaly detection in medical imaging, addressing two key questions: (1) Can language models enhance the interpretability of anomaly detection maps? and (2) Can anomaly maps improve the generalizability of language models in open-set anomaly detection tasks? To investigate these questions, we introduce a new dataset for multi-image visual question-answering on brain magnetic resonance images encompassing multiple conditions. We propose KQ-Former (Knowledge Querying Transformer), which is designed to optimally align visual and textual information in limited-sample contexts. Our model achieves a 60.81% accuracy on closed questions, covering disease classification and severity across 15 different classes. For open questions, KQ-Former demonstrates a 70% improvement over the baseline with a BLEU-4 score of 0.41, and achieves the highest entailment ratios (up to 71.9%) and lowest contradiction ratios (down to 10.0%) among various natural language inference models. Furthermore, integrating anomaly maps results in an 18% accuracy increase in detecting open-set anomalies, thereby enhancing the language model’s generalizability to previously unseen medical conditions. The code and dataset are available at: https://github.com/compai-lab/miccai-2024-junli?tab=readme-ov-file.

Original language	English
Title of host publication	Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 Workshops - LDTM 2024, MMMI/ML4MHD 2024, ML-CDS 2024, Held in Conjunction with MICCAI 2024, Proceedings
Editors	Anna Schroder, Xiang Li, Tanveer Syeda-Mahmood, Neil P. Oxtoby, Alexandra Young, Alessa Hering, Tejas S. Mathai, Pritam Mukherjee, Sven Kuckertz, Tiantian He, Isaac Llorente-Saguer, Andreas Maier, Satyananda Kashyap, Hayit Greenspan, Anant Madabhushi
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	113-123
Number of pages	11
ISBN (Print)	9783031845246
DOIs	https://doi.org/10.1007/978-3-031-84525-3_10
Publication status	Published - 2025
Event	Workshop on Longitudinal Disease Tracking and Modeling with Medical Images and Data, LDTM 2024, 5th International Workshop on Multiscale Multimodal Medical Imaging, MMMI 2024, 1st Workshop on Machine Learning for Multimodal/-sensor Healthcare Data, ML4MHD2024 and Workshop on Multimodal Learning and Fusion Across Scales for Clinical Decision Support, ML-CDS 2024 held in conjunction with the 27th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2024 - Marrakesh, Morocco Duration: 6 Oct 2024 → 10 Oct 2024

Publication series

Name	Lecture Notes in Computer Science
Volume	15401 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	Workshop on Longitudinal Disease Tracking and Modeling with Medical Images and Data, LDTM 2024, 5th International Workshop on Multiscale Multimodal Medical Imaging, MMMI 2024, 1st Workshop on Machine Learning for Multimodal/-sensor Healthcare Data, ML4MHD2024 and Workshop on Multimodal Learning and Fusion Across Scales for Clinical Decision Support, ML-CDS 2024 held in conjunction with the 27th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2024
Country/Territory	Morocco
City	Marrakesh
Period	6/10/2024 → 10/10/2024

Keywords

Multimodal Learning
Vision-Language Models
VQA

Access to Document

10.1007/978-3-031-84525-3_10

Cite this

Li, J., Kim, S. H., Müller, P., Felsner, L., Rueckert, D., Wiestler, B., Schnabel, J. A., & Bercea, C. I. (2025). Language Models Meet Anomaly Detection for Better Interpretability and Generalizability. In A. Schroder, X. Li, T. Syeda-Mahmood, N. P. Oxtoby, A. Young, A. Hering, T. S. Mathai, P. Mukherjee, S. Kuckertz, T. He, I. Llorente-Saguer, A. Maier, S. Kashyap, H. Greenspan, & A. Madabhushi (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 Workshops - LDTM 2024, MMMI/ML4MHD 2024, ML-CDS 2024, Held in Conjunction with MICCAI 2024, Proceedings (pp. 113-123). (Lecture Notes in Computer Science; Vol. 15401 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-84525-3_10

Li, Jun ; Kim, Su Hwan ; Müller, Philip et al. / Language Models Meet Anomaly Detection for Better Interpretability and Generalizability. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 Workshops - LDTM 2024, MMMI/ML4MHD 2024, ML-CDS 2024, Held in Conjunction with MICCAI 2024, Proceedings. editor / Anna Schroder ; Xiang Li ; Tanveer Syeda-Mahmood ; Neil P. Oxtoby ; Alexandra Young ; Alessa Hering ; Tejas S. Mathai ; Pritam Mukherjee ; Sven Kuckertz ; Tiantian He ; Isaac Llorente-Saguer ; Andreas Maier ; Satyananda Kashyap ; Hayit Greenspan ; Anant Madabhushi. Springer Science and Business Media Deutschland GmbH, 2025. pp. 113-123 (Lecture Notes in Computer Science).

@inbook{4f577ec64b824508bc9f601b687a647b,

title = "Language Models Meet Anomaly Detection for Better Interpretability and Generalizability",

abstract = "This research explores the integration of language models and unsupervised anomaly detection in medical imaging, addressing two key questions: (1) Can language models enhance the interpretability of anomaly detection maps? and (2) Can anomaly maps improve the generalizability of language models in open-set anomaly detection tasks? To investigate these questions, we introduce a new dataset for multi-image visual question-answering on brain magnetic resonance images encompassing multiple conditions. We propose KQ-Former (Knowledge Querying Transformer), which is designed to optimally align visual and textual information in limited-sample contexts. Our model achieves a 60.81% accuracy on closed questions, covering disease classification and severity across 15 different classes. For open questions, KQ-Former demonstrates a 70% improvement over the baseline with a BLEU-4 score of 0.41, and achieves the highest entailment ratios (up to 71.9%) and lowest contradiction ratios (down to 10.0%) among various natural language inference models. Furthermore, integrating anomaly maps results in an 18% accuracy increase in detecting open-set anomalies, thereby enhancing the language model{\textquoteright}s generalizability to previously unseen medical conditions. The code and dataset are available at: https://github.com/compai-lab/miccai-2024-junli?tab=readme-ov-file.",

keywords = "Multimodal Learning, Vision-Language Models, VQA",

author = "Jun Li and Kim, {Su Hwan} and Philip M{\"u}ller and Lina Felsner and Daniel Rueckert and Benedikt Wiestler and Schnabel, {Julia A.} and Bercea, {Cosmin I.}",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.; Workshop on Longitudinal Disease Tracking and Modeling with Medical Images and Data, LDTM 2024, 5th International Workshop on Multiscale Multimodal Medical Imaging, MMMI 2024, 1st Workshop on Machine Learning for Multimodal/-sensor Healthcare Data, ML4MHD2024 and Workshop on Multimodal Learning and Fusion Across Scales for Clinical Decision Support, ML-CDS 2024 held in conjunction with the 27th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2024 ; Conference date: 06-10-2024 Through 10-10-2024",

year = "2025",

doi = "10.1007/978-3-031-84525-3_10",

language = "English",

isbn = "9783031845246",

series = "Lecture Notes in Computer Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "113--123",

editor = "Anna Schroder and Xiang Li and Tanveer Syeda-Mahmood and Oxtoby, {Neil P.} and Alexandra Young and Alessa Hering and Mathai, {Tejas S.} and Pritam Mukherjee and Sven Kuckertz and Tiantian He and Isaac Llorente-Saguer and Andreas Maier and Satyananda Kashyap and Hayit Greenspan and Anant Madabhushi",

booktitle = "Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 Workshops - LDTM 2024, MMMI/ML4MHD 2024, ML-CDS 2024, Held in Conjunction with MICCAI 2024, Proceedings",

address = "Germany",

}

Li, J, Kim, SH, Müller, P, Felsner, L, Rueckert, D, Wiestler, B, Schnabel, JA & Bercea, CI 2025, Language Models Meet Anomaly Detection for Better Interpretability and Generalizability. in A Schroder, X Li, T Syeda-Mahmood, NP Oxtoby, A Young, A Hering, TS Mathai, P Mukherjee, S Kuckertz, T He, I Llorente-Saguer, A Maier, S Kashyap, H Greenspan & A Madabhushi (eds), Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 Workshops - LDTM 2024, MMMI/ML4MHD 2024, ML-CDS 2024, Held in Conjunction with MICCAI 2024, Proceedings. Lecture Notes in Computer Science, vol. 15401 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 113-123, Workshop on Longitudinal Disease Tracking and Modeling with Medical Images and Data, LDTM 2024, 5th International Workshop on Multiscale Multimodal Medical Imaging, MMMI 2024, 1st Workshop on Machine Learning for Multimodal/-sensor Healthcare Data, ML4MHD2024 and Workshop on Multimodal Learning and Fusion Across Scales for Clinical Decision Support, ML-CDS 2024 held in conjunction with the 27th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2024, Marrakesh, Morocco, 6/10/2024. https://doi.org/10.1007/978-3-031-84525-3_10

Language Models Meet Anomaly Detection for Better Interpretability and Generalizability. / Li, Jun; Kim, Su Hwan; Müller, Philip et al.
Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 Workshops - LDTM 2024, MMMI/ML4MHD 2024, ML-CDS 2024, Held in Conjunction with MICCAI 2024, Proceedings. ed. / Anna Schroder; Xiang Li; Tanveer Syeda-Mahmood; Neil P. Oxtoby; Alexandra Young; Alessa Hering; Tejas S. Mathai; Pritam Mukherjee; Sven Kuckertz; Tiantian He; Isaac Llorente-Saguer; Andreas Maier; Satyananda Kashyap; Hayit Greenspan; Anant Madabhushi. Springer Science and Business Media Deutschland GmbH, 2025. p. 113-123 (Lecture Notes in Computer Science; Vol. 15401 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

TY - CHAP

T1 - Language Models Meet Anomaly Detection for Better Interpretability and Generalizability

AU - Li, Jun

AU - Kim, Su Hwan

AU - Müller, Philip

AU - Felsner, Lina

AU - Rueckert, Daniel

AU - Wiestler, Benedikt

AU - Schnabel, Julia A.

AU - Bercea, Cosmin I.

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

PY - 2025

Y1 - 2025

N2 - This research explores the integration of language models and unsupervised anomaly detection in medical imaging, addressing two key questions: (1) Can language models enhance the interpretability of anomaly detection maps? and (2) Can anomaly maps improve the generalizability of language models in open-set anomaly detection tasks? To investigate these questions, we introduce a new dataset for multi-image visual question-answering on brain magnetic resonance images encompassing multiple conditions. We propose KQ-Former (Knowledge Querying Transformer), which is designed to optimally align visual and textual information in limited-sample contexts. Our model achieves a 60.81% accuracy on closed questions, covering disease classification and severity across 15 different classes. For open questions, KQ-Former demonstrates a 70% improvement over the baseline with a BLEU-4 score of 0.41, and achieves the highest entailment ratios (up to 71.9%) and lowest contradiction ratios (down to 10.0%) among various natural language inference models. Furthermore, integrating anomaly maps results in an 18% accuracy increase in detecting open-set anomalies, thereby enhancing the language model’s generalizability to previously unseen medical conditions. The code and dataset are available at: https://github.com/compai-lab/miccai-2024-junli?tab=readme-ov-file.

AB - This research explores the integration of language models and unsupervised anomaly detection in medical imaging, addressing two key questions: (1) Can language models enhance the interpretability of anomaly detection maps? and (2) Can anomaly maps improve the generalizability of language models in open-set anomaly detection tasks? To investigate these questions, we introduce a new dataset for multi-image visual question-answering on brain magnetic resonance images encompassing multiple conditions. We propose KQ-Former (Knowledge Querying Transformer), which is designed to optimally align visual and textual information in limited-sample contexts. Our model achieves a 60.81% accuracy on closed questions, covering disease classification and severity across 15 different classes. For open questions, KQ-Former demonstrates a 70% improvement over the baseline with a BLEU-4 score of 0.41, and achieves the highest entailment ratios (up to 71.9%) and lowest contradiction ratios (down to 10.0%) among various natural language inference models. Furthermore, integrating anomaly maps results in an 18% accuracy increase in detecting open-set anomalies, thereby enhancing the language model’s generalizability to previously unseen medical conditions. The code and dataset are available at: https://github.com/compai-lab/miccai-2024-junli?tab=readme-ov-file.

KW - Multimodal Learning

KW - Vision-Language Models

KW - VQA

UR - http://www.scopus.com/inward/record.url?scp=105003862554&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-84525-3_10

DO - 10.1007/978-3-031-84525-3_10

M3 - Conference paper

AN - SCOPUS:105003862554

SN - 9783031845246

T3 - Lecture Notes in Computer Science

SP - 113

EP - 123

BT - Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 Workshops - LDTM 2024, MMMI/ML4MHD 2024, ML-CDS 2024, Held in Conjunction with MICCAI 2024, Proceedings

A2 - Schroder, Anna

A2 - Li, Xiang

A2 - Syeda-Mahmood, Tanveer

A2 - Oxtoby, Neil P.

A2 - Young, Alexandra

A2 - Hering, Alessa

A2 - Mathai, Tejas S.

A2 - Mukherjee, Pritam

A2 - Kuckertz, Sven

A2 - He, Tiantian

A2 - Llorente-Saguer, Isaac

A2 - Maier, Andreas

A2 - Kashyap, Satyananda

A2 - Greenspan, Hayit

A2 - Madabhushi, Anant

PB - Springer Science and Business Media Deutschland GmbH

T2 - Workshop on Longitudinal Disease Tracking and Modeling with Medical Images and Data, LDTM 2024, 5th International Workshop on Multiscale Multimodal Medical Imaging, MMMI 2024, 1st Workshop on Machine Learning for Multimodal/-sensor Healthcare Data, ML4MHD2024 and Workshop on Multimodal Learning and Fusion Across Scales for Clinical Decision Support, ML-CDS 2024 held in conjunction with the 27th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2024

Y2 - 6 October 2024 through 10 October 2024

ER -

Li J, Kim SH, Müller P, Felsner L, Rueckert D, Wiestler B et al. Language Models Meet Anomaly Detection for Better Interpretability and Generalizability. In Schroder A, Li X, Syeda-Mahmood T, Oxtoby NP, Young A, Hering A, Mathai TS, Mukherjee P, Kuckertz S, He T, Llorente-Saguer I, Maier A, Kashyap S, Greenspan H, Madabhushi A, editors, Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 Workshops - LDTM 2024, MMMI/ML4MHD 2024, ML-CDS 2024, Held in Conjunction with MICCAI 2024, Proceedings. Springer Science and Business Media Deutschland GmbH. 2025. p. 113-123. (Lecture Notes in Computer Science). doi: 10.1007/978-3-031-84525-3_10