MUTUAL: Towards Holistic Sensing and Inference in the Operating Room

TY - CHAP

T1 - MUTUAL: Towards Holistic Sensing and Inference in the Operating Room

AU - Quarez, Julien

AU - Li, Yang

AU - Irzan, Hassna

AU - Elliot, Matthew

AU - MacCormac, Oscar

AU - Knight, James

AU - Huber, Martin

AU - Mahmoodi, Toktam

AU - Dasgupta, Prokar

AU - Ourselin, Sebastien

AU - Raison, Nicholas

AU - Shapey, Jonathan

AU - Granados Martinez, Alejandro

PY - 2024/7/17

Y1 - 2024/7/17

N2 - Embodied AI (E-AI) in the form of intelligent surgical robotics and other agents is calling for data platforms to facilitate its development and deployment. In this work, we present a cross-platform multimodal data recording and streaming software, MUTUAL, successfully deployed on two clinical studies, along with its ROS 2 distributed adaptation, MUTUAL-ROS 2. We describe and compare the two implementations of MUTUAL through their recording performance under different settings. MUTUAL offers robust recording performance at target configurations for multiple modalities, including video, audio, and live expert commentary. While this recording performance is not matched by MUTUALROS2, we demonstrate its advantages related to real-time streaming capabilities for AI inference and more horizontal scalability, key aspects for E-AI systems in the operating room. Our findings demonstrate that the baseline MUTUAL is well-suited for data curation and offline analysis, whereas MUTUAL-ROS 2, should match the recording reliability of the baseline system under a fully distributed manner where modalities are handled independently by edge computing devices. These insights are critical for advancing the integration of E-AI in surgical practice, ensuring that data infrastructure can support both robust recording andreal-time processing needs.

AB - Embodied AI (E-AI) in the form of intelligent surgical robotics and other agents is calling for data platforms to facilitate its development and deployment. In this work, we present a cross-platform multimodal data recording and streaming software, MUTUAL, successfully deployed on two clinical studies, along with its ROS 2 distributed adaptation, MUTUAL-ROS 2. We describe and compare the two implementations of MUTUAL through their recording performance under different settings. MUTUAL offers robust recording performance at target configurations for multiple modalities, including video, audio, and live expert commentary. While this recording performance is not matched by MUTUALROS2, we demonstrate its advantages related to real-time streaming capabilities for AI inference and more horizontal scalability, key aspects for E-AI systems in the operating room. Our findings demonstrate that the baseline MUTUAL is well-suited for data curation and offline analysis, whereas MUTUAL-ROS 2, should match the recording reliability of the baseline system under a fully distributed manner where modalities are handled independently by edge computing devices. These insights are critical for advancing the integration of E-AI in surgical practice, ensuring that data infrastructure can support both robust recording andreal-time processing needs.

M3 - Conference paper

BT - Medical Image Computing and Computer Assisted Interventions

ER -