Development and validation of a tool for non-technical skills evaluation in robotic surgery-the ICARS system

Nicholas Raison, Thomas Wood, Oliver Brunckhorst, Takashige Abe, Talisa Ross, Ben Challacombe, Mohammed Shamim Khan, Giacomo Novara, Nicolo Buffi, Henk Van Der Poel, Craig McIlhenny, Prokar Dasgupta, Kamran Ahmed

Research output: Contribution to journalArticlepeer-review

50 Citations (Scopus)
402 Downloads (Pure)


BACKGROUND: Non-technical skills (NTS) are being increasingly recognised as vital for safe surgical practice. Numerous NTS rating systems have been developed to support effective training and assessment. Yet despite the additional challenges posed by robotic surgery, no NTS rating systems have been developed for this unique surgical environment. This study reports the development and validation of the first NTS behavioural rating system for robotic surgery.

METHODS: A comprehensive index of all relevant NTS behaviours in robotic surgery was developed through observation of robotic theatre and interviews with robotic surgeons. Using a Delphi methodology, a panel of 16 expert surgeons was consulted to identify behaviours important to NTS assessment. These behaviours were organised into an appropriate assessment template. Experts were consulted on the feasibility, applicability and educational impact of ICARS. An observational trial was used to validate ICARS. 73 novice, intermediate and expert robotic surgeons completed a urethrovesical anastomosis within a simulated operating room. NTS were tested using four scripted scenarios of increasing difficulty. Performances were video recorded. Robotic and NTS experts assessed the videos post hoc using ICARS and the standard behavioural rating system, NOn-Technical Skills for Surgeons (NOTSS).

RESULTS: 28 key non-technical behaviours were identified by the expert panel. The finalised behavioural rating system was organised into four principle domains and seven categories. Expert opinion strongly supported its implementation. ICARS was found to be equivalent to NOTSS on Bland-Altman analysis and accurately differentiated between novice, intermediate and expert participants, p = 0.01. Moderate agreement was found between raters, Krippendorff's alpha = 0.4. The internal structure of ICARS was shown to be consistent and reliable (median Cronbach alpha = 0.92, range 0.85-0.94).

CONCLUSION: ICARS is the first NTS behavioural rating system developed for robotic surgery. Initial validation has shown it to be an effective and reliable tool. Implementation of ICARS will supported structured training and assessment of NTS within the robotic surgical curriculum.

Original languageEnglish
Pages (from-to)5403-5410
Number of pages8
JournalSurgical endoscopy
Issue number12
Early online date20 Jun 2017
Publication statusPublished - Dec 2017


  • Clinical Competence/standards
  • Curriculum
  • Educational Measurement
  • Humans
  • Operating Rooms
  • Program Evaluation
  • Robotic Surgical Procedures/education
  • Surgeons/education
  • Video Recording


Dive into the research topics of 'Development and validation of a tool for non-technical skills evaluation in robotic surgery-the ICARS system'. Together they form a unique fingerprint.

Cite this