Session 1: AI in Medical Image Analysis: Rethinking Validation and Performance Reporting Practices

COURSE INFORMATION

This series is designed for all those interested in radiology research, with a particular focus on young radiologists eager to engage in scientific discovery. Whether exploring statistical best practices, navigating a research career, working with synthetic data, or ensuring research integrity, these sessions provide valuable insights and practical skills. Through expert-led discussions and real-world applications, participants will gain the knowledge needed to contribute meaningfully to the future of radiology research.

The Metrics Reloaded initiative has highlighted critical flaws in how medical imaging AI performance is evaluated, emphasizing the need for more rigorous and transparent use of metrics. A recent analysis of MICCAI 2023 papers found that many reported improvements lack statistical support and often fall within uncertainty margins given dataset size, meaning that some claims of progress may be false. This course aims to underscore the urgency of adopting larger benchmarks, performance uncertainty reporting, and stricter evaluation standards to ensure credible and clinically meaningful advances.

LEARNING OBJECTIVES

• To recognise common methodological and statistical pitfalls in the evaluation of AI systems for medical imaging
• To interpret performance metrics within the context of uncertainty
• To critically appraise AI outperformance claims and identify when reported advances may fall within uncertainty margins or lack sufficient statistical evidence.
• To get familiarised with best practices for transparent and reliable validation in medical imaging AI.

PROGRAMME ONLINE

Thursday, October 30, 2025