RPS 1305 - Strategic deployment of AI

Author Block: A. S. Alic¹, D. Arce Grilo¹, M. Birhanu², E. Bron², V. Kalokyri³, T. Kussel⁴, K. Lang⁵, K. Majcen⁵, I. Blanquer¹; ¹Valencia/ES, ²Rotterdam/NL, ³Heraklion/GR, ⁴Heidelberg/DE, ⁵Graz/AT
Purpose: EUCAIM (https://cancerimage.eu/) is a pan-European federated infrastructure for cancer images, fueling AI innovations.
Methods or Background: This federated infrastructure is built upon a set of core services that comprise a public metadata catalogue, a federated search service following a common hyperontology, an access negotiation system, a coherent AAI and a distributed processing service. EUCAIM has recently released an early prototype with 40 image datasets from nine cancer types (breast, colon, lung, prostate, rectum, liver, diffuse intrinsic pontine glioma, neuroblastoma, glioblastoma) registered, related to the five projects in the AI4HI network (EUCANIMAGE, ProCAncer-I, INCISIVE, CHAIMELEON and PRIMAGE - https://future-ai.eu/), for a total of more than 200,000 image series from approximately 20,000 individuals. These collections follow a common metadata model defined in the EUCAIM project.
Results or Findings: This early prototype comprises a dashboard with guiding instructions, a public catalogue, a federated search engine and an access negotiation system in beta version.
Conclusion: This platform will permit users to discover, search, request, access and process medical imaging and associated clinical data in a flexible manner, supporting federated providers with different access levels and a future centralised repository. EUCAIM is based on cloud and container technologies, and it will be linked to intensive computing infrastructures such as EGI and supercomputing centres.
Limitations: The access negotiation service is currently in beta version and access requests will be forwarded to the providers.
Funding for this study: This project is co-funded by the European Union under grant agreement 101100633
Has your study been approved by an ethics committee? Not applicable
Ethics committee - additional information: The project acts as a broker for accessing data and relies on the ethical approvals of the providers and requesters.

Author Block: B-J. Boverhof¹, K. Redekop¹, D. Bos¹, M. P. A. Starmans¹, J. Birch², A. G. Rockall³, J. J. Visser¹; ¹Rotterdam/NL, ²Poole/UK, ³Godalming/UK
Purpose: The aim is to provide a comprehensive framework for value assessment of AI for radiology.
Methods or Background: This paper presents the RADAR framework, which has been adapted from Fryback and Thornbury's imaging efficacy framework and facilitates valuation of radiology artificial intelligence (AI) from conception to local implementation. Special attention is placed on local efficacy to underscore the importance of appraising an AI system in its local environment. The RADAR framework is illustrated through a myriad of study designs that help conduct adequate valuation.
Results or Findings: The RADAR approach constitutes a seven-levelled-hierarchy, providing radiologists, researchers, and decision-makers with a conceptual framework for comprehensive AI valuation in radiology. RADAR is dynamic, catering to varying valuation throughout the AI's developmental cycle. Technical and diagnostic efficacy (RADAR-1 and RADAR-2) is assessed before clinical implementation and can be addressed by in-silico clinical trials and cross-sectional studies. The next phases, encompassing diagnostic thinking to patient outcome efficacy (RADAR-3 to RADAR-5) necessitate clinical integration and can be addressed through randomised controlled trials and cohort studies. Societal efficacy (RADAR-6) delves into broader societal implications, assessed through health-economic evaluations. Concluding the hierarchy, the extent to which previous assessments generalise locally (RADAR-7) are gauged with budget impact analysis and multi-criteria decision analysis.
Conclusion: The RADAR framework stands as a comprehensive solution for valuing radiology. With its progressive and hierarchical approach, as well as an emphasis on local efficacy, RADAR provides a comprehensive framework to illustrate radiology AI's value conform to the notion of value-based radiology.
Limitations: No limitations were identified.
Funding for this study: No funding was received for this study.
Has your study been approved by an ethics committee? Not applicable
Ethics committee - additional information: No information provided by the submitter.

Author Block: T. B. T. Moeller, P. F. W. Sögner; Dillingen/DE
Purpose: The objective of this study was to answer the question of whether the use of AI is already having a quality-improving effect in routine teleradiological reporting throughout Germany and Austria.
Methods or Background: We performed a study of 2,707 native CCT scans from the CT departments of 140 hospitals in Germany and Austria between March and April 2022, which were analysed using AI with haemorrhage analysis. The results were compared with the findings of more than 70 teleradiologists who did not have the AI results at that time. Possible discrepant findings were evaluated by two radiologists with specific neuroradiological CCT experience.
Additionally, the in-house error statistics from 2021 and 2022-23 were reviewed.
Results or Findings: Of the 2,707 CCT examined by both radiologists and AI, 189 cases (approximately seven percent) were found to have intracranial haemorrhage described by both radiologists and AI. In 30 patients there was a discrepancy: the AI had seen a haemorrhage that had not been described by the radiologist. These cases were subsequently re-evaluated. Twelve (approximately 40%) of the 30 unclear examinations were classified as false positives by the AI, eight cases as questionable positives, and 10 cases as true positives. Thus, there were 199 cases with ICB in the studied patient group, of which > 5% were primarily missed by radiologists without AI support.
A review of in-house error statistics also revealed a significant decrease in reported false findings for intracranial haemorrhage (from 16 in 2021 to one between 08/31/2022 and 08/31/2023).
Conclusion: The positive effects of AI on the quality of radiological reporting postulated in several studies can also be confirmed in practice and especially in the teleradiological context.
Limitations: This assumption should be substantiated by further studies.
Funding for this study: No funding was received for this study.
Has your study been approved by an ethics committee? Not applicable
Ethics committee - additional information: No information provided by the submitter.

Author Block: H. Huhtanen, M. J. Nyman, A. Karlsson, J. Hirvonen; Turku/FI
Purpose: Advancements in AI-driven models for natural language processing has offered opportunities to automate many menial tasks that require understanding written text. Automating the protocolling of incoming MRI referrals could reduce interruptions in radiologists’ workflow. The purpose of this study is to test different AI models in assigning a suitable protocol and the need for contrast medium for emergency brain MRI referrals.
Methods or Background: For training and testing the models, we collected 1,563 and 390 Finnish emergency brain MRI referral texts, respectively. Data was labelled according to suitable imaging protocol and the need for contrast medium. We trained baseline machine learning (ML) models (three different algorithms) and newer deep learning (DL) models (BERT and GPT3) for classification. We also tested whether using less training data (50% of the training set) or using less data but upsampling it with augmentation affected model performance.
Results or Findings: In protocol and contrast medium prediction, GPT3 outperformed other models with accuracies of 84% and 91%, respectively. BERT models had accuracies of 78% and 89%, and the best ML models 77% and 86%, respectively. For DL models, using less training data affected performance negatively. Upsampling the data with augmentation boosted BERT’s accuracy in the protocol task but not in the contrast medium task. For ML models, neither dataset size nor augmentation seemed to affect performance.
Conclusion: Our results show that there is potential in using AI in automatic protocolling. Although GPT3 outperformed other algorithms, BERT and ML models also performed well. However, the DL models seem to have more potential to improve performance with increasing dataset size, than the ML models.
Limitations: The limitations of this study are the high imbalance between MRI protocol classes and using data from only one institute.
Funding for this study: Funding was provided by the Emil Aaltonen Foundation (grant number: 230049), and the Radiological Society of Finland.
Has your study been approved by an ethics committee? Not applicable
Ethics committee - additional information: Review by the ethics committee was waived due to the retrospective nature of this study in accordance with national legislation.