RPS 1005b - Artificial intelligence and machine learning in reporting and workflow

Purpose:

To automatically extract clinical and epidemiological information from past narrative radiological reports on low-dose computed tomography (CT) for suspected urolithiasis using natural language processing (NLP) and to correlate findings with clinical information.

Methods and materials:

Narrative reports of low dose CT-examinations of the retroperitoneum from 04/2016 to 07/2018 (n=1714) were analysed using NLP. Reports were automatically structured based on RadLex® concepts. Manual feedback was used to test and train the NLP engine to reach adequate test accuracy (F1-Score ≥0.80). Chi-square test, phi-coefficient, and logistic regression were performed to determine the effects of clinical information on the positive hit rate of urolithiasis.

Results:

Urolithiasis was affirmed in 72% of the reports. In 38% of the reports, at least one stone was described in the kidneys and in 45% in the ureter. In 22% of the reports, patients suffered from combined nephrolithiasis and ureterolithiasis.

Affirmed clinical information such as previous stone history and obstructive uropathy could be found significantly more frequently in reports with affirmed urolithiasis (p=.001). The combination of obstructive uropathy and loin pain showed the highest rate for positive urolithiasis with an odds ratio of 1.16.

Conclusion:

Generating data from past radiological reports allows for the calculating of positive hit rates for pathologies, which can be used for epidemiological studies or to evaluate the appropriateness of CT-examinations. The evaluation of our data indicates that the occurrence of clinical information such as stone history and obstructive uropathy, or a combination of obstructive uropathy and loin pain, was higher in patients with affirmed urolithiasis. NLP can collect and monitor the data automatically in high quality.

Limitations:

Clinical information may not be revealed correctly or entirely by the referring physicians which could lead to limitations.

Ethics committee approval

n/a

Funding:

No funding was received for this work.

Purpose:

We propose an artificial intelligence-based method for the comprehension of textual radiological reports. Thousands of reports are created annually which represent textual information that is often expressed in the native language of the radiologist. Comparison and later examination of their content are rather cumbersome. To our knowledge, we are the first to introduce an intelligent comprehension and detailed visualisation technique that supports Hungarian language reports.

Methods and materials:

Our method relies on deep learning and various artificial intelligence procedures. The classification builds upon data from almost 500 anonymised reports manually annotated by two radiologists with 0.79 Cohen's kappa agreement. The classification process detects anatomic locations and pathologies with quantification, which are further specified with natural language processing techniques. Several linguistic characteristics are also addressed and negative statements and non-pathological disorders are separated.

Results:

Our AI classification model achieves a three-class classification with an F-score of 90-96%. The recognised disorders are displayed in a tree structure and also highlighted in the text itself. Furthermore, they are also visualised on a schematic illustration indicating the anatomical position of the disorders. This detailed view is obtained automatically by AI and NLP methods and is not yet available in existing RIS applications.

Conclusion:

We introduce an AI-based method for the automatic comprehension and visualisation of radiological reports, working currently with lumbar spine MRI reports written in Hungarian. The results can be used to filter the core of the report content and for quality assurance.
Our process can be extended to other languages and to any field where large amounts of text are created routinely and would benefit from automatic processing.

Limitations:

Our method currently supports reports written in Hungarian and concerning the spinal region.

Ethics committee approval

n/a

Funding:

No funding was received for this work.

Purpose:

To validate a sematic search tool by testing the search results for complex terms.

Methods and materials:

The tool consists of two pipelines: an offline indexing pipeline and a querying pipeline. The raw text from both reports and queries were first passed through a set of pre-processing steps; sentence tokenisation, spelling correction, negation detection, and word sense disambiguation. It was transformed into a concept plane followed by indexing or querying. During querying, additional concepts were added using a query expansion technique to include nearby related concepts. The validation was done on a set of 30 search queries, carefully curated by two radiologists. The reports that are related to the search queries were randomly selected with the help of keyword search and the text was re-read to determine its suitability to the queries. These reports formed the "related" group. Similarly, the reports that were not exactly satisfying the context of the search queries were categorised as the "not related" group. A set of 5 search queries and 250 reports were used for tuning the model initially. A total of 500 reports of the 10 search queries formed the corpus of the test set. The search results for each test query were evaluated and appropriate statistical analysis was performed.

Results:

The average precision and recall rates on 10 unseen queries on a small corpus for respective queries containing related and unrelated reports were 0.54 and 0.42. On a larger corpus containing 60 K reports, the average precision for these 15 queries was 0.6.

Conclusion:

We describe a method to clinically validate a sematic search tool with high precision.

Limitations:

A small corpus of test reports.

Ethics committee approval

n/a

Funding:

No funding was received for this work.

Purpose:

Occupational processes can only run smoothly if all participants are getting relevant and essential information accurately, clearly, and without time lag. Current table-style process information from a single system (RIS) does not fulfil those criteria and presumably leads to inefficiencies.

Methods and materials:

Patient-, process-, and time-information was obtained from the RIS relational database (GE, Oracle) and from the enterprise resource planning program (SAP) by ETL-programs. Time-stamps were directly extracted from DICOM images in a modality-independent manner. Process progression was calculated from those time points. A machine-learning tool (Prophet) was implemented to forecast daily patient numbers and increase the project ability. Data was visualised in a browser via an open-source client-server application or via a business intelligence software (Qliksense).

Results:

RadCount prepares key numbers like throughput times, equipment occupancy, and patient lists in a visual and self-explaining manner. Predictive analytics technology was used to forecast the expected workload. Data was presented live and team-specifically, and thus facilitated the planning of radiological examination and diagnostic processes. Moreover, by combining data from different sources, typical interface problems were being alerted and hence could be prevented.

Conclusion:

Radcount eliminated the problem of poor information transfer about VRE- and MRSA-infections at the interfaces. It also provided the data needed to devise an intelligent MR scheduling based on overbooking to manage no-shows. Its predictive functionality allows for resource management. Our solution is designed to be platform-independent and may be applicable to other platforms or high-throughput clinical processes. Thus, the tool proves oneself to enhance efficiency and safety.

Limitations:

Radcount is expandable by further resolving the time sequence of the radiology processes.

Ethics committee approval

IRB exemption was obtained.

Funding:

No funding was received for this work.

Purpose:

To assess a software prototype performance in measuring concordance between the indication and conclusion sections in standardised structured reports (SSRs) by comparing the results of automated and human evaluation.

Methods and materials:

200 randomly chosen French-language SSRs with "indication", "description" and "conclusion" sections were analysed. Indication and conclusion concordance regarding anatomy and disease was assessed by two expert radiologists and by a prototype software using falling rule-lists aided by MeSH (medical subject headings) terminology.

The software measured semantic similarity between any two MeSH codes with MeSHSim and an R-package with ”-nodeSim” functionality. L’Extracteur de concepts multi-terminologique (ECMT) combining a rule-based and an NLP (natural language processing)-based approach was used to extract health-related concepts from French-language texts using French-language terminologies in HeTOP (health terminology/ontology portal), housing 70 health terminologies in 32 languages including MeSH and radiology lexicon (RadLex).

Results:

Upon total concordance assessment between the indication and conclusion, the prototype reported 45% concordance for the anatomy information, while manual evaluation reported a 61% concordance. For disease information, the prototype reported 80% concordance versus 88% with the peer review. Based on ground truth, the algorithm accuracy for measuring anatomy concordance was 84% and for disease 92%. It took the experienced radiologists about 1.5 hours to conduct manual quality checks, but the prototype did it in a matter of seconds.

Conclusion:

The automated prototype shows good performance compared to expert radiologists for the assessment of indication and conclusion concordance in SSRs. It also reduces the time and cost required for quality control.

Limitations:

The prototype success is subject to the ability of ECMT to accurately extract the concepts from free texts and also upon availability of a particular concept in the MeSH vocabulary.

Ethics committee approval

n/a

Funding:

No funding was received for this work.

Purpose:

To evaluate a high-sensitivity deep learning algorithm for normal/abnormal chest x-ray (CXR) classification by deploying it in a real clinical setting.

Methods and materials:

A commercially available deep learning algorithm (QXR, Qure.ai, India) was integrated into the clinical workflow for a period of 3 months at an outpatient imaging facility. The algorithm, deployed on-premise, was integrated with PACS and RIS such that it automatically analysed all adult CXRs and reports for those which were determined to be “normal” were automatically populated in the RIS using HL7 messaging. Radiologists reviewed the CXRs as part of their regular workflow and ‘accepted’ or changed the pre-populated reports. Changes in reports were divided into ‘clinically insignificant’ and ‘clinically significant’ following which those CXRs with clinically significant changes were reviewed by a specialist chest radiologist with 8 years’ experience.

Results:

A total of 1,970 adult CXRs were analysed by AI, out of which 388 (19.69%) were identified to be normal. 361/388 (93.04%) of these were accepted by radiologists and in 14/388 (3.60%) clinically less significant changes (e.g. increased broncho-vascular markings) were made in reports. Upon review of the balance 13/388 (3.35%) CXRs, it was found that 12 had truly clinically significant missed findings by AI, including 3 with opacities, 3 with lymphadenopathy, 3 with blunted CP angle, 2 with nodules, and 1 with consolidation.

Conclusion:

This study shows that there is a great potential to automate the identification of normal CXRs to a great degree, with very high sensitivity.

Limitations:

The evaluation is limited to the normal/abnormal classification of chest x-ray.

Ethics committee approval

Institutional ethics committee approval was obtained.

Funding:

No funding was received for this work.