Artificial Intelligence

Applications for
Wor k f l o w, Proc e s s
O p t i m i z a t i o n a n d P re d i c t i v e
Analytics
Laurent Letourneau-Guillon, MD, MSca,b,*, David Camiranda,
Francois Guilbert, MDa,b, Reza Forghani, MD, PhDc,d,e,f,g,h

KEYWORDS
 Artificial intelligence  Machine learning  Workflow  Decision support  Smart device  Radiology
 Medical imaging  Operational efficiency

KEY POINTS
 In addition to the use of artificial intelligence (AI) for medical image analysis for diagnostic classifi-
cation and interpretation tasks, there are many additional important applications as it pertains to
workflow optimization and support for interpretation tasks, quality and safety, and operational
efficiency.
 Successful implementation of AI workflow applications are contingent on a robust and integrated
informatics infrastructure.
 Process and workflow optimization applications of AI in health care may be easier to achieve in the
near term both from a technical standpoint and from the perspective of other barriers, such as reg-
ulatory barriers and user acceptance and adoption.

INTRODUCTION radiology, advances in machine learning ap-

proaches, especially the development of deep
Applications of artificial intelligence (AI) in health learning, coupled with tremendous increases in
care, especially machine learning, have garnered computational processing power, have
significant attention in the last decade. In

Funding: R. Forghani is a clinical research scholar (chercheur-boursier clinicien) supported by the Fonds de
recherche en santé du Québec (FRQS) and has an operating grant jointly funded by the FRQS and the Fonda-
tion de l’Association des radiologistes du Québec (FARQ).
a
Department of Radiology, Centre Hospitalier de l’Université de Montréal (CHUM), 1051, rue Sanguinet, Mon-
tréal, Quebec H2X 0C1, Canada; b Centre de Recherche du CHUM (CRCHUM), 900 St Denis St, Montréal,
Quebec H2X 0A9, Canada; c Augmented Intelligence & Precision Health Laboratory (AIPHL), Department of
Radiology & Research Institute of the McGill University Health Centre, 5252 Boulevard de Maisonneuve
Ouest, Montréal, Quebec H4A 3S5, Canada; d Department of Radiology, McGill University, 1650 Cedar Avenue,
neuroimaging.theclinics.com

Montréal, Quebec H3G 1A4, Canada; e Segal Cancer Centre, Lady Davis Institute for Medical Research, Jewish
General Hospital, 3755 Cote Ste-Catherine Road, Montréal, Quebec H3T 1E2, Canada; f Gerald Bronfman
Department of Oncology, McGill University, Suite 720, 5100 Maisonneuve Boulevard West, Montréal, Quebec
H4A3T2, Canada; g Department of Otolaryngology - Head and Neck Surgery, Royal Victoria Hospital, McGill
University Health Centre, 1001 boul. Decarie Boulevard, Montréal, Quebec H3A 3J1, Canada; h 4intelligent
Inc., Cote St-Luc, Quebec H3X 4A6, Canada
* Corresponding author. Department of Radiology, Centre Hospitalier de l’Université de Montréal (CHUM),
1051, rue Sanguinet, Montréal, Quebec H2X 0C1, Canada.
E-mail address: laurent.letourneau-guillon.1@umontreal.ca

Neuroimag Clin N Am 30 (2020) e1–e15

https://doi.org/10.1016/j.nic.2020.08.008
1052-5149/20/Ó 2020 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC-
ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
e2 Letourneau-Guillon et al

revolutionized computerized medical image anal- vocabulary of the dataset will be represented by
ysis.1–4 There is great potential in the application a single unique binary vector. From this, it is
of machine learning for image analysis diagnostic possible to represent an entire document using a
decision support tasks and precision medicine.5,6 bag-of-words vectoral approach (Fig. 2), which
Many are discussed in detail in multiple accompa- in turn allows the creation of a term frequency–
nying articles in this issue. However, the potential inverse document frequency (TF-IDF) matrix.8–12
applications of AI in radiology go beyond image TF-IDF matrices are a popular approach to text
analysis tasks such as lesion classification or seg- representation because they emphasize the less
mentation alone. AI has the potential to enhance all common words in the corpus that may carry
levels of the radiology workflow and practice, more predictive value.10 However, there are
including workflow optimization and support for many drawbacks to using this form of vector rep-
interpretation tasks, quality and safety, and opera- resentation, including loss of word context (ie,
tional efficiency (Box 1, Fig. 1). This article reviews similar medical terms have independent binary
these important potential applications of infor- representations) and creation of very sparse vec-
matics and AI related to process improvement tors.7 This high vector space can become prob-
and operations in the radiology department. This lematic and reduce classification optimization by
article focuses on the potential additive value of overfitting the dataset.12 More recent techniques
AI, but before discussing the different applica- incorporating deep learning implement word
tions, it is important to keep the following funda- embedding, allowing word context preservation
mental principles in mind. For the different and dimension reduction; common models used
applications, any successful implementation of AI for this purpose include Word2vec and Glove.7,13
in the radiology department or the broader health Pretrained models are available but training a
care system is contingent on a robust informatics personalized model on a corpus related to the
network. Furthermore, AI applications are a sup- medical domain field may be more advisable.
plement to other important informatics tools and Similar to other machine learning algorithms, an
software that do not constitute AI. Therefore, NLP classifier’s performance is greatly dependent
even though not explicitly mentioned or repeated on the quality of its dataset, an important feature
in this article, this important and fundamental prin- being its size. Spasic and Nenadic14 (2020)
ciple applies to all of the applications described. showed in their systematic review that most data-
sets used to train machine learning NLP algorithms
NATURAL LANGUAGE PROCESSING in the clinical health care setting are only hundreds
or thousands of documents in length. In the few
There has been much attention paid to machine studies pertaining to radiology protocol selection,
learning applications for image analysis but this dataset size roughly ranged from 6700 to 18,000
should not detract from the importance of natural documents.8–11,13 One reason for these small
language processing (NLP) for developing applica- sizes is the annotation bottleneck effect commonly
tions and tools that enhance radiology practice. afflicting supervised machine learning models.14
Because of the importance of NLP for many of the For example, for creating an automated protocol-
potential applications discussed in this article, a brief ing engine using supervised machine learning, se-
review is provided of NLP before proceeding to a dis- lection requires input data coupled with an output
cussion of specific AI applications for process and class from a predefined set of standard proto-
workflow optimization. A historical discussion of cols.8–11,13 Generating standard protocols from
the evolution of NLP is beyond the scope of this free-text ones and manually annotating them to
article but, for current practical purposes, NLP can the dataset can be strenuous and time consuming.
be considered as a specific subfield of AI that con- When standard protocols are not readily available
verts narrative text into structured data.7 NLP can in an institution’s electronic health record (EHR),
be used to process unstructured text data from req- semiautomatic techniques with manual verification
uisitions (ie, clinical and demographic information) are one possible approach used to convert free-
and render it interpretable to computers and algo- text protocols to a defined standard form.11,13
rithms. Once the raw text is transformed into struc- String searching algorithms such as regular ex-
tured data, a final step involves vectorization. pressions are some examples sought out to find
Because algorithms only understand numerical rep- and replace such protocols. Other approaches
resentation as their inputs, vectorization transforms proposed to facilitate protocol annotation yet to
words into vector form. be tested include active learning (Fig. 3) and unsu-
Traditional NLP techniques normally represent pervised learning methods (ie, K-clustering and hi-
every word as a 1-hot encoded one-dimensional erarchical clustering); however, the efficiency of
vector.7–11 That is, every word found in the these techniques depends on the algorithm’s
 Artificial Intelligence Applications e3

Box 1 capability to correctly classify and annotate the

Potential applications of machine learning in free-text protocol, which could result in long
radiology retraining time cycles if not optimal.14
Before imaging examinations or procedures
In addition, 2 important performance metrics are
commonly calculated in text classification: precision
Clinical decision supporta and recall.12 Precision is defined as the probability
Patient scheduling and screening that a given document (eg, a computed tomography
Protocoling [CT] requisition) classified under a given category (ie,
During imaging examinations or procedures a standard protocol) is correct.12 Precision grossly
correlates to positive predictive value. Recall is
Image acquisition
defined as the probability that, if a random document
Image optimization (eg, motion artifact, noise should be classified under a given category, this de-
reduction)
cision be correctly taken by the classifier.12
Image generation (eg, low-dose gadolinium,32 syn-
thetic MR imaging) tp tp
precision 5 recall 5
Supplementary magnetic resonance (MR) imaging tp1fp tp1fn
sequence suggestions based on findings
where tp 5 true positives, fp 5 false positives,
After imaging examinations or procedures (before fn 5 false negatives.
interpretation)
Two approaches from which different inferences
Automated radiation dose estimation can be formulated are used to calculate precision
Image postprocessing (reformations, segmenta- and recall. Macroaveraging initially calculates the
tions, quantifications) classifier’s precision and recall for each category,
Advanced image analysis, including radiomics and and the mean of these scores is then calculated.12
delta-radiomics extraction This approach highlights how well a classifier does
DICOM routing and workflow orchestration with regard to categories having smaller samples.
Worklist prioritization Analogously, weighted precision and recall ac-
Preliminary interpretation
count for category imbalance by assigning a
weight to every score according to the category’s
Pre-dictation
distribution in the dataset. Weighted recall is
After imaging examinations or procedures (during commonly referred as a classifier’s accuracy. A
interpretation) combined metric such as the F-measure is usually
Contextual report template recommendation compiled; the F-measure is a harmonic mean of
Assisted segmentations precision and recall and reaches its best values
Electronic medical record extraction and correla- as it approaches the value of 1.12 Moreover, other
tion with other relevant data for interpretation metrics, such as inter-radiologist discordance
Prognostic inference measurements, may provide additional insight on
potential sources of model error.10
Quality (including peer review) and safety
When developing NLP applications, it is impor-
After interpretation tant to understand the strengths and pitfalls in al-
Urgent finding notification gorithm development in order to develop robust
International Classification of Diseases (ICD) code and reliable applications. With this brief overview
extraction from report of NLP, this article next discusses different AI ap-
Billing plications, using machine learning and NLP, for
Compliance
process and workflow improvement in the radi-
ology department. More in-depth reviews of
Abuse and fraud detection (insurer or Medicare
level)
NLP, different machine learning approaches, and
a
approaches for the evaluation of machine learning
Clinical decision support for ordering of radiology
algorithms are provided elsewhere in this issue.
scans discussed here should not be confused with
the more generic use of the term in the radiomics
and machine learning literature discussing the use of PRE-IMAGE ACQUISITION WORKFLOW
image analysis with radiomics and AI for diagnostic
decision support, alluding to diagnostic information
ENHANCEMENT: SCAN ORDERING,
extracted from imaging studies16 for augmenting PROTOCOLING, AND SCHEDULING
the expert radiologist’s final interpretation. Scan Ordering and Clinical Decision Support
Tools
Adapted from Choy.4
There has been significant interest in imaging and
the use of clinical decision support software to
e4 Letourneau-Guillon et al

Fig. 1. There are many potential applications of AI, more specifically machine learning and natural language pro-
cessing, in the radiology department. In addition to important augmented diagnostic assistant tools for precision
medicine, many other important applications are process and workflow related. This figure provides an example
of some of these applications relevant to different processes and workflow in the radiology department. CDS,
clinical decision support; EMR, electronic medical record; 3D, three-dimensional.

optimize the process of radiology scan ordering imaging studies for enhancing or augmenting the
and reduce ordering of unnecessary scans over expert radiologist’s final interpretation.6,18 Prelimi-
the past 2 decades.15–17 For greater clarity, the nary studies suggesting that AI-based frameworks
clinical decision support for ordering of radiology may be able to perform robustly and approximate
scans discussed here should not be confused the human decision-making process in complex
with the more generic use of the term in the radio- and uncertain environments suggests that AI en-
mics and machine learning literature discussing gines supporting traditional clinical decision sup-
the use of image analysis with radiomics and AI port software have great potential for optimizing
for diagnostic decision support (see Box 1), scan ordering, reducing wasteful use or erroneous
alluding to diagnostic information extracted from scan ordering and protocoling, and improving
overall patient care.4,17,19 Automated retrieval of
clinical data from the EHR can even be envisioned,
improving the quality and efficiency of clinical de-
cision support software in the future.20

Patient Screening, Scan Protocoling, and

Scheduling
Scan protocoling and scheduling are discussed
together because, in the radiology department,
these are inevitably intertwined. Efficient and
streamlined scheduling of radiology examinations
poses unique administrative and operational chal-
lenges because the proper scheduling of scans is
contingent on specific medical information. This
situation can pose a challenge for administrative
personnel with no formal medical background.
The typical process where the input from an expert
Fig. 2. Example of text vectorization based on a bag- radiologist and/or radiology technologist is
of-words model. required in the chain of mainly administrative tasks
 Artificial Intelligence Applications e5

Fig. 3. Active learning tackles the annotation bottleneck phenomenon allowing much less annotation effort.14
This method requires a professional to manually label a sample of the free-text protocol dataset and then iter-
atively train a machine learning algorithm over it in order to predict the annotation of the remaining dataset.
Measures of uncertainty, probability, and/or density can be used to guide the algorithm’s confidence level;
when the algorithm is uncertain about its annotation for a given input, the expert can manually label the
free-text protocol. The newly labeled input is then reinserted and the algorithm retrained with the modified da-
taset. This process is done until it can annotate the standard protocol of the whole dataset with a high confi-
dence level. c is the predefined ratio of the database that will be used for manual labeling (ML). DB,
database containing free-text protocols.

creates undesirable inefficiencies, which can be involves using NLP techniques to extract and pro-
disruptive for the administrative personnel, radiol- cess pertinent clinical text data from the clinical
ogists, and patients in providing a streamlined and indication/history field and from prior imaging re-
efficient service. ports and build a training dataset. Subsequently
A well-structured informatics system with the annotating this training dataset with predefined
support of AI has great potential in streamlining standard protocols then allows for text classifica-
this process (see Box 1).21 With increasing sophis- tion using supervised learning algorithms. Com-
tication of NLP and machine learning applications, mon techniques used to process raw text in
there is the possibility of efficient safety screening studies of automated protocol selection include
of patients before scheduling a scan, identifying conversion to lowercase, removal of stop words
contraindications or creatinine when required for (ie, words with little predictive value such as
scheduling of a contrast-enhanced study. This “and,” “or,” “the,” “please”), removal of punctua-
screening has the potential concurrently improve tion and numerical values, tokenization, lemmati-
patient safety and operational efficiency. zation, and stemming.8–10
Scan protocoling is a time-consuming and re- Algorithms used in studies of imaging protocol
petitive task and can take up a considerable automation have included support vector ma-
portion of a radiologist’s time.22 Moreover, the chine, gradient boosting machine, random forest,
accumulation of requisitions to be protocoled and K-nearest neighbor.8–11 These studies have
can contribute to outpatient imaging wait times. shown promising results.8–10 These models can
Although computerized clinical decision support all be tagged as traditional NLP techniques or
systems using machine learning techniques are shallow learning models that require a process
increasingly popular, few have studied their appli- called feature selection (ie, input variable selec-
cability for protocoling. Constructing support sys- tion) before they can start training on the dataset.
tems for automated protocol selection initially Common features (or variables) that were used
e6 Letourneau-Guillon et al

as input to the learning models included clinical provided in the requisition. It is not practically
indication and history (processed via NLP tech- feasible to consult and evaluate in detail every
niques as stated earlier), patient demographics patient’s chart when performing protocoling
(age and sex), and ordering service.8–10,13 How- tasks, which introduces certain important defi-
ever, a greater role is now to be expected from ciencies and inaccuracies in the process.
emerging deep learning methods based on their Moreover, for follow-up studies, the optimal
performance for other NLP tasks.7 These timeline for follow-up relative to the prior exam-
methods include a broad range of classifiers, ination may not be evident unless specified by
ranging from convolutional neural networks the ordering physician. In the future, robust
(CNNs) to recurrent neural networks and long well-structured informatics systems combined
short-term memory. These new methods have with NLP are likely to overcome these chal-
the advantage of not requiring human involve- lenges based on the analysis of the patient’s
ment in feature selection; the computer automat- electronic medical record and even automated
ically emphasizes features that have more selection of the scan time and optimal scan
predictive impact while in training.7 Two studies type. These systems may alert the referring
of CNNs for musculoskeletal image protocoling, physician and/or radiology department when a
Trivedi and colleagues11 (2018) and Lee13 recommended follow-up was not scheduled,
(2018), have shown promising results for auto- provided yet another level of quality control
matic determination of intravenous contrast and safety measures in the system. Further-
administration and tumor versus routine protocol more, such systems may identify redundant
selection (accuracy of 80% and 94% respec- scans or poorly timed follow-up scans (eg, a
tively). More recently, Kalra and colleagues10 prematurely booked follow-up scan), providing
(2020) showed the effectiveness of a deep neural an opportunity for the referring physician and/
network for CT and magnetic resonance (MR) im- or radiologist to be alerted in order to rectify
aging protocol selection, reaching 95% accuracy the situation. These future applications of AI
for the assignment of 69% of their protocols are important potential and likely feasible as
where optimal confidence thresholds were met proper health informatics infrastructures are
and 92% accuracy for the remaining 31% by developed and optimized.
providing a top-3 protocol suggestion for clinical
review. ARTIFICIAL INTELLIGENCE AI APPLICATIONS
Successful implementation of a semiautomatic DURING IMAGE ACQUISITION: SMART
or automatic protocoling engine can significantly DEVICES
improve and enhance the administrative and
scheduling processes in the radiology depart- AI has great potential for improving the radiology
ment, and free up the radiologists for more scan acquisition process, including as assistive
important, higher-level activities. By automating technology for improving radiology technologists
this process and taking the radiologist out of and enabling increased scan quality and acquisi-
this mainly administrative loop, much more tion efficiency (see Box 1).4 As an example, in
streamlined scheduling is possible where most MR imaging there is potential for AI-supported
scans are automatically protocoled based on guidance and assistance in patient positioning,
the information provided in the requisition and/ or matching of sequences with prior studies
or the information in the EHR. In this scenario, when appropriate. AI-supported engines may
only a small percentage of ambiguous requests also identify outliers or cases where additional
or complex cases may be flagged for review sequences may be required based on a pa-
and protocoling by the radiologist. In robust and tient’s past scans. All of these steps can
properly set up systems, it may eventually even improve quality at the acquisition, providing
be possible to support live interactive booking important assistive support to the imaging tech-
by the patient in most cases, which would nologist. In CT, AI not only has the potential to
improve patient satisfaction in addition to opera- assist in patient positioning but also can opti-
tional efficiency. mize CT dose. As an example, rather than pick-
Beyond the simple but important operational ing categories such as small, medium, or large
impact of automation of most protocoling- for radiation dose modulation, AI has the poten-
scheduling tasks, in the future, AI applications tial to truly personalize radiation dose to a pa-
can be foreseen that could improve the pro- tient’s body habitus based on the scout view
cess beyond what is currently done. In most that has been obtained.23 In cases where there
radiology departments, the selection of a scan are metallic implants or other sources of artifact,
protocol is done based on limited information appropriate automated processes can be
 Artificial Intelligence Applications e7

preprogramed that would enable optimal acqui- and DICOM image routing (see Box 1). AI has the
sition and generation of additional reconstruc- potential to enhance existing informatic solutions
tions. For example, in the neck, detection of for end-to-end workflow, image management,
dental material on the scout view could auto- and analytics, including creation of intelligent
matically prompt the need for the acquisition of worklists harmonizing and optimizing case assign-
an angled gantry or open-mouth view (depend- ment and workflow within the department at 1 hos-
ing on a specific scanner and practice). Metal pital or across different hospital or clinic sites.
prostheses could also automatically trigger Thereafter, an example of the use of AI for
reconstruction using a metal artifact reduction improving workflow efficiency is automatic crea-
algorithm. This possibility has the potential to tion of hanging protocols.37,38 Although this may
standardize and enhance the quality of the ex- seem simplistic, having optimal hanging protocols
aminations at acquisition, performing this effi- with proper fetching and matching of prior studies
ciently and enabling both quality improvement is difficult to achieve based on DICOM metadata
and streamlining of operational processes. The alone. Using AI engines with additional image
scans could even be preprogramed to automat- analytical capabilities, this can be enhanced,
ically detect quality deficits prompting repeat resulting in overall improvement of productivity.
scans when appropriate (eg, extreme motion) Pertaining to image processing, the applications
in either MR imaging or CT with automated of AI are numerous, ranging from generating
application of necessary interventions. These various reconstructions to complex image pro-
developments hold great potential for AI in the cessing and analytical tasks, including feature
future in streamlining the scanning process. extraction and radiomics. A discussion of these
In addition to the possibilities discussed earlier, is beyond the scope of this article but can be found
there are potential applications of machine in accompanying articles in this issue or different
learning for image reconstruction in CT, MR imag- reviews on the topic.4–6,18
ing, or other modalities in order to improve image
quality and/or operational efficiency (see Box 1).
Radiology Worklist Prioritization:
In CT, other than the patient-tailored dose estima-
Prioritization of Critical Findings
tion discussed earlier, deep learning–based recon-
struction approaches can be used for noise At present, a widely used application of AI for
reduction and radiation dose optimization.24–27 In workflow is reading list prioritization (see Box 1).
MR imaging, deep learning similarly can be used In this scenario, images are analyzed and those
to enhance the image acquisition process by with potential critical findings are flagged and
improving general image quality, performing mo- pushed to the top of the list for more immediate
tion correction, and reducing sequence acquisi- reading. This application can improve turnarounds
tion time, resulting in improved operational for examinations with critical findings where time
efficiency.28–31 Most of these manipulations are could be of the essence for patient management.
performed on the raw data, but certain deep In health systems where there are large backlogs,
learning–based manipulations and improvements such engines also have the potential to flag and
can also be done in the image space after scan enable a more rapid read of examinations that
acquisition. There are also potentially more could be abnormal.
advanced applications of deep learning. These ap- Detection of urgent actionable findings on imag-
plications include real-time image analysis to ing study is important to prevent patient morbidity
recommend modifications to sequences or addi- and mortality as well as to mitigate medicolegal
tion of other sequences to a scan.4 They also risks. It is identified as an important goal by the
include applications for reducing contrast agent Joint Commission in their National Patient Safety
dose or generation of various synthetic Goal.39 The American College of Radiology rec-
images.32–36 ommends nonroutine communication of “findings
that suggest a need for immediate or urgent inter-
vention.”40 A secondary benefit of rapid detection
POST–IMAGE ACQUISITION WORKFLOW
of such findings is increased productivity gained
ENHANCEMENT: BEFORE AND DURING
from the reduction of interruptions related to in-
IMAGE INTERPRETATION
quiries about urgent reports.
DICOM Image Routing, Workflow
To accelerate the identification of such findings,
Orchestration, and Image Postprocessing
most radiology practices use a worklist prioritiza-
Another area where a properly set up informatic tion strategy. The most basic implementation is
system with AI can be of great help would be through a rule-based system, such as filters
more efficient and optimal workflow orchestration already available within picture archiving and
e8 Letourneau-Guillon et al

communication systems (PACS). These rules can and neck studies, such as pulmonary52 or thyroid
consider, for example, the patient’s location for in- nodules,53 can also be performed.
patients (eg, emergency room or intensive care
unit) or examination priority for outpatients. So-
Preliminary Radiology Report Generation
called stat creep can hamper the implementation
of such strategies over time.41,42 Alternatives One possible extension and more advanced appli-
include server-based enterprise collaboration cation of preliminary automated image analysis,
software 43 or EHR-driven solutions,41 which allow such as that used for worklist prioritization, could
increased granularity and efficiency compared be a more sophisticated diagnostic support
with PACS-based strategies. Several companies (such as the example provided earlier for inci-
offer worklist prioritization software with PACS dental findings). However, another application by
integration. A major drawback of these non–AI- linking to NLP is the generation of preliminary re-
based strategies is that they may not be able to ports (see Box 1). Interpretation of radiology ex-
properly prioritize scans with potential unexpected aminations requires high-level expertise and
critical findings. adaptation to the appropriate clinical scenario.4,54
Contrary to these prioritization strategies, ma- A recent study combining deep learning and
chine learning–driven worklist prioritization typi- bayesian networks, mimicking perceptual and
cally uses the information available within the cognitive tasks performed by radiologists, was
images. Among critical findings, intracranial hem- able to infer differential diagnoses approaching
orrhage is frequently cited as the most important the accuracy of neuroradiologists and surpassing
to detect rapidly.44,45 The 2019 RSNA AI challenge trainees’ performance.55 In this study, the addition
was aimed at detection and classification of acute of clinical features added significant diagnostic ac-
intracranial hemorrhage (https://www.kaggle. curacy to the model. This task was aimed at a
com/c/rsna-intracranial-hemorrhage-detection). particular entity (fluid-attenuated inversion recov-
Algorithms used for such tasks are typically image ery hyperintensities on brain imaging) and remains
classifiers and can detect critical findings such as limited in scope but shows the potential of such
hemorrhage,46 mass effect, midline shift,47,48 hy- technology to augment interpretation of imaging.
drocephalus, acute ischemic stroke, large vessel Automatic radiology report generation is
occlusion,49 or a combination of these condi- another complex task that must integrate accurate
tions44,50 before formal reading, allowing rapid analysis of medical images with appropriate natu-
notification of radiologists and worklist ral language description. This ability has been
prioritization. leveraged by advances in computer vision and lan-
Diagnostic performance of such implementa- guage translation.56,57 The first study specifically
tions are variable and depend notably on the aimed at radiology reports was described by
type of model and dataset used for training the al- Shin and colleagues in 2016.57 Earlier research
gorithm. Prevedello and colleagues44 built a deep on this topic was based on image captioning,
learning algorithm to detect hemorrhage, mass ef- which is a simpler task compared with the gener-
fect, and hydrocephalus with 90% sensitivity and ation of full radiology reports.56–58
85% specificity. Similar to human performance, a A typical approach to tackle this problem is to
separate algorithm dedicated to the identification combine a CNN for image detection and classifi-
of suspected acute infarct, which is generally cation with a recurrent neural network for text gen-
more difficult to detect than hemorrhages, had eration.57,58 In particular, recurrent neural network
lower accuracy, with 62% sensitivity and 96% encoder-decoder architectures with long short-
specificity.44 Chilamkurthy and colleagues50 used term memory typically used in language transla-
a much larger dataset to identify different types tion have been used to generate complete senten-
of intracranial hemorrhages, calvarial fracture, ces.57,58 Furthermore, generation of long
mass effect, and midline shift. Using high- sentences and paragraphs, such as in the findings
sensitivity operating points, the sensitivity for section of a radiology report, is a task that requires
these findings on different test datasets was be- hierarchical neural networks.58,59 So far, most
tween 90% and 95% and specificity between studies have focused on the interpretation of chest
60% and 88% depending on the finding. Non- radiographs using a publicly available dataset of
CNS or head and neck findings, such as the iden- limited size (the Indiana University Chest X-ray
tification of pneumothorax, could also be inte- Collection).60 Different approaches have been
grated in such critical finding detection models.51 used, often mimicking the radiologist’s perceptual
Finally, in addition to critical findings, the identifi- and cognitive tasks.56,58,59 Interpretability of such
cation of common incidental findings on head models can be enhanced by the use of attention
maps.56,59 In the model built by Yuan and
 Artificial Intelligence Applications e9

colleagues,59 activation maps attract expert atten- Extracting unstructured information from free
tion to findings of uncertain significance, allowing text has been applied with good accuracy to radi-
additional human input in the final interpretation, ology reports using ML techniques (see Box 1).
which appears especially important since the Historically, traditional NLP models have been
source of errors in the automated model appeared used for such tasks, but this requires significant
to be different than the one observed in humans. development and domain-specific feature engi-
Thus it is possible that such models allowing neering with generalizability issues.68 Description
both image detection and classification with of NLP techniques applied to radiology is beyond
descriptive text generation eventually augment the scope of this article and has been reviewed
the radiologist’s work, potentially improving pro- in an accompanying article in this issue and else-
ductivity and decreasing the error rate.61 The avail- where.69,70 Deep learning approaches, including
ability of larger training datasets will probably the use of neural networks (CNNs), can also be
allow increased performance of these automated used with high accuracy for the purpose of free-
radiology report generation models in the text annotation without the aforementioned draw-
future.58,59 Ultimately, such application would backs associated with traditional NLP
lighten the radiologists’ work, allowing more focus methods.68,71 Furthermore, similar to saliency or
on perceptual and interpretation tasks and less on class-activation masks used in image classifica-
report generation. Finally, automatic integration of tion tasks, visualization tools are available to in-
basic measurements made on the screen (or from crease the interpretability of such neural
an ultrasonography report) into the final radiology networks.68 Further improvements in model so-
report could be integrated during reporting. phistication for such tasks are expected66,68 and
would allow better labeling of image data from un-
structured reports, but up to a certain limit. Indeed,
Extracting Information from Radiology
in 1 study, even some statements were difficult to
Reports: Report Processing
classify by both the observers and algorithms,
Both structured and contextual reports, the latter highlighting a potential ceiling in future
allowing for adaptation of the report to the disease improvements.71
or clinical indication compared with a generic During radiology study interpretation, integra-
structured reports,62 are increasingly used. tion of prior examinations and reports is important,
Several advantages of structured reports have a process that is time consuming but must be per-
been reported, including completeness,63 homo- formed thoroughly to avoid missing important prior
geneity,64 as well as increased ability to perform findings. Summarizing such information from prior
data mining either for teaching files or research reports is important, and NLP has been applied for
purposes.64 This process is notably important this purpose.72 Other applications of information
and difficult in oncology follow-up, where the num- extraction from radiology reports include identifi-
ber of prior examinations can be significant. In ma- cation of the need for follow-up imaging73,74 and
chine learning studies, data labeling is often the target lesion measurements in oncologic follow-
limiting step as it requires expert annotation of ups.75 NLP models can also be used to extract
large imaging datasets, which can rapidly become important information from other medical sources,
prohibitive. The alternative is to use the informa- such as pathology or clinical records, during imag-
tion contained within radiology reports. In this re- ing interpretation.
gard, structured radiological reports could Radiology report translation will probably
potentially facilitate machine learning research in become increasingly important with globalization.
radiology by facilitating the pairing of labels to im- A recent report shows that, although promising,
aging data. Despite the aforementioned progres- the use of machine translation is not currently ac-
sive shift toward structured reporting in curate enough to be adopted in routine clinical
radiology, there are potential drawbacks, including practice.76 Translation of radiology reports based
the lack of widespread adoption.65 Consequently, on a structured reporting format may be a potential
unstructured radiology reports still represent an approach to improve the performance of such
important source of data for many applications applications.77
since this reporting format remains the one linked
to most prior imaging studies currently archived in
Quality and Safety
many centers. Such reports can be mined to
extract weak labels for the purpose of neural At this end of this spectrum, AI can play a role in
network training,66 which is especially important improving intradepartmental and interdepart-
given the need for large datasets to achieve mental as well as interdisciplinary communication
optimal performances with these models.67 for quality assurance (QA), enhanced patient
e10 Letourneau-Guillon et al

safety, and educational purposes. AI tools have infrastructure. However, AI engines can further
the potential to be used as semiautomated feed- enhance these processes.
back QA tools for technologists. AI can also be
used for radiologist QA in different ways, including
AI-supported peer review, which can be either OTHER ARTIFICIAL INTELLIGENCE
retrospective or prospective, the latter overlapping APPLICATIONS FOR DIAGNOSTIC AND
with computer-assisted diagnosis/diagnostic de- WORKFLOW ENHANCEMENT
cision support functionalities of AI.78 If sufficiently Predictive Analytics: Diagnostic Versus
robust and well integrated into the EHR, such tools Workflow Related
can also be used for interdisciplinary feedback Image-based prediction of treatment response or
and peer review based on, for example, a final prognosis is currently the focus of important
pathologic diagnosis, enabling a level of contin- research, especially in oncology, which includes
uous practice improvement and educational feed- radiomic approaches based on handcrafted fea-
back that would not be feasible without such tures or those using deep learning (Fig. 4).6
robust informatics systems in most health care Although not always characterized as such, these
settings. In a study, a tool allowing retrieval of models can be under the broader umbrella of pre-
pertinent follow-up information from EHRs using dictive analytics. For example, such models can
previously reported cases from PACS helped radi- predict genetic mutations,80 pseudoprogression
ologists learn from their prior interpretations.79 AI- versus true progression,81 and survival82 in pa-
supported tools can also be used to improve result tients with gliomas. Another application outside
communication, including critical result communi- of oncology is the prediction of hematoma expan-
cation. Similar to other applications that are dis- sion83,84 or identification of underlying vascular
cussed, successful implementation of these malformation in acute intra-cerebral hema-
solutions requires a well-designed informatics tomas.85 Even overall longevity can be inferred

Fig. 4. Overview of the radiomic workflow. The major steps in the radiomic workflow consist of lesion identifi-
cation and localization, segmentation, feature extraction, and prediction model construction. In the short term,
lesion detection and localization may be facilitated by directing the algorithm using a bounding-box approach by
the radiologist, or alternatively by the radiologist pointing at the lesion of interest using a cursor. However, in the
long term, lesion identification may also be performed automatically by the algorithm. Although much of the
focus in the literature has been on the diagnostic-related tasks in the radiomic workflow, any future widespread
implementation and adoption in the clinical setting for routine patient care is likely to require streamlined inte-
gration into the radiologists’ workflow. AI-assisted tools can also play an essential role in improving and over-
coming barriers related to workflow as it pertains to advanced diagnostic medical image analysis tools and
radiomics. Note that, regardless of the degree of automation, these steps would be performed under the super-
vision of an expert radiologist with the ability to make adjustment or modifications as needed. (Modified from
Forghani R. Precision Digital Oncology: Emerging Role of Radiomics-based Biomarkers and Artificial Intelligence
for Advanced Imaging and Characterization of Brain Tumors. Radiology: Imaging Cancer. 2020;2(4):e190047; with
permission.)
 Artificial Intelligence Applications e11

using radiomics extracted from medical imaging.86 optimal manpower needs. For example, at an aca-
A discussion of these potential AI-assisted or demic institution, a large percentage of neuroradi-
augmented diagnostic support tools is beyond ologists are likely to be off service during at least
the scope of this article. However, the application part of the period of some society meetings. Using
of predictive analytics is not limited to diagnostic predictive analytics, this information can be used
support alone (see Box 1). First, even as it pertains to preferentially schedule a greater percentage of
to diagnostic predictive tasks such as that in the nonneuroradiological, elective, nonurgent studies
radiomic workflow (Fig. 5), any future widespread during that period.
implementation and adoption in the clinical setting
for routine patient care is likely to require stream- Billing and Compliance
lined integration into the radiologists’ workflow. Accurate coding of examinations and proced-
AI-assisted tools can play an essential role in ures is important for appropriate billing. In
improving and overcoming barriers related to many jurisdictions, task is performed by identi-
workflow that otherwise are likely to hinder the fying appropriate International Classification of
adoption of such tools in the clinical setting. Diseases (ICD) codes. Extracting such codes
In addition, there are other nondiagnostic pre- from electronic medical records is important for
dictive analytical tasks that are important for the accurate payment to the hospital and doctors,
operations in a radiology department. One and avoiding errors that may lead to unfair bills
example relates to applications for improving ex- to patients, insurers, and the society. This task
amination scheduling. In 1 study, machine learning can be performed by ML models (see Box 1).
and predictive analytics were used to identify pa- Corpora of electronic medical records available
tients who were at high risk of not attending their to train such models, including specific radiology
appointments.87 Such applications can be impor- corpus, have been available for more than a
tant for alerting the team and implementing miti- decade.88 Similar to other fields, neural networks
gating strategies to ensure that patients obtain have been shown to improve the accuracy of
the required care, along with optimizing depart- such classification tasks.89 Further improvement
mental operations and efficiency. Another has been obtained using transfer learning using
example is the use of predictive analytics to ensure classification of medical subject heading

Fig. 5. A potential future radiology report incorporating radiomic features and machine learning for predictive
modeling. A preliminary draft could even be envisaged of the report being generated automatically based on
automated image analysis combined with NLP, which can then be modified by the expert radiologist as required.
Seamless integration and workflow enhancement will be important requirements for successful implementation
of such AI-assisted diagnostic tools. A robust, integrated informatics system powered by AI is also likely to play a
role in practical aspects related to the implementation of these advanced tools, in addition to diagnostic-related
functionalities. MGMT, O6-methylguanine-DNA methyltransferase. (Modified from Forghani R. Precision Digital
Oncology: Emerging Role of Radiomics-based Biomarkers and Artificial Intelligence for Advanced Imaging and
Characterization of Brain Tumors. Radiology: Imaging Cancer. 2020;2(4):e190047; with permission.)
e12 Letourneau-Guillon et al

(MeSH) from medical articles indexed by 3. Erickson BJ, Korfiatis P, Akkus Z, et al. Machine
PubMed as the source task.90 Learning for Medical Imaging. Radiographics
A significant amount of monetary and human 2017;37(2):505–15.
health care resources is lost yearly to abuse and 4. Choy G, Khalilzadeh O, Michalski M, et al. Current
fraud. With the advent of EHRs and computerized Applications and Future Impact of Machine
systems, implementation of machine learning Learning in Radiology. Radiology 2018;288(2):
technologies for the purpose of automated fraud 318–28.
detection becomes feasible.91 Using a combina- 5. Forghani R, Savadjiev P, Chatterjee A, et al. Radio-
tion of ML and statistical methods, data mining mics and Artificial Intelligence for Biomarker and
strategies now allow the identification of a small Prediction Model Development in Oncology. Com-
subset of claims for further verification. Supervised put Struct Biotechnol J 2019;17:995–1008.
and unsupervised ML techniques can be used for 6. Forghani R. Precision digital oncology: emerging
this purpose. One potential drawback of the role of radiomics-based biomarkers and artificial in-
former method is that supervised models need to telligence for advanced imaging and characteriza-
be updated for new fraud patterns developing tion of brain tumors. Radiology: Imaging Cancer
over time, whereas unsupervised methods may 2020;2(4):e190047. https://pubs.rsna.org/doi/10.
be more able to identify new types of fraud or 1148/rycan.2020190047.
abuse.92 Hybrid methods combining both super- 7. Sorin V, Barash Y, Konen E, et al. Deep Learning for
vised and unsupervised methods have been Natural Language Processing in Radiology-
used for this purpose.91 Fundamentals and a Systematic Review. J Am Coll
Radiol 2020;17(5):639–48.
8. Brown AD, Marotta TR. A Natural Language
SUMMARY AND FUTURE DIRECTIONS
Processing-based Model to Automate MRI Brain
Although diagnostic support and precision medi- Protocol Selection and Prioritization. Acad Radiol
cine applications of machine learning have 2017;24(2):160–6.
garnered much attention in both the scientific 9. Brown AD, Marotta TR. Using machine learning for
community and the general public, the other sequence-level automated MRI protocol selection
important potential applications of AI in health in neuroradiology. J Am Med Inform Assoc 2018;
care should not be forgotten, including those 25(5):568–71.
related to process improvement and operational 10. Kalra A, Chakraborty A, Fine B, et al. Machine
efficiency. Various examples of such applications Learning for Automation of Radiology Protocols for
are discussed in this article. These applications Quality and Efficiency Improvement. J Am Coll Ra-
have the potential to have significant impact on diol 2020;17(9):1149–58. https://doi.org/10.1016/j.
health care delivery, and by nature are likely to jacr.2020.03.012.
face fewer regulatory hurdles and easier adoption 11. Trivedi H, Mesterhazy J, Laguna B, et al. Automatic
compared with more complex diagnostic image Determination of the Need for Intravenous Contrast
interpretation tasks. Regardless of the specific ap- in Musculoskeletal MRI Examinations Using IBM
plications, machine learning–based algorithms are Watson’s Natural Language Processing Algorithm.
not currently well integrated into clinical ecosys- J Digit Imaging 2018;31(2):245–51.
tems and often require parallel pipelines for anal- 12. Sebastiani F. Machine learning in automated text
ysis.4,93 A robust informatics infrastructure and categorization. ACM Comput Surv 2002;34(1):1–47.
seamless data integration are needed to fully 13. Lee YH. Efficiency Improvement in a Busy Radiology
exploit the potential of machine learning–derived Practice: Determination of Musculoskeletal Mag-
breakthroughs in health care, and they need to netic Resonance Imaging Protocol Using Deep-
be prioritized to support important developments Learning Convolutional Neural Networks. J Digit Im-
and innovations in health care that not only will aging 2018;31(5):604–10.
improve the quality of patient care but also have 14. Spasic I, Nenadic G. Clinical Text Data in Machine
the potential to improve health care system effi- Learning: Systematic Review. JMIR Med Inform
ciency and sustainability. 2020;8(3):e17984.
15. Khorasani R, Hentel K, Darer J, et al. Ten command-
ments for effective clinical decision support for im-
REFERENCES
aging: enabling evidence-based practice to
1. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature improve quality and reduce waste. AJR Am J Roent-
2015;521(7553):436–44. genol 2014;203(5):945–51.
2. Chartrand G, Cheng PM, Vorontsov E, et al. Deep 16. Hentel K, Menard A, Khorasani R. New CMS Clinical
Learning: A Primer for Radiologists. Radiographics Decision Support Regulations: A Potential
2017;37(7):2113–31.
 Artificial Intelligence Applications e13

Opportunity with Major Challenges. Radiology 2017; weighted Imaging of the Brain with a Multi-shot
283(1):10–3. Echo-planar Sequence: Deep-Learning-based
17. Sutton RT, Pincock D, Baumgart DC, et al. An over- Denoising. Magn Reson Med 2020. https://doi.org/
view of clinical decision support systems: benefits, 10.2463/mrms.tn.2019-0081.
risks, and strategies for success. NPJ Digit Med 31. Do WJ, Seo S, Han Y, et al. Reconstruction of multi-
2020;3(1):17. contrast MR images through deep learning. Med
18. Gillies RJ, Kinahan PE, Hricak H. Radiomics: Images Phys 2020;47(3):983–97. https://doi.org/10.1109/
Are More than Pictures, They Are Data. Radiology TMI.2020.2987026.
2016;278(2):563–77. 32. Gong E, Pauly JM, Wintermark M, et al. Deep
19. Bennett CC, Hauser K. Artificial intelligence frame- learning enables reduced gadolinium dose for
work for simulating clinical decision-making: a Mar- contrast-enhanced brain MRI. J Magn Reson Imag-
kov decision process approach. Artif Intell Med ing 2018;48(2):330–40. https://doi.org/10.1002/mp.
2013;57(1):9–19. 13047.
20. Rousseau JF, Ip IK, Raja AS, et al. Can Automated 33. Wang G, Gong E, Banerjee S, et al. Synthesize high-
Retrieval of Data from Emergency Department quality multi-contrast magnetic resonance imaging
Physician Notes Enhance the Imaging Order Entry from multi-echo acquisition using multi-task deep
Process? Appl Clin Inform 2019;10(2):189–98. generative model. IEEE Trans Med Imaging 2020.
21. Marella WM, Sparnon E, Finley E. Screening elec- https://doi.org/10.1002/mrm.28432.
tronic health record-related patient safety reports 34. Emami H, Dong M, Nejad-Davarani SP, et al. Gener-
using machine learning. J Patient Saf 2017;13(1): ating synthetic CTs from magnetic resonance im-
31–6. ages using generative adversarial networks. Med
22. Schemmel A, Lee M, Hanley T, et al. Radiology work- Phys 2018.
flow disruptors: a detailed analysis. J Am Coll Radiol 35. Sanders JW, Chen HS, Johnson JM, et al. Synthetic
2016;13(10):1210–4. generation of DSC-MRI-derived relative CBV maps
23. Yin Z, Yao Y, Montillo A, et al. Acquisition, prepro- from DCE MRI of brain tumors. Magn Reson Med
cessing, and reconstruction of ultralow dose volu- 2020.
metric CT scout for organ-based CT scan planning. 36. Bahrami A, Karimian A, Fatemizadeh E, et al. A new
Med Phys 2015;42(5):2730–9. deep convolutional neural network design with effi-
24. You C, Yang Q, Shan H, et al. Structurally-sensitive cient learning capability: Application to CT image
Multi-scale Deep Neural Network for Low-Dose CT synthesis from MRI. Med Phys 2020. https://doi.
Denoising. IEEE Access 2018;6:41839–55. org/10.1002/mp.14418.
25. Zhao T, McNitt-Gray M, Ruan D. A convolutional 37. Luo H, Hao W, Foos DH, et al. Automatic image
neural network for ultra-low-dose CT denoising and hanging protocol for chest radiographs in PACS.
emphysema screening. Med Phys 2019;46(9): IEEE Trans Inf Technol Biomed 2006;10(2):302–11.
3941–50. 38. Filice RW, Frantz SK. Effectiveness of Deep Learning
26. Racine D, Becce F, Viry A, et al. Task-based charac- Algorithms to Determine Laterality in Radiographs.
terization of a deep learning image reconstruction J Digit Imaging 2019;32(4):656–64.
and comparison with filtered back-projection and a 39. The Joint Commission. National Patient Safety
partial model-based iterative reconstruction in Goals. Available at: https://www.jointcommission.org/
abdominal CT: A phantom study. Phys Med 2020; standards/national-patient-safety-goals/hospital-2020-
76:28–37. national-patient-safety-goals/. Accessed June 22,
27. Wolterink JM, Leiner T, Viergever MA, et al. Genera- 2020.
tive Adversarial Networks for Noise Reduction in 40. Radiology. ACo. ACR PRACTICE PARAMETERFOR
Low-Dose CT. IEEE Trans Med Imaging 2017; COMMUNICATION OF DIAGNOSTIC IMAGING
36(12):2536–45. FINDINGS. Available at: https://www.acr.org/-/media/
28. Lin DJ, Johnson PM, Knoll F, et al. Artificial Intelli- ACR/Files/Practice-Parameters/CommunicationDiag.
gence for MR Image Reconstruction: An Overview pdf. Accessed June 22, 2020.
for Clinicians. J Magn Reson Imaging 2020. 41. Wildman-Tobriner B, Thorpe MP, Said N, et al. Mov-
https://doi.org/10.1002/jmri.27078. ing Radiology Workflow to the Electronic Health Re-
29. Haskell MW, Cauley SF, Bilgic B, et al. Network cord: Quantitative and Qualitative Experience From
Accelerated Motion Estimation and Reduction a Large Academic Medical Center. Acad Radiol
(NAMER): Convolutional neural network guided 2020;27(2):253–9.
retrospective motion correction using a separable 42. Wesp W. Using STAT properly. Radiol Manage 2006;
motion model. Magn Reson Med 2019;82(4): 28(1):26–30. quiz 31–23.
1452–61. 43. McDonald JE, Kessler MM, Hightower JL, et al.
30. Kawamura M, Tamada D, Funayama S, et al. Accel- Server-based enterprise collaboration software
erated Acquisition of High-resolution Diffusion-
e14 Letourneau-Guillon et al

improves safety and quality in high-volume PET/CT adsabs.harvard.edu/abs/2019arXiv190310122L. Ac-

practice. J Nucl Med Technol 2013;41(4):289–91. cessed March 01, 2019.
44. Prevedello LM, Erdal BS, Ryu JL, et al. Automated 57. Shin H-C, Roberts K, Lu L, et al. Learning to Read
Critical Test Findings Identification and Online Notifi- Chest X-Rays: Recurrent Neural Cascade Model for
cation System Using Artificial Intelligence in Imag- Automated Image Annotation. 2016. arXiv:
ing. Radiology 2017;285(3):923–31. 1603.08486. Available at: https://ui.adsabs.harvard.
45. Honig SE, Honig EL, Babiarz LB, et al. Critical find- edu/abs/2016arXiv160308486S. Accessed March
ings: timing of notification in neuroradiology. AJNR 01, 2016.
Am J Neuroradiol 2014;35(8):1485–92. 58. Xue Y, Xu T, Rodney Long L, et al.Multimodal
46. Grewal M, Mayank Srivastava M, Kumar P, et al. recurrent model with attention for automated radi-
RADNET: Radiologist Level Accuracy using Deep ology report generation. In: Frangi A, Schnabel J,
Learning for HEMORRHAGE detection in CT Scans. Davatzikos C, et al, editors. Medical Image
arXiv 2017. arXiv:1710.04934. Available at: https:// Computing and Computer Assisted Intervention -
ui.adsabs.harvard.edu/abs/2017arXiv171004934G. MICCAI 2018. MICCAI 2018. Lecture Notes in
Accessed October 01, 2017. Computer Science, vol 11070. Springer, Cham.
47. Hooshmand M, Soroushmehr SMR, Williamson C, https://doi.org/10.1007/978-3-030-00928-1_52.
et al. Automatic Midline Shift Detection in Traumatic 59. Yuan J, Liao H, Luo R, et al. Automatic Radiology
Brain Injury. Paper presented at: 2018 40th Annual Report Generation based on Multi-view Image
International Conference of the IEEE Engineering in Fusion and Medical Concept Enrichment. 2019. ar-
Medicine and Biology Society (EMBC). Honolulu Xiv:1907.09085. Available at: https://ui.adsabs.
(HI), July 18–21, 2018. harvard.edu/abs/2019arXiv190709085Y. Accessed
48. Wang H-C, Ho S-H, Xiao F, et al. A Simple, Fast and July 01, 2019.
Fully Automated Approach for Midline Shift Measure- 60. Demner-Fushman D, Kohli MD, Rosenman MB, et al.
ment on Brain Computed Tomography. arXiv 2017. Preparing a collection of radiology examinations for
arXiv:1703.00797. Available at: https://ui.adsabs. distribution and retrieval. J Am Med Inform Assoc
harvard.edu/abs/2017arXiv170300797W. Accessed 2016;23(2):304–10.
March 01, 2017. 61. Hosny A, Parmar C, Quackenbush J, et al. Artificial
49. Amukotuwa SA, Straka M, Dehkharghani S, et al. intelligence in radiology. Nat Rev Cancer 2018;
Fast Automatic Detection of Large Vessel Occlu- 18(8):500–10.
sions on CT Angiography. Stroke 2019;50(12): 62. Mamlouk MD, Chang PC, Saket RR. Contextual
3431–8. Radiology Reporting: A New Approach to Neurora-
50. Chilamkurthy S, Ghosh R, Tanamala S, et al. Deep diology Structured Templates. AJNR Am J Neurora-
learning algorithms for detection of critical findings diol 2018;39(8):1406–14.
in head CT scans: a retrospective study. Lancet 63. Marcovici PA, Taylor GA. Journal Club: Structured
2018;392(10162):2388–96. radiology reports are more complete and more
51. Li X, Thrall JH, Digumarthy SR, et al. Deep learning- effective than unstructured reports. AJR Am J
enabled system for rapid pneumothorax screening Roentgenol 2014;203(6):1265–71.
on chest CT. Eur J Radiol 2019;120:108692. 64. Brady AP. Radiology reporting-from Hemingway to
52. Halder A, Dey D, Sadhu AK. Lung Nodule Detection HAL? Insights Imaging 2018;9(2):237–46.
from Feature Engineering to Deep Learning in 65. Powell DK, Silberzweig JE. State of structured re-
Thoracic CT Images: a Comprehensive Review. porting in radiology, a survey. Acad Radiol 2015;
J Digit Imaging 2020;33(3):655–77. 22(2):226–33.
53. Buda M, Wildman-Tobriner B, Hoang JK, et al. Man- 66. Steinkamp JM, Chambers C, Lalevic D, et al. Toward
agement of Thyroid Nodules Seen on US Images: Complete Structured Information Extraction from
Deep Learning May Match Performance of Radiolo- Radiology Reports Using Machine Learning.
gists. Radiology 2019;292(3):695–701. J Digit Imaging 2019;32(4):554–64.
54. Velikova M, Lucas PJF, Samulski M, et al. On the 67. Tang A, Tam R, Cadrin-Chenevert A, et al. Canadian
interplay of machine learning and background Association of Radiologists White Paper on Artificial
knowledge in image interpretation by Bayesian net- Intelligence in Radiology. Can Assoc Radiol J 2018;
works. Artif Intell Med 2013;57(1):73–86. 69(2):120–35.
55. Rauschecker AM, Rudie JD, Xie L, et al. Artificial In- 68. Chen MC, Ball RL, Yang L, et al. Deep Learning to
telligence System Approaching Neuroradiologist- Classify Radiology Free-Text Reports. Radiology
level Differential Diagnosis Accuracy at Brain MRI. 2018;286(3):845–52.
Radiology 2020;295(3):626–37. 69. Cai T, Giannopoulos AA, Yu S, et al. Natural Lan-
56. Li CY, Liang X, Hu Z, et al. Knowledge-driven Encode, guage Processing Technologies in Radiology
Retrieve, Paraphrase for Medical Image Report Gen- Research and Clinical Applications. Radiographics
eration. 2019. arXiv:1903.10122. Available at: https://ui. 2016;36(1):176–91.
 Artificial Intelligence Applications e15

70. Pons E, Braun LMM, Hunink MGM, et al. Natural Patients with High-Grade Gliomas. AJNR Am J Neu-
Language Processing in Radiology: A Systematic roradiol 2020;41(1):40–8.
Review. Radiology 2016;279(2):329–43. 83. Li H, Xie Y, Wang X, et al. Radiomics features on
71. Spandorfer A, Branch C, Sharma P, et al. Deep non-contrast computed tomography predict early
learning to convert unstructured CT pulmonary angi- enlargement of spontaneous intracerebral hemor-
ography reports into structured reports. Eur Radiol rhage. Clin Neurol Neurosurg 2019;185:105491.
Exp 2019;3(1):37. 84. Shen Q, Shan Y, Hu Z, et al. Quantitative parameters
72. Goff DJ, Loehfelm TW. Automated Radiology Report of CT texture analysis as potential markers for early
Summarization Using an Open-Source Natural Lan-
prediction of spontaneous intracranial hemorrhage
guage Processing Pipeline. J Digit Imaging 2018; enlargement. Eur Radiol 2018;28(10):4389–96.
31(2):185–92.
85. Zhang Y, Zhang B, Liang F, et al. Radiomics features
73. Lou R, Lalevic D, Chambers C, et al. Automated
on non-contrast-enhanced CT scan can precisely
Detection of Radiology Reports that Require
classify AVM-related hematomas from other sponta-
Follow-up Imaging Using Natural Language Pro-
neous intraparenchymal hematoma types. Eur Ra-
cessing Feature Engineering and Machine Learning
diol 2019;29(4):2157–65.
Classification. J Digit Imaging 2020;33(1):131–6.
74. Oliveira L, Tellis R, Qian Y, et al. Follow-up Recom- 86. Oakden-Rayner L, Carneiro G, Bessen T, et al. Pre-
mendation Detection on Radiology Reports with Inci- cision Radiology: Predicting longevity using feature
dental Pulmonary Nodules. Stud Health Technol engineering and deep learning methods in a radio-
Inform 2015;216:1028. mics framework. Sci Rep 2017;7(1):1648.
75. Bozkurt S, Alkim E, Banerjee I, et al. Automated 87. Glover Mt, Daye D, Khalilzadeh O, et al. Socioeco-
Detection of Measurements and Their Descriptors nomic and Demographic Predictors of Missed Op-
in Radiology Reports Using a Hybrid Natural Lan- portunities to Provide Advanced Imaging Services.
guage Processing Algorithm. J Digit Imaging 2019; J Am Coll Radiol 2017;14(11):1403–11.
32(4):544–53. 88. Pestian JP, Brew C, Matykiewicz P, et al. A shared
76. Dew KN, Turner AM, Choi YK, et al. Development of task involving multi-label classification of clinical free
machine translation technology for assisting health text. Proceedings of the Workshop on BioNLP 2007:
communication: A systematic review. J Biomed Biological, Translational, and Clinical Language Pro-
Inform 2018;85:56–67. cessing; 2007, Prague, Czech Republic https://dl.
77. Sobez LM, Kim SH, Angstwurm M, et al. Creating acm.org/doi/10.5555/1572392.1572411.
high-quality radiology reports in foreign languages 89. Karimi S, Dai X, Hassanzadeh H, et al. Automatic
through multilingual structured reporting. Eur Radiol diagnosis coding of radiology reports: a comparison
2019;29(11):6038–48. of deep learning and conventional classification
78. Rao B, Zohrabian V, Cedeno P, et al. Utility of Artifi- methods. Paper presented at: BioNLP 2017; 2017.
cial Intelligence Tool as a Prospective Radiology Available at: https://www.aclweb.org/anthology/
Peer Reviewer - Detection of Unreported Intracranial W17-2342/.
Hemorrhage. Acad Radiol 2020.
90. Rios A, Kavuluru R. Neural transfer learning for as-
79. Kovacs MD, Mesterhazy J, Avrin D, et al. Correlate:
signing diagnosis codes to EMRs. Artif Intell Med
A PACS- and EHR-integrated Tool Leveraging Natu-
2019;96:116–22.
ral Language Processing to Provide Automated Clin-
ical Follow-up. Radiographics 2017;37(5):1451–60. 91. Joudaki H, Rashidian A, Minaei-Bidgoli B, et al. Us-
80. Chang P, Grinband J, Weinberg BD, et al. Deep- ing data mining to detect health care fraud and
abuse: a review of literature. Glob J Health Sci
Learning Convolutional Neural Networks Accurately
Classify Genetic Mutations in Gliomas. AJNR Am J 2014;7(1):194–202.
Neuroradiol 2018;39(7):1201–7. 92. Abdullah S, Rothenberg S, Siegel E, et al. School of
81. Jang BS, Jeon SH, Kim IH, et al. Prediction of Pseu- Block-Review of Blockchain for the Radiologists.
doprogression versus Progression using Machine Acad Radiol 2020;27(1):47–57.
Learning Algorithm in Glioblastoma. Sci Rep 2018; 93. Dikici E, Bigelow M, Prevedello LM, et al. Integrating
8(1):12516. AI into radiology workflow: levels of research, pro-
82. Han W, Qin L, Bay C, et al. Deep Transfer Learning duction, and feedback maturity. J Med Imaging
and Radiomics Feature Prediction of Survival of (Bellingham) 2020;7(1):016502.