Journal of the American Medical Informatics Association, 27(11), 2020, 1688–1694

doi: 10.1093/jamia/ocaa154
Advance Access Publication Date: 27 September 2020
Research and Applications

Research and Applications

A machine learning–based clinical decision support

Downloaded from https://academic.oup.com/jamia/article/27/11/1688/5912186 by Withers user on 04 September 2023

system to identify prescriptions with a high risk of
medication error
Jennifer Corny ,1 Asok Rajkumar,1 Olivier Martin,2 Xavier Dode,3,4
Jean-Patrick Lajonchère,5 Olivier Billuart,6 Yvonnick Be
zie,1 and Anne Buronfosse6

1
Pharmacy Department, Groupe Hospitalier Paris Saint Joseph, Paris, France, 2Lumio Medical, Paris, France, 3Centre National
edicament, Paris, France, 4Pharmacy Department, Hospices Civils de Lyon University Hospital,
Hospitalier d’Information sur le M
Lyon, France, Groupe Hospitalier Paris Saint Joseph, Paris, France, 6Medical Information Department, Groupe Hospitalier Paris
5

Saint Joseph, Paris, France

Corresponding author: Jennifer Corny, PharmD, Pharmacy Department, Groupe Hospitalier Paris Saint Joseph, 185 rue
Raymond Losserand, 75014 Paris, France; jcorny@hpsj.fr
Received 3 April 2020; Revised 10 June 2020; Editorial Decision 19 June 2020; Accepted 30 June 2020

ABSTRACT
Objective: To improve patient safety and clinical outcomes by reducing the risk of prescribing errors, we tested
the accuracy of a hybrid clinical decision support system in prioritizing prescription checks.
Materials and Methods: Data from electronic health records were collated over a period of 18 months. Inferred
scores at a patient level (probability of a patient’s set of active orders to require a pharmacist review) were cal-
culated using a hybrid approach (machine learning and a rule-based expert system). A clinical pharmacist ana-
lyzed randomly selected prescription orders over a 2-week period to corroborate our findings. Predicted scores
were compared with the pharmacist’s review using the area under the receiving-operating characteristic curve
and area under the precision-recall curve. These metrics were compared with existing tools: computerized
alerts generated by a clinical decision support (CDS) system and a literature-based multicriteria query prioritiza-
tion technique. Data from 10 716 individual patients (133 179 prescription orders) were used to train the algo-
rithm on the basis of 25 features in a development dataset.
Results: While the pharmacist analyzed 412 individual patients (3364 prescription orders) in an independent val-
idation dataset, the areas under the receiving-operating characteristic and precision-recall curves of our digital
system were 0.81 and 0.75, respectively, thus demonstrating greater accuracy than the CDS system (0.65 and
0.56, respectively) and multicriteria query techniques (0.68 and 0.56, respectively).
Discussion: Our innovative digital tool was notably more accurate than existing techniques (CDS system and
multicriteria query) at intercepting potential prescription errors.
Conclusions: By primarily targeting high-risk patients, this novel hybrid decision support system improved the
accuracy and reliability of prescription checks in a hospital setting.

Key words: supervised machine learning, electronic prescribing, clinical pharmacy information systems, medication errors,
decision support systems, clinical

C The Author(s) 2020. Published by Oxford University Press on behalf of the American Medical Informatics Association.
V
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/),
which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact
journals.permissions@oup.com 1688
Journal of the American Medical Informatics Association, 2020, Vol. 27, No. 11 1689

INTRODUCTION is making prediction at the patient level, rather than through predic-
tions about individual prescription orders.
Medical errors are a major public health problem and a leading
cause of mortality. With some 250 000 deaths per year in the United
States, medical errors now rank after heart disease and cancer as the
MATERIALS AND METHODS
third leading cause of death.1 Even back in 1999, the Institute of
Medicine highlighted the need for technologies to prevent the esti- Setting
mated 44 000 to 98 000 annual deaths resulting from medical The study was conducted in a large, private, nonprofit hospital (592
errors.2 The problem is global, and the findings from the United beds) in Paris that provides both surgical and medical activities.
States are readily supported by data from other countries that have With the exception of neonatology and intensive care units that rely
also reported substantial rates of health care–related adverse on a dedicated software program, all patient files are typically digi-
events.3–7 During hospitalization, the majority of adverse events are tized and recorded using DxCare medical software (Dedalus, Le

Downloaded from https://academic.oup.com/jamia/article/27/11/1688/5912186 by Withers user on 04 September 2023

attributed to invasive procedures, hospital-acquired infections, and Plessis-Robinson, France). Prescription orders and medication
health products (drugs and medical devices).7 While at least 30% of reviews by clinical pharmacists are managed with the same soft-
adverse events are likely easily preventable, one study showed that ware, which includes all medical and nursing notes, laboratory
adverse events associated with negligence (including medical errors) results, and vital signs. All drugs can be prescribed (formulary or
were more likely to be associated with injury to patients than other nonformulary). The software also includes a CDS system and is
adverse events.8 interfaced with a national drug database. However, given the associ-
Improving patient safety by reducing medication errors has ated workload, systematic comprehensive medication reviews are
therefore become a top priority. Although mistakes can occur at any typically restricted to certain medical wards—namely cardiology,
stage of the medical treatment process, prescribing and administra- endocrinology, internal medicine, and rheumatology—where
tion of drugs are the most frequent sources of medication errors.9,10 patients are considered to have a high risk of exposure to prescrip-
Numerous tools have been developed in an attempt to improve tion errors.
point-of-care prescription processes, including electronic prescrib-
ing—termed computerized prescriber order entry (CPOE)—and Datasets
computerized clinical decision support (CDS) alert systems. How- Data collection
ever, CPOE is known to generate other types of prescribing errors Data collected over an 18-month period from January 2017 to Au-
such as the wrong drug being prescribed,11 and CDS systems are gust 2018 were extracted directly from the electronic health records
known to generate numerous unnecessary alerts, leading to alert fa- by the hospital Medical Information Department: medication order
tigue (desensitization) and subsequent inefficiency.12,13 data, laboratory reports, demographics, medical history, and vital
Medication review by clinical pharmacists is currently the gold- signs. Together these data comprise the development dataset used to
standard method of verification and is increasingly recognized as a train the algorithm.
critical step in the prevention of potential adverse drug events.14,15 Throughout the study, all data were stored on a secure local
Review can trigger pharmaceutical interventions, which are actions server. Patient identities were pseudonymized: each patient was allo-
implemented in response to—or to prevent—a drug-related problem cated an identification number that was saved on an identification
in an individual patient. However, the process is time-consuming table on the server.
and, as with any human process, not always reproducible. In addi-
tion, interventions of this type need to be targeted to ensure that Prediction target-pharmacist interventions
patients who have the greatest risk of errors in their prescription All the prescription orders included in the development dataset and
orders are given priority. We recently showed that polypharmacy, used to train the algorithm (labeled data) had already undergone at
patient age, and impaired renal function were associated with more least 1 medication order review. Details of pharmacist interventions
frequent drug-related problems, and consequently, more frequent were routinely affixed to each medication order as a comment. Each
pharmaceutical interventions.16 However, neither individual predic- intervention was categorized by a clinical pharmacist according
tor strategies nor multivariate model-based strategies have reliably to the ACT-IP classification of the French Society of Clinical
demonstrated an ability to detect high-risk patients.17,18 While these Pharmacy.20
models have marginally improved the efficacy of medication review
by targeting patients at the greatest risk, other parameters—includ- Score design
ing clinical presentation or progression of the patient, laboratory Input features
findings, drug regimens, and drug interactions—also need to be A large number of datasets were compiled to provide the most com-
taken into consideration. The increasingly widespread availability of prehensive overview of the context of each prescription order and
electronic health records and the development of big data analytics the patient’s medical condition, and thus mitigate any possible bias
are currently paving the way for the use of artificial intelligence, in the selection of features. Each feature is commonly used for medi-
which relies on sophisticated algorithms with the capacity to analyze cation review in routine clinical pharmacy practice, and corre-
vast quantities of data to make medication review more effective sponded to a set of individual patient data: laboratory reports (eg,
and thus help pharmacists predict or intercept drug-related renal function, serum potassium, international normalized ratio),
problems, and therefore make potential medication errors more demographics (sex, age), medical history (information on allergies
accurately.19 extracted from free text comment fields), and physiological data
In this context, we tested the accuracy of Lumio Medication (de- (vital signs, such as weight, heart rate, and arterial pressure).
veloped by Lumio Medical, Paris, France), a hybrid artificial intelli- For each prescription order, a specific set of rule-based alerts re-
gence decision support system—combining machine learning and a lating to the medication (eg, dosage, frequency, route) were used.
rule-based expert system—in a typical hospital setting. This system The number of times each type of alert was raised represented a
1690 Journal of the American Medical Informatics Association, 2020, Vol. 27, No. 11

categorical feature (discrete number). Alerts were either extracted Metrics

from the French online drug database Theriaque (edited by Centre Prior to the start of the study, several widely acknowledged metrics
National Hospitalier d’Information sur le Medicament),21 or built were selected to assess the algorithm’s capacity to predict the risk of
specifically by the hospital pharmacy team on the basis of published an individual patient prescription to contain at least 1 drug-related
literature, and avoiding any overlap with the drug database. For ex- problem (binary classification). The area under the receiver-
ample, a prescription order of a concentration >4 g/L of potassium operating characteristic curve (AUROC) does not rely on prevalence
chloride was considered as inappropriate, and a rule was built to ad- and is therefore the most widely used metric for model comparisons
dress this rare but potentially harmful prescribing error. in statistics. The receiver-operating characteristic (ROC) curve is
created by plotting the true positive rate (also known as sensitivity
Classifier architecture or recall) against the false positive rate at various threshold settings.
We sought to generate a score representing the risk for a given However, the area under the precision-recall curve (AUCPR) pro-
vides a more accurate representation of the impact the algorithm

Downloaded from https://academic.oup.com/jamia/article/27/11/1688/5912186 by Withers user on 04 September 2023

patient’s prescription order to contain at least 1 drug-related prob-
lem (considered as a medication error), thus answering the question: can have on the pharmacist’s work,22 as the prevalence of pharma-
“Does this patient’s set of active orders require a pharmacist ceutical interventions on prescription orders was 3.6%; it is created
review?” To this end, we trained a binary classifier to identify by plotting the positive predictive value (also known as precision)
patients who were likely to have at least 1 drug-related error in their against the true positive rate at various threshold settings.
prescription order. The classifier selected was derived from We also calculated the F1 score (ie, the harmonic mean of preci-
LightGBM, a gradient-boosting framework based on decision tree sion and recall). The 95% confidence interval was calculated for
algorithms and developed by Microsoft (Redmond, WA). The 2 AUCPR and AUROC scores using a bootstrap method after sam-
types of data were combined as inputs: patient-related data and pling with replacement from 10 000 random samples.23
prescription-related alerts. Permutation tests (also called exact tests) were computed with
For each prescription order in the development dataset, the label- 10 000 permutations to assess the statistical significance of the
ing was therefore binary: 1 ¼ a pharmaceutical intervention was car- results compared.
ried out; 0 ¼ no pharmaceutical intervention.

Preprocessing Comparators
The binary classifier used 25 features engineered from heteroge- Two other prioritization processes were used as comparators to
neous inputs: numerical quantities, date/time objects, categorical evaluate the performance of our algorithm: one based on patient-
values, and natural language open text fields. During preprocessing, related data (the multicriteria query), and the other based on medi-
all numerical features were calibrated for outliers, standardized and cation orders (CDS alert system). The latter makes use of a certified
imputed, while categorical features were encoded, using scikit-learn drug database to provide alerts after analyzing patient prescription
and scikit-learn machine learning–compatible libraries. orders. The CDS alert system thus raised alerts relating to drug inter-
actions, dosage errors, and contraindications for renal insufficiency.
Testing protocol We also used a multicriteria query strategy based on 4 easily
We tested the performance of the tool on a separate validation data- available and widely recognized criteria to target high-risk patients
set (independent of the one used for model development). The accu- (ie, age, renal function [glomerular filtration rate], serum potassium,
racy of the algorithm was assessed and compared with the and international normalized ratio).17,18 The score was determined
prescription orders reviewed by pharmacists and also with classic using the following thresholds with the calculation of the number of
techniques: a CDS alert system and a multicriteria query approach. alerts raised:

• Age
Methodology • >75 years: 1
Over a 2-week period, a fully trained clinical pharmacist routinely • Other: 0
analyzed randomly selected patient prescription orders on all wards • Estimated glomerular filtration rate (Modification of Diet in Re-
and made a note of the interventions that followed. The selection of
nal Disease) formula:
prescription orders came from an automatic daily extraction from • <30 mL/min: 1
the medical software documenting all patients who had at least 1 • Other: 0
drug prescription. The pharmacist reviewed as many patients as pos- • Serum potassium level
sible over the 2-week period. • <3 mmol/L: 1
The data scientists were blinded to the actual pharmaceutical • >5 mmol/L: 1
interventions. Data relating to these prescription orders were then • Other: 0
used as inputs for the algorithm, which was thus tested on all wards. • International normalized ratio
All predicted scores (a continuous variable: probability for a pre- • <5: 0
scription order to contain errors) were then compared with the bi- • Other: 1
nary score: 1 ¼ a pharmaceutical intervention was carried out
during medication review; 0 ¼ no pharmaceutical intervention. For example:
For drug-related problems that were not intercepted by the tool • a 40-year-old patient with no other criteria: score, 0
(false negatives), a group of 2 physicians and 2 pharmacists ranked • an 80-year-old patient with an international normalized ratio of
the level of severity (from 1 [minor] to 4 [life-threatening]). To de- 4: score, 1
termine this risk, the patient’s file was reviewed and the potential • a 76-year-old patient with a glomerular filtration rate of 27 mL/
immediate or midterm harm was assessed. min: score, 2
Journal of the American Medical Informatics Association, 2020, Vol. 27, No. 11 1691

Ethics approval For continuous scoring, such as the output of the hybrid algo-
Institutional review board approval was obtained from the local rithm (a probability), recall and precision were calculated by select-
ethics committee. Considering the type of study, international re- ing the classification threshold that maximizes the F1 score.
view board approval was not required. The decision support algorithm outperformed classic systems in
its capacity to both detect patients with a medication error (recall,
also known as sensitivity), and to limit the number of false alerts
RESULTS (precision, also called the positive predictive value).
Figures 1 and 2 show the results with regard to AUCPR and
Data collection and development
AUROC.
Over an 18-month period, data were collected on 94 720 hospital-
izations and a total of 61 611 patients (mean length of stay, 4.1
days; mean age, 69 years; female/male, 49.8%/50.2%), with a mean Accuracy of medication review using the algorithm

Downloaded from https://academic.oup.com/jamia/article/27/11/1688/5912186 by Withers user on 04 September 2023

of 9.4 prescription orders per hospitalization. In the independent validation dataset, with the classification thresh-
During this period, pharmacists reviewed a total of 10 716 indi- old that maximizes the F1 score, the Lumio Medication algorithm
vidual patient files (133 179 prescription orders), along with each intercepted 74% of all prescription orders that required pharmacist
patient’s individual data (laboratory findings, demographics, medi- intervention while also demonstrating 74% precision. Of the
cal history, and vital signs) as part of their daily practice, and these remaining 26% prescription orders that required pharmacist inter-
data were used to train the algorithm. This comprised the dataset vention (false negatives) that were not intercepted by the algorithm,
used for model development. none were life-threatening. The ensuing F1 score thus showed
A pharmaceutical intervention was recommended for 2163 indi- 15.6% greater accuracy than multicriteria query techniques and
vidual patients (20%), meaning 20% of the patients whose all pre- 21% greater accuracy than the CDS alert system.
scription orders were reviewed were at risk of a potential The AUCPR and AUROC scores showed the greater accuracy
medication error. Based on the ACT-IP classification, the main (þ33% and þ19%, respectively) of the algorithm vs multicriteria
drug-related problems requiring pharmaceutical intervention were query techniques, and vs the CDS alert system (þ33% and þ24%,
overdosing (33%; dose adaptation suggested), underdosing (16%; respectively).
dose adaptation suggested), and noncompliance with the drug for-
mulary (16%; replacement by a therapeutic equivalent suggested).
Only 3.6% of prescription orders required a pharmaceutical inter- Statistical significance
vention. The permutation tests showed the statistically significant superiority
In all, 72.7% of the input to the development dataset was from of the AUCPR and AUROC scores of the hybrid machine learning
the endocrinology, cardiology, rheumatology, internal medicine, system over those of random variables and vs the comparator tech-
and vascular medicine wards. niques: P < .00001 vs the CDS alert system, and P ¼ .001 (AUCPR)
and P ¼ .0152 (AUROC) vs the multicriteria query technique.

Comparative performance of the different techniques

Of the 412 individual patients (3364 prescription orders) that were DISCUSSION
randomly selected for analysis in the validation dataset, at least 1
pharmaceutical intervention was recommended in 174 (42%) This study presents a hybrid machine learning–based decision sup-
patients. In all, there were 211 pharmaceutical interventions (ie, port system for reviewing the accuracy of medical prescription
6.3% of all prescription orders). In the validation dataset, 64.7% of orders. Our findings confirm that the algorithm outperformed clas-
the input was from the accident and emergency department, and sic systems in its capacity to limit the number of false alerts without
vascular medicine, maternity, orthopedics and endocrinology wards. overlooking patients with prescription order errors.
CPOE has proven its efficacy in reducing medication errors. A
meta-analysis published in 2014 found that compared with paper
Performance of the algorithm vs classic prescription order entry, CPOE was associated with 50% less medication errors,
order analysis tools although new types of medication errors were associated with
The accuracy of the hybrid decision support algorithm was com- CPOE.11
pared with the CDS alert system and the multicriteria query CDS systems included in CPOE are a known source of alert
(Table 1). fatigue. Ancker et al24 found that the likelihood of alerts being

Table 1. Comparative performance of classic prescription order analysis tools versus the Lumio Medication algorithm in terms of recall, pre-
cision, and F1 scores, as well as AUCPR and AUROC

Metric CDS alert system Multicriteria query Lumio Medicationalgorithm

Recall 0.69 0.66 0.74a

Precision 0.54 0.62 0.74a
F1 0.61 0.64 0.74a
AUCPR 0.56 (95% CI, 0.50-0.62; P < .00001) 0.56 (95% CI, 0.51-0.61; P ¼ .001) 0.75 (95% CI, 0.70-0.80)a
AUROC 0.65 (95% CI, 0.61-0.69; P < .00001) 0.68 (95% CI, 0.64-0.72; P ¼ .0152) 0.81 (95% CI, 0.78-0.84)

AUCPR: area under the precision-recall curve; AUROC: area under receiver-operating characteristic curve; CDS: clinical decision support; CI, confidence
interval.
1692 Journal of the American Medical Informatics Association, 2020, Vol. 27, No. 11

Downloaded from https://academic.oup.com/jamia/article/27/11/1688/5912186 by Withers user on 04 September 2023

Figure 1. Accuracy of the decision support system: ROC AUC: area under receiver operating characteristic curve.

accepted decreased by 30% for each additional reminder received, adverse events, with interesting results.25,26 However, these only fo-
and by 10% for each 5% increase in the number of repeated alerts. cused on selected adverse events and did not consider all potential
Given that current day processes of medication review do not medication errors.
have the capacity to cover all medical prescription orders, the pro- Our hybrid decision support system combining machine learning
cess is in urgent need of improvement. Pharmaceutical interventions with a rule-based expert system was notably more accurate at
are still relatively scarce and therefore prioritization of medication detecting medication errors compared with other tools described in
reviews, based on the likelihood of drug-related problems and there- the literature. Two of 3 individual patient prescription orders
fore medication errors, is essential. This study provided compelling reviewed by our tool triggered a pharmaceutical intervention, a fig-
evidence of the accuracy of artificial intelligence in identifying those ure that compares very favorably with the 20% in our development
patients with the greatest risk of errors in their prescription orders dataset or approximately 17% in the study by Nguyen et al.17 The
(ie, prioritizing patients in whom medication review is justified). sensitivity of our tool was also significantly higher than that of the
Earlier studies identified several risk factors for medication errors, CDS alert system or multicriteria query techniques. The hybrid
including patient age, renal dysfunction, and the number of drugs model we have developed uses both knowledge-driven (expert sys-
prescribed, to help pharmacists target interventions more effectively. tem) and data-driven (machine learning) approaches. It can there-
However, we previously found that these risk factors accounted for fore be expected to overcome the main shortcomings of both these
only 34% of the variations in the number of pharmaceutical inter- techniques by (1) not overfitting and consequently reproducing the
ventions.16 A multicriteria model-based strategy was also developed same error patterns that occurred in the development dataset; (2)
to identify patients whose prescription orders presented a high risk addressing the issue of certain infrequent though critical medical
of containing errors. This model was based on 11 predictors, of errors such as the so-called never-events, that is, serious incidents
which patient age and the number of drugs on the prescription were that are wholly preventable, as highlighted by the French Agency for
the most significant, with a C-statistic of 0.72.17 Nevertheless, of a the Safety of Health Products; and (3) reducing the number of false
total of 303 individual patients, 6 still needed to be reviewed for a positive alerts typically seen with tools such as CDS alert systems.
drug-related problem to be detected, demonstrating the need for in- This tool can also be easily adjusted by the addition of specific
novative approaches to make this activity more effective. Other rules to account for noisy or conflicting categories that the algorithm
studies reported a C-score model to detect only previously identified has not yet learnt to deal with.
Journal of the American Medical Informatics Association, 2020, Vol. 27, No. 11 1693

Downloaded from https://academic.oup.com/jamia/article/27/11/1688/5912186 by Withers user on 04 September 2023

Figure 2. Accuracy of the decision support system: AP: area under the precision–recall curve.

One recently published study presented the results of an outlier cation review over the 18-month data collection and development
detection machine learning–based tool in a real-life setting that period; and (2) input with regard to pharmaceutical interventions
exhibited 89% accuracy in terms of the alerts raised.27 However, for the validation dataset and the development dataset came from
only 0.4% of the prescription orders generated an alert, whereas in different wards.
our study, 6.3% of prescription orders were associated with a phar- The next step is to deploy our system throughout other hospitals,
maceutical intervention. In addition, the authors did not report any thus extending the patient population covered. This will also enable
data on the sensitivity of this tool; it is therefore legitimate to specu- us to benefit from the experience of a greater number of clinical
late that because the system focused solely on outlier detection, pharmacists to confirm our findings. Adjustments are currently be-
other common medication errors likely went undetected. ing made to the algorithm to integrate unstructured data to further
The capacity of our hybrid system to constantly be adjusted as improve the performance of this tool. As an example, while the algo-
more prescription orders are reviewed by clinical pharmacists and rithm used in this study does not yet identify wrong-patient errors,
more potential error patterns identified gives it a significant advan- adjustments presently underway will enable it to address potential
tage over existing CDS systems. errors on medical notes (free text) associated with CPOE. Finally, a
Our findings are currently limited in scope in as far as the study real-life evaluation is currently being conducted to assess the perfor-
was conducted in a single hospital setting. In addition, neonatology mance of this tool in daily medication reviews.
and intensive care unit patients were not included because they are
managed by a different medical software. Consequently, evidence of
the accuracy of the algorithm to identify prescription order errors in
these units has yet to be demonstrated and our results can therefore
CONCLUSION
not be applied to these patients. Importantly, more pharmaceutical A hybrid machine learning–based decision support system has been
interventions were recommended during the test phase than in the developed to intercept prescription orders with a high risk of con-
development dataset. There are 2 possible explanations: (1) in the taining at least 1 medication error. Given that it is based on machine
validation dataset, prescription orders were reviewed by an experi- learning– and rule-based alerts, this decision support system has the
enced clinical pharmacist, whereas several pharmacists (junior and advantage of not overfitting errors, of decreasing alert fatigue, and
senior) with different levels of experience were involved in the medi- also of addressing infrequent but nevertheless potentially critical
1694 Journal of the American Medical Informatics Association, 2020, Vol. 27, No. 11

errors. The positive clinical implication of this model, which has nationale-sur-les-evenements-indesirables-graves-associes-aux-soins
demonstrated superior accuracy to both CDS alert systems and mul- Accessed October 31, 2019.
ticriteria query prioritization strategies, is the 3-fold improvement in 8. Brennan TA, Leape LL, Laird NM, et al. Incidence of adverse events and
negligence in hospitalized patients. Results of the Harvard Medical Prac-
the efficacy and productivity of medication review in a hospital set-
tice Study I. N Engl J Med 1991; 324 (6): 370–6.
ting and thus greater patient safety.
9. Bates DW, Boyle DL, Vander Vliet MB, et al. Relationship between medi-
cation errors and adverse drug events. J Gen Intern Med 1995; 10 (4):
199–205.
FUNDING 10. Bobb A, Gleason K, Husch M, et al. The epidemiology of prescribing
This research received no specific grant from any funding agency in the pub- errors: the potential impact of computerized prescriber order entry. Arch
lic, commercial or not-for-profit sectors. The algorithm developed by Lumio, Intern Med 2004; 164 (7): 785–92.
and that will soon be available on the market for all hospitals, was provided 11. Villama~ nan E, Larrubia Y, Ruano M, et al. Potential medication errors as-
without cost to the hospital pharmacy to enable the tool to undergo rigorous sociated with computer prescriber order entry. Int J Clin Pharm 2013; 35

Downloaded from https://academic.oup.com/jamia/article/27/11/1688/5912186 by Withers user on 04 September 2023

testing in a real-life setting. (4): 577–83.
12. Backman R, Bayliss S, Moore D, et al. Clinical reminder alert fatigue in
healthcare: a systematic literature review protocol using qualitative evi-
AUTHOR CONTRIBUTIONS dence. Syst Rev 2017; 6 (1): 255.
13. McCoy AB, Thomas EJ, Krousel-Wood M, et al. Clinical decision support
JC, AR, OM, XD, J-PL, OB, YB, and AB all made substantial contributions
alert appropriateness: a review and proposal for improvement. Ochsner J
to the conception of the work and the acquisition, analysis, and interpretation
2014; 14 (2): 195–202.
of data for the work; drafted the work and revised it critically for important
14. Renaudin P, Baumstarck K, Daumas A, et al. Impact of a pharmacist-led
intellectual content; approved the final version of the work to be published;
medication review on hospital readmission in a pediatric and elderly pop-
and agreed to be accountable for all aspects of the work in ensuring that ques-
ulation: study protocol for a randomized open-label controlled trial. Trials
tions related to the accuracy or integrity of any part of the work are appropri-
2017; 18 (1): 65.
ately investigated and resolved.
15. Holland R, Smith R, Harvey I. Where now for pharmacist led medication
review? J Epidemiol Community Health 2006; 60 (2): 92–3.
16. Cornuault L, Mouchel V, Phan Thi TT, et al. Identification of variables
ACKNOWLEDGMENTS influencing pharmaceutical interventions to improve medication review
The authors thank Drs Menage, Phan Thi, and Voisin for their help in defin- efficiency. Int J Clin Pharm 2018; 40 (5): 1175–9.
ing specific rules and analyzing pharmaceutical interventions, with a special 17. Nguyen TL, Leguelinel-Blache G, Kinowski JM, et al. Improving medica-
mention to Dr Helène Beaussier for fruitful discussions. They also thank tion safety: Development and impact of a multivariate model-based strat-
Mathilde Vermeire for English-language editing and Moyra Barbier for edito- egy to target high-risk patients. PLoS One 2017; 12 (2): e0171995.
rial assistance. 18. Jarre C, Bouchet J, Hellot-Guersin M, et al. ACESO: une requ^ete pour la
selection des ordonnances  a risque. 2016. http://docplayer.fr/43107190-
Aceso-une-requete-pour-la-selection-des-ordonnances-a-risque.html
CONFLICT OF INTEREST STATEMENT Accessed 15, 2020.
19. Jiang F, Jiang Y, Zhi H, et al. Artificial intelligence in healthcare: past, pre-
JC, AR, OB, XD, J-PL, YB, and AB have no competing interests to declare.
sent and future. Stroke Vasc Neurol 2017; 2 (4): 230–3.
O Martin is one of the founders of Lumio Medical.
20. French Society of Clinical Pharmacy. ACT-IP platform. http://www.actip.
sfpc.eu/actip/index Accessed October 31, 2019.
21. Husson M-C. Theriaque: independent-drug database for good use of
REFERENCES drugs by health practitioners. Ann Pharm Fr 2008; 66 (5–6): 268–77.
1. Makary MA, Daniel M. Medical error–the third leading cause of death in 22. Davis J, Goadrich M. The relationship between precision-recall and ROC
the US. BMJ 2016; 353: i2139. curves. In: proceedings of the 23rd international conference on Machine
2. Institute of Medicine Committee on Quality of Health Care in America. learning-ICML’06. Pittsburgh, PA: ACM Press; 2006.
To Err is Human: Building a Safer Health System. Washington, DC: Na- 23. Bertail P, Clemençon S, Vayatis N. On constructing accurate confidence
tional Academies Press; 2000. bands for ROC curves through smooth resampling; 2008. https://hal.
3. Donaldson L. An organisation with a memory. Clin Med 2002; 2 (5): 452–7. archives-ouvertes.fr/hal-00335232 Accessed October 31, 2019.
4. Aranaz-Andres JM, Aibar-Rem on C, Vitaller-Burillo J, et al. Impact and 24. Ancker JS, Edwards A, Nosal S, et al. Effects of workload, work complex-
preventability of adverse events in Spanish public hospitals: results of the ity, and repeated alerts on alert fatigue in a clinical decision support sys-
Spanish National Study of Adverse Events (ENEAS). Int J Qual Health tem. BMC Med Inform Decis Mak 2017; 17 (1): 36.
Care 2009; 21 (6): 408–14. 25. Jeon N, Staley B, Johns T, et al. Identifying and characterizing preventable
5. Hogan H, Zipfel R, Neuburger J, et al. Avoidability of hospital deaths and adverse drug events for prioritizing pharmacist intervention in hospitals.
association with hospital-wide mortality ratios: retrospective case record Am J Health Syst Pharm 2017; 74 (21): 1774–83.
review and regression analysis. BMJ 2015; 351: h3239. 26. Winterstein AG, Staley B, Henriksen C, et al. Development and validation
6. Schoen C, Osborn R, Huynh PT, et al. Taking the pulse of health care sys- of a complexity score to rank hospitalized patients at risk for preventable
tems: experiences of patients with health problems in six countries. Health adverse drug events. Am J Health Syst Pharm 2017; 74 (23): 1970–84.
Aff (Millwood) 2005; 24 (Suppl 1): W5–509–25. 27. Segal G, Segev A, Brom A, et al. Reducing drug prescription errors and ad-
7. French Health Ministry. National survey on serious adverse events associ- verse drug events by application of a probabilistic, machine learning based
ated with care. https://drees.solidarites-sante.gouv.fr/etudes-et-statistiques/ clinical decision support system in an inpatient setting. J Am Med Inform
publications/documents-de-travail/serie-etudes-et-recherche/article/enquete- Assoc 2019; 26 (12): 1560–5.