Circulation: Cardiovascular Quality and Outcomes

ORIGINAL ARTICLE

Automated and Interpretable Patient ECG

Profiles for Disease Detection, Tracking,
and Discovery

See Editorial by Kao Geoffrey H. Tison, MD,

MPH*
BACKGROUND: The ECG remains the most widely used diagnostic Jeffrey Zhang, BA*
test for characterization of cardiac structure and electrical activity. We Francesca N. Delling, MD,
hypothesized that parallel advances in computing power, machine MPH
Rahul C. Deo, MD, PhD
learning algorithms, and availability of large-scale data could substantially
expand the clinical inferences derived from the ECG while at the same
time preserving interpretability for medical decision-making.
METHODS AND RESULTS: We identified 36 186 ECGs from the
University of California, San Francisco database that would enable training
of models for estimation of cardiac structure or function or detection of
Downloaded from http://ahajournals.org by on September 15, 2019

disease. We segmented the ECG into standard component waveforms

and intervals using a novel combination of convolutional neural networks
and hidden Markov models and evaluated this segmentation by
comparing resulting electrical intervals against 141 864 measurements
produced during the clinical workflow. We then built a patient-level
ECG profile, a 725-element feature vector and used this profile to
train and interpret machine learning models for examples of cardiac
structure (left ventricular mass, left atrial volume, and mitral annulus
e-prime) and disease (pulmonary arterial hypertension, hypertrophic
cardiomyopathy, cardiac amyloid, and mitral valve prolapse). ECG
measurements derived from the convolutional neural network-hidden *Dr Tison and J. Zhang contributed
Markov model segmentation agreed with clinical estimates, with median equally to this work.

absolute deviations as a fraction of observed value of 0.6% for heart rate Key Words: heart rate ◼ hypertension
◼ machine learning ◼ mitral valve
and 4% for QT interval. Models trained using patient-level ECG profiles prolapse ◼ work flow
enabled surprising quantitative estimates of left ventricular mass and
© 2019 The Authors. Circulation:
mitral annulus e′ velocity (median absolute deviation of 16% and 19%, Cardiovascular Quality and Outcomes
respectively) with good discrimination for left ventricular hypertrophy is published on behalf of the American
Heart Association, Inc., by Wolters
and diastolic dysfunction as binary traits. Model performance using our Kluwer Health, Inc. This is an open
approach for disease detection demonstrated areas under the receiver access article under the terms of
the Creative Commons Attribution
operating characteristic curve of 0.94 for pulmonary arterial hypertension, Non-Commercial-NoDerivs License,
0.91 for hypertrophic cardiomyopathy, 0.86 for cardiac amyloid, and 0.77 which permits use, distribution, and
reproduction in any medium, provided
for mitral valve prolapse. that the original work is properly cited,
the use is noncommercial, and no
CONCLUSIONS: Modern machine learning methods can extend the 12- modifications or adaptations are made.
lead ECG to quantitative applications well beyond its current uses while https://www.ahajournals.org/journal/
preserving the transparency that is so fundamental to clinical care. circoutcomes

Circ Cardiovasc Qual Outcomes. 2019;12:e005289. DOI: 10.1161/CIRCOUTCOMES.118.005289 September 2019 1

 Tison et al; Automated, Interpretable ECG Patient Profiles

ECG signals from multiple leads, or small visually imper-

WHAT IS KNOWN ceptible yet informative changes which may exist in the
signal, particularly in early disease stages.
• Although computerized interpretation algorithms
Given the physiological and structural correlates
for ECGs have existed for decades, they have been
constrained in that they aim to replicate the rules- of ECG signals, we hypothesized that a modern algo-
based approach to ECG analysis used by human rithmic approach could be used to expand the clini-
readers. cal inferences derived from ECGs. Algorithmic analysis
• Conventional ECG reading approaches can- should aspire to estimate continuous attributes of car-
not readily account for high-level interactions diac disease and structure, as well as capture change
between ECG signals from multiple leads, or small in these attributes over time. Moreover, such an analy-
visually imperceptible yet informative changes sis should inform the clinician which components of
which may exist in the signal, particularly in early the ECG signal are responsible for any given diagno-
disease stages. sis,8 thus providing the algorithmic transparency need-
ed to reassure physicians and patients about the basis
WHAT THE STUDY ADDS and possible validity of resulting automated diagnosis.
• Using a combination of machine learning methods, Although several recent efforts have applied machine
including convolutional neural networks and hidden learning techniques to ECG analysis,9,10 most of these
Markov models, we have developed an automated, innovative strategies suffer from being largely uninter-
scalable, interpretable method to perform detailed pretable. In high-stakes fields such as medicine, this
longitudinal tracking and comparison of ECGs.
limits the ability to understand successes or trouble-
• As demonstration examples, we indicate how a
shoot failures, potentially dampening physician adop-
personalized ECG vector profile can be used to
estimate continuous measures of cardiac struc- tion of an unfamiliar technology. Presently, there does
ture and function such as left ventricular mass and not exist an automated, scalable, interpretable meth-
mitral annular e′ velocity. od to perform detailed longitudinal tracking and com-
• We also use the ECG vector to train models to parison of ECGs.
detect and track diseases such as cardiac amyloid, In this work, we develop and test an algorithmic
hypertrophic cardiomyopathy, and pulmonary framework that facilitates scalable analysis of ECG
arterial hypertension. data while preserving interpretable parallels to cardiac
Downloaded from http://ahajournals.org by on September 15, 2019

physiology. We demonstrate this framework on several

examples of cardiac structure and disease, although we

T
he ECG is the most commonly performed cardio- maintain that this approach can be used broadly across
vascular diagnostic procedure, with >100 million the spectrum of cardiac abnormalities. This approach
ECGs obtained annually in the United States,1 aspires to expand the flexibility and scalability of algo-
including use in 21% of annual health examinations2 rithmic ECG analysis, laying the foundation to per-
and 17% of emergency department visits.3 The ECG form a wide range of novel ECG-based tasks, includ-
tracing contains a large amount of information that ing improving accuracy, estimating quantitative cardiac
directly reflects underlying cardiac physiology since its traits, performing longitudinal tracking of serial ECGs,
morphological and temporal features are produced and monitoring disease progression and risk.
from cardiac electrical and structural variations. How-
ever, the existing techniques physicians use to interpret
ECGs4—using sets of rules that were initially established
MATERIALS AND METHODS
The source code for this project, including model weights, is
by empirical, manual review of ECGs from disease co-
available at https://bitbucket.org/rahuldeo/ecgai/.
horts5—consider only a fraction of the total information
available in the ECG.
Although computerized interpretation algorithms Human Subjects Research
for ECGs have existed for decades,6 they have been The University of California, San Francisco (UCSF) Institutional
constrained in that they aim to replicate the rules- Review Board approval was obtained for this study.
based approach to ECG analysis used by human read-
ers. Moreover, the prevailing policy, whether performed Overview: Automated and Interpretable
by humans or algorithms, aims to detect the presence ECG Profiling for Disease Detection,
or absence of disease (eg, left ventricular hypertrophy Tracking, and Discovery
[LVH] or not), evaluating fairly simple criteria on only a We sought to develop an automated, scalable, and interpre-
small subset of the total information contained in the table method to characterize (1) cardiac structure, (2) diastolic
ECG, such as how the height of the R wave in lead aVL function; and (3) detect and track disease using patient-spe-
exceeding 11 mm suggests LVH.7 Such an approach can- cific ECG profiles. Figure 1 demonstrates the analysis pipe-
not readily account for high-level interactions between line, data inputs, and number of ECGs that were used in each

Circ Cardiovasc Qual Outcomes. 2019;12:e005289. DOI: 10.1161/CIRCOUTCOMES.118.005289 September 2019 2

 Tison et al; Automated, Interpretable ECG Patient Profiles

We extracted standard ECG MUSE measurements, as well

as final cardiologist-confirmed ECG diagnostic interpreta-
tions. Data from the UCSF electronic health record were
obtained for relevant patients, including medical diagnoses,
medications, specialty clinic referrals, and echocardiographic
measurements.

Selection of Studies for Model

Development
To facilitate model development, we restricted the analyses to
those ECGs for which the GE/UCSF rhythm interpretation was
normal sinus rhythm.
For cardiac structure models, we searched the UCSF
echocardiographic database for all instances of patients with
echocardiograms and ECGs collected within 30 days of one
another and who had recorded measurements either of left
ventricular mass or left atrial volume. We found 10 082 (Table
I and Figure I in the Data Supplement) and 8289 (Table II and
Figure II in the Data Supplement) studies, respectively, that
met these criteria. For cardiac diastolic function, we per-
formed a similar search and found 4205 instances of patients
with an ECG and a recorded mitral annulus medial e′ value on
echocardiographic within 30 days of each other (Table III and
Figure III in the Data Supplement). There were fewer instances
of lateral e′ values recorded within our database and we thus
focused our efforts on the medial e′ metric.
Figure 1. Workflow for ecgAI project. We selected 4 diseases for which to perform a clinical dem-
The workflow consisted of training an ECG segmentation model and using onstration of automated detection and tracking of disease
a selected group of ECGs to train interpretable models to estimate cardiac using patient ECG profiles: pulmonary arterial hypertension
structure and function and detect and track disease. Concordance with mea-
surements from the GE MUSE system was used for validation of segmentation
(PAH), hypertrophic cardiomyopathy (HCM), cardiac amyloi-
Downloaded from http://ahajournals.org by on September 15, 2019

as well as a filter for segmentation quality. The number of ECGs used for the dosis (CA), and mitral valve prolapse (MVP). We previously
various tasks is indicated in parentheses. For disease detection, a slash sepa- identified the PAH, HCM, and CA patients as part of a parallel
rates the number of cases and control ECGs used. Curved rectangles represent study on developing a computer vision pipeline for automated
training data; ellipses represent algorithms; and standard rectangles represent
other data types. CNN indicates convolutional neural network; HCM, hy-
echocardiographic interpretation.11 Briefly, on chart review
pertrophic cardiomyopathy; HMM, hidden Markov model; MVP, mitral valve HCM patients met guideline-based criteria12 (Figure IV in the
prolapse; NSR, normal sinus rhythm; PAH, pulmonary arterial hypertension; Data Supplement); CA patients had both echocardiographic
LA, left atrium; and LV, left ventricle. evidence of hypertrophy and confirmation of amyloidosis by
biopsy or imaging (Figure V in the Data Supplement); and
step of algorithm development and validation. We termed the PAH patients had an echocardiographic-indication of PAH and
entire approach as ecgAI—referring to artificial intelligence. were on one of 4 PAH specific medications (Figure VI in the
Data Supplement). Patients with MVP were identified by que-
rying the UCSF echocardiographic database for patients with
ECG Data single or bileaflet MVP (Figure VII in the Data Supplement).
We selected a subset of ECGs from the UCSF ECG database Echocardiographic studies were subsequently over-read by a
to train models for estimation of cardiac structure and func- second board-certified cardiologist to confirm the diagnosis.
tion and detection of disease. Standard 12-lead ECG data We selected all ECGs corresponding to these patients that
from 2010 to 2017 was obtained in XML format from the were available in XML format. To build classification models,
UCSF clinical MUSE ECG database (MUSE Version 9.0 SP4, GE we also matched each ECG to up to 5 ECGs matched by age
Healthcare, Wauwatosa, WI). Based on the presence of con- (in 10 years bins), sex, year of study, and race (the patient
current clinical and echocardiographic information which was demographic information for ECGs in our archive has been
used for cardiac structure and disease correlations (described organized in a python dictionary to facilitate the control selec-
below), we selected 36 186 ECGs from the UCSF database, tion process). Patient and study characteristics are described
each of which included the standard 10 seconds of raw ECG in Tables IV through VII in the Data Supplement.
voltage data across each of the 12 individual leads; 60%
of data had been sampled at a frequency of 500 Hz, and
40% had been sampled at 250 Hz. As part of routine clinical A Novel Machine Learning-Based
care, each clinical ECG undergoes initial analysis by the GE Approach to ECG Segmentation
software (MAC 5500 HD, Version 10, Revision F; Marquette To develop novel models with the goal of expanding ECG
12SL; GE Healthcare, Wauwatosa, WI), and the interpretation clinical utility, we needed an efficient way to derive patient-
is subsequently changed or confirmed by a UCSF cardiologist. specific ECG profiles consisting of vectors of uniform length

Circ Cardiovasc Qual Outcomes. 2019;12:e005289. DOI: 10.1161/CIRCOUTCOMES.118.005289 September 2019 3

 Tison et al; Automated, Interpretable ECG Patient Profiles

that capture the variation in ECG voltage over different leads. 5 averaged values were included as 5 additional components
This first required a method to segment ECGs into their stan- in the ECG vector—resulting in a total of 725 components.
dard, interpretable components.
We first trained a convolutional neural network (CNN)-
based model to delineate individual segments within the
ECG-Derived Estimates of Cardiac
ECG. As training data, we downloaded raw ECG voltage data Structure and Function
from 2 sources: 112 ECGs from the Physikalisch-Technische The 725-component ECG patient vector can be used as an
Bundesanstalt diagnostic database13 and 58 ECGs from the input to train models to estimate a variety of cardiac struc-
UCSF database. For each ECG, we extracted a 2-second strip ture and functional estimates. In this article, we examined the
and manually assigned to each 1 ms block 1 of 6 possible ability to estimate continuous measurements of left ventricu-
labels: P wave, PR segment (termination of P wave to start of lar mass (indexed for body surface area), left atrial volume
QRS), QRS complex, ST segment, T wave, and TP segment. (indexed), and mitral annular medial e′ (medial e′) as demon-
Before manual labeling, we performed linear interpolation so stration examples. Anticipating complex interactions among
that, regardless of the initial ECG sampling frequency, 2 sec- input features, as well as heterogeneity among patients,16
onds corresponded to an input vector of length 2000. we used a machine learning algorithm known as a Gradient
We then trained a multilayered neural network to detect Boosted Machine (GBM),17 which is an ensemble regression-
these segments within an ECG. The architecture of our net- tree based technique. Individual GBM models were trained
work was based on the U-net network14 (Figure 2A). Our net- to estimate the 3 continuous structure and function metrics.
work accepted a 12×2000 input array and was composed of We also generated dichotomous measures for each of these,
sequential contracting and expanding paths with a total of treating controls as individuals with values below (for left
32 convolutional layers, 5 max pool layers, and 3 deconvolu- ventricular mass indexed and left atrial volume indexed) or
tional layers. The output of this CNN is a 6×2000 array of ECG above (medial e′) the median value, and cases as individu-
segment classes, identical in length to the input vector, and als above or below the tenth percentile (Tables VIII through
including a probability for each potential segment label at each X in the Data Supplement). Given that we noted occasional
position along the ECG trace (Note I in the Data Supplement). inaccuracy in both our CNN-HMM segmentation model as
Although U-Nets can provide accurate segmentation of well as in the MUSE values, we limited our models to ECGs
objects, they fail to take advantage of the obligate ordering of with substantial agreement (mean difference <10%) across
elements in a typical ECG. We thus trained a hidden Markov the RR, PR, QRS, and QT intervals. This filter merely served as
model (HMM) to accept the output of the U-Net and provide a quick check on the quality of our segmentation, which is
improved segmentation.15 As a final step, we introduced a essential to building an accurate phased patient profile vector.
simple heuristic filter to eliminate implausibly short ECG com- There was no appreciable difference in patient characteristics
Downloaded from http://ahajournals.org by on September 15, 2019

plexes (ie, <10 ms), which was a consistent hallmark of poor for this subset (Tables I through III in the Data Supplement).
HMM performance. The combined CNN-HMM–heuristic filter Models were fit using the GBM function in the R caret pack-
pipeline was run on all ECGs. ECG segmentation was validated age. Tuning parameters were selected in an automated man-
by comparison to manual segmentation and by compari- ner using 3-fold cross validation.
son against measurements a total of 141 864 measurements Accuracy was assessed using 5-fold cross validation,
derived from the GE muse software for 35 466 ECGs (Note II in with area under the receiver operating characteristic curve
the Data Supplement). These ECGs were primarily selected for (AUROC) curves used to evaluate classification tasks and
the various classification and estimation tasks related to cardiac absolute differences (50th, 75th, and 95th percentiles) and
structure, function, and disease detection described below and Bland-Altman18 plots used for continuous measures. Variable
so thus may be biased towards more challenging cases. importance was extracted for each of the 725 features and
averaged over cross-validation runs. To facilitate interpreta-
tion of variable importance rankings, we binned the variable
Deriving Patient-Level ECG Profiles importance scores for the lead-specific voltage values so that
Because ECG waveforms and intervals have corollaries to each segment (eg, QRS) was represented by 5 rather than 20
electrical and structural cardiac physiology, a crucial principle bins. We note that there is still redundancy between voltages
to our approach aimed to create a representation of the raw in highly similar leads (eg, leads I and aVL), but we elected not
ECG data which preserves these features while still decreas- to bin across different leads. Overall, the redundancy in volt-
ing the feature space, making it tractable for analysis by ages within and across leads may reduce the variable impor-
interpretable machine learning algorithms. To achieve this, tance of these features compared with minimally redundant
we developed a 725-component ECG vector representation measures such as the ECG intervals.
derived from segments of the CNN/HMM-segmented ECG.
For 3 ECG segments—the PR interval, the QRS complex, and
the ST-T-wave complex (including both the ST segment and Disease Detection and Tracking
the T wave)—the vector of raw-voltage amplitude from each In addition to quantifying cardiac structure, we also trained
of the 12 leads was resized to 20 samples by linear interpola- GBM models to detect PAH, HCM, CA, and MVP. Separate
tion, averaged across all cardiac cycles, and included as ECG GBM models were trained to output a probability for each dis-
vector components (totaling 720 components). The PR inter- ease based on an input ECG vector using a similar approach as
val, P wave duration, QRS interval, heart rate, and QT intervals outlined above for structure and function. We also assessed
were calculated based on the segmentation boundaries and the discriminative ability of conventional features, such as
averaged across all cardiac cycles and across 12 leads, and the maximum voltage in aVL (for HCM) and maximum voltage in

Circ Cardiovasc Qual Outcomes. 2019;12:e005289. DOI: 10.1161/CIRCOUTCOMES.118.005289 September 2019 4

 Tison et al; Automated, Interpretable ECG Patient Profiles
Downloaded from http://ahajournals.org by on September 15, 2019

Figure 2. ecgAI method of ECG segmentation.

A, Architecture of convolutional neural network used for ECG segmentation. Gray rectangles represent layers with dimensions listed below. The notation for each
layer indicates the size of the input (eg, 2000 ms, initially) by the number of leads (eg, 12) by the number of filters. The size of the filter is specified in the body of
the rectangle. B, Architecture of hidden Markov model (HMM) used after convolutional neural network (CNN) based segmentation. Gray boxes represent states
that are traversed in order in the ECG. C, Example of CNN-HMM output for an ECG. CNN-HMM based classes are shown below the image. The ST and T wave
segments have been combined.

lead V1 (for PAH). To demonstrate the use of this approach to Statistical Methods
track longitudinal changes in disease over time, we selected All analyses were performed using R 3.3.2 or Python 2.7.
all patients who had ECGs in 2 or more years and took the Differences between case and control characteristics for the
median score per patient for each year. Scores were plotted disease detection models were performed using 2-tailed
as a function of year. Wilcoxon-Mann-Whitney, t, or χ2 tests. Only a single value

Circ Cardiovasc Qual Outcomes. 2019;12:e005289. DOI: 10.1161/CIRCOUTCOMES.118.005289 September 2019 5

 Tison et al; Automated, Interpretable ECG Patient Profiles

was taken per patient in these pairwise comparisons. The (ρ=0.77–0.98, Figure 3; Figure VIII in the Data Supple-
AUROC for disease detection models were computed using ment). The HMM and, to a lesser extent, the heuristic fil-
held-out values from 5-fold cross validation with the help of ter, contributed substantially to the accuracy of interval
the pROC and hmeasure packages in R. CIs for the AUROC estimation (Table XIII in the Data Supplement).
were generated by the nonparametric method of Delong,19
as implemented in the pROC package. The only predictor for
these models was the patient-level disease score, as output ecgAI Performance to Quantify Cardiac
by the GBM model.
Structure and Function
CNNs were developed using the TensorFlow Python pack-
age.20 Signal manipulation (such as linear interpolation for In contrast to the binary detection of cardiac structural
resizing) was performed using scikit-image.21 diagnosis on ECG using existing methods, the ecgAI
approach enables estimation of the severity of struc-
tural abnormalities using continuous metrics. For the 3
RESULTS demonstration examples of cardiac structural and func-
tion, we used echocardiographic measurements for
Validation of ecgAI Machine Learning-
training and validation. Median absolute deviation of
Based ECG Segmentation ecgAI predictions against reference echocardiographic
Our ecgAI segmentation pipeline (Figure 2A and 2B) was measurements varied by structure: the lowest deviation
trained on 170 manually segmented ECGs and deployed was for left ventricular mass indexed (16.5%), inter-
on 36 186 sinus rhythm ECGs (Figure 1). ECG segmenta- mediate deviation was for mitral annulus medial e′
tion forms the basis for subsequent steps in the ecgAI (19.1%), and the greatest deviation was for left atrial
pipeline and example output from the ecgAI segmenta- volume indexed (22.9%; Table XI in the Data Supple-
tion model is shown in Figure 2C, with every time-step ment). For all 3 structural measurements, there was
along the ECG tracing being classified as belonging to a tendency to overestimate low values and underesti-
one of the 6 segments (illustrated in the Figure by sepa- mate high values (Figure 4A and 4B; Figure IX in the
rate colors). The ecgAI segmentation performed reason- Data Supplement), suggesting a more limited dynamic
ably well versus manual annotations as demonstrated range for ECG compared with echocardiography. When
by the IoU metrics of 91±3 (P wave), 85±2 (PR seg- the continuous measurements for the cardiac structures
ment), 94±4 (QRS complex), 88±3 (ST segment), 91±3 were dichotomized, the model demonstrated strong
(T wave), and 92±5 (TP segment). As a second indirect
Downloaded from http://ahajournals.org by on September 15, 2019

discrimination for both LVH and diastolic dysfunction

validation of segmentation performance, standard ECG with AUROCs of 0.87 (95% CI, 0.86–0.89) and 0.84
interval measurements were calculated based on ecgAI (95% CI, 0.82–0.86), respectively (Figure 4C and 4D).
segmentation on 35 466 ECGs not included in the train- Left atrial enlargement had a much lower AUROC of
ing set and compared against the reference MUSE val- 0.62 (95% CI, 0.60–0.64), most likely reflecting a fail-
ues (Tables XI and XII in the Data Supplement), overall ure of the ECG to correctly estimate large atrial volumes
demonstrating good agreement with MUSE calculated (Figure IX in the Data Supplement).
intervals. Intervals from ecgAI-measurements dem- The ecgAI approach also enables the identification
onstrated a strong correlation with those from MUSE of which ECG components (waveform voltages and

Figure 3. Comparison of ecgAI (hidden

Markov model [HMM]+convolutional
neural network [CNN]) derived measure-
ments and MUSE/University of California,
San Francisco (UCSF) values for 4 commonly
reported ECG measurements.
The scatterplot depicts 35 466 comparisons. The
line y=x is drawn to help identify any bias. The
unit for heart rate is beats per minute while that
of the other 3 metrics is milliseconds.

Circ Cardiovasc Qual Outcomes. 2019;12:e005289. DOI: 10.1161/CIRCOUTCOMES.118.005289 September 2019 6

 Tison et al; Automated, Interpretable ECG Patient Profiles
Downloaded from http://ahajournals.org by on September 15, 2019

Figure 4. Estimating cardiac structure and function using patient-level ECG profiles.
Bland-Altman plots comparing estimation of left ventricular mass index (LVMi; A) and mitral annulus medial e′ (B) values using ECG alone compared with echo-
derived values. Number of studies depicted in comparison is shown. Orange, red, and blue dashed lines delineate the central 50%, 75%, and 95% of patients, as
judged by difference between automated and manual measurements. The solid gray line indicates the median. Receiver operating characteristic (ROC) curves for
classification models for left ventricular hypertrophy (C) and diastolic dysfunction (D). The area under the ROC curve (AUROC) is indicated. Variable importance for
LVMi (E) and mitral annulus e′ (F) estimation models. The predictors most important for each model are highlighted with the relative importance indicated by the
shading (white to blue). Informative intervals are depicted below the plot while lead-specific segments of the ECG are highlighted on the voltage trace.

intervals from the 725-component patient-level ECG V1 (1.8, segments 8–12), and heart rate (1.2). For left
profile) most strongly contributed to classification for atrial volume indexed, top predictors were QT dura-
each cardiac structural abnormality (Figure 4E and F; tion (4.6), P wave duration (4.5), QRS duration (1.4),
Table XIV in the Data Supplement). For left ventricular PR duration (1.3), and the middle portion of the QRS
mass index, QRS duration was the strongest predictor from lead V6 (0.97).
with a variable score of 4.0, followed by P wave dura-
tion (3.3), QT duration (1.7), the middle portion of the
QRS from lead V3 (1.5, segments 8–12 out of a total of
ecgAI Performance for Cardiac Disease
20), and the middle portion of the ST-T complex from Detection
lead V1 (1.3, segments 12–16; Figure 4E; Table XIV in In addition to quantifying cardiac structure, we applied
the Data Supplement). Collectively, these reflect many ecgAI toward classification of 4 example diseases,
of the classic criteria for LVH,7 while also highlighting accompanied by the discovery of ECG predictors for
potential new ECG-based predictors of LVH. each disease (Figure 5; Figure XI in the Data Supple-
For medial e′ the strongest ECG predictors were ment for precision-recall curves). The strongest discrimi-
PR duration (3.1), QT duration (2.9), P wave duration nation was observed for a model for PAH, which had an
(2.4), the middle portion of the ST-T complex from lead AUROC of 0.94 (95% CI, 0.93–0.95). Key predictors for

Circ Cardiovasc Qual Outcomes. 2019;12:e005289. DOI: 10.1161/CIRCOUTCOMES.118.005289 September 2019 7

 Tison et al; Automated, Interpretable ECG Patient Profiles
Downloaded from http://ahajournals.org by on September 15, 2019

Figure 5. Detecting disease using patient-level ECG profiles.

Receiver operating characteristic (ROC) curves (with area under the ROC curve [AUROC] indicated) for disease detection models for pulmonary arterial hyperten-
sion (PAH; A), hypertrophic cardiomyopathy (HCM; C), and cardiac amyloid (CA; E). Corresponding variable importance plots (B, D, F, and H) with coloring as in
Figure 4. Violin plots indicating distribution of the top predictive feature in cases and controls for PAH (G), HCM (H), and CA (I). Precision-recall curves are depicted
in Figure II in the Data Supplement.

PAH included the middle portion of QRS from lead V1 and 5B; Table XV in the Data Supplement). The GBM
(variable score =4.5, segments 8–12), reflecting a tall R′ model for PAH was better than one constructed solely
(Figure 5G, P<2×10−16), followed by the latter and mid- using the maximum height of the QRS complex in V1
dle portions of the QRS from lead V1 (1.6, segments (as a proxy for right ventricular hypertrophy), which
12–16; 1.4 segments 12–16), reflecting a deep S wave; yielded an AUROC of 0.77 (95% CI, 0.73–0.78).
and the early portion of the P-PR complex from lead V3 HCM had the next strongest discrimination with an
(0.9, segments 4–8) and aVR (0.9, segments 4–8), pre- AUROC of 0.91 (95% CI, 0.90–0.92). The strongest
sumably reflecting right atrial enlargement (Figure 5A predictors of HCM were the latter portion of the ST-T

Circ Cardiovasc Qual Outcomes. 2019;12:e005289. DOI: 10.1161/CIRCOUTCOMES.118.005289 September 2019 8

 Tison et al; Automated, Interpretable ECG Patient Profiles

complex from lead V1 (3.8, segments 12–16), which

can be markedly deeper in some patients with HCM
(Figure 5H, P<2×10−16), the P wave duration (3.5), QT
duration (2.7), PR duration (2.4), and the middle por-
tion of the QRS from lead aVR (1.3, segments 12–16;
Figure 5C and 5D; Table XV in the Data Supplement).
The GBM model for HCM was superior to one con-
structed solely using the maximum height of the QRS
complex in lead aVL (as a proxy for LVH), which yielded
an AUROC of 0.69 (95% CI, 0.67–0.71).
CA had an AUROC of 0.86 (95% CI, 0.82–0.89),
and the strongest predictors in this model were the
early portion of the QRS from lead aVR (3.0, segments
4–8), which is blunted in voltage in CA patients (Fig-
ure 5I, P=3×10−7), QRS duration (1.3), the middle and
early portions of the QRS from lead I (1.2, segments
8–12; 1.1, segments 4–8), and the earliest portion of
the QRS from lead V1 (1.1, segments 0–4; Figure 5E
and 5F; Table XV in the Data Supplement).
The MVP showed the weakest discrimination, with
an AUROC of 0.77 (95% CI, 0.76–0.78; Figure III in
the Data Supplement), a disease not known to strongly
impact ECG morphology. The top predictors for MVP
included PR duration (3.3), the early portion of the QRS
from lead V2 (1.2, segments 4–8), the earliest portion
of the QRS from lead V3 (1.2, segments 0–4), P wave
duration (1.1), and QT duration (0.97; Figure X and
Table XV in the Data Supplement).
Downloaded from http://ahajournals.org by on September 15, 2019

Serial ECGs Analysis With ecgAI to

Perform Within-Patient Disease Tracking Figure 6. ECG-profile based models can be used to track changes in
patient ECGs in pulmonary arterial hypertension (PAH).
By applying ecgAI to serial ECGs of patients with PAH, A, Scores for PAH detection for individual patients with measurements for 2
we obtained a progression of scores over time corre- or more years. A median of all scores for each year is computed, and lines are
sponding to the degree to which the model estimated drawn connecting scores for different years for each patient. The blue dashed
line indicates a score threshold with 90% specificity and 80% sensitivity for a
likelihood of PAH based on ECG features (Figure 6A). diagnosis of PAH. Purple, red, and yellow lines highlight score trajectories for 3
The dashed blue line represents the PAH score at which patients with dramatic variation in scores, crossing this threshold. B, Variation
PAH is identified with 80% sensitivity and 90% speci- in ECG patterns for leads I and V1 from 2010 to 2017 for patient highlighted
in purple in A. Over time, as the PAH score increases, the QRS axis swings
ficity. Patients typically have scores that remain with a progressively rightward (lead I), the P wave height grows, and the R′ wave in
narrow range, but there are some exceptions—and we lead V1 increases in size.
highlight the 3 most prominent ones. Figure 6B shows
a time course of ECG tracings for the individual depict-
ed by the purple trajectory in Figure 6A). In 2010 and tor in lead I. In contrast (and for unclear reasons), the
2011, ECG tracings do not have any marked abnormali- subsequent low PAH score ECG tracings for both indi-
ties. In 2015 and 2017, ECG tracings appear increas- viduals appear substantially different and more normal,
ingly abnormal, with a prominent R wave and T wave with a decrease in R wave prominence in V1 and nor-
inversion in lead V1, and QRS changes and a tall promi- malization of the QRS in lead I (Figure XIIB and XIIC in
nent P wave in lead I. These progressive ECG changes the Data Supplement). The GBM PAH score thus tracks
over time correspond with the increasing PAH scores well with visible morphological change in the ECG.
from 2010 to 2017.
Two other patients (trajectories colored in red and
yellow) had precipitous decreases followed by subse- DISCUSSION
quent increases in score (Figure XII in the Data Supple- There has been a dramatic increase in publications
ment). In both cases, the ECG tracings from the high applying machine learning methods to perform rou-
PAH score year appear abnormal, featuring prominent tine diagnostic tasks in medicine. Most of these have
R waves in V1 and a more negatively directed QRS vec- emphasized matching or even outperforming practic-

Circ Cardiovasc Qual Outcomes. 2019;12:e005289. DOI: 10.1161/CIRCOUTCOMES.118.005289 September 2019 9

 Tison et al; Automated, Interpretable ECG Patient Profiles

ing physicians, whether it be for interpreting retino- Although we used a machine learning approach to
grams,22 skin disorders,23 bone x-rays,24 or heart rhythm segment ECGs in this work, other methods, including
abnormalities.24 Here, we outline a machine learning any of the existing heuristic-based segmentation algo-
approach to ECG interpretation that differs crucially rithms, could also be used to derive patient-level ECG
from these prior works in emphasizing 3 facets which profiles.31 A limitation of our machine learning ECG
are crucial when applying machine learning toward segmentation pipeline is that they are currently only
medical applications: (1) the use of machine learning optimized to analyze ECGs in normal sinus rhythm.
to extend the utility of a diagnostic tool to applications Similarly, we limited our cardiac disease and structure
beyond what would be possible by human readers; (2) models to segmented ECGs with substantial agreement
the focus on eliciting interpretable features which can with clinical measurements. While these limitations
be used to both justify an automated diagnosis within of our existing pipeline would need to be optimized
clinical care and inspire new research on physiological before clinical deployment, we made these decisions to
correlates of disease; and (3) the demonstration of a demonstrate the performance of our approach of using
flexible framework that permits estimation or classifica- ECG profiles for this proof of concept. An additional
tion for a broad range of cardiac metrics and diseases. limitation is that our data, although large in scale, is
Our machine learning approach not only outperforms derived from a single medical center.
existing rule-based binary diagnostic criteria for ECG Our primary intended applications of this work
diagnosis against which it was compared but also it all relate to augmenting clinical practice, rather than
more importantly expands the utility of the ECG as a replacing what is already performed by skilled practi-
clinical tool beyond present human capability. We dem- tioners. Patients with uncommon diseases, such as
onstrate the ability to estimate continuous metrics of PAH, HCM, and cardiac amyloidosis, all of which have
cardiac structure and function while also performing approved or emerging therapies, would benefit from
disease detection and longitudinal tracking of predicted early detection and referral to a specialty center. Com-
disease. We think this constitutes a new paradigm in bining low-cost testing—potentially even with mobile
ECG analysis by expanding the clinical inferences that ECG devices—with an automated detection algorithm
can be drawn from an ECG, increasing its potential util- can help recognize and triage these individuals. Of
ity to novel applications. course, criteria such as precision (ie, positive predictive
We think that the enormous potential of applying value) and more broadly, a decision analysis regarding
machine learning to medicine must lie in its ability to illu- the costs of a false positive or negative result, should
Downloaded from http://ahajournals.org by on September 15, 2019

minate patterns across large quantities of data in a way come into play when evaluating the viability of any
that preserves clinical interpretability, both to maintain such approach.31 The ECG is also currently not used as
physician and patient agency in decision-making and to a quantitative detection of disease progression in an
enable knowledge discovery. In the case of ECG-disease individual, although several studies suggest this is fea-
correlates, there is considerable evidence that previous- sible.32–34 Our approach provides an additional method
ly recognized ECG predictors represent only a fraction by which to achieve quantitative tracking of disease
of informative features of any disease,25,26 making the progression which benefits from being automatic and
case for data-driven discovery of novel ECG correlates, not limited to predefined disease criteria. ECG fea-
which our approach uniquely enables. Most prior dis- tures that co-occur with hypertension35 and even obe-
ease-focused studies have highlighted the association sity,36 diabetes mellitus,37 coronary artery disease,38 and
of various ECG features with disease status, rather than aging39 may occur at variable rates in different individu-
describing the global discrimination performance24,27– als and artificial intelligence-assisted monitoring of ECG
30
—limiting our ability to directly compare our perfor- features using our approach may provide a low-cost
mance. In work bearing the most similarity to ours,24 noninvasive window into systemic processes that can
the investigators used various ECG features to iden- be slowed with existing or emerging therapies.40 Such
tify HCM patients, however, their focus remained the rates of minute change in ECG tracings would other-
optimization of predictive performance rather than wise be challenging to assess with the human eye and
enabling clinical interpretability. In our study, the ECG- will need automated systems for the development and
based features identified as most strongly contributing the validation of quantitative models.
to prediction for each disease have clear physiological Such quantitative patient tracking using the output
parallels—such as ECG correlates of right ventricular of multidimensional models is not performed routinely
hypertrophy in PAH and myocardial infiltration in CA— for ECGs or even echocardiographic, in part, because of
which conforms to our expectations based on patho- long-standing fears that it might obscure the diagnostic
physiology, and increases confidence in and acceptance process.41,42 With the current widespread availability of
of model performance. Furthermore, the novel predic- digital data, we strongly believe such concerns should
tors identified by our models may provide inroads into be revisited, both for the benefit of the physician and
future investigation. patient. To this end, a primary motivation of this work is

Circ Cardiovasc Qual Outcomes. 2019;12:e005289. DOI: 10.1161/CIRCOUTCOMES.118.005289 September 2019 10

 Tison et al; Automated, Interpretable ECG Patient Profiles

to demonstrate how we can extract much more knowl- 8. Lipton ZC. The mythos of model interpretability. Communications of the
ACM. 2018;61:36–43.
edge from our current low-cost input data, all in an 9. Attia ZI, Kapa S, Lopez-Jimenez F, McKie PM, Ladewig DJ, Satam
automated manner, and yet remain transparent to phy- G, Pellikka PA, Enriquez-Sarano M, Noseworthy PA, Munger TM,
sicians, patients, and researchers about the provenance Asirvatham SJ, Scott CG, Carter RE, Friedman PA. Screening for cardiac
contractile dysfunction using an artificial intelligence-enabled electrocar-
of these insights. diogram. Nat Med. 2019;25:70–74. doi: 10.1038/s41591-018-0240-2
10. Hannun AY, Rajpurkar P, Haghpanahi M, Tison GH, Bourn C, Turakhia
MP, Ng AY. Cardiologist-level arrhythmia detection and classification in
ARTICLE INFORMATION ambulatory electrocardiograms using a deep neural network. Nat Med.
2019;25:65–69. doi: 10.1038/s41591-018-0268-3
Received October 22, 2018; accepted July 15, 2019.
11. Zhang J, Gajjala S, Agrawal P, Tison GH, Hallock LA, Beussink-
The Data Supplement is available at https://www.ahajournals.org/doi/
Nelson L, Lassen MH, Fan E, Aras MA, Jordan C, Fleischmann KE,
suppl/10.1161/CIRCOUTCOMES.118.005289.
Melisko M, Qasim A, Shah SJ, Bajcsy R, Deo RC. Fully automated echocar-
diogram interpretation in clinical practice. Circulation. 2018;138:1623–
Correspondence 1635. doi: 10.1161/CIRCULATIONAHA.118.034338
12. Gersh BJ, Maron BJ, Bonow RO, Dearani JA, Fifer MA, Link MS, Naidu SS,
Rahul C. Deo, MD, PhD, One Brave Idea Science Innovation Center, Division of
Nishimura RA, Ommen SR, Rakowski H, Seidman CE, Towbin JA, Udelson JE,
Cardiovascular Medicine, Brigham and Women’s Hospital, 360 Longwood Ave,
Yancy CW; American College of Cardiology Foundation/American Heart
Box 201, Boston, MA 02215. Email rdeo@bwh.harvard.edu
Association Task Force on Practice Guidelines. 2011 ACCF/AHA guideline
for the diagnosis and treatment of hypertrophic cardiomyopathy: a report
Affiliations of the American College of Cardiology Foundation/American Heart As-
sociation Task Force on Practice Guidelines. Developed in collaboration
Division of Cardiology, Department of Medicine (G.H.T., F.N.D., R.C.D.), Bakar
Institute for Computational Health Sciences (G.H.T., R.C.D.), Center for Digital with the American Association for Thoracic Surgery, American Society of
Health Innovation (G.H.T., R.C.D.), Cardiovascular Research Institute (J.Z.), and Echocardiography, American Society of Nuclear Cardiology, Heart Failure
Institute for Human Genetics (R.C.D.), University of California, San Francisco Society of America, Heart Rhythm Society, Society for Cardiovascular An-
Department of Electrical Engineering and Computer Science, University of Cali- giography and Interventions, and Society of Thoracic Surgeons. J Am Coll
fornia at Berkeley, CA (J.Z., R.C.D.). California Institute for Quantitative Biosci- Cardiol. 2011;58:e212–e260. doi: 10.1016/j.jacc.2011.06.011
ences, San Francisco, CA (R.C.D.). One Brave Idea and Division of Cardiovascu- 13. Goldberger AL, Amaral LA, Glass L, Hausdorff JM, Ivanov PC, Mark RG,
lar Medicine, Brigham and Women's Hospital, Boston, MA (R.C.D.). Mietus JE, Moody GB, Peng CK, Stanley HE. Physio bank, physiotoolkit,
and physionet: components of a new research resource for complex
physiologic signals. Circulation. 2000;101:E215–E220. doi: 10.1161/01.
Acknowledgments cir.101.23.e215
We acknowledge the assistance of Edward Mahoney and John Cooper in ac- 14. Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks For Bio-
cessing records from the MUSE server. We thank Dr Pulkit Agrawal for advice medical Image Segmentation. Accessed April 1, 2017. https://arxiv.org/
on the ECG segmentation model. abs/1505.04597.
15. Rabiner LR. A tutorial on hidden Markov models and selected applications
in speech recognition. Proc IEEE. 1989;77:257–286.
Downloaded from http://ahajournals.org by on September 15, 2019

Sources of Funding 16. Deo RC. Machine learning in medicine. Circulation. 2015;132:1920–
The funding for this work was through National Institutes of Health (NIH)/Na- 1930. doi: 10.1161/CIRCULATIONAHA.115.001593
tional Heart, Lung, and Blood Institute (NHLBI) DP2 HL123228 to Dr Deo and 17. Friedman JH. Greedy function approximation: a gradient boosting ma-
NIH/NHLBI K23HL135274 to Dr Tison. chine. Ann Stat. 2001;29:1189–1232.
18. Bland JM, Altman DJ. Regression analysis. Lancet. 1986;1:908–909. doi:
10.1016/s0140-6736(86)91008-1
Disclosures 19. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under
None. two or more correlated receiver operating characteristic curves: a non-
parametric approach. Biometrics. 1988;44:837–845.
20. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S,
Irving G, Isard M, Kudlur M, Levenberg J, Monga R, Moore S, Murray DG,
REFERENCES Steiner B, Tucker P, Vasudevan V, Warden P, Wicke M, Yuan Y, Zheng X. Ten-
1. Drazen E, Mann N, Borun R, Laks M, Bersen A. Survey of computer-assist- sorflow: a system for large-scale machine learning. OSDI. 2016;16:265–283.
ed electrocardiography in the United States. J Electrocardiol. 1988;21(sup 21. van der Walt S, Schönberger JL, Nunez-Iglesias J, Boulogne F, Warner JD,
pl):S98–104. Yager N, Gouillart E, Yu T; scikit-image contributors. Scikit-image: image
2. Bhatia RS, Bouck Z, Ivers NM, Mecredy G, Singh J, Pendrith C, Ko DT, processing in Python. PeerJ. 2014;2:e453. doi: 10.7717/peerj.453
Martin D, Wijeysundera HC, Tu JV, Wilson L, Wintemute K, Dorian P, 22. Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A,
Tepper J, Austin PC, Glazier RH, Levinson W. Electrocardiograms in low- Venugopalan S, Widner K, Madams T, Cuadros J, Kim R, Raman R, Nelson PC,
risk patients undergoing an annual health examination. JAMA Intern Mega JL, Webster DR. Development and validation of a deep learning
Med. 2017;177:1326–1333. doi: 10.1001/jamainternmed.2017.2649 algorithm for detection of diabetic retinopathy in retinal fundus photo-
3. Pitts SR, Niska RW, Xu J, Burt CW. National hospital ambulatory medical graphs. JAMA. 2016;316:2402–2410. doi: 10.1001/jama.2016.17216
care survey: 2006 emergency department summary. Natl Health Stat Re- 23. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep
port. 2008;7:1–38. convolutional neural networks. Commun ACM. 2017;60:84–90.
4. Schläpfer J, Wellens HJ. Computer-interpreted electrocardiograms: bene- 24. Rahman QA, Tereshchenko LG, Kongkatong M, Abraham T, Abraham MR,
fits and limitations. J Am Coll Cardiol. 2017;70:1183–1192. doi: 10.1016/j. Shatkay H. Utilizing ECG-based heartbeat classification for hypertrophic
jacc.2017.07.723 cardiomyopathy identification. IEEE Trans Nanobiosci. 2015;14:505–512.
5. Blackburn H, Keys A, Simonson E, Rautaharju P, Punsar S. The electro- doi: 10.1109/TNB.2015.2426213
cardiogram in population studies. A classification system. Circulation. 25. Qi Z, Wu M, Fu Y, Huang T, Wang T, Sun Y, Feng Z, Li C. Palmitic acid curcum-
1960;21:1160–1175. doi: 10.1161/01.cir.21.6.1160 in ester facilitates protection of neuroblastoma against oligomeric Aβ40
6. Pipberger HV, Stallman FW, Berson AS. Automatic analysis of the P-QRS-T insult. Cell Physiol Biochem. 2017;44:618–633. doi: 10.1159/000485117
complex of the electrocardiogram by digital computer. Ann Intern Med. 26. Nahar J, Imam T, Tickle KS, Chen YP. Computational intelligence for heart
1962;57:776–787. doi: 10.7326/0003-4819-57-5-776 disease diagnosis: a medical knowledge driven approach. Expert Syst
7. Casale PN, Devereux RB, Kligfield P, Eisenberg RR, Miller DH, Chaudhary Appl. 2013;40:96–104.
BS, Phillips MC. Electrocardiographic detection of left ventricular hyper- 27. Cheng Z, Zhu K, Tian Z, Zhao D, Cui Q, Fang Q. The findings of electrocar-
trophy: development and prospective validation of improved criteria. J Am diography in patients with cardiac amyloidosis. Ann Noninvasive Electro-
Coll Cardiol. 1985;6:572–580. doi: 10.1016/s0735-1097(85)80115-7 cardiol. 2013;18:157–162. doi: 10.1111/anec.12018

Circ Cardiovasc Qual Outcomes. 2019;12:e005289. DOI: 10.1161/CIRCOUTCOMES.118.005289 September 2019 11

 Tison et al; Automated, Interpretable ECG Patient Profiles

28. Dumont CA, Monserrat L, Soler R, Rodríguez E, Fernandez X, Peteiro J, 35. Okin PM, Devereux RB, Jern S, Kjeldsen SE, Julius S, Nieminen MS,
Bouzas A, Bouzas B, Castro-Beiras A. Interpretation of electrocardiographic Snapinn S, Harris KE, Aurup P, Edelman JM, Wedel H, Lindholm LH,
abnormalities in hypertrophic cardiomyopathy with cardiac magnetic res- Dahlöf B; LIFE Study Investigators. Regression of electrocardiographic left
onance. Eur Heart J. 2006;27:1725–1731. doi: 10.1093/eurheartj/ehl101 ventricular hypertrophy during antihypertensive treatment and the predic-
29. Lyon A, Mincholé A, Martínez J, Laguna P, Rodriguez B. Computational tion of major cardiovascular events. JAMA. 2004;292:2343–2349. doi:
techniques for ECG analysis and interpretation in light of their contribu- 10.1001/jama.292.19.2343
tion to medical advances. J Roy Soc Interface. 2018;15:20170821. 36. Simonson E, Keys A. The effect of age and body weight on the elec-
30. Fikrle M, Paleček T, Kuchynka P, Němeček E, Bauerová L, Straub J, Ryšavá R. trocardiogram of healthy men. Circulation. 1952;6:749–761. doi:
Cardiac amyloidosis: a comprehensive review. Cor Vasa. 2013;55:e60– 10.1161/01.cir.6.5.749
e75. 37. Kalliomaki JL, Mollerstrom J, Sollberger A. Observations on the standard
31. Greenland S. The need for reorientation toward cost-effective prediction: electrocardiogram in diabetics with clinically normal hearts. Acta Med
comments on ‘Evaluating the added predictive ability of a new marker: Scand. 1956;156:211–220.
from area under the ROC curve to reclassification and beyond’ by M. J. 38. Simonson E. The effect of age on the electrocardiogram. Am J Cardiol.
Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). Stat Med. 1972;29:64–73. doi: 10.1016/0002-9149(72)90417-1
2008;27:199–206. doi: 10.1002/sim.2995 39. Bachman S, Sparrow D, Smith LK. Effect of aging on the electrocardiogram.
32. Tonelli AR, Baumgartner M, Alkukhun L, Minai OA, Dweik RA. Electro- Am J Cardiol. 1981;48:513–516. doi: 10.1016/0002-9149(81)90081-3
cardiography at diagnosis and close to the time of death in pulmonary 40. Levy D, Salomon M, D’Agostino RB, Belanger AJ, Kannel WB. Prognos-
arterial hypertension. Ann Noninvasive Electrocardiol. 2014;19:258–265. tic implications of baseline electrocardiographic features and their se-
doi: 10.1111/anec.12125 rial changes in subjects with left ventricular hypertrophy. Circulation.
33. Henkens IR, Gan CT, van Wolferen SA, Hew M, Boonstra A, Twisk JWR, 1994;90:1786–1793. doi: 10.1161/01.cir.90.4.1786
Kamp O, van der Wall EE, Schalij MJ, Vonk Noordegraaf A, Vliegen HW. 41. Brohet CR, Robert A, Derwael C, Fesler R, Stijns M, Vliers A, Braasseur LA.
ECG monitoring of treatment response in pulmonary arterial hypertension Computer interpretation of pediatric orthogonal electrocardiograms: sta-
patients. Chest. 2008;134:1250–1257. doi: 10.1378/chest.08-0461 tistical and deterministic classification methods. Circulation. 1984;70:255–
34. Waligóra M, Tyrka A, Podolec P, Kopeć G. Corrigendum to “ECG markers 262. doi: 10.1161/01.cir.70.2.255
of hemodynamic improvement in patients with pulmonary hypertension”. 42. Kors JA, van Bemmel JH. Classification methods for computerized interpre-
Biomed Res Int. 2018;2018:1541709. doi: 10.1155/2018/1541709 tation of the electrocardiogram. Methods Inf Med. 1990;29:330–336.
Downloaded from http://ahajournals.org by on September 15, 2019

Circ Cardiovasc Qual Outcomes. 2019;12:e005289. DOI: 10.1161/CIRCOUTCOMES.118.005289 September 2019 12