Comparison of an Informally Structured

Triage System, the Emergency Severity

Index, and the Manchester Triage System to
Distinguish Patient Priority in the Emergency
Department
Marja N. Storm-Versloot, RN, MSc, Dirk T. Ubbink, MD, PhD, Johan Kappelhof, MSc,
and Jan S. K. Luitse, MD

Abstract
Objectives: The objective was to compare the validity of an existing informally structured triage system
with the Emergency Severity Index (ESI) and the Manchester Triage System (MTS).
Methods: A total of 900 patients were prospectively triaged by six trained triage nurses using the three
systems. Triage ratings of 421 (48%) patients treated only by emergency department (ED) physicians
were compared with a reference standard determined by an expert panel. The percentage of undertriage,
the sensitivity, and the specificity for each urgency level were calculated. The relationship between
urgency level, resource use, hospitalization, and length of stay (LOS) in the 900 triaged patients was
determined.
Results: The percentage of undertriage using the ESI (86 of 421; 20%) was significantly higher than in
the MTS (48 of 421; 11%). When combining urgency levels 4 and 5, the percentage of undertriage was
8% for the informally structured system (ISS), 14% for the ESI, and 11% for the MTS. In all three systems, sensitivity for all urgency levels was low, but specificity for levels 1 and 2 was high (>92%). Sensitivity and specificity were significantly different between ESI and MTS only in urgency level 4. In all 900
patients triaged, urgency levels across all systems were associated with significantly increased resource
use, hospitalization rate, and LOS.
Conclusions: All three triage systems appear to be equally valid. Although the ESI showed the highest
percentage of undertriage and the ISS the lowest, it seems preferable to use a verifiable, formally
structured triage system.
ACADEMIC EMERGENCY MEDICINE 2011; 18:822829 2011 by the Society for Academic Emergency
Medicine

atients arriving at the emergency department

(ED) are often confronted with long waiting
times. These may be caused by arrival volumes,
order of arrival, or clinical urgency. Although EDs
always use some form of triage, either formal or
From the Departments of Surgery (MNS, DTU), Quality
Assurance & Process Innovation (DTU), and Emergency
Medicine (JK, JSKL), Academic Medical Center, Amsterdam,
The Netherlands.
Received November 1, 2010; revisions received February 9 and
February 23, 2011; accepted February 27, 2011.
The authors have no relevant financial information or potential
conflicts of interest to disclose.
Supervising Editor: Clifton W. Callaway, MD, PhD.
Address for correspondence and reprints: Marja N. StormVersloot, RN, MSc; e-mail: m.n.storm@amc.uva.nl.

822

ISSN 1069-6563
PII ISSN 1069-6563583

informal,1 overcrowding of EDs makes accurate triaging

essential to avoid delays in critical patient care, which
may result in long waiting times and poor outcomes.2
There are a number of three-level systems that rank
patients as emergent, urgent, or nonurgent. However, as
five-level triage systems have been shown to be more
reliable and valid than three-level systems, they are likely
the systems of choice.3,4
Worldwide, there are four five-level triage systems in
use: the Australasian Triage Scale, the Canadian Triage
and Acuity Scale, the Manchester Triage System (MTS),
and the Emergency Severity Index (ESI). The MTS and
the ESI are the most commonly used in the Netherlands.5
The MTS was developed in the United Kingdom and
is widely used.6 The MTS contains 52 flow charts, each
representing a presenting complaint. The presenting

2011 by the Society for Academic Emergency Medicine

doi: 10.1111/j.1553-2712.2011.01122.x

ACADEMIC EMERGENCY MEDICINE August 2011, Vol. 18, No. 8

complaint determines which flow chart should be followed. Each flow chart is based on a five-step decision
process that uses discriminators at each step to assign
patients to one of the five triage categories.6 A color
indicates the level of urgency and its associated
maximum waiting time: red = immediate care by a
physician; orange = 10 minutes; yellow = 60 minutes;
green = 2 hours; and blue = 4 hours. Interobserver and
intraobserver agreement on the MTS has been found to
be substantial to excellent.7,8 Under- and overtriage as
determined by an expert panel ranges from 5% to
25%.7,8 Validity has been tested in several studies, however, only in children and in patients with chest pain.911
Therefore, the overall validity of MTS in daily clinical
practice is not supported in the literature.5
The ESI system was developed in Boston, Massachusetts. This system uses one algorithm, with ratings
ranging from level 1 (the most acutely ill patients) to
level 5 (the least resource-intensive patients). The triage
nurse estimates the number of resources needed to discharge the patient from the ED in those patients who
do not meet ESI level 1 or 2 criteria.12 The ESI system
is valid for both children and adults and has a high
interobserver agreement.8,13,14 When triage urgency
levels estimated by nurses were compared with the real
urgency level, version 3 of the ESI had an undertriage
rate of 9% and an overtriage rate of 11%.14
The two triage systems have been designed for different purposes. The MTS is meant to place patients in
order of priority and to assure that patients do not have
to wait longer than is safe, given the presenting complaint. The ESI integrates acuity and estimated resource
consumption to determine treatment priority. Both ESI
and MTS seem to be useful, but to date no studies have
compared the validity of these systems within the same
patient mix, nor to an informally structured triage practice. We defined validity as the agreement of classifications by the triage systems with a reference standard
and with actual resource utilization. In this prospective
observational comparative study, we determined and
compared the percentage of undertriage; the validity of
both structured triage systems and a local informally
structured triage system (ISS); and their relation to
resource use, hospital admission, and length of stay
(LOS).
METHODS
Study Design
This was a prospective, single-center, observational
comparative study combined with a retrospective chart
review to determine the validity of the ISS, the ESI, and
the MTS triage systems. The study was explained to
patients, and all gave oral informed consent. Our local
institutional review board waived the requirement for
written informed consent.
Study Setting and Population
This study was conducted between November and
December 2005 at an urban tertiary care academic
teaching hospital with a Level I trauma center. The ED
sees almost 31,000 patients annually, of whom approximately 1,000 (3%) are seen in a trauma room. Almost

www.aemj.org

823

71% of the patients were self-referrals, while 29% of

the patients were referred by a general physician (GP)
to a specialist. The overall admission rate was approximately 18%, and 15% of the patients were younger
than 15 years.
In 2005, when the study began, no formally structured triage system was in use on our ED, but rather
an ISS was in place. This system was based on clinical
expertise, but not on explicit criteria and information.
When patients were registered, the patients appearance and presenting complaints were judged, and the
nurse or receptionist implicitly answered the question:
Could this patient wait safely before being seen?
Options were: patient could not wait at all, patient
should be seen as soon as possible, or patient could
wait.
For the purpose of this study, six ED nurses received
a 6-hour combination of didactic and practical training
in each triage system (ESI and MTS), in accordance
with national standards. At random and on different days of the week between 12 noon and 10 PM, the
nurses triaged all patients entering the ED consecutively. Patients already triaged before hospital arrival by
ambulance staff, and who met the criteria for treatment
at the trauma room according to current guidelines,
were not triaged again, but classified as level 1 (ESI) or
red (MTS) patients. Patients who left the ED without
being seen by a physician, or whose records were not
available, were excluded from analysis.
Study Protocol
Patients were first registered by the department receptionist and then prospectively classified by one of the
trained triage nurses, using all three systems (ISS, ESI,
and MTS). The treating ED nurse and physician were
blinded to the classification codes. If the triage nurse
classified the patient as needing to be seen immediately,
the patient was turned over to the treating ED nurse.
Otherwise, the patient was sent to the waiting room
and followed the usual procedure.
The information needed to identify the level of acuity
is different in each of the triage systems. For this reason, the triage nurse first classified the patient according to informally structured practice, which required
the least information, then by means of the ESI system,
and finally with the MTS, which required the most
information. For triage following the informally structured practice, based on the patients appearance and
complaints presented, the nurse answered the question:
Under difficult circumstances (e.g., an overcrowded
waiting room), what is the maximum possible time that
this patient will be able to wait before being seen? The
answer options were: patient could not wait or was
able wait up to a maximum of 15 minutes, 1 hour, or
4 hours.
Level 1 was defined as the most urgent category,
comprising patient could not wait (ISS), level 1 (ESI),
and red (MTS). Level 2 comprised the urgency levels
was able to wait up to a maximum of 15 minutes
(ISS), level 2 (ESI), and orange (MTS); level 3 comprised
1 hour (ISS), level 3 (ESI), and yellow (MTS); level 4
comprised level 4 (ESI) and green (MTS); and level 5
comprised 4 hours (ISS), level 5 (ESI), and blue (MTS).

824

Storm-Versloot et al.

Reference Standard. To determine the effectiveness of

the triage systems in those patients treated only by an
ED physician, the classification of these systems was
compared with the reference standard. (Patients
referred to meet a specialist were excluded from this
analysis, because they had already undergone some
prehospital triage and commitment to specific resource
utilization.) This reference standard was determined by
an expert panel consisting of seven experienced ED
physicians. Each physician evaluated all cases individually and was blinded to the conclusions of the other
panel members. The physicians evaluated retrospectively what the real degree of urgency would have
been, based on the ED data, results of diagnostic tests,
and the final diagnosis. Knowing the outcome is prerequisite to determine which patients were really in danger and were misclassified by the systems. Except for
age and sex, patient data were deidentified. The main
question for the panel was: Under difficult circumstances, what is the maximum possible time that this
patient would have been able to wait before being
seen? The answer options were: patient could not wait
or was able to wait up to a maximum of 15 minutes,
1 hour, 2 hours, or 4 hours. In each case the decision of
the majority was applied, and this decision was defined
as the real degree of urgency. If there were multiple
majority decisions, the panel reviewed the case until
consensus was reached.
Data Collection and Definitions. The following patient
data were collected from case report forms, ED reports,
and electronic hospital information systems: patient
demographics, mode of arrival, triage ratings by the triage nurse, urgency classification by the expert panel of
ED physicians, ED resources used, hospital admission
rates (including death), and LOS. LOS was defined as
the time in minutes from registration to discharge or
admission. Death in the ED was coded as a death and
considered as a hospital admission.
The number of resources was counted in accordance
with the ESI (version 3) definitions. Resources used
included labs, electrocardiogram, radiology, specialist
consultation, intravenous (IV) fluids or hydration, IV or
imtramuscular medication, simple procedures, and
complex procedures. For each patient we documented
if one of these resources was used or not.
Data Analysis
Descriptive statistics with continuous data are presented either as means with standard deviation (SD)
or as medians, based on the distribution of the data.
Categorical data are presented as the percentage frequency occurrence. p values of <0.05 were considered
to indicate a statistically significant difference. Differences in distribution of urgency levels were tested by
means of the Friedmans test. In all 900 triaged patients,
we determined the number of resources used to diagnose the patient, the number of admissions, and LOS in
the ED. To evaluate the relationship between triage
classification and these aspects, the Spearmans correlation coefficient was calculated.
Of the subset (patients treated only by emergency
physicians), the data of the reference standard were

COMPARISON OF INFORMAL TRIAGE, ESI, AND MTS

entered into a text file and imported into SPSS, version

16.0 (SPSS Inc., Chicago, IL), for statistical analysis. To
determine the validity of all three systems in patients
treated by an emergency physician, we compared the
ESI and the MTS triage classifications with the reference standard as both five-level and four-level systems.
The latter was achieved by combining the urgency levels 4 and 5. The percentage of patients who were
under- or overtriaged was calculated. The sensitivity,
specificity, predictive values, and likelihood ratios and
their 95% confidence intervals (CIs) for each of the
five urgency levels were calculated. Sensitivity and specificity were defined in terms of correct or overtriage
classification, and undertriage was defined as a misclassification.
RESULTS
A total of 900 patients were triaged. Of these, 10 patients
were lost due to missing ED or triage notes, leaving 890
patients for analysis. Complete triage notes were available for 875 patients (97%) triaged using the ISS, 876
(97%) using the ESI, and 872 (97%) using the MTS.
Patient Characteristics
The characteristics for all 900 patients are presented in
Table 1. Patients referred by the GP to the specialist
(mean SD age = 48 27 years) were significantly
older than self-referred patients (mean SD age = 33
20 years; p < 0.001). Patients arriving by ambulance
(mean SD age = 54 28 years) were significantly
older than patients arriving by private vehicle (mean
SD age = 34 21 years; p < 0.001). In self-referred
patients, no significant differences in age were found
between patients treated by the ED physician only and
those referred from ED physician to a specialist.
Distribution of Urgency Levels
In all patients available for analysis, the number of
patients in each urgency level in each of the triage systems is shown in Figure 1. When the three triage systems were compared as four-level systems, the
distribution of urgency levels was shown to be significantly different (Friedman test, p < 0.006). The same
was true when the ESI and the MTS were compared as
five-level systems (Friedman test, p < 0.001). In the ISS,
more patients were scored very urgent than in the
ESI and the MTS. Furthermore, according to the MTS,
significantly fewer patients belonged to level 5 than
according to the other systems.
Number of Resources
In 890 patients available for analysis, the number of
resources was strongly associated with the urgency
level in all triage systems. The mean number of
resources by urgency level and triage system is
presented in Figure 2.
Number of Admissions
In 890 patients, the number of patient admissions was
strongly associated with the urgency level in all triage
systems. The distribution of admission by urgency level
and triage system is presented in Figure 3.

ACADEMIC EMERGENCY MEDICINE August 2011, Vol. 18, No. 8

825

presented in Figure 4. Patients in the highest urgency

levels had the longest LOS, except for patients assigned
to urgency level 1.

Table 1
Patient Characteristics
All Triaged
Patients
(n = 900)
Sex
Male
Female
Age, yr
Mean (SD)
Median (range)
IQR (2575)
Age distribution, yr
<15
1530
3045
4560
6075
>75
Mode of arrival
Private vehicle
or otherwise
By ambulance
Mode of referral
Self-referral
Only seen by EP
(ED physician)
Referred (by EP)
to specialist
Referred by GP
or otherwise
Number of
admissions
Number of patients
triaged out of
hospital during
measurement period

www.aemj.org

475 (53)
421 (47)
37 (23)
36 (0102)
1953
163
212
207
170
82
61

(18)
(24)
(23)
(19)
(9)
(7)

Patients Only
Seen by ED
Physician (n = 428)
243 (57)
185 (43)
31 (18)
30 (079)
1744
81
134
116
69
20
4

(19)
(32)
(27)
(16)
(5)
(1)

784 (87)

407 (96)

115 (13)

17 (4)

664 (74)
428
236
231 (26)

Reference Standard
Of 890 patients, a total of 428 (48%) were treated by an
ED physician only. Seven forms were incomplete and
were therefore excluded. Statistically significant differences were found in the percentages of under- and
overtriage when all systems were compared as fourlevel systems. The percentages of undertriage were
8.3% for the ISS, 13.5% for ESI, and 11.2% for the
MTS. The highest agreement (64.8%) with the reference
standard was found for the ISS, while the highest
overtriage (29%) was found for the MTS (Figure 5).
When comparing ESI and MTS as five-level systems
with the reference standard, agreement decreased
and overtriage increased, while significant differences
remained.
Details of the sensitivity, specificity, predictive value,
and likelihood ratio for each of the five urgency levels
are shown in Table 2. Overall, sensitivity and positive
predictive values were low for all urgency levels in each
system, whereas specificity and negative predictive
values were over 95% in urgency levels 1 and 2. The fivelevel ESI and MTS systems showed significant differences in sensitivity and specificity only in urgency level 4.

162 (18)

DISCUSSION
27

Data are reported as n (%) unless otherwise specified.

GP = general physician; IQR = interquartile range.

LOS
In 890 patients, the length of ED stay was strongly
associated with the urgency level in all triage systems.
Median LOS per urgency level and triage system is

In this study we demonstrated that when investigated

in an ED setting, the validity of these three triage systems is similar. Undertriage was seen most frequently
when using the ESI system. Furthermore, in all triage
systems, higher urgency levels were associated with
increased resource use, higher rate of hospitalization,
and increased LOS. Based on these results, not one of
these systems appears superior. However, we do recommend the use of a formally structured triage system
in order to obtain verifiable systematic judgments,
transparency, and uniformity in triage.

Figure 1. Distribution of the urgency (Urg) levels for the ISS, the ESI, and the MTS. Urg 1 = patients could not wait (ISS), level 1
(ESI), red (MTS). Urg 2 = patients could wait up to 15 minutes (ISS), level 2 (ESI), orange (MTS). Urg 3 = patients could wait up to
1 hour (ISS), level 3 (ESI), yellow (MTS). Urg 4 = level 4 (ESI), green (MTS). Urg 5 = patients could wait up to 4 hours (ISS), level 5
(ESI), blue (MTS). ESI = Emergency Severity Index; ISS = informally structured system; MTS = Manchester Triage System.

826

Storm-Versloot et al.

COMPARISON OF INFORMAL TRIAGE, ESI, AND MTS

Figure 2. Mean resource use by urgency level and triage system in 890 patients. Spearmans correlation coefficient: ISS = 0.425;
ESI = 0.531; and MTS = 0.374; p<0.001. ESI = Emergency Severity Index; ISS = informally structured system; MTS = Manchester
Triage System; Urg = urgency level.

Figure 3. Patient admission rates by urgency (Urg) level and triage system in 890 patients. Spearmans correlation coefficient:
ISS = 0.396; ESI = 0.379; MTS = 0.398; p < 0.001. ESI = Emergency Severity Index; ISS = informally structured system; MTS = Manchester Triage System.

It is difficult to say which level of sensitivity or specificity is acceptable to conclude that a certain triage system is safe. To reach a high sensitivity (i.e., an
acceptable degree of undertriage), the specificity will be
so low that the potential for saving resources would be
marginal at best.
In our study we found that the ESI had a much lower
sensitivity than the 75% found by Travers et al.4 They
compared ratings by the triage nurse with the triage
decisions taken by two reviewers. These decisions were
based on the original triage notes. Also, we found the
sensitivity of the MTS to be lower than has been
reported in previous studies.9,1517 These differences
may be explained by the fact that most of these studies

were performed in selected patient groups.9,15,16 In contrast, we studied the systems validity in patients treated
by an ED physician only, of whom only 30 (7%) were
triaged to urgency levels 1 or 2.
Using both formally structured systems, the majority
of patients deemed by the expert panel to belong in
urgency levels 1 or 2 were undertriaged. This difference
between the reference standard and the formally structured systems may be due to the fact that the expert
panel of ED physicians knew what happened to the
patient. Therefore, they may have retrospectively evaluated such patients as being less (or more) urgent than
they would otherwise have done using a formally
structured system and before knowing the outcome.

ACADEMIC EMERGENCY MEDICINE August 2011, Vol. 18, No. 8

www.aemj.org

827

Figure 4. Median LOS by urgency level and triage system in 890 patients. Pearsons correlation: ISS = 0.264; ESI = 0.339; MTS =
0.260; p < 0.001. ESI = Emergency Severity Index; ISS = informally structured system; LOS = length of stay; MTS = Manchester
Triage System.

Figure 5. Agreement with the reference standard per triage system (n = 421). Reference standard determined retrospectively by an
expert panel using all available information including the final diagnosis. ESI 4-level = ESI as a four-level system; levels 4 and 5
combined. MTS 4-level = MTS as a four-level system; levels green and blue combined. Spearmans correlation coefficient with the
reference standard: ISS = 0.466; ESI 4-level = 0.276; and MTS 4-level = 0.272; p<0.001. ESI 5-level and MTS 5-level = ESI and MTS
as five-level systems. Spearmans correlation coefficient with the reference standard: ESI 5-level = 0.172; MTS 5-level = 0.240;
p<0.001. ESI = Emergency Severity Index; ISS = informally structured system as a four-level system; MTS = Manchester Triage
System.

Of the total patient group, only a few patients were

categorized to level 1 in accordance with the ESI and
MTS, because patients triaged in the prehospital setting
by ambulance paramedics, and those who met the current criteria for treatment in the trauma room, were
treated immediately. For this reason, these patients
were not present in our sample.
In addition, it is possible that the ISS and the reference standard contained other priorities deemed to be
more important and that caused the users to classify a
patient to a higher (or lower) level than was the case
with the formally structured triage systems. For

example, patients with cerebrovascular accidents

should be seen immediately, to start thrombolysis as
soon as possible. Therefore, in the ISS these patients
are classified to the highest level of urgency. Using the
formally structured systems these patients would be
classified as level 2 (ESI) or orange (MTS). To state the
issue clearly, decisions made following the ISS are not
transparent; relevant information remains implicit and
cannot be retrieved from the ED form.
In general, predicting hospital admission is difficult.17
Nevertheless, the lowest levels of urgency determined
by our study were associated with very low admission

828

Storm-Versloot et al.

COMPARISON OF INFORMAL TRIAGE, ESI, AND MTS

Table 2
Sensitivity, Specificity, Predictive Value, and Likelihood Ratio for Each Cutoff Point of the ISS, ESI, and MTS Triage Systems (n = 421)
Sensitivity
(95% CI)

Cutoff
Urgency
ISS
ESI
MTS
Urgency
ISS
ESI
MTS
Urgency
ISS
ESI
MTS
Urgency
ISS
ESI
MTS

Specificity
(95% CI)

PPV
(95% CI)

NPV
(95% CI)

LR+
(95% CI)

LR
(95% CI)

40 (585)
NA
17 (464)

98 (9799)
100 (99100)
100 (99100)

20 (356)
NA
100

99 (98100)
99 (97100)
99 (97100)

20.8 (1.7142)
NA
NA

0.61 (0.150.98)
1 (0.51)
0.83 (0.40.97)

47 (2965)
36 (2055)
34 (1953)

92 (8994)
95 (9397)
95 (9397)

32 (1947)
40 (2359)
37 (2056)

95 (9397)
95 (9297)
95 (9297)

5.7 (2.611.5)
7.8 (2.819.6)
7.0 (2.517.7)

0.58 (0.370.8)
0.67 (0.460.86)
0.69 (0.480.88)

76 (6585)
50 (3961)
60 (4971)

73 (6878)
74 (6978)
66 (6171)

41 (3349)
32 (2440)
30 (2338)

93 (8995)
86 (8290)
87 (8391)

2.8 (2.13.8)
1.9 (1.32.8)
1.8 (1.32.5)

0.33 (0.200.51)
0.68 (0.500.89)
0.60 (0.410.83)

NA
63 (5671)x
98 (94100)x

NA
44 (3850)*
2 (15)*

NA
36 (3042)
34 (2939)

NA
70 (6377)
67 (3093)

NA
1.1 (0.91.4)
1.0 (0.91.0)

NA
0.84 (0.581.17)
0.97 (0.097.50)

ESI = Emergency Severity Index; ISS = Informally structured system; LR = likelihood ratio; MTS = Manchester Triage System;
NA = not applicable; NPV = negative predictive value; PPV = positive predictive value.
*p < 0.05.

rates. The predictions of resource use, hospital admission, and LOS per urgency level in each of the three triage systems were consistent with research from the
ESI group in Boston, as well as other results from
within the Netherlands.13,1820
LIMITATIONS
The limitations of our study design were first, the lack
of standardized criteria for determining the reference
standard. However, even when working with explicit
criteria, a high rate of disagreement among experts has
been demonstrated.21 For the purpose of our study, we
tried to compensate for possible disagreement by forming a panel of seven experienced ED physicians and settling for a majority consensus. As they did not see the
patients, the expert panel had to rely on a verbal
description. On the other hand, having the diagnosis
and diagnostic test results offered the opportunity to
identify potentially urgent patients who were not identified by the triage systems.22
Second, for pragmatic reasons we chose to triage at
random on different days and between 12 noon and 10
PM. The distribution of urgency levels within this time
frame might differ from other times of the day. However, patient age, sex, and types of condition were similar to a consecutive series of patients in previous
research in our department. Additionally, the distribution of urgency levels in our center (according to the
ESI and the MTS classifications) was consistent with
previous Dutch reports on both systems.18,23
Third, the fact that three triage systems were
applied sequentially in every patient might have
biased the results. If the MTS, which was always used
last, had received the least attention, it would have
shown the lowest validity, but this was not the case.
Hence, we do not think this has been an important
source of bias.

Fourth, data collected from one center may merely

reflect that particular institutions practice. In a previous study comparing inter- and intraobserver agreement between inexperienced and experienced triage
nurses, agreement was found to be the same for the
ESI triage nurses, but lower than in experienced MTS
triage nurses. However, overall the MTS showed a
greater inter- and intraobserver agreement than the
ESI.8 Still, if the use of nurses extensively experienced
in ED practice but inexperienced in triage did result in
underestimation of the sensitivity and specificity, this
would have affected the results of both formally structured systems equally.
Finally, in our ED we saw a relatively high number of
referrals by GPs and percentage of patients sent for
specialty services. If this would substantially influence
the distribution of the triage levels allocated to these
patients, i.e., would lead to generally higher or lower
triage levels, this could influence the predictive values
we found for the triage systems. In turn, this may be of
influence on the generalization of our results. Therefore, the predictive values were determined for the subset of patients treated by the ED physician only. Hence,
the reader should check whether this distribution in his
or her own ED is similar or dissimilar to ours.
CONCLUSIONS
Informally and formally structured triage systems
appear to have equal validity, although the Emergency
Severity Index tends to undertriage patients. To ensure
transparency and uniformity, a verifiable, formally
structured triage system for ED patients is advocated.
The authors thank all nurses and physicians who contributed to
the realization of this study. Thanks also to the English language
editor Daphne Lees for checking the manuscript. Furthermore, we
are grateful to the members of the study group: E. R. Schinkel,
R. Khlinger, and D. C. Schutte.

ACADEMIC EMERGENCY MEDICINE August 2011, Vol. 18, No. 8

References
1. Fernandes CM, Wuerz R, Clark S, Djurdjev O. How
reliable is emergency department triage? Ann
Emerg Med. 1999; 34:1417.
2. Sprivulis PC, Da Silva JA, Jacobs IG, Frazer AR,
Jelinek GA. The association between hospital overcrowding and mortality among patients admitted
via Western Australian emergency departments.
Med J Aust. 2006; 184:20812.
3. Wuerz RC, Fernandes CM, Alarcon J. Inconsistency
of emergency department triage. Ann Emerg Med.
1998; 32:4315.
4. Travers D, Waller A, Bowling JM, Flowers D, Tintinalli J. Five-level triage system more effective than
three-level in tertiary emergency department. J
Emerg Nurs. 2002; 28:395400.
5. HAN University of applied sciences, the Dutch
Emergency Nurses Association (NVSHV), Netherlands Centre of Excellence in Nursing (LEVV).
[Guideline triage on the emergency department;
2008.] Available in Dutch at: http://www.levv.nl/
fileadmin/sites/LEVV/PDF-PP/richtlijnen/080819_
Rapport_Triage_WM.pdf. Accessed May 22, 2011.
6. Mackway-Jones K. Emergency Triage: Manchester
Triage Group. London: BMJ Publishing Group,
1997.
7. van der Wulp I, van Baar ME, Schrijvers AJ. Reliability and validity of the Manchester Triage System
in a general emergency department patient population in the Netherlands: results of a simulation
study. Emerg Med J. 2008; 25:4314.
8. Storm-Versloot MN, Ubbink DT, Chin a Choi V,
Luitse JS. Observer agreement of the Manchester
Triage System and the Emergency Severity Index: a
simulation study. Emerg Med J. 2009; 26:55660.
9. Speake D, Teece S, Mackway-Jones K. Detecting
high-risk patients with chest pain. Emerg Nurse.
2003; 11:1921.
10. Roukema J, Steyerberg EW, van Meurs A, Ruige
M, van der Lei J, Moll HA. Validity of the Manchester Triage System in paediatric emergency care.
Emerg Med J. 2006; 23:90610.
11. van Veen M, Moll HA. Reliability and validity of triage systems in paediatric emergency care. Scand J
Trauma Resusc Emerg Med. 2009; 17:38.

www.aemj.org

829

12. Gilboy N, Tanabe P, Travers DA, Rosenau AM, Eitel

DR. Emergency Severity Index, Version 4: Implementation Handbook and DVDs. Available at: http://
www.ahrq.gov/research/esi. Accessed May 22, 2011.
13. Wuerz RC, Milne LW, Eitel DR, Travers DA, Gilboy
N. Reliability and validity of a new five-level triage
instrument. Acad Emerg Med. 2000; 7:23642.
14. Tanabe P, Gimbel R, Yarnold PR, Kyriacou DN,
Adams JG. Reliability and validity of scores on The
Emergency Severity Index version 3. Acad Emerg
Med. 2004; 11:5965.
15. Cooke MW, Jinks S. Does the Manchester triage
system detect the critically ill? J Accid Emerg Med.
1999; 16:17981.
16. Van der Linden C. [Triage of cardiac patients at an
emergency department] [in Dutch]. Cordiaal. 2005;
1:126.
17. Brillman JC, Doezema D, Tandberg D, et al. Triage:
limitations in predicting need for emergent care
and hospital admission. Ann Emerg Med. 1996;
27:493500.
18. Boeije T, Frederikse MP, van der Heijden FH, Mencl
F. The Emergency Severity Index (version 3): a
good predictor of admission, length of stay and
mortality in a European emergency department
[abstract]. Eur J Emerg Med. 2006; 13:62.
19. Tanabe P, Travers D, Gilboy N, et al. Refining
Emergency Severity Index triage criteria. Acad
Emerg Med. 2005; 12:497501.
20. Wuerz RC, Travers D, Gilboy N, Eitel DR, Rosenau
A, Yazhari R. Implementation and refinement of the
emergency severity index. Acad Emerg Med. 2001;
8:1706.
21. Gill JM, Reese CL 4th, Diamond JJ. Disagreement
among health care professionals about the urgent
care needs of emergency department patients. Ann
Emerg Med. 1996; 28:4749.
22. Asplin BR. Undertriage, overtriage, or no triage? In
search of the unnecessary emergency department
visit. Ann Emerg Med. 2001; 38:2825.
23. van der Wulp I, Schrijvers AJ, van Stel HF. Predicting admission and mortality with the Emergency
Severity Index and the Manchester Triage System:
a retrospective observational study. Emerg Med J.
2009; 26:5069.