SPECIAL ARTICLE

Defining a High-Performance lCU System for the _,

21st Century: A Position Paper

Daniel Teres, MD, FCCM,·

Thomas Higgins, MD, FCCM·
Jay Steingrub, MD,·
Laurie Loiacono, MD,·
\X!illiam McGee, MD, MHA,·
Lori Circeo, MD,·
Mary Brunton, MS, RN,·
Karen Giuliano, RN, MSN·
Marty Burns, PhD, RN,·
Jean Roger Le Gall, MD,t
Antonio Artigas, MD,*
Martin Strosberg, PhD,S
and Stanley Lemeshow, PhD,··

Teres D, Higgins T, Steingrub], Loiacono L, McGee \VI, Circeo shown to validate well in new settings. We feel that by
L, llrunton M, Giuliano K, Burns M, Le Gall]R, Artigas A, Strosb- focusing on the episode of critical illness rather than each
erg M, Lemeshow S_ Defining a high-performance lCU system individual lCU admission and by going be}'ond the traditional
for the 21st century: a position paper.] Intensive Care Moo 1998; acute hospital discharge to determine whether the patient
13:195-205.
lives or dies, we can bctter e\":lluatc critical carc systelll
In the fall of 1997 George D. Lundberg and]ohn E. Wennberg performance and cost-effectivcncss. The incentives for high
wrote an editorial in]AMA calling for comprehensh'e qualitl' quality/low cost should favor integrated comprehensive criti-
improvement programs to become the drh-er of the Ameri- cal care delivery systems. Programs that score well should
can health care sl'stem. The suggestion came dUring the be identified as high quality and be honored as medallion
Second European Forum on Qualitl' Improvement in Health level 1 lCUs. We challenge national and international critical
Care held in Paris, France, in April 1997 and was based on care societies to e\":lluate and then debate the described
comments made bl' Donald Berwick. The concept was to definitions and recommendations as a call to action.
focus on an organized response to problem identification
and proposed solutions to improve patient care and protect
the health of the public. Critical care medicine represents
a large segment of health care and is undergoing dramatic
changes during our managed care revolution. General lCU In order to facilitate the study of outcomes in critical
severity of illness models have been developed, tested, and care medicine for the 21st century, it is important
shown to pro\'idc a useful estimate of hospital mortalitl' to better define what constitutes a high-perfor-
for populations of criticalll' ill patients. These systems have mance leu system. leus do not function in isolation
captured the imagination of clinical researchers and have in the process of caring for an acute critical ill-
become an integral component of a large number of publica-
tions as well as a part of manl' lCU databases. These risk ness, and comparisons of leu mortality and length
adjustment severitl' models are remarkably robust for hetero- of stay are directly affected by the growing use of
geneous patient populations but the models have not been intermediate or step-down units and subacute or
long-term ventilator facilities not necessarily linked
From the ·Center for Health Services Research, Departments of to an leu or hospital. Measurement of system per-
Medicinc, Surgery, Anesthesia, and Nursing, Baystate Medical formance should include mortality, functional out-
Center, Springfield, l\1A, and the Tufts Univcrsity School of Medi- come, and resource consumption in the leU,
cine, Boston, l\1A; tHopital Saint Louis, Paris, France; 'Consorci
Hospitalari del Pare Tauli, Sabadell, Spain; Iinstitute of Adminis- intermediate care unit, and regular nursing units, as
tration and Management, Union College, Schenectady, NY; and well as status beyond hospital discharge. Lundberg
··School of Public Health and Health Scienccs, Unh'ersity of and Wennberg [II, based on a suggestion by Ber-
Massachusetts, Amherst, l\1A.
wick, recently requested proposals that can provide
Received Dec 9, 1997, and in revised form Mar 5, 1998. Accepted
for publication 1\Iar 9, 1998
directions for measuring the quality of major seg-
ments of our evolving health care system. We pro-
Address correspondence to Dr Daniel Teres, Baystate Medical
Center, Center for Health Serviccs Research, 759 Chestnut St., pose measuring quality of care for leu systems by
Springfield, 1\IA 01l99. focusing specifically on the first major episode of

Copyright © 1998 lllackwell Science, Inc. 195

Downloaded from jic.sagepub.com at University of York on June 5, 2016

196 Jourml of Intensive Care Medicine Vol 13 No 4 July/August 1998

ICU care and following outcomes to defined end among national and international critical care socie-
points beyond the hospital stay. ties to study, critique, evaluate, and pretest our defi;
Risk-adjusted severity-of-illness measurements nition, rules, and methodology and then to organize
have gained widespread acceptance in general the initiative through local/regional business and
medicaVsurgical ICUs but have not successfully community hospital coalitions.
achieved field or external validation. Several practi-
cal questions arise regarding evaluation of quality
of care across multiple institutions, including stan- Current Measurement of ICU Clinical
dardization of the time to start measuring the major
critical episode and identifying which particular
Performance
ICU admission should be counted. There are now
ICU severity models have reached a high degree
problems using hospital discharge to decide vital
of maturity with a published track record based
status when ventilator-dependent patients are
transferred to chronic or subacute care facilities. on large databases with high discrimination and
calibration, internal validation, and widespread ap-
We propose (1) defining a set of simple rules
plications [2-61. APACHE, l\IPM, and SAPS have
to systematize a time to trigger the clock to start
dominated the literature compared to other meth-
measurements; (2) measuring a standardized mor-
tality ratio (see Table 1) based on" a target patient ods of ICU assessment such as trauma scores and
multiorgan failure modeling [7]. These models are
sample of 300 consecutive evaluable patients twice
simple, have achieved "face validity," and have the
per year. For high-volume ICUs this would translate
practical advantage of easy access to collecting pa-
into two 3 month data collection efforts. ICUs with
tient data at one place and time (lCU at admission)
smaller patient volume may need an extended time
and an easy-to-obtain end point of vital status at
to reach the target sample size, or a minimum of
hospital discharge. When used accurately, the gen-
200 patients; (3) Establishing a fixed time interval
eral models are remarkably robust for heteroge-
of 90 days after the first major ICU admission to
neous patient populations, although separate
determine the predicted and observed mortality of
approaches arc reqUired for cardiac surgery, myo-
the acute episode of care; and (4) Generating a
cardial infarction/chest pain patients, pediatrics, se-
simple weighted hospital or care scale to measure
resource consumption as a cost proxy over the vere sepsis, and burn patients [8-131.
Despite their popularity, however, the general
same 90 day time period. We feel that the risk-
ICU severity models have not been shown to vali-
adjusted outcomes approach combining clinical
date well in new settings [14-17]. There are now
performance and resource consumption is prefera-
numerous publications documenting poor external
ble to structural, organizational, or procedural ap-
validation using formal goodness-of-fit testing in-
proaches in defining a high-performance system.
cluding APACHE III [18-22). Could severity models
The severity-adjusted methodology, uniquely avail-
deteriorate over time? Is there a problem with qual-
able in critical care, is inherently simple and power-
ity of care? Is there inherent "noise" related to multi-
ful and provides benchmark data for future
ple admissions and patients being transferred from
comparisons.
one ICU to another? We suggest that the current
We define a high-performance lCU as one stan-
models will remain useful, but they need to be
dard deviation better than the benchmark for both
updated by new rules regarding start and end time
clinical and resource measures at 90 days post-acute
of a critical illness.
ICU episode of illness. We suggest collaboration

Table 1. Standardized Mortality Ratio (Observed Approaches for Improving Models

Mortality/Mean Predicted Mortality)
There are three possible approaches to improving
Advantages
• Measures performance pre-ICU and post-ICU model performance. One is based on creating ever-
discharge (system performance) larger databases and ultimately developing multiple
• Need large sample size unique severity models based on admitting diagno-
• Show confidence interval sis and presenting problem [211. Another possibility
• Show goodness-of-fit table is developing models such as sepsis models or
Problems multiorgan failure models which are not necessarily
• When does ICU care start? restricted to the time of ICU presentation [23-30).
• What does hospital discharge mean?
We propose a third approach: to focus on the epi-
• What about multiple admissions?
sode of acute care, including post-ICU care up to

Downloaded from jic.sagepub.com at University of York on June 5, 2016

 Teres et al.: Defining a High-Performance leu System 197

90 days, and evaluate ICU care in the context of team and family and they decide to write a do-not-
system performance. resuscitate order, do not transfer him back to an
Consider the following two cases that reflect acute care facility, and limit therapy to support mea-
changes in clinical practice. sures. The patient dies 1 week later.
[The patient died 81 days after the acute onset
of a critical illness. Which severity measure should
Case Presentation.
count? Even if he had survived, his functional status
PATIENT A: l\IULTICO:\IPARTl\IENTAL IIEALTII CARE SYSTEM. would have reflected a poor outcome.]
A 67-year-old man is admitted to a community gen-
eral hospital ICU with vomiting, dehydration, back PATIENT 13: INTEGRATED DELIVERY SYSTEM. A 67-year-
and abdominal pain, hypotension, and tachycardia old man is admitted to community general hospital
with an irregular rhythm. Based on his initial pre- with vomiting, dehydration, back and abdominal
sentation a physiology-based score "estimates" a pain, hypotension, and rapid atrial fibrillation. Dur-
hospital mortality of30%. Because of the atrial fibril- ing resuscitation in the ICU he has an abdominal
lation he has an extensive cardiac evaluation. An cr scan which is reviewed via telemedicine by the
evaluation of the abdomen suggests a differential vascular radiology department at the central, re-
diagnosis of expanding aortic aneurysm, dissecting gional tertiary medical center. They strongly suspect
aneurysm, or embolism to the small intestine. He an expanding aortic aneurysm and, in conjunction
is treated with fluid, Cardizem, and digitalis and his with the ICU triage officer at the medicaVsurgical
heart rate improves. He has extensive x-rays and critical care center, the patient is resuscitated en
the local radiology department has not come up route, stabilized, and transferred to the operating
with a definitive answer. By the third day he has room. Because of the short time since the develop-
received about 3 L of fluid, and his blood pressure ment of this expanding aortic aneurysm, he has
is better. He then develops a reduction in his urine five units of blood loss, has maintained good urine
output. Because of his new onset of oliguria and output, but has developed acute lung injury with
diagnostic uncertainty, arrangements are made to the development of a low partial pressure of oxygen
transfer him to the medical ICU of the regional requiring FiO l of 70%. The patient is admitted to
tertiary center. the ICU with a physiology-based mortality estimate
After having been resuscitated at community gen- of 30%. Acute respiratory distress syndrome devel-
eral, his physiology-based mortality estimate upon ops with some degree of multiorgan dysfunction
presentation at the tertiary medical ICU is now 20%. but not failure. The patient does not require dialysis
He is evaluated, has a repeat cr scan, and the but does require prolonged mechanical ventilation
vascular radiology department confirms that his and a tracheostomy is performed on the tenth day.
problem is an expanding aortic aneurysm. He is On day 14 the patient has improved and is starting
evaluated by the vascular surgery service and ar- to show some attempts at weaning.
rangements are made to operate on an urgent basis. He is transferred to a subacute ventilator facility
Three hours later the operation proceeds and is in which the critical care faculty rotate. After 3
accompanied by massive blood loss because of the weeks the patient is ambulatory, has been weaned
3 day expansion of the aneurysm, further progres- off the ventilator, has improved nutritional status,
sion of oliguria into renal failure, and worsening and is discharged home after another week with a
pulmonary compliance. The patient is now admit- home care program including supplemental oxy-
ted to the surgical ICU with a physiology-based gen therapy.
mortality estimate of 70%, acute respiratory distress
syndrome and multiorgan failure develop, and the Discussion of the Two Cllilical Case Examples.
patient ends up dependent on hemodialysis and In the first case example, the patient had three ICU
mechanical ventilatory support. A tracheostomy is admissions and it is not clear which ICU admission
performed 10 days postoperatively. After 3 weeks should really "count." There is "delayed lead time
in the SICU, he is transferred to the surgical interme- bias" due to the patient staying in the first hospital
diate care unit; he now needs dialysis on a routine where an attempt is initially made at definitive eval-
basis. After three more weeks he is then transferred uation. The patient is moved to a medical ICU be-
to a remote subacute facility for prolonged weaning cause of renal and cardiac involvement and then
from mechanical ventilation and continued dialysis. ultimately to a surgical ICU. There are delays associ-
Three weeks later he develops pneumonia with a ated with each move.
resistant organism and treatment is stepped up. In the second example, the patient has only a
After another week, based on impending septic minimal (but crucial) involvement in the commu-
shock, there is discussion between the medical nity hospital with stabilization but not definitive

Downloaded from jic.sagepub.com at University of York on June 5, 2016

198 Journal of Intensive Care l\Iedicine Vol 13 No 4 July/August 1998

treatment. There is only one probability of mortality ful measures and the burdens of data collection. In
described for this patient, defined at presentation the most straightforward approach, the ratio of tl~~
to the tertiary center ICU after surgery. The patient total number of deaths observed to the number of
has spent only a few hours in the community ICU. hospital deaths predicted by a model is a cmde
The other problem illustrated by these two case measure of clinical performance provided that this
examples is that hospital discharge no longer re- is based on a reasonably large number of consecu-
flects the relevant outcome status. In both cases, tive ICU admissions. This defined measurement is
the patient is transferred to a subacute facility so the standardized mortality ratio (SMR) which, with
that it appears that, for the community hospital, a calculated confidence interval [361, allows catego-
the patient was discharged alive and at the tertiary rizing ICU systems as being at benchmark level or
center the patient was discharged alive.. However, significantly above or below. For ICU systems to
a 90 day time period would demonstrate that the be considered high performance, the SMR (i.e., the
patient with delay in treatment, associated with observed mortality divided by expected mortality)
multiple complications, had a poor outcome and will be significantly less than 1.0, based on a calcu-
had died. There would be a large difference in lated confidence interval. The timing of the mea-
resource use. surement, the choice of the severity model, and
the clinical and physiologic variables must all be
described for operational use, along with the train-
New Focus on Acute Episode of Critical ing requirements for data collectors and gUidelines
Illness for data management.

Our approach to refining severity models for the Recommendations.

21st century is to take into account changes in medi-
1II0DEL SELEGnON. Because of their Widespread ac-
cal practice and to focus on the acute episode of
ceptance, our primary selection is to employ physi-
critical illness. We would like to maintain the gen-
ologically based severity models measured at 24
eral integrity of the severity of illness models as
hours after ICU admission. These models are the
much as possible. It has been repeatedly demon-
APACHE II/III , SAPS II [3,4,61, or the condition-
strated that simple physiology and conditions do
based MPM at 24 hours [51 (see Table 2). The physi-
correlate with patient mortality following acute criti-
ology score with conversion to a probability (i.e.,
cal illness [31-341. Lead time bias has been recog-
APACHE II/III or SAPS II) is the most widely used
nized in a comparison of medical patients admitted
method, but the severity models perform compara-
directly or through an emergency department [35].
bly [371.
The APACHE III system does have a factor for lead
An alternative approach is to use multiorgan fail-
time bias but does not reflect the complexity of the
ure models [26,281. Although organ failure models
issues [3].
have the advantage of "average" laboratory values
We believe it is possible to identify guidelines as
associated with organ dysfunction/failure as op-
to when to start the ICU clock to minimize issues
posed to determining the highest or lowest physio-
related to delayed lead time bias. Likewise we feel
logic variable, they still need further refinement and
that hospital discharge is no longer a valid end
internal validation. Organ failure models are not spe-
point and suggest a 90 day time period to capture
cifically designed for early probability modeling.
acute critical illness. We feel that additional research
Patients with chest pain or myocardial infarction,
will be needed to study whether our suggestions
children, burn victims, and cardiac surgery patients
improve severity model and ICU evaluation and
typically have been excluded from consideration
performance. Additional research also would be
by general models. Post-cardiac surgery models
necessary to better define the time period of acute
should be used for cardiac surgery patients rather
critical illness. Obviously a patient with diabetic
than preoperative risk-assessment models [38].
ketoacidosis or acute severe asthma would recover
MPM at ICU presentation is useful for case-mix eval-
well before the 90 day mark. A patient with severe
ARDS or severe head trauma might be in a plateau
status at 90 days. We feel that as an initial step the
Table 2. GeneraliCU Severity l\Iodeis for Adult Patients
90 day period is a reasonable compromise.
It is therefore time to develop an improved meth- -APACHE II 11II @ 24 hours
odology to compare the performance of an ICU -SAPS II @ 24 hours
relative to other similar ICUs or to one ICU over -l\IPl\I II at presentation and 24 hours
-Postoperative Cardiac Surgery l\Iodels
time. In order to accomplish this goal, a balance
-l\lultiorgan Failure @ 24 hours
must be achieved between the creation of meaning-

Downloaded from jic.sagepub.com at University of York on June 5, 2016

 Teres et al.: Defining a High-Performance leu System 199

uation [391 and specific models may be necessary the ICU and under similar medical management,
in highly specialized ICU's that reach critical levels then perhaps the PACU can be counted as ICU care.
of patient subpopulations [40]. Specific models exist However, if the lCU is distant, and the PACU does
for many specialized patient populations, but inde- not proVide comprehensive organized care for the
pendent validation is generally lacking. ICU admis- occasional long-stay patient, then the PACU is not
sion diagnoses (DRG) and ICU procedure codes equivalent to the ICU. Patients also may start out
also should be collected. in one hospital, be transferred to a single-organ
ICU, and then moved to a multidisciplinary (or qua-
DATA COLLEcnON: SAMPLE SIZE. \Ve support intermit- ternary) ICU (see Fig n. Typically a critical patient
tent rather than continuous data collection because may move several times, including readmission to
of the costs associated with maintaining high quality the ICU. When does the clock start?
data over an extended time period. We recommend
a target population of 300 evaluable patients which RECO~t~IENDATIONS AND SUGGESTED RULES FOR START-
would allow calculation of Sl\IR plus confidence ING TIlE CLOCK. To minimize and simplify complex
interval (see Table 3). We recommend two separate issues of lead time bias, we suggest an arbitrary
data collection periods per year. Patients with multi- start time to be the first major admission to a critical
ple or previous ICU admissions during the same care unit at the highest level hospital, even if the
hospitalizations would be excluded since the focus patient is subsequently transferred from a single-
is on "first episode" critical illness admission. Small organ ICU to a multidisciplinary unit (see Fig n. In
ICUs would need a longer time period to achieve case example A, the admission to the medical ICU
a minimum of 200 first episode patients. Hospitals at the regional tertiary hospital would be when the
with separate medical and surgical ICUs should not "clock" starts. The "incentive" is to get the acutely
combine their numbers if each unit can achieve ill patient to the definitive evaluation and treatment
target patient numbers. There are also fluctuations ICU as soon as pOSSible. (Of course, the key factors
in microbiologic flora within an ICU. A 3 month were prompt diagnosis and surgery.)
sample, repeated later, should minimize the influ-
ence of a single episode of an ICU epidemic of
SUGGESTED SPECIFIC RULES AND EXCEPTIONS.
resistant organisms (see Table 3).

• Severity of illness and resource utilization should

Problem: When Does ICU Care Begin? De-
be measured in consecutive patients admitted for
pending on local practice, postoperative patients
the first major ICU episode, until the target sample
may be directly admitted to an ICU or they may
size is achieved.
spend several hours to a day in the postanesthesia
• One exception would be for a patient who is
recovery unit (PACU). If the PACU is contiguous to
held in a postanesthesia care unit overnight (>
12 hours) while on life support and awaiting ICU
admission. If average PACU time is less than 3
Table 3. Suggested Definition of a High-Performance
leu hours, then a rule can be created to start the ICU
clock at hour 3.
• Low severity adjusted mortality 90 days after first major • For a patient held in an emergency unit, ICU
leu experience at highest level hospital' measurement should be started when the patient
• Low severity adjusted resource use 90 days after first arrives in the ICU. In a parallel structure to the
major leu exposure including leu stay, hospital stay,
and skilled nursing facility stay PACU, if an occasional patient has a prolonged
• Among patients who die ~72 hours after leu and unstable stay in the emergency department,
admission, low percentage of terminal CPR the ICU clock should start 3 hours after the start
• Low readmission rate at 72 hours of ER treatment.
o Low severity adjusted mortality at hospital discharge • If a patient has a brief overnight ICU admission
o Low percentage of monitor patients following high-risk surgery (and is not on any life
o Low reintubation rate support beyond postoperative ventilation) with
o Low percentage of resistant nosocomial infections transfer in less than 24 hours, and is subsequently
o High satisfaction score from patients and/or families
readmitted with a new diagnosis, the second ad-
o High le\"t~l of provider satisfaction
mission becomes the first major ICU exposure
'Based on 300 consecutive, evaluable patients; for ICUs with for reporting purposes. This approach would not
smaller patient volume, the minimum would be 200 patients. penalize ICU systems where high-risk monitor
• = Recommended measures.
o=Associated with high performance leu but not easily patients are not admitted to the ICU. For a patient
measured. admitted from any other location, the ICU admis-

Downloaded from jic.sagepub.com at University of York on June 5, 2016

200 Journal of Intensive Care l\ledicine Vol 13 No 4 July/August 1998

Points of Entr)' Acute Hospital Sub-Acute

Chronic Hospital

Full Chronic Care

Chronic Ventilator Vital Status

90 Da)'s
Sub-Acute Rehab
(TimelLocation
of Death)
Skilled Nursing Home

Transition Care

Home

Fig 1. Episode of critical illness (when to start the ICU clock).

sion counts as the major admission even if the We would anticipate that a national or international
patient is not on life support. committee would deliberate extensively on these
• For a patient who begins at a community-level suggested guidelines. Repeat admissions back to
hospilal lCU and then is transferred directly to the lCU are an important quality indicator. Defining
another lCU at a higher-level hospital, start the the start of the critical illness is all the more im-
lCU performance clock at the higher-level hospi- portant to establish the baseline.
tal. For patients with the typical short-term lead-
time bias, we would argue for starting the clock Problem: When Does Hospital Care End? Since
with the first lCU admission at the tertiary hospi- the current ICU severity models were developed
tal, as in case A. More problematic are the patients using hospital discharge status, they will need to
with a longer stay at a community lCU and then be recalibrated for the 90 day window and possible
transfer to an lCU in another hospital. We would changes in outcome associated with increased use
suggest keeping track of all lCU-lCU transfers of step-down and ventilator rehabilitation facilities.
as an indirect way of differentiating specialty or It should be recognized that recalibration of the
tertiary lCUs from general lCUs. Other ap- models is not a trivial task. Although it is likely that
proaches would involve more complex defini- the same admission variables will "predict" 90 day
tions and data collection efforts. Although these mortality, the coefficients will probably change. We
mles are based on compromise, it is nevertheless would also calculate the hospital-based SMR plus
important to fix the time when lCU system care the 90 day SMR.
starts for the major acute episode of critical ill-
ness, without reducing the integrity of the lCU RECO:lI:lIENDATIONS AND SUGGESTED RULES FOR STOP-
severity systems. Plt-:G TIlE CLOCK. We recommend the end point of
• Also problematic are patients with multiple ad- vital status (live/die) be at 90 days follOWing the
missions to the lCU. For patients admitted from a first major admission to the lCU (see Table 3). Al-
home environment with recurrence of a previous though it is more than theoretically possible that a
problem, such as pulmonary edema or capo patient could die of an unrelated accidental death
exacerbation, we would count each lCU admis- following an acceptable discharge, it would seem
sion as a separate event. For patients with an that 90 days would better capture the "episode" of
extended illness and lCU admissions during the acute illness for the vast majority of patients. More
previous 4 months and readmission to the lCU research will be needed to better define the episode
during the data collection period, we would in- of care. A patient with severe asthma with respira-
clude the measurements but not count the patient tory failure would recover well before 90 days,
as evaluable. while patients with severe ARDS or neurologic head

Downloaded from jic.sagepub.com at University of York on June 5, 2016

 Teres et al.: Defining a High-Performance ICU System 201

trauma might need a longer time before reaching provide an approximate measure of the poorly
a plateau or stabilization. functioning yet surviving patient (see Table 4). The
level of care proVided by different types of units
from intermediate to long-term care will be re-
Resource Measures flected by nurse and rehabilitation support staffing
at these different sites. Some ICUs may provide
Resource utilization costs can best be estimated by flexible ICU care including discharge to home or
using a simple weighted hospital care day approach to a rehabilitation unit. Such an approach minimizes
that includes postacute hospital care. Our meth- the number of patient moves through the system
odolgy is based on a proposal by Rapoport et ai. and minimizes error, but alters the definition of
[411 using ICU and hospital days. The suggested what is meant by an ICU day [431. If the patient is
weights for ICU, step-down, hospital, and posthos- receiving intermediate care in the ICU, then the
pital facilities are listed in Table 4. Although this step-down resource unit can be used. The resource
methodology doesn't measure the intensity of care, measure should foHow the NIH gUidelines on cost-
it stands in contrast to a daily measure of TISS effectiveness analyses [441.
(therapeutic intervention scoring system) points By knOWing the location of the patient at the 90
which are too complex for a practical data coHec- day mark, it is be possible to distinguish among
tion effort. Also, the methodolgy is not dependent patients with high sustained critical care require-
on hospital charges and costs, which are not readily ments (e.g., a patient having a liver transplant, then
available, believable, nor comparable. There may rejection, and then a repeat successful transplant),
be value in adding the simplified or reduced TISS those with prolonged rehabilitation (e.g., high spi-
score at ICU admission to better identify low-cost nal injury), and those with very poor function or
monitored patients and high-cost unstable patients persistent vegetative state (with chronic facility
[421. placement) compared to patients who go home
To match a 90 day mortality end point, the re- (see Table 5). This location status may be a proxy
source consumption measurement should foHow for functional outcome.
the same time period. Long-term high resource usc
such as a chronic or subacute ventilator unit would
New Definition of a High-Performance leu
System
Table 4. Suggested Formula for Resource Use 90 Days
Post First Major ICU Exposure" Bya combination of relatively easy to coHect clinical
• ICU admission day (including readmission):
measures, we suggest it is possible to develop a
= 14 units if on life support' new definition of a high-performance ICU system
= 10 units if not on life support (see Tables 3 and 4). This high-performance ICU
• Next 6 days in ICU: system has a low observed:mean expected mortal-
= 12 u,nits/day if on more than one life supportt ity ratio at 90 days after the first major ICU admis-
= 10 units/day if on a ventilator
sion, and a low severity adjusted resource
= 6 units/day if not on a ventilator
• After 7 days in ICU:
utilization including posthospital care during the
= 8 units/day if on a ventilator same 90 day period.
= 6 units each day not on a ventilator Both the mortality rate and resource utilization
• Intermediate/step-down: would be assessed relative to one standard devia-
= 4 units each day tion from the benchmark. We would suggest meas-
• General ward:
= 2 units each day
• High-level rehabilitation unit (including ventilator Table 5. Levels of Care at 90 Days
unit):
= 2 units each day • Intensive care unit
• Nursing home or extensive home care: • Step-down or ventilator unit in hospital
= 0.5 unit each day • Acute care hospital '
• Full support chronic care facility: • Ventilator unit in chronic care facility
= 2 units each day • Level I subacute rehabilitation
• Level II skilled nursing facility
•Arbitrary scale generally reflecting physician (including
consultants), nurse, support staff, and mid-level care pro\'ider • Level III Custodial facility
differenccs among levels of carc: needs testing, including • Level IV rest home
sensitivity analysis, • Home with home care support
'For example, mechanical ventilation, acute dial}'sis, or • Independent living
vasopressors.

Downloaded from jic.sagepub.com at University of York on June 5, 2016

202 Journal of Intensive Care I\lcdicine Vol 13 No 4 July/August 1998

uring the percentage of patients receiving terminal community-hospital coalition would be more politi-
ePR in the ICU (see Table 3 and Appendix) [451. cally acceptable.
Other measures associated with a high-perfor- The cost of doing such an undertaking is not
mance ICU are not easily captured. trivial and is important to address. There is no easy
answer except to state that no one wants to pay
for data to "evaluate" a major segment of the health
\Vhat About a "Level" Designation for ICUs care system. Information services or administrative
databases cannot substitute for bedside datu collec-
Similar to Trauma Centers?
tion since many key variables, such as neurologic
The proposed severity outcomes approach appears assessment, need clinical interpretation in order to
maintain high quality and integrity in the program.
simple, but obviously requires considerable effort
to follow patients for three consecutive months Unfortunately the Cleveland model is very expen-
sive and overly ambitious. It is always tempting
measured twice and tracking their status over 90
days. However, by following the suggested rules, to collect more, evaluate more, study more. Our
proposal is "bare-bones," including only the mini-
the benefits of benchmarking both clinical outcome
and cost, even with substantial compromises, far mally acceptable data elements with emphasis on
quality data. A national or international committee
outweigh a more complex and arbitrary process-
would likely recommend continuous data collec-
based or structural-based approach to identifying
tion plus some process and structural elements. An
high-performance leu systems such as level I
additional argument against continuous data is that
trauma centers or medallion ICUs. It is unclear what
organization pattern or management structure is intermittent data allows for an exit strategy to stop
once goals are reached.
best associated with focusing on the acute episode
The cost of the proposed system is small com-
of care. The ICU faculty would need to change
pared to its potential benefit and provides incen-
focus, be more inclusive and multidisciplinary. We
tives to encourage early transfer to a definitive ICU.
have much to learn about measuring and managing
Monitoring a system improves the system. Costs
the acute episode of care before we assign designa-
should be shared among hospital systems, insur-
tion. ICUs that consistently achieve the high-perfor-
mance outcome-based definition should be ance carriers, the business community, and state
and local governments.
recognized as medallion-level ICUs. The ICU in the
integrated delivery system, as in case example B,
should be rewarded and designated as medallion
status for high quality low-cost performance.
Summary

General leu severity models have become ac-

cepted and are in Widespread usc, have an exten-
How to Organize a Monitoring System: sive published literature, and contain "simple"
Through a Business-Community-Hospital clinical values. They are robust when used for out-
Partnership come-based population analyses of large numbers
of heterogeneous patients but do poorly in external
\Ve recommend that the structure for organizing field applications. Although there are recognized
data monitoring and data management efforts be a problems with using the standardized mortality
regional coalition with the involvement of multiple ratio as a measure of outcome, many can be mini-
local stakeholders, including the public (or con- mized by the described rules on when to start and
sumers). One example is the Cleveland Health stop the clock. By adopting an "official" definition
Quality Choice Program [45]. The main rationale is of the start of leu care and using a 90 day observa-
that medicine is still primarily provided at the local tion period, the standardized mortality ratio (with
level. National rankings have little relevance to most confidence interval) can be used to compare quality
consumers, who need care locally. Analysis and of care among similar ICU systems for local, na-
reporting preferably should be through a larger co- tional, and international application.
alition. At all levels there should be active involve- It should be acknowledged that the SMR is not the
ment by clinicians and medical and nursing sole measure of quality. However, other important
professional organizations. The coalition should outcomes need to be better defined. We propose
not be structured primarily through legislative or measuring the rate of terminal DNR orders in the
regulatory mechanisms. There could be other ap- ICU and the patient's location at 90 days as a proxy
proaches similar to the New York State cardiac for functional outcome. Asimple weighted resource
surgery report card system, but a private business- scale also collected over 90 days will provide a

Downloaded from jic.sagepub.com at University of York on June 5, 2016

 Teres et aI.: Defining a High-Performance ICU System 203

proxy measure for the "costs" of the episode of These latter patients could be identified at 90 days
critical illness. leU systems with high clinical perfor- by being located in a chronic custodial care facility,
mance and low resource use should be identified Other indicators of poorly performing units might
(with documentation by external audiO, recog- include the following:
nized, acknowledged, and studied. These leU sys-
tems should be identified as medallion level Ileus. • Do-not-resuscitate orders would rarely be used
The above definitions and recommendations for or used only at the terminal stage of prolonged
evaluating critical care medicine should be debated multiorgan failure [46], pOSSibly quantifiable by
by the major national and international critical care a short interval between the time of the DNR
societies. Our approach coincides with the new order and the time of death.
lAMA proposal and call to action, as suggested by • Reintubation rates might be low because patients
Berwick, for a major quality of care initiative [II. are kept on ventilators for longer time periods.
• Readmission rates would be low but in the setting
of long lengths of stay.
Appendix • For patients who die, the stays in the leu would
be excessively long and have high cost.
DefIning a "Poorly" Performing ICU. Even with • The family satisfaction scale would be low, since
rules to regulate the standardized mortality ratio, the level of communication between physicians
the SMR may be insufficient by itself to define a and families would be poor.
poorly functioning leU system. There are at least
two categories of "poor" hospital performance that None of the measures above are currently standard,
can be better identified by a combination ofseverity reliable, or easily collected and each needs to be
scores and other factors. evaluated. Terminal ePR would be the easiest factor
to measure [461.
The irony is that the above description of a poorly
TYPE I-AGGRESSIVE CARE APPROACH. The first ex- functioning, overly aggressive leU might have been
treme example is an leU that pushes hard to save considered the high-performance leu of the 1980s.
every patient, regardless of the patient's wishes, No wonder that the SMR by itself (using hospital
preferences, or advance directives (see Table 6). discharge status) is not sufficient to measure quality.
The reason for starting with this type of leu is In fact, as IeUs increasingly encourage discussion
that the standardized mortality ratio would possibly of withdrawal of care, the SMR might increase
look very good since observed hospital mortality (more observed deaths) over the course of the hos-
might be low compared to mean expected mortal- pital stay, although this might be mitigated when
ity. It is possible that a very aggressive leU (or an examining 90 day mortality. The high-performing
excessively paternalistiC leU) could produce some leU should provide a "dignified" death for patients
hospital survivors among patients who might other- who develop a very poor prognosis for meaningful
wise choose to have their care limited. However, recovery despite aggressive care, assuming good
since we do not have acceptable measures of func- communication with the family and appropriate use
tional outcome or quality of life, the standardized of advanced preferences.
mortality ratio alone, using hospital discharge,
would be inadequate to evaluate these hospitals. TYPE 2-POOR PERFORMA"'CE DUE TO DOWNSlZl"'G OR
Presumably there would be an "excess" of patients Ltl\tlTED RESOURCES. The more difficult assessment
who would die after hospital discharge or who is related to the downsiZing of leus due to cost
would remain in a very poor or low functional state. containment efforts resulting in inadequate leu
staffing or beds and/or limited intermediate or high-
dependency resources (see Table 7). With the re-
Table 6. Can We Identify ICUs Which Push Hard to Save duction of leu capacity, or with the maintenance
Every Patient (No Matter Wh:lt)?·
of leu capacity but reduced bedside staffing with
• SMR shows low observed:mean predicted deaths qualified leu nurses, there should be a tendency
• High severity adjusted resource use toward "excess mortality." eare on regular floors
• High percentage of patients who die with terminal CPR also would deteriorate, creating a constant pressure
• DNR rarely used early to triage patients into and out of the leu [47]. If we
• Long stays for patients who die had good benchmark measures, then it might be
• Low family satisfaction possible to identify negative outcomes. In order to
·Speculati\·e and we have only a limited uatab:Ise from selected show negative outcomes, the standardized mortal-
lCUs voluntarily participating in research datab:Iscs. ity ratio using hospital discharge status should show

Downloaded from jic.sagepub.com at University of York on June 5, 2016

204 Journal of Intensive Care Medicine Vol 13 No 4 July/August 1998

Table 7. Can We Identify ICUs With Excess Mortality as quaternary cancer centers [40]. It also should be
(Due to Downsizing or Reduced Staffing?)"
acknowledged that using two acceptable modelo?
• SMR shows high observed:mean predicted hospital on the same set of patients will yield different prob-
mortality. . . . but patients could be sent to other abilities for individual patients but not necessarily
locations where they would die. for the aggregate population [49,50J.
• Other measures It would be potentially useful to monitor the per-
-High triage frequency centage of patients transferred from one ICU to
-High reintubation rate
-Low readmission rate another hospital ICU. However, the transfer meas-
-Low percentage of low-risk monitor patients ure may be inadequate to accurately reflect a hospi-
-Low family satisfactiont tal with a high-level or specialized ICU. We also
would identify the patient's location at 90 days
·Speculative and we do not have benchmark data.
'Family satisfaction measure for ICVs has not been systematically (home, nursing home, rehabilitation facility, etc.)
studied; families are under stress and are not in a position to or if the patient is deceased, including the date and
judge dangerous staffing patterns. location of death.

high observed: mean predicted mortality, particU- Acknowledgments

larly for low- and moderate-severity patients. But
we do not have good benchmark data. Measures The authors thank Arlen Collins, MD, and John
oflCUs with excess triage activity need to be devel- Rapoport, PhD, for reviewing the manuscript and
oped and tested. adding helpful suggestions and Ann de Roode for
There may be other situations where there could manuscript preparation.
be a good performance ICU but in a poor hospital
or rehabilitation environment. These examples may
apply for ICUs with both high or low acuity. References

Case Mix Issues-Additional Approaches. Care 1. Lundberg GD, \X'ennbergJE. AJAMA thcme issue on quality
must be taken to assure that differences in case mix of care: a ncw proposal and a call to action [editorial]. JAl\tA
1997;278: 1615-1616
do not inappropriately label hospitals as high or low 2. Pollack MM, Runim:mn VE, Getson PRo Pediatric risk of
performers. A common misapplication of severity mortality (PRIS~I) score. Crit Carc Med 1988;16:1110-1116
adjustment models is to selected subsets of patients 3. Knaus \VA, Wagner DP, Drapcr EA, et al. TIle APACHE 1II
rather than all comers. Murphy-Filkins et al. [39J prognostic system: risk prediction of hospital mortality for
have shown that model performance deteriorates critically ill hospitalized adults. Chest 1991;100:1619-1636
4. Le Gall JR, Lemcshow S, Saulnier F. A ncw simplified acute
when critical percentages of risk factors are ex- physiology score (SAPS 11) based on EuropeanINorth Amcri-
ceeded, such as in specialty ICUs. At a basic level, can multicenter study. JAl\lA 1993;270:2957-2963
certain classes of patients (pediatric, burn, cardiac 5. Lemeshow S, Teres D, Klar J, et al. Mortality probability
surgery) must be excluded from analysis unless spe- models (MPM II) based on an international cohort of inten-
cialized models are used. For the general severity sive care unit patients. JAl\lA 1993;270:2478-2486
6. Knaus WA, Draper EA, Wagner DP, et al. APACHE II: a
models, there is debate ~egarding disease- or diag- severity of disease classification systcm. Crit Care Mcd 1985;
nosis-specific models (APACHE) compared to dis- 13:8HHl29
ease-neutral models (MPM, SAPS, PRISM). The 7. \X'citzen S, Higgins T, Tcrcs D. Four year literature survey
specific coefficients for disease categories are rarely of uses of ICU sevcrity models. Personal communication,
used in APACHE II and are only available through unpublished data
8. Selker HP, GriffithJL, D'Agostino RB. A time-insensiti\"e pre-
a proprietary database with APACHE III. dictive instrument for acute myocardial infarction mortality:
Use of the Charlson Index [48] to evaluate medi- a multicenter study. Mcd Care 1991;29:1190-1211
cal comorbidities entails additional data collection 9. Ryan CM, Schoenfcld DA, TIlOrpe \'W, ct al. Objective esti-
and has not been well studied for ICU patients. mates of the probability of death from burn injuries. N Engl
Another approach would be to select high-fre- J Med 1998;338:362-366
10. Edwards FH, Clark RE, Schwartz M. Coronary artery bypass
quency medical and surgical DRGs and limit data grafting: the Society of Thoracic Surgeons National Database
collection to these patients. The concern here experience. Ann Thorac Surg 1994;57:12-19
would be smaller sample size and problems with 11. Higgins n, Estafanous FG, Loop FD, et al. Stratification of
statistical significance. morbidity and mortality outcome bypreoperath'e risk factors
The use of more complex models or specialized in coronary artery bypass patients: a clinical scverity score.
JAl\IA 1992;67:2344-2348
models for different subgroups of patients would 12. Hannan EL, Kilburn H, Racz M, et al. Improving the out-
complicate data collection efforts. An exception comes of coronary artcry b}'pass surgery in New York State.
should be made for highly specialized units such JMlA 1994;271:761-767

Downloaded from jic.sagepub.com at University of York on June 5, 2016

 Teres et a\.: Defining a High-Performance leu System 205

13, Pollack M~I, Patel KM, RUllimann UE. PRIS~I III: an updated 33. Schuster Dp. Prediding outcomc after ICU admission: the
pediatric risk of mortality score, Crit Care Med 1996;24: art and science of assessing risk. Chest 1992; 102:1861-1870
743-752 34. Hunt Jp, Meyer A. Predicting sun'h'al in the intensive care'
14. Beck DH, Taylor GL, Millar D, et al. Prediction of outcome unit. Curr Probl Surg 1997;34:534-599
from intensive care: a prospective cohort study comparing 35. F_~carce)), Kelley I\IA. Admission source to the medicallCU
acute physiology and chronic health evaluation 11 and III predicts hospital death independent of APACHE II score.
prognostic systems in a United Kingdom intensive care unit. )MIA 1990;264:2389-2394
Crit Care Med 1997;25:9-15 36. Hosmer DW, Lemeshow S. Confidence inten'al estimates of
15. Sirio CA, Tajimi DT, Tase C. An initial comparison ofintensive an index ofquality performance based on logistic regression
care in)apan and the United States. Crit Care Med 1992;20: models. Stat Med 1995;14:2161-2172
1207-1215 37. Castella X, Artigas A, Bion), et al. A comparison of severity
16, 1Iastos PG, Sun X, Wagner DP, et al. TIle Brazil APACHE of illness scoring systems for intensi\'e care unit patients:
III Study Group: application of the APACHE III prognostic results of a multicenter, multinational study. Crit Care Med
system in Brazilian intensive care units. A prospedive 1995;23:1327-1335
multicenter study. Intensive Care I\led 1996;22:564--570 38. Higgins n, Estafanous FG, Loop FD, et aI. lCU admission
17. Oh TE, Hutchinson R, Short S, et al. Verification of the acute score for predicting morbidity and mortality risk after coro-
physiology and chronic health evaluation scoring system in nary artery bypass grafting. Ann TIlorac Surg 1997;64:
a Hong Kong intensive care unit. Crit Care Med 1993;21: 1050-1058
698-705 39. Murphy-Filkins RL, Teres D, Lemeshow S, et al. Effect of
18. Apolone G, Bertolini G, D'Amico R, et al. TIle performance changing patient mLx on the performance of an intensive
of SAPS 11 in a cohort of patients admilled to 99 Ilalian lCUs: care unit severity-of-illness model: how to distinguish a gen-
results from GiViTi. Intensh'e Care Med 1996;22:1368-1378 eral from a specialty intensive care unit. Crit Care Med 1996;
19. Moreno R, Reis Miranda D, Fidler V. Evaluation of two out- 24:1968-1973
come prediction models on an independent database. Crit 40. Groeger)S, Lemeshow S, Price K, et al. Multicenter outcome
Care Mcd 1998;26:50-61 study of cancer patients admilled to the intensh'e care unit:
20. Nouira S, Belghith M, E!atrous S, et al. Predicti\'e value of a probability of mortality model.) Clin Oncol, 1998; 16(2):
severity scoring systems: comparison multicenter study of 761-770
four models in Tunisian adult ICUs. Cril Care Med 1998; 41. Rapoport), Teres D, Lemeshow S, et al. A method for as-
26(5):852-859 sessing the clinical performance and cost-effectivcness of
21. Zimmerman)E, \\~agner DP, Draper EA, et al. Evaluation of intensh'e care units: a multicenter inception cohort study.
acute physiolo!,'Y and chronic health evaluation (APACHE) Crit Care Med 1994;22:1385--1391
III predictions of hospital mortality in an independent data- 42. Reis Miranda D, De Rijk A, Schaufeli WB. Simplified thera-
base. Crit Care Med 1998 (in press). pcutic inten'ention scoring system: the T1SS-28 items--
22. Teres D, Lemeshow S. As American as apple pie and results from a multicenter study. Crit Care Med 1996;24:
APACHE. Crit Care Med 1998 (in press). 64-73
23. Escarce)), Kelley I\IA. Admission source to the medicallCU 43. Desserman E, Logan A, Brennan M, et al. Reducing in-house
predicts hospital death independent of APACHE 11 score. transfers through flexible ICU and step,down care [abstradl.
)MIA 1990;264:2389-2394 Crit Care Med 1998; 26, l(suppl.):A42
24. Le Gall)R, Lemeshow S, Hillard), et al. Customized probabil- 44. Siege1)E, Weinstein MC, Russell LB, et al. Recommendations
ity models for early sepsis in adult intensive care patients. for reporting cost-effediveness anal)·ses. )MIA 1996;276:
)MIA 1995;273:6-i-i-Q50 1339-1341
25. Knaus \'(~, Harrell FE, Fisher C) )r, et al. TIle clinical evalua- 45. Rosenthal GE, Harper DL. Cleveland health quality choice:
tion of new drugs for sepsis: a prospedive study design a model for collaborath'e community-based outcomes as-
based on survival analysis, )MIA 1993;270:1233-1241 sessment. Joint Comm) Quallmprm'e 1994;20:425--442
26. Hebert PC, Drummond A), Singer), et al. A simple multiple
46. Prendergast T), Luce )~1. A national sun'ey of withdrawal
system organ failure scoring system predicts mortality of
of life support from critically ill patients [abstrad). Am )
patients who have sepsis syndrome. Chest 1993;104:230-235
Respir Crit Care Med 1996;153:A360
27. Marshall)C, Cook D), Cristou JI.'V, et al. Multiple organ dys-
47. Strosberg M, Teres n. Gatekeeping in the intensive care unit:
fundion score: a reliable descriptor of a complex clinical
even harder choices ahead. Chicago: Health Administration
outcome. Crit Care Med 1995;23:1638-1652
Press, 1997
28. Barriere SL, Lowry SF. An overview of mortality risk predic-
48. Charlson M, Szatrowski TP, Peterson), et al. Validation of
tion in sepsis. Crit Care Med 1995;23:376-393
a combined comorbidity index. ) Clin Epidemiol 1994;47:
29, Le Gall )R, Klar ), Lemeshow S, et al. TIle logistic organ
1245--1251
dysfunction system. )MIA 1996;276:802-810
49. Lemeshow S, K1ar), Teres D. Outcome prediction for individ-
30. Vincent)L, Mendonca A, CantrJine F, et al. The SOFA (sepsis-
ual intensive care patients: useful, misused or abused? Inten-
related organ failure assessment) score to describe organ
sive Care Med 1995;21:770-776
dysfunctiOn/failure. Intensive Care Med 1996;22:707-710
50. Orr RK, Maini BS, Sottile FD, et al. A comparison of four
31. Kollef I\IH, Schuster Dp. Prediding intensive care unit out-
scverity-adjusted models to predict mortality after coronary
come with scoring systems. Crit Care Clin 1994;10:1-18
artery bypass graft surgef)'. Arch Surg 1995;130:301-306
32. Becker RB, Zimmerman)E. ICU scoring systems allow pre-
didion of patient outcomes and comparison of ICU pcrfor-
mance. Crit Care Clin 1996;12:503-514

Downloaded from jic.sagepub.com at University of York on June 5, 2016