International Emergency Nursing xxx (xxxx) xxxx

Contents lists available at ScienceDirect

International Emergency Nursing

journal homepage: www.elsevier.com/locate/aaen

Modeling emergency department nursing workload in real time: An

exploratory study
Edwin L. Cloptona, , Eira Kristiina Hyrkäsb
⁎

a
Emergency Department, Southern Maine HealthCare, One Medical Center Drive, Biddeford, ME 04005 USA
b
Center for Nursing Research & Quality Outcomes, Maine Medical Center, 22 Bramhall Street, Portland, ME 04102 USA

ARTICLE INFO ABSTRACT

Keywords: Study of emergency department (ED) nursing workload has been largely subsumed under the related but se-
Workload parate phenomenon of ED crowding. Nursing workload is difficult to quantify directly. This observational study
Emergency nursing explored modeling ED nursing workload indirectly, in real time, from quantitative data available from the pa-
Emergency department tient tracking computer system (PTCS).
ESI
Methods: Data on 2793 patient visits plus departmental statistics were collected during 167 60-minute survey
Triage
Clinical staffing
periods (SP) in a 25-bed hospital ED in the United States. The charge nurse assessed a perceived workload score
Scales (WLS) according to pre-determined criteria following each SP as a validation measure.
Data analysis: Correlations were calculated between the data and WLS, and strongly correlating variables were
incorporated into linear regression models that sought to approximate WLS.
Results: A measure of aggregate patient acuity derived from the Emergency Severity Index (ESI) was the
strongest predictor of WLS (r = 0.7991). The best-performing model agreed with WLS in 64% of SPs.
Conclusions: Good agreement between model output and WLS suggests that ED nursing workload can be esti-
mated indirectly in real time using data from a PTCS. Strong correlation between the ESI derivative and WLS
further validates ESI and suggests a new application for the ESI score.

1. Introduction summarize work done over a period of time or anticipate work that will
be required in the future.
Emergency nurses bear a substantial workload. Each year in the Time-and-motion studies of nursing date to the industrial effi-
United States approximately 175 000 registered nurses [1,2] devote more ciency era of the 1910s–1920s [12]. (In ergonomics and industrial
than 250 million person-hours to patient care in emergency departments, engineering the time-and-motion study is a standard observational
providing direct patient care during some 140 million emergency de- tool used to analyze work by measuring the time and describing the
partment (ED) patient visits [3]. Nursing workload is known to be related physical movements required to complete a given task [13]). Studies
to patient care and outcomes [4–6] and to staff morale [7,8]. that explicitly address “nursing workload” date from the 1970s on-
Nursing workload is challenging to quantify. The ephemeral, process- ward. The Therapeutic Intervention Scoring System (TISS) [14] of
intensive nature of the work of nursing contributes to the puzzle: much of 1974 and subsequent measures derived from it [15] were developed
the work that nurses do leaves behind no tangible product for objective using a timed task/activity approach (Table 1, discussed below) to
measurement. Researchers agree that quantifying nursing workload facil- quantify overall nursing workload in the intensive care unit (ICU).
itates effective staffing, but despite decades of research and discussion in Those systems and others like them report nursing workload with a
the literature, the complex and diverse nature of nursing has thus far defied resolution of entire work shifts or days rather than describing the
attempts to formulate a broadly accepted quantitative measure [9–11]. current situation with a resolution of hours or minutes. Such systems
often require manual retrospective reporting of nursing activities by
2. Background and literature review the nurses themselves, which adds yet another task to their overall
workload [16]. Although staff nurses receive initial training on ap-
Nursing workload originally was measured to facilitate financial plying the particular workload measurement system adopted by their
analysis and personnel planning. Measures developed for those purposes institution, ongoing quality assurance tends to be lacking, and as a

⁎
Corresponding author.
E-mail addresses: elclopton@smhc.org (E.L. Clopton), hyrkak@mmc.org (E.K. Hyrkäs).

https://doi.org/10.1016/j.ienj.2019.100793
Received 27 January 2019; Received in revised form 22 August 2019; Accepted 5 September 2019
1755-599X/ © 2019 Elsevier Ltd. All rights reserved.

Please cite this article as: Edwin L. Clopton and Eira Kristiina Hyrkäs, International Emergency Nursing,
https://doi.org/10.1016/j.ienj.2019.100793
E.L. Clopton and E.K. Hyrkäs International Emergency Nursing xxx (xxxx) xxxx

Table 1
Characteristics of nursing workforce planning systems as presented by Hurst (2003).
Nursing workforce planning system Characteristics of conceptualized variables and analysis methods

Professional judgment Depends on experience and professional knowledge to determine staffing needs.

Top-down formula-driven methods Utilize metrics such as nurses per occupied bed and nurse:patient ratios that are set forth by regulatory bodies and professional
associations.

Acuity-quality methods Assign patients to categories of “dependency” from which nursing needs are inferred as determined by nursing activity analysis and facility-
specific work sampling.

Timed task/activity methods Standardized time requirements (established by work sampling) for individual tasks included in nursing care plans are summed to estimate
the time that should be (or should have been) required to care for a patient under a given care plan.

Mathematical regression analysis Identifies relationships between variables (e.g., percentage of bed occupancy and nursing hours worked) and extrapolates from those
relationships to predict future staffing needs.

consequence, classification and reporting of tasks—and thus the drawn continuously from a patient tracking computer system (PTCS)
quality of the reports produced—can be inconsistent [16]. and without manual reporting by nurses. The goal was to apply linear
In 1998, Maxwell reported on an automated computer implementa- regression to a set of variables identified among PTCS data to create a
tion of the GRASP (GRAphical System for Presentation; now a trademark model that generates a numerical score corresponding to ED nursing
of Infor, New York, New York) system of accounting for ED patient care workload intensity as perceived by ED charge nurses. To our knowl-
hours based on discharge diagnoses [17]. The implementation’s auto- edge, this study represents one of the first attempts at automatic, real-
mated estimates correlated well with manual tabulations when reporting time quantitative analysis of nursing workload.
monthly, but the author noted that the system performed less sa-
tisfactorily at shorter time scales due to loss of the regression effect 3. Methods
whereby random variations among individual patients’ cases cancel out
over time. Although not adapted to real-time application, Maxwell’s 3.1. Setting
system is notable as apparently the earliest publication in the literature of
an automated nursing workload monitoring system (i.e., one not driven The setting for this observational study was a regional hospital with
by manual input) and for focusing specifically on the ED. 120 inpatient beds whose 25-bed emergency department received ap-
Reflecting growing interest in the study of nursing workload, in 2003 proximately 40 000 patient visits annually during the study. The PTCS
Hurst systematically reviewed 500 publications related to nursing in use throughout the study was Meditech Client Server™ P5.66
workforce planning and workload. He identified five “nursing workforce (Meditech Corporation, Westwood, Massachusetts). Staff size and
planning systems” reported in the literature to quantify nursing workload composition, ED and hospital organization, and computer system re-
for the purpose of determining appropriate staffing levels [18]. Hurst’s mained essentially unchanged during the study.
categories, listed in Table 1 and discussed in detail by Twigg & Duffield
[11], summarize past and present approaches to characterizing nursing 3.2. Measures
workload ranging from intuitive to rigorously analytical.
Research on ED performance challenges has been dominated by The Emergency Severity Index (ESI) triage tool is used by many EDs
studies of the related but separate phenomenon of ED crowding, resulting in the United States to estimate the acuity of patients presenting for
in a substantial body of work [19,20]. Reeder & Garrison’s READI (Real- treatment [31,32]. The two highest acuity levels assigned by ESI reflect
time Emergency Analysis of Demand Indicators) model of 2001 [21] was the immediacy with which intervention is indicated (1 = emergent,
the first of several in-depth studies of ED crowding to be published 2 = urgent). The remaining three levels reflect the triage nurse’s esti-
[22–27]. These studies tend to describe the overall crowding outcome mate of the number of predefined diagnostic and treatment resources
without addressing nursing workload per se. Models of crowding are of the patient will require in the ED (3 = several resources, 4 = 1 re-
limited usefulness in studying nursing workload across the spectrum of source, 5 = no resources). EDs typically use ESI scores to assign pa-
working conditions, as indicated by Jones et al’s finding that the four ED tients to appropriate treatment areas, such as to a low-acuity “fast
crowding scales that they studied “lack scalability and do not perform as track,” and to help prioritize care when multiple patients are waiting.
designed in EDs where crowding is not the norm” [28]. We defined workload broadly as the portion of the available finite
Though related, the concepts of crowding and nursing workload capacity for work that is required to meet the present need, following
differ significantly. Characteristics of the construct nursing workload O’Donnell & Eggemeier: “The term workload refers to that portion of the
include bed occupancy, patient acuity, and available staff resources, operator’s limited capacity actually required to perform a particular
whereas physical bed occupancy approaching or exceeding 100% is the task.” [33] This definition is consistent with Swiger et al.’s more recent
dominant characteristic of crowding even though other factors may be concept analysis-based definition of nursing workload [29]. We as-
included in studies or models [22,24,29]. Wretborn et al observe that sumed that the number of tasks pending at a particular time would
“most of the negative effects of crowding … are mediated via a high reflect the portion of the nurses’ collective capacity for work that is
staff workload” and present further insightful discussions of the re- required to meet the needs of the ED at that time. The workload score
lationship between the concepts of workload and crowding in the ED instrument developed for the present study (Fig. 1) corresponds closely
and the merits of using separate validation measures for studies of the to the instrument developed and validated by Bernstein et al [22] in
two phenomena [24]. These researchers and others [26,27,30] also their study of ED crowding.
agree on the value of continuous data collection and analysis in Staffing level was the number of nurses present and available for
managing a complex system such as an ED. As with nursing workload, direct patient care during the data collection survey period (SP, dis-
no clear definition or standard measure of ED crowding has emerged cussed below) and was not adjusted for personal activities such as meal
from extensive research into the phenomenon [19,20,22,24,26,29]. breaks. Staffing data were collected manually because the PTCS did not
The purpose of this paper is to report on an exploratory study that track this variable.
extends existing work on characterizing nursing workload by devel- Direct, automated measurement of nursing workload from the
oping a model to quantify nursing workload in real time based on data electronic record is challenging. While designing the present study, an

2
E.L. Clopton and E.K. Hyrkäs International Emergency Nursing xxx (xxxx) xxxx

Workload: The portion of the available capacity to perform work that is required to satisfy the
present demand for work.

Choose one number to characterize workload in the ED for the past 60 minutes:

ED resources greatly exceeded demand. Required patient care and

1 None or
Very Light other duties (if any) performed promptly with ample time to spare.

ED resources somewhat exceeded demand. Required patient care

2 Light
and other tasks performed promptly with some time to spare.

ED resources approximately equal to demand. Required patient

3 Moderate
care and other tasks performed promptly with little or no time to

spare. Few or no tasks deferred. Few or no patients in waiting room.

Demand somewhat exceeded ED resources. Some tasks deferred

4 Heavy
in order to perform to perform more urgent tasks. Tasks are pending

part of the time. Some patients in waiting room.

Demand greatly exceeded ED resources. Many tasks deferred to

5 Overwhelming
perform more urgent tasks; tasks are pending most or all of the time.

Many patients or long waits in waiting room.

Fig. 1. Workload Score (WLS) Instrument.

experienced data collector (the first author, E.C.) found it difficult to assessing the workload for purposes of the study. By surveying only the
extract and categorize nursing activities accurately from the PTCS, and charge nurse we minimized intrusion upon the work of staff nurses.
we determined that the task was not feasible for a computer algorithm
of reasonable complexity. We observed that whether and how a given 3.3. Data collection
nursing activity was reflected in the electronic record varied according
to the type of activity, from nurse to nurse, and from time to time for a Data were collected by the first author (E.C.) during a sample of
given nurse. Some nursing activities were charted in response to explicit one-hour survey periods (SPs) stratified to represent times of day and
orders, and some were reflected in narrative notes but not as discrete, days of the week approximately equally. The goal was to obtain a ba-
readily countable events. Also, importantly for a system operating in lanced, representative data set from which to construct a model. We
real time, some activities that contributed substantially to nursing prepared a stratification worksheet (Fig. 2) dividing the seven week-
workload were charted long after they occurred or not at all. A further days into three categories of days and the 24-hour day into eight 3-hour
example of intangible, undocumented factors impacting nursing blocks. Based on first-hand experience in the studied ED, we assumed
workload is the frequency of interruptions as reported by Forsyth et al that the days included in each category would have similar character-
[34]. Therefore we opted to model nursing workload indirectly by istics and thus would be approximately equivalent within categories for
seeking objective indicators among the quantitative data available from data collection purposes. SPs were a convenience sample selected to
the PTCS that correlated strongly with an independent validation cover each time block within each day category. Additional SPs (shaded
measure. Our methodological assumption was that a model based on cells) were selected to sample under-represented workload levels.
indirect but relevant objective measures would be more stable and During each 60-minute SP we collected data on 11 variables for
more reliable than a model based on direct measurement of variables each patient and four department-wide measures (Table 2). Within
such as chart entries of nursing activities that are characterized by 15 min following the end of the SP the data collector (E.C.) asked the
nurses’ individual documentation styles and subjective observations. charge nurse to assess the workload for the past hour using the work-
In the absence of a broadly recognized objective measure of nursing load instrument (Fig. 1) posted at the charge desk and recorded the
workload, we adopted provider perception (specifically that of the ED workload score (WLS).
charge nurse) as the validation measure. Perception is acknowledged to
be subjective and susceptible to bias, but Crane et al. emphasize the
unique ability of provider perception to capture the cumulative effect on 3.4. Data analysis
workload of complex interacting factors that may elude detection in
studies of isolated objective measures [10]. We assumed that the ED The ESI algorithm assigns its smallest numerical value (1) to the
charge nurses’ global view of the department would enable them to most seriously ill/injured patients and its largest value (5) to the least
render an accurate assessment of the overall nursing workload at a given seriously ill/injured. For consistency with the other variables studied
time and that they would give professional consideration to the task of whose values increase with increasing volume or severity, we calcu-
lated an “inverted ESI” for use within the models, IESI ≡ 6 – ESI, to

3
E.L. Clopton and E.K. Hyrkäs International Emergency Nursing xxx (xxxx) xxxx

Fig. 2. Stratification worksheet for data collection. This worksheet guided selection of 60-minute survey periods (SPs) for the study as explained in the text. For
clarity, years have been omitted from dates, and not all additional (shaded) SPs are shown.

return values ranging from 1 for the lowest-acuity patient to 5 for the not take staffing level into account. To test this hypothesis we created a
highest. Bernstein et al. [22] and Epstein & Tian [26] employed the set of 15 “per-RN” variables by dividing each sum variable by the
same technique in their models. number of nurses available for direct patient care during the SP.
We analyzed Pearson’s correlations between WLS and 34 variables Preliminary results prompted us also to test the sensitivity of WLS to
as described below (Table 3). The number of laboratory orders for each the volume of diagnostic orders entered during each 60-minute SP (“60-
patient was limited to 12 (the largest number of laboratory orders for a minute variables”) as opposed to the volume of orders cumulative since
non-admitted patient in the sample) to reduce the influence of in- registration in the ED (“cumulative variables”). To that end we re-
patients boarded in the ED who had aberrantly large numbers of in- trieved 60-minute order entry volumes retrospectively from a stratified
patient laboratory orders (81 patients, 2.9%) that did not directly affect post hoc sample of SPs included in the study. An online calculator [35]
their care in the ED. However, inpatient orders were not excluded en- was used to determine a sample size providing a 97% confidence level.
tirely because some admitted patients were boarded in the ED and thus Sixty-minute order volume data were obtained from 36 SPs (21.6%)
contributed to the ED nursing workload. comprising 570 patients (20.4%).
From the collected variables (Table 2) we derived “sum variables” All variables were screened for collinearity prior to inclusion in
for each SP: the sum and mean IESI scores, sums of the respective ca- regression models to avoid overfitting. In addition to studying the linear
tegories of orders entered (both cumulative and 60-minute), and the regression statistics, we evaluated the percent of model outputs,
sum of all orders entered (both cumulative and 60-minute), as well as rounded to the nearest integer value, that coincided with the perceived
sums of patients in the ED, patient arrivals and departures, crisis pa- WLS stated by the charge nurses for the respective SPs. This percentage
tients (defined in footnote, Table 2), and nurses available for direct served as the primary outcome metric for comparison among models.
patient care. In addition, we hypothesized that an individual nurse’s Data were analyzed using R version 3.4.2 (The R Foundation for
share of the work represented by each sum variable would correlate Statistical Computing, Vienna, Austria.)
more strongly with WLS than would the overall sum variables that did

Table 2
Variables collected during each survey period (SP).
For Each Patient (measure) For the Department (measure)

Age (years) Patient census (number of patients occupying an ED bed at any time during SP)
Sex (female/male) RN staffing (number of RNs available for direct patient care)
Arrival during SP (yes/no) Waiting room time (longest waiting time in minutes at the end of SP)
Arrival by EMS or law enforcement (yes/no) Workload score (WLS—see text)
Crisis patient1 (yes/no)
Departure during SP (yes/no)
ESI score (1–5)
ECG tests (number of orders)
Laboratory tests (number of orders)
Radiology tests (number of orders)
Respiratory therapy interventions (number of orders)

1
Crisis patient: a patient presenting with chief complaint of mental health or behavioral disorder, acute alcohol or drug intoxication, or seeking
detoxification for chemical dependency.

4
E.L. Clopton and E.K. Hyrkäs International Emergency Nursing xxx (xxxx) xxxx

Table 3
Independent variables and correlations with perceived workload (WLS).
WLS vs. r WLS vs. r

Aggregate IESI 0.7991 Sum of all orders (60 min) 0.3798

Patient census 0.7591 Respiratory Therapy orders (cumulative) 0.3747
Sum of all diagnostic orders (cumulative)1 0.7495 Sum of all orders (60 min) per RN 0.3721
Radiology orders (cumulative) 0.7291 Laboratory orders (60 min) per RN 0.3662
Laboratory orders (cumulative) 0.7196 Total patient departures from ED 0.2833
Aggregate IESI per RN2 0.7016 Respiratory Therapy orders (cumulative) per RN 0.2767
ECG orders (cumulative) 0.6610 Radiology orders (60 min) per RN 0.2521
Patient census per RN 0.6447 Total patient arrivals by any means per RN 0.2425
Radiology orders (cumulative) per RN 0.6321 Total patient arrivals by EMS and law enforcement 0.2231
Sum of all orders (cumulative) per RN 0.6290 Radiology orders (60 min) 0.2216
Laboratory orders (cumulative) per RN 0.5876 Number of crisis patients in ED 0.2163
Maximum time in waiting room 0.5618 ECG orders (60 min) per RN 0.1962
ECG orders (cumulative) per RN 0.5132 Total patient departures per RN 0.1599
Number of RNs on duty 0.4896 ECG orders (60 min) 0.1525
Total patient arrivals by any means 0.4177 Respiratory Therapy orders (60 min) 0.0513
Mean IESI 0.4110 Number of crisis patients in ED per RN 0.0386
Laboratory orders (60 min)3 0.4073 Respiratory Therapy orders (60 min) per RN −0.0190

Notes:
1
Cumulative: sum of orders entered for all current emergency department patients, cumulative from start of each patient’s present ED visit.
2
Per RN: the stated variable divided by the number of nurses available for patient care during the survey period. Per-RN variables are italicized in the table.
3
60 min: sum of orders entered for all emergency department patients during the 60-minute survey period; based on a subset of 36/167 study periods (see text).

Table 4 entered during the 60-minute SP) (p ≈ 0.004).

Description of the data set. Several models, some derived from linear regression analysis and
Variables Numerical counts/ranges Median some constructed manually, achieved correlations with WLS greater
than 0.75 and agreement with WLS of 60% to 64% (Table 5). The fol-
Study Periods (SP) 167 lowing formula produced the best-performing model:
Patient Visits 2793
Sex: Female 1527 (55%) y= 0.02425 m+ 0.004301 n+ 0.003172 p+ 1.0256
Male 1266 (45%)
Age (years) Range: 0–100 53 where m = sum of IESI scores; n = sum of all orders, cumulative since
Patients in ED per SP Range: 3–36 17 registration; p = longest time (minutes) since registration among pa-
Arrivals in ED per SP Range: 0–11 3 tients in waiting room at end of SP. Fig. 5 compares the output of the
Departures from ED per SP Range: 0–11 best-performing model with WLS.
RNs available for direct patient care per SP Range: 3–8 5

5. Discussion
4. Results
The workload measure proposed here is consistent with other
The data (Table 4) consisted of 2793 patient visits during 167 SPs models of nursing workload and of ED crowding that seek to distill
on 108 different dates, from 7 April 2015 to 1 January 2017. The data useful measures of nursing workload from the complex tangle of vari-
set was complete except for two variables, number of patient departures ables that comprise a functioning ED [15,20]. The model presented
and maximum waiting room wait, that were not recorded for several here can be considered an “acuity-quality” method as described by
consecutive SPs early in the study. Distributions of ESI triage scores and Hurst [18] (Table 1) because it extrapolates primarily from nursing
of WLS scores reported by charge nurses both were heavily centrally assessments of the acuities of individual patients at triage to estimate
weighted (Figs. 3 and 4). We attributed the small number of ESI Level 4 the amount of work required by the current patient population at any
and Level 5 patients seen in the ED to lower-acuity patients seeking given time.
treatment at urgent care walk-in clinics operated by the hospital The present model differs from most other nursing workload models
throughout its service area. in being focused specifically on the ED rather than on the ICU or other
Table 3 reports correlations between the 34 studied variables and inpatient units, and in being adaptable to automated real-time appli-
WLS in descending order of strength of correlation. The correlations cation rather than being updated once per shift or once per day based
varied between positive and moderately strong (Aggregate IESI, on manual reporting of nursing activities. The model also differs from
r = 0.7991) and weakly negative (Respiratory Therapy orders [60 min] ED crowding research in concentrating on ED nursing workload. Recent
per RN, r = −0.0190). Our assumption for further analysis was that a Australian [36,37] and Brazilian [38,39] formula-driven analyses that
greater value of r, indicating strong correlation between that variable address workload directly share some similarities with our study but
and the perceived workload stated by the charge nurses (WLS), sug- concentrate on nurse:patient ratios and timed tasks without acuity
gested that the variable might be a good surrogate indicator of nursing weighting, and they report on a scale of work shifts or full days rather
workload. than being adaptable to real-time application. The methodology of our
Most per-RN variables (e.g., number of ECG orders per RN) corre- study did not interfere with the work of ED staff nurses, in contrast with
lated less strongly with WLS than did the corresponding sum variables studies that administered questionnaires or other assessments to nurses
(e.g., sum of all ECG orders), but the difference was not statistically during their work shifts [34].
significant (p ≈ 0.095). WLS correlated significantly more strongly The most interesting and significant finding of our study relates to
with cumulative order volumes (e.g. number of ECG orders cumulative the ESI triage algorithm. The aggregate acuity measure derived from
since registration) than with 60-minute order volumes (e.g., ECG orders ESI scores accounted statistically for almost two-thirds of the variance
in WLS (r2 = 0.64); a model based solely on that measure agreed with

5
E.L. Clopton and E.K. Hyrkäs International Emergency Nursing xxx (xxxx) xxxx

Fig. 3. Distribution of ESI triage scores (N = 2793).

Level 5 (Most
Intense) n = 10 (6.0%)

Level 4 n = 43 (25.7%)

Level 3 n = 51 (30.5%)

Level 2 n = 48 (28.7%)
Level 1 (Least
Intense) n = 15 (9.0%)

0 10 20 30 40 50
Fig. 4. Distribution of WLS intensities (n = 167).

perceived nursing workload (WLS) in 63% of SPs. This finding both suggests that the work required to care for a given patient depended
further validates the ESI triage algorithm and suggests a novel appli- more on the overall complexity of patients’ cases, as implied by the total
cation for the ESI triage score in predicting overall ED nursing work- volume of diagnostic tests ordered, than on the volume of diagnostic
load. orders being entered during any one-hour period during the visit.
The relationship between correlations of WLS with the per-RN vs. Performance of the models was encouraging: their output agreed
sum variables was not expected. This finding implies that the workload with perceived workload (WLS) for up to 64% of SPs. Confidence in the
measure was not sensitive to staffing level. As noted above, we hy- findings would have been enhanced if the model had been validated
pothesized that the per-RN variables would correlate more strongly using a second sample of randomly selected time periods. In the case of
with WLS than would the sum variables. However, in 13 of the 15 per- the present study, however, impending hospital-wide implementation
nurse/sum-variable pairings, the sum variables correlated more of a new PTCS and the subsequent disruption of work flow in the ED
strongly with WLS. The difference in correlations was not statistically would have meant a significant delay before validation could have
significant, but even where all outcomes are equally likely, the simple begun, and direct comparability of data derived from the two systems
probability of the correlation of one category of variable being greater could not be assured.
by chance in 13 of 15 cases is approximately 0.3%. This finding may be
related to the subjectivity of the validation measure. At times of low
patient census and low staffing, the relatively quiet environment might 6. Limitations
lead charge nurses to underestimate the workloads on individual
nurses, and thus also to underestimate the WLS for those SPs. The sample size for analysis was reasonable, but all data were col-
The decision not to count nursing orders or activities may have lected at a single site. We do not know the extent to which workflow,
contributed to the statistically significant difference between correla- staffing patterns, and PTCS characteristics unique to this site might
tions of WLS with 60-minute order volumes vs. cumulative order vo- have affected the results.
lumes. Because the majority of the work involved in diagnostic tests is The heavy central weighting of the WLS scale left the distribution
performed by personnel other than nurses, the order volumes we stu- rather sparse at the extremes of its range (Fig. 4). That situation limited
died did not directly involve ED nurses to a substantial degree. The the power of the regression models to estimate values in those ranges
stronger correlation between WLS and cumulative order volume accurately as shown in Fig. 5. Sampling over a longer period of time
would have been required to strengthen representation of WLS Level 1

Table 5
Comparison of selected models.
Formula Designation Correlation (r) with WLS r2 Agreement Remarks

Formula 1 0.7818 0.6112 60% Linear regression on all variables except per-RN and 60-minute variables
Formula 2 0.7991 0.6385 63% Single regression on aggregate triage acuity alone
Formula 3 0.7527 0.5665 60% Regression on the second- and third-strongest correlates, patient census and total diagnostic orders
Formula 4 0.8064 0.6504 64% Best-performing linear regression model overall (see text for formula)

6
E.L. Clopton and E.K. Hyrkäs International Emergency Nursing xxx (xxxx) xxxx

Fig. 5. Graphic comparison of computer model output with perceived WLS. The grey dots show the range and distribution of the computer model’s estimated
workload scores for survey periods assessed at the indicated levels of perceived workload intensity (WLS) by charge nurses. Dot size indicates the number of survey
periods for which the model produced that output value. Boxes represent the second and third quartile range of model outputs (bar marks the median) for each survey
period. Median model outputs for levels 2 through 5 were close to the WLS values they sought to reflect, and the range of model outputs for level 3 was closest to the
value sought. The narrow interquartile range for level 1 indicates the greatest precision, although the accuracy at that level was suboptimal.

and Level 5 SPs in the database. 7. Implications for emergency nursing and recommendations for
The PTCS does not reflect disruptive and workload-intensive events future research
such as an emergent intubation in a timely or readily countable way
even though such events are immediately apparent to a human ob- An understanding of nursing workload is widely acknowledged to
server. Therefore the model proposed here could not capture the real- be fundamental to managing delivery of high-quality health care.
time impact of those patient care events on nursing workload. Equally widely acknowledged is the need for further work on the un-
The ESI acuity level assigned at triage, as applied at the study fa- derlying problem of defining and characterizing nursing workload so
cility, does not dynamically reflect a patient’s condition. Only occa- that it can be effectively measured and reported [9,10,11,29]. To our
sionally is a triage level revised upward if the patient’s condition knowledge, this exploratory study is one of the first attempts to develop
changes sufficiently to warrant a higher priority for treatment. A pa- an automated, real-time measure of ED nursing workload. Although
tient who arrives at a high acuity level, is stabilized, and subsequently preliminary, our results suggest that it is feasible to characterize ED
requires fewer resources for the remainder of his/her ED visit retains nursing workload in real time from data extracted continuously from a
the initial high acuity level until disposition. Therefore the aggregate PTCS, and without manual data reporting to compound the already
ESI measure in this study slightly over-estimated the current acuity in substantial workload of nurses. The reliability and validity of our ap-
the ED. proach should be tested with larger samples, at additional sites, and in
The subjective nature of the WLS makes the performance of the hospitals in other cultures and health care systems, and should be
models challenging to assess. A study design that obtained WLS ratings compared with existing methodologies.
from multiple raters instead of from the charge nurse alone would have Real-time analysis of operations helps organizations adapt quickly
permitted assessment of inter-rater reliability, although identifying a to changing needs and conditions [27]. The availability of real-time
second rater with a perspective of the unit comparable to that of the workload data could facilitate decision-making by ED staff and man-
charge nurse would have been a challenge. The WLS criteria in the agement, for example by supporting activation of a “full-capacity pro-
workload instrument (Fig. 1) were clear, but assessment of WLS by the tocol”, a set of pre-determined actions to be implemented throughout
charge nurse was still subject to interpretation which may have affected the hospital to help alleviate overload conditions in the ED [40]. The
the reliability of the validation measure. On a few occasions charge measure described here may also be a useful metric for administrators
nurses acknowledged stating WLS scores at variance with the stated and researchers. Further research may identify similar measures ap-
criteria but that they felt were warranted by other factors. plicable to other nursing units.
The Emergency Severity Index (ESI) is well established in the United
States as an ED triage tool. In addition to further validating the ESI

7
E.L. Clopton and E.K. Hyrkäs International Emergency Nursing xxx (xxxx) xxxx

algorithm, our observations suggest a new application for the score in the working environment on emergency department nurses: a cross-sectional study.
predicting ED nursing workload. Potential use of this information be- Int Emerg Nurs 2017;31:9–14. https://doi.org/10.1016/j.ienj.2016.04.005.
[8] Vahey DC, Aiken LH, Sloane DM, Clarke SP, Vargas D. Nurse burnout and patient
yond triage imparts added importance to continued conscientious ap- satisfaction. Med Care 2004;42(2 Suppl):II56–66. https://doi.org/10.1097/01.mlr.
plication of the algorithm by triage nurses. 0000109126.50398.54.
The inherent standardization of automated workload measures can [9] Myny D, Van Hecke A, De Bacquer D, Verhaeghe S, Gobert M, Defloor T, et al.
Determining a set of relevant and measurable factors affecting nursing workload in
enhance the transparency and consistency of information shared within the acute care hospital setting: a cross-sectional study. Int J Nurs Studies
departments and across organizations. While this was an exploratory 2012;49(4):427–36. https://doi.org/10.1111/j.1365-2648.2011.05689.x.
study, our vision for its eventual application in emergency nursing in- [10] Crane PW, Zhou Y, Sun Y, Lin L, Schneider S. Entropy: A conceptual approach to
measuring situation-level workload within emergency care and its relationship to
cludes a more fluid, efficient work environment with potential to en- emergency department crowding. J Emerg Med 2014;46(4):551–9. https://doi.org/
hance perceptions of the overall ED experience on the part of patients 10.1016/j.jemermed.2013.08.113.
and of job success on the part of staff. Even the simple advantage of [11] Twigg D, Duffield C. A review of workload measures: A context for a new staffing
methodology in Western Australia. Int J Nurs Studies 2009;46:132–40. https://doi.
enabling staff to help patients understand why they are waiting can
org/10.1016/j.ijnurstu.2008.08.005.
reflect positively in an organization’s patient satisfaction scores [41]. [12] Reverby S. The search for the hospital yardstick. In: Leavitt JW, Numbers RL,
editors. Sickness and Health In America. second ed.Madison: University of
8. Conclusions Wisconsin Press; 1985. p. 206–16.
[13] Lopetegui M, Yen Po-Yin, Lai A, Jeffries J, Embi P, Payne P. Time and motion
studies in health care: what are we talking about? J Biomed Informatics
The present study indicates that a measure derived from the ESI 2014;49:292–9. https://doi.org/10.1016/j.jbi.2014.02.017.
triage algorithm, the aggregate inverted ESI score, can form the basis of [14] Cullen DJ, Civetta JM, Briggs BA, et al. Therapeutic Intervention Scoring System: A
method for quantitative comparison of patient care. Crit Care Med 1974;2:57–60.
an indirect quantitative measure of ED nursing workload. The results [15] Guccione A, Morena A, Pezzi A, Iapichino G. I carichi di lavoro infermieristico.
presented here further validate the ESI triage algorithm and suggest Minerva Anestesiol 2004;70:411–6.
that it is feasible to model ED nursing workload indirectly in real time [16] Registered Nurses Association of Ontario. Reporting on nursing workload mea-
surement systems—a discussion of the issues. Dated January 2005. https://rnao.ca/
based on data readily available from a PTCS. sites/rnao-ca/files/storage/related/1554_RNAO_Workload_Measurement.pdf, ac-
cessed 10/8/2018.
Ethical statement [17] Maxwell DM. A computer-generated measure of nursing workload for the emer-
gency department based on discharge diagnosis. Can J Nurs Admin
1998;11(1):52–64.
The present study conformed to the principles of the Declaration of [18] Hurst K. Selecting and applying methods for estimating the size and mix of nursing
Helsinki and constituted non-human subject research. The study was teams. Leeds, UK: Nuffield Institute for Health; 2003.
[19] Solberg LI, Asplin BR, Weinick RM, Magid DJ. Emergency department crowding:
strictly observational, using existing, de-identified available data. No
consensus development of potential measures. Ann Emerg Med 2003;42(6):824–34.
interventions or clinical investigations were involved. https://doi.org/10.1016/mem.2003.423.
[20] Hwang U, McCarthy ML, Aronsky D, Asplin B, Crane PW, Craven CK, et al. Measures
Funding source of crowding in the emergency department: a systematic review. Acad Emerg Med
2011;18(5):527–38. https://doi.org/10.1111/j1553-2712.2011.01054.
[21] Reeder TJ, Garrison HG. When the safety net is unsafe: real-time assessment of the
The study received no grant or other funding from entities in the overcrowded emergency department. Acad Emerg Med 2001;8:1070–4. https://doi.
public, commercial, or not-for-profit sectors. org/10.1111/j.1553-2712.2001.tb01117.x.
[22] Bernstein SL, Verghese V, Leung W, Lunney A, Perez I. Development and validation
of a new index to measure emergency department crowding. Acad Emerg Med
Declaration of Competing Interest 2003;10:938–42. https://doi.org/10.1197/S1069-6563(03)00311-7.
[23] Weiss SJ, Derlet R, Arndahl J, Ernst AA, Richards J, Fernández-Frankelton M,
Schwab R, Stair TO, Vicellio P, Levy D, Brautigan M, Johnson A, Nick TG.
The authors declare that they have no known competing financial Estimating the degree of emergency department overcrowding in academic medical
interests or personal relationships that could have appeared to influ- centers: results of the National ED Overcrowding Study (NEDOCS). Acad Emerg
ence the work reported in this paper. Med 2004;11(1):38–50. https://doi.org/10.1197/j.aem.2003.07.017.
[24] Wretborn J, Khoshnood A, Wieloch M, Ekelund U. Skåne Emergency Department
assessment of patient load (SEAL)—A model to estimate crowding based on
Acknowledgments workload in Swedish emergency departments. PLoS One
2015;10(6):e010020https://doi.org/10.1371/journal.pone.0130020.
[25] Asplin BR, Magid DJ, Rhodes KV, Solberg LI, Lurie N, Camargo Jr. CA. A conceptual
The authors express their appreciation to John Dziodzio, Data
model of emergency department crowding. Ann Emerg Med 2003;42:173–80.
Analyst, Pulmonary and Critical Care Medicine, Maine Medical Center, https://doi.org/10.1067/mem.2003.302.
Portland, Maine, for his valuable contributions to data analysis for this [26] Epstein SK, Tian L. Development of an emergency department work score to predict
project. ambulance diversion. Acad Emerg Med 2006;13:421–3. https://doi.org/10.1197/j.
aem.2005.11.081.
[27] Martin N, Bergs J, Eerdekens D, Depaire B, Verelst S. Developing an emergency
References department crowding dashboard: a design science approach. Int Emerg Nurs
2018;39:68–76. https://doi.org/10.1016/j.ienj.2017.08.001.
[28] Jones SS, Allen TL, Flottemesch TJ, Welch SJ. An independent evaluation of four
[1] Budden JS, Moulton P, Harper KJ, Brunell ML, Smiley R. The 2015 nursing work- quantitative emergency department crowding scales. Acad Emerg Med
force survey, registered nurse results. J Nurs Reg 2016;7(1). https://doi.org/10. 2006;13(11):1204–11. https://doi.org/10.1197/j.aem.2006.05.021.
1016/S2155-8256(16)31058-4. Supplement. [29] Swiger PA, Vance DE, Patrician PA. Nursing workload in the acute-care setting: a
[2] American College of Emergency Physicians. Emergency medicine statistical profile. concept analysis of nursing workload. Nurs Outlook 2016;64:244–54. https://doi.
http://www.acep.org/content.aspx?id=25234#sm. org/10.1016/j.outlook.2016.01.003.
000144xisrm294e7mt6529kgx5nzk7, accessed April 9, 2018. [30] Bergs J, Vandyjck D, Hoogmartens O, Heerinckx P, Van Sassenbroeck D, Depaire B,
[3] Rui P, Kang K. National Hospital Ambulatory Medical Care Survey: 2014 Emergency et al. Emergency department crowding: time to shift the paradigm from predicting
Department Summary Tables. http://www.cdc.gov/nchs/data/nhamcs/web_tables/ and controlling to analyzing and managing. Int Emerg Nurs 2016;24:74–7. https://
2014_ed_web_tables.pdf, accessed 3/2/2018. doi.org/10.1016/j.ienj.2015.05.004.
[4] Aiken LH, Clarke SP, Sloane DM, Sochalski JA, Silber JH. Hospital nurse staffing and [31] Gilboy N, Tanabe P, Travers DA. The emergency severity index version 4: changes
patient mortality, nurse burnout, and job dissatisfaction. J Amer Med Assoc to ESI level 1 and pediatric fever criteria. J Emerg Nurs 2005;31(4):357–62. https://
2002;288(16):1987–93. https://doi.org/10.1001/jama.288.16.1987. doi.org/10.1016/j.jen.2005.05.011.
[5] Korn R, Mansfield M. ED overcrowding: an assessment tool to monitor ED registered [32] ESI, Emergency severity index. A valid and reliable five-level emergency depart-
nurse workload that accounts for admitted patients residing in the emergency de- ment triage system. http://www.esitriage.org. Accessed November 15, 2017.
partment. J Emerg Nurs 2008;34:441–6. https://doi.org/10.1016/j.jen.2007.06. [33] O’Donnell D, Eggemeier T. Workload assessment methodology. Cognitive process
025. and performance. In: Boff K, Thomas J, editors. Handbook of Perception and
[6] Duffield C, Diers D, O-Brien-Pallas L, Aisbett C, Roche M, King M, Aisbett K. Nurse Human Performance. New York: Wiley; 1986. p. 42.1–9.
staffing, nursing workload, the work environment and patient outcomes. Appl Nurs [34] Forsyth KL, Hawthorne HJ, El-Sherif N, Varghese RS, Ernste VK, Koenig J, et al.
Res 2011;24:244–55. https://doi.org/10.1016/aprn.2009.12.004. Interruptions experienced by emergency nurses: implications for subjective and
[7] Crilly J, Greenslade J, Lincoln C, Timms J, Fisher A. Measurement of the impact of objective measures of workload. J Emerg Nurs 2017;44:614–23. https://doi.org/10.

8
E.L. Clopton and E.K. Hyrkäs International Emergency Nursing xxx (xxxx) xxxx

1016/j.jen.2018.02.001. 2013(Spec):225–32. https://doi.org/10.1590/S0104-11692013000700028.

[35] http://www.raosoft.com/samplesize. Accessed 9/13/2018. [39] Rossetti AC, Gaidzinski RR, Bracco MM. Determining workload and size of nursing
[36] Lyneham J, Cloughessy L, Martin V. Workloads in Australian emergency depart- team in the pediatric emergency department. Einstein 2014;12(2):217–22. https://
ments a descriptive study. Int Emerg Nurs 2008;16:200–6. https://doi.org/10. doi.org/10.1590/S1679-45082014AO2945.
1016/j.ienj.2008.05.006. [40] Willard E, Carlton EF, Moffat L, Barth B. A full-capacity protocol allows for in-
[37] Varndell W, Ryan E, Jeffers A, Marquez-Hunt N. Emergency nursing workload and creased emergency patient volume and hospital admissions. J Emerg Nurs
patient dependency in the ambulance bay: a prospective study. Australasian Emerg 2017;43(5):413–8. https://doi.org/10.1016/j.jen.2017.01.007.
Nurs J 2016;19:210–6. https://doi.org/10.1016/j.aenj.2016.09.002. [41] Sonis JD, Aaronson EL, Lee RY, Philpotts LL, White BA. Emergency department
[38] Rossetti AC, Gaidzinski RR, Fugulin FM. Nursing workload in the emergency de- patient experience: a systematic review of the literature. J Patient Experience
partment: a methodological proposal. Rev Latino-Am Enfermagem 2018;5(2):101–6. https://doi.org/10.1177/2374373517731359.