A proactive transfer policy for critical patient flow management

Health Care Management Science https://doi.org/10.1007/s10729-018-9437-7 A proactive transfer policy for critical patient flow management Jaime González1 · Juan-Carlos Ferrer1 · Alejandro Cataldo1 · Luis Rojas2 Received: 11 September 2017 / Accepted: 2 February 2018 © Springer Science+Business Media, LLC, part of Springer Nature 2018 Abstract Hospital emergency departments are often overcrowded, resulting in long wait times and a public perception of poor attention. Delays in transferring patients needing further treatment increases emergency department congestion, has negative impacts on their health and may increase their mortality rates. A model built around a Markov decision process is proposed to improve the efficiency of patient flows between the emergency department and other hospital units. With each day divided into time periods, the formulation estimates bed demand for the next period as the basis for determining a proactive rather than reactive transfer decision policy. Due to the high dimensionality of the optimization problem involved, an approximate dynamic programming approach is used to derive an approximation of the optimal decision policy, which indicates that a certain number of beds should be kept free in the different units as a function of the next period demand estimate. Testing the model on two instances of different sizes demonstrates that the optimal number of patient transfers between units changes when the emergency patient arrival rate for transfer to other units changes at a single unit, but remains stable if the change is proportionally the same for all units. In a simulation using real data for a hospital in Chile, significant improvements are achieved by the model in key emergency department performance indicators such as patient wait times (reduction higher than 50%), patient capacity (21% increase) and queue abandonment (from 7% down to less than 1%). Keywords Markov decision process · Approximate dynamic programming · Emergency department · Critical care beds · Patient flow 1 Introduction Recent years have witnessed a growing problem of emergency departments (ED) at Chilean hospitals overwhelmed by patient demand volumes. This excess demand is due in part to increasing ED visit rates that are overtaxing emergency care systems [25] and hampering their ability to attend promptly to all patients. Responses to these pressures are further conditioned by the overcrowding in other hospital units, preventing the timely transfer out of ED of patients needing further treatment. To cope with this situation, ED’s typically use triage systems to classify incoming patients into different levels of urgency ranging from those requiring immediate attention  Jaime González jggonzalez@uc.cl 1 School of Engineering, Pontificia Universidad Católica de Chile, Vicuña Mackenna 4860, Macul, Santiago, Chile 2 School of Medicine, Pontificia Universidad Católica de Chile, Libertador Bernardo O’Higgins 340, Santiago, Chile to those whose needs are least urgent. For patients in the former classification, once they have been attended to they may well require further treatment in a critical/intensive care unit. If transfer to this unit is delayed, however, the patient may deteriorate to the point where their life is at risk. To ensure patients in critical condition receive prompt attention, an emergency services law was passed in Chile some years ago guaranteeing, among other things, that any person in a life-threatening situation be offered immediate treatment at the nearest medical centre. The cost of providing the service until the patient is stabilized must be picked up by the government. In cases where no public hospital can receive the patient, however, he or she must be treated at the nearest private clinic, considerably increasing the cost. As an indicator of the problem, out of the 25,620 patients who were diverted from public hospital ED’s for further treatment between 2009 and 2013, more than 50% were transferred to private clinics where a critical care bed costs three times more than it does in the public sector [24]. The availability of such resources at public hospital emergency departments would have substantially lowered the total cost to the government. J. González et al. It may happen, however, that patients in critical condition are not classified for immediate transfer under the emergency services law and end up waiting for extended periods before being hospitalized. This leads to serious problems, for as Lara et al. [16] have shown, such patients not only have higher mortality rates once they arrive at a critical/intensive care unit but their median hospital stays are also longer, exacerbating hospital overcrowding and reducing the number of other patients that can be treated. This is confirmed by Singer et al. [32], who find a positive correlation between ED stay and patient mortality. In short, overcrowding in emergency departments and critical/intensive care units can have negative consequences both for patient health and hospital or government health budgets. Reducing overcrowding is thus of obvious importance in cutting the mortality risk. If the problem is not addressed, this risk will continue at high levels for a significant number of patients. The obvious solution at the hospital level would appear to be the acquisition of more critical care beds, thereby lowering the probability of full bed occupancy when a patient arrives in critical condition. However, this is a very expensive solution due to the cost of the equipment and infrastructure involved and in many cases will simply not be feasible. An alternative approach would be to find ways of making better use of the existing resources so as to reduce overcrowding both in ED’s and other hospital units, thereby improving patient experience while simultaneously lowering costs. More efficient management is therefore key to cutting patients’ vital risk. The present study posits that the flow of patients through hospital beds designed for the three main levels of patient complexity (low, medium and high) directly influences ED overcrowding and performance measures. A patient transfer policy that manages bed use more intelligently should be able to improve ED indicators and have a positive impact on patient health. More specifically, we will analyze how doctors responsible for transfer decisions at each hospital unit (hereafter simply “the doctor”) should best respond to an increase in the ED patient arrival rate. This involves a number of specific issues. How, for example, should a patient transfer policy for a specific hospital unit change in response to variations in the arrival rate at that unit alone? Would it be different for a hospital with different characteristics such as size? How would the optimal decision change if the arrival rate changed proportionally at all of the units? And if the rate changed only at one unit, would it affect the decisions made by the others? If so, the doctor at, say, the intermediate care unit would not only need to know the arrival rate for that unit but also how the rates are behaving at other units. And finally, how is the entire system affected if appropriate decisions are not made at each moment? The present paper sets out to develop a model that can answer these questions. For this purpose our approach will consider all adult patients at a hospital and the different units responsible for their care. We will further assume that the hospital has three large units: the intensive care (high complexity) unit (ICU), the intermediate care (medium complexity) unit, also known as Step-Down Unit (SDU) and the low complexity unit (WARD). 2 Literature survey In recent years a number of authors have addressed the problem of improving the efficiency of health systems, particularly as regards the flow of users of the systems, both outpatients and inpatients. In the latter case, research has focused on coordinating the flows between the various units within a single hospital, bringing to bear a range of different operations research methods. Among the many resources that can be optimized are staff doctors at health facilities, who can be better managed through efficient organization of medical appointments [8, 13]. Surgery schedules can also be organized so as to make better use of operating theatres [18, 28, 33]. Emergency departments, the context for the present article, have also been investigated [1, 6, 9]. A full description of bed assignment and bed management is addressed by Hall [11], where the author relates this resource with other units within the hospital. Operational issues at other health facility units have also been studied. Since priority attention must be given to critical patients, improving management of the scarce available resources can have a major positive impact on the overall system. Kolker [15] studies intensive care unit bed occupancy rates in terms of the number of elective surgeries that are scheduled daily; Cochran and Roche [5] analyze how to estimate daily demand for beds in an American hospital; and Mallor and Azcárate [21] model an intensive care unit, showing that the decision-makers themselves should be incorporated into the modeling. Although all of these authors are focused on improving hospital systems, little work has been done on the interactions that occur between critical patients and the various hospital units such as intensive care and emergency. The dynamics of the relationship between different units has not been investigated in depth. De Bruin et al. [7] study the behavior of an ED and a coronary care unit, developing a model to determine bed assignments that reduce waiting times and optimize the process. Various methods have been employed to address the above-mentioned problems and issues. One of the most common methods is simulation [14, 35, 36], which can be used to model complex systems incorporating many A proactive transfer policy for critical patient flow management possible decisions, patient flows and uncertain events. More particularly, it allows different system scenarios and configurations to be compared, a key factor in providing support for decision-making. Deterministic linear programming models have also been used to identify optimal bed allocations based on the objectives set for a given situation. Ma and Demeulemeester [20] use this method in a three-stage approach for optimizing hospital planning. Ben Bachouch et al. [4] propose an integer linear programming model for assigning hospital beds. Integrating simulation with optimization is the approach taken by Ahmed and Alkhamis [1] to define appropriate medical staffing levels. To model patient flows and random patient arrivals (such as occurs in emergency departments), however, different methods are needed to incorporate the element of randomness. Cochran and Roche [5] assert that daily patient demand data do not reflect the demand variability over the course of the day. In such cases, stochastic modeling techniques such as queuing theory are required. Among authors already mentioned here for their work on emergency departments, Cochran and Roche [6] apply queuing theory to an ED with different classes of patients while Elalouf andWachtel [9] combine queuing theory with simulation. Rashwan et al. [27] show how the same technique can be used to reduce ED congestion, incorporating their observation that acute bed occupancy can influence the number of persons waiting for emergency attention. In two very recent works, Luscombe and Kozan [19] propose a real-time tool for the efficient management of emergency rooms while Niyirora and Zhuang [22] develop methods for use with emergency resources that minimize wait times and staffing costs. The above-mentioned problems are generally aimed at determining how the system should be organized in terms of bed configurations to improve efficiency and patient flow, or simply describe certain situations in order to identify the mathematical relationship between different agents. Naturally, the question arises as to how to increase efficiency with available resources, particularly for critical patients. Optimization problems can improve management by assigning beds that already exist without raising thornier issues about how many beds a system should have, but as noted earlier, they are not sufficient for dealing with dynamic processes that involve randomness. Andersen et al. [3] optimize the distribution of hospital beds within the facility using a continuous-time Markov chain model to model patient flow. Sauré et al. [29] address a dynamic scheduling problem for radiation therapy patients using a Markov decision process (MDP). As well as employing Markov processes to handle the randomness issue, these studies incorporate decision-making subject to existing resource capacities. MDP’s have also been used with other health-related problems. Schaefer et al. [30] review numerous works addressing problems using this technique, including the control of an epidemic process [17], managing kidney transplants [2] and managing the treatment of heart disease [12]. Relatively little research seems to have been done on patient flow, however. One study that has been done is by Thompson et al. [34], who analyze how hospital patient admissions should be allocated using an MDP to identify an optimal patient admission and proactive transfer policy that anticipates demand for beds in order to reduce bed wait times. In the present study, we attempt to generate a model similar to Thompson et al. [34] but with certain modifications, primarily in the classification of patients within a hospital unit and the cost assigned to ED patients waiting for a bed. Also, Thompson et al. [34] discretizes time into very short periods of just 15 minutes, with decisions made at the start of each one. In practice, however, patient allocation decisions are not made at such a rapid rate. Our analysis therefore uses a division into periods of 8 hours. A Markov decision process is used that will determine a proactive patient transfer policy aimed at reducing ED wait times to better safeguard patients’ health. 3 Description of the problem The patient hospitalization process begins at either the ED or the General Admissions department. In the former case, a nurse in triage makes a rapid assessment of the condition of an arriving patient, who is then sent to a cubicle where he or she is stabilized by a doctor. The patient is then either discharged or transferred to one of the other hospital units. In the latter case, the patient is assigned a low-, mediumor high-complexity bed, depending on the severity of the condition. Patients entering through General Admissions, on the other hand, are being hospitalized for a scheduled procedure and are classified as elective. Typically, such patients will remain in the facility for a number of days. They are first assigned a pre-operative bed and then transferred to the unit where the operation will be carried out. From there they are moved to a post-surgical recovery room and later to a high-, medium- or low-complexity bed depending on their condition. Once patients have been admitted, the hospital is responsible for their condition until they can be safely discharged and sent home. In many cases they will not be in the same bed during the length of their stay. Patients assigned to a ICU bed are in a critical condition requiring high-complexity care and will have to remain hospitalized for some time. At some point they will be J. González et al. transferred to the medium complexity SDU, thus freeing a ICU bed for an incoming patient in critical condition. An analogous chain of events occurs between SDU and the low complexity WARD. This idealized inpatient flow from ICU to SDU to WARD is depicted in Fig. 1. The dashed lines are flows in the reverse direction, which are less frequent and refer to patients whose condition has deteriorated to the point where they require more specialized treatment. It should be noted that this figure do not consider patients arriving at ED which do not need an inpatient bed. They are important in the ED performance, but they do not participate in the hospitalization process. When the demand for hospitalization is high relative to the number of beds available, congestion may result in the various units including the ED. For example, consider the case where there are patients waiting to be attended to in the ED and the ICU is full to capacity. A patient in critical condition then arrives at the ED requiring immediate attention and may later need a bed in the ICU. Since there are no ICU beds available, the patient cannot be transferred and so a problem arises. Until a solution is found the patient remains in an ED cubicle, perhaps for several hours, and his or her condition deteriorates. Meanwhile, because the cubicle remains occupied it cannot be used for a new arrival, thus reducing the number of patients that can be attended to and increasing ED congestion and wait times for all other arrivals. In addition, to free up a bed the ICU must either bring forward the transfer of a current patient or simply wait until one is in a condition to be transferred, at which point there will be an additional delay while the bed or the entire room is cleaned up and readied for the next incoming patient. Such situations occur on a daily basis. Finding ways to reduce their impact depends on a better understanding of the dynamics of daily patient demand and methods for managing patient assignments. Discussions with hospital staff and observation of the hospitalization process have brought certain key aspects of the problem to light. One of these is that demand is uncertain due to the effect of emergency arrivals. Whereas elective admissions are scheduled and therefore can be anticipated, emergency arrivals cannot be predicted with precision. They can, however, be estimated from the unit’s historical records. For present purposes, days were divided into three periods: morning (6am to 2pm); afternoon (2pm to 10 pm) and night (10 pm to 6 am). Knowledge of demand levels broken down by these intervals will be useful in designing a better action policy. The decisions doctors make on patient transfers can reduce congestion in the system. A transfer decision involves determining at what point to move a patient from one unit to another. At any given moment, a patient recovering in a unit may still be in a condition serious enough that they cannot yet be transferred, while at some later moment the patient has recovered to the point where they can be transferred to a lower complexity bed, thus freeing up the higher complexity bed they had been occupying for someone else. Not to carry out the transfer at that point would result in an inefficient use of a valuable resource and an unnecessary provision of a service. There exists, however, an intermediate point at which the patient’s condition is no longer so serious that they must necessarily remain in the same unit, but not yet good enough that transferring them to a lower complexity unit is clearly called for. The doctor must then use his or her judgment in deciding whether to move the patient to a different unit or let them stay an additional period. This will depend on the doctor’s opinion as to where another bed is needed most. If the doctor decides to transfer the patient to a lower complexity unit, a bed in the higher complexity unit becomes available for a new patient but the original patient loses the advantages of remaining in it. If, on the other hand, the doctor decides not to order a transfer, the original patient retains those advantages but the new patient cannot yet be accommodated. Of course, if the original patient is still in serious condition, the transfer decision does not even arise; only for patients whose condition is genuinely good enough to qualify for a transfer will such a move be contemplated. The determination is thus up to the doctor, and whatever he or she decides will directly affect the management of hospital resources. If the decision is made in an informed manner, however, the use of critical beds can be optimized and congestion levels lowered in ED as well as the other hospital units. 4 Model formulation Fig. 1 Inpatient flows between units The problem of how many patients to transfer and when to transfer them can be formulated as a Markov decision A proactive transfer policy for critical patient flow management process (MDP). In this section we present the elements of our proposed model of the transfer problem, the various states of the patient flow system, the actions taken as a result of the doctors’ decisions, the random variables defining the probabilities of the transitions between the various system states, the benefit function and the equations for deriving an optimal transfer policy. The notation used for the different elements of the model is set out in Table 1. The model is focused only in patients needing hospitalization, so patients who are discharged directly from ED are not being consider in arrival rates. Another assumption is that we consider the ideal flow of patients (ICU to SDU to WARD, not backwards), because this flow plus the incoming demand from ED and General Admissions represent the majority of the demand. The reverse flow showed in Fig. 1 is always less than 13% of incoming demand in each unit. So, for the sake of simplicity, we consider that a patient can not deteriorate its health condition. 4.1 System state space A decision regarding how many patients to transfer is made at the start of each time period, that is, at 6 am, 2 pm and 10 pm. The interval between the decisions (i.e., the length of the periods) was originally chosen to fit the three-period division of the 24-hour day used by the hospital this model was first applied at. In our view it is also a realistic interval for general application in that it is long enough for there to be significant changes in patient conditions but short enough that all such changes will be responded to promptly. At the moment each decision is made, a certain number of beds in each unit are occupied. This number is used to define each unit’s state. Also important is the condition of each patient. We therefore distinguish between the numbers of patients who are in serious condition (i.e., cannot be transferred), fair condition (can be either transferred to another unit or maintained where they are) and good condition (implying the complexity level of the unit they are in is too high for their current condition so they must necessarily be transferred). Each time a patient is transferred to a less critical unit, their condition is automatically reclassified as serious, from which they move up to fair and then good condition as they improve. Before defining the system states, we must define the sets to be used. Let I be the set of different units, J the set of different patient conditions within each unit and T Table 1 Model notation Sets I J T System states and decision variables S s ui sij qi t As a xi yi Random variables Fi Gij Ei Parameters Ci deit r(s, a) Bi CUi CYi Set of hospital units {ICU,SDU,WARD}. Set of possible patient conditions {serious, fair, good}. Set of time periods {morning, afternoon, night}. Set of all possible system states. Vector representing the current state of the system. s = (ui , sij , qi , t). The number of patients in the Emergency Department (ED) waiting for a bed in unit i. The number of patients occupying a bed in unit i in condition j . The number of patients in unit i waiting for a bed in unit i + 1. The time period. Set of all possible actions for a state s. Vector representing an action. a = (xi , yi ). Number of patients in fair condition it is decided to transfer out of unit i. Number of unit i operations cancelled. Number of patients arriving at the ED who will need a bed in unit i. Number of patients in unit i in condition j who improve to condition j + 1. Number of patients transferred from ED to unit i. Number of beds in unit i. Elective demand for unit i in period t. Function defining the benefits of a state-action pair. Benefit derived from having a bed in unit i occupied. Cost of having a patient in ED waiting for a bed in unit i. Cost of cancelling an operation in unit i. J. González et al. the set of possible periods. Thus, I = {ICU,SDU,WARD}, J = {serious, fair, good} and T = {morning, afternoon, night}. To simplify the presentation, the index used for the elements in the three sets may just be the corresponding natural number (1,2,3). Also, in our MDP model we define S as the set of all possible states and As as the set of all possible actions for a state s. We also want to study how the hospitalization flow affects the ED. For this, we need to know the current state of attention to patients there, and more specifically, how many patients are ready to be assigned to a bed but are still in a cubicle, blocking it from use by an incoming ED patient and thus reducing the number of ED patients that can be attended to simultaneously. The state space is therefore expressed as follows: s = (ui , sij , qi , t), where ui is the number of patients waiting in ED to be transferred to unit i, sij is the number of beds occupied in unit i ∈ I by patients in condition j ∈ J , qi is the number of patients in unit i ∈ {I CU, SDU } waiting for transfer to the next unit (note that q3 does not exist since WARD patients are not waiting for a bed in another unit, their next move in fact being discharge from the hospital), and finally, t indicates the current period (morning, afternoon or night). The flow diagram for the state space is shown in Fig. 2. The solid lines are the normal patient flow, the dotted lines represent flows resulting from a doctor’s decision to bring patient transfers forward and the dashed lines indicate the flows when the units are full to capacity and transfer decisions are delayed. It is assumed that ED has patients blocked in cubicles only if the unit they are waiting to be assigned to is full given that freeing cubicles and transferring emergency patients have priority. Note that the state space is finite given that each of its vector components is limited by the number of beds in the correspondingunit. For example, if in ICU (i = 1) there are s1j + q1 ≤ n. The same can be said for ui n beds, then j ∈J and for t, in the latter case given that there are only three periods. 4.2 Actions At the start of each time period, the doctor at each unit knows how many patients are in good condition and must be transferred to the next (that is, lower complexity) unit. The doctor must also decide how many of the patients in fair condition to transfer to the next unit (or discharged if they are in WARD). An elective procedure can be suspended if the bed is needed for a more critical patient. These possible actions are represented by the following vector: a = (xi , yi ), where xi is the number of patients in fair condition it is decided to transfer to unit i and yi is the number of procedure cancellations for unit i. In practice such cancellations are rare, but are included here to ensure the problem’s feasibility. Each set of possible actions is defined in terms of each state. Thus, for a state s = (ui , sij , qi , t) the set must satisfy xi ≤ si2 yi ≤ deit ∀i ∈ I, ∀i ∈ I, t ∈ T , (1) (2) where deit is the elective patient demand in period t for unit i. Constraint (1) imposes that the maximum number of transfers of fair condition patients from i is equal to the number of such patients in that unit. Constraint (2) ensures that there cannot be more procedure cancellations than the number that were scheduled. Thus, all of the doctor’s possible actions are bounded above, and are also bounded below by the nature of the variables as nonnegative integers. 4.3 Random variables Once the decisions regarding patient transfers and procedure cancellations have been made, the state at the end of the period will depend on two random factors. The first factor is the ED demand. Although this phenomenon can in some sense be predicted, it is by nature random and thus the number of arrivals cannot be known a priori. A probability distribution function is therefore used to define the likelihood of arrivals at ED for later transfer to each of the three units. The second factor determining the end-of-period state is the condition of the patients, which is the more difficult of the two factors to quantify. The system has been modeled in such a way that serious patients will not remain permanently in that condition, but will not necessarily be ready for transfer to the next unit after just one period. An estimated length of stay was derived for each unit, which was then used to determine, for a certain number of patients initially in serious condition, the probabilities that any given number of them will improve to fair condition. As for the patients in fair condition, those whom the doctor decides at the start of a given period not to transfer will have a certain probability of continuing in that condition, while others will improve to good condition and therefore must be transferred at the start of the next period. The probability that they will become serious patients in the next unit is thus equal to 1 (recall that new arrivals to a A proactive transfer policy for critical patient flow management Fig. 2 State space flow diagram unit are automatically classified as serious even though the transfer reflects an improvement in their condition). We therefore have two random variables: – – Fi : A random variable representing the demand of ED patients for unit i. Following the common practice in the literature for modeling random ED arrivals, we assume the variable follows the distribution Poisson(λit ), where λit is the rate of ED arrivals in period t for later transfer to unit i. Gij : A random variable representing the number of patients that progresses to the next condition within their unit, that is, how many patients in i pass from condition j to condition j + 1. The variable does not need to be defined for condition j = 3 (i.e., good condition) since by assumption, all such patients are transferred in the current period. The variable’s distribution is assumed to be Binomial(sij , pij ), where pij is a function of the average stay in days for unit i and patient condition j . We also define Ei as the number of patients transferred from ED to unit i. This variable, however, is deterministic rather than random and depends on how many patients there are in each unit. It will always tend to be as high as possible given that it is preferable for a patient to be in a unit rather than an ED cubicle. si1 ′ = si1 +Ei +si−1,3 +xi−1 −Gi1 +deit −yi si2 ′ = si2 − xi + Gi1 − Gi2 si3 ′ = Gi2 ∀i ∈ I, (4) ∀i ∈ I, (6) ∀i ∈ I,    qi′ = max qi + si3 + xi − Ci+1 − si+1,1 + si+1,2   ′ − xi+1 + qi+1 ,0 ∀i ∈ {ICU,SDU} , t′ =  (5) t + 1 , t = 1, 2 1 , t = 3, (7) (8) where Ci is the number of beds in unit i, the other variables being previously defined. We define s0,3 and x0 equal to 0, to ensure the correct definition of the equations. Observe that when a patient is first transferred to a given unit, they are classified in serious condition. The probability of passing to a state s′ from state s after action a has been taken is then found by calculating the probability that all of the random variables take the necessary values to satisfy (3)–(8). Solving for each random variable, we have ′ Gi2 = si3 (9) ∀i ∈ I, 4.4 System state transition probabilities The equations for the transitions between system states are as follows: ui ′ = ui + Fi − Ei ∀i ∈ I, (3) ′ ′ Gi1 = si2 − si2 + xi + si3 ∀i ∈ I, (10) ′ ′ − si1 − si−1,3 − xi−1 + si2 − si2 + xi Fi = u′i −ui + (si1 ′ +si3 − deit + yi ) ∀i ∈ I . (11) J. González et al. The desired probability is thus given by  ′ ′ ′ { Pr(Gi2 = si3 p(s′ |s, a) = )·P r(Gi1=si2 −si2 +xi +si3 ) The Bellman equation for finding the maximum value and, more especially, the optimal decision policy that will yield this value, is as follows: i∈I ′ · P r(Fi = u′i − ui + (si1 − si1 − si−1,3 − xi−1 ′ ′ + si2 − si2 + xi + si3 − deit + yi )) } . v ∗ (s) = max r(s, a) + λ Note that the probability is defined as the product of the probability each random variable takes a certain value given that all the random variables are independent. This is so because the arrival of a given patient is independent of both the arrival and the condition of every other patient. 4.5 Costs and benefits Although the broad objective of the proposed model is to make fuller use of existing resources and reduce congestion in the hospital system, for the sake of simplicity the objective will be defined as maximizing hospital income. The benefits accruing to each state derive from the income generated by the hospital per day per occupied bed in unit i. So, let Bi be the benefit of having a bed occupied in unit i. Note that in the case of a patient definitely in good enough condition to be moved to the next unit due to the lack of a bed there has not yet been transferred, the benefit is counted at the lower bed value of the next unit rather than that of the higher complexity bed the patient continues to occupy. The costs included in the benefit function relate to the decisions taken by the doctor. CYi is the cost of cancelling a procedure in unit i. A cost CUi is also assigned to the presence of a patient waiting to be transferred from ED. The benefit function can then be written as r(s, a) = Bi sij + i∈I j ∈J CUi ui Bi+1 qi − i∈{1,2} CYi yi − i∈I ∀s ∈ S, a ∈ As . i∈I 4.6 Optimality equations The value function in our MDP formulation, denoted vk (s) for a period k, gives the expected total value discounted over an infinite time horizon. The equation defining this value is written as a∈As a∈As p(s′ |s, a)vk+1 (s′ ) ∀s ∈ S, s′ ∈S where λ < 1 is the discount factor. Taking the limit over the time horizon, we have v ∗ (s) = lim vk (s). k→∞ ∀s ∈ S. s′ ∈S (12) Finding a solution to this equation directly is in practice almost impossible due to its dimensionality, considered by de Farias and Van Roy [10] as the limiting factor for solving stochastic problems. In the real-world case treated here below in Section 6.4, the number of states can be as high as 1024 (not all of them feasible) and the number of state-action pairs may reach 1030 . 5 A solution approach Various methods for approaching problems with highdimensional spaces have been proposed in the literature. One of the most commonly used methods is approximate dynamic programming, which consists in using a linear programming approach together with an approximation of the maximization function in the original problem. First set out in Schweitzer and Seidmann [31], it has since been studied and applied by other authors such as de Farias and Van Roy [10], Patrick et al. [23] and Sauré et al. [29]. Following their example, we tackled our problem in five steps. First, the MDP is transformed into its linear programming (LP) equivalent. Second, an approximate objective function is identified with a known structure. Third, using this objective function, the approximate linear programming (ALP) problem is defined. Fourth, the ALP problem is solved. Finally, the optimal decision policy is stated. According to Puterman [26], an MDP discounted over an infinite time horizon always has an associated linear programming problem from which the desired value can be found using the Bellman equation. It is written as follows: LP ) vk (s) = max r(s, a)+λ p(s′ |s, a)v ∗ (s′ ) s.t.  α(s)v(s) min s  r(s, a) + λ p(s′ |s, a)v(s′ ) ≤ v(s) ∀s, a, s′ ∈S where α(s) is a positive number for each s. If we impose the condition that α(s) = 1, then α(s) can be taken as the s initial probability distribution of the system. The associated A proactive transfer policy for critical patient flow management dual problem directly gives the optimal decision policy in terms of the values taken by the decision variables:  r(s, a)x(s, a) DLP ) max s,a s.t.  x(s, a) − λ   p(s|s′ , a′ )x(s′ , a′ ) = α(s) ∀s s′ ∈S a′ ∈As′ a∈As  Ui ui +  Sij sij + i∈I j ∈J i∈I  Qi qi i∈{1,2} s In this approximation function, Ui is the marginal cost discounted over an infinite time horizon of having a patient in an ED cubicle waiting for a bed to become free in unit i, Sij is the marginal benefit discounted over an infinite time horizon of having a bed occupied in unit i by a patient in condition j , and Qi is the marginal benefit discounted over an infinite time horizon of having a bed occupied in unit i by a patient ready to be transferred to a lower complexity unit. Following the literature, we impose that Ui , Sij and Qi are all non-negative, while W0 has no restriction on its sign. Thus, the ALP is ALP )    min W0 − Ui Eα (ui ) + Sij Eα (sij ) + Qi Eα (qi ) i∈I j ∈J s.t. i∈{1,2} s.t.   (1 − λ)W0 − Ui Es,a (ui ) + Sij Es,a (sij ) i∈I i∈I j ∈J  + Qi Es,a (qi ) ≥ r(s, a) ∀s, a i∈{1,2} Ui , Sij , Qi ≥ 0 ∀i, j W0 ∈ R, a 1 −Eα (ui ) ∀i Eα (sij ) ∀i, j Eα (qi ) ∀i s∈S a∈As x(s, a) ≥ 0 ∀s, a. The above program is the master problem to be solved by column generation. It starts with a small set of initial variables representing the initial solution. New columns are added to the problem by adding the state-action pair that most violates the primal constraint. This pair is found using the pricing problem model, an optimization problem written as follows: Ui , Sij , Qi ≥ 0 W0 ∈ R. i∈I  x(s, a)r(s, a)  (1 − λ) x(s, a) =   s∈S a∈As − Es,a (ui )x(s, a) ≤ s∈S a∈A  s Es,a (sij )x(s, a) ≤ s∈S  a∈A s Es,a (qi )x(s, a) ≤ DALP ) max x(s, a) ≥ 0 ∀s, a. As we have already seen, this model comes with the dimensionality curse, so we must resort to an approximation function. In recent decades much work has been done on approximate dynamic programming, which attempts to find an approximation using a series of specific functions that act as a base. Choosing a function that will give a good approximation remains a difficult challenge, however, as no satisfactory method for making the choice has yet been found. As Patrick et al. [23] has noted, the task is still more of an art than a science. Based on approaches found in the literature, we propose to approximate v(s) in the following manner: v(ui , sij , qi , t) = W0 − This approximate model has a low number of variables but the number of constraints is still high. We therefore turn to the method of column generation to solve its dual, which has a high number of variables but a reasonable number of constraints. Thus, pricing) max s,a +  i∈I j ∈J  r(s, a) − (1 − λ)W0∗ − Sij∗ Es,a (sij ) +  i∈{1,2}  i∈I Ui∗ Es,a (ui ) Q∗i Es,a (qi ) . (13) If this generates a strictly positive value, it means there exists a state-action pair not considered in the master problem that would improve its optimal value. This pair is then added to the master problem as a new column and the column generation algorithm is again iterated until either the optimal value is 0 (no primal constraint is violated) or the improvement in the objective function is marginal (< 0.00001). The master problem will have only a small set of positive variables and thus cannot directly indicate the optimal policy. For that, an optimization problem will have to be solved for each state. Thus, once the DALP problem is ∗ solved and values obtained for W0∗ , Ui∗ , Sij∗ and  Qi∗, the ∗ approximation function  ∗ v(ui , sij , qi , t) = W0 − i Ui ui +   ∗ S s + ij i j ij i Qi qi is inserted in the right-hand side of Eq. 12 and we obtain where Eα (ui ) = Eα (sij ) = Eα (qi ) =  s∈S  s∈S  s∈S α(s)ui α(s)sij α(s)qi Es,a (ui ) = ui − λ s′ ∈S Es,a (sij ) = sij − λ Es,a (qi ) = qi − λ  ′ ∈S s s′ ∈S p(s′ |s, a)u′i v(s) = max r(s, a)+λ a∈As p(s′ |s, a)sij′ p(s′ |s, a)qi′ . +  i∈I j ∈J Sij∗ sij′  s′ ∈S +  i∈{1,2}  p(s′ |s, a) W0∗− Ui∗ u′i Q∗i qi′ i∈I ∀s ∈ S. (14) J. González et al. 6 Results In what follows we set out the main results obtained with the model proposed in the previous section for obtaining an approximate optimal patient transfer policy. Since it would not be practical to determine what should be the optimal action in every possible state, we confine our presentation to the behavior of the solution, its robustness and an application of the optimal policy using a computer simulation. 6.1 Two instances Due to the complexity of the problem as described in the preceding section, it would be difficult to find exact mathematical expressions for all the relevant relationships between the various hospital units. We therefore focus our analysis on two extreme instances as regards the dimensions of the problem and compare the behavior of their respective solutions. Any patterns common to the two can then be interpreted as tendencies that could be replicated for hospitals with different bed capacities and patient arrival rates. The characteristics of the two instances are summarized in Table 2. In the first instance, based on real data, the size of the problem is similar to that of the real case we will take up below in Section 6.4. Its bed capacity and arrival rates are considerably larger than those of the second instance (hypothetical data), a “small problem” representing a hospital with lower patient demand and fewer beds. Average lengths of stay, on the other hand, are the same in both instances given that the condition of patients should not depend on the size of the institution they are hospitalized in. 6.2 Interpretation of transfer policy For the first instance we studied many different possible states, focusing on those that appeared to be particularly representative and therefore might best indicate what Table 2 Characteristics of two instances of model to determine an approximate optimal patient transfer policy guidelines should be adopted and how to interpret the solutions. In what follows we describe what is suggested by the optimal approximate policy. It was observed that the policy decision always involves maintaining enough free beds in each unit to accommodate the demand that will materialize in the next 8-hour period. For a unit during a given period there may be incoming patients from General Admissions (elective demand), patients already waiting for a free bed, and patients arriving at ED. In the first two cases the demand is certain. In addition to this demand plus any patients it is decided to transfer to the unit (i.e., those ready to be transferred and those whose transfer is brought forward), a certain number of beds must be kept free as a function of the random arrival rate from ED. This policy implies coordination between the various units given that if unit i decides to transfer n patients, unit i + 1 will have to accommodate them above and beyond the other above-cited sources of demand. How many of these free beds there should be will depend on the specifics of the situation at hand. For the case to be discussed below in Section 6.4, the numbers aimed for in the morning period are three in ICU, three in SDU and ten in WARD. Thus, if at the start of the period there is one free bed in ICU and two in SDU, the doctor will attempt to transfer two patients from ICU to SDU and three from SDU to WARD, given that each unit is also subject to demand from higher units. But what if the number of desired beds cannot be freed? At that point, a sort of competition between units comes into play. A free bed in either ICU or SDU is always preferred to a bed in WARD despite the high demand there may be for the latter. As between ICU and SDU, one free bed in each is preferable to two in either one of them, but if there is only one free bed, the optimal policy suggests that it should be in SDU. This preference is due to an assumption of the model according to which a patient is assigned to a bed at a level of complexity no higher than the one most closely corresponding to their condition. In reality, however, it is Number of beds in ED Number of beds in ICU Number of beds in SDU Number of beds in WARD ED daily patient arrival rate, for xfer to ICU ED daily patient arrival rate, for xfer to SDU ED daily patient arrival rate, for xfer to WARD Average stay in days, in ICU Average stay in days, in SDU Average stay in days, in WARD Instance 1 Instance 2 30 35 50 100 3.3 5.1 14.1 3 4 6 12 12 18 40 2,2 3,0 7,5 3 4 6 Arrival rates are averages of the time-dependent data used in the model A proactive transfer policy for critical patient flow management Fig. 3 Number of patient transfers from ICU by number of free beds in SDU. Each curve represents a different ICU arrival rate (patients/day). a represents Instance 1, b represents Instance 2 better to have a single free bed assigned to ICU given that if necessary, a patient only requiring SDU can still receive the required level of care in ICU. This is not optimal, of course, but it may be preferable to prolonging patient wait times in ED. 6.3 Sensitivity analysis A sensitivity analysis was conducted to gauge the robustness of the solution to changes in the problem parameters. The tests focused on the impact of changes to the ED arrival rate. In both instances they were conducted for the morning period given that it accounts for a relatively large part of ED demand (more than 40%) and less elective demand than the afternoon period, making it particularly suitable for studying ED dynamics. The analysis was based on observations made in the various units and is presented here in the form of answers to the questions raised in the introduction. Note that although the accompanying graphs show the trends as curves, the results are in fact discrete (integers). Fig. 4 Number of patient transfers from ICU by number of free beds in SDU. Each curve represents a proportional variation in arrival rates at all units (patients/day). Curves are for Instance 2 How should a hospital unit’s optimal patient transfer policy change in response to variations in the arrival rate at that unit alone? Figure 3 shows how the number of patients in fair condition to be transferred from a unit changes as a function of bed occupancy in the next unit. In this case, it is a question of the number of transfers from the ICU (initially full) versus the number of free beds in SDU. The solid line represents the base case while the broken lines represent different variations in the ED arrival rate of patients destined for ICU. The more free beds there are in SDU, the more patients it will be decided to transfer, but the trend increases at a decreasing rate until eventually no more ICU beds need to be freed. Also, the curve is more pronounced when the arrival rate is relatively high. This suggests that the change in transfer decisions is more sensitive when there are more beds available to transfer patients given that the separation of the curves is greater. If there are only one or two beds free in SDU, the change is smaller because there is less margin for taking transfer actions. J. González et al. Table 3 Number of patient transfers from SDU by number of free beds in WARD at different WARD arrival rates, for Instance 2 WARD arrival rate 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0 Free beds in WARD 0 1 2 3 4 5 6 7 8 9 10 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 2 2 2 1 1 1 1 1 3 2 2 2 2 2 1 1 3 3 3 3 2 2 2 2 4 3 3 3 3 3 2 2 5 4 4 3 3 3 3 3 5 5 5 4 4 3 3 3 5 5 5 5 4 4 4 3 5 5 5 5 5 5 4 4 How do the two instances compare? Comparing the two graphs in Fig. 3 it can be seen that with the same ICU arrival rates, the curves are similar when the rates are less than 3, but grow apart at higher rates given that in Instance 2 the optimal policy calls for the transfer of an additional patient. This difference is due to the fact that the SDU arrival rate is lower so that it is preferable to leave fewer beds free in that unit. How does the optimal decision change if the arrival rate changes proportionally in all units? Here we are interested in how the number of ICU transfers varies if the arrival rates at all units (including ICU) change. The variation in the number of transfers from ICU as bed availability in SDU increases for different arrival rates is depicted in Fig. 4 for Instance 2. Only the ICU rate is shown but the rates for each unit were varied in the same proportion. As can be seen, the behavior is similar to the previous case when the rate is low (less than 3). At higher rates, however, the curves tend to converge. To compare Figs. 3b and 4, we define a metric δ n (λ1 ; λ2 ) as the difference between the number of patient transfers at arrival rates λ1 and λ2 when there are n free beds in ICU. When there are 8 SDU beds available and the rate changes from 1.2 to 2.2, the difference in transfers δ 8 (1.2; 2.2) is 1 in both cases, but when the rate changes from 2.2 to 8.2, the difference δ 8 (1.2; 2.2) is 3 in Fig. 3b and only 1 in Fig. 4. This suggests that when the arrival rate changes proportionately in all units, a rate increase of 3 or 4 times the base rate will not result in equal increases in the optimal transfer decisions. Finally, at low values of n the difference between the optimal decisions for the two figures is not significant. If the arrival rate changes in only one unit, will it affect the decisions made in the others units? The number of transfers from SDU that must be made in the morning when it is full (i.e., the number of beds that must be freed in SDU) versus the number of free beds in WARD, for different WARD arrival rates, is shown in Table 3. As with the previous case, the number of transfers increases with availability in the next unit until it reaches a maximum. If there are enough free beds in WARD, the number of transfers reaches 5 in almost every arrival rate series given that by assumption, the SDU arrival rate remains the same. Note also that the number of transfers reaches the maximum more quickly at low WARD arrival rates. This occurs because with fewer arrivals, more “weight” is given to a free bed in SDU and thus the number of transfers from SDU to WARD is greater. Also, in the extreme case where the WARD arrival rate is 10.0, the optimal number of transfers is only four. This is so because there are not enough free beds in WARD to permit transferring five patients. Finally, how is the system affected if appropriate decisions are not made at each moment? To answer this question we studied the percentage variation in the value of Eq. 14 when the number of patient transfers from ICU is one or two patients more or less than the number indicated by the approximate optimal policy. Thus, various ICU arrival rates were tested assuming there were enough free beds in SDU to receive them. The results are set out in Table 4. As can be seen, if, for example, the arrival rate in Instance 2 is 2.2 patients per day, transferring one fewer patient than the number indicated by the optimal policy generates a drop in Table 4 Percentage change in objective function value for transfers of one or two patients more or less than the optimal number, at different arrival rates (patients/day) for each instance Arrival rate 1.2 2.2 3.2 4.2 5.2 6.2 Instance 1 Instance 2 −2 −1 +1 +2 −2 −1 +1 +2 10.9% 1.7% 2.9% 1.4% 2.4% 1.2% 0.7% 0.1% 0.7% 0.1% 0.6% 0.1% 0.6% 0.7% 0.3% 0.5% 0.2% 0.5% 1.4% 1.6% 1.1% 1.4% 0.9% 1.3% – – – 9.0% 13.6% 8.4% – 2.2% 5.2% 1.6% 4.4% 1.4% 2.0% 4.5% 1.4% 4.0% 1.3% 4.0% 9.2% 12.4% 7.7% 11.7% 7.5% 12.0% A proactive transfer policy for critical patient flow management the objective function value of 2.2%. If, on the other hand, one or two more than the optimal number are transferred, the OF value falls by 4.5% and 12.4%, respectively. Also apparent is that regardless of the arrival rate, the change in the OF value when the optimal number of transfers is varied up or down by 1 patient is almost always less than 5%. But if the variation is 2 patients, the OF value change can be as much as 13.6%. For Instance 1, the OF value change for a variation of one or two transfers was smaller, remaining below 5% in every case except one in which the value fell 10.9% when the arrival rate was 1.2 and 2 patients fewer than the optimum were transferred. This atypically high percentage was due to the fact that in this case, no bed was left free for an elective inpatient and a procedure had thus to be cancelled, generating a high cost in the objective function. These results demonstrate that in the two instances representing hospitals of different sizes, transferring one patient more or one patient less than is optimally indicated will not have a significant negative impact as long as there are enough free beds for the patients who are certain to arrive at each unit. 6.4 Simulation of a real-world case Having determined the approximate optimal policy, it was then put to the test in simulations of a real hospital in Santiago, Chile following the process depicted in Fig. 1. The model was coded in C# using the SimSharp package available on the World Wide Web. The time horizon for the simulation was five months (August to December) and 100 replications were run for each scenario tested. Data for the simulations were obtained directly from hospital records, which contained information on the number of patients entering and leaving each unit, lengths of stay in each unit, time of ED arrivals and the number of daily elective admissions. For items not available in the records, estimates were made and validated by experts. Four different policies were tested. Policy 0, the base case, represents the current situation at the hospital in which there is no proactive demand management decision-making. Policies 1 and 2 are based on demand estimates. In Policy 1, the number of free beds maintained by each unit equals the minimum demand indicated by the hospital data while Table 6 Number of Emergency Department arrivals in each scenario Patient complexity ICU patients SDU patients WARD patients Patients not needing hospitalization ICU-SDU / Triage lvl 1 WARD / Triage lvl 2 Not needing hospitalization / Triage lvl 3-4-5 Real data 17.73 (1.037) 45.08 (1.187) 61.53 (1.580) 15.01 (0.933) 43.18 (0.992) 62.53 (0.939) P0 P1 P2 P3 526 801 2204 30988 527 799 2196 31006 524 803 2185 30961 526 799 2200 31011 in Policy 2, the number equals the maximum demand. Each unit thus acts independently, transferring the minimum (Policy 1) or maximum (Policy 2) number of patients on the basis of historical demand data. If, for example, we assume that the number of patients arriving at SDU, including elective demand, ED and other unit transfers, is 3 with a standard deviation of 1, Policy 1 will maintain 2 free beds whereas Policy 2 will maintain 4. Finally, Policy 3 is the approximate optimal policy described in the previous section. The reason we include Policy 0 and we do not compare the other policies with real data is because we did not have all the desired indicators. Then, it is necessary to compare Policy 0 with real data, to validate our model and to show the base case is like the hospital of study. Waiting time in ED was the only indicator we could obtain of the data. The other indicators were not able in the data of the hospital. Table 5 shows a comparison of the waiting time of patients in ED, one of the main indicators. However, this comparison is not directly because the data were not accurate. We group the different patients in three:patients with level 1 of Triage, patients with level 2, and patients with level 3-4-5. Each one is associated with patients needing a bed in ICU or SDU, patients needing a bed in WARD, and patients needing no hospitalization, respectively. The number of patients arriving at ED over the time horizon is shown in Table 6. The differences between the various scenarios are just pseudo-random noise, so the Table 7 Number of patients transferred reactively, by unit and policy Number of transferred patients Table 5 Comparison of Policy 0 and real data Policy 0 Number of ED Arrivals From ICU From SDU From WARD From ICU reactively From SDU reactively From WARD reactively P0 P1 P2 P3 728 1592 6133 209 349 3387 740 1604 6142 4 308 3413 738 1607 6131 4 233 3387 733 1601 6167 58 116 770 J. González et al. Table 8 Average Emergency Department wait time Average ED wait time (minutes) ICU patients SDU patients WARD patients Patients not needing hospitalization Policy 0 Policy 1 Policy 2 Policy 3 17.4 (1.011) 17.96 (1.049) 45.08 (1.187) 61.53 (1.580) 13.38 (1.227) 13.62 (1.250) 31.28 (1.477) 40.43 (1.914) 13.12 (1.248) 13.43 (1.270) 30.56 (1.498) 39.11 (1.945) 6.42 (1.844) 6.47 (1.842) 9.43 (2.242) 9.76 (2.404) Figures in parentheses are coefficients of variation four policies were subject to the same conditions and the indicators could be compared directly. The idea behind the design of these policies is to highlight the benefits of anticipating demand and handling patient transfers in a proactive rather than a reactive manner. By a reactive transfer is meant one that is made to free a bed upon a request for a bed from another unit, with the inconvenience that the patient from that other unit is already waiting and that there will be additional wait time to ready the freed bed. Table 7 shows, for each policy, the number of transfers from each unit (patients ready for transfer plus transfers brought forward), indicating how many were made reactively. As can be seen, under Policy 0 the number of reactive transfers is considerably greater than under the other policies. Under Policy 3, by contrast, this indicator improves considerably for all units. As for Policies 1 and 2, there is significant improvement on transfers from ICU only given that each unit makes its own decisions, with no communication between them on decision-making. Thus, although ICU improves, transfers from SDU and WARD are not properly incorporated, especially when transfer levels are high. Many transfers therefore end up being made at the last minute. Emergency Department wait time, measured as the number of minutes patients wait between triage and entry into a cubicle, is shown in Table 8. This indicator also improves under the preventive policies, particularly Policy 3 where the wait times show declines of 63% to 84%. The explanation in this case is that ED congestion is caused by patients waiting not only for transfer to ICU but also for a bed in SDU or WARD, which Policy 3 handles more efficiently than the others. Table 9 Emergency Department cubicle availability ED cubicle availability Closely related to ED wait time is ED cubicle availability, measured as unoccupied cubicles per patient waiting to be hospitalized. The results for this indicator are set forth in Table 9. Once again, Policy 3 shows the greatest improvement over the base policy, increasing the rate by 21% and thereby boosting the department’s patient capacity. This latter effect leads in turn to lower ED queue abandonment rates. To appreciate the magnitude of the decline we first assume that patients arriving at ED who need to be hospitalized will not abandon (though they may be diverted to other hospitals if the queue is long). For those who do not need hospitalization and therefore are susceptible under this assumption to abandonment, we further assume their ED wait times before reneging follow a uniform distribution from 4 to 9 hours. Under Policy 0, it gives an abandonment rate of about 7%, which accords with the real data. This figure falls to just 0.1% under Policy 3, pointing clearly to an increase in the number of patients ED can attend to. The last indicator is the average occupancy for each unit, displayed in Table 10. As may be observed, Policy 3 shows lower rates for all units compared to Policy 0. This result was to be expected given that beds under Policy 3 are unoccupied for longer periods because they become available earlier. The biggest improvement is seen in WARD, where patient turnover is greatest. Policy 3 also improves on Policies 1 and 2 in that it makes better use of critical beds in ICU. This can be seen in the Policy 3 figure of 29.9 occupied beds versus 26.6 and 26.3 for Policies 1 and 2, a difference that is greater than that between Policy 3 and the base case Policy 0 figure of 31.4. Policy 0 Policy 1 Policy 2 Policy 3 24.6 (0.240) 26.6 (0.180) 26.6 (0.181) 29.7 (0.033) Figures in parentheses are coefficients of variation A proactive transfer policy for critical patient flow management Table 10 Average number of beds occupied, by policy and unit Unit (bed capacity) Policy 0 Policy 1 Policy 2 Policy 3 ICU (35 beds) SDU (50 beds) WARD (120 beds) 31.4 (0.124) [90%] 43.7 (0.123) [87%] 116.7 (0.051) [97%] 26.6 (0.173) [76%] 43.6 (0.125) [87%] 116.8 (0.049) [97%] 26.3 (0.176) [75%] 43.4 (0.123) [87%] 116.7 (0.051) [97%] 29.9 (0.128) [86%] 41.4 (0.131) [83%] 111.6 (0.067) [93%] Figures in parentheses are coefficients of variation, and percentages are utilization of capacity Lastly, we note that the results presented above were validated by the head of the SDU at the hospital that supplied the historical data for this study. 7 Discussion and conclusions The ultimate purpose of the proposed model is not, of course, to implement a mere simulation but rather to put the formulation into practice on a daily basis at a hospital facility. How this might be done was discussed with the doctor in charge at the SDU of the hospital that supplied the historical data for this study. The first conclusion to emerge was that the simulation results we obtained reflect well the real situation of a hospital in that beds were freed by the model on the basis of what patient arrivals are estimated to be in the immediately upcoming time period. A second conclusion is that it was reasonable for the model to be set up in such a way as to give priority to patients waiting in ED to be hospitalized. However, there are other certain extreme situations that were not well covered by the model. The hospital in our study assigns all beds to patients through General Admissions. If, for example, there is only one free bed in SDU when there should be three, General Admissions will request that SDU transfer out two of its patients. It will then have to be decided whether the free bed will be assigned to a patient transferring from ICU or one coming from ED. The optimal policy would call for assigning the bed to the latter patient in order to reduce ED congestion and its associated cost. In practice, however, if ICU is full, the decision may be to assign the SDU bed to the patient currently in ICU in order to free up a bed in the latter unit given that ED arrivals needing ICU care must have first priority. A third conclusion is that in the cases we tested, the results demonstrated that the model tends to give priority to a free bed in SDU rather ICU. This occurs mainly because of the difference in ED demand for transfer to the two units, emergency arrivals destined for SDU being up to 50% greater. In Instance 1 there were a total of 35 beds in ICU and 50, or only about 40% more, in SDU, yet the latter unit had to handle 50% more demand from ED as well as incoming transfers from ICU. In practice, therefore, the hospital prefers to maintain a free bed in ICU because if necessary an SDU patient can be treated there. Strictly speaking, this is an inefficient solution since it means the ICU bed is underutilized, but such a situation is preferable to leaving a patient waiting in a corridor. Our assumption that a patient requiring a bed in a given unit can only be transferred to that unit is what leads the model to give priority to SDU. The model further assumes an idealized patient flow, which does not allow for transfers from lower to higher complexity units or for discharges directly from ICU or SDU. This simplification was adopted even though both phenomena do occur in the real world after observing that their frequency is in fact very low. In particular, discharge from higher complexity units is not a recommended practice. The exclusion of these possibilities should therefore have little effect on the long run functioning of the proposed model. Their inclusion, on the other hand, would have significantly increased the complexity of the model without contributing much to a better solution. The comparisons of the simulation results with the real hospital data also confirmed that the former fell within the margins of what would be expected. As for the main ED performance indicators, significant improvements were recorded. Patient wait times declined 63% to 84% depending on the type of patient while patient capacity increased by 21%. This latter figure led in turn to a fall in the ED queue abandonment rate from 7% to 0.1%, thus boosting the number of patients the hospital could attend to. In practical terms, although obtaining an optimal policy is not trivial, a policy that makes efficient use of beds can be readily established if two points are observed: – – Demand in each unit should be estimated daily. This can be done using the facility’s historical data. Decisions made by the various units should be coordinated with each other. By having a single person charged specifically with this task (as is the case with the General Admissions unit in our case study), patient flows can be coordinated without much difficulty so that an efficient use of resources can be achieved. Finally, the present study assumed three decision-making levels corresponding to the major hospital units defined as high, medium and low complexity. In a future study, J. González et al. these units could be disaggregated to reflect specific patient requirements based on their individual diagnoses. The idealized flow assumption could be relaxed in order to investigate whether alternative flow possibilities would result in better solutions. Yet another extension to the proposed model would be to consider a network of hospitals in which patients may be diverted from one facility to another. Acknowledgements The authors thank their colleagues and students for helpful discussions and feedback at various stages of this research project. The authors also thank the three anonymous referees and the editor for helpful comments on earlier versions of this paper. Finally, the authors would like to thank for the financial support provided by FONDEF (Chile) grant no. CA13I10319. Funding This study was funded by FONDEF (grant number CA13I10319). Compliance with Ethical Standards Conflict of interests The authors declare that they have no conflict of interest. References 1. Ahmed MA, Alkhamis TM (2009) Simulation optimization for an emergency department healthcare unit in Kuwait. Eur J Oper Res 198(3):936–942 2. Ahn JH, Hornberger JC (1996) Involving patients in the cadaveric kidney transplant allocation process: a decisiontheoretic perspective. Manag Sci 42(5):629–641 3. Andersen AR, Nielsen BF, Reinhardt LB (2017) Optimization of hospital ward resources with patient relocation using Markov chain modeling. Eur J Oper Res 260(3):1152–1163 4. Ben Bachouch R, Guinet A, Hajri-Gabouj S (2012) An integer linear model for hospital bed planning. Int J Prod Econ 140(2):833–843 5. Cochran JK, Roche K (2008) A queuing-based decision support methodology to estimate hospital inpatient bed demand. J Oper Res Soc 59(11):1471–1482 6. Cochran JK, Roche KT (2009) A multi-class queuing network analysis methodology for improving hospital emergency department performance. Comput Oper Res 36(5):1497–1512 7. De Bruin AM, Van Rossum AC, Visser MC, Koole GM (2007) Modeling the emergency cardiac in-patient flow: an application of queuing theory. Health Care Manag Sci 10(2):125–137 8. De Vuyst S, Bruneel H, Fiems D (2014) Computationally efficient evaluation of appointment schedules in health care. Eur J Oper Res 237(3):1142–1154 9. Elalouf A, Wachtel G (2015) An alternative scheduling approach for improving patient-flow in emergency departments. Oper Res Health Care 7:94–102 10. de Farias DP, Van Roy B (2003) The linear programming approach to approximate dynamic programming. Oper Res 51(6):850–865 11. Hall R (2012) Bed assignment and bed management. In: Hall R (ed) Handbook of healthcare system scheduling. Springer, Boston, pp 177–200 12. Hauskrecht M, Fraser H (2000) Planning treatment of ischemic heart disease with partially observable Markov decision processes. Artif Intell Med 18(3):221–244 13. Kaandorp GC, Koole G (2007) Optimal outpatient appointment scheduling. Health Care Manag Sci 10(3):217–229. arXiv:1011.1669v3 14. Kolker A (2008) Process modeling of emergency department patient flow: effect of patient length of stay on ED diversion. J Med Syst 32(5):389–401 15. Kolker A (2009) Process modeling of ICU patient flow: effect of daily load leveling of elective surgeries on ICU diversion. J Med Syst 33(1):27–40 16. Lara BA, Cataldo A, Castro R, Aguilera PR, Ruiz C, Andresen M (2016) Medicina de urgencia y unidades de cuidados intensivos. Una alianza necesaria en busca de la mejorı́a de la atenciȯn de pacientes crı́ticos. Revista Mėdica de Chile 144(7):917–924 17. Lefevre C (1981) Optimal control of a birth and death epidemic process. Oper Res 29(5):971–982 18. Lehtonen JM, Kujala J, Kouri J, Hippelȧinen M (2007) Cardiac surgery productivity and throughput improvements. Int J Health Care Quality Assur 20(1):40–52 19. Luscombe R, Kozan E (2016) Dynamic resource allocation to improve emergency department efficiency in real time. Eur J Oper Res 255(2):593–603 20. Ma G, Demeulemeester E (2013) A multilevel integrative approach to hospital case mix and capacity planning. Comput Oper Res 40(9):2198–2207 21. Mallor F, Azcȧrate C (2014) Combining optimization with simulation to obtain credible models for intensive care units. Ann Oper Res 221(1):255–271 22. Niyirora J, Zhuang J (2017) Fluid approximations and control of queues in emergency departments. Eur J Oper Res 261(3):1110– 1124 23. Patrick J, Puterman ML, Queyranne M (2008) Dynamic multipriority patient scheduling for a diagnostic resource. Oper Res 56(6):1507–1525 24. Pepper S, Sepúlveda K, Campos J (2013) Informe Final Programa Compra a Privados Del Programa De Prestaciones Valoradas. Tech. rep., Ministerio de Salud. http://www.dipres.gob.cl/574/articles89691 doc pdf.pdf\r\n 25. Pines JM, Hilton JA, Weber EJ, Alkemade AJ, Al Shabanah H, Anderson PD, Bernhard M, Bertini A, Gries A, Ferrandiz S, Kumar VA, Harjola VP, Hogan B, Madsen B, Mason S, Ȯhlėn G, Rainer T, Rathlev N, Revue E, Richardson D, Sattarian M, Schull MJ (2011) International perspectives on emergency department crowding. Acad Emerg Med 18(12):1358–1370 26. Puterman ML (1994) Markov decision processes: discrete stochastic dynamic programming. Wiley 27. Rashwan W, Abo-Hamad W, Arisha A (2015) A system dynamics view of the acute bed blockage problem in the Irish healthcare system. Eur J Oper Res 247(1):276–293 28. Santibȧṅez P, Chow VS, French J, Puterman ML, Tyldesley S (2009) Reducing patient wait times and improving resource utilization at British Columbia Cancer Agency’s ambulatory care unit through simulation. Health Care Manag Sci 12(4):392– 407 29. Saurė A, Patrick J, Tyldesley S, Puterman ML (2012) Dynamic multi-appointment patient scheduling for radiation therapy. Eur J Oper Res 223(2):573–584 30. Schaefer AJ, Bailey MD, Shechter SM, Roberts MS (2004) Modeling medical treatment using Markov decision processes. Oper Res Health Care, 593–612 31. Schweitzer PJ, Seidmann A (1985) Generalized polynomial approximations in Markovian decision processes. J Math Anal Appl 110(2):568–582 32. Singer AJ, Thode HC, Viccellio P, Pines JM (2011) The association between length of emergency department boarding and mortality. Acad Emerg Med 18(12):1324–1329 A proactive transfer policy for critical patient flow management 33. Testi A, Tanfani E, Torre G (2007) A three-phase approach for operating theatre schedules. Health Care Manag Sci 10(2):163– 172 34. Thompson S, Nunez M, Garfinkel R, Dean MD (2009) OR practice--efficient short-term allocation and reallocation of patients to floors of a hospital during demand surges. Oper Res 57(2):261–273 35. Troy PM, Rosenberg L (2009) Using simulation to determine the need for ICU beds for surgery patients. Surgery 146(4):608– 620 36. Zeng Z, Ma X, Hu Y, Li J, Bryant D (2012) A simulation study to improve quality of care in the emergency department of a community hospital. J Emerg Nurs 38(4):322–328