A Reinforcement Learning Approach for Inventory Replenishment

in Vendor-Managed Inventory Systems With Consignment Inventory

Zheng Sui, Terra Technology

Abhijit Gosavi, Missouri University of Science and Technology
Li Lin, University at Buffalo, SUNY

Abstract: In a Vendor-Managed Inventory (VMI) system, • Electronics industry: Intel (Kanellos, 1998) and Hewlett-
the supplier makes decisions of inventory management for the Packard (Waller et al., 1999),
retailer; the retailer is not responsible for placing orders. There • Food manufacturing: Kraft Inc. and Mott’s USA (Emigh,
is a dearth of optimization models for replenishment strategies 1999),
for VMI systems, and the industry relies on well-understood, • Petrochemical: Shell Chemical (Hibbard, 1998)
but simple models, e.g., the newsvendor rule. In this article, • Men’s undergarments: Fruit of the Loom (Cetinkaya and Lee,
we propose a methodology based on reinforcement learning, 2000).
which is rooted in the Bellman equation, to determine a
replenishment policy in a VMI system with consignment The use of VMI at Wal-Mart on a large scale has attracted a
inventory. We also propose rules based on the newsvendor great deal of attention in the industrial world.
rule. Our numerical results show that our approach can In a VMI program, the supplier assumes control of the
outperform the newsvendor. inventory management for one or more retailers (Fry, 2002). The
supplier monitors the inventory level at the retailers and makes the
Keywords: Vendor-Managed Inventory, Supply Chains, decisions related to the quantities of replenishment and the timing
Simulation of shipments to the retailers. It has been claimed that this can
play a significant role in reducing the bull-whip effect. Compared
EMJ Focus Areas: Quantitative Methods & Models to traditional retailer-managed inventory (RMI) programs in
which retailers are responsible for placing orders and are entirely
responsible for their own inventory shortages or excesses, VMI

I
can bring benefits to both retailers and manufacturer, as discussed
n the past two decades, interest in supply chain management next. Some of the advantages for the retailer are as follows:
(SCM) has grown rapidly because of the increasing • Increase in the service level: As the supplier has access to
competition in today’s global markets. The introduction databases at the retailer and to the current demand at the
of products with shortening life cycles and the heightened retailer, the supplier can better coordinate its own activities
expectations of customers have forced businesses to invest in of supplying materials to its various customers. As a result,
and focus attention on their supply chain. A driving force behind the out-of-stock situation is rarely encountered at the retailer,
effective SCM is the effective collaboration of the numerous improving its service levels.
activities involved. A lack of coordination in the associated • Reduction of inventory levels: Since the supplier has access
activities in general results in the well-known bullwhip effect, to inventory databases at the retailer, the supplier is able
and more specifically in low service levels, high inventory, and to coordinate replenishment decisions in a way that keeps
high transportation costs. To overcome these problems, many inventory levels at the retailer from being excessive. This
forms of supply chain coordination, such as vendor-managed leads to reduced holding costs and increased inventory turns
inventory (VMI) and continuous replenishment and collaborative at the retailer.
forecasting, planning, and replenishment (CPFR), have been • Reduction of ordering and planning costs: Ordering and
implemented in recent years in supply chain software. Known as planning costs are eliminated (or reduced) for the retailer
direct replenishment or supplier managed inventory in the early since the responsibility of ordering/planning is shifted to
days, VMI was popularized in the late 1980s and is now widely the supplier.
used in various industries. According to a 2003 technology
survey (Automotive Warehouse Distributors Association, 2003), From the supplier’s perspective, the advantages are:
over 60% of manufacturers in the U.S. have implemented VMI. A • Ease in coordination of supply process: Since the supplier has
survey of electronic components (Survey, 2003) and a survey of access to the retailer’s inventory database, as well as its recent
the grocery industry (Kurt Salmon Associates, 2002) also support sales transactions, the supplier can more easily coordinate
the contention for a broad use of VMI. VMI has been widely used its own activities of supplying goods to various retailers. In
in the following industries: particular, it is not overloaded by orders at the beginning of
• Grocery: Campbell Soup Company (Clark and McKenny, the week or month as is the case in RMI. Often in RMI the
1994), and Barilla SpA (Hammond, 1994), supplier simply cannot meet all the demand since orders

Refereed management tool manuscript. Accepted by Associate Editor Murray.

44 Engineering Management Journal Vol. 22 No. 4 December 2010

 from different retailers tend to come at the same time. As a VMI systems in practice, and developed a solution that satisfied
result, the supplier can increase its own profits. a “newsvendor–type” relationship. Cheung and Lee (2002) built
• Reduced transporters: Ideally, since the supplier can better a model of coordinated shipment and stock rebalancing in the
coordinate its activities, it can do with fewer trucks or, at VMI system and examined the benefits of two initiatives to the
the very least, it can reduce the number of trips in which supplier and the retailers. Bernstein and Federgruen (2003)
the truck load is not full; a trip with “less-than-truckload” considered a “centralized” system, which is similar to a VMI
shipments are relatively expensive. system, in which the supplier determines sales quantities and the
complete chain-wide replenishment strategy. They considered
In a consignment-inventory VMI system, the supplier retains a problem in which the pricing of the product is an additional
ownership of the inventory at the retailer. Until the item is sold at decision variable for the retailer. Subramaniam and Gosavi (2007)
the retailer, payment is not made to the supplier, so the inventory- developed a simulation-based optimization approach based on
holding costs are absorbed by the supplier. In this article, we simultaneous perturbation to optimize the inventory dispatching
present a model for VMI systems with consignment inventory. policies; however, the policies they obtained are static and do not
Our optimization model will be built from the perspective of depend on the dynamically changing state of the inventory levels
the supplier, but VMI systems have significant benefits for both at the retailers.
suppliers and retailers.
The sudden shift in responsibility of managing retailers’ Contributions of this Article
inventory and dealing with the associated risks is “like jumping This research is focused on determining the replenishment
into a cold pool early in the morning” (Betts, 1994). Betts also quantities and the number of trucks dispatched by the vendor
quotes a vice president at a Cleveland firm as saying, “…if the to each retailer, assuming that truck routes are fixed and known.
scheme does not change the production process or squeeze out We make many assumptions that real-world systems share. In
excess costs and inventory, then VMI has really just shifted particular, we consider multiple non-identical retailers with
costs to the vendor.” In addition, VMI sometimes involves non-zero transportation times between retailers and the vendor
high transportation costs, since the supplier has to ship more and assume the transportation times to be random variables.
frequently to achieve inventory-turn targets at the retailers Transportation costs are also assumed to be non-negligible. We
(Copacino, 1993). These issues make it quite challenging for believe transportation costs should be an important part of any
the supplier to operate a VMI program. Some of the important VMI model, since transport frequency can be quite high in a VMI
questions that arise are: How much should one replenish, i.e., setting. Our solution methodology is based on reinforcement
what should the replenishment quantities be? If trucks are used learning (RL), which is an approximate dynamic programming
as transporters, how many truckloads should be dispatched in method rooted in the Bellman equation for the semi-Markov
a given day? decision process. To show the usefulness of our methodology, we
This article develops models that address these questions. conduct a series of numerical tests. In any reinforcement learning
Poor decision-making on these problems can prevent the supplier based application, unless there is a way to determine the optimal
from enjoying the benefits of VMI. Some MRP programs, such as solution, it is customary to compare the solution’s performance to
SAP and I2, have addressed this issue in designing their software; that of some other well-accepted approach, i.e., benchmark. This
however, the algorithms used are not always transparent to the is because RL requires a great deal of tuning and experimentation,
user, so it is unclear how appropriate these solutions are. and its behavior can be highly unpredictable; therefore, without
The literature on solving supply-demand problems is such a comparison, one is usually unsure of the quality of the
extensive. Versions of the problem we consider have been solution. We use the newsvendor model as the benchmark.
studied since Veinott (1965) and Evans (1967), where the focus Our numerical results show that our method outperforms the
was on developing optimal strategies for inventory allocation newsvendor model.
in multiple retailer scenarios. More recently, Higginson and The choice of the newsvendor as a benchmark is natural
Bookbinder (1995) used a Markov decision process in their here for at least three reasons. First, the newsvendor rule
model for shipment consolidation. Van Roy et al. (1997) also elegantly captures the tradeoff between under-stocking (penalty)
developed a model based on Markov decision processes and and over-stocking (holding) costs in a single-period periodic
use neuro-dynamic programming for solution purposes. Their review setting, to which class our problem belongs. Second,
model assumed identical retailers, no transportation costs, and the newsvendor model can be easily used by any engineering
fixed (non-random) transportation time, which allowed them manager within spreadsheet-based software. The usefulness of
to use a Markov decision process. DeCroix and Arreola-Risa new methods is usually best shown via comparisons to methods
(1998) have characterized an optimal policy for production and that can be easily implemented in practice, because favorable
inventory control under a finite resource and have also developed comparisons are likely to increase their practical appeal. Also,
a heuristic for solving the problem studied. Cetinkaya and Lee although the newsvendor rule was proposed a long time ago, its
(2000) present a model based on renewal (reward) theory in variants continue to be used in supply chain software today. The
which the timing of the shipment is determined along with its newsvendor, like the Economic Order Quantity (EOQ), is a well-
quantity. Axsater (2001) provided a simple procedure and an exact understood, transparent model that is also quite versatile and can
optimization procedure for the model. Chaouch (2001) developed be adapted easily to a large number of periodic-review settings
a VMI model with demand consisting of two components: one in supply chains. Finally, the recent work of Suo et al. (2004)
deterministic and one random. Their result showed that for a and Wong et al. (2008) in VMI systems is reflective of the fact
fixed delivery rate, the order-up-to level can be determined much that the newsvendor continues to be a baseline VMI model for
like the optimal stock level in the newsvendor model. Fry et al. academic research, while the solution in Fry et al. (2001) exploits
(2001) built an interesting model for a type of VMI agreement, a newsvendor-type relation for setting the upper and lower limits
called the (z, Z) contract, based on their analysis of a number of of their contract.

Engineering Management Journal Vol. 22 No. 4 December 2010 45

 Most of the existing analytical models in the literature are with demand size equaling 1. Also, the normal distribution is not
developed under a specific set of system assumptions that deviate easily usable in a low demand scenarios because it can become
from ours. The model in Bernstein and Federgruen (2003) negative (Nahmias and Smith, 1993). In addition, we assume
considers a pricing-cum-replenishment problem in a game- that the demand is signaled as either “high” or “low,” where a
theoretic setting in which they extensively use the EOQ formula. high demand is characterized by a higher arrival rate and a low
The model in Fry et al. (2001) does not consider transportation demand by a lower arrival rate. With this we have attempted to
costs, and is developed for a single-product single-retailer capture a modern trend in modeling demand in supply chains
combination. Further, theirs is not a consignment VMI system, called forecast updating (Sethi et al., 2001). Forecast updating is a
unlike ours, and their holding costs at the supplier and the retailer broader concept and requires updating of the entire forecast and
are modeled separately. Hence, under the assumptions that we perhaps even the demand distributions on the basis of signals
make, comparison of these models to our simulation-based received about the state of the demand. The customer’s demand,
methodology is not feasible. D, at any retailer for a given product in one cycle, can be modeled
N
as: D = d where
Problem Description
∑
i =1
i

In this article, we focus on a two-echelon system with one • di denotes the demand from the ith customer, which has a
Distribution Center (DC) and the retailers that the DC serves discrete uniform distribution.
(see Exhibit 1). We will assume that the route followed by • N is a Poisson distributed random variable with parameter λ;
the truck (or fleet of trucks) is already known, and we make essentially N stands for the number customers that arrive in
no attempt to optimize it. The supplier pays the holding cost one cycle, and the value of λ depends on i, the retailer.
of the inventory at the retailers (in a consignment inventory
VMI system). Whenever a stock-out occurs at the retailer, the Some additional notation that we need is:
supplier is penalized. The manufacturer has to decide upon the • cT: the transportation cost is the cost of operating one truck
timing and quantity of the order from the manufacturer to the for unit time.
DC, which we assume is done via a (Q, R) policy. The review of • hr: the holding cost of one item for unit time at a given
the inventory at the retailers is performed when the truck fleet retailer
comes back from the retailers to the DC, and the new decision is • pr: the cost of stock-out penalty at a given retailer.
to be made regarding the number of trucks to be sent in the next • Rev: the revenue generated for the supplier when the sale of
period. We refer to the period between two successive inventory a product takes place at the retailer.
reviews as a cycle. • Cap: the capacity of one truck.
• tDC-RET: the time of travel between DC and a retailer.
Exhibit 1. The 2-Echeleon System Considered in this Article • tRET-RET: the time of travel between retailers.
• tRET: the service time at the retailer.
Retailer Customer • tDC: the service time at the DC.
• t0: the lead time for the DC’s order from the manufacturer.
…

DC One of the other decisions to be made at the DC, which

our RL approach does not seek to optimize, is its own inventory
management policy. We will assume that it uses a (Q, R) policy
Retailer Customer (Askin and Goldberg, 2003). When such a policy is pursued, the
inventory level is observed continuously. As soon as it becomes
less than R, an order of size Q is delivered from the manufacturer
to the DC. There are a number of algorithms to find the optimal
values of Q and R (Askin and Goldberg, 2002). We let to denote
In one cycle, the following events occur: Customer demand the lead time for the DC’s order from the manufacturer.
arrives at the retailers. We model this via a compound Poisson At the start of each cycle, the inventory level of the retailers,
process. The retailer’s demand forecast update refers to whether x⃗ the inventory level of the DC, y,⃗ and the demand forecast of the
the demand has been predicted to be high or low at the retailer. ⃗
retailers, I , are observed. Then the decision to be made is related
The demand forecast for retailer i is denoted by I(i), and the to (i) the number of trucks to be sent and (ii) the allocation of
n-tuple I⃗ collectively denotes the demand forecast at all retailers. the amounts of each product to the total amount within each
The retailers’ current inventory levels are denoted by the n-tuple truck. Sub-optimal decisions can either lead to excessive holding
⃗ with x(i) denoting the inventory at the ith retailer. Similarly,
x, costs or stock-out costs at the retailers. Indeed, it can also lead
the DC’s inventory level is denoted by another n-tuple y. ⃗ After to excessive holding costs for one product and stock-out costs
the products are loaded onto the trucks, the truck fleet departs for some other product. We will discuss our RL approach in the
from the DC. The majority of the literature assumes that the next section. We now describe a robust approach based on a well-
distribution of the demand at the retailers is either normal known paradigm.
(Gavirneri et al., 1999; Jackson, 1988) or Poisson with a demand
of size always equaling 1 (Cetinkaya and Lee, 2000; Deuermeyer Newsvendor Solution
and Schwarz, 1981; Graves, 1996). In this work, we assume the One approach to solving this problem is to use the newsboy or
customer demand to be a compound Poisson process, where the newsvendor rule. The newsvendor rule is designed for perishable
demand size is assumed to have the discrete uniform distribution. items that have a holding cost and a stock-out cost. In order to
Our use of this distribution is motivated by the fact that the adapt the model to our problem setting, we need some additional
compound Poisson process is more general than a Poisson process work that we now present.

46 Engineering Management Journal Vol. 22 No. 4 December 2010

Exhibit 2. Lead Time and Cycle Time

Current Next Lead time Truck fleet arrives at

decision decision (L) the retailer

Current cycle Next cycle

(T)

The value of the retailer’s order up-to level S* is determined in and variance σ2next of the customer demand can be computed as
the newsvendor model by solving: follows:
p µnext = E ( I ) ⋅ E ( N ) ⋅ E (d )
F ( S *) =
p+h E ( I ) ⋅ λ ( a + b)
where p is the penalty cost, h is the holding cost, and F(.) is = ,
2
the cumulative distribution function of the customer demand in
the cycle. Let T denote our cycle time and L denote the lead time
2
σ current = E 2 ( I ) ⋅  E ( N )Var (d ) + Var ( N ) E 2 (d ) 
for the retailer in question (see Exhibit 2). L is equal to the truck
fleet transportation time from the DC to the retailer in question. 2  (b − a ) 2 a+b 2
Hence, the inventory replenishment decision made at the start of = E ( I ) ⋅ λ ⋅ + λ ⋅( ) 
 12 2 
the current cycle will affect the inventory system from the time
the current cycle starts until the truck fleet arrives at the retailer 2 2 2
E ( I ) ⋅ λ (a + b + ab)
in the next cycle. In our setting, the customer demand is assumed = .
to be a compound Poisson process with a demand forecast update 3
and is defined by: N Again, assuming that L is a discrete random variable, the
D = ∑ di mean μlead and the variance σ2lead of the demand experienced
i =1 during L time units in the next cycle can be computed as:
where di, the amount of demand for ith customer, has a discrete
uniform distribution, (U(a, b)). In the current cycle, the value µlead = µnext ⋅ µ L ,
of the demand forecast update (I) is known and will be denoted 2 2 2
σ lead = µ Lσ next + µnext σ L2 ,
by g. So in unit time, the mean μcurrent and variance σ2current of the
customer demand can be given as (Ross, 2002): where μL and σ2L are the mean and the variance of L respectively.
µcurrent = g ⋅ E ( N ) ⋅ E (d ) Then, via the central limit theorem, the customer demand in the
replenishment cycle (T) and lead time (L) can be approximated
g λ ( a + b) (Ross, 2002) as:
= , µ = µcycle + µlead ,
2
σ 2 = σ cycle
2 2
+ σ lead .
2
σ current = g 2  E ( N )Var (d ) + Var ( N ) E 2 (d ) 
If the demand is normally distributed with mean μ and
 (b − a ) 2 a+b 2 variance σ2, the optimal order up-to level (S*) is:
= g 2 λ ⋅ + λ ⋅( )  p
 12 2  S * = φ −1 ( ) ⋅σ + µ ,
2 2 2 p+h
g λ (a + b + ab)
= . −1
where φ (.) is the inverse of the cumulative distribution
3 function of the standard normal distribution. Let S*(i,k) denote
Now if we assume T to be a discrete random variable, the the order up-to level of retailer i and product k based on the
mean μcycle and the variance σ2cycle of the demand during T time newsvendor heuristic. From these values, one can determine the
units can be computed (Nahmias, 2001) as: number of trucks to be sent as follows:
µcycle = µcurrent ⋅ µT ,
∑�𝑖=� ∑� ∗
𝑘=�(𝑆 (𝑖, 𝑘) − 𝑥𝑖𝑘 )
2
σ cycle 2
= µT σ current 2
+ µcurrent σ T2 , 𝑎= � �
𝑐𝑎𝑝
where μT and σ2T are the mean and the variance of T. During
L time units in the next cycle, the value of the demand forecast where ⟦. ⟧denotes the nearest integer of the quantity inside
update I is unknown; therefore we use the expected value of I, the brackets and xik denotes the current inventory for retailer i
E(I), to estimate it. In unit time of the next cycle, the mean μnext and product k.
Engineering Management Journal Vol. 22 No. 4 December 2010 47
Cost Structure 𝐸{∑��=�(𝑟(𝑖� , 𝜋(𝑖� ), 𝑖��� )|𝑖� = 𝑠)}
We summarize our cost economics model used in our simulation liminf
model. We have four elements for the costs and revenues: the
�→� 𝐸{∑��=�(𝑡(𝑖� , 𝜋(𝑖� ), 𝑖��� )|𝑖� = 𝑠)}
holding cost (hr) at the retailers for each product (note that this
cost is absorbed by the supplier in our model), the revenues where E denotes the expectation operator and im denotes the state
(Rev) transmitted to the supplier when a sale occurs, the stock- in the mth jump of the embedded Markov chain associated with
out penalty (pr) which is transmitted to the supplier, and the policy π. It can be shown that if the Markov chain associated
transportation cost (cT) per truck per unit time for the supplier. with every policy is “regular,” then the above criterion does not
depend on the starting state. Our goal in average-reward SMDPs
Solution Methodology is to maximize the above function, which is also called the long-
The problem that we have presented above can also be solved via run average reward of the SMDP. As 𝛾 → 0 , which implies that
a more detailed model that looks at the Markov chains underlying exp(−𝛾𝑡)
→ 1, from the Bellman equation for discounted reward,
the VMI system. In this section, we first present the underlying it can be shown that optimization with respect to discounting
stochastic process that we have used for developing the model becomes equivalent to optimizing with respect to the average-
and then discuss the method used for solution. reward criterion. Use of such a small value of 𝛾, so that the discount
factor tends to 1, is called a vanishing-discount approach. In this
Semi-Markov Decision Process article, we will use a discounting algorithm, but via the vanishing
The stochastic process that we use in our model is called the semi- discount approach will solve an average-reward problem.
Markov decision process (SMDP) (Bertsekas, 1995). The SMDP is SMDPs can be solved by via classical dynamic programming
characterized by a set of states, actions, rewards, transition times, method (Bertsekas, 1995), provided the number of states is small
and transition probabilities. In each decision-making state, the and the system has a tractable transition probability model; however,
agent can choose from a set of actions. The random trajectory for a problem with a large number of states, which is true of the
followed by the system depends on the action chosen in each state problem considered in this article, it generally becomes difficult to
and upon the random variables that govern the system’s behavior. store all the transition probabilities, p(.,.,.), and the elements of the
Underlying the semi-Markov process is an embedded Markov value function, J(.). This is attributed to the curse of dimensionality
chain. Let p(i,a,j) denote the transition probability of transitioning (Bellman, 1957). If the transition probabilities are too difficult to
from state i to state j under the influence of action a in the Markov find in an exact manner because of the complex stochastics of
chain associated to action a. Similarly, let r(i,a,j) denote the the underlying random process, the analysis is said to have the
immediate reward (revenue) earned in the same transition. Then, curse of modeling (Sutton and Barto, 1998); this may be true even
the expected immediate reward earned in state i by selecting of problems with a few states. One approach to avoid these twin
action a is: 𝑟̅(𝑖, 𝑎) = � 𝑝(𝑖, 𝑎, 𝑗)𝑟(𝑖, 𝑎, 𝑗). In an SMDP model, the time curses of dynamic programming is to use the methodology called
of transition from one 𝑗
state to the next is also a random variable reinforcement learning (RL), which we next describe.
with an arbitrary distribution. So if t(i,a,j) denotes the mean time
in transiting from state i to state j under a, the mean time in any Reinforcement Learning (RL)
transition out of i under a is given by RL has attracted a considerable amount of attention recently (see
𝑡̅(𝑖, 𝑎) = � 𝑝(𝑖, 𝑎, 𝑗)𝑡(𝑖, 𝑎, 𝑗). Sutton and Barto, 1998; Bertsekas and Tsitsiklis, 1996; Gosavi,
𝑗 2003) for textbooks on this topic). Watkins (1989) discovered
Typically, when we study the system over an infinite time the Q-Learning algorithm, which is one of the most popular
horizon, the objective function is to maximize a cumulative algorithms in RL, to solve the MDP in a simulator. It was
function of the immediate rewards (or costs) earned in each extended to the SMDP in Bradtke and Duff (1995) where they
state transition. Two performance metrics frequently used used a continuous reward rate. We use the algorithm proposed
in stochastic control theory are: the average reward, which in Gosavi (2003) for the lumpsum reward that is applicable to the
is the expected reward per unit time calculated over an problem here. The algorithm’s convergence has been established
infinitely long trajectory, and the discounted reward, which in Gosavi (2007). The algorithm is described next.
is the expected total discounted reward calculated over an
infinitely long trajectory. Solving the SMDP with respect to the Steps in the Q-Learning Algorithm
discounted reward metric requires solution of the following Step I. Let S denote the set of states, and A(i) denote the set of
Bellman equation: actions allowed in state i. Initialize the Q-factors, Q(i,u) = 0 for
∞ all i Є S and all u Є A(i). Set m=0 and γ to a very small positive
𝐽(𝑖) = max𝑎Є�(𝑖) �� 𝑝(𝑖, 𝑎, 𝑗) �𝑟(𝑖, 𝑎, 𝑗) + � exp(−𝛾𝑡) 𝐽(𝑗)𝐹𝑖𝑎 (𝑡) 𝑑𝑡��
, number, e.g., 0.001. Compute the step-size using a rule such as α
�
𝑗
= A/(B+m), where A and B are constants, e.g., A= 99 and B=100.
where Fia(t) denotes the cumulative distribution function of the Set MAX_STEPS to a large positive integer, e.g., 10,000. Start
time spent in transition out of i under a, 𝛾 denotes the discount system simulation.
rate (with exp(−𝛾𝑡) being the discount factor during a time interval
of length t), A(i) denotes the set of actions allowed in state i, and Step II. While m < MAX_STEPS do:
J(i) denotes the value function for state i. The solution to the Let the system start in state i.
SMDP can be expressed as a policy π, which prescribes π (i) as the 1. With probability of 1/|A(i)| (note that |X| denotes the
action to be taken in state i. number of elements in set X), choose an action a Є A(i) that
We now consider the average reward performance metric. maximizes Q(i,a). (In other words, compare all the Q-factors
If the starting state for the system is s, the average reward for state i, and choose the action for which the Q-factor is
(performance criterion) of a given policy π can be defined as: the maximum).

48 Engineering Management Journal Vol. 22 No. 4 December 2010

2. Simulate the chosen action a. Let the system state at the next In most complex problems, however, the Q-factors tend to be
decision epoch be j. non-linear functions, whose closed forms are unknown. In such
3. Update Q(i,a) using the following rule: a scenario, the use of back-propagated neural networks has been
𝑄(𝑖, 𝑎) ← (1 − 𝛼)𝑄(𝑖, 𝑎) + 𝛼[𝑟(𝑖, 𝑎, 𝑗) + exp (−𝛾𝑡(𝑖, 𝑎, 𝑗)max𝑏��(𝑗) 𝑄(𝑗, 𝑏)]
widely cited in the literature on RL (Tesaru, 1995; Crites and
Barto, 1998). Hence, in this article, we use a back-propagated
4. Set the current state i to the new state j. Increment m by 1, neural network (Werbos, 1974) to approximate the Q-factors.
re-compute α, and then go to Step II(1 ). See Exhibit 3.
Although the RL algorithm can help identify the number of
When the algorithm has run for MAX_STEPS, we can trucks to be dispatched at the start of the cycle, it cannot solve
identify the policy π returned from the Q-factors as follows. For the problem of allocating the inventory to different products
all i Є S,
𝜋(𝑖) = argmax𝑎��(𝑖) 𝑄(𝑖, 𝑎) . and retailers. To this end, we present a technique to allocate the
It is to be noted that the algorithm given above does not inventory that must “help” the RL algorithm.
need the transition probabilities of the underlying Markov
chains, but can be used within a simulator. Thus the algorithm Inventory Allocation
avoids the curse of modeling. Dynamic programming algorithms An allocation method for the products to be sent in a given truck
require these transition probabilities, which are notoriously hard has been discussed in the literature of inventory management since
to compute in a real-world problem such as the one we study the late 1950s. Simpson (1959) discusses an optimum allocation
here; however, the problem of having to store a large number of policy to minimize the weighted number of lost sales, in which
Q-factors remains. This issue is resolved with the use of function the necessary condition is to equalize a weighted probability.
approximation that we next describe. Simpson’s allocation method has two drawbacks that make it
unsuitable for the problem in this research. First, only lost sales
Function Approximation are considered in the allocation method; the holding cost at each
The idea underlying function approximation is to model the retailer is also not taken into account. The other drawback is that
Q-factors for a given action as a function of the state, and it is not easily implemented when the retailers are not identical.
store only the relatively smaller number of scalars that define This drawback is also seen in other works (Jackson, 1988; Jonsson
a function instead of storing all the Q-factors explicitly. This is and Silver, 1987). Bertrand and Bookbinder (1998) consider non-
said to avoid the curse of dimensionality. Of course, the function identical retailers, and use a search procedure to determine the
that we store should be an appropriate one. In other words, the allocation method. Using these ideas in the literature, we propose
function should return a reasonably accurate estimate of the a rule for inventory allocation, which can work for non-identical
Q-factor’s value when the relevant state and action are fed into retailers, and is computationally easy. The aim here is to make the
it. For instance for a 2-action problem, consider the following ratio of the retailer’s inventory after allocation, which equals the
linear representation (where the state is a scalar): Q(i,a) = Ω + sum of the current retailer’s inventory and the allocated inventory,
κi for a =1 and Q(i,a)=ψ + ωi for a=2, where i denotes the state. to the retailer’s order up-to inventory level (computed via the
Then instead of storing the Q-factors for all state-action pairs, we newsvendor model), the same for each retailer and each product.
𝑥 + 𝐼𝑛𝑣(𝑖, 𝑘)
can do with storing only 4 scalars: Ω, ω, κ, and ψ. This strategy That is, 𝑆 (𝑖, 𝑘) should be roughly equal for all (i, k) pairs, where
𝑖𝑘
∗

works well if the Q-factors are linear or nearly linear functions. xik is the current inventory of retailer i and product k, and Inv(i,k)

Exhibit 3. The Neural-Network Architecture for the Function Approximator Used

Sum of retailers’ inventory x

i
i

Feature extraction

State space

Region

layers layers

Inputs output … Inputs output

(x, y, I) (x, y, I)

Back -propagation neural network Back -propagation neural network

Engineering Management Journal Vol. 22 No. 4 December 2010 49

is the inventory allocated to retailer i and product k. We must Exhibit 5. Additional Parameters for Our Numerical Experiments
also ensure that � � 𝐼𝑛𝑣(𝑖, 𝑘) = 𝑎 . 𝑐𝑎𝑝. . The computational scheme for this
𝑖 𝑘

can be described as follows: For every (i,k) combination, compute Values

the following:
Parameters
(∑𝑖 ∑𝑘 𝑥𝑖𝑘 + 𝑎. 𝑐𝑎𝑝)𝑆 ∗ (𝑖, 𝑘) Product 1 Product 2
𝑌(𝑖, 𝑘) = � � − 𝑥𝑖𝑘 .
∑𝑖 ∑𝑘 𝑆 ∗ (𝑖, 𝑘)
hDC 0.005 0.005

(In the above, ⟦. ⟧ denotes the nearest integer for the quantity pDC 1 1
inside the brackets.) If Y(i,k)< 0, Inv(i,k) =0, and otherwise
Inv(i,k)= Y(i,k). Note that this rule uses the newsvendor (Q, R) (500, 150) (500, 150)
relationship.
K 50 50
Numerical Experiments
In this section, we describe the results of our numerical tDC-ret Uniform (2, 4)
experiments with the technique developed in the previous
section. We will consider a scenario with two products and ten tret-DC Uniform (2, 4)
retailers. The parameters of each product at different retailers are
listed in Exhibit 4. The values of the parameters of each product tret-ret Uniform (0.5, 1)
at the DC, the transportation time, and truck-related information
are listed in Exhibit 5. The parameters in Exhibits 4 and 5 present tDC Uniform (0.2, 0.3)
our baseline case. The parameters for the other cases are defined
in Exhibit 6. Exhibit 7 presents the results obtained with all the tret Uniform (0.01, 0.015)
eight cases that we study.
t0 Uniform (30, 50) Uniform (30, 50)
Exhibit 4. Parameters (Demand Arrival, Holding Costs, Stock-Out
Penalties and Revenues) for the Retailers and the Products
cT 10

Cap 100
Demand
Product Retailer λ hr pr Rev
Uniform(a,b)

1 0.25 (1, 2) 0.06 4 5 Exhibit 6. Designed Experiment for Parameter Values

2 0.5 (0.5, 1.5) 0.05 4 5

Factors Level (-1) Level (+1)
3 0.3 (1, 2) 0.03 4 5

4 0.25 (1, 2) 0.04 4 5 pr Original values Increase the value by 50%

5 0.1 (2, 4) 0.03 4 5
1 hr Original values Increase the value by 100%
6 0.2 (1, 3) 0.05 4 5

7 0.3 (1, 1.5) 0.03 4 5 λ Original values Increase the value by 50%

8 0.5 (0.5, 1.5) 0.06 4 5

9 0.15 (2, 3) 0.04 4 5 We now describe how our value function is approximated.
The inventory at each retailer and DC is encoded using
10 0.2 (1, 3) 0.05 4 5 “buckets” (Sutton and Barto,
 1998). The state is composed of:
1 0.5 (0.5, 1.5) 0.04 3 5 the retailers’ inventory x = {x11 ,..., xmn } , where xkj denotes
 of the kth product at the ith retailer; the
the retailer’s inventory
2 0.1 (2, 3) 0.06 3 5 DC’s inventory y = { y1 ,..., ym } , where yk denotes the DC’s
3 0.15 (1, 3) 0.04 3 5 inventory of the kth product; and the retailers’ demand forecast
update I = {I11 ,..., I mn } , where Iik denotes the forecast update
4 0.3 (0.5, 2) 0.05 3 5 of the kth product at the ith retailer; m is the number of the
product items and n is the number of the retailers. The actual
5 0.35 (0.5, 1.5) 0.03 3 5
2 state space is too large to store all the Q-factors explicitly. We
6 0.25 (1, 2) 0.06 3 5 used a neural network for which typically the state space must
first be encoded. We encode the inventory to generate signals
7 0.4 (0.5, 1.5) 0.04 3 5 (levels) for the neural network as follows. Let the inventory for
8 0.2 (1, 3) 0.03 3 5 a given retailer have a maximum value of b and a minimum
value of a, with c, d, and e, chosen in a manner such that:
9 0.15 (1.5, 2.5) 0.05 3 5 a < c < d < e <b and c – a = d – c = e – d = b –e. Inventory values
10 0.25 (1, 2) 0.03 3 5 from a to c will be assumed to have a signal of 1, those from c to

50 Engineering Management Journal Vol. 22 No. 4 December 2010

Exhibit 7. Results of Using the RL Algorithm and the Newsvendor Heuristic

Newsvendor
Case pr hr λ RL-based algorithm % Improvement
heuristic

1 -1 -1 -1 15.41 16.31 5.84

2 -1 -1 +1 25.68 26.75 4.17

3 -1 +1 -1 7.55 8.15 7.95

4 -1 +1 +1 15.01 15.86 5.66

5 +1 -1 -1 14.86 15.7 5.65

6 +1 -1 +1 24.99 25.93 3.76

7 +1 +1 -1 6.39 6.80 6.42

8 +1 +1 +1 13.60 14.21 4.49

d will generate a signal of 2, those from d to e a signal of 3, and Conclusions

those from e to b a signal of 4. For the case of two products and ten Optimizing the replenishment quantities for a VMI system
retailers, we used 5 signal levels for the inventory of each product is a problem that has been studied in the literature mostly
at the retailers, 3 signal levels for inventory of each product at via analytical models. Usually, in the process of constructing
the DC, and 2 signal levels of each demand forecast update (one analytical models, one has to make simplifying assumptions
for high and one for low) for each product. We assume three about the system. In this article, we presented a simulation-
actions: 0 trucks, 1 truck and 2 trucks. The number of these based approach to determine the replenishment quantities. The
encoded Q-factors is 52×10 × 32 × 22×10 × 3 = 2.7 ×1021 . (Note attractive feature of the simulation-based approach is that one
that before the encoding, the state space is even larger!) With can make realistic assumptions about the system. We used a
the look-up table method in which the values of the encoded reinforcement-learning model based on semi-Markov decision
Q-factors are stored explicitly, if it takes one byte to store a value processes for solution purposes. Our model requires solution
of Q-factor, we will need a memory of 2.7 ×1012 GB to store all of the newsvendor problem. As a by-product, our analysis
the Q-factors, which is clearly unrealistic for current computer also resulted in a robust newsvendor heuristic. Our approach
technologies. The state space is separated into 50 regions. In outperformed the newsvendor solution by at least 4% in all
each region, we place a neural network to approximate the our experiments. Our computer programs generated solutions
values of the Q-factors in that region (See Exhibit 3). The (Q,R) within minutes for the problems we solved, which we view as
policy for the DC is computed using the technique in Nahmias a positive outcome. As such, our approach can easily be used
(2001). We run each case for 1,000,000 units of simulation time by an engineering manager. The newsvendor heuristic can be
in the learning stage, and 10 replications with each replication implemented within any standard spreadsheet software.
lasting 100,000 units of simulation time in the frozen stage. For There are some clear implications of our work for the
Case 1 of our experimental setup (see Exhibit 7), the average practicing engineering manager. When VMI systems are used,
profit for the entire VMI system using the RL-based algorithm they need to be carefully optimized. Newsvendor-based rules
proposed in the work is 16.31. Via the newsvendor heuristic, and simulation-based approaches, such as the one proposed here,
the average profit is 15.41. The RL-based algorithm outperforms can produce a significant impact on system profits. The computer
the newsvendor heuristic by 5.84%. Note that the RL algorithm programs needed for these approaches can produce solutions
uses the newsvendor in deriving its allocation strategy. The within minutes. The newsvendor-based rules can be programmed
highest improvement over the newsvendor is about 8%, and within spreadsheet software.
the lowest improvement is about 4%. The problems that we We should point out that our simulation-based approach
solved did not require more than 10 minutes on a PC. These are could potentially be used in any supply chain where the supplier
encouraging results because they show that our RL algorithm has to determine the shipment quantities to be sent to multiple
outperforms a newsvendor. The newsvendor heuristic, retailers. That is a problem with a broader scope; however, some
which is essentially a by-product of our analysis, appears to of the assumptions that we have made, e.g., holding and stock-out
be a solid method in its own right; what is attractive about costs absorbed by suppliers, would have to be altered to construct
the newsvendor is that it can be implemented easily on any a more general model. It is not difficult to make such changes
spreadsheet software. in the simulation model, and hence this can potentially form an

Engineering Management Journal Vol. 22 No. 4 December 2010 51
avenue for additional research. A number of other directions for Innovations,” in Harvard Business School Case, Harvard
future research can be envisioned. Business School, Harvard University (1994).
First, the policy considered in this article belongs to the Crites, Robert, and Andrew Barto, “Elevator Group Control
periodic review class in which the inventory is reviewed at the Using Multiple Reinforcement Learning Agents,” Machine
start of the cycle, when the truck (or trucks) comes back. There is Learning, 33:2-3 (1998), pp. 235-262.
a body of literature that looks at (Z,z) type of policies (including in DeCroix, Gregory A., and Antonio Arreola-Risa, “Optimal
particular Fry at al, 2001) for VMI systems. An exciting challenge Production and Inventory Policy for Multiple Products under
would be to use reinforcement learning for deriving such policies. Resource Constraints,” Management Science, 44:7 (1998),
Second, optimization of the routes for the truck fleet along with pp. 950-961.
optimization of the replenishment quantities is another direction Deuermeyer, Brian L., and Leroy Schwarz, “The Model for the
in which the simulation-optimization procedure could be Analysis of System Service Level in Warehouse/Retailer
developed. It turns out that there is an interesting tabu-search Distribution Systems: The Identical Retailer Case,” in Multi-
procedure (Gendreau et al, 1994) that could be integrated within Level Production/Inventory Control Systems: Theory and
the simulation-optimization framework. Third, optimizing the Practice, North-Holland (1981).
system for both the retailer and the supplier would require a game- Emigh, Jacqueline, “Vendor-Managed Inventory,” Computerworld,
theoretic formulation. This would perhaps require first showing 33 (1999), pp. 52-55.
the existence of Nash equilibria, but identifying a solution that is Evans, Richard V., “Inventory Control of a Multiproduct System
optimal for both the retailers and the suppliers should provide with a Limited Production Resource,” Naval Research
insights that our single-agent optimization model cannot Logistics Quarterly, 14:2 (1967), pp. 173-184.
provide. Finally, including the manufacturer in the optimization Fry, Michael J., Collaborative and Cooperative Agreements in the
model will form a very interesting topic for further research. The Supply Chain, unpublished Ph. D dissertation, University of
full impact of the bull-whip effect can only be studied if all three Michigan (2002).
levels are taken into account. It will be also interesting to consider Fry, Michael J., Roman Kapuscinski, and Tava L. Olsen,
the impact of risk (Bahill and Smith, 2009) within the analysis. “Coordinating Production and Delivery Under a (z, Z)-Type
Vendor-Managed Inventory Contract,” Manufacturing &
References Service Operations Management, 3:2 (2001), pp. 151-173.
Askin, Ronald., and Jeffrey Goldberg, Design and Analysis of Gavirneni, Srinagesh, Roman Kapuscinski, and Sridhar Tayur,
Lean Production Systems, Wiley (2002). “Value of Information in Capacitated Supply Chains,”
Automotive Warehouse Distributors Association, 2003 AWDA Management Science, 45:1 (1999), pp. 16-24.
Technology Survey, www.awda.org (2003). Gendreau, Michel, Alain Hertz, and Gilbert Laporte, “A
Axsater, Sven, “A Note on Stock Replenishment and Shipment Tabu Search Heuristic for the Vehicle Routing Problem,”
Scheduling for Vendor-Managed Inventory Systems,” Management Science, 40:10 (1994), pp. 1276-1290.
Management Science, 47:9 (2001), pp. 1306-1310. Gosavi, Abhijit, Simulation-Based Optimization: Parametric
Bahill, Terry, and Eric D. Smith, “An Industry Standard Risk Optimization Techniques and Reinforcement Learning,
Analysis Technique,” Engineering Management Journal, 21:4 Springer (2003).
(2009), pp. 16-29. Gosavi, Abhijit, “Adaptive Critics for Airline Revenue
Bellman, Richard E., Dynamic Programming, Princeton University Management,” in Proceedings of the 18th Annual Conference of
Press (1957). the Production and Operations Management Society (2007).
Bernstein, Fernando, and Awi Federgruen, “Pricing and Graves, Stephen C., “A Multi-Echelon Inventory Model With
Replenishment Strategies in a Distribution System with Fixed Reorder Intervals,” Technical Report, Sloan School of
Competing Retailers,” Operations Research, 51:3 (2003), pp. Management, MIT, Cambridge (1996).
409-426. Hammond, Janice, “Barilla SpA (A) and (B),” In Harvard Business
Bertrand, Louise P., and James H. Bookbinder, “Stock Redistribution School Case. Cambridge, MA: Harvard Business School,
in Two-Echelon Logistics Systems,” The Journal of the Harvard University (1994).
Operational Research Society, 49:9 (1998), pp. 966-975. Hibbard, Janet, “Supply-Side Economics,” InformationWeek, 707
Bertsekas, Dimitri P., Dynamic Programming (2nd ed.), Athena (1998), pp. 85-87.
Scientific (1995). Higginson, James, and James Bookbinder, “Markovian Decision
Bertsekas, Dimitri P., and John Tsitsiklis, Neuro-Dynamic Processes in Shipment Consolidation,” Transportation
Programming, Athena Scientific (1996). Science, 29:3 (1995), pp. 242-255.
Betts, Mitch, “Manage My Inventory Or Else,” Computerworld, Jackson, Peter L., “Stock Allocation in a Two Echelon Distribution
28:5 (1994), pp. 93-95. System or ‘What to do Until Your Ship Comes In’,” Management
Cetinkaya, Sila, and Chung Y. Lee, “Stock Replenishment and Science, 34 (1988), pp. 880-895.
Shipment Scheduling for Vendor-Managed Inventory Jonsson, Horace, and Silver, Edward A., “Analysis of a Two-
Systems,” Management Science, 46:2 (2000), pp. 217-232. Echelon Inventory Control System with Complete
Chaouch, Benjamin A., “Stock Levels and Delivery Rates in Redistribution,” Management Science, 33 (1987),
Vendor-Managed Inventory Programs,” Production and pp. 215-227.
Inventory Management, 10:1 (2001), pp. 31-44. Kanellos, Michael, “Intel to Manage PC Inventory,” http://ww.cnet.
Cheung, Ki L., and Hau Lee, “The Inventory Benefit of Shipment com (1998).
Coordination and Stock Rebalancing in a Supply Chain,” Kurt Salmon Associates., Survey of Supply Chain Effectiveness,
Management Science, 48:2 (2002), pp. 300-306. Food Distribution International (2002).
Clark, Theodore, and James L. McKenny, “Campbell Soup Nahmias, Steven, Production and Operations Analysis (4th ed.),
Company: A Leader in Continuous Replenishment McGraw-Hill (2001).

52 Engineering Management Journal Vol. 22 No. 4 December 2010

Nahmias, Steven, and Stephen A. Simth, “Mathematical Models Werbos, Paul J., Beyond Regression: New Tools for Prediction
of Retailer Inventory Systems: A Review,” in R. Sarin (Ed.), and Analysis of Behavioral Sciences, Ph.D. thesis, Harvard
Perspectives on Operations Management (pp. 249-278), University, Cambridge, MA (1974).
Kluwer Academic Publishers (1993). Wong, Wai-Keung, Jian Qi, and Sunney Leung, “Coordinating
Ross, Sheldon M., Introduction to Probability Models (8th Ed.) Supply Chains with Sales Rebate Contracts and Vendor-
Academic Press (2002). Managed Inventory,” to appear in International Journal of
Sethi, Suresh P., Han Yan, and Qin Zhang, “Peeling Layers of Production Economics (2009).
an Onion: Inventory Model with Multiple Delivery Modes
and Forecast Updates,” Journal of Optimization Theory and Acknowledgements
Applications, 108:2 (2001), pp. 253-281. The authors thank the two anonymous reviewers for numerous
Simpson, Kenneth F., “A Theory of Allocation of Stocks to comments that improved the quality of the article and the special
Warehouses,” Operations Research, 7:6 (1959), pp. 797-805. issue editor Dr. Susan Murray for her suggestions.
Subramaniam, Ganesh, and Abhijit Gosavi, “Simulation-Based
Optimization for Material Dispatching in a Retailer Network,”
International Journal of Simulation and Process Modeling, 3:4 About the Authors
(2007), pp. 238-245. Zheng Sui holds a BS in plastic forming engineering, an MS in
Suo, Hansheng, Jingchun Wang, and Yihui Jin., “Coordinating a mechanical engineering from Shanghai Jiao Tong University,
Loss-Averse Newsvendor with Vendor Managed Inventory,” and a PhD in industrial engineering from University at
in Proceedings of the IEEE Conference on Systems, Man, and Buffalo, SUNY. He currently serves as a product manager
Cybernetics (2004), pp. 6026-6030. in Terra Technology, where he uses his knowledge of supply
Survey, Distributor Customer Evaluation Survey on electronic chain and optimization to design and develop supply chain
components Electronic Buyers News (2003). software which has been used in some of the world’s largest
Sutton, Richard S., and Andrew Barto, Reinforcement Learning: consumer-packaged goods companies.
An Introduction, MIT Press (1998). Abhijit Gosavi has a BE and an M.Tech in mechanical
Tesauro, Gerald, “Temporal Difference Learning and Td- engineering, and a PhD in industrial engineering from the
gammon,” Communications of the Association for Computing University of South Florida. He is currently an assistant
Machinery, 38:3 (1995), pp. 58-68. professor in the department of engineering management
Van Roy, Benjamin, Dimitri P. Bertsekas, Yuchus Lee, and John and systems engineering in Missouri University of Science
N. Tsitsiklis, “A Neuro-Dynamic Programming Approach and Technology. His research interests are in simulation,
to Retailer Inventory Management,” Proceedings of the 36th reinforcement learning, and manufacturing.
IEEE Conference on Decision and Control, pp. 4052-4057 Li Lin is professor of industrial and systems engineering,
(1997). University at Buffalo. His research interests include computer
Veinott, Arthur F., “Optimal Policies for a Multi-Product, simulation, manufacturing and healthcare system analysis and
Dynamic Non-Stationary Inventory Model with Several design. He has published 60 refereed papers in major research
Demand Classes,” Operations Research, 13:5 (1965), pp. 761- journals. His current research focuses on the delivery of
778. healthcare using system engineering methods. As a volunteer
Waller, Matt, Eric Johnson, and Tom Davis, “Vendor-Managed he serves on the Board of Directors of Catholic Health System
Inventory in the Retailer Supply Chain,” Journal of Business in Buffalo, NY.
Logistics, 20 (1999), pp. 183-203. Contact: Dr. Abhijit Gosavi, Missouri University of Science
Watkins, Chris J., Learning from Delayed Rewards, unpublished and Technology, 210 Engineering Management, Rolla, MO
Ph.D. thesis, Kings College, Cambridge, England (1989). 65409; phone: 573-341-4624; gosavia@mst.edu

Engineering Management Journal Vol. 22 No. 4 December 2010 53