J. Math. Anal. Appl.

287 (2003) 118–140

www.elsevier.com/locate/jmaa

Optimal control problems on manifolds:

a dynamic programming approach
I. Chryssochoos and R.B. Vinter ∗
EEE Department, Imperial College of Science, Technology and Medicine, London SW7 2BT, UK
Received 19 September 2002
Submitted by H. Frankowska

Abstract
Dynamic programming identifies the value function of continuous time optimal control with a
solution to the Hamilton–Jacobi equation, appropriately defined. This relationship in turn leads to
sufficient conditions of global optimality, which have been widely used to confirm the optimality
of putative minimisers. In continuous time optimal control, the dynamic programming methodology
has been used for problems with state space a vector space. However there are many problems of
interest in which it is necessary to regard the state space as a manifold. This paper extends dynamic
programming to cover problems in which the state space is a general finite-dimensional C ∞ mani-
fold. It shows that, also in a manifold setting, we can characterise the value function of a free time
optimal control problem as a unique lower semicontinuous, lower bounded, generalised solution of
the Hamilton–Jacobi equation. The application of these results is illustrated by the investigation of
minimum time controllers for a rigid pendulum.
 2003 Elsevier Inc. All rights reserved.

Keywords: Optimal control; Nonlinear control; Hamilton–Jacobi equation; Dynamic programming;

Differentiable manifolds; Nonsmooth analysis

1. Introduction

In optimal control theory, dynamic programming links the function describing how the
minimum cost is affected by changes in initial data (“the value function”) and solutions
to the Hamilton–Jacobi equation. In contrast to techniques for solving optimal control

* Corresponding author.
E-mail address: r.vinter@ic.ac.uk (R.B. Vinter).

0022-247X/$ – see front matter  2003 Elsevier Inc. All rights reserved.
doi:10.1016/S0022-247X(03)00535-3
 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140 119

problems based on the maximum principle and related necessary conditions, dynamic pro-
gramming provides sufficient conditions of optimality. A frequent application of dynamic
programming is to confirm optimality of a putative minimiser. Here, necessary conditions
of optimality provide guesses regarding the nature of minimising controls for arbitrary ini-
tial data; this information is used to construct a suitable solution to the Hamilton–Jacobi
equation, to validate the “minimiser” under consideration, a procedure referred to as “con-
structing a field of extremals.” The technique has been used, for example, to confirm
optimality of extremals (i.e., arcs satisfying Euler–Lagrange type conditions) for many
classical variational problems, including the brachystochrone problem [12], for dynamic
optimisation problems arising in resource economics [4] and for variational problems in
nonlinear elasticity [5].
Research on the dynamic programming approach in optimal control has been very ex-
tensive. Much of this centres on the task of providing appropriate solution concepts, to
ensure that the value function can be characterised as the unique solution to the Hamilton–
Jacobi equation. This is far from straightforward—to take account of numerous optimal
control problems of interest where the value functions lack the requisit smoothness to be
regarded as classical solutions to the Hamilton–Jacobi equation (we examine one such
example in Section 8), we must allow “solutions” which are nondifferentiable or even dis-
continuous. The use of elementary notions of solutions to the Hamilton–Jacobi equation,
based on satisfaction of the equation at points of differentiability, fails to guarantee unique-
ness of solutions. Sophisticated analytical machinery is now available however, dealing
with these difficulties (viscosity solutions, generalised proximal normal solutions, etc). We
refer to [2,11] for background discussion.
However there remain important classes of optimal control problems not directly cov-
ered by current theory. A simple example (one which we shall examine in detail) is
minimum time control of a rigid pendulum, driven by a tangential force (the “control”),
subject to saturation, and confined to the vertical plane. The target set is the configuration
in which the pendulum is stationary and makes a zero angle with the downward vertical.
For this problem, the state space M is a two-dimensional differentiable manifold, the
tangent bundle of S 1 . It is not difficult to come up with plausible candidates for mini-
mum controls for this problem. However, standard techniques cannot be used directly to
verify optimality, because the state space is a finite-dimensional C ∞ manifold, not home-
omorphic to an open subset of a finite-dimensional space, and “continuous time” dynamic
programming has hitherto been developed only in a vector space setting.
In the free time problem described above, one of a number of “fixes” can be used to
provide a test of optimality based on dynamic programming for optimal control problems
on vector spaces. We can narrow the problem by introducing state constraints which con-
fine motion to one local coordinate chart. Alternatively we can regard M as the image of
all of R2 , under some C ∞ mapping which is onto, but not one-to-one; in this way, we
can reformulate the original problem as a vector space problem with target set comprising
multiple copies of the original target point. However this is a very simple problem and we
would expect a great deal of ingenuity would be required in more complicated situations,
to provide an alternative formulation to which vector space dynamic programming can be
applied.
120 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140

What is required is a framework for dynamic programming on manifolds, say general

finite-dimensional C ∞ manifolds. The aim of this paper is to bridge this gap. Specifically
we provide

(1) A description of control systems in terms of differential inclusions on manifolds;

(2) Notions of subdifferentials for lower semicontinuous functions on a manifold and re-
lated concepts of generalised solutions to Hamilton–Jacobi equations on manifolds;
(3) A characterisation of the value function as the unique lower semicontinuous, lower
bounded, generalised solution to the Hamilton–Jacobi equation.

When specialised to a vector space setting (M = Rn ), (3) provides the information that
the value function V satisfies the minimum time problem Hamilton–Jacobi equation and
accompanying boundary conditions on the target set C;

1 − H (x, −η) = 0 for all η ∈ ∂ P V (x), x ∈ Rn \ C,

1 − H (x, −η)
0 for all η ∈ ∂ P V (x), x ∈ C,
V (x) = 0 for all x ∈ C.
Here H is the Hamiltonian associated with the dynamics and ∂ P V is the proximal
subgradient.

The proximal subdifferential ∂ P V (x) is the empty set if x ∈

/ dom V (x) (:= {x : V (x )
< ∞) and, otherwise, comprises vectors η, corresponding to which there exist M > 0 and
 > 0 such that
η · (x − x)  V (x ) − V (x) + M|x − x|2 for all x ∈ x + B.

This reproduces the ‘state of the art’ characterisation of value functions for free time
problems given, for example, in [13] with one modification: we have an extra hypothesis,
not present in [13] excluding state trajectories with finite escape times. The target set is
an arbitrary closed subset of the state space and no controllability-type hypotheses are
imposed, whose purpose is to ensure continuity of the value function.
For a fixed time optimal control problem with state space a vector space, Frankowska
and Plaskacz [8] provide a Hamilton–Jacobi type characterisation of the value function, in
situations where the state is constrained to lie in a closed subset D (with, possibly, non-
empty interior). In the case when the manifold M can be embedded in R N , for some N ,
in such a manner that F has a locally Lipschitz extension to all of R N and a certain
‘outward-pointing’ condition is satisfied, we expect that use of the methods of [8] (based
on interpreting D as M) provide an alternative approach to dynamic programming on man-
ifolds, though one that does not emphasise the coordinate free character of the problem.
The idea of posing optimal control problems on manifolds and of developing intrin-
sic formulations of certain optimality conditions such as the maximum principle are by
no means new [7,10]. Properties of solutions of partial differential equations on mani-
folds, too, have been the subject of extensive investigations. The novel feature of this paper
is that it extends to a manifold setting optimality conditions of dynamic programming
 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140 121

type, and provides the appropriate machinery for interpreting ‘nonsmooth’ solutions of the
Hamilton–Jacobi equation.
The main results of this paper, and also a more detailed description of the solution to
rigid pendulum problem of the final section, were reported in [3].
Throughout the paper, B denotes the open unit ball in Euclidean space.

2. A free time optimal control problem on a manifold

Take a
C ∞ n-dimensional manifold M, a closed set C ⊂ M and a multifunction

F : M ❀ p∈M Tp M such that F (p) ⊂ Tp M for each p ∈ M. Here Tp M denotes the
tangent space at p.
Fix p ∈ M. We shall investigate the time optimal control problem

 Minimize T over arcs (q, T ) satisfying
d
P (p) : q∗ dt (t) ∈ F (q(t)) a.e. t ∈ [0, T ],

q(0) = p, q(T ) ∈ C.
The domain of this problem comprises arcs q : [0, T ] → M satisfying the dynamic con-
straint (elucidated below) and the initial point condition. Arcs are written (q, T ), to em-
phasise that the end-time T is a choice variable. The cost of (q, T ) is T if q(T ) ∈ C. We
define the cost of (q, T ) to be +∞ if q(T ) ∈
/ C.
The arcs (q, T ) considered are locally Lipschitz continuous, in the sense that, for any
t ∈ [0, T ] and any local coordinate chart (φ, U ) such that q(t) ∈ U , φ(q(·)) is Lipschitz
continuous on some neighbourhood of t ∈ [0, T ]. The assertion that an arc (q, T ) satisfies
the dynamic constraint
 
d  
q∗ (t) ∈ F q(t) a.e. t ∈ [0, T ] (1)
dt
is interpreted as follows: for any t ∈ [0, T ] and local coordinate chart (φ, U ) satisfying
q(t) ∈ U , we have
 
ẋ(t) ∈ F̂ x(t) a.e. on some neighbourhood of t.
Here x(·) = φ(q(·)) and F̂ : φ(U ) ❀ Rn is the relevant local representation of F , namely
d  
F̂ (x) := v ∈ Rn : vi = φ∗ (p)ζ and ζ ∈ F φ −1 (x)
dxi
i
for x ∈ φ(U ). ((1) is merely notation indicating that the dynamic constraint is satisfied
in the above sense; (1), which makes reference to the tangent space mapping q∗ , induced
by the arc q, is meaningful, strictly speaking, only when q is continuously differentiable.)
Arcs are referred to also as F -trajectories.
Our description of the system dynamics in terms of a differential inclusion (1) is now
widely used in the optimal control literature. It subsumes descriptions involving a control
dependent vector field
 
d    
q∗ (t) = f q(t), u(t) a.e., u(t) ∈ Ω q(t) a.e.,
dt
122 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140

which conform to the above description (1) with

F (q) := f (q, u): u ∈ Ω(q) .
The merits of the differential inclusion formulation are discussed at length in [11].
In this paper, interest centres on the value function V : M → R ∪ {+∞} for the family
of optimal control problems {P (p): p ∈ M}. V (p) is the infimum cost of P (p),
V (p) = inf P (p).
We follow the usual convention, setting V (p) = +∞, if there exist no arcs (q, T ) satisfying
the constraints of P (p).

3. Nonsmooth analysis on smooth manifolds

If we are to develop dynamic programming techniques for free time optimal control
problems on manifolds of any generality, we must allow for possibly nondifferentiable
value functions. It is necessary therefore to extend to a manifold setting vector space ma-
chinery for analysing the local properties of nonsmooth functions, in terms of gradient-like
constructs. Many of the analytic tools from nonsmooth analysis (including proximal subd-
ifferentials, strict subdifferentials, limiting subdifferentials, lower Dini directional deriva-
tives and viscosity-type subdifferentials) have analogues in a manifold setting. Of primary
interest here will be a generalisation of the proximal subdifferential since this alone is
required in our approach to dynamic programming on manifolds. Recall that M is an n-
dimensional C ∞ manifold.

Definition 3.1. Take a function f : M → R ∪ {+∞} and a point p ∈ M. We say that a

cotangent vector σ ∈ Tp∗ M is a proximal subgradient of f at p if p ∈ domf and there
exists a C 2 function m, defined on some neighbourhood W of p, such that σ = dm(p) and
f (p ) − f (p)
m(p ) − m(p) for all p ∈ W. (2)
The set of all proximal subgradients of f at p is called the proximal subdifferential of f
at p and is denoted ∂ P f (p).

The following proposition links the above definition of proximal subdifferential to the
familiar proximal subdifferential defined on finite-dimensional vector spaces.

Proposition 3.1. Take a function f : M → R ∪ {+∞}, a point p ∈ domf and a cotangent

vector σ ∈ Tp∗ M.Then the following statements are equivalent:

(i) σ is a proximal subgradient of f at p;

(ii) For any local coordinate chart (φ, U ) such that p ∈ U , and any C 2 function m such
that σ = dm (p), define fˆ : φ(U ) → R ∪ {+∞} and η ∈ (Rn )∗ according to
fˆ(x) = (f ◦ φ −1 )(x), η = ∇(m ◦ φ −1 )(x)|x=φ(p).
 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140 123

(Here, ∇ denotes the derivative of a function on a vector space, differentiable at its

base point.) Then fˆ(φ(p)) < +∞. Furthermore, there exist constants M > 0 and
 > 0 such that φ −1 (φ(p) + B) ⊂ U and
   
fˆ(x ) − fˆ φ(p)
η · x − φ(p) − M|x − φ(p)|2 for all x ∈ φ(p) + B.
(3)

Proof. (i) ⇒ (ii) Suppose σ ∈ ∂ P f (p). Then (2) is satisfied for some neighbourhood W
of p and some C 2 function m : W → R. Write x̄ = φ(p). Take any local coordinate chart
(φ, U ) and  > 0 such that φ −1 (φ(p) + B) ⊂ U ∩ W and any C 2 function m such that
σ = dm (p). By (2) we have, for all x ∈ x̄ + B,
f ◦ φ −1 (x ) − f ◦ φ −1 (x̄)
m ◦ φ −1 (x ) − m ◦ φ −1 (x̄).
Since σ = dm = dm and m ◦ φ −1 and m ◦ φ −1 are both C 2 functions, we have
∇(m ◦ φ −1 )(x̄) = ∇(m ◦ φ −1 )(x̄)
and, for some M > 0 sufficiently large,
m ◦ φ −1 (x ) − m ◦ φ −1 (x̄)
∇(m ◦ φ −1 )(x̄) · (x − x̄) − M|x − x̄|2
for all x ∈ x̄ + B.
We have confirmed (ii).
(ii) ⇒ (i) Assume that (ii) is true. Then f (p) < +∞ and σ = dm (p). (2) is satisfied
with W = φ −1 (φ(p) + B) and
   2
m(p ) = f (p) + ηT φ(p ) − φ(p) − M φ(p ) − φ(p) .
We have confirmed (i). ✷

A related construct, that of the proximal subdifferential, is obtained by relaxing the

conditions in Definition 3.1 to require merely that the function m is continuous on a neigh-
bourhood of p and differentiable at p. This Frechet type subdifferential is an analogue,
in a smooth manifold setting, of the vector space generalised subdifferentials used in the
viscosity and viability theory literature, and could be used as an alternative device for char-
acterising the value function as a generalised solution to the Hamilton–Jacobi equation to
that used in this paper.

4. Main results: characterisation of the value function

We state the main results of the paper. These relate to the free time optimal control
problem P (p) of Section 2. It is asserted that the value function
V (p) := inf P (p), p ∈ M,
is the unique solution (appropriately defined) to the Hamilton–Jacobi equation
1 − H (p, −dV (p)) = 0, p ∈ M \ C,
(4)
V (p) = 0, p ∈ C,
124 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140

where H : {(p, ξ ): p ∈ M, ξ ∈ T ∗ M(p)} → R is the Hamiltonian

H (p, ξ ) := sup v, ξ .
v∈F (p)
In the case that V is continuously differentiable dV (p) is interpreted as the differential of
V at p, to make sense of (4). However, a more general concept of “solution” is required to
characterise the value function.

Definition 4.1. A lower semicontinuous function V : M → R ∪ {+∞} is said to be a prox-

imal solution to (4) if

 1 − H (p, −ξ ) = 0 for all ξ ∈ ∂ P V (p) and p ∈ M \ C,
1 − H (p, −ξ )
0 for all ξ ∈ ∂ P V (p) and p ∈ M \ C,

V (p) = 0 for all p ∈ C.

Notice that the condition “1 − H (p, −ξ ) = 0 for all ξ ∈ ∂ P V (p)” is automatically

satisfied at all points p ∈ M \ C, where ∂ P V (p) is empty and, in particular, at points p ∈
/
dom V . The second condition, too, need only be tested at points where ∂ P V is nonempty.
We shall make reference to the following hypotheses:

(H1) F (p) is a nonempty, compact, convex subset of Tp M for each p ∈ M;

(H2) (Local boundedness and Lipschitz continuity). For any p, there exists a local coordi-
nate chart (φ, U ), such that φ(U ) is a bounded set, and constants kφ > 0 and cφ > 0
such that
F̂ (x) ⊂ F̂ (x ) + kφ |x − x |B and F̂ (x) ⊂ cφ B
for all x, x ∈ φ(U ), where
d  
F̂ (x) = v ∈ Rn : vi ∈ φ∗ F φ −1 (x) ;
dxi
i
(H3) (Linear growth). There exists a continuously differentiable function s : M → R and
K > 0 such that
s −1 (I ) is compact for all compact intervals I ⊂ R
and
    
max v · ds(p)  K 1 + s(p) for all p ∈ M.
v∈F (p)

We are ready to state our characterisation of the value function for the free time optimal
control problem on a manifold, in terms of generalised solutions of the Hamilton–Jacobi
equation.

Theorem 4.1. Assume that the data for problem P (p) satisfy hypotheses (H1)–(H3). Then
the value function V is the unique lower semicontinuous function which is a proximal
solution to the Hamilton–Jacobi equation (4).

The theorem is proved in Section 7.

 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140 125

5. Existence of minimisers

As a prelude to establishing existence of minimisers, we prove two lemmas. They tell

us that values of F -trajectories, with initial conditions confined to a compact subset and
defined on uniformly bounded time intervals, are covered by a finite number of local coor-
dinate charts.

Lemma 5.1. Fix a compact set G ⊂ M and T > 0. Assume (H1)–(H3). Then there exists
a compact set D such that for all T ∈ [0, T ] and F -trajectories q : [0, T ] → M such that
q(0) ∈ G, we have
q(t) ∈ D for all t ∈ [0, T ].

Proof. Take any arc q : [0, T ] → M as described in the lemma. We deduce from (H3) that
 
d      
 s q(t)   K 1 + s q(t)  a.e. t ∈ [0, T ]. (5)
 dt 
Now t → s(q(t)) and t → |s(q(t))| are locally Lipschitz continuous. So there exists a
subset Q ⊂ [0, T ], of full measure, on which both functions are differentiable. We claim
d       
s q(t )   K 1 + s q(t )  (6)
dt
for all t ∈ Q. Indeed, (6) follows from (5) if s(q(t)) = 0, since then
d    d  
s q(t )  = ± s q(t ) .
dt dt
On the other hand, if s(q(t )) = 0, t → |s(q(t))| is minimised at t = t so
d        
s q(t )  = 0  K 1 + s q(t )  .
dt
We see that (6) is satisfied in this case also.
By Gronwall’s lemma,
      
s q(t)   eKt s q(0) + 1  L for all t ∈ [0, T ],

for some constant L (independent of q). We have used the fact that the continuous function
p → s(p) is bounded on the compact set G. The assertions of the lemma now follow from
the assumed properties of s(·). ✷

Lemma 5.2. Assume (H1)–(H3). Fix a compact set G ⊂ M and T̄ > 0. Then there exists
a finite collection of local coordinate charts {(φj , Uj )} and constants k̄ > 0, c̄ > 0 and
¯ > 0 with the following properties. Given any T ∈ [0, T̄ ] and F -trajectory q : [0, T ] → M
satisfying q(0) ∈ G, there exists a local coordinate chart (φ, U ) ∈ {(φj , Uj )} such that
   
φ −1 φ q(T ) + B ¯ ⊂U
and
F̂ (x ) ⊂ F̂ (x) + k̄|x − x|B and F̂ (x) ⊂ c̄B for all x ∈ φ(U ),
126 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140

for all x, x ∈ φ(U ), where

d  
F̂ (x) = v ∈ Rn : vi ∈ φ∗ F φ −1 (x) .
dxi
i

Proof. We choose an atlas {(φp , Up ): p ∈ M}, such that, for each p ∈ M, we have p ∈ Up ,
constants kp > 0 and cp > 0 governing the Lipschitz and boundedness of F̂ on Up (see
(H2) and (H3)) and a constant ¯ > 0 such that
 
φp−1 φp (p) + 2p B ⊂ Up
and
F̂p (x) ⊂ cp B for all x ∈ φp (p) + 2p B,
where
d  
F̂p (x) = v: vi ∈ (φp )∗ F φp−1 (x) .
dxi
i

Take any T ∈ [0, T̄ ] and any F -trajectory q : [0, T ] → M satisfying q(0) ∈ G. By

Lemma 5.1 there exists a compact set D ⊂ M (which does not depend on (q, T )) such
that q(t) ∈ D for all t ∈ [0, T ]. D is covered by the family of open sets
 
φp−1 φ(p) + p B : p ∈ M .
Since D is compact, it is covered also by a finite subfamily
 
φp−1 φ(p) + p B : p ∈ Σ ,
in which Σ is a finite subset of M.
Let k̄ > 0, c̄ > 0 and ¯ be the constants
k̄ = min{kp : p ∈ Σ}, c̄ = min{cp : p ∈ Σ} and ¯ = min{p : p ∈ Σ},
and define
  
(φj , Uj ) := φp , φp−1 φp (p) + ¯ B : p ∈ Σ .
It is straightforward to show that these constants and this choice of local coordinate charts
have the desired properties. ✷

Proposition 5.3. Take any p ∈ M. Then P (p) has a minimiser.

Proof. We can assume that p ∈ / C since, if p ∈ C, P (p) has the trivial solution (q ≡ p,
T ∗ = 0). We can also assume that inf P (p) < +∞ for, otherwise, by convention, the min-
imum cost is +∞, which is achieved by any arc (q, T ) with left endpoint p.
Let T ∗ = inf P (p) (< +∞). Since p ∈ / C, and in view of (H2), T ∗ > 0. By definition of
the infimum cost, there exists a sequence of F -trajectories qi : [0, Ti ] → M with qi (0) = p,
qi (Ti ) ∈ C, such that Ti ↓ T ∗ . Extend (without relabelling) each qi to [0, +∞), by constant
extrapolation to the right.
 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140 127

Let F = {(φi , Ui )} be the finite family of local coordinate charts of Lemma 5.2 corre-
sponding to the initial set G = {p} and T̄ , some upper bound on the Ti ’s. Let k̄ > 0, c̄ > 0
and ¯ > 0 be the constants of Lemma 5.2.
Choose δ > 0 and a positive integer N
2 such that
δ < ¯ /3c̄, δ  T ∗, (N − 1)δ < T ∗ < Nδ.
By eliminating initial terms in the sequence {(qi , Ti )}, we can arrange that
 
Ti ∈ (N − 1)δ, Nδ for all i.
We now construct a sequence of arcs
 
q[j ] : (j − 1)δ, j δ → M, j = 1, . . . , N − 1,
and
 
q[N] : (N − 1)δ, (Nδ) ∧ T ∗ → M.
According to the properties of the family of local coordinate charts F , we can choose a
local coordinate chart (φ[1] , U[1] ) ∈ F such that
−1  
φ[1] φ[1] (p) + ¯ B ⊂ U[1] . (7)
Write
d  −1 
F̂[1] (x) = v ∈ Rn : vj ∈ φ[1]∗ F φ[1] (x) .
dxj
j

Then F̂ is a compact, convex multifunction which is bounded and Lipschitz continuous on

φ(p) + ¯ B.
Note that, for each i, qi (t) ∈ U[1] for 0  t  δ, by (7) and since c̄δ  . Write
 
xi[1] (t) = φ[1] qi (t) for t ∈ [0, δ].
By the compactness of trajectories theorem [11, Theorem 2.5.2, p. 81], there exists an
F̂[1] -trajectory x[1] : [0, δ] → Rn satisfying x[1] (0) = φ(p) such that, following extraction
of subsequences (we do not re-label), we have
xi (t) → x[1] (t) uniformly on [0, δ]
and
 
ẋ[1] (t) ∈ F̂[1] x[1] (t) a.e. t ∈ [0, δ].
−1
Write q[1] (t) = φ[1] (x[1] (t)), 0  t  δ. Then q[1] : [0, δ] is an F -trajectory on [0, δ] such
that q[1] (0) = p,
q[1] (t) ∈ U[1] for t ∈ [0, δ]
and
−1 −1
φ[1] qi (t) → φ[1] q[1] (t) uniformly as i → ∞.
Next, choose a local coordinate chart (φ2 , U[2] ) ∈ F such that
−1    
φ[2] φ[2] q[1] (δ) + B
¯ ⊂ U[2] .
128 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140

By further extraction of subsequences, we can ensure that

    
φ[2] qi (δ) − φ[2] q i (δ)   ¯
[1]
3
for all i. Then, since δ  /3
¯ c̄,
−1    
φ[2] φ[2] qi (t) + (¯ /3)B ⊂ U[2] for t ∈ [δ, 2δ], i = 1, 2, . . . .
Write
 
xi[2] (t) = φ[2] qi (t) , δ  t  2δ.
Following extraction of a further subsequence, we have that
xi[2] → x[2] (t) uniformly on [δ, 2δ],
for some arc x[2] : [δ, 2δ] → φ[2] (U[2] ) such that
 
ẋ[2] (t) ∈ F̂[2] x[2] (t) a.e. t ∈ [δ, 2δ ∧ T ∗ ].
Let
−1  
q[2] (t) = φ[2] x[2] (t) for t ∈ [δ, 2δ ∧ T ∗ ].
Define q[2] : [0, 2δ ∧ T ∗ ] → M to be the F -trajectory obtained by concatenating q[1] and
q [2] .
Suppose that N = 2. In this case δ < T ∗ < 2δ. We have
 −1    
xi[2] (T ∗ ) ∈ φ[2] C ∩ φ[2] ¯
φ[2] q1 (δ) + (2¯ /3)B (8)

for all i. Since T i ↓ T ∗ , since xi[2] ’s converge uniformly to x[2] , xi[2] ’s are uniformly Lip-
schitz continuous and the set on the right side of (8) is closed, we conclude that
x[2] ∈ φ[2] (C ∩ U[2] ).
It follows that the arc (q[2] , T ∗ ) achieves the minimum cost. The proposition is proved in
the case N = 2.
If, on the other hand, N > 2, we construct also arcs q[j ] : [(j − 1)δ, j δ] → M, and local
coordinate charts (φ[j ] , Uj ) ∈ F , for j = 3, 4, . . . , N , recursively, in a manner which pat-
terns the construction of q[2] from q[1] , etc., extracting suitable subsequences at each step.
A repeat of the preceding analysis establishes that the arc (q, T ∗ ), obtained by concatenat-
ing q[1] , . . . , q[N] , is a minimiser for P (p). ✷

Proposition 5.4. The value function V is lower semicontinuous.

Proof. Take any p ∈ M. Take any pi → p. We must show that

lim inf V (pi )
V (p). (9)
i→∞
The condition is automatically satisfied if lim infi→∞ V (pi ) = +∞. In view of Propo-
sition 5.3, we can therefore assume, without loss of generality, that there exists a sequence
of F -trajectories qi : [0, Ti ] → M such that qi (0) = pi , qi (Ti ) ∈ C and V (pi ) = Ti for
 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140 129

each i, and Ti → T ∗ for some T ∗ . (9) will follow if we can construct an F -trajectory
q : [0, T ∗ ] → M such that q(0) = p and q(T ∗ ) ∈ C. The construction of q is along almost
identical lines to that of the minimising F -trajectories in the proof of Theorem 5.3, and
is not repeated here. (The only difference, which does not affect the analysis, is that the
condition “qi (0) = p” is replaced by “qi (0) → p.”) ✷

6. Monotonicity properties

In the system theoretic approach to dynamic programming, monotonicity properties of

solutions to the Hamilton–Jacobi equation along state trajectories play a key role. These
monotonicity properties are closely linked to invariance properties of solutions to differ-
ential inclusions systematically studied in the viability literature [1] and elsewhere. In this
section, we establish such properties, in a manifold setting.

Proposition 6.1 (Weak monotonicity). Take a lower semicontinuous function V : M →

R ∪ {+∞} and a point p ∈ M. Suppose that, for some neighbourhood U of p ∈ M, we
have
1 − H (p , −η)  0 for all η ∈ ∂ P V (p ), p ∈ U. (10)
Then there exist  > 0 and an F -trajectory q : [0, ] → M such that q(0) = p and
 
V q(t) + t  V (p) for all t ∈ [0, ]. (11)

Proof. We can assume that V (p) < +∞ since, otherwise, the assertions of the proposition
are trivial. We can also assume that U is the open set associated with some local coordinate
chart (φ, U ). Write x0 = φ(p) and choose  > 0 such that x0 +  B ⊂ φ(U ). Write
 
V̂ (x) = V φ −1 (x) ,
d  
F̂ (x) = v: vi ∈ φ∗ F φ −1 (x)
dxi
i

for all x ∈ U . By hypothesis, we can find c > 0 such that

F̂ (x) ⊂ cB for all x ∈ x0 +  B.
Also, expressing (10) in local coordinates, we have
1 + min η · v  0 for all η ∈ ∂ P V̂ (x), x ∈ x0 +  B. (12)
v∈F̂ (x)

Consider now the differential inclusion and constraint


 (ẋ(t), τ̇ (t), ż(t)) ∈ F̃ (x(t), τ (t), z(t)), t
0,
(x(0), τ (0), z(0)) = (x0 , 0, V̄ (x0 )),

(x(t), τ (t), z(t)) ∈ D, t
0,
in which
130 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140

F̃ (x, τ, z) := F̂ (x) × {1} × {0} if x ∈ x0 +  B,

cB × [0, 1] × {0} if x ∈/ x0 +  B,
D := epi (x, τ ) → Ṽ (x) + τ ,
where

Ṽ (x) := V̂ (x) if x ∈ x0 +  B,
+∞ if x ∈ / x0 +  B.
Well-known arguments (see, e.g., [6,11]) permit us to deduce from (12) that
min η · v + η1 · σ + η2 · α  0 for all (η, η1 , η2 ) ∈ ND
P
(x, τ, z) (13)
(v,σ,α)∈F̃(x,τ,z)

for all (x, τ, z) ∈ Rn × R × R. Here, ND P denotes the (vector space) proximal normal cone

to the set D (at the relevant basepoint). (Notice that, for |x − x0 |
, F̃ (x, τ, z) contains
the point (0, 0, 0), so (13) is automatically satisfied.)
We now apply the proximal normal version of the weak invariance theorem (see,
e.g., [11]) in the standard vector space setting, noting that the relevant hypotheses, namely
F̃ is an upper semicontinuous, bounded, convex, nonempty multifunction and satisfies
the “inward pointing” condition (13) (see also [6]). We conclude that there exists an F -
trajectory (x(t), τ (t), z(t)), 0  t < ∞, with initial value (x0 , 0, 0), and taking values in D.
Choose  <  /c. Then |x(t) − x0 | <  for 0  t   . It follows that Ṽ (x(t)) = V̂ (x(t))
and τ̇ (t) = 1 for 0  t  . For (x(t), τ (t), z(t)) ∈ D, z(0) = V (x0 ) we deduce
 
V̂ (x0 )
V̂ x(t) + t, 0  t  .

Define q(t) = φ −1 (x(t)). We have q(0) = p. Also, q is an F -trajectory on [0, ].

Since x(t) ∈ x0 + B for all t ∈ [0, ] and V̂ (x) = V (φ −1 (x)) on x0 + B, we conclude
V (q(t)) + t  V (q(0)) for all t ∈ [0, ]. ✷

Proposition 6.2 (Strong monotonicity). Suppose that, for some lower semicontinuous func-
tion W : M → R ∪ {+∞},
1 − H (p, −η)
0 for all η ∈ ∂ P W (p), p ∈ M.
Then, for any F -trajectory q : [0, T ] → M and t ∈ (0, T ],
   
W q(0)  t + W q(t) . (14)

Proof. Take any t ∈ (0, T ]. We aim to show (14). It may be assumed that W (q(t)) < +∞
since, otherwise, the inequality is automatically satisfied. There exists a chart (φ, U ) and
 > 0 such that A := φ −1 [φ(q(t)) + B] ⊂ U and
   
φ −1 φ q(s) + (/2)B ⊂ A for all s ∈ [t − , t].

Write x(s) = φ(q(s)) for s ∈ [t − , t]. Let F̂ be the local representation of F with respect
to (φ, U ) and define Ŵ : (−∞, +∞) × A → R ∪ {+∞} to be
 
Ŵ (s, x) := s + W φ −1 (x) .
 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140 131

In local coordinates the Hamilton–Jacobi inequality takes the form

min η0 + η1 · e
0 for all (s, x) ∈ (−∞, +∞) × A, (η0 , η1 ) ∈ ∂ P Ŵ (s, x).
e∈F̂ (x)

(∂ P Ŵ here has its customary, vector space, meaning.) Bearing in mind that F̂ is Lipschitz
continuous, bounded and has values closed, convex sets, and recalling that Ŵ (t, x(t)) =
t + W (q(t)) < +∞, we can show, using a local analysis along the lines of the proof of [11,
Theorem 12.2.4], that
   
Ŵ s, x(s)  Ŵ t, x(t) for all s ∈ [t − , t],
i.e.,
   
W x(s) + s  W t, x(t) + t for all s ∈ [t − , t].
Define
   
smin = min σ ∈ [0, t]: W x(s) + s  W t, x(t) + t for all s ∈ (σ, t] .
We must have smin = 0, for, otherwise, the preceding analysis tells us that the interval
(σ, t] could be extended to the left, while preserving the inequality, in contradiction of the
definition of smin . We have shown that W (x(s)) + s  W (t, x(t)) + t for all s ∈ (0, T ]. It
is true also for s = 0, by lower semicontinuity. ✷

7. Proof of Theorem 4.1

We observe at the outset that the defining conditions on V , to be a proximal solution to

the Hamilton–Jacobi equation (see Definition 4.1) can be equivalently expressed as

1 − H (p, −η) = 0 for all η ∈ ∂ P V (p), p ∈ M \ C, (15)

1 − H (p, −η)
0 for all η ∈ ∂ V (p), p ∈ M,
P
(16)
V (p) = 0 for all p ∈ C. (17)
Notice, here, that the second condition (16) is required now to hold for all p ∈ M (not just
p ∈ M \ C). The equivalence follows from the fact that the final condition (17) implies
∂ P V (p) = {0} for all p ∈ intC, and hence that 1 − H (p, −η)
0 for all p ∈ C and η ∈
∂ P V (p).
Let V be the value function. Then V is nonnegative valued and therefore bounded below.
We have shown that V is lower semicontinuous (Proposition 5.4). V obviously satisfies the
boundary condition (17).
Take any p ∈ M \ C. We show that V satisfies
1 − H (p, −η)  0 for all η ∈ ∂ P V (p). (18)
This condition is trivially satisfied if V (p) = +∞. So assume that V (p) < +∞. By Propo-
sition 5.3 P (p) has a minimiser q : [0, T ] → M. We have
   
V q(t) + t = V q(0) for all t ∈ [0, T ], (19)
132 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140

by well-known monotonicity properties of the value function (“the principle of optimali-

ty”). Let (φ, U ) be a local coordinate chart such that p ∈ U . Let V̂ (x) = V (φ −1 (x)) and
 −1
F̂ = {e: i ei (d/dxi ) ∈ φ∗ F φ } be local representations of V and F (near p). Write
x(t) = φ(q(t)).
It can be deduced from the convexity and upper semicontinuity of F̂ that there exists
e ∈ F̂ (x(0)) such that, for some sequence ti ↓ 0,
 
e = lim ti−1 x(ti ) − x(0) . (20)
i
For each i, by (19),
     
ti−1 V̂ x(0) + x(ti ) − x(0) − V̂ x(0) + 1 = 0. (21)

Take any η ∈ ∂ P V (p). Let ζ be the vector such that v ζi dxi = φ ∗−1 η. Then
     2
V̂ (x) − V̂ x(0)
ζ · x − x(0) − M x − x(0) for all x ∈ x(0) + B,
for some M > 0,  > 0. We deduce from (20) and (21) that
 
1 + ζ · e = 1 + lim ti−1 ζ · x(ti ) − x(0)
i
     
 1 + lim ti−1 V̂ x(0) + x(ti ) − x(0) − V̂ x(0) + 0 = 0.
i
We have shown that
 
1+ζ ·e=0 for some e ∈ F̂ x(0) .
But then
1 − H (p, −η) = 1 + min η, v = 1 + min ζ, e   0.
v∈F (p) e ∈F̂ (x(0))

This confirms (7.4).

Next take p ∈ M. We show that
1 − H (p, −η)
0 for all η ∈ ∂ P V (p). (22)
This condition is trivially satisfied if V (p) = +∞, so assume V (p) < +∞. Let (φ, U ) be
a local coordinate chart such that p ∈ U . Write x0 = φ(p). Denote by V̂ and F̂ the local
representations of V and F .
 Choose any v ∈ F (p) and (φ, U ) such that p ∈ U . Let the vector e be such that
i ei (d/dxi ) = φ∗ v. Filippov’s existence theorem, applied in reverse time, implies ex-
istence of α > 0 and an F -trajectory x : [−α, 0] → Rn such that x(0) = p and
 
lim t −1 x(0) − x(−t) = e.
t ↓0

Write q(t) = φ −1 (x(t)). We have

   
V q(−t) − t  V q(0) for all t ∈ [0, α],
by monotonicity properties of the value function, whence
    
−1 + lim inf t −1 V q(−t) − V q(0)  0.
t ↓0
 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140 133


Take any η ∈ ∂ P V (p). Write ζ for the vector satisfying i ξi dxi = φ ∗−1 η. Then
     2
V̂ (x) − V̂ x(0)
ζ · x − x(0) − M x − x(0) for all x ∈ x(0) + B,
for some M > 0 and  > 0. We have
 
1 + η, v = 1 + ζ · e = 1 − lim t −1 ζ · x(−t) − x(0)
t ↓0
−1
    

1 + lim inf t V q(−t) − V q(0)
0.
t ↓0

But v was an arbitrary point in F (p). We conclude

1 − H (p, −η) = 1 + min η, v
0.
v∈F (p)

(22) is confirmed. Since (18) and (22) imply (15) and (16), we have confirmed that V
satisfies the stated conditions.
It remains to show that V is the unique such function.
Let W be any lower semicontinuous function, bounded from below, vanishing on C,
which satisfies the “equivalent” conditions
1 − H (p, −η)  0 for all η ∈ ∂ P W (p), p ∈ M \ C,
1 − H (p, −η)
0 for all η ∈ ∂ P W (p), p ∈ M.
Take any p ∈ M. We show

(i) For any F -trajectory q : [0, T ] → M such that q (0) = p and q (T ) ∈ C we have
W (p)  T ;
(ii) There exists an F -trajectory q : [0, T ] → M such that q(0) = p, q(T ) ∈ C and
W (p)
T .

These properties will establish that

W (p) = inf P (p) = V (p),
completing the proof.
Property (i) follows from Proposition 6.2, which tells us that, for any F -trajectory
q : [0, T ] → M such that q (0) = p, q(T ) ∈ C,
 
W (p)  T + W q(T ) = T + 0.
(We have used the fact that W vanishes on C.)
It remains to verify property (ii). Assume then that W (p) < +∞. We can assume also
that p ∈/ C since, otherwise, (ii) is trivially satisfied.
We shall use Zorn’s lemma [9]: A set V with a partial ordering “$” has a maximal
element if every chain in V (i.e., totally ordered subset) has a maximal element.
We take V to be the set of F -trajectories (q, T ) satisfying
q(0) = p,
q(t) ∈ M \ C for all t ∈ [0, T ),
 
W q(t) + t  W (p) for all t ∈ [0, T ]. (23)
134 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140

The partial ordering is

(q, T ) $ (q , T ) ⇔ T T and q|[0,T ] = q |[0,T ] .
Let L be a lower bound on W . Then
(q, T ) ∈ V ⇒ T  T̄ (< +∞), (24)
where T̄ = W (p) − L. This follows from (23).
It follows from Lemma 5.2 and (24) that there exists a compact set D ⊂ M such that,
for any (q, T ) ∈ V and t ∈ [0, T ],
q(t) ∈ D.
Now take an arbitrary chain C in V. We show that C has a maximal element. Define
T ∗ := sup T : (q, T ) ∈ C .
Notice that, in view of Proposition 5.3, and (24), T ∗ is a finite, positive number. It is
a straightforward matter to establish the existence of a function q̄ : [0, T ∗ ] → M with the
following properties.
For any T ∈ [0, T ∗ ), q̄ restricted to [0, T ] is an F -trajectory with q̄(0) = p.
q̄(t) ∈ M \ C for all t ∈ [0, T ∗ ),
 
V q̄(t) + t  V (p), t ∈ [0, T ∗ ).
Given any (q, T ) ∈ C then T  T ∗ and q(t) = q̄(t) for t ∈ [0, T ].
If there exists (q, T ) ∈ C with T = T ∗ then q̄ is a maximal element in C. So we can
suppose that T < T ∗ for every (q, T ) ∈ C.
Choose a sequence ti ↑ T ∗ such that ti ∈ {T : (q, T ) ∈ C} for each i. Since the sequence
{q̄(ti )} lies in the compact set D, we can arrange by subsequence extraction that there
exists p∗ ∈ M and a local coordinate chart (φ, U ) such that p ∈ U and φ(q̄(ti )) → φ(p∗ )
as i → ∞. Choose  > 0, c > 0 and an index value k such that φ(p∗ ) + B ⊂ U ,
F̂ (x) ⊂ cB for all x ∈ φ(p∗ ) + B, (25)
 −1   
φ q̄(tk ) − φ −1 (p∗ ) + c|T ∗ − tk | < , (26)
where F̂ is the local representation of F , with respect to (φ, U ).
Define x(t) = φ(q̄(t)) for t ∈ [tk , T ∗ ). Then, in view of (25) and (26) we can extend x
as a Lipschitz continuous function to all of [tk , T ∗ ], by setting
t
∗
x(T ) = x(tk ) + lim∗ ẋ(s) ds.
t ↑T
tk

Redefine, if necessary, the value of q̄ at T ∗ ,

 
q̄(T ∗ ) = φ −1 x(T ∗ ) .
Then q̄ is a locally Lipschitz arc on [0, T ∗ ] satisfying
 
d
φ∗ q̄ ∈ F (q̄) a.e.
dt
 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140 135

In other words, (q̄, T ∗ ) is an F -trajectory. In view of the other properties listed above,
(q̄, T ∗ ) is a maximal element for the chain C.
Invoking Zorn’s lemma, we deduce existence of a maximal element (q, T ) in V. By
definition of elements in V, (q, T ) is an F -trajectory such that

q(0) = p, q(t) ∈ M \ C for all t ∈ [0, T ), (27)

 
W q(t) + t  W (p) for all t ∈ [0, T ). (28)
We claim
q(T ) ∈ C.
Indeed, if this were not the case, there exists, by Lemma 6.1, an F -trajectory defining an
element in V, which is a strict extension of (q, T ). In such circumstances, (q, T ) cannot be
a maximal element in V. So (q, T ) is an F -trajectory with initial value p and terminating
in C. Since V vanishes on C, we deduce from (28) and the lower semicontinuity of V that
T  W (p).
We have exhibited an arc with the required properties; proof of the theorem is complete.
✷

8. An example

We study an example, one of the simplest where the state space is not homeomorphic
to an open subset of Rn , to illustrate the role of Theorem 4.1, regarding the confirmation
of minimisers in a manifold setting.
The system is the rigid pendulum in the vertical plane, to which an amplitude limited
torque (the control) is applied. The state space is a two-dimensional manifold, namely
the tangent bundle of S 1 . Fix a number α ∈ (0, π/2). The configuration of the system is
described by local coordinates, (x1 , x2 ), in each of two local coordinate charts.
The Upper Chart. In this chart, x2 is the angular velocity and x1 describes the angle
from the downward vertical, in the range +π/2 − α  x1  +3π/2 + α.
The Lower Chart. In this chart, x2 is the angular velocity and x1 describes the angle
from the downward vertical, in the range −π/2 − α  x1  +π/2 + α.
The charts are illustrated in Figs. 1 and 2. Assume zero gravity. Assume also that the
pendulum has unit moment of inertia and the controlling torque u is required to satisfy the
constraints −1  u  +1. Then, with respect to local coordinates in either chart domain,
the dynamic constants are
d
(x1 , x2 ) ∈ (x2 , u): u ∈ [−1, +1] .
dt
A local analysis of extremals would indicate that the value function (in either local coordi-
nate chart) is described by the function V : D → R, where
D = (x1 , x2 ) ∈ R 2 : −π/2 − α  x1  3π/2 + α
136 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140

Fig. 1. The upper chart domain.

Fig. 2. The lower chart domain.

and
V (x1 , x2 ) := min Vk (x1 , x2 ). (29)
k∈Z

Here, V0 : R 2 → R is the function

  1/2
2 x1 + 12 x22 + x2 if (x1 , x2 ) ∈ A1 ∪ A2 ,
V0 (x1 , x2 ) =  
1 2 1/2
2 −x1 + 2 x2 − x2 if (x1 , x2 ) ∈ A3 ∪ A4 ,
in which

A1 := (x1 , x2 ): x2 > − 2|x1| and x1 > 0 ,

A2 := (x1 , x2 ): x2
+ 2|x1| and x1  0 ,

A3 := (x1 , x2 ): x2 < + 2|x1| and x1 < 0 ,

A4 := (x1 , x2 ): x2  − 2|x1| and x1
0 ,
and the functions Vk : R2 → R, k ∈ Z, are defined according to
Vk (x1 , x2 ) = V0 (x1 + 2πk, x2 ).

Proposition 8.1. The function V defined above is continuous. Furthermore, V is the value
function, expressed in local coordinates, w.r.t. either chart. (Notice that V unambiguously
defines a function on the manifold, since V is periodic, with period 2π .)

Proof. We shall apply Theorem 4.1. This is justified, since the system dynamics satisfy
the compactness, convexity, and local Lipschitzity and boundedness hypotheses (H1) and
(H2), and also the linear growth hypothesis (H3), with functional s : M → R chosen to be
s(x1 , x2 ) = |x2 |2 , w.r.t. local coordinates in either chart.
 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140 137

We shall establish presently

Claim. V is a continuous function. Furthermore

(i) 1 + η1 x2 + min|u|
1 η2 · u = 0, ∀(η1 , η2 ) ∈ ∂ P V0 (x1 , x2 ) and (x1 , x2 ) ∈ R2 \ {0};

(ii) 1 + η1 x2 + min|u|
1 η2 · u
0, ∀(η1 , η2 ) ∈ ∂ P V0 (0, 0) and (x1 , x2 ) ∈ R2 .

The proposition follows simply from the claim, as we now demonstrate.

(A) We show that V is continuous and V (0, 0) = (0, 0).

V (0, 0) = 0 since V0 (0, 0) = 0 and the Vk ’s are nonnegative valued. V0 is clearly con-
tinuous. Notice next that, for any bounded set E ⊂ R2 ,
lim inf V0 (x1 + 2πk, x2 ) = +∞. (30)
k→∞ (x1 ,x2 )∈E
It follows that for each (x1 , x2 ) ∈ E, we can limit attention to a (fixed) finite collection of
functions {Vk }k∈S , when evaluating V (x1 , x2 ) (see (29)). We deduce that V is continuous
since, locally, it is a pointwise infimum over a finite number of continuous functions.

(B) Fix (x1 , x2 ) = (0, 0). We show that

1 + η1 · x2 + min η2 · u = 0, ∀(η1 , η2 ) ∈ ∂ P V (x1 , x2 ).
|u|
1

Since π/2 − α  x1  +3/π + α, we know that

x1 + 2πk = 0 for k ∈ Z. (31)
Suppose that (η1 , η2 ) ∈ ∂ P V (x 1 , x2 ).
Then there exists  > 0, M > 0 such that
 2
V (y1 , y2 ) − V (x1 , x2 )
(η1 , η2 ) · (y1 − x1 , y2 − x2 ) − M (y1 − x1 , y2 − x2 )
(32)
for all (y1 , y2 ) ∈ (x1 , x2 ) + B. Since we can take the infimum defining V (x1 , x2 ) over a
finite number of index values, the infimum is achieved at some index value k. We have then
Vk (x1 , x2 ) = V (x1 , x2 ). Since Vk (y1 , y2
V (y1 , y2 ) for any (y1 , y2 ) ∈ (x1 , x2 ) + B, we
deduce from (32) that
 2
Vk (y1 , y2 ) − Vk (x1 , x2 )
(η1 , η2 ) · (y1 − x1 , y2 − x2 ) − M (y1 − x1 , y2 − x2 )
for all (y1 , y2 ) ∈ (x1 , x2 ) + B. It follows that (η1 , η2 ) ∈ ∂ P V0 (x1 + 2πk, x2 ). But then, by
(31) and the first claimed assertion
1 + η1 · x2 + min η2 · u = 0,
|u|
1
as required.

(C) We show that

1 + η1 · x2 + min η2 · u = 0, ∀(η1 , η2 ) ∈ ∂ P V (0, 0).
|u|
1
138 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140

Notice that V0 (0, 0) = 0 and V (2πk, 0) > 0 for all k. It follows that
V (0, 0) = V0 (0, 0).
Arguing as in step (B) above, we deduce from the second claimed assertion that
1 + η1 · x2 + min η2 · u
0,
|u|
1

as required.
Reviewing properties (A)–(C), we see that they amount to the assertions of Proposi-
tion 8.1. It remains then to validate the claim. We attend, finally, to this task.
Validation of the Claim. Take any (x1 , x2 ) ∈ R2 and (η1 , η2 ) ∈ ∂ P V0 (x1 , x2 ). Then there
exists M > 0 and  > 0 such that
 2
V0 (y1 , y2 ) − V0 (x1 , x2 )
(η1 , η2 ) · (y1 − x1 , y2 − x2 ) − M (y1 − x1 , y2 − x2 )
(33)
for all (y1 , y2 ) ∈ (x1 , x2 ) + B.

(i) Assume (x1 , x2 ) = (0, 0). We must show that

1 + η1 · x2 + min η2 · u = 0. (34)
|u|
1

If (x1 , x2 ) ∈ int A1 ∪ · · · ∪ int A4 then V0 is analytic on a neighbourhood of (x1 , x2 ) and, as

is easily calculated,
1 + ∂/∂x1 V0 (x1 , x2 ) + min ∂/∂x2 V0 (x1 , x2 ) · u = 0.
|u|
1

This implies (34). The other two cases to attend to are

(a) (x1 , x2 ) = (−t 2 /2, t) for some t > 0,

(b) (x1 , x2 ) = (t 2 /2, −t) for some t > 0.

(These relationships provide parametric descriptions of two curves, the union of whose
graphs comprise nonzero points in the complement of int A1 ∪ · · · ∪ int A4 in R2 .) We
consider only case (a). (Case (b) is treated analogously.) For all  > 0 sufficiently small,
 
1
− (t ± ) , t ±  ∈ A2 .
2
2
Inserting
 
1
(y1 , y2 ) = − (t ± )2 , t ± 
2
into (33), dividing across by  and passing to the limit as  ↓ 0 gives
−1 = +η1 t − η2 . (35)
For  > 0, (x1 , x2 ) + (0, −1) ∈ A3 . Inserting
(y1 , y2 ) = (x1 , x2 ) − (0, −1)
 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140 139

Fig. 3. Contours of the value function (lower chart).

Fig. 4. Lower chart.

into (33), dividing across by  and passing to the limit as  ↓ 0 gives

η2
0.
This inequality and (35) imply (36).
140 I. Chryssochoos, R.B. Vinter / J. Math. Anal. Appl. 287 (2003) 118–140

(ii) Assume (x1 , x2 ) = (0, 0). We must show that

1 + η1 · x2 + min η2 · u
0. (36)
|u|
1

For  > 0, (− 2 /2, +) ∈ A4 and (+ 2 /2, −) ∈ A4 . Inserting
 
1 2
(y1 , y2 ) = ∓  , ±
2
into (33), dividing across by  and passing to the limit as  ↓ 0, we deduce that η2 = 0.
Since x2 = 0, (36) follows. We have validated the claim. Proof of the proposition is com-
plete. ✷

Figure 3 shows contours of the value function for the rigid pendulum minimum time
problem, in local coordinates for the lower chart. For a fixed initial condition, the optimal
control is bang bang. Figure 4 shows switching surfaces in local coordinates for the lower
chart, separating the regions where the optimal control takes value +1 from the regions
where the optimal control takes value −1.

References

[1] J.-P. Aubin, Viability Theory, Birkhäuser, Boston, 1991.

[2] M. Bardi, I. Capuzzo-Dolcetta, Optimal Control and Viscosity Solutions of Hamilton–Jacobi Equations,
Birkhäuser, Boston, 1997.
[3] I. Chryssochoos, R.B. Vinter, Optimal Control on Manifolds, in: Proc. ECC Conference, Porto, 2001.
[4] F.H. Clarke, Optimization and Nonsmooth Analysis, Wiley–Interscience, New York, 1983.
[5] F.H. Clarke, R.B. Vinter, On the conditions under which the Euler equation or the maximum principle hold,
Appl. Math. Optim. 12 (1984) 73–79.
[6] H. Frankowska, Lower semi-continuous solutions of the Hamilton–Jacobi equation, SIAM J. Control Op-
tim. 31 (1993) 257–272.
[7] V. Jurdjevic, Geometric Control Theory, in: Cambridge Studies in Advanced Mathematics, Vol. 51, 1997.
[8] H. Frankowska, S. Plaskacz, Semicontinuous solutions of Hamilton–Jacobi–Bellman equations with degen-
erate state constraints, J. Math. Anal. Appl. 251 (2000) 818–838.
[9] J.F. Kingman, S.J. Taylor, Introduction to Measure and Probability, Cambridge Univ. Press, Cambridge,
1966.
[10] A. Krener, The higher order maximum principle and its application to singular extremals, SIAM J. Control
Optim. 12 (1977) 43–51.
[11] R.B. Vinter, Optimal Control, Birkhäuser, Boston, 2000.
[12] L.C. Young, Lectures on the Calculus of Variations and Optimal Control Theory, Saunders, Philadelphia,
1969.
[13] P.R. Wolenski, Y. Zhuang, Proximal analysis and the minimal time function, SIAM J. Control Optim. 36
(1998) 1048–1072.