(Stuart Geman, Mark Johnson (Auth.), Mark Johnson, (B-Ok - Xyz) PDF

The IMA Volumes
in Mathematics
and its Applications
Volume 138
Series Editors
Douglas N. Amold Fadil Santosa
Springer Science+Business Media, LLC

Institute for Mathematics and
its Applications (IMA)
The Institute for Mathematics and its Applications was estab-
lished by a grant from the National Science Foundation to the University
of Minnesota in 1982. The primary mission of the IMA is to foster research
of a truly interdisciplinary nature, establishing links between mathematics
of the highest caliber and important scientific and technological problems
from other disciplines and industry. To this end, the IMA organizes a wide
variety of programs, ranging from short intense workshops in areas of ex-
ceptional interest and opportunity to extensive thematic programs lasting
a year. IMA Volumes are used to communicate results of these programs
that we believe are of particular value to the broader scientific community.
The fulllist of IMA books can be found at the Web site of the Institute
for Mathematics and its Applications:
http://www.ima.umn.edu/springer/full-list-volumes.html.
Douglas N. Arnold, Director of the IMA
**********
IMA ANNUAL PROGRAMS
1982-1983 Statistical and Continuum Approaches to Phase Transition

1983-1984 Mathematical Models for the Economics of Decentralized
Resource Allocation
1984-1985 Continuum Physics and Partial Differential Equations
1985-1986 Stochastic Differential Equations and Their Applications
1986-1987 Scientific Computation
1987-1988 Applied Combinatorics
1988-1989 Nonlinear Waves
1989-1990 Dynamical Systems and Their Applications
1990-1991 Phase Transitions and Free Boundaries
1991-1992 Applied Linear Algebra
1992-1993 Control Theory and its Applications
1993-1994 Emerging Applications of Probability
1994-1995 Waves and Scattering
1995-1996 Mathematical Methods in Material Science
1996-1997 Mathematics of High Performance Computing
1997-1998 Emerging Applications of Dynamical Systems
1998-1999 Mathematics in Biology
Continued at the back

Mark J ohnson Sanjeev P. Khudanpur
Mari Ostendorf Roni Rosenfeld
Editors
Mathematical Foundations
of Speech and
Language Processing
With 56 Illustrations
Springer
Mark Johnson Sanjeev P. Khudanpur Mari Ostendorf
Dept. of Cognitive and Dept. of ECE and Dept. of Dept. of Electrical Engineering
Linguistic Studies Computer Science University of Washington
Brown University Johns Hopkins University Seattle, WA 98195
Providence, RI 02912 Baltimore, MD 21218 USA
USA USA
Roni Rosenfeld Series Editors:

School of Computer Science Douglas N. Amold
Carnegie Mellon University Padil Santosa
Pittsburgh, PA 15213 Institute for Mathematics and
USA its Applications
University of Minnesota
Minneapolis, MN 55455
USA
http://www.ima.unm.edu
Mathematics Subject Classification (2000): 68T10, 68T50, 94A99, 68-06, 94A12, 94A40,
6OJ22, 6OJ20, 68U99, 94-06
Library of Congress Cataloging-in-Publication Data

Mathematical foundations of speech and language processing / Mark Johnson ... [et al.]
p. cm. - (IMA volumes in mathematics and its applications ; v. 138)
Includes bibliographical references.
ISBN 978-1-4612-6484-2
1. Speech processing systems-Mathematical models. 1. Johnson, Mark Edward, 1970-
II. Series.
TK7882.S6S.M38 2004
006.4'S4--<lc22 2003065729
ISBN 978-1-4612-6484-2 ISBN 978-1-4419-9017-4 (eBook)

DOI 10.1007/978-1-4419-9017-4
© 2004 Springer Scienee+Business Media New York

Originally published by Springer-Verlag New York, Ine. in 2004
Softeover reprint of the hardcover 1st edition 2004
AII rights reserved. This work may not be translated or copied in whole or in part without the
written permission of the publisher (Springer Science+Business Media, LLC), except for brief
excerpts in connection with reviews or scholarly analysis. Use in connection with anY form of
information storage and retrieval, electronic adaptation, computer software, or by similar or
dissimilar methodology now known or hereafter developed is forbidden. The use in this publication
of trade names, trademarks, service marks, and similar terms, even if they are not identified as such,
is not to be taken as an expression of opinion as to whether or not they are subject to proprietary
rights.
Authorization to photocopy items for internal or personal use, or the internal or personal use of
specific clients, is granted by Springer-Verlag New York, Inc., provided that the appropriate fee is
paid directly to Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, USA
(Telephone: (508) 750-8400), stating the ISBN number, the title of the book, and the first and last
page numbers of each article copied. The copyright owner's consent does not include copying for
general distribution, promotion, new works, or resale. In these cases, specific written permission
must frrst be obtained from the publisher.
9 8 7 6 5 4 3 2 l SPIN 10951453
Springer-Verlag is part of Springer Science+Business Media
springeron/ine.com
FOREWORD
This IMA Volume in Mathematics and its Applications
MATHEMATICAL FOUNDATIONS OF SPEECH

AND LANGUAGE PROCESSING
contains papers presented at two successful one-week workshops : Math-

ematical Foundations of Speech Processing and Recognition and Mathe-
matical Foundations of Natural Language Modeling. Both workshops were
integral to the 2000-2001 IMA annual program on Mathematics in Multi-
media.
Sanjeev Khudanpur (Department of Electrical and Computer Engi-
neering and Department of Computer Science, Johns Hopkins University) ,
Mari Ostendorf (Signal and Image Processing, University of Washington) ,
and Roni Rosenfeld (School of Computer Science, Carnegie Mellon Univer-
sity) were the organizers of the first workshop held on September 18-22,
2000. The second workshop which took place on October 30-November 3,
2000 was also organized by Khudanpur and Rosenfeld. They were joined by
Mark Johnson (Department of Cognitive and Linguistic Sciences, Brown
University) and Frederick Jelinek (Center for Language and Speech Pro-
cessing, Johns Hopkins University)
We are grateful to all the organizers for making the events successful.
We further thank Mark Johnson, Sanjeev P. Khudanpur , Mari Ostendorf,
and Roni Rosenfeld for their superb role in editing the proceedings.
We take this opportunity to thank the National Science Foundation
for its support of the IMA. We are also grateful to the Office of Naval
Research for providing additional funds to support the Multimedia annual
program.
Series Editors
Douglas N. Arnold, Director of the IMA
Fadil Santosa, Deputy Director of the IMA
v
PREFACE
The importance of speech and language technologies continues to grow

as information, and information needs, pervade every aspect of our lives
and every corner of the globe. Speech and language technologies are used to
automatically transcribe, analyze , route and extract information from high-
volume streams of spoken and written information. Equally important,
these technologies are also used to create natural and efficient interfaces
between people and machines.
The workshop on Mathematical Foundations of Speech Processing and
Recognition, (September 18-22,2000), and the one on Mathematical Foun-
dations of Natural Language Modeling (October 30-November 3, 2000),
were held at the University of Minnesota's NSF-sponsored Institute for
Mathematics and Its Applications (IMA), as part of the "Mathematics in
Multimedia" year-long program. These workshops brought together prac-
titioners in the respective technologies on one hand , and mathematicians
and statisticians on the other hand, for an intensive week of introduction
and cross-fertilization. The intent of these workshops was (1) to provide
the mathematicians and statisticians with an accelerated introduction to
the state-of-the-art in the aforementioned technologies, and the mathemat-
ical challenges lying therein; (2) to expose the practitioners to the state-
of-the-art in various mathematical and statistical disciplines of potential
relevance to their field; (3) to create an environment for the emergence of
cross-fertilization and break-through ideas; and (4) to encourage and facil-
itate new long-term collaborations. Judging from the level of enthusiasm
during the workshops and the long off-hours discussions, the first three
goals achieved unqualified success. As for the fourth goal, some collabora-
tion between practitioners and mathematicians had already begun during
the workshop planning; only time will tell the long-term effects of such
beginnings.
There is a long history of benefit from introducing mathematical tech-
niques and ideas to speech and language technologies. Examples include
applying the source-channel paradigm from information theory to auto-
matic speech recognition (and later also to machine translation and infor-
mation retrieval); applying hidden Markov models to acoustic modeling
and hidden variables to speech and language modeling more generally; ap-
plying decision trees , singular value decomposition and exponential models
to the modeling of natural language ; and applying formal languages theory
to parsing. It is likely that new mathematical techniques, or novel appli-
cations of existing techniques, will once again prove pivotal for moving the
field forward . For example, recent work on making Monte Carlo Markov
Chain techniques more computationally feasible holds promise for breaking
away from point estimation (e.g. maximum likelihood and discriminative
vii
viii PREFACE
criteria) towards full Bayesian modeling in both the acoustic and linguistic
domains.
The role of mathematics and statistics in speech and language tech-
nologies cannot be overestimated. The rate at which we continue to ac-
cumulate speech and language training data is far greater than the rate
at which our understanding of the speech and language phenomena grows.
As a result, the relative advantage of data driven techniques continues to
grow with time, and with it, the importance of mathematical and statistical
methods that make use of such data.
In this volume, we have compiled papers representing some original
contributions presented by participants during the two workshops. More
information about the various workshop presentations and discussions can
be found online, at http://www.ima.umn.edu/multimedia/. In this vol-
ume, chapters are organized starting with four contributions related to
language processing, moving from more general work to specific advances
in structure and topic representations in language modeling. The fifth pa-
per on prosody modeling provides a nice transition, since prosody can be
seen as an important link between acoustic and language modeling. The
next five papers relate primarily to acoustic modeling, starting with work
that is motivated by speech production models and acoustic-phonetic stud-
ies, and then moving toward more general work on new models. The book
concludes with two contributions from the statistics community that we
believe will impact speech and language processing in the future.
Finally, we would like to express our gratitude to the National Science
Foundation for making these workshops possible via its funding of the IMA
and its activities; to the IMA's staff for so ably organizing and adminis-
tering the workshops and related events; and to all the participants for
contributing to the success of these workshops in particular and "the Year
of Mathematics in Multimedia" in general.
Mark Johnson
Department of Cognitive and Linguistic Sciences
Brown University
Sanjeev P. Khudanpur
Department of Electrical and Computer Engineering and Department of
Computer Science
Johns Hopkins University
Mari Ostendorf
Signal and Image Processing
University of Washington
Roni Rosenfeld
School of Computer Science
Carnegie Mellon University
CONTENTS
Foreword v
Pr eface vii
Probability and st atistics in comput at ional
linguistics , a brief review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Stuart Geman and Mark Johnson
Three issues in modern language modeling. . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Dietrich Klakow
Stochastic analysis of Structured Language

Modeling 37
Frederick Jelinek
Latent semantic language modeling for speech

recognition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Jerome R . Bellegarda
Prosody modeling for automatic speech

recognition and understanding 105
Elizabeth Shriberg and Andreas Stolcke
Switching dynamic system models for speech

art iculat ion and acoustics 115
Li Deng
Segmental HMMS: Modeling dynamics and

underlying structure in speech 135
Wendy J. Holmes
Modelling graph-based observation spaces

for segment-based speech recognition 157
James R . Glass
Towards robust and adapt ive speech

recognition models 169
Herve Bourlard, Samy Bengio , and
Katrin Weber
ix
x CONTENTS
Graphical models and automat ic speech

recognition 191
Jeffrey A . Bilmes
An introduction to Markov chain Monte

Carlo methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 247
Julian B esag
Semiparametric filtering in speech processing " 271

Benjamin K edem and K onstantinos Fokianos
List of workshop participants 283

PROBABILITY AND STATISTICS IN
COMPUTATIONAL LINGUISTICS, A BRIEF REVIEW
STUART GEMAN* AND MARK JOHNSON*
1. Introduction. Computational linguistics studies the computat-

ional processes involved in language learning, production, and comprehen-
sion. Computational linguists believe that the essence of these processes
(in humans and machines) is a computational manipulation of informa-
tion. Computational psycho linguistics studies psychological aspects of hu-
man language (e.g., the time course of sentence comprehension) in terms
of such computational processes.
Natural language processing is the use of computers for processing nat-
ural language text or speech. Machine translation (the automatic transla-
tion of text or speech from one language to another) began with the very
earliest computers [Kayet al., 1994]. Natural language interfaces permit
computers to interact with humans using natural language, e.g., to query
databases. Coupled with speech recognition and speech synthesis, these
capabilities will become more important with the growing popularity of
portable computers that lack keyboards and large display screens . Other
applications include spell and grammar checking and document summa-
rization. Applications outside of natural language include compilers , which
translate source code into lower-level machine code, and computer vision
[Foo, 1974, Foo, 1982].
The notion of a grammar is central to most work in computational
linguistics and natural language processing. A grammar is a description
of a language; usually it identifies the sentences of the language and pro-
vides descriptions of them, e.g., by defining the phrases of a sentence, their
inter-relationships, and perhaps also aspects of their meanings . Parsing
is the process of recovering a sentence 's description from its words, while
generation is th e process of translating a meaning or some other part of a
sentence's description into a grammatical or well-formed sentence. Parsing
and generation are major research topics in their own right. Evidently,
human use of language involves some kind of parsing and generation pro-
cess, as do many natural language processing applications. For example, a
machine translation program may parse an input language sentence into a
(partial) representation of its meaning , and then generate an output lan-
guage sentence from that representation.
Although the intellectual roots of modern linguistics go back thousands
of years, by the 1950s there was considerable interest in applying the then
newly developing ideas about finite-state machines and other kinds of au-
tomata, both deterministic and stochastic, to natural language . Automata
'Department of Cognitive and Linguistic Sciences, Brown Univers ity, Providence,

RI 02912, USA.
M. Johnson et al. (eds.), Mathematical Foundations of Speech and Language Processing

© Springer Science+Business Media New York 2004
2 STUART GEMAN AND MARK JOHNSON
are Markov-like machines consisting of a set of states and a set of allowed

state-to-state transitions. An input sequence, selected from a finite input
alphabet, moves the machine from state to state along allowed transitions.
[Chomsky, 1957] pointed out clearly the inadequacies of finite-state ma-
chines for modelling English syntax. An effect of Chomsky 's observations,
perhaps unintended, was to discourage further research into probabilistic
and statistical methods in linguistics. In particular, stochastic grammars
were largely ignored. Instead, there was a shift away from simple automata,
both deterministic and stochastic, towards more complex non-stochastic
grammars, most notably "transformational" grammars. These grammars
involved two levels of analyses , a "deep structure" meant to capt ure more-
or-less simply the meaning of a sentence , and a "surface structure" which
reflects the actual way in which the sentence was constructed. The deep
structure might be a clause in the active voice, "Sandy saw Sam," whereas
the surface structure might involve the more complex passive voice, "Sam
was seen by Sandy."
Transformational grammars are computationally complex, and in the
1980s several linguists came to the conclusion that much simpler kinds
of grammars could describe most syntactic phenomena, developing Gen-
eralized Phrase-Structure Grammars [Gazdar et al., 1985] and Unification-
based Grammars [Kaplan and Bresnan, 1982, Pollard and Sag, 1987] ,
[Shieber, 1986] . These grammars generate surface structures directly; there
is no separate deep structure and therefore no transformations. These kinds
of grammars can provide very detailed syntactic and semantic analyses of
sentences, but as explained below, even today there are no comprehensive
grammars of this kind that fully accommodate English or any other natural
language.
Natural language processing using hand-crafted non-stochastic gram-
mars suffers from two major drawbacks. First, the syntactic coverage of-
fered by any available grammar is incomplete, reflecting both our lack of
understanding of even relatively frequently occuring syntactic constructions
and the organizational difficulty of manually constructing any artifact as
complex as a grammar of a natural language. Second, such grammars al-
most always permit a large number of spurious ambiguities, i.e., parses
which are permitted by the rules of syntax but have unusual or unlikely se-
mantic interpretations. For example, in the sentence I saw the boat with the
telescope, the prepositional phrase with the telescope is most easily inter-
preted as the instrument used in seeing, while in I saw the policeman with
the rifle, the prepositional phrase usually receives a different interpretation
in which the policeman has the rifle. Note that the corresponding alterna-
tive interpretation is marginally accessible for each of th ese sentences : in
the first sentenc e one can imagine that the telescope is on the boat, and in
the second, that the rifle (say, with a viewing scope) was used to view the
policeman.
PROBABILITY AND STATISTICS IN COMPUTATIONAL LINGUISTICS 3
In effect, there is a dilemma of coverage. A grammar rich enough to

accommodate natural language, including rare and sometimes even "un-
grammatical" constructions, fails to distinguish natural from unnatural in-
terpretations. But a grammar sufficiently restricted so as to exclude what
is unnatural fails to accommodate the scope of real language. These obser-
vations lead, in the 1980's, to a renewed interest in stochastic approaches to
natural language, particularly to speech. Stochastic finite-state automata
became the basis of speech recognition systems by out-performing the best
of the systems based on deterministic hand-crafted grammars. Largely in-
spired by the success of stochastic approaches in speech recognition, com-
putational linguists began applying them to other natural language pro-
cessing applications. Usually, the architecture of such a stochastic model
is specified manually (e.g., the possible states of a stochastic finite-state
automaton and the allowed transitions between them), while the model's
parameters are estimated from a training corpus, i.e., a large representative
sample of sentences.
As explained in the body of this paper, stochastic approaches re-
place the binary distinctions (grammatical versus ungrammatical) of non-
stochastic approaches with probability distributions. This provides a way
of dealing with the two drawbacks of non-stochastic approaches. Ill-formed
alternatives can be characterized as extremely low probability rather than
ruled out as impossible , so even ungrammatical strings can be provided
with an interpretation. Similarly, a stochastic model of possible interpre-
tations of a sentence provides a method for distinguishing more plausible
interpretations from less plausible one.
The next section, §2, introduces formally various classes of grammars
and languages. Probabilistic grammars are introduced in §3, along with
the basic issues of parametric representation, inference, and computation.
2. Grammars and languages. The formal framework, whether used

in a transformational grammar, a generalized phrase-structure grammar, or
a more traditionally styled context-free grammar, is due to [Chomsky, 1957]
and his co-workers. In this section, we will present a brief introduction to
this framework . But for a thorough (and very readable) presentation we
highly recommend the book by [Hopcroft and Ullman, 1979].
If T is a finite set of symbols, let T* be the set of all strings (i.e.,
finite sequences) of symbols of T, including the empty string, and let T+
be the set of all nonempty strings of symbols of T . A language is a subset
of T*. A rewrite grammar G is a quadruple G = (T, N, S, R), where T and
N are disjoint finite sets of symbols (called the terminal and non -terminal
symbols respectively), SEN is a distinguished non-terminal called the
start symbol, and R is a finite set of productions. A production is a pair
(a ,j3) where a E N+ and (3 E (N U T)* ; productions are usually written
a -* (3. Productions of the form a -* f, where f is the empty string,
are called epsilon productions. In this paper we will restrict attent ion to
grammars without epsilon productions, i.e., f3 E (NUT)+, as this simplifies

the mathematics considerably.
A rewrite grammar G defines a rewriting relation =}a ~ (N U T)* x
(N U T) * over pairs of strings consisting of terminals and nonterminals as
follows: "(afJ =} "(f3fJ iff a -+ f3 E Rand "(, fJ E (N U T)* (the subscript G
is dropped when clear from the context) . The reflexive, transitive closure
of =} is denoted =} * . Thus =} * is the rewriting relation using arbitrary
finite sequences of productions. (It is called "reflexive" because the identity
rewrite, a =}* a, is included) . The language generated by G , denoted La,
is the set of all strings w E T+ such that S =}* w.
A terminal or nonterminal X E NuT is useless unless there are
"(,fJ E (N U T)* and w E T* such that S =}* "(XfJ =}* w. A production
a -+ f3 E R is useless unless there are "(, fJ E (N U T) * and w E T* such
that S =}* "(afJ =} "(f3fJ =}* w. Informally, useless symbols or productions
never appear in any sequence of productions rewriting the start symbol
S to any sequence of terminal symbols, and the language generated by a
grammar is not affected if useless symbols and productions are deleted from
the grammar.
Example 1. Let the grammar G 1 = (Tl,N1,S,RI), where Tl =
{grows, rice, wheat}, N 1 = {S,NP,VP} and R 1 = {S -+ NPVP,NP-+
rice, NP -+ wheat, VP -+ grows}. Informally, the nonterminal S rewrites to
sentences or clauses, NP rewrites to noun phrases and VP rewrites to verb
phrases. Then Lal = {rice grows, wheat grows}. G 1 does not contain any
useless symbols or productions.
Rewrite grammars are traditionally classified by the shapes of their produc-
tions . G = (T, N , S , R) is a context-sensitive grammar iff for all productions
a -+ f3 E R, [o] ~ 1f31, i.e., the right-hand side of each production is not
shorter than its left-hand side. G is a context-free grammar iff lal = 1, i.e.,
the left-hand side of each production consists of a single non-terminal. G
is a left-linear grammar iff G is context-free and f3 (the right-hand side of
the production) is either of the form Aw or of the form w where A E N
and wE T* ; in a right-linear grammar f3 always is of the form wA or w. A
right or left-linear grammar is called a regular grammar.
It is straight-forward to show that the classes of languages generated
by these classes of grammars stand in equality or subset relationships;
Specifically, the class of languages generated by right-linear grammars is the
same as the class generated by left-linear grammars; this class is called the
regular languages, and is a strict subset of the class of languages generated
by context-free grammars, which is a strict subset of the class of languages
generated by context-sensitive grammars, which in turn is a strict subset
of the class of languages generated by rewrite grammars.
The computational complexity of deciding whether a string is gener-
ated by a rewrite grammar is determined by the class that the grammar
belongs to. Specifically, the recognition problem for grammar G takes a
string w E T+ as input and returns TRUE iff w E La. Let 9 be a class of
PROBABILITY AND STATISTICS IN COMP UTAT IONAL LIN GUISTI CS 5
gr ammars. The universal recognition problem for 9 t akes as input a string

wE T+ and a grammar G E 9 and returns TRUE iff w E Le.
There ar e rewriting grammars that generate languages that are recur-
sively enumerable but not recursive. In essence, a langu age Le is recursively
enumerable if there exist s an algorithm that is guaranteed to halt and emit
TRUE when ever W E La , and may halt and emit NOT TRUE, or may not
halt, whenever W 'I. L a . On the other hand, a language is recursive if
there exists an algorit hm that always halts, and emit s TR UE or NOT TR UE
depending on whether W E La or W 'I. Le, respectively.' Obviously, the
set of recursive languages is a subset of the set of recursively enumera ble
languages. The recognition problem for a language that is recursively enu-
merable but not recursive is said to be undecidable. Since such languages
do exist, generated by rewrite gr ammars, the universal recognition problem
for rewrite grammars is undecidable.
The universal recognition problem for context-sensitive grammars is
decidable, and furthermore is in PSPACE (space polynomial in the size of
G and w), but there are context-sensitive grammars for which the recog-
nition problem is PSPACE-complete [Gar ey and Johnson, 1979], so the
universal recognition problem for context-sensit ive grammars is PSPACE-
complete also. Since NP~PSPACE, we should not expe ct to find a poly-
nomial-time recognition algorithm for arbit rary context-sensit ive gram-
mars . The universal recognition problem for context-free gr ammars is de-
cidable in time polynomial in the size of w and linear in the size of G; as far
as we are aware a tight upp er bound is not known . Finally, the universal
recognition problem for regular grammars is decidable in time linear in w
and G.
It turns out that conte xt-sensit ive gra mmars (where a production
rewrites more than one nonterminal) have not had many applications in
natural language pro cessing , so from here on we will concentrate on context-
free grammars , where all productions t ake the form A -+ {3, where A E N
and {3 E (N U T)+ .
An appealing property of grammar s with productions in this form is
that they induce tree structures on the strings that they genera te. And ,
as we shall see shortly (§3), this is the basis for bringing in probability
distributions and the theory of inference. We say that the context-free
grammar G = (T, N, S, R) generates the labelled, ordered tree 'l/J iff the
root node of 'l/J is labelled S, and for each node n in 'l/J , either n has no
children and its label is a memb er of T (i.e., it is labell ed with a termina l)
or else there is a production A -+ {3 E R where the label of n is A and the
left-to-right sequence of labels of n's immediate childr en is {3. It is straight
forward to show that w is in L e iff G gener ates a tree 'l/J whose yield (i.e.,
the left-to-right sequ ence of t erminal symbols labelling 'l/J's leaf nodes) is
1 A rigorous definition req uires a proper introduction t o Tu ring mach ines . Again , we
recommend [Hopcroft and Ullm an , 1979].
w ; 'I/J is called a parse tree of w (with respect to G) . In what follows, we

define We to be the set of parse tre es generated by G, and YO to be the
function th at maps trees to their yields.
Example 1 (continued). The grammar G 1 defined above generat es
the following two trees, 'l/J1 and 'l/J2.
S S
'l/J1 = »<.VP
NP
'l/J2 = -<.VP
NP
I
rice
I
grows
I
wheat
I
grows
In this example, Y('l/Jd = rice grows and Y('l/J2) = wheat grows

A string of terminals w is called ambiguous iff w has two or more parse trees.
Linguistically, each parse tre e of an ambiguous string usually corresponds
to a distinct interpretation.
Example 2. Consider G 2 = (T2, N 2, S, R2), where T2 = {I, saw,
the, man, with , telescope}, N 2 = {S, NP , N, Det , VP , V, PP, P} and R 2 =
{S -> NPVP,NP -> I,NP -> DetN,Det -> the,NP -> NPPP,N ->
man , N -> telescope, VP -> V NP, VP -> VP PP, pp -> P NP , V -> saw,
P ~ with} . Informally, N rewrites to nouns , Det to determiners, V to
verbs , P to prepositions and PP to prepositional phrases . It is easy to
check that the two trees 'l/J3 and 'l/J4 with the yields Y('l/J3) = Y('l/J4) =
I saw the man with the telescope are both generated by G 2 . Linguisti-
cally, these two parse tr ees represent two different synt actic analyses of
the sentence. The first analysis corresponds to the interpretation where
the seeing is by means of a telescope, while the second corresponds to th e
interpretation where the man has a telescope .
NP--------
--------
~
VP
S
VP
~
PP
V NP P NP
-<. .r>;
Det N Det N
I I I I
I saw the man with the telescope
P ROBAB ILIT Y AND STAT IST ICS IN COM PU TATIONAL LINGUISTI CS 7
--------
S
---------
NP VP
--------
V NP
NP PP
-<. ~
Det N P NP
»<.
Det N
I I
I saw th e man with t he telescope
There is a close relati onship between linear gr amm ar s and finite-state

machines. A finite-state machine is a kind of auto mato n that makes
state-to -state transitions driven by letters from an inpu t alpha bet (see
[Hopcroft and Ullman , 1979]) for det ails) . Each finite-st ate machine has
a corres ponding right-linear gra mmar which has t he property t hat t he set
of strings accepted by t he machine is t he same as t he set of strings gener-
ated by t he gramma r (mod ulo an end marker, as discussed below), and t he
nonterminals of t his grammar are exactly t he set of states of t he machine.
Moreover, t here is an isomorphism between accept ing computations of t he
machine and pa rse t rees generated by t his gra mmar: for each sequence of
states that t he machine transitions through in an accepting computation
t here is a parse t ree of the corresponding gra mmar containing exact ly the
sa me sequence of states (t he exa mple below clarifies this) .
The grammar G M = (T ,N , 5, R) t hat corres ponds to a finite-stat e
machine M is one where t he nonterminal symbols N are t he states of M , t he
start symbo l 5 is t he start state of M , and t he termina l symbols T are t he
input symbols to M together with a new symbol '$' , called t he endma rker,
t hat does not appear in the t ra nsit ion lab els of M . T he prod uctions R
of GM come in two kind s. R can cont ain productions A ----+ b B, where
A , B E N and bET, iff there is a transit ion in M from state A to state
B on input symbol b. R contains the product ion A ----+ $ iff A is a final
state in M . Informally, M accepts a st ring W E T * iff w is t he sequence
of inputs along a path from M 's start state to some final state. It is easy
to show t hat G M generates t he st ring w$ iff M accepts t he string w. (If
we permitted epsilon pro ductions t hen it would not be necessar y to use an
endmarker ; R would contain a produ ction A ----+ E, where E is t he empty
string, iff A is a final state in M) .
Example 3. Cons ider t he right-li near gramma r G 3 = (T3 , N 3 , S, R3 ) ,
where T3 = {a,b, $}, N 3 = {S,A} and R3 = {S ----+ b,S ----+ as , S ----+
b A, A ----+ a A, A ----+ $}. G3 corresponds to t he finite-state machine depicted
below, where t he '> ' attached to the state labelled S indicates it is t he
8 STUART GE MAN AND M A R K JOH NSON
start state, and th e double circle indicat es that t he state lab elled A is a
fina l state. The parse t ree for 'aaba$ ' with respect to G 3 t ra nslates im-
mediately into t he sequence of states t hat the machine t ra nsitio ns t hrough
when accept ing 'aaba' .
a a
--------
S
--------
a S
a S
b A
a A
I
$
As rem arked earlier, context-sensitive and unrestricted rewrite gra m-

mars do not seem to be useful in many natural language processing appli-
cations. On t he other hand, th e not ation of context-free gra mmars is not
ideally suited to formul ating natural language gra mma rs. Fur therm ore, it is
possible to show th at some natural languages ar e not context-free langu ages
[Culy, 1985, Shieber , 1985]. These two factors have led to t he developm ent
of a variety of different kinds of gra mmars. Many of these can be describ ed
as annotated phrase structure grammars, which are extensions of context-
free grammars in which the set of nonterminals N is very large, possibly in-
finite, and Nand R possess a linguist ically motivated st ructure. In Gener-
alized Phrase Structure Gr ammars [Gazda r et al., 1985] N is finite, so these
gra mma rs always generate context -free languages, but in unification gra m-
mar s such as Lexical-Fun ctional Grammar [Ka plan and Bresnan , 1982] or
Head-driven Phrase Structure Grammar [Pollard and Sag, 1987] N is infi-
nite and t he lan guages such grammars generate need not be context -free
or even recur sive.
Example 4. Let G 4 = (T4, N 4, S, R4) where T4 = {a , b}, N 4 = {S} u
{A, B}+ (Le., N 4 consists of S and nonempty strings over t he alpha bet A, B)
and R 4 = {S -> o o : a E {A, B}+} U {A -> a, B -> b} U {Aa -> a o, Ba->
b a : a E {A, B}+}. G 4 generat es the language {ww : w E {a, b}+}, which
is not a context-free language. A parse tree for aa baab is shown below.
----
---- ----
AAB
S
AAB
a
---- ----
a
AB
B
I
b
a
a
AB
B
I
b
PROBABILITY AND STATISTICS IN COMPU TAT IONAL LINGUISTICS 9
3. Probability and statistics. Obviously broad coverage is desir-

able-nat ural language is rich and diverse, and not easily held to a small
set of rules. But it is hard to achieve broad coverage without massive
ambiguity (a sentence may have tens of thousands of parses), and this
of course complicat es applications like language interpr etation, langu age
translation, and speech recognition . This is th e dilemma of coverage that
we referred to earlier , and it sets up a compelling role for probabilistic and
st atistical methods.
We will review the main probabilistic grammars and their associat ed
theories of inference. We begin in §3.1 with probabilist ic regular gram-
mars , also known as hidden Markov models (HMM), which are the foun-
dation of modern speech recognition systems. In §3.2 we discuss prob a-
bilistic context-free grammars, which turn out to be essent ially th e same
thing as branching processes. We review the estim ation problem, the
computation problem , and th e role of criti cality. Finally, in §3.3, we
t ake a more general approach to placing probabilities on grammars, which
leads to Gibbs distributions, a role for Besag's pseudolikelihood method
[Besag, 1974, Besag, 1975]' various computational issues, and, all in all, an
act ive area of research in comput at ional linguistics .
3.1. Hidden Markov models and regular grammars. Recall that
a right-linear grammar G = (T, N, 5, R) corresponding to a finite-state
machine is char acterized by rewrite rules of th e form A --+ b B or A --+ $,
where A, BEN, bET, and $ E T is a special terminal th at we call
an endmarker. Th e connection with Hidden Markov Models (HMM's) is
tr ansp arent : N defines th e states, R defines th e allowed transitions (A can
go to B if th ere exists a production of the form A --+ b B) , and th e string
of te rminals defines the "observation. " The process is "hidden" since, in
general, th e observat ions do not uniquely define th e sequence of states .
In general, it is convenient to work with a "normal form" for right-
linear grammars: all rules are either of the form A --+ b B or A --+ b, where
A , BEN and bET. It is easy to show that every right-line ar gra mmar
has an equivalent normal form in the sense that the two gra mmars produce
the same language. Essenti ally nothing is lost , and we will usually work
with a normal form.
3.1.1. Probabilities. Assume th at R has no useless symbols or pro-
ductions . Th en th e grammar G can be made into a prob abilisti c grammar
by assigning to each nont erminal A E N a prob ability distribution p over
productions of the form A --+ a: E R : for every A E N
(1) p(A --+ o) = 1 .

a E(N UT) +
s.t . (A ->a) ER
Recall that We is the set of parse trees generated by G (see §2). If G is

linear, then 'IjJ E We is characterized by a sequence of productions, start ing
from S. It is, then, straightforward to use p to define a probability P on

We : just take P(1jJ) (for 1jJ E we) to be the product of the associated
production probabilities.
Example 5. Consider the right-linear grammar Gs = (Ts , Ns; 8, R s ),
with Ts = {a, b}, Ns = {8,A} and the productions (Rs ) and production
probabilities (p):
8 --+ a8 p= .80
8 --+ b8 p= .01
8 --+ bA p= .19
A --+ bA p= .90
A --+ b p=.10 .
The language is the set of strings ending with a sequence of at least two
b's. The grammar is ambiguous: in general, a sequence of terminal states
does not uniquely identify a sequence of productions. The sentence aabbbb
has three parses (determined by the placement of the production 8 --+ b A),
but the most likely parse, by far, is 8 --+ a S, 8 --+ a S, 8 --+ bA, A --+ bA,
A --+ b A, A --+ b (P = .8· .8 · .19·.9· .1), which has a posterior probability
of nearly .99. The corresponding parse tree is shown below.
----------- a 8
----------- a 8
----------- b A
a, b b
----------- b A
-----------I
ObOb
~®~
b A
An equivalent formulation is through the associated three-state (8, A,

and F) two-output (a and b) HMM also shown above: the transition prob-
ability matrix is
.81 .19 .00 )

.00 .90 .10
( .00 .00 1.00
where the first row and column represent 8, the next represent A, and the
last represent F; and the output probabilities are based on state-to-state
pairs ,
PROBABILITY AND STAT IST ICS IN COMPUTATIONAL LINGUISTICS 11
(S,S) ~{ :
prob = 80/81
prob = 1/81
prob = 0
(S, A) ~{ :
prob = 1
(A,F)~{: prob = 0
prob = 1 .
3.1.2. Inference. Th e problem is to est imate the tr ansition proba-

bilities, p(.), either from parsed data (examples from 'lJ a) or just from
sentenc es (examples from La) . Consider first the case of parsed data ( "su-
pervised learning") , and let 'ljJl, 'ljJ2," " 'ljJn E 'lJ be a sequence taken iid
according to P . If f(A - t o ; 'IjJ ) is the counting function, counting the
number of times the transition A - t a E R occurs in 'IjJ , then the likelihood
function is
n
(2) L = L(P i 'ljJl,"" 'ljJn ) = IT IT p(A -t a) !( A --+Q;!/J;) .
i= l A--+QER
Th e maximum likelihood estimate is, sensibly, th e relative frequency est i-

mator:
(3)
If a nont erminal A does not appear in the sample, t hen the numerator
and denominator are zero, and p(A - t a ), a E (N U T)+ , can be assigned
arbitra rily, provided it is consiste nt with (1).
The problem of est imat ing P from sentences (" unsupervised learning")
is more interesting , and more important for applications. Recall th at Y( 'IjJ )
is t he yield of 'IjJ , i.e. t he sequence of terminals in 'IjJ. Given a sentence W E
T+, let 'lJ w be the set of parses which yield w: 'lJ w = {'IjJ E 'lJ : Y( 'IjJ ) = w} .
Th e likelihood of a sent ence W E T+ is the sum of the likelihoods of its
possible parses:
L(P i w) = L P( 'IjJ) = L IT p(A -t a)! (A--+Q;!/J) .

!/JEilI", !/JE ilI", A--+QER
Imagine now a sequence 'ljJl, . . . , 'ljJn , iid according to P , for which only th e
corr esponding yields, Wi = Y( 'ljJi), 1 ::; i ::; n , are observed. Th e likelihood
function is
n
(4) L=L(Pi Wl , ... ,Wn) = IT L IT p (A -t a)! (A--+Q;!/J; ) .
i = l !/JE ilI",; A--+QER
To get the maximum likelihood equation, take logarithms, introduce La-

grange multipliers to enforce (1), and set the derivative with respect to
p(A ---t 0:) to zero:
(5)
Introduce E p [' ]' meaning expectation under the probability on P induced

by p , and solve for p(A ---t 0:):
(6)
We can't solve, directly, for P, but (6) suggests an iterative approach

[Baum, 1972]: start with an arbitrary Po (but positive on R). Given Pt,
t = 1,2, ... , define Pt+! by using Pt in the right hand side of (6):
(7)
Evidently, Pt = Pt+! if and only if we have found a solution to the likeli-

hood equation, 8p(1-;a)L = 0, \fA ---t 0: E R. What's more, as shown by
[Baum, 1972], it turns out that L(Pt+l;Wl, . . . ,Wn ) ~ L(Pt;Wl, ... ,Wn ) ,
and the procedure finds a local maximum of the likelihood. It turns out ,
as well, that (7) is just an instance of the EM algorithm, which of course
is more general and was discovered later by [Dempster et al., 1977].
Needless to say, nothing can be done with this unless we can actu-
ally evaluate, in a computationally feasible way, expressions like Ep[f(A ---t
0:; 1/1)11/1 E ww] . This is one of several closely related computational prob-
lems that are part of the mechanics of working with grammars.
3.1.3. Computation. A sentence W E T+ is parsed by finding a se-
quence of productions A ---t b B E R which yield w. Depending on the
grammar, this corresponds more or less to an interpretation of w. Often ,
there are many parses and we say that W is ambiguous. In such cases, if
there is a probability p on R then there is a probability P on W, and a
reasonably compelling choice of parse is the most likely parse:
(8) arg max P(1/I) .

l/JEW,,,
This is the maximum a posteriori (MAP) estimate of 1/I-obviously it min-

imizes the probability of error under the distribution P.
What is the probability of w? How are its parses computed? How is
the most likely parse computed? These computational issues turn out to be
PROBABILITY AND STATISTICS IN COMP UTATIONAL LINGUIST ICS 13
more-or-less the same as the issue of computing Ep[f(A -+ 0:; "p)I"p E 'lJ wl
that came up in our discussion of inference. The basic structure and cost
of the comput at ional algorithm is the same for each of the four problems-
compute the prob ability of w, compute the set of parses, compute the
best parse, compute E p. For regular grammars, there is a simple dynamic
programming solution to each of these problems, and in each case the
complexity is of the order n ·IRI, where n is the length of w , and IRI is the
number of productions in G.
Consider the representative problem of producing the most likely
parse, (8). Let w = (b1 , . . • , bn ) E T", Th ere are n - 1 productions of
the form A k -+ bk+l Ak+l for k = 0, . . . , n - 2, with Ao = S, followed by
a single terminating production A n - 1 -+ bn . Th e most likely sequen ce of
productions can be computed by a dynamic-programming type iteration:
for every A E N initi alize with
A 1(A) =S
V1(A) = p(S -+ b1A) .
Then , given AdA) and Vk(A), for A E N and k = 1,2, . .. , n - 2, compute

A k+1(A) and Vk+l(A) from
Ak+l(A) = arg max p(B -+ bk+1A)VdB)

BE N
Vk+l(A) = p(Ak+l(A) -+ bk+1 A)Vk(Ak+1(A)) .
Finally, let
Consider the most likely sequence of productions from S at "t ime 0" to
A at "time k ," given bi , . . . , bk' k = 1, . . . , n -1. Ak(A) is the st ate at tim e
k - 1 along this sequen ce, and Vk(A) is the likelihood of this sequence.
Therefore, An- 1 d;j A n is the state at time n - 1 associated with the
most likely parse, and working backwards, the best st at e seque nce overall
is Ao, A1 , • . . , An - 1 , where
There can be ties when more than one sequence achieves the optimum.
In fact , the pro cedure genera lizes easily to produce the best l parses , for
any l > 1. Another modification produces all parses, while st ill anot her
computes expe ctations E p of the kind th at app ear in the EM iterati on (7)
or probabilities such as P{Y("p) = w} (thes e last two are, essentially, just
a matter of replacing argmax by summation).
3.1.4. Speech recognition. An outstanding application of proba-

bilistic regular grammars is to speech recognition . The approach was first
proposed in the 1970's (see [Jelinek, 1997] for a survey), and has since be-
come the dominant technology. Modern systems achieve high accuracy in
multi-user continuous-speech applications. Many tricks of representation
and computation are behind the successful systems, but the basic technol-
ogy is nevertheless that of probabilistic regular grammars trained via EM
and equipped with a dynamic programming computational engine. We
will say something here, briefly and informally, about how these systems
are crafted.
So far in our examples T has generally represented a vocabulary of
words, but it is not words themselves that are observable in a speech recog-
nition task. Instead, the acoustic signal is observable, and a time-localized
discrete representation of this signal makes up the vocabulary T. A typical
approach is to start with a spectral representation of progressive, overlap-
ping windows, and to summarize this representation in terms of a relatively
small number, perhaps 200, of possible values for each window. One way
to do this is with a clustering method such as vector quantization. This
ensemble of values then constitutes the terminal set T.
The state space, N, and the transition rules, R, are built from a hierar-
chy of models, for phonemes (which correspond to letters in speech), words,
and grammars. A phoneme model might have, for example , three states
representing the beginning, middle, and end of a phoneme's pronunciation,
and transitions that allow, for example, remaining in the middle state as a
way of modeling variable duration. The state space, N, is small-maybe
three states for each of thirty or forty phonemes-making a hundred or
so states. This becomes a regular grammar by associating the transitions
with elements of T, representing the quantized acoustic features. Of course
a realistic system must accommodate an enormous variability in the acous-
tic signal, even for a single speaker, and this is why probabilities are so
important.
Words are modeled similarly, as a set of phonemes with a variety of al-
lowed transition sequences representing a variety of pronunciations choices.
These representations can now be expanded into basic units of phoneme
pronunciation, by substituting phoneme models for phonemes. Although
the transition matrix is conveniently organized by this hierarchical struc-
ture, the state space is now quite large: the number of words in the system's
vocabulary (say 5,000) times the number of states in the phoneme models
(say 150). In fact many systems model the effects of context on articulation
(e.g. co-articulation), often by introducing states that represent triplets of
phonemes ("triphones"), which can further increase the size of N, possibly
dramatically.
The sequence of words uttered in continuous speech is highly con-
strained by syntactic and semantic conventions. These further constraints,
which amount to a grammar on words, constitute a final level in the hi-
PROBABILITY AND STAT IST ICS IN COMPUTAT IONAL LINGUISTICS 15
era rchy. An obvious candidate model would be a regular grammar , with

N made up of syntactically meaningful parts of speech (verb , noun , noun
phr ase, art icle, and so on) . But implementations generally rely on th e much
simpler and less structured trigram. The set of states is th e set of ordered
word pairs, and the tr ansitions are a priori only limited by notin g t hat t he
second word at one unit of tim e must be the same as the first word at th e
next . Obviously, t he trigram model is of no utility by itself; once again
prob abilities play an essent ial role in meaningfully restricting t he coverage.
Trigrams have an enormous effective state space, which is made all
th e larger by expanding th e words themselves in terms of word models.
Of course the actual number of possible, or at least reasonable, transitions
out of a state in th e resulting (expanded) grammar is not so large. This
fact, together with a host of computational and representational tricks and
compromises, renders the dynamic programming computation feasible, so
that training can be carri ed out in a matter of minutes or hours, and
recognition can be performed at real time , all on a single user's PC.
3.2. Branching processes and context-free grammars. Despit e
th e successes of regular grammars in speech recognition , the problems of
language und erstanding and translat ion are generally better addressed with
th e more structured and more powerful context-free gra mmars. Following
our development of probabilistic regular grammars in the previous section ,
we will address here the inter-related issues of fitting context-free gram-
mars with probability distributions, estimating t he parameters of these
dist ribut ions, and computing various function als of t hese distributions.
Th e context-free grammars G = (T , N , S , R) have rules of t he form
A -; 0:,0: E (N U T )+ , as discussed previously in §2. Th ere is again a
norm al form, known as the Chomsky normal form, which is particularly
convenient when developing probabilistic versions. Specifically, one can
always find a context-free grammar G' , with all productions of th e form
A -; BC or A -; a, A , B , C, EN, a E T , which produces t he same language
as G: L c' = Lc. Henceforth, we will assume th at context-free grammars
are in th e Chomsky normal form.
3.2.1. Probabilities. The goal is to put a prob ability distribution
on th e set of parse trees generated by a context-free grammar in Chomsky
norm al form. Ideally, the distribution will have a convenient parametric
form, that allows for efficient inference and computation.
Recall from §2 th at context-free grammars generate labeled, ordered
tr ees. Given sets of nont erminals N and terminals T , let III be the set of
finite trees with :
(a) root node labeled S;
(b) leaf nodes labeled with elements of T ;
(c) interior nodes labeled with elements of N ;
(d) every nont ermin al (interior) node having either two children la-
beled with nonterminal s or one child labeled with a te rminal.
Every 'Ij; E '1' defines a sentence W E T+ : read the labels off of the terminal
nodes of 'Ij; from left to right . Consistent with the notation of §3.1, we will
write Y( 'Ij;) = w. Conversely, every sentence W E T+ defines a subset of
'1' , which we denot e by ww , consisting of all 'Ij; with yield w (Y('Ij;) = w).
A context-free grammar G defines a subset of '1', We, whose collection of
yields is the language, L e , of G. We seek a probability distribution P on
\[1 which concent rates on \[1 e.
The time-honored approach to probabilistic context-free grammars is
through the production probabilities p : R --+ [0, 1], with
(9) p(A --+ a) = 1 .

OI E N 2Ul'
s.t. (A-+ OI)ER
Following the development in §3.1, we introduce a counting function f(A --+

a ; 'Ij; ), which counts th e number of instances of the rule A --+ a in the tree
7jJ, i.e. the number of nont erminal nodes A whose daughter nodes define,
left-to-right, th e string a. Through f , p induces a prob ability P on '1' :
(10) P('Ij;) = II p(A --+ a)!(A-+ OI ;1/!) .
(A-+OI)ER
It is clear enough that P concentrates on We, and we shall see shortly th at

this parameteriz ation , in terms of products of probabilities p, is particularly
workable and convenient. The pair , G and P , is known as a probabilistic
context-free grammar, or PCFG for short.
Branching Processes and Criticality. Notice the connection to
branching processes [Harris, 1963] : Starting at S, use R, and th e associ-
ated prob abilitie s p(') , to expand nodes into daughter nodes until all leaf
nodes are labeled with terminals (elements of T) . Since branching pro-
cesses display critical behavior, whereby they mayor may not termin ate
with prob ability one, we should ask ourselves whether p truly defines a
prob ability on we -bearing in mind th at '1' includes only finite tr ees. Ev-
idently, for p to induce a probability on We (P(\[1e) = 1), the associated
bran ching process must terminate with probability one. This may not hap-
pen , as is most simply illustrated by a bare-boned example:
Example 6. G6 = (T6 , N 6 , S, R 6) , T6 = {a}, N 6 = {S}, and R 6
includes only
S--+SS
S--+a.
Let p(S --+ S S) = q and p(S --+ a) = 1 - q, and let Sh be the tot al
prob ability of all trees with depth less th an or equal to h. Then S2 = 1 - q
(corresponding to S --+ a) and S3 = (1 - q) + q(1 - q)2 (corresponding to
S --+ a or S --+ S S followed by S --+ a, S --+ a). In general, Sh+l = l - q + q S~,
P ROBABILIT Y AND STAT IST ICS IN COMPU TATIONAL LING UIST ICS 17
which is nonincreasing in q and converges to min(l , ~) as q i 1. Hence

P (W e ) = 1 if and only if q S .5.
More genera lly, it is not diffic ult to characterize production probabili-
ties that put full mass on finite trees (so that P (W e ) = 1), see for example
[Grenander, 1976] or [Harris, 1963]. But t he issue is largely irrelevant , since
maximum likelihood est imated proba bilities always have th is property, as
we shall see short ly.
3.2.2. Inference. As with probabilistic regular gra mmars, t he pro-
duction probabilities of a context- free grammar, which amount to a par am-
ete rizat ion of th e distribut ion P on We , can be estimated from examples.
In one scena rio, we have access to a sequence 'l/Jl, . . " v«from We under P.
Thi s is "supervised learning," in t he sense th at sentences come equipped
with parses. More interestin g is the problem of "unsupervised learning,"
wherein we observe only the yields, Y('l/Jl) , . . . ,Y('l/Jn).
In either case, the treat ment of maximum likelihood est imat ion is es-
sent ially identical to the tr eatment for regular grammars. In particular,
th e likelihood for fully observed dat a is again (2), and the maximum like-
lihood estimat or is again the relative frequency estimator (3). And , in the
unsup ervised case, the likelihood is again (4) and this leads to t he same
EM-type iteration given in (7).
Criticality. We remarked earlier th at t he issue is largely irr elevant.
T his is because estimated probabilities p are always proper probabilities:
p(w) = 1 whenever P is induced by p computed from (3) or any iterat ion
of (7) [Chi and Geman , 1998].
3.2.3. Computation. There are four basic computations: find the
probab ility of a sente nce W E T+ , find a 'l/J E W (or find all 'l/J E w)
satisfying Y('l/J) = w ( "parsing"); find
arg max P('l/J )

,pEW s.t .
Y( ,p)=w
(" maximum a post eriad ' or "opt imal" parsing); compute expectations
of t he form E p, (J(A ---+ a ; 'l/J)I'l/J E ww ] th at arise in iterative est ima-
t ion schemes like (7). Th e four comput ations turn out to be more-or-
less t he same, as was the case for regular grammars (§3.1.3), and there
is a common dynamic-programmin g-like solution [Lari and Young , 1990,
Lari and Young , 1991].
We illustr at e wit h t he problem of finding t he probability of a string
(sente nce) w, under a gra mmar G, and under a probability dist ribution
P concent rating on We. For PCFG s, t he dynamic-programming algo-
rit hm involves a recursion over substrings of th e st ring w t o be parsed.
If w = W I . . . W m is the string to be parsed, then let Wi,j = Wi . .. Wj be
the substring consisting of terminals i through i , with t he convention t hat
W i ,i = Wi · Th e dynamic-programm ing algorit hm works from smaller to
larger substrings Wi,j, calculating the probability that A :::;..* Wi,j for each
nonterminal A E N . Because a substring of length 1 can only be gener-
ated by a unary production of the form A -+ x, for each i = 1, . .. , m,
P(A :::;..* Wi,i) = p(A -+ Wi)' Now consider a substring Wi,j of length 2
or greater. Consider any derivation A ~* Wi,j ' The first production used
must be a binary production of the form A -+ B C, with A, B , C EN.
That is, there must be a k between i and j such that B ~ * Wi,k and
C :::;..* Wk+l,j ' Thus the dynamic programming step involves iterating from
smaller to larger substrings Wi,j, 1 ::::; i, j ::::; m, calculating:
p(A-+B C) L P(B~* Wi,k)P(C:::;,,* Wk+l,j).

H ,G EN k=i,j-l
S .t .A~BGEn
At the end of this iteration, P(w) = P(S ~* Wl,m) ' This calculation
involves applying each production once for each triple of "string positions"
o < i ::::; k < j ::::; m, so the calculation takes O(IRlm 3 ) time .
3.3. Gibbs distributions. There are many ways to generalize. The
coverage of a context-free grammar may be inadequate, and we may hope,
therefore, to find a workable scheme for placing probabilities on context-
sensitive grammars, or perhaps even more general grammars. Or, it may be
preferable to maintain the structure of a context-free grammar, especially
because of its dynamic programming principle, and instead generalize the
class of probability distributions away from those induced (parameterized)
by production probabilities. But nothing comes for free. Most efforts
to generalize run into nearly intractable computational problems when it
comes time to parse or to estimate parameters.
Many computational linguists have experimented with using Gibbs
distributions, popular in statistical physics, to go beyond production-based
probabilities, while nevertheless preserving the basic context-free structure.
We shall take a brief look at this particular formulation, in order to illus-
trate the various challenges that accompany efforts to generalize the more
standard probabilistic grammars.
3.3.1. Probabilities. The sample space is the same: W is the set
of finite trees, rooted at S, with leaf nodes labeled from elements of T
and interior nodes labeled from elements of N. For convenience we will
stick to Chomsky normal form, and we can therefore assume that every
nonterminal node has either two children labeled from N or a single child
labeled from T. Given a particular context-free grammar 0, we will be
interested in measures concentrating on the subset WG of W. The sample
space, then, is effectively IJ1 G rather than 1J1 .
Gibbs measures are built from sums of more-or-less simple functions ,
known as "potent ials" in statistical physics, defined on the sample space.
In linguistics, it is more natural to call these features rather than poten-
tials. Let us suppose, then, that we have identified M linguistically salient
features it, ... , f M, where fk : q,G - t R, through which we will character-

ize the fitness or appropriateness of a structure 7/J E q,G. More specifically,
we will construct a class of probabilities on q,G which depend on 7/J E q,G
only through it (7/J) , .. . , fM(7/J) . Examples of features are the number of
times a particular production occurs, the number of words in the yield ,
various measures of subject-verb agreement, and the number of embedded
or independent clauses .
Gibbs distributions have the form
(11)
where 81 " " 8M are parameters, to be adjusted "by hand" or inferred from
data, 8 = (81 " " 8M), and where Z = Z(8) (known as the "partition
function") normalizes so that Po(q,) = 1. Evidently, we need to assume or
ensure that L 1/JE>va exp{L~ 8di(7/J)} < 00. For instance, we had better
require that 81 < 0 if M = 1 and it (7/J) = IY( 7/J )I (the number of words in
a sentence), unless of course Iq,GI < 00.
Relation to Probabilistic Context-Free Grammars. The feature
set {J(A - t 0:; 7/J )} A->QER represents a particularly important special case:
The Gibbs distribution (11) takes on the form
Evidently, we recover probabilistic context-free grammars by taking 8A -> Q

= loge p(A - t 0:), where p is a system of production probabilities consistent
with (9), in which case Z = 1. But is (12) more general? Are there
probabilities on q,G of this form that are not PCFGs? The answer turns
out to be no, as was shown by [Chi, 1999] and [Abneyet al., 1999]: Given
a probability distribution P on q,G of the form of (12), there always exists
a system of production probabilities p under which P is a PCFG.
One interesting consequence relates to the issue of criticality raised
in §3.2.1. Recall that a system of production probabilities p may define
(through 10) an improper probability P on q,G: P(q,G) < 1. In these cases
it is tempting to simply renormalize, P(7/J) = p(7/J)/P(q,G) , but then what
kind of distribution is P? It is clear enough that P is Gibbs with feature
set {J(A - t 0:; 7/J )} A->QER , so it must also be a PCFG, by the result of Chi
and Abney et al. What are the new production probabilities, p(.)?
For each A E N , consider the grammar GA which "starts at A," i.e.
replace S, the start symbol , by A. If q,A is the resulting set of tree struc-
tures (rooted at A) , then (12) defines a measure PA on q,A , which will
have a new normalization ZA' Consider now the production A - t Be,
A, B, C E N . Chi's proof of the equivalence between PCFGs and

Gibbs distributions of the form (12) is constructive:
is, explicitly, the production probability under which P is a PCFG. For a

terminal production, A - t a,
Consider again example 6, in which S - t S S with probability q and

S - t a with probability 1 - q. We calculated P(iJ!c) = min(l, .!.=.9.) , so
renormalize and define q
P(ol.) - P( 7/J) iJ!

'P - • (1.!.=.9.)
mm , q 7/J E c·
Then P = P when q ::; .5. In any case, P is Gibbs of the form (12), with
B_s-+s s = loge q, BS-+ a = 10ge(1 - q), and Zs = min(l, ~). Accordingly,
P is also a PCFG with production probabilities
min(l,.!.=.9.) min(l,.!.=.9.) 1_ q
p(S - t S S) =q .q 1- q = qmin(l, - - )
mm(1 ' 7 ) q
and
_ 1
p(S - t a) = (1 - q).
mm(l, 7) 1
In particular, p = p when q ::; .5, but p(S - t S S) = 1-q and p(S - t a) = q

when q > .5.
3.3.2. Inference. The feature set {fdi=I, ...,M can accommodate ar-
bitrary linguistic attributes and constraints, and the Gibbs model (11),
therefore, has great promise as an accurate measure of linguistic fitness.
But the model depends critically on the parameters {Bdi=I ,...,M , and the
associated estimation problem is, unfortunately, very hard. Indeed, the
problem of unsupervised learning appears to be all but intractable.
Of course if the features are simply production frequencies, then (11)
is just a PCFG, with Z = 1, and the parameters are just log production
probabilities (BA-+a = 10gep(A - t 0:)) and easy to estimate (see §3.2.2).
More generally, let B = (01 , . . • ,OM) and suppose that we observe a sample
'l/Jl ,' " 'l/Jn E iJ!e ("supervised learning") . Writing Z as Z(O) , to emphasize
the dependency of the normalizing constant on 0, the likelihood function is
which leads to the likelihood equations (by setting 8~i log L to 0):
(13)
where Eo is expectation under (11). In general, We is infinite and depend-

ing on the features {fi}i=I ,...,M and the choice of 0, various sums (like Z(O)
and Eo) could diverge and be infinite. But if these summations converge,
then the likelihood function is concave . Furthermore, unless there is a lin-
ear dependence among {fih=I ,...,M on {1Pih=I ,...,n, then the likelihood is
in fact strictly concave, and there is a unique solution to (13). (If there is
a linear dep endence, then there are infinitely many () values with the same
likelihood.)
The favorable shape of L(O; WI . .. , Wn) suggests gradient ascent, and
in fact the OJ component of the gradient is proportional to ~ I:7=1 f j( Wi ) -
Eo[!J(W)]. But Eo[J] is difficult to compute (to say the least) , except
in some very special and largely uninteresting cases. Various efforts to
use Monte Carlo methods to approximate Eo[fl, or related quantities that
arise in other approaches to estimation, have been made [Abney, 1997]. But
realistic grammars involve hundreds or thousands of features and complex
feature structures, and under such circumstances Monte Carlo methods ar e
notoriously slow to converge . Needless to say, the important problem of
unsupervised learning, wher ein only yields are seen, is even more daunting.
This state of affairs has prompted a number of suggestions in way of
compromise and approximation. One example is the method of pseudolike-
lihood, which we will now discuss.
Pseudolikelihood. If the primary goal is to select good parses, then
perhaps the likelihood function
(14)
asks for too much , or even the wrong thing. It might be more rele-
vant to maximize the likelihood of the observed parses , given the yields
Y( WI), . . . ,Y(Wn) [Johnson et al. , 1999]:
n
(15) II Po(wiIY(Wi)) .
i=1
One way to compare these criteria is to do some (loose) asymptotics.
Let P( W) denote the "t rue" distribution on We (from which WI, · · · , Wn
are presumably drawn, iid), and in each case (14 and 15) compute the
large-sample-size average of the log likelihood:
1 1
II L log Po(7/Ji)
n n
-log Po(7/Ji) = -
n i=l n i=l
~ L P(7/J) logPo(7/J)
'l/JEw e
= L P(7/J)logP(7/J) - L P(7/J) log ~~~)

'l/JEw e 'l/JEw e
1 1 n
II Po (7/JiIY(7/Ji)) = -n Llog Po (7/JdY(7/Ji))
n
-log
n i=l i=l
~ L P(w) L P(7/JIY(7/J)) log Po (7/JIY(7/J))

wET+ 'l/JEwe
s.t. Y('I/J)=w
= L P(w) L P(7/JIY(7/J)) log P(7/JIY(7/J))

wET+ 'l/JEwe
s.t . y('I/J)=w
" " P(7/JIY(7/J))

- L..J P(w) L..J P(7/JIY(7/J)) log Po(7/JIY(7/J))
wET+ 'l/JEw e
s.t . y('I/J)=w
Therefore, maximizing the likelihood (14) is more or less equivalent to min-

imizing the Kullback-Leibler divergence between P(7/J) and Po(7/J), whereas
maximizing the "pseudolikelihood" (15) is more or less equivalent to min-
imizing the Kullback-Leibler divergence between P(7/JIY(7/J)) and Po(7/J1
Y( 7/J) )-averaged over yields. Perhaps this latter minimization makes more
sense, given the goal of producing good parses.
Maximization of (15) is an instance of Besag's remarkably effective
pseudolikelihood method [Besag, 1974, Besag, 1975], which is commonly
used for estimating parameters of Gibbs distributions. The computations
involved are generally much easier than what is involved in maximizing
the ordinary likelihood function (14). Take a look at the gradient of the
logarithm of (15): the ()j component is proportional to
(16)
Compare this to the gradient of the likelihood function , which involves

EO[!j(7/J)] instead of ~ L:~=1 Eo[Ij(7/J)!Y(7/J) = Y(7/Ji)] ' Eo [Ij(7/J)] is essen-
tially intractable, whereas Eo [Ij (7/J) IY(7/J)] can be computed directly from
the set of parses of the sentence Y(7/J). (In practice there is often massive
ambiguity, and the number of parses may be too large to feasibly consider.
Such cases require some form of pruning or approximation.)
PROBABILITY AND STAT IST ICS IN COMPU TAT IONAL LINGUISTICS 23
Thus gradient ascent of th e pseudolikelihood function is (at least ap-

proximat ely) computationally feasible. This is particularly useful since the
Hessian of the logarithm of the pseudolikelihood function is non-positive,
and therefore there are no local maxima. What's more, under mild condi-
tions pseudolikelihood est imators (i.e. maximizers of (15)) are consistent
[Chi, 1998].
4. Generalizations and other directions. There are a large num-

ber of extensions and applications of the grammatical tools just outlined.
Treebank corpora, which consist of the hand-constructed parses of tens of
thousands of sentences, are an ext remely important resource for develop-
ing stochastic grammars [Marcus et al., 1993]. For example, the parses in
a tr eebank can be used to generate, more or less automatically, a PCFG.
Productions can be simply "read off" of the parse tr ees, and production
probabilities can be estimated from relative frequencies, as explained in
§3.1.2. Such PCFGs typically have on the order of 50 nont erminals and
15,000 productions. While the average number of parses per sentence is
astronomical (we estimate greater than 1060) , th e dyn amic programming
methods described in §3.2.3 are quite tractable, involving perhaps only
hundr eds of thousands of operations.
PCFGs derived from tr eebanks are moderately effective in parsing nat-
ural language [Charniak, 1996]. But the actual probabilities generat ed by
these models (e.g. t he probability of a given sentence) are considerably
worse th an tho se generated by oth er much simpler kinds of models, such
as t rigra m models. This is presumably because th ese PCFGs ignore lexi-
cal dependencies between pairs or t riples of words. For example, a typical
t reebank PCFG might contain the productions VP -+ V NP , V -+ eat and
NP -+ pizza, in order to generat e the string eat pizza. But since noun
phr ases such as airpl anes are presumably also generated by productions
such as NP -+ airplanes, this grammar also generates unlikely st rings such
as eat airplanes.
One way of avoiding this difficulty is to lexicalize t he gra mmar, i.e., to
"split" the nonterminals so that they encode the "head" word of the phrase
that they rewrit e to . In the previous example , the corresponding lexicalized
productions are VP eat -+ Vea t NP pizza , Vea t -+ eat and NP piz za -+ pizza.
This permits th e grammar to capt ure some of th e lexical selectional pref-
erences of verbs and other heads of phrases for specific head words. This
technique of splitting the nont erminals is very general, and can be used to
encode oth er kinds of nonlocal dependencies as well [Gazdar et al., 1985] .
In fact , th e state of the art probabilistic parsers can be regard ed as PCFG
parsers operating with very large, highly structured, nonterminals. Of
course, t his nont ermin al splitting dramatically increases th e number of
nonterminals N and the number of productions R in t he gra mmar, and
this complicates both th e computational problem [Eisner and Satta, 1999]
and, more seriously, inference. While it is str aight-forward to lexicalize
24 ST UART GEMAN AND MARK JOHNSON
the productions of a context-free grammar, many or even most produc-

tions in the resulting grammar will not actually appear even in a large
treebank. Developing methods for accurately estimating the probability
of such productions by somehow exploiting the structure of the lexically
split nonterminals is a central theme of much of the research in statistical
parsing [Collins, 1996, Charniak, 1997].
While most current statistical parsers are elaborations of the PCFG
approach just specified, there are a number of alternative approaches that
are attracting interest. Because some natural languages are not context-
free languages (as mentioned earlier), most linguistic theories of syntax
incorporate context-sensitivity in some form or other. That is, according
to these theories the set of trees corresponding to the sentences of a hu-
man language is not necessarily generated by a context-free grammar, and
therefore the PCFG methods described above cannot be used to define a
probability distribution over such sets of trees . One alternative is to employ
the more general Gibbs models, discussed above in §3.3 (see for example
[Abney, 1997]). Currently, approaches that apply Gibbs models build on
previously existing "unification grammars" [Johnson et al., 1999], but this
may not be optimal, as these grammars were initially designed to be used
non-stochastically.
REFERENCES
[Abney et al., 1999] ABNEY , STEVEN, DAVID McALLESTER, AND FERNANDO PEREIRA .
1999. Relating probabilistic grammars and automata. In Proceedings of
the 37th Annual Meeting of the Association for Computational Linguistics,
pages 542-549, San Francisco. Morgan Kaufmann.
[Abney, 1997] ABNEY , STEVEN P. 1997. Stochastic Attribute-Value Grammars. Com-
putational Linguistics, 23(4) :597-617.
[Baum, 1972] BAUM, L.E . 1972. An inequality and associated maximization techniques
in statistical estimation of probabilistic functions of Markov processes. In-
equalities, 3 :1-8.
[Besag , 1974] BESAG , J . 1974. Spatial interaction and the statistical analysis of lattice
systems (with discussion). Journal of the Royal Statistical Society, Series
D,36:192-236 .
[Besag, 1975] BESAG , J . 1975. Statistical analysis of non-lattice data. The Statistician ,
24:179-195.
[Charniak, 1996] CHARNIAK, EUGENE. 1996. Tree-bank grammars. In Proceedings of the
Thirteenth National Conference on Artificial Inteliigence , pages 1031-1036,
Menlo Park. AAAI Press/MIT Press .
[Charniak, 1997] CHARNIAK , EUGENE. 1997. Statistical parsing with a context-free
grammar and word statistics. In Proceedings of the Fourteenth National
Conference on Artificial Inteliig ence, Menlo Park. AAAI Press/MIT Press.
[Chi , 1998] CHI, ZHIYi. 1998. Probability Models for Complex Syst ems. PhD thesis ,
Brown University.
[Chi , 1999J CHI, ZHIYi. 1999. Statistical properties of probabilistic cont ext -free gr am-
mars. Computational Linguistics, 25(1) :131-160.
[Chi and Geman, 1998] CHI, ZHIYI AND STUART GEMAN. 1998. Estimation of proba-
bilistic context-free grammars. Computational Linguistics, 24(2):299-305.
PROB ABILITY AND STATISTICS IN COMPUTATIONAL LINGUISTICS 25
[Cho msky, 1957] C HOMSKY, NOAM. 1957. Syntactic Structures. Mouton, The Hague.
[ColI ins , 1996] COLLINS, M .J . 1996. A new statistical parser based on bigram lexical
d ep endencies. In Th e Proceedings of the 34th A nnual Meeting of the Asso-
ciation for Com putational Linguistics, pages 184-191, San Fr an cisc o. The
Association for Co m p utationa l Linguistics, Morgan Kaufmann.
[C uly, 1985] C ULY, C HRISTOPHER. 1985. The complexity of t he vocab ula ry of Bambara.
Linguistics and Philosophy, 8(3 ):345-352 .
[Dempster et al ., 1977] DEMPSTER, A ., N . LAIRD , AND D . RUBIN. 1977. Maximum
likelihood from incomplet e data via the EM a lgo rit h m . Jou rnal of the
Royal Statistical Society, Series B, 39:1- 38.
[Ei sner and Satta, 1999] EISNER, J ASON AND GIORGIO SATTA. 1999. Efficient parsing
for bilexical context-free gr ammars and head automaton gr ammars. In Pro-
ceedings of the 37th Annual Meeting of the As sociat ion for Com putational
Linguistics, pages 457-464, 1999.
[Foo, 1974] Foo , K.S. 1974. Syntactic Methods in Pattern Recognition . Academic
Press.
[Foo , 1982] Foo , K.S. 1982. Syntacti c Patt ern Recognition and Applications. Prentice-
HalI .
[Garey and Johnson, 1979] GAREY, MICHAEL R. AND DAVID S . JOHNSON . 1979. Com-
puters and Introctability: A Guide to the Theory of NP- Completeness.
W.H. Freeman and Company, New York.
[Gazdar et al ., 1985] GAZDAR, GERALD, EWAN KLEIN , GEOFFREY P ULLUM , AND IVAN
SAG. 1985. Generolized Phrose Structure Grommar. Basil Bl ackwelI, Ox-
ford .
[G re na nd er , 1976] GRENANDER, ULF. 1976. Lectures in Pattern Th eory . Volum e 1:
Pattern Synthesis. Springer , Berlin.
[Harr is, 1963] HARRIS, T .E. 1963. Th e Th eory of Bron ching Processes. Sp ringer, Berlin.
[Ho p cro ft and UlIma n, 1979] HOPCROFT, JOHN E . AND JEFFREY D . ULLMAN. 1979.
Introdu ction to Automata Th eory, Languages and Com putation. Addison-
Wesley.
[Jelinek , 1997] JELINEK , FREDERICK. 1997. Statistical Methods f or Speech Recognition .
The MIT Press, Cambr id ge, Massachusetts .
[Johnson et a l., 1999] JOHNSON, MARK , STUART GEMAN , STEPHEN CANON, ZHIYI C HI,
AND STEFA N RIEZLER. 1999. Estimators for stochastic "u n ification-bas ed "
grammars. In Th e Proceedings of the 37th Annual Confe rence of the Associ-
ation for Com putational Lin guistics, pages 535- 541, San Francis co. Morgan
Kaufmann .
[Kaplan and Bresnan, 19821 KAPLAN , RONALD M . AND JOAN BRESNA N. 1982. Lexical-
Functional Grammar : A formal system for grammatical representation. In
Joan Bresnan, editor, Th e Mental Representation of Grommatical Rela-
tions , Ch apter 4, pages 173-281. The MIT Press.
[Kay et al ., 1994] KAY, MARTIN , JEAN MARK GAVRON , AND PET ER NORVIG . 1994. Verb-
mobil : a tronslation system for face-to-face dialog. CSLI Press, Stanford,
California.
[Lari and Young, 1990] LARI, K . AND S .J. YOUNG . 1990. The est imat ion of Stochastic
Context-Free Grammars using the Inside-Outside a lgori t h m . Computer
Speech and Language, 4(3 5-56) .
[Lari a nd Young, 1991J LARI, K . AND S.J . YOUNG . 1991. Applications of Stochastic
Context-Free Grammars using the Inside-Outside a lgor it h m . Com puter
Speech and Language, 5 :237-257.
[Marcus et a l., 1993] MARCUS, MICHELL P. , BEATRICE SANTORINI , AND MARY A NN
MARCINKIEWICZ. 1993. Building a large annotated corpus of English: The
P enn Treebank. Com putational Linguistics, 19 (2 ):313-330.
[PolIard and Sag, 1987] POLLARD, C ARL AND IVAN A. SAG. 1987. Information-based
Syntax and Semantics. Number 13 in CSLI Lecture Notes Seri es. C h icago
University Press , C h icago.
[Shieber, 1985] SHIEBER, STUART M . 1985 . Evidence against the Context-Freeness of

natural language. Linguistics and Philosophy , 8(3) :333-344.
[Shieber, 1986J SHIEBER, STUART M. 1986. An Introduction to Unification -based Ap-
proaches to Gmrnmar. CSLI Lecture Notes Series. Chicago University Press,
Chicago.
THREE ISSUES IN MODERN LANGUAGE MODELING
DIETRICH KLAKOW'
Abstract. In this pap er we discuss t hree issues in mod ern language mode ling. The
first one is the question of a quality measure for language models, the second is lang uage
mod el smooth ing and the third is t he quest ion of how t o bu ild good long-r ang e language
mod els. In all three cases some results are given indicating possible directions of further
resear ch.
Key words. Language models, quality measures, perpl exity, smoot hing, long-range
correlat ions.
1. Introduction. Language models (LM) are very often a compo-

nent of speech and natural language processing systems. They assign a
prob ability to any sentence of a language. The use of language models in
speech recognition syst ems is well known for a long tim e and any modern
commercial or academic speech recognizer uses them in one form or other
[1] . Closely related is th e use in machine translation syst ems [2] where a
langu age model of th e t arget language is used. Relatively new is the lan-
guage model approach to information retrieval [3] where the query is the
languag e model history and th e documents are to be predicted. It may
even be applied to question answering. When asking an open question like
"T he name of the capital of Nepal is X?" filling the open slot X is just
t he language modeling tas k given the previous words of th e questi on as t he
history.
This pap er extends the issues raised by the aut hor being a panelist at
the language modeling workshop of the Institute of Mathemati cs and its
Applications. Th e three sect ions will discuss the three issues raised in the
abst ract.
2. C orrelation of word error rate and perplexity. How can the
value of a language model for speech recognition be evaluated with little
effort ? Perplexity (PP, it measures the predictive power of a language
model) is simple to calculat e. It is defined by:
(1) I PP = _ " Ntest(w, h) .1 ( Ih)

og LJ N (I) OgPLM w
w, h test t
where Ntest(w, h) and Ntest(h) are frequencies on a test corpus for word w
following a history of words h and PLM(wlh) is the probability assigned to
th at sequence by th e langu age model.
The possible correlation of word-error-rat e (WER, th e fraction of
word s of a text , miss-recognized by a speech recognizer) words) and per-
plexity has been an issue in th e liter ature for quit e some time now, but
' P hilips GmbH Forschungslab oratorien , Weisshau sst r.2, 0 -52066 Aachen, Germany
(dietrich.klakow@philips.com ).
27
28 DIETRICH KLAKOW
the value of perplexity as a quality measure for language models in speech

recognition has been questioned by many people. This triggered the de-
velopment of new evaluation metrics [4-8] . The topic of a good quality
measure for language models deserves much attention for two reasons:
• Target function of LM development: a clear and mathe-
matically tractable target function for LM construction allows for
mathematically well defined procedures. Such a target function
completely defines the problem. Then the LM task is just to find
better language model structures and to optimize their free param-
eters .
• Fast development of speech recognition systems: An in-
direct quality measure (as compared to WER) of LMs allows LM
development mostly decoupled from acoustic training and optimiz-
ing the recognizer as such. This is essential to speed up the process
of setting up new speech recognition systems.
Perplexity is one of the quality measures that used to be very popu-
lar but has been questioned during the last years. For the reasons given
above perplexity has clear advantages. We only have to know how well it
correlates with word-error-rate.
An important new aspect from our investigations [9] is the observation
that both perplexity and WER are subject to measurement errors . This is
mostly due to the finite size of our test samples. It is straightforward to
develop uncertainty measures for WER and PP and details can be found
in [9] or any basic mathematics book [10]. For all results shown here we
picked a 95% confidence level.
Based on this we can derive uncertainty measures for the correlation
coefficient of WER and PP. On top of the uncertainty coming from the
measurement for one individual LM we now also have to take into account
the number of LMs used to measure the correlation. The more LMs built
the better.
Motivated by Fig. 1 our hypothesis is that there is a power-law relation
of the form
(2) WER = bPp a
where a and b are free parameters. We found that they depend on the data
set (e.g. WSJ or Broadcast News) under investigation.
To actually test the correlation we built 450 different language mod-
els using the whole variety of available techniques : backing-off models,
linear interpolation, class-models, cache-models and FMA-adapted models
[11] . Note, that the language models with the lowest perplexities shown
are highly tuned state-of-the-art trigrams. All those models were trained
on different portions of Wall Street Journal Corpus (WSJ) which contains
about 80000 articles (40 million running words). We use a vocabulary of
64000 words. For testing the adaptation spoke of the 1994 DARPA eval-
uation was used. There, articles (development-set and test-set with about
THREE ISSUES IN MODERN LANGUAGE MODELING 29
50
47.5
45
42.5
40
ta: 37.5
w
~ 35
32.5
30
27.5
..
25
300 350 400 500 600 BOO 1000 1200 1500 2000 2500
Perplexity
FIG. 1. Correlation of WER and perplexity tested on data from the DARPA 1994
evaluation (spoke 4). Only a sm all fmction of the error bars are shown , to keep the
power law fit visible.
2000 words) with special topics (like "Jackie Kennedy") were provided.
The task was to optimizes and adapt language models to the particular
topics.
The results of our experiments given are in Fig. 1. Each point cor-
responds to one of the 450 language models. Only a small fraction of the
error bars are shown to keep the power law fit visible. We observe that
the power law fit nicely runs through the error-bars. The optimal fit pa-
rameters in (2) are a = 0.270 ± 0.002 and b = 6.0 ± 0.01. Those are not
universal values but they depend on the corpus!
The correlation coefficient is given in Tab . 1. In addition we now also
show results for the 1996 and 1997 DARPA Hub4-evaluation data. For a
perfect correlation, the correlation coefficient r should be one. We observe
that r = 1 is always within the error-bars and hence we have no indication
that the power-law relation (2) is not valid. Please note that the fact that
both values (and all correlation-coefficient values given in the literature)
are smaller than one is not the result of a systematic deviation but a fact
coming from the definition of th e correlation coefficient.
In summary: we have observed no indication, that WER and per-
plexity are not perfectly correlated. However, these are only two data-sets
investigated. We will perform the same analysis on all future speech recog-
30 DIETRICH KLAKOW
TABLE 1
Measured correlation coefficients r and their error bars.
Data Correlation r
Hub4: 96 + 97 0.978 ± 0.073
DARPA Eval 1994: "Kennedy" 0.993 ± 0.048
nition tasks we are going to work on, but collecting data for a huge number
of really different language models is a time-consuming endeavor . We would
like to invite others to join the undertaking.
3. Smoothing of language models. Smoothing of language models
attracted much attention for a very long time. However for backing-off
language models the discussion calmed down during the last few years
as most people started to think that there is very little room for further
improvement.
A well established method is absolute discounting with marginal
backing-off [12]. It is defined by a very simple structure:
Count(hN, w)-d ) ) . ( )
p(wlhN)= Count(hN) +a(h N · ,8(wlhN-1 If Count hN,W >0,
{
a(hN)' ,8(wlhN- d if Count(hN,w)=O,
with the discounting parameter d (0::; d ::; 1) and the dedicated backing-off
distribution ,8(wlh) which is normalized 2: ,8(wlh) = 1.
w
Absolute discounting refers to the fact that d is subtracted from the
observed counts . Marginal backing-off means that special backing-off dis-
tributions are used rather than smoothed relative frequencies. How to cal-
culate the optimal backing-off distributions was described by Kneser and
Ney [12] .
Can we do better smoothing? To answer this question we want to
first turn to a very basic observation: Zipfs law [13, 14]. To observe this
behavior on a corpus, the frequency of the words in the corpus is counted,
this list is sorted by frequency and then for each word the position in the
list (the rank) is plotted versus the frequency on a doubly logarithmic scale.
The result is shown in Fig. 2. We looked at two different "texts" . One is
the novel "Crime and Punishment" by the Russian poet Dostoevsky. The
other is the Philips research speech recognizer, a typical piece of C-code.
The observation is that in both cases the statistics is nearly a power law
even though the exponents differ.
This was observed for the first time by Zipf and since then several
models have been developed, the most well known by Mandelbrot [15]. He
used the model of a "string of letters" chopped randomly into pieces. This
behavior is very general and can be observed very often in nature. Some
examples for systems with power law distributions:
Zipfs Law
0.1 ,--~-~~,--~-"""".......,r--""""'-""""'""""-""""'--'-",,,,--'--~"""
C-Program (Speech Recognizer)
Fit -- -----
Russian Novel(Crimeand Punishment) +
Fit ..
0.01
0.001
>-
0
c:
Q)
:>
xr
l'!
u. 0.0001
Q)
>
1a
-~~~
a;
a:
le-OS
<, ~~
....•... " .
le-06 .....•........•
<,
le-07
1 10 100 1000 10000 100000
Rank
FIG. 2. Zipf law demonstrated for a natural text and a C-program.
• Smash a piece of glass like a window pane and do statistics of the

size of the fragments
• Measure the file sizes of your hard disc and plot the results accord-
ingly
• Consider on the strength of earth quakes
Always a nearl y power law behavior can be observed. The function de-
scribing the relation is:
(3) f w _ Count(w)
( ) - TotalCount ;::j (c + r(w))B
where r(w) is the rank of the word in the sorted list and f(w) the corr e-
sponding frequency. This function has two parameters: B and c. Here J..L
serves as a scaling but can also be used to normalize the distribution. For
all the experiments described above only these parameters vary.
We can and should use this observation to create a new smoothed LM
type. There is no need to actually estimate probabilities or frequencies.
All we have to do is to est imate the three parameters and the rank of the
words. This has to be compared with the task of estimating probabilities
for about a hundred thousand words . Hence we have very much simplified
the estimation problem.
To actually do language modeling we proceed as follows:
• Estimate the rank of each word: All words are sorted ac-
cording to their frequency in the training corpus. In case of equal
32 DIETRICH KLAKOW
frequency, a huge background corpus is used to decide which word

to rank higher .
• Obtain Probabilities: The actual probabilities can either be
estimated from relative frequency in a huge background corpus
or from (3) where the parameters are estimated on the training
corpus .
The experiments were again performed on the DARPA Spoke 4 data
from the 1994 evaluation. In addition to the topic "Kennedy" we also used
stories about "Korea". The background corpus are the 40 million words
from Wall Street journal. The results are given in Tab. 2. Training on
the background corpus WSJ gives very high perplexities (first line). The
domain specific training material supplied for the evaluation gave much
better results (next two lines). But the models are undertrained as most
words from the 64000-word vocabulary are not observed in this small 2000-
running-words training data. When we now perform the sorting procedure
as described above and use the probabilities from the full WSJ-corpus we
get a very significant improvement (last but one line) and using the Mandel-
brot function (3) with parameters tuned on the domain specific adaptation
corpus gives an additional boost (last line).
We can conclude that we have demonstrated a new smoothing method
which doesn't have any zero-frequency problem because it uses ranks of
words and no probabilities have to be estimated. We did this to show that
there is still room for improved smoothing.
TABLE 2
Improved smoothing of unigrams (BO : Backing-off smoothing).
Model PPKennedy PPKorea

BO Unigram WSJ 2318 1583
BO Unigram "Kennedy" 1539 -
BO Unigram "Korea" - 1277
Zipf Unigram (WSJ) 1205 892
Zipf Unigram (Fit) 1176 886
4. Modeling long range dependencies. So far trigrams are very

popular in speech recognition systems . They have a very limited context
but they seem to work quite well. Still, speech recognition systems and
statistical translation systems tend to produce output that locally looks
reasonable but globally is inconsistent and humans can easily spot this .
One approach to cure this is the combination of grammars with trigrams
[16, 17]. But this is not the only way to approach long-range dependencies
in language.
To motivate our approach we want to start with simple observations
from the Wall Street Journal Corpus, which we also hold true on other
corpora (British National Corpus, Broadcast News, Verbmobil,...).
AND SEVEN
,J'l .l
:\ ,1' . it
.
. ... .... .. .. . . .J..~I.l ~.~:~~:~:!:!.•_~:., , ' .. ~ j\
I~ i
7
~~ i \
.I \/ ~ I \ i
!
5 I
~ I \ '\
~. ! \ ! 'J,'\t\
: ' 3 . Y 'l·" !_.
i.·Ii-f_,
................................................... ..................~:~:!:~.1:t......_...........
D ,Il:~~t. ( l . BlgI'amj DiI ~.", . (1 . BIQIl mj
PRESIDENT HE
FIG. 3. Pair autocorrelation function s.
In Fig. 3 the pair auto-correlation function
_ Pd(W,W)
(4) Cd W
( ) - p(w)2
is given for four example words. Here, Pd(W, w) is the probability that the
word W occurs now and is observed again after skipping d words and p(w)
is the unigram distribution. We have the obvious prop erty
(5) lim Cd(W) = 1.

d-eoo
For four different words Fig. 3 shows that it is indeed the case that af-
ter skipping about a thousand words in between th e value 1 is reached .
However each word has its individual pattern as to how it approaches this
limit . A short-function word like "and" shows at short distances a strong
anti-correlation and then approaches this limit rapidly. "President" shows
a broad bump stemming from purely semantic relations within one newspa-
per article. The other two examples show mixed behavior . The very short
range positive correlation for "seven" comes from patterns like 7x7 where x
is another digit , which relates to references to Boeing airplanes. In general ,
34 DIETRICH KLAKOW
we observed a very individual pair-correlation function for every word and

also each pair-correlation of any pair of words has its own characteristics.
We developed a method to combine generalized pair correlation func-
tions which we called log-linear interpolation[18]. It is a generalization of
an adaptation technique we proposed in [11]. Related work on maximum-
entropy models can be found in [19] and [20] . Log-linear interpolation
can be viewed as a simplified version of maximum-entropy models and is
defined by
(6) p(wlh) = Z,X~h) llpi(wlh)'xi

I
where Pi are the different component language models to be combined and

Z,X(h) is the normalization. The free parameters to be optimized are the Ai.
The component language models may be usual trigram or distance bigrams
where one word at a certain distance in the history predicts w. This would
model the same information as measured by the pair-correlation function.
Of course the component models could also be distance trigrams or even
higher order models of any pattern of skipped words.
To build a general long-range language model using log-linear interpo-
lation we propose the following scheme:
• Start with a base model like a trigram.
• Add all distance bigram models up to a defined range and opti-
mize the parameters (i.e. exponents) using a maximum likelihood
criterion.
• For all possible skip patterns look at the distance trigrams that
add information on top of the already existing model built in the
previous step . This algorithm should work like the feature selection
algorithms proposed for maximum entropy language models [21] .
• Add all selected distance trigrams to the model built so far and
optimize the exponents.
• ...repeat the previous two steps for all possible distance 4-grams,
5-grams ...
To get an impression about the potential of the above described scheme
we built a language model with an effective lO-gram context. As the base
model we used a backing-off 5-gram and combined it with distance bigrams
and distance trigrams. We have used the specialized structure
(7) p(wlh) = _1_ P5(wlh)'x5 rr(p~(Wlh--i))'x; iI(d(WlhOh_j-l))'x~

Z,X(h) i=O p(w) j=3 p(wlho)
where ho is the word immediately preceding wand all older words of the
history have negative indices. Also, A~ and A~ are the exponents of the i-th
distance-bigram and the j-th distance-trigram respectively.
In Tab . 3 we give the results for increasing context length . The results
are produced on the WSJ-corpus. The training corpus is the same as
THREE ISSUES IN MODERN LAN GUAGE MODELING 35
describ ed in the previous section but the test data is a closed vocabulary
task again from Wall Str eet Journal with a vocabulary of 5000 words . We
observe a steady improvement when increasing th e context. The bigram
and trigram are traditional backing-off models. The other models follow
t he formula (7) only th e upper bounds of t he products and t he rang e of t he
base 5-gram may be adjusted to the effective language model range . The
10-gram has a perplexity 30% lower than the trigram. The corresponding
speech recognition experiments are described in [22] .
TABLE 3
Perplexit ies for an increasing language mod el mnge.
LM-Range 2 3 4 6 10 .1
PP 112.7 60.4 50.4 45.4 43.3 .
Given the scheme described above several questions arise

• How does this scheme perform in general ?
• Is this scheme simpler th an using a grammar?
• Is it more powerful than using a grammar?
• What happens if such a model and a grammar are combined?
5. Conclusion. In conclusion we can observe th at langu age modeling
as a research topic has st ill important unresolved issues. Three of th em have
been illustrated:
• Perplexity as a quality measure may be better t han often thought.
In particular a careful correlat ion analysis shows that t here is no
indication that word-error-rate and perplexity are not correlated.
However investigations on other corpora, on other langu ages and
other groups should be done along the lines out lined.
• The smoothing of language models is not a closed topic . We
demonstrated by a simple toy language model th at even unigr ams
can be dramati cally improved by better smoothing.
• Long-range language models will lead to more consistent output of
our syst ems. We suggested a method of modeling but its relation
to grammars needs further investigation.
Acknowledgment. The author would like to thank Jochen Peters,
Roni Rosenfeld and Harry Printz for many stimulating discussions .
REFERENCES
[1] F . JELINEK, Statsitical Method 's for Speech Recognition , The MIT Press, 1997.
[2] H. SAWAF, K. SCH UTZ, AN D H. NEV, On the Use of Gr ammar Based Language Mod-
els for Sta tistical Machine Translation, Proc. 6t h Intl . Workshop on Parsing
Techn ologies, 2000, pp . 231.
[3J J . P ONTE AND W . CROFT, A Language Modeling Approach to Information Retri eval,
Research and Development in Information Ret rieval, 1998, pp . 275.
36 DIETRICH KLAKOW
[4J P. CLARKSON AND T . ROBINSON, Towards Improved Language Model Evaluation

Measures, Proc. Eurospeech, 1999, pp . 2707.
[5] A. ITO, M. KOHDA, AND M . OSTENDORF, A new Metric for stochastic Language
Model Evaluation, Proc. Eurospeech, 1999 , pp. 1591.
[6J R. IYER, M . OSTENDORF , AND M . METEER Analyzing and Predicting Language
Model Improvements, Proc. ASRU , 1997 , pp. 254.
[7J S . CHEN, D. BEEFERMAN , AND R. ROSENFELD , Evaluation metrics for language mod-
els, Proc. DARPA Broadcast news transcription and understanding workshop,
1998.
[8J H . PRINTZ AND P . OLSEN, Theory and Practice of Acoustic Confusability, Proc.
ASR, 2000 , pp. 77.
[9] D. KLAKOW AND J . PETERS, Testing the Correlation of Word Error Rate and Per-
plexity, accepted for publication in Speech Communications .
[10] LN . BRONSHTEIN AND K.A . SEMENDYAYEV, Handbook of Mathematics, Springer,
1997 .
[11] R . KNESER, J. PETERS, AND D. KLAKOW, Language Model Adaptation using Dy-
namic Marginals, Proc Eurospeech, 1997 , pp. 1971.
[12J R. KNESER AND H . NEY, Improved backing-off for m-gram language modeling,
Proc. ICASSP, 1995, pp. 181.
[13] G . ZIPF, The Psycho-biology of Language, Houghton Mifflin, Boston, 1935 .
[14] C . MANNING AND H . SCHUTZE, Foundation of Statistical Natural Language Pro-
cessing, The MIT Press, 1999.
[15J B. MANDELBROT, The Fractal Geometry of Nature , W .H . Freeman and Company,
1977.
[16] C. CHELBA AND F . JELINEK, Exploiting syntactic structure for language modeling,
Proc. of COLING-ACL, 1998 .
[17J E . CHARNIAK, Immediate-Head Parsing for Language Models, Proc. ACL, 2001 ,
pp. 124 .
[18] D . KLAKOW, Log-Linear Interpolation of Language Models, Proc. ICSLP, 1998 ,
pp. 1695 .
[19] R . ROSENFELD, A Maximum Entropy Approach to Adaptive Statistical Language
Modeling , Computer, Speech, and Language, 1996 , pp . 10.
[20] S . CHEN, K. SEYMORE, AND R. ROSENFELD, Topic Adaptation for Language Mod-
eling Using Unnormalized Exponential Models, Proc. ICASSP , 1998 , pp. 681.
[21J H. PRINTZ, Fast Computation of Maximum Entropy/Minimum Divergence Feature
Gain, Proc. ICSLP, 1998, pp. 2083.
[22] C. NEUKIRCHEN , D . KLAKOW , AND X . AUBERT Generation and Expansion of Word
Graphs using Long Span Context Information, Proc. ICASSP, 2001 , pp. 41.
STOCHASTIC ANALYSIS OF
STRUCTURED LANGUAGE MODELING
FREDERICK JELINEK'
Abstract. As previously introduced, the Structured Language Model (SLM) op-

erated with the help of a stack from which less probable sub-parse entries were purged
before further words were generated. In this article we generalize the CKY algorithm to
obtain a chart which allows the direct computation of language model probabilities thus
rendering the stacks unnecessary. An analysis of the behavior of the SLM leads to a gen-
eralizat ion of the Inside - Outside algorithm and thus to rigorous EM type re-estimation
of the SLM parameters. The derived algorithms are computationally expensive but their
demands can be mitigated by use of appropriate thresholding.
1. Introduction. The structured language model (SLM) was devel-

oped to allow a speech recognizer to assign a priori probabilities to words
and do so based on a wider past context than is available to the state-of-
the-art trigram language model. It is then not surprising that the use of
the SLM results in lower perplexities and lower error probabilities [1 , 2].1
The SLM generates a string of words wo, WI, W2, .. . , W n , wn +! , where
Wi, i = 1, ..., n are elements of a vocabulary V, Wo = < s > (the beginning
of sentence marker) and wn +! = < / s > (the end of sentence marker).
During its operation, the SLM also generates a parse consisting of a binary
tree whose nodes are marked by headwords. The headword at the apex of
the final tree is < s > . The tree structure and its headwords arise from
the operation of a SLM component called the constructor (see below).
When its performance was previously evaluated [1], the SLM's opera-
tion involved a set of stacks containing partial parses, the less probable of
the latter were being purged. The statistical parameters of that version of
the SLM were trained by a re-estimation procedure based on N -best final
parses.
This article deals with the stochastic properties of the SLM that, for
the sake of exposition, is at first somewhat simplified as compared with
the full-blown SLM presented previously [1] . This simplified SLM (SSLM)
is fully lexical - no non-terminals or part-of-speech tags are used. As a
direct consequence, the resulting parses contain no unary branches . All
simplifications are removed in Section 6 and all algorithms are extended to
the complete SLM version introduced in earlier publications [1, 2] .
Among the stochastic properties with which we will be concerned are
the following ones:
• The probability of the generated sentence based on a generalization
of the CKY algorithm [4-6].
'Center for Language and Speech Processing, Johns Hopkins University, 3400 N.
Charles se., Baltimore, MD 21218.
1 Additional results related to the original 8LM formulation can be found in the
following references: [11-19J.
37
38 FREDERICK JELINEK
• The probability of the next word given the sentence prefix.

• The probability of the most probable parse.
• A full EM style re-estimation algorithm for the statistical parame-
ters underlying the 8LM - a generalization of the Inside - Outside
algorithm [7].
• Various subsidiary probabilities necessary for the computation of
the above quantities of interest.
The algorithms of this article allow the running of the 8LM without the
use of any stacks thus, unfortunately, increasing the required computational
load. Thus we pay with complexity for rigor.
2. A simplified structured language model. The simplified struc-

tured language model (88LM) generates a string of words Wo, WI, W2, ... , W n,
Wn+l, where Wi, i = 1, ..., n are elements of a vocabulary V, Wo = < s >
(the beginning of sentence marker) and Wn+1 = < / s > (the end of sentence
marker). During its operation, the 88LM also generates a parse consisting
of a binary tree whose nodes are marked by headwords. The headword
at the apex of the final tree will be < s > . Headwords arise from the
operation of the constructor (see below).
A node of a parse tree dominates a phrase consisting of the sequence of
words associated with the leaves of the sub-tree emanating from the node.
Intuitively and ideally, the headword associated with the node should be
that word of the phrase (dominated by the node) which best represents the
entire phrase and could function as that phrase. Immediately after a word
is generated it belongs only to its own phrase whose headword is the word
itself.
The 88LM operates left-to-right, building up the parse structure in
a bottom-up manner. At any given stage of the word generation by the
88LM, the exposed headwords are those headwords of the current partial
parse which are not (yet) part of a higher phrase with a head of its own
(i.e., are not the progeny of another headword). Thus, in Figure 1, at the
time just after the word AS is generated, the exposed headwords are < s >
, SHOW, HAS, AS. We now specify precisely the operation of the 88LM. It
is based on constructor moves and predictor moves. These are specifically
arranged so that
• A full sentence is parsed by a complete binary tree
• The trigram language model is a special case of the 88LM. It
is the result of a degenerate choice of the constructor statistics:
2
Q(nulljh_ 2 = v, h_ 1 = V ) = 1 for all v, v E V .
I I
The 88LM operation is then as follows:

1. Constructor moves: The constructor looks at the pair of
right-most (last) exposed headwords, h_ 2 , h_ 1 and performs an
action a with probability Q(al h_ 2 , h_ l ) where a E {adjoin right,
2The meaning of constructor statistics Q(alh_2 , h : i) will be made clear shortly.

STOCHAST IC ANALYSIS OF ST RUC TU RED LANGUAGE MODEL ING 39
<s>
<s> A Flemish game show has as its host a
FIG. 1. Pars e by the simpl ified structured language model .
adjoin rrght", adjoin left, adjoin left", null} . The specifications

of t he five possible act ions are:3
• adjoin right: create an apex, mar k it by t he ident ity of h_ 1
and connect it by a leftward branch'' to t he (formerly) ex-
posed headword h_ 2 and by a rightward branch to the ex-
posed headword h_ 1 (i.e., t he headword h_ 1 is percolated up
by one tree level). Increase t he indices of t he current exposed
headwords h_ 3 , h_4 , . . , by 1. Th ese headwords together wit h
h_ 1 become t he new exposed headwords h~ 1 , h~ 2' h ~3' '" I.e.,
h~l = h_ 1 , and h~ i = h- i - 1 for i = 2, 3, ...
• adjoin rlght": create an apex, mark it by t he identi ty of
the word corresponding to h_ 1 , attac h to it t he marker *,5
and connect it by a leftward branch to the (formerly) ex-
posed headword h_ 2 and by a rightward branch to t he ex-
posed headword h:« (i.e., the headword h_1 is percolat ed up
by one tree level). Increase the indices of the current exposed
3T he actions adjoin right ' and adjo in left' are necessary to assure that the trigram
language model be a spec ial case of the SSLM. T his case will resu lt from a degenerate
choice of t he constructor statistics: Q( n u ll lh_2 = v, h-l = v') = 1 for all v , v' E V ,
4 Aiming down from the apex.
5I.e., if either h-l = v, or h - l = v', then the newly created apex will be marked
by h~l = u" .
headwords h_ 3 , h_ 4 , . •. by 1. These headwords together with

(h_ I )* become the new exposed headwords h~I' h~2 ' h~3 ' ...
I.e., h~I = (h_ I )* , and h~i = h- i - I for i = 2,3 , ...
• adjoin left: create an apex, mark it by the identity of h_ 2 and
connect it by a rightward branch to the (formerly) exposed
headword h_ 2 and by a leftward branch to the exposed head-
word h_ I (i.e., the headword h_ 2 is percolated one tree level
up). Increase the indices of the new apex, as well as those
of the current exposed headwords h_3 , h_ 4 , . . . by 1. These
headwords together with h_ I become the new exposed head-
words h~I,h~2,h~3' ... I.e., h~I = h_ 2, and h~i = h- i - I for
i = 2,3, ...
• adjoin left* : create an apex , mark it by the identity of the
word corresponding to h- 2 , attach to it the marker * , and
connect it by a rightward branch to the (formerly) exposed
headword h- 2 and by a leftward branch to the exposed head-
word h_ I (i.e., the headword h_ 2 is percolated one tree level
up) . Increase the indices of the new apex, as well as those
of the current exposed headwords h.:«,h_ 4 , ... by 1. These
headwords thus become the new exposed headwords h~I' h~2'
h~3' ... I.e., h~I = (h_ 2)*, h~i = h- i - I for i = 2,3, ...
• null: leave headword indexing and current parse structure as
they are and pass control to the predictor.
If a ~null, then the constructor stays in control and chooses the
next action a' E {adjoin right, adjoin right* , adjoin left, adjoin
left*, null} with probability Q(a' I h~2' h~I) where the latest (pos-
sibly newly created) headword indexation is used. If a = null, the
constructor suspends operation and the control is passed to the
predictor.
Note that a null move means that the right-most exposed head-
word will eventually be connected to the right. An adjoin move
connects the right-most exposed headword to the left.
2. Predictor moves: The predictor generates the next word Wj with
probability P( Wj = vi h_ 2 , h.: I), v E V U < / s > . The indexing
of the current headwords h-I, h_ 2 , h_ 3 , ••. is decreased by 1 and the
newly generated word becomes the right-most exposed headword
so that h~I = Wj , h~i = h- H I for i = 2,3 , .... Control is then
passed to the constructor.
The operation of the 88LM ends when the parser completes the tree,
marking its apex by the headword < s > .6
6It will turn out that the operation and statistical parameter values of the SSLM
are such that the only possible headword of a complete tree whose leaves are < 8 >
,Wl ,W2, . .. ,Wn, < /8> is < 8 > .
STOCHASTIC ANALYSIS OF STRUCTURED LANGUAGE MODELING 41
To complete the description of the operation of the parser, we have to

prescribe particular values to certain statistical par ameters:
Start of operation: Th e predictor generates the first word WI with
prob ability P1(WI = v) = P(WI = vi < s », v E V. The ini-
tial headwords (both exposed) become h- z = < s > , h_ l = WI .
Control is passed to the const ructor.
Special constructor probabilities:
• If h- l = v, v E V then
if a = null
(1) Q(alh_ z = < s >, h_ l ) = { ~ otherwise
• If h_ l = v* , v E V then
if a = adjoin left
(2) Q(alh_ z = < s >, h- d = { ~ otherwise
• If h- z = v, v E V then
if a = adjoin left"
(3) Q(alh_z ,h_ l = < [s » = { ~ oth erwise
• If h_ l = v* , v E V and h- z -# < s > then

(4) Q(alh_ z, h_ l ) = 0 for a E {adjoin right, adjoin left, null}
Th e special constructor probabilities have th e following consequences:
1. Formula (1) assures that the < s > marking the beginning of
the sentence will not be a part of the parse tre e until the tr ee is
completed.
2. Formula (2) assures th at the complet ed parse tr ee will have < s >
at tached to its apex.
3. Formulas (3) and (4) assure th at once th e end of sentence marker
< / s > is generated by th e predictor , the parse tr ee will be com-
pleted.
4. Formula (3) highlights th e last exposed headword h- z of the sen-
tence and by attaching an asterisk to it marks it for forced attach-
ment to the previous exposed headword.
5. Formula (4) forces attachment of the two last exposed headwords
h- z and h_1into a phr ase with either h- z or h_ l being percolat ed
up as the headword of the new phr ase.
The special constructor and predictor probabilities assure th at the final
parse of any word string has th e appearance of Figur e 2.
Exp erienced readers will note that the 88LM is a generalization of a
shift - redu ce parser with adjoin corresponding to reduce and predict to
shift. Th e particular non-context free nature of the 88LM is interesting
because its word generation depends on exposed headwords and th erefore
pot enti ally on the ent ire word string already generated.
<s> Wz • • • • • • • <is>
FIG. 2. Form of a com plete SLM parse.
3. Some notation. Let the generated sequence be denoted by W =

< s >, WI, W2 , ..., Wn , < / s > and let T denote a complete (bin ar y) parse
1
of W, that is, one whose sole exposed headword is < s > . Further, let
W i = < s >, WI, ..., Wi denote a prefix of the complete sente nce W , and let
T i denote a partial parse structure built by the SSLM const ru ctor on t op
of w- . Clearly, W = Wn+l and WH 1 = w -, WH 1. Fin ally, let h: j (Ti)
denote the i" exposed headword of the structure T i , j = 1,2 , ..., k where
Lk (T i ) = < s > .
We will be int erested in such quantities as P(W) , P(T , W) , p(Wi) ,
P(T i , w 'j, P(wi+lI W i) , etc.
Because of the nature of the SSLM operation specified in Section 2, the
computation of P(T , W) or P(Ti, Wi) is straight-forward. In fact, given
a "legal" parse st ru cture T i of a prefix w -, there is a unique sequence
of const ructor and predi ctor moves that results in the pair T i , w'. For
inst an ce, for the parse of Figure 1, the sub-parse T 6 corr esponding to the
prefix < s > A FLEMISH GAME SHOW HAS AS results from the following
sequence of SSLM moves: predtx) , null, pred(FLEMISH), null, pred(GAME),
null, predlsnow), adjoin right, adjoin right, adjoin right, null, pred(HAs),
null, predfAs}, null. P (Ti , Wi) is simply equal to the product of the
probabilities of the SSLM moves that result in T i , w'. More formall y,
II P(Wjlh_ 2(Tj-I) , h_ (T j - I ))
i
P(T i , Wi) = 2
j=I
(5) m (j)
.
x II Q(aj,z1TJ- , Wj, aj, I, ..., aj ,I-I)
1
1=1
where aj,m(j) = null, aj,l E {adjoin left, adjoin right}, l = 1,2, ...,
m(j) - 1 are the actions taken by the constructor after Wj (and before
wj+t) has been generated by the predictor. Tj results from Tj-1 after
actions aj,l, . .. ,aj,m(j) have been performed by the constructor, and Tj-1
is the built-up structure just before control passes to the predictor which
will generate Wj' Furthermore, in (5) the actions aj,l, ... , aj,l-l performed
in succession on the structure [Tj-1 , Wj] result in a structure having some
particular pair of last two exposed headwords h(l - 1)-2, h(l - 1)-1, and
we define
Strictly speaking, (5) applies to i < n + 1 only. It may not reflect the
"end moves" that complete the parse.
4. A chart parsing algorithm. We will now develop a chart parsing
algorithm [4-6] that will enable us to calculate P(W) when the word string
was generated by the SSLM. The results of this algorithm will facilitate
the calculation of p(Wi) and thus allow an implementation of the SSLM
language model that is alternative to the one in [1]. Furthermore, it will
lead to a Viterbi-like determination of the most probable parse
(7) T= arg max P(T, W)

T
and form the basis of a parameter re-estimation procedure that is a gener-

alization of the Inside - Outside algorithm of Baker [7].
4.1. Calculating P(W). As before, W denotes a string of words
wo, W1, W2, ... , W n, Wn+lthat form the complete sentence, where Wi, i =
1, ..., n are elements of a vocabulary V, Wo = < s > (the beginning of
sentence marker, generated with probability 1) and Wn+l = < [s > (the
end of sentence marker). The first word, W1 is generated with probability
P1(W1) = P(w11 < s », the rest with probabilities P(wilh-2,h_l) where
h_ 2 , h_ l are the most recent exposed headwords valid at the time of gener-
ation of Wi. The algorithm we will develop will be computationally complex
(see below) exactly because with different probability different headword
pairs h_ 2, h_ l determine the parser's moves, and h_ 2, h_ l can in principle
be any pair of succeeding words belonging to the sentence prefix Wi-I.
Our algorithm will proceed left-to-right." The probabilities of phrases
covering word position spans < i,j >, i E {O, 1, ... ,j} 8 will be calculated
7S0 can the famous CKY algorithm [4-6] that will turn out to be similar to ours. As
a matter of fact , it will be obvious from formula (9) below that the presented algorithm
can also be run from bottom up, just as the CKY algorithm usually is, but such a
direction would be computationally wasteful because is could not take full advantage of
the thresholding suggested in Section 8.
8The span < i,j > refers to the words Wi,Wi+l , ... ,Wj '
•
FIG. 3. Diagmm illustmting inside probability recursions .
after the corresponding information concerning spans < k, j - 1 >, k =

0,1 , ... .i - 1 and < l,j >, l = i + 1, ... .i had been determined.
We will be interested in the inside probabilities P(wf+l ,y[i,jllwi'x)
that, given that x is the last exposed headword preceding time i and that
ui; is generated, the following words wi+l
= Wi+l"" Wj are generated and
y becomes the headword of the phrase Wi ,Wi+l " " Wj.9 Figure 3 illustrates
two ways in which the described situation may arise.
In fact, the first way to generate Wi+l , ... , Wj and create a phrase span-
ning < i, j > whose headword is y , given that the headword of the preceding
phrase is x and the word Wi was generated, is:
• a string wi+ 1, . .. , WI is generated,
• a phrase spanning < i, l > is formed whose headword is y (and
preceding that phrase is another phrase whose headword is x),
• the word WI+l is generated "from" its two preceding headwords
(i.e., x , y)
i 1)
P(W{+ l' y[i ,jllwi , x) == P(WH1> ... , Wj, h(wi, WHl , ..., Wj) = ylwi , h_l(T - = x)
where h(Wi ,Wi+l, .. .,Wj) = e (empty symbol) if Wi,Wi+l , .. ., Wj do not form a phrase,
and
if y = Wi
otherwise
STOCHAST IC ANALYSIS OF ST RUCTU RED LANGUAGE MODELING 45
• t he string Wl+2, ...,Wj is generate d and t he span < l + 1, j > forms

a following phrase whose head word is, say, z (and t he headword of
its precedin g phrase must be y!)
• and finally, t he two phrases are joined as one phrase whose head -
word is y .
T he just described pr ocess can be embodied in t he formula
j-1
L L P*(wl+llx, y) P(w~+l ' y[i, lllwi, x)
l= i z
where
The second way!" to crea te a phrase whose headword is y and to

generate W i+ 1 , . . . , W j , given that the headword of the precedin g phrase is x
and t he word Wi was generated , is almost the same as t he one describ ed
above, except that t he first of t he two phrases is headed by some headword
u and t he second phrase by head word y, and when th ese two phrases are
joined it is t he second headword , y, which is percolat ed upward t o head
t he overa ll phrase. Of course , in t his case Wl+1 is generated "from" its
preced ing two headword s, x and u. T his second process is embo died in t he
formul a
j- 1
L L P*(wl+llx ,u) P(w~+l ' uri,lJlwi, X)
l= i u
X P (W{+2' y[l + l , j ]!WI+ 1,u) Q(right lu, y)
We may t hus conclude t hat for j > i , i E {O, 1, 2, ..., n }
P (w{+l ,y[i , jJ!Wi' X) =

j-1
LL P*(Wl+1I x , y) P(w~+l ' y[i, lllwi, x)
l=i z
(9) X p(wl+ 2 , z [l + l,j]lwl+b y) Q (leftjy, z )
j-1
+ LLP*(wl+llx ,u) P(w~+l ' uti, lllwi, X)
l=i u
x p(wl+ 2 , y[l + l , jllwl+l , u) Q(rightl u, y)
where
P (W{+ l> y[i,jllwi, x ) = 0
(10)
if x tj. {wo, ...,wi - d or Y tj. {Wi, ...,Wj } or i > j .
lOSee t he seco nd diagram of F igure 3.

The boundary cond itions for the recursion (9) ar e
P(W~+I ,y [i , ill wi ' x) = P(h(wi) = yIWi ,h_ 1 (T i- l ) = x) = 1

(11)
for x E {wo" " ,wi-d ,Y = Wi
and the probability we are inter ested in is given by
It may be useful to illustrate the carrying out of the above recurs ion by
a simp le example. We will create the corres ponding chart (also referred to
as the parse triangle) for the sentence fragment < s > FRESHMAN BASKET-
BALL PLAYER < / s > . To simplify t he presentation we will abbreviate the
preceding by < s > F B P < / s > . To fit into into Tab le 1 we will further
simp lify the probabilities P(W~+I' uri, lJ!Wi , x) by omitting the red undant
w~+l ' thus obtaining P(u[i ,lJlWi , x) , one of the entries in the i t h row and
Ith column of the parse tri angle. The parse tri angle of Table 1 cont ains all
the cases that the recursion (9) generates. As a further illustration, the
probability P(p[2, 3J1B ,F) would be computed as follows:
P(p[2,3J1B ,F) = Q(null IF,B) x P(p IF,B) x P(B[2 , 2J1B ,F)

x P(p[3, 3]lp,B) x Q(rig ht l H,P).
TABLE 1
Parse tri angle of the SSLM.
<s> I
Ifreshman (F) basketball (B) player (r-) I < I s>
Ip (<s > !n.oll <s> .< js» P ( <js> !n.111 < s>. < j s»
P (rlJ . ~lI r .< s»
P (Il[I .211 r ,<s»
P( r[l,lllr.<s» P(Il {1 .3I1r.<s» P ( < j s> [1,411 r ,<s»
P (r[1,211 r .<s> )
P ( r [I .:lll r ,< s»
P( f'12 ,~lIn .r )
P(n[2,2I1n,r)
P(uI2.:11ln,I.')
P (f'I:l ,:lll r ,D)
P ( I' I :l , ~lI f' , I')
P« js > IM II < js > ,r)
4.2. Comput ing P(Wi+t1Wi) . We can now use the concepts and
notation developed in Section 4.1 to compute left-to-right probabilities of
word generation by th e SSLM. Let p(Wi+I , x) denote the probability that
t he sequence Wo ,WI, W2,...,Wi,Wi+ 1 is generated in a mann er resulting in
STOCHAST IC ANA LYSIS OF STRUCTURED LANGUAGE MODELING 47
<s> WI W2 • • • • • • • • • • • • Wi _I Wi ••• • •••
F IG. 4. Diagm m illustmting the basis of the recursion (13) .
any construction 11 T i whose last exposed headword is x.t 2 Further , define

t he set of words Wi = {wQ ,WI, W2, ..., w;}. Then we have the following
recursion:
I
(13)
p (WI+I , x ) =:L :L P (Wi , y)P (w~+I , x [i ,lll wi , y)P* (wI +!I Y , x)
i= 1 yEWi-1
for x E W I
wit h t he init ial conditio n
P (WI , x ) = { PI ~WI ) x= <s >

xi=<s >.
The situation corres ponding to the genera l term in the sum (13) is depicted
in Figure 4.
For t he exa mple sente nce < s > FRES HMAN BASKETBALL PLAYER
< / s > t he probability p (W3,B) is given by t he following formula:
P (W 3 , B) = P(W 2, F) P (B[2, 21IB,F) Q(nullIF,B) P(pIF ,B)
+ P (W I , < s » P (B[1, 21IF, < s »
x Q(P(p l < s > ,B).
It follows directly from the definition of P (Wi+ I , x) that
P (wQ,WI, W2, ...,Wi, Wi+!) = :L P (Wi+ I , x)

xEW'
11 By constructio n T i we mean a sub-tree covering W i that can be generated in the

process of ope rating the SSLM.
12T hat is, T i is a construction "coveri ng" W i ::: WO ,Wl,W2 , ..., Wi, t he constructor
passes control to the predictor which then generates the next word Wi+l . Thus
"+ 1 ~ " "
P( W ' , x) ::: L P(W' ,wi+l ,h-l (T ') ::: x) .
and therefore
(14)
4.3. Finding the most probable parse T. We are next interested

in finding the most prob able parse T specified by (7). Consider any parse
T of the full sentence W. Since the pair T , W is generated in a unique
sequence of parser actions, we may label every node < i, j > 13 of th e pars e
tr ee by the appropriate headword pair x, y, where y is th e headword of th e
phrase < i, j > corresponding to the node, and x is th e preceding exposed
headword at the time the constructor created the node. Let us denote the
subtree corresponding to the node in question by V( x , y, i , j) . Now consider
any other possible subtree spanning the same interval < i, j > and having
y as its headword, and denote this subtree by V i (x, y, i, j ).J4 Clearly, if
th e prob ability of generating WHI, ...,Wj (given Wi and the structure of
T pertaining to < 1, i - I » and creating the subtr ee V i (x, y, i, j ) ex-
ceeds the corresponding probability of generating Wi+!, ...,Wj and creat ing
V( x , y, i,j) , then P(T', W) >P (T , W) where T' arises from T by replacing
in the latter the subtree V(x , y, i, j) by th e subtree Vi (x , y, i , j).
From the above observation it is now clear how to find the most prob-
able parse T. Namely, given that x is the last exposed headword corre-
sponding to W i-I and Wi is generated, let R(w{+! ,y[i ,jJlwi, X) denot e
th e probability of the most probable sequence of moves th at generate th e
following words WHI...,Wj with y becoming th e headword of the phrase
Wi, WH l .. . , Wj' Then we have for j > i , i E {O, 1,2, ..., n} th at
R(W{+l ,y[i,jJlwi,X) =
max { max [P*(wl+llx , y)R(W~+l ' y[i, lJl wi, x )

IE{ ' ,J-l} ,z
(15) x R(wl+2' z[t + l , jJlwl+l' y) Q(leftly, z )],
max [P*(WI+r1 x , u)R(W~+l ' u[i, jJlwi, x )

IE{i, j-l} ,u
X R(wl+2' y[t + l,jllwi, u) Q(rightlu, y)]}
where
R(W{+ l ,y[i,jJ!Wi, X) = 0
if x't{wo, ...,wi- r} or y't{Wi, ...,Wj} or i > j .
131n a given parse tree T , every nod e cor responds to some particular phrase span
< i,j > and is therefore uniqu ely identifi ed by it.
14Th e preceding exposed headword does not change, so it must still be x .
The boundary condit ions for th e recursion (15) are
R(w1+1'wi!i ,i]l wi, x) = P(h(Wi) = wilwi,h_1(Ti-1) = x) = 1

for x E {wo, ...,wi- d
and th e probability we are interested in will be given by
1
P(T, W) = R(w~+ ,< /8 > [1 ,n+ 111wl , < 8 » P1(Wl ).
~ ~
Obviously, th e tree '1' itself can be obt ained by a back-trace of relations

(15) st arting from the tre e apex '1'« 8 >, < /8 >, O,n + 1).
5. An EM-type re-estimation algorithm for the SSLM. We
need to derive re-estim ation formulas for the basic statistical parameters of
the SSLM: P(wlh_2 ' LI) and Q(alh_ 2, h_ 1). We will be inspired by t he
Inside - Outside algorithm of Baker [7] . We will generalize that approach
to the SSLM whose structure is considerab ly more complex than that of a
context free grammar.
5.1. Computing the "outside" probabilities. We will first derive
'formulas for P(W,x , y[i,j]), the probability th at W was produced by some
tree T th at has a phr ase spanning < i, j > whose headword is y (not
necessarily exposed) and the immediately preceding exposed headword is
x . More formally,
(16)
P(W , x , y[i, j])
~ P( wo,WI , , Wn+l, h_ 1(wo , ..., Wi-I) = x, h(Wi, ...,Wj) = y)
= P(WO ' WI, ,Wi , h_ 1(wo , ...,Wi - I) = x )
X P(Wi+l, , ui] , h(Wi , ...,Wj) = ylWi, h_ 1(wo, ...,Wi-I) = X, )
x p(Wj+l , ,Wn+llh_ 1(wo, ...,Wi - I) = x , h(Wi, ...,Wj) = y).
Now th e middle term on the right-hand side of (16) was designated by
P (w{+l ,y[i,j]lwi'X) (an "inner" prob ability) and can be computed by th e
recursion (9). We need a way to compute the product of th e outer terms
(out er probabilities) 15
P(w~, w jtt, x li - 1] ; y[i,j])

(17) ~ p(wo ,WI, ,Wi, h_ 1(wo, ...,Wi - I) = x )
XP(Wj+l , , wn+llh_ 1(wo , ...,Wi-I) = x, h(Wi, ...,Wj ) = y) .
We thus have
(18) P(W ,x, y[i ,j]) = P(wh ,wjtt ,x[i -1] ; y[i,j]) P(w{+l,y[i, jJlwi' X).
15In P(wh, w'ltl, x li - 1] ; y[i, j]) we use a semicolon rather that a vertical slash to
indicate that t his is a pr oduct of probabiliti es and not a probab ility itself. The semicolon
avoids a possible pr obl em in equat ion (18).
We will obtain a recursion for P(wb, wj:t ,xli - 1]; Y[i,j]) based on
the four cases presented in Figure 6 that illustrate what the situation of
Figure 5 (that pertains to P(wb, wj:t , xli - 1] ; y[i,j])) may lead to. The
four cases correspond to the four double sums on the right-hand side of the
following equation valid for x E Wi-1 ,y E {Wi, ..., Wj } :16
P(wh, wj:t, xli - 1] ; y[i,j])

i- l
= L L I,
[P(wh- wj:l, z[i - i - 1]; xli -i,j])
1=1 zEW i- l-l
X P(w~=f+l,x[i -i,i -11Iwi_l,z)P*(wilz,x)Q(leftlx,y)]

i-l
+ L L [P(wb-
l,wj:l,z[
i - i -1] ;y[i-i,j])
1=1 ZEWi-l-l
(19) X P(w~=f+l ,x[i -i, i - l]lwi-l, z)P*(wilz,x)Q(rightjx, y)]

n-j+l
+ L L [P(wb,wj:~+l,X[i-11;Y[i,j+m])
m=1 uEWjt;"
x P(w~t~\U[j + 1,j + m]lwj+1,y)P*(wj+1Ix,y)Q(leftly,u)]
n-j+l
+ L L [P(wh,wj:~+l' X[i-1];u[i,j+m])
m=1 uEWJtt
x p(w]t;n, u[j+1,j+mllwj+l, Y)P*(Wj+llx, y)Q(rightIY, u)]
where P* (wj+llx , y) is defined by (8). Of course,
P(wh, wjtt, xli - 1]; y[i,j]) ,;" 0

(20)
if either x ¢ W i - 1 or y ¢ {Wi, ...,Wj} .
The above recursion allows for the computation of
P(wb ,wjtt,x[i -1] ; Y[i,j]) provided the values of P(w1+l,y[i,j]/Wi'Z)
are known beforehand (they were presumably obtained using (9)), as
are the values P(wh-I,wj:l,z[i -i-1];x[i -i,j]), l = 1,2, ... , i - 1
and P(wb , wj:~+l, x[i -1] ;y[i,j + m]), m = 1,2 , ... ,n - j + 1.
In order to start the recursion process (19) we need the boundary
condition
if x=<s >, y=</s>
otherwise
which reflects the requirement, pointed out in Section 2, that the final parse
has the appearance of Figure 2.
16Below we use the set notation WI == {Wi ,Wi+l, .. .,Wj} .

• •
FIG. 5. Diagmm illustmting the parse situation p(wb , -u; x li - 1] ; y[i, j]) .
We can make a partial check on the correctness of these boundary

conditions by substituting into (18) for values i = 1, j = n + 1, x = < 8 > ,
Y =< /8> :
P(W, < 8 >, < /8 > [1, n + 1])

= P(w~ ,w~t~ , < 8 > [i -1] ; < /8 > [1,n+ 1])
x P(w~+1 , < /8 > [i, j]jWl, < 8 »
= Pl(wdP(w~+1, < /8 > [i,j]jWl, < 8 »
which agrees with (12).

For the example sent ence < 8 > FRESHMAN BASKETBALL PLAYER
< /8 > the probability P(F ; B[2,2]) is given by the following formula(in the
formula , we use a simplified not ation, replacing P(wh, witt,
x[i-I] ; y[i , j ])
by P(x;y[i, j]) which unambiguously specifies the former) :
P( F; B[2,2])
= P( < S>; B[I ,2])P(F[1 ,1]IF, < s> )Q(nullj < S>, F)
x P(BI < s>, F)Q (r ig ht jF,B)
(22) + P( < S>; F[I ,2])P(F[I ,I]IF , < S> )Q (nulll < S> ,F)
x P(BI < s> ,F)Q (le ft IF,B)
+ P(F; B[2,3])P(p[3 ,3]jp,B)Q(nullIF,B)P(pIF ,B)Q(le ft IB,p)
+ P(F; p[2 ,3])P (p[3,3]lp, B)Q(nullIF,B)P(P\F,B)Q(right\B,P).
5.2. The re-estimation formulas. We now need to use the inside

and outside probabilities derived in Sections 4.1 and 5.1 to obtain formulas
for re-estimating P(vlh_ 2 , h-d and Q(alh_ 2 , h_ 1 ) . We will do so with the
help of the following quantities found on the right-hand side of (19):
FIG. 6. Diagmms illustmting outside probability recursions .

STO CHASTIC ANALYSIS OF ST RUCT URED LANGUAGE MODELING 53
CK(x ,y, i,j,le ft) ~ P(~K)P(wi+l , Y[i, j] lwi ' X)

i- 1
(23) X LL [P(wb-l , w'l:N , z [i -l- 1] ; xli - l,j])
z 1=1
X P(W~::::+l ' x li - l, i - l]lwi-l , z) P*(Wi [z, X) Q(left lx , y)]
CK(X, y, i,j, r ight) ~ P(~K)P(W{+1 , Y[i, jJlwi' X)

i- 1
(24) X LL [P(wb- I , w'lt f , z[i -l - 1]; y[i -l ,j])
z 1=1
X P(W~:::t+ 1 ' x li -l , i - l ]lwi-l , z)P*(wilz,x)Q (rightlx, y)]
1 .
CK(X, y, i,j, nu ll) ~ P (WK )P(wi+l, y[i , j llwi' x )
n-j+ 1
X {~1=1 [P(wb,w'lt;'+l ,X[i- 1] ;y[i,j+m])
(25) X p(w;~;n , u [j + 1, j +mJl wj+l,y) P*(Wj+l!x ,y)Q(le ftly ,u)]
n-j+ 1
+ L L
u m= 1
[P(w~ ,w'lt;'+1 , x[i-11 ; u[i , j + m])
X p(w;~;n , u [j+ 1, j+ m]lwj+l , y)P*(wj+1I x , y)Q (rightly, u )] }

where t he index K refers to t he K th of t he M sentences const ituting t he
training data.!"
It is clear t hat C K (x, y, i , j , left) corres ponds to the first case depicted
in Figure 6, CK(x , y,i, j, right) to t he second case, and CK(X, y, i,j, null)
to t he last two cases of Figure 6. T hus, defining counter "contents" (nK is
t he length of t he Kth sente nce)
M nK+1nK+1
CC (x , y, left) ~ L L L CK(x,y,i,j, left)
K= 1 i=1 j=i
M nK+ 1nK+1
CC(x,y, r ight) ~ L L L CK(x ,y ,i,j, r ight)
K=1 i=1 j=i
M nK nK
CC( x , y, null) ~ L LL CK(X, y, i.i, null)
K=1 i=1 j=i
7
1 Not to complicate t he notation, we did not both er to associate the index K wit h
t he words Wj and subseq uences w~ of t he K t h sentence W K . However , the mea ning is
impli ed .
we get the re-estimates
, CC(x ,y,a)
(26) Q (a1L 2 = x,h_ 1 = y) = L a' CC(
x,y ,a
T
We can similarly use the quantities (25) for re-estimating
P(vIL 2 , h_t} . In fact, let
M nK nK
(27) CC(x, u, v) == L L L CK(x, y, i,j, null) o(v, wi+d

K=I i=1 j=i
then
, CC(x, y, v)
(28) P (vlh_ 2 = X , h_ 1 = y) = L v' CC(
x ,y,v
')'
Of course, PI (v) need not be re-estimated. It is equal to the relative fre-

quency in the M training sentences of the initial words WI (K) being equal
to v :
(29)
6. Extension of training to full structured language models.

We will now extend our results to the complete structured language model
(SLM) that has both binary and unary constructor actions [1] . It has a
more complex constructor than does the SSLM and an additional module,
the tagger. Headwords h will be replaced by heads h = (h l ,h 2 ) where hI
is a headword and h 2 is a tag or a non-terminal. Let us describe briefly
the operation of the SLM:18
• Depending on the last two exposed heads, the predictor generates
the next word Wi with probability P(wilh-2' h_t}.
• Depending on the last exposed head and on Wi, the tagger tags Wi
by a part of speech 9 E 9 with probability P(gIWi, h_t} .
- Heads shift: h~i-I = h_ i , i = 1,2, ...
- A new last exposed head is created: h~1 = (h~I ' h~l)
(Wi, g)
• The constructor operates essentially as described in Section 2 ac-
cording to a probability Q(alh_ 2 , h_t}, but with an enlarg ed ac-
tion alphabet. That is, a E {(rightIIJ) , (right*lh/) , (leftll,) ,
(left*IIJ) , (upIIJ), null} where, E I', the set of non-t erminal
symbols.
1 8 We will be brief, basing our exposition on the assumption that the reader is by now
familiar with the operation of the SSLM as described in Section 2.
STOCHAST IC ANALYSIS OF STRUCTU RED LANGUAGE MO DELIN G 55
- (r ightlb) means create an apex with downward connections

1
to h_ 2 and h_ 1. Label t he apex by h_ 1 = (h_1 , I )' Let
I
h~ i = h - i- 1, i = 2,3 , ...
- (r ight*lll) means create an apex with downward connections
to h_ 2 and h_ 1. Label t he apex by h~ 1 = (h~1'1)*' Let
h~ i = h- i- 1, i = 2,3, ...
- (leftl b) means create an apex wit h downward connections
1
to h_ 2 and h _ 1. Label t he apex by h _ 1 = (h_ 2,1) ' Let
I
h~ i = h - i- 1, i = 2,3 , .,.
- (Ieft" Ib ) means create an apex with downward connections
to h_ 2 and h_ 1. Label th e apex by h~ 1 = ( h~2'1)* ' Let
h~ i = h- i-1 , i = 2,3 , ...
- (up lb) means create an apex with a downward connection
to h_ 1 only. Label the apex by h~1 = (h~1' 1) ' Let h~i =
h_ i , i = 2 , 3, ,..
- null means pass control to the predictor.
T he operation of the 8LM ends when th e parser marks its apex by the
head < s > .19
Start of operation: The predictor generates t he first word W1 wit h
probability P1(W1 = v) = P (WI = vi < s » , v E V. Th e
tagge r t hen tags W1 by t he part of speech 9 wit h probability
P (glw1 , < s », 9 E Q. The initi al heads (bot h exposed) become
h_ 2 = « s >, < s », h_ 1 = (W l ,g). Cont rol is passed to t he
constructo r.
Restriction: For all h_ 2 , h ~ 1' 10 and j = 0, 1, ...20
(30) Q(( u plllo)lh_ 2, (h~ 1 ' Ij)) IT Q((up lll i)lh- 2, (h ~ 1 ' I i-d) = O.
i= 1
Special constructor probabilities:

• If h_ 1 = (v,I), v E V th en
if a = null
(31) Q(alh_2 = « s >, < s », h - d = {~ ot herwise.
• If h_ 1 E { (v,I) *, « / s >, < / s >)}, v E V t hen

. {I if a = (Ieftl ] < s »
(32) Q(alh_ 2 = « s >, < s », h - d = 0 o t herwise.
.
19 Formally, t his head is h = ( < s > . < s » . but we rimy sometimes omit writ ing
t he second compo nent . Simila rly, the head corres ponding to the end of sentence symbo l
is h = ( < / s > , < / s » . This, of course, mea ns that when t he tagger is called upon
to tag < / s >, it tags it wit h probability one by the part of speech < / s > .
2°I.e., up actions cannot cycle.
56 FREDERICK J ELINEK
<s>,<s>
(has, s)*
<s> Det JJ NN NN VBZ PRP PN NN Det NNP <Is>

<s> A Flemish game show has as its host a Belgian </s>
FIG. 7. Parse by the complete structured language model.
• If h- z = (v,')') , v E V then
if a = (left*II')')
(33) Q(alh_z,h_ 1 = < [s » = {~ otherwise.
• If h.i , = (v, ')')*, v E V and h_ 2 =I « 8 >,< 8 » then

(34) Q(a/h_ 2 , h-d = a for a E {right, left, null}.
Special predictor probabilities:
• If h- z =I < 8 > then
(35) P« /8 > Ih-z ,h_d = o.

Figure 7 illustrates one possible parse (the "correct" one) resulting
from the operation of the complete 8LM on the senten ce of Figur e 1.
A good way to regard the increase in complexity of th e full 8LM (com-
pared to th e original, simplified version) is to view it as an enlargement of
th e headword vocabulary. We will now adjust the recursions of Sections 4.1,
4.2 and 5.1 to reflect th e new sit uat ion.P! We will find th at certain scalar
arguments in th e preceding formulas will be replaced by th eir appropriat e
vect or counte rparts denoted in boldface.
21 Adjustment of Sect ion 4.3 is left to t he reader since it is very similar to that of
Section 4.1. In fact, th e only difference between formula (9) and (15) is th at sums in
th e former are replaced by maxim a in the latter.
We will first adjust t he equat ions (9) through (11). We get:

For j > i ,i E {0,1 , 2, ... ,n }
P (W{+l ' y [i , j ]lwi, x)

= L P (w{+t , (yl , , )[i , j llwi,x)Q ((uplly2) IX,(yl , ,))
'i'Er(x ,y )
j- l
+L LL [P*(WI+ l lx, (yl, ,)) P(W~+l ' (yl , , )[i , l]lwi, x )

(36) l=i z 'i'
X P (w{+2' z[l + 1, jllWI+ l , (yl"n Q( (leftl ly2) l(yl, ,), z)]
j- l
+ LLL [P* (WI+t1 x ,U)P(W\+l ,U[i ,l]!Wi,X )

l=i u 'i'
X P(w{+2' (yl ,, )[l + 1,j]lwl+t , u) Q((rightlly2) [u, (yl, ,n]
where
and
P (W{+l ' (yl, ,)[i, jllwi, x) = 0

(38)
if xl ¢. {Wo, "" wi- d or yl ¢. {Wi, ...,Wj} or i > j
and I'(x , y ) is an appropriate subset of t he non-terminal set r as discussed

below.
T he bounda ry conditions for the recursion (36) are
P(W~+ l ' (Wi , ,)[i, iJlWi , x) = P (h = (Wi, ,)!Wi, h_ l(T i- l) = x)

(39)
= Pblwi, x ) for xl E {Wo, ...,Wi - d
and the final probabilit y we are interested in remains
A certain subtlety must be observed in evaluat ing (36): For every pair
(x,yl) th e prob abilities P (W{+l ' (yl, ,) [i, j ]IWi , x) must be evaluated for dif-
ferent non-termin als , in a part icular order. Because of the restriction (30)
such an order exists assuring t hat t he values of P (W{+l , (yl ,, )[i, j ]l wi'X )
for, E I' (x .y) are fully known before P (w{+t ,y[i, j Jlwi,X) is computed.
T his completes t he adjust ment of the formulas of Section 4.1.
Before pro ceeding furth er we can return to t he examp le fragment
< S > FR ESHMAN BASKETB ALL PLAYER < / s > and perform for it
the operations (36) when the basic produ ction probabilities are as
given in Tab le 2 (these are realistic qua ntities obtained after training on
TABLE 2
Probabiliti es needed to compute inside-outside probabilities .
Probability I Value
P(rl« /s > , </s» ,« s> . <s > )) 7.05E-5
P(B/( <S>, <S> ),(r ,NN» I. 26E-I
P(pl ( <s> . <s> ).(B.N P) 3.22E-6
P(I'I (·"N N ),(B,NN)) 2.37E-l
P (pl(" ,NN) ,(B,NNP» 3.18E-5
P( < I s> /( <s>. <s> ),(p ,NP» 3.56E-l
P( NNlr, <s» 9.95E-l
P( NNI B,NN) 9.94E-l
P( NNPIB,NN) 5.03E-3
P( NNII',NP) 9.34E-l
P( NNPlr ,NP) 5.62E-2
P( NNlr,NN) 9.65E-l
P( NNPjl' ,NN) 3.46E-2
P(NNl r ,NNP) 6.65E-l
P(N NPl r ,NNP) 3.31E-l
P ( NULL_/(-cs», <S> ).(r ,NN) 8.59E-l
P (N UL "-I( <S> , <S> ),(B,NP ) 1.00
P( NULL_/( ,,,NN ),(B,NN» 6.64E-l
P(N LJLL_I(r ,NN) ,(B,NNP)) 7.85E-l
P(AR_NPI( r ,NN) ,(B,NN » 1.23E-l
P( AILNP/( r ,NN ),(D,NNP» 3.l1E-2
P(AR_NP'I(D ,NN) ,(I" NN)) 8.98E-l
P (AR-NP 'I( D,NNP) ,(r ,NN» 7.42E-l
P( AR _NI"I( D,NN) ,(" ,NNP» 1.24E-l
P(AIL NI"I(D,NNP) ,(p ,NNI') 2.13E-l
P(AR_NPI (Il ,NN) ,(p ,NN)) 4.93E-2
P (AILNPI( n,NNP),(r,NN» 5.38E-2
P(AR_NPI (n,NN) ,(r,NNP )) 2.82E-l
P(AR _NPI(B ,NNP ),(I',N NP)) 2.47E-l
P( A R_NPI( B,NP ),(I' ,NN)) 4.55E-l
P( AR_NPI(n ,NP ),( r ,NNP» 2.82E-2
P(A IL NPI (" ,NN) ,(r ,NP» 1.66E-2
P( AILNPI (r, NN),(p,NP' ) 8.45E-l
the U Penn Treebank [3]). Table 3 then represents the parse triangle
chart containing the inside probabilities (36) for all the relevant spans.
Finally, the following is the detailed calculation for the inside
probability P(BP,(p,NP)[l ,3]IF,« s >, < s > )) which we abbreviate as
P((p,NP)[1 ,3]1F,« s >, < s » ):
T A BLE 3 CIl
Ins id e probabili ty ta ble. '"':l
o
o
I <s> I freshman (F) I basketball (B) I player (p) I < Is> I :r:
:>
CIl
I P( ~8; . < 8> )1 < 8>. ( < / 8>. < /8») P« </8 >. </8» 1< 8> .« / 8> . < /8») '"':l
= 3 .llE-7 (3
P «F.NN)lr ..« 8> . <8» ) P « a .NP lIr.« 8> . <8» ) P«p.NP) lr.«8> . <8 ») P« </ 8>. < / 8> )l r .« 8> . < 8» )
:>
z
= 9 .95E - l = 1. 32 E -2 = 1.24E-2 = 4.4 1E-3 :>
P «a.NN lI B.(r .NN)) P «p .N P) la .(r.NN) ~
CIl
= 9 .94E - l = 9 .0 E-3 en
P ( (B.NNP)lIl.( r .N N)) P « p.NP ' )la .(r .NN» o
'lj
= 5 .0 3E - 3 = 1. 3 6 E - l CIl
'"':l
P«p .NN )lp .( Il.NP»
= 9 .3 4E - l
~
o
P«p .NNP)lp.(a .N P) '"':l
c:::
= 5 .62E - 2 ;:0
t.l:J
P« ".N N)lp·(B.NN» tl
= 9 .65E - l e-
:>
P« p.NNP )lp.( a.KN» Z
= 3 .4 6 E- 2
o
c:::
P « p.NN)lp .(B.NN P) :>
ot.l:J
= 6 .65E- l
P « p.NN PlIp ·(a .NNP ) ~
o
= 3.3 1E- l tl
t.l:J
P« < / 8> . < / 8> 1I < / 8> .(p.NP )) r-
=1 Z
o
en
<0
60 FREDERICK JE LINEK
P((p ,NP)[1,3]IF,( <s>, <S» )

= P((F ,NN)[l ,l]IF,( <s> , <s» ) XP((p ,NP)[2 ,3]IB,(F,NN))
X P(NULLJ( -cs», <S>),(F,NN))
X P(BI( <S>, <S>),(F,NN)) XP(AR_NPI( F,NN) ,(p,NP))
+ P((F ,NN)[l ,l]IF,( <s> , <s» ) XP((p ,NP') [2 ,3]IB,(F,NN))
X P(NULL_I( <s> , <s> ),(F,NN))
X P(BI( <8>, <S>),(F,NN)) XP(AR_NPI(F ,NN) ,(p,NP'))
+ P((B ,NP)[1 ,2]IF,( <8> , <S» ) XP((p ,NN )[3,3]Jp,(B,NP))
X P(NULL_I( <S>, <S>),(B,NP))
(40)
X P(pl( <S>, <S> ),(B,NP)) XP(AR_NPI(B,NP) ,(p,NN ))
+ P((B ,NP)[1 ,2]IF,( <S>, <S» ) XP((p ,NNP)[3,3]!p ,(B,NP) )
X P(NULL_I( <S>, <S>),(B,NP))
X P(pj( <S>, <8> ),(B,NP)) X P(AR_NPI(B,NP) ,(p,NNP))
= 0.995 X 9.0 X 10- 3 X 0.859 X 0.126 X 0.0166
+ 0.995 X 0.136 X 0.859 X 0.126 X 0.845
+ 0.0132 X 0.934 X 1 X 3.22 X 10- 6 X 0.455
+ 0.0132 X 0.136 X 1 X 3.22 X 10- 6 X 0.0282
= 1.24 X 10- 2
We are now in th e position to compute th e probabilities P(Wi) .22 Th e

required recursion replacing (13) is
I
P(WI+l ,x) = L L L [P(Wi,(yl ,"I))
i = l y1 E W i-l -y
(41)
X P(W~+I ' xli, l]l wi,(yl, "I)) P*(wI+ ll(y l, "I), x)]
for x l E Wi = {WI, W2, ...,WL}
with the initi al condit ion
x= < s >, < s >

x # < s >, < s > .
It then follows from (41) that
22Compare with t he resul ts of Section 4.2.

STOCHAST IC ANA LYSIS OF STRUCTURED LANGUAGE MO DE LING 61
In refining the formulas of Section 5.1 we get new recursions for th e

outer probabilities P (w~ , wj tl , x li - 1] ; Y[i , j]) involved in
(42) p eW, x , Y[i , j ]) =p(wb,wj t l , x li-I] ; y [i, j ]) P(w{+1> y[i ,jJlWi, x ).
For x l E W i- l and yl E {Wi , ...,Wj }, the required formulas are
p (wb,w j tl , x li - 1] ; Y[i , j])

= L p (wb,wj t l ,x[i-1] ; (yl , , )[i ,j ])Q((up lb ) lx ,y)
"'Y E ~ ( x , y )
L L[P(wb-l , -n: z[i -1-1] ; (x l ,,)[i -l , j ])

i- l
+L
1= 1 z "'Y
X P(wt:J+1' x[i-l , i -I]\wi_l , z) P* (wilz , x) Q((leftl b) [x, y)]

i- l
+ LL L [p(w~-I , w jt11, z[i -I - 1]; (y1 , , )[i - I, j ])
(43) 1= 1 z "'Y
X P(W~=t+1 ' x li - I, i- 11Iwi- l, z)P* (wi lz, x )Q((right lb ) [x, y )]

n-j+1
+ L LL[P(wb ,wjt~+l ,X[i-I] ; (y l , ,)[i, j + m])
m= l u "'Y
x p (W}t ;rt, u [j +1 ,j + m]lwJ+l ,y)P*(wJ+t1x,y)Q((left ll, ) Iy, u )]

n-j+l
+ L LL 1
[P(wb, wjt~+l ,x[i -1]; (u ,,)[i,j + m])
m =l u "y
X P (W}t ;' , u [j +1 , j + m]lwJ+1,y )P*(wJ+l lx , y )Q((right ll, ) Iy, u)]

where P*(Wj+l jX, y ) is defined by (37). If either xl ¢ Wi-l or yl ¢
{Wi , ...,Wj } then we define23
(44) i ,wjn+l
P (WO +l ,x [.z - 1]', yz,
[. ].]) -. .:. . 0.
Again , th e prob abilities (43) for (x, yl ) must be evaluated in an appro-

priat e order to assur e th at P(w~ , w jtl , xli-I] ; (yl , , )[i, j ]) for , E ~(x, y)
is known. This order is act ually the reverse order implied by rex, y) and
can be determined because th e restriction (30) applies.
In order to use (43) we need the bound ary conditions
P(w5, w~t~ , x[O] ; Y[I, n + 1])

(45) = { P l (wd if x = < s >, < s > and y = < [s >, < [s >
o ot herwise.
23T his definit ion is made use of in the summations Lz and Lu in (43) and else-
where, thus simplifying not at ion. Consequ ently Lz could also have been written
LzIEWi- I-1 L OE r , etc .
We now return to the example fragment < 8 > FRESHMAN BAS-

KETBALL PLAYER < /8 > for the last tim e and perform for it the op-
erations (43). Table 4 th en represents th e parse tri angle chart cont ain-
ing the outside probabilities (36) for all the relevant spans. The fol-
lowing is th e detailed calculation for the outsid e probability P( < S > FB,
P < /8 >, (F, N N )[1]; (B, N N )[2, 2]) which we abbreviate as P((F,NN) ;
(B, N N)[2, 2]):
P((F ,NN) ; (B,NN)[2,2])

= P((F,NN) ; (p,NP)[2 ,3]) x P((p,NN)[3,3ll p,(B,NN))
x P(NULL_I(F,NN),(B ,NN)) x P(pl(F,NN) ,(B,NN))
x P(AR_NPI(B ,NN) ,(p,NN))
+ P((F,NN) ; (p,NP ')[2 ,3]) x P((p ,NN)[3 ,3llp,(B,NN))
x P(NULL_I(F,NN),(B,NN)) x P(pl(F,NN) ,(B,NN))
x P(AR_NP'I(B ,NN),(p,NN))
+ P((F ,NN) ; (p,NP)[2 ,3]) x P((p,NNP)[3 ,3J1p ,(B,NN))
x P(NULL_I(F,NN) ,(B,NN)) x P(pl(F ,NN) ,(B,NN))
x P(AR_NPI(B ,NN) ,(p,NNP))
+ P((F ,NN) ; (p,NP')[2,3]) x P((p,NNP)[3,31Ip,(B,NN))
x P(NULL!(F,NN) ,(B,NN)) x P(p!(F ,NN) ,(B ,NN))
(46) x P(AR_NP 'I(B,NN) ,(p,NNP))
+ P( ( < S>, < S» ; (B ,NP)[1 ,2])
x P((F ,NN)[l,lJ1F ,(< S>, < S» )
x P(NULL_I( -cs» , < S> ),(F,NN))
x P(BI( <S>, < S> ),(F,NN))
x P(AR_NP\(F ,NN) ,(B,NN))
= 4.49 x 10- 8 x 0.965 x 0.664 x 0.237 x 0.0493
+ 2.28 x 10- 6 x 0.965 x 0.664 x 0.237 x 0.898

+ 4.49 x 10- 8 x 0.0346 x 0.664 x 0.237 x 0.282
+ 2.28 x 10- 6 x 0.0346 x 0.664 x 0.237 x 0.124
+ 3.45 x 10- 11 x 0.995 x 0.859 x 0.126 x 0.123
= 3.13 x 10- 7 .
Finally, to obt ain formulas for re-estimating P(vlh _2 , h _ 1 ) and

Q(alh_ 1 , h_ 2 ) we proceed as follows:
TA BLE 4 en
Outside pro bability ta ble. '"'3
o
o
I <s> I freshman (F) I bas ketball (B) I player (p) I <Is> I ::r:
;p-
en
I P« </ s>. </ S» :«S>. <S») P« </ s>. < / s»: « / s>. < Is» ~ ) '"'3
= 3 . 11 E- 7 = 1 o;p-
P « -csc-. <s> ): ( r .N N )) P « -cs» . <S> ) :( B.N P» P « -cs», <s> ) :( p.N P)) P « -cs» . -cs» l:( <Is> . <I s> l) z
;p-
= 3 . 11E - 7 = 3 . 45 E- 11 = 2 .5 1E -5 = 7 .05E - 5
P « F.N N ) :( B.N N » P «r.N N ) :( p.NP )) ~
en
= 3 . 13E - 7 = 4 .4 9 E - 8 en
P « r .N N ) :( B.N N P )) P «r.N N ) :(p. NP ' )) o
"%j
= 3 . 23E- 11 = 2 .28 E - 6 en
'"'3
P «B .NP) :(p .N N »
= 4 .8 5 E - 13
~
o
P «B .NP) :( p.N NP» '"'3
C
= 3 .0 1E- 14 g;
P «B .NN ) :(p .N N » t:'
= 3 .2 1E- 7 t-<
;p-
P « B.N N ) :(p .K N P )) Z
= 4 .6 2E- 8
o
c
;p-
P « B.N K P ) :(p .N N ))
O
= 2 . 13 E - 13 t':l
P « B.N N P ) :( p.K N P )) ~
o
= 6 .24E - 14 t:'
t':l
P « p.NP):( <Is> . </s» ) t-<
= 3 . 11 E- 7 Z
o
O'l
W
i-I
(47) x LL [P(W~-I,wjtl,z[i-l-l];(Xl,"f)[i-l,j])
z 1=1
i-I
(48) xL L [p(W~-I, wjtl, z[i - l - 1]; (yl, "f)[i -l,j])
z 1=1
CK(X, y , i ,j, (uplb)) ~ P(~K) P(wi+l' y[i,jllwi' x)

(49)
X L P(W~, wjtl ,x[i -1] ; (yl , f')[i,j])Q((uplb) [x.y]
"Y E A (x ,y)
CK(x,y,i,j,null) ~ P(~K)P(W{+I,Y[i,j]lwi'X)
n-j+l
X {L L L[P(w~,wjt~+I,X[i-l];(yl '''f)[i,j+m])
u m=1 "y
(50) X p(W}t;n ,u[j+ l,j+m]lwj+l , y)P*(wj+dx, y)Q((leftlb) Iy, u)]

n-j+l
+L L L [P(w& ,wjt~+I,X[i -1] ;(u 1 ,"f)[i,j +m])
u m=1 "y
Thus, defining counter "contents" (nK is the length of the K t h

sentence)
STOCHAST IC ANALYSIS OF ST RUCTU RED LANGUAGE MODE LING 65
M nK+ lnK + l
CC( X, y , (left lh)) ~ L L L CK(x , y , i, j, (le ftlh'))
K= l i= 1 j=i
M nK + l nK+l
CC(x , y , (r ight IIT)) ~ L L L CK(X, y , i, j, (r ight IIT))
K =1 i= 1 j=i
M nK + l n K + l
CC(x, y ,(u pll,)) ~ L L L CK(x, y ,i,j,( u pll ,))
K=l i= 1 j=i
M n« nK
CC(x, y, null) ~ L L L CK(x ,y,i , j ,null)
K= 1 i= 1 j = i
we get t he re-estim at es
, CC (x ,y,a)
(51) Q (alh_ 2 = x , h_ 1 = y ) = 2:= a' CC( x ,y, a ')
We can similarly use t he quantities (47) t hrough (50) for re-est imating
P(vlh_ 2 , h- d. In fact , let
M n K nK
CC( x,y, v) ~ L L L CK(X,y, i, j ,null) t5(v,wj+t}
K =1 i = 1 j = i
t hen
, CC( x , y , v)
(52) P (v lh_2 = X , h_ 1 = y ) = ~
L.J v'
CC(
x,y ,v
')
As before, PI (v) need not be re-estimated:

1
P1 (v) = M L t5(v ,wl(K)) .
M
K=l
Fina lly, t he re-esti mation of tagger probabilit ies is given by the for-
mula 24
CC (x, (y,g))
(53) P(g ly, h _ 1 = x) = 2:= g'EQ CC(x ,(y,g '))
wher e
CC( x , y ) = CC( x , y , null )
+ L [CC(x , y , (leftl l,)) + CC(x, y , (r ight ll,))]

"fE r
24 Note t hat a headword y can be t agged by a part of sp eech 9 only if t he phrase

whose head is y consists of th e sing le word y .
7. The problem of complexity. The recursion formulas (9) and

(19) for the 88LM are very computing intensive, and the formulas (36) and
(43) even more so. Referring to (9), the < i, j > element of the "inside"
chart need in general contain i x (j - i + 1) entries , one for each permissible
headword pair x, y. The more complex chart for (36) has for each word
pair xl, yl which appears in it as many as K L entries where K and L are
the numbers of different non-terminals that the headwords Xl and yl can
represent, respectively.
The question then is: what shortcuts can we take? The following
observation shows that the 88LM by itself would not produce adequate
parses:
Consider the parse of Figure 7. On the third level the headword pair
HAS, AS forms a phrase having the headword HAS. But we would not have
wanted to join HAS with AS on the first level, that is, prematurely! What
prevents this joining in the 8LM are the parts of speech VBZ and PRP by
which the tagger had tagged HAS and AS, respectively. At the same time,
the joining of HAS with AS is facilitated on the third level by their respective
attached non-terminals VBZ and PP.
80 if we wished to simplify, we could perhaps get away with the
parametrization
while the tagger distribution would continue to be given by P(glw, x).

Alas, such a simplification would not materially reduce the computing effort
required to carry out the recursions (36) and (43).
It is worth noting that from the point of view of sparseness of data, we
could in principle be able to estimate constructor and tagger probabilities
having the parametric forms Q(alh_ 3 , h_ 2 , h-d and P(glw, h_ 2 , h_ 1 ) . In-
deed, P(glw, h_ 2 , h-d has the memory range involved in standard HMM
tagging, and Q(alh_ 3 , h_ 2 , h_ 1 ) would enhance the power of the construc-
tor by moving it decisively beyond context freedom. Unfortunately, it
follows from the recursions (36) and (43) that the computational price for
accommodating this adjustment would be intolerable.
8. Shortcuts in the computation of the recursion algorithms.
It is the nature of the 8LM that a phrase spanning < i, j > can have, with
positive probability, as its headword any of the words {Wi, ...,Wj}. As a
result, analysis of the chart parsing algorithm reveals its complexity to be
proportional to n 6 . This would make all the algorithms of this article im-
practical, unless schemes can be devised that would purge from the charts
a substantial fraction of their entries .
8.L Thresholding in the computation of inside probabilities.
Note first that the product P(Wi,x)P(w{+l,y[i,jJ/Wi'X) denotes the
probability that Wi+j is generated, the last exposed head of Wi-l is x,
and that the span < i,j > is a phrase whose head is y.
STOCHAST IC ANALYSIS OF STRUCTURED LANGUAGE MODE LING 67
Observe next 25 t hat for a fixed span < i,j > the products P(Wi,x)
P(W{+l' y [i , jllwi, x ) are compara ble to each other regardless of t he iden-
t ity of x l E {wo,Wl, ..., wi- d and yl E {Wi, ...,Wj}. They can thus be
t hresholded with respect to maxv, P (W i , v ) P (w{+l ' z [i , j !lWi' v ). Th at
°
is, for further computation of inside probabilities P (W{+l' y [i , j llwi, x ) can
be set to i£2 6
P(W i , x) P(w{+l' y [i , j!lwi' x)

(54) i , v ) P (W{+l' z[i , jl lwi' v).
« maxP(W
v .z
Of course, it must be kept in mind th at , as always, thresholding is only

an opportunistic device: Th e fact that (54) holds does not mean with
probability 1 that P(w{+l,y[i,jllwi'x) will not become useful in some
highly probable parse. For inst ance, P(wj+2,Z[j + 1,kll wj+l ,y) may be
very large and thus compensate for the relatively small value of
P (Wi ,x)P(w{+l ,y[i ,jllwi'x ). Thus the head y might be "needed" to
complete t he parse th at corresponds to th e prob ability P (W ~l ' z[i , kJl wi, x ).
Next not e t hat if P(Wi, y ) « max, P (Wi, z) t hen it is unlikely t hat
a high prob ability parse will account for th e interval < 0, i - I > wit h a
sub-parse whose last exposed head is y . In such a case then t he calculat ion
of P (w{+l ' x [i , j Jlwi' y) , j E {i + 1, ..., n + I} will probably not be needed
(for any x ) because the sub-parse corresponding to P(w{+l' x [i, j llwi, y ) is
a cont inuation of sub-parses whose total probability is very low. Again, t he
fact t hat P(W i , y ) is small does not mean t hat th e head y cannot become
useful in producing t he future. I.e., it is st ill possible (though unlikely)
that for some x and i . P(w{+l,x[i, j Jlwi'y ) will be so large that at least
some parses over t he interval < 0, j > t hat have y as t he last exposed head
at time i - I will have a substantial probability mass.
So, if we are willing to take th e risk th at t hresholding involves, th e
probabilities (36) and (41) should be computed as follows:
1. Once P(Wi+l,x) and P (w~+l ,y[j, iJl wj ,x) , j = O,l , ... ,i, i =
0,1 , ..., l are known for all allowed values of x and y ,27 probabiliti es
P(w~~\ ,z[k ,l + 111Wk , v ) are computed in the sequence k = l +
°
1, l , ..., for each allowed z and th ose values of v th at have not
been previously zeroed out .28
25Aga in, anyt hing relating to the qua nt ities P (W{+I ' Y[i, jJlWi , x) applies equa lly to
the qu an t it ies R (w {+l ,y[i , j llwi'x ).
26T he author is indebted to Mark Jo hnson who pointed out t his improvement of t he
thresholding regime.
27Allowed are x = (x l, , ) and y = (y l , ,) where x l E {wo, ..., W j _ i ] and yl E
{ Wj , .. . ,w;} .
28Z eroing is carried out in step 4 that follows.
2. For each span < k, l + 1 > just computed, set to 0 all probabilities
P(w~:t.\,y[k,l + 11Iwk ,x) satisfying
p(Wk,x)p(W~:t.ll'y[k ,l + 11Iwk,x)
« maxP(Wt,
v,z
. l+l
v) P(wk+l' v[k, l + IJlwk ' z) .
3. Use equation (41) to compute P(W1+l , x) for the various heads x .

4. Zero out all prob abilities P(w7+2 ,Z[l + l,kllwl+l,v), k = l +
2, l + 3, ..., n + 1 for all allowed heads v such that P(Wl+ I , v) «
max; p(Wl+l , x) . The zeroed out probabilities will not be com-
puted in the futur e when th e tim e comes to compute th e corre-
sponding positions in the chart.
Obviously, the thresholds implied in st eps 2 and 4 above must be
selected experimentally. Using them will short-cut th e computation process
while carrying the danger th at occasional desirabl e parses will be discarded.
8 .2 . Thresholding in computation of outside probabilities.
Having taken care of limiting the amount of comput ation for (36) and
(41), let us consider the recursion (43). It is clear from equation (42)
th at P(wb, wjtl, xli - IJ ;y[i,j]) will not be needed in any re-estimati on
formula if P(w{+l' y[j, illwj , x) = O. This can also be seen from th e counte r
contribut ions (47) through (50).
However, we must check whether P(WQ ,wjtll ,X[i -1] ;y[i ,j]) may
not have to be computed because it might be needed on the right-hand
side of th e recurrence (43). Fortunately such is not th e case. In fact , if
P(W, x , y[i, j]) = 0 then the event
W is generated , y is a head of the span < i,j >, and x is
th e preceding exposed headword
just cannot arise, so this situ ation is entirely analogous to the one where
either Xl ~ Wi-lor yl ~ {Wi, ...,Wj}' Consequently, if P(W , x, y[i,j]) = 0
th en th e reduction 29 implied by P(wb,wjtl , xli -IJ ;y[i, j]) is illegitim at e
and cannot be used in any formula comput ing oth er probabilities.
The conclusion of the preceding par agraph is valid if
P(w {+I ,y[j,i]lwj ,x) = 0, but we want to cut down on computati on by
coming to th e same conclusion even if all we know is th at
P(W{+I' y[j, iJlwj, x) ~ O. Thus we are advocating here the not always
valid 30 approximation of setting P(wb, wjtl ,x[i - 1];y [i,j]) to 0 when-
ever our previous thresholding already set P(W{+I' y[j , i]lwj, x) to O.
A final saving in comput ation may be obt ained by tr acing
back th e chart corresponding to the inside probabilities. Th at is, st art-
ing with th e chart contents for the span < 1, n > (corresponding to
2 9 We ar e usin g here th e terminology of sh ift - reduce parsing.

3 0 For th e reasons given in t he preceding sub-section.
P(w2',y[1,nllw1' < s » for various values of y) find and mark the sub-
parse pairs P(w~,x[1,j]lw1' < s », P(wJ+! ,z[j + 1,nllwH1,x) that re-
sulted in P(w2' , yjl , n]lw1, < s » . Perform this recursively. When the
process is completed, eliminate from the chart (i.e., set to 0) all the sub-
parses that are not marked . Computations P(w&, wj:f, xli - 1] ; y[i, j])
will thus be performed only for those positions which remain in the chart.
8.3. Limiting non-terminal productions. The straight-forward
way of training the Structured Language Model is to initialize the statistics
from reliable parses taken either from an appropriate treebank [3], or from
a corpus parsed by some automatic parser [8-10]. This is what was done
in previous work on the SLM [1, 2].
If this initialization is based on a sufficiently large corpus, then it is
reasonable to assume that all allowable reductions 1'1, 1'2 -+ I' and 1'0 -+ I'
have taken place in it. This can be used to limit the effort arising from the
sums 2:1' appearing in (36) and (43).
If we assume that the initialization corpus is identical to the training
corpus then the problem of out-of-vocabulary words does not arise. Never-
theless, we need to smooth the initial statistics. This can be accomplished
by letting words have all part of speech tags the dictionary allows, and
by assigning a positive probability only to reductions (xL 1'1), (x~, 1'2) -+
(xL 1') that correspond to reductions 1'1,1'2 -+ I' that were actually found
in the initialization corpus.
9. Smoothing of statistical parameters. Just as in a trigram lan-
guage model, the parameter values extracted in training will suffer from
sparseness. In order to use them on test data, they will have to be sub-
jected to smoothing. Let us, for instance, consider the predictor in the
SSLM setting. The re-estimation formulas specify its values in equation
(28) which we repeat in a slightly altered form
CC(x, y, v)
(55) f( VIh -2=X, h -l=Y ) = " , CC( ')
LJv' X,Y,V
with the function CC defined in (27). The value of f(vlL 2 = x, h_ 1 = y)

of interest is the one obtained in the last iteration of the re-estimation
algorithm.
Assuming linear interpolation, the probability used for test purposes
would be given by
1\(vlh_ 2 = x, h_ 1 = y)
(56)
= A f(vlh_ 2 = x, h_ 1 = y) + (1 - A) P(vlh_ 1 = y)
where P(v\h_ 1 = y) denotes a bigram probability smoothed according
to the same principles being described here. The value of A in (56) is
a function of the "bucket" that it belongs to. Buckets would normally
depend on counts, and the appropriate count would be equal to CC(x, y) ==

L:v' CC(x , y, v') obtained during training.
Unfortunately, there is a potential problem. The counts CC(x, y)
are an accumulation of fractional counts which have a different character
from what we are used to in trigram language modeling. In the latter,
counts represent the number of times a situation actually arose. Here the
pair x , y may represent a combination of exposed headwords that is totally
unreasonable from a parsing point of view. From every sentence in which
the pair appears it will then contribute a small value to the total count
CC(x, y). Nevertheless, there may be many sentences in which the word
pair x, y does appear. So the eventual count CC(x , y) may end up to
be respectable. At the same time, there may exist pairs x' ,y' that are
appropriate heads which appear in few sentences and as a result the count
CC(x', y') may fall into the same bucket as does CC(x , y). But we surely
want to use different X's for the two situations!
Th e appropriate solution may be obtained by noticing that the count
can be thought of as made up of two factors: the number of times, M, the
x , y pair could conceivably be headwords (roughly equal to the number of
sentences in which they appear) , and the probability that if they could be
headwords, they actually are. So therefore
Now X's must be estimated by running the re-estimation algorithm on

heldout data. Assuming that the headword pair x , y belongs to the k t h
bucket''! and that CCH denotes the CC value extracted from the heldout
set ,32 the contribution to the new value A'(k) due to the triplet x, y, v
will be33
A(k) f(vlh_ 2 = x, h_ l = y)
CCHX,y,V
()
y) + (1 - A(k)) P(vlh_ l
A
A(k) f(vlh_ 2 = x, h_ l = = y)
where A(k) denotes the previous A value for that bucket .

Generalization of this smoothing to SLM is straight forward, as is the
specification of smoothing of constructor and tagger parameters.
Acknowledgement. The author wishes to thank Peng Xu, who con-

structed the tables presented in this paper and carried out the necessary
computations. Mr. Xu took care of the formatting and held invaluable
discussions with the author concerning the SLM.
31Even though the buckets are two-dimensional, they can be numbered in sequence.
32Values depend on the probabilities i\(vlh_2 = X,h-l = y) and QA(alh-2 =
x, h_ 1 = y) which change with each iteration as the values of the >.-parameters change.
33We assume it erati ve re-estimation.
REFERENCES
[1] C . CHELBA AND F . JELINEK , "Structured Language Modeling," Computer Speech

and Language, Vol. 14, No.4, October 2000.
[2] C. CHELBA AND F . JELINEK, "Exploit ing Syntactic Structure for Language Model-
ing," Proceedings of COLING - ACL, Vol. 1 , pp . 225 - 231, Montreal, Canada,
August 10-14 , 1998.
[3] M. MARCUS AND B. SANTORINI , "Building a Large Annotated Corpus of English:
the Penn Treebank," Computational Linguistics, Vol. 19 , No.2, pp . 313-330,
June 1993.
[4] J . COCKE, unpublished notes.
[5] T . KASAMI , "An efficient recognition and syntax algorithm for context-free lan-
guages ," Scientific Report A FCRL-65-758, Air Force Cambridge Research
Lab. , Bedford MA, 1965.
[6] D .H. YOUNGER, "Recognition and Parsing of Context Free Languages in Time
N3, " Information and Control , Vol. 10, pp . 198-208, 1967.
[7] J .K . BAKER, "Trainable Grammars for Speech Recognition," Proceedings of the
Spring Conference of the Acoustical Society of America, pp . 547-550, Boston
MA,1979.
[8] A. RATNAPARKHI, "A Linear Observed Time Statistical Parser Based on Maximum
Entropy Models ," Proceedings of the Second Conference on Empirical Methods
in Natural Language Processing, pp . 1-10, Providence, RI, 1997.
[9] E . CHARNIAK, "Tr eebank Grammars," Proceedings of the Thirteenth National
Conference on Artificial Intelligence, pp . 1031-1036, Menlo Park, CA , 1996.
[10] M.J. COLLINS, "A New Statistical Parser Based on Bigram Lexical Dependencies,"
Proceedings of the 34th Annual Meeting of the Associations for Computational
Linguistics, pp . 184-191 , Santa Cruz, CA, 1996.
[11] C . CHELBA , "A Structured Language Model," Proceedings of ACL/EACL'97 Stu-
dent Session, pp. 498-500, Madrid, Spain, 1997.
[12] C. CHELBA AND F . JELINEK , "Refinement of a Structured Language Model," Pro-
ceedings of ICAPR-98, pp . 225-231, Plymouth, England, 1998
[13] C. CHELBA AND F . JELINEK , "St ruct ured Language Modeling for Speech Recogni-
tion," Proceedings of NLDB99, Klagenfurt, Austria, 1999
[14] C. CHELBA AND F . JELINEK , "Recognit ion Performance of a Structured Language
Model, " Proceedings of Eurospeech'99, Vol. 4 , pp . 1567-1570 , Budapest, Hun-
gary, 1999.
[15] F. JELINEK AND C. CHELBA , "Putting Language into Language Modeling," Pro-
ceedings of Eurospeech '99, Vol. 1 , pp . KN-I-6, Budapest, Hungary, 1999.
[16] C. CHELBA AND P. Xu , "Richer Syntactic Dependencies for Structured Language
Modeling," Proceedings of the Automatic Speech Recognition and Understand-
ing Workshop , Madonna di Campiglio, Italy, 2001.
[17] P . Xu, C. CHELBA, AND F . JELINEK, "A Study on Richer Syntactic Dependen-
cies for Structured Language Mod eling," Proceedings of ACL'02, pp . 191-198 ,
Philadelphia, 2002.
[18] D.H . VAN UVSTEL , D. VAN COMPERNOLLE, AND P . WAMBACQ , "Naximum-
Likelihood Training of the PLCG-Based Language Model," Proceedings of
the Automatic Speech Recognition and Understanding Workshop, Madonna
di Campiglio, Italy, 2001.
[19] D .H. VAN UVSTEL, F . VAN AELTEN , AND D. VAN COMPERNOLLE, "A Structured
Language Model Based on Context-Sensitive Probabilistic Left-Corner Pars-
ing," Proceedings of 2nd Meeting of the North American Chapter of the ACL,
pp. 223-230, Pittsburgh, 2001.
LATENT SEMANTIC LANGUAGE MODELING FOR
SPEECH RECOGNITION
J EROME R . BELLEGARDA'
Abstract. St at istical language models used in lar ge voca bulary speech recognition
must properly capt ure th e vario us constraints, both local and global, present in the lan-
guage. Wh ile n-gram modeling readily accounts for the former , it has been more difficult
to handle t he latter , and in par t icular long-t erm semantic dep end encies, within a suit able
data-driven formalism. This pap er focuses on the use of latent semantic analysis (LSA)
for this purpose. The LSA paradigm auto matic ally uncovers meaningful associations in
the lan guage based on word-docum ent co-occurrences in a given corpus. The resulting
sema nt ic knowledge is encapsulat ed in a (cont inuous) vector space of compa rat ively low
dimension, where ar e mapped all (discrete) words and documents considered . Compar-
ison in this space is done through a simple similarity measure, so famili ar clustering
tec hniques can be applied. This leads to a powerful fram ework for both automatic se-
mantic classification and semantic language modeling. In the latter case, th e large-span
nature of LSA models makes them particularly well suited to complement convent ional
n-grams. This synergy can be harnessed through an integrative formulation, in which
lat ent semantic knowledge is exploited to judiciously adjust the usual n-gram probabil-
ity. The paper concludes with a discussion of intrinsic trade-offs, such as t he influenc e
of t raining data selection on th e result ing performance enhancement.
Key words. St atist ical language modeling, multi-span integra tion , n-grarns , latent
sema nt ic an alysis, speech recognition.
1. Introduction. Th e well-known Bayesian formulation of automatic

speech recognition requires a prior model of th e language, as pert ains to th e
domain of interest [34,491 . T he role of t his prior is to quantify which word
sequences are acceptable in a given language for a given tas k, and which
are not: it must th erefore encapsulate as much as possible of the syntac tic,
semant ic, and pragmatic characteristics of t he domain. In t he past two
decades, st at ist ical n-gram modeling has steadily emerged as a practical
way to do so in a wide range of applications [15] . In this approach, each
word is predicted conditioned on th e current context, on a left to right basis.
An comprehensive overview of the subject can be found in [52], including
an insightful perspective on n-grams in light of other techniques, and an
excellent tutorial on relat ed tr ade-offs. Prominent among the challenges
faced by n-gram modeling is the inherent locality of its scope, as is evident
from the limited amount of context available for predicting each word.
1.1. Scope locality. Central to this problem is th e choice of n , which
has implications in terms of predictive power and par ameter reliability.
Although larger values of n would be desirable for more predictive power ,
in practice , reliable estimation demands low values of n (see, for example,
[38,45 ,46]) . This in turn imposes an art ificially local horizon to th e model,
impeding its ability to capt ure large-span relation ships in the langu age.
'Spoken Language Group , Apple Comput er Inc., Cupert ino, CA 95014.

73
74 JEROME R. BELLEGARDA
To illustrate, consider, in each of the two equivalent phrases:
(1.1) stocks fell sharply as a result of the announcement

(1.2) stocks, as a result of the announcement, sharply fell
the problem of predicting the word "fell" from the word "stocks." In (1.1),
the prediction can be done with the help of a bigram language model
(n = 2). This is straightforward with the kind of resources currently
available [50]. In (1.2), however, the value n = 9 would be necessary, a
rather unrealistic proposition at the present time. In large part because of
this inability to reliably capture large-span behavior, the performance of
conventional n-gram technology has essentially reached a plateau [52].
This observation has sparked interest in a variety of research direc-
tions, mostly relying on either information aggregation or span extension
[5]. Information aggregation increases the reliability of the parameter esti-
mation by taking advantage of exemplars of other words that behave "like"
this word in the particular context considered. The trade-off, typically, is
higher robustness at the expense of a loss in resolution . This paper is
more closely aligned with span extension, which extends and/or comple-
ments the n-gram paradigm with information extracted from large-span
units (i.e., comprising a large number of words). The trade-off here is in
the choice of units considered, which has a direct effect on the type of long
distance dependencies modeled. These units tend to be either syntactic or
semantic in nature. We now expand on these two choices.
1.2. Syntactically-driven span extension. Assuming a suitable
parser is available for the domain considered, syntactic information can be
used to incorporate large-span constraints into the recognition . How these
constraints are incorporated varies from estimating n-gram probabilities
from grammar-generated data [61] to computing a linear interpolation of
the two models [36] . Most recently, syntactic information has been used
specifically to determine equivalence classes on the n-gram history, resulting
in so-called dependency language models [13, 48], sometimes also referred
to as structured language models [14, 35, 57].
In that framework, each unit is in the form of the headword of the
phrase spanned by the associated parse sub-tree. The standard n-gram
language model is then modified to operate given the last (n -1) headwords
as opposed to the last (n - 1) words. Said another way, the structure of
the model is no longer pre-determined: which words serve as predictors
depends on the dependency graph, which is a hidden variable [52] . In the
example above, the top two headwords in the dependency graph would be
"stocks" and "fell" in both cases, thereby solving the problem .
The main caveat in such modeling is the reliance on the parser, and
particularly the implicit assumption that the correct parse will in fact be as-
signed a high probability [60]. The basic framework was recently extended
LATENT SEMANTIC LANGUAGE MODELING 75
to operat e efficiently in a left-to-right manner [14, 35], through careful op-

t imization of both chart par sing [58] and search modules. Also noteworthy
is a somewhat complement ary line of research [59], which exploits the syn-
tactic st ruct ure contained in th e sentences prior to the one featuring th e
word being predicted.
1.3. Semantically-driven span extension. High level semantic in-

formation can also be used to incorporate large-span const raints into the
recognition. Since by nature such information is diffused across t he en-
tir e text being created, this requires t he definition of a document as a
semant ically homogeneous set of sentences. Then each document can be
cha racterized by drawing from a (possibly large) set of topi cs, usually pre-
defined from a hand-labelled hierarchy, which covers th e relevant semantic
domain [33, 54, 55]. The main uncertainty in this approach is the granu-
larity required in the topic clustering procedure [25]. To illustrate, in (1.1)
and (1.2) , even perfect knowledge of the general topi c (most likely, "stock
market trends") does not help much.
An alternat ive solution is to use long distance depend encies between
word pairs which show significant correlat ion in th e tr aining corpus . In the
above example, suppos e that the training data reveals a significant corre-
lation between "stocks" and "fell ." Then the presence of "s tocks" in the
document could automat ically trigg er "fe ll," causing its prob ability esti-
mate to change. Because thi s behavior would occur in both (1.1) and in
(1.2), proximity being irrelevant in this kind of model, the two phrases
would lead to th e same result. In t his approach, t he pair (s tocks, f ell )
is said to form a word trigger pair [44]. In practice, word pairs with high
mutual information are searched for inside a window of fixed duration. Un-
fortunat ely, trigger pair selection is a complex issue: different pairs display
markedly different behavior, which limits the potential of low frequency
word triggers [51]. Still, self-tri ggers have been shown to be particularly
powerful and robust [44]' which underscores the desirability of exploit ing
correlat ions between th e current word and features of th e document history.
Recent work has sought to extend the word tri gger concept by using
a more comprehensive framework to handle the trigg er pair select ion [2-
4, 6, 18, 28, 30]. This is based on a paradigm originally formulated in
the context of information retrieval , called latent semantic analys is (LSA)
[10,21 ,24,26,31 ,42,43,56] . In this paradigm, co-occurr ence analysis still
t akes place across the span of an ent ire document, but every combinat ion
of words from the vocabulary is viewed as a potential tri gger combinat ion.
This leads to the systematic integration of long-term semantic dependencies
into the analysis.
Th e concept of document assumes that th e available training data
is tagged at the document level, i.e., there is a way to identify article
bound aries. This is the case, for example, with th e ARPA North American
Business (NAB) News corpus [39]. Once thi s is done, t he LSA par adigm
can be used for word and document clustering [6, 28, 30], as well as for
language modeling [2, 18]. In all cases, it was found to be suitable to
capture some of the global semantic constraints present in the language. In
fact, hybrid n-gram+LSA language models, constructed by embedding LSA
into the standard n-gram formulation, were shown to result in a substantial
reduction in average word error rate [3, 4].
1.4. Organization. The focus of this paper is on semantically-driven

span extension only, and more specifically on how the LSA paradigm can
be exploited to improve statistical language modeling. The main objectives
are : (i) to review the data-driven extraction of latent semantic information,
(ii) to assess its potential use in the context of spoken language processing,
(iii) to describe its integration with conventional n-gram language model-
ing, (iv) to examine the behavior of the resulting hybrid models in speech
recognition experiments, and (v) to discuss a number of factors which in-
fluence performance.
The paper is organized as follows . In the next two sections, we give an
overview of the mechanics of LSA feature extraction, as well as the salient
characteristics of the resulting LSA feature space. Section 4 explores the
applicability of this framework for general semantic classification. In Sec-
tion 5, we shift the focus to LSA-based statistical language modeling for
large vocabulary recognition . Section 6 describes the various smoothing
possibilities available to make LSA-based language models more robust.
In Section 7, we illustrate some of the benefits associated with hybrid n-
gram+LSA modeling on a subset of the Wall Street Journal (WSJ) task.
Finally, Section 8 discusses the inherent trade-offs associated with the ap-
proach, as evidenced by the influence of the data selected to train the LSA
component of the model.
2. Latent semantic analysis. Let V, IVI = M, be some underly-

ing vocabulary and T a training text corpus, comprising N articles (docu-
ments) relevant to some domain of interest (like business news, for example,
in the case of the NAB corpus [39]). The LSA paradigm defines a mapping
between the discrete sets V, T and a continuous vector space 5, whereby
each word Wi in V is represented by a vector Ui in 5, and each document
d j in T is represented by a vector Vj in 5 .
2.1. Feature extraction. The starting point is the construction of

a matrix (W) of co-occurrences between words and documents. In marked
contrast with n-gram modeling, word order is ignored, which is of course
in line with the semantic nature of the approach [43] . This makes it an
instance of the so-called "bag-of-words" paradigm, which disregards collo-
cational information in word strings: the context for each word essentially
becomes the entire document in which it appears. Thus, the matrix W is
accumulated from the available training data by simply keeping track of
which word is found in what document.
This accumul ation involves some suitable function of t he word count,

i.e., th e number of tim es each word appears in each document [6] . Var-
ious implementations have been investigated by th e informat ion retrieval
community (see, for example, [23]). Evidence points to th e desirability
of normalizing for document length and word entropy. Thus, a suitable
expression for the (i, j ) cell of W is:
(2.1)
where Ci ,j is the numb er of tim es W i occurs in dj , nj is th e total numb er of

words present in dj , and Ci is the normalized entropy of W i in the corpus
T. The global weighting implied by 1 - Ci reflects th e fact that two words
appearing with the same count in dj do not necessarily convey the same
amount of information about th e document; this is subordinated to the
distribution of the words in th e collection T.
If we denote by t i = I:j Ci ,j th e total number of times W i occurs in T ,
th e expression for Ci is easily seen to be:
(2.2)
By definition , 0 ::; Ci:s1, with equality if and only if Ci ,j = t i and Ci, j =

tilN , respectively. A value of Ci close to 1 indicat es a word distributed
across many docum ents throughout the corpus, while a value of Ci close to
o means t hat th e word is present only in a few specific documents. The
global weight 1 - s, is th erefore a measure of th e indexing power of th e
word Wi .
2.2. Singular value decomposition. The (M x N ) word-document
matrix W resulting from t he above feature ext ract ion defines two vector
repr esentations for th e words and th e documents . Each word Wi can be
uniquely associated with a row vector of dimension N, and each document
dj can be uniquely associat ed with a column vector of dimension M. Un-
fortunately, these vector represent ations are unpr actical for three related
reasons. First, the dimensions M and N can be extremely large; second,
th e vectors W i and dj are typi cally very sparse; and third , th e two spac es
are distinct from one other.
To address thes e issues, one solution is to perform th e (order-R) sin-
gular value decomposition (SVD) of W as [29]:
(2.3)
where U is th e (M x R) left singular matrix with row vectors U i (1 ::;

i ::; M) , S is the (R x R) diagonal matrix of singular values 8 1 ?: 82 ?:
.. . ?: sn > 0, V is the (N x R) right singular matrix with row vectors Vj
:s
(1 j ::; N) , R « min (M , N) is t he order of th e decomposition , and T
denotes matrix transposition. As is well known, both left and right singular
matrices U and V are column-orthonormal, i.e., UTU = VTV = IR (the
identity matrix of order R) . Thus, th e column vectors of U and Veach
define an orthornormal basis for the space of dimension R spanned by the
(R-dimensional) u/s and vj's . Furthermore, the matrix W is the best
rank-R approximation to the word-document matrix W , for any unitarily
invariant norm (cf., e.g., [19]) . This entails, for any matrix A of rank R :
(2.4) min
{A : rank(A)=R}
IIW -All = IIW - WII = SR+l,
where II . II refers to the L 2 norm , and S R+l is the smallest singular value
retained in the order-(R+ 1) SVD of W . Obviously, SR+l = 0 if R is equal
to the rank of W .
Upon projecting the row vectors of W (Le., words) onto the orthonor-
mal basis formed by the column vectors of V, the row vector UiS charac-
terizes the position of word Wi in the underlying R-dimensional space , for
1 :S i :S M. Similarly, upon projecting the column vectors of W [i.e., docu-
ments) onto the orthonormal basis formed by the column vectors of U, the
row vector VjS characterizes the position of document dj in th e same space,
for 1 :S j :S N. We refer to each of the M scaled vectors iii = UiS as a word
vector, uniquely associated with word Wi in the vocabulary, and each of the
N scaled vectors Vj = "i S as a document vector, uniquely associated with
document dj in the corpus. Thus, (2.3) defines a transformation between
high-dimensional discrete entities (V and T) and a low-dimensional contin-
uous vector space 5, the R-dimensional (LSA) space spanned by the u/s
and vi's . The dimension R is bounded from above by the (unknown) rank
of the matrix W, and from below by the amount of distortion tolerable in
the decomposition. It is desirable to select R so that W captures the major
structural associations in W, and ignores higher order effects.
2.3. Properties. By construction, the "closeness" of vectors in the
LSA space 5 is determined by the overall pattern of the language used in
T , as opposed to specific constructs. Hence, two words whose representa-
tions are "close" (in some suitable metric) tend to appear in the same kind
of documents, whether or not they actually occur within identical word
contexts in those documents. Conversely, two documents whose represen-
tations are "close" tend to convey the same semantic meaning, whether
or not they contain the same word constructs. In the same manner, from
the bidiagonalization process inherent in the SVD, we can expect that the
respective representations of words and documents that are semantically
linked would also be "close" in the LSA space S.
Of course, the optimality of this framework can be debated, since the
L 2 norm may not be the best choice when it comes to linguistic phenom-
ena . For example, the Kullback-Leibler divergence provides a more elegant
(probabilistic) interpretation of (2.3) [31] , albeit at the expense of requiring
a conditional independence assumption on the words and the documents
L AT ENT SEMANT IC LANGUAGE MODELING 79
Intra-Topic, Original Space

a
-, I·
a ~ l(lter-TQPic
~~ \ O.gm,1 .~'"
:0
'"
.0 a
£~
C;
o
..J a
"r
~ '--. -..1-J..-...,.... ,.- ---'r_- ' --O....-_ _--,--'
0.0 0.5 1.0 1.5 2.0

. .. Distance
Expected Distributions In Uflqlnal Space and LSA Space
FIG. 1. Im proved Topic Sepambility in LSA Space {Aft er (47J).
[32J. This caveat notwith st anding, t he corres pondence between closeness

in LSA space and semant ic relatedness is well documented. In ap plications
such as information retrieval, filtering, indu ction, and visua lization, t he
LSA framework has repeatedly proven remarkabl y effective in capturing
semant ic inform ation [10, 21, 24, 26,32 ,42 ,43, 56J.
Such behavior was recentl y illustrated in [47], in t he context of an (ar-
t ificial) inform ati on retri eval tas k with 20 distin ct to pics and a vocabulary
of 2000 words. A probabilist ic corpus model generated 1000 documents,
each 50 to 100 words long. The prob ability distribution for each to pic was
such t hat 0.95 of its probability density was equally distributed among topic
words, and th e remaining 0.05 was equally distributed among all th e 2000
words in the vocabul ary. The authors of the study measured th e distance!
between all pairs of document s, both in the original space and in the LSA
space obtained as above, with R = 20. This leads to th e expected dist ance
distributions depicted in Figur e 1, where a pair of document s is considered
"Int ra-Topic" if t he two documents were generated from t he same top ic
and "Inte r-Topic" ot herwise.
It can be seen t hat in t he LSA space th e average distance between
inter-topic pairs stays about the same, while t he average distance between
intra-topic pairs is dra matically reduced. In addit ion, t he standa rd de-
1 T he relevant definition for t his quantity will be d iscussed in detail shor tly, cr. Sec-
tion 3.3.
viation of the intra-topic distance distribution also becomes substantially

smaller. As a result, separability between intra- and inter-topic pairs is
much better in the LSA space than in the original space. Note that this
holds in spite of a sharp increase in the standard deviation of the inter-
topic distance distribution, which bodes well for the general applicability of
the method. Analogous observations can be made regarding the distance
between words and/or between words and documents.
2.4. Computational effort. Clearly, classical methods for determin-
ing the SVD of dense matrices (see, for example , [11]) are not optimal for
large sparse matrices such as W . Because these methods apply orthog-
onal transformations (Householder or Givens) directly to the input ma-
trix, they incur excessive fill-in and thereby require tremendous amounts
of memory. In addition, they compute all the singular values of W; but
here R « min(M, N), and therefore doing so is computationally waste-
ful. Instead, it is more appropriate to solve a sparse symmetric eigenvalue
problem, which can then be used to indirectly compute the sparse singu-
lar value decomposition. Several suitable iterative algorithms have been
proposed by Berry, based on either the subspace iteration or the Lanczos
recursion method [9] . Convergence is typically achieved after 100 or so
iterations.
3. LSA feature space. In the continuous vector space 5 obtained
above, each word Wi E V is represented by the associated word vector of
dimension R, Ui = UiS, and each document dj E T is represented by the
associated document vector of dimension R, Vj = vjS. This opens up
the opportunity to apply familiar clustering techniques in 5 , as long as
a distance measure consistent with the SVD formalism is defined on the
vector space. Since the matrix W embodies, by construction, all structural
associations between words and documents, it follows that, for a given
training corpus , W W T characterizes all co-occurrences between words,
and W T W characterizes all co-occurrences between documents.
3.1. Word clustering. Expanding W WT using the SVD expression
(2.3), we obtain (henceforth ignoring the distinction between Wand W):
(3.1)
Since S is diagonal, a natural metric to consider for the "closeness" between

words is therefore the cosine of the angle between UiS and ujS:
(3.2)
for any 1 ::; i,j ::; M. A value of K(Wi,Wj) = 1 means the two words
always occur in the same semantic context, while a value of K (Wi, Wj) < 1
means the two words are used in increasingly different semantic contexts.
Cluster 1
Andy, antique, antiques, art , artist, artist's, artists , artworks,

auctioneers, Christie 's, collector, drawings, gallery, Gogh, fetched,
hysteria, m asterpiece, museums, painter, painting, paintings, Picasso,
Pollock, reproduction, Sotheby 's, van, Vincent, Warhol
Cluster 2
appeal, appeals, attorney, at torney's, counts, court, court's, courts,

condemned, convictions, criminal, decision, defend, defendant,
dismisses, dismissed, hearing, here, indicted, indictm ent, indictm ents,
judge, judicial, judiciary, j ury, juries, lawsuit, leniency, overt urned,
plaintiffs, prosecute, prosecution, prosecutions, prosecutors, ruled,
ruling, sentenced, sentencing, suing, suit, suits , witness
FIG. 2. Word Cluster Exam ple (After (2J).
While (3.2) does not define a bona fide distance measure in t he space S , it
easy leads to one. For exam ple, over t he interval [0,71'], the measure:
(3.3)
readily satisfies t he properties of a dist ance on S. At this point , it is

st raightforward to proceed with t he clusteri ng of the word vectors Ui , using
any of a variety of algorithms (see, for instance, [1]). T he outco me is a set
of cluste rs Ck , 1 ::; k ::; K , which can be thought of as revealing a partic ular
layer of semantic knowledge in t he space S .
3.2. Word cluster example. For th e purpose of illustr ation, we re-
call here t he result of a word clustering experiment originally reported in
[2] . A corpus of N = 21, 000 documents was randomly selected from th e
WSJ portion of the NAB corpus. LSA training was t hen performed with an
underlying vocabulary of M = 23, 000 words, and the word vectors in the
resultin g LSA space were clustered into 500 disjoint cluste rs using a combi-
nation of K-means and bottom-up cluster ing (d. [4]). Two representative
examples of th e clusters so obtai ned are shown in Figure 2.
Th e first thing to note is that t hese word clusters comprise words with
different part of speech, a marked difference with conventional class n-gra m
techniques (d. [45]) . This is a direct consequence of t he semantic nature
of the derivatio n. Second , some obvious words seem to be missing from
the cluste rs: for examp le, the singular noun "drawing" from cluster 1 and
the present tense verb "rule" from cluster 2. This is an instance of a phe-
nomenon called polysemy: "drawing' and "rule" are more likely to appear in
the training text with their alternative meanings (as in "drawing a conclu-
sion" and "breaking a rule," respectively), thus resulting in different cluster
assignments. Finally, some words seem to contribute only marginally to the
clusters: for example, "hysteria" from cluster 1 and "here" from cluster 2.
These are the unavoidable outliers at the periphery of the clusters.
3.3. Document clustering. Proceeding as above, the SVD expres-
sion (2.3) also yields:
(3.4)
As a result, a natural metric to consider for the "closeness" between doc-

uments is the cosine of the angle between ViS and vjS, i.e.:
(3.5)
for any 1 :S i, j :S N . This has the same functional form as (3.2); thus,
the distance (3.3) is equally valid for both word and document clustering.f
The resulting set of clusters De, 1 :S f :S L, can be viewed as revealing
another layer of semantic knowledge in the space S.
3.4. Document cluster example. An early document clustering ex-
periment using the above measure was documented in [30]. This work was
conducted on the British National Corpus (BNC), a heterogeneous corpus
which contains a variety of hand-labelled topics. Using the LSA framework
as above, it is possible to partition BNC into distinct clusters, and compare
the sub-domains so obtained with the hand-labelled topics provided with
the corpus. This comparison was conducted by evaluating two different
mixture trigram language models : one built using either the LSA sub-
domains, and one built using the hand-labelled topics. As the perplexities
obtained were very similar [30], this validates the automatic partitioning
performed using LSA.
Some evidence of this behavior is provided in Figure 3, which plots
the distributions of four of the hand-labelled BNC topics against the ten
document sub-domains automatically derived using LSA. While clearly not
matching the hand-labeling, LSA document clustering in this example still
seems reasonable. In particular, as one would expect, the distribution
for the natural science topic is relatively close to the distribution for the
applied science topic (cf. the two solid lines), but quite different from the
two other topic distributions (in dashed lines). From that standpoint, the
data-driven LSA clusters appear to adequately cover the semantic space.
2In fact, the measure (3.3) is precisely the one used in the study reported in Figure l.
Thus, the distances on the x-axis of Figure 1 are V( di , dj) expressed in radians.
LATENT SEMANTIC LANGUAGE M ODEL ING 83
LO
+
X
Natwal Science
ApPlied Science
c:i Social S<;:ience
~ Imaginative
-
c:i
o
c:i
2 4 6 8 10
. . Sub-domain (Cluster) Index
Probab ilitv Distribut ions 01Four BNG TOPIcs AQalnst lSA Document Clusters
FIG. 3. Document Cluster Example (A fter [3D)) .
4. Semantic classification. As just seen in the previous two sec-

tion s, th e lat ent semantic framework has a number of interesting properti es,
includin g: (i) a single vector represent ation for both words and document s
in the same cont inuous vector space, (ii) an underlying topological st ruc-
ture reflecting semant ic similarity, (iii) a well-mot ivated, natural metric to
measur e the distance between words and between documents in that space,
and (iv) a relatively low dimensionality which makes clustering meaningful
and practical. T hese properties can be exploited in several areas of spo-
ken language processing. In t his section, we address the most immediate
domain of application, which follows direct ly from t he previous cluste ring
discussion: (data-dr iven) semantic classification [7, 8, 12, 16, 27].
4.1. Framework extension. Semant ic classification refers to the

t ask of determining, for a given document , which one of severa l pre-defined
topics t he document is most closely aligned with . In cont rast with th e
clusterin g set up discussed above, such document will not (normally) have
been seen in t he training corpus. Hence, we first need to exte nd t he LSA
framework accordingly. As it turns out, under relatively mild assumptions,
finding a representation for a new document in the space S is st ra ightfor-
ward .
Let us refer to the new document as dp, with p > N, where the tilde
symbol denotes t he fact th at the document was not part of the t ra ining
dat a. First , we construct a feature vector containing, for each word in
the underlying vocabulary, the weighted counts (2.1) with j = p. This

feature vector dp , a column vector of dimension M, can be thought of as
an additional column of the matrix W . Thus, provided the matrices U and
S do not change, the SVD expansion (2.3) implies:
(4.1)
where the R-dimensional vector fi;

act as an additional column of the
matrix V T . This in turn leads to the definition:
(4.2) vc; p = vp S = d-pT U .

-
v
The vector p , indeed seen to be functionally similar to a document vector,
corresponds to the representation of the new document in the space S .
To convey the fact that it was not part of the SVD extraction, the
new document dp is referred to as a pseudo-document. Recall that the
(truncated) SVD provides, by definition, a parsimonious description of the
linear space spanned by W. As a result, if the new document contains
language patterns which are inconsistent with those extracted from W, the
SVD expansion (2.3) will no longer apply. Similarly, if the addition of dp
causes the major structural associations in W to shift in some substantial
manner.i' the parsimonious description will become inadequate. Then U
and S will no longer be valid, in which case it would be necessary to re-
compute (2.3) to find a proper representation for dp- If, on the other hand ,
the new document generally conforms to the rest of the corpus T , then the
v
pseudo-document vector p in (4.2) will be a reasonable representation for
s;
Once the representation (4.2) is obtained, the "closeness" between the
new document dp and any document cluster De can then be expressed as
V(dp , De), calculated from (3.5) in the previous section.
4.2. Semantic inference. This can be readily exploited in such com-
mand-and-control tasks as desktop user interface control [7] or automated
call routing [12]. Suppose that each document cluster De can be uniquely
associated with a particular action in the task. Then the centroid of each
cluster can be viewed as the semantic anchor of this action in the LSA
space . An unknown word sequence (treated as a new "document") can
thus be mapped onto an action by evaluating the distance (3.3) between
that "document" and each semantic anchor. We refer to this approach
as semantic inference [7, 8]. In contrast with usual inference engines (cf.
3For example, suppose training was carried out for a banking application involving
the word "bank" taken in a financial context. Now suppose dp is germane to a fishing
application, where "bank" is referred to in the context of a river or a lake. Clearly,
the closeness of "bank" to, e.g., "money" and "account," would be irrelevant to dp .
Conversely, adding dp to W would likely cause such structural associations to shift
substantially, and perhaps even disappear altogether.
... what is the day

day
• word
£ command
co D. new variant
o
5
'iii
c
<D (0
E .
es O what
-is
o
> what time is the meeting
...
-
(/) whatis ...
"tJ
C
"¢
. the time
0 time
-,
0 b,
o
<D
(/) when is the meeting
meeting cancel
C\I the
o meeting
•
...
cancel
o the
o -
0.0 0.2 0.4 0.6 0.8 1.0
. First SVD Dimension
Two-Dirnerisional lfushatiori 01 LSASpace
FIG. 4. An Example of Semantic Inference for Command and Control (R = 2) .
[20]), semantic inference thus defined does not rely on formal behavioral
principles extracted from a knowledge base. Instead, the domain knowledge
is automatically encapsulated in the LSA space in a data-driven fashion.
To illustrate, consider an application with N = 4 actions (documents),
each associated with a unique command: (i) "what is the time," (ii) "what
is the day," (iii) "what time is the meeting," and (iv) "cancel the meeting."
In this simple example, there are only M = 7 words in the vocabulary, with
some interesting patterns: "what" and "is" always co-occur, "the " appears
in all four commands, only (ii) and (iv) contain a unique word , and (i) is
a proper subset of (iii). Constructing the (7 x 4) word-document matrix
as described above, and performing the SVD, we obtain the 2-dimensional
space depicted in Figure 4.
This figure shows how each word and each command is represented
in the space S. Note that the two words which each uniquely identify a
command-"day" for (ii) and "cancel" for (iv)-each have a high coordi-
nate on a different axis. Conversely, the word "the ," which conveys no in-
formation about the identity of a command, is located at the origin. On the
other hand, the semantic anchors for (ii) and (iv) fall "close" to the words
which predict them best-"day" and "cancel" , respectively. Similarly, the
semantic anchors for (i) and (iii) fall in the vicinity of their meaningful
components-"what-is" and "time" for (i) and "time" and "meeting" for
(iii)-with the word "time," which occurs in both, indeed appearing "close"
to both.
Now suppose that a user says something outside of the training setup,
such as "when is the meeting" rather than "what time is the meeting."
This new word string turns out to have a representation in the space S
indicated by the hollow triangle in Figure 4. Observe that this point is
closest to the representation of command (iii). Thus , the new word string
can be considered semantically most related to (iii), and the correct action
can be automatically inferred . This can be thought of as a way to perform
"bottom-up" natural language understanding.
By replacing the traditional rule-based mapping between utterance
and action by such data-driven classification, semantic inference makes it
possible to relax some of the typical command-and-control interaction con-
straints. For example, it obviates the need to specify rigid language con-
structs through a domain-specific (and thus typically hand-crafted) finite
state grammar. This is turn allows the end user more flexibility in ex-
pressing the desired command/query, which tends to reduce the associated
cognitive load and thereby enhance user satisfaction [12J .
4.3. Caveats. Recall that LSA is an instance of the "bag-of-words"
paradigm, which pays no attention to the order of words in the sentence.
This is what makes it well-suited to capture semantic relationships between
words. By the same token, however, it is inherently unable to capitalize
on the local (syntactic, pragmatic) constraints present in the language.
For tasks such as call routing, where only the broad topic of a message is
to be identified, this limitation is probably inconsequential. For general
command and control tasks, however, it may be more deleterious.
Imagine two commands that differ only in the presence of the word
"not" in a crucial place. The respective vector representations could con-
ceivably be relatively close in the LSA space, and yet have vastly differ-
ent intended consequences. Worse yet, some commands may differ only
through word order. Consider, for instance, the two MacOS 9 commands:
change popup to window

(4.3)
change window to popup
which are mapped onto the exact same point in LSA space . This makes
them obviously impossible to disambiguate.
As it turns out, it is possible to handle such cases through an extension
of the basic LSA framework using word agglomeration. The idea is to move
from words and documents to word n-tuples and n-tuple documents, where
each word n-tuple is the agglomeration of n successive words, and each
(n-tuple) document is now expressed in terms of all the word n-tuples it
contains. Despite the resulting increase in computational complexity, this
extension is practical in the context of semantic classification because of
the relatively modest dimensions involved (as compared to large vocabulary
recognition) . Further details would be beyond the scope of this manuscript,
but the reader is referred to [8] for a complete description.
LATE NT SE MANT IC LANGUAGE MODELING 87
5. N-gram+LSA language modeling. Anoth er major area of ap-

plication of t he LSA framework is in stat ist ical language modeling, where
it can readily serve as a paradigm for semantically-driven span exte nsion.
Because of th e limitation just discussed, however, it is best applied in con-
junction with t he standa rd n-gram approach. This section describes how
t his can be done.
5.1. LSA component. Let w q denote t he word about to be pre-
dicted, and H q - 1 the admissible LSA history (context) for this particular
word. At best this history can only be the current document so far, i.e.,
up to word W q - l, which we denote by dq - 1 . Thus , in general terms , the
LSA language model probability is given by:
(5.1)
where the condit ioning on 5 reflects the fact th at the prob ability depends
on th e particular vector space arising from the SVD represent ation. In this
expression, Pr (wq!dq_ 1 ) is computed directly from th e represent ations of
wq and dq - 1 in the space 5 , i.e., it is inferred from th e "closeness" between
the associated word vector and (pseudo-)document vector in 5 . We there-
fore have to specify both the appropriate pseudo-document representation
and th e relevant probability measure.
5.1.1. Pseudo-document representation. To come up with a pseudo-
document representation , we leverage th e results of Section 4.1, with some
slight modification s due to t he time-varying nature of th e span considered.
From (4.2), t he context dq - 1 has a represent ation in t he space 5 given by:
(5.2) c;
V q- l = -
Vq- l
S = d-q-l
T U.
As mentioned before, t his vector representation for dq - 1 is adequate under

some consistency condit ions on the general patterns present in th e domain
considered. The difference with Section 4.1 is th at , as q increases, th e con-
tent of the new document grows, and th erefore th e pseudo-document vector
moves around accordingly in th e LSA space. Assuming t he new document
is semant ically homogeneous, eventually we can expect th e resulting tra-
jectory to settle down in th e vicinity of the document cluster corr esponding
to th e closest semantic conte nt .
Of course, here it is possible to take advantage of redundancies in
tim e. Assume, without loss of generality, that word Wi is observed at tim e
q. Then, dq - 1 and dq differ only in one coordinate, corresponding to th e
index i. Assume furth er th at th e t raining corpus T is large enough, so that
the normalized ent ropy e, (1 :::; i :::; M) does not change appreciably with
the addit ion of each pseudo-document . This makes it possible, from (2.1),
to express dq as:
- nq - 1 - 1 - Ci
(5.3) d q = - - dq - 1 + - - [0 . . . 1. . . 0]T ,
nq nq
where the "1" in the above vector appears at coordinate i. This is turn
implies, from (5.2):
(5.4)
As a result, the pseudo-document vector associated with the large-span

context can be efficiently updated directly in the LSA space.
5.1.2. LSA probability. To specify a suitable "closeness" measure, we
now follow a reasoning similar to that of Section 3. Since, by construc-
tion , the matrix W embodies structural associations between words and
documents, and, by definition, W = USV T , a natural metric to consider
for the "closeness" between word Wi and document dj is the cosine of the
angle between ui81 /2 and v j8 1/ 2. Applying the same reasoning to pseudo-
documents, we arrive at :
(5.5)
for any q indexing a word in the text data. A value of K(wq,dq-d = 1

means that dq - 1 is a strong semantic predictor of wq , while a value of
K (wq , dq - 1 ) < 1 means that the history carries increasingly less informa-
tion about the current word. Interestingly, (5.5) is functionally equivalent
to (3.2) and (3.5), but involves scaling" by 8 1/ 2 instead of S . As before, the
mapping (3.3) can be used to transform (5.5) into a real distance measure.
To enable the computation of Pr (wqldq-d, it remains to go from that
distance measure to an actual probability measure. One solution is for the
distance measure to induce a family of exponential distributions with perti-
nent marginality constraints. In practice, it may not be necessary to incur
this degree of complexity. Considering that dq - 1 is only a partial docu-
ment anyway, exactly what kind of distribution is induced is probably less
consequential than ensuring that the pseudo-document is properly scoped
(cf. Section 5.3 below). Basically, all that is needed is a "reasonable"
probability distribution to act as a proxy for the true (unknown) measure.
We therefore opt to use the empirical multivariate distribution con-
structed by allocating the total probability mass in proportion to the dis-
tances observed during training. In essence, this reduces the complexity to
a simple histogram normalization, at the expense of introducing a potential
"quant izat ion-like" error. Of course, such error can be minimized through
a variety of histogram smoothing techniques. Also note that the dynamic
range of the distribution typically needs to be controlled by a par ameter
4Not surprisingly, this difference in scaling exactly mirrors the squ are root relation-
ship between the singular values of Wand the eigenvalues of the (square) matrices
WTWand WWT .
LATENT SEMANT IC LANGUAGE MODELIN G 89
t hat is opt imized empirically, e.g., by an exponent on the distance t erm, as

discussed in [18].
Intuitively, Pr (wqldq- d, in mark ed contrast with a convent iona l bag-
of-words (unigra m) mod el, reflects the "relevance" of word w q t o the ad-
missible history, as observed through dq - 1 . As such, it will be highest
for word s whose meaning aligns most closely with t he semant ic fabric of
dq - 1 (i.e., relevant "conte nt" words) , and lowest for word s which do not
convey any particular information about this fabric (e.g., "function" words
like "the") . This beh avior is exactly the opposite of that observed with
t he convent ional n-gram formalism , which tends to assign higher probabil-
ities to (frequent) function words t han to (rarer) conte nt words . Hence the
at t rac t ive synergy potential between the two paradigms.
5.2. Integration with N-grams. Exploiting this potential requires
integr ating the two together. This kind of integration can occur in a num-
ber of ways, such as simple interpolation [18, 34], or within th e maximum
ent ropy framework [22, 41, 57]. Alt ernatively, und er relatively mild as-
sumpt ions, it is also possible to derive an integrated formulation directly
from the expression for the overa ll language mod el pr obability. We st art
with t he definition:
(n+ l) (n)
Pr (wqIH q_ 1 ) = Pr (wqIHq_ 1 , H q_ 1
(I )
(5.6) ,
where H q - 1 denotes, as before , some suitable admissible history for word

w q, and th e superscripts (n), (I) , and (n+l) refer t o t he n-gra m component
(Wq- lWq-2 ... Wq-n+ l, with n > 1), t he LSA component (d q- 1), and the
integr ation thereof, resp ectively," This expression can be rewritten as:
P ( q H (I ) IH(n)
(5.7) Pr (w IH(n+I ) = r w , q-l q - l
q q-l "" P ( . H (I) IH (n» ,
~ r Wt , q -l q-l
w;E V
where the summat ion in the denominator extends over all words in V.
Expanding and re-arranging, the numerator of (5.7) is seen to be:
Pr(Wq ,H~~lIH~~)l) = Pr(wqIH~~)d . Pr(H~~dwq,H~~)l)

(5.8) = Pr (WqIWq-1Wq_2 . . . wq-n+d
. Pr(dq -l!Wq Wq-1Wq-2 . .. Wq_n+l)'
Now we make the assumpt ion that the probability of the document history
given the current word is not affect ed by the imm ediate context preceding
it . This is clearly an approximation, since WqWq_ l Wq-2 . .. Wq -n+l may
5Hencefort h we make t he assumpt ion that n > 1. Wh en n = 1, t he n-gra m history

becom es null, and t he integrated history t herefore degenerat es t o t he LSA history a lone,
bas ically red ucin g (5.6) to (5.1).
well reveal more information about the semantic fabric of the document
than wq alone. This remark notwithstanding, for content words at least ,
different syntactic constructs (immediate context) can generally be used to
carry the same meaning (document history) . Thus the assumption seems
to be reasonably well motivated for content words. How much it matters
for function words is less clear [37], but we conjecture that if the document
history is long enough, the semantic anchoring is sufficiently strong for the
assumption to hold. As a result, the integrated probability becomes:
Pr (wqIH(n+I))
q-l =
Pr (WqIWq_l Wq-2 . . . Wq-n+l) Pr (dq_1Iwq)
(5.9)
L Pr(wilwq-lWq-2 . . .Wq-n+t}Pr(dq-llwi)·
w;EV
If Pr (dq-1!wq) is viewed as a prior probability on the current document

history, then (5.9) simply translates the classical Bayesian estimation of
the n-gram (local) probability using a prior distribution obtained from
(global) LSA. The end result, in effect, is a modified n-gram language
model incorporating large-span semantic information.
The dependence of (5.9) on the LSA probability calculated earlier can
be expressed explicitly by using Bayes' rule to get Pr (dq_1Iw q) in terms of
Pr (wqldq_1). Since the quantity Pr (dq- 1) vanishes from both numerator
and denominator, we are left with:
(5.10)
where Pr (wq ) is simply the standard unigram probability. Note that this
expression is meaningful" for any n > 1.
5.3. Context scope selection. In practice, expressions like (5.9)-
(5.10) are often slightly modified so that a relative weight can be placed on
each contribution (here, the n-gram and LSA probabilities) . Usually, this is
done via empirically determined weighting coefficients. In the present case,
such weighting is motivated by the fact that in (5.9) the "prior" probability
Pr (dq_1Iw q) could change substantially as the current document unfolds.
Thus, rather than using arbitrary weights, an alternative approach is to
dynamically tailor the document history dq - 1 so that the n-gr am and LSA
cont ribut ions remain empirically balanced.
60 bserve that with n = 1, the right hand side of (5.10) degenerates to the LSA
probability alone, as expected.
L AT ENT SEMANTIC LANGUAGE MODELING 91
T his approach, referr ed to as context scope selection, is more closely

aligned with t he LSA framework , beca use of th e underlying cha nge in be-
havior between t raining and recognition. During t ra ining, t he scope is fixed
to be t he cur rent document . Dur ing recognition , however , t he concept of
"current document" is ill-defined, because (i) its length grows wit h each
new word , and (ii) it is not necessar ily clear at which point completion
occurs . As a result , a decision has to be made regard ing what to con-
sider "cur rent," versus what to consider part of an earlier (presuma bly less
relevant) document .
A st raightforward solut ion is to limit the size of t he history considered,
so as to avoid relying on old, possibly obsolete fragment s to const ruct t he
curre nt context. Altern ati vely, to avoid making a hard decision on the size
of t he caching window, it is possible to assume an exponent ial decay in
t he relevance of the context [3]. In this solution, exponent ial forgetting is
used to progressively discount older utterances. Assuming 0 < A :::; 1, this
approac h corresponds to modifying (5.4) as follows:
(5.11)
where the par ameter A is chosen according to the expected heterogeneit y

of t he session.
5.4. Computational effort. From the above, t he (on-line) cost in-
curred during recognition has t hree components: (i) t he const ruction of
t he pseud o-document repr esentation in 5 , as generally done via (5.11); (ii)
t he computation of t he LSA probability Pr (wq Jdq-d in (5.1); and (iii) the
integration proper, in (5.10). It can be shown (cf. [3, 4]) t hat t he total
cost of t hese operations, per word and pseudo-document , is O(R2). T his is
obviously more expensive t ha n t he usual table look-up required in conven-
t iona l n-gra m language modeling. On t he ot her hand , for ty pical values of
R, the resulting overh ead is, argua bly, quit e modest . This.allows hybrid
n-gra m+ LSA language modeling to be t aken advantage of in early st ages
of the sear ch [3].
6. Smoothing. Since the derivation of (5.10) does not depend on a
particular form of the LSA probabilit y, it is possible to t ake advantage
of t he additional layer (s) of knowledge uncovered earlier through word
(in Section 3.1) and document (in Section 3.3) clusterin g. Basically, we
can expect words and/or documents relat ed to t he curre nt document to
cont ribute with more synergy, and unr elat ed words and/or documents to
be bet te r discounted. Said anot her way, clusterin g provides a convenient
smoothing mechanism in t he LSA space [2, 3].
6.1. Word smoothing. Using t he set of word clusters C k , 1 :::; k :::;
K , produ ced in Secti on 3.1 leads to word-based smoot hing. In t his case,
we expand (5.1) as follows:
K
(6.1) Pr (wqldq_1 ) = L Pr (WqICk) Pr (Ckldq-l) '
k=l
which carries over to (5.10) in a straightforward manner. In (6.1), the
probability Pr(Ckldq-l) is qualitatively similar to (5.1) and can therefore
be obtained with the help of (5.5), by simply replacing the representation
of the word wq by that of the centroid of word cluster Ck. In contrast,
the probability Pr(wq!Ck) depends on the "closeness" of W'j relative to this
(word) centroid. To derive it, we therefore have to rely on the empirical
multivariate distribution induced not by the distance obtained from (5.5),
but by that obtained from the measure (3.2) mentioned in Section 3.1.
Note that a distinct distribution can be inferred on each of the clusters Ck,
thus allowing us to compute all quantities Pr (wiICk) for 1 ::; i ::; M and
1 ::; k ::; K.
The behavior of the model (6.1) depends on the number of word clus-
ters defined in the space S. Two special cases arise at the extremes of
the cluster range. If there are as many classes as words in the vocabulary
(K = M), then with the convention that P(wiICj) = tS ij , (6.1) simply re-
duces to (5.1). No smoothing is introduced, so the predictive power of the
model stays the same as before. Conversely, if all the words are in a single
class (K = 1), the model becomes maximally smooth: the influence of spe-
cific semantic events disappears, leaving only a broad (and therefore weak)
vocabulary effect to take into account . The effect on predictive power is,
accordingly, limited. Between these two extremes , smoothness gradually
increases , and it is reasonable to postulate that predictive power evolves in
a concave fashion.
The intuition behind this conjecture is as follows. Generally speaking,
as the number of word classes Ck increases, the contribution of Pr(wqjCk)
tends to increase, because the clusters become more and more semantically
meaningful. By the same token, however, the contribution of Pr(Ckldq-d
for a given dq - 1 tends to decrease, because the clusters eventually become
too specific and fail to reflect the overall semantic fabric of dq - 1 • Thus,
there must exist a cluster set size where the degree of smoothing (and
therefore the associated predictive power) is optimal for the task consid-
ered. This has indeed been verified experimentally, cf. [2] .
6.2. Document smoothing. Exploiting instead the set of document
clusters De, 1 ::; £, ::; L, produced in Section 3.3 leads to document-based
smoothing. The expansion is similar:
L
(6.2) Pr (wqldq- 1 ) = L Pr (wqIDe) Pr (Deldq-l) '
e=l
except that the document clusters De now replace the word clusters Ci,
This time, it is the probability Pr(wqJDe) which is qualitatively similar to
LATENT SEM AN T IC LANGUAGE MODELING 93
(5.1), and can therefore be obtained with th e help of (5.5). As for the
probability Pr(Deldq-1) , it depends on the "closeness" of dq- 1 relative to
t he cent roid of document cluster De. Thus , it can be obt ained t hrough th e
empirical multivari ate distribution induced by th e dist ance derived from
(3.5) in Section 3.3.
Again , the behavior of the model (6.2) depends on t he number of
document clusters defined in the space S. Compared to (6.1), however ,
(6.2) is more difficult to interpr et at th e extremes of t he cluste r range (i.e.,
L = 1 and L = N) . If L = N , for example, (6.2) does not reduce to (5.1),
because dq - 1 has not been seen in th e training dat a, and therefore cannot
be identified with any of the exist ing clusters. Similarly, t he fact t hat all
th e documents are in a single cluster (L = 1) does not imply th e degree
of degenerescence observed previously, because th e cluster itself is strongly
indicative of the general discourse domain (which was not genera lly true
of the "vocabulary cluster" above). Hence, depending on th e size and
st ruct ure of the corpus , th e model may still be adequate to capt ure general
discourse effects.
To see that, we apply L = 1 in (6.2), whereby th e expression (5.10)
becomes:
(6.3)
since th e quantity Pr (D 1 !dq - d vanishes from both numerator and denom-

inator. In t his expression D 1 refers to t he single document cluster en-
compassing all documents in the LSA space. In case t he corpus is fairly
homogeneous, D 1 will be a more reliable represent ation of t he underlying
fabr ic of the domain t han dq - 1 , and th erefore act as a robust proxy for
t he context observed. Interestin gly, (6.3) amount s to est imating a "correc-
tion" factor for each word, which depends only on the overall topic of the
collection. This is clearly similar to what is done in th e cache approach
to language model adaptation (see, for example , [17, 40]), except th at, in
th e present case, all words are tr eat ed as though th ey were already in th e
cache.
More generally, as the number of document classes De increases, th e
contribut ion of Pr( w qIDe) tends to increase, to the extent th at a more
homogeneous topic boosts th e effects of any related content words. On
t he ot her hand , t he contribut ion of Pr(Deldq-d te nds to decrease, because
t he clusters represent more and more specific topics, which increases th e
cha nce th at t he pseudo-document dq - 1 becomes an out lier. Thus, again
t here exists a cluster set size where t he degree of smoot hing is optimal for
t he task considered (d . [2]).
6.3. Joint smoothing. Finally, an expression analogous to (6.1) and

(6.2) can also be derived to take advantage of both word and document
clusters. This leads to a mixture probability specified by:
K L
(6.4) Pr(wqldq_1) = LLPr(wqICk,Dt)Pr(Ck,Dtldq-1) '
k=lt=l
which, for tractability, can be approximated as:
K L
(6.5) Pr (wqldq- 1) = L L Pr (WqICk) Pr (CkIDt) Pr (Dtldq- 1).
k=l t=l
In this expression, the clusters Ck and Dt are as previously, as are the
quantities Pr (wqICk ) and Pr(Dtjdq- 1). As for the probability Pr(CkIDt),
it is qualitatively similar to (5.1), and can therefore be obtained accordingly.
To summarize, any of the expressions (5.1), (6.1), (6.2), or (6.5) can be
used to compute (5.10), resulting in four families of hybrid n-gram+LSA
language models. Associated with these different families are various trade-
offs to become apparent below.
7. Experiments. The purpose of this section is to illustrate the be-
havior of hybrid n-gram+LSA modeling on a large vocabulary recognition
task." The general domain considered was business news, as reflected in
the WSJ portion of the NAB corpus. This was convenient for comparison
purposes since conventional n-gram language models are readily available ,
trained on exactly the same data [39] .
7.1. Experimental conditions. The text corpus T used to train
the LSA component of the model was composed of about N = 87,000
documents spanning the years 1987 to 1989, comprising approximately 42
million words. The vocabulary V was constructed by taking the 20,000
most frequent words of the NAB corpus, augmented by some words from
an earlier release of the WSJ corpus, for a total of M = 23,000 words.
The test set consisted of a 1992 test corpus of 496 sentences uttered by
12 native speakers of English. In all experiments, acoustic training was
performed using 7,200 sentences of data uttered by 84 speakers (a stan-
dard corpus known as WSJO SI-84). On the above test data, our baseline
speaker-independent, continuous speech recognition system (described in
detail in [3]) produced reference error rates of 16.7% and 11.8% across the
12 speakers considered, using the standard (WSJO) bigram and trigram
language models, respectively.
We performed the singular value decomposition of the matrix of co-
occurrences between words and documents using the single vector Lanczos
7The reader is referred to [4] for additional results in this application, and to [8J for
experiments involving semantic inference.
TA BLE 1
Word Error Rat e (WER) Results Using Hybrid Bi-LSA and Tri -LSA Models.
Word Error Rate Bigram Trigram

< WER Reduction> n=2 n=3
I Conventional n-Gram 16.7 % 11.8 %
Hybrid, No Smoothing 14.4 % < 14 %> 10.7 % < 9 %>
Hybrid, Document Smoothing 13.4 % < 20 %> 10.4 % < 12 %>
Hybrid, Word Smoothing 12.9 % < 23 %> 9.9 % < 16 %>
Hybrid, Joint Smoothing 13.0 % < 22 %> 9.9 % < 16 %>
method [9] . Over the course of this decomposition, we experimented with

different numbers of singular values retained, and found that R = 125
seemed to achieve an adequate balance between reconstruction error-
minimiz ing S R + l in (2.4)-and noise suppression-minimizing th e ratio
between order-Rand ord er-(R + 1) traces l:i Si . This led to a vector space
S of dimension 125.
We th en used this LSA space to const ruct the (unsmoothed) LSA
model (5.1), following the procedure described in Sect ion 5. We also con-
structed the various clustered LSA models present ed in Section 6, to imple-
ment smoothing based on word clusters-word smoothing (6.1) , docum ent
clusters-document smoothing (6.2), and both-joint smoothing (6.5). We
experimented with different values for th e number of word and /or docu-
ment clust ers (cf. [2]), and ended up using K = 100 word clust ers and
L = 1 document clust er . Finally, using (5.10), we combined each of these
models with either th e standard WSJO bigram or th e st andard WSJO t ri-
gra m. The resulting hybrid n-gram+ LSA language models, dubbed bi-LSA
and tri-LSA models, respectively, were then used in lieu of th e standard
WSJO bigram and trigram models.
7.2. Experimental results. A summ ary of th e results is provid ed

in Table 1, in terms of both absolute word error rat e (WER) numb ers
and WER reduction observed (in angle brackets). Without smoot hing, th e
bi-LSA langu age model leads to a 14% WER reduction compared to th e
st andard bigram . The corresponding tri-LSA language model leads to a
somewhat smaller (just below 10%) relative improvement compa red to th e
st and ard trigram . With smoothing, th e improvement brought about by
t he LSA component is more marked : up to 23% in th e smoot hed bi-LSA
case, and up to 16% in th e smooth ed tri-LSA case. Such results show th at
the hybrid n-gram+ LSA approach is a promising avenue for incorp orating
large-span semantic information into n-gram modeling.
The qualitativ e behavior of t he two n-gram+ LSA language models ap-
pears to be quite similar. Quantitatively, th e average reduction achieved
by tri-LSA is about 30% less th an th at achieved by bi-LSA. This is most
likely related to the greater predictive power of the trigram compared to the
bigram, which makes the LSA contribution of the hybrid language model
comparatively smaller. This is consistent with the fact that the latent se-
mantic information delivered by the LSA component would (eventually)
be subsumed by an n-gram with a large enough n . As it turns out, how-
ever, in both cases the average WER reduction is far from constant across
individual sessions, reflecting the varying role played by global semantic
constraints from one set of spoken utterances to another.
Of course, this kind of fluctuations can also be observed with the
conventional n-gram models, reflecting the varying predictive power of the
local context across the test set. Anecdotally, the leverage brought about
by the hybrid n-LSA models appears to be greater when the fluctuations
due to the respective components move in opposite directions. So, at least
for n ::; 3, there is indeed evidence of a certain complementarity between
the two paradigms.
7.3. Context scope selection. It is important to emphasize that
the recognition task chosen above represents a severe test of the LSA com-
ponent of the hybrid language model. By design, the test corpus is con-
structed with no more than 3 or 4 consecutive sentences extracted from
a single article. Overall, it comprises 140 distinct document fragments,
which means that each speaker speaks, on the average, about 12 different
"mini-documents." As a result, the context effectively changes every 60
words or so, which makes it somewhat challenging to build a very accurate
pseudo-document representation. This is a situation where it is critical
for the LSA component to appropriately forget the context as it unfolds,
to avoid relying on an obsolete representation. To obtain the results of
Table 1, we used the exponential forgetting setup of (5.11) with a value
A = 0.975.8
In order to assess the influence of this selection, we also performed
recognition with different values of the parameter A ranging from A = 1
to A = 0.95, in decrements of 0.01. Recall from Section 5 that the value
A = 1 corresponds to an unbounded context (as would be appropriate for
a very homogeneous session), while decreasing values of A correspond to
increasingly more restrictive contexts (as required for a more heterogeneous
session) . Said another way, the gap between A and 1 tracks the expected
heterogeneity of the current session.
Table 2 presents the corresponding recognition results, in the case of
the best bi-LSA framework (l.e., with word smoothing) . It can be seen
that, with no forgetting, the overall performance is substantially less than
the comparable one observed in Table 1 (13% compared to 23% WER
reduction). This is consistent with the characteristics of the task, and
underscores the role of discounting as a suitable counterbalance to frequent
8To fix ideas, this means that the word which occurred 60 words ago is discounted
through a weight of about 0.2.
TABLE 2
Influ ence of Cont ext Scope Selection on Word Error Rate.
Word Error Rate Bi-LSA with

< WER Reduction> Word Smoothing
>. = 1.0 14.5 % < 13 %>
>. = 0.99 13.6 % < 18 %>
>. = 0.98 13.2 % < 21 %>
>. = 0.975 12.9 % < 23 %>
>. = 0.97 13.0 % < 22 %>
>. == 0.96 13.1 % < 22 %>
>. = 0.95 13.5 % < 19 %>
context changes. Perform ance rapidly improves as >. decreases from >. =
1 to >. = 0.97, presumably because the pseudo-do cument representation
gets less and less contaminated with obsolete dat a. If forget ting becomes
too aggressive, however , the performance st arts degrading, as the effective
context no longer has an equivalent length which is sufficient for the t ask
at hand. Here, th is happ ens for>' < 0.97.
8. Inherent trade-offs. In th e previous section , the LSA component
of the hybrid langu age model was trained on exact ly t he same data as its
n-gram component. This is not a requirement, however , which ra ises the
question of how crit ical th e selection of th e LSA tr aining dat a is to the
performance of the recognizer. This is particularly interesting since LSA is
known to be weaker on heterog eneous corpora (see, for example, [30]) .
8.1. Cross-domain training. To ascert ain the matter, we went back
to calculat ing the LSA component using the original, unsmoothed model
(5.1). We kept th e same underlying vocabul ary V, left the bigram com-
ponent unchanged, and repeat ed the LSA tr aining on non-WSJ data from
th e same general period . Three corpora of increasing size were consid-
ered, all corr espond ing to Associat ed Pr ess (AP ) dat a: (i) Ti , composed
of N, = 84,000 document s from 1989, comprising approximately 44 mil-
lion words; (ii) 72 , composed of N 2 = 155, 000 documents from 1988 and
1989, comprising approximately 80 million words; and (iii) 73 , composed
of N 3 = 224,000 document s from 1988-1990, comprising approximately
117 million words. In each case we proceeded with th e LSA t raining as
described in Section 2. The results are reported in Table 3.
Two things are immediately apparent. First, th e performance im-
provement in all cases is much smaller th an previously observed (recall
th e corresponding reduction of 14% in Table 1). Larger training set sizes
notwithstanding, on the average t he hybrid model tr ained on AP dat a is
about 4 tim es less effect ive than that tr ained on WSJ data. This suggest s a
relatively high sensitivity of the LSA component to th e domain considered.
TABLE 3
Model Sensitivity to LSA Training Data.
Word Error Rate Bi-LSA with

<WER Reduction> No Smoothing
7j : N l = 84, 000 16.3 % <2 %>
72: N z = 155, 000 16.1 % <3 %>
73: N 3 = 224, 000 16.0 % <4 %>
To put this observation into perspective, recall that: (i) by definition, con-
tent words are what characterize a domain; and (ii) LSA inherently relies
on content words, since, in contrast with n-grams, it cannot take advantage
of the structural aspects of the sentence. It therefore makes sense to expect
a higher sensitivity for the LSA component than for the usual n-gram.
Second, the overall performance does not improve appreciably with
more training data, a fact already observed in [2] using a perplexity mea-
sure . This supports the conjecture that, no matter the amount of data
involved, LSA still detects a substantial mismatch between AP and WSJ
data from the same general period . This in turn suggests that the LSA
component is sensitive not just to the general training domain, but also to
the particular style of composition, as might be reflected, for example, in
the choice of content words and/or word co-occurrences. On the positive
side, this bodes well for rapid adaptation to cross-domain data, provided a
suitable adaptation framework can be derived.
8.2. Discussion. The fact that the hybrid n-gram+LSA approach is
sensitive to composition style underscores the relatively narrow semantic
specificity of the LSA paradigm. While n-grams also suffer from a possi-
ble mismatch between training and recognition, LSA leads to a potentially
more severe exposure because the space S reflects even less of the prag-
matic characteristics for the task considered. Perhaps what is required is
to explicitly include an "authorship style" component into the LSA frame-
work. 9 In any event, one has to be cognizant of this intrinsic limitation,
and mitigate it through careful attention to the expected domain of use.
Perhaps more importantly, we pointed out earlier that LSA is inher-
ently more adept at handling content words than function words. But, as
is well-known, a substantial proportion of speech recognition errors come
from function words, because of their tendency to be shorter, not well ar-
ticulated, and acoustically confusable. In general, the LSA component will
not be able to help fix these problems. This suggests that, even within a
well-specified domain, syntactically-driven span extension techniques may
9In [47J , for example, it has been suggested to define an M x M stochastic matrix (a
matrix with non-negative entries and row sums equal to 1) to account for the way style
modifies the frequency of words . This solution, however, makes the assumption-not
always valid-that this influence is independent of the underlying subject matter.
be a necessary complement to the hybrid approach. On that subject, note

from Section 5 that the integrated history (5.6) could easily be modified
to reflect a headword-based n-gram as opposed to a conventional n-gram
history, without invalidating the derivation of (5.10). Thus, there is no
theoretical barrier to the integration of latent semantic information with
structured language models such as described in [14, 35]. Similarly, there
is no reason why the LSA paradigm could not be used in conjunction with
the integrative approaches of the kind proposed in [53, 57], or even within
the cache adaptive framework [17, 40] .
9. Conclusion. Statistical n-grams are inherently limited to the cap-
ture of linguistic phenomena spanning at most n words. This paper has
focused on a semantically-driven span extension approach based on the
LSA paradigm, in which hidden semantic redundancies are tracked across
(semantically homogeneous) documents. This approach leads to a (con-
tinuous) vector representation of each (discrete) word and document in a
space of relatively modest dimension . This makes it possible to specify
suitable metrics for word-document, word-word, and document-document
comparisons, which in turn allows well-known clustering algorithms to be
applied efficiently. The outcome is the uncovering, in a data-driven fashion,
of multiple parallel layers of semantic knowledge in the space, with variable
granularity.
An important property of this vector representation is that it reflects
the major semantic associations in the training corpus, as determined by
the overall pattern of the language, as opposed to specific word sequences
or grammatical constructs. Thus, language models arising from the LSA
framework are semantic in nature, and therefore well suited to complement
conventional n-grams. Harnessing this synergy is a matter of deriving an
integrative formulation to combine the two paradigms. By taking advan-
tage of the various kinds of smoothing available, several families of hybrid
n-gram+LSA models can be obtained. The resulting language models sub-
stantially outperform the associated standard n-grams on a subset of the
NAB News corpus.
Such results notwithstanding, the LSA-based approach also face some
intrinsic limitations. For example, hybrid n-gram+LSA modeling shows
marked sensitivity to both the training domain and the style of composi-
tion . While cross-domain adaptation may ultimately alleviate this problem,
an appropriate LSA adaptation framework will have to be derived for this
purpose. More generally, such semantically-driven span extension runs the
risk of lackluster improvement when it comes to function word recogni-
tion. This underscores the need for an all-encompassing strategy involving
syntactically motivated approaches as well.
REFERENCES
[1] J .R. BELLEGARDA, Context-Dependent Vector Clustering for Speech Recognition,

Chapter 6 in Automatic Speech and Speaker Recognition: Advanced Topics,
C.-H. Lee, F .K Soong , and KK Paliwal (Eds .), Kluwer Academic Publishers ,
NY, pp . 133-157 , March 1996.
[2] J .R. BELLEGARDA, A Multi-Span Language Modeling Framework for Large Vo-
cabulary Speech Recognition, IEEE Trans. Speech Audio Proc., Vol. 6, No.5,
pp . 456-467, September 1998.
[3J J .R . BELLEGARDA, Large Vocabulary Speech Recognition With Multi-Span Sta-
tistical Language Models, IEEE Tr ans . Speech Audio Proc. , Vol. 8 , No.1,
pp . 76-84, January 2000.
[4] J .R . BELLEGARDA, Exploiting Latent Semantic Information in Statistical Language
Modeling, Proc. IEEE, Spec . Issue Speech Recog . Understanding, B.H. Juang
and S. Furui (Eds.) , Vol. 88, No.8, pp . 1279-1296 , August 2000.
[5] J .R. BELLEGARDA, Robustness in Statistical Language Modeling : Review and Per-
spectives, Chapter 4 in Robustness in Language and Speech Technology, J.C.
Junqua and G.J.M. van Noord (Eds .), Kluwer Academic Publishers, Dortrecht ,
The Netherlands, pp . 101-121, February 2001.
[6] J .R. BELLEGARDA , J .W . BUTZBERGER, Y.L. CHOW, N.B . COCCARO, AND D. NAIK,
A Novel Word Clustering Algorithm Based on Latent Semantic Analysis, in
Proc. 1996 Int . Conf . Acoust ., Speech , Sig. Proc., Atlanta, GA , pp. I172-I175,
May 1996.
[7] J .R . BELLEGARDA AND KE.A. SILVERMAN, Toward Unconstrained Command and
Control : Data-Driven Semantic Inference, in Proc. Int. Conf. Spoken Language
Proc., Beijing, China, pp . 1258-1261, October 2000.
[8] J .R . BELLEGARDA AND KE.A. SILVERMAN, Natural Language Spoken Interface
Control Using Data-Driven Semantic Inference, IEEE Trans. Speech Audio
Proc., Vol. 11 , April 2003.
[9] M.W. BERRY, Letge-Scele Sparse Singular Value Computations, Int . J. Sup er-
cornp . Appl ., Vol. 6, No.1, pp . 13-49, 1992.
[10] M.W . BERRY, S.T. DUMAIS, AND G.W. O 'BRIEN, Using Linear Algebra for In-
telligent Information Retrieval, SIAM Review , Vol. 37, No.4, pp . 573-595,
1995.
[l1J M. BERRY AND A. SAMEII, An Overview of Parallel Algorithms for the Singular
Value and Dense Symmetric Eigenvalue Problems , J . Computational Applied
Math., Vol. 27, pp . 191-21 3, 1989.
[12] B . CARPENTER AND J . CIIU-CARROLL, Natural Language Call Routing: A Robust,
Self-Organized Approach, in Proc. Int . Conf. Spoken Language Proc. , Sydney,
Australia, pp . 2059-2062 , December 1998.
[13] C. CHELBA , D. ENGLE, F . JELINEK , V. JIMENEZ , S. KIIUDANPUR, L. MANGU ,
H. PRINTZ, E .S. RISTAD , R. ROSENFELD , A. STOLCKE, AND D. Wu, Structure
and Performance of a Dependency Language Model, in Proc. Fifth Euro , Conf.
Speech Comm, Technol., Rhodes, Greece , Vol. 5, pp . 2775-2778 , September
1997.
[14] C. CIIELBA AND F . JELINEK, Recognition Performance of a Structured Language
Model, in Proc. Sixth Euro. Conf. Speech Comm. Technol. , Budapest, Hun-
gary, Vol. 4 , pp . 1567-1570, September 1999.
[15] S. CIIEN, Building Probabilistic Models for Natural Language, Ph .D. Thesis, Har-
vard University, Cambridge, MA, 1996.
[16] J . CIIU-CARROLL AND B. CARPENTER, Dialog Management in Vector-Based Call
Routing, in Proc. Conf. Assoc. Comput. Linguistics ACL/COLING, Montreal,
Canada, pp . 256-262, 1998.
[17] P .R . CLARKSON AND A.J . ROBINSON, Language Model Adaptation Using Mix-
tures and an Exponentially Decaying Cache, in Proc, 1997 Int . Conf. Acoust.,
Speech , Signal Proc., Munich , Germany, Vol. 1, pp. 799-802, May 1997.
LATE NT SEMANTIC LANGUAGE MODELING 101
[18] N. COCCARO AND D. JURAFSKV, Towards Better Integration of Semantic Predictors

in Statistical Language Modeling , in Proc. Int . Conf. Spoken Language Proc.,
Sydney, Australia, pp . 2403-2406 , December 1998.
[19] J .K. CULLUM AND R .A. WILLOUGHBV, Lanczos Algorithms for Large Symmetric
Eigenvalue Computations - Vol. 1 Theory, Chapter 5: Real Rectangular Ma-
trices, Brickhauser, Boston, MA, 1985.
[20] R . DE MORI, Recognizing and Using Knowledge Structures in Dialog Systems, in
Proc. Aut. Speech Recog. Understanding Workshop, Keystone , CO , pp . 297-
306, December 1999.
[21] S. DEERWESTER, S.T . DUMAIS, G.W . FURNAS, T .K. LANDAUER, AND R. HARSH-
MAN, Indexing by Latent Semantic Analysis, J. Am . Soc. Inform. Science,
Vol. 41, pp . 391-407, 1990.
[22J S. DELLA PIETRA , V. DELLA PIETRA , R . MERCER, AND S. ROUKOS, Adaptive
Language Model Estimation Using Minimum Discrimination Estimation, in
Proc. 1992 Int . Conf . Acoust. , Speech, Signal Processing, San Francisco, CA ,
Vol. I, pp . 633-636, April 1992.
[23] S.T. DUMAIS, Improving the Retrieval of Information from External Sources, Be-
havior Res . Methods, Instrum., Computers, Vol. 23, No.2, pp . 229-236, 1991.
[24] S.T . DUMAIS, Latent Semantic Indexing (LSI) and TREC-2, in Proc. Second
Text Retrieval Conference (TREC-2) , D. Harman (Ed .), NIST Pub. 500-215,
pp . 105-116, 1994.
[25] M. FEDERICO AND R . DE MORI, Language Modeling, Chapter 7 in Spoken Di-
alogues with Computers, R. De Mori (Ed.) , Academic Press, London, UK ,
pp . 199-230, 1998.
[26] P .W. FOLTZ AND S.T . DUMAIS, Personalized Information Delivery: An Analysis of
Information Filtering Methods, Commun. ACM , Vol. 35, No. 12, pp. 51-60,
1992.
[27] P .N . GARNER, On Topic IdentifJcation and Dialogue Move Recognition, Computer
Speech and Language, Vol. 11 , No.4, pp . 275-306 , 1997.
[28] D. GILDEA AND T . HOFMANN, Topic-Based Language Modeling Using EM, in
Proc. Sixth Euro. Conf . Speech Comm. Technol. , Budapest, Hungary, Vol. 5,
pp . 2167-2170, September 1999.
[29] G . GOLUB AND C . VAN LOAN, Matrix Computations, Johns Hopkins, Baltimore,
MD , Second Ed ., 1989.
[30] Y. GOTOH AND S. RENALS, Document Space Models Using Latent Semantic Analy-
sis, in Proc. Fifth EUTo. Conf. Speech Comm. Technol. , Rhodes, Greece, Vol. 3 ,
pp . 1443-1448, September 1997.
[31J T . HOFMANN, Probabilistic Latent Semantic Analysis, in Proc. Fifteenth Conf.
Uncertainty in AI, Stockholm, Sweden, July 1999.
[32J T. HOFMANN, Probabilistic Topic Maps: Navigating Through Large Text Col-
lections, in Lecture Notes Compo Science. , No. 1642, pp . 161-172 , Springer-
Verlag , Heidelberg, Germany, July 1999.
[33] R. IVER AND M. OSTENDORF, Modeling Long Distance Dependencies in Language:
Topic Mixtures Versus Dynamic Cache Models, IEEE Trans. Speech Audio
Proc., Vol. 7, No. 1, January 1999.
[34] F. JELINEK, Self-Organized Language Modeling for Speech Recognition, in Read-
ings in Speech Recognition, A. Waibel and K.F. Lee (Eds .), Morgan Kaufmann
Publishers, pp . 450-506, 1990.
[35] F . JELINEK AND C. CHELBA, Putting Language into Language Modeling, in
Proc. Sixth Euro. Conf. Speech Comm. Technol., Budapest, Hungary, Vol. 1 ,
pp . KNI-KN5 , September 1999.
[36] D. JURAFSKY, C. WOOTERS, J . SEGAL, A. STOLCKE, E . FOSLER, G . TAJCHMAN ,
AND N. MORGAN, Using a Sto chastic Context-Free Grammar as a Language
Model for Speech Recognition, in Proc. 1995 Int . Conf . Acoust ., Speech, Signal
Proc. , Detroit, MI, Vol. I, pp . 189-192, May 1995.
[37J S. KHUDANPUR, Putting Language Back into Language Modeling, presented at

Workshop-2000 Spoken Lang. Reco. Understanding, Summit, NJ , February
2000.
[38] R . KNESER, Statistical Language Modeling Using a Variable Context, in Proc. Int.
Conf. Spoken Language Proc., pp. 494-497, Philadelphia, PA, October 1996.
[39J F . KUBALA, J.R. BELLEGARDA , J .R. COHEN , D . PALLETT , D .B . PAUL, M.
PHILLIPS , R. RAJASEKARAN , F . RICHARDSON , M. RILEY , R. ROSENFELD , R.
ROTH , AND M. WEINTRAUB , The Hub and Spoke Paradigm for CSR Evalua-
tion, in Proc. ARPA Speech and Natural Language Workshop, Morgan Kauf-
mann Publishers, pp . 40--44, March 1994.
[40J R . KUHN AND R. DE MORI, A Cache-based Natural Language Method for Speech
Recognition , IEEE Trans. Pattern Anal. Mach. Int el. , Vol. PAMI-12, No.6,
pp , 570--582, June 1990.
[41] J.D. LAFFERTY AND B. SUHM, Cluster Expansion and Iterative Scaling for Maxi-
mum Entropy Language Models, in Maximum Entropy and Bayesian Methods,
K Hanson and R. Silver (Eds.), Kluwer Academic Publishers, Norwell , MA ,
1995.
[42] T .K . LANDAUER AND S.T . DUMAIS, Solution to Plato's Problem : The Latent
Semantic Analysis Theory of Acquisition, Induction , and Representation of
Knowledge in Psychological Review, Vol. 104, No.2, pp. 211-240, 1997.
[43] T.K. LANDAUER, D. LAHAM , B . REHDER, AND M.E . SCHREINER, How Well Can
Passage Meaning Be Derived Without Using Word Order: A Comparison of
Latent Semantic Analysis and Humans, in Proc. Conf. Cognit. Science Soc .,
Mahwah, NJ, pp . 412-417, 1997.
[44J R . LAU, R. ROSENFELD, AND S. ROUKOS , Trigger-Based Language Models : A
Maximum Entropy Approach, in Proc. 1993 Int . Conf. Acoust., Sp eech , Signal
Proc., Minneapolis, MN , pp . II45-48, May 1993.
[45J H. NEY, U. ESSEN AND R. KNESER, On Structuring Probabilistic Dependences
in Sto chastic Language Modeling, Computer , Speech, and Language, Vol. 8 ,
pp . 1-38, 1994.
[46J T . NIESLER AND P . WOODLAND, A Variable-Length Category-Based N-Gram Lan-
guage Model, in Proc. 1996 Int . Conf. Acoust., Speech , Sig. Proc., Atlanta,
GA , pp . I164-I167, May 1996.
[47J C.H . PAPADIMITRIOU, P . RAGHAVAN , H. TAMAKI, AND S. VEMPALA , Latent Se-
mantic Indexing: A Probabilistic Analysis, in Proc. 17th ACM Symp. Princip.
Database Syst., Seattle, WA, 1998. Also J . CompoSyst. Sciences, 1999.
[48J F .C . PEREIRA , Y. SINGER, AND N. TISHBY, Beyond Word n-Grams, Computational
Linguistics, Vol. 22 , June 1996.
[49J L.R. RABINER, B .H. JUANG , AND C .-H . LEE, An Overview of Automatic Speech
Recognition, Chapter 1 in Automatic Speech and Speaker Recognition : Ad-
vanced Topics, C .-H. Lee, F.K Soong, and KK Paliwal (Eds.), Kluw er Aca-
demic Publishers, Boston, MA, pp . 1-30, 1996.
[50J R . ROSENFELD, The CMU Statistical Language Modeling Toolkit and its Use in the
1994 ARPA CSR Evaluation , in Proc. ARPA Speech and Natural Language
Workshop, Morgan Kaufmann Publishers, March 1994.
[51J R . ROSENFELD, A Maximum Entropy Approach to Adaptive Statistical Language
Modeling , Computer Speech and Language, Vol. 10, Academic Press, London,
UK, pp. 187-228, July 1996.
[52J R. ROSENFELD, Two Decades of Statistical Language Modeling : Where Do We
Go From Here, Proc. IEEE, Spec . Issue Speech Recog . Understanding, B.H.
Juang and S. Furui (Eds.), Vol. 88, No.8, pp . 1270--1278, August 2000.
[53] R . ROSENFELD, L. WASSERMAN , C. CAl, AND X .J. ZHU, Interactive Feature In-
duction and Logistic Regression for Whole Sentence Exponential Language
Models, in Proc. Aut. Speech Recog. Understanding Workshop, Keystone,
CO, pp . 231-236, December 1999.
[54] S. ROUKOS, Language Representation, Chapter 6 in Survey of the State of the Art
in Human Language Technology, R. Cole (Ed.) , Cambridge University Press,
Cambridge, MA, 1997.
[55] R . SCHWARTZ , T . IMAI , F . KUBALA , L. NGUYEN, AND J . MAKHOUL, A Maximum
Likelihood Model for Topic Classification of Broadcast News, in Proc. Fifth
Euro. Conf . Speech Comm. Technol., Rhodes, Greece, Vol. 3, pp . 1455-1458,
September 1997.
[56] R.E . STORY, An Explanation of the Effectiveness of Latent Semantic Indexing by
Means of a Bayesian Regression Model, Inform. Processing & Management,
Vol. 32, No.3, pp. 329-344, 1996.
[57] J . Wu AND S. KIIUDANPUR, Combining Nonlocal, Syntactic and N-Gram Depen-
dencies in Language Modeling, in Proc. Sixth Euro. Conf. Speech Comm.
Technol. , Budapest, Hungary, Vol. 5, pp. 2179-2182, September 1999.
[58] D .H . YOUNGER, Recognition and Parsing of Context-Free Languages in Time N 3,
Inform. & Control, Vol. 10 , pp . 198-208, 1967.
[59] R . ZHANG , E . BLACK , AND A. FINCH, Using Detailed Linguistic Structure in Lan-
guage Modeling , in Proc. Sixth Euro. Conf. Speech Comm. Technol., Bu-
dapest, Hungary, Vol. 4 , pp . 1815-1818, September 1999.
[60] X.J . ZHU, S.F . CHEN, AND R. ROSENFELD, Linguistic Features for Whole Sen-
tence Maximum Entropy Language Models, in Proc. Sixth Euro. Conf. Speech
Comm. Technol. , Budapest, Hungary, Vol. 4 , pp . 1807-1810, September 1999.
[61] V . ZUE, J. GLASS, D. GOODINE, H. LEUNG , M. PHILLIPS, J. POLIFRONI , AND S.
SENEFF, Integration of Speech Recognition and Natural Language Processing
in the MIT Voyager System , in Proc. 1991 IEEE Int . Conf. Acoust., Speech,
Signal Processing, Toronto, Canada, pp . 713-716, May 1991.
PROSODY MODELING FOR AUTOMATIC SPEECH
RECOGNITION AND UNDERSTANDING*
ELIZABETH SHRIBERGt AND ANDREAS STOLCKEt
Abstract. This paper summarizes statistical modeling approaches for the use of
prosody (th e rhythm and melody of speech) in automatic recognition and understanding
of speech. We outline effective prosodic feature extraction, model architectures, and
techniques to combine prosodic with lexical (word-based) information. We then survey
a number of applications of the framework, and give results for automatic sentence
segmentation and disfluency detection, topic segmentation, dialog act labeling, and word
recognition.
Key words. Prosody, speech recognition and understanding, hidden Markov

models.
1. Introduction. Prosody has long been studied as an important

knowledge source for speech understanding. In recent years there has been
a large amount of computational work aimed at prosodic modeling for au-
tomatic speech recognition and understanding. 1 Whereas most current
approaches to speech processing model only the words, prosody provides
an additional knowledge source that is inherent in, and exclusive to, spo-
ken language. It can therefore provide additional information that is not
directly available from text alone, and also serves as a partially redundant
knowledge source that may help overcome the errors resulting from faulty
word recognition.
In this paper, we summarize recent work at SRI International in the
area of computational prosody modeling, and results from several recog-
nition tasks where prosodic knowledge proved to be of help. We present
only a high-level perspective and summary of our research; for details the
reader is referred to publications cited .
2. Modeling philosophy. Most problems for which prosody is a
plausible knowledge source can be cast as statistical classification problems.
By that we mean that some linguistic unit U (e.g., words or utterances)
is to be classified as one of several target classes S. The role of prosody
"The research was supported by NSF Grants IRI-9314967, IRI-9618926 , and

IRI-9619921, by DARPA contract no. N66001-97-C-8544, and by NASA contract
no. NCC 2-1256. Additional support came from the sponsors of the 1997 CLSP Work-
shop [7, 11] and from the DARPA Communicator project at UW and ICSI [8] . The
views herein are those of the authors and should not be interpreted as representing the
policies of the funding agencies .
tSRI International, 333 Ravenswood Ave., Menlo Park, CA 94025 ({ees,stolcke}@
speech.sri.com) . We thank our many colleagues at SRI, ICSI , University of Washington
(formerly at Boston University), and the 1997 Johns Hopkins CLSP Summer Workshop,
who were instrumental in much of the work reported here.
IToo much work in fact , to cite here without unfair omissions. We cite some sp ecif-
ically relevant work below; a more compr ehensive list can be found in th e papers cited .
105
106 E. SHRIBERG AND A. STOLCKE
is to provide us with a set of features F that can help predict S. In a

probabilistic framework, we wish to estimate P(SIF). In most such tasks
it is also a good idea to use the information contained in the word sequence
W associated with U, and we therefore generalize the modeling task to
estimate P(SIW, F). In fact, Wand F are not restricted to pertain only
to the unit in question; they may refer to the context of U as well. For
example, when classifying an utterance into dialog acts, it is important to
take the surrounding utterances into account .
Starting from this general framework, and given a certain labeling
task, many decisions must be made to use prosodic information effectively.
What is the nature of the features F to be used? How can we model the
relationship between F and the target classes S? How should we model the
effect of lexical information Wand its interaction with prosodic properties
F? In the remainder of this paper we give a general overview of approaches
that have proven successful for a variety of tasks.
2.1. Direct modeling of target classes. A crucial aspect of our

work, as well as that of some other researchers [6, 5] is that the depen-
dence between prosodic features and target classes (e.g., dialog acts, phrase
boundaries) is modeled directly in a statistical classifier-without the use
of intermediate abstract phonological categories, such as pitch accent or
boundary tone labels. This bypasses the need to hand-annotate such la-
bels for training purposes, avoids problems of annotation reliability, and
allows the model to choose the level of granularity of the representation
that is best suited for the task [2] .
2.2. Prosodic features. As predictors of the target classes, we ex-

tract features from a forced alignment of the transcripts (usually with
phone-level alignment information), which can be based on either true
words, or on (errorful) speech recognition output. Similar approaches are
used by others [2]. This yields a rich inventory of "raw" features reflecting
FO, pause and segment durations, and energy. From the raw features we
compute a wide range of "derived" features-devised (we hope) to capture
characteristics of the classes-which are normalized in various ways, con-
ditioned on certain extraction regions, or conditioned on values of other
features.
Phone-level alignments from a speech recognizer provide durations of
pauses and various measures of lengthening (we have used syllable, rhyme,
and vowel durations for various tasks) and speaking rate. Pitch-based
features benefit greatly from a postprocessing stage that regularizes the
raw FO estimates and models octave errors [10]. As a byproduct of the
postprocessing, we also obtain estimates of the speaker's FO baseline, which
we have found useful for pitch range normalizations.
Combined with FO estimates, the recognizer output also allows com-
putation of pitch movements and contours over the length of utterances
PROSODY MODELING FOR SPEECH 107
or individual words, or over the length of windows positioned relative to a

location of interest (e.g., around a word boundary) . The same applies to
energy-based features .
2.3. Prosodic models. Any number of statistical classifiers that can

deal with a mix of categorical and real-valued features may be used to model
P(SIF, W) . These requirements, as well as our desire to be able to inspect
our models (both to understand patterns and for sanity checking), have led
us to use mainly decision trees as classifiers. Decision trees have two main
problems, however, which we have tried to address . First, to help overcome
the problem of greediness, we wrap a feature subset selection algorithm
around the standard tree growing algorithm, thereby often finding better
classifiers by eliminating detrimental features up front from consideration
by the tree [9]. Second, to make the trees sensitive to prosodic features in
the case of highly skewed class sizes, we train on a resampled version of
the target distribution in which all classes have equal prior probabilities.
This approach has additional benefits. It allows prosodic classifiers to be
compared (both qualitatively and quantitatively) across different corpora
and tasks. In addition, classifiers based on uniform prior distributions are
well suited for integration with language models, as described below.
2.4. Lexical models. Our target classes are typically cued by both
lexical and prosodic information; we are therefore interested in optimal
modeling and combination of both feature types . Although in principle
one could add words directly as input features to a prosodic classifier, in
practice this is often not feasible since it results in too large a feature
space for most classifiers. Approaches for cardinality reduction (such as
inferring word classes via unsupervised clustering [4]) offer promise and are
an area we are interested in investigating. To date, however, we have used
statistical language models (LMs) familiar from speech recognition. One
or more LMs are used to effectively model the joint distribution of target
classes S and words W, P(W, S) . With labeled training data, such models
can usually be estimated in a straightforward manner. During testing on
unlabeled data, we compute P(SIW) to predict the possible classes and
their posterior probabilities, or simply to recover the most likely target
class given the words.
2.5. Model combination. The prosodic model may be combined

with a language model in different ways, including
• Posterior interpolation: Compute P(SIF, W) via the prosodic
model and P(SIW) via the language model and form a linear
combination of the two. The weighting is optimized on held-out
data. This is a weak combination approach that does not attempt
to model a more fine-grained structural relationship between the
knowledge sources, but it also does not make any strong assump-
tions about their independence.
• Posteriors as features: Compute P(SIW) and use the LM posterior

estimate as an additional feature in the prosodic classifier. This
approach can capture some of the dependence between the knowl-
edge sources. However, in practice it suffers from the fact that the
LM posteriors on the training data are often strongly biased , and
therefore lead the tree to over-rely on them unless extra held-out
data is used for training.
• HMM-based integration: Compute likelihoods P(FjS, W) from the
prosody model and use them as observation likelihoods in a hid-
den Markov model (HMM) derived from the LM. 2 The HMM is
constructed to encode the unobserved classes S in its state space.
By associating these states with prosodic likelihoods we obtain a
joint model of F, S, and W , and HMM algorithms can be used to
compute the posteriors P(SIF, W) that incorporate all available
knowledge .
This approach models the relationship between words and prosody
at a detailed level, but it does require the assumption that prosody
and words are conditionally independent given the labels S. In
practice, however, this model often works very well even if the
independence assumption is clearly violated.
For a detailed discussion of these approaches, and results showing their
relative success under various conditions, see [12, 9, 15].
3. Applications. Having given a brief overview of the key ideas in our
approach to computational prosody, we now summarize some applications
of the framework.
3.1. Sentence segmentation and disfluency detection. The
framework outlined was applied to the detection of sentence boundaries
and disfluency interruption points in both conversational speech (Switch-
board) and Broadcast News [12, 9]. The target classes S in this case were
labels at each word boundary identifying the type of event : sentence bound-
ary, various types of disfluencies (e.g., hesitations, repetitions, deletions)
and fluent sentence-internal boundaries. The prosodic model was based on
features extracted around each word boundary, capturing pause and phone
durations, FO properties, and ancillary features such as whether a speaker
change occurred at that location.
The LM for this task was a hidden event N-gram, i.e., an N-gram
LM in which the boundary events were represented by tags occurring be-
tween the word tokens. The LM was trained like a standard N-gram model
from tagged training text; it thus modeled the joint probability of tags and
words. In testing, we ran the LM as an HMM in which the states corre-
spond to the unobserved (hidden) boundary events. Prosodic likelihood
2By equating the class distributions for classifier training, as advocated above, we
obtain posterior estimates that are proportional to likelihoods, and can therefore be used
directly in the HMM.
PROSODY MODELING FOR SPEECH 109
scores P( FIS, W) for t he bounda ry events were attached to t hese states

as described above, to condition the HMM tagg ing output on the prosodic
feat ures F.
We tested such a model for combined sentence segmentation and disflu-
ency detection on conversational speech, where it gave about 7% boundary
classification error using correct word transcripts. The results for var ious
knowledge sources based on true and recognized words are summarized in
Table 1 (adapted from [12]). For both test conditions, the prosodic model
improves t he accuracy of an LM-only classifier by about 4% relat ive.
TABLE 1
Sentence boundary and disfluency event tagging error rates for the Switchboard
corpus . The higher chance error rate for recognized words is due to incorrect word
boundary hypotheses.
Model Tru e words Recognized words

LM only 7.3 26.2
Prosody only 11.1 27.1
Comb ined 6.9 25.1
Chance 18.2 30.8
We also carried out a comparative st udy of sentence segmentation

alone, comparing Switchboard (SWB) te lephone conversations to Broad-
cast News (BN) speech. Results are given in Tab le 2 (adapted from [9]).
Again the combination of word and prosodic knowledge yielded the best
results, wit h significant improvements over eit her knowledge source alone.
TABLE 2
Sentence boundary tagging erro r rates for two different speech corpora: Switchboard
(SWB) and Broadcast News (BN) .
SWB BN
Model Tr ue words Rec. words Tr ue words Rec. words
LM only 4.3 22.8 4.1 11.8
P rosod y only 6.7 22.9 3.6 10.9
Combined 4.0 22.2 3.3 10.8
Cha nce 11.0 25.8 6.2 13.3
A striking result in BN segmentation was t hat the prosodic model

alone performed better t ha n t he LM alone. T his was t rue even when
t he LM was using the correct words, and even tho ugh it was trained on
two orders of magnitude more data than the prosody model. Pause du-
ration was universally the most useful feat ure for these tasks; in addition,
SWB classifiers relied primarily on phone duration features, whereas BN
classifiers made considerable use of pitch range features (mainly distance

from the speaker 's estimated baseline). We attribute the increased impor-
tance of pitch features in BN to the higher acoustic quality of the audio
source, and the preponderance of professional speakers with a consistent
speaking style.
3.2. Topic segmentation in broadcast news. A second task we

looked at was locating topic changes in a broadcast news stream, following
the DARPA TDT [3] framework. For this purpose we adapted a baseline
topic segmenter based on an HMM of topic states, each associated with a
unigram LM that models topic-specific word distributions [17] . As in the
previous tagging tasks, we extracted prosodic features around each poten-
tial boundary location, and let a decision tree compute posterior probabil-
ities of the events (in this case, topic changes). By resampling the training
events to a uniform distribution, we ensured that the posteriors are pro-
portional to event likelihoods, as required for HMM integration [9, 15].
The results on this task are summarized in Table 3. We obtained a
large , 24-27% relative error reduction from combining lexical and prosodic
models. Also, similar to BN sentence segmentation, the prosodic model
alone outperformed the LM. The prosodic features selected for topic seg-
mentation were similar to those for sentence segmentation, but with more
pronounced tendencies . For example, at the end of topic segments, a
speaker tends to pause even longer and drop the pitch even closer to the
baseline than at sentence boundaries.
TABLE 3
Topic segmentation weighted error on Broadcast News data. The evaluation metric
used is a weighted combination of false alarm and miss errors [S}.
Model True words Recognized words

LM only 0.1895 0.1897
Combined 0.1377 0.1438
Chance 0.3000 0.3000
3.3. Dialog act labeling in conversational speech. The third

task we looked at was dialog act (DA) labeling. In this task the goal was
to classify each utterance (rather than each word boundary) into a number
of types, such as statement, question, acknowledgment, and backchannel.
In (7] we investigated the use of prosodic features for DA modeling, alone
and in conjunction with LMs. Prosodic features describing the whole ut-
terance were fed to a decision tree . N-gram language models specific to
each DA class provided additional likelihoods. These models can be ap-
plied to DAs in isolation, or combined with a statistical dialog grammar
PROSODY MODELING FOR SPE EC H 111
t hat models t he contextual effects of nearby DAs. In a 42-way classifi-

cation of Switchb oard utterances, t he prosody component impr oved the
overa ll classification accuracy of such a combined model [11]. However , we
found t hat prosodic features were most useful in disambiguatin g certain
DAs t hat are part icularl y confusable based on t heir words alone. Ta ble 4
shows results for two such binar y DA discrimin ation tas ks: disti nguish-
ing questions from statements, and backchann els ("uh-huh" , "right" ) from
agreeme nts ( "Right !") . Again , adding prosody boosted accuracy substan-
t ially over a word-only model. The features used for these and ot her DA
disambiguat ion tas ks, as might be expecte d, depend on t he DAs involved,
as described in [7] .
TA BL E 4
Dialog act classification error on highly ambiguous DA pairs in the Switchboard
corpus.
Classification task Tru e words Rec. words

Knowledge source
Questions vs. Statements
LM only 14.1 24.6
P rosody only 24.0 24.0
Combined 12.4 20.2
Agreements vs. Backchannels
LM only 19.0 21.2
Combined 15.3 18.3
Chance 50.0 50.0
3.4. Word recognition in conversational speech. All applica-

tions discussed so far had the goal of adding st ructural, semantic, or prag-
matic informati on beyond what is contained in the raw word t ra nscripts.
Word recognition itself, however , is still far from perfect , raising the ques-
tion: can prosod ic cues be used to impr ove speech recognition accurac y?
An early approach in this area was [16] , using prosody to evaluate possible
parses for recognized words, which in t urn would be t he basis for reranking
word hypotheses. Recently, t here have been a numb er of approaches t hat
essentially condition t he language model on prosodic evidence, t hereby con-
st ra ining recognition. T he dialog act classification tas k mentioned above
can serve this purpose, since many DA types are characterized by spe-
cific word pat terns . If we can use prosodic cues to predict t he DA of
an ut terance, we can t hen use a DA-specific LM to constrain recognit ion.
Thi s ap proac h has yielded improved recognit ion in task-oriented dialogs
[14], but significant improvements in large-vocabulary recognitio n remain
elusive [11].
We have had some success using the hidden event N-gram model (pre-
viously introduced for sentence segmentation and disfluency detection) for
word recognition [13]. As before, we computed prosodic likelihoods for
each event type at each word boundary, and conditioned the word portion
of the N-gram on those events . The result was a small , but significant 2%
relative reduction in Switchboard word recognition error. This improve-
ment was surprising given that the prosodic model had not been optimized
for word recognition. We expect that more sophisticated and more tightly
integrated prosodic models will ultimately make substantive contributions
to word recognition accuracy.
3.5. Other corpora and tasks. We have recently started applying
the framework described here to new types of data, including multiparty
face-to-face meetings. We have found that speech in multiparty meetings
seems to have properties more similar to Switchboard than to Broadcast
News, with respect to automatic detection of target events [8] . Such data
also offers an opportunity to apply prosody to tasks that have not been
widely studied in a computational framework. One nice example is the
modeling of turn-taking in meetings. In a first venture into this area, we
have found that prosody correlates with the location and form of overlap-
ping speech [8].
We also studied disfluency detection and sentence segmentation in the
meeting domain, and obtained results that are qualitatively similar to those
reported earlier on the Switchboard corpus [1]. A noteworthy result was
that event detection accuracy on recognized words improved slightly when
the models were trained on recognized rather than true words . This indi-
cates that there is systematicity to recognition errors that can be partially
captured in event models.
4. Conclusions. We have briefly summarized a framework for com-
putational prosody modeling for a variety of tasks. The approach is based
on modeling of directly measurable prosodic features and combination with
lexical (statistical language) models . Results show that prosodic informa-
tion can significantly enhance accuracy on several classification and tagging
tasks, including sentence segmentation, disfluency detection, topic segmen-
tation, dialog act tagging, and overlap modeling. Finally, results so far show
that speech recognition accuracy can also benefit from prosody, by con-
straining word hypotheses through a combined prosody/language model.
More information about individual research projects is available
at http://www.speech.srLcom/projects/hidden-events.html, http://www.
speech .sri.com/projects/sleve/, and http://www.clsp.jhu .edu/ws97/dis-
course;'
PROSODY MO DELING FOR SP E EC H 113
REFERENCES
[1] D . BARON, E . SHRIBERG , AN D A. STOLCKE, Automatic punctuation an d disfiuency

detection in multi-party mee tings using prosodic and lexical cues, in Proceed-
ings of the Intern ational Confere nce on Sp oken Lan guage P rocessing, Denv er ,
Sept. 2002.
[2] A . BATLINER, B . MOBIUS, G . MOHLER, A. SCHWEITZER, AN D E . NOTH, Prosodic
models, automatic speech un derstanding, and speech synthesis: toward the
common ground, in Proceed ings of t he 7th European Conference on Sp eech
Communication and Technol ogy, P. Dalsgaard, B. Lindberg , H. Benn er, and
Z. Tan , eds ., Vol. 4 , Aalb org , Denmark, Sept. 2001, pp . 2285-2288.
[3J G . DODDINGTON, Th e Topic Detection and Tracking Phase 2 (TDT2) evaluati on
plan, in Proceedings DARPA Broadcast News Transcription and Understand-
ing Workshop, Lansdowne , VA, Feb. 1998, Morgan Kaufm ann, pp . 223-229.
Revised version available from http) /www.nist .gov/speech/testsjtdt/tdt98/ .
[4] P . HEEMAN AND J . ALLEN, Int ernational boundaries , speech repairs, and discour se
markers: Modeling spoken dialog, in Proceedings of th e 35t h Annual Meeting
and 8th Conference of the European Chapter, Madrid, Ju ly 1997, Association
for Computational Linguist ics.
[5] J . HIRSCHBERG AND C. NAKATANI, Acoustic ind icators of topic segme ntation, in
Proceedings of th e International Confere nce on Spoken Language Processing,
R.H . Mannell and J . Rob ert-Ribes, eds ., Sydney, Dec. 1998, Aust ralian Speech
Scienc e and Technology Associat ion , pp . 976-979.
[6] M. MAST, R KOMPE, S. HARBECK, A. KIESSLING, H . NIEMA NN , E. NOTH, E .G.
SCHUKAT-TALAMAZZINI, AND V . WARNKE, Dialog act classification with the
help of prosody, in Proceed ings of t he Int ern ati on al Conference on Sp oken
Language Processing , H.T. Bunnell and W . Ids ardi, eds., Vol. 3 , Philadelphia ,
Oct . 1996, pp . 1732-1735.
[7] E. SHRIBERG, R BATES, A. STOLCKE, P. TAYLOR, D. J URA FSKY, K . RIES, N. COC-
CARO, R. MARTIN , M. METEER, AND C . VA N Ess-D YKEMA, Can prosody aid
the automatic classification of dialog acts in conversationa l speech?, Lan gu age
an d Speech, 41 (1998), pp . 439-487.
[8] E . SHRIBERG, A. STOLCKE, AND D. BARON, Can prosody aid the automatic pro-
cessin g of mul ti-party mee tings? Evidence from predicting punctuation, dis-
jiuencies, and overlapping speech, in P roceed ings ISCA Tu tori al and Resear ch
Workshop on P rosod y in Speech Recogn iti on and Unde rstanding, M. Bacchi-
ani, J . Hirs chb erg , D. Litman , and M. Ost endorf , eds ., Red Bank, NJ , Oct .
2001, pp . 139-1 46.
[9] E . SHRIBERG , A. STOLCKE, D . HAKKANI-TuR, AND G . T UR, Prosody-based auto-
matic segmentation of speech in to sentences and topics, Speec h Communica-
t ion, 3 2 (2000) , pp. 127-154 . Special Issue on Accessin g Information in Spoken
Audio.
[10] K. SONMEZ, E . SHRIBERG, L. HECK , AND M. WEINTRAU B, Modeling dynamic
prosodic variation for speaker verification, in Proceedings of the Int ernational
Conference on Spoken Language Processing, RH. Mannell and J . Robert-
Ribes , eds ., Vol. 7 , Sydney, Dec. 1998, Australian Speech Science and Tech-
nology Association, pp . 3189-3192.
[11] A . STOLCKE, K. RIES, N . COCCARO, E . SHRIBERG , D . JURAFSKY , P . TAYLOR,
R . MARTIN, C. VAN Es s-DYKEMA , AN D M. METEER, Dialogue act modeli ng
for automatic tagging and recognition of conversational speech, Com putat ional
Lingu istics, 26 (2000), pp . 339- 373.
[12] A . STOLCKE, E. SHRIBERG, R . BATES, M. OSTENDORF, D . HAKKANI , M. PLAUCHE,
G . T UR, AND Y. Lu, Automatic detection of sen tence boundaries and disfi u-
encies based on recognized words, in Proceedings of t he Internati onal Con-
ferenc e on Spoken Lan gu age P rocess ing, RH. Mannell and J . Robert-Ribes,
eds ., Vol. 5 , Sydney, Dec. 1998, Australian Speech Science a nd Techn ology
Association, pp . 2247-2250.
[13] A. STOLCKE , E . SHRIBERG , D. HAKKANI-TuR, AND G . TUR, Modeling the prosody

of hidden events for impro ved word recognition, in Proceedings of the 6th
European Conference on Speech Communication and Technology , Vol. 1, Bu-
dapest, Sept. 1999, pp . 307-310.
[14J P . TAYLOR , S . KING , S . ISARD , AND H . WRIGHT, Intonation and dialog con-
text as constraints for speech recognition, Language and Speech , 41 (1998) ,
pp . 489-508.
[15J G. TUR, D. HAKKANI-T UR, A. STOLCKE , AND E. SHRIBERG, Int egrating prosodic
and lexical cues for automatic topic segmentation, Computational Linguistics,
27 (2001), pp . 31-57.
[16J N.M. VEILLEUX AND M . OSTENDORF , Prosody/parse scoring and its applications
in ATIB, in Proceedings of the ARPA Workshop on Human Language Tech-
nology, Pl ainsboro, NJ, Mar. 1993, pp. 335-340.
[17] J . YAMRON , I. CARP , L. GILLICK, S . LOWE , AND P. VAN MULBREGT, A hidden
Markov model approach to text segmentation and event tracking, in Proceed-
ings of the IEEE Conference on Acoustics, Speech , and Signal Processing,
Vol. I , Seattle, WA, May 1998, pp . 333-336.
SWITCHING DYNAMIC SYSTEM MODELS
FOR SPEECH ARTICULATION AND ACOUSTICS
LI DENC"
Abstract. A statistical generative model for the speech process is described that
embeds a substantially richer structure than the HMM currently in predominant use for
automatic speech recognition. This switching dynamic-system model generalizes and
integrates the HMM and the piece-wise stationary nonlinear dynamic system (state-
space) model. Depending on the level and the nature of the switching in the model
design , various key properties of the speech dynamics can be naturally represented in
the model. Such properties include the temporal structure of the speech acoustics, its
causal articulatory movements, and the control of such movements by the multidimen-
sional targets correlated with the phonological (symbolic) units of speech in terms of
overlapping articulatory features.
On e main challenge of using this multi-level switching dynamic-system model for suc-
cessful speech recognition is the computationally intractable inference (decoding with
confidence measure) on the posterior probabilities of the hidden states. This leads to
computationally intractable optimal parameter learning (training) also . Several versions
of BayesNets have been devised with detailed dependency implementation specified to
represent the switching dynamic-system model of speech. We discuss the variational
technique developed for general Bayesian networks as an efficient approximate algo-
rithm for the decoding and learning problems. Some common operations of estimating
phonological states' switching times have been shared between the variational technique
and the human auditory function that uses neural transient responses to detect temporal
landmarks associated with phonological features. This suggests that the variation-style
learning may be related to human speech perception under an encoding-decoding the-
ory of speech communication, which highlights the critical roles of modeling articulatory
dynamics for speech recognition and which forms a main motivation for the switching
dynamic system model for speech articulation and acoustics described in this chapter.
Key words. State-space model, Dynamic system, Bayesian network, Probabilistic

inference, Speech articulation, Speech acoustics, Auditory function, Speech recognition.
AMS(MOS) subject classifications. Primary 68TIO.
1. Introduction. Speech recognition technology has made dramatic

progress in recent years (cf. [30, 28]), attributed to the use of powerful
statistical paradigms, availability of increasing quantities of speech data
corpus, and to the development of powerful algorithms for model learning
from the data. However, the methodology underlying the current tech-
nology has been founded on weak scientific principles. Not only does the
current methodology require prohibitively large amounts of training data
and lack robustness under mismatch conditions, its performance also falls
at least one order of magnitude short of that of human speech recognition
on many comparable tasks (cf. [32,43]) . For example, the best recognizers
"Microsoft Research, One Microsoft Way, Redmond, WA 98052 (deng@

microsoft.com) . The author wishes to thank many useful discussions with and sug-
gestions for improving the paper presentation by David Heckerman, Mari Ostendorf,
Ken Stevens, B. Frey, H. Attias, C. Ramsay, J . Ma, L. Lee, Sam Roweis, and J . Bilmes.
115
M. Johnsonet al. (eds.), Mathematical Foundations of Speech and Language Processing
116 LI DENG
today still produce errors in more than one quarter of the words in natu-
ral conversational speech in spite of many hours of speech material used as
training data. The current methodology has been primarily founded on the
principle of statistical "ignorance" modeling. This fundamental philosophy
is unlikely to bridge the performance gap between human and machine
speech recognition. A potentially promising approach is to build into the
statistical speech model most crucial mechanisms in human speech com-
munication for use in machine speech recognition. Since speech recognition
or perception in humans is one integrative component in the entire closed-
loop speech communication chain, the mechanisms to be modeled need to
be sufficiently broad - including mechanisms in both speech production
and auditory perception as well as in their interactions.
Some recent work on speech recognition have been pursued along this
direction [6, 18, 13, 17, 46, 47] . The approaches proposed and described in
[1, 5, 49] have incorporated the mechanisms in the human auditory process
in speech recognizer design. The approaches reported in [18, 21, 19, 44, 3,
54] have advocated the use of the articulatory feature-based phonological
units which control human speech production and are typical of human
lexical representation, breaking away from the prevailing use of the phone-
sized, "beads-on-a-string" linear phonological units in the current speech
recognition technology. The approaches outlined in [35, 12, 11, 14, 13] have
emphasized the functional significance of the abstract, "task" dynamics in
speech production and recognition . The task variables in the task dynamics
are the quantities (such as vocal tract constriction locations and degrees)
that are closely linked to the goal of speech production, and are nonlinearly
related to the physical variables in speech production. Work reported and
surveyed in [10, 15, 38, 47] have also focused on the dynamic aspects in
the speech process, but the dynamic object being modeled is in the space
of speech acoustics, rather than in the space of the production-affiliated
variables.
Although dynamic modeling has been a central focus of much recent
work in speech recognition, the dynamic object being modeled either in
the space of "task" variables or of acoustic variables does not and may
not be potentially able to directly take into account the many important
properties in true articulatory dynamics. Some earlier work used [16, 22]
either quantized articulatory features or articulatory data to design speech
recognizers, employing highly simplistic models for the underlying artic-
ulatory dynamics . Some other earlier proposals and empirical methods
exploited pseudo-articulatory dynamics or abstract hidden dynamics for
the purpose of speech recognition [2, 4, 23, 45], where the dynamics of a
set of pseudo-articulators is realized either by FIR filtering from sequen-
tially placed, phoneme-specific target positions or by applying trajectory-
smoothness constraints. Such approaches relied on simplistic nature in
the use of the pseudo-articulators. As a result, compensatory articulation,
which is a key property of human speech production and which requires
MODELS FO R SP EEC H ARTIC ULATION AND ACOUSTI CS 117
modeling correlations among a set of articulators , could not be taken into

account . This has dr astic ally diminished the power of such models for
potentially successful use in speech recognition.
To incorporate crucial properties in human articulatory dynamics -
including compensatory art iculat ion, t arget behavior, and relatively con-
st rained dynamics (due to biomechanical prop erties of the articulatory
orga ns) - in a statistical model of speech, it appears necessary to use
t rue, multidimension al art iculators, rather than the pseudo-articulators at-
tempted in the past. Given that much of the acoustic variation observed in
speech that makes speech recognition difficult can be at t ribute d to art icu-
latory phenomena, and given that articulat ion is one key component in the
closed-loop human speech communication chain, it is reasonable to expect
that incorporating a faithful and explicit articulatory dynamic model in the
statist ical structure of automatic speech recognizer will cont ribute to bridg-
ing the performance gap between human and machine speech recognition.
Based on this motivation, a general framework for speech recognition using
a stat ist ical description of the speech articulation and acoust ic processes
is developed and outlined in this chapter. Central to this fram ework is a
switching dynamic syst em model used to cha racte rize the speech articula-
tion (wit h its cont rol) and the related acoustic pro cesses, and the Bayesian
network (BayesNet ) repr esent ation of this model. Before presenting some
det ails of this model, we first introduce an encoding-decoding t heory of
hum an speech perception which formalizes key roles of modeling speech
articulation.
2. Roles of articulation in encoding-decoding theory of speech
perception. At a global and functional level, hum an speech communica-
tion can be viewed as an encoding-decoding process, where the decodin g
pro cess or perception is an active pro cess consisting of aud itory reception
followed by phoneti c/linguistic interpretation. As an encoder implemented
by t he speech production system, the speaker uses knowledge of meanings
of words (or phrases), of gramm ar in a langu age, and of the sound rep-
resentations for the intended linguistic message. Such knowledge can be
mad e analogous to the keys used in engineering communicat ion systems.
The phonetic plan, derived from the semantic, syntactic, and phonologi-
cal pro cesses, is then execute d through the motor-articulatory system to
produce speech waveforms.
As a decoder which aims to accomplish speech per ception, the list ener
uses a key, or the internal "generative" model , which is compat ible with the
key used by the speaker to interpret the speech signal received and trans-
formed by the peripheral auditory system . This would ena ble the listener to
reconstruct , via (probabilistic) analysis-by-synt hesis st rate gies, the linguis-
t ic message int ended by t he spea ker. 1 This encoding-decoding t heory of
1 Wh ile it is not universally accepted t hat list eners actua lly do a na lysis-by-sy nt hesis
in speech per ception, it would be useful to use such a fram ewor k to int er pret t he roles
118 LIDENG
human speech communication, where the observable speech acoustics plays

the role of the carrier of deep, linguistically meaningful messages, may be
likened to the modulation-demodulation scheme in electronic digital com-
munication and to the encryption-decryption scheme in secure electronic
communication. Since the nature of the key used in the phonetic-linguistic
information decoding or speech perception lies in the strategies used in the
production or encoding process, speech production and perception are in-
timately linked in the closed-loop speech chain. The implication of such a
link for speech recognition technology is the need to develop functional and
computational models of human speech production for use as an "internal
model" in the decoding process by machines. Fig. 1 is a schematic diagram
showing speaker-listener interactions in human speech communication and
showing the several components in the encoding-decoding theory.
FIG. 1. Speaker-listener interactions in the encoding-decoding theory of speech

perception.
The encoding-decoding theory of speech perception outlined above

highlights crucial roles of speech articulation for speech perception. In
summary, the theory consists of three basic, integrated elements : 1) ap-
proximate motor-encoding - the symbolic phonological process interfaced
with dynamic phonetic process in speech production; 2) robust auditory
reception - speech signal transformation prior to the cognitive process;
of articulation in speech perception.

MODELS FOR SPEECH ARTICULATION AND ACOUSTICS 119
3) cognitive decoding - optimal (by statistical criteria) matching of the

auditory transformed signal with the "internal" model derived from a set
of motor encoders distinct for separate speech classes. In this theory, the
"internal" model in the brain of the listener is hypothesized to have been
"approximately" established during the childhood speech acquisition pro-
cess (or during the process of learning foreign languages in adulthood).
The speech production process as the approximate motor encoder in
the above encoding-decoding theory consists of the control strategy of
speech articulation, the actual realized speech articulation, and the acous-
tic signal as the output of the speech articulation system. On the other
hand, the auditory process plays two other key roles. First, it transforms
the acoustic signal of speech to make it robust against environmental vari-
ations. This provides the modified information to the decoder to make its
job easier than otherwise. Second, many transient and dynamic properties
in the auditory system's responses to speech help create temporal land-
marks in the stream of speech to guide the decoding process [50, 53, 54].
(See more detailed discussions on the temporal landmarks in Section 4.3).
As will be shown in this chapter, the optimal decoding using the switch-
ing dynamic system model as the encoder incurs exponentially growing
computation. Use of the temporal landmarks generated from the audi-
tory system's responses may successfully overcome such computational dif-
ficulties, hence providing an elegant approximate solution to the otherwise
formidable computational problem in the decoding.
. In addition to accounting for much of the existing human speech per-
ception data, the computational nature of this theory, with some details
described in the remaining of this chapter with special focus on statisti-
cal modeling of the dynamic speech articulation and acoustic processes,
enables it to be used as the basic underpinning of computer speech recog-
nition systems.
3. Switching state space model for multi-level speech dynam-
ics. In this section, we outline each component of the multi-level speech
dynamic model. The model serves as a computational device for the ap-
proximate encoder in the encoding-decoding theory of speech perception
outlined above. We provide motivations for the construction of each model
component from principles of speech science, present a mathematical de-
scription of each model component, and justify assumptions made to the
mathematical description. The components in the overall model consists
of a phonological model, a model for the segmental target, a model for the
articulatory dynamics, and a model for the mapping from articulation to
acoustics. We start with the phonological-model component.
3.1. Phonological construct. Phonology is concerned with sound
patterns of speech and the nature of discrete or symbolic units that form
such patterns. Traditional theories of phonology differ in the choice and
interpretation of the phonological units . Early distinctive feature based
120 LIDENG
theory [8] and subsequent autosegmental, feature-geometry theory [9] as-

sumed a rather direct link between phonological features and their phonetic
correlates in the articulatory or acoustic domain . Phonological rules for
modifying features represented changes not only in the linguistic structure
of the speech utterance, but also in the phonetic realization of this struc-
ture. This weakness has been recognized by more recent theories, e.g.,
articulatory phonology [7], which emphasize the importance of accounting
for phonetic levels of variation as distinct from those at the phonological
levels.
In the framework described here, it will be assumed that the linguistic
function of phonological units is to maintain linguistic contrasts and is
separate from phonetic implementation. It is further assumed that the
phonological unit sequence can be described mathematically by a discrete-
time, discrete-state homogeneous Markov chain. This Markov chain is
characterized by its state transition matrix A = [aij] where aij = P(Sk =
j!Sk-l = i).
How to construct sequences of symbolic phonological units for any ar-
bitrary speech utterance and how to built them into an appropriate Markov
state (i.e., phonological state) structure will not be dealt with here. We
merely mention that for effective use of the current framework in speech
recognition, the symbolic units must be of multiple dimensions that overlap
with each other temporally, overcoming beads-on-a-string limitations. We
refer the readers to some earlier work for ways of constructing such over-
lapping units, either by rules or by automatic learning, which have proved
effective in the HMM-like speech recognition framework [21, 19, 18, 56] .
3.2. Articulatory control and targets. After a phonological model

is constructed, the processes for converting abstract phonological units into
their phonetic realization need to be specified. This is a central issue
in speech production. It concerns the nature of invariance and variabil-
ity in the processes interfacing phonology and phonetics , and specifically,
whether the invariance is more naturally expressed in the articulatory or
acoustic/auditory domains. Early proposals assumed a direct link between
abstract phonological units and physical measurements. The "quantal the-
ory" [53] proposed that phonological features possessed invariant acoustic
correlates that could be measured directly from the speech signal. The
"motor theory" [31] proposed instead that articulatory properties are as-
sociated with phonological symbols. No conclusive evidence supporting
either hypothesis has been found without controversy, however.
In the current framework, a commonly held view in the phonetics
literature is adopted that discrete phonological units are associated with a
temporal segmental sequence of phonetic targets or goals [34, 29, 40, 41, 42].
The function of the articulatory motor control system is to achieve such
targets or goals by manipulating the articulatory organs according to some
control principles subject to the articulatory inertia and possibly minimal-

energy constraints.
Compensatory articulation has been widely documented in the pho-
netics literature where trade-offs between different articulators and non-
uniqueness in the articulatory-acoustic mapping allow for the possibilities
that many different articulatory target configurations may be able to real-
ize the same underlying goal, and that speakers typically choose a range
of possible targets depending on external environments and their interac-
tions with listeners [29]. In order to account for compensatory articulation,
a complex phonetic control strategy need be adopted. The key modeling
assumptions adopted regarding such a strategy is as follows. First, each
phonological unit is associated with a number of phonetic parameters that
are described by a state-dependent distribution. These measurable param-
eters may be acoustic, articulatory or auditory in nature, and they can
be computed from some physical models for the articulatory and audi-
tory systems. Further, the region determined by the phonetic correlates
for each phonological unit can be mapped onto an articulatory parame-
ter space . Hence the target distribution in the articulatory space can be
determined simply by stating what the phonetic correlates (formants, artic-
ulatory positions, auditory responses, etc .) are for each of the phonological
units (many examples are provided in [55]), and by running simulations in
suitably-detailed articulatory and auditory models.
A convenient mathematical representation for the distribution of the
articulatory target vector t is a multivariate Gaussian distribution, de-
noted by
t rv N(tj m(s), ~(s)).
Since the target distribution is conditioned on a specific phonological unit

(such as a bundle of overlapped features represented by an HMM state s)
and since the target does not switch until the phonological unit changes,
the statistics for the temporal sequence of the target process follows that
of a segmental HMM. A most recent review of the segmental HMM can
found in [26].
3.3. Articulatory dynamics. At the present state of knowledge, it
is difficult to speculate how the conversion of higher-level motor control
into articulator movement takes place. Ideally, modeling of articulatory
dynamics and control would require detailed neuromuscular and biome-
chanical models of the vocal tract, as well as an explicit model of the
control objectives and strategies. This is clearly too complicated to imple-
ment . A reasonable, simplifying assumption would be that the combined
(non-linear) control system and articulatory mechanism behave, at a func-
tional level, as a linear dynamic system that attempts to track the control
input equivalently represented by the articulatory target in the articula-
tory parameter space. Articulatory dynamics can then be approximated
122 LI DENG
as the response of a dynamic vocal tract model driven by a random target

sequence (as a segmental HMM). (The output of the vocal tract model
then produces a time-varying tract shape which modulates the acoustic
properties of the speech signal as observed data.)
This simplifying assumption then reduces the generic nonlinear state
equation:
z(k + 1) = gs[z(k), t s, w(k)]
into a mathematically tractable linear one:
(3.1) z(k + 1) = <Jlsz(k) + (1 - <Jls)t s + w(k),

where z E R" is the articulatory-parameter vector, 1 is the identity ma-
trix, w is the IID and Gaussian system noise (w(k) N[w(k) ; 0, QSk])' t s
"J
is the HMM-state dependent, target vector (expressed in the articulatory

domain) , and <Jl s is the HMM-state-dependent system matrix. The depen-
dence of the t., and <Jl s parameters of the above dynamic system on the
phonological state is justified by the fact that the functional behavior of an
articulator depends on the particular goal it is trying to implement, and
on the other articulators with which it is cooperating in order to produce
compensatory articulation.
3.4. Acoustic model. While a truly consistent framework we are
striving for based on explicit knowledge of speech production and percep-
tion ideally should include detailed high-order state-space models of the
physical mechanisms involved, this becomes unfeasible due to excessive
computational requirements. The simplifying assumption adopted is that
the articulatory and acoustic state of the vocal tract can be adequately
described by low-order vectors of variables representing respectively the
relative positions of the major articulators, and the corresponding time-
averaged spectral parameters derived from the acoustic signal (or other
parameters computed from auditory models) . Given further that an ap-
propriate time scale is chosen, it will also be assumed that the relationship
between articulatory and acoustic representations can be modeled by a
static memory less transformation, converting a vector of articulatory pa-
rameters into a vector of acoustic (or auditory) measurements.
This noisy static memoryless transformation can be mathematically
represented by the followingobservation equation in the state-space model:
(3.2) o(k) = h[z(k)] + v(k).

where 0 E n m is the observation vector, v is the IID observation noise
vector (v(k) rv N[v(k) ; 0, R]) uncorrelated with the state noise w, and h[.]
is the static memoryless transformation from the articulatory vector to its
corresponding acoustic observation vector.
There are many ways of choosing the static nonlinear function for h[z] .
Let us take an example of multi-layer perceptron (MLP) with three layers
(input, hidden and output) . Let Wjl be the MLP weights from input to
hidden units and Wij be the MLP weights from hidden to output units,
where l is the input node index, j the hidden node index and i the output
node index. Then the output signal at node i can be expressed as a (non-
linear) function h( .) of all the input nodes (making up the input vector)
according to
(3.3) hi(z) = tw
j=l
ij . s(tWjl'
1=1
ZI) ' 1:S i:S I ,
where I, J and L are the numbers of nodes at the output, hidden and input
layers, respectively. s(.) is the hidden unit's nonlinear activation function,
taken as the standard sigmoid function of
1
(3.4) s( z) - ----;----:-
-1+exp(-z)
The derivative of this sigmoid function has the following concise form:
(3.5) s'(») = s(z)(1 - s(z)),

making it convenient for use in many computations.
Typically, the analytical forms of nonlinear functions, such as the MLP,
make the associated nonlinear dynamic systems difficult to analyze and
make the estimation problems difficult to solve. Approximations are fre-
quently used to gain computational simplifications while sacrificing accu-
racy for approximating the nonlinear functions.
One most commonly used technique for the approximation is the trun-
cated (vector) Taylor series expansion. If all the Taylor series terms of order
two and higher are truncated, then we have the linear Taylor series approx-
imation that is characterized by the Jacobian matrix J and the point of
Taylor series expansion Zo:
(3.6) h(z) ~ h(zo) + J(zo)(z - zo).
Each element of the Jacobian matrix J is partial derivative of each vector

component of the nonlinear output with respect to each of the input vector
components. That is,
8hdzo) 8hdzo) 8hdzo)
~ 8Z2 . . . az;;:-
8h 2(zo) 8h2(ZO) 8h2(ZO)
8h ~ 8Z2 . . . az;;:-
(3.7) J(zo) = - =
8zo
124 LIDENG
As an example, for the MLP nonlinearity of Eqn. 3.3, the (i, l)-th element
of the Jacobian matrix is
J
(3.8) L W ij . Sj(Y) . (1 - Sj(Y)) . Wjl, 1 :s i < I, 1:S l :s L,
j=l
where Y = 2:~=1 Wjl'Zll.

Use of the radial basis function as the nonlinearity in the general
nonlinear dynamic system model, as an alternative to the MLP described
above, can be found in [24].
3.5. Switching state space model. Eqns. 3.1 and 3.2 form a spe-
cial version of the switching state-space model appropriate for describing
multi-level speech dynamics. The top-level dynamics occurs at the discrete-
state phonology, represented by the state transitions of S with a relatively
long time scale. The next level is the target (t) dynamics ; it has the same
time scale and provides systematic randomness at the segmental level. At
the level of articulatory dynamics, the time scale is significantly shortened.
This is continuous-state dynamics driven by the target process as input,
which follows HMM statistics. The state equation 3.1 explicitly describes
this dynamics in z, with index of S (which takes discrete values) implic-
itly representing the switching process. At the lowest level of acoustic
dynamics, there is no switching process. Since the observation equation
3.2 is static, this simplifying speech model assumes that acoustic dynamics
results solely from articulatory dynamics.
4. BayesNet representation of the segmental switching dy-
namic speech model. Developed traditionally by machine-learning re-
searchers, BayesNets have found many useful applications. A BayesNet
is a graphical model that describes dependencies in the probability dis-
tributions defined over a set of variables. A most interesting class of the
BayesNet, as relevant to speech modeling, is dynamic BayesNets that are
specifically aimed at modeling time series statistics. For time series data
such as speech vector sequences, there are causal dependencies between
random variables in time . The causal dependencies give some specific,
left-to-right BayesNet structures. Such specific structures either permit
development of highly efficient algorithms (e.g., for the HMM) for the prob-
abilistic inference (i.e., computation of conditional probabilities for hidden
variables) and for learning (i.e., model parameter estimation), or enable
the use of approximate techniques (such as variational techniques) to solve
the inference and learning problems .
Both the HMM and the stationary (Le., no switching) dynamic sys-
tem model are two of the simplest examples of a dynamic BayesNet, for
which the efficient algorithms developed already in statistics and in speech
processing [51, 38, 20] turn out to be identical to those based on the more
MODELS FOR SPEECH ART ICU LAT ION AND ACOUSTICS 125
general principles of BayesNet theory applied to the special network struc-

tures associated with these models. However , for th e more complex speech
model such as the switching dynamic syste m model described above, no ex-
act solut ions for inference and learning are available without exponentially
growing computation with the size of t he dat a. Approxim ate solutions
have been provided for some simple versions of the the switching dynamic
system model in literatures of statistics [52], speech processing [33], and
of neural computation and BayesNet [25, 39J. T he BayesNet fra mework
allows us to take a fresh view on the complex computational issues for such
a model, and provides guidance and insights to the algorit hm development
as well as model refinement.
4.1. Basic BayesNet model. We now discuss how t he particular

multi-component speech model described in Section 3 can be represent ed
and implement ed by BayesNets. Fig. 2 shows one ty pe of depe ndency struc-
ture (indicat ed by the direction of arrows) of th e model, where (discrete)
t ime index runs from left to right . The t op-row random variables s(k)
take discrete values over the set of phonological states (overlap ped feature
bundles), and th e remaining random varia bles for t he targets, articulato rs,
and acoustic vectors are continuously valued for each time index.
F IG . 2. Dynamic Baye sNet for a basic version of the switching dynamic system
model of speech. The mndom variables on Row 1 are discrete, hidden linguistic state s
with the Markov- chain tempoml structure. Those on Row 2 are continuous, hidden
arti culatory targets as ideal articulation . Those on Row :3 are continuous, hidden states
represen ting physical articulat ion with the Markov tempoml struc ture also. Thos e on
Row 4 are continuous, observed acousti c/auditory vectors.
126 LI DENG
Each dependency in the above BayesNet can be implemented by speci-

fying the associated conditional probability. In the speech model presented
in Section 3, the horizontal (temporal) dependency for the phonological
(discrete) states is specified by the Markov chain transition probabilities:
(4.1)
The vertical (level)2 dependency for the target random variables is specified
by the following conditional density function:
(4.2)
Possible structures in the covariance matrix ~(Sk) in the above target

distribution can be explored using physical interpretations of the targets
as idealized articulation. For example, the velum component is largely
uncorrelated with other components; so is the glottal component. On the
other hand, tongue components are correlated with each other and with
the jaw component. For some linguistic units (lui for instance) , some
tongue components ar e correlated with the lip components. Therefore, th e
covariance matrix ~(Sk) has a block diagonal structure. If we represent
each component in the target vector in the BayesNet, then each target
node in Fig. 2 will contain a sub-network.
The joint horizontal and vertical dependency for the articulatory (con-
tinuous) state is specified, based on state equation 3.1, by the conditional
density function:
The vertical dependency for the observation random variables is speci-

fied, based on observation equation 3.2, by the conditional density function:
Po[o(k)lz(k)] = Pv[o(k) - h(z(k))]

(4.4)
= N[o(k)); h(z(k)), R] .
Eqns. 4.1, 4.2, 4.4, and 4.5 then completely specify the switching dynamic
model in Fig. 2 since they define all possible dependencies in its BayesNet
representation. Note that while the phonological state Sk and its associ-
ated target t(k) in principle are at a different time scale than the phonetic
variables z(k) and o(k), for simplicity purposes and as one possible imple-
mentation, Eqns. 4.1-4.5 have placed them at the same time scale.
Note also that in Eqn . 4.5 the "forward" condit ional probability for
the observation vector (when the corresponding articulatory vector z(k) is
known) is Gaussian, as is the measurement noise vector's distribution. The
2This refers to the level of the speech production chain as t he "encoder" .

mean of the Gaussian is the prediction of the nonlinear function h(z(k)) .

However, the "inverse" or "inference" conditional probability p[z(k)lo(k)]
will not be Gaussian due to the nonlinearity of h( .) as well as the switching
process that controls the dynamics in z(k). The fact that the conditional
distribution for z(k) is not Gaussian is one major source of difficulty for
the inference and learning problems associated with the nonlinear switching
dynamic system model.
4.2. Extended BayesNet model. One modification and extension

of the basic BayesNet model of Fig. 2 is to explicitly represent parallel
streams of the overlapping phonological features and their associated artic-
ulatory dimensions. As discussed in Section 3.1, the phonological construct
of the model consists of multidimensional symbols (feature bundles) over-
lapping in time. The BayesNet for this expanded model is shown in Fig. 3,
where the individual components of the articulator vector from the paral-
lel overlapping streams are ultimately combined to generate the acoustic
vectors.
FIG. 3. Dynamic BayesNet for an expanded version of the switching dynamic sys-
tem model of speech. Pamllel streams of the overlapping phonological features and their
associated articulatory dimensions are explicitly represented. The articulators from the
pamllel streams are ultimately combined to jointly determine the acoustic vectors .
128 LIDENG
Another modification of the basic Bayesian-net model of Fig. 2 is to

incorporate the segmental constraint on the switching process for the dy-
namics of the target random vector t(k). That is, while random, t(k)
remains fixed until the phonological state Sk switches. The switching of
target t(k) is synchronous with that of the phonological state, and only at
the time of switching, t(k) is allowed to take a new value according to its
probability density function . This segmental constraint can be described
mathematically by the following conditional probability density function:
8[t(k) - t(k - 1)] if Sk = Sk-l,

p[t(k)lsk , Sk-l, t(k - 1)] =
{ N(t(k); m(sk), :E(Sk)) otherwise.
This adds the new dependency of random vector of t( k) on Sk-l and t(k-
1), in addition to the existing Sk as in Fig. 2. The modified BayesNet
incorporating this new dependency is shown in Fig. 4.
FIG. 4. Dynamic BayesNet for the switching dynamic system model of speech
incorporating the segmental constraint for the target random variables.
4.3. Discussions. Given the BayesNet representations of switching

dyn amic system models for speech, rich tools for approximate inference and
learning can be exploited and further developed . Since the exact inference
is impossible, at least in theory, the success of applying such a model to
speech recognition crucially depends on the accuracy of the approximate
algorithms.
It is worth noting that while the exact optimal inference for the phono-
logical states (the speech recognition problem) has exponent ial compl ex-
ity in computation, once the approximate times of the switching in the
phonological states become known, computational complexity can be sub-
st antially reduced . With the applicat ion of the variational te chnique (e.g.,
[27]) developed for BayesNet inference and learning to some generic, un-
structured versions of the switching state-space mod el [39, 25]), one can
separ ate the discrete states from the remaining portion of the network. (Re-
cent research [37J also provides evidence that approximate methods such
variat ional learning work well for a speech model called loosely-coupled
HMM.) For the structured switching state-space model of speech dyn amics
as pres ented in this pap er, this allows one to iteratively est imate the poste-
rior distributions of t he discrete phonological and continuous art iculatory
state s. Infer ence on the phonologic al states becomes essent ially a search
for the state swit ching times with soft decisions. For exa mple, when one
uses the Gaussian mixture distribution to approximate t he t rue posteriors
in the speech model discuss ed so far , the E-step (needed for the recog-
nizer's MAP decoding procedure) in the variational EM algorithm can be
shown to be a solut ion to a set of algebr aic nonlin ear system of equations.
Achieving efficient and accurate solutions to these closely coupl ed equa-
tions for the purpose of decoding the optimal phonological st ate sequence
ca n be greatly facilitated when some crude esti mates (e.g., within the rang e
of sever al frames) of the phonological state boundaries, which we call t he
landmarks , are made available.
Interestingly, such an important role of the phonological state bound-
ary estimates fits closely with the encoding-decoding theory of speech per-
cept ion outlined in Section 2. As we discussed in Section 2, one crucial
role of auditory reception for human speech perception is to provide tern-
por allandmarks for the phonological features via the many transient neu-
ral response properties in th e auditory system [50, 53, 54]. Recall that
in the switching dyn amic system model of speech pr esent ed in this pa-
per , the phonological units are repr esented not in t erms of phones that
consist of a bundle of synchronously aligned features, but in terms of in-
dividual features. Therefor e, the temporal landmarks associated with the
individual features that may be detected by transient neur al responses in
the auditory system have important functional roles to play in providing
the crude boundary information to facilitate the decoding of phonological
st ates (speech perception). This common operation performed by the au-
ditory system and by one aspect of the vari ational technique suggest s that
130 LIDENG
the variational-style decoding algorithms may be closely related to human

speech perception.
5. Summary and discussions. We outlined an encoding-decoding

theory of speech perception in this chapter, which highlights the importance
and critical role of modeling articulatory dynamics in speech recognition .
This is an integrated motor-auditory theory where the motor or produc-
tion system provides the internal model for the listener's speech decoding
device, while the auditory system provides sharp temporal landmarks for
phonological features to constrain the decoder's search space and to mini-
mize possible loss of decoding accuracy.
Most of current speech systems are very fragile. For further progress
in the field, the author believes that it is necessary to bring in human-
like intelligence of speech perception into computer systems. The switch-
ing dynamic system models discussed in this chapter offer one powerful
mathematical tool for implementing the encoding-decoding mechanism of
human speech communication. We have shown that the BayesNet frame-
work allows us to take a fresh view on the complex computational issues in
inference (decoding) and in learning, and to provide guidance and insights
to the algorithm development .
It is hoped that the framework presented here will help integrate re-
sults from speech production and advanced machine learning within the
statistical paradigms for speech recognition . An important, long-term goal
will involve development of computer systems to the extent that they can
be evaluated efficiently on realistic, large speech databases, collected in a
variety of speaking styles (conversational styles in particular) and for a
large population of speakers .
The ultimate goal of the research, whose components are described
in some detail in this chapter, is to develop high-performance systems
for integrated speech analysis, coding, synthesis, and recognition within
a consistent statistical framework. Such a development is guided by the
encoding-decoding theory of human speech communication, and is based on
computational models of speech production and perception. The switch-
ing dynamic system models of speech and their BayesNet representations
presented are a significant extension of the current highly simplified statis-
tical models used in speech recognition. Further advances in this research
direction will require greater integration within a statistical framework of
existing research in modeling speech production, speech recognition, and
advanced machine learning.
REFERENCES
[1] J . ALLEN. "How do humans process and recognize speech," IEEE Trans . Speech
Audio Proc., Vol. 2, 1994, pp . 567-577.
[2] R . BAKIS. "Coart iculat ion modeling with continuous-state HMMs, " Proc. IEEE
Workshop Automatic Speech Recognition, Harriman, New York, 1991, pp .
20-21.
[3] N. BITAR AND C . ESPy-WILSON . "Speech parameterization based on phonetic fea-
tures: Application to speech recognition," Proc. Eurospeech, Vol. 2 , 1995, pp.
1411-1414.
[4] C. BLACKBURN AND S. YOUNG . "Towards improved speech recognition using a
speech production model ," Proc. Eurospeech ; Vol. 2 , 1995, pp . 1623-1626.
[5] H. BOURLARD AND S. DUPONT. "A new ASR approach based on independent pro-
cessing and recombination of partial frequency bands," Proc. ICSLP, 1996,
pp . 426-429.
[6] H. BOURLARD , H. HERMANSKY, AND N. MORGAN . "Towards increasing speech
recognition error rates," Speech Communication, Vol. 18 , 1996, pp . 205-231.
[7] C. BROWMAN AND L. GOLDSTEIN . "Art iculatory phonology: An overview, " Pho-
netica, Vol. 49 , pp . 155-180, 1992.
[8] N. CHOMSKY AND M. HALLE. The Sound Pattern of English, New York: Harper
and Row, 1968.
[9] N. CLEMENTS. "T he geometry of phonological features," Phonology Yearbook, Vol.
2, 1985, pp . 225-252 .
[10] L. DENG . "A generalized hidden Markov model with state-conditioned trend func-
tions of time for th e speech signal," Signal Processing, Vol. 27, 1992, pp . 65-78 .
[11] L. DENG . "A computational model of the phonology-phonetics interface for auto-
matic speech recognition," Summary Report, SLS-LCS , Massachusetts Insti-
tute of Technology, 1992-1993 .
[12] L. DENG . "Design of a feature-based speech recognizer aiming at integration of
auditory processing, signal modeling, and phonological structure of speech."
J. Acoust . Soc. Am., Vol. 93, 1993, pp . 2318.
[13] L. DENG . "Computational models for speech production," in Computational Mod-
els of Speech Pattern Processing (NATO ASI), Springer-Verlag, 1999, pp .
67-77.
[14J L. DENG . "A dynamic, feature-based approach to the interface between phonology
and phonetics for speech modeling and recognition," Speech Communication ,
Vol. 24 , No.4, 1998, pp. 299-323.
[15J L. DENG , M. AKSMANOVIC , D. SUN , AND J. Wu. "Speech recognition using hidden
Markov models with polynomial regression functions as nonstationary states,"
IEEE Trans. Speech Audio Proc., Vol. 2, 1994, pp . 507-520.
[16] L. DENG , AND K. ERLER. "Structural design of a hidd en Markov model based
speech recognizer using multi-valued phonetic features: Comparison with seg-
mental speech units," J. Acoust. Soc. Am. , Vol. 92 , 1992, pp . 3058-3067.
[17] L. DENG AND Z. MA. "Spont aneous speech recognition using a statistical coarticu-
latory model for th e hidden vocal-tract-resonance dynamics," J. Acoust. Soc.
Am., Vol. 108, No.6, 2000, pp . 3036-3048.
[18] L. DENG, G. RAMSAY, AND D. SUN . "Production models as a structural basis for
automatic speech recognition," Speech Communication, Vol. 22 , No.2, 1997,
pp . 93-11 1.
[19] L. DENG AND H. SAMET!. "Transitional speech units and their representation by the
regressive Markov states: Applications to speech recognition," IEEE Trans .
Speech Audio Proc., Vol. 4, No.4, July 1996, pp . 301-306 .
[20] L. DENG AND X. SHEN. "Max imum likelihood in st atistical estimation of dynamic
systems: Decomposition algorithm and simulation results" , Signal Processing,
Vol. 57, 1997, pp . 65-79 .
[21] L. DENG AND D. SUN . "A statistical approach to automatic speech recognition
using the atomic speech units constructed from overlapping articulatory fea-
tures," J. Acoust . Soc. Am ., Vol. 95, 1994, pp . 2702-2719.
[22] J . FRANKEL AND S. KING . "ASR - Articulatory speech recognition" , Proc. Eu-
rospeech, Vol. 1, 2001, pp . 599-602 .
132 LIDENG
[23J Y. GAO, R. BAKIS, J. HUANG , AND B. ZHANG , "Multistage coarticulation mod el

combining articulatory, formant and cepstral features" , Proc. ICSLP, Vol. 1,
2000, pp. 25-28 .
[24J Z. GHAHRAMANI AND S. ROWElS. "Learning nonlinear dynamic systems using an
EM algorithm" . Advances in Neural Information Processing Systems, Vol. 11 ,
1999, 1-7.
[25] Z. GHAHRAMANI AND G. HINTON. "Variational learning for switching state-space
model" . Neural Computation, Vol. 12, 2000, pp . 831-864.
[26] W . HOLMES. "Segmental HMMs: Modeling dynamics and underlying structure
in speech, " in M. Ostendorf and S. Khudanpur (eds .) Mathematical Founda-
tions of Speech Recognition and Processing, Volume X in IMA Volumes in
Mathematics and Its Applications, Springer-Verlag, New York, 2002.
[27] M. JORDAN , Z. GHAHRAMANI, T . JAAKKOLA , AND L. SAUL. "In introduction to
variational methods for graphical models," in Learning in Graphical Models
M. Jordon (ed .), The MIT Press, Cambridge, MA, 1999.
[28] F. JUANG AND S. FURUI (eds.), Proc. of the IEEE (special issue) , Vol. 88 , 2000.
[29] R. KENT, G. ADAMS, AND G. TURNER. "Models of speech production," in Prin-
ciples of Experimental Phonetics, N. Lass (ed.), Mosby: London, 1995, pp.
3-45.
[30] C.-H . LEE, F . SOONG , AND K. PALIWAL (eds.) Automatic Speech and Speaker
Recognition - Advanced Topics, Kluwer Academic, 1996.
[31J A. LIBERMAN AND I. MATTINGLY. "The motor theory of speech perception revised "
Cognition , Vol. 21 , 1985, pp . 1-36 .
[32] R. LIPPMAN. "Speech recognition by human and machines," Speech Communica-
tion, Vol. 22, 1997, pp . 1-15.
[33] Z. MA AND L. DENG . "A path-stack algorithm for optimizing dynamic regimes in a
statistical hidden dynamic model of speech ," Computer Speech and Language,
Vol. 14 , 2000, pp. 101-104.
[34] P . MACNEILAGE. "Motor control of serial ordering in speech ," Psychological Re-
view, Vol. 77, 1970, pp . 182-196 .
[35] R. MCGOWAN . "Recovering articulatory movement from formant frequen cy trajec-
tories using task dynamics and a genetic algorithm: Preliminary model tests,"
Speech Communication, Vol. 14, 1994, pp . 19-48.
[36] R. MCGOWAN AND A. FABER. "Speech production parameters for automatic speech
recognition," J. Acoust. Soc. Am., Vol. 101 , 1997, pp . 28.
[37] H. NOCK. Techniques for Modeling Phonological Processes in Automatic Speech
Recognition , Ph.D. thesis, Cambridge University, 2001, Cambridge, U.K.
[38J M. OSTENDORF , V. DIGALAKIS , AND J . ROHLICEK. "From HMMs to segment mod-
els: A unified view of stochastic modeling for speech recognition" IEEE Trans .
Speech Audio Proc., Vol. 4 , 1996, pp . 360-378 .
[39J V. PAVLOVIC, B. FREY , AND T . HUANG . "Variat ional learning in mixed-state dy-
namic graphical models," Proc. Annual Conf. in Uncertainty in Artificial In-
telligence, 1999, UAI-99.
[40J J . PERKELL, M. MATTHIES, M. SVIRSKY, AND M. JORDAN. "Goal-based speech mo-
tor control: a th eoretical framework and some preliminary data, " J. Phonetics,
Vol. 23 , 1995, pp . 23-35 .
[41] J . PERKELL. "Properties of the tongue help to define vowel categories: hypotheses
based on physiologically-oriented modeling," J. Phon etics Vol. 24 , 1996, pp.
3-22.
[42J P . PERRIER, D . OSTRY, AND R. LABOISSIERE. "T he equilibrium point hypothesis
and its application to speech motor control," J. Speech fj Hearing Research,
Vol. 39, 1996, pp. 365-378.
[43J L. POLS . "Flexible human speech recognition," Proceedings of the 1997 IEEE
Workshop on Automatic Speech Recognition and Understanding, Santa Bar-
bara, 1997, pp . 273-283.
[44] M. RANDOLPH. "Speech analysis based on articulatory behavior," J. Acoust. Soc.

Am. , Vol. 95, 1994, pp . 195.
[45] H. RICHARDS, AND J . BRIDLE. "The HDM: A segmental hidden dynamic model of
coarticulation", Proc. ICASSP, Vol. 1, 1999, pp . 357-360.
[46J R . ROSE, J . SCHROETER, AND M. SONDHI. "T he potential role of speech production
models in automatic speech recognition," J. Acoust . Soc. Am., Vol. 99 , 1996,
pp . 1699-1709.
[47] M. RUSSELL. "P rogress towards speech models that model speech ," Proceedings of
the 1997 IEEE Workshop on Automatic Speech Recognition and Understand-
ing, Santa Barbara, 1997, pp . 115-123.
[48] J . SCHROETER AND M. SONDHI. "Techniques for estimating vocal-tract shapes
from '" he speech signal ," IEEE Trans . Speech Audio Proc., Vol. 2, 1994, pp .
133-150 .
[49] H . SHEIKHZADEH AND L. DENG. "Speech analysis and recognition using interval
statistics generated from a composite auditory model ," IEEE Trans . Speech
Audio Proc., Vol. 6, 1998, pp. 50-54.
[50] H. SHEIKHZADEH AND L. DENG . "A layered neural network int erfaced with a
cochlear model for the study of speech encoding in the auditory system,"
Computer Speech and Language, Vol. 13, 1999, pp . 39-64.
[51] R. SHUMWAY AND D. STOFFER. "An approach to time series smoothing and fore-
casting using the EM algorithm," J. Tim e Series Analysis, Vol. 3 , 1982, pp .
253-264.
[52] R . SHUMWAY AND D. STOFFER. "Dynamic linear models with switching" , J. Amer-
ican Statistical Association, Vol. 86, 1991, pp . 763-769 .
[53] K. STEVENS. "On the quantal nature of speech, " J. Phonetics, Vol. 17, 1989, pp .
3-45.
[54] K. STEVENS . "From acoustic cues to segments, features and words ," Proc. ICSLP,
Vol. 1, 2000, pp . AI -A8.
[55] K. STEVENS . Acoustic Phonetics, The MIT Press , Cambridge, MA, 1998.
[56] J. SUN , L. DENG , AND X. JING . "Dat a-d riven model construction for continu-
ous speech recognition using overlapping articulatory features," Proc. ICSLP,
Vol. 1, 2000, pp . 437-440.
SEGMENTAL HMMS: MODELING DYNAMICS AND
UNDERLYING STRUCTURE IN SPEECH
WENDY J. HOLMES"
Abstract. The mot ivat ion und erlying th e development of segment al hidd en Markov
mod els (SHMMs) is to overcome import ant sp eech-modeling limit at ions of conventional
HMMs by representing sequences (or 'seg ments') of features and incorporating the con-
cept of a trajectory to describe how features change over time. This pap er presents an
overview of investigations that have been carried out into the properties and recognition
performa nce of various SHMMs , highlighting some of the issues th at have been ident ified
in using these models successfully. Recognition resul ts are presented showing t hat t he
best recognition performance was obt ained when combining a trajectory model with a
form ant representation, in comparison both with a conventional cepstrum-based HMM
syst em and with syst ems that incorporated eit her of the developments individually.
An attractive characteristic of a formant-based trajectory model is that it applies
easily to speech synthesis as well as to speech recognition, and thus can provide the
basis for a 'unified' approach to both recognition and synthesis and t o spee ch modeling
in gener al. One practical application is in very low bit-rate sp eech coding, for which
a form ant trajectory description provides a compact means of coding an utterance . A
demonstration system has been developed that typically codes sp eech at 600-1000 bits/s
with good intelli gibility, whilst preserving speaker characteristics.
Key words. Segment al HMM , dynamics, trajectory, formant, unified model , low
bit-r ate speech coding.
AMS(MOS) subject classifications . Primary 1234, 5678, 9101112.
1. Intro duction. Th e acoust ic-phonet ic components of the most sue-

cessful large-vocabulary automat ic recognition (ASR) syste ms to date are
almost exclusively based on hidden Markov models (HMMs) of some pho-
netically defined subword units, ty pically using a large inventory of context-
dependent phone models th at are trained on a vast quantity of speech data.
The models thems elves tend to be gender-dependent but are otherwise
'speaker-independent', alt hough th ere is often on-line adapt at ion to any
particular speaker . Using thi s type of approach, impressive performance
has been achieved on recognition of read speech. For example, in the 1998
US Defense Advanced Resear ch Proj ects Agency (DARPA) evaluat ions us-
ing broad cast news mat erial, for th e 'planned' portion of th e test set (read
speech in quiet conditions) a word error rate of 7.8% was obtained [21] .
However, the general level of performance drops if the recording condit ions
and speaking style are less controlled, with th e lowest error rate on the
spontaneous portion of the 1998 test set being 14.4% [21] . Conversational
speech is particularly challenging, especially when the conversations are
between individu als who know each other very well, when the percentage
of recognition errors may be several times that for read speech, even if
"20/ 20 Speech Lim ited , Malvern Hills Science Park , Gera ldine Road , Malvern ,
Wores., WR1 4 3SZ, UK.
135
136 WENDY J . HOLMES
the vocabulary size is much smaller. For example, on the CallHome cor-
pus of telephone conversations between family members, typical word error
rates exceed 30% [7] . Although there are many aspects to the problems
associated with this type of recognition task, the extent of the drop in per-
formance when moving beyond constrained domains suggests that there
may be inherent deficiencies in the acoustic modeling paradigm.
HMMs provide a framework which is broadly appropriate for mod-
eling speech patterns, accommodating both variability in timescale and
short-term spectral variability. However these models are simply general
statistical pattern matchers, and do not take advantage of the constraints
inherent in the speech production process and make certain assumptions
that conflict with what is known about the nature of speech production
and its relationship with acoustic realization. In particular, the follow-
ing three assumptions which are made by the HMM formalism are clearly
inappropriate for modeling speech patterns:
• Piece-wise stationarity. It is assumed that a speech pattern is
produced by a piece-wise stationary process, with instantaneous
transitions between stationary states.
• The independence assumption. The probability of a given acoustic
vector corresponding to a given state depends only on the vec-
tor and the state, and is otherwise independent of the sequence
of acoustic vectors preceding and following the current vector and
state. The model therefore takes no account of the dynamic con-
straints of the physical system which has generated a particular
sequence of acoustic data, except inasmuch as these can be incor-
porated in the feature vector associated with a state. In a typical
speaker-independent HMM recognizer where each modeling unit is
represented by a multi-modal Gaussian distribution to include all
speakers, the model in effect treats each frame of data as if it may
have been spoken by a different speaker.
• State duration distribution. A consequence of the independence
assumption is that the probability of a model staying in the same
state for several frames is determined only by the 'self-loop' transi-
tion probability. Thus the state duration in an HMM conforms to
a geometric probability distribution which assigns maximum prob-
ability to a duration of one frame and exponentially decreasing
probabilities to longer durations.
The impact of these inappropriate assumptions can be reduced by, for ex-
ample, using a generous allocation of states which allows a sequence of
piece-wise stationary segments to better approximate the dynamics and
also makes a duration of one frame per state more appropriate. A popu-
lar way of mitigating the effects of the independence assumption is to use
an acoustic feature vector which includes information over a time span of
several frames . This is most usually achieved by including the first and
sometimes also the second time-derivative of the original static features.
SEGM ENTAL HMMS 137
However , alt hough such tec hniques have been shown to be of pr actic al
benefit in improving recognition performance, t he independence assump-
t ion actua lly becomes even less valid because t he observed data for anyone
frame are used to cont ribute to a time spa n of several feature vecto rs.
Rather t ha n t rying to mod ify t he data to fit t he mode l, it should be
better to adopt a mod el t hat is more appropriate for t he cha racteristics
of speech signals. The inappropriate assumptions of HMMs are linked
wit h t heir frame-synchronous cha racteristic, whereby states are associated
with single acoustic feature vecto rs. In order to improve t he underlying
mod el, it is necessary somehow to incorporat e t he concept of mod eling
frame sequences rather t ha n individual fram es, wit h t he aim of providing
an accurate mod el of speech dynamics across an utterance in a way which
t akes into account predict able factor s such as speaker cont inuity. At the
sa me time, it is import ant to retain the many desirable attributes of t he
HMM framework, such as rigorous dat a-driven tr aining, optimal decoding
and delayed decision making. The desire to overcome the above limit ations
of HMMs by modeling frame sequences has motivat ed t he developm ent of a
variety of exte nsions, modifications and alte rnat ives to HMMs (see [20] for
a review). Models of t his type are usua lly referr ed to as "segment models"
[20] , or as "segmental HMMs" [16].
2. Segment models. A "segmental HMM" (SHMM) can be defined

in genera l te rms as a Markov model where segments, rather t ha n frames,
are t he homogeneous units which are t reated as probabilistic funct ions of
t he model states. T he relationship betwee n successive acoustic feature
vecto rs rep resenti ng a sub- phonemic speech segment can be approximated
by some form of trajectory t hro ugh t he feature space .
Severa l workers have suggeste d segment models t hat incorp orate t he
notion of a t ra jectory (e.g. [3, 11, 16]). The model described in [16] is a
par ti cular form of segmental HMM t hat was invest igated by t he aut hor,
working wit h Prof. M.J. Russell. The main char acterist ic t hat distinguishes
t he mod el is the repr esent ation of feature vari abili ty, which includ es th e
concept of a 'probabilistic' trajectory.
A probabilist ic-traj ectory SHMM (PTSHMM) for a speech sound pro-
vides a representation of t he range of possible und erlyin g t ra jectories for
that sound, wher e the traj ectories ar e of vari able dur ation and each dur a-
t ion has a st ate-depend ent prob ability. To accommod ate t he fact t hat an
obse rved sequence of feature vecto rs will in genera l not follow any underly-
ing t ra jectory exactly, a trajectory is modeled as 'noisy' . A segment is thus
describ ed by a stochastic process whose mean cha nges as a function of ti me
accor ding to t he par ameters of t he trajectory. The model t herefore makes
a disti ncti on between two types of varia bility: t he first is extra-segm ental
var iation in t he underlying trajectory, and t he second is intra-segmental
var iation of t he observations aro und anyone t rajectory. Intuit ively, extra-
segme ntal variations repr esent general factors, such as differences between
138 WENDY J. HOLMES
speakers or chosen pronunciation for a speech sound, which would lead

to different trajectories for the same sub-phonemic unit. Intra-segmental
variations can be regarded as representing the much smaller frame-to-frame
variation that exists in the realization of a particular pronunciation in a
given context by anyone speaker. For reasons of mathematical tractability
and of trainability, all variability associated with PTSHMMs is modeled
with Gaussian distributions assuming diagonal covariance matrices, and
only parametric trajectory models are considered.
The theory and implementation of PTSHMMs have been presented
and discussed in detail in [14] and [16] . The first part of the current paper
will concentrate on giving an overview of linear trajectory models, including
the linear PTSHMM and other segmental HMMs that can be viewed as
simplified forms of it. The main aim of this overview will be to show
some of the ways in which the conventional HMM can be extended, and
also some of the practical issues involved in achieving success with these
extended models . The second part of the paper will concentrate on the
choice of features on which the trajectory model is based, and on some of
the potential advantages that can be gained from choosing features that
have a closer relationship with the underlying speech production system
than the currently more popular features such as mel-frequency cepstrum
coefficients (MFCCs).
3. Linear-trajectory segmental HMMs. A simple model of dy-
namics is one in which it is assumed that the underlying trajectory vector
changes linearly over time. Previous studies of trajectory representations
of MFCCs [14] have suggested that a linear trajectory model is sufficient
to capture typical time-evolving characteristics (at least when using three
segments per phone) . The adequacy of a linear model is also supported
by Gish and Ng [11], who found that a linear trajectory was sufficient for
most sounds, even when only using one segment per phone.
3.1. Theory. A linear trajectory f(m ,c) is defined by its slope m and
the segment mid-point value c, such that f(m,c)(t) = c + m(t - ~T) if the
trajectory has a duration of T + 1 frames. In a linear PTSHMM, extra-
segmental variability takes the form of variation in the slope and mid-point
parameters. The distributions of the two trajectory parameters for a given
state can be defined by Gaussian distributions Np. ,-y and NI/'''' for the slope
and mid-point respectively. Intra-segment variation can be represented by
a Gaussian distribution with fixed variance r , All distributions are assumed
to have diagonal covariance matrices, and for notational simplicity all .ob-
servation sequences are therefore assumed to be one-dimensional. The joint
probability of a sequence of observations Y = Yo, ..., YT and any particular
values of the slope m and mid-point c is thus defined as follows:
T
P(y, m, c) = Np.,-y(m)NI/,"I(c) IT Nfm,c(t),r(Yd·
t=O
SEGMENTAL HMMS 139
The PTSHMM emission probability P(y) can be computed by integrating

P(y, m, c) over the unknown trajectory parameters m and c:
oo Joo Np"oy(m)Nv,.,.,(c) gNf"" c(t) ,r(Yt)dcdm.
T
(3.1) P(y) =
J-00 -00
In [14] it was shown that the integral in the above expression can be
evaluated to give a tractable expression that does not depend on the values
of m or c, thus:
P(y) = J(T+l~1J + TV
~( 1 )
~
J21rT
(T+l)
1 ((T+l) (v - c'(y))2 q (JL - m'(y))2

(3.2) x exp - -
(
+--'-----'-'-'--
2 (T+l)1J+T q,+T
+ ~ (~y' -(T +1) c'(y)' - qm'(Y)'))) ,

where
In Equation 3.2 above,
m'( ) = l:i'=o(t - ~)Yt c'( ) = l:i'-o Yt

Y "\'T
L..,t=O
(
t
_ 1:)2 '
2
Y T+ 1
define the slope and mid-point value respectively of the linear trajectory
which provides the least-squared error best fit to the sequence of observa-
tions y. HMMs are usually trained by a parameter re-estimation procedure
called the Baum-Welch algorithm [1], which is an example of the more gen-
eral method known as the expectation-maximization (EM) algorithm[2].
Extended Baum-Welch parameter re-estimation formulae were derived for
linear-trajectory PTSHMMs [14] [16], following the approach of Liporace
[19], whereby an auxiliary function was introduced and new values were
calculated for the model parameters in order to maximize that auxiliary
function .
Returning to the linear-PTSHMM expression for P(y) given above,
it is interesting to consider the characteristics of the models that arise if
certain parameters are fixed to have values of zero. If the slope mean JL and
the slope variance, are both fixed at zero, the trajectory is defined by just
the mid-point distribution and is thus constant over time . The expression
for P(y) becomes:
(3.3)
This model is a static PTSHMM, equivalent to the model suggested in [8].

Static PTSHMMs distinguish between extra-segmental and intra-segmental
variability, and therefore impose some degree of continuity constraint be-
tween successive observations within a segment . This type of model cannot,
however, capture local frame-to-frame dynamic characteristics.
If the mid-point variance 1] is also fixed at zero, the expression for P(y)
is further simplified as follows:
T
(3.4) P(y) = II N'I,r(yt},
t=O
which is equivalent to the standard-HMM probability calculation (although

here a maximum segment duration will be applied, and there is also the
possibility of easily including a realistic duration model, as in the approach
of [26] for example).
A standard HMM can be regarded as a static PTSHMM with zero
extra-segment variance. Similarly, by taking a linear PTSHMM and setting
the extra-segment variances 1] and 'Y of the mid-point and slope both to
zero, the mid-point and slope values will always be equal to the model
means and therefore define a single linear trajectory, thus:
T
(3.5) P(y) = II Nf",v(t),r(yt}.
t=O
This model represents a linear "fixed-trajectory" segmental HMM (FT-

SHMM), equivalent to linear trajectory-based segment models such as those
suggested by [11] and by [3].
If only the slope variance 'Y is set to zero, the probability expression
becomes:
(3.6)
This model is a constrained form of linear PTSHMM, whereby the mid-

point of the trajectory can vary across examples of a model unit, but the
slope is always fixed at the model mean value. In future discussion it will be
convenient to refer to this type of linear PTSHMM as having "const rained
slope" , while the linear PTSHMM with variability in all its parameters and
represented in Equation (3.1) has "flexible slope" .
The following sections summarize the findings of experimental inves-
tigations into linear PTSHMMs with full flexibility in all their parameters
and also into the range of simplified cases described above.
3.2. Digit recognition experiments. The recognition task was se-
lected with the aim of providing the basis for meaningful comparisons to be
SEG MENTA L HMMS 141
made between different segmental HMMs and conventional HMMs, while

also enabling the properti es of the segmental model to be investigat ed.
A spea ker-independent connected-digit recognition t ask using phone-based
models was chosen, because it requires a connected-word recognit ion algo-
rithm such t hat segmentat ion occurs simulta neously with recognit ion but
is a simple t ask with a small vocabulary so t hat analysis of recognition er-
rors is relat ively st raightforward. In addit ion, th e small vocabulary offers
a faster experiment t urn-a round time than is possible with larger vocabu-
laries. The increase in computational load associated wit h segment models
is such that it was considered import ant to begin with a small t ask when
investigating th e properties of PTSHMM s.
3.2.1. Speech data. Th e test dat a were four list s of 50 digit triples
spoken by each of 10 male speakers. The training dat a were from 225
different male speakers , each reading 19 four-digit st rings taken from a vo-
cabulary of 10 strings. The available speech data had been sampled at
19.98 kHz and analyzed using a 27-channel crit ical-band filterbank span-
ning t he range 0-10 kHz, producing output channel amplitudes quantiz ed
in units of 0.5 dB at a rate of 100 frames/so A cosine transform was applied,
and the first eight cosine coefficients together with an average amplit ude
parameter were used as t he feature set .
3.2.2. Experimental framework. The experiments described here
used cont inuous-density Gaussian HMMs wit h a single-Gaussian out put
distri bution per state and diagonal covariance mat rices. The emphasis has
been on relative performance of different models rath er tha n on achieving
the best possible absolute performance. Th erefore, alt hough t he systems
are intended to provide a sufficiently good level of baseline performance to
make comparisons meaningful, no attempt was made to optimize det ails of
the feature analysis, model inventor y and so on.
Three-st at e context-independent monophon e models and four single-
state non-speech models were used. Th e simple left-right model st ructure
that is typically used in most HMM systems was adopted. Self-loop tr an-
sitions were allowed in t he standard HMMs. For the segmental HMMs
representin g speech sounds , only tr ansitions from a state on to th e imme-
diat ely following state were allowed and each state was assigned a maximum
segment duration of 10 frames. This model st ruct ure thu s imposes a max-
imum phone dur ation of 300 ms, which was considered adequate for most
speech sounds in connected speech. The self-loops were retained for th e
non-speech models, to provide a simple way of accommodating any long
periods of silence. Because t hese experiments were intended to evaluate
different models of acoustics, rather th an dur ati on-modeling differences,
all transitions from a state were assigned the same trans ition prob abil-
ity, and for the segmental HMMs all the segment dur ations were assigned
equal probability divided between the allowed dur ation range. None of t he
transition probabiliti es or duration probabilities were re-estimated.
The model means for the standard HMMs were initialized based on
a small quantity of hand-annotated training data, and all model variances
were initialized to the same arbitrary value. The model means and vari-
ances were then trained with five iterations of Baum-Welch re-estimation.
After five training iterations, these standard HMMs gave a word error rate
of 8.6% on the test data. These models provided the starting point for train-
ing different sets of segmental HMMs, all of which were trained with five
iterations of the appropriate Baum-Welch type re-estimation procedure.
For performance comparisons, the standard HMMs were also subjected to
a further five iterations of re-estimation.
3.2.3. Segmental HMM training. The simplest segmental models
were those represented in Equation (3.4), which used the standard-HMM
probability calculation but incorporated the constraint on maximum du-
ration. These simple static segmental HMMs and the linear FTSHMMs
(shown in Equation (3.5)) were both initialized directly from the trained
standard-HMM parameters, with the intra-segment variances being copied
from the standard-HMM variances and the extra-segment variances being
set to zero. The slope means of the FTSHMMs were initialized to zero, so
that the starting point was the same as for the simplest segmental HMMs,
with the difference being that when training the FTSHMMs the slope mean
was allowed to deviate away from its initial value.
Initial estimates for the different sets of PTSHMMs were computed
by first using the trained standard HMMs to segment the complete set
of training data, and using the statistics of these data to estimate the
various parameters of each set of PTSHMMs according to the relevant
modeling assumptions. In all cases the means and variances of the mid-
points were initialized from the distributions of the sample means for the
individual segments . The initialization strategies that were used for the
other parameters were different for the different types of PTSHMM and are
summarized in Table 1. Each set of segmental models was trained with five
iterations of the appropriate Baum-Welch-type re-estimation procedure
[14] [16] .
3.2.4. Results. The connected-digit recognition results are shown in
Table 2 for the different sets of segmental models, compared with the base-
line HMMs after both five and 10 training iterations. The main findings
are summarized in the following paragraphs.
The simple segmental HMMs with a maximum segment duration of 10
frames gave an error rate of 6.6%, which is lower than that of the conven-
tional HMMs even when further training had been applied (8.4%). Thus,
in these experiments there were considerable advantages in constraining
the maximum segment duration, which acts to prevent unrealistically long
occupancies for the speech model states.
The lowest word error rate achieved with the static PTSHMMs was
7.5%, which is not quite as good as the 6.6% obtained with the simplest
SEGMENTAL HMMS 143
TABLE 1
Ini tializati on stm tegy f or various model pammeter s of different sets of PTSHMMs .
Model set Slope parameters Intra-segment

variance
°
static PTSHMM mean and variance variance of observations
(Eq . (3.3)) both fixed at about segment mean
linear PTSHMM mean and variance of variance of observations
(flex. slope) slopes of best-fit about best-fit linear
(Eq . (3.1)) linear trajectories trajectories
linear PTSHMM mean and variance variance of observations
(constr. slope) both set to 0, but mean about segment mean
(Eq. (3.6)) can vary in training (i. e. line with zero slope)
segmental HMMs. Both sets of models appe ared to be adequately tr ained

after five iterations of re-estimation , as performing a further five iterations
did not reduce the word error rate . It therefore appears that, for a static
trajectory assumption, there is no benefit from adopting th e PTSHMM
approach of separating out intra- from ext ra-segmental variability.
The linear FTSHMMs gave a word error rate of 4.9%, which is an
improvement over the 6.6% error rate achieved with the simplest segment al
HMMs. This result demonstrates the benefits of incorporating a linear
tr ajectory representation to describe how features change over time .
The best performance achieved with th e linear PTSHMMs was an
erro r rat e of 2.9%, which represents a reduction in error rate of 40% over
the result with the linear FTSHMMs . Considerable furth er advantage was
thus gained by separating out extra- from intra-segmental variability, in
addit ion to the benefits of th e linear tr ajectory description.
Th e linear PTSHMMs with constrained slope gave th e best recognition
performance, whereas the linear PTSHMMs with flexible slope performed
worse than the baseline standard HMMs. This finding suggests that lin-
ear PTSHMMs provide better discrimination when th ey represent extra-
segmental variability in th e mid-point but not in the slope par ameters .
3.2.5. Comparisons with HMMs using time-derivative fea-
tures. Th e experiments described so far have demonstrated recognition
performance improvements by incorporating a linear model of temporal
dyn amics within a segment-based framework. However, successful conven-
t ional HMM-bas ed recognizers almost always include some repr esentation
of dynamic chara cterist ics within the acoust ic feature vectors themselves.
Comparisons were th erefore made with models using conventional HMM
prob ability calculat ions with time-derivative features, computed for each
frame using the typical approach of applying linear regression over a win-
dow of five frames cent red on th e current frame. Both HMMs and then
simple segmental HMMs were tr ained using an acoustic feature set which
TABLE 2
Connected-digit recognition results for different sets of segmentalllMMs compared
with conventional HMMs .
Model type I % Sub. I % Del. I % Ins. I % Err. I

HMM (five training iterations) 6.2 1.5 0.9 8.6
HMM (10 training iterations) 6.0 1.6 0.8 8.4
Simple segmental HMM 5.2 0.7 0.7 6.6
Static PTSHMM 5.2 2.2 0.1 7.5
Linear FTSHMM 3.8 0.5 0.6 4.9
Linear PTSHMM (flex. slope) 4.9 4.0 0.0 9.0
Linear PTSHMM (constr. slope) 2.0 0.8 0.1 2.9
included time-derivative features of the original nine instantaneous fea-

tures, to give a total of eighteen features . The performance of these models
was significantly improved over that which had been obtained using only
instantaneous features, to give error rates of 1.5% and 1.6% for the HMMs
and simple segmental HMMs respectively. Thus, when derivative features
were included, the maximum-duration constraints provided by the simple
segmental HMM did not give any advantage over the standard HMM. The
conventional HMMs with time-derivative features have given an error rate
of only 1.5%, whereas the best error rate achieved with the linear PT-
SHMMs (using only instantaneous features) was 2.9%. The result of this
comparison is disappointing, but can be explained by differences in the
extent to which the two models are able to represent dynamics . Although
the use of derivative features only provides implicit modeling of dynamics,
some representation of change is provided for every frame. However, the
segmental models studied here have been limited to representing dynamics
within anyone segment, so further performance advantages may be ob-
tained by using derivative features with the segmental models, as has been
found by other researchers, for example [5]. Given that the error rate was
already very low with conventional HMMs when including time-derivative
features, it was not considered worthwhile trying this approach for the digit
recognition task. However, the next section describes some experiments on
a much more challenging task, which have been carried out both with and
without time-derivative features.
3.3. Phone classification experiments. Phone classification in-

volves determining the identity of speech segments with specified phonetic
boundaries, so providing a means to investigate and compare phonetic-
modeling capabilities for different speech sounds. Studying classification
rather than recognition has computational advantages, but also allows for
the investigation of description and discrimination abilities separately from
segmentation properties. A useful set of data for evaluating phonetic classi-
fication performance is the DARPA TIMIT acoustic-phonetic continuous-
SEGMENTAL HMMS 145
speech database of American English [10]' for which all the utterances
have been phonetically transcribed, segmented and labeled. TIMIT was
designed to provide broad phonetic coverage, and is therefore particularly
appropriate for comparing approaches to acoust ic-phonetic modeling .
When classifying the data segments , all phones were treated as equally
likely (no langu age model was used to constrain the allowed phone se-
quen ces). This approach was considered appropriate for investigating im-
proved acoustic-phonetic modeling, but it does make th e task very difficult.
As with the digit experiments, the emphasis here is on th e relative perfor-
mance of the different segment al HMMs.
3.3 .1. Speech data and model sets. Th e experiments reported
here used the TIMIT designated training set and the core test set, using
data only from the male speakers . The available data had been analyzed
by applying a 20 ms Hamming window to the 16 kHz-sampled speech at a
rate of 100 framesjs and computing a fast Fourier tr ansform. The output
had been converted to a mel scale with 20 chann els, and a cosine transform
had been applied. The first 12 cosine coefficients together with an aver-
age amplitude feature formed the basic feature set, but some experiment s
also included time-derivative features computed for each frame by applying
linear regression over a five-frame window cent red on th e current frame.
The inventory of model units was defined as th e set of 61 symbols which
are used in the tim e-aligned phonetic tr anscriptions provided with TIMIT.
However, the two different silence symbols used in these transcriptions
were represented by a single silence model, to give 60 model units. In
common with most other work using TIMIT, when scoring recognition
output the 60-symbol set was reduced to the 39-category scoring set given in
[18] . Experiments were carried out with context-independent (monophone)
models, and also with right-context-dependent biphones, which depend on
only the immediately-following phoneme context .
Th e basic model structure for both the conventional and the segment al
HMMs was the same as the one used for th e digit experiments , with three
states per speech model and single-state non-speech models. However , this
structure imposes a minimum duration of three frames for every speech
unit, whereas some of the labeled phone segments are shorter than three
frames. In order to accommodate th ese very short utterances, the structure
of all the speech models was extended to allow transitions from the initial
st at e to all emitt ing states with a low probability.
3.3.2. Training procedure. First, a set of standard-HMM mono-
phone models was initialized and t rained with five iterations of Baum-
Welch re-estim ation. These models were then used to initi alize different
set s of monophone segmental HMMs, using the same appro ach as the one
adopted in the digit-recognition experiments. The discussion here will fo-
cus on comparing the performance of linear FTSHMMs and const rained-
slope linear PTSHMMs with that of simple segmental HMMs (models us-
ing conventional HMM probabilities but with a maximum-duration con-

straint) . For all sets of segment models, five iterations of the appropri-
ate Baum-Welch-type re-estimation were applied, the trained monophone
models were used to initialize biphone models and three further iterations
of re-estimation were then carried out.
3.3.3. Classification results. Table 3 shows the classification error
for different segmental HMMs, for both monophone and biphone models,
with and without time-derivative features . From these results it can be seen
that, under all the different experimental conditions, the linear FTSHMMs
gave some performance improvement over the simple segmental HMMs,
and there were further performance benefits from linear PTSHMMs that
incorporated a model of variability in the mid-point parameter.
TABLE 3
Classification results for the male portion of the TIMIT core test set using biphone
models . The percentage of phone errors is shown for different types of linear-tmjectory
segmental HMMs compared with the baseline provided by simple segmental HMMs.
Model type I Cepstrum features only I Include delta features I

Simple SHMM 43.0 29.4
Linear FTSHMM 39.0 27.4
Linear PTSHMM 38.2 26.8
The relative performance of the linear PTSHMMs and the simple seg-
mental HMMs was analyzed as a function of phone class, in order to de-
termine whether the segmental model of dynamics was more beneficial for
some types of sound than others. The results of this analysis are shown in
Table 4 for a selection of sound classes. The linear PTSHMMs improved
performance for all the phone classes, but were most beneficial for the diph-
thongs and the semivowels and glides. These sounds are characterized by
highly dynamic continuous changes, for which the trajectory model should
be particularly advantageous. The performance gain was smallest for the
stops, which have rather different characteristics involving abrupt changes
between relatively steady-state regions. Some model of dynamics across
segment boundaries may be the best way to represent these changes.
3.4. Discussion. The experiments described above have shown some
improvements in recognition performance through modeling trajectories of
mel-cepstrum features, although for best performance it was necessary to
also include time-derivatives in the feature set. These experiments all used
a fixed model structure with three segments per phone. There are various
aspects of the acoustic modeling which could be developed to take greater
account of the characteristics of speech patterns. For example , rather than
using three segments to model every phone, it would seem more appropriate
to represent each phone with the minimum number of segments to describe
typical trajectories in order to maximize the benefit from the constraints
SEGMENTAL HMMS 147
TABLE 4
Classificati on performance of linear PTSHMMs relative to that of simple segmental
HMMs , shown for different phon e classes (using biphone models with mel-cepstrum
f eatures) .
Phone class No. SHMM PTSHMM % PTSHMM

examples (% err.) (% err.) improvement
Stops 566 56.7 54.8 3.4
(p, t , dx , k, b, d, g)
Fricatives (f, v, 710 41.7 38.9 6.8
th , dh, s, z, sh, hh)
Semivowels and 497 39 .2 33.2 15.4
glides (I, r, y, w)
Vowels (iy, ih, eh, 1178 53 .8 48.9 9.1
ae, ah , uw, uh, er)
Diphthongs 376 48.9 41.2 15.8
(ey, ay, oy, aw, ow)
provided by the segment model. With a linear trajectory model, three

segments are probably necessary for cert ain sounds (such as diphthongs),
but for many other sounds (e.g. nasals, voiceless fricatives and short vowels)
one segment should be sufficient. It may also be beneficial to employ phone-
dependent constraints on th e range of allowed segment durations.
Another aspect of th e modeling concerns the choice of acoust ic fea-
tures. While it has been shown that recognition perform ance can be im-
proved by representing t raj ectories of mel-cepstrum features, the motiva-
t ion for modeling dynamics in speech comes from the continuous dyn amic
nature of speech production . It may therefore be better to apply th e tra-
jecto ry model to features that are more directly relat ed to the mechanisms
of speech prod uction . A useful functional represent ation of speech produc-
tion is provided by th e vocal tr act resonances, or formant s. Exp eriments
using formant-trajector y models togeth er with a phone-dependent model
structure are described in the next sect ion.
4 . Recognition using formant trajectories. Although it is well

known th at the frequencies of th e formants are extremely important for
determining the phonetic content of speech sounds, formant frequencies
are not normally used as features for ASR due to a number of practical
difficulties that tend to arise when attempt ing to ext ract and use formant
information in recognition. For exa mple, the formants are often not clearly
apparent as dist inct peaks in th e short-t erm spect rum. In the extreme, for-
mants do not provide the required information for making certain distinc-
tions , such as identifying silence. Furthermore , formant-labeling difficulties
can arise even when the spect rum shape shows a clear simple resonance
structure because, for example, two formants that are close together may
148 WENDY J. HOLMES
merge to form a single spectral peak. A consequence of all these factors is

that it is difficult to identify formants independently from the recognition
process that decides on the phone identities.
A method of formant analysis has been developed [13] that includes
techniques to largely overcome the difficulties normally associated with
extracting and using formant information. In addition, improvements in
recognition performance were demonstrated by incorporating formant in-
formation in a standard-HMM recognition system [13]. Work on applying
the formant-based system to linear-trajectory segmental-HMM recognition
is described below.
4.1. Formant analysis. The formant analyzer is described in some

detail in [13]' but a brief overview is given here. The system is based on a
codebook of spectral cross-sections and associated formant labelings that
has been previously set up by a human expert. Given an input speech sig-
nal , short-term spectral cross-sections are matched against this codebook
to find the entries that give the best spectral match and hence derive pos-
sible formant labelings for each frame of speech. Continuity constraints
and other constraints are employed to eliminate many possibilities but, in
those cases where there is still uncertainty about how the formants should
be allocated to spectral peaks , alternative sets of formant frequencies are
offered to the recognition process. To indicate cases where the formants are
not well defined by the spectrum shape, an empirical degree-of-confidence
measure is provided for each estimated formant frequency. The confidence
measure is calculated using information about formant amplitude and spec-
trum curvature. When the amplitude is low or the formant structure is not
well defined (as in many voiceless fricatives), the confidence will be much
lower than when there is a peaky spectrum shape (typical of vowels).
Figure 1 shows a spectrogram with superimposed formant tracks indi-
cating the output of the formant analyzer. It can be seen that the analyzer
has tracked the formants quite accurately with smooth trajectories that
capture dynamics such as the movement of F2 in the I all diphthong of
the word "nine" . In the word "two", there are two alternative formant
trajectory choices that have been suggested and the recognizer would be
required to select between these two choices.
4.2. Using the formant analyzer output in recognition. In ad-

dition to the formant frequencies, features giving some measure of spectrum
level and spectrum shape are needed to perform recognition . Low-order
cepstrum features are a convenient way of providing this information. To
make use of the formant alternatives and confidence measures that are pro-
duced by the analyzer, some modifications to the recognition calculations
are required . When the analyzer offered alternative formant allocations,
for each model state the set was chosen that gave the highest probability of
generating the observations. The choice between alternatives was made on
SEGMENTAL HMMS 149
:r:-;.~=T-'=----.....4
300 700 800
FIG . 1. Spectrogram of an utterance of the words "nine one two" , with superimposed
formant tracks showing alternative formant allocations offered by the analyz er for Fl ,
F2 and F3. Tracks are not plotted when there is no confidence in their accuracy.
a frame-by-frame basis when performing standard-HMM recognition, and

on a segment-by-segment basis when using segment al HMMs.
To use the confidence measure in recognition , it was represented as th e
variance of a notion al Gaussian distribution of th e true formant frequency
about the estimated value. By representin g th e formant confidence est i-
mat es as varianc es, it is st raightforward to incorpor ate t hem in t he HMM
prob ability calculations in a way th at can be justified theoret ically (see [9]
for details). In this interpr etati on, th e formant analyzer emits th e par am-
eters of a norm al distribution representin g its belief about the position of
each formant. When t he confidence is high, the variance is low to represent
strong belief in th e estim ate. Conversely, when th e confidence is low, t he
variance will be large, so representing almost equal belief in all possible
frequencies. With this Bayesian interpretation, the confidences are incor-
por at ed simply into th e recognition calculations by adding the appropriate
confidence varianc e to t hat of the model state output distribution. The
same approach can be applied to extend the probability calculat ions for
FTSHMMs. The situation is more complicated for PTSHMMs due to the
prob abilistic nature of the trajectory model, and so far all segmental HMM
experiments with formants have used the fixed trajectory model.
4.3. Digit recognition experiments.
4.3.1. Method. Using the same connected-digit recognition task that
was used in the experiments described in Section 3.2, and with both stan-
dard and segmental HMMs, recognition performance when using formant
features to describ e fine spect ral det ail was compared with t hat obt ained
when using more ty pical mel-cepstrum features. In order to assess the use-
fulness of th e formants directly, t he same tot al number of features was used
for both feature sets and exactly the same low-order cepstrum features were
used for describing general spectrum shape. The output of an excitation-

synchronous FFT was therefore used both to estimate formant fequencies
with associated confidence measures and to compute a mel-cepstrum. One
feature set for the experiments comprised the first eight cepstrum coeffi-
cients and an overall energy feature, while for the other feature set cepstrum
coefficients 6, 7 and 8 were replaced by three formant features.
4.3.2. Model sets. For these segmental HMM experiments, each
phone was modeled by an appropriate number of segments in order to
describe its spectral characteristics using linear trajectories, with the num-
ber of segments assigned based on phonetic knowledge. Three segments
were used to model voiceless stops, affricates and some diphthongs, with
two segments being used to represent voiced stops, most diphthongs and a
few long monophthongs. When using linear trajectories, one segment was
considered sufficient for nasals, fricatives, semivowels and most monoph-
thongal vowels. For each segment, a minimum and maximum segment
duration was set to allow a plausible range of durations for each phone .
Linear-trajectory segmental HMMs were compared with the simplest type
of 'segmental' models, which used the same state allocation and maximum-
duration constraints, but with standard-HMM emission probability calcu-
lations. Further comparisons were carried out with standard HMMs using
the phone-dependent state allocation, and with standard HMMs using the
more conventional allocation of three states per phone. All experimental
conditions used single-state non-speech models.
4.3.3. Results. The results are summarized in Table 5. The baseline
is provided by the standard HMMs using mel-cepstrum features and three
states per phone. These models gave an error rate of 3.5%.1 Incorporating
the variable-state allocation designed for the linear-trajectory model made
the performance of the standard HMMs worse, which can be explained by
the lower number of states that were then available to capture the acoustic
characteristics of many of the phones. This disadvantage of the smaller
number of states was overcome by introducing a segmental structure with
segment-duration constraints. A considerable further advantage was gained
by incorporating the linear trajectory model. For all the model sets, there
was a small but consistent advantage gained from including formants in
the feature vector, for the same total number of features. The best overall
performance is provided by the formant-based linear-trajectory segmental
HMMs. The total number of free parameters in this model set is in fact
fewer than the number in the standard HMMs with three states per phone.
Although the linear trajectory requires additional parameters, this addition
is more than compensated for by the reduction in the total number of states.
IThese results are not directly comparable with the earlier results that were shown
in Table 2 for a number of reasons, including the use of a different set of features and
also some other minor differences in the experimental framework .
SEGMENTAL HMMS 151
TABLE 5
Conne cted-digit recognition results for different sets of standard and segm ental
HMMs . Word error rates for a f eature set com prising the first eight MFCCs (and
an overall en ergy feature) are com pared with those for a feature set in which the three
highest MFCCs are replaced by three formant featur es.
Model typ e I 8 MFCCs I 5 MFCCs + 3 formants I

Stand ard HMM 3.5 2.5
with three states per phone
Standard HMM 6.4 5.9
with variabl e state allocat ion
Simple segmental HMMs 3.2 2.9
Linear FTSHMMs 2.6 2.3
4 .3.4. Discussion. The results of th e experiments described above

have indicated t hat using segmental HMMs to model formant tr aj ectories
seems to be a promising appro ach to ASR. Another attractive aspect of
modeling formant trajectories is that such a model naturally lends itself
to speech synthesis as well as to recognition. Th ere is thus the possibility
of using the same model for both recognition and synthe sis [25], which in
turn leads to a compact model for low bit-rate speech coding [15] . The
applicat ion of the formant-based linear tr ajectory model to speech coding
is discussed in the next section .
5. Recognition-based speech coding. Successful speech coding at

low data rates of a few hundred bits/s requires a compact , low-dimensional
representation of the speech signal, which is generally applied to variable-
length 'segments' of speech. Automatic speech recognition is potentially a
powerful way of identifying useful segments for coding. If th e segment s are
meaningful in phonetic terms, knowledge of segment ident ity can be used
to guide th e coding. In the ext reme, very low data rates can be achieved by
tr ansmitting only phoneme identity information. A number of recognition-
based coders have been suggested that use HMMs (e.g. [22, 17,23,27]) . In
all of th ese syst ems, an utteran ce is coded in terms of phone-based recogni-
tion units and relevant dur ation information. The main differences are in
th e schemes that are used to reconstruct th e utterance at t he receiver. One
possibility is to use the HMMs th emselves, but simple use of th e HMM st ate
means will tend to lead to inappropriate discontinuities in the synthesized
speech. [27] used a more elabor ate scheme which prod uced smoother se-
quences by also using information from time-d erivative features. However,
the underlying assumptions of piecewise-stationarity and of independence
are such that HMMs are inherently limited as speech production models.
Another limitation when using HMMs for synthesis is that typic al feature
sets such as LPC coefficients [22] or mel-frequency cepstral coefficients [27]
impose limits on the coded speech quality. Other syst ems have regenerat ed
the utterances using completely separate systems , such as time-normalized

versions of complete segments [23] or a synthesis-by-rule system [17].
A formant-trajectory segment model of the type described in the previ-
ous section can naturally lend itself to speech synthesis as well as to speech
recognition. This type of model can therefore provide the basis for a 'uni-
fied' approach to speech coding in which the same (appropriate) model of
speech recognition is used as the basis for both the recognition step and
the synthesis step. In this way it is possible to address the issues associated
with achieving successful recognition-synthesis coding at low bit-rates [15] .
5.1. A general framework for a 'unified' speech coding model.

A good model for recognition-based coding needs to provide a compact
representation of speech, while offering both accurate recognition perfor-
mance and high quality synthesis. Such a model could be used for coding
at a range of data rates, by trading bits against retention of speaker char-
acteristics. With no limitations on vocabulary size, at the lowest bit rates
speech could be generated from a phoneme sequence. At higher data rates,
the coding could be applied directly to speech production parameters. The
principles of operation for such a 'unified' model for speech recognition and
synthesis, and its application to speech coding, are illustrated in Figure 2.
The approach requires an accurate model for speech dynamics that pre-
serves the distinctions between different speech sounds and also a suitable
representation of speech production mechanisms.
UreCOgnifiOn
message
nSynthesiS
SYMBOLIC less fine detail

-tCODING and fewer speaker
LEVEL
1'l'characteristics
Upreserved ,low
underlying trajectories bit-rate
capturing dynamics of -tCODING
n
production mechanisms
INTERMEDIATE
high quality,
LEVELS • relatively high
detailed time-evolving
production parameters -t CODING bit-rate
for utterance
SURFACE
LEVEL
,illU~~~~~lLLLLllLLL~ peech
'1~lT"'waveform
FIG. 2. Schematic representation of a unified model for speech recognition and

synthesis, showing its application to speech coding across a range of data rates.
SEGMENTA L HM M S 153
5.2. A simple coding scheme to illustrate the approach. Work

so far has concent rated on demonstratin g the principle of recognit ion-
synt hesis coding using t he same linear formant- tr ajectory model for both
recognition and synth esis. The coding is applied to analyzed formant tra-
jectories, and so is at t he high bit-r ate end of t he range of coding schemes
discussed above. The main stages in t his simple coding scheme are illus-
t rated in Figure 3. Th e recognition uses the system described in t he pre-
vious sect ion, with formant-based linear-tr ajectory segmental HMMs, and
the synt hesis uses the JSRU parallel-formant synt hesizer [12]. Th e speech
is first analyzed in terms of formant tra ject ories and ot her information
t hat is needed for synth esis (formant amplit udes, fundament al frequency
and degree of voicing). Th e recognizer is used to determ ine the segment
identities and th e locations of t he bound aries of t he segments. All t he
synt hesizer cont rol par ameter s are t hen coded as linear trajectories for th e
segments that have been identified in recognition. Th e meth od of cod-
ing and the bit allocation for th e different parameters was chosen to be
reasonably economical, but was not optimized for maximum efficiency.
A particular aim of this work has been to use information from t he
recognition process to assist in th e formant coding. For example, when a
sequence involving a vowel followed by anot her vowel or a sonora nt was
recognized, at the coding stage a cont inuity constr aint was imposed on t he
form ant t ra jecto ries across the segment bound ary (in future t here is the
possibility of also incorp oratin g continuity const raints within the recogni-
tion process itself). When the recognizer selected between two alternative
formant t rajectories offered by t he initial form ant analysis, t he t ra jecto ry
chosen in recognition was the one that was used in t he coding. More det ail
about the coding method can be found in [15].
Although formants have been used as t he basis for oth er speech cod-
ing schemes (e.g. [6, 28]), the use of traj ectory-based recognition is an
import ant distin guishing feature of t he approac h being pursued here.
5.3. Coding experiments. Th e coding method has been tested on

t he spea ker-independent connected-digit recognition tas k that was used in
the speech recognition experiments described above, and also on a speaker-
dependent t ask of recognizing spoken airborne reconn aissance mission
(ARM) reports using a 500-word vocabulary. For each ut terance coded, th e
bit rate was calculated (see [15] for details) and th e quality of th e coding
was evaluated by informal listening test s. The segment-coded utteran ces
were genera lly perceived as somewhat stylized in compar ison with the orig-
inal utterances. However , speaker characteristics were ret ained for all t he
variety of speakers tested, and the speech was generally highly int elligible.
For t he digit data, typical coding rates were 600- 800 bits/so For t he
ARM task, t he rates te nded to be higher at about 800-1000 bits/so These
rates reflect the nature of t he speech material: with this coding scheme,
t he bit rate does not simply depend on th e vocabulary size, but it does
tran$mrlter
RecognIZe uSing J-i

Iinear.trajectory
segmenta l HMMs Code synthesizer
~
segments
tr
Analyse : information ~:~~~~~:;~rs c
Input _:-i speech to
esbmate : from .. trajectories for
speech • recognlbon each recognized
formants
DerIVe I--i segment
L. frame·by.frame
synthesizer control
Resynthesized
parameters speech
FIG. 3. Block diagram showing the main stages in a simple recognition-synthesis

coding scheme using linear form ant trajectories in both recognition and synthesis.
depend on the number of segments identified per second of speech. The

higher bit rates for the ARM tas k arose because this data set included
more acoustically complex words and the reports were spoken rat her more
quickly than the digit strings that were used in the other tas k.
6. Summary and di scussion. This paper has given an overview of
a range of studies that have been carried out investigating the application
of t rajectory-based segmental HMMs to ASR and speech coding, and more
generally to modeling the characteristics of speech signals. Improvements
in recognition performance have been demonstrated by modeling linear tra-
jectories of bot h mel-cepstrum and formant features, with the overall best
performance being obtained by adopting the trajectory model and includ-
ing forma nts in the feature set. Although the formant ana lyzer used in
these exper iments includes a number of special characteristics to reduce
prob lems due to formant analysis errors, there were still some errors from
the formant ana lyzer that were propagated to the recognizer. Any system
that performs formant ana lysis as a first stage prior to recognition will tend
to suffer from these problems, which are likely to become worse under more
difficult environmental conditions . The difficulties involved in extracting
and using formant (or articulatory) information have led a number of work-
ers (e.g. [25, 24, 4]) to suggest that this information is best incorporated in
a multiple-level framework, incorporating some production-related model
of dynamics as an intermediate level between the abstract phonological
units and the observed acoustic features. The aim is for the trajectory
model to enforce production-related constraints on possible acoustic re-
alizations without requiring explicit extraction of articulatory or formant
information. This type of extension can be applied to the linear-trajectory
segmental HMM described here (see [25]), which should enable the model
to enforce more powerful constraints while not suffering from analysis er-
rors . Such a model should be especially useful in difficult environmental
conditions and whenever the acoustic signal is degraded.
SEGMENTAL HMMS 155
REFERENCES
[1] L. BAUM, An inequality and associated maximization technique in statistical esti-

mation for probabilistic functions of Markov processes, Inequalities, III (1972),
pp. 1-8.
[2J A. DEMPSTER, N. LAIRD, AND D . RUBIN, Maximum likelihood from incomplete
data via the em algorithm , Journal of the Royal Statistical Society, Series B,
39 (1977) , pp. 1-38.
[3J L. DENG , M . AKSMANOVIC, D . SUN , AND J . Wu, Speech recognition using hidden
Markov models with mixtures of trend functions, IEEE Trans Speech and
Audio Processing, 2 (1994), pp . 507-520.
[4] L. DENG AND J . MA, Spontaneous speech recognition using a statistical coarticula-
tory model for the vocal-tract-resonance dynamics, Journal of the Acoustical
Society of America, 108 (2000) , pp . 1-13.
[5] V. DIGALAKIS, Segment-based stochastic models of spectml dynamics for continu-
ous speech recognition, PhD thesis, Boston University, 1992.
[6] B.C. DUPREE, Formant coding of speech using dynamic progmmming, Electronics
Letters, 20 (1980) , pp. 279-280.
[7] J . FISCUS, W . FISHER, A. MARTIN , M. PRZYBOCKI , AND D. PALLETT, 2000 nist
evaluation of conversational speech recognition over the telephone : English
and mandarin performance results, in Proceedings of the 2000 Speech Tran-
scription Workshop, University of Maryland, 2000.
[8J M.J . GALES AND S.J . YOUNG , Segmental hidden Markov models, in EU-
ROSPEECH, Berlin, 1993, pp . 1611-1614.
[9] P . GARNER AND W. HOLMES, On the robust incorpomtion of formant features into
hidden Markov models for automatic speech recognition, in ICASSP, Seattle,
1998, pp . 1-4.
[lOJ J. GAROFOLO , L.F . LAMEL, W . FISHER, J . FISCUS, D . PALLETT, AND
N. DAHLGREN, The darpa timit acoustic-phonetic continuous speech corpus
cdrom, ntis order number pbOl-100354 , available from Ide, 1993.
[11] H . GISH AND K . NG, A segmental speech model with applications to word spotting,
in ICASSP, Minneapolis, 1993, pp . 447-450.
[12] J.N. HOLMES, A pamllel-formant synthesizer for machine voice output, in Com-
puter Speech Processing, 1985.
[13J J .N . HOLMES, W.J . HOLMES, AND P .N . GARNER, Using formant frequencies in
speech recognition, in EUROSPEECH, Rhodes, 1997.
[14] W .J . HOLMES, Modelling segmental variability for automatic speech recognition,
PhD thesis, University of London, 1997.
[15] - - , Towards a unified model for low bit-mte speech coding using a recognition-
synthesis approach, in ICSLP, Sydney, 1998.
[16J W .J . HOLMES AND M .J . RUSSELL, Probabilistic-tmjectory segmental HMMs , Com-
puter Speech and Language, 13 (1999) , pp . 3-37.
[17J M. ISMAIL AND K .M . PONTING, Betwe en recognition and synthesis - 300
bits/second speech coding, in EUROSPEECH, Rhodes, 1997, pp. 441-444.
[18] K.F . LEE AND H.W . HON, Speaker-independent phone recognition using hidden
Markov models, ASSP, 31 (1989), pp . 1641-1648.
[19] L.A . LIPORACE, Maximum likelihood estimation for mult ivariate observations of
Markov sources, IT , 28 (1982) , pp . 729-734.
[20] M . OSTENDORF , V.V . DIGALAKIS, AND O .A. KIMBALL, Prom HMM's to segment
models: A unified view of stochastic modeling for speech recognition, IEEE
Trans Speech and Audio Processing, 4 (1996), pp . 360-378.
[21] D. PALLETT , J. FISCUS , J . GAROFOLO, A. MARTIN , AND M. PRZYBOCKI, 1998
broadcast news benchmark test results: English and non- english word error
mte performance measures, in Proceedings of the DARPA Broadcast News
Workshop, Virginia, 1999.
[22] J . PICONE AND G . DODDINGTON, A phonetic vocoder, in ICASSP, Glasgow , 1989,

pp . 580-583.
[23J C. RIBEIRO AND I. TRANCOSO, Improving speaker recognisability in phonetic
uocoders, in ICSLP, Sydney, 1998, pp. 2611-2614.
[24J H.B . RICHARDS AND J .S. BRIDLE, The hdm : a segm en tal hidden dynamic model
of coarticulation, in ICASSP, Phoenix, 1999.
[25] M.J . RUSSELL AND W .J . HOLMES, Progress towards a unified model for speech
pattern processing, Proc. ·IOA, 20 (1998), pp. 21-28.
[26] M.J . RUSSELL AND R.K. MOORE, Explicit modelling of state occupancy in hidden
Markov models for automatic speech recognition, in ICASSP, 1985, pp . 5-8.
[27J K. TOKUDA , T . MASUKO , J . HIROI , T . KOBAYAHI , AND T . KITAMURA, A very
low bit rate speech coder using HMM-based speech recognition/synthesis tech-
niques, in ICASSP, Seattle, 1998, pp. 609-612.
[28J P . ZOLFAGHARI AND T. ROBINSON, A segmental formant vocoder based on linearly
varying mixture oj Gaussians, in EUROSPEECH, Rhodes, 1997, pp . 425-428.
MODELLING GRAPH-BASED OBSERVATION SPACES
FOR SEGMENT-BASED SPEECH RECOGNITION
JA MES R. GLASS'
Abstract . Most speech recognizers use an observat ion space which is bas ed on
a t empora l sequence of sp ectral "frames." T here is ano th er class of recogn izer which
further processes th ese fram es to produce a segment-based network , and repr esents each
segment by a fixed-dim ensional "feat ure." In such feature-based recognizers t he obser-
vatio n space takes t he form of a temp oral graph of feature vectors, so t hat any singl e
segmentat ion of an utterance will use a subset of all possible feature vectors. In this
work we describe a max imum a posteriori decoding strat egy for feature-based recogniz-
ers and der ive two normalization crit era useful for a segment-based Viterbi or A ' sear ch.
We show how a segment-based recognizer is able to obtain good results on th e tasks of
ph oneti c and word recognition .
Key words. Segment-based sp eech recognition, phon etic recogniti on.
1. Introduction. Th e fundamental goal of automat ic speech recog-

nizers is to identify the spoken words in a speech waveform. This process
genera lly begins with a signal processing stage which converts th e recorded
waveform to some form of acoustic representation. Subsequently, one or
more search stages, which incorpor ate linguistic const ra ints such as acoustic
and langu age models, attempt to identify th e most likely word hypoth eses.
For many years the prototypical acoust ic represent ation has consisted of a
time-frequency represent ation , which is computed at regular intervals over
t he speech signal (e.g., every 10 ms). This sequence of spect ra l observa-
tions, or frames, is intended to capture the salient time-frequency dynamics
of t he underlying phonological units present in the speech waveform. Most
acoust ic modelling techniques use t he spectral frame as t he input to pat-
te rn classifiers which attempt to determine th e probability th at a frame
was produced by a particular linguistic unit .
Over t he past two decades, first-ord er hidden Markov models (HMMs)
have emerged as the domin ant stochast ic model for speech recognition [25] .
HMMs use classifiers, such as Gaussian mixtures or art ificial neural net-
works, to emit a state-dependent likelihood or post erior prob ability on a
frame-by-frame basis. In contrast to HMMs and other frame-based process-
ing techniques, th e SUMMIT speech.recognizer developed by our group uses
a segment- based framework for its acoustic-phonet ic represent ation of th e
speech signal [7] . In this framework, acoustic feature vectors are ext racted
both over hypothesized segments , and at th eir bound aries, for phonetic
analysis. The resulting graph-based observation space differs considerably
from th e more conventional frame-based sequential observations.
• MIT Lab oratory for Comp ute r Science, 200 Technology Square, Ca mbridge, MA
02139, USA (glass@mit.edu) .
157
158 JAMES R. GLASS
Our interest in exploring segment-based approaches is based on our

experience with phonetic classification. One of our findings has been that
a homogeneous acoustic representation can compromise performance. For
example , the classification performance of stop consonants, / ptkbdg/, and
nasal consonants, /mnu/, have opposite dependencies on the duration of
the spectral analysis window [11]. While the ability to discriminate between
the stops improves with shorter analysis windows, the nasal consonant per-
formance improves as the analysis window is lengthened . Another moti-
vation for heterogeneous measurements is the observation that different
temporal basis vectors can significantly change phonetic classification per-
formance . For example, Halberstadt found that smoothly varying cosine-
basis functions are better suited for discriminating between vowels and
nasal consonants, while piecewise-constant basis vectors (e.g., averages) do
better for discriminating fricatives and stop consonants [10] . These results
support the notion that combining information sources could reduce overall
error.
By combining different sources of information and combining classifiers
we have been able to achieve state-of-the-art phonetic classification perfor-
mance. On the TIMIT acoustic phonetic corpus [5]' using standard training
and test sets, we have been able to achieve an 18.3% context-independent
error rate on a 39 phone classification problem [12]. These results are the
best that we have seen on this task. More generally, we have found that a
segment-based approach, whereby acoustic modelling is performed over an
entire segment, has provided us with a powerful framework for exploring a
variety of ways in which to incorporate acoustic-phonetic information into
the speech recognizer.
In this paper we describe the probabilistic formulation we currently use
for our segment-based recognizer. This recognizer differs considerably from
most frame-based decoding techniques, as well as from many other segment-
based approaches. In the following section we provide a brief survey of
other segment-based approaches which have been explored in the speech
recognition literature. This is followed by a derivation of the MAP decoding
techniques we have developed for our recognizer. Finally, we report some
current results on phonetic and word recognition experiments.
2. Background. There have been many segment-based approaches

which have been explored in the speech community [22]. In HMM-based
approaches, which retain many similarities to HMM-based techniques, re-
searchers have explored variable frame-rate analysis [24]' segment-based
HMMs [19], and the segmental HMM [15, 30]. In trajectory-based mod-
elling approaches, there have been stochastic segment models [23], para-
metric trajectory models [6], as well as stochastic and statistical trajectory
models [9] . These methods typically try to model dynamic behavior over
the course of a phonetic segment, although the modelling is ultimately per-
formed at the individual frame level. Finally, there are a class of approaches
SEGMENT-BASED SPEECH RE COGNIT ION 159
1.107. to 1.%1'.
S_t
Iy 1.tl
ey 1.01
n -1. 51
.... -l.n
-%.%1
Ih -% .%7
I -%.1'
... -%.5%
a
_
-%.n
-%.1%
F IG . 1. Th is figure contains seveml displays from the SUMMIT segment-based speech

recognizer. T he top two displays contain the speech waveform and associated spec-
trogmm, respectively. B elow them , a segment-ne twork display shows hypothesized seg-
ments; each segment spans a tim e mnge. T he darker colored segments show the segmen-
tat ion which achieved the highest score during search. T he two tmnscriptions below the
segment-network contain the best-scoring phone tic, and word sequences , respectively.
Th e pop-up menu shows the mnked log-likelihood mtio scores for the [ iJ in the word
"three" .
which could be called feature-based. Examples in t his area include the FEA-
TURE syste m [4], the SUMMIT system described in t his paper, as well t he
L A F F syste m [31].
T he SUMMIT segment-based syste m developed in our group uses a
segment- and landmark-based framework to represent the speech signal.
Acoustic or prob abilistic landm arks form t he basis for a phonetic network ,
or gra ph, as shown in Figur e 1. Acoustic features are ext racted over hy-
pothesized phonetic segments and at important acoust ic landmarks. Gaus-
sian mixture classifiers are th en used to provide phonet ic-level prob abilities
for both segment al and landm ark features. Words are represent ed as pro-
nunciat ion graphs whereby phonemic baseforms are expanded by a set of
phonological rules [14]. Phone probabilities are determined from tra ining
data. A prob abilistic MAP decoding framework is used. A modified Vit erbi
beam search finds t he best path through both acoustic-p honetic and pro-
nunciation gra phs while incorporating language const raints . All graph rep-
resentat ions are based on weighted finite-st ate tr ansducers [8, 21]. One of
the prop erties of t he segment-based framework is t hat, as will be described
later , the models must incorpor ate both positive and negative exa mples of
lexical units. Fin ally, a secondary A * search can provide word-gra ph or
N - best sentence outputs for furthe r processing.
160 JAMES R. GLASS
3. MAP decoding. In most probabilistic formulations of speech

recognition the goal is to find the sequence of words W* = WI, . . • , W N ,
which has the maximum a posteriori (MAP) probability P(WIA), where
A is the set of acoustic observations associated with the speech waveform:
(3.1) W* = argmax P(WIA) .
w
In most speech recognizers, decoding is accomplished by hypothesizing
(usually implicitly) a segmentation, S, of the waveform into a sequence
of sub-word states or linguistic units , U. A full search would consider all
possible segmentations for each hypothesized word and sub-word sequence,
so that Equation 3.1 can be rewritten as:
(3.2) W* = arg max

s,u,w
I: P(S, U,WIA) .
VS,U
If some form of dynamic programming or graph search, such as Viterbi or

A", is used to find the best path, this expression can be simplified to
(3.3) S*, U*, W* = arg max P(S, U, WIA).

s ,u,w
In subsequent equations, S* and U* will be dropped for notational simplic-
ity. Using Bayes rule, the term P(S, U, WIA) can be decomposed:
(3.4) P(S U WjA) = P(AjS, u,W)P(SIU, W)P(UIW)P(W)

, , P(A) '
Since P(A) is independent of S, U, and W, it will not affect the outcome

of the search , and is usually ignored unless it is being used as a normal-
ization mechanism . The term P(W) is estimated by a language model
which predicts the a priori probability of a particular sequence of words
being spoken . The term P(UIW) can be considered to be a phonologi-
cal model which predicts the probability of a sequence of sub-word units
being generated by the given word sequence W . In an HMM this term
might correspond to the state sequence corresponding to a word sequence,
and could be completely deterministic. Some speech recognizers incorpo-
rate a stochastic component at this level to attempt to model phonological
variations in the way a word can be realized in fluent speech (e.g., "did
you" being realized as the phoneme sequence /dljU/) [14, 26]. The term
P(SIU, W) models the probability of the segmentation itself, and typically
depends only on U. In HMMs for example, this term corresponds to the
likelihood of a particular state sequence, and is generated by the state
transition probabilities. More generally, this term can be considered as a
duration model which predicts the probability of individual segment dura-
tions . Many researchers have explored the use of more explicit duration
models for recognition, especially in the context of segment-based recogni-
tion [18, 23] . The remaining term in Equation 3.4, P(AIS, U,W) , relates to
SEGMENT -BASED SPEECH RECOGNITION 161
F IG. 2. Tw o segm entat ions (in solid lines) through a simple five segme nt gmph
with acoustic observations {al , . . . , as} . Th e top segm entation is associated with obser-
vations {al , a3,as} , while the bott om segmen tation is associated with {al , a2, a4, as} .
the ASR acoustic mod elling component, and is the subject of this paper.
For simplicity we will assume that acoust ic likelihoods are condit ionally
ind epend ent of W so that P (A IS, U, W ) = P(A IS, U). This assumption is
also standa rd in convent ional HMM-b ased syste ms.
In conventional speech recognizers, the acoustic observation space, A ,
cor res ponds to a temp oral sequence of acoustic frames (e.g., spect ra l slices).
Each hypoth esized segment , s., is repr esent ed by t he series of frames com-
puted bet ween segment start and end tim es. Thus, t he acoust ic likelihood
P(AIS , U), is derived from t he same observat ion spac e for all word hy-
potheses. In feat ure-based recognition, as illustrat ed in Figur e 2, each
segment , s., is repr esent ed by a single feature vector, ai. Given a par-
ti cular segment at ion, S, (a contiguous sequence of segments spanning the
ent ire ut terance), A consists of X , t he feature vectors associate d with the
segments in S, as well as Y , t he featu re vecto rs associate d with segments
not in S, such t hat A = X u Y and X n Y = 0. In order to compa re
different segmentations it is necessary to predict the likelihood of both X
and Y , since P (A IS, U) = P (X ,Y IS, U). Thus, in addit ion to th e obser-
vati ons X associated with t he segments in S, we must consider all ot her
possible observations in t he space Y, corre sponding to the set of all other
possible segments, R . In the top path in Figure 2, X = {al ,a3, as}, and
Y = {a2,a4}. In the bottom path , X = {al , a2,a4,as} , and Y = {a3}'
The to t al observation space A , contains both X and Y , so for MAP de-
coding it is necessary to est imate P (X ,YIS, U). Note t hat since X implies
S we can say P (X ,Y IS, U) = P(X ,Y IU). The following sections describe
two methods we have develop ed to account for t he ent ire observation space
in our segment-based recognizer .
3 .1. M o d elling non-lexical units. One possible method for mod-

elling Y is to assign them to a non-lexical unit , or anti-phone, Ci (e.g.,
too big or too small) . Given a segmentat ion, S, assign feature vectors
162 JAMES R. GLASS
in X to valid linguistic units, and all others to Y to the non-unit, n:.

Since P(X, YIn:) is a constant for any given segment graph, we can write
P(X, YIU) assuming independence between X and Y:
_ P(XIn:) P(XIU)
(3.5) P(X, YIU) = P(XIU)P(Yla) P(XIn:) oc P(XIn:) '
Thus, we need onlyconsider segments in S during search
(3.6)
Where Ui is the linguistic unit assigned to segment Si, and its associated
feature vector observation Xi .
The advantage of the anti-phone normalization framework for decod-
ing is that it models the entire observation space, using both positive and
negative examples . Log likelihood scores are normalized by the anti-phone
so that good scores are all positive, while bad scores will be negative . In
Figure 1 for example, only the top two log likelihood ratio 's for the Iii seg-
ment were positive. Invalid segments, which do not correspond to a valid
phone, are likely to have negative scores for all phonetic hypotheses. Note
that the anti-phone is not used for lexical access, but purely for normaliza-
tion purposes. In general, it is useful for pattern matching problems with
graph-based observation spaces.
3.2. Near-miss modelling. Anti-phone modelling divides the ob-
servation space into essentially two parts; subsets of segments which are
either on or off a hypothesized segmentation S. The anti-phone model is
quite general since it must model all examples of observations which are
not phones . A larger inventory of anti-phones might better model segments
which are near misses of particular phones. One such method, called near-
miss modelling, was developed by Chang [1, 2] . Near-miss modelling par-
titions the observation space into a set of near-miss subsets, where there is
one near-miss subset, Ai, associated with each segment, Si, in the segment
network (although Ai could be empty) . During recognition , observations in
a near-miss subset are mapped to the near-miss model of the hypothesized
phone . The net result can be represented as:
n
(3.7) W* = arg max
s,u,w .
II P(xdui)P(Ailui)P(silui)P(UIW)P(W)
t=l
where P(Ailui) is computed as the product of observations in Ai being

generated by the near-miss model associated with Ui (i.e., Ui)'
Near-miss models can be an anti-phone, but can potentially be more
sophisticated. The challenge is to partition the observation space during
the search so that all observations in A are included . Chang developed
SEGMENT-BASED SPEECH RECOGNITION 163
an effective criterion for creating near-miss subsets. By definition, the

near-miss subsets associated with any segmentation S must be mutually
exclusive, and collectively exhaustive so that A = U(Xi UA i ) "lSi E Sand
Ai n A j = 0 "lSi, Sj E S, i i- j . Note that for any given segmentation,
S, X = U Xi, and Y, Y = UAi. Chang recognized that a temporal crite-
rion could be used to guarantee proper near-miss subset creation. This is
because any particular segmentation through a segment network accounts
for all times exactly once. Thus, segments which all span the same time
naturally form a near-miss set. Using a common reference point, such as
the segment mid-point, appropriate near-miss subsets can be defined which
satisfy the necessary near-miss conditions [1] .
3.3. Modelling landmarks. In addition to modelling segments, it
is also possible to model phonetic transitions at hypothesized landmarks
or phonetic boundaries. If we represent landmark-based observations by
Z, it is necessary to determine P(X, Y, ZIS, U) . Assuming conditional in-
dependence between the segmental and landmark-based observations, we
can say P(X, Y, ZIS, U) = P(X, YIS, U)P(ZIS, U). Further, if we assume
conditional independence between landmark-based observations, we can
represent P(ZIS, U) by
m
(3.8) P(ZIS, U) = II P(zdSU)
i=l
where Zi is the feature vector extracted at all m hypothesized landmarks

in the speech waveform. Since every segmentation accounts for every land-
mark there is no need for the normalization procedures discussed for seg-
mental observations. Note that depending on the hypothesized segmenta-
tion, some landmarks will be considered to be transitions between lexical
units, whereas others will be considered to be internal to a unit.
Landmark-based models are able to capture the relative dynamics
which occur between phones. These models have been used to effectively
generate an initial segmentation graph for our segment-based recognizer
by using them in a forward Viterbi pass followed by a backwards A *
search [2]. Block processing enables the segment network to be computed
with pipelined computation [17] .
4. Experiments. We have evaluated our segmental framework in
both phonetic and word recognition experiments. The experiments de-
scribed in the next two sections have been presented in more detail else-
where [12, 32] . All experiments have made use of the SUMMIT segment-
based speech recognition system [7]. This recognizer combines segment-
and landmark-based classifiers. Feature extraction is typically based on
averages and derivatives of Mel-frequency cepstral coefficients (MFCCs),
plus additional information such as energy. Principal component analysis
is used to normalize and whiten the feature space . Acoustic models are
based on mixtures of diagonal Gaussians.
164 JAMES R. GLASS
4.1. Phonetic recognition. Phonetic recognition experiments were

carried out the the TIMIT acoustic-phonetic corpus [5] . This corpus has been
used by many researchers to report phonetic classification and recognition
experiments. In our experiments, we used the standard 462 speaker train-
ing set , and 24 speaker core-test set. By convention, phonetic recognition
errors were reduced to the common 39 classes. Segmental and landmark
representations were based on five variations of averages and derivatives
of 12 Mel-Frequency and PLP cepstral coefficients, plus energy and du-
ration [12] . 4-fold aggregation was used to improve the robustness of the
Gaussian mixtures [13] . The language model used in all experiments was
a phone bigram based on the training data.
The phonetic error rates obtained on the core test set varied from
30.1% to 24.4% depending on the sophistication of the acoustic measure-
ments [12] . The best results were obtained using an anti-phone model and
a committee-based classifier to combine the outputs of five different seg-
mental, and landmark models. Table 1 compares this result with the best
results reported in the literature.
TABLE 1
Reported phonetic recognition error rates on the TIMIT core test set.
Method % Error
Triphone CDHMM [16] 27.1
Recurrent Neural Network [27] 26.1
Bayesian Triphone HMM [20] 25.6
Anti-Phone, Heterogeneous classifiers [12] 24.4
4.2. Word recognition. Word recognition experiments have been

performed on a spontaneous-speech, telephone-based, conversational inter-
face task in the weather domain [34] . For these experiments, a 50,000 ut-
terance training set was used, and an 1806 utterance test set was used con-
taining in-domain queries. The recognizer for this task used a vocabulary
of 1957 words, as well as a class bigram and trigram language model [8, 32] .
As shown in Table 2, the word error rate (WER) obtained by the system
using only context-dependent segmental models was 9.6%. When landmark
models were used, the WER decreased to 7.6%. The overall performance
decreased to 6.1% when both models were combined.
TABLE 2
Word recognition error rates for a weather domain task.
Method % Error
Segment models 9.6
Landmark models 7.6
Combined 6.1
5. Discussion. The results shown in Table 1 compare a number of

published results on phonetic recognition which have been based on the
TIM IT core test set . There are still differences regarding the complexity
of the acoustic and language models, thus making a direct comparison
somewhat difficult. Nevertheless, we believe our results are competitive
with those obtained by others, and that our performance will improve when
we increase the complexity of our models.
The framework we have outlined in this paper provides flexibility to ex-
plore the relative advantages of segment versus landmark representations.
As we have shown, it is possible to use only segment-based feature vectors,
or landmark-based feature vectors (which could reduce to frame-based pro-
cessing), or a combination of both.
The anti-phone normalization criterion can be interpreted as a like-
lihood ratio. In this way it has similarities with techniques being used
in word-spotting, which compare acoustic likelihoods with those of "filler"
models [28, 29, 33] . The likelihood or odds ratio was also used by Cohen
to use HMMs for segmenting speech [3] .
One of the most questionable assumptions made in this work was the
independence assumption made between the observations in X, and those
in Y. Segments that temporally overlap with each other are clearly re-
lated to each other to some degree. In the future, it would be worthwhile
examining alternative methods for modelling the joint X, Y space .
6. Summary. In this paper we have described a probabilistic de-
coding framework for decoding a graph-based observation space. This
method is particularly appropriate for segment-based speech recognizers
which transform the observation space from a sequence of frames, to a
graph of features. Graph-based observation spaces allow for a wide-variety
of alternative modelling methods to be explored , than could be achieved
with frame-based approaches. We have developed two techniques for de-
coding graph-based observation spaces, based on anti-phone or near-miss
modelling , and have achieved good results on phonetic recognition . We
have also observed improved performance when combining segmental mod-
els into a landmark-based word recognizer.
Acknowledgements. There are a number of colleagues, past and
present, who have contributed to this work including Jane Chang, Andrew
Halberstadt, T .J. Hazen, Lee Hetherington, Michael McCandless, Nikko
Strom, and Victor Zue. The author would also like to thank Mari Ostendorf
for providing many useful comments that helped improve the paper. This
research was supported by DARPA under contract N66001-99-C-1-8904
monitored through the Naval Command, Control and Ocean Surveillance
Center.
166 JAMES R. GLASS
REFERENCES
[1] J . Chang. Near-miss modeling: A segment-based approach to speech recognition.

Ph .D. thesis, EECS , MIT, June 1998.
[2] J . Chang and J. Glass. Segmentation and modeling in segment-based recognition.
In Proc. Eurospeech , pages 1199-1202 , Rhodes, Greece , October 1997.
[3] J . Cohen . Segmenting speech using dynamic programming. Journal of the Acoustic
Society of America, 69(5) :1430-1438 , May 1981.
[4J R. Cole, R. Stern, M. Phillips, S. Brill, A. Pilant, and P. Specker . Feature-based
speaker-independent recognition of isolated letters. In Proc. ICASSP, pages
731-733, Boston, MA, April 1983.
[5] J . Garofolo, L. Lamel , W. Fisher, J . Fiscus, D. Pallet, and N. Dahlgren. The
DARPA TIMIT acoustic-phonetic continuous speech corpus CDROM . NTIS
order number PB91-505065 , October 1990.
[6J H. Gish and K. Ng. A segmental speech model with applications to word spotting.
In Proc. ICASSP, pages 447-450, Minneapolis, MN, April 1993.
[7J J . Glass , J . Chang, and M. McCandless. A probabilistic framework for feature-
based speech recognition. In Proc. ICSLP, pages 2277-2280, Philadelphia,
PA, October 1996.
[8] J . Glass , T . Hazen, and L. Hetherington. Real-time telephone-based speech recog-
nition in the Jupiter domain. In Proc. ICASSP, pages 61-64, Phoenix, AZ,
March 1999.
[9] W. Gold enthal. Statistical trajectory models for phonetic recognition . Technical
report MIT / LCS/ T R-642, MIT Lab . for Computer Science, August 1994.
[10] A. Halberstadt. Heterogeneous acoustic measurements and multiple classifiers for
spee ch recognition. Ph .D. thesis, MIT Dept. EECS , November 1998.
[I1J A. Halberstadt and J . Glass . Heterogeneous measurements for phonetic classifica-
tion . In Proc. Eurospeech, pages 401-404, Rhodes, Greece , September 1997.
[12] A. Halberstadt and J. Glass . Heterogeneous measurements and multiple classifiers
for speech recognition. In Proc. ICSLP, pages 995-998, Sydney, Australia,
December 1998.
[13J T . Hazen and A. Halberstadt . Using aggregation to improve the performance of
mixture Gaussian acoustic models . In Proc. ICASSP, pages 653-656, Seattle,
WA, May 1998.
[14J L. Hetherington . An efficient implementation of phonological rules using finit e-
st ate transducers. In Proc. Eurospeech, pages 1599-1602, Aalborg, Denmark,
September 2001.
[15] W . Holmes and M. Russell. Modeling speech variability with segmental HMMs .
In Proc. ICASSP, pages 447-450, Atlanta, GA , May 1996.
[16] L. Lamel and J.L . Gauvain. High performance speaker-independent phone recog-
nition using CDHMM . In Proc . Eurospeech; pages 121-124, Berlin , Germany,
September 1993.
[17] S. Lee and J . Glass . Real-time probabilistic segmentation for segment-based speech
recognition. In Proc. ICSLP, pages 1803-1806, Sydney, Australia, December
1998.
[18] K. Livescu and J. Glass. Segment-based recognition on the PhoneBook task: Initial
results and observations on duration modeling. In Proc. Eurospeech; pages
1437-1440, Aalborg, Denmark, September 2001.
[19J J . Marcus. Phonetic recognition in a segment-based HMM. In Proc. [CASSP,
pages 479-482, Minneapolis, MN, April 1993.
[20] J. Ming and F . Smith. Improved phone recognition using bayesian triphone models .
In Proc. ICASSP, pages 409-412, Seattle, WA, May 1998.
[21] M. Mohri. Finite-state transducers in language and speech processing. Computa-
tional Linguistics, 23(2) :269-311, June 1997.
[22J M. Ostendorf, V. Digilakis, and O. Kimball. From HMM's to segment models : a
unified view of stochastic modelling for speech recognition. IEEE Trans. SAP,
4(5) :360-378, September 1996.
[23J M. Ostendorf and S. Roucos. A stochastic segment model for phoneme-based con-
tinuous spee ch recognition. IEEE Trans. ASSP, 37(12):1857-1869, December
1989.
[24J K. Ponting and S. Peeling. The use of variable frame rate analysis in speech
recognition. Computer Speech and Language, 5 :169-179,1991.
[25] L. Rabiner. A tutorial on hidden Markov models and selected applications in
speech recognition. Proceedings of the IEEE, 1989.
[26J M. Riley and A. Ljolje . Lexical access with a statistically-derived phonetic network.
In Proc. Eurospeech, pages 585-588, Genoa, Italy, September 1991.
[27] A. Robinson. An application of recurrent nets to phone probability estimation.
IEEE Trans. Neural Networks, 5(2) :298-305, March 1994.
[28] J . Rohlicek, W . Russell, S. Roucos, and H. Gish . Continuous hidden Markov
modelling for speaker-independent word spotting. In Proc. ICASSP, pages
627-630, Glasgow, Scotland, May 1989.
[29] R . Rose and D. Paul. A hidd en Markov model based keyword recognition system.
In Proc. ICASSP, pages 129-132, Albuquerque, NM, April 1990.
[30] M. Russell . A segmental HMM for speech pattern modelling. In Proc. ICASSP,
pages 499-502, Minneapolis, MN, 1993.
[31] K. Stevens. Lexical access from features . In Workshop on speech technology for
man-machine interaction, Bombay, India, 1990.
[32] N. Strom, L. Hetherington, T . Hazen , E. Sandness, and J . Glass . Acoustic mod -
elling improvements in a segment-based speech recognizer. In Proc. IEEE
Automatic Speech Recognition and Understanding Workshop, pages 139-142,
Keystone, CO , 1999.
[33] J . Wilpon, L. Rabiner, C.H. Lee, and E . Goldman. Automatic recognition of
keywords in unconstrained speech using hidden Markov models . IEEE Trans .
A SSP, 38(11) :1870-1878, November 1990.
[34] V. Zue, S. Seneff, J . Glass, J . Polifroni, C. Pac, T . Hazen , and L. Hetherington.
Jupiter: A telephone-based conversational interface for weather information .
IEEE Trans . Speech and Audio Proc., 8(1) :85-96, January 2000.
TOWARDS ROBUST AND ADAPTIVE SPEECH
RECOGNITION MODELS
HERVE BOURLARD', SAMY BENGIOt, AND KATRIN WEBER't
Abstract. In this paper, we discuss a family of new Automatic Speech Recognition

(ASR) approaches, which somewhat deviate from the usual ASR approaches but which
have recently been shown to be more robust to nonstationary noise, without requiring
specific adaptation or "mult i-style" training. More specifically, we will motivate and
briefly describe new approaches based on multi-stream and subband ASR. These ap-
proaches extend the standard hidden Markov model (HMM) based approach by assum-
ing that the different (frequency) streams representing the speech signal are processed
by different (independent) "experts" , each expert focusing on a different characteristic
of the signal, and that the different stream likelihoods (or posteriors) are combined at
some (temporal) stage to yield a global recognition output. As a further extension to
multi-stream ASR, we will finally introduce a new approach, referred to as HMM2, where
the HMM emission probabilities are estimated via state specific feature based HMMs
responsible for merging the stream information and modeling their possible correlation.
Key words. Robust speech recognition, hidden Markov models , subband process-
ing, multi-stream processing.
1. Introduction. Current automatic speech recognition systems are

based on (context-dependent or context-independent) phone models de-
scribed in terms of a sequence of hidden Markov model (HMM) states,
where each HMM state is assumed to be characterized by a stationary
probability density function. Furthermore, time correlation, and conse-
quently the dynamics of the signal, inside each HMM state are also usually
disregarded (although the use of delta and delta-delta features may cap-
ture some of this correlation) . Consequently, apart from the dependencies
captured via the topology of the HMM, most temporal dependencies are
usually very poorly modeled.' Ideally, we want to design a particular HMM
that is able to accommodate multiple time-scale characteristics so that we
can capture phonetic properties, as well as syllable structures, which seem
to have many attractive properties [9], including invariants that are more
'Daile Molle Institute for Perceptual Artificial Intelligence (IDIAP) , and Swiss Fed-
eral Institute of Technology at Lausanne (EPFL) , 4, Rue du Simplon, CH-1920 Martigny,
Switzerland (bourlard@idiap.ch) .
tDalle Molle Institute for Perceptual Artificial Intelligence (IDIAP), 4, Rue du Sim-
pion , CH-1920 Martigny, Switzerland (bengio@idiap.ch).
t weber@idiap.ch.
IThis problem is not specific to the fact that phone models are generally used . Whole
word models , or syllable models, built up as sequences of HMM states will suffer from
exactly the same drawbacks, the only potential advantage of moving towards "larger"
units being that one can then have more word (or syllable) specific distributions (usually
resulting in more parameters and an increased risk of undersampled training data) .
Consequently, building an ASR system simply based on syllabic HMMs will not alleviate
the limitations of the current recognizers since those models will still be based on the
short-term piecewise stationary assumptions mentioned above.
169
170 HERVE BOURLARD ET AL.
robust to noise. For example, acoustic features such as the modulation

spectrogram/ exhibit some correlation with syllabic features and can be
used to improve state-of-the-art ASR systems [33] . It is, however, clear
that those different time-scale features will also exhibit different levels of
stationarity and will require different HMM topologies to capture their
dynamics.
There are many potential advantages to such a multi-stream approach,
including:
1. The definition of a principled way to merge different temporal
knowledge sources such as acoustic and visual inputs, even if the
temporal sequences are not synchronous and do not have the same
data rate - see [28] and [29] for further discussion about this.
2. The possibility to incorporate multiple time resolutions (as part of
a structure with multiple unit lengths, such as phone and syllable) .
3. Multiband-based ASR [6, 14] involving the independent processing
and combination of partial frequency bands is a very particular case
of multi-stream recognition. Although this will not be explicitly
discussed here, there are many potential advantages to this multi-
band approach, including (i) better robustness to speech impaired
by narrowband noise, and (ii) the possibility of applying different
time/frequency tradeoffs and different recognition strategies in the
subbands.
In the following, we will not discuss the underlying algorithms ("com-
plex" variants of Viterbi decoding, if one wants to take the possible asyn-
chrony into account), nor detailed experimental results (see [11] for recent
results). Instead, we will mainly discuss different combination strategies
pointing towards the same formalism.
2. Psycho-acoustic evidence. It seems to me that what can happen
in the future is... that experiments get harder and harder to make, more and
more expensive... and scientific discovery gets slower and slower. (Richard
Feynman, 1918-1988, The Character of Physical Law, Cambridge, MA,
p.172.)
2.1. Product of errors rule and its interpretation. The work of
Fletcher and his colleagues (see the insightful review of his work in [1]) sug-
gests that human decoding of the linguistic message is based on decisions
within narrow frequency sub bands that are processed quite independently
of each other. Empirical evidence suggests that the combination of deci-
sions from these subbands is done at some intermediate level and in such
a way that the global error rate is equal to the product of error rates in
the subbands. In other words, if we have two frequency bands (streams)
Cl and C2, and each of them is resp ectively yielding a probability of error
(error rate) e(qjlx 1 ) and e(qjlx 2 ) for a particular class qj and an input
2Initially proposed as a way to assess room acoustics [16J .

TOWARDS ROBUST SPEECH RECOGNITION MODELS 171
pattern x = {Xl , x 2 } , where xl and x 2 represent the output features of the

two frequency streams" , the total error rate e( qj lXi, x 2 ) resulting from the
simultaneous use of the two streams is given by:
(2.1)
Although this conclusion is often questioned by the scientific community" ,
it is probably not worth arguing too long about it since it is pretty clear
that (2.1) is anyway the optimal rule to obtain the best performance out
of a possibly noisy multi-stream system (but it requires perfect knowledge
about which stream, if any, is noisy) . Moreover, a similar rule can usually
explain some of the empirical observations in audio-visual processing (see,
e.g., [24] and [20]).
Although pretty simple, rule (2.1) is not always easy to interpret (and
even less for engineers!). So let us have a closer look at it . Since the prob-
ability of being correct whenever we assign a particular observation x to a
class q is equal to the a posteriori probability P(qlx) (i.e., the probability
of error is equal to 1 - P(qlx) , see [8], page 12)5, rule (2.1) can also be
written as:
(2.2)
2 2
= 1- L P(qjl x k
) + IT P(qjlx k
) ,
k=l k=l
where P(qj\x k ) denotes the class posterior probabilities obtained for the
k-th input stream. Rewriting (2.2) in terms of (total) correct recognition
probability (P(qj lxi, x 2 ) = 1 - e(qj Ixl, x 2 )) , we have:
2 2
(2.3) P(qjlx l , x 2 ) = L P(qjl x k ) - II P(qjlx k
) .
k=l k=l
In the case of K streams, the above expression will have 2K - 1 terms,

containing all possible stream combinations.
These expressions are quite reasonable since they also reflect a stan-
dard property of probabilities of joint events." Actually, this product of
3Since we decided not to deal with the temporal const raints, this notation is over-
simplified. In th e case of temporal sequences, xl and x 2 will be sequences (possibly
of different lengths and different rates) of features, and qj will be a sequence of HMM
st ates.
4Since the relevant Fletcher experiments were done (i) with nonsense syllables only,
and (ii) using high-pass or low-pass filters (i.e., two streams) only, it is not clea r whether
or not this is an accurate statement for disparate bands in cont inuous speech.
5See Section 3 in this ar ticl e for further evidence.
6The probability of union of two events P(A or B) = P(A) + P(B) - P(A, B), which
is also equal to P(A) + P(B) - P(A)P(B) if events A and B are independent. Indeed, in
estimating the proportions of a sequence of trials in which A and B occur, respectively,
one counts twice those trials in which both occur.
172 HERVE BOVRLARD ET AL.
0.9 r - - _
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

pI
FIG . 1. "Optimal" classification stmtegy based on two (independent) observation

streams yielding posterior probabilities P(qjlx 1 ) and P(qjlx 2 ) . The grey level represents
the "total" probability of correct recognition (with white corresponding to the maximum
probability), and the different curves represent the equal recognition probability curves
(as a function of P(qlx 1 ) and P(qlx 2 ) ) above which the probability of correct recognition
will be higher than a prescribed value.
errors rule tells us that the probability of correct classification on human

full-band hearing is equal to the probability that there is correct (human)
classification in any subband. Consequently, this also means that human
hearing seems capable of processing numerous bands and selecting the one
that gives correct recognition.
The resulting (very simple but nonlinear) product of errors function is
illustrated in Figure 1 for all possible values of P( qj Ix l ) (horizontal axis)
and P(qjlx 2 ) (vertical axis). From this figure, it is interesting to note how
much flexibility an "optimal" multi-stream system potentially has in keep-
ing the (total) probability of correct recognition above a certain threshold,
even if one of the streams is extremely noisy (and yields high error rates) .
This can indeed be measured by the area above a given equal recognition
rate curve. For example, for P( qj Ix l , x2 ) = 0.9, nearly one third of the
space is available! It is clear that this property cannot be achieved by
using the usual product of likelihoods, where if one of the likelihoods is
poorly estimated, the whole product deteriorates.7
7In addition to the fact that it is usually difficult to compare/combine likelihoods

TOWARDS ROBUST SP EECH RECOG NITION MODELS 173
This conclusion remains valid for more than two st reams. Actually, it
can even be shown th at th e area above a given (multi-dimensional) equal
error rat e surface grows exponenti ally with th e number of st rea ms. To
make th e link easier with what follows in t he remainder of t his art icle,
observe t hat, in t he case of t hree input st reams, (2.3) becomes:
3 3
P (qj IX1 , X2 , X3 ) = L P(qj lx k) + II P(qj lx k)
k=l k= l
(2.4)
3
L P (qjl xk)P(q j lx l ) .
l > k= l
Obviously, this reflects a "perfect" world. In actual engineering systems

th ough the poste rior probabilities P(qjlx k) will have to be est imated on
th e basis of a set of parameters 8 , and , in the case of two streams , (2.3)
should be written:
2 2
(2.5) P(qjl x 1 , x 2 ,8) = L P (qj lxk, 8 ) - II P(qjl x k,8) .
k= l k= l
Figure 1 does not change, but the position in the space depends on 8 , as
well as on the different st ream features. Ideally, robust tra ining and ad ap-
tation should be performed in the 8 space to guarantee t hat P (qj lx 1 , x 2 , 9 )
is always above a cert ain threshold, or to directly maxim ize (2.5). In t he
following, we discuss approaches going in t hat direction.
2.2. Discussion. Th e above analysis allows us to draw a few conclu-
sions and to design t he features of an "opt imal" ASR system:
1. Human hearin g performs combinat ion of frequency st reams accord-
ing to th e product of errors rule discussed above. In t his case (and
assuming th at the subbands are independent , which is false), cor-
rect classification of any subb and is empirically equivalent to cor-
rect full-band classification . In subband-based ASR systems, this
m eans that we should design the syst em and the traini ng criterion
to maximize the classification performance on subbands, while also
making sure that the subband errors are ind ependent .
2. As a direct consequence of th e above, it is also obvious that th e
more subbands we use, th e higher th e full-band correct classifica-
tion rate will be. As done in human hearing , A SR system s should
thus use a large number of subbands to have a better chance to
increase recognition rat es. It is interestin g to note here th at this
t rend has recently been followed in [15].
computed from feat ur es in different spaces , possibly of di fferent d imensions (since like-
lihoods, as usually computed (assu ming Gaussia n densiti es wit h diagonal covariance
mat rices), are "d imensional" , i.e., dependent on the dimension of t he feat ure space).
3. In order to estimate the reliability of each stream, ASR systems

should be able to estimate subband posteriors as accurately as pos-
sible. We will show in the next section that this is not impossible.
4. If ASR systems can reliably estimate local posteriors, we can im-
plement the product of errors rule, which should guarantee the
minimum of errors (if the above conditions are satisfied) . Further-
more, each time we improve the classification rate in any subband,
the recognition rate should improve.
3. Estimating posteriors. The purpose of models is not to fit the
data but to sharpen the questions. (Samuel Karlin, 1923-, 11th R.A. Fisher
Memorial Lecture, Royal Society, 20 April 1983.)
From the discussion above, it seems clear that we should work on the basis
of a posteriori probabilities" . Given that we often work in the framework of
hybrid HMM/ANN systems [5] (using artificial neural networks (ANN) for
estimating local posterior probabilities which are transformed into scaled
likelihoods used as HMM emission probabilities), and although some of the
arguments below will also be valid for likelihood-based systems, we will
focus our discussion on posteriors.
As initially reported in [5], Figure 2 illustrates the fact that an ANN
can reliably estimate local posterior probabilities P(qjlx) . Indeed, recalling
the properties of posterior probabilities discussed in the previous section,
good estimates of posterior probabilities should also be a measure of the
fraction of correct classification. Consequently, when representing the cor-
rect classification rate as a function of the posterior probabilities as esti-
mated at the output of a neural network, the ideal Bayes (posterior-based)
classifier would yield a diagonal, which is quite the case for both the train-
ing data and the cross-validation data (not used for training, but for which
correct classification was known).
Dividing these local posterior probabilities by the prior probabilities
P( qj) as estimated on the training set, yields scaled local likelihoods that
can be used to compute [12]
P(MIX) P(XIM)
(3.1)
P(M) P(X)
where M represents a complete HMM (modeling a specific subword unit,

a word, or a sentence) composed of several units computing p(q(;I
p q,
.x)), and
X an observation sequence associated with M. This can then be simply
multiplied (as in usual HMMs) by P(M) to include external knowledge
sources (such as a language model).
4. Multi-stream and mixture of experts. From an engineering
perspective, one way to introduce the multi-stream formalism in a pattern
8Which are known , anyway, to yield the minimum error rate solution.
T OWARDS ROBUST SPEECH RECOGNIT ION MODELS 175
Trained Network Out put vs . Fr acti on Corre ct ly Classifi ed
+
+ <>
0 .9 + <>
+6 0 0
0 .8 + 0°
o
o
0 .7
.,o
"
~
~
0
0.6
U
~ 0 .5
.,
-~
t!
0 .4
'"
~
"-
0 .3
0.2
0.1
0
0 0 .1 0.2 0 .3 0 .4 0 .5 0 .6 0.7 0 .8 0 .9
Out pu t Pro babi lity
FIG. 2. It is possible to genemte "good" posterior probabilities out of a neuml

network, and these are ind eed good measures of the probability of being correct . Th is
plot was genemted on real speech data by collecti ng statistics over the acoustic pamme-
ters from 1750 Resource Managem ent speaker-in dependent tm ining sen tences and 500
cross-v alidation sent ences (no t used for tm ining, but for which correct classificati on was
known).
classificati on task (such as ASR) is to use the approac h of mixture of ex-

perts, as proposed in t he framework of neur al networks [4]. The genera l
idea of a mixt ure of experts is to process t he (same) inpu t x according to
different linear or nonlin ear (neural network ) functi ons ("experts"), and
to combine the out puts of each expert according to a weighted sum, and
where the weights also result from a (linear or nonlin ear ) functi on of the
input pattern x .
Typically, this approac h (as for HMMs) can be formu lated in term s
of latent vari ab les, where the missing variable is the set of experts who
ar e reliable at each ste p. As illustr at ed in Figur e 3, let M repr esent the
hypo thesized mod el (HMM) associated with an input sequence X. If £ =
{E 1 , . . . , Ek , . . . , E K} repr esent s a set of mutua lly exclusive and exhaustive
experts'' (and where P (Ek ) is defined as the probabili ty t ha t Ek is t he most
reliable expert ), then P (M IX) can be est imated as:
9 As d iscussed lat er , the initial multi-st ream ap proach (Sectio n 5.1) was not using
stri ctl y exha ust ive expe rts since t hey did not cover all possible stream combina t ions.
The fuJI combinat ion approac h, as discussed in Secti on 5.3, will act ually use a ll possible
combinat ions.
FIG. 3. Posterior-based mixture of experts. Experts (e.g., neural networks) are

extracting their own posterior estimates, which are then combined through weights also
estimated (by the "gating network") from the data. These weights could also be adapted
online.
K
P(MIX) = LP(M,EkIX)
k=1
K
(4.1) = LP(MIEk,X)P(EkI X)
k=1
K
~ L P(MkIXk)P(EkI X)
k=1 .
where x» represents the respective inputs of expert/function Ek 10, M k

the model for the speech unit M used to process x», and P(EkIX) the
(relative) reliability of expert Ek given the whole inputY The approx-
imations in (4.1) result from the assumptions that (i) the probability of
a model M given a particular expert Ek is only estimated from the sub-
model Mk associated with the expert, and (ii) that expert-specific model is
only looking at its specific input features. As briefly recalled in Section 3,
segment-based posteriors in (4.1) can easily be estimated when using arti-
ficial neural networks.
Ideally, as discussed in [6, 14,29] and illustrated in Figure 4, the expert
combination presented above should take place at the level of M, i.e., at the
level of the particular (non-emitting) states denoted "Q9" . However, this is
not trivial and will often require a significant adaptation of the recognizer.
lOIn the case of multi-stream inputs, x» will typically be a subset of X (containing

the features relative to Ek)'
llSince, as illustrated in Figure 4, each sequence Xk will be processed with a differ-
ent/specific HMM.
TOWARDS ROBUST SPEECH RECOGNIT ION MODELS 177
Q9 = Recombination at the sub-unit level

F IG. 4. General form of a K -streams recognizer wit h anchor points between speech
units (to f orce synchrony between the different streams) . Note that the model topology
is not necessarily the same f or the different sub-systems.
It is only in the case of segment likelihoods combination (by products)

t hat one can develop a t ractable solut ion to t his opt imizatio n problem .
In t his case, it can be shown t hat t he product of segment-based, expert-
specific, likelihood s can be expressed as local likelihood products of an
equivalent 1st orde r HMM. In t his case, some adaptation of t he t ransition
probabilit ies [32] may be required. T his algorithm, referred to as "HMM
comb ination" , is t hen implemented as a stra ight forward adaptation of the
HMM decomp osit ion algorithm presented in [30] .
In the case of more complex combination criteria, as in the case of
a mixt ure of experts or the approach discussed is Sect ion 5 (related to
the mixture of experts mode l and t he psycho-acoustic evidence discussed
in Section 2), HMM combination/decompos ition is no longer a tractable
solution. In t his case, ot her approaches have to be used, e.g., using a two
level dy namic programming algorithm or using (4.1) to rescore an N-best
list of hypotheses (providing us with a set of possible segmentation/anchor
points) .
Alt houg h it is clear t hat:
1. The empirica l results discussed in Section 2 were obtained on t he
basis of segments (non-sense syllables),
2. only the segment level combinat ion can allow for asy nchro ny be-
tween the st reams'P,
we will mainly focus, in t he remaind er of this pap er, on t he combinat ion
at t he state level.
5. Multiband-based ASR with latent variables.
5.1. General formalism. As a parti cular case of multi-stream pro-
cessing, we have been invest igating an ASR approach based on independe nt
12 Although not using the nonlinear (optimal?) combination functions discussed in

this paper, preliminary results presented in [6, 14] suggested that asynchrony was not a
major factor - see, though , [21] and [29J for further discussion about this.
Acoustic Processing
Frequency band I
Recombined
Acoustic --t--e>t Acoustic Processing Recombination Result
Frequency bandk
Acoustic Processing
Frequency bandK
FIG. 5. Typical multiband-based ASR architecture . In multiband speech recognition,

the frequency rnnge is split into severnl bands, and information in the bands is used for
phonetic probability estimation by independent modules. These probabilities are then
combined for recognition later in the process at some segmental level.
processing and combination of frequency subbands. The general idea, as

illustrated in Fig. 5, is to split the whole frequency band (represented in
terms of critical bands) into a few subbands on which different recognizers
are independently applied. The resulting probabilities are then combined
for recognition later in the process at some segmental level (here we con-
sider the state level). Starting from critical bands, acoustic processing is
now performed independently for each frequency band, yielding K input
streams, each being associated with a particular frequency band.
In this case, each of the K sub-recognizers (streams) uses the informa-
tion contained in a specific frequency band X k = {xt, x~, . . . ,x~, . .. ,x1v},
where each x~ represents the acoustic (spectral) vector at time n in the
k-th stream. In (4.1), P(MkIX k ) represents the a posteriori probability
of a sub-model M k (k-th frequency band model for M) and can be esti-
mated from local posteriors P(qjlx~) (e.g., estimated at the output of an
ANN), where qj denotes a state j of model M k) . P(EkIX) represents the
"reliability" of expert Ek, working on the k-th frequency band, and can be
estimated in different ways (e.g., based on SNR).
As discussed in the previous section, combination at the segment level
according to the criteria discussed here is not easy. However, combina-
tion at the HMM-state level, by combining local posteriors P(qjlx~), can
be done in many ways [6], including untrained or trained linear functions
(e.g., as a function of automatically estimated local SNR), as well as trained
nonlinear functions (e.g., by using neural networks). This is quite simple
to implement and amounts to performing a standard Viterbi decoding in
which local (log) probabilities are obtained from a linear or nonlinear com-
bination of the local subband probabilities. For example, in our initial
subband-based ASR, local posteriors P(qjlx n ) (or scaled likelihoods) were
estimated according to:
K
(5.1) P(qjlx n ) = I: wkP(qjlx~,8k)
k=l
TOWARDS ROB UST SPEECH RECOGNITION MODELS 179
where, in our case, each P (qj l x~ , 8 k ) is computed with a band-specific

ANN of par amet ers 8k and with x~ (possibly with te mpora l context ) at
its inpu t . The weighting factor s can be assigned a uniform distribution
(already performing very well [6]) or be proportional to t he esti mated SNR.
Over the last few years, several results were reported showing th at such a
simple approach was usually quit e robust to band limited noise.
In Section 5.3 below, we discuss a new approach th at was recently
developed at IDIAP, and presented in [3, 11, 23], and show (i) how it
significant ly enhances t he baseline multi band approach, and (ii) how it
relates to t he above discussions (and psycho-acoustic evidence).
5.2. Motivations and drawbacks. The multib and approach has
several pot ential advantages, which are briefly discussed here.
Better robustness to band-limited noise - The speech signal may
be impaired (e.g., by noise, stream characterist ics, reverb eration,...) only
in some specific frequen cy bands. When recognition is based on several in-
dependent decisions from different frequency subbands , th e decoding of a
linguistic message need not be severely impair ed, as long as th e remaining
clean subbands supply sufficiently reliable information. This was confirmed
by severa l experiment s (see, e.g., [6]). Surprisingly, even when the combi-
nation is simply performed at t he HMM st at e level, it is observed th at th e
multi band approach yields better perform ance and noise robu stn ess th an
a regular full-band system. P
Similar conclusions also hold in t he framework of t he missing feature
t heory [19, 22] . In t his case, it was shown th at , if one knows the posit ion of
the noisy f eatures, significant ly better classificat ion performance could be
achieved by disregarding th e noisy data (using marginal distributions) or
by integrating over all possible values of the missing dat a condit ionally on
t he clean features - See Section 5.3 for furth er discussion about this. In
t he multi band approach, we do not tr y to explicitly identify th e noisy band
(and to disregard it). Instead , we process all th e subb ands independently
(to avoid "spreading" t he noise across all components of t he feature vector
or in the local probability estimate ) and recombine them according to a
particular weighting scheme that should de-emph asize (or cancel out) th e
noisy bands .
Better modeling - As for a regular full-band syst em, it was shown in [6]
t ha t all-pole modeling significantly improved th e perform ance of multiband
13 It could however be argued that , in t his case , t he multi ba nd approach boils down
to a regular full-band recognizer in which several likelihoods of (assu med) indepe nde nt
features are est imated and multiplied tog et her to yield local likelihoods (since, in likeli-
hood based syste ms, expected values for t he full-ba nd is t he same t han t he concat enated
expected values of subband s). This is however not true when using posterior based sys-
tems (such as hybrid HMM/ ANN systems) where t he subbands are presented to different
nets th at are independently trained in a discri min ant way on each ind ividua l subband .
Fina lly, as discussed in t his pap er, we a lso believe t hat the comb ination crit er ion should
be different t ha n a simple product of (scaled) likelihoods or posteriors.
systems. However, as an additional advantage of the subband approach, it

can be shown or argued that:
1. This all-pole modeling may be more robust if performed on sev-
eral subbands (low dimensional spaces) than on the full-band sig-
nal [27] .
2. Since the dimension of each (subband) feature space is smaller, it
is easier to estimate reliable statistics (resulting in a more robust
parameterization).
Stream asynchrony - Transitions between more or less stationary seg-
ments of speech (corresponding to an HMM state) do not necessarily occur
at the same time for different frequency bands [21] , which makes the piece-
wise stationary assumption more fragile for HMMs. The subband approach
may have the potential of relaxing the synchrony constraint inherent in cur-
rent HMM systems .
Stream specific processing and modeling - Different recognition
strategies might ultimately be applied in different subbands. For example,
different time/frequency resolution tradeoffs could be chosen (e.g., time
resolution and width of analysis window depending on the frequency sub-
band). Finally, some subbands may be inherently better for certain classes
of speech sounds than others.
Major objections and drawbacks - There are a few, related, draw-
backs to this multiband approach [21]:
1. One of the common objections to this separate modeling of each
frequency band has been that important information in the form of
correlation between bands may be lost . Although this may be true,
several studies [21], as well as the good recognition rates achieved
on small frequency bands [10, 15], tend to show that most of the
phonetic information is contained in each frequency band (possibly
provided that we have enough temporal information)!".
2. To define and independently process frequency bands, it is obvi-
ously necessary to start from spectral coefficients (critical bands) ,
which, however, are not orthogonal and do not permit competitive
performance for clean speech. In standard ASR systems, these co-
efficient are typically orthogonalized using a DCT (cepstral) trans-
formation. Even in the case of ANN probability estimation (where
ANN is supposed to extract and model the correlation across coeffi-
cients), it has been observed that orthogonalization of the features
still helped a bit. However, in the case of narrowband additive
noise, we obviously want to subtract as much as possible of the
noise before the DCT transform to avoid spreading the noise across
14 And, indeed, the discussion in Section 2, as well as many other psycho-acoustic

experiments, seem to suggest that human hearing can actually extract a lot of pho-
netic/syllabic information from band limited signals.
TOWARDS ROB UST SP EEC H RECOGNITION MODELS 181
all the feature components. For subband ASR systems, a partial

but effective solut ion to this problem consists in performing an
independ ent DCT in each subband [6, 26] .
Alternative solutions to this problem have recently been proposed
in which an attempt is made to decorrelate as much as possible the
(temporally) successive output energies of each individual filter-
bank - see, e.g., [18, 7, 25]. This is usually obtained by perform-
ing some kind of temporal filtering (and , consequently, spreading
th e possible noise over tim e instead of over frequency) or frequency
filtering (and consequently spreading t he possible noise over a lim-
ited frequency range only).
3. As opposed to th e empirical evidence discussed in Section 2, th e
initi al subband-based ASR system presented in [6] did not make
use of all possible subband combinat ions. This will be fixed by th e
method presented next .
5.3. Full combination subband ASR. Following th e developments

and discussions above, it seems reasonable to assume that a subb and ASR
system should simult aneously deal with all th e L = 2K possible subband
combinat ions Se (wit h £ = 1, . . . , L, including th e empty set l 5 ) resultin g
from an initial set of K frequency (criti cal) bands x k . However, while it is
pretty easy to quickly est imate any subb and likelihood or margin al distri-
bution when working with Gaussian or multi-G aussian densiti es [19], this
is harder when using ANN to estim at e posterior probabilities. In this latter
case, indeed, it would be necessary to train (and run , during recognition)
2K neur al networks , which would become very quickly intract able.
In the following, we briefly present the solut ion recently proposed
in [11] and [23], and discuss its relationships with the themes developed
in th e curre nt paper.
Ideally, we would t hus like to compute the post erior probabilities for
each of th e L = 2K possible combinations S; (including all possible single
bands, pairs of bands, triples, etc) of th e K subb ands x~ . Indeed, since we
do not know a priori where the noise is locat ed, we should integrate over
all possible positions'" , Using the formalism of mixture of experts, we can
thus writ e:
L
P(qjlx n , 8 ) = L P(qj , Eelxn, 8 )
e=1
15Which would corr espond to th e case where all th e band s ar e unreliable. In t his
case, t he best post erior est imate is th e prior probability P (qj ) , and one of t he L te rms
in the following equations will cont ain only t his prior inform ation.
16This amounts to assuming that t he position of th e noise or , in ot her words , the
position of th e reliable frequency bands, is a hidden (lat ent) variable on which we will
int egrate to maximize th e posterior probabilities (in the spirit of the EM algorit hm).
L
(5.2) = LP(qjIEe ,xn,8)P(EeI Xn)
e=l
L
= L P(qjIS~ , 8e)P(Eel xn)
e=l
where 8 represents the whole parameter space, while Se denotes the set
of (ANN) parameters used to compute the subband posteriors. Of course ,
implementation of (5.2) requires the training of L neural networks to es-
timate all the posteriors P(qjIS~ , 8e} that have to be combined according
to a weighted sum, with each weight representing the relative reliability
of a specific set of subbands. In the case of stationary interference, this
reliability could be estimated on the basis of the average (local) SNR in the
considered set . Alternatively, it could also be estimated as the probability
that the local SNR is above a certain threshold, and where the threshold
has been estimated to guarantee a prescribed recognition rate (e.g., lying
above a certain equal recognition rate curve in Figure 1) [3] .
Typical1y, training of the L neural nets would be done once and for al1
on clean data, and the recognizer would then be adapted online simply by
adjusting the weights P(Eelxn) (still representing a limited set of L weights)
to increase the global posteriors. This adaptation could be performed by
online estimation of the SNR or by an online version of the EM (deleted-
interpolation) algorithm. Although this approach is not really tractable, it
has the advantage of avoiding the independence assumption between the
subbands of a same set, as well as allowing any DCT transformation of the
combination before further processing. Consequently, this combination,
referred to as Full Combination, was actually implemented [10] for the case
of four frequency subbands (each containing several critical bands), thus
requiring the training of 16 neural nets, and used as an "opt imal" reference
point.
An interesting approximation to this "optimal" solution though con-
sists in simply train one neural network per subband for a total of K
models, and to approximate all the other subband combination probabili-
ties directly from these. In other words, re-introducing the independence
assumption!" between subbands, subband combination posteriors would be
estimated as [10, 11] :
(5.3)
Experimental results obtained from this approximated Full Combina-

tion approach in different noisy conditions are reported in [10, 11], where
17 Actually, it is shown in [10, 11] that we only need to introduce a weak (conditional)
independence assumption.
TOWARDS ROB UST SPEECH RECOGNITION MODELS 183
t he performance of t his approximat ion above was also compared to the

"optima l" est imators (5.2). Int erestingly, it was shown that this indepen-
dence assumpt ion did not hurt us much and that the resultin g recognition
perform an cel '' was similar to t he performance obtained by training and
recombining all possible L nets (and significant ly better than the origi-
nal single subband approach) . In both cases, the recognition rate and t he
robu stness to noise were greatly impr oved compared to t he initial sub-
band approach (5.1). This furth er confirms that we do not seem to lose
"critically" important information when neglecting the correlation between
band s.
Fin ally, it is particularly int eresting to note here that using (5.3)
in (5.2) yields something very similar to th e "optimal" product of errors
rule (2.4) observed empirically:
with Ce = P(qj ) (n t-l ) , and ne being the numb er of subbands in s'. In [10],
it is shown that this norm alization factor is important to achieve good
perform an ce. This Full Combin at ion rule thus takes exact ly the same form
as the product of errors rule [such as (2.4) or (2.5)], apart from the fact
t hat t he weighting factor s are different. In (5.4), the weighting factors can
be int erpreted as (scaled) prob abilities est imati ng t he relative reliability of
each combina tion, while in t he product of errors rule t hese are simply equal
to + 1 or -1. Another difference is that t he product of erro rs rule involves
2K - 1 te rms while th e Full Combination rule involves 2K terms, one of
t hem representing the cont ribut ion of the prior prob ability.
In the next section, we discuss a further exte nsion of t his approach
where the segment ation into subbands is no longer done explicitly, but is
achieved dyn amically over tim e, and where t he int egration over all possible
frequ ency segmentations is part of the same formalism.
6. HMM2: Mixture of HMMs. All HMM emission probabilities
discussed in the previous models are typically modeled through Gaussian
mixtures or artificial neur al networks . Also, in the multiband based recog-
nizers discussed above, we have to decide a priori the numb er and position
of the subbands being considered. As also briefly discussed above, it is not
always clear what the "opt imal" recombination criterion should be. In the
following, we introduce a new approach, referred to as HMM2, wher e the
emission prob abilities of the HMM (now referred to as "te mporal HMMs")
are est imat ed through a secondary, state-dependent, HMM (referred to as
"feat ure HMMs") specifically workin g along the feature vector. As briefly
discussed below (see references such as [2] and [31] for further detail) , this
180 btained on t he Number s'95 database , containing te lephone-based speaker ind e-

pe ndent free format numbers, on which NOISEX noise was added.
184 HERVE BO URLARD ET AL.
FIG. 6. HMM2: the emi ssion distributions of the temporal HMM are estim ated by
secon dary, state-specific, feature HMMs .
model will then allow for dynamic (time and st at e dependent) subb and
(frequency) segmentation as well as "optimal" recombination according to
a standard maximum likelihood criterion (although other crit eria used in
st and ard HMMs could also be used) .
In HMM2, as illustrated in Figure 6, each temporal feature vector X n
is considered as a fixed length sequence of S components X n = (x~ , .. ,x~ ),
which is supposed to have been generat ed at time n by a specific fea-
ture HMM associated with a specific st at e qj of th e tempor al HMM. Each
feature HMM st ate ri is thus emitting individu al feature components x~ ,
whose distributions are modeled by, e.g., one dimensional Gaussian mix-
tures. Th e feature HMM thus looks at all possible subb and segmentations
and automatically performs the combination of th e likelihoods to yield a
single emission probability. The resulting emission probability can th en be
used as emission probability of the temporal HMM. As an alternat ive, we
can also use the resulting feature segmentat ion in multi band systems or as
addit ional acoustic features features in a st and ard ASR system. Ind eed, if
HMM2 is applied to th e spectral domain , it is expected th at the feature
HMM will "segment" the feature vector into piecewise st ationary spectral
regions, which could thus follow spectr al peaks (formant s) and/or spect ral
valley regions.
In the example illustrated in Figure 6, the HMM2 is composed of

a temporal HMM that handles sequences of features through time, and
feature HMMs assigned to the different temporal HMM states. The tem-
poral HMM is composed of 3 left-to-right connected states (q1' q2 and
q3), while the state-specific feature HMM is composed of 4 ("top-down")
states (r1 ' r2 r a and r4)' Although not reflected in Figure 6, each feature
HMM {r1' r2,r3, r4} is specific to a temporal HMM state (emission proba-
bility distribution), with different parameters, and possibly different HMM
topologies. More formally, as done in [2] and [17], the feature state should
have been denoted rj, with k representing the associated temporal state
index and j the feature state index.
Of course, the topology of the feature HMM, extracting the correla-
tion information within feature vectors, could take many forms, includ-
ing ergodic HMMs and/or topologies with a number of states larger the
number of feature components, in which case "high-order" correlation in-
formation could be modeled . In the following though , we constrained th e
feature HMM to a strictly "top-down" topology. Moreover, since we were
interested in extracting information in the spectral domain and in possi-
ble relationships with multiband ASR systems, we considered features in
the spectral domain. Each of the feature HMM states is then supposed to
model one of the K frequency bands, where the positions and bandwidths
of these bands are determined dynamically.
In [2] , we introduced an EM algorithm to jointly train all the param-
eters of such an HMM2 in order to maximize the data likelihood . This
derivation is based on the fact that an HMM is a special kind of mixture
of distributions, and therefore HMM2, as a mixture of HMMs, can be con-
sidered as a more general kind of mixture distribution. During decoding ,
the Viterbi algorithm is used to find the path through the HMM2 which
best explains the input data. Local st ate likelihoods of the temporal HMM
can however be estimated using either Viterbi or the complete likelihood
calculation, summing over all possible paths through the feature HMM:
s
(6.1) p(xnlqj) = LP(rolqj) rrp(x~lrl,qj)P(rdrl-1,qj)
R 8= 1
where qj is the temporal HMM state at time n, rl the feature HMM state
at feature s, R the set of all possible paths through the feature HMM,
P(rolqt) the initial state probability of the feature HMM, p(x~ln,qj) the
probability of emitting feature component x~ while in feature HMM state
rl oftemporal state qj, and P(rL!rl-t,qj) the transition probabilities ofthe
feature HMM in temporal state qj.
We believe that HMM2 (which includes the classical mixture of Gaus-
sian HMMs as a particular case) has several potential advantages, includ-
ing:
1. Better feature correlation modeling through the feature HMM
topology (e.g., working in the frequency domain) . Also, the com-
plexity of this topology and the probability density function asso-

ciated with each state easily control the number of parameters.
2. Automatic non-linear spectral warping . In the same way the
conventional HMM does time warping and time integration, the
feature-based HMM performs frequency warping and frequency
integration.
3. Dynamic formant trajectory modeling. As further discussed below,
the HMM2 structure has the potential to extract some relevant for-
mant structure information, which is often considered as important
to robust speech recognition .
To illustrate these advantages and the relationship of HMM2 with
dynamic multi-band ASR, we trained all parameters of an HMM2, using
frequency filtered filterbank features [25] . We employed the HMM2 topol-
ogy as shown in Figure 6. Training was done with the EM algorithm, and
decoding was performed using the Viterbi algorithm for both the temporal
arid the frequency HMM. Figure 7 illustrates (on unseen test data) the tem-
poral and frequency segmentation obtained as a by-product from Viterbi,
plotted onto a spectrogram of our features . At each time step, we kept the
3 positions where the feature HMM changed its state during decoding (for
instance, at the first time frame, the feature HMM goes from state Tl to
state T2 after the second feature) . We believe that this segmentation gives
cues about some structures of the speech signal such as formant positions.
In fact , in [31] it has been shown that this segmentation information can be
used as (additional) features for speech recognition, being (1) discriminant
and (2) rather robust in the case of speech degraded by additive noise.
7. Conclusions. In this paper, we have discussed a family of new
ASR approaches that have recently been shown to be more robust to noise,
without requiring specific adaptation or "mult i-style" training.
From all this discussion, and the convergence of independent experi-
ments , we can draw the following preliminary conclusions:
1. Multiband ASR does not seem to be inherently inferior to a full-
band approach, although some correlation information is lost due
to the division of the frequency space into subbands.'? Further-
more, it is not clear either that human hearing uses this kind of
correlation information.
2. When training subband systems, we should not aim at maximiz-
ing the classification performance for every subband. When using
the right combination rule, it is better to increase the number of
subbands while making sure that at any time at least one subband
will be guessing the right answer.r''
19probably because the advantages of subband based ASR can outweight the slight
problem due to independent processing of subbands.
20This conclusion is very similar to what is proved mathematically in [4]' p. 369,
para. 1 (also p . 424) .
FIG. 7. Frequency filtered filterbanks and HMM2 resulting (Viterbi) segmentation

for a test example of phoneme "w".
3. Doing this, we should also look at the potential for improvement in

subband modeling when combining longer time-scale information
streams (trading frequency information for temporal information) .
4. The full combination approach discussed here has the potential
of providing us with new adaptation schemes in which only the
combination weights are automatically adapted (e.g., according to
an online EM algorithm) .
Finally, it is clear that several key problems remain to be addressed, in-
cluding:
1. Need for improved expert weighting
2. Need for methods which are robust to noise but still perform well
for clean speech .
In subband processing, there is also a need to properly choose the
frequency subband, and it is expected that those sub bands should be dy-
namically defined, e.g, following some formant structure. In this respect,
the HMM2 formalism also presented here can be considered as a generaliza-
tion of subband approaches, allowing for optimal (according to a maximum
likelihood criterion) subband segmentation and recombination.
188 HERVE BOURLARD ET AL .
Acknowledgments. The content and themes discussed in this paper

largely benefited from the collaboration with our colleagues Andrew Morris
and Astrid Hagen . This work was partly supported by the Swiss Federal
Office for Education and Science (FOES) through the European SPHEAR
(TMR, Training and Mobility of Researchers) and RESPITE (ESPRIT
Long term Research) projects. Additionally, Katrin Weber is supported by
a the Swiss National Science Foundation project MULTICHAN.
REFERENCES
[1] ALLEN J ., "How do humans process and recognize speech?," IEEE Trans . on
Speech and Audio Processing, Vol. 2 , no . 4, pp , 567-577, 1994.
[2] BENGIO S., BOURLARD H., AND WEBER K. , "An EM Algorithm for HMMs with
Emission Distributions Represented by HMMs," IDIAP Research Report,
IDIAP-RR-OO-11 , 2000.
[3] BERTHOMMIER F . AND GLOTIN H., "A new SNR-feature mapping for robust multi-
stream speech recognition," Inti . Conf. of Phonetic Sci ences (ICPhS'99) (San
Francisco) , to appear, August 1999.
[4J BISHOP C.M ., Neural Networks for Pattern Recognition, Clarendon Press (Oxford),
1995.
[5] BOURLARD H. AND MORGAN N ., Connectionist Speech Recognition - A Hybrid
Approach, Kluwer Academic Publishers, 1994.
[6] BOURLARD H . AND DUPONT S., "A new ASR approach based on independent
processing and combination of partial frequency bands," Proc. of Inti . Conf.
on Spoken Language Processing (Philadelphia), pp . 422-425, October 1996.
[7] DE VETH J ., DE WET F ., CRANEN B. , AND BOVES L., "Missing feature theory
in ASR: make sure you miss the right type of features," Proceedings of the
ESCA Workshop on Robust Speech Recognition (Tampere, Finland) , May 25-
26, 1999.
[8] DUDA R.O . AND HART P .E ., Pattern Classification and Scene Analysis, John Wi-
ley, 1973.
[9] GREENBERG S., "On the origins of speech int elligibility in the real world," Proc.
of the ESCA Workshop on Robust Speech Recognition for Unknown Commu-
nication Channels, pp . 23-32, ESCA, April 1997.
[10] HAGEN A ., MORRIS A., AND BOURLARD H., "Subband-base d speech recognition
in noisy conditions: The full combinat ion a pproach," IDIAP Research Report
no. IDIAP-RR-98-15, 1998.
[11) HAGEN A., MORRIS A. , AND BOURLARD H ., "Different weighting schemes in the
full combination subbands approach for noise robust ASR," Proceedings of the
Workshop on Robust Methods for Speech Recognition in Adverse Conditions
(Tampere, Finland), May 25-26, 1999.
[12] HENNEBERT J., RIS C., BOURLARD H., RENALS S., AND MORGAN N. (1997),
"Est im at ion of Global Posteriors and Forward-Backward Training of Hybrid
Systems," Proceedings of EUROSPEECW97 (Rhodes, Greece, Sep . 1997) ,
pp . 1951-1954.
[13J HERMANSKY H. AND MORGAN N ., "RA STA processing of speech," IEEE Trans .
on Speech and Audio Processing, Vol. 2, no . 4, pp . 578-589, October 1994.
[14J HERMANSKY H ., PAVEL M. , AND TRIBEWALA S., "Towards ASR using partially cor-
rupted sp eech," Proc. of Inti . Conf. on Spoken Language Processing (Philadel-
phia), pp . 458-461 , October 1996.
[15] HERMANSKY H. AND SHARMA S., "Te mporal patterns (TRAPS) in ASR noisy
speech," Proc. of the IEEE Inti . Conf. on Acoustics, Speech, and Signal Pro-
cessing (Phoenix, AZ) , pp . 289-292, March 1999.
TOWARDS ROB UST SP EECH RE CO GNITION MODELS 189
[16] HOUTGAST T . AND STEENEKEN H.J .M., "A review of th e MTF concept in room
acoust ics and its use for est imating speech int elligibility in auditoria ," J.
Ac ous t. S oc. Am. , Vol. 77, no. 3, pp . 1069-1077, March 1985.
[17J IKBAL S., BOURLARD H., BENGIO S ., AN D WEBER K ., "ID IAP HMM /HMM2 Sys-
t em : Theoretical Basis and Software Specificat ions" IDIAP Research R eport,
IDIAP-RR-01-27 , 200 l.
[18] KINGSBURY B., MORGA N N., AND GREENBERG S., "Robust sp eech recogni t ion us-
ing t he modulati on spect rogram," Speech Communication, Vol. 25 , nos. 1-3,
pp . 117-132, 1998.
(19] LIPPMA NN R.P. AND CARLSON B.A. , "Using missing feature theory to act ively
select features for rob ust sp eech recogn ition with int erruptions, filtering a nd
noise ," Proc. Euro speech '97 (Rhodes, Greec e, September 1997), pp . KN37-40 .
(201 MCGURK H. AND McDoNALD J ., "Hearing lips and seeing voices," Nat ure, no. 264,
pp.746-748, 1976.
[21] MIRGHAFORI N. AND MORGA N N., "Transmissions and transi ti ons: A study of two
common assumptions in multi-band ASR ," Int! . IEEE Conf. on A coustics ,
Speech , and Signal Processin g (Seattle, WA, May 1997) , pp . 713-716.
[22] MORRIS A.C ., COOKE M.P ., AND GREEN P .D ., "Some solutions to the miss-
ing features problem in data classification, with application to noise robust
ASR," Proc. Int! . Conf on A coustics, Speech , and Si gn al Processing , pp . 737-
740, 1998.
(23] MORRIS A. C ., HAGEN A., AND BOURLARD H., "T he full combina ti on subbands
approach to noise robust HMM /ANN-based ASR ," Proc. of Eurospeech '99
(Budapest, Sep . 99), t o appear.
[24] MOORE B .C .J ., An Introduction to the Ps ychology of Hearing (4t h edition) , Aca-
demic Press, 1997.
[25] NADEU C. , HERNANDO J ., AND GORRICHO M., "On th e decorrelati on of filter-
bank energies in sp eech recogniti on ," Proc. of Eu rospeech '95 (Mad rid , Spain),
pp. 1381-1384, 1995.
[26] OKAWA S., BOCCHIERI E. , AND P OTAMIANOS A., "Mult i-ba nd speec h recogn it ion in
noisy environment ," Proc . IEEE In ti. Conf. on Ac ous tics, Speech, and Signal
Processing , 1998.
[27] RAO S. AND PEARLMA N W .A., "Analysis of linear pr edict ion, coding, and sp ectral
est imation from subb ands, " IE EE Tran s. on Information Th eory , Vol. 42 ,
pp . 1160-1178, July 1996.
[28J TOMLINSON J ., RUSSEL M.J ., AND BROOKE N.M ., "Integrat ing audio and visua l
information to provid e highly robust sp eech recogniti on ," Proc. of IEEE Inti.
Conf. on A coustics, Speech, and Signal Processing (Atlanta), May 1996.
[29J T OMLINSON M.J ., RUSSEL M.J ., MOORE R.K ., BUCKLAN A.P ., AND FAWLEY M.A .,
"Modelling asynchrony in speech using elementary single-signal decomposi-
tion," Proc. of IEEE Int!. Conf . on A coustics, Speech, and Signal Processing
(Munich) , pp , 1247-1250, April 1997.
[30J VARGA A. AND MOORE R., "Hidden markov model decompositi on of speech and
noise," Proc. IEEE Int! . Conf. on Acoustics, Speech and Si gnal Processing,
pp . 845-848, 1990.
[31] WEBER K ., BENGIO S., AND BOURLARD H ., "HMM2- Extraction of Formant Fea-
tures and their Use for Robust ASR " , Proc. of Euro speech, pp . 607- 610, 2001.
[32] WELLEKENS C.J . , KANGASHARJU J ., AND MILESI C., "T he use of met a-HMM in
multistream HMM training for automatic spe ech recognition," Proc. of Int!.
Confe rence on Spoken Language Processing (Sydney), pp . 2991-2994, Decem-
ber 1998.
[33] W u S.-L., KINGSB URY B.E ., MORGAN N. , AND GREENBERG S., "Pe rfor mance im-
pr ovements th rou gh combining phone and syllable-scale information in auto-
mati c sp eech recogniti on ," Proc. Int! . Conf. on Spoken Language Pro cessing
(Sydney) , pp. 459- 462, Dec. 1998.
GRAPHICAL MODELS AND AUTOMATIC
SPEECH RECOGNITION*
JEFFREY A. BILMESt
Abstract. Graphical models provide a promising pa radigm to study both existing

and novel techniques for automatic speech recognition. This paper first provides a brief
overview of graphical models and their uses as statistical models . It is then shown that
the statistical assumptions behind many pattern recognition techniques commonly used
as part of a speech recognition system can be described by a graph - this includes Gaus-
sian distributions, mixture models , decision trees, factor analysis, principle component
analysis, linear discriminant analysis, and hidden Markov models . Moreover , this paper
shows that many advanced models for spee ch recognition and language processing can
also be simply described by a graph, including many at the acoust ic-, pronunciation-,
and language-modeling levels. A number of speech recognition techniques born directly
out of the graphical-models paradigm are also surveyed. Additionally, this paper in-
cludes a novel graphical analysis regarding why derivative (or delta) features improve
hidden Markov model-based speech recognition by improving structural discriminabil-
ity. It also includes an example where a graph can be used to represent language model
smoothing constraints. As will be seen , the space of models describable by a graph is
quite large . A thorough exploration of this space should yield techniques that ultimately
will supersede the hidden Markov model.
Key words. Graphical Models, Bayesian Networks , Automatic Speech Recognition,

Hidden Markov Models , Pattern Recognition, Delta Features, Time-Derivative Features,
Structural Discr iminability, Language Modeling.
1. Introduction. Since its inception, th e field of automatic speech

recognition (ASR) [129, 39, 21, 164, 83, 89, 117, 80] has increasingly come
to rely on statistical methodology, moving away from approaches that were
initially proposed such as templ ate matching, dynamic time warping , and
non-probabilistically motivated distortion measures . While there are still
many successful instances of heuristically motivated techniques in ASR, it
is becoming increasingly apparent that a st atistical understanding of the
speech process can only improve the performance of an ASR system. Per-
haps the most famous example is the hidden Markov model [129], current ly
the predominant approach to ASR and a statistical generalization of earlier
template-based practices.
A complete state-of-the-art ASR system involves numerous separate
components, many of which are statistically motivated. Developing a thor-
ough understanding of a complete ASR syst em, when it is seen as a collec-
tion of such conceptually distinct entities, can take some time . A impres-
sive achievement would be an over-arching and unifying framework within
which most statistical ASR methods can be accurately and succinctly de-
"This material is based upon work supported in part by the National Science Foun-
dation under Grant No. 0093430.
tDepartment of Electrical Engineering, University of Washington, Seattle, Washing-
ton 98195-2500.
191
192 JEFFREY A. BILMES
scribed. Fortunately, a great many of the successful algorithms used by

ASR systems can be described in terms of graphical models.
Graphical models (GMs) are a flexible statistical abstraction that have
been successfully used to describe problems in a variety of domains rang-
ing from medical diagnosis and decision theory to time series prediction
and signal coding. Intuitively, GMs merge probability theory and graph
theory. They generalize many techniques used in statistical analysis and
signal processing such as Kalman filters [70], auto-regressive models [110],
and many information-theoretic coding algorithms [53]. They provide a
visual graphical language with which one may observe and reason about
some of the most important properties of random processes, and the un-
derlying physical phenomena these processes are meant to represent. They
also provide a set of computationally efficient algorithms for probability
calculations and decision-making . Overall, GMs encompass an extremely
large family of statistical techniques.
GMs provide an excellent formalism within which to study and under-
stand ASR algorithms. With GMs, one may rapidly evaluate and under-
stand a variety of different algorithms, since they often have only minor
graphical differences. As we will see in this paper, many of the existing
statistical techniques in ASR are representable using GMs - apparently
no other known abstraction possesses this property. And even though the
set of algorithms currently used in ASR is large, this collection occupies
a relatively small volume within GM algorithm space. Because so many
existing ASR successes lie within this under-explored space, it is likely that
a systematic study of GM-based ASR algorithms could lead to new more
successful approaches to ASR.
GMs can also help to reduce programmer time and effort. First, when
described by a graph, it is easy to see if a statistical model appropriately
represents relevant information contained in a corpus of (speech) data.
GMs can help to rule out a statistical model which might otherwise require
a large amount of programming effort to evaluate. A GM moreover can be
minimally designed so that it has representational power only where needed
[10] . This means that a GM-based system might have smaller computa-
tional demands than a model designed without the data in mind, further
easing programmer effort. Secondly, with the right set of computational
tools, many considerably different statistical algorithms can be rapidly eval-
uated in a speech recognition system. This is because the same underlying
graphical computing algorithms are applicable for all graphs, regardless of
the algorithm represented by the graph. Section 5 briefly describes the new
graphical models toolkit (GMTK)[13], which is one such tool that can be
used for this purpose.
Overall , this paper argues that it is both pedagogically and scientifi-
cally useful to portray ASR algorithms in the umbrage of GMs. Section 2
provides an overview of GMs showing how they relate to standard statistical
procedures. It also surveys a number of GM properties (Section 2.5), such
GRAP HICAL MODELS FOR ASR 193
as probabilistic inference and learning. Section 3 casts many of the meth-

ods commonly used for automat ic speech recognition (ASR) as inst ances
of GMs and t heir associate d algorithms. This includes principle compo-
nent analysis [44], linear discriminant analysis (and its quadr atic and het-
eroscedastic generalizations) [102]' factor analysis, independent component
analysis, Gaussian densit ies, multi-l ayered perceptrons, mixture models,
hidden Markov models, and many language models. This pap er further
arg ues t hat developing novel ASR techniques can benefit from a GM per-
spective. In doing so, it surveys some recent techniqu es in speech recogni-
t ion, some of which have been developed without GMs explicit ly in mind
(Section 4), and some of which have (Sect ion 5).
In this paper, capit al letters will refer to rand om vari ables (such as X ,
Y and Q) and lower-case letters will refer to values t hey may take on . Sets
of variables may be referr ed to as X A or Q B where A and B are sets of
indices. Sets may be referred to using a Matl ab-like range not ation, such
as 1 :N which indicat es all indices between 1 and N inclusive. Using this
not ati on , one may refer to a length T vector of rand om vari able t aking on
a vector of values as P (X 1:T = X l :T) .
2. Overview of graphical models. This sect ion briefly reviews

gra phical models and t heir associated algorit hms - t hose well-versed in
t his methodology may wish to skip directly to Sect ion 3.
Bro adly spea king, gra phical models offer two prim ar y features to t hose
interested in workin g with statistical systems. Fir st , a GM may be viewed
as an abstract, form al, and visual language t hat can depict important prop-
erties (conditional independence) of natural syste ms and signa ls when de-
scribed by multi-vari ate rand om processes. There are math emat ically pre-
cise rules that describ e what a given gra ph means, rules t hat associate with
a gra ph a family of prob ability distributions. Natural signals (t hose t hat
are not pur ely rand om) have significant st ati sti cal structure, and t his can
occur at multiple levels of gra nular ity. Gr aph s can show anyt hing from
causa l relati ons between high-level concepts [122] down to t he fine-grained
dependencies exist ing within t he neur al code [5]. Second, along with GMs
come a set of algorit hms for efficiently performing probabilistic inference
and decision making. Typically intractable, the GM inference proc edures
and their approximations exploit the inherent st ruct ure in a graph in a way
th at can significantly redu ce computational and memory demands relative
to a naive implement ation of prob abilistic inference.
Simply put, gra phical models describ e condit ional independ ence prop-
ert ies amongst collections of rand om variables. A given GM is identi cal to
a list of condit ional independence state ments, and a gra ph repr esents all
dist ribut ions for which all t hese independ ence statements are true. A ran-
dom vari able X is condit ionally independent of a different rand om variable
Y given a t hird rand om variable Z und er a given prob abilit y dist ribu ti on
p(.), if t he following relat ion holds:
p(X = x , Y = ylZ = z) = p(X = xlZ = Z)p(Y = ylZ = z)

for all x , y, and z. This is written X lLYIZ (notation first introduced
in [37]) and it is said that "X is independent of Y given Z under p(.)".
This has the following intuitive interpretation: if one has knowledge of Z,
then knowledge of Y does not change one's knowledge of X and vice versa .
Conditional independence is different from unconditional (or marginal) in-
dependence. Therefore, neither X lLY implies X lLYIZ nor vice versa.
Conditional independence is a powerful concept - using conditional inde-
pendence, a statistical model can undergo enormous changes and simplifi-
cations. Moreover, even though conditional independence might not hold
for certain signals, making such assumptions might yield vast improvements
because of computational, data-sparsity, or task-specific reasons (e.g., con-
sider the hidden Markov model with assumptions that obviously do not
hold for speech [10], but that nonetheless empirically appear benign, and
actually beneficial as argued in Section 3.9). Formal properties of condi-
tional independence are described in [159, 103, 122, 37].
A GM [103, 34, 159, 122, 84] is a graph 9 = (V, E) where V is a set
of vertices (also called nodes or random variables) and the set of edges
E is a subset of the set V x V. The graph describes an entire family
of probability distributions over the variables V . A variable can either
be scalar- or vector-valued , where in the latter case the vector variable
implicitly corresponds to a sub-graphical model over the elements of the
vector. The edges E, depending on the graph semantics (see below), encode
a set of conditional independence properties over the random variables . The
properties specified by the GM are true for all members of its associated
family.
Four items must be specified when using a graph to describe a particu-
lar probability distribution: the GM semantics, structure, implementation,
and parameterization. The semantics and the structure of a GM are inher-
ent to the graph itself, while the implementation and parameterization are
implicit within the underlying model.
2.1. Semantics. There are many types of GMs, each one with differ-
ing semantics. The set of conditional independence assumptions specified
by a particular GM, and therefore the family of probability distributions it
represents, can be different depending on the GM semantics. The seman-
tics specifies a set of rules about what is or is not a valid graph and what set
of distributions correspond to a given graph. Various types of GMs include
directed models (or Bayesian networks) [122,84],1 undirected networks (or
Markov random fields) [27], factor graphs [53, 101], chain graphs [103, 133]
which are combinations of directed and undirected GMs, causal models
[123], decomposable models (an important sub-family of models [103]),
1 Note that the name "Bayesian network" does not imply Bayesian statistical infer-
ence . In fact, both Bayesian and non-Bayesian Bayesian networks may exist .
GRAPHICAL MODELS FOR ASR 195
dependen cy networks [76] , and many others. In general, different graph

semantics will correspond to different families of distributions, but overlap
can exist (meaning a particular distribution might be describable by two
graphs with different semant ics).
A Bayesian network (BN) [122, 84, 75] is one type of directed GM
where th e graph edges are directed and acyclic. In a BN, edges point from
parent to child nodes, and such graphs implicitly portray factorizations
th at are simplifications of th e chain rule of prob ability, namely:
P(X1:N) = II p(XiIX1:i-d = II p(X iIX1r; ) .
i
The first equality is the probabilistic chain rule, and the second equality
holds under a particular BN, where 7ri designates node i's parents according
to the BN. A Dynamic Bayesian Network (DBN) [38, 66, 56] has exactly
the same semantics as a BN, but is structured to have a sequence of clusters
of connected vertices , where edges between clusters point in the direction
of increasing time. DBNs are particularly useful to describe t ime signals
such as speech, and as can be seen from Figur e 2 many techniques for ASR
fall under this or the BN category.
Several equivalent schemat a exist that formally define a BN's condi-
tional independence relationships [103, 122, 84]. The idea of d-separation
(or directed separation) is perhaps the most widely known: a set of variables
A is condit ionally independent of a set B given a set C if A is d-separated
from B by C . D-separ ation holds if and only if all paths t hat connect any
node in A and any oth er node in B are blocked. A pat h is blocked if it
has a node v with eit her: 1) t he arrows along th e path do not converge
at v (i.e., serial or diverging at v) v E C ; or 2) the arrows along t he path
do converge at v, and neith er v nor any descendant of v is in C . Note
t hat C can be the empty set in which case d-separation encodes standa rd
stat ist ical independ ence.
From d-separation, one may compute a list of conditional indepen-
dence statements made by a graph. This set of probability distributions
for which this list of st at ements is true is precisely the set of distributions
represented by the graph. Graph properties equivalent to d-separation in-
clude the directed local Markov property [103] (a variable is condit ionally
independent of its non-descendants given its parents), and th e Bayes-ball
procedur e [143] which is a simple algorithm that one can use to read condi-
t ional independence st at ements from graphs, and which is arguably simpler
th an d-separ ation. It is assumed henceforth th at t he reader is familiar with
eit her d-separation or some equivalent rule.
Conditional independence prop erti es in undir ected graphical models
(UGMs) are much simpler than for BNs, and are specified using graph
sepa ration. For exa mple, assuming that X A, XB , and X c are disjoint
set s of nodes in a UGM, X AJlXB IX c is true when all paths from any
node in XA to any node in XB intersect some node in Xc . In a UGM,

a distribution may be described as the factorization of potential functions
where each potential function operates only on collections of nodes that
form a clique in the graph. A clique is a set of nodes that are pairwise
connected [84].
BNs and DGMs are not the same. Despite the fact that BNs have
complicated semantics, they are useful for a variety of reasons. One is
that BNs can have a causal interpretation, where if node A is a parent
of B, A might be thought of as a cause of B . A second reason is that
the family of distributions associated with BNs is not the same as the
family associated with UGMs - there are some useful probability models
that are concisely representable with BNs but that are not representable
at all with UGMs (and vice versa) . This issue will arise in Section 3.1
when discussing Gaussian densities . UGMs and BNs do have an overlap,
however, and the family of distributions corresponding to this intersection
is known as the decomposable models [103] . These models have important
properties relating to efficient probabilistic inference (see below).
In general , a lack of an edge between two nodes does not imply that
the nodes are independent. The nodes might be able to influence each other
indirectly via an indirect path. Moreover, the existence of an edge between
two nodes does not imply that the two nodes are necessarily dependent
- the two nodes could still be independent for certain parameter values
or under certain conditions (see later sections) . A GM guarantees only
that the lack of an edge implies some conditional independence property,
determined according to the graph's semantics . It is therefore best, when
discussing a given GM, to refer only to its (conditional) independence rather
than its dependence properties - it is more accurate to say that there is
an edge between A and B than to say that A and B are dependent.
Originally BNs were designed to represent causation, but more re-
cently, models with semantics [123] more precisely representing causality
have been developed. Other directed graphical models have been designed
as well [76], and can be thought of as the general family of directed graph-
ical models (DGMs).
2.2. Structure. A graph's structure, the set of nodes and edges, de-
termines the set of conditional independence properties for the graph un-
der a given semantics. Note that more than one GM might correspond
to exactly the same conditional independence properties even though their
structure is entirely different (see the left two models in Figure 1). In this
case, multiple graphs will correspond to the same family of probability dis-
tributions. In such cases, the various GMs are said to be Markov equivalent
[153, 154, 77] . In general, it is not immediately obvious with complicated
graphs how to visually determine if Markov equivalence holds, but algo-
rithms are available that can determine the members of an equivalence class
[153, 154, 114, 30].
GRAP HICAL MODELS FOR ASR 197
F IG . 1. Th is figure shows fou r BNs with different arrow directions over the same
rand om varia bles, A, B, and C . On the left side, the variables form a three-variable
first -order Mar kov chain A ~ B ~ C . In the middl e graph, the same conditional
ind ependence statement is realized even though on e of the arrow directions has been
reversed. Both these netwo rks state that AlLCIB . Th e right network corresponds to
the property AlLC but not that AlLCIB .
Nodes in a graphical model can be either observed, or hidd en. If a

vari able is observed, it means t hat its value is known, or t hat dat a (or
"evidence" ) is available for th at variable. If a variable is hidden, it current ly
does not have a known .value, and all that is available is the conditional
distribution of t he hidden variables given the observed variables (if any).
Hidden nodes are also called confounding, lat ent , or unobserved variables.
Hidden Markov models are so named because th ey possess a Markov chain
that , in some cases, contains only hidden variables. Note that th e gra phs in
GMs do not show th e zeros t hat exist in t he stochastic t ra nsit ion matrices of
a Markov chain - GMs, rath er, encode st atis tical independence properties
of a model (see also Section 3.7).
A node in a gra ph might sometimes be hidden and at ot her t imes
be observe d. With an HMM, for exa mple, th e "hidden" chain might be
observed during t raining (because a phonetic or state-level alignment has
been provided) and hidden during recognition (because t he hidden variable
values are not known for test speech data). Wh en making the query "is
AJLBIC?" , it is implicitly assumed t hat C is observed. A and B are the
nodes being queried, and any ot her nodes in the network not listed in th e
query are considered hidden. Also, when a collection of sampled dat a exists
(say as a t raining set ), some of t he dat a samples might have missing values
each of which would correspond to a hidden variable. The EM algorithm
[40]' for example, can be used to t rain t he par ameters of hidden vari ables.
Hidden variables and t heir edges reflect a belief about th e underlying
generat ive process lying behind t he phenomenon th at is being statist ically
represented. This is because th e data for these hidden variables is either
unavailable, is to o cost ly or impossible to obtain, or might not exist since
t he hidd en variables might only be hypothetical (e.g., specified based on
human-acquired knowledge about t he underlying domain). Hidden vari-
ables can be used to indicate t he underlying causes behind an information
source. In speech, for example, hidden variables can be used to represent
t he phonet ic or art iculato ry gestures, or more ambit iously, t he originat ing
semantic t hought behind a speech waveform. One common way of using
GMs in ASR, in fact , is to use hidden variables to represent some condition
known dur ing tra ining and unknown during recognition (see Section 5).
Certain GMs allow for what are called switching dependencies [65,
115, 16]. In this case, edges in a GM can change as a function of other
variables in the network. An important advantage of switching dependen-
cies is the reduction in the required number of parameters needed by the
model. A related construct allows GMs to have optimized local probability
implementations [55] using, for example , decision trees .
It is sometimes the case that certain observed variables are used only
as conditional variables. For example, consider the graph B --+ A which
implies a factorization of the joint distribution P(A, B) = P(AIB)P(B). In
many cases, it is not necessary to represent the marginal distribution over
B. In such cases B is a "conditional-only" variable, meaning is always and
only to the right of the conditioning bar . In this case, the graph represents
P(AIB). This can be useful in a number of applications including classi-
fication (or discriminative modeling) , where we might only be interested
in posterior distributions over the class random variable, or in situations
where additional observations (say Z) exist that are marginally indepen-
dent of a class variable (say C) but that are dependent conditioned on other
observations (say X) . This can be depicted by the graph C --+ X f - Z,
where it is assumed that the distribution over Z is not represented.
Often, the true (or the best) structure for a given task is unknown .
This can mean that either some of the edges or nodes (which can be hid-
den) or both can be unknown. This has motivated research on learning
the structure of the model from the data, with the general goal to produce
a structure that accurately reflects the important statistical properties in
the data set. These can take a Bayesian [75, 77] or frequentist point of
view [25, 99, 75] . Structure learning is akin to both statistical model se-
lection [107, 26] and data mining [36] . Several good reviews of structure
learning are presented in [25, 99, 75]. Structure learning from a discrimina-
tive perspective, thereby producing what is called discriminative generative
models, was proposed in [10] .
Figure 2 depicts a topological hierarchy of both the semantics and
structure of GMs, and shows where different models fit in, including several
ASR components to be described in Section 3.
2.3. Implementation. When two nodes are connected by a depen-

dency edge, the local conditional probability representation of that depen-
dency may be called its implementation. An edge between variable X and
Y can represent a lack of independence in a number of ways depending
on if the variables are discrete or continuous. For example, one might use
discrete conditional probability tables (CPTs) [84], compressed tables [55],
decision trees [22], or even a deterministic function (in which case GMs
may represent data-flow [1] graphs, or may represent channel coding algo-
rithms [53]) . A node in a GM can also depict a constant input parameter
since random variables can themselves be constants. Alternatively, the
dependence might be linear regression models, mixtures thereof, or non-
GRAPHI CAL MODELS FOR ASR 199
Other Semantics
Causal Models
DependencyNetworks
FIG. 2. A topology of gmphical model semantics and structure.
linear regression (such as a multi-layered perceptron [19] , or a STAR [149]

or MARS [54] model) . In general , different edges in a graph will have
different implementations.
In UGMs, conditional distributions are not explicitly represented.
Rather a joint distribution over all the variables is constructed using a
product of clique potential functions as mentioned in Section 2.1. In gen-
era l the clique potentials can be arbitrary functions , although certain types
are commonly used such as Gibbs or Boltzmann distributions [79] . Many
such models fall under what are known as exponential models [44] . The
implementation of a dependency in an UGM, therefore, is implicitly spec-
ified via these functions in that they specify the way in which subsets of
variables , depending on their values, can influence the resulting probability
2.4. Parameterization. The parameterization of a model corresp-

onds to the parameter values of a particular implementation in a particular
structure. For example , with linear regression, parameters are simply the
regression coefficients; for a discrete probability table the parameters are
the table entries. Since parameters of random distributions can themselves
be seen as nodes, Bayesian approaches are easily represented [75] with GMs.
Many algorithms exist for training the parameters of a graphical
model. These include maximum likelihood [44] such as the EM algorithm
[40], discriminative or risk minimization approaches [150], gradient descent
[19], sampling approaches [109], or general non-linear optimization [50].
The choice of algorithm depends both on the structure and implementa-
tion of the GM. For example, if there are no hidden variables, an EM
approach is not required. Certain st ructural properties of the GM might

render certain t raining procedures less crucial to t he performance of t he
model [16, 47] .
2.5 . Efficient probabilistic inference. A key application of any
statistical model is to compute t he probability of one subset of random
variables given values for some ot her subset , a procedure known as proba-
bilistic inference. Inference is essential both to make predictions based on
t he model and to learn the model parameters using, for example, the EM
algorit hm [40, 113]. One of t he critical advantages of GMs is t hat t hey offer
proced ures for making exact inference as efficient as possible, much more
so than if conditional independence is ignored or is used unwisely. And
if t he resultin g savings is not enough, there are GM-inspired approximate
inference algorit hms that can be used.
Exact inference can in general be quite computationally costly. For ex-
ample, suppose t here is a joint distri bution over 6 variables p(a, b, c, d, e, I)
and t he goal is to compute p(all). T his requires both p(a , I) and p(J), so
the variables b, c, d, e must be "marginalized", or integrated away to form
p(a, I). Th e naive way of performing this computation would entail the
following sum:
p(a ,1) = L p(a ,b ,c,d, e,l).

b.c.d ,e
Suppos ing that each variable has K possible values, this comput at ion re-
quires O(K6) operations, a quantity that is exponential in the number of
variables in the joint distribution. If, on t he othe r hand, it was possible
to factor t he joint distribution into factors containing fewer variables, it
would be possible to reduce computation significantly. For example, under
t he graph in Figure 3, the above distr ibut ion may be factored as follows:
p(a , b, c, d, e, I) = p(alb)p(b lc)p(cld, e)p(dl e, I)p( ell)p(J)
so that the sum
p(a, I) = p(J ) LP(alb) L P(blc) LP(cld, e) L P(dje, I) p(ell)

b ed e
requires only O(K 3) computation. Inference in GMs involves formally de-

fined manipulations of gra ph data st ructures and then operations on t hose
data st ructures. T hese operations provably correspo nd to valid operations
on probability equations, and they reduce computation essentially by mov-
ing sums, as in the above, as far to the right as possible in th ese equat ions.
T he graph operations and data st ructures needed for inference are
typically described in the ir own light , without needing to refer back to t he
original probability equations. One well-known form of inference proce-
dure , for example , is the jun ction tree (JT) algorithm [122, 84]. In fact ,
FIG. 3. The graph's independence properties are used to move sums inside of factors .
the commonly used forward-backward algorithm [129] for hidden Markov

models is just a special case of the junction tree algorithm [144], which is
a special case of the generalized distributive law [2].
The JT algorithm requires that the original graph be converted into a
junction tree, a tree of cliques with each clique containing nodes from the
original graph. A junction tree possesses the running intersection property,
where the intersection between any two cliques in the tree is contained
in all cliques in the (necessarily) unique path between those two cliques.
The junction tree algorithm itself can be viewed as a series of messages
passing between the connected cliques of the junction tree . These messages
ensure that the neighboring cliques are locally consistent (i.e., that the
neighboring cliques have identical marginal distributions on those variables
that they have in common) . If the messages are passed in a particular order,
called the message passing protocol [85], then because of the properties of
the junction tree, local consistency guarantees global consistency, meaning
that the marginal distributions on all common variables in all cliques are
identical, meaning that inference is correct. Because only local operations
are required in the procedure, inference can be fast.
For the junction tree algorithm to be valid, however, a decomposable
model must first be formed from the original graph. Junction trees exist
only for decomposable models, and a message passing algorithm can prov-
ably be shown to yield correct probabilistic inference only in that case. It
is often the case, however, that a given DGM or UGM is not decomposable.
In such cases it is necessary to form a decomposable model from a general
GM (directed or otherwise), and in doing so make fewer conditional inde-
pendence assumptions. Inference is then solved for this larger family of
models. Solving inference for a larger family still of course means that in-
ference has been solved for the smaller family corresponding to the original
(possibly) non-decomposable model.
Two operations are needed to transform a general DGM into a de-
composable model: moralization and triangulation. Moralization joins the
unconnected parents of all nodes and then drops all edge directions. This
procedure is valid because more edges means fewer conditional indepen-
dence assumptions or a larger family of probability distributions. Moral-
ization is required to ensure that the resulting UGM does not disobey any
of the conditional independence assumptions made by the original DGM. In
other words, after moralizing, it is assured that the UGM will make no in-
dependence assumption that is not made by th e original DGM. Otherwise,

inference might not be correct.
After moralization, or if starting from a UGM to begin with, trian-
gulation is necessary to produce a decomposable model. The set of all
triangulated graphs corresponds exactly to the set of decomposabl e mod-
els. The triangulation operation [122, 103] adds edges until all cycles in
the graph (of length 4 or greater) contain a pair of non-consecutive nodes
(along the cycle) that are connected by an edge (i.e., a chord) not part
of the cycle edges. Triangulation is valid because more edges enlarge the
set of distributions represented by the graph. Triangulation is necessary
because only for triangulated (or decomposable) graphs do junction trees
exists . A good survey of triangulation techniques is given in [98] .
Finally, a junction tree is formed from the triangulated graph by, first,
forming all maximum cliques in the graph, next connecting all of the cliques
together into a "super" graph, and finally finding a maximum spanning tree
[32] amongst that graph of maximum cliques. In this case, the weight of an
edge between two cliques is set to the number of variables in the intersection
of the two cliques.
For a discrete-node-only network, junction tree complexity is
O(LcEC TIVEc Ivl) where C is the set of cliques in the junction tree, c is the
set of variables contained within a clique, and Ivl is the number of possible
values of variable v - i.e., the algorithm is exponential in the clique sizes,
a quantity important to minimize during triangulation. There are many
ways to triangulate [98] , and unfortunately the operation of finding the
optimal triangulation is itself NP-hard. For an HMM, the clique sizes are
N 2 , where N is the number of HMM states, and there are T cliques leading
to the well known O(T N2) complexity for HMMs. Further information on
the junction tree and related algorithms can be found in [84, 122, 34, 85].
Exact inference, such as the above, is useful only for moderately com-
plex networks since inference is NP-hard in general [31]. Approximate
inference procedures can, however, be used when exact inference is not
feasible. There are several approximation methods including variational
techniques [141 ,81,86], Monte Carlo sampling methods [109]' and loopy
belief propagation [156]. Even approximate inference can be NP-hard how-
ever [35]. Therefore, it is always important to use a minimal model, one
with least possible complexity that still accurately represents the important
aspects of a task.
3. Graphical models and automatic speech recognition. A wide

variety of algorithms often used in state-of-the-art ASR systems can easily
be described using GMs, and this section surveys a number of them. While
many of these approaches were developed without GMs in mind, they turn
out to have surprisingly simple and elucidating network structures. Given
an understanding of GMs, it is in many cases easier to understand the
technique by looking first at the network than at the original algorithmic

description.
As is often done, the following sections will separate ASR algorithms
into three categories: acoustic, pronunciation, and language modeling.
Each of these are essentially statistical models about how the speech data
that we observe is generated. Different statistical models, and inference
within these models, leads us to the different techniques , but each are es-
sentially special cases of the more general GM techniques described above.
3.1. Acoustic modeling: Gaussians. The most successful and
widely used density for acoustic modeling in ASR systems is the multi-
dimensional Gaussian. The Gaussian density has a deceptively simple
mathematical description that does not disclose many of the useful prop-
erties this density possesses (such as that first and second moments com-
pletely characterize the distribution). In this section, it will be shown how
Gaussians can be viewed as both undirected and directed GMs, and how
each of these views describe distinct properties of the Gaussian.
An N-dimensional Gaussian density has the form:
p(x) = P(Xl :N) = N(Xl :N; /L, E) = 127rE\-1/2e - ! (x - /-l )T r; - 1(X-/-l)
where /L is an N-dimensional mean vector, and E is an N x N covari-

ance matrix. Typically, K = E- 1 refers to the inverse covariance (or the
concentration) matrix of the density.
It will be useful to form partitions of a vector x into a number of parts.
For example, a bi-partition of x = [XA XB] may be formed, where XA and
XB are sub-vectors of x [68], and where the sum of the dimensions of XA and
XB equals N . Tri-partitions x = [XA XB xc] may also be formed. In this
way, the mean vector /L = [/LA /LBjT, and the covariance and concentration
matrices can be so partitioned as
and
Conventionally, EA~ = (EAA)-l , so that the sub-matrix operator takes

precedence over the matrix inversion operator. A well known prop erty of
Gaussians is that if E AB = 0 then XA and XB are marginally independent
(XAJLxB) '
A more interesting and less well-known property of a Gaussian is that
for a given tri-partition x = [XA XB xc] of x , and corresponding tri -
partitions of /L and K, then xAJLxBlxc if and only if, in the corresponding
tri-partition of K , KAB = 0, a property that may be proven quite readily.
For any distribution, the chain rule of probability says
When p(x) is a Gaussian density, the marginal distribution p(XB) is also

Gaussian with mean /LB and covariance EBB. Furthermore, p(xAlxB) is a
Gaussian having a "condit ional" mean and covariance [111, 4]. Specifically,
t he distribut ion for XA given XB is a conditional Gaussian with an X B-
depe ndent mean vector
and a fixed covariance matrix
T his means th at , if t he two vectors X A and X B are joint ly Gaussian , t hen

given knowledge of one vector, say X B , th e result is a Gaussian distribution
over XA th at has a fixed variance for all values of XB but has a mean th at
is an affine transformati on of t he particular value of x B . Most importantl y,
it can be shown that K AA , th e upp er-left partition of th e original concen-
tration matrix K, is th e inverse of th e conditional covariance, specifically
K A A = B AllB [103 , 159].
Let t he partition X A be furth er partitioned to form t he sub-bi-part it-
ion XA = [X A a XAb], meaning t hat x = [X A a XAb X B ]. A similar sub-
pa rt it ion is formed of t he concentration mat rix
K A A aa A A ab
K AA = ( KK ) .
KAAba AAbb
Setting K A A ab = 0 implies that XAaJLxAb, but only when conditio ning

on x B . T his yields t he result desired, but with t he matrix and vector
pa rtitions renamed. T herefore, zeros in t he inverse covariance mat rix result
in conditional independence properties for a Gaussian, or more specifically
if K i j = 0 th en X iJLXjIX{1 :N}\{ i,j} '
T his prop erty of Gaussians corresponds to their view as an UGM. To
see this, first consider a fully connected UGM wit h N nodes, somet hing that
represents all Gaussians. Setting an entry, say K i j , to zero corres ponds
to t he independence property above, which correspo nds in the UGM to
removing an edge between variable Xi and Xj (see [103] for a form al proof
where th e pairwise Markov property and th e global Markov prop ert y are
relat ed) . This is shown in Figure 4. Therefore, missing edges in a Gaussian
UGM correspond exactly to zeros in t he inverse covariance matri x.
FIG . 4. A Gaussian viewed as an UGM. On the left, there are no ind ependence
assumptions . On the right, X 2JLX3 !{X1 ,X4 }.
A Gaussian may also be viewed as a BN, and in fact many BNs. Unlike
with a UGM, to form a Gaussian BN a specific variable ordering must first
be chosen, the same ordering used to factor the joint distribution with the
chain rule of probability. A Gaussian can be factored
according to some fixed but arbitrarily chosen variable ordering. Each

factor is a Gaussian with conditional mean
and conditional covariance
both of which are unique for a given ordering (these are an application
of the conditional Gaussian formulas above, but with A and B set to the
specific values {i} and {(i + l):N} respectively). Therefore, the chain rule
expansion can be written:
An identical decomposition of this Gaussian can be produced in a

different way. Every concentration matrix K has a unique factorization
K = UTDU where U is a unit upper-triangular matrix and D is diagonal
[111, 73]. A unit triangular matrix is a triangular matrix that has ones on
the diagonal, and so has a unity determinant (so is non-singular), therefore,
IK I = ID I· This corresponds to a form of Cholesky factorization K = R T R,
where R is upper triangular, Dl/ 2 = diag(R) is the diagonal portion of R,
and R = D 1 / 2U . A Gaussian density can therefore be represented as:
The unit triangular matrices, however, can be "brought" inside the squared
linear terms by considering the argument within the exponential
(x - J.LfUT DU(x -IL) = (U(x - J.L)fD(U(x - J.L))

= (Ux-ilfD(Ux-il)
= ((1 - B)x - ilf D((I - B)x - il)
= (x - Bx - ilf D(x - Bx - il)
where U = I - B, I is the identity matrix, B is an upper triangular ma-
trix with zeros along the diagonal, and [L = UJ.L is a new mean . Again,
this transformation is unique for a given Gaussian and variable ordering.

This process exchanges K for a diagonal matrix D, and produces a linear
auto-regression of x onto itself, all while not changing the Gaussian nor-
malization factor contained in D . Therefore, a full-covariance Gaussian
can be represented as a conditional Gaussian with a regression on x itself,
yielding the following:
p(Xl :N) = (211' )-d/2IDI 1/ 2e-! (X-Bx-iL)T D(x-Bx-iL).

In this form the Gaussian can be factored where the i t h factor uses only
the i t h row of B:
(3.2) p(Xl:N) = II(211')-1/2 Di/ 2 e-!(x i-B; ,iH N x iHN-ji;j2 o; .
When this is equated with Equation (3.1), and note is taken of the unique-
ness of both transformations, it is the case that
and that iii = J-li - B i,i+l :N J-li+l:N . This implies that the regression coef-
ficients within B are a simple function of the original covariance matrix.
Since the quantities in the exponents are identical for each factor (which
are each an appropriately normalized Gaussian), the variance terms Di,
must satisfy:
meaning that the D ii values are conditional variances.

Using these equations we can now show how a Gaussian can be viewed
as a BN. The directed local Markov property of BNs states that the joint
distribution may be factorized as follows:
where lI'i ~ {( i + 1): N} are parents of the variable Xi . When this is con-
sidered in terms of Equation (3.2), it implies that the non-zero entries of
B i,Hl :N correspond to the set of parents of node i, and the zero entries
correspond to missing edges. In other words (under a given variable order-
ing) the B matrix determines the conditional independence statements for
a Gaussians when viewed as a DGM, namely XiJlX {(i+l) :N}\ 11'i IX11'i if and
only if the entries B i ,{(Hl):N}\11'i are zero.2
It is important to realize that these results depend on a particular
ordering of the variables X 1 :N • A different ordering might yield a different
2Standard notation is used here, where if A and B are sets, A \ B is the set of
elements in A that are not in B .
B matrix, possibly implying different indep endence state ments (depending

on if the gra phs are Markov equivalent, see Sect ion 2.2). Moreover , a B
matrix can be sparse for one ordering, but for a different ord ering t he B
matrix can be dense, and zeros in B might or might not yield zeros in
K = (I - Bf D(I - B) or E = K- 1 , and vice versa.
Thi s means t hat a full covariance Gaussian with N(N + 1)/2 non-
zero covariance parameters might act ually employ fewer than N(N + 1)/2
para mete rs, since it is in t he directed domain where sparse pattern s of
independence occur. For example, consider a 4-dimensiona l Gaussian wit h
a B matr ix such that B 12 = B 13 = B 14 = B 24 = B 34 = 1, and along
wit h the ot her zero B entries, take B 23 = o. For this B matri x and when
D = I , neit her the concent ration nor t he covariance matrix has any zeros,
alt hough they are both full rank and it is t rue t hat X2JL X 3 IX 4 . It must be
th at K possesses redundancy in some way, but in t he undirected form alism
it is impossible to encode this independence st at ement and one is forced to
generalize and to use a model th at possesses no independence properties.
T he opposite can occur as well, where zeros exist in K or E, and less
sparsity exists in t he B matrix. Take, for example t he matrix,
K=( H ! ~) .
4 0 3 6
This concent rat ion matrix states that X 1JLX31{X2 , X 4} and X 2JLX41{X1 ,
X 3 } , but the corresponding B matrix has only a single zero in its upp er
portion reflecting only t he first independence statement .
It was mentioned earlier t hat UGMs and DGMs represent different
families of probability distributio ns, and t his is reflected in the Gaussian
case above by a reduction in spa rsity when moving between certain B and
K matrices. It is interestin g to note that Gaussians are able to repre-
sent any of the dependency st ruct ures captured eithe r in a DGM (via an
appropriate orde r of t he variab les and zeros in t he B matrix) or a UGM
(wit h appropriately placed zeros in t he concentration matrix K) . T here-
fore, Gaussians, along with many ot her interestin g and desirable t heoret ical
propert ies, are quite genera l in terms of th eir ability to possess condit ional
indepen dence relationship s.
T he question t hen becomes what form of Gaussian should be used, a
DGM or a UGM, and if a DGM, in what varia ble order . A common goal
is to minimize the total number of free parameters. If this is the case, t he
Ga ussian should be represented in a "natural" domai n [10], where t he least
degree of par ameter redun dancy exists. Sparse matrices often provide the
answer, assuming no additional cost exists to represent sparse matrices,
since the sparse pattern itself might be considered a par ameter needing
a representation. Thi s was exploited in [17], where t he natural directe d
Gaussian representation was solicited from dat a, and where a negligible
penalty in WER performance was obtained with a factored sparse covari-

ance matrix having significantly fewer parameters.
Lastly, it is important to realize that while all UGM or DGM depen-
dency structures can be realized by a Gaussian, the implementations in
each case are only linear and the random components are only univariate
Gaussian. A much greater family of distributions, other than just a Gaus-
sian , can be depicted by a UGM or DGM, as we begin to see in the next
sections.
3.2. Acoustic modeling: PCA/FA/ICA. Our second example of
GMs for speech consists of techniques commonly used to transform speech
feature vectors prior to being used in ASR systems . These include principle
component analysis (PCA) (also called the Karhunen-Loeve or KL trans-
form), factor analysis (FA), and independent component analysis (ICA).
The PCA technique is often presented without any probabilistic interpre-
tation. Interestingly, when given such an interpretation and seen as a
graph, PCA has exactly the same structure as both FA and ICA - the only
difference lies in the implementation of the dependencies .
The graphs in this and the next section show nodes both for random
variables and their parameters. For example, if X is Gaussian with mean
J.L, a J.L node might be present as a parent of X . Parameter nodes will be
indicated as shaded rippled circles. For our purposes, these nodes constitute
constant random variables whose probability score is not counted (they
are conditional-only variables , always to the right of the conditioning bar
in a probability equation). In a more general Bayesian setting [77, 75,
139], however, these nodes would be true random variables with their own
distributions and hyper-parameters.
Starting with PCA , observations of a d-dimensional random vector X
are assumed to be Gaussian with mean J.L and covariance E. Th e goal
of PCA is to produce a vector Y that is a zero-mean uncorrelated linear
transformation of X . The spectral decomposition theorem [146] yields the
factor ization E = I' AfT, where I' is an orthonormal rotation matrix (the
columns of r are orthogonal eigenvectors, each having unit length), and
A is a diagonal matrix containing the eigenvalues that correspond to the
variances of the elements of X. A transformation achieving PCA's goal is
Y = rT(X - J.L). This follows since E[yyTJ = rTE[(X - J.L)(X - J.LfJf
= rTEr = A. Alternatively, a spherically distributed Y may be obtained
by the following transformation: Y = (A- 1/ 2 f) T(X - J.L) = CT(X - J.L)
with C = A-l/2f.
Solving for X as a function of Y yields the following:
X =fY +J.L.
Slightly abusing notation, one can say that X '" N(rY + J.L, 0), meaning
that X, conditioned on Y, is a linear-conditional constant "Gaussian" -
Le., a conditional-Gaussian random variable with mean ry + J.L and zero
vari ance.i' In this view of PCA , Y consists of the latent or hidden "causes"
of th e observed vector X , where Y rv N(O , A), or if the C-transformation
is used above, Y rv N(O , I) where I is the identity matrix. In eit her case,
the variance in X is entirely explained by the variance within Y, as X is
simply a linear transformation of th ese underlying causes. PCA tr ansforms
a given X to the most likely values of the hidden causes. This is equal to
the conditional mean E [YIXj = rT(X - p,) since p(ylx) rv N(rT(x - p,), 0).
The two left graphs of Figure 5 show the probabilis tic interpretations
of PCA as a GM, where the depend ency implementations are all linear.
Th e left graph corresponds to th e case where Y is spherically distributed.
Th e hidden causes Yare called th e "principle components" of X. It is often
the case that only th e components (i.e., elements of Y) corres ponding to
the largest eigenvalues of E are used in the model, the other elements of
Yare removed, so that Y is k-dimensional with k < d. There are many
prop erties of PCA [111] - for example, using the principle k elements of
Y leads to the smallest reconstruction error of X in a mean-squared sense.
Another notable property (which motivates factor analysis below) is that
PCA is not scale invariant - if the scale of X changes (say by converting
from inches to centimeters) , both r and A will also change, leading to
different components Y. In this sense, PCA explains th e variance in X
using only variances found in th e hidden causes Y.
PCA : X=CY+1.1 PCA : x-rv-u PPCA: X=CY+I.1+E FA: X=CY+I.1+E
FIG. 5. Left two gmphs: two views of principle component s analysis (PCA) ; middle:
probabilistic PCA; right ; factor analysis (FA) . In geneml, the gmph corresponds to the
equation X = CY + /l + e, where Y rv N(O,A) and € rv N(O, 111). X is a random
conditional Gaussian with mean C Y + /l and variance CAC T + 111 . With PCA , 111 = 0
so that € = 0 with probability 1. Also, either (far left) A = I is the identity matrix
and C is general, or (second from left) A is diagonal and C = r is orthonorm al. With
PPCA , 111 = a 2 I is a spheri cal covariance matrix, with diagonal term s a 2 • With FA, 111
is diagonal . Other genemlizations are of possible, but they can lead to an in determi nacy
of the parameters.
Factor analysis (the right-most graph in Figur e 5) is only a simple

modification of PCA - a single random variable is added onto t he PCA
equat ion above, yielding:
3This of course corresp onds to a degenerate Gaussian, as t he covariance matrix is

singular.
X=CY+J1.+f
where Y rv N(O, 1), and f rv N(O, '11) with '11 a non-negative diagonal matrix.
In factor analysis, C is the factor loading matrix and Y the common factor
vector . Elements of the residual term f = X - CY - J1., are called the specific
factors, and account both for noise in the model and for the underlying
variance in X. In other words, X possesses a non-zero variance, even
conditional on Y, and Y is constrained to be unable to explain the variance
in X since Y is forced to have I as a covariance matrix. C , on the other
hand, is compelled to represent just the correlation between elements of
X irrespective of its individual variance terms, since correlation can not
be represented by f. Therefore, unlike PCA, if the scale of an element of
X changes, the resulting Y will not change as it is e that will absorb the
change in X 's variance. As in PCA, it can be seen that in FAX is being
explained by underlying hidden causes Y, and the same graph (Figure 5)
can describe both PCA and FA.
Probabilistic PCA (PPCA) (second from the right in Figure 5) [147,
140] while not widely used in ASR is only a simple modification to FA,
where '11 = (J2 I is constrained so that f is a spherically-distributed Gaus-
sian .
In all of the models above, the hidden causes Yare uncorrelated Gaus-
sians, and therefore are marginally independent. Any statistical depen-
dence between elements of X exist only in how they are jointly dependent
on one or more of the hidden causes Y. It is possible to use a GM to make
this marginal Y dependence explicit, as is provided on the left in Figure 6
where all nodes are now scalars . In this case, Yj rv N(O, 1), fj rv N(O,'lfJj),
and p(Xi) rv N(~j CijYj + J1.i, 'lfJi) where 'lfJj = 0 for PCA .
FIG. 6. Left: A graph showing the explicit scalar variables (and therefore their
statistical dependencies) for PCA, PPCA, and FA. The graph also shows the parameters
for these models. In this case, the dependencies are linear and the random variables
are all Gaussian . Right: The graph for PCA/PPCA/FA (parameters not shown) which
is the same as the graph for ICA. For ICA, the implementation of the dependencies
and the random variable distributions can be arbitrary, different implementations lead
to different ICA algorithms . The key goal in all cases is to explain the observed vector
X with a set of statistically independent causes Y.
The PCAjPPCAjFA models can be viewed without the parameter

and noise nodes, as shown on the right in Figure 6. This , however, is the
general model for independent component analysis (ICA) [7, 92], another
method that explains data vectors X with independent hidden causes. Like
PCA and FA, a goal of ICA is to first learn the parameters of the model
that explain X . Once done, it is possible to find Y, the causes of X, that
are as statistically independent as possible. Unlike PCA and FA, however,
dependency implementations in ICA neither need to be linear nor Gaussian.
Since the graph on the right in Figure 6 does not depict implementations,
the vector Y can be any non-linear and/or non-Gaussian causes of X .
The graph insists only that the elements of Yare marginally independent,
leaving alone the operations needed to compute E[YIX] . Therefore, ICA
can be seen simply as supplying the mechanism for different implementation
of the dependencies used to infer E[YIXj . Inference can still be done using
the standard graphical-model inference machinery, described in Section 2.5.
Further generalizations of PCA/FA/ICA can be obtained simply by
using different implementations of the basic graph given in Figure 6. Inde-
pendent factor analysis [6] occurs when the hidden causes Yare described
by a mixture of Gaussians. Moreover, a multi-level factor analysis algo-
rithm, shown in Figure 7, can easily be described where the middle hidden
layer is a possibly non-independent explanation for the final marginally in-
dependent components. The goal again is to train parameters to explain
X, and to compute E[ZJX] . With graphs it is therefore easy to understand
all of these techniques, and simple structural or implementation changes
can lead to dramatically different statistical procedures.
FIG. 7. Multi-level tcs.
3.3. Acoustic modeling: LDA/QDA/MDA/QMDA. When the

goal is pattern classification [44] (deciding amongst a set of classes for X),
it is often beneficial to first transform X to a space spanned neither by the
principle nor the independent components, but rather to a space that best
discriminatively represents the classes. Let C be a variable that indicates
the class of X, ICI the cardinality of C. As above, a linear transformation
can be used, but in this case it is created to maximize the between-class
covariance while minimizing the within-class covariance in the transformed
space. Specifically, the goal is to find the linear transformation matrix A

to form Y = AX that maximizes tr(BW- 1 ) [57] where
W = I>(C = i)Ep(Ylc=i)[(Y -11~)(Y -11;f]

and
B = LP(C = i)(l1; -I1Y)(I1; -l1yf
i
where J.Lt is the class conditional mean and l1y is the global mean in the
transformed space. This is a multi-dimensional generalization of Fisher's
original linear discriminant analysis (LDA) [49] .
LDA can also be seen as a particular statistical modeling assumption
about the way in which observation samples X are generated. In this case,
it is assumed that the class conditional distributions in the transformed
space P(YIC = i) are Gaussians having priors P(C = i). Therefore, Y is
a mixture model p(y) = L:i p(C = i)p(yIC = i), and classification of y is
optimally performed using the posterior:
. p(yli)p(i)
p(C = ~Iy) = L: jP (") (l
YJ PJ
For standard LDA, it is assumed that the Gaussian components p(yJj) =
N(y; I1j, E) all have the same covariance matrix, and are distinguished only
by their different means. Finally, it is assumed that there is a linear trans-
form relating X to Y . The goal of LDA is to find the linear transformation
that maximizes the likelihood P(X) under the assumptions given by the
model above. The statistical model behind LDA can therefore be graphi-
cally described as shown on the far left in Figure 8.
LDA IIDAlQDA MDA IIMDA Semi ·lied
FIG. 8. Linear discriminant analysis (left), and its genemlizations.
There is a intuitive way in which these two views of LDA (a statistical

model or simply an optimizing linear transform) can be seen as identi-
cal. Consider two class-conditional Gaussian distributions with identical
covariance matrices. In this case, the discriminant functions are linear,

and effectively project any unknown sample down to an affine set", in this
case a line, that points in the direction of the difference between the two
means [44]. It is possible to discriminate as well as possible by choosing a
threshold along this line - the class of X is determined by the side of the
threshold X's projection lies.
More generally, consider the affine set spanned by the means of ICI
class-conditional Gaussians with identical covariance matrices. Assum-
ing the means are distinct, this affine set has dimensionality min{ICI -
1, dim(X)} . Discriminability is captured entirely within this set since the
decision regions are hyperplanes orthogonal to the lines containing pairs of
means [44] . The linear projection of X onto the ICI- 1 dimensional affine
set Y spanned by the means leads to no loss in classification accuracy, as-
suming Y indeed is perfectly described with such a mixture. If fewer than
C -1 dimensions are used for the projected space (as is often the case with
LDA), this can lead to a dimensionality reduction algorithm that has a
minimum loss in discriminative information. It is shown in [102] that the
original formulation of LDA (Y = AX above) is identical to the maximum
likelihood linear transformation from the observations X to Y under the
model described by the graph shown on the left in Figure 8.
When LDA is viewed as graphical model, it is easy to extend it to
more general techniques. The simplest extension allows for different co-
variance matrices so that p(xji) = N(x; Mi, ~i), leading to the GM second
from the left in Figure 8. This has been called quadratic discriminant
analysis (QDA) [44, 113]' because decision boundaries are quadratic rather
than linear, or heteroscedastic discriminant analysis (HDA) [102]' because
covariances are not identical. In the latter case, it is assumed that only a
portion of the mean vectors and covariance matrices are class specific -
the remainder corresponds in the projected space to the dimensions that
do not carry discriminative information.
Further generalizations to LDA are immediate. For example, if the
class conditional distributions are Gaussians mixtures, every component
sharing the same covariance matrix, then mixture discriminant analysis
(MDA) [74) is obtained (3rd from the left in Figure 8). A further gen-
eralization yields what could be called be called heteroscedastic MDA,
as described 2nd from the right in Figure 8. If non-linear dependencies
are allowed between the hidden causes and the observed variables, then
one may obtain non-linear discriminant analysis methods, similar to the
neural-network feature preprocessing techniques [51, 95, 78) that have re-
cently been used.
Taking note of the various factorizations one may perform on a posit-
ive-definite matrix [73], a concentration matrix K within a Gaussian dis-
tribution can be factored as K = A"r A. Using such a factorization, each
4 An affine set is simply a translated subspace [135].

Gaussian component in a Gaussian mixture can use one each from a shared
pool of As and I's, leading to what are called semi-tied covariance matrices
[62, 165]. Once again, this form of tying can be described by a GM as
shown by the far right graph in Figure 8.
3.4. Acoustic modeling: Mixture models. In speech recognition,
hidden Markov model observation distributions rarely use only single com-
ponent Gaussian distributions. Much more commonly, mixtures of such
Gaussians are used. A general mixture distribution for p(x) assumes the
existence of a hidden variable C that determines the active mixture com-
ponent as in:
p(x) = 2:p(x, C = i) = 2:p(C = i)p(xIC = i)

i
where p(xlC = i) is a component of the mixture. A GM may simply

describe a general mixture distribution as shown in the graph C --t X .
Conditional mixture generalizations, where X requires Z, are quite easy to
obtain using the graph Z --t C --t X, leading to the equation:
p(xlz) = 2:p(x, C = i, z) = 2:p(C = ilz)p(xIC = i) .

i i
Many texts such as [148, 112] describe the properties of mixture distribu-
tions, most of which .can be described using graphs in this way.
3.5. Acoustic modeling: Acoustic classifier combination. It
has often been found that when multiple separately trained classifiers are
used to make a classification decision in tandem, the resulting classification
error rate often decreases. This has been found in many instances both
empirically and theoretically [82, 100, 124, 160, 19]. The theoretical results
often make assumptions about the statistical dependencies amongst of the
various classifiers, such as that their errors are assumed to be statistically
independent. The empirical results for ASR have found that combination
is useful at the acoustic feature level [12, 94, 72, 95], the HMM state level
[96], the sub-word or word level [163], and even at the utterance level [48] .
Assume that Pi(clx) is a probability distribution corresponding to the
i t h classifier, where c is the class for feature set x . A number of classi-
fication combination rules exist such as the sum rule [97] where p(clx) =
Li p(i)Pi(clx), or the product rule where p(clx) ex [1 Pi(clx). Each of these
schemes can be explained statistically, by assuming a statistical model that
leads to the particular combination rule . Ideally, the combination rule that
performs best will correspond to the model that best matches the data. For
example, the sum rule corresponds to a mixture model described above, and
the product rule can be derived by the independence assumptions corre-
sponding to a naive Bayes classifier [18] . Additional combination schemes,
moreover, can be defined under the assumption of different models, some
of which might not require the errors to be statistically independent.
GRAPHICAL MODEL S FOR ASR 215
More advanced combination schemes can be defined by, of course, as-

suming more sophisticat ed models. One such example is shown in Figure 9,
where a true class variable C drives several error-full (noisy) versions of the
class Ci , each of which gener ates a (possibly quite dependent ) set of feature
vectors. By viewing the process of classifier combination as a graph, and
by choosing the right graph , one may quickly derive combination schemes
t ha t best match th e dat a available and that need not make assumpt ions
which might not be true.
FIG . 9. A GM to describe the process of classifier combination . Th e model does

not require in all cases that the errors are statistically independent.
3.6. Acoustic modeling: Adaptation. It is typically th e case that

additional ASR WER improvements can be obtained by additional adap-
t ation of the model parameters after training has occurr ed, but before th e
final utterance hypothesis is decided upon . Broadly, th ese t ake the form
of vocal-tract length norm alization (VTLN) [91], and explicit parameter
ada pt at ion such as maximum-likelihood linear regression (MLLR) [104]. It
turns out that th ese procedur es may also be described with a OM.
VTLN corresponds to augmenting an HMM model with an addit ional
global hidden variable that indicates the vocal tract length. This variable
determines the transformation on th e acoustic feature vectors th at should
be performed to "normalize" the affect of vocal-tract length on these fea-
t ures. It is common in VTLN to perform all such transformations, and
th e one yielding th e highest likelihood of the data is ultimately chosen to
produce a prob ability score. Th e graph in Figur e 10 shows t his model,
where A indicates vocal-tr act length and t hat can pot entially affect the en-
tire model as shown (in this case, t he figure shows a hidden Markov model
which will be described in Section 3.9). In a "Viterbi" approach, only th e
most probable assignment of A is used to form a probability score. Also,

A is often a conditional-only variable (see Section 2.2) so the prior P(A) is
not counted. If a prior is available, it is also possible to integrate over all
values to produce the final probability score.
FIG. 10. A GM to describe various adaptation and globalparameter tmnsformation

methods,such as VTLN, MLLR, and SAT. The variable A indicates that the parameters
of the entire model can be adapted.
MLLR [104], or more generally speaker adaptation [164], corresponds

to adjusting parameters of a model at test time using adaptation data that
is not available at training time. In an ASR system, this takes the form of
training on a speaker or an acoustic environment that is not (necessarily)
encountered in training data. Since supervised training requires supervi-
sory information, either that is available (supervised speaker adaptation) ,
or an initial recognition pass is performed to acquire hypothesized answers
for an unknown utterance (unsupervised speaker adaptation) - in either
case, these hypotheses are used as the supervisory information to adjust
the model. After this is done, a second recognition pass is performed. The
entire procedure may also be repeated. The amount of novel adaptation
information is often limited (typically a single utterance), so rather than
adjust all the parameters of the model directly, typically a simple global
transformation of those parameters is learned (e.g., a linear transformation
of all of the means in a Gaussian-mixture HMM system). This procedure is
also described in Figure 10, where A in this case indicates the global trans-
formation. During adaptation, all of the model parameters are held fixed
except for A which is adjusted to maximize the likelihood of the adaptation
data.
Finally, speaker adaptive training (SAT) [3] is the dual of speaker
adaptation. Rather than learn a transformation that maps the parameters
from being speaker-independent to being speaker-dependent and doing so
at recognition time, in SAT such a transformation is learned at training
time. With SAT, the speaker-independent parameters of a model along
with speaker-specific transformations are learned simultaneously. This pro-
cedure corresponds to a model that possesses a variable that identifies the
speaker, is observed during training, and is hidden during testing. The
speaker variable is the parent of the transformation mapping from speaker-
independent to speaker-dependent space, and the transformation could po-

tent ially affect all the remaining parameters in th e system . Figure 10 once
again describes the basic structure, with A the speaker variable. During
recognition , either the most likely transformation can be used (a Viterbi
approach), or all speaker transform ations can be used to form an integra-
tive score.
In the cases above, novel forms of VTLN , MLLR, or SAT would arise
simply by using different implementations of the edges between A and th e
rest of the model.
3.7. Pronunciation modeling. Pronunciation modeling in ASR
syst ems involves examining each word in a lexicon, and finding sets of
phone st rings each of which describes a valid instance of the corresponding
word [28, 134,45,52,89]. Often th ese strings are specified probabilistically,
where the probability of a given phone depends on the preceding phone (as
in a Markov chain) , thus producing probabilities of pronunci ation variants
of a word. The pronunciation may also depend on th e acoustics [52] .
Using the chain rule , the probability of a string of T phones V1:T ,
where Vi is a phone , can be written as:
If it is assumed th at only the previous K phones are relevant for determin-

ing the curr ent phone probability, this yields a Kth-order Markov chain .
Typically, only a first-ord er model is used for pronunciation modeling, as
is depicted in Figure 11.
FI G. 11. A simple fir st- order Markov chain . Th is graph encodes the relation ship
QtlLQ l:t- 2IQt-l .
Phones are typically shar ed across multiple words. For example, in th e

two words "bat" and "bag", th e middle phone / ae/ is th e same. Therefore,
it is advant ageous in th e acoust ic Gaussian model for /ae/ to be shared
between these two words. With a first-order model, however, it is possible
only to select the distribution over the next state given th e curre nt one.
This seems to present a problem since P(VtI/ae/) should choose a ItI for
"bat" and a Igl for "bag" . Clearly, then, there must be a mechanism, even
in a first order case, to specify th at th e following Vt might need to depend
on more than just the current phone.
Fortunatel y, there are several ways of achieving this issue. The easiest
way is to expand the cardinality of Vi (i.e., increase the state space in th e
Markov chain ). That is, the set of values of Vt represents not only the
different phones , but also different positions of different words. Different
values of Vt, corresponding to the same phone in different words, would

then correspond to the same acoustic Gaussians, but the distribution of
Vt+l given Vt would be appropriate for the word containing Vt and the
position within that word. This procedure is equivalent to turning a x».
order Markov chain into a first-order chain [83].
Another way to achieve this effect is rather than condition on the pre-
vious phone, condition instead on the word Wt and the sequential position
of a phone in the word St, as in P(vtIWt, St). The position variable is
needed to select the current phone. A Markov chain can be used over the
two variables Wt and St . This approach corresponds to expanding the
graph in Figure 11 to one that explicitly mentions the variables needed to
keep track of and use each phone, as further described in Section 5.
A GM view of a pronunciation model does not explicitly mention the
non-zero entries in the stochastic matrices in the Markov chain . Stochastic
finite state automata (SFSA) [130] diagrams are ideally suited for that
purpose. A GM, rather, explains only the independence structure of a
model. It is important to realize that while SFSAs are often described
using graphs (circles and arrows) , SFSA graphs describe entirely different
properties of a Markov chain than do the graphs that are studied in this
text.
Pronunciation modeling often involves a mapping from base-forms
(isolated word dictionary-based pronunciations) to surface forms (context-
dependent and data-derived pronunciations more likely to correspond to
what someone might say). Decision trees are often used for this purpose
[28, 134], so it is elucidative at this point to see how they may be described
using GMs [86]5. Figure 12 shows a standard decision tree on the right,
and a stochastic GM version of the same decision tree on the left. In the
graphical view, there is an input node I, an output node 0 , and a series
of decision random variables Di. The cardinality of the decision variables
D, is equal to the arity of the corresponding decision tree node (at roughly
the same horizontal level in the figure) - the figure shows that all nodes
have an arity of two (l.e., correspond to binary random variables) .
In the GM view, the answer to a question at each level of the tree
is made with a certain probability. All possible questions are considered ,
and a series of answers from the top to the bottom of the tree provides
the probability of one of the possible outputs of the decision tree. The
probability of an answer at a node is conditioned on the set of answers
higher in the tree that lead to that node. For example, D 1 = 0 means the
answer is 0 to the first question asked by the tree . This answer occurs with
probability P(D 1 = iiI). The next answer is provided with probability
P(D 2 = jID 1 ,!) based on the first decision, leading to the graph on the
left of the figure.
5These GMs also describe hierarchical mixtures of experts [87J .

FIG. 12. Left : A GM view of a decision tree, which is a probabilistic generalization

of th e more familiar decision tree on the right.
In a normal decision tree, only one decision is made at each level in the
tree. A GM can represent such a "crisp" tree by insistin g th at th e distribu-
tion s at each level De (and th e final decision 0) of th e tree are Dirac-delta
functions, such as P(D e = ii I) = Ji,!t U,du _ 1 ) where fe(I , dU-l ) is a de-
terrninistic function of th e input I and previousl y made decisions dU-l ,
where de = fe(I , du-d· Th erefore, with the appropriate implementation
of dependencies, it can be seen that th e GM-view is a probabilistic gener-
alization of normal decision trees.
3.8. Language modeling. Similar to pronunciation modeling, th e

goal of language modeling is to provide a probability for any possible string
of words W1 :T in a langu age. Th ere are many varieties of language models
(LMs) [83,136,118,29], and it is beyond th e scope of this paper to describe
t hem all. Nevertheless, th e following section uses GMs to portray some of
th e more commonly and successfully used LMs.
At this time, the most common and successful language model is the
n-gram. Similar to pronunciation modeling, the chain rule is applied to
a joint distribution over words P(W1 :T)' Within each condit ional factor
p(WtIWl:t-d , the most distant parent variables are dropped until an (n-
l)th order Markov chain results p(WtlWt-n+1 :t-I) = p(WtIHt} , where H,
is th e length n - 1 word history. For a bi-gram (n = 2), this leads to a
gra ph identical to th e one shown in Figure 11. In general , trigrams (i.e.,
2nd-ord er Markov chains) have so far been most successful for language
modeling among all values n [29].
While a graphical model showing an (n - l) th- order Markov chain
accurately depicts th e statistical independence assumptions made by an
n-gram, it does not portray how the parameters of such a model are typi-
cally obtained, a procedure that can be quite involved [29]. In fact , much
research regarding n-grams involves methods to cope with data-sparsity
- because of insufficient training data, "smoothing" methodology must
be employed , whereby a Kth order model is forced to provide probability
for length K + 1 strings of words that did not occur in training data. If
a purely maximum-likelihood procedure was used, these strings would be
given zero probability.
•••
FIG. 13. A GM view oj a LM. The dashed arcs indicate that the parents are
switching, The hidden switching parents St switch between the word variables Wt [orm-
ing either a zeroth (S« = 1), first (St = 2), or second (S« = 3) order Markov chain.
The switching parents also possess previous words as parents so that the probability oj
the Markov- chain order is itself context dependent .
Often, smoothing takes the form of mixing together higher- and lower-
order sub-models, with mixing weights determined from data not used for
training any of the sub-models [83, 29] . In such a case, a language model
mixture can be described by the following equation:
p(WtIWt-l' Wt-2) = 03(Wt-l, Wt-2)!(Wt!Wt-b Wt-2)

+ 02(Wt-l, Wt-2)!(WtI Wt-l)
+ 01 (Wt-1, Wt-2)!(Wt)
where I:i 0i = 1 for all word histories, and where the 0 coefficients are some
(possibly) history-dependent mixing values that determine how much each
sub-model should contribute to the total probability score. Figure 13 shows
this mixture using a graph with switching parents (see Section 2.2). The
variables St correspond to the 0 coefficients, and the edges annotated with
values for St exist only in the case that St has those values. The dashed
edges between S, and W t indicate that the S, variables are switching rather
than normal parents. The graph describes the statistical underpinnings of
many commonly used techniques such as deleted interpolation [83], which
is a form of parameter training for the St variables . Of course, much of the
success of a language model depends on the form of smoothing that is used

[29], and such methods are not depicted by Figure 13 (but see Figure 15).
A common extension to the above LM is to cluster words together and
then form a Markov chain over the word group clusters, generally called a
class-based LM [24]. There are a number of ways that these clusters can be
formed, such as by grammatical category or by data-driven approaches that
might use decision trees (as discussed in [83, 24]). Whatever the method,
the underlying statistical model can also be described by a GM, as shown in
Figure 14. In this figure, a Markov chain exists over a (presumably) much
lower dimensional class variables C rather than the high-dimensional word
variables. This representation can therefore considerably decrease model
complexity and therefore lower parameter estimation variance.
FIG. 14. A class-based language model. Here, a Markov chain is used to model the
dynamics of word classes mther than the words themselves .
The class-based language model can be further extended to impose

certain desirable constraints with respect to words that do not occur in
training material, or the so-called unknown words. It is common in a
language model to have a special token called unk indicating the unknown
word. Whenever a word is encountered in a test set that has not occurred in
the training set, the probability of unk should be given as the probability of
the unknown word. The problem, however, is that if maximum likelihood
estimates of the parameters of the language model are obtained using the
training set, the probability of unk will be zero. It is therefore typical to
force this token to have a certain non-zero probability and in doing so,
essentially "steal" probability mass away from some of the tokens that do
indeed occur in the training set. There are many ways of implementing such
a feature, generally called language model back-off [83]. For our purposes
here, it will be sufficient to provide a simple model, and show how it can
be enforced by an explicit graph structure."
Suppose that the vocabulary of words W can be divided into three
disjoint sets: W = [unk} USUM, where unk is the token representing the
unknown word, S is the set of items that have occurred only one time in the
training set (the singletons), and M is the set of all other lexical items. Let
6Thanks to John Henderson who first posed the problem to me of how to represent
this construct using a graphical model, in the context of building a word-tagger.
us suppose also that we have a maximum-likelihood distribution Pml over

words in Sand M, such that LwPml(W) = 1, Pml(unk) = 0, and in general
N(w)
Pml(w) = Jr'
where N(w) is the number of times word W occurs in the training set,
and N is the total number of words in the training set. This means, for
example, that N(w) = 1,w E S.
One possible assumption is to force the probability of unk to be 0.5
times the probability of the entire singleton set, i.e., p(unk) = 0.5*Pml(S) =
0.5 * LWESPml(W) . This requires that probability be taken away from
tokens that do occur in the training set . In this case probability is removed
from the singleton words, leading to the following desired probability model
Pd(W) :
0.5Pml(S) if W = unk
(3.3) Pd(W) = 0.5Pml(w) if wE S
{ Pml (w) otherwise.
This model, of course, can easily be modified so that it is conditioned on

the current class pd(wlc) and so that it uses the conditional maximum
likelihood distribution pml(wlc). Note that this is still a valid probability
model, as LwPd(wlc) = 1.
The question next becomes how to represent the constraints imposed
by this model using a directed graph. One can of course produce a large
conditional probability table that stores all values as appropriate, but the
goal here is to produce a graph that explicitly represents the constraints
above, and that can be used to train such a model. It might seem at first
not to be possible because the variable W must be used both to switch in
different distributions and, once a given distribution has been selected, as
a probabilistic query variable. The variable W can not exist both on the
left of the conditioning bar (where it is possible to produce a probability
of W = w) and also on the right of the conditioning bar (where it can be
used to select the current distribution).
Surprisingly, there exists a solution within the space of directed graphi-
cal models shown in Figure 15. This graph, a modified class-based language
model, includes a child variable vt at each time that is always observed
to be vt = 1. This means, rather than compute p(Wt Icd at each time
step, we compute p(Wt, vt = 1Ict). The goal is to show that it is possi-
ble for p(Wt, vt = lied = Pd( Wt led. The variables B, and K; are both
binary valued for all t, and these variables are also switching parents (see
Section 2.2) of W t . From the graph, we see that there are a number of
conditional distributions that need to be defined. Before doing that, two
auxiliary distributions are defined so as to make the definitions that follow
simpler:
Pml(wIC)
ifw EM
(3.4) PM(wlc) ~ p(~lc)
{
else
~
where p(Mlc) = EwEMPml(wlc), and
Pml(wlc)
ifw E S
(3.5) ps(wlc) ~ p(~lc)
{
else
where p(Slc) ~ Z=WES Pml(wlc). Note that both PM and PS are valid nor-
malized distributions over all words. Also, p(Slc) + p(Mlc) = 1 since these
two quantities together use up all the probability mass contained in the
maximum likelihood distribution.
FIG. 15. A class-based language model that forces the probability of unk to be a
times the probability of all singleton words. The Vt variables are shaded , indicating that
they are observed, and have value V t = 1, Vt. The K, and B; variables are switching
parents of W t .
The remaining distributions are as follows. First, B, has a binary

uniform distribution:
(3.6) p(B t = 0) = p(Bt = 1) = 0.5.

The observation variable lit = 1 simply acts as in indicator, and has a
distribution that produces probability one only if certain conditions are
met, and otherwise produces probability zero:
(3.7) p(lIt = llwt, kt ) = l{(W,ES,k,=l) or (wtEM,k,=O) or (Wt=unk,k,=l)}

where 1A is a binary indicator function that is unity only when the event
A is true, and is zero otherwise. Next, the word distribution will switch
between one of three distributions depending on the values of the switching
parents K, and Be; as follows:
PM(wtlcd if k t = 0
(3.8) p(Wt/kt, bt, cd = ps(wtlcd ~f k t = 1 and bt = 1
{
8{wt=unk} If k t = 1 and bt = O.
Note that the third distribution is simply a Dirac-delta distribution, giving

probability one only when Wt is the unknown word. Last, the distribution
for K, is as follows:
(3.9)
This model correctly produces the probabilities that are given in Equa-
tion 3.3. First, when Wt = unk:
p(Wt = unk, Vt = 1) = p(Vt = llkt = 1, Wt = unk)

x p(Wt = unklkt = 1, bt = 0, Ct)
x p(bt = 0) x p(k t = llcd
= 1 x 1 x 0.5 x p(Slct)
= 0.5p(Slct)
as desired. This follows because the other terms, for different values of the
hidden variables, are all zero. Next, when Wt E S,
p(w(, Vt = 1) = p(Vt = 11 kt = 1, Wt E S) x p(Wt Ik« = 1, b; = 1, cd

x p(bt = 1) x p(k t = llcd
= 1 x ps(wtlcd x 0.5 x p(Slct)
= 0.5 * Pml(wtlcd
again as desired. Lastly, when Wt E M,
p(Wt, Vt = 1) = p(Vt = llkt = 0, Wt EM) x ( L p(wtlkt = 0, bi, Ct)P(bt))

b,E{O,l}
x p(k t = 0lct)
= 1 x PM (wtlct} x p(MICt)
= pml(WtICt) .
In this last case, B, has no influence as it is marginalized away - this
is because the event K; = a removes the parent B t from Wt. Once the
graph structures and implementations are set up, standard GM learning
algorithms can be used to obtain smoothed parameters for this distribution.
Many other such models can be described by directed graphs in a sim-
ilar way. Moreover, many language models are members of the family of
exponential models[44]. These include those models whose parameters are
learned by maximum entropy methods [126, 83, 9, 137], and are derived
by establishing a number of constraints that the underlying probability

distribution must possess. The goal is to find a distribution satisfying
these constraints and that otherwise has maximum entropy (or minimum
KL-divergence with some desired distribution [126]). Note that such an
approach can also be used to describe the distribution over an entire sen-
tence[138] at a time , rather than a conditional distribution of the current
word Wt given the current history tu, Such maximum entropy models can
be described by UGMs, where the edges between words indicate that there
is some dependency induced by the constraint functions . In many cases,
the resulting graphs can become quite interesting.
Overall, however, it is clear that there are a multitude of ways to depict
language models with GMs, and this section has only begun to touch upon
this topic.
3.9. GMs for basic speech models. The Hidden Markov model
(HMM) is still the most successful statistical technique used in ASR. The
HMM encompasses standard acoustic, pronunciation, and most language
modeling into a single unified framework. This is because pronunciation
and language modeling can be seen as a large finite-state automata that
can be "flattened" down to a single first-order Markov chain [116, 83]. This
Markov chain consists of a sequence of serially connected discrete hidden
variables during recognition, thus the name HMM.
Most generally, a hidden Markov model (HMM) is collection of T dis-
crete scalar random variables QI :T and T other variables X1:T that may be
either discrete or continuous (and either scalar- or vector-valued). These
variables, collectively, possess the following conditional independence prop-
erties:
(3.10) Qt :TJLQ 1:t - 2 IQ t - 1
and
(3.11)
for each es i : T . Q~t refers to all variables Qr except for the one at time
T = t. The length T of these sequences is itself an integer-valued random
variable having a complex distribution. An HMM consists of a hidden
Markov chain of random variables (the unshaded nodes) and a collection
of nodes corresponding to the speech utterance (the shaded nodes). In
most ASR systems, the hidden chain corresponds to sequences of words ,
phones, and sub-phones.
This set of properties can be concisely described using the GM shown
in Figure 16. The figure shows two equivalent representations of an HMM,
one as a BN and another as an UGM. They are equivalent because moral-
izing the BN introduces no edges, and because the moralized HMM graph
is already triangulated and therefore decomposable . The UGM on the
right is the result of moralizing the BN on the left. Interestingly, the same
graph describes the structure of a Kalman filter [70] where all variables are
continuous and Gaussian and all dependency implementations are linear.
Kalman filter operations are simply applications of the formulas for con-
ditional Gaussians (Section 3.1), used in order to infer conditional means
and covariances (the sufficient statistics for Gaussians).
FIG. 16 . A hidden Markov model (HMM) , viewed as a gmphical model. Note that
an HMM may be equivalently viewed either as a directed (left) or an undirected (right)
model, as in this case the conditional independence properties are the same .
In a standard HMM with Gaussian mixture observation densities, each

value of the hidden variable (i.e., each state) corresponds to a separate
(possibly tied) mixture distribution (Figure 17). Other forms of HMM also
exist, such as when there is a single global pool of Gaussians, and each
state corresponds to a particular mixture over this global pool. This is
often called a semi-continuous HMM (similar to vector quantization [69]),
and corresponds to the state-conditional observation equation:
p(xlQ = q) = I>(C = ilQ = q)p(xIC = i) .

i
In other words, each state uses a mixture with components from this glob-
ally shared set of distributions. The GM for such an HMM loses an edge
between Q and X as shown on the right in Figure 17. In this case, all of
the represented dependence occurs via the hidden mixture variable at each
time.
FIG. 17. An HMM with mixture observation distributions (left) and a semi-
continuous HMM (right) .
Still another modification of HMMs relaxes one of the HMM condi-

tional independence statements, namely that successive feature vectors are
conditionally independent given the state. Auto-regressive, or correlation
HMMs [157, 23, 120], place an additional edges between successive obser-
vation vectors. In other words, the variable X, might have as a parent not
only the variable Qt but also the variables Xt-l for l = 1,2, . .. , K for some
K . The case where K = 1 is shown in Figure 18. When the additional de-
pendencies are linear and Gaussian, these are sometimes called conditional
Gaussian HMMs [120].
FIG. 18. An Auto-regressive HMM as a GM .
Note that although these models are sometimes called vector-valued

auto-regressive HMMs, they are not to be confused with auto-regressive,
linear predictive, or hidden filter HMMs [127, 128, 88, 129]. These latter
models are HMMs that have been inspired from the use of linear-predictive
coefficients for speech [129] . They use the observation distribution that
arises from random Gaussian noise sources passed through a hidden-state
dependent auto-regressive filter . The filtering occurs at the raw acoustic
(signal) level rather than on the observation feature vector (frame) level.
These earlier models can also be described by an GM that depicts state-
condit ioned auto-regressive models at the speech sample level.
Our last example of an augm ented HMM is something often called an
input-output HMM [8] (See Figure 20). In this case, there ar e vari ables
at each time frame corresponding both to the input and the output. The
output variables are to be inferred. Given a complete input feature stream
X 1:T , one might want to find E[YIX], the most likely values for the output.
These HMMs can therefore be used to map from a continuous vari able
length input feature streams to output stream . Such a model shows promise
for speech enh ancement.
Whil e HMMs account for much of the te chnology behind existing ASR,
GMs include a much lar ger spac e of models. It seems quite improbable
that within this space, it is the HMM alone that is somehow intrinsically
superior to all other models. While there are of course no guar antees to the
following, it seems reasonable to assume that because the space of GMs is
larg e and diverse, and because it includes HMMs, that there exists some
model within this space that will greatly outperform the HMM. Section 4
begins to explore more advanc ed speech models as viewed from a GM
persp ective.
3.10. Why delta features work. State-of-the-art ASR systems aug-
ment HMM feature vecto rs X; with approximations to their first and second
ord er time-derivatives (called delt a- and delt a-delta- features [46, 58-60] ,
or just "dynamic" features). Most often , estimat es of the derivative are
obtained using linear regression [129], namely:
K
L kx,
k=-K
it = K
L
k=-K
k
2
where K in this case is the number of points used to fit the regression. This
can be viewed as a regression because
K
Xt = L akXt-k +e
k=-K
where ak are defined accordingly, and e can be seen as a Gaussian error

term. A new feature vector is then produced that consists of Xt and Xt
appended together.
It is elucidating to expand the joint distribution of the features and the
deltas, namely p(Xl :T , Xl :T) = L: q 1 :T p( Xl:T, Xl :Tlql:T )p(Ql :T) . The state
conditioned joint distribution within the sum can be expanded as:
The conditional distribution P(Xl :TIQ1:T) can be expanded as is normal for

an HMM [129, 11], but
P(Xl :Tl xl :T, Ql:T) = II p(xtlparents(Xt)) .

t
This last equation follows because, observing the process to generate delta
features ,x, is independent of everything else given its parents. The parents
of X t are a subset of X l:T and they do not include the hidden variables
Qt . This leads to the GM on the left in Figure 19, a generative model
for HMMs augmented with delta features . Note that the edges between
the feature stream Xt , and the delta feature stream X t correspond to de-
terministic linear implementations. In this view, delta-features appear to
be similar to fixed-dependency auto-regressive HMMs (Figure 18), where
each child feature has additional parents both from the past and from the
future. In this figure, however, there are no edges between x, and Qt,
because XtJLQtlparents(Xt} . This means that parents(Xt} contain all the
information about Xt, and Qt is irrelevant.
It is often asked why delta features help ASR performance as much
as they do. The left of Figure 19 does not portray the model typically
used with delta features. A goal of speech recognition is for the features
to contain as much information as possible about the underlying word
sequence as represented via the vector Ql :T. The generative model on the
left in Figure 19 shows, however, that there is zero information between the
x, and Qt. When the edges between x,
and its parents parents(Xt} are
F IG. 19. An GM-based explanation of why delta features work in HMM-based ASR
systems. The left figure gives a GM that shows the generative process of HMMs with
delta features. The right figure shows how delta features are typically used in an HMM
system, where the information between x, and Qt is greatly increased relative to the
left figure.
removed, the mutual information [33] between x,

and Qt can only increase
(from zero to something greater) relative to the generative model. The right
of Figure 19 thus shows the standard model used with deltas, where it is
not the case that XtJLQt. Since in the right model, it is the case that more
information about x, and Qt exist, it might be said th at this model has a
structure that is inherently more discriminative (see Section 5).
Interestingly, the above analysis demonstrates th at additional condi-
tional independence assumptions (i.e., fewer edges) in a model can increase
the amount of mutual information that exists between random variables.
When edges are added between the delta features and th e generative par-
ents X t , the delta features become less useful since there is less (or zero)
mutual information between them and Qt .
Therefore, the very conditional independence assumptions that are
commonly seen as a flaw of th e HMM provide a benefit when using delta
features. More strongly put, the incorrect statistical independence proper-
ties made by the HMM model on the right of Figure 19 (relative to truth, as
shown by the generative model on the left) are the very thing that enable
delta features to decrease recognition error. The standard HMM model
with delta features seem to be an instance of a model with an inherently
discriminative structure [16, 47] (see also Section 5).
In general, can the removal of edges or additional processing lead to
and overall increase in the information between the entire random vectors
X 1:T and Ql:T? The data processing inequality [33] says it can not. In the
above, each feature vector (X t , X t ) will have more information about the
temporally local hidden variable Qt - this can sometimes lead to better
word error scores. This same analysis can be used to better understand
other feature processing strategies derived from multiple frames of speech ,
such as PCA or LDA preprocessing over multiple windows [71] and other
non-linear generalizations [51, 95, 78].
It has often been found that conditionally Gaussian HMMs (as in

Figure 18) often do not provide an improvement when delta features are
included in the feature stream [20, 23, 93, 158] . The above provides one
possible explanation, namely that by having a delta feature x,
include as
its parent say X t - 1 , the mutual information between x,
and Qt decreases
(perhaps to zero). Note, however, that improvements were reported with
the use of delta features in [161, 162] where discriminative output distri-
butions were used. In [105, 106], successful results were obtained using
delta features but where the conditional mean , rather than being linear,
was non-linear and was implemented using a neural network . Furthermore,
Buried Markov models [16] (to be described below) also found an improve-
ment with delta features and additional dependencies, but only when the
edges were added discriminatively.
FIG . 20. An input-output HMM. Xl :T the input is tmnsformed via integmtion over
a Markov chain Ql :T into the output Yl :T .
4. GMs for advanced speech models. Many non-HMM models

for speech have been developed outside the GM paradigm but turn out
to be describable fairly easily as GMs - this section to describe some of
them. While each of these models are quite different from one another,
they can all be described with only simple modifications of an underlying
graph structure.
The first example presented is a factorial HMM [67]. In this case,
rather than a single Markov chain, multiple Markov chains are used to
guide the temporal evolution of the probabilities over observation distribu-
tions (see Figure 21). The multiple hidden chains can be used to represent
a number of real-world phenomena. For example, one chain might rep-
resent speech and another could represent an independent and dynamic
noise source [90]. Alternatively, one chain could represent the speech to
be recognized and the other chain could represent confounding background
speech [151, 152]7, or the two chains might each represent two underlying
7 A related method to estimate the parameters of a composite HMM given a col-

lection of separate, independent, and already trained HMMs is called parallel model
combination [64J .
concurrent and independent sub-processes governing the realization of the

observation vectors [61, 155, 108]. Such factored hidden state representa-
tions have also been called HMM decomposition [151, 152] in the past.
FIG. 21. A factorial HMM where there are multiple hidden Markov chains.
One can imagine many modifications of this basic structure, where

edges are added between variables at each time step . Often, these sepa-
rate Markov chains have been used for modeling separate loosely coupled
streams of hidden articulatory information [131, 132] or to represent a cou-
pling between phonetic and articulatory information [167, 145].
It is interesting to note that the factorial HMMs described above are
all special cases of HMMs. That is, they are HMMs with tied parame-
ters and state transition restrictions made according to the factorization.
Starting with a factorial HMM consisting of two hidden chains Qt and Rt,
an equivalent HMM may be constructed by using IQII~I states and by re-
stricting the set of state transitions and parameter assignments to be those
only allowed by the factorial model. A factorial HMM using M hidden
Markov chains each with K states that all span T time steps can have time
complexity O(TMKM+l) [67]. If one translates the factorial HMM into
an HMM having K M states, the complexity becomes O(TK 2M ) which is
significantly larger. An unrestricted HMM with K M states will, however,
have more expressive power than a factorial HMM with M chains each
with K states because in the HMM there are no required state transition
restrictions and any form of correlation may be represented between the
separate chains. It is possible, however, that such an expanded state space
would be more flexible than needed for a given task. Consider, as an exam-
ple, the fact that many HMMs used for ASR have only simple left-to-right
Markov chain structures.
As mentioned earlier, the GM for an HMM is identical to that of a
Kalman filter - it is only the nodes and the dependency implementations
that differ. Adding a discrete hidden Markov chain to a Kalman filter
allows it to behave in much more complex ways than just a large joint
Gaussian. This has been called a switching Kalman filter, as shown in
Figure 22. A version of this structure, applied to ASR, has been called
a hidden dynamic model [125] . In this case, the implementations of the
dependences are such that the variables are non-linearly related.
FIG. 22. The GM corresponding to a switching Kalman filter (SKM) . The Q

variables are discrete, but the Y and X variables are continuous . In the standard
SKM, the implementations between continuous variables are linear Gaussian, other
implementations can be used as well and have been applied to the ASR problem.
Another class of models well beyond the boundaries of HMMs are

called segment or trajectory models [120] . In such cases, the underlying
hidden Markov chain governs the evolution not of the statistics of individual
observation vectors. Instead, the Markov chain determines the allowable
sequ ence of observation segments, where each segment may be described
using an arbitrary distribution. Specifically, a segment model uses the joint
distribution over a variable length segment of observations conditioned on
the hidden state for that segment. In the most general form, the joint
distribution for a segment model is as follows:
p(X1 :T=Xl:T) =
(4.1)
L L L II P(Xt(i,l)' Xt(i ,2),' .. , Xt(i,£;) , l'ilqi,r)p(qi!qi-l , r)p(r).
T
T ql :T £l :T i=l
There are T time frames and r segments where the i th segment is of a

hypothesized length l'i. The collection of lengths are constrained such that
L:~=l l'i = T. For a particular segmentation and set of lengths, the i th
segment starts at time frame t(i ,l) = f(ql :71l'l:71i,l) and ends at time
frame t(i ,l'i) = f(ql :T,l'l:T,i,l'i) . In this general case, the time variable t
could be a general function fO of the complete Markov chain assignment
ql:71 the complete set of currently hypothesized segment lengths l'l :71 the
segment number i, and the frame position within that segment 1 through
l'i' It is assumed that f(ql :71l'l :71i,l'i) = f(q1:T,l'l :71i + 1,1) -1 for all
values of all quantities.
Renumbering the time sequence for a segment starting at one , an ob-
servation segment distribution is given by:
where P(Xl, X2, . . " xdl', q) is the length l' segment distribution under hid-
den Markov state q, and p(l'lq) is the explicit duration model for state q.
GRAPHICAL MODEL S FOR ASR 233
A plain HMM may be repr esented using t his framework if p(t'jq) is a

geometric distribution in t' and if
e
p(Xl,X2,'" , xll t',q) = IIp(Xjlq)
j=l
for a st ate specific distribution p(xlq). One of the first segment models [121]
is a generalization that allows observations in a segment to be addit ionally
dependent on a region within a segment
l
P(Xl ' X2, "" Xl!t', q) = II p(Xjh ,q)
j=l
where rj is one of a set of fixed regions within th e segment . A more general

model is called a segmental hidden Markov model [63]
p(Xl, X2, . . . ,xl lt',q) = J

p(Jllq)
j= l
e
II p(Xj IJl, q)dJl
where Jl is the multi-dim ensional conditional mean of th e segment and
where th e resulting distribution is obtained by integrat ing over all pos-
sible st ate- condit ioned means in a Bayesian sett ing. More general still ,
in tr ended hidden Markov models [41, 42], the mean trajectory within a
segment is described by a polynomial function over tim e. Equation 4.1
gener alizes many models including th e condit ional Gaussian methods dis-
cussed above. A summ ary of segment models, th eir learning equations, and
a complete bibliography is given in [120].
One can view a segment model as a GM as shown in Figure 23. A
single hidden variable T is shown t hat determines t he number of segments.
Within each segment, additional dependencies exist . The segment model
allows for th e set of dependencies within a segment to be arbi tr ary, so it is
likely th at many of t he dependencies shown in th e figure would not exist in
practice. Moreover, there may be addit ional dependencies not shown in the
figure, since it is th e case that th ere must be constra ints on th e segment
lengths. Nevertheless, this figure quickly details the essenti al st ruct ure
behind a segment model.
5. GM-motivated speech recognition. Th ere have been several
cases where graph ical models have t hemselves been used as t he cruxes of
speech recognition systems - this section explores several of them.
Perhaps the easiest way to use a gra phical model for speech recognition
is to start with the HMM graph given in Figure 16, and extend it with
either addit ional edges or additional variables. In th e former case, edges
can be added between the hidden variables [43, 13] or between observed
variables [157,23 , 14]. A crucial issue is how should th e edges be added, as
•••
•••
•••
FIG. 23. A Segment model viewed as a GM.
mentioned below. In the latter case, a variable might indicate a condition

such as noise level or quality, gender, vocal tract length, speaking mode,
prosody, pitch, pronunciation, channel quality, microphone type, and so
on. The variables might be observed during training (when the condition
is known) , and hidden during testing (when the condition can be unknown) .
In each case, the number of parameters of the system will typically increase
- in the worst of cases, the number of parameters will increase by a factor
equal to the number of different conditions.
In Section 3.7 it was mentioned that for an HMM to keep track of the
differences that exist between a phone that occurs in multiple contexts, it
must expand the state space so that multiple HMM states share the same
acoustic Gaussian mixture corresponding to a particular phone. It turns
out that a directed graph itself may be used to keep track of the necessary
parameter tying and to control the sequencing needed in this case [1671.
The simplest of cases is shown in Figure 24, which shows a sequence of
connected triangles - for each time frame a sequence variable St , a phone
variable Qt, and a transition variable R; is used. The observation variable
X, has as its parent only Qt since it is only the phone that determines the
observation distribution. The other variables are used together to appro-
priately sequence through valid phones for a given utterance.
In this particular figure, straight lines are used to indicate that the
implementations of the dependencies are strictly deterministic, and rippled
lines are used to indicate that the implementations correspond to true
random dependencies . This means, for example, that p(St+1 = ilR t , Sd =
bi,J(Rt ,S,) is a Dirac-delta function having unity probability for only one
possible value of St+l given a particular pair of values for R; and St .
In the figure, St is the current sequence number (i.e., 1,2,3, etc .) and
indicates the sub-word position in a word (e.g., the first, second, or third
FIG. 24. A BN used to explicitly represent parameter tying. In this figure, the
straight edges correspond to deterministic implementations and the rippled edges corre-
spond to stochastic implementations.
phone). St does not determine the identity of the phone. Often, S, will
be a monotonically increasing sequence of successive integers , where either
St+l = S, (the value stays the same) or St+l = St+1 (an increment occurs).
An increment occurs only if R; = 1. R t is a binary indicator variable that
has unity value only when a transition between successive phone positions
occurs. R, is a true random variable and depending on the phone (Qd,
R t will have a different binary distribution, thereby yielding the normal
geometric duration distributions found in HMMs. Qt is a deterministic
function of the position St . A particular word might use a phone multiple
times (consider the phone laal in the word "yamaha"). The variable S,
sequences, say, from 1 through to 6 (the number of phones in "yamaha") ,
and Qt then gets the identity of th e phone via a deterministic mapping
from St to Qt for each position in the word (e.g., 1 maps to IyI, 2 maps to
laa/ , 3 maps to [va], and so on) . This general approach can be extended
to multiple hidden Markov chains, and to continuous speech recognition
to provide graph structures that explicitly represent the control structures
needed for an ASR system [167, 13, 47].
As mentioned above, factorial HMMs require a large expansion of the
state space and therefore a large number of parameters. A recently pro-
posed system that can model dependencies in a factorial HMM using many
fewer parameters are called mixed memory Markov models [142] . Viewed
as a GM as in Figure 25, this model uses an additional hidden variable for
each time frame and chain. Each normal hidden variables possesses an ad-
ditional switching parent (as depicted by dotted edges in the figure, and as
described in Section 2.2). The switching conditional independence assump-
tions for one time slice are that QtJLRt-liSt = 0, QtJLQt-liSt = 1 and
the symmetric relations for R t . This leads to the following distributional
simplification:
p(QtIQt-l, R t- 1 ) = p(QtIQt-l, S, = O)P(St = 0)

+ p(QtIRt-l' S, = l)P(St = 1)
which means that, rather than needing a single three-dimensional table
for the dependencies, only two two-dimensional tables are required. These
models have been used for ASR in [119] .
([)
I
I
@ I
I
CD I
I
G I
I
FIG. 25 . A mixed-memory hidden Markov model. The dashed edges indicate that
the S and the W nodes are switching parents.
A Buried Markov model (BMM) [16, 15, 14] is another recently pro-
posed GM-based approach to speech recognition . A BMM is based on the
idea that one can quantitatively measure where the conditional indepen-
dence properties of a particular HMM are poorly representing a corpus of
data. Wherever the model is found to be most lacking, additional edges
are added (i.e., conditional independence properties are removed) relative
to the original HMM. The BMM is formed to include only those data-
derived, sparse, hidden-variable specific, and discriminative dependencies
(between observation vectors) that are most lacking in the original model.
In general, the degree to which Xt-1JLXtIQt is true can be measured us-
ing conditional mutual information I(Xt-1;XtIQd [33]. If this quantity
is zero, the model needs no extension, but if it is greater than zero, there
is a modeling inaccuracy. Ideally, however, edges should be added dis-
criminatively, to produce a discriminative generative model, and when the
structure is formed discriminatively, the notion has been termed structural
discriminability [16, 47, 166,47]. For this purpose, the "EAR" (explaining
away residual) measure has been defined that measures the discriminative
mutual information between a variable X and its potential set of parents
Z as follows:
EAR(X, Z) ~ [(X ; ZIQ) - [(X; Z) .
It can be shown that choosing Z to optimize the EAR measure can be

equivalent to optimizing th e post erior probability of th e class Q [16] . Since
it attempts to minimally correct only thos e measured deficiencies in a par-
ticular HMM, and since it does so discriminatively, this approach has th e
potential to produce better performing and more parsimonious models for
speech recognition.
0,./- s., 0, - q,
F IG. 26. A Bu ried Markov Model (BMM) with two hidden Markov chain assign-
me nts, Ql :T = ql :T on the left, and Ql :T = q~ :T on the right.
It seems apparent at this point that the set of models that can be
described using a graph is enormous. With th e options th at are available
in choosing hidden variables, th e different sets of dependencies between
those hidden variables, t he dependencies between observations, choosing
switc hing dependencies, and considering the variety of different possible
implementations of thos e depend encies and the various learning techniques,
it is obvious that th e space of possible models is practically unlimited.
Moreover, each of thes e modeling possibilities , if seen outside of the GM
paradigm, requires a large software development effort before evaluation
is possible with a large ASR system. This effort must be spent without
having any guarantees as to the model's success.
In answer to these issues, a new flexible GM-bas ed software toolkit has
been developed (GMTK) [13]. GMTK is a graphical models toolkit that has
been optimized for ASR and oth er time-series processing tasks. It supports
EM and GEM parameter training, sparse linear and non-linear dependen-
cies between observations, arbit rary parameter sharing , Gaussian vanish-
ing and splitting, decision-tr ee implementations of dependenci es, sampling,
swit ching parent function ality, exact and log-space inference, multi-rate
and multi-stream processing, and a textual graph programming language.
The toolkit supports structural discriminability and arbitra ry model selec-
tion, and makes it much easier to begin to experiment with GM-based ASR
systems.
6. Conclusion. This paper has provided an introductory survey of
graphical models, and then has provided a number of examples of how
many existing ASR techniques can be viewed as instances of GMs. It is
hoped that this paper will help to fuel the use of GMs for further speech
recognition research. While the number of ASR models described in this
document is large, it is of course the case that many existing ASR tech-
niques have not even been given a mention. Nevertheless, it is apparent
that ASR collectively occupies a relatively minor portion of the space of
models representable by a graph. It therefore seems quite improbable that
a thorough exploration of the space of graphical models would not ulti-
mately yield a model th at performs better than the HMM. The search
for such a novel model should ideally occur on multiple fronts: on the
one hand guided by our high-level domain knowledge about speech and
thereby utilize phonetics, linguistics, psycho-acoustics , and so on. On the
other hand, the data should have a strong say, so there should be signifi-
cant data-driven model selection procedures to determine the appropriate
natural graph structure [10] . And since ASR is inherently an instance of
pattern classification, the notion of discriminability (parameter training)
and structural discriminability (structure learning) might playa key role
in this search. All in all, graphical models opens many doors to novel
speech recognition research.
REFERENCES
[I] A.V . AHO , R . SETHI, AND J.D . ULLMAN. Compilers : Principle s, Techniques and
Tools. Addison-Wesley, Inc., Reading, Mass ., 1986.
[2J S.M . AJI AND R.J . McELIECE. The generalized d istributive law. IEEE Transac-
tions in Information Theory, 46 :325-343, March 2000.
[3] T . ANASTASAKOS , J . McDONOUGH , R . SCHWARTZ , AND J MAKHOUL. A compact
model for speaker adaptive training. In Proc. Int. Conf. on Spoken Language
Processing, pp . 1137-1140, 1996.
[4] T .W. ANDERSON. An Introduction to Multivariate Statistical Analysis. Wiley
Series in Probability and Statistics, 1974.
[5] J .J. ATICK. Could information theory provide an ecological theory of sensory
processing? Network, 3 :213-251 , 1992.
[6] H. ATTIAS. Independent Factor Analysis. Neural Computation, 11(4) :803-851 ,
1999.
[71 A .J . BELL AND T .J. SEJNOWSKI. An information maximisation approach to blind
separation and blind deconvolution. Neural Computation, 7(6) :1129-1159,
1995.
[8J Y . BENGIO. Markovian models for sequential data. Neural Computing Surveys,
2:129-162, 1999.
[9J A. L. BERGER, S.A . DELLA PIETRA, AND V.J. DELLA PIETRA . A maximum
entropy approach to natural language processing. Computational Linguistics,
22(1) :39-71, 1996.
[lOJ J . BILMES. Natural Statistical Models for Automatic Speech Recognition. PhD
thesis, U.C. Berkeley, Dept. of EECS, CS Division, 1999.
[11] J . BILMES . What HMMs can do . Technical Report UWEETR-2002-003, Univer-

sity of Washington, Dept. of EE , 2002.
[12] J . BILMES , N. MORGAN, S.-L. Wu , AND H. BOURLARD. Stochastic perceptual
speech models with durational dependence. Int!. Conference on Spoken Lan-
guage Processing, November 1996.
[13] J . BILMES AND G . ZWEIG . The Graphical Models Toolkit: An open source soft-
ware system for speech and time-series processing. Proc. IEEE Int!. Conf.
on Acoustics, Speech, and Signal Processing, 2002.
[14] J .A. BILMES. Data-driven extensions to HMM statistical dependencies. In Proc.
Int. Conf. on Spoken Language Processing, Sidney , Australia, December
1998.
[15J J .A. BILMES . Buried Markov models for speech recognition. In Proc. IEEE Ititl.
Conf. on Acoustics, Speech, and Signal Processing, Phoenix, AZ, March
1999.
[16] J .A . BILMES. Dynamic Bayesian Multinets . In Proceedings of the 16th conf. on
Uncertainty in Artificial Intelligence . Morgan Kaufmann, 2000.
[17] J .A. BILMES. Factored sparse inverse covariance matrices. In Proc. IEEE Int!.
Conf. on Acoustics, Speech, and Signal Processing, Istanbul, Turkey, 2000.
[18] J .A . BILMES AND K . KIRCHHOFF . Directed graphical models of classifier com-
bination: Application to phone recognition. In Proc. Int. Conf. on Spoken
Language Processing, Beijing , China, 2000.
[19] C. BISHOP. Neural Network s for Patt ern Recognition. Clarendon Press, Oxford,
1995.
[20] H. BOURLARD . Personal communication, 1999.
[21J H . BOURLARD AND N. MORGAN . Connectionist Speech Recognition : A Hybrid
Approach. Kluwer Academic Publishers, 1994.
[22] L. BREIMAN, J .H. FRIEDMAN , R .A. OLSHEN , AND C.J . STONE. Classification and
Regression Trees. Wadsworth and Brooks, 1984.
[23] P .F . BROWN . The Acoustic Modeling Problem in Automatic Speech Recognition .
PhD thesis, Carnegie Mellon University, 1987.
[24] P.F . BROWN , V.J . DELLE PIETRA, P .V . DESOUZA , J .C . LAI, AND R .L. MERCER.
Class-based n-gram models of natural language. Computational Linguistics,
18(4):467-479, 1992.
[25] W . BUNTINE. A guide to the literature on learning probabilistic networks from
data. IEEE Trans. on Knowledge and Data Engineering, 8:195-210, 1994.
[26] K .P . BURNHAM AND D.R. ANDERSON . Model Selection and Inference : A Prac-
tical Information- Theoretic Approach. Springer-Verlag, 1998.
[27] R. CHELLAPPA AND A. JAIN, eds . Markov Random Fields: Theory and Applica-
tion . Academic Press, 1993.
[28] FRANCINE R . CHEN . Identification of contextual factors for pronunciation net-
works . Proc, IEEE Inil. Conf. on Acoustics, Speech, and Signal Processing,
pp . 753-756, 1990.
[29] S.F. CHEN AND J. GOODMAN . An empirical study of smoothing techniques for
language modeling. In Arivind Joshi and Martha Palmer, editors, Proceedings
of the Thirty-Fourth Annual Meeting of the Association for Computational
Linguistics, pp . 310-318, San Francisco, 1996. Association for Computational
Linguistics, Morgan Kaufmann Publishers.
[30] D.M . CHICKERING . Learning from Data: Artificial Intelligence and Statistics,
chapter Learning Bayesian networks is NP-complete, pp. 121-130. Springer-
Verlag, 1996.
[31] G. COOPER AND E . HERSKOVITS. Computational complexity of probabilistic in-
ference using Bayesian belief networks. Artificial Intelligence , 42:393-405,
1990.
[32] T .H. CORMEN , C.E. LEISERSON , AND R.L. RIVEST. Introduction to Algorithms.
McGr aw Hill, 1990.
[33] T .M. COVER AND J .A. THOMAS. Elements of Information Theory. Wiley, 1991.
[34] R.G . COWELL, A.P . DAWID , S.L. LAURITZEN , AND D.J. SPIEGELHALTER. Proba-
bilistic Networks and Expert Systems. Springer-Verlag, 1999.
[35J P . DAGUM AND M. LUBY. Approximating probabilistic inference in Bayesian
belief networks is NP-hard. Artificial Intelligence, 60(141-153), 1993.
[36J Data mining and knowledge discovery . Kluwer Academic Publishers. Maritime
Institute of Technology, Maryland.
[37] A.P . DAWID. Conditional independence in statistical theory. Journal of the Royal
Statistical Society B, 41(1) :1-31 , 1989.
[38J T. DEAN AND K. KANAZAWA. Probabilistic temporal reasoning. AAAI, pp . 524-
528, 1988.
[39] J.R. DELLER, J .G . PROAKIS, AND J .H.L. HANSEN. Discrete-time Processing of
Speech Signals. MacMillan, 1993.
[40J A.P. DEMPSTER, N.M. LAIRD, AND D.B . RUBIN. Maximum-likelihood from in-
complete data via the EM algorithm. J . Royal Statist. Soc. Ser . B., 39,
1977.
[41] L. DENG , M. AKSMANOVIC, D. SUN , AND J. Wu . Speech recognition using hidden
Markov models with polynomial regression functions as non-stationary states.
IEEE Trans. on Speech and Audio Proc., 2(4) :101-119, 1994.
[42] L. DENG AND C. RATHINAVELU . A Markov model containing state-conditioned
second-order non-stationarity: application to speech recognition. Computer
Speech and Language, 9(1) :63-86, January 1995.
[43J M. DEVIREN AND K. DAOUD/. Structure learning of dynamic bayesian networks
in speech recognition. In European Conf. on Speech Communication and
Technology {Eurospeecli}, 2001.
[44J R .O. DUDA , P.E. HART, AND D.G. STORK. Pattern Classification . John Wiley
and Sons, Inc., 2000.
[45J E . EIDE. Automatic modeling of pronunciation variations. In European Conf. on
Speech Communication and Technology (Eurospeech), 6th , 1999.
[46J K. ELENIUS AND M. BLOMBERG. Effects of emphasizing transitional or stationary
parts of the speech signal in a discrete utterance recognition system. In Proc.
IEEE Inti. Conf. on Acoustics, Speech, and Signal Processing, pp. 535-538,
1982.
[47] J . BILMES et al. Discriminatively structured graphical mod els for speech recog-
nition: JHU-WS-2001 final workshop report. Technical report, CLSP, Johns
Hopkins University, Baltimore MD, 2001.
[48] J .G . FISCUS. A post-processing system to yield reduced word error rates: Rec-
ognizer output voting error reduction (ROVER) . In Proceedings of IEEE
Workshop on Automatic Speech Recognition and Understanding, Santa Bar-
bara, California, 1997.
[49J R .A . FISHER. The use of multiple measurements in taxonomic problems. Ann.
Eugen ., 7:179-188, 1936.
[50J R. FLETCHER. Practical Methods of Optimization. John Wiley & Sons , New
York, NY, 1980.
[51] V . FONTAINE, C. RIS, AND J.M .BOITE. Nonlinear discriminant analysis for im-
proved speech recognition. In European Conf. on Speech Communication
and Technology (Eurospeech), 5th, pp . 2071-2074 , 1997.
[52J E . FOSLER-LusSIER. Dynamic Pronunciation Models for Automatic Speech
Recognition . PhD thesis, University of California, Berkeley., 1999.
[53] B . FREY. Graphical Models for Machine Learning and Digital Communication.
MIT Press, 1998.
[54J J.H . FRIEDMAN. Multivariate adaptive regression splines. The Annals of Statis-
tics, 19(1):1-141, 1991.
[55J N. FRIEDMAN AND M. GOLDSZMIDT. Learning in Graphical Models, chapter
Learning Bayesian Networks with Local Structure. Kluwer Academic Pub-
lishers, 1998.
GRAPHICAL MODELS FO R ASR 241
[56] N. FRIEDMAN , K. MURPHY , AND S. RUSSELL. Learning the st ructure of dynamic

probabilistic networks. 14th Conf. on Uncertainty in Artificial Intelligence,
1998.
[57] K . F UKUNAGA . Introduct ion to Statistical Pattern Recognition , 2nd Ed. Academic
Press, 1990.
[58] S. FURUI. Cepstral analysis technique for automatic speaker verification. IEEE
Transa ctions on Acoustics, Speech, and Signal Processing, 29(2) :254-272 ,
April 1981.
[59] S. F URUI. Speaker-independent isolated word recogni tion using dyn amic features
of speech spectrum. IEEE Transa ctions on Acoustics, Speech, and Signal
Processing, 34(1):52-59, February 1986.
[60J S. FURUI. On the role of spectral transition for sp eech perception. Journal of the
Acoustical Society of America, 80(4):1016- 1025, October 1986.
[61] M.J .F . GALES AND S. YOUNG . An improved approach to the hidden Markov
model decomposition of speech and noise. In Proc. IEEE Int!. Conf. on
Acoustics, Speech, and Signal Processing, pp . 1-233-236, 1992.
[62] M.J .F . GALES. Semi-tied covariance matrices for hidden Markov models. IEEE
Transactions on Speech and Audio Processing, 7 (3):272- 281, May 1999.
[63] M .J .F. GALES AND S.J . YOUNG . Segmental hidden Markov models. In Euro-
pean Conf. on Speech Communication and Technology (Eurospeech), 3rd,
pp . 1579- 1582, 1993.
[64J M.J .F . GALES AND S.J . YOUNG . Robust speech recognition in additive and con-
volutional noise using parallel model combination. Computer Speech and
Language, 9 :289-307, 1995.
[65J D. GEIGER AND D. HECKERMAN . Knowledge representation and inference in
similarity networks and Bayesian multinets. Artificial Intellig ence, 8 2 :45-
74, 1996.
[66] Z. GHAHRAMANI. Lecture Notes in Artificial Intelligence, Chapter Learning Dy-
namic Bayesi an Networks. Springer-Verlag, 1998.
[67] Z. GHAHRAMANI AND M. JORDAN. Factorial hidden Markov models. Machin e
Learning, 29 , 1997.
[68] G .H. GOLUB AND C .F. VAN LOAN. Matrix Computations. Johns Hopkins, 1996.
[69] R.M. GRAY AND A. GERSHO . Vector Quant ization and Signal Compression.
Kluwer , 1991.
[70J M.S. GREWAL AND A.P . ANDREWS. Kalman Filtering : Th eory and Pract ice.
Prentice Hall , 1993.
[71] X.F . Guo , W .B. ZHU, Q . Sill , S. CHEN , AND R . GOPINATH. The IBM LVCSR
system used for 1998 mandarin broadcast news transcription evaluation. In
Th e 1999 DARPA Broadcast News Workshop , 1999.
[72J A.K. HALBERSTADT AND J .R . GLASS . Heterogeneous measurements a nd multiple
classifiers for speech recognition . In Proc. Int. Conf. on Spoken Language
Processing, pp . 995-998, 1998.
[73J D.A . HARVILLE. Matrix Algebm from a Statistician's Perspectiv e. Spri nger-
Verlag, 1997.
[74] T . HASTIE AND R . TIBS111RANI. Discriminant analysis by Gaussian mixtures.
Journal of the Royal Statistical Society series B, 58: 158-176, 1996.
[75] D . HECKERMAN . A tutorial on learning with Bayesian networks. Technical Report
MSR-TR-95-06, Microsoft, 1995.
[76] D. HECKERMAN , MAX CHICKERING, CHRIS MEEK , ROBERT ROUNTHWAITE, AND
CARL KADIE. Dependency networks for density estimation , collaborative fil-
te ring, and data visu alization. In Proceedings of the 16th conf. on Uncertainty
in Artificial Intellig ence. Morgan Kaufmann, 2000.
[77] D. HECKERMAN , D . GEIGER, AND D.M . C111CKERING. Learning Bayesian net-
works : The combination of knowledge and statistic al dat a. Techn ical Report
MSR-T R-94-09, Microsoft , 1994.
[78] H. HERMANSKY, D. ELLIS, AND S. SHARMA. Tandem conn ectionist feature st ream
extraction for conventional HMM systems. InProc. IEEE Inti. Conf. on
Acoustics, Speech, and Signal Processing , Istanbul, Turkey, 2000.
[791 J . HERTZ , A. KROGH , AND R.G . PALMER. Introduction to the Th eory of Neural
Computation. Allan M. Wylde, 1991.
[80] X.D . HUANG , A. ACERO, AND H.-W . HON . Spok en Languag e Processing: A
Guide to Theory, Algorithm, and System Development. Prentice Hall, 200l.
[81] T .S. JAAKKOLA AND M.l. JORDAN. Learning in Graphical Models, chapter Im-
proving the Mean Field Approximations via the use of Mixture Distributions.
Kluwer Academic Publishers, 1998.
[82] R .A. JACOBS . Methods for combining experts' probability assessments. Neural
Computation, 1 :867-888, 1995.
[83] F . JELINEK. Statistical Methods for Speech Recognition. MIT Press, 1997.
[84] F.V . JENSEN . An Introduction to Bayesian Networks. Springer-Verlag, 1996.
[85J M.l. JORDAN AND C.M. BISHOP, eds . An Introduction to Graphical Models . to
be published, 200x.
[861 M.l. JORDAN , Z. GHAHRAMANI, T .S. JAAKKOLA , AND L.K. SAUL. Learning
in Graphi cal Models, chapter An Introduction to Variational Methods for
Graphical Models. Kluwer Academic Publishers, 1998.
[87] M.I . JORDAN AND R . JACOBS. Hierarchical mixtures of experts and the EM
algorithm. Neural Computation, 6:181-214, 1994.
[88J B.-H . JUANG AND L.R. RABINER. Mixture autoregressive hidden Markov models
for speech signals. IEEE Trans. Acoustics, Speech, and Signal Processing,
33(6):1404-1413, December 1985.
[89J D. JURAFSKY AND J .H . MARTIN . Speech and Language Processing. Prentice Hall ,
2000.
[90] M. KADIRKAMANATHAN AND A.P . VARGA. Simultaneous model re-estimation from
contaminated data by composed hidd en Markov modeling. In Proc. IEEE
Inti . Conf. on Acoustics, Speech, and Signal Processing, pp. 897-900, 1991.
[91J T . KAMM , G. ANDREOU, AND J . COHEN. Vocal tract normalization in speech
recognition comp ensating for systematic speaker variability. In Proc. of the
15th Annual speech research symposium, pp . 175-178. CLSP, Johns Hopkins
Univers ity, 1995.
[92] J . KARHUNEN. Neural approaches to independent component analysis and source
separation . In Proc 4th European Symposium on Artificial Neural Networks
(ESANN '96), 1996.
[93J P. KENNY, M. LENNIG , AND P . MERMELSTEIN . A linear predictive HMM for
vector-valued observations with applications to speech recognition. IEEE
Transactions on Acoustics, Speech, and Signal Processing , 38(2) :220-225,
February 1990.
[94J B.E .D . KINGSBURY AND N. MORGAN. Recognizing reverberant speech with
RASTA-PLP. Proceedings ICASSP-97, 1997.
[95] K . KIRCHHOFF. Combining acoustic and articulatory information for speech
recognition in noisy and reverberant environments. In Proceedings of the
International Conference on Spoken Language Processing , 1998.
[96] K. KIRCHHOFF AND J . BILMES. Dynamic classifier combination in hybrid spee ch
recognition systems using utterance-level confidence values . Proceedings
ICASSP-99, pp . 693-696, 1999.
[97J J. KITTLER, M. HATAF , R.P.W. DUIN , AND J. MATAS. On combining classi-
fiers. IEEE Transactions on Pattern Analysis and Machine Intelligence,
20(3) :226-239, 1998.
[98J K. KJAERULFF. Tri angulation of graphs - algorithms giving small total space.
Technical Report R90-09, Department of Mathematics and Computer Sci-
ence. Aalborg University, 1990.
[99] P . KRAUSE. Learning probabilistic networks. Philips Research Labs Tech. Report,
1998.
[100] A. KROGH AND J. VEDELSBY . Neur al network ensembles, cross validation, and
acti ve learning. In Advan ces in N eural Information Processing Systems 7.
MIT Press, 1995.
[101] F.R. KSCHISCHANG, B. FREY , AND H.-A . LOELIGER. Fact or graphs and the sum-
product algorithm. IEEE fun s. Inform. Th eory, 47(2):498-519, 200 l.
[102] N. KUMAR. In vestigation of Si licon Auditory Model s and Generalization of Lin-
ear Discriminant Analysis fo r Imp roved Speech Recognitio n. PhD thesis,
Johns Hopkins University, 1997.
[103] S.L. LAURITZEN. Graph ical Mod els. Oxford Science Publications , 1996.
[104] C.J. LEGGETTER AND P. C. WOODLAND . Max imum likelihood linear regression for
speaker ad aptation of conti nuous density hidden Marko v models . Com pu ter
Speech and Lang uage, 9:171-1 85, 1995.
[105] E . LEVIN . Word recognition using hidden cont rol neur al architecture. In Proc.
IEEE Int! . Conf. on A cousti cs, Speech, and Signal Processing, pp . 433-436.
IEEE, 1990.
[106J E . LEVIN. Hidden control neural architecture modeling of nonlin ear time varying
systems and its applicat ions. IEEE funs . on N eural N etworks, 4(1):109-
116, January 1992.
[107] H. LINHART AND W . ZUCCHINI. Model Selection. Wiley, 1986.
[108] B .T . LOGAN AND P.J . MORENO. Factorial HMMs for acoust ic mod eling. Proc.
IEEE Intl . Conf. on A coustics, Speech, and Signal Processing , 1998.
[109J D.J .C. MACKAY. Learn ing in Graphical Models , chapter Introduction to Monte
Carlo Methods. Kluwer Acad emic Publishers, 1998.
[110J J . MAKHOUL. Linear predict ion: A tutorial review. Proc . IEEE, 63 :561-580,
April 1975.
[111] K.V. MARDlA , J .T . KENT, AND J.M. BIBBY. Mult ivari ate Analysis. Academic
Press, 1979.
[112] G.J . McLACHLAN. Fin ite Mixture Model s. Wiley Series in Probability and Statis-
tics , 2000.
[113] G .J . McLACHLA N AND T . KRISHNAN. Th e EM A lgori thm and Extensions . W iley
Series in Probabi lity and St atistics, 1997.
[114] C. MEEK. Ca usa l inference and causa l explanat ion with background knowledge.
In Besnard, Philippe and Ste ve Hanks, editors , Proceeding s of the 11th Con-
ference on Uncert ainty in Artific ial Intelligen ce (U A I'95), pp. 403-410, San
Francisco, CA, USA, Augus t 1995. Morgan Kaufmann Publishers.
[115] M. MElLA. Learning wi th MixtUTeS of Tree s. PhD t hesis, MIT, 1999.
[116] M. MOHRI , F .C.N. PEREIRA , AND M. RILEY. The design pr inciples of a weighted
finite-state t ransducer library. Th eoretical Com pute r Scie nce , 231 (1) :17-32,
2000.
[117] N. MORGAN AND B. GOLD. Speech and Audio Signal Processing. John Wiley and
Sons, 1999.
[118] H. NEY, U . ESSEN, AND R. KNESER. On structuring probabilistic dependencies
in stochastic language modelling. Computer Speech and Language, 8 :1-38,
1994.
[119] H.J . NOCK AND S.J . YOUNG . Loosely-coupled HMMs for ASR . In Proc . Int.
Conf. on Spok en Language Processing, Beijing , China , 2000.
[120] M. OSTENDORF , V. DIGALAKIS , AND O.KIMBALL. From HMM 's to segment
models : A unified view of st ochastic modeling for speech recognition. IEEE
Trans . Speech and Audio Proc., 4 (5), Sept emb er 1996.
[121] M. OSTENDORF , A. KANNAN , O. KIMBALL, AND J . ROHLICEK . Cont inuous word
recognition based on th e st ochast ic segment model. Proc. DARPA Workshop
CS R , 1992.
[122] J. PEARL. Probabilistic Reasoning in Intelligent S yst em s: N etwo rks of Plaus ible
Inferen ce. Morgan Kaufmann, 2nd printing editi on , 1988.
[123J J . PEARL. Caus ality . Ca mbridge, 2000.
[124] M.P . PERRONE AND L.N . COOPER. When networks disagree: ensemble methods
for hybrid neural networks. In RJ . Mammone, editor, N eural Networks for
Speech and Image Processing, page Chapter 10, 1993.
[125] J . PICONE, S. PIKE, R REGAN, T . KAMM, J. BRIDLE, L. DENG , Z. MA,
H. RICHARDS , and M. Schuster. Initial evaluation of hidden dynamic models
on conversational speech . In Proc . IEEE Inti. Conf. on Acoustics, Speech,
and Signal Processing, 1999.
[126] S.D . PIETRA, V.D . PIETRA , AND J . LAFFERTY. Inducing features of random fields.
Technical Report CMU-CS-95-144, CMU, May 1995.
[127J A.B. PORITZ. Linear predictive hidden Markov models and the speech signal.
Proc . IEEE Intl. Conf. on Acoustics, Speech, and Signal Processing, pp .
1291-1294, 1982.
[128] A.B. PORlTZ. Hidden Markov models : A guided tour. Proc. IEEE Inil. Conf.
on Acoustics, Speech, and Signal Processing, pp . 7-13 , 1988.
[129] L.R . RABINER AND B.-H . JUANG . Fundamentals of Speech Recognition. Prentice
Hall Signal Processing Series, 1993.
[130] L.R RABINER AND B.H. JUANG . An introduction to hidden Markov models .
IEEE ASSP Magazine, 1986.
[131] M. RICHARDSON, J . BILMES, AND C. DIORIO. Hidden-articulator markov models
for speech recognition. In Proc. of the ISCA ITRW ASR2000 Workshop ,
Paris, France, 2000. LIMSI-CNRS.
[132J M. RICHARDSON, J . BILMES, AND C. DIORIO. Hidden-articulator markov models :
Performance improvements and robustness to noise. In Proc . Int. Conf. on
Spoken Language Processing, Beijing, China, 2000.
[133] T .S. RICHARDSON . Learning in Graphical Models, chapter Chain Graphs and
Symmetric Associations. Kluwer Academic Publishers, 1998.
[134] M.D . RILEY. A statistical model for generating pronunciation networks. Proc .
IEEE Int!. Conf. on Acoustics, Speech, and Signal Processing, pp . 737-740,
1991.
[135] R .T. ROCKAFELLAR. Convex Analysis. Princeton, 1970.
[136] R ROSENFELD. Adaptive Statistical Language Modeling: A Maximum Entropy
Approach. PhD thesis, School of Computer Science, CMU, Pittsburgh, PA,
April 1994.
[137J R ROSENFELD. Two decades of statistical language modeling: Where do we go
from here? Proceedings of the IEEE, 88(8) , 2000.
[138] R ROSENFELD, S.F . CHEN, AND X. ZHU. Whole-sentence exponential language
models: a vehicle for linguistic-statistical integration. Computer Speech and
Language, 15(1), 200l.
[139] D.B . ROWE. Multivariate Bayesian Statistics: Models for Source Separation adn
Signal Unmixing. CRC Press, Boca Raton, FL, 2002.
[140] S. ROWElS AND Z. GHAHRAMANI. A unifying review of linear gaussian models.
Neural Computation, 11 :305-345 , 1999.
[141] L.K . SAUL, T. JAAKKOLA , AND M.1. JORDAN. Mean field theory for sigmoid belief
networks. lAIR, 4 :61-76 , 1996.
[142J L.K. SAUL AND M.1. JORDAN . Mixed memory markov models: Decomposing
complex stochastic processes as mixtures of simpler ones. Machine Learning,
1999.
[143] RD . SHACHTER. Bayes-ball: The rational pastime for determining irrelevance
and requisite information in belief networks and influence diagrams. In Un-
certainty in Artificial Intelligence, 1998.
[144J P . SMYTH , D. HECKERMAN, AND M.I . JORDAN . Probabilistic independence net-
works for hidden Markov probability models. Technical Report A.I. Memo
No. 1565, C.B.C.L . Memo No. 132, MIT AI Lab and CBCL, 1996.
[145] T . STEPHENSON, H. BOURLARD , S. BENGIO , AND A. MORRIS. Automatic speech
recognition using dynamic bayesian networks with both acoustic and artie-
ulatory variables. In Proc . Int . Conf. on Spoken Language Processing, pp .

951-954, Beijing, China, 2000.
[146] G . STRANG. Linear Algebra and its applications, 3rd Edition. Saunders College
Publishing, 1988.
[147J M .E . TIPPING AND C.M . BISHOP. Probabilistic principal component analysis.
Journal of the Royal Statistical Society, Series B , 61(3) :611-622, 1999.
[148J D .M . TITTERINGTON , A.F .M . SMITH , AND U.E . MAKOV. Statistical Analysis of
Finite Mixture Distributions. John Wiley and Sons , 1985.
[149] H. TONG . Non-linear Time Series : A Dynamical System Approach. Oxford
Statistical Science Series 6. Oxford University Press, 1990.
[150] V. VAPNIK . Statistical Learning Theory. Wiley, 1998.
[151] A.P . VARGA AND R.K MOORE. Hidden Markov model decomposition of speech
and noise . In Proc. IEEE Int!. Conf. on Acoustics, Speech, and Signal Pro-
cessing, pp. 845-848, Alburquerque, April 1990.
[152J A.P . VARGA AND R.K. MOORE. Simultaneous recognition of concurrent speech
signals using hidden makov model decomposition. In European Conf. on
Speech Communication and Technology (Eurospeech), 2nd, 1991.
[153] T. VERMA AND J. PEARL. Equivalence and synthesis of causal models. In Un-
certainty in Artificial Intelligence. Morgan Kaufmann, 1990.
[154J T . VERMA AND J. PEARL. An algorithm for deciding if a set of observed indepen-
dencies has a causal explanation. In Uncertainty in Artificial Intelligence.
Morgan Kaufmann, 1992.
[155] M.Q. WANG AND S.J. YOUNG . Speech recognition using hidden Markov model
decomposition and a general background speech model. In Proc . IEEE Inti.
Conf. on Acoustics, Speech, and Signal Processing, pp. 1-253-256, 1992.
[156] Y . WEISS. Correctness of local probability propagation in graphical models with
loops . Neural Computation, 12(1):1-41, 2000.
[157] C.J . WELLEKENS. Explicit time correlation in hidden Markov models for speech
recognition. Proc . IEEE Int!. Conf. on Acoustics, Speech , and Signal Pro-
cessing, pp . 384-386, 1987.
[158] C .J . WELLEKENS. Personal communication, 2001.
[159] J . WHITTAKER. Graphical Models in Applied Multivariate Statistics. John Wiley
and Son Ltd ., 1990.
[160] D.H. WOLPERT. Stacked generalization. Neural Networks, 5 :241-259, 1992.
[161] P .C . WOODLAND . Optimizing hidden Markov models using discriminative output
distributions. In Proc , IEEE Int!. Conf. on Acoustics, Speech , and Signal
Processing, 1991.
[162] P .C . WOODLAND . Hidden Markov models using vector linear prediction and
discriminative output distributions. In Proc. IEEE Int! . Con]. on Acoustics,
Speech, and Signal Processing, pp . 1-509-512, 1992.
[163J Su-LIN Wu , MICHAEL L. SHIRE, STEVEN GREENBERG, AND NELSON MORGAN .
Integrating syllable boundary information into speech recognition. In Proc .
IEEE Inil. Con]. on Acoustics, Speech , and Signal Processing, Vol. 1, Mu-
nich, Germany, April 1997. IEEE.
[164] S . YOUNG . A review of large-vocabulary continuous-speech recognition. IEEE
Signal Processing Magazine, 13(5):45-56, September 1996.
[165] KH . Yuo AND H.C . WANG. Joint estimation of feature transformation parame-
ters and gaussian mixture model for speaker identification. Speech Commu-
nications, 3(1) , 1999.
[166] G . ZWEIG , J . BILMES , T . RICHARDSON , K FILALI, K LIVESCU, P . XU, K JACK-
SON , Y. BRANDMAN, E . SANDNESS , E . HOLTZ , J . TORRES , AND B . BYRNE.
Structurally discriminative graphical models for automatic speech recogni-
tion - results from the 2001 Johns Hopkins summer workshop. Proc. IEEE
Int!. Con]. on Acoustics, Speech , and Signal Processing, 2002.
[167] G . ZWEIG AND S. RUSSELL. Speech recognition with dynamic Bayesian networks.
AAAI-98, 1998.
AN INTRODUCTION TO MARKOV CHAIN
MONTE CARLO METHODS
JULIAN BESAG·
Abstract. This article provides an introduction to Markov chain Monte Carlo

methods in statistical inference. Over the past twelve years or so, these have revolu-
tionized what can be achieved computationally, especially in the Bayesian paradigm.
Markov chain Monte Carlo has exactly the same goals as ordinary Monte Carlo and
both are intended to exploit the fact that one can learn about a complex probability
distribution if one can sample from it . Although the ordinary version can only rarely
be implemented, it is convenient initially to presume otherwise and to focus on the ra-
tion ale of the sampling approach, rather than computational details. The article then
moves on to describe implementation via Markov chains, especially the Hastings algo-
rithm, including the Metropolis method and the Gibbs sampler as special cases . Hidden
Markov models and the autologistic distribution receive some emphasis, with the noisy
binary channel used in some toy examples . A brief description of perfect simul at ion is
also given . The account concludes with some discussion .
Key words. Autologistic distribution; Bayesian computation; Gibbs sampler; Hast-

ings algorithm; Hidden Markov models ; Importance sampling; Ising model; Markov
chain Monte Carlo; Markov random fields; Maximum likelihood estimat ion; Met ropolis
method; Noisy binary channel; Perfect simulation; Reversibility; Simulated annealing.
1. The computational challenge.

1.1. Introduction. Markov chain Monte Carlo (MCMC) methods
have had a profound influence on computational statistics over the past
twelve years or so, especially in the Bayesian paradigm. The intention
here is to cover the basic ideas and to provide references to some more
specialized topics . Other descriptions include the books by (or edited by)
Fishman (1996), Gilks et al. (1996), Newman and Barkema (1999), Robert
and Casella (1999), Chen et al. (2000), Doucet et al. (2001), Liu (2001)
and MacCormick (2002). Although none of these addresses speech per se,
the last three include descriptions of sequential Monte Carlo methods and
particle filters, with applications to on-line signal processing and target
tracking, for example. These books may therefore be of particular interest
to readers of this volume.
In the remainder of this section, we introduce the basic computational
task in MCMC . In Section 2, we discuss ordinary Monte Carlo meth-
ods and their conceptual relevance to Bayesian inference, especially hid-
den Markov models, to maximum likelihood estimation and to function
optimization. Unfortunately, ordinary Monte Carlo is rarely practicable
for high-dimensional problems, even for minor enhancements of hidden
Markov models. However, the underlying ideas transfer quite smoothly
to MCMC, with random samples replaced by dependent sampl es from a
• Department of Statistics, University of Washington, Box 354322, Seattle, WA 98195,

USA (julian@stat.washington.edu) .
247
248 JULIAN BESAG
Markov chain, as we discuss in Section 3. We also describe the Hastings

algorithm, including the Metropolis method and the Gibbs sampler, and
perfect MCMC simulation via monotone coupling from the past. The pa-
per includes a few toy examples based on the noisy binary channel. Finally,
Section 4 provides some discussion. The paper is mostly a distillation of
Besag (2001), where some applications and more specialized topics, such
as MCMC p-values (Besag and Clifford; 1989, 1991), cluster algorithms
(Swendsen and Wang, 1987), Langevin-Hastings algorithms (Besag, 1994a)
and reversible jump MCMC (Green, 1995), can be found. For the most re-
cent developments in a rapidly expanding field, the interested reader should
consult the MCMC website at
http ://www.statslab .cam .ac.ukj~mcmc/
1.2. The main task. Let X denote a random quantity: in practice,

X will have many components and might represent, for example, a random
vector or a multi-way contingency table or a grey-level pixel image (per-
haps augmented by other variables) . Also, some components of X might
be discrete and others continuous . However, it is most convenient for the
moment to think of X as a single random variable (r.v.), having a finite
but huge sample space. Indeed, in a sense, such a formulation is perfectly
general because ultimately all our calculations are made on a finite ma-
chine. It is only in considering specific MCMC algorithms, such as the
Gibbs sampler, or applications, such as hidden Markov models, that we
need to address the individual components of X.
Thus, let {7r(x) : XES} denote the probability mass function (p.m.f.)
of X, where S is the corresponding support of X ; that is, S = {x : 7r(x) >
O} . We assume that 7r(x) is known up to scale, so that
(1) 7r(X) = h(x)jc, XES,
where h(x) is completely specified, but that the normalizing constant
(2) c=Lh(x)
xES
is not known in closed form and that S is too unwieldy for c to be found
numerically from the sum in (2). Nevertheless, our goal is to compute
expectations of particular functions 9 under 7r; that is, we require
(3) Eng = L g(X)7r(X),

xES
for any relevant g, where again the summation in equation (3) cannot be
evaluated directly.
As an especially important special case, note that (3) includes the
probability of any particular event concerning X . Explicitly, for any relevant
subset B of S,
MARKOV CHAIN MONTE CARLO METHODS 249
(4) Pr(X E B) =L l[x E B]n(x),

xES
where 1[. ] is the usual indicator function; that is, 1 [x E BJ = 1 if the

outcome x implies that the event B occurs and 1 [x E BJ = 0 otherwise.
Indeed, one of the major strengths of MCMC is that it can focus directly
on probabilities, in contrast to the usual tradition in statistics of indirect
calculations based on large sample asymptotics.
2. Ordinary Monte Carlo calculations.
2.1. Monte Carlo estimation. Suppose that, despite the complex-
ity of 5, we can generate random draws from the target p.m.f. n(x). If we
produce m such draws, x(l), . . . , x(m), then the natural estimate of Eng is
the empirical mean,
(5)
This is unbiased for Eng and its sampling variance can be assessed in the
usual way.
Thinking ahead, we remark that (5) may provide an approximation
to Eng even when (x(l), .. . ,x(m)) is not a random sample from tt . In
particular, this occurs when m is sufficiently large and x(l) , x(2) , . . ., seeded
by some x(O) E 5, are successive observations from a Markov chain with
(finite) state space 5 and limiting distribution n . This extension provides
the basis of MCMC when random sampling from n is no longer feasible.
It requires that useful general recipes exist for constructing appropriate
Markov chains, as in Section 3.
2.2. Bayesian computation. For a description of parametric Bayes-
ian inference, see e.g. Gelman et al. (1995). Here we begin in the simplest
possible context. Thus, let x now denote a meaningful constant whose value
we wish to estimate. Suppose we know that this parameterx lies in a finite
space 5 and that our initial beliefs about its value can be represented by
a prior p.m.f. {p(x) : X E 5}. If y denotes relevant discrete data, then the
probability of y given x, viewed as a function of x, is called the likelihood
L(ylx) . In the Bayesian paradigm, the prior information and the likelihood
are combined via Bayes theorem to produce the posterior p.m.f.
(6) 1T(xIY) <X L(Ylx)p(x), x E 5,
for x given y . All inferences are based on (6) and require the evaluation
of corresponding expectations. In terms of (1), (2), (3) and (4), we replace
1T(X) by 1T(xIY) and let h(x) be proportional to L(Ylx)p(x). Note that
L(ylx) and p(x) need be known only up to scale.
Unfortunately, it is rarely possible to calculate the expectations di-
rectly. A potential remedy is to generate a large random sample x(l), ... ,
250 JULIAN BESAG
x(m) from the p.m.f. (6) and then use (5) to approximate the correspond-
ing posterior mean and variance, to evaluate posterior probabilities about
x and to construct corresponding credible intervals . In principle, this ap-
proach extends to multicomponent parameters but it is generally impossible
to implement because of the difficulty of sampling directly from complex
multivariate p.m.I, 's. This hindered applied Bayesian inference until it was
recognized that ordinary Monte Carlo could be replaced by MCMC.
As an example of how the availability of random samples from 7r(xly)
would permit trivial solutions to ostensibly very complicated problems,
consider a clinical, industrial or agricultural trial in which the aim is to
compare different treatment effects (}i. Then x = ((},</», where () is the vec-
tor of (}i'S and </> is a vector of other, possibly uninteresting, parameters in
the posterior distribution. A natural quantity of interest from a Bayesian
perspective is the posterior probability that any particular treatment effect
is best or is among the best three, say, where here we suppose best to mean
having the largest effect. Such calculations are usually far beyond the capa-
bilities of conventional numerical methods, because they involve sums (or
integrals) of non-standard functions over awkward regions of the parameter
space S. However, in a sampling approach, we can closely approximate the
probability that treatment i is best, simply by the proportion of simulated
(}(t)'s among which (}~t) is the largest component; and the probability that
treatment i is one of the best three by the proportion of (}(t)'s for which
e?) is one of the largest three components. Note that the values obtained
for components that are not of immediate interest are simply ignored and
that this procedure is entirely rigorous.
Ranking and selection is just one area in which the availability of
random samples from posterior distributions would have had a profound
influence on applied Bayesian inference. Not only does MCMC deliver
what is beyond ordinary Monte Carlo methods but also it encourages the
investigator to build and analyze more realistic statistical models. Indeed,
one must sometimes resist the temptation to build representations whose
complexity cannot be justified by the underlying scientific problem or by
the available data.
2.2.1. Hidden Markov models. Basic hidden Markov models
(HMM's) are among the most complex formulations for which ordinary
Monte Carlo can be implemented. In addition to their popularity in speech
recognition (e.g. Rabiner, 1989; Juang and Rabiner, 1991), HMM's also
occur in neurophysiology (e.g. Fredkin and Rice, 1992), in computational
biology (e.g. Haussler et al., 1993; Eddie et al., 1995; Liu et al., 1995), in cli-
matology (e.g. Hughes et al., 1999), in epidemiologic surveillance (Le Strat
and Carrat, 1999), and elsewhere. For a useful survey, see MacDonald and
Zucchini (1997). Here, we briefly describe HMM's and the corresponding
role of the Baum et al. (1970) algorithm.
Let Xl ,X2, .. ., where Xi E {O,1, . .. ,s}, denote successive states of a

process . These states are unknown but, for each i = 1, .. . , n independently,
Xi generates an observation Yi with probability f(Xi' Yi), so that the data
Y ~ (Yl, . . . , Yn) has probability
n
(7) L(ylx) II f(Xi, Yi) ,
i=l
given x . Now suppose Xl, X2, .. . is (modeled as) the output of a Markov
chain , with transition probability q(Xi' xHI) of the it h component Xi being
followed by Xi+!, so that the prior probability for X = (Xl, .. . , Xn ) is
n-l
(8) p(x) = q(Xl) II q(Xi ,Xi+!) , XES = {O,1 , . . . ,s}n ,
i=l
where q(.) is the p.m.f. of Xl . Then the post erior probability of x, given Y,
is
n
(9) 1r(xly) ex: q(XI)f(Xl,yI) II q(Xi-l,Xi)f(Xi,Yi), xES.
i=2
The goal is to make inferences about the true X and perhaps about some
future Xi'S. For example, we might require the marginal posterior modes
(MPM) estimate x* of x, defined by
(10) X; = argmax 1r(Xily)'

Xi
where 1r(Xijy) is the posterior marginal p.m.f, for Xi. Note here that x*
is generally distinct from the maximum a posteriori (MAP) estimate X,
defined by
(11) x= argmax 1r(xly),

X
as we discuss later for the noisy binary channel.

We remark in passing that, if the above Markov formulation provides
a physically correct model, then there is nothing intrinsically Bayesian in
the use of Bayes theorem to obtain the p.m.f. (9). Furthermore, if the
three p.mJ.'s on the right-hand side are known, then the Baum algorithm
can be used directly to evaluate many expectations (3) of interest, with-
out any need for simulation; and, if the p.mJ.'s are unknown, then Baum
can be preceded by the Baum-Welch algorithm, which determines their
nonparametric maximum likelihood estimates. Thus , it may seem that a
Bayesian formulation and a sampling approach to the analysis of HMM's
are both somewhat irrelevant. However, the Bayesian paradigm has dis-
tinct advantages when, as is usual, the HMM is more a conceptual rather
252 JULIAN BESAG
than a physical model of the underlying process; and more important here,
a sampling approach is viable under almost any modification of the basic
HMM formulation, but with ordinary Monte Carlo replaced by MCMC,
for which the Baum algorithm retains its relevance if an HMM lives within
the overall formulation. We comment further at the end of Section 3.7; see
also Robert et al. (2000).
The Baum et al. (1970) recursions for (9) exploit the fact that 1r(x) ==
1r(xly) inherits th e Markov property, though its transition probabilities are
functions of Y and therefore nonhomogeneous. Specifically,
n
(12) 1r(xly) = 1r(xIly) II 1r(xilxi-l'Y~ i)'
i=2
where Y~i = (Yi , . .. ,Yn), a form of notation we use freely below. The
factorization (12) provides access to the calculation of expected values and
to sampling from {1r(xly) : XES}, except that the conditional probabilities
in (12) are not immediately available. It can easily be shown that
n
(13) 1r(x~ilxi-l,Y~i) oc II q(Xk-l ,Xk)!(Xk,Yk), i = 2, . .. ,n ,
k=i
which determines 1r(xilxi-l, Y~i) by summing over X>i but the summations
are impracticable unless n is tiny. The Baum algorithm avoids the problem
by using the results,
(14) 1r(xIly)oc !(Xl,yI)q(xI)Pr(Y>llxI) ,

(15) 1r(XdXi-l,Y~i) oc !(Xi,Yi)q(Xi-l,xdPr(Y>i!xi), i=2 , . .. ,n.
Here, Pr(Y>nlxn) == 1 and the other Pr(Y>i!xi)'S for Xi = 0,1 , . . . , s can be

evaluated successively for i = n-1 , . . . , 1 from the backward recursion,
(16) Pr(Y>il xi) = L Pr(Y>HdxHI) !(Xi+l, YHd q(Xi' xH d .

X i.+l
Then (14) and (15) are used forwards to calculate expectations or to sample
from 1r(xly). Some care is needed in using (16) because the probabilities
quickly become vanishingly small. However, as they are required only up
to scale in (14) and (15), a dummy normalization can be carried out at
each stage to remedy the problem .
Example. Noisy binary channel. The noisy binary channel provides a
convenient numerical illustration of sampling via the Baum algorithm. We
additionally impose symmetry and st ationarity, merely to ease the notation.
The noisy binary channel is not only the simplest HMM but is also a rather
special case of the autologistic distribution, which we consider in Section
3.4 and which is not generally amenable to the Baum algorithm.
Thus, suppose that both the hidden Xi'S and the observed Yi'S are
binary and that the posterior probability (9) of a true signal XES given
data YES, where S = {O, l}" , is
where 1[. ] again denotes the usual indicator function . Here a is the log-
odds of correct to incorrect transmission of each Xi and (3 is the log-odds
in favor of Xi+! = Xi. In particular, we set a = In 4, corresponding to a
corruption probability of 0.2, and (3 = In 3, so that Xi+! is a repeat of Xi
with probability 0.75.
As a trite example, suppose y = 11101100000100010111, so that lSI =
220 = 1048576. For such a tiny space, we can calculate expected values
simply by enumeration but we also applied the Baum algorithm to generate
a random sample of size 10000 from 1T(xly). First, consider the posterior
marginal probabilities for the x/so We obtained Xl = 1 in 8989 of the
samples x(t) and hence our estimate of the corresponding probability is
0.899, versus the exact value 0.896; for X2 = I, we obtained 0.927 versus
0.924; and so on. Hence, the MPM estimate x* of x, defined by (10),
is correctly identified as x* = 11111100000000010111 . Clearly, x* is a
smoothed version of the data, with two fewer isolated bits . The xi's for
positions i = 4,12,16 and 17 are the most doubtful, with estimated (exact)
probabilities of Xi = 1 equal to 0.530 (0.541), 0.421 (0.425),0.570 (0.570)
and 0.434 (0.432). Note that, neither component 16 nor 17 flips in the
MPM estimate but that, if we examine them as a single unit, the posterior
probabilities of 00, 10, 01 and 11 are 0.362 (0.360), 0.203 (0.207), 0.068
(0.070) and 0.366 (0.362), respectively. Thus, there is a preference for 00
or 11, rather than the 10 obtained in x* .
The previous point illustrates how an estimate may be affected by
choosing either a marginal or a multivariate criterion. Indeed, at the op-
posite extreme to MPM is the MAP estimate (11), which here is equally
11111100000000011111or 11111100000000000111, both of which are easily
seen to have the same posterior probability. In our random sample, they
were indeed the two most frequent configurations, occurring 288 and 323
times, respectively, compared with the exact probability 0.0304. Note that
x* and Y itself occurred 138 and 25 times, compared with the exact proba-
bilities 0.0135 and 0.0027. If one requires a single-shot estimate of x, then
the choice of a particular criterion, ultimately in the form of a loss func-
tion , should depend on the practical goals of the analysis. For example,
the MAP estimate corresponds to zero loss for the correct X and unit loss
for any incorrect estimate, regardless of the number of errors among its
components; whereas MPM arises from a componentwise loss function and
minimizes the expected total number of errors among all the components.
The writer's own view is that a major benefit of a sampling approach is
254 JULIAN BESAG
that it enables one to investigate various aspects of the posterior distribu-

tion, rather than forcing one to concentrate on a single criterion; but note
that sampling from the posterior is not generally suitable for finding the
MAP estimate, which we address in Section 2.5 on simulated annealing.
As a more taxing toy example, we applied the Baum algorithm to gen-
erate a 100 realizations x from a noisy binary channel, again with a: = In 4
and (3 = In 3 but now with y = 111001110011100 . .., a vector of length
100000, so that 181 = 2100000. The MPM and MAP estimates of x, obtained
in the conventional manner from Baum and Viterbi algorithms, both coin-
cide with the data y in this case. The majority vote classification from the
100 random samples was correct for all 100000 components, although the
average success rate for a single sample was only 77.7%, with a maximum
of 78.1%. For a sample of size 10000, these figures were 77.7% and 78.2%,
respectively. We return to this example in Sections 2.5 and 3.8.
Finally, we mention some modifications of HMM's that one might want
to make in practice. For instance, the three p.m.f.'s on the right-hand
side of (9) could be partially unknown, with their own (hyper)priors; the
degradation mechanism forming the data y might be more complex, with
each Yi depending on several components of x; there could be multiple y's
for each x ; the Markov formulation x might be inappropriate in known
or unknown segments of x; and so on. In such cases, it is very likely
that standard methods for HMM's break down but a sampling approach
via MCMC can still be adopted. Slowing down of the algorithm can be
countered by sequential Monte Carlo; see below. An interesting and largely
unexplored further possibility is to cater for complications by incorporating
MCMC in an otherwise deterministic algorithm.
2.3. Importance sampling. The notion of learning about an other-
wise intractable fixed probability distribution ?T via Monte Carlo simulation
is of course quite natural. However, we now describe a more daunting task
in which the goal is to approximate E 1r* g for distributions ?T* that are close
to a baseline distribution ?T from which we have a random sample. For ex-
ample, in Bayesian sensitivity analysis, we need to assess how changes in
the basic formulation affect our conclusions. This may involve posterior
distributions that have different functional forms and yet are not far apart.
An analogous problem arises in difficult maximum likelihood estimation,
as we discuss in the next section. Importance sampling also drives sequen-
tial Monte Carlo methods and particle filters, in which observations on a
process arrive as a single or multiple time series and the goal is to update
inference as each new piece of information is received, without the need to
run a whole new simulation; see especially Doucet et al. (2001), Liu (2001)
and MacCormick (2002). Particle filters provide the most relevant MCMC
methods for problems in speech recognition, though the writer is not aware
of any specific references. We now describe how ordinary importance sam-
pling works.
MARKOV CHA IN MONTE CA RLO METHODS 255
Suppose we have a random sample x(1), . . . ,x(m) from 1l"(x) = h(x)je >
o for xES but that our real interest lies in E 11". g, for some specific g, where
1l"*(x) = h*(x) je* > 0, x E So,
wit h h* known and crucially S* ~ S. Now
(18) E 11" gh* = ~ g(x)h*(x) h(x) = e* ~ g(x) h*(x) =

h LJ h(x) e e LJ e*
xES xE s •
so we can estimate the right-hand side of (18) by the mean value of

g(x(t»)h*(x(t»)jh(x(t») . Usually, e*[c is unknown but, as a specia l case
of (18),
so t hat, as our eventual approx imation to E 11". g, we adopt the ratio esti-
mate,
m
(19) L w(x(t») g(x(t») ,
t= l
where
Note that the w(x(t») 's are independent of 9 and are well defined because
S* ~ S. T he estimate (19) is satisfactory if (5) is adeq uate for E 11"g and
there are no large weights among the w(x(t») ,s. In practice, the latter con-
dition requires that hand h* are not too far apart. There are modifications
of th e basic method described here that can extend its range (e.g. umbre lla
sampling).
2.4. Monte Carlo maximum likelihood estimation. Let x(O) de-
note an observatio n, generally a vector, from a p.m.f.
1l"(x;B) = h(x;B) j e(B) , X E S, BE 8,
where e(B) = L XES h(x; B) . Suppose we require the maximum likelihood

estimate,
of B but that, although h is quite manageable, e(B) and its derivatives

cannot be calculated directly, even for partic ular values of o.
256 JULIAN BESAG
Instead, suppos e th at we can generate a random sample from 1r(x; B)

for any given B, and let (x(1), . . . , x(m)) denot e such a sample for B = B, a
curre nt approximat ion to e.
Then, trivially, we can always write
1r(x(O); B) { h(x(O); B) c(B) }

(20) B = ar max In = ar max In In
A
v v - - v- •
g BEe 1r(x(O); B) g BEe h(x(O); B) c( B)
The first quotient on the right-hand side of (20) is known and the second
can be approximated using (18), where c(B), c(B), h(x(O) ; B) and h(x(O) ; B)
play the roles of c", c, h * and h, respectively. That is,
c(B) = ~ h(x; B) _ ~ h(x; B) ( . B V
1r X,
)
v L...J L...J v - v
c(B) xES c(B) XES h(x; B)
can be approximated by th e empirical average,

1 m h(x(t) ;B)
m L
t=l
h(x(t)· B) ,
'
for any B in th e neighborhood of B. It follows th at, at least when B is one-

or two-dimensional , an improved approximat ion to can be found by di-e
rect search , thou gh, in higher dimensions, it is necessary to implement a
more sophisticat ed appro ach, usually involving derivatives and correspond-
ing approximat ions. In practice, several stages of Monte Carlo sampling
may be required to reach an acceptable approximation to e.
Unfortunately, in most applications where st andard maximum likeli-
hood est imat ion is problemati cal, so to o is th e t ask of producing a random
sample from tt . The above approach must then be replaced by an MCMC
version, as introduced by Penttinen (1984), in spati al st atis t ics, and by
Geyer (1991) and Geyer and Thompson (1992), in more general settings.
For an exception to this rule, see Besag (2003), though in fact this is a
swindle because it uses perfect MCMC to generate th e random samples!
For a quit e complicated example of genuine MCMC maximum likelihood ,
see Tj elmeland and Besag (1998).
2.5. Simulated annealing. Simulated annealing (Kirkpatrick et al.,
1983) is a general purpose MCMC algorithm for th e optimization of dis-
crete high-dimensional functions. Here we describe a toy version, based
on ordinary Monte Carlo sampling , and comment briefly on the MCMC
implementation th at is required in practice.
Let {h( x) : X E S} , with S finite, denote a bound ed non-negative
function , specified at least up to scale. Let x = arg max, h(x) . We assume
for th e moment th at x is unique but th at S is too complicated for x to be
found by complete enumeration and t hat h does not have a sufficiently nice
structure for x to be determined by simple hill-climbing methods. In oper-
at ions research , where such problems abound, h is sometim es amena ble to
mathematical programming techniques; for example, the simplex method

applied to the traveling salesman problem . However, here we make no such
assumption.
Let {7I"(x) : XES} denote the corresponding finite p.m.f. defined by
(1) and (2), with c generally unknown. Clearly, x = arg max; 7I"(x) and,
indeed, the original task may have been to locate the global mode of 71", as
in our example below. The goal in simulated annealing is not to produce a
random draw from 71" but to bias the selection overwhelmingly in favor of
the most probable value x.
We begin by defining a sequence of distributions {7I"k (x)} for k =
1,2, . . ., where
(21) XES,
and the mk 's form a specified increasing sequence. Then, each of the distri-
butions has its mode at x and, as k increases, the mode becomes more and
more prominent. Thus, if we make a random draw from each successive
7I"k(X), eventually we shall only produce x, with the proviso that, if there
are multiple global maxima, observations are eventually drawn uniformly
from among the corresponding x's.
Example. Noisy binary channel. We return to the second case of the
noisy binary channel in Section 2.2.1, with y = 111001110011100..., a
vector of length 100000. The ordinary Viterbi algorithm identifies y itself
as the mode of 7I"(xly) but we also deduced this by sampling from 7I"k(X) ex
{71"( xly)}k, which requires a trivial amendment of the original sampling
algorithm. Thus, we generated x's from 7I"k(X) for mk = 1 (done already), 2,
. .. ,25 and noted the number of disagreements with y. For mk = 1,2,3,4,8,
°
12,16,20,21 ,22,23,24,25, there were 22290, 11928,6791,3826,442,30, 14,
0, 0, 2, 0, 0, discrepancies, respectively. Although still a toy example,
7I"(yly) ~ 5 X 10- 324 , so the task was not entirely trivial from a sampling
perspective.
Of course, in the real world, it is typical that, if x cannot be found
directly, then nor can we generate draws from 7I"k(X) . In that case, we
must implement an MCMC version in which successive 7I"k'S in a single run
of the algorithm are sampled approximately rather than exactly. This re-
quires some care in selecting a "schedule" for how the mk's in (21) should
increase, because the observation attributed to 7I"k must also serve as an
approximate draw from 7I"k+1 ' It is typical that eventually the mk's must
increase extremely slowly at a rate closer to logarithmic than to linear. Sim-
ulated annealing can also be extended to continuous functions via Langevin
diffusion; see Geman and Hwang (1986).
3. Markov chain Monte Carlo calculations.
3.1. Markov chains, stationary distributions and ergodicity.
In ordinary Monte Carlo calculations, we require perfect draws from the
258 JULIAN BESAG
target distribution {-7l"(x) : XES}. We now assume that this is imprac-

ticable but that we can construct a Markov transition probability matrix
(t.p.m.) P with state space S and limiting distribution 1f and that we
can generate a very long realization from the corresponding Markov chain.
In Section 3.2, we discuss some general issues in the construction and im-
plementation of suitable t.p.m.'s. At present, this may all seem bizarre:
generally S is astronomically large, tt is an arbitrary probability distribu-
tion on S and, even if we can find a suitable P, we cannot possibly store
it! Nevertheless, in Sections 3.3 to 3.7, we describe a general recipe for
any at; due to Hastings (1970), with the Gibbs sampler and the Metropolis
algorithm as special cases. Section 3.8 considers the more specialized topic
of perfect MCMC.
We begin by recalling some useful definitions and results for Markov
chains with finite or countable state spaces . Our notation differs from that
for the Markov chains in Section 2.2.1 but is chosen for consistency with
Section 2.1. Thus, let X(O), X(1), . . . denote a Markov chain with state
space Sand t .p.m. P, whose (x, x') element P(x, x') is the probability of
a one-step transition from xES to x' E S. Define Po to be the row vector
representing the p.m .f. of the initial state x(O) . Then the marginal p.m .f
Pt of X(t) is given by
(22) t = 0,1, . . . ,
and, if 1f is a probability vector satisfying general balance
(23) tt P = tt ;
then 1f is called a stationary distribution for P. That is, P maintains rr:

if Po = n, then Pt = 1f for all t = 1,2, . . .. What we require is something
more : that, given 1f (up to scale), we can always find a P for which Pt ~ 1f
as t ~ 00, irrespective of Po. The additional condition is that P should be
ergodic; that is, irreducible and aperiodic, in which case 1f in (23) is unique.
Irreducible means that there exists a finite path between any pair of states
x, x' E S that has non-zero probability. Aperiodic means that there is
no state that can recur only after a multiple of d steps, where d ~ 2. A
sufficient condition for an irreducible P to be aperiodic is that at least one
diagonal element P(x,x) of P is non-zero, which is automatically satisfied
by almost any P in MCMC . More succinctly, P is ergodic if and only if all
elements of P'" are positive for some positive integer m. It then follows
that g, defined in (5) or, more correctly, the corresponding sequence of
r.v.'s, also converges almost surely to E71"g as m ~ 00. Furthermore, as in
ordinary Monte Carlo, the sampling variance of 9 can be assessed and is of
order 11m. For details, see almost any textbook covering Markov chains.
Stationarity and irreducibility are somewhat separate issues in MCMC .
Usually, one uses the Hastings recipe in Section 3.3 to identify a whole
collection of t .p.m. ts Pk, each of which maintains 1f and is simple to apply
but is not individu ally irreducible with respect to S . One th en combines

th ese Pk 's appropriately to achieve irreducibility. In particular , note that,
if PI , . .. , Pn maintain 7r , then so do
(24)
equivalent to applying PI , . . . , Pn in turn, and
(25)
equivalent to choosing one of the Pk 'S at random. Amalgamations such

as (24) or (25) are very common in practice. For example, (25) ensures
th at , if a transition from x to x' is possible using any single Pk , t hen
this is inherited by P . In applicat ions of MCMC, where x E S has many
individu al components, x = (Xl , . . . ,X n ) , it is typical to specify a Pi for
each i, where Pi allows change only in Xi . Then P in (24) allows change in
each component in turn and (25) in any single component of x, so that, in
eit her case, irreducibility is at least plausible.
Ideally, we would like x(O) to be drawn directly from n , which is the
goal of perfect MCMC algorit hms (Section 3.8) but generally this is not
viable. The usual fix is to ignore the output during a burn -in phase before
collecting th e sampl e x(l) , .. . , x(m) for use in (5). There are no hard and
fast rules for choosing th e burn-in but assessment via form al analysis (e.g.
autocorrelat ion tim es) and informal graphical meth ods (e.g. par allel box-
and-whisker plots of t he output) is usually adequate, though simple tim e-
series plots can be misleading.
There are some contexts in which burn-in is a crucia l issue; for exam-
ple, with the Ising model in statistical physics and in some applications in
genetics. It is th en desirable to const ruct special purp ose algorithms; see,
among ot hers, Sokal (1989), Marin ari and Parisi (1992), Besag and Green
(1993) and Geyer and Th ompson (1995). Some keywords include auxiliary
varia bles, multigrid m ethods and simulated tempering (which is relat ed to
but distinct from simulat ed annealing).
When X is very high-dimensional , storage of MCMC samples can
become problematic. Stor age can be minimized by calculat ing (5) on the
fly for any given g, but often th e g's of eventual inter est are not known in
advance. Because successive st at es X (t) , X(t+l ) usually have high positive
autocorrelation, little is lost by subsampling th e output. However, thi s
has no intrinsic merit and it is not generally intended t hat the gaps be
sufficiently large to produce in effect a random sample from 7r. No new
theory is required for subsampling: if the gap length is r , t hen P is merely
replaced by the new Markov t .p.m. P", Therefore, we can ignore this
aspect in const ruct ing appropriate P ' s , thou gh event ually X( l), . . . , x(m) in
(5) may refer to a subsample. Note also that burn-in and collection tim e are
somewhat separate issues: the rate of convergence to 7r is enha nced if the
260 JULIAN BESAG
second-largest eigenvalue of P is small in modulus, whereas a large negative

eigenvalue can improve the efficiency of estimation. Indeed, one might
use different samplers during the burn-in and collection phases. See, for
example, Besag et al. (1995), especially the rejoinder, for some additional
remarks and references.
Lastly here , we mention that the capabilities of MCMC have occa-
sionally been undersold, in that the convergence of the Markov chain is
not merely to the marginals of 'Tr but to its entire multivariate distribution.
Corresponding functionals (3), whether involving a single component or
many, can be evaluated with equal ease from a single run. Of course, there
are some obvious limitations: for example, one cannot expect to approxi-
mate the probability of some very rare event with high relative precision
without a possibly prohibitive run length.
3.2. Detailed balance. We need a method of constructing Pk'S to
satisfy (23). That is, we require Pk'S such that
(26) L 'Tr(x) Pk(x, x') = 'Tr(x/),

xES
for all x' E S . However, we also need to avoid the generally intractable
summation over the state space S. We can achieve this by demanding
a much more stringent condition than general balance, namely detailed
balance,
(27)
for all x, x' E S. Summing both sides of (27) over xES implies that
general balance is satisfied; moreover, detailed balance is much simpler to
confirm, particularly if we insist that Pk(X, x') = 0 = Pdx ', x) for the vast
majority of x , x' E S . Also note that (27) need only be checked for x' =1= x ,
which is helpful in practice because the diagonal elements of Pk are often
quite complicated. The physical significance of (27) is that, if a stationary
Markov chain . .. , X( -1) , X(O), X(1), . . . satisfies detailed balance, then it is
time reversible, which means that it is impossible to tell whether a film of
a sample path is being shown forwards or backwards.
It is clear that, if PI, . . . , Pn individually satisfy detailed balance with
respect to 'Tr, then so does P in (25). Time reversibility is not inherited in
the same way by P in (24) but it can easily be resurrected by assembling
the Pk'S as a random rather than as a fixed permutation at each stage.
The maintenance of time reversibility has some theoretical advantages (e.g.
the Central Limit Theorem of Kipnis and Varadhan, 1986, and the Initial
Sequen ce Estimators of Geyer, 1992) and is worthwhile in practice if it adds
a negligible computational burden.
3.3. Hastings algorithms. Hastings (1970) provides a remarkably
simple general construction of t .p.m.Is Pk satisfying detailed balance (27)
MARK OV CHA IN M ONTE CARLO METHODS 261
with respect to 7L Thus, let Rk be any Markov t .p.m. having state space S
and elements Rk(X, x' ), say. Now define the off-diagonal elements of Pk by
(28) x' f; XE S ,
where Ak(x , x' ) = 0 if Rk(X, x' ) = 0 and oth erwise

l)Rd xl,
(29) I • { 7r(x x)}
Ak(X,X) = min 1 , 7r(x) Rk(x, x') ,
with Pk(X, x ) obtained by subtraction to ensure that Pk has unit row sums,
which is achievable since Rk is itself a t .p.m. Th en, to verify th at de-
t ailed balance (27) is satisfied for x' f; x, eit her Pk(X, X') = 0 = Pk(X', X)
and there is nothing to prove or else direct substitution of (28) produ ces
min {7r( x) Rk(X, X') , 7r (x') Rdx ' , x )} on both sides of th e equation. Thus,
7r is a st ationary distribution for Pk, despite the arbitrary choice of Rk'
t hough note that we might as well have insisted that zeros in Rk occur
symmet rically. Note also that Pk depends on 7r only through h(x ) in (1)
and th at the usually unknown and problemati c normalizing constant c can-
cels out . Of course, t hat is not quite t he end of th e story: it is necessary to
check th at P , obt ained via an amalgamat ion of different Pk'S , is sufficientl y
rich to guarantee irreducibility with respect to 7r but usually this is simple
to ensure in any par ticular case.
Operationally, any Pk is applied as follows . When in state x, a pro-
posal x* for the subsequent state x' is generated with probability Rk(X, z"},
T his requires calculat ing th e non- zero elements in row x of Rk on the fly,
rath er tha n storing any matri ces. Th en eit her x' = x*, with t he acceptance
probability Ak(X, x*), or else x' = x is ret ained as th e next state of the
chain. Note that (28) does not apply to the diagonal elements of P: two
successive st at es x and x' can be th e same eit her because x happens to be
proposed as the new state or because some other state x* is proposed but is
not accepte d. Also note that the procedur e differs from ordinary rejection
sampling, where proposals x* are made until one is accepte d, which is not
valid in MCMC.
3.4. Componentwise Hastings algorithms. In practice, we still
need to choose a particular set of Rk's. It is important th at proposals and
decisions on their acceptance are simple and fast to make. We now openly
acknowledge t hat X has many components and write X = (X l," " X n ),
where each Xi is univariate (though this is not essential). Then, the most
common approach is to devise an algorit hm in which a proposal matrix R ; is
assigned to each individu al component X i' Th at is, if x is t he current state,
th en R; propos es replacing th e ith component Xi by xi, while leaving th e
remainder X-i of x unaltered. Note th at we can also allow some continuous
components: th en t he corresponding Ri' s and Pi' s become transition ker-
nels rath er t han matri ces and have elements that are conditional densities
rath er than probabilities. Although t he underlying Markov chain t heory
262 JULIAN BESAG
must then be reworked in terms of general state spaces (e.g. Nummelin,

1984), the modifications in practice are entirely straightforward. For con-
venience here, we continue to adopt discrete state space terminology and
notation.
In componentwise Hastings algorithms, the acceptance probability for
xi can be rewritten as
(30) A t.(x, x *) -- nun

. {I 7r(XiI
7r (I
X-i)Ri(X*,x)}
Xi X-i ) R;(X, x*) '
,
which identifies the crucial role played by the full conditionals 7r(xilx-i) .
Note that these n univariate distributions comprise the basic building
blocks of Markov random field formulations in spatial statistics (Besag,
1974), where formerly they were called the local characteristics of X.
The full conditionals for any particular 7r(x) follow from the trivial
but, at first sight, slightly strange-looking result,
(31)
where the normalizing constant involves only a one-dimensional summation
over Xi. Even this drops out in the ratio (30) and, usually, so do many other
terms because likelihoods, priors and posteriors are typically formed from
products and then only those factors in (31) that involve Xi itself need to
be retained. Such cancelations imply enormous computational savings.
In terms of Markov random fields, the neighbors ai of i comprise the
minimal subset of -i such that 7r(xilx-i) = 7r(xilx8i)' Under a mild pos-
itivity condition (see Section 3.5), it can be shown that, if j E Bi , then
i E oj, so that the n neighborhoods define an undirected graph in which
there is an edge between i and j if they are neighbors. Similar considera-
tions arise in graphical models (e.g. Lauritzen, 1996) and Bayesian networks
(e.g. Pearl, 2000) in constructing the implied undirected graph from a di-
rected acyclic graph or from a chain graph, for example. Note that conven-
tional dynamic simulation makes use of directed graphs, whereas MCMC is
based on undirected representations or a mix of the two, as in space-time
(chain graph) models, for example. Generally speaking, dynamic simula-
tion should be used and componentwise MCMC avoided wherever possible .
Example. Autologistic and related distributions. The autologistic dis-
tribution (Besag, 1974) is a pairwise-interaction Markov random field for
dependent binary r.v.'s. It includes binary Markov chains, noisy binary
channels and finite-lattice Ising models as special cases, so that simulation
without MCMC can range from trivial to taxing to (as yet) impossible.
We define X = (Xl, " " X n ) to have an autologistic distribution if its
p.m.f. is
(32) 7r(x) oc exp (LC¥iXi + ~,Bij1[Xi = Xj]) , XES = {O, 1}n,

t t<J
where the indices i and j run from 1 to n and the {3ij'S control the de-
pendence in the system. The simplification with respect to a saturated
binary model is that no terms involve interactions between three or more
components. The autologistic model also appears under other names: in
Cox and Wermuth (1994), as the quadratic exponential binary distribution
and, in Jordan et al. (1998), as the Boltzmann distribution, after Hinton
and Sejnowski (1986).
It is convenient to define {3ij = {3ji for i > j . Then, as regards the spe-
cial cases mentioned above, the r.v.'s Xl, . .. , X n form a simple symmetric
Markov chain if O:i = 0 and {3ij = {3 for Ii - j! = 1, with (3ij = 0 otherwise.
The noisy binary channel (17) is obtained when n(x) becomes n(xIY) with
y fixed, O:i = (2Yi - 1)0: and {3ij = {3 whenever Ii - j\ = 1, with {3ij = 0
otherwise. For the symmetric Ising model, the indices i are identified with
the sites of a finite d-dimensional regular array, O:i = 0 for all i and {3ij = {3
for each pair of adjacent sites, with (3ij = 0 otherwise. In each case, the
asymmetric version is also catered for by (32).
It follows from (31) that the full conditional of Xi in the distribution
(32) is
(33) n(xilx-i) ex exp Io.z; + L{3ij1[xi = Xj]), Xi = 0,1,

Hi
so that i and j are neighbors if and only if {3ij 1= o. i. For example, in the
noisy binary channel (17)
where Xo = Xn+l = -1 to accommodate the end points i = 1 and i = n ;

and correspondingly, 8i = {i - 1, i + I}, unless i = 1 or n for which i has
a single neighbor.
Although it is trivial to evaluate the conditional probabilities in (33),
because there are only two possible outcomes, we again emphasize that
the normalizing constant is not required in the Hastings ratio, which is
important in much more complicated formulations. We now describe the
two most widely used componentwise algorithms, the Gibbs sampler and
the Metropolis method.
3.5. Gibbs sampler. The term "Gibbs sampler" (Geman and Ge-
man, 1984) is motivated by the simulation of Gibbs distributions in statis-
tical physics, which correspond to Markov random fields in spatial statis-
tics, the equivalence being established by the Hammersley-Clifford theorem
(Besag, 1974). The Gibbs sampler can be interpreted as a componentwise
Hastings algorithm in which proposals are made from the full conditionals
themselves; that is,
(35)
264 JULIAN BESAG
so that the quotient in (30) has value 1 and the proposals are always ac-
cepted. The n individual Pi'S can then be combined as in (24), producing
a systematic scan of all n components, or as in (25), giving a random scan,
or otherwise. Systematic and random scan Gibbs samplers are aperiodic,
because Ri(x,x) > a for any x E S; and they are irreducible under the pos-
itivity condition that the minimal support S of X is the Cartesian product
of the minimal supports of the individual Xi'S. Positivity holds in most
practical applications and can be relaxed somewhat (Besag, 1994b) to cater
for exceptions. To see its relevance, consider the trite example in which
X = (X}, X 2 ) and S = {OO, 11}, so th at no movement is possible using 'a
componentwise updating algorithm. On the other hand, if S = {OO, 01,11} ,
then positivity is violated but both the systematic and random scan Gibbs
samplers are irreducible. Severe problems occur most frequently in con-
strained formulations and can be tackled by using block updates of more
than one component at a time, to which we return in Section 3.7, or by
augmenting S by dummy states (e.g. {01} in the above example) .
Although the validity of the Gibbs sampler is ensured by the general
theory for Hastings algorithms, there is a more direct justification, which
formalizes the argument that, if X has distribution Jr and any of its com-
ponents is replaced by one sampled from the corresponding full conditional
induced by n , to produce a new vector X', then X' must also have distri-
bution Jr. That is, if x' differs from x in its ith component at most, so that
x'-i = X-i, then
Xi
Example. Autologistic distribution. In the systematic scan Gibbs sam-

pler for the distribution (32), every cycle addresses each component Xi in
turn and immediately updates it according to its full conditional distribu-
tion (33).
3.6. Metropolis algorithms. The original MCMC algorithm is that
of Metropolis et al. (1953), used for the Ising and other models in statistical
physics. This is a componentwise Hastings algorithm, in which R; is chosen
to be a symmetric matrix, so that the acceptance probability (30) reduces
to
(36)
independent of R i ! For example , if Xi supports only a small number of

values, then R; might select xi uniformly from these, usually excluding the
current value Xi. If Xi is continuous, then it is common to choose xi ac-
cording to a uniform or Gaussian or some other easily-sampled symmetric
distribution, centered on Xi and with a scale factor determined on the basis
of a few pilot runs to give acceptance rates in the range 20 to 60%, say. A
little care is needed here if Xi does not have unbounded support, so as to

maintain symmetry near an endpoint; alternatively a Hastings correction
can be applied.
The intention in Metropolis algorithms is to make proposals that can
be generated and accepted or rejected very fast . Note that consideration
of 7r arises only in calculating the ratio of the full conditionals in (36) and
that this is generally a much simpler and faster task than sampling from
a full conditional distribution, unless the latter happens to have a very
convenient form. Thus, the processing time per step is generally much less
for Metropolis than for Gibbs; and writing a program from scratch is much
easier.
Example. Autologistic distribution. When updating X i in (32), the ob-
vious Metropolis proposal is deterministic, with xi = 1- X i . This generally
results in more mobility than in the corresponding Gibbs sampler, because
Ai(x,x*) 2: 7r(xilx-i), and therefore increases statistical efficiency. The
argument can be formalized (Peskun, 1973, and more generally, Liu, 1996)
and provides one reason why physicists prefer Metropolis to Gibbs for the
ferromagnetic Ising model, though these days they would usually adopt a
cluster algorithm (e.g. Swendsen and Wang, 1987).
3.7. Gibbs sampling versus other Hastings algorithms. The
Gibbs sampler has considerable intuitive appeal and one might assume
from its ubiquity in the Bayesian literature that it represents a panacea
among componentwise Hastings algorithms. However, we have just seen
that this is not necessarily so. The most tangible advantage of the general
Hastings formulation over the Gibbs sampler is that it can use the current
value Xi of a component to guide the choice of the proposal xi and improve
mobility around the state space S . For some further discussion, see Besag
et al. (1995, Section 2.3.4) and Liu (1996). Even when Gibbs is statistically
more efficient, a simpler algorithm may be superior in practice if 10 or 20
times as many cycles can be executed in the same run time . Hastings also
facilitates the use of block proposals, which are often desirable to increase
mobility and indeed essential in constrained formulations. As one example,
multivariate proposals are required in the Langevin-Hastings algorithm
(Besag , 1994a).
Nevertheless , there are many applications where efficiency is relatively
unimportant and componentwise Gibbs is quite adequate. In the contin-
uous case, difficult full conditionals are often log-concave and so permit
the use of adaptive rejection sampling (Gilks, 1992). Also, approximate
histogram-based Gibbs samplers can be corrected by appropriate Hastings
steps; see Tierney (1994) and, for related ideas involving random proposal
distributions, Besag et al. (1995, Appendix 1).
Finally, smart block updates can sometimes be incorporated in Gibbs.
For example, if 7r(xAlx_A) , in a particular block A, is multivariate Gaus-
sian, then updates of X A can be made via Cholesky decomposition. Simi-
266 JULIAN BESAG
larly, if ?T(xAlx-A) is a hidden Markov chain, then the Baum algorithm can
be exploited when updating X A . It may also be feasible to manufacture
block Gibbs updates by using special recursions (Bartolucci and Besag,
2002) .
3.8. Perfect MCMC simulation. We have noted that it may be dif-
ficult to determine a suitable burn-in period for MCMC. Perfect MCMC
simulation solves the problem by producing an initial draw that is exactly
from the target distribution ?T. The original method, called monotone cou-
pling from the past (Propp and Wilson, 1996), in effect runs the chain from
the infinite past and samples it at time zero, so that complete convergence
is assured. This sounds bizarre but can be achieved in several important
special cases, including the Ising model, even at its critical temperature on
very large arrays (e.g. 2000 x 2000) . Perfect MCMC simulation is a very
active research area; see
http ://research.microsoft.com/~dbYilson/exact/
Here we focus on coupling from the past (CFTP) and its implementation
when ?T is the posterior distribution (17) for the noisy binary channel.
We assume that ?T has finite support S . We can interpret burn-in
of fixed length mo as running our ergodic t.p .m. P forwards from time
-mo and ignoring the output until time O. Now imagine that, instead of
doing this from a single state at time -mo , we do it from every state in
S, using the identical stream of random numbers in every case, with the
consequence that, if any two paths ever enter the same state, then they
coalesce permanently. In fact, since S is finite, we can be cert ain that, if
mo is large enough, coalescence of all paths will occur by time 0 and we
obtain the same state x(O) regardless of x( -m o) . It also ensures that we will
obtain x(O) if we run the chain from any state arbit rarily far back in time,
so long as we use the identical random number stream during the final mo
steps. Hence x(O) is a random draw from zr. It is crucial in this argument
that the timing of the eventual draw is fixed. If we run the chain forwards
from every possible initialization at time 0 and wait for all the paths to
coalesce, we obtain a random stopping time and a corresponding bias in
the event ual state. As an extreme example, suppose that P(x' , x") = 1 for
two particular states x' and x" but that P(x,x") = 0 for all x 1= x' , Then
?T(x") = ?T(x') but the state at coalescence is never x".
At first sight, useful implementation of the above idea seems hopeless.
Unless S is tiny, it is not feasible to run the chain from every st at e in S
even for mo = 1. However, we can sometimes find a monotonicity in the
paths which allows us to conclude that coalescence from certain extremal
states implies coalescence from everywhere. We discuss this here merely
for the noisy binary channel but the reasoning is identical to that in Propp
and Wilson (1996) for th e ostensibly much harder Ising model and also
extends immediately to the general autologistic model (32), provided the
f3ij 'S are non-negative .
Thus, again consider the posterior distribution (17), with Q: and (3 > 0
known. We have seen already that it is easy to implement a systematic
scan Gibbs sampler based on (34). We presume that the usual inverse dis-
tribution function method is used at every stage : that is, when addressing
component X i , we generate a uniform deviate on the unit interval and, if
its value exceeds the probability for Xi = 0, implied by (34), we set the new
Xi = 1, else Xi = O.
Now imagine that, using a single stream of random numbers, we run
the chain as above from each of two states x' and x" E S such that x' :S x"
componentwise. Then the corresponding inequality is inherited by the new
pair of states obtained at each iteration, because (3 > O. Similarly, consider
three initializations, 0 (all zeros), 1 (all ones) and any other xES . Because
o :S X :S 1 elementwise, it follows that the corresponding inequality holds
at every subsequent stage and so all paths must coalesce by the time the
two ext reme ones do so. Hence, we need only monitor the two extremal
paths. Note that coalescence occurs much faster than one might expect,
because of the commonalit ies in the simulation method.
However, we must still determine how far back we need to go to ensure
that coalescence occurs by time O. A basic method is as follows. We
begin by running simulations from time -1 , initialized by x( -1) = 0 and
1, respectively. If the paths do not coalesce at t ime 0, we repeat the
procedure from time -2, ensuring that the previous random numbers are
used again between times -1 and O. If the paths do not coalesce by time
0, we repeat from time -3, ensuring that the previous random numbers
are used between times -2 and 0; and so on. We terminate the process
when coalescence by time 0 first occurs and t ake the corresponding x(O) as
our random draw from n . We say coalescence "by" rather than "at" time
o because, in the final run, this may occur before time O. In practice, it is
generally more efficient to use increasing increments between the starting
times of successive runs, again with duplication of the random numbers
during the common intervals . There is no need to identify the smallest m
for which coalescence occurs by time zero.
For a numerical illustration, we again chose Q: = In 4 and (3 = In 3
in (17), with y = 111001110011100..., a vector of length 100000. Thus,
the state space has 2100000 elements. Moving back one step at a tim e,
coalescence by time 0 first occurred when running from time -15, with
an approximate halving of the discrepancies between each pair of paths,
generation by generation, though not even a decrease is guaranteed. Co-
alescence itself occurred at time -2. There were 77759 matches between
the CFTP sample x(O) and the MPM and MAP estimates, which recall
are both equal to y in this case. Of course, the performance of CFTP
becomes hopeless if (3 is too large but, in such cases, it may be possible
to adopt algorithms that converge faster but still preserve monotonicity.
Indeed, for the Ising model, Propp and Wilson (1996) use Sweeny's cluster
algorit hm rather than th e Gibbs sampler. An alternative would be to use
268 JULIAN BESAG
perfect block Gibbs sampling (Bartolucci and Besag, 2002). Fortunately,

in most Bayesian formulations, convergence is relatively fast because the
information in the likelihood dominates that in the prior .
4. Discussion. The most natural method of learning about a complex

probability model is to generate random samples from it by ordinary Monte
Carlo methods. However, this approach can only rarely be implemented.
An alternative is to relax independence. Thus, in recent years, Markov
chain Monte Carlo methods, originally devised for the analysis of com-
plex stochastic systems in statistical physics, have attracted much wider
attention. In particular, they have had an enormous impact on Bayesian
inference, where they enable extremely complicated formulations to be ana-
lyzed with comparative ease, despite being computationally very intensive .
For example, they are now applied extensively in Bayesian image analysis;
for a recent review, see Hurn et al. (2003). There is also an expanding
literature on particle filters, whose goal is to update inferences in real time
as additional information is received. At the very least, MCMC encourages
the investigator to experiment with models that are beyond the limits of
more traditional numerical methods.
Acknowledgment. This research was supported by the Center for
Statistics and the Social Sciences with funds from the University Initiatives
Fund at the University of Washington.
REFERENCES
BARTOLUCCI F . AND BESAG J .E . (2002). A recursive algorithm for Markov random

fields. Biometrika, 89, 724-730.
BAUM L.E ., PETRIE T ., SOULES G ., AND WEISS N. (1970). A maximization technique
occurring in the statistical analysis of probabilistic functions of Markov chains.
Annals of Mathematical Statistics, 41, 164-171.
BESAG J .E . (1974). Spatial interaction and th e statistical an alysis of lattice systems
(with Discussion) . Journal of the Royal Statistical Society B , 36, 192-236.
BESAG J .E . (1994a) . Discussion of paper by U. Grenander and M. I. Miller. Journal of
the Royal Statistical Society B, 56, 591-592 .
BESAG J .E . (1994b) . Discussion of paper by L.J . Tierney. Annals of Statistics, 22,
1734-1741.
BESAG J .E . (2001). Markov chain Monte Carlo for statistical inference. Working Paper
No.9, Center for Statistics and the Social Sciences, University of Washington,
pp .67.
BESAG J .E . (2003). Likelihood analysis of binary data in time and space. In Highly
Structured Stochastic Systems (eds. P.J . Green, N.L. Hjort, and S. Richardson) .
Oxford University Press.
BESAG J .E . AND CLIFFORD P . (1989). Generalized Monte Carlo sign ificance tests.
Biometrika, 76, 633-642.
BESAG J .E . AND CLIFFORD P . (1991). Sequential Monte Carlo p-values. Biometrika,
78 , 301-304.
BESAG J .E. AND GREEN P .J . (1993). Spatial st atistics and Bayesian computation (with
Discussion) . Journal of the Royal Statistical Society B, 55, 25-37.
BESAG J .E ., GREEN P .J ., HIGDON D.M ., AND MENGERSEN K.L. (1995) . Bayesian com-
putation and stochastic systems (with Discussion) . Statistical Science, 10, 3-66.
CHEN M .-H ., SHAO Q .-M ., AND IBRAHIM J.G . (2000) . Monte Carlo Methods in Bayesian
Computation. Springer: New York .
Cox D .R. AND WERMUTH N. (1994). A note on the quadratic exponential binary dis-
tribution. Biometrika, 81, 403-408.
DOUCET A. , DE FREITAS N. , AND GORDON N. (2001) . Sequential Monte Carlo Methods
in Practice . Springer: New York.
EDDIE S .R., MITCHISON G ., AND DURBIN R . (1995) . Maximum discrimination hidden
Markov models of sequence concensus. Journal of Computational Biology, 2, 9-24.
FISHMAN G .S . (1996) . Monte Carlo: Concepts, Algorithms, and Applications. Springer
Verlag: New York.
FREDKIN D .R. AND RICE J.A . (1992). Maximum likelihood estimation and identification
directly from single-channel recordings. Proceedings of the Royal Society of London
B, 249, 125-132.
GELMAN A ., CARLIN J .B ., STERN H .S., AND RUBIN D .B . (1995) . Bayesian Data Anal-
ysis. Chapman and Hall/CRC: Boca Raton .
GEMAN S . AND GEMAN D. (1984) . Stochastic relaxation, Gibbs distributions and the
Bayesian restoration of images. Institute of Electrical and Electronics Engineers ,
Transactions on Pattern Analysis and Machine Intelligence, 6, 721-741.
GEMAN S. AND HWANG C .-R. (1986) . Diffusions for global optimization. SIAM Journal
on Control and Optimization, 24, 1031-1043.
GEYERC .J . (1991) . Markov chain Monte Carlo maximum likelihood. In Computing Sci-
ence and Statistics: Proceedings of the 23m Symposium on the Interface (ed . E .M .
Keramidas) , 156-163. Interface Foundation of North America, Fairfax Station, VA.
GEYER C .J . (1992) . Practical Markov chain Monte Carlo (with Discussion) . Statistical
Science, 7, 473-511.
GEYERC.J . AND THOMPSON E .A . (1992). Constrained Monte Carlo maximum likelihood
for dependent data (with Discussion) . Journal of the Royal Statistical Society B,
54, 657-699.
GEYER C.J. AND THOMPSON E .A. (1995) . Annealing Markov chain Monte Carlo with
applications to ancestral inference. Journal of the American Statistical Association,
90, 909-920.
GILKS W.R. (1992) . Derivative-free adaptive rejection sampling for Gibbs sampling. In
Bayesian Statistics 4 (eds . J .O . Berger, J .M . Bernardo, A.P . Dawid, and A.F .M .
Smith), 641-649. Oxford University Press.
GILKS W .R. , RICHARDSON S ., AND SPIEGELHALTER D . (eds.) (1996). Markov Chain
Monte Carlo in Practice . Chapman and Hall : London.
GREEN P .J . (1995). Reversible jump Markov chain Monte Carlo computation and
Bayesian model determination. Biometrika, 82, 711-732.
HASTINGS W.K . (1970) . Monte Carlo sampling methods using Markov chains and their
applications. Biometrika, 57,97-109.
HAUSSLER D ., KROGH A ., MIAN S., AND SJOLANDER K . (1993) . Protein modeling us-
ing hidden Markov models: analysis of glob ins . In Proceedings of the Hawaii In-
ternational Conference on System Sciences . IEEE Computer Science Press: Los
Alamitos, CA .
HINTON G .E . AND SEJNOWSKI T . (1986) . Learning and relearning in Boltzmann ma-
chines. In Parallel Distributed Processing (eds . D.E . Rumelhart and J .L. McClel-
land). M.l.T Press.
HUGHES J .P ., GUTTORP P ., AND CHARLES S.P. (1999) . A nonhomogeneous hidden
Markov model for precipitation. Applied Statistics, 48 , 15-30.
HURN M ., HUSBY 0 ., AND RUE H . (2003) . Advances in Bayesian image analysis. In
Highly Structured Stochastic Systems (eds. P.J . Green, N.L. Hjort, and S. Richard-
son). Oxford University Press.
JORDAN M.L , GHAHRAMANI Z., JAAKKOLA T.S ., AND SAUL L.K . (1998) . An introduction
to variational methods for graphical models. In Learning in Graphical Models
(ed. M.l. Jordan) . Kluwer Academic Publishers.
270 JULIAN BESAG
JUANG B .H. AND RABINER L.R . (1991). Hidden Markov models for speech recognition.
Technometrics, 33, 251-272 .
KIPNIS C . AND VARADHAN S.R.S. (1986). Central limit theorem for additive functionals
of reversible Markov processes and applications to simple exclusions. Communica-
tions in Mathematical Physics, 104, 1-19 .
KIRKPATRICK S., GELATT C .D ., AND VECCHI M.P. (1983). Optimization by simulated
annealing. Science, 220, 671-680.
LAURITZEN S.L . (1996). Graphical Models. Clarendon Press: Oxford.
LE STRAT Y. AND CARRAT F . (1999). Monitoring epidemiologic surveillance data using
hidden Markov models. Statistics in Medicine , 18, 3463-3478.
LIU J .S. (1996). Peskun's theorem and a modified discrete-state Gibbs sampler. Bio-
metrika, 83, 681--682.
LIU J .S. (2001). Monte Carlo Strategies in Scientific Computing. Springer: New York.
LIU J.S ., NEUWALD A.F. , AND LAWRENCE C .E . (1995). Bayesian models for multiple
local sequence alignment and Gibbs sampling strategies. Journal of the American
Statistical Association, 90, 1156-1170.
MACCORMICK J . (2002). Stochastic Algorithms for Visual Tracking. Springer: New
York .
MACDoNALD I.L. AND ZUCCHINI W . (1997). Hidden Markov and Other Models for
Discrete-valued Time Series . Chapman and Hall : London.
MARINARI E . AND PARISI G . (1992). Simulated tempering: a new Monte Carlo scheme.
Europhysics Letters, 19, 451-458 .
METROPOLIS N., ROSENBLUTH A.W., ROSENBLUTH M.N ., TELLER A.H. , AND TELLER
E . (1953) . Equations of state calculations by fast computing machines. Journal of
Chemical Physics, 21, 1087-1092 .
NEWMAN M.E.J . AND BARKEMA G .T . (1999). Monte Carlo Methods in Statistical
Physics . Clarendon Press: Oxford.
NUMMELIN E . (1984). General Irreducible Markov Chains and Non-Negative Operators.
Cambridge University Press.
PEARL J . (2000).] Causality . Cambridge University Press.
PENTTINEN A. (1984). Modeling interaction in spatial point patterns: parameter estima-
tion by the maximum likelihood method. Jyviiskylii Studies in Computer Science,
Economics and Statistics, 7.
PESKUN P.H . (1973). Optimum Monte Carlo sampling using Markov chains. Biometri-
ka, 60, 607-612.
PROPP J .G . AND WILSON B.M . (1996). Exact sampling with coupled Markov chains
and applications to statistical mechanics. Random Structures and Algorithms, 9,
223-252.
RABINER L.R. (1989). A tutorial on hidden Markov models and selected applications
in speech recognition. Proceedings of the Institute of Electrical and Electronics
Engineers, 77, 257-284 .
ROBERT C.P. AND CASELLA G . (1999). Monte Carlo Statistical Methods. Springer: New
York.
ROBERT C.P ., RYDEN T ., AND TITTERINGTON D.M. (2000). Bayesian inference in hidden
Markov models through the reversible jump Markov chain Monte Carlo method.
Journal of the Royal Statistical Society B, 62, 57-75.
SOKAL A.D . (1989). Monte Carlo methods in statistical mechanics: foundations and
new algorithms. Cours de Troisieme Cycle de la Physique en Suisse Romande,
Lausanne.
SWENDSEN R.H. AND WANG J .-S . (1987). Non-universal critical dynamics in Monte
Carlo simulations. Physics Review Letters, 58, 86-88 .
TIERNEY L.J. (1994). Markov chains for exploring posterior distributions (with Discus-
sion). Annals of Statistics, 22, 1701-1762.
TJELMELAND H. AND BESAG J. (1998). Markov random fields with higher-order inter-
actions. Scandinavian Journal of Statistics, 25, 415-433 .
SEMIPARAMETRIC FILTERING IN SPEECH PROCESSING
BENJAMIN KEDEM' AND KONSTANTINOS FOKIANOSt
Abstract. We consider m data sets where the first m - 1 are obtained by sampling
from multiplicative exponential distortions of the mth distribution, it being a refer-
ence. The combined data from m samples, one from each distribution, are used in the
semipararnetric large sample problem of estimating each distortion and the reference
distribution, and testing the hypothesis that the distributions are identical. Possible
applications to speech processing are mentioned.
1. Introduction. Imagine the general problem of combining sources

of information as follows. Suppose there are m related sources of data,
of which the mth source, called the "reference", is the most reliable . Ob-
viously, the characteristics of the reference source can be assessed from
its own information or data. But since the sources are related , they all
contain pertinent information that can be used collectively to improve the
estimation of the reference characteristics. The problem is to combine all
the sources, reference and non-reference together, to better estimate the
reference characteristics and deviations of each source from the reference
source .
Thus, throughout this paper the reader should have in mind a "ref-
erence" and deviations from it in some sense, and the idea of combining
"good" and "bad" to improve the quality of the "good" .
We can think of several ways of applying this general scheme to speech
processing. The idea could potentially be useful in the combination of
several classifiers of speech where it is known that one of the classifiers
is more reliable than the rest. Conceptually, our scheme points to the
possibility of improving the best classifier by taking into consideration also
the output from the other classifiers.
The idea could possibly be useful also in speech processing to account
for channel effects in different segments of the acoustic training data, for
changes in genre or style in language modeling text, and in other situations
where the assumption that the training material is temporally homogeneous
is patently false, but the training data may be segmented into contiguous
portions within which some homogeneity may be reasonable to assume.
Interestingly, the celebrated statistical problem of analysis of variance
under normality is precisely a special case of our general scheme but without
the burden of the normal assumption. We shall therefore provide a conve-
nient mathematical framework formulated in terms of "reference" data or
'Department of Mathematics, University of Maryland, College Park, MD 20742,

U.S.A (bnk@math.umd.edu).
tDepartment of Mathematics & Statistics, University of Cyprus, P.O. Box 20537,
Nicosia 1678, Cyprus (fokianos@ucy.ac.cy).
271
272 BENJAMIN KEDEM AND KONSTANTINOS FOKIANOS
their distribution and deviations from them in some sense . The theory will
be illustrated in terms of autoregressive signals akin to speech.
The present formulation of the general scheme follows closely the re-
cent development in Fokianos, et al. (2001) which extends Fokianos, et, al.
(1998), and Qin and Zhang (1997) . Qin and Lawless (1994) is the prede-
cessor to all this work . Related references dealing with more general tilting
or bias are the pioneering papers of Vardi (1982), (1986) .
2. Mathematical formulation of source combination. In our
formalism, "sources" are identified with "dat a" . Deviations are formu-
lated in terms of deviations from a reference distribution. Thus, a data set
deviates from a reference set in the sense that its distribution is a distortion
of a reference distribution.
To motivate this, consider the classical one-way analysis of variance
with m = q + 1 independent normal random samples,
Xql, ,xqn'l rv gq(x)

Xml, ,X mn", rvgm(x)
where gj(x) is the probability density of N(Jlj , (12), j = 1, ..., m . Then,

holding gm(x) as a reference distribution, we can see that
gj(X)
(1) -(-)
gm X
= exp(aj + (3jx), j = 1, ..., q
where
{3. - Jlj - Jlm j = 1, ... , q

J - (12 '
It follows that the test Ho : JlI = . . . = Jlm is equivalent to Ho : {31 = .. . =

{3q = o. Clearly (3j = 0 implies aj = 0, j = 1, ..., q.
This sets the stage for the following generalization. With 9 == gm
denoting the reference distribution, we define deviations from 9 by the
exponential tilt,
(2) gj(X) = exp(aj + (3jh(x))g(x), j = 1, ..., q

where aj depends on (3j, and h(x) is a known function. The data set
Xj = (Xjl , ..., XjnJ' corresponding to gj deviates from the reference data
set Xm = (Xml, ..., Xmn"')' corresponding to 9 == gm in the sense of (2) .
Our goal is to estimate 9 and all the aj and {3j from the combined data
xj , ... , X q , X m .
SEMIPARAMETRI C FILTERING IN SPEECH PRO CESSING 273
Expression (2) is what we mean here by filtering. It really is an oper-

at ion applied to g to produce its filtered versions gj , j = 1, ..., q.
Th e combined data set from th e m samples is th e vector t ,
where Xj = (Xjl ' ..., XjnJ' is the j th sample of length nj, and n = nl +
' " + nq + nm ·
Th e statistical semipara met ric est imat ion/ testing problems using the
com bined data tare.
1. Nonparametric est imat ion of G(x) , th e cdf correspondin g to g(x).
2. Estimation of the par ameters 0: = (0:1, ..., O:q)' , {3 = ({31, ... , {3q)' , and
the study of th e large sample properties of the est imators.
3. Testing of the hypothesis Ho : {31 = .. . = {3q = O.
Evidently, the general const ruct ion does not require normality or even
symmetry of the distributions, the variances need not be th e same, and the
model does not require knowledge of the reference distribution . The main
assumption is th e form of the distortion of the reference distribution.
2.1. Estimation and large sample results. A maximum likelihood
estimator of G(x) can be obt ained by maximizing the likelihood over the
class of ste p cdf's with jumps at the observed values t1 , ..., tn' Accordingl y,
if Pi = dG(td , i = 1, .., n , t he likelihood becomes,
n nl nq
(3) £ (0:,{3, G) = IT Pi IT exp(O:I + !it h(XIj )) ' " IT exp(O:q + {3qh(Xqj )).
i= 1 j=1 j=1
We follow a profiling procedur e whereby first we express each Pi in terms

of 0:, {3 and then we subst itute the Pi back into t he likelihood to produce a
function of 0:, {3 only. When 0:, (3 are fixed, (3) is maximized by maximizin g
only th e th e product term flZ:
1 Pi , subject to the m constraints
n n n
LPi = 1, LPd w 1(ti) - 1] = 0, ..., LPdwq(ti) - 1] = 0
i=1 i=1 i= 1
where the summation is over all the ti and
Wj(t) = exp(O:j + {3j h(t )), j = 1, ..., q.

We have
(4)
where Pj = nj/n m, j = 1, ..., q, and t he value of t he profile log-likelihood

up to a const ant as a function of 0:, {3 only is,
n
l = - L log]l + P1W1(ti) + ... + PqWq(ti)]
i=1
(5)
nt nq
+ L[a1 + tJ1h(X1j)] + ... + L[aq + tJqh(xqj)].
j=1 j=l
The score equations for j = 1, ..., q, are therefore,
~=-:t pjwj(td +nj=O
8aj i=l 1 + P1 W1(ti) + ... + pqwq(td
(6)
~= _ ~ pjh(ti)wj(ti) + ~ h(Xji) = o.
8tJj 6 1 + P1 W1(ti)+ ...+ PqWq(ti) 6
The solution of the score equations gives the maximum likelihood estima-
tors &, (3, and consequently by substitution also the estimates
(7)
and therefore, the estimate of G(t) from the combined data is
It can be shown that the estimators &, {3, are asymptotically normal,
o)
(9) Vii ( &-a
(3 _ f3 ~ N(O,~)
0
as n -+ 00 . Here ao and f3 0 denote the true parameters and ~ = S-lYS-1 ,

where the matrices Sand Yare defined in the appendix.
2.2. Hypothesis testing. We are now in a position to test the hy-
pothesis Ho : f3 = 0 that all the m populations are equidistributed.
We shall use the following notation for the moments of h(t) with re-
spect to the reference distribution:
E(t k) == J hk(t)dG(t)
Var(t) == E(e) - E 2(t) .
2.2.1. The Xl test. Under Ho : f3 = O-so that all the moments of
h(t) are taken with respect to g-consider the q x q matrix All whose jth
diagonal element is
pj[1 + Lk-lj Pk]

[1 + Lk=l PkF
SEMIPARAMETRIC FILTERING IN SPEECH PROCESSING 275
and ot herwise for j =f:.l, th e jl element is

-PjPj l
For m = 2,q = 1, All reduces to a scalar pI/(l + pt}2. For m = 3,q = 2,
where
and the eigenvalues of the matrix on the right are 1,1 + PI + P2.
The elements are bounded by 1 and the matrix is nonsingular,
IA 11 I = [1
ITk-1 Pk
+ ",q ]m > 0
6k=1 Pk
and can be used to represent 8 ,
8 _ ( 1 E(t)) A
E(t) E(e ) 0 11
with 0 denoting the Kronecker product . It follows that 8 is nonsingular ,
and,
2
8- 1 1 (E (t ) -1E (t) ) 0 A } .
= Var (t) - E (t ) J
On the other hand , V is singular,
V=Var(t) ( ~ ~ll)
as is
2(t)
~ = 8- 1V8- I = _1_
,u
(E - E (t ) )
1 0
A 1
11 ·
V ar(t ) -E(t)
Since Au is nonsingular we have from (9),
(10)
It follows under Ho : (3 = 0
A' A
(11) Xl = nVar(t){3 A l1 {3
is approximately distributed as X2 (q), and Ho can be rejected for large

A' A
values of nVar(t){3 A l1 {3.
In practice, V ar(t) needed for Xl, and defined above as the variance
of h(t) (not of t unless h(t) = t), is estimated from
2.2.2. A power study. In Fokianos et al. (2001) the power of Xl

defined in (11) was compared via a computer simulation with the power of
the two sample t and Wilcoxson rank sum tests for m = 2, and with the F
and Kruskal-Wallis tests for m = 3.
The simulation study shows that the Xl-test is more powerful than the
t-test in lognormal and gamma cases, meaning that departures from the
classical normal and variance equality assumptions can weaken the t-test
considerably. In addition, the Xl test dominated the Wilcoxon rank sum
and Kruskal-Wallis tests in all the cases considered . Interestingly, under
normality and variance equality the Xl-test is not dominated by the t-test
and F tests.
2.2.3. Testing the linear hypothesis. We can further test the gen-
eral linear hypothesis HO = c where H is a p x 2q predetermined matrix
of rank p (p < 2q), 0 = (al, . . . ,aq,{3I, . .. ,{3q)', and c is a vector in RP.
Then, using (9), we have under the hypothesis
vn(HO - c) :::} N(O, H~H')

Thus, the random variable
(12)
has an asymptotic chi-square distribution with p degrees of freedom pro-

vided the inverse exists (Sen and Singer 1993 p. 239). A consistent estima-
tor of ~ can be obtained by replacing all the parameters by their maximum
likelihood estimates.
Note that in general the results obtained from (11) and (12) are dif-
ferent, since in (11) we substitute the exact value {3 = 0 in ~, while (12)
requires the maximum likelihood estimate of () instead.
3. Illustration of the theory. We illustrate the semiparametric ap-
proach in terms of first order autoregressive (AR(l)) time series. Exten-
sions to more complex models are straightforward. We have to distinguish
between marginal and conditional models.
3.1. Marginal analysis. Suppose m = 3 and that the data come

from
(13)
where qJi is a parameter such that IcPi I < 1 for i = 1,2,3, and {Ed is a
sequence of independent normal random variables with mean 0 and vari-
ance 0'2. Then it is well know that for sufficiently large t, X; is normally
distributed with mean 0 and variance 0'2/1 - cPr, that is
(14) li(xd =
- cPr exp (1
VI.ji.;- - cPr 2)
---2-Xt , i = 1,2,3.
a 21T a
With this (2) becomes,
(15)
where
D:i =
1'2 [1 -cP
log 1 - cP~
2
]
,
(16)
(3t. -- cPr - cP~
20'2 .
The last three equations show that the exponential distortion model (2)
holds for Gaussian AR(I) data with h(x) = x 2 . Thus we expect our method
to produce consistent estimates for independent samples from sufficiently
long time series. We show that this is indeed the case by resorting to the
following limited simulation study.
We assume throughout that Et follows the normal distribution with
mean 0 and variance 1, and that N, denotes the length of time series
i . Practically independent samples were obtained by sampling at random
from long time series. Each entry in the Table 1 was obtained from 100
independent time series.
In Table 1 the reference cP3 = 0.1 is rather small meaning that the
reference signal is close to being white noise. The table shows that there is
a good agreement between the true parameter values and their estimates.
To demonstrate the asymptotic normality, Figure 1 displays the Q-Q plots
of the estimators when N 1 = N 2 = N 3 = 200. The plots indicate that the
asymptotic distribution of the estimators is close to normal.
3.2. Conditional analysis. The exponential tilt model (2) also holds
for the conditional one-step transition probability density function of the
AR(I) model (13). First observe that the conditional density of X; given
X t - 1 is given by
(17) i = 1,2,3.
TABLE 1
Parameters estimates and their standard errors for different sample sizes for model
(15). The data were gen erated according to (13) with <PI = -0.5, <P2 = - 0.3, and
<P3 = 0.1, so that 01 = -0.139, {31 = 0.12, 0 2 = -0.042, {32 = 0.04.
Sample Sizes I Estimated Parameters and Standard Errors I

N1 N2 N3 ih /31 0:2 /32
200 200 200 -0.120 0.118 -0.021 0.034
0.067 0.059 0.073 0.071
200 500 200 -0.133 0.129 - 0.037 0.049
0.075 0.067 0.056 0.055
300 500 200 - 0.136 0.131 - 0.038 0.048
0.067 0.063 0.069 0.067
(1) (2)
d
~
0
,;
,;
~
~
N
9
0
9
·2 -1 -2 -1
(3) (4)
0
....
9 51
,;
'"
0
9 ~
-: g
0;- ,;
0 0
-2 -1 -2 -1 0
FIG. 1. Q-Q plots of the parameter estim ates under model (15) . Th e data were
generated according to (13) with <PI = -0.5, <P2 = -0.3, <P3 = 0.1, and N, = N2 =
N3 = 200. Th e numb er of runs is 100. (1) ch (2) 131 (3) 0:2 (4) 132 .
Put h (xt \xt-d for the reference distribution. Then a simple calculat ion
shows that
(18) 1og [f!i (Xt IXt- d ] _ ¢i - ¢3 + ¢5 - ¢; 2

(I
3 Xt Xt- 1
) - (7
2 XtXt- 1 2(72 Xt- I'
for i = 1,2 . Thus, we are led to t he following density ratio model
(19) 1og [ f!i (Xt

(I )
IXt- d ]
= ai
C
+
{3c X X C 2
i t t- 1 + "Ii Xt- 1
3 Xt Xt- I
for i = 1,2. Comparison of expressions (18) and (19) reveals th at if th e

data are generated according to th e Gaussian autoregressive process (13),
then
af = 0
{3c = ¢i - ¢3
t (72
( 20) C ¢5 - ¢;
"Ii = 2(72 '
for i = 1,2. A theory similar to the marginal case can be developed for th e
condit ional case as well, but this requires further resear ch.
4. Summary. We have out lined a method for combining data from
several sources. This can be used in testing the similarity of m signals given
a reference signal, and in th e estimat ion of the margin al distribution of the
reference signal from the combined dat a as in (8). Th e mathematical devel-
opment assumes exponential distortions or deviations of the form (2), but
the t heory can be developed for other types of distortions. The analysis was
illustrated in terms of Gaussian AR(I ) data where h( x ) = x 2 . In general ,
h(x) is not known and must be estimated. The choice/ estimat ion of h( x )
can be approached in severa l ways. One possibility is to approximate h(x)
by a polynomial or by B-splines. Anoth er possibility is to employ some
type of kernel estimation applied to the log-ratio of prob ability densities.
For skewed geophysical dat a h(x) = log(x) or even h(x ) = log2(x ) may be
helpful.
APPENDIX
We prove the asserted asymptotic normality (9). First define
'l = (a:1,...,a:q'a~1 "" a~q)'

Then it is easy to see E ['ll (a , 13)] = O. To obtain th e score second moments
=
it is convenient to define Pm 1, W m (t) 1, =
Ej(t) =Jh(t )wj (t) dG (t ), V arj (t) =Jh 2(t)
wj(t )dG (t ) - EJ(t)
and,
Ao(j ,j') === JWj(t~j,(t)dG(~\

+ PkWk t
Jh~t)W~~Wj'(t)d~~t)
1 k=l
A1(j,j') ==
+ PkWk
Jh2(t)WfYWj'(t)df~t)
k=l
A 2(j,j') ==
+ 1 k=l PkWk t
for j, jf = 1, ..., q. Then, the entries in
V == Var [In Vl(a ,,B)]

are ,
n
(",ucx)
81 .) = 1 +~l Pk [Ao(j,j) - r=l
.!..Var
k=l
f>rA6(j,r)]
-1 C ov (81 8l) 1 +""q

~,~
PjPj' [A (J,. J'f) - ~
= L.J Pr A (J,. r )A ('f
0J ,r )] 0 0
n ucx) ucx)' L..-k=l Pk r=l
~Cov (~/ ~J = 1+ iLl Pk [Ao(j,j)Ej(t) - ~prAo(j,r)Adj, r)]

1 (81 8l) =
-Cov ~''''(3
PjPj'
2: q
[ . .f
Ao(],J )Ej'(t
)
n UCXj U j ' 1+ k=l Pk
- ~prAo(j,r)Al(jf,r)]
.!..var(!(3l) =
n U ) 1+
~l k=l Pk
[-A2(j,j)+2A1(j ,j)Ej(t)
- fprAi(j ,r)]
r=l
+
1+
-fJ
k=l Pk
Varj(t)
1 (81"'(3 .' "'(38l)

' = 1 ""q
PjPj' [ 2( . 'f) A ( . 'f)( j () E ())
-Cov
n U ) U i' + L..-k=l Pk -A J,J + J,J E t + i' t 1
- ~prAl(j,r)Al(jf,r)].
Next , as n -. 00 ,
1
--VV fl(a,{3) -. S
n
where S is a 2q x 2q matrix with entries corresponding to j , j' = 1, ..., q,
_.!.. o2l
n ocx;
-. Pj
1 + 2:%=1 Pk
J + 2:t
+ 2:%=1
[1
1
h pkWk(t)]Wj(t) dG(t)
PkWk(t)
The entries are obtained by a repeated application of the facts JdG(t) =

1 and Jwj(t)dG(t) = 1, j = 1, ..., q. It should be noted that due to
profiling, the matrix S is not the usual information matrix although it
plays a similar role.
Thus , when (2) holds with true parameters ao, f30 , it follows under
regularity conditions 0 , j3 are both consistent and asymptotically normal
(See Sen and Singer 1993, Ch. 5),
(A.l) vn ( 0 - ao )
j3 _ f3
0
:::} N(0, :E)
where E = S-lVS- 1 .
Acknowledgments. Research supported by NASA contract
NAG52783. The authors are very grateful to the referee for a number of
helpful suggestions which were incorporated in the paper.
REFERENCES
[1] FOKIANOS K. , B. KEDEM , J . QIN , AND D.A. SHORT (2001), "A semiparametric
approach to the one-way layout ," Technometrics, 43 , 56-64.
[2] FOKIANOS K. , B. KEDEM , J . QIN, J. HAFERMAN , AND D.A. SHORT (1998) , "On
Combining Instruments." Journal of Applied Meteorology, 37, 220-226 .
[3] QIN J . AND J .F . LAWLESS (1994), "Empirical likelihood and gener al estimating
equations," The Annals of Statistics, 22, 300-325.
[4] QIN J . AND B. ZHANG (1997), "A goodness of fit test for logistic regression models
based on case-control data." in Biometrika, 84, 609-618.
[5] SEN P .K. AND J .M. SINGER (1993) , Large Sampl e Methods in Statistics, An In-
troduction With Applications, London : Chapman & Hall.
[6] VARD! Y. (1982), "Nonparamet ric estimation in th e presence of length bias ,"
Annals of Statistics, 10, 616-620.
[7] VARD! Y. (1986) , "Empirical distributions in selection bias models," Annals of
Statistics, 13, 178-203 .
LIST OF WORKSHOP PARTICIPANTS
• Abeer Alwan, Department of Electrical Engineering , University of

California, Los Angeles
• Steven Abney, AT&T Labs-Research
• Joan Bachenko , Linguistech Technologies
• J ames Baker
• Marian Barry
• Jerome Bellegarda, Spoken Language Group, Apple Computer,
Inc.
• Julian E. Besag, Department of Statistics, University of Washing-
ton
• Peter Bickel, Department of Statistics, University of California,
Berkeley
• Jeffrey A. Bilmes, Department of Electrical Engineering, Univer-
sityof
Washington
• Daniel Boley, Department of Computer Science & Engineering,
University of Minnesota
• Herve Bourlard, Daile Molle Institute for Perceptual Artificial In-
telligence (IDIAP), Swiss Federal Institute of Technology at Lau-
sanne (EPFL)
• Mireille Boutin, School of Mathematics, University of Minnesota
• J amylle Carter, Institute for Mathematics and its Applications
• David Castanon, ENG Electrical & Computer Engineering, Boston
University
• Olivier Catoni, Probabilites et Modeles Aleatoires , C.N.R.S.
• Eugene Charniak, Department of Computer Science, Brown Uni-
versity
• Li-Tien Cheng , Institute for Mathematics and its Applications
• Zhiyi Chi, Department of Statistics, University of Chicago
• Jordan Cohen, Voice Signal Technologies, Inc.
• Michael Collins, AT&T Labs-Research
• Zoran Cvetkovic, Information Sciences Research, AT&T Shannon
Laboratory
• Akira Date, Advanced Brain Signal Processing Lab., Riken Brain
Science Institute
• Li Deng, Microsoft Research
• Mukund Deshpande, Department of Computer Science, University
of Minnesota
• Fred Dulles, Institute for Mathematics and its Applications
283
284 LIST OF WORKSHOP PARTICIPANTS
• Jason Eisner, Department of Computer Science, University of

Rochester
• Selim Esedoglu, Institute for Mathematics and its Applications
• Hollis Fitch, Institute for Defense Analysis
• Paul Garrett, School of Mathematics, University of Minnesota
• Stuart Geman , Applied Mathematics, Brown University
• Charles Geyer, School of Statistics, University of Minnesota
• Larry Gillick, Vice President of Research, Dragon Systems, Inc.
• Maria Gini, Computer Science and Engineering, University of Min-
nesota
• Herb Gish, BBN Systems and Technologies
• James Glass, Laboratory for Computer Science Massachusetts In-
stitute of Technology
• Fikri Goksu, Department of Electrical Engineering , University of
Minnesota
• Ramesh A. Gopinath, Speech Transcription Algoithms , HLT De-
partment, IBM T .J. Watson Research Center
• Mitch Helle-Morrissey, Department of Research, Lernout and
Hauspie
• Wendy J . Holmes, 20/20 Speech Limited
• Kirk Jackson, DIRNSA National Security Agency
• Frederick Jelinek, Center for Language, and Speech Processing,
• Chuanshu Ji , Department of Statistics, University of North Car-
olina at Chapel Hill
• Mark Johnson, Department of Cognitive and Linguistic Sciences,
Brown University
• Benjamin Kedem, Department of Mathematics, University of
Maryland at College Park
• Irving Kessler, IDA-Center for Communications Research
• Sanjeev Khudanpur, Center for Language and Speech Processing,
• Seongjai Kim, Department of Mathematics, University of Ken-
tucky
• Dietrich Klakow, Lehrstuhl ff Sprachsignalverarbeitung, Univer-
sitat des Saarlandes
• Christopher Lang, Indiana University Southeast
• Chin-Hui Lee, Dialogue Systems Research, Lucent Technologies
• Lillian Lee, Department of Computer Science, Cornell University
• Zhijun Liu, Department of Mathematics and Statistics, Mississippi
State University
• Elizabeth Lovance, Linguistic Technology
LIST OF WORKSHOP PARTICIPANTS 285
• Michael Macon, Center for Spoken Language Understanding, Ore-

gon Graduate Institute of Science & Tech.
• Martin Maiers, Department of Computer Science, University of
Minnesota
• Christopher Manning, Department of Computer Science, Stanford
University
• David McKoskey, Research/Development, Linguistic Technologies,
Inc.
• Dan Melamed, Westgroup
• Willard Miller, Institute for Mathematics and its Applications
• Mehryar Mohri, AT&T Labs-Research
• Alejandro Murua, Department of Statistics, University of Wash-
ington
• Partha Niyogi, Bell Laboratories, Lucent Technologies
• Alexei Novikov, Applied & Computational Mathematics, Califor-
nia Institute of Technology
• Peter Olver, School of Mathematics, University of Minnesota
• Mari Ostendorf, Department of Electrical Engineering , University
of Washington
• Sergey Pakhomov, ILLASL (Linguistics Program), University of
Minnesota
• Fernando Pereira, Whiz Bang! Labs
• Harry Printz, IBM T .J . Watson Research Center
• Yingyong Qi, Qualcomm Inc.
• Jianliang Qian, Institute for Mathematics and its Applications
• Kashif Riaz, Strategic Development, West Group
• Stefan Riezler, Institut fur Maschinelle Sprachverarbeitung, Uni-
versitat Stuttgart
• Ya'acov Ritov, Department of Statistics, The Hebrew University
of Jerusalem
• Roni Rosenfeld, School of Computer Science, Carnegie Mellon Uni-
versity
• Yoshinori Sagisaka, ATR Interpreting Telecommunications Res.
Lab
• Ananth Sankar , Nuance Communications
• Lawrence Saul, AT&T Labs-Research
• Guergana Savova, LTI/L&H, Lernout and Hauspie
• Judith Schlesinger, IDA Center for Computing Sciences
• Michael Schonwetter,
• Jackie Shen, School of Mathematics, University of Minnesota
• Kemal Sonmez, SRI International
• Richard Stern, Department of Engineering , Carnegie Mellon Uni-
versity
286 LIST OF WORKSHOP PARTICIPANTS
• Andreas Stolcke, SRI International

• Paul Thompson, West Group
• Jean-Philippe Vert, DMA, Ecole Normale Superieure
• Shaojun Wang, Center for Automated Learning and Discovery
Carnegie Mellon University
• Ken Williams, University of Minnesota
• Jack Xin, Department of Mathematics, University of Texas at
Austin
• Xiangsheng Xu, Department of Mathematics and Statistics, Mis-
sissippi State University
• Steve Young, Department of Engineering, Cambridge University
1999-2000 Reactive Flows and Transport Phenomena
2000-2001 Mathematics in Multimedia
2001-2002 Mathematics in the Geosciences
2002-2003 Optimization
2003-2004 Probability and Statistics in Complex Systems: Genomics,
Networks , and Financial Engineering
2004-2005 Mathematics of Materials and Macromolecules: Multiple Scales,
Disorder , and Singularities
2005-2006 Imaging
IMA SUMMER PROGRAMS
1987 Robotics
1988 Signal Processing
1989 Robust Statistics and Diagnostics
1990 Radar and Sonar (June 18-29)
New Directions in Time Series Analysis (July 2-27)
1991 Semiconductors
1992 Environmental Studies : Mathematical, Computational, and
Statistical Analysis
1993 Modeling, Mesh Generation, and Adaptive Numerical Methods
for Partial Differential Equations
1994 Molecular Biology
1995 Large Scale Optimizations with Applications to Inverse Problems,
Optimal Control and Design, and Molecular and Structural
Optimization
1996 Emerging Applications of Number Theory (July 15-26)
Theory of Random Sets (August 22-24)
1997 Statistics in the Health Sciences
1998 Coding and Cryptography (July 6-18)
Mathematical Modeling in Industry (July 22-31)
1999 Codes, Systems, and Graphical Models (August 2-13, 1999)
2000 Mathematical Modeling in Industry: A Workshop for Graduate
Students (July 19-28)
2001 Geometric Methods in Inverse Problems and PDE Control
(July 16-27)
2002 Special Functions in th e Digital Age (July 22-August 2)
2003 Probability and Partial Differential Equations in Modern
Applied Mathematics (July 21-August 1)
2004 n-Categories: Foundations and Applications (June 7-18)
IMA "HOT TOPICS" WORKSHOPS

• Challenges and Opportunities in Genomics: Production, Storage,
Mining and Use, April 24-27 , 1999
• Decision Making Under Uncertainty: Energy and Environment al
Models, July 20-24, 1999
• Analysis and Modeling of Optical Devices, September 9-10 , 1999
• Decision Making under Uncertainty: Assessment of the Reliability
of Mathematical Models, September 16-17, 1999
• Scaling Phenomena in Communication Networks , Octob er 22-24,
1999
• Text Mining, April 17-18, 2000
• Mathematical Challenges in Global Positioning Systems (GPS) ,
August 16-18,2000
• Modeling and Analysis of Noise in Integrated Circuits and Systems,
August 29-30,2000
• Mathematics of the Internet: E-Auction and Markets, December
3-5, 2000
• Analysis and Modeling of Industrial Jetting Processes, January
10-13, 2001
• Special Workshop : Mathematical Opportunities in Large-Scale Net-
work Dynamics , August 6-7, 2001
• Wireless Networks, August 8-102001
• Numeri cal Relativity, June 24-29 , 2002
• Operational Modeling and Biodefense: Problems, Techniques , and
Opportunities, September 28, 2002
• Data-driven Control and Optimization, December 4-6, 2002
• Agent Based Modeling and Simulation, November 3-6, 2003
SPRINGER LECTURE NOTES FROM THE IMA:
Th e Mathematics and Physi cs of Disordered Media

Editors: Barry Hughes and Barry Ninham
(Lecture Notes in Math., Volume 1035, 1983)
Orienting Polymers
Editor: J .L. Ericksen
(Lecture Notes in Math. , Volume 1063, 1984)
New Perspectives in Thermodynamics

Editor: J ames Serrin
(Sprin ger-Verlag, 1986)
Models of Economic Dynamics

Editor: Hugo Sonnenschein
(Lecture Notes in Econ., Volume 264, 1986)
Forthcoming Volume:
Time Series Analysis and Applications to Geophysical Systems

(Stuart Geman, Mark Johnson (Auth.), Mark Johnson, (B-Ok - Xyz) PDF

Uploaded by

Copyright:

Available Formats

(Stuart Geman, Mark Johnson (Auth.), Mark Johnson, (B-Ok - Xyz) PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

(Stuart Geman, Mark Johnson (Auth.), Mark Johnson, (B-Ok - Xyz) PDF

Uploaded by

Copyright:

Available Formats

The IMA Volumes

Springer Science+Business Media, LLC

1982-1983 Statistical and Continuum Approaches to Phase Transition

Continued at the back

Roni Rosenfeld Series Editors:

Library of Congress Cataloging-in-Publication Data

ISBN 978-1-4612-6484-2 ISBN 978-1-4419-9017-4 (eBook)

© 2004 Springer Scienee+Business Media New York

Springer-Verlag is part of Springer Science+Business Media

This IMA Volume in Mathematics and its Applications

MATHEMATICAL FOUNDATIONS OF SPEECH

contains papers presented at two successful one-week workshops : Math-

The importance of speech and language technologies continues to grow

Three issues in modern language modeling. . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Stochastic analysis of Structured Language

Latent semantic language modeling for speech

Prosody modeling for automatic speech

Switching dynamic system models for speech

Segmental HMMS: Modeling dynamics and

Modelling graph-based observation spaces

Towards robust and adapt ive speech

Graphical models and automat ic speech

An introduction to Markov chain Monte

Semiparametric filtering in speech processing " 271

List of workshop participants 283

1. Introduction. Computational linguistics studies the computat-

'Department of Cognitive and Linguistic Sciences, Brown Univers ity, Providence,

M. Johnson et al. (eds.), Mathematical Foundations of Speech and Language Processing

are Markov-like machines consisting of a set of states and a set of allowed

In effect, there is a dilemma of coverage. A grammar rich enough to

2. Grammars and languages. The formal framework, whether used

grammars without epsilon productions, i.e., f3 E (NUT)+, as this simplifies

gr ammars. The universal recognition problem for 9 t akes as input a string

w ; 'I/J is called a parse tree of w (with respect to G) . In what follows, we

In this example, Y('l/Jd = rice grows and Y('l/J2) = wheat grows

There is a close relati onship between linear gr amm ar s and finite-state

As rem arked earlier, context-sensitive and unrestricted rewrite gra m-

3. Probability and statistics. Obviously broad coverage is desir-

(1) p(A --+ o) = 1 .

Recall that We is the set of parse trees generated by G (see §2). If G is

from S. It is, then, straightforward to use p to define a probability P on

An equivalent formulation is through the associated three-state (8, A,

.81 .19 .00 )

3.1.2. Inference. Th e problem is to est imate the tr ansition proba-

Th e maximum likelihood estimate is, sensibly, th e relative frequency est i-

L(P i w) = L P( 'IjJ) = L IT p(A -t a)! (A--+Q;!/J) .

To get the maximum likelihood equation, take logarithms, introduce La-

Introduce E p [' ]' meaning expectation under the probability on P induced

We can't solve, directly, for P, but (6) suggests an iterative approach

Evidently, Pt = Pt+! if and only if we have found a solution to the likeli-

(8) arg max P(1/I) .

This is the maximum a posteriori (MAP) estimate of 1/I-obviously it min-

Then , given AdA) and Vk(A), for A E N and k = 1,2, . .. , n - 2, compute

Ak+l(A) = arg max p(B -+ bk+1A)VdB)

3.1.4. Speech recognition. An outstanding application of proba-

era rchy. An obvious candidate model would be a regular grammar , with

(9) p(A --+ a) = 1 .

Following the development in §3.1, we introduce a counting function f(A --+

(10) P('Ij;) = II p(A --+ a)!(A-+ OI ;1/!) .