Ok What Is An Algorithm
Ok What Is An Algorithm
Yuri Gurevich
Microsoft Research
Technical Report
MSR-TR-2011-116, July 2011
Microsoft Research
Microsoft Corporation
One Microsoft Way
Redmond, WA 98052
0 Preamble
We present a two-part exposition on the notion of algorithm and foun-
dational analyses of computation. The first part is below, and the sec-
ond is here: http://research.microsoft.com/en-us/um/people/gurevich/
Opera/210.pdf. This preamble was added in February 2012.
In the first part, after a short introduction, we clarify some common
misconceptions related to Turing’s analysis of computation, consider whether
the notion of algorithm can be rigorously defined in full generality, discuss
what kind of entities algorithms are, and examine two approaches to the
title problem in the literature. The first part appeared in “SOFSEM 2012:
Theory and Practice of Computer Science,” Springer LNCS 7147, 31–42.
The second part is devoted entirely to fundamental analyses of computa-
tions in the literature, by Turing and by others including this author. It will
appear in the proceedings of “Turing Centenary Conference, CiE 2012: How
the World Computes,” to be held in Cambridge, England, in June 2012.
1
What is an algorithm?
Yuri Gurevich
Microsoft Research
1 Introduction
Two articles in a recent book [10] present two approaches to the title problem
and offer different answers. Article [19] presents an approach developed by
Yiannis Moschovakis. “A characteristic feature of this approach is the adop-
tion of a very abstract notion of algorithm that takes recursion as a primitive
operation and is so wide as to admit ‘non-implementable’ algorithms” [19,
p.87]. The article starts thus.
2
by Wilfried Sieg. In article [22], Sieg uses Gandy machines to “dispense with
[Church’s and Turing’s] theses”. The article starts thus.
Acknowledgments
Many thanks to Andreas Blass for numerous illuminating discussions, and to
Serge Grigorieff and Oron Shagrir for useful suggestion.
3
2 Can the notion of algorithm be rigorously
defined?
Two articles [19] and [22], mentioned in §1, give different answers to the
title question. The two answers are not at all equivalent. A question arises
whether the notion of algorithm can be defined at all. The answer is yes and
no. Let us explain.
4
(and Gödel and other experts) had in mind only sequential algorithms, which
we believe they did. The axiomatic definition was extended to synchronous
parallel algorithms in [3] and to interactive sequential algorithms in [6, 7].
5
length on the Turing tape. More generally, the functions computed by ruler-
and-compass algorithms are not Turing computable. And let us emphasize
that ruler and compass were practical tools in ancient Greece and that a
number of ruler-and-compass algorithms were practical algorithms.
It is common in mathematics to consider algorithms that work on abstract
objects. The functions computed by these algorithms may not be Turing
computable. One example is Gaussian elimination. Here is another example:
a bisection algorithm that, given a real ε > 0 and a continuous function f on
a real segment [a, b] with f (a) < 0 < f (b), computes a point c ∈ [a, b] with
|f (c)| < ε.
while |f ((a + b)/2)| ≥ ε do
if f ((a + b)/2) < 0 then a := (a + b)/2 else b := (a + b)/2
c := (a + b)/2
One can argue that these functions are not truly computable, that in
practice we can only approximate them. But then there are analog computers
in practical use that work in real time and compute functions that are not
Turing computable.
Of course Turing would not be surprised by our examples. He explicitly
restricted his analysis to “symbolic” (symbol-pushing, digital) algorithms.
He implicitly restricted his analysis to sequential algorithms, essentially the
only algorithms in his time. It is interesting that it turned out easier to
axiomatize all sequential algorithms [12], whether symbolic or not, including
the ruler-and-compass algorithms, Gaussian elimination and the bisection
algorithm (but excluding analog algorithms which are not sequential in our
sense).
What about quantum algorithms? Do they compute functions that are
not Turing computable? Erich Grädel and Antje Nowack checked that the
quantum computing models in the literature can be faithfully simulated by
parallel abstract state machines [11]. And, as we mentioned above, functions
computed by parallel ASMs are Turing computable.
There is also a rather common misunderstanding that Turing defined the
notion of algorithm, albeit restricted to symbolic sequential algorithms. Let
us restrict attention to such algorithms for a moment. Suppose that your
computation model (e.g. a programming language) is Turing complete. Does
it mean that the model allows you to express all algorithms? Not necessarily.
Turing machines simulate faithfully only the input-output behavior of algo-
rithms. But there may be much more to algorithms than their input-output
6
behavior. Turing completeness does not mean algorithmic completeness. It
means only that, for every Turing computable function f , the language allows
you to program an algorithm that computes f .
For illustration consider Turing machines with one tape that may be mul-
tidimensional. The model is obviously Turing complete. On any such ma-
chine, the problem of palindrome recognition requires Θ(n2 / log n) time [2].
But the problem is trivially solvable in linear time on a Turing machine with
two one-dimensional tapes. For a deeper dive into algorithmic completeness,
see [26, §3].
Note “successive states”. Gandy machines work in sequential time. This type
of parallelism is called synchronous. In the rest of this section, parallelism
will by default be synchronous.
Gandy pioneered the use of axioms in the analysis of computation. He
came up with four principles (or constraints, or axioms) satisfied, he claimed,
by all discrete deterministic machines. Contrast this with Turing’s analysis.
While Turing’s analysis was convincing, it is hard to isolate first principles
that, in Turing’s opinion, are satisfied by all symbolic sequential computa-
tions.
Wilfried Sieg adopted Gandy’s approach and reworked Gandy’s axioms;
see [23] and references there.
7
Critical remarks
In a 2002 article [20], Oron Shagrir suggests that “there is an ambiguity re-
garding the types of machines that Gandy was postulating”. He offers three
interpretations: “Gandy machines as physical machines”, “Gandy machines
as finite-physical machines”, and “Gandy machines as a mathematical no-
tion”. Shagrir concludes that none of the three interpretations “provides the
basis for claiming that Gandy characterized finite machine computation.”
This agrees with our own analysis. By the way, for our purposes, there is no
difference between Gandy’s original axioms and Sieg’s versions of the axioms.
So we will just speak about Gandy’s axioms.
8
• How does Gandy’s parallel computation model compare to other
parallel computation models? By now, there are numerous models of
synchronous parallelism in the literature, e.g. parallel random access ma-
chines, circuits, alternating Turing machines, first-order logic with the least
fixed-point operator, and parallel abstract state machines. What are the ad-
vantages, if any, of Gandy’s model over the other models? Neither Gandy
nor Sieg addressed this question. Gandy’s model seems quite awkward for
programming or specifying algorithms.
9
transits from one state to the next until, if ever, it halts or breaks. The very
first postulate in our axiomatizations of sequential and synchronous parallel
algorithms [12, 3] is that the algorithms in question are sequential time.
The question arises what kind of entities states are. In our view, rather
common in computer science, algorithms are not humans or devices; they are
abstract entities. According to the second postulate in the axiomatizations
of sequential and synchronous parallel algorithms, the states are (first-order)
structures, up to isomorphism. This admittedly involves a degree of mathe-
matical modeling and even arbitrariness. A particular form of structures is
used; why this particular form? But this is a minor detail. Structures are
faithful representations of states, and that is all that matters for our purposes.
It is convenient to declare that states are structures, up to isomorphism; but
there is no need to do so.
The point of view that sequential-time algorithms are state transi-
tion systems extends naturally to other classes of algorithms. In particu-
lar, a sequential-time interactive algorithm (until now we considered non-
interactive algorithms) is a state transition system where a state transition
may be accompanied by sending and receiving messages. A distributed al-
gorithm is an ensemble of communicating sequential-time interactive algo-
rithms.
exp(x + 1, 0) = 1
(
0 if x = 0
exp(x, y + 1) =
x × exp(x, y) if x > 0
10
again, obtaining additionally exp(x, 1) = x for all x, and so on. After ω steps
(where ω is the first infinite ordinal), you reach a fixed point; now exp(x, y)
is defined for all x, y except x = y = 0. Often the evolution toward the fixed
point involves not only the function that you are computing but also some
auxiliary functions.
In 1934, Gödel formulated a recursion-based calculus of (in general par-
tial) numerical functions. Gödel’s calculus can be seen as a specification
language where a specification of a function f is a system of recursive equa-
tions that, taking into account some global conventions, suggests a particular
(possibly inefficient) way to compute f . Church’s thesis (extended to partial
functions by Kleene) asserts that every “effectively calculable”, that is com-
putable by an algorithm, function on natural numbers is programmable in
Gödel’s calculus.
Recursive specification of functions has much appeal. It is declarative and
abstracts from computation details. It is often concise. There has been much
progress since the 1930s. Logicians developed recursion theory. McCarthy
created a functional (that is recursion-based) programming language LISP,
and many other functional languages followed.
The key ideas of Moschovakis’s approach appear already in the 1984 ar-
ticle [16] that seems to be the very first publication on the subject.
11
He gives recursive equations for the mergesort algorithm on a set X and
proceeds to prove that at most n · log2 (n) comparisons are required to sort
n elements.
Moschovakis’s views have been evolving.
Critical remarks
• Recursors vs. algorithms We think that Moschovakis was right the
first time around, in [16, p. 295] when he refrained from identifying (what
he later called) recursors with algorithm “because the time-honored intuitive
concept of algorithm carries many linguistic and intensional connotations”
which are contrary to such identification.
Consider the system of two recursive equations (and thus a recursor) for
the exponentiation in the beginning of this section. Is it an algorithm or not?
The recursor certainly looks like an algorithm, and in many functional pro-
gramming languages, this recursor would be a legitimate program (modulo
12
syntactic details of no importance to us here). Typically exp(xy ) would be
interpreted as a function call and, for example, the evaluation of 32 would
proceed thus:
32 = 3 · 31 = 3 · (3 · 30 ) = 3 · (3 · 1)) = 3 · 3 = 9.
13
• Recursion is but one aspect of an algorithm The theory of algo-
rithms does not reduce to recursion. For one thing, there are clever data
structures. For many linear-time algorithms, for example, it is crucially im-
portant that an algorithm does not manipulate large objects directly; instead
it manipulates only pointers to those objects. Such aspects of complexity
analysis seem below the abstraction level of recursors.
T (x, y) ← R(x, y)
T (x, y) ← R(x, z), T (z, y)
U (x, y) ← T (x, y)
V (x, y) ← T (x, y), R(x0 , z 0 ), T (z 0 , y 0 ), ¬T (x0 , y 0 )
C(x, y) ← ¬T (x, y), U (x0 , y 0 ), ¬V (x0 , y 0 )
Explanation. At every step all rules are fired. By the first two rules, the
computation of T proceeds in the usual way. Since the domain is finite, the
computation of T completes after some number k of steps. The pairs of T
are stored in U with a delay of one step, so the computation of U completes
after k + 1 steps. The computation of V is identical to that of U , except that
at the step k + 1, when U is completed, the last batch of pairs from T is not
stored in V . The final rule is idle during the first k steps but on step k + 1
it stores the complement of T into C.
14
• More examples, please. It would be much useful to have more example
of recursors of interest to computer scientists. All current examples of that
sort seem to be present already in the 1984 article [16].
References
[1] Serge Abiteboul and Victor Vianu, “Datalog extensions for database
queries and updates”, J. of Computer and System Sciences 43 (1991)
62–124.
[3] Andreas Blass and Yuri Gurevich, “Abstract state machines capture par-
allel algorithms,” ACM Transactions on Computational Logic 4:4 (2003)
578–651. Correction and extension, same journal, 9:3 (2008) article 19.
[4] Andreas Blass and Yuri Gurevich, “Algorithms vs. machines”, Bull. Eu-
ropean Association for Theoretical Computer Science 77 (2002), 96–118.
[5] Andreas Blass and Yuri Gurevich, “Algorithms: A quest for absolute
definitions”; in Current Trends in Theoretical Computer Science, World
Scientific, 2004, 195–225; also in Church’s Thesis after 70 Years (eds. A.
Olszewski et al.), Ontos Verlag, 2006, 24–57.
[6] Andreas Blass and Yuri Gurevich, “Ordinary interactive small-step algo-
rithms”, ACM Trans. Computational Logic 7:2 (2006) 363–419 (Part I),
plus 8:3 (2007), articles 15 and 16 (Parts II, III).
[7] Andreas Blass, Yuri Gurevich, Dean Rosenzweig, and Benjamin Ross-
man, “Interactive small-step algorithms, Logical Methods in Computer
Science 3:4 (2007), papers 3 and 4 (Part I and Part II).
15
[9] Robin Gandy, “Church’s thesis and principles for mechanisms”, In The
Kleene Symposium (eds. J. Barwise et al.), North-Holland, 1980, 123–
148.
[10] S. Barry Cooper, Benedict Löwe and Andrea Sorbi, editors, “New Com-
putational Paradigms: Changing Conceptions of what is Computable”,
Springer, 2008.
[11] Erich Grädel and Antje Nowack, “Quantum computing and abstract
state machines”, Springer Lecture Notes in Computer Science 2589
(2003), 309–323.
16
[20] Oron Shagrir, “Effective computation by humans and machines”, Minds
and Machines 12 (2002) 221–240.
17