Agile Project Management With SCRUM:

Theory and Practice

Jeff Sutherland, Ph.D.
Chief Technology Officer
jeff.sutherland@computer.org
http://jeffsutherland.com
The Zen of SCRUM
 So simple, anyone can implement it!
 So easy, all can benefit!
 So subtle, few achieve transcendent performance …
 How is it that a project manager does nothing, and achieves
everything?
 Interlocking principles emerge product like jigsaw puzzle.
 Novice removes one piece -- engine never fires on all
cylinders …
 Who can know why?
 ScrumMaster must understand deeply and practice rigorously.
 Only then will team members say, “This experience changed
my life!”
Complex Adaptive Systems (cas)
 Interacting agents respond to stimuli.
 Stimulus-response behavior is defined in terms of rules.
 Agents adapt by changing rules as experience accumulates.
 Agents aggregate into meta-agents whose behavior is emergent.
 How can a collection of dumb things emerge smart system behavior?

Web services? Chaos

Fragmentation
1998 Agents
1995 Components cas
1993 Business Objects Self Organization

1980 Classes
Maamar, Zakaria and Sutherland, Jeff (2000) Toward Intelligent Business
Frozen Objects: Focusing on Techniques to Enhance
1970 Procedures
Business Objects that Exhibit Goal-Oriented Behaviors. Communications of the ACM 40:10:99-101.
 Enterprise Systems are cas
 Business entities are examples of complex adaptive
systems.
 Modification time is on the order of months or years,
roughly time required to change software.
 Automating business processes renders parts of the
business in software.
 Business systems have severely constrained rule
sets, making ideal test bed for cas concepts. 

Sutherland, Jeff and van den Heuvel, Willem-Jan (2002) Developing and integrating
enterprise components and services: Enterprise application integration and complex
adaptive systems. Communications of the ACM 45:10:59-64.
Objects Meet Requirements for
Evolution
 Variation: there is a continuing abundance of different
elements (class libraries).
 Heredity or replication: the elements have the capacity
to create copies or replicas of themselves (inheritance).
 Differential "fitness": the number of copies of an element
that are created in a given time varies, depending on
interactions between the features of that element and
the features of the environment in which it persists
(reuse).

Daniel Dennett. Darwin's Dangerous Idea: Evolution and the Meanings of Life. Simon and
Schuster, 1995, p. 343.
Do Programmers Meet Evolutionary
Requirements?

 Algorithms create movement through design space

– Simple minded repetitive procedure
– Incremental change to adjacent designs
 "Cranes" accelerate movement through design space
– Sex and genetic engineers
– Evolutionary prototyping and smart people
– Emergent architecture

Brooks, Fred. No Silver Bullet: Essence and Accidents of Software Engineering. IEEE
Computer, Vol. 20, No. 4, April 1987, pp. 10-19.
 Change is Imperative:
Wasserman's 7 Factors Driving Change

 Criticality of time to market

 Shift in computing economics
Too many projects fail
 Powerful desktop computers
 Extensive networks and the Web
 Growing availability of object technology
 WIMP (windows, icons, menus, pointers)
 Unpredictability of the waterfall model of software
development

Tony Wasserman. Toward a Discipline of Software Engineering. IEEE Software 13:6:23-31, Nov 1996.
"Why Are Systems Late, Over Budget, Wrong?"
"The Waterfall Methodology!" (Paul Bassett)

Analysis Paralysis
– static modeling overused
– specs are stale baked
Design-from-Scratch
– no generic models
– no standard architectures
Large Project Teams
User Intermediaries
No Early Warning Signals

Bassett, Paul G. Framing Software Reuse: Lessons from the Real World. Yourdon
Press Computing Series, 1997.
 Wicked Problems: Righteous Solutions
Out of a total cost of $37B for the sample set, 75% of [DOD] projects failed or were never used,
and only 2% were used without extensive modification. Jarzombek. The 5th Annual JAWS S3 Proceedings,
1999.

 Wicked problems have no definitive formulation. Each attempt at creating a solution

changes your understanding of the problem.
 Wicked problems have no stopping rule. The problem-solving process ends when
resources are depleted, stakeholders lose interest or political realities change.
 Solutions to wicked problems are not true-or-false, but good-or-bad … getting all
stakeholders to agree that a resolution is "good enough" can be a challenge.
 There is no immediate or ultimate test of a solution to a wicked problem.
 Every implemented solution to a wicked problem has consequences.
 Wicked problems don't have a well-described set of potential solutions. Various
stakeholders have differing views of acceptable solutions.
 Each wicked problem is essentially unique. Part of the art of dealing with wicked
problems is the art of not knowing too early what type of solution to apply.
 Each wicked problem can be considered a symptom of another problem. A wicked
problem is a set of interlocking issues and constraints that change over time,
embedded in a dynamic social context.
 The causes of a wicked problem can be explained in numerous ways.
 The planner (designer) has no right to be wrong.

Rittel, H and Webber M. Dilemmas in a General Theory of Planning. Policy Sciences, Vol. 4. Elsevier, 1973.
Degrace and Hulet's book, Wicked Problems, Righteous Solutions, Prentice Hall, 1990
Software Development is an Empirical
Process

 Ziv's Uncertainty Principle in Software Engineering -

uncertainty is inherent and inevitable in software
development processes and products [Ziv, 1996].
 Humphrey's Requirements Uncertainty Principle - for a
new software system, the requirements will not be
completely known until after the users have used it.
 Wegner's Lemma - it is not possible to completely
specify an interactive system [Wegner, 1995].
Productivity: All at Once Models
 Single Super-Programmer
 Handcuffing two programmers
together
 Brook’s Surgical Team
 Borland Quattro project
 Goldratt’s “The Goal”
 Senge’s systems thinking
 Holland’s complex adaptive systems
Team Size: Development Effort in Months

 The smaller the better.

 491 medium sized projects with 35,000-95,000 SLOC (source
lines of code)
Putnam, Lawrence H. and Myers, Ware. Familiar Metrics Management: Small is Beautiful--
Once Again. IT Metrics Strategis IV:8:12-16, Cutter Information Corp., August 1998.
Team Size: Development Time in Months

 Sweet spot is 5-7 people

 491 medium sized projects with 35,000-95,000 SLOC
(source lines of code)
Putnam, Lawrence H. and Myers, Ware. Familiar Metrics Management: Small is Beautiful--
Once Again. IT Metrics Strategis IV:8:12-16, Cutter Information Corp., August 1998.
Bell Labs Report on most productive project
ever: Borland Quattro for Windows
1,000,000 lines of BWP Industry standard
C++ code
Time in months 31 >50 Jones, Capers. Applied
Software Measurement, Second
Staff 8 >100 Edition. McGraw Hill, 1997.
Function points 77 2
per staff month

 BQW
 James Coplien. Borland Software Craftsmanship: A New
Look at Process, Quality, and Productivity. Proceedings
of the 5th Annual Borland International Conference,
Orlando, 1994.
 One of most remarkable organizations, processes, and
development cultures seen in AT&T Bell Laboratories Pasteur
process research project
 Project management, product management, QA integral to team,
all making technical contributions
 Higher communication saturation than 89% of projects
 More even distribution of workload
 “Anti-schismogenetic” – no cliques
 Highly iterative development
 Strong architectural interaction with implementation
 More time spent in project team meetings than anything else –
several hours a day
 Gerry Weinberg notes that CMM Level 1 and 2 teams need strong
managerial direction. Level 3 paradigm shift is self-directing team.
Borland team was clearly in this category, although not by
commonly accepted criteria.
Team comments on Quattro project

 “We are satisfied by doing real work.”

 “Software is like a plant that grows. You can’t
predict its exact shape, or how big it will grow.”
 “There are no rules for this kind of thing—it’s
never been done before.”

“Evolutionary development is best technically, and it saves time

and money.”
Report of the Defense Science Board Task Force on Military Software. Oct 1987.
History of Iterative and Incremental
Development (IID)
 1956 – Benington’s stagewise model – USAF SAGE
System
 1957 – IBM Service Bureau Corp, Project Mercury, IBM
Federal Systems Devision – Gerry Weinberg
 1960 – Weinberg teaching IID at IBM Systems
Research Institute
 1969 - Earliest published reference to IID:
– Robert Glass. Elementary Level Discussion of
Compiler/Interpreter Writing. ACM Computing Surveys, Mar
1969
Larman, Craig and Basili, Vic. A History of Iterative and Incremental
Development. IEEE Computer, June 2003 (in press)
History of Iterative and Incremental
Development (IID)
 1971 – IBM Federal Systems Division
– Mills, Harlan. Top-down programming in Large Systems. In
Debugging Techniques in Large Systems. Prentice Hall, 1971
 1972 – TRW uses IID on $100M Army Site Defense software
 1975 – First original paper devoted to IID
– Gasili, Vic and Turner, Albert. Iterative Enhancement: A Practical
Technique for Software Development. IEEE Transactions on
Software Engineering. Dec 1975.
 1977-1980 – IBM FSD builds NASA Space Shuttle software in
17 iterations over 31 months, averaging 8 weeks per iteration
– Madden and Rone. Design, Development, Integration: Space Shuttle
Flight Software. Communications of the ACM, Sept 1984.

Larman, Craig and Basili, Vic. A History of Iterative and Incremental Development.
IEEE Computer, June 2003 (in press)
History of Iterative and Incremental
Development (IID)
 1985 – Barry Boehm’s Spiral Model
– Boehm, Barry. A Spiral Model of Software Development and Enhancement.
Proceedings of an International Workshop on Software Process and Software
Environments. March, 1985
 1986 – Brooks, Fred. No Silver Bullet. IEEE Computer, April 1987
– Nothing … has so radically changed my own practice, or its effectiveness [as
incremental development].
 1994 – First SCRUM at Easel Corporation

 1994 – DOD must manage programs using iterative development

– Report of the Defense Science Board Task Force on Acquiring Defense
Software Commercially. June 1994.
 1995 – Microsoft IID published
– McCarthy, Jim. Dynamics of Software Development. Microsoft Press, 1995.
 1996 – Kruchten. A Rational Development Process. Crosstalk. July.
– Origins of RUP
Larman, Craig and Basili, Vic. A History of Iterative and Incremental Development. IEEE Computer, June
2003 (in press)
History of Iterative and Incremental
Development (IID)
 1996 – Kent Beck Chrysler Project
– Origin of XP
 1996 – Larman meets with principal author of DD-STD-2167
– David Maibor expressed regret for the creation of the waterfall-based
standard. He had not learned of incremental development at the time and
based his advice on textbooks and consultants advocating the waterfall
method. With the hindsight of iterative experience, he would recommend IID.
 1997 – Coad and DeLuca rescue Singapore project
– Origin of Feature-Driven Development
 1998 – Standish Group CHAOS Project
– Top reason for massive project failures was waterfall methods. “Research
also indicates that smaller time frames, with delivery of software components
early and often, will increase success rate.
 1999 – Publication of extensive DOD failures
– Out of a total cost of $37B for the sample set, 75% of projects failed or were
never used, and only 2% were used without extensive modification.
Jarzombek. The 5th Annual JAWS S3 Proceedings, 1999.
Larman, Craig and Basili, Vic. A History of Iterative and Incremental Development. IEEE Computer, June
2003 (in press)
History of Iterative and Incremental
Development (IID)
 2001 – 17 process expert “anarchists” meet at
Snow Bird
– Agile Manifesto initiated 100s of books and papers on agile
development
 2001 – MacCormack’s study of key success factors
– MacCormack, Alan. Product-Develoment Practices that Work.
MIT Sloan Management Review 42:2, 2001.
Larman, Craig and Basili, Vic. A History of Iterative and Incremental
Development. IEEE Computer, June 2003 (in press)
Manifesto for Agile Software Development
We are uncovering better ways of developing
software by doing it and helping others do it.

Through this work we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on

the right, we value the items on the left more.
 MacCormack Process Evolution
 Waterfall model – sequential process maintains a
document trail
 Rapid-Prototyping Model – disposable prototype helps
establish customer preference
 Spiral Model – series of prototypes identifies major
risks
 Incremental or Staged Delivery Model – system is
delivered to customer in chunks
 Evolutionary Delivery Model – iterative approach in
which customers test an actual version of the software
MacCormack, Alan. Product-Development Practices That Work: How Internet Companies Build
Software. MIT Sloan Management Review 42:2:75-84, Winter 2001.
 MacCormack Success Factors

 Early release of evolving product design to customers.

 Daily incorporation of new software code and rapid
feedback on design changes
 A team with broad-based experience in shipping
multiple projects
 Major investment in design of product architecture

MacCormack, Alan. Product-Development Practices That Work: How Internet Companies Build
Software. MIT Sloan Management Review 42:2:75-84, Winter 2001.
SCRUM Origins: Takeuchi and Nonaka
Lessons from Fuji-Xerox, Canon, Honda, NEC, Epson, Brother,
3M, Xerox, HP
 Old model – Relay Race
– Speed and flexibility not adequate in today’s market
 New model - Rugby

Takeuchi, Hirotaka and Nonaka, Ikujiro. 1986. The new new product development
game. Harvard Business Review 64:1:137-146 (Jan/Feb), reprint no. 86116.
Moving the SCRUM downfield
Takeuchi and Nonaka Success Factors
 Built-in instability
 Self-organizing project teams
 Overlapping development phases
 “Multilearning”
 Subtle control
 Organizational transfer of learning
“These characteristics are like pieces of a jigsaw puzzle. Each element, by itself,
does not bring about speed and flexibility. But taken as a whole, the characteristics
can product a powerful new set of dynamics that will make a difference.”
Factor 1: Built-in instability
 Top management kicks off development process by signaling
broad goal.
 Project team is offered extremely challenging goals with wide
measure of freedom.
– Example: Fuji-Xerox gave FX-3500 project team two years to come
up with a copier that cut costs in half
 Top management creates an element of tension in the project
team through challenging requirements with wide freedom to
achieve strategic objective.
– Honda Executive: “It’s like putting the team members on the second
floor, removing the ladder, and telling them to jump or else. I believe
creativity is born by pushing people against the wall and pressuring
them almost to the extreme.”
Factor 2: Self-organizing project teams
 A project team takes on a self-organizing character as it is driven
to a state of “zero information” – where prior knowledge does not
apply.
 Left to stew, the process begins to create its own dynamic order.
 The project team begins to operate like a start-up company.
 A group possesses a self-organizing capability when it exhibits
three conditions.
– Autonomy
– Self-transcendence
– Cross-fertilization
 At some point, the team begins
to create its own concept.

ScrumDown at the Radcliffe Rugby Club

Autonomy
 Headquarters involvement is limited to providing
guidance, money, and moral support at the outset.
 On a day to day basis, management seldom intervenes
and the team is free to set its own direction.
 In a way, top management acts as a venture capitalist
– “We open our purse and keep our mouth closed.”
 Example: IBM development of personal computer
 Example: Honda City project team, average age 27,
“Develop the kind of car that the youth segment would like to
drive.”
Self-transcendence
 The project teams appear to be absorbed in the never-
ending quest for “the limit.”
 They elevate their goals through the development
process.
 By pursuing what appear to be contradictory goals, they
devise ways to
override the status quo and
make the big discovery.
 Example: Canon AE-1 team

Flyhalf Sarah Schooler of Radcliffe fending off Boston College

Cross-fertilization
 Team with wide variety of specializations, thought
processes, and behavior patterns carries out new
product development.
 Working in one large room is best (Fuji-Xerox).
 “When all team members
are in one room, others
information becomes yours
without even trying.”

Radcliffe Rugby Football Club

Factor 3: Overlapping Development Phases
 Self-organizing character of the team produces unique
dynamic or rhythm
 Sashimi system – Fuji Xerox Rugby system – Honda
 Hard merits (demerits)
– Speed and flexibility (watch out for muck and mall)
 Soft merits
– Share responsibility and cooperation
– Stimulates involvement and commitment
– Sharpens a problem-solving focus
– Develops initiative and diversified skills
– Heightens sensitivity to market conditions
Factor 4: Multilearning
 Learning by doing in two dimensions
– Across organization
– Across specialty
 Enhanced learning opportunities
– 15% of time devoted to “dreams” – 3M
– Peer pressure to study
– Send team to Europe to look around – Honda
– Bring in top academics and consultants – HP
 Everyone learns multiple skills
Factor 5: Subtle Control
 Management establishes checkpoints
– Prevents instability, ambiguity, and tension from turning into
chaos
 Emphasis on self-control, control by peer pressure,
control by love = “subtle control”
 Management responsible for:
– Selecting team members for balanced team
– Creating an open working environment
– Encouraging engineers to go out in the field
– Establishing rewards based on group performance
– Tolerating and anticipating mistakes
– Encouraging suppliers to become self-organizing
Factor 6: Organizational Transfer of Learning
 Transfer knowledge outside group
– Scatter successful team to new projects
– Institutionalize practice (monthly demos at IDX)
 Consciously pursue unlearning
– Next generation must be 40% better
– Cut product cycle by 80%
– Scrap old parts, processes, tools
Challenges and Opportunities
 Winding the Rubber Band Principle: Broad mandate and
demanding goals create tension.
 Anti-Waterfall Principle: Operational decisions are made
incrementally. Strategic decisions delayed to last moment.
 Push/Pull Principle: Differentiation in concept phase,
integration dominates in implementation phase
 Spread the Wealth Principle: Non-experts take on new
tasks.
 Cuckoo Principle: Successful SCRUMs become company
models (or they can get crushed because they are different).
 Control Anti-Pattern: Seniority based companies have
difficult time.
– But in times of desperation, SCRUMs are easily created.
Spiral Methodology

 Barry Boehm introduced the Spiral Methodology to "fix" problems

with the Waterfall Methodology. This is the most commonly used
variant of the Waterfall today.
 The Spiral methodology "peels the onion", progressing through
"layers" of the development process. A prototype lets users
determine if the project is on track, should be sent back to prior
phases, or should be ended. However, the phases and phase
processes are still linear.

Boehm, B.W. A Spiral Model of Software Development and Enhancement.

Proceedings of an International Workshop on Software Process and Software
Environments, Coto de Caza, Trabuco Canyon, California, March 27-29, 1985.
Boehm, Barry. A Spiral Model of Software Development and Enhancement.  ACM
SIGSOFT Software Engineering Notes, August 1986.
Boehm, Barry. A Spiral Model of Software Development and Enhancement.  IEEE
Computer, vol.21, #5, May 1988, pp 61-72.
Iterative Methology

 The Iterative methodology improves on the Spiral

methodology.
 Each iteration consists of all of the standard Waterfall phases,
but each iteration only addresses one subsystem. Further
iterations can add resources to the project while ramping up
the speed of delivery.
 Improves cost control, reduces risk, ensures delivery of
(sub)systems, and improves overall flexibility.
 Still assumes that the underlying development processes are
defined and linear.
SCRUM
Methodology

 The first and last phases (Planning and Closure) consist of

defined processes.
 The Sprint phase is an empirical process. It is treated as a
black box that requires external controls.
 Sprints are nonlinear and flexible. Sprints are used to evolve
the final product.
 The project is open to the environment until the Closure
phase. The deliverable can be changed at any time.
 The deliverable is determined during the project based on the
environment.
Methodology Comparison
Risk with Current
Methodologies
 Any methodology is better than
nothing.
 Current approaches rests on the
fallacy that the development
processes are defined, predictable
processes.
 They lack flexibility needed to cope
with the unpredictable results and
respond to a complex environment.

Schwaber, Ken. SCRUM Development Process.

Business Object Design and Implementation (Eds.
Jeff Sutherland et al.). London: Springer-Verlag,
1997.
SCRUM Lowers
Risk

 Development teams need to operate adaptively

within a complex environment using imprecise
processes.
 SCRUM can accelerate closure by inducing the
phenomenon known as "punctuated equilibrium"
seen in the evolution of biological species.
Levy, Steven. Artificial Life: A Report from the Frontier Where Computers Meet
Biology. Vintage Books, 1993.
Lewin, Roger. Complexity: Life at the Edge of Chaos. Collier Books, 1994.
Agile Project Management With SCRUM:
Theory and Practice
Jeff Sutherland, Ph.D.
Chief Technology Officer
jeff.sutherland@computer.org
http://jeffsutherland.com