Introduction To Software Testing Why Do We Test Software?: Paul Ammann & Jeff Offutt

Introduction to Software Testing
(2nd edition)
Chapter 1
Why Do We Test Software?
Paul Ammann & Jeff Offutt
h //
http://www.cs.gmu.edu/~offutt/softwaretest/
d / ff / f
/
Updated August 2012

First version, 28 August 2011
Testing in the 21st Century
Software defines behavior

network routers, finance, switching networks, other infrastructure
Todays software market :
Industry is going
through
h
h a revolution
l i in
i
what testing means to
the success of software
products
Embedded Control Applications
iis muchh bigger
bi
is more competitive
has more users
airplanes, air traffic control

spaceships
watches
ovens
remote controllers
PDAs
memory seats
DVD players
garage door openers
cell phones
Agile processes put increased pressure on testers

Programmers must unit test with no training, education or tools !
Tests are key to functional requirements but who builds those tests ?
Introduction to Software Testing, Edition 2 (Ch 1)
Ammann & Offutt
Testing in the 21st Century

More safety critical, real-time software
Embedded software is ubiquitous check your pockets
Enterprise
p
applications
pp
means bigger
gg pprograms,
g
, more
users
Paradoxically, free software increases our expectations !
Security is now all about software faults
Secure software is reliable software
Thee web o
offers
e s a new
ew deployment
dep oy e platform
pa o
Very competitive and very available to more users
Web apps are distributed
Web apps must be highly reliable
Industry desperately needs our inventions !

Ammann & Offutt
Software is a Skin that Surrounds

Our Civilization
Quote due to Dr. Mark Harman

Ammann & Offutt
Software Faults, Errors & Failures
Software Fault : A static defect in the software
Software Failure : External, incorrect behavior with respect to the

requirements or other description of the expected behavior
Software Error : An incorrect internal state that is the manifestation

of some fault
Faults in software are equivalent to design mistakes in

hardware.
Software does not degrade.
Ammann & Offutt
Fault and Failure Example
A patient gives a doctor a list of symptoms

Failures
The doctor tries to diagnose the root cause, the ailment

Fault
The doctor may look for anomalous internal conditions

(high blood pressure, irregular heartbeat, bacteria in the
blood stream)
Errors
Most medical problems result from external attacks
(bacteria, viruses) or physical degradation as we age.
They were there at the beginning and do not appear when
a part wears out.
Ammann & Offutt
A Concrete Example
Fault: Should start
searching at 0, not 1
p
public
static int numZero ((int [ ] arr))
Test 1
{ // Effects: If arr is null throw NullPointerException
[ 2, 7, 0 ]
// else return the number of occurrences of 0 in arr
Expected: 1
int count = 0;
Actual: 1
for (int i = 1; i < arr.length; i++)
{
Error: i is 1, not 0,
Test 2
if (arr [ i ] == 0)
on the first iteration
[ 0, 2, 7 ]
{
Failure: none
Expected: 1
count++;
t++
Actual: 0
}
}
Error: i is 1, not 0
return count;
Error propagates to the variable count
}
Failure: count is 0 at the return statement
Ammann & Offutt
The Term Bug
Bug is used informally

Sometimes speakers mean fault, sometimes error, sometimes failure
often the speaker doesnt know what it means !
precise defined,
defined and
This class will try to use words that have precise,
unambiguous meanings
an analyzing process
BUG
It
so in
The
has been just
j
i all off my inventions.
i
i
first step is an intuition, and comes with a burst,
then difficulties arisethis thing gives out and
[it is] then that 'Bugs'as such little faults and
difficulties are calledshow themselves and
months of intense watching, study and labor are
requisite. . . Thomas Edison
Ammann & Offutt
must equally have been

performed in order to
furnish the Analytical
Engine with the necessary
operative data; and that
herein may also lie a
possible
ibl source off error.
Granted that the actual
mechanism is unerring in
its processes, the cards
may give it wrong orders.
Ada, Countess Lovelace
(notes on Babbages
Analytical Engine)
8
Spectacular Software Failures
NASAs Mars lander: September 1999, crashed

due to a units integration fault
Mars Polar
THERAC-25 design
Lander crash
site?
THERAC-25 radiation machine : Poor testing of

safety-critical software can cost lives : 3 patients Ariane 5:
were killed
exception-handling
Ariane 5 explosion : Very expensive
bug : forced self
Intels Pentium FDIV fault : Public relations
nightmare
destruct on maiden
flight (64-bit to 16-bit
conversion: about
370 million $ lost)
We need our software to be dependable

Testing is one way to assess dependability
Ammann & Offutt
Northeast Blackout of 2003

508 generating
units and 256
power plants shut
down
Affected 10 million
people in Ontario,
Canada
Affected 40 million
people in 8 US
states
Financial losses of
$6 Billion USD
The alarm system in the energy management system failed
due to a software error and operators were not informed of
the power overload in the system
Ammann & Offutt
10
Costly Software Failures
NIST report, The Economic Impacts of Inadequate

Infrastructure for Software Testing (2002)
Inadequate software testing costs the US alone between $22 and
$59 billion annuallyy
Better approaches could cut this amount in half
Huge losses due to web application failures

Financial services : $6.5 million per hour (just in USA!)
Credit card sales applications : $2.4 million per hour (in USA)
In Dec 2006,, amazon.coms BOGO offer turned into a

double discount
2007 : Symantec says that most security vulnerabilities are
due to faulty software
World--wide monetary loss due to poor software is staggering
World
Ammann & Offutt
11
What Does This Mean?
Software testing is getting more

important
What are we trying to do when we test ?

What are our goals ?
Ammann & Offutt
12
Validation & Verification (IEEE)
Validation : The process of evaluating software at the end of

software development to ensure compliance with intended usage
Verification : The process of determining whether the products of a

given phase of the software development process fulfill the
requirements established during the previous phase
IV&V stands for independent verification and validation
Ammann & Offutt
13
Testing Goals Based on Test Process

Maturity
Level 0 : Theres no difference between testing and
debugging
Level 1 : The purpose of testing is to show correctness
Level 2 : The purpose of testing is to show that the
software doesnt work
Level 3 : The purpose of testing is not to prove anything
specific, but to reduce the risk of using the software
Level 4 : Testing is a mental discipline that helps all IT
professionals develop higher quality software
Ammann & Offutt
14
Level 0 Thinking
Testing
is the same as debugging
Does
not distinguish between incorrect behavior

and mistakes in the program
Does
not help develop software that is reliable or
safe
This is what we teach undergraduate CS majors
Ammann & Offutt
15
Level 1 Thinking
Purpose
is to show correctness
Correctness is impossible to achieve
What do we know if no failures?
Good software or bad tests?
Test
engineers have no:
Strict goal
Real stopping rule
Formal test technique
Test managers are powerless
This is what hardware engineers often expect

Ammann & Offutt
16
Level 2 Thinking
Purpose
is to show failures
Looking
g
for failures is a negative

g
activityy
Puts
testers and developers into an adversarial

relationship
What
if there are no failures?

This describes most software companies.
How can we move to a team approach ??
Ammann & Offutt
17
Level 3 Thinking
Testing
can only show the presence of failures
Whenever
we use software,, we incur some risk
Risk
may be small and consequences unimportant
Risk
may be great and consequences catastrophic
Testers
and developers cooperate to reduce risk
This describes a few enlightened software companies

Ammann & Offutt
18
Level 4 Thinking
A mental discipline that increases quality
Testing is only one way to increase quality
Test engineers can become technical leaders of the project
Primary responsibility to measure and improve software

quality
Their expertise should help the developers

This is the way traditional engineering works
Ammann & Offutt
19
Where Are You?

Are you at level 0, 1, or 2 ?
Is your organization at work at level 0,
1, or 2 ?
Or 3?
We hope to teach you to become
change agents in your workplace
Advocates for level 4 thinking
Ammann & Offutt
20
10
Tactical Goals : Why Each Test ?

If you dont know why youre conducting
each test, it wont be very helpful
Written
test objectives and requirements must be

documented
What are your planned coverage levels?
How much testing is enough?
objective spend the budget test
until the ship-date
Common
Sometimes called the date criterion

Ammann & Offutt
21
Here! Test This!

My first professional job
Big
software big
program
MicroSteff
software
f
system
for the mac
V.1.5.1
Jan/2007
Jan/2011
MF2-HD
1.44 MB
DataLife
Verdatim
A stack of computer printoutsand no documentation

Ammann & Offutt
22
11
Why Each Test ?

If you dont start planning for each test when the
functional requirements are formed, youll never
know why youre conducting the test
1980:
The software shall be easily maintainable
Threshold
What
reliability requirements?
fact does each test try to verify?
Requirements
definition teams need testers!
Ammann & Offutt
23
1990s Thinking
Naw - I already
dont plow as good
as I know how...
Theyre teaching a new way

of plowing over at the Grange
tonight
you going?
g -y
g g
Fail !!!
Knowing is not enough, we must apply. Willing is not enough, we must do.
do.
Goethe
Ammann & Offutt
24
12
Cost of Not Testing

Poor Program Managers might say:
say:
Testing is too expensive.
Testing
is the most time consuming and

expensive part of software development
Not testing is even more expensive
If we have to little testing effort early, the cost of
testing increases
Planning for testing after development is
prohibitively expensive
Ammann & Offutt
25
Cost of Late Testing

60
Assume $1000 unit cost, per fault, 100 faults
50
40
Fault origin (%)
30
Fault detection (%)
20
Unit cost (X)
10
0
Software Engineering Institute; Carnegie Mellon University; Handbook CMU/SEI-96-HB-002

Ammann & Offutt
26
13
Summary: Why Do We Test Software ?

A testers goal is to eliminate faults as
early
l as possible
ibl
Improve quality
Reduce cost
Preserve customer satisfaction
Ammann & Offutt
27
14

Introduction To Software Testing Why Do We Test Software?: Paul Ammann & Jeff Offutt

Uploaded by

Copyright:

Available Formats

Introduction To Software Testing Why Do We Test Software?: Paul Ammann & Jeff Offutt

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Introduction To Software Testing Why Do We Test Software?: Paul Ammann & Jeff Offutt

Uploaded by

Copyright:

Available Formats

Introduction to Software Testing

Updated August 2012

Testing in the 21st Century

Software defines behavior

Todays software market :

airplanes, air traffic control

Agile processes put increased pressure on testers

Introduction to Software Testing, Edition 2 (Ch 1)

Ammann & Offutt

Testing in the 21st Century

Secure software is reliable software

Industry desperately needs our inventions !

Ammann & Offutt

Software is a Skin that Surrounds

Quote due to Dr. Mark Harman

Ammann & Offutt

Software Faults, Errors & Failures

Software Fault : A static defect in the software

Software Failure : External, incorrect behavior with respect to the

Software Error : An incorrect internal state that is the manifestation

Faults in software are equivalent to design mistakes in

Introduction to Software Testing, Edition 2 (Ch 1)

Ammann & Offutt

Fault and Failure Example

A patient gives a doctor a list of symptoms

The doctor tries to diagnose the root cause, the ailment

The doctor may look for anomalous internal conditions

Introduction to Software Testing, Edition 2 (Ch 1)

Ammann & Offutt

Introduction to Software Testing, Edition 2 (Ch 1)

Ammann & Offutt

The Term Bug

Bug is used informally

Ammann & Offutt

must equally have been

Spectacular Software Failures

NASAs Mars lander: September 1999, crashed

THERAC-25 radiation machine : Poor testing of

We need our software to be dependable

Ammann & Offutt

Northeast Blackout of 2003

Ammann & Offutt

Costly Software Failures

NIST report, The Economic Impacts of Inadequate

Huge losses due to web application failures

In Dec 2006,, amazon.coms BOGO offer turned into a

Introduction to Software Testing, Edition 2 (Ch 1)

Ammann & Offutt

What Does This Mean?

Software testing is getting more

What are we trying to do when we test ?

Ammann & Offutt

Validation & Verification (IEEE)

Validation : The process of evaluating software at the end of

Verification : The process of determining whether the products of a

IV&V stands for independent verification and validation

Introduction to Software Testing, Edition 2 (Ch 1)

Ammann & Offutt

Testing Goals Based on Test Process