04/03/2017
Dhirubhai Ambani Institute of Information and Communication
Technology (DA-IICT), Gandhinagar
Software Testing
&
Applications, Tools and Technologies
Saurabh Tiwari, PhD
DA-IICT Gandhinagar
A Self Assessment Test
Write a set of test cases specific set of data to properly test a
relatively Simple program.
Create a set of test data for the program .
The program reads three integer values from an input dialog. The three
value represent the lengths of the sides of a triangle. The program
displays a message that states whether the triangle is scalene,
isosceles, or equilateral.
Test Action and Data Expected Result
2, 5, 10 ??
1
04/03/2017
A Self Assessment Test
The function triangle takes three integer parameters that are
interpreted as the lengths of the sides of a triangle. It returns whether
the triangle is equilateral (three lengths equal), isosceles (two lengths
equal), scalene (no lengths equal), or invalid (impossible lengths).
Solution?
Tester Action and Data Expected Result
Erroneous input partition
a, b, or c a non-number an error message
Scalene triangle partition
2, 3, 4 scalene
1, 2, 3 an error message
Isosceles triangle partition
2, 2, 1 isosceles
2, 2, 0 an error message
Equilateral triangle partition
1, 1, 1 equilateral
1, 1, -1 an error message
User interface tests
Try a leading space for a, b, c an error message
Many more
2
04/03/2017
Why Testing?
Uncover as many as errors as possible in a given timeline.
Demonstrate a given software product matching its
requirements specification.
Validate the quality of a software
Software testing is arguably the least understood part of the development
process. Through a four-phase approach, the author shows why
eliminating bugs is tricky and why testing is a constant trade-off
[Whittaker]
Testing Purpose
1. Avoid Redundancy
Source Code
Requirement/Design/Maintenance/Change
Management
2. Reducing Cost
Cost of detection, Cost of Prevention
Internal Failure (at the time of development or before
you release the software product)
External Failure (finding bugs after deploying the
software product)
3. Correctness CONFIDENCE
4. Quality Assurance
3
04/03/2017
Trivial example
Try to move a file from folder A to another folder B
What are the possible scenarios?
Trying to move the file when it is open
You do not have the security rights to paste the file in folder B
Folder B is on a shared drive and storage capacity is full
Folder B already has a file with the same name
Taxonomy of Bugs
Verification?
Error?
Validation?
Defects ? Doesnt work as defined
Fault?
Failure?- Non functioning of system
Testing? Bugs? error in the system
Debugging?
Quality?
4
04/03/2017
BUG
Bugs
The word bug, originated in engineering.
The term's application to computing has been attributed to the
pioneer programmer, Grace Hopper. In 1944, Hopper, who was then
a young Naval Reserve officer, went to work on the Mark I computer
at Harvard.
He later described an incident in which a technician is said to have
pulled an actual bug (a moth, in fact) from between two electrical
relays in the Mark II computer.
5
04/03/2017
Error, Defects, Fault and Failure
Example:
You are driving a car and you are on road while on driving now
there is two way on the road
1. left--> Mumbai
2. right--> Delhi
Now you have to go to Delhi, it means you have to turn the
steering to the right, but by mistake you turn the steering to
the left, from that position that is called as "Error
and now Fault is there till you will not reach the Mumbai, but
when you reach Mumbai that is a final stage which is called
"Failure" because you had to reach Delhi but now you are in
Mumbai.
A mistake in coding is called Error, error found by tester is called
defect, defect accepted by development team then it is called bug, build
does not meet the requirements then it Is failure.
Coding: Error, Defects, Fault and Failure
The program is required to add two numbers
1 #include<stdio.h>
2
3 int main ()
4 { 5 + 3 should be 8, but the result is 2. There could
5 int value1, value2, ans; be various reasons as to why the program displays
6 the answer 2 instead of 8. For now we have
7 value1 = 5; detected a failure.
8 value2 = 3;
9
10 ans = value1 - value2;
11
12 printf("The addition of 5 + 3 = %d.", ans);
13
14 return 0;
15 }
As the failure has been detected a defect can be raised.
6
04/03/2017
The program is required to add two numbers
1 #include<stdio.h>
2
3 int main () Now lets go back to the program and analyze what
4 { was the fault in the program.
5 int value1, value2, ans;
6 There is a - sign present instead of + sign. So
7 value1 = 5; the fault in the program is the - sign.
8 value2 = 3;
9
10 ans = value1 - value2; // Bug, Defect
11
12 printf("The addition of 5 + 3 = %d.", ans);
13
14 return 0;
15 }
Error is the mistake I made by typing - instead
of + sign.
Error, Defects, Fault and Failure
A tester does not necessarily have access to the code and may be just
testing the functionality of the program. In that case the tester will
realize the output is faulty and will raise a defect.
Error: A mistake made by a programmer
Example: Misunderstood the requirements.
Defect/fault/bug: Manifestation of an error in a program.
Failure: Manifestation of one or more faults
in the observed program behavior
7
04/03/2017
Testing and Debugging
Testing: Finding inputs that cause the software to fail
Finding and localization of a defect
Intention behind is to find as many as defects possible
Debugging: The process of finding a fault given a failure
Fixing that defect
Intention is to remove those defects
...testing can be a very effective way to show the presence of bugs, but
is hopelessly inadequate for showing their absence. The only effective
way to raise the confidence level of a program significantly is to give a
convincing proof of its correctness.
Edsger Dijkstra
http://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/EWD340.html
Who Tests the Software?
Developer
Independent Tester
Understands the system but, Must learn about the
will test "gently and, is driven system, but, will attempt to
by "delivery" break it and, is driven by
quality
8
04/03/2017
Why Testing is necessary
Suppose you have a 15 input fields each one having 5
possible values.
How many combinations to be tested?
5^15 = 30517578125!!!
Why Testing is necessary
Why dont we test everything ?
System has 20 screens
Average 4 menus / screen
Average 3 options / menu
Average of 10 fields / screen
2 types of input per field
Around 100 possible values
Approximate total for exhaustive testing
20 x 4 x 3 x 10 x 2 x 100 = 480,000 tests
Test length = 1 sec then test duration = 17.7 days
Test length = 10 sec then test duration = 34 weeks
Test length = 1 min then test duration = 4 years
Test length = 10 mins then test duration = 40 years!
It is not a matter of time. But,time is money ( salary is taken by hour. So second
is valuable for software houses)
9
04/03/2017
Testing Technique Hierarchies
Software Testing
Techniques
Execution-based Non-execution based
testing testing
Program-based Combined Specification- Inspections
testing Testing based testing
Ad-hoc Checklist Scenario-
based
Structure-based Fault-based Error-based
criterion criterion criterion
19
Testing Strategy Test Case Design
Testing Strategy
Black-box White-box
(Specification based) Static Analysis
(Code based)
Requirement
Design
Equivalence class Boundary Value Cause-Effect Implementation
Partitioning Analysis Graphing
Model-Based Data Flow Mutation Control Flow
Testing based based based
Statement coverage, Decision coverage, Condition
coverage, Decision-condition coverage, Multiple-condition
coverage
20
10
04/03/2017
New : Test Coverage Criteria
A testers job is simple : Define a model of the software,
then find ways to cover it
Test Requirements : Specific things that must be satisfied or
covered during testing
Test Criterion : A collection of rules and a process that define
test requirements
Testing researchers have defined dozens of criteria, but
they are all really just a few criteria on four types of
structures
Criteria Based on Structures
Structures : Four ways to model software
1. Graphs
2. Logical Expressions
(not X or not Y) and A and B
A: {0, 1, >1}
3. Input Domain B: {600, 700, 800}
Characterization C: {swe, cs, isa, infs}
if (x > y)
z = x - y;
4. Syntactic Structures else
z = 2 * x;
11
04/03/2017
Black-box Testing
Black-Box : Test to Specification
Inputs causing
anomalous
Input test data I behaviour
e
System
Outputs which reveal
the presence of
Output test results Oe defects
24
12
04/03/2017
Equivalence Class Partitioning
Partition the program input domain into equivalence
classes (according to the specifications)
The rationale is that test of a representative value of
each class is equivalent to a test of any other value
of the same class.
Identify valid as well as invalid equivalence classes
One test case from each equivalence class
25
A Set of Guidelines
If an input condition specifies a range of values (e.g., the item count can
be from 1 to 999), identify one valid equivalence class (1<item count<999)
and two invalid classes (item count <1 and item count >999)
If an input condition specifies the number of values (e.g., one through six
owners can be listed for the automobile), identify one valid equivalence
class and two invalid equivalence classes (no owners and more than six
owners)
If an input condition specifies a set of input values and there is a reason to
believe that the program handles each differently (type of vehicle must be
BUS, TRUCK, TAXICAB, PASSENGER, or MOTORCYCLE), identify a valid
equivalence class for each and one invalid equivalence class (e.g.,
"TRAILER)
If an input condition specifies a must be situation such as first character
of the identifier must be a letter, identify one valid equivalence class (it is
a letter) and one invalid equivalence class (it is not a letter)
26
13
04/03/2017
A Summary: Set of Guidelines
Input Condition Valid Eq. Classes Invalid Eq. Classes
Range of values (1-200) One valid (value two invalid
within the range)
Number N valid values One valid Two invalid (none,
more than N)
Set of input values One valid Eq. class One
each handled differently by for each value (e.g. any value not
the (total n) in valid input set)
program (e.g. A, B, C, (n))
Must be condition (e.g., id One (e.g., it is a One (e.g., it is not
name must begin with a letter) a letter)
letter)
27
Example
If there is any reason to believe that the program does not
handle elements in an equivalence class identically, split the
equivalence class into smaller equivalence classes.
Example:
Input condition 0 <= x <= Max
Valid Equivalence Classes 0 <= x <= Max
Invalid Equivalence Classes x < 0; x > Max
3 Test Cases
28
14
04/03/2017
Example
Example of Equivalence class partitioning
A text field permits only numeric characters
Length must be 6-10 characters long
Partition according to the requirement should be like this:
29
Identifying Test cases
Assign a unique number to each equivalence class
Until all valid equivalence classes have been covered by test
cases, write a new test case covering as many of the uncovered
valid equivalence classes as possible.
Each invalid equivalence class cover by a separate test case.
30
15
04/03/2017
Identifying Test cases
A individual test case could not cover invalid cases.
For example, if the specification is
enter book type (HARDCOVER, SOFTCOVER, or LOOSE) and
amount (1-999)
The test case, XYZ, 0, expressing two error conditions (invalid book
type and amount) will probably not exercise the check for the
amount.
(Program may say XYZ is UNKNOWN BOOK TYPE, and not bother
to examine the remainder of the input.
31
Question?
Consider that wordCount method takes a word w and a
filename f as input and returns the number of occurrences of
w in the text contained in the file named f. An exception is
raised if there is no file with name f. Using the partitioning
method, we obtain the following equivalence classes.
Equivalence Class W F
E1 Non-null Exists, not empty
E2 Non-null Does not exist
E3 Non-null Exists, empty
E4 Null Exists, not empty
E5 Null Does not exist
E6 Null Exists, empty
32
16
04/03/2017
Question?
Consider an application that requires two integer inputs x and y. Each
of these inputs is expected to lie in the following ranges:
3 <= x <= 7 and 5 <= y <= 9
E1: x<3 E2: 3<=x<=7 E3: x>7 (y ignored)
Six equivalent Classes
E4: y<5 E5: 5<=y<=9 E6: y>9 (x ignored)
E1: x<3, y<5
E2: x<3, 5<=y<=9
E3: x<3, y>9
E4: 3<=x<=7, y<5 Nine equivalent Classes
E5: 3<=x<=7, 5<=y<=9 X x Y
E6: 3<=x<=7, y>9
E7: x>7, y<5
E8: x>7, 5<=y<=9
E9: x>7, y>9
33
Boundary-Value Analysis
Boundary value analysis is a test selection technique that targets
faults in applications at the boundaries of equivalence classes.
Generally combined with Equivalence Class Partitioning
Design test cases that exercise values that lie at the boundaries
of an input equivalence class.
Also identify output equivalence classes, and write test cases to
generate o/p at the boundaries of the output equivalence
classes.
Example: input condition 0 <= x <= max
Test for values : 0,1, x, max-1, max ( valid inputs)
-1, max+1 (invalid inputs)
34
17
04/03/2017
3 11
4 7 10
Less than 4 Between 4 and 10 More than 10
Number of input values
9999 100000
10000 50000 99999
Less than 10000 Between 10000 and 99999 More than 99999
Input values
35
Example 1
Suppose you have very important tool at office, accepts valid User
Name and Password field to work on that tool, and accepts
minimum 8 characters and maximum 12 characters. Valid range 8-
12, Invalid range 7 or less than 7 and Invalid range 13 or more than
13.
Write Test Cases for Valid partition value, Invalid partition value and
exact boundary value.
36
18
04/03/2017
Test Cases 1: Consider password length less than 8.
Test Cases 2: Consider password of length exactly 8.
Test Cases 3: Consider password of length between 9 and 11.
Test Cases 4: Consider password of length exactly 12.
Test Cases 5: Consider password of length more than 12.
37
Exercise??
Consider a program for determining the previous date. Its input is
triple of day, month and year with the following ranges 1 <= month
<= 12, 1 <= day <= 31, 1900 <= year <= 2015.The possible output
dates would be previous date or invalid date. Design the
equivalence class test cases?
Equivalence classes:-
1. year<1900
2. year>2015
3. 1900<=year<=2015
a. Month<1
b. Month>12
c. 1<=Month<=12
i. Day<1
ii. Month has 30 days a.)Day>30 b.)1<=Day<=30
iii. Month has 31 days a.)Day>31 b.)1<=Day<=31
iv. Month is February
1. Year is leap year a.) Day>29 b.)1<=Day<=29
2. Year is common year a.)Day>28 b.)1<=Day<=28
38
19
04/03/2017
White-Box Testing
39
White-Box Testing
Testing based on analysis of internal logic (design, code,
etc.). (But expected results still come from requirements.)
Also know as structural testing.
White-box testing concerns techniques for designing tests; it
is not a level of testing.
White-box testing techniques apply primarily to lower levels
of testing (e.g., unit and component).
20
04/03/2017
Control-flow-based Testing
A traditional form of white-box testing
Step 1: From the source, create a graph describing the flow of
control
Called the control flow graph
The graph is created (extracted from the source code)
manually or automatically
Step 2: Design test cases to cover certain elements of this
graph
Nodes, edges, paths
Control-flow-Graph
21
04/03/2017
Elements of a CFG
Three kinds of nodes:
Statement nodes: represent single-entry-single-exit
sequences of statements
Predicate nodes: represent conditions for branching
Auxiliary nodes: (optional) for easier understanding (e.g.,
join points for IF, etc.)
Edges: represents possible flow of control
It is relatively easy to map standard constructs from
programming languages to elements of CFGs
IF-THEN, IF-THEN-ELSE, SWITCH
if (c) if (c) switch (c)
then then case 1:
// join point else case 2:
// join point // join point
..
22
04/03/2017
Example
switch (position)
case CASHIER
if (empl_yrs > 5)
bonus := 1;
else
bonus := 0.7; .
case MANAGER .
.
bonus := 1.5;
if (retiring_soon) .
bonus := 1.2 * bonus
case
endswitch
Mapping for Loops
while (c) {
Note: other loops (e.g., FOR, DO-WHILE,) are mapped
similarly. Figure out how this is done.
23
04/03/2017
Example
int f1(int x,int y){
1 while (x != y){
2 if (x>y) then
3 x=x-y;
4 else y=y-x;
5}
6 return x; }
Example Control Flow Graph
1
2
3 4
5
6
24
04/03/2017
White-Box Test Case Design
Statement coverage
write enough test cases to execute every statement at least once
Decision coverage (Branch coverage)
write test cases to exercise the true and false outcomes of every
decision
Condition coverage (Predicate coverage)
write test cases such that each condition in a decision takes on
all possible outcomes at least once
may not always satisfy decision coverage
Example
void function eval (int A, int B, int X )
{
if ( A > 1) and ( B = 0 )
then X = X / A;
if ( A = 2 ) or ( X > 1)
then X = X + 1;
}
Statement coverage test cases:
1) A = 2, B = 0, X = ? ( X can be
assigned any value)
25
04/03/2017
Example
void function eval (int A, int B, int X )
{
if ( A > 1) and ( B = 0 )
then X = X / A;
if ( A = 2 ) or ( X > 1)
then X = X + 1;
}
Decision coverage test cases:
1) A = 3 B = 0 X = 1 (acd)
2) A = 2, B = 1, X = ? (abe)
Example
Condition coverage test cases must
cover conditions
A > 1, A <=1, B = 0 , B!=0
A = 2, A!=2, X > 1, X <=1
Test Cases:
1) A = 1, B = 0, X = 3 (abe)
2) A =2, B = 1, X = 1 (abe)
Doesnt satisfy decision coverage
26
04/03/2017
Exercise
Consider the following function which determines whether a given
code is a valid Canadian postal code or not. The Canadian postal
code is a seven-character uniformly structured, alphanumeric code in
the form ANA NAN where `A' represents an alphabetic character
and `N' represents a numeric character.
Exercise
[a] Statement Coverage; [b] Branch
Coverage; [c] Condition Coverage; [d]
Equivalence Class Partitioning
27
04/03/2017
JUnit Testing Framework
Example: Old way vs. New way
int max(int a, int b) { @Test
if (a > b) { void testMax() {
return a; assertEquals(7, max(3, 7));
} else { assertEquals(3, max(3, -7));
return b;
} }
}
void testMax() {
int x = max(3, 7);
if (x != 7) {
System.out.println("max(3, 7) gives " + x);
}
x = max(3, -7);
if (x != 3) {
System.out.println("max(3, -7) gives " + x);
}
}
public static void main(String[] args) {
new MyClass().testMax();
}
28
04/03/2017
Junit, In PICTURES
test suite test runner
another unit test
test case (for one method) A unit test tests the methods in a
another test case single class
A test case tests (insofar as
another unit test possible) a single method
another test case You can have multiple test cases
another test case
for a single method
another test case A test suite combines unit tests
The test fixture provides
software support for all this
unit test (for one class)
test case (for one method)
The test runner runs unit tests or
an entire test suite
another test case
Integration testing (testing that it
all works together) is not well
test fixture
supported by JUnit
Writing a JUnit test class, I
Start by importing these JUnit 4 classes:
import org.junit.*;
import static org.junit.Assert.*; // note static import
Declare your test class in the usual way
public class MyProgramTest {
Declare an instance of the class being tested
You can declare other variables, but dont give them initial
values here
public class MyProgramTest {
MyProgram program;
int someVariable;
29
04/03/2017
Writing a JUnit test class, II
Define a method (or several methods) to be executed before
each test
Initialize your variables in this method, so that each test starts
with a fresh set of values
@Before
public void setUp() {
program = new MyProgram();
someVariable = 1000;
}
You can define one or more methods to be executed after each
test
Typically such methods release resources, such as files
Usually there is no need to bother with this method
@After
public void tearDown() {
}
A Simple Example
Suppose you have a class Arithmetic with methods int multiply(int x, int y), and
boolean isPositive(int x)
import org.junit.*;
import static org.junit.Assert.*;
public class ArithmeticTest {
@Test
public void testMultiply() {
assertEquals(4, Arithmetic.multiply(2, 2));
assertEquals(-15, Arithmetic.multiply(3, -5));
}
@Test
public void testIsPositive() {
assertTrue(Arithmetic.isPositive(5));
assertFalse(Arithmetic.isPositive(-5));
assertFalse(Arithmetic.isPositive(0));
}
}
30
