Unit-I

SOFTWARE TESTING:

Introduction, Evolution, Myths & Facts, Goals, Psychology, Definition, Model for testing,
Effective Vs Exhaustive Software Testing. Software Testing Terminology and Methodology:
Software Testing Terminology, Software Testing Life Cycle, relating test life cycle to
development life cycle Software Testing Methodology.

Introduction:

Software has pervaded our society, from modern households to spacecraft’s. It has become an
essential component of any electronic device or system.

This is why software development has turned out to be an exciting career for computer engineers
in the last 10–15 years. However, software development faces many challenges.

Software is becoming complex, but the demand for quality in software products has increased.
This rise in customer awareness for quality increases the workload and responsibility of the
software development team. That is why software testing has gained so much popularity in the
last decade. Job trends have shifted from development to software testing. Today, software
quality assurance and software testing courses are offered by many institutions. Organizations
have separate testing groups with proper hierarchy. Software development is driven with testing
outputs. If the testing team claims the presence of bugs in the software, then the development
team cannot release the product. However, there still is a gap between academia and the demand
of industries.

The practical demand is that passing graduates must be aware of testing terminologies, standards,
and techniques. But students are not aware in most cases, as our universities and colleges do not
offer separate software quality and testing courses. They study only software engineering. It can
be said that software engineering is a mature discipline today in industry as well as in academia.
On the other hand, software testing is mature in industry but not in academia. Thus, this gap must
be bridged with separate courses on software quality and testing so that students do not face
problems when they go for testing in industries. Today, the ideas and techniques of software
testing have become essential knowledge for software developers, testers, and students as well.
This book is a step forward to bridge this gap.

We cannot say that the industry is working smoothly, as far as software testing is concerned.
While many industries have adopted effective software testing techniques and the development is
driven by testing efforts, there are still some loopholes. Industries are dependent on automation
of test execution. Therefore, testers also rely on effi cient tools. But there may be an instance
where automation will not help, which is why they also need to design test cases and execute
them manually. Are the testers prepared for this case? This requires testing teams to have
knowledge of testing tactics and procedures of how to design test cases.

How do industries measure their testing process? Since software testing is a complete process
today, it must be measured to check whether the process is suitable for projects. CMM
(Capability Maturity Model) has measured the development process on a scale of 1–5 and
companies are running for the highest scale. On the same pattern, there should be a measurement
program for testing processes. Fortunately, the measurement technique for testing processes has
also been developed. But how many managers, developers, testers, and of course students, know
that we have a Testing Maturity Model (TMM) for measuring the maturity status of a testing
process? This book gives an overview of various test process maturity models and emphasizes
the need for these. Summarizing the above discussion, it is evident that industry and academia
should go parallel. Industries constantly aspire for high standards.

On the other side, organizations cannot run with the development team looking after every stage,
right from requirement gathering to implementation. Testing is an important segment of software
development and it has to be thoroughly done. Therefore, there should be a separate testing
group with divided responsibilities among the members.

EVOLUTION OF SOFTWARE TESTING

In the early days of software development, software testing was considered only a debugging
process for removing errors after the development of software. By 1970, the term ‘software
engineering’ was in common use. But software testing was just a beginning at that time. In 1978,
G. J. Myers realized the need to discuss the techniques of software testing in a separate subject.
He wrote the book The Art of Software Testing [2] which is a classic work on software testing.
He emphasized that there is a requirement that undergraduate students must learn software
testing techniques so that they pass out with the basic knowledge of software testing and do not
face problems in the industry. Moreover, Myers discussed the psychology of testing and
emphasized that testing should be done with a mindset of fi nding errors and not to demonstrate
that errors are not present.

By 1980, software professionals and organizations started emphasizing on quality. Organizations

realized the importance of having quality assurance teams to take care of all testing activities for
the project right from the beginning. In the 1990s, testing tools fi nally came into their own.
There was a fl ood of various tools, which are absolutely vital to adequate testing of software
systems. However, they do not solve all problems and cannot replace a testing process.

Gelperin and Hetzel [79] have characterized the growth of software testing with time. Based on
this, we can divide the evolution of software testing into the following phases

Debugging-oriented Phase (Before 1957) This phase is the early period of testing. At that time,
testing basics were unknown. Programs were written and then tested by the programmers until
they were sure that all the bugs were removed. The term used for testing was checkout, focused
on getting the system to run. Debugging was a more general term at that time and it was not
distinguishable from software testing. Till 1956, there was no clear distinction between software
development, testing, and debugging. Demonstration-oriented Phase (1957–78) The term
‘debugging’ continued in this phase. However, in 1957, Charles Baker pointed out that the
purpose of checkout is not only to run the software but also to demonstrate the correctness
according to the mentioned requirements. Thus, the scope of checkout of a program increased
from program runs to program correctness. Moreover, the purpose of checkout was to show the
absence of errors. There was no stress on the test case design. In this phase, there was a
misconception that the software could be tested exhaustively.

Destruction-oriented Phase (1979–82) This phase can be described as the revolutionary turning
point in the history of software testing. Myers changed the view of testing from ‘testing is to
show the absence of errors’ to ‘testing is to fi nd more and more errors.’ He separated debugging
from testing and stressed on the valuable test cases if they explore more bugs. This phase has
given importance to effective testing in comparison to exhaustive testing. The importance of
early testing was also realized in this phase.

Evaluation-oriented Phase (1983–87) With the concept of early testing, it was realized that if
the bugs were identifi ed at an early stage of development, it was cheaper to debug them as
compared to the bugs found in implementation or post-implementation phases. This phase
stresses on the quality of software products such that it can be evaluated at every stage of
development. In fact, the early testing concept was established in the form of verifi cation and
validation activities which help in producing better quality software. In 1983, guidelines by the
National Bureau of Standards were released to choose a set of verifi cation and validation
techniques and evaluate the software at each step of software development.

Prevention-oriented Phase (1988–95) The evaluation model stressed on the concept of bug-
prevention as compared to the earlier concept of bug-detection. With the idea of early detection
of bugs in earlier phases, we can prevent the bugs in implementation or further phases. Beyond
this, bugs can also be prevented in other projects with the experience gained in similar software
projects. The prevention model includes test planning, test analysis, and test design activities
playing a major role, while the evaluation model mainly relies on analysis and reviewing
techniques other than testing.

Process-oriented Phase (1996 onwards) In this phase, testing was established as a complete
process rather than a single phase (performed after coding) in the software development life
cycle (SDLC). The testing process starts as soon as the requirements for a project are specifi ed
and it runs parallel to SDLC. Moreover, the model for measuring the performance of a testing
process has also been developed like CMM. The model for measuring the testing process is
known as Testing Maturity Model (TMM). Thus, the emphasis in this phase is also on quantifi
cation of various parameters which decide the performance of a testing process.
The evolution of software testing was also discussed by Hung Q. Nguyen and Rob Pirozzi in a
white paper [81], in three phases, namely Software Testing 1.0, Software Testing 2.0, and
Software Testing 3.0. These three phases discuss the evolution in the earlier phases that we
described. According to this classifi cation, the current state-of-practice is Software Testing 3.0.
These phases are discussed below.

Software Testing 1.0 In this phase, software testing was just considered a single phase to be
performed after coding of the software in SDLC. No test organization was there. A few testing
tools were present but their use was limited due to high cost. Management was not concerned
with testing, as there was no quality goal.

Software Testing 2.0 In this phase, software testing gained importance in SDLC and the
concept of early testing also started. Testing was evolving in the direction of planning the test
resources. Many testing tools were also available in this phase.

Software Testing 3.0 In this phase, software testing is being evolved in the form of a process
which is based on strategic effort. It means that there should be a process which gives us a
roadmap of the overall testing process. Moreover, it should be driven by quality goals so that all
controlling and monitoring activities can be performed by the managers. Thus, the management
is actively involved in this phase.

SOFTWARE TESTING— MYTHS AND FACTS

Some of the myths are there in testing, in its growing phase.

Myth Testing is a single phase in SDLC .

Truth This myth is more in the minds of students who have just passed out or are going to pass
out of college and want to start a career in testing. So the general perception is that, software
testing is an easy job, wherein test cases are executed with testing tools only. But in reality, tools
are there to automate the tasks and not to carry out all testing activities. Testers’ job is not easy,
as they have to plan and develop the test cases manually and it requires a thorough understanding
of the project being developed with its overall design. Overall, testers have to shoulder a lot of
responsibility which sometimes make their job even harder than that of a developer.

Myth Software development is worth more than testing.

Truth This myth prevails in the minds of every team member and even in fresher’s who are
seeking jobs. As a fresher, we dream of a job as a developer. We get into the organization as a
developer and feel superior to other team members. At the managerial level also, we feel happy
about the achievements of the developers but not of the testers who work towards the quality of
the product being developed. Thus, we have this myth right from the beginning of our career,
and testing is considered a secondary job. But testing has now become an established path for
job-seekers. Testing is a complete process like development, so the testing team enjoys equal
status and importance as the development team.

Myth Complete testing is possible.

Truth This myth also exists at various levels of the development team. Almost every person who
has not experienced the process of designing and executing the test cases manually feels that
complete testing is possible. Complete testing at the surface level assumes that if we are giving
all the inputs to the software, then it must be tested for all of them. But in reality, it is not
possible to provide all the possible inputs to test the software, as the input domain of even a
small program is too large to test. Moreover, there are many things which cannot be tested
completely, as it may take years to do so. This will be demonstrated soon in this chapter. This is
the reason why the term ‘complete testing’ has been replaced with ‘effective testing.’ Effective
testing is to select and run some select test cases such that severe bugs are uncovered first.

Myth Testing starts after program development.

Truth Most of the team members, who are not aware of testing as a process, still feel that testing
cannot commence before coding. But this is not true. As mentioned earlier, the work of a tester
begins as soon as we get the specifications. The tester performs testing at the end of every phase
of SDLC in the form of verification (discussed later) and plans for the validation testing
(discussed later). He writes detailed test cases, executes the test cases, reports the test results, etc.
Testing after coding is just a part of all the testing activities.

Myth The purpose of testing is to check the functionality of the software.

Truth Today, all the testing activities are driven by quality goals. Ultimately, the goal of testing
is also to ensure quality of the software. But quality does not imply checking only the
functionalities of all the modules. There are various things related to quality of the software, for
which test cases must be executed.

Myth Anyone can be a tester.

Truth This is the extension of the myth that ‘testing is easy.’ Most of us think that testing is an
intuitive process and it can be performed easily without any training. And therefore, anyone can
be a tester. As an established process, software testing as a career also needs training for various
purposes, such as to understand (i) various phases of software testing life cycle, (ii) recent
techniques to design test cases, (iii) various tools and how to work on them, etc. This is the
reason that various testing courses for certified testers are being run.

GOALS OF SOFTWARE TESTING

To understand the new concepts of software testing and to define it thoroughly, let us first
discuss the goals that we want to achieve from testing. The goals of software testing may be
classified into three major categories

Short-term or immediate goals: These goals are the immediate results after performing testing.
These goals may be set in the individual phases of SDLC. Some of them are discussed below.

Bug discovery: The immediate goal of testing is to find errors at any stage of software
development. More the bugs discovered at an early stage, better will be the success rate of
software testing.

Bug prevention: It is the consequent action of bug discovery. From the behavior and
interpretation of bugs discovered, everyone in the software development team gets to learn how
to code safely such that the bugs discovered should not be repeated in later stages or future
projects. Though errors cannot be prevented to zero, they can be minimized. In this sense, bug
prevention is a superior goal of testing.

Long-term goals These goals affect the product quality in the long run, when one cycle of the
SDLC is over. Some of them are discussed here.

Quality: Since software is also a product, its quality is primary from the users’ point of view.
Thorough testing ensures superior quality. Therefore, the first goal of understanding and
performing the testing process is to enhance the quality of the software product.

Though quality depends on various factors, such as correctness, integrity, efficiency, etc.,
reliability is the major factor to achieve quality. The software should be passed through a
rigorous reliability analysis to attain high quality standards. Reliability is a matter of confidence
that the software will not fail, and this level of confidence increases with rigorous testing. The
confidence in reliability, in turn, increases the quality.
Customer satisfaction From the users’ perspective, the prime concern of testing is customer
satisfaction only. If we want the customer to be satisfied with the software product, then testing
should be complete and thorough. Testing should be complete in the sense that it must satisfy the
user for all the specified requirements mentioned in the user manual, as well as for the
unspecified requirements which are otherwise understood. A complete testing process achieves
reliability, reliability enhances the quality, and quality in turn, increases the customer
satisfaction.

Risk management Risk is the probability that undesirable events will occur in a system. These
undesirable events will prevent the organization from successfully implementing its business
initiatives. Thus, risk is basically concerned with the business perspective of an organization.

Risks must be controlled to manage them with ease. Software testing may act as a control, which
can help in eliminating or minimizing risks.

Thus, managers depend on software testing to assist them in controlling their business goals. The
purpose of software testing as a control is to provide information to management so that they can
better react to risk situations [4]. For example, testing may indicate that the software being
developed cannot be delivered on time, or there is a probability that high priority bugs will not be
resolved by the specified time. With this advance information, decisions can be made to
minimize risk situation. Hence, it is the testers’ responsibility to evaluate business risks (such as
cost, time, resources, and critical features of the system being developed) and make the same a
basis for testing choices. Testers should also categorize the levels of risks after their assessment
(like high-risk, moderate-risk, low-risk) and this analysis becomes the basis for testing activities.
Thus, risk management becomes the long-term goal for software testing.
Post-implementation goals These goals are important after the product is released. Some of
them are discussed here.

Reduced maintenance cost The maintenance cost of any software product is not its physical
cost, as the software does not wear out. The only maintenance cost in a software product is its
failure due to errors. Post-release errors are costlier to fix, as they are difficult to detect. Thus, if
testing has been done rigorously and effectively, then the chances of failure are minimized and in
turn, the maintenance cost is reduced.

Improved software testing process A testing process for one project may not be successful and
there may be scope for improvement. Therefore, the bug history and post-implementation results
can be analyzed to find out snags in the present testing process, which can be rectified in future
projects. Thus, the long-term post-implementation goal is to improve the testing process for
future projects.

PSYCHOLOGY FOR SOFTWARE TESTING

Software testing is directly related to human psychology. Though software testing has not been
defined till now, but most frequently, it is defined as, Testing is the process of demonstrating
that there are no errors.

The purpose of testing is to show that the software performs its intended functions correctly. This
defi nition is correct, but partially. If testing is performed keeping this goal in mind, then we
cannot achieve the desired goals (described above in the previous section), as we will not be able
to test the software as a whole. Myers first identified this approach of testing the software. This
approach is based on the human psychology that human beings tend to work according to the
goals fixed in their minds. If we have a preconceived assumption that the software is error-free,
then consequently, we will design the test cases to show that all the modules run smoothly. But it
may hide some bugs. On the other hand, if our goal is to demonstrate that a program has errors,
then we will design test cases having a higher probability to uncover bugs.

Thus, if the process of testing is reversed, such that we always presume the presence of bugs in
the software, then this psychology of being always suspicious of bugs widens the domain of
testing. It means, now we don’t think of testing only those features or specifications which have
been mentioned in documents like SRS (software requirement specification), but we also think in
terms of finding bugs in the domain or features which are understood but not specified. You can
argue that, being suspicious about bugs in the software is a negative approach. But, this negative
approach is for the benefit of constructive and effective testing. Thus, software testing may be
defined as, Testing is the process of executing a program with the intent of finding errors.

This definition has implications on the psychology of developers. It is very common that they
feel embarrassed or guilty when someone finds errors in their software. However, we should not
forget that humans are prone to error. We should not feel guilty for our errors. This psychology
factor brings the concept that we should concentrate on discovering and preventing the errors and
not feel guilt about them. Therefore, testing cannot be a joyous event unless you cast out your
guilt.

According to this psychology of testing, a successful test is that which finds errors. This can be
understood with the analogy of medical diagnostics of a patient. If the laboratory tests do not
locate the problem, then it cannot be regarded as a successful test. On the other hand, if the
laboratory test determines the disease, then the doctor can start an appropriate treatment. Thus,
the destructive approach of software testing, the definitions of successful and unsuccessful
testing should also be modified.

SOFTWARE TESTING DEFINITIONS

Many practitioners and researchers have defined software testing in their own way. Some are
given below

Testing is the process of executing a program with the intent of finding errors.

Testing is the process of executing a program with the intent of finding errors. Myers [2]

A successful test is one that uncovers an as-yet-undiscovered error. Myers [2]

Testing can show the presence of bugs but never their absence. W. Dijkstra [125]

Program testing is a rapidly maturing area within software engineering that is receiving
increasing notice both by computer science theoreticians and practitioners. Its general aim
is to affi rm the quality of software systems by systematically exercising the software in
carefully controlled circumstances. E. Miller[84]

Testing is a support function that helps developers look good by fi nding their mistakes
before anyone else does. James Bach [83]

Software testing is an empirical investigation conducted to provide stakeholders with

information about the quality of the product or service under test, with respect to the
context in which it is intended to operate. Cem Kaner [85]
The underlying motivation of program testing is to affirm software quality with methods
that can be economically and effectively applied to both large-scale and small-scale
systems. Miller [126]

Testing is a concurrent lifecycle process of engineering, using and maintaining test ware
(i.e. testing artifacts) in order to measure and improve the quality of the software being
tested. Craig [117]

Since quality is the prime goal of testing and it is necessary to meet the defined quality
standards, software testing should be defined keeping in view the quality assurance terms. Here,
it should not be misunderstood that the testing team is responsible for quality assurance. But the
testing team must be well aware of the quality goals of the software so that they work towards
achieving them.

Moreover, testers these days are aware of the definition that testing is to find more and more
bugs. But the problem is that there are too many bugs to fix. Therefore, the recent emphasis is on
categorizing the more important bugs first. Thus, software testing can be defined as,

Software testing is a process that detects important bugs with the objective of having better
quality software.

MODEL FOR SOFTWARE TESTING

Testing is not an intuitive activity; rather it should be learnt as a process. Therefore, testing
should be performed in a planned way. For the planned execution of a testing process, we need
to consider every element and every aspect related to software testing. Thus, in the testing model,
we consider the related elements and team members involved

The software is basically a part of a system for which it is being developed. Systems consist of
hardware and software to make the product run. The developer develops the software in the
prescribed system environment considering the testability of the software. Testability is a major
issue for the developer while developing the software, as a badly written software may be
difficult to test. Testers are supposed to get on with their tasks as soon as the requirements are
specified. Testers work on the basis of a bug model which classifies the bugs based on the SDLC
phase in which the testing is to be performed. Based on the software type and the bug model,
testers decide a testing methodology which guides how the testing will be performed. With
suitable testing techniques decided in the testing methodology, testing is performed on the
software with a particular goal. If the testing results are in line with the desired goals, then the
testing is successful; otherwise, the software or the bug model or the testing methodology has to
be modified so that the desired results are achieved. The following describe the testing model.

Software and Software Model

Software is built after analysing the system in the environment. It is a complex entity which
deals with environment, logic, programmer psychology, etc. But complex software makes it very
difficult to test. Since in this model of testing, our aim is to concentrate on the testing process,
therefore the software under consideration should not be so complex such that it would not be
tested. In fact, this is the point of consideration for developers who design the software. They
should design and code the software such that it is testable at every point. Thus, the software to
be tested may be modeled such that it is testable, avoiding unnecessary complexities.

Bug Model Bug model provides a perception of the kind of bugs expected. Considering the
nature of all types of bugs, a bug model can be prepared that may help in deciding a testing
strategy. However, every type of bug cannot be predicted. Therefore, if we get incorrect results,
the bug model needs to be modifi ed.

Testing methodology and Testing Based on the inputs from the software model and the bug
model, testers can develop a testing methodology that incorporates both testing strategy and
testing tactics. Testing strategy is the roadmap that gives us well-defined steps for the overall
testing process. It prepares the planned steps based on the risk factors and the testing phase. Once
the planned steps of the testing process are prepared, software testing techniques and testing
tools can be applied within these steps. Thus, testing is performed on this methodology.
However, if we don’t get the required results, the testing plans must be checked and modified
accordingly.

EFFECTIVE SOFTWARE TESTING VS EXHAUSTIVE SOFTWARE TESTING

Exhaustive or complete software testing means that every statement in the program and every
possible path combination with every possible combination of data must be executed. But soon,
we will realize that exhaustive testing is out of scope. That is why the questions arise: (i) When
are we done with testing? or (ii) How do we know that we have tested enough? There may be
many answers for these questions with respect to time, cost, customer, quality, etc. This section
will explore that exhaustive or complete testing is not possible. Therefore, we should concentrate
on effective testing which emphasizes efficient techniques to test the software so that important
features will be tested within the constrained resources.
The testing process should be understood as a domain of possible tests (see Fig. 1.7). There are
subsets of these possible tests. But the domain of possible tests becomes infinite, as we cannot
test every possible combination.

This combination of possible tests is infinite in the sense that the processing resources and time
are not sufficient for performing these tests. Computer speed and time constraints limit the
possibility of performing all the tests. Complete testing requires the organization to invest a long
time which is not cost-effective. Therefore, testing must be performed on selected subsets that
can be performed within the constrained resources. This selected group of subsets, but not the
whole domain of testing, makes effective software testing. Effective testing can be enhanced if
subsets are selected based on the factors which are required in a particular environment

Now, let us see in detail why complete testing is not possible.

The Domain of Possible Inputs to the Software is too Large to Test

If we consider the input data as the only part of the domain of testing, even then, we are not able
to test the complete input data combination. The domain of input data has four sub-parts: (a)
valid inputs, (b) invalid inputs, (c) edited inputs, and (d) race condition inputs

Valid inputs It seems that we can test every valid input on the software. But look at a very
simple example of adding two-digit two numbers. Their range is from –99 to 99 (total 199). So
the total number of test case combinations will be 199 × 199 = 39601. Further, if we increase the
range from two digits to four-digits, then the number of test cases will be 399,960,001. Most
addition programs accept 8 or 10 digit numbers or more. How can we test all these combinations
of valid inputs?

Invalid inputs Testing the software with valid inputs is only one part of the input sub-domain.
There is another part, invalid inputs, which must be tested for testing the software effectively.
The important thing in this case is the behavior of the program as to how it responds when a user
feeds invalid inputs. The set of invalid inputs is also too large to test. If we consider again the
example of adding two numbers, Then the following possibilities may occur from invalid inputs:

i. Numbers out of range

ii. Combination of alphabets and digits
iii. Combination of all alphabets
iv. Combination of control character
v. Combination of any other key on the keyboard

Edited inputs If we can edit inputs at the time of providing inputs to the program, then
many unexpected input events may occur. For example, you can add many spaces in the
input, which are not visible to the user. It can be a reason for non-functioning of the
program. In another example, it may be possible that a user is pressing a number key,
then Backspace key continuously and finally after sometime, he presses another number
key and Enter. Its input buffer overflows and the system crashes. The behavior of users
cannot be judged. They can behave in a number of ways, causing defect in testing a
program. That is why edited inputs are also not tested completely.
Race condition inputs The timing variation between two or more inputs is also one of
the issues that limit the testing. For example, there are two input events,
A and B. According to the design, A precedes B in most of the cases. But, B can also
come first in rare and restricted conditions. This is the race condition; whenever B
precedes A. Usually the program fails due to race conditions, as the possibility of
preceding B in restricted condition has not been taken care, resulting in a race condition
bug. In this way, there may be many race conditions in the system, especially in
multiprocessing systems and interactive systems. Race conditions are among the least
tested.
There are too Many Possible Paths through the Program to Test
A program path can be traced through the code from the start of a program to its
termination. Two paths differ if the program executes different statements in each, or
executes the same statements but in different order. A testing person thinks that if all the
possible paths of control flow through the program are executed, then possibly the
program can be said to be completely tested. However, there are two flaws in this
statement.
(i) The number of unique logic paths through a program is too large. This was
demonstrated by Myers[2] with an example shown in Fig. 1.9. It depicts a 10–20
statements program consisting of a DO loop that iterates up to 20 times. Within
the body of the DO loop is a set of nested IF statements. The number of all the
paths from point A to B is approximately 1014. Thus, all these paths cannot be
tested, as it may take years of time.

See another example for the code fragment shown in Fig. 1.10 and its corresponding fl
ow graph in Fig. 1.11 (We will learn how to convert the program into a fl ow graph in
Chapter 5).
Now calculate the number of paths in this fragment. For calculating the number of paths,
we must know how many paths are possible in one iteration. Here in our example, there
are two paths in one iteration. Now the total number of paths will be 2n + 1, where n is
the number of times the loop will be carried out, and 1 is added, as the for loop will exit
after its looping ends and it terminates. Thus, if n is 20, then the number of paths will be
220 + 1, i.e. 1048577. Therefore, all these paths cannot be tested, as it may take years.
(ii) The complete path testing, if performed somehow, does not guarantee that there
will not be errors. For example, it does not claim that a program matches its
specification. If one were asked to write an ascending order sorting program but
the developer mistakenly produces a descending order program, then exhaustive
path testing will be of little value. In another case, a program may be incorrect
because of missing paths. In this case, exhaustive path testing would not detect
the missing path.
Every Design Error Cannot be Found
Manna and Waldinger [15] have mentioned the following fact: ‘We can never be sure
that the specifications are correct.’ How do we know that the specifications are
achievable? Its consistency and completeness must be proved, and in general, that is a
provably unsolvable problem [9]. Therefore, specification errors are one of the major
reasons that make the design of the software faulty. If the user requirement is to have
measurement units in inches and the specification says that these are in meters, then the
design will also be in meters. Secondly, many user interface failures are also design
errors.
The study of these limitations of testing shows that the domain of testing is infinite and
testing the whole domain is just impractical. When we leave a single test case, the
concept of complete testing is abandoned. But it does not mean that we should not focus
on testing. Rather, we should shift our attention from exhaustive testing to effective
testing. Effective testing provides the flexibility to select only the subsets of the domain
of testing based on project priority such that the chances of failure in a particular
environment is minimized.
Software Testing Terminology and Methodology
Since software testing or rather effective software testing is an emerging fi eld, many
related terms are either undefi ned or yet to be known. We tend to use the terms ‘error’,
‘bug’, ‘defect’, ‘fault’, ‘failure’, etc. interchangeably. But they don’t mean the same.
Bugs in general, are more important during software testing. All the bugs do not have the
same level of severity. Some bugs are less important, as they are not going to affect the
product badly. On the other hand, bugs with high severity level will affect the product
such that the product may not be released. Therefore, bugs need to be classified according
to their severity. A bug has a complete cycle from its initiation to death. Moreover, these
bugs also affect a project cost at various levels of development. If there is a bug at the
specification level and it is caught, then it is more economical to debug it as compared to
the case if it is found in the implementation stage. If the bug propagates further into later
stages of software development life cycle (SDLC), it would become more costly to
debug. As we have discussed in the previous chapter, software testing today is recognized
as a complete process, and we need to identify various stages of software testing life
cycle. These stages, like SDLC stages, define various tasks to be done during the testing
of software in a hierarchy. Besides this, what should be our complete methodology to test
the software? How should we go for testing a complete project? What will be the
techniques used during testing? Should we use testing tools during testing? This chapter
defines all the terms related to software testing, some concepts related to these terms, and
then discusses the development of a testing methodology.
2.1 SOFTWARE TESTING TERMINOLOGY
2.1.1 DEFINITIONS
As mentioned earlier, terms like error, bug, failure, defect, etc. are not synonymous and
hence these should not be used interchangeably. All these terms are defined below
Failure
When the software is tested, failure is the first term being used. It means the inability of a
system or component to perform a required function according to its specification. In
other words, when results or behavior of the system under test are different as compared
to specified expectations, then failure exists.
Fault/Defect/Bug
Failure is the term which is used to describe the problems in a system on the output side,
as shown in Fig. 2.1. Fault is a condition that in actual causes a system to produce failure.
Fault is synonymous with the words defect or bug. Therefore, fault is the reason
embedded in any phase of SDLC and results in failures. It can be said that failures are
manifestation of bugs. One failure may be due to one or more bugs and one bug may
cause one or more failures. Thus, when a bug is executed, then failures are generated. But
this is not always true. Some bugs are hidden in the sense that these are not executed, as
they do not get the required conditions in the system. So, hidden bugs may not always
produce failures. They may execute only in certain rare conditions.

Error Whenever a development team member makes a mistake in any phase of SDLC,
errors are produced. It might be a typographical error, a misleading of a specifi cation, a
misunderstanding of what a subroutine does, and so on. Error is a very general term used
for human mistakes. Thus, an error causes a bug and the bug in turn causes failures, as
shown in Fig.
Suppose the module shown above is expected to print the value of x which is critical for
the use of software. But when this module will be executed, the value of x will not be
printed. It is a failure of the program. When we try to look for the reason of the failure,
we fi nd that in Module A(), the while loop is not being executed. A condition is
preventing the body of while loop to be executed. This is known as bug/defect/fault. On
close observation, we fi nd a semicolon being misplaced after the while loop which is not
its correct syntax and it is not allowing the loop to execute. This mistake is known as an
error.
Test case Test case is a well-documented procedure designed to test the functionality of a
feature in the system. A test case has an identity and is associated with a program
behaviour. The primary purpose of designing a test case is to fi nd errors in the system.
For designing the test case, it needs to provide a set of inputs and its corresponding
expected outputs. The sample for a test case is shown in Fig.

Test case ID is the identification number given to each test case.

Purpose defines why the case is being designed.
Preconditions for running the inputs in a system can be defined, if required, in a test
case.
Inputs should not be hypothetical. Actual inputs must be provided, instead of general
inputs. For example, if two integer numbers have to be provided as input, then
specifically mention them as 23 and 56. Expected outputs are the outputs which should
be produced when there is no failure. Test cases are designed based on the chosen testing
techniques. They provide inputs when the system is executed. After execution, observed
results are compared with expected outputs mentioned in the test case.
Test ware The documents created during testing activities are known as test ware.
Taking the analogy from software and hardware as a product, test ware are the
documents that a test engineer produces. It may include test plans, test specifications, test
case design, test reports, etc. Test ware documents should also be managed and updated
like a software product. Incident When a failure occurs, it may or may not be readily
apparent to the user. An incident is the symptom(s) associated with a failure that alerts
the user about the occurrence of a failure. Test oracle An oracle is the means to judge the
success or failure of a test, i.e. to judge the correctness of the system for some test. The
simplest oracle is comparing actual results with expected results by hand. This can be
very timeconsuming, so automated oracles are sought. 2.1.2 LIFE CYCLE OF A BUG
It should be clear that any member of the development team can make an error in any
phase of SDLC. If an error has been produced in the requirement specification phase and
not detected in the same phase, then it results in a bug in the next phase, i.e. the design
phase. In the design phase, a bug has come from the previous stage, but an error can also
be produced in this stage. Again, if the error in this phase is not detected and it passes on
to the next stage, i.e. coding phase, then it becomes a bug. In this way, errors and bugs
appear and travel through various stages of SDLC, as shown in Fig. 2.4. It means, one
stage may contain errors as well as bugs and the bugs which come from the previous
stage are harder to detect and debug. In the testing phase, we analyse the incidents when
the failure occurs. On the basis of symptoms derived from the incidents, a bug can be
classified into certain categories. After this, the bug can be isolated in the corresponding
phase of SDLC and resolved by finding its exact location. The whole life cycle of a bug
can be classified into two phases: (i) bugs-in phase and (ii) bugs-out phase.
Bugs-In Phase This phase is where the errors and bugs are introduced in the software.
Whenever we commit a mistake, it creates errors on a specific location of the software
and consequently, when this error goes unnoticed, it causes some conditions to fail,
leading to a bug in the software. This bug is carried out to the subsequent phases of
SDLC, if not detected. Thus, a phase may have its own errors as well as bugs received
from the previous phase. If you are not performing verification (discussed later in this
chapter) on earlier phases, then there is no chance of detecting these bugs.
Bugs-Out Phase If failures occur while testing a software product, we come to the
conclusion that it is affected by bugs. However, there are situations when bugs are
present, even though we don’t observe any failures. That is another issue of discussion. In
this phase, when we observe failures, the following activities are performed to get rid of
the bugs.
Bug classification In this part, we observe the failure and classify the bugs according to
its nature. A bug can be critical or catastrophic in nature or it may have no adverse effect
on the output behaviour of the software. In this way, we classify all the failures. This is
necessary, because there may be many bugs to be resolved. But a tester may not have
sufficient time. Thus, categorization of bugs may help by handling high criticality bugs
first and considering other trivial bugs on the list later, if time permits. We have taken
various considerations to classify different bugs (discussed later in this chapter).
Bug isolation Bug isolation is the activity by which we locate the module in which the
bug appears. Incidents observed in failures help in this activity. We observe the
symptoms and back-trace the design of the software and reach the module/files and the
condition inside it which has caused the bug. This is known as bug isolation.
Bug resolution Once we have isolated the bug, we back-trace the design to pinpoint the
location of the error. In this way, a bug is resolved when we have found the exact
location of its occurrence.
2.1.3 STATES OF A BUG
Based on the above discussion, a bug attains the following different states in its life cycle
(see Fig. 2.5)
New The state is new when the bug is reported first time by a tester.
Open The new state does not verify that the bug is genuine. When the test leader (test
organization hierarchy will be discussed in Chapter 9) approves that the bug is genuine,
its state becomes open.
Assign An open bug comes to the development team where the development team verifi
es its validity. If the bug is valid, a developer is assigned the job to fi x it and the state of
the bug now is ‘ASSIGN’.
Deferred The developer who has been assigned to fi x the bug will check its validity and
priority. If the priority of the reported bug is not high or there is not suffi cient time to test
it or the bug does not have any adverse effect on the software, then the bug is changed to
deferred state which implies the bug is expected to be fi xed in next releases.
Rejected It may be possible that the developer rejects the bug after checking its validity,
as it is not a genuine one.
Test After fi xing the valid bug, the developer sends it back to the testing team for next
round of checking. Before releasing to the testing team, the developer changes the bug’s
state to ‘TEST’. It specifi es that the bug has been fi xed by the development team but not
tested and is released to the testing team.
Verifi ed/fi xed The tester tests the software and verifi es whether the reported bug is fi
xed or not. After verifying, the developer approves that the bug is fi xed and changes the
status to ‘VERIFIED’. Reopened If the bug is still there even after fi xing it, the tester
changes its status to ‘REOPENED’. The bug traverses the life cycle once again. In
another case, a bug which has been closed earlier may be reopened if it appears again. In
this case, the status will be REOPENED instead of OPEN.
Closed Once the tester and other team members are confirmed that the bug is completely
eliminated, they change its status to ‘CLOSED’.

2.1.4 WHY DO BUGS OCCUR?

This is a very basic question for testers. The following points can answer this best:
To Err is Human
As discussed earlier, errors are produced when developers commit mistakes. Sample this.
The phone rang while coding and the developer got distracted, he pressed a wrong key
and the results of that mistake produced a bug in the system. But this is not the only
reason for bugs. Consider the second point.
Bugs in Earlier Stages go Undetected and Propagate
The phases of SDLC are connected to each other and the output of one phase becomes
the input to the next. Therefore, an undetected bug easily propagates into subsequent
phases. These propagated bugs, in later phases, are harder to detect and if found, are
costlier to debug. It is a proven fact that requirement specification and design phases
contain the largest percentage of bugs.
This point also establishes the fact that testing should be performed at each level of
SDLC. Errors are inevitable; however, we can prevent critical bugs from propagating into
later stages by checking minute details at each level of SDLC.
How do bugs get into a software product? There may be various reasons: unclear or
constantly changing requirements, software complexity, programming errors, timelines,
errors in bug tracking, communication gap, documentation errors, deviation from
standards, etc. Some examples are given below: „
 Miscommunication in gathering requirements from the customer is a big source of
bugs.
 If the requirements keep changing from time to time, it creates a lot of confusion
and pressure, both on the development as well as the testing team. Often, a new
feature added or an existing feature removed can be linked to other modules or
components in the software. „
 If the effect of changes in one module to another module is overlooked, it causes
bugs.
 Rescheduling of resources, re-doing or discarding an already completed work,
and changes in hardware/software requirements can affect the software. Assigning
a new developer to the project midway can cause bugs. If proper coding standards
have not been followed, then the new developer might not get all the relevant
details of the project. Improper code documentation and ineffective knowledge
transfer can limit the developer’s ability to produce error-free codes. Discarding a
portion of the existing code might just leave its trail behind in other parts of the
software. Overlooking or not eliminating such code can cause bugs. Serious bugs
can especially occur with larger projects, as it gets tougher to identify the problem
area. „
 Complexity in keeping a track of all the bugs can in turn cause more bugs. This
gets harder when a bug has a very complex life cycle, i.e. when the number of
times it has been closed, re-opened, not accepted, ignored, etc. goes on increasing.
2.1.5 BUGS AFFECT ECONOMICS OF SOFTWARE TESTING
Studies have demonstrated that testing prior to coding is 50% effective in detecting
errors and after coding, it is 80% effective. Moreover, it is at least 10 times as costly
to correct an error after coding as before, and 100 times as costly to correct a
production error (post-release error). This is how the bugs affect the economics of
testing.
There is no guarantee that all the errors will be recognized after testing and that the
testing phase will remove all of them. This is not possible practically. If the bugs
embedded in earlier stages go undetected, it is more difficult to detect them in later
stages. So the cost to remove bugs from a system increases, as the cost of a bug is
equal to Detection Cost + Correction Cost.
A bug found and fixed during early stages when the specification is being written,
cost next to nothing. The same bug, if not found until the software is coded and
tested, might cost ten times the cost in early stages. Moreover, after the release of the
product, if a customer fi nds the same bug, the cost will be 100 times more. Figure 2.6
outlines the relative cost to correct a defect depending on the SDLC in which it is
discovered. Therefore, cost increase is logarithmic.

2.1.6 BUG CLASSIFICATION BASED ON CRITICALITY

Bugs can be classified based on the impact they have on the software under test. This
classification can be used for the prioritization of bugs, as all bugs cannot be resolved
in one release. Since all bugs are not of the same criticality, prioritization will put
high criticality bugs on top of the list. We have divided bugs based on their criticality
in the following broad categories:
Critical Bugs This type of bugs has the worst effect such that it stops or hangs the
normal functioning of the software. The person using the software becomes helpless
when this type of bug appears. For example, in a sorting program, after providing the
input numbers, the system hangs and needs to be reset.
Major Bug This type of bug does not stop the functioning of the software but it
causes a functionality to fail to meet its requirements as expected. For example, in a
sorting program, the output is being displayed but not the correct one.
Medium Bugs Medium bugs are less critical in nature as compared to critical and
major bugs. If the outputs are not according to the standards or conventions, e.g.
redundant or truncated output, then the bug is a medium bug.
Minor Bugs These types of bugs do not affect the functionality of the software.
These are just mild bugs which occur without any effect on the expected behaviour or
continuity of the software. For example, typographical error or misaligned printout.
2.1.7 BUG CLASSIFICATION BASED ON SDLC
Since bugs can appear in any phase of SDLC, they can be classified based on SDLC
phases which are described below [9, 48].
Requirements and Specifications Bugs The fi rst type of bug in SDLC is in the
requirement gathering and specification phase. It has been observed that most of the
bugs appear in this phase only. If these bugs go undetected, they propagate into
subsequent phases. Requirement gathering and specification is a difficult phase in the
sense that requirements gathered from the customer are to be converted into a
requirement specification which will become the base for design. There may be a
possibility that requirements specified are not exactly what the customers want.
Moreover, specified requirements may be incomplete, ambiguous, or inconsistent.
Specification problems lead to wrong missing, or superfl uous features.
Design Bugs Design bugs may be the bugs from the previous phase and in addition
those errors which are introduced in the present phase. The following design errors
may be there. Control flow bugs If we look at the control flow of a program (through
control flow graph—discussed in Chapter 5), then there may be many errors. For
example, some paths through the flow may be missing; there may be unreachable
paths, etc.
Logic bugs Any type of logical mistakes made in the design is a logical bug. For
example, improper layout of cases, missing cases, improper combination of cases,
misunderstanding of the semantics of the order in which a Boolean expression is
evaluated.
Processing bugs Any type of computation mistakes result in processing bugs.
Examples include arithmetic error, incorrect conversion from one data representation
to another, ignoring overfl ow, improper use of logical operators, etc.
Data flow bugs Control flow cannot identify data errors. For this, we use data flow
(through data flow graph—discussed in Chapter 5). There may be data flow anomaly
errors like un-initialized data, initialized in wrong format, data initialized but not
used, data used but not initialized, redefined without any intermediate use, etc.
Error handling bugs There may be errors about error handling in the software.
There are situations in the system when exception handling mechanisms must be
adopted. If the system fails, then there must be an error message or the system should
handle the error in an appropriate way. If you forget to do all this, then error handling
bugs appear.
Race condition bugs Race conditions (discussed in Chapter 1) also lead to bugs.
Sometimes these bugs are irreproducible.
Boundary-related bugs Most of the time, the designers forget to take into
consideration what will happen if any aspect of a program goes beyond its minimum
and maximum values. For example, there is one integer whose range is between 1 to
100. What will happen if a user enters a value as 1 or 101? When the software fails at
the boundary values, then these are known as boundary-related bugs. There may be
boundaries in loop, time, memory, etc. User interface bugs There may be some
design bugs that are related to users. If the user does not feel good while using the
software, then there are user interface bugs. Examples include inappropriate
functionality of some features; not doing what the user expects; missing, misleading,
or confusing information; wrong content in the help text; inappropriate error
messages, etc.
Coding Bugs There may be a long list of coding bugs. If you are a programmer, then
you are aware of some common mistakes made. For example, undeclared data,
undeclared routines, dangling code, typographical errors, documentation bugs, i.e.
erroneous comments lead to bugs in maintenance.
Interface and Integration Bugs
 External interface bugs include invalid timing or sequence assumptions related to
external signals, misunderstanding external input and output formats, and user
interface bugs. Internal interface bugs include input and output format bugs,
inadequate protection against corrupted data, wrong subroutine call sequence, call
parameter bugs, and misunderstood entry or exit parameter values. Integration bugs
result from inconsistencies or incompatibilities between modules discussed in the
form of interface bugs. There may be bugs in data transfer and data sharing between
the modules.
System Bugs There may be bugs while testing the system as a whole based on various
parameters like performance, stress, compatibility, usability, etc. For example, in a real-
time system, stress testing is very important, as the system must work under maximum
load. If the system is put under maximum load at every factor like maximum number of
users, maximum memory limit, etc. and if it fails, then there are system bugs.
Testing Bugs One can question the presence of bugs in the testing phase because this
phase is dedicated to finding bugs. But the fact is that bugs are present in testing phase
also. After all, testing is also performed by testers – humans. Some testing mistakes are:
failure to notice/report a problem, failure to use the most promising test case, failure to
make it clear how to reproduce the problem, failure to check for unresolved problems just
before the release, failure to verify fixes, failure to provide summary report.
2.1.8 TESTING PRINCIPLES
Now it is time to learn the testing principles that are largely based on the discussion
covered in the fi rst chapter and the present one. These principles can be seen as
guidelines for a tester. Effective testing, not exhaustive testing
All possible combinations of tests become so large that it is impractical to test them all.
So considering the domain of testing as infi nite, exhaustive testing is not possible.
Therefore, the tester’s approach should be based on effective testing to adequately cover
program logic and all conditions in the component level design.
Testing is not a single phase performed in SDLC
Testing is not just an activity performed after the coding in SDLC. As discussed, the
testing phase after coding is just a part of the whole testing process. Testing process starts
as soon as the specifications for the system are prepared and it continues till the release of
the product. Destructive approach for constructive testing
Testers must have the psychology that bugs are always present in the program and they
must think about the technique of how to uncover them (this is their art of creativity).
This psychology of being always suspicious about bugs is a negative/destructive
approach. However, it has been proved that such a destructive approach helps in
performing constructive and effective testing. Thus, the criterion to have a successful
testing is to discover more and more bugs, and not to show that the system does not
contain any bugs.
Early testing is the best policy
When is the right time to start the testing process? As discussed earlier and we will
explore later that testing process is not a phase after coding, rather it starts as soon as
requirement specifications are prepared. Moreover, the cost of bugs can be reduced
tenfold, as bugs are harder to detect in later stages if they go undetected. Thus, the policy
in testing is to start as early as possible.
Probability of existence of an error in a section of a program is proportional to the
number of errors already found in that section Suppose the history of a software is
that you found 50 errors in Module X, 12 in Module Y, and 3 in Module Z. The software
was debugged but after a span of time, we fi nd some errors again and the software is
given to a tester for testing. Where should the tester concentrate to fi nd the bugs? This
principle says that the tester should start with Module X which has the history of
maximum errors. Another way of stating it is that errors seem to come in clusters. The
principle provides us the insight that if some sections are found error-prone in testing,
then our next testing effort should give priority to these error-prone sections.
Testing strategy should start at the smallest module level and expand towards the
whole program This principle supports the idea of incremental testing. Testing must
begin at the unit or module level, gradually progressing towards integrated modules and
fi nally the whole system. Testing cannot be performed directly on the whole system. It
must start with the individual modules and slowly integrate the modules and test them.
After this, the whole system should be tested.
Testing should also be performed by an independent team
 When programmers develop the software, they test it at their individual modules.
However, these programmers are not good testers of their own software. They are
basically constructors of the software, but testing needs a destructive approach.
Programmers always think positively that their code does not contain bugs. Moreover,
they are biased towards the correct functioning of the specifi ed requirements and not
towards detecting bugs. Therefore, it is always recommended to have the software tested
by an independent testing team. Testers associated with the same project can also help in
this direction, but this is not effective. For effective testing, the software may also be sent
outside the organization for testing.
Everything must be recorded in software testing
As mentioned earlier, testing is not an intuitive process; rather it is a planned process. It
demands that every detail be recorded and documented. We must have the record of
every test case run and the bugs reported. Even the inputs provided during testing and the
corresponding outputs are to be recorded. Executing the test cases in a recorded and
documented way can greatly help while observing the bugs. Moreover, observations can
be a lesson for other projects. So the experience with the test cases in one project can be
helpful in other projects. Invalid inputs and unexpected behaviour have a high
probability of finding an error
Whenever the software is tested, we test for valid inputs and for the functionality that the
software is supposed to do. But thinking in a negative way, we must test the software
with invalid inputs and the behaviour which is not expected in general. This is also a part
of effective testing.
Testers must participate in specifi cation and design reviews
Testers’ role is not only to get the software and documents and test them. If they are not
participating in other reviews like specifi cation and design, it may be possible that either
some specifi cations are not tested or some test cases are built for no specifi cations. Let
us consider a program. Let S be the set of specifi ed behaviours of the program, P be the
implementation of the program, and T be the set of test cases. Now consider the
following cases (see Fig. 2.7):
(i) There may be specifi ed behaviours that are not tested (regions 2 and 5).
(ii) Test cases that correspond to unspecifi ed behaviours (regions 3 and 7)
 (iii) Program behaviours that are not tested (regions 2 and 6).
The good view of testing is to enlarge the area of region 1. Ideally, all three sets S, P, and T must
overlap each other such that all specifi cations are implemented and all implemented specifi
cations are tested. This is possible only when the test team members participate in all discussions
regarding specifi cations and design.

2.2 SOFTWARE TESTING LIFE CYCLE (STLC)

Since we have recognized software testing as a process, like SDLC, there is need for a
well-defi ned series of steps to ensure successful and effective software testing. This
systematic execution of each step will result in saving time and effort. Moreover, the
chances are that more number of bugs will be uncovered. The testing process divided into
a well-defi ned sequence of steps is termed as software testing life cycle (STLC). The
major contribution of STLC is to involve the testers at early stages of development. This
has a signifi cant benefi t in the project schedule and cost. The STLC also helps the
management in measuring specifi c milestones. STLC consists of the following phases
(see Fig. 2.8):
Test Planning The goal of test planning is to take into account the important issues of
testing strategy, viz. resources, schedules, responsibilities, risks, and priorities, as a
roadmap. Test planning issues are in tune with the overall project planning. Broadly,
following are the activities during test planning: „
 Defining the test strategy. „
 Estimate the number of test cases, their duration, and cost. „
 Plan the resources like the manpower to test, tools required, documents required.
„
 Identifying areas of risks. „
 Defi ning the test completion criteria. „
 Identifi cation of methodologies, techniques, and tools for various test cases. „
 Identifying reporting procedures, bug classification, databases for testing, bug
severity levels, and project metrics.
Based on the planning issues as discussed above, analysis is done for various testing
activities. The major output of test planning is the test plan document. Test plans are
developed for each level of testing. After analysing the issues, the following activities are
performed: „
 Develop a test case format. „
 Develop test case plans according to every phase of SDLC. „
 Identify test cases to be automated (if applicable). „
 Prioritize the test cases according to their importance and criticality. „
  Defi ne areas of stress and performance testing. „
 Plan the test cycles required for regression testing.
Test Design One of the major activities in testing is the design of test cases. However,
this activity is not an intuitional process; rather it is a well-planned process. The test
design is an important phase after test planning. It includes the following critical
activities.
Determining the test objectives and their prioritization
This activity decides the broad categories of things to test. The test objectives reflect the
fundamental elements that need to be tested to satisfy an objective. For this purpose, you
need to gather reference materials like software requirements specification and design
documentation. Then on the basis of reference materials, a team of experts compile a list
of test objectives. This list should also be prioritized depending upon the scope and risk.
Preparing list of items to be tested
The objectives thus obtained are now converted into lists of items that are to be tested
under an objective.
Mapping items to test cases
After making a list of items to be tested, there is a need to identify the test cases. A
matrix can be created for this purpose, identifying which test case will be covered by
which item. The existing test cases can also be used for this mapping.
Thus it permits reusing the test cases. This matrix will help in:
(a) Identifying the major test scenarios.
(b) Identifying and reducing the redundant test cases.
(c) Identifying the absence of a test case for a particular objective and as a result,
creating them
Designing the test cases demands a prior analysis of the program at functional or
structural level. Thus, the tester who is designing the test cases must understand the
cause-and-effect connections within the system intricacies. But look at the rule quoted by
Tsuneo Yamaura—There is only one rule in designing test cases: Cover all features, but
do not make too many test cases.
Some attributes of a good test case are given below:
(a) A good test case is one that has been designed keeping in view the criticality and
high-risk requirements in order to place a greater priority upon, and provide added depth
for testing the most important functions [12].
(b) A good test case should be designed such that there is a high probability of fi nding an
error. (c) Test cases should not overlap or be redundant. Each test case should address a
unique functionality, thereby not wasting time and resources.
(c) Although it is sometimes possible to combine a series of tests into one test case, a
good test case should be designed with a modular approach so that there is no complexity
and it can be reused and recombined to execute various functional paths. It also avoids
masking of errors and duplication of test-creation efforts [7, 12].
(d) A successful test case is one that has the highest probability of detecting an as-yet-
undiscovered error [2]
Selection of test case design techniques
While designing test cases, there are two broad categories, namely black-box testing and
white-box testing. Black-box test case design techniques generate test cases without
knowing the internal working of a system. These will be discussed later in this chapter.
The techniques to design test cases are selected such that there is more coverage and the
system detects more bugs.
Creating test cases and test data
The next step is to create test cases based on the testing objectives identified. The test
cases mention the objective under which a test case is being designed, the inputs required,
and the expected outputs. While giving input specifications, test data must also be chosen
and specified with care, as this may lead to incorrect execution of test cases.
Setting up the test environment and supporting tools
The test created above needs some environment settings and tools, if applicable. So
details like hardware configurations, testers, interfaces, operating systems, and manuals
must be specifi ed during this phase.
Creating test procedure specification This is a description of how the test case will be
run. It is in the form of sequenced steps. This procedure is actually used by the tester at
the time of execution of test cases.
Thus, the hierarchy for test design phase includes: developing test objectives, identifying
test cases and creating their specifications, and then developing test case procedure
specifications as shown in Fig. 2.9. All the details specified in the test design phase are
documented in the test design specification. This document provides the details of the
input specifications, output specifications, environmental needs, and other procedural
requirements for the test case.

Test Execution
In this phase, all test cases are executed including verification and validation.
Verification test cases are started at the end of each phase of SDLC. Validation test cases
are started after the completion of a module. It is the decision of the test team to opt for
automation or manual execution. Test results are documented in the test incident reports,
test logs, testing status, and test summary reports, as shown in Fig. 2.10.

Responsibilities at various levels for execution of the test cases are outlined in Table 2.1.
Table 2.1 Testing level vs responsibility
Test Execution Level Person Responsible
Unit Developer of the module
Integration Testers and Developers
System Testers, Developers, End-users
Acceptance Testers, End-users

Post-Execution/Test Review
 As we know, after successful test execution, bugs will be reported to the concerned
developers. This phase is to analyse bug-related issues and get feedback so that maximum
number of bugs can be removed. This is the primary goal of all test activities done earlier.
As soon as the developer gets the bug report, he performs the following activities.
Understanding the bug
The developer analyses the bug reported and builds an understanding of its whereabouts.
Reproducing the bug
Next, he confi rms the bug by reproducing the bug and the failure that exists. This is
necessary to cross-check failures. However, some bugs are not reproducible which
increases the problems of developers.
Analysing the nature and cause of the bug
After examining the failures of the bug, the developer starts debugging its symptoms and
tracks back to the actual location of the error in the design. The process of debugging has
been discussed in detail in Chapter 17.
After fixing the bug, the developer reports to the testing team and the modifi ed portion
of the software is tested once again.
After this the results from manual and automated testing can be collected.
The final bug report and associated metrics are reviewed and analysed for overall testing
process. The following activities can be done: „
 Reliability analysis can be performed to establish whether the software meets the
predefined reliability goals or not. If so, the product can be released and the
decision on a release date can be taken. If not, then the time and resources
required to reach the reliability goals are outlined.
 Coverage analysis can be used as an alternative criterion to stop testing. „
 Overall defect analysis can identify risk areas and help focus our efforts on
quality improvement.
2.3 SOFTWARE TESTING METHODOLOGY
Software testing methodology is the organization of software testing by means of which the test
strategy and test tactics are achieved, as shown in Fig. 2.11. All the terms related to software
testing methodology and a complete testing strategy is discussed in this section.
2.3.1 SOFTWARE TESTING STRATEGY
Testing strategy is the planning of the whole testing process into a well-planned series of
steps. In other words, strategy provides a roadmap that includes very specific activities
that must be performed by the test team in order to achieve a specific goal.
In Chapter 1, risk reduction was described as one of the goals of testing. In fact, when a
test strategy is being developed, risk reduction is addressed first. The components of a
testing strategy are discussed below:
Test Factors
Test factors are risk factors or issues related to the system under development. Risk
factors need to be selected and ranked according to a specific system under development.
The testing process should reduce these test factors to a prescribed level.
Test Phase This is another component on which the testing strategy is based. It refers to
the phases of SDLC where testing will be performed. Testing strategy may be different
for different models of SDLC, e.g. strategies will be different for waterfall and spiral
models.
2.3.2 TEST STRATEGY MATRIX
A test strategy matrix identifi es the concerns that will become the focus of test planning
and execution. In this way, this matrix becomes an input to develop the testing strategy.
The matrix is prepared using test factors and test phase (Table 2.2). The steps to prepare
this matrix are discussed below.
Select and rank test factors
Based on the test factors list, the most appropriate factors according to specifi c systems
are selected and ranked from the most signifi cant to the least. These are the rows of the
matrix. Identify system development phases
Different phases according to the adopted development model are listed as columns of
the matrix. These are called test phases.
Identify risks associated with the system under development
In the horizontal column under each of the test phases, the test concern with the strategy
used to address this concern is entered. The purpose is to identify the concerns that need
to be addressed under a test phase. The risks may include any events, actions, or
circumstances that may prevent the test program from being implemented or executed
according to a schedule, such as late budget approvals, delayed arrival of test equipment,
or late availability of the software application. As risks are identified, they must be
assessed for impact and then mitigated with strategies for overcoming them, because
risks may be realized despite all precautions having been taken. The test team must
carefully examine risks in order to derive effective test and mitigation strategies for a
particular application. Concerns should be expressed as questions so that the test strategy
becomes a high-level focus for testers when they reach the phase where it’s most
appropriate to address a concern.
Creating a test strategy Let’s take a project as an example. Suppose a new operating
system has to be designed, which needs a test strategy. The following steps are used
(Table 2.3).

Select and rank test factors

A critical factor to be considered for the development of an operating system is
portability. This is the effort required to transfer a program from one hardware
configuration to another. This factor matters the most, as the operating system has to be
compatible with most hardware configurations.
Identify the test phases
In this step, all test phases affected by the selected test factors are identified. All the
affected test phases can be seen according to the test factors in Table 2.3.
Identify the risks associated with each test factor and its corresponding test phase
All the risks are basic concerns associated with each factor in a phase, and are expressed
in the form of a question. For example, ‘Is testing performed successfully on INTEL,
MIPS H/W platforms?’
Plan the test strategy for every risk identified
After identifying the risks, it is required to plan a strategy to tackle them. It helps testers
to start working on testing so that risks are mitigated.
Test strategies generally must concentrate on risks associated with cost overruns,
schedule slippage, critical software errors, and other failures. Thus test-strategy design is
developed with risks, constrained resources, time limits, and budget restrictions.
2.3.3 DEVELOPMENT OF TEST STRATEGY
When the project under considration starts and progresses, testing too starts from the first
step of SDLC. Therefore, the test strategy should be such that the testing process
continues till the implementation of project. Moreover, the rule for development of a test
strategy is that testing ‘begins from the smallest unit and progresses to enlarge’. This
means the testing strategy should start at the component level and fi nish at the
integration of the entire system. Thus, a test strategy includes testing the components
being built for the system, and slowly shifts towards testing the whole system. This gives
rise to two basic terms—Verification and Validation—the basis for any type of testing. It
can also be said that the testing process is a combination of verification and validation.
The purpose of verification is to check the software with its specification at every
development phase such that any defect can be detected at an early stage of testing and
will not be allowed to propagate further. That is why verification can be applied to all
stages of SDLC. So verification refers to the set of activities that ensures correct
implementation of functions in a software. However, as we progress down to the
completion of one module or system development, the scope of verification decreases.
The validation process starts replacing the verification in the later stages of SDLC.
Validation is a very general term to test the software as a whole in conformance with
customer expectations. According to Boehm [5]—
Verification is ‘Are we building the product right?’
Validation is ‘Are we building the right product?
You can relate verification and validation to every complex task of daily life. You divide
a complex task into many sub-tasks. In this case, every sub-task is developed and
accomplished towards achieving the complex task. Here, you check every sub-task to
ensure that you are working in the right direction. This is verification. After the sub-tasks
have been completed and merged, the entire task is checked to ensure the required task
goals have been achieved. This is validation.
Verification is checking the work at intermediate stages to confirm that the project is
moving in the right direction, towards the set goal. When a module is prepared with
various stages of SDLC like plan, design and code, it is verified at every stage. But there
may be various modules in the project.
These needs to be integrated, after which the full system is built. However, if we simply
integrate the modules and build the system for the customer, then we are leaving open
spaces for the bugs to intrude. Therefore, after building individual modules, the following
stages need to be tested: the module as a whole, integration of modules, and the system
built after integration. This is validation testing.
2.3.4 TESTING LIFE CYCLE MODEL
Verification and validation (V&V) are the building blocks of a testing process. The
formation of test strategy is based on these two terms only. V&V can be best understood
when these are modeled in the testing process. This model is known as the Testing Life
Cycle Model. Life cycle involves continuous testing of the system during the
development process. At predetermined points, the results of the development process are
inspected to determine the correctness of implementation. These inspections identify
defects as early as possible. But, life cycle testing is dependent upon the completion of
predetermined deliverables at a specified point in the development life cycle. It also
shows the paths for various types of testing.
V-Testing Life Cycle Model
In V-testing concept [4], as the development team attempts to implement the software,
the testing team concurrently starts checking the software. When the project starts, both
the system development and the system test process begin. The team that is developing
the system begins the system development process and the team that is conducting the
system test begins planning the system test process, as shown in Fig. 2.12. Both teams
start at the same point using the same information. If there are risks, the tester develops a
process to minimize or eliminate them.
Figure 2.12 shown above can also be expanded. In Fig. 2.13, on the left arm of the V, the
development cycle is progressing and on the right arm, the corresponding testing stages
are moving. As said earlier, in the early stages of SDLC, testing comprises more
verification activities and towards the later stages, the emphasis is on validation.

The V&V process, in a nutshell, involves (i) verification of every step of SDLC (all
verification activities are explained in the next chapter) and (ii) validation of the verified
system at the end.
2.3.5 VALIDATION ACTIVITIES
Validation has the following three activities which are also known as the three levels of
validation testing. Unit Testing It is a major validation effort performed on the smallest
module of the system. If avoided, many bugs become latent bugs and are released to the
customer. Unit testing is a basic level of testing which cannot be overlooked, and
confirms the behaviour of a single module according to its functional specifications.
Integration Testing It is a validation technique which combines all unit-tested modules
and performs a test on their aggregation. One may ask, when we have tested all modules
in unit testing, where is the need to test them on aggregation? The answer is interfacing.
Unit modules are not independent, and are related to each other by interface
specifications between them. When we unit test a module, its interfacing with other
modules remain untested. When one module is combined with another in an integrated
environment, interfacing between units must be tested. If some data structures, messages,
or other things are common between some modules, then the standard format of these
interfaces must be checked during integration testing, otherwise these will not be able to
interface with each other.
But how do we integrate the units together? Is it a random process? It is actually a
systematic technique for combining modules. In fact, interfacing among modules is
represented by the system design. We integrate the units according to the design and
availability of units. Therefore, the tester must be aware of the system design.
System Testing
This testing level focuses on testing the entire integrated system. It incorporates many
types of testing, as the full system can have various users in different environments. The
purpose is to test the validity for specific users and environments. The validity of the
whole system is checked against the requirement specifications.
2.3.6 TESTING TACTICS
The ways to perform various types of testing under a specific test strategy are discussed
below. Software Testing Techniques
In the previous sections, it has been observed that complete or exhaustive testing is not
possible. Instead, our effort should be on effective testing. How -ever, effective testing is
a real challenge in the domain of testing. At this stage, testing can be defined as the
design of effective test cases such that most of the testing domains will be covered
detecting the maximum number of bugs. As Myers [2] said—
Given constraints on time, cost, computer time, etc., the key issue of testing becomes —
What subset of all possible test cases has the highest probability of detecting the most
errors?
Therefore, the next objective is to find the technique which will meet both the objectives
of effective test case design, i.e. coverage of testing domain and detection of maximum
number of bugs. The technique used to design effective test case is called Software
Testing Technique.
Till now, we have discussed the overall strategy for testing methodology. V&V and the
levels of testing under this process describe only the organization and planning of
software testing. Actual methods for designing test cases, i.e. software testing techniques,
implement the test cases on the software. These techniques can be categorized into two
parts: (a) static testing and (b) dynamic testing.
Static Testing It is a technique for assessing the structural characteristics of source code,
design specifications or any notational representation that conforms to well-defined
syntactic rules [16]. It is called as static because we never execute the code in this
technique. For example, the structure of code is examined by the teams but the code is
not executed.
Dynamic Testing
All the methods that execute the code to test software are known as dynamic testing
techniques. In this technique, the code is run on a number of inputs provided by the user
and the corresponding results are checked.
This type of testing is further divided into two parts: (a) black-box testing and (b) white-
box testing.
Black-box testing
This technique takes care of the inputs given to a system and the output is received after
processing in the system. What is being processed in the system? How does the system
perform these operations? Black-box testing is not concerned with these questions. It
checks the functionality of the system only. That is why the term black-box is used. It is
also known as functional testing. It is used for system testing under validation.
White-box testing
This technique complements black-box testing. Here, the system is not a black box.
Every design feature and its corresponding code is checked logically with every possible
path execution. So, it takes care of the structural paths instead of just outputs. It is also
known as structural testing and is used for unit testing under verification.
Testing Tools
Testing tools provide the option to automate the selected testing technique with the help
of tools. A tool is a resource for performing a test process. The combination of tools and
testing techniques enables the test process to be performed. The tester should first
understand the testing techniques and then go for the tools that can be used with each of
the techniques.
2.3.7 CONSIDERATIONS IN DEVELOPING TESTING METHODOLOGIES
The considerations in developing a testing methodology are described below [4].
Determine Project Risks
A test strategy is developed with the help of another team familiar with the business risks
associated with the software. The major issue in determining the strategy is to identify the
risks associated with the project. What are the high priority risks? What are the
consequences, if risks are not handled?
Determine the Type of Development Project
The environment or methodology to develop the software also affects the testing risks.
The risks associated with a new development project will be different from a
maintenance project or a purchased software.
Identify Test Activities According to SDLC Phase
After identifying all the risk factors in an SDLC phase, testing can be started in that
phase. However, all the testing activities must be recognized at all the SDLC phases.
Build the Test Plan
A tactical test plan is required to start testing. This test plan provides: „
 Environment and pre-test background „
 Overall objectives „
 Test team „
 Schedule and budget showing detailed dates for the testing „
 Resource requirements including equipment, software, and personnel „
 Testing materials including system documentation, software to be tested, test
inputs, test documentation, test tools „
 Specified business functional requirements and structural functions to be tested „
 Type of testing technique to be adopted in a particular SDLC phase or what are
the specific tests to be conducted in a particular phase
Since the internal design of software may be composed of many components, each of
these components or units must also have its own test plan. The idea is that units
should not be submitted without testing. So an extra effort must be put in preparing
unit test plans also.