Methods of Software Testing

There are two basic methods of performing software testing:

1. Manual testing
2. Automated testing
Manual Software Testing
As the name would imply, manual software testing is the process of an individual or individuals manually
testing software. This can take the form of navigating user interfaces, submitting information, or even trying
to hack the software or underlying database. As one might presume, manual software testing is labor-
intensive and slow. There are some things for which manual software testing is appropriate, including:
 User interface or usability testing
 Exploratory/ad hoc testing (where testers do not follow a 'script', but rather testers 'explore' the
application and use their instincts to find bugs)
 Testing areas of the application which experience a lot of change.
 User acceptance testing (often, this can also be automated)
The time commitment involved with manual software testing is one of its most significant drawbacks. The
time needed to fully test the system will typically range from weeks to months. Variability of results
depending on who is performing the tests can also be a problem. For these reasons, many companies look to
automation as a means of accelerating the software testing process while minimizing the variability of results.
Automated Software Testing
Automated software testing is the process of creating test scripts that can then be run automatically,
repetitively, and through many iterations. Done properly, automated software testing can help to minimize
the variability of results, speed up the testing process, increase test coverage (the number of different things
tested), and ultimately provide greater confidence in the quality of the software being tested.
There are, however, some things for which automated software testing is not appropriate. These include:
 End user usability testing is not typically a good candidate for automated testing.
 Tests which will not be run more than a couple of times are typically not a good candidate for
automated tasting, since the payoff of in test automation comes after many test executions.
 Tests for areas of the application which experience a lot of change are also not a good candidate for
automation since this can lead to substantial maintenance of test automation scripts. Such areas of the
application may be more effectively tested manually.
It is important to note that test automation is software, and just like the software you are building for internal
or external customers, it must be well-architected. A good test automation architecture, such as a keyword-
driven testing framework, will reduce the overall cost of ownership of your test automation by minimizing
maintenance expense and increasing the number of automated tests, allowing your organization to run more
tests (and achieve higher quality) for the same investment of time and money.
Testing is a process used to help identify the correctness, completeness and quality of developed computer
software. With that in mind, testing can never completely establish the correctness of computer software.

There are many approaches to software testing, but effective testing of complex products is essentially a
process of investigation, not merely a matter of creating and following rote procedure. One definition of
testing is "the process of questioning a product in order to evaluate it", where the "questions" are things the
tester tries to do with the product, and the product answers with its behavior in reaction to the probing of the
tester. Although most of the intellectual processes of testing are nearly identical to that of review or
inspection, the word testing is connoted to mean the dynamic analysis of the product—putting the product
through its paces.

The quality of the application can and normally does vary widely from system to system but some of the
common quality attributes include reliability, stability, portability, maintainability and usability. Refer to the
ISO standard ISO 9126 for a more complete list of attributes and criteria.

Testing helps is Verifying and Validating if the Software is working as it is intended to be working. Thins
involves using Static and Dynamic methodologies to Test the application.

Because of the fallibility of its human designers and its own abstract, complex nature, software development
must be accompanied by quality assurance activities. It is not unusual for developers to spend 40% of the
total project time on testing. For life-critical software (e.g. flight control, reactor monitoring), testing can cost
3 to 5 times as much as all other activities combined. The destructive nature of testing requires that
the developer discard preconceived notions of the correctness of his/her developed software.

Software Testing Fundamentals

Testing objectives include

1. Testing is a process of executing a program with the intent of finding an error.

2. A good test case is one that has a high probability of finding an as yet undiscovered error.
3. A successful test is one that uncovers an as yet undiscovered error.

Testing should systematically uncover different classes of errors in a minimum amount of time and with a
minimum amount of effort. A secondary benefit of testing is that it demonstrates that the software appears to
be working as stated in the specifications. The data collected through testing can also provide an indication of
the software's reliability and quality. But, testing cannot show the absence of defect -- it can only show that
software defects are present

Testing Start Process

Testing is sometimes incorrectly thought as an after-the-fact activity; performed after programming is done
for a product. Instead, testing should be performed at every development stage of the product. Test data sets
must be derived and their correctness and consistency should be monitored throughout the development
process. 

If we divide the lifecycle of software development into “Requirements Analysis”, “Design”,

“Programming/Construction” and “Operation and Maintenance”, then testing should accompany each of the
above phases. If testing is isolated as a single phase late in the cycle, errors in the problem statement or
design may incur exorbitant costs. Not only must the original error be corrected, but the entire structure built
upon it must also be changed. Therefore, testing should not be isolated as an inspection activity. Rather
testing should be involved throughout the SDLC in order to bring out a quality product.

Testing Activities in Each Phase

The following testing activities should be performed during the phases:

1. Requirements Analysis

- Determine correctness 
- Generate functional test data.

2. Design

- Determine correctness and consistency 

- Generate structural and functional test data.

3. Programming/Construction

- Determine correctness and consistency 

- Generate structural and functional test data 
- Apply test data 
- Refine test data

4. Operation and Maintenance

- Retest

Requirements Analysis

The following test activities should be performed during this stage:

1.1 Invest in analysis at the beginning of the project - Having a clear, concise and formal statement of the
requirements facilitates programming, communication, error analysis and test data generation.

The requirements statement should record the following information and decisions:

a. Program function - What the program must do?

b. The form, format, data types and units for input.

c. The form, format, data types and units for output.

d. How exceptions, errors and deviations are to be handled.

e. For scientific computations, the numerical method or at least the required accuracy of the solution.

f. The hardware/software environment required or assumed (e.g. the machine, theoperating system, and the
implementation language).

Deciding the above issues is one of the activities related to testing that should be performed during this stage.
1.2 Start developing the test set at the requirements analysis phase - Data should be generated that can be
used to determine whether the requirements have been met. To do this, the input domain should be
partitioned into classes of values that the program will treat in a similar manner and for each class a
representative element should be included in the test data. 

In addition, following should also be included in the data set:

(1) boundary values 

(2) any non-extreme input values that would require special handling.

The output domain should be treated similarly.

Invalid input requires the same analysis as valid input.

1.3 The correctness, consistency and completeness of the requirements should also be analyzed - Consider
whether the correct problem is being solved, check for conflicts and inconsistencies among the requirements
and consider the possibility of missing cases.

Design

The design document aids in programming, communication, and error analysis and test data generation. The
requirements statement and the design document should together give the problem and the organization of
the solution i.e. what the program will do and how it will be done.

The design document should contain:

 Principal data structures.

 Functions, algorithms, heuristics or special techniques used for processing.

 The program organization, how it will be modularized and categorized into external and internal
interfaces.

 Any additional information.

Here the testing activities should consist of:

- Analysis of design to check its completeness and consistency - the total process should be analyzed to
determine that no steps or special cases have been overlooked. Internal interfaces, I/O handling
and data structures should specially be checked for inconsistencies.

- Analysis of design to check whether it satisfies the requirements - check whether both requirements and
design document contain the same form, format, units used for input and output and also that all functions
listed in the requirement document have been included in the design document. Selected test data which is
generated during the requirements analysis phase should be manually simulated to determine whether the
design will yield the expected values.
- Generation of test data based on the design - The tests generated should cover the structure as well as the
internal functions of the design like the data structures, algorithm, functions, heuristics and general program
structure etc. Standard extreme and special values should be included and expected output should be
recorded in the test data.

- Re-examination and refinement of the test data set generated at the requirements analysis phase.

The first two steps should also be performed by some colleague and not only the designer/developer.

Programming/Construction

Here the main testing points are:

- Check the code for consistency with design - the areas to check include modular structure, module
interfaces, data structures, functions, algorithms and I/O handling.

- Perform the Testing process in an organized and systematic manner with test runs dated, annotated and
saved. A plan or schedule can be used as a checklist to help theprogrammer organize testing efforts. If errors
are found and changes made to the program, all tests involving the erroneous segment (including those which
resulted in success previously) must be rerun and recorded.

- Asks some colleague for assistance - Some independent party, other than the programmer of the specific
part of the code, should analyze the development product at each phase. The programmer should explain the
product to the party who will then question the logic and search for errors with a checklist to guide the
search. This is needed to locate errors the programmer has overlooked.

- Use available tools - the programmer should be familiar with various compilers and interpreters available
on the system for the implementation language being used because they differ in their error analysis and code
generation capabilities.

- Apply Stress to the Program - Testing should exercise and stress the program structure, the data structures,
the internal functions and the externally visible functions or functionality. Both valid and invalid data should
be included in the test set.
- Test one at a time - Pieces of code, individual modules and small collections of modules should be exercised
separately before they are integrated into the total program, one by one. Errors are easier to isolate when the
no. of potential interactions should be kept small. Instrumentation-insertion of some code into the program
solely to measure various program characteristics – can be useful here. A tester should perform array bound
checks, check loop control variables, determine whether key data values are within permissible ranges, trace
program execution, and count the no. of times a group of statements is executed.

- Measure testing coverage/When should testing stop? - If errors are still found every time the program is
executed, testing should continue. Because errors tend to cluster, modules appearing particularly error-prone
require special scrutiny.

The metrics used to measure testing thoroughness include statement testing (whether each statement in the
program has been executed at least once), branch testing (whether each exit from each branch has been
executed at least once) and path testing (whether all logical paths, which may involve repeated execution of
various segments, have been executed at least once). Statement testing is the coverage metric most frequently
used as it is relatively simple to implement.

The amount of testing depends on the cost of an error. Critical programs or functions require more thorough
testing than the less significant functions.

Operations and Maintenance

Corrections, modifications and extensions are bound to occur even for small programs and testing is required
every time there is a change. Testing during maintenance is termed regression testing. The test set, the test
plan, and the test results for the original program should exist.

Modifications must be made to accommodate the program changes, and then all portions of the program
affected by the modifications must be re-tested. After regression testing is complete, the program and test
documentation must be updated to reflect the changes.

SDLC models

Introduction

There are various software development approaches defined and designed which are used/employed during
development process of software, these approaches are also referred as "Software Development Process
Models". 

Each process model follows a particular life cycle in order to ensure success in process of software
development.

Waterfall Model

Waterfall approach was first Process Model to be introduced and followed widely in Software Engineering to
ensure success of the project. In "The Waterfall" approach, the whole process of software development is
divided into separate process phases.

The phases in Waterfall model are: Requirement Specifications phase, Software Design, Implementation and
Testing & Maintenance. All these phases are cascaded to each other so that second phase is started as and
when defined set of goals are achieved for first phase and it is signed off, so the name "Waterfall Model". All
the methods and processes undertaken in Waterfall Model are more visible.
The stages of "The Waterfall Model" are: 

Requirement Analysis & Definition: All possible requirements of the system to be developed are captured
in this phase. Requirements are set of functionalities and constraints that the end-user (who will be using the
system) expects from the system. The requirements are gathered from the end-user by consultation, these
requirements are analyzed for their validity and the possibility of incorporating the requirements in the
system to be development is also studied. Finally, a Requirement Specification document is created which
serves the purpose of guideline for the next phase of the model.

System & Software Design: Before a starting for actual coding, it is highly important to understand what we
are going to create and what it should look like? The requirement specifications from first phase are studied
in this phase and system design is prepared. System Design helps in specifying hardware and system
requirements and also helps in defining overall system architecture. The system design specifications serve
as input for the next phase of the model.

Implementation & Unit Testing: On receiving system design documents, the work is divided in
modules/units and actual coding is started. The system is first developed in small programs called units,
which are integrated in the next phase. Each unit is developed and tested for its functionality; this is referred
to as Unit Testing. Unit testing mainly verifies if the modules/units meet their specifications.

Integration & System Testing: As specified above, the system is first divided in units which are developed
and tested for their functionalities. These units are integrated into a complete system during Integration
phase and tested to check if all modules/units coordinate between each other and the system as a whole
behaves as per the specifications. After successfully testing the software, it is delivered to the customer.

Operations & Maintenance: This phase of "The Waterfall Model" is virtually never ending phase (Very long).
Generally, problems with the system developed (which are not found during the development life cycle) come
up after its practical use starts, so the issues related to the system are solved after deployment of the system.
Not all the problems come in picture directly but they arise time to time and needs to be solved; hence this
process is referred as Maintenance.

Advantages

The advantage of waterfall development is that it allows for departmentalization and managerial control. A
schedule can be set with deadlines for each stage of development and a product can proceed through the
development process like a car in a carwash, and theoretically, be delivered on time. Development moves
from concept, through design, implementation, testing, installation, troubleshooting, and ends up at operation
and maintenance. Each phase of development proceeds in strict order, without any overlapping or iterative
steps.

Disadvantages

The disadvantage of waterfall development is that it does not allow for much reflection or revision. Once
an application is in the testing stage, it is very difficult to go back and change something that was not well-
thought out in the concept stage. Alternatives to the waterfall model include joint application development
(JAD), rapid applicationdevelopment (RAD), synch and stabilize, build and fix, and the spiral model.

Iterative Model

An iterative lifecycle model does not attempt to start with a full specification of requirements. Instead,
development begins by specifying and implementing just part of the software, which can then be reviewed in
order to identify further requirements. This process is then repeated, producing a new version of the
software for each cycle of the model. Consider an iterative lifecycle model which consists of repeating the
following four phases in sequence: 

A Requirements phase, in which the requirements for the software are gathered and analyzed. Iteration
should eventually result in a requirements phase that produces a complete and final specification of
requirements. - A Design

phase, in which a software solution to meet the requirements is designed. This may be a new design, or an
extension of an earlier design.
- An Implementation and Test phase, when the software is coded, integrated and tested.

- A Review phase, in which the software is evaluated, the current requirements are reviewed, and changes
and additions to requirements proposed.

For each cycle of the model, a decision has to be made as to whether the software produced by the cycle will
be discarded, or kept as a starting point for the next cycle (sometimes referred to as incremental
prototyping). Eventually a point will be reached where the requirements are complete and the software can
be delivered, or it becomes impossible to enhance the software as required, and a fresh start has to be made.

The iterative lifecycle model can be likened to producing software by successive approximation. Drawing an
analogy with mathematical methods that use successive approximation to arrive at a final solution, the
benefit of such methods depends on how rapidly they converge on a solution.

The key to successful use of an iterative software development lifecycle is rigorous validation of
requirements, and verification (including testing) of each version of the software against those requirements
within each cycle of the model. The first three phases of the example iterative model is in fact an abbreviated
form of a sequential V or waterfall lifecycle model. Each cycle of the model produces software that requires
testing at the unit level, for software integration, for system integration and for acceptance. As the software
evolves through successive cycles, tests have to be repeated and extended to verify each version of the
software.

V-Model

The V-model is a software development model which can be presumed to be the extension of the waterfall
model. Instead of moving down in a linear way, the process steps are bent upwards after the coding phase, to
form the typical V shape. The V-Model demonstrates the relationships between each phase of the
development life cycle and its associated phase of testing.
Verification Phases

1. Requirements analysis:

In this phase, the requirements of the proposed system are collected by analyzing the needs of the user(s).
This phase is concerned about establishing what the ideal system has to perform. However, it does not
determine how the software will be designed or built. Usually, the users are interviewed and a document
called the user requirements document is generated. The user requirements document will typically describe
the system’s functional, physical, interface, performance, data, security requirements etc as expected by the
user. It is one which the business analysts use to communicate their understanding of the system back to the
users. The users carefully review this document as this document would serve as the guideline for the system
designers in the system design phase. The user acceptance tests are designed in this phase.

2. System Design:

System engineers analyze and understand the business of the proposed system by studying the user
requirements document. They figure out possibilities and techniques by which the user requirements can be
implemented. If any of the requirements are not feasible, the user is informed of the issue. A resolution is
found and the user requirement document is edited accordingly.

The software specification document which serves as a blueprint for the development phase is generated.
This document contains the general system organization, menu structures, data structures etc. It may also
hold example business scenarios, sample windows, reports for the better understanding. Other technical
documentation like entity diagrams, data dictionary will also be produced in this phase. The documents for
system testing is prepared in this phase.

3. Architecture Design:

This phase can also be called as high-level design. The baseline in selecting the architecture is that it should
realize all which typically consists of the list of modules, brief functionality of each module, their interface
relationships, dependencies, database tables, architecture diagrams, technology details etc. The integration
testing design is carried out in this phase.

4. Module Design:

This phase can also be called as low-level design. The designed system is broken up in to smaller units or
modules and each of them is explained so that the programmer can start coding directly. The low level design
document or program specifications will contain a detailed functional logic of the module, in pseudocode -
database tables, with all elements, including their type and size - all interface details with complete API
references- all dependency issues- error message listings- complete input and outputs for a module. The unit
test design is developed in this stage.

Spiral Model

History

The spiral model was defined by Barry Boehm in his 1988 article A Spiral Model of Software Development
and Enhancement. This model was not the first model to discuss iterative development, but it was the first
model to explain why the iteration matters. As originally envisioned, the iterations were typically 6 months to
2 years long. Each phase starts with a design goal and ends with the client (who may be internal) reviewing
the progress thus far. Analysis and engineering efforts are applied at each phase of the project, with an eye
toward the end goal of the project.

The Spiral Model

The spiral model, also known as the spiral lifecycle model, is a systems development method (SDM) used
in information technology (IT). This model of development combines the features of the prototyping model
and the waterfall model. The spiral model is intended for large, expensive, and complicated projects.

The steps in the spiral model can be generalized as follows:

1. The new system requirements are defined in as much detail as possible. This usually involves
interviewing a number of users representing all the external or internal users and other aspects of
the existing system.

2. A preliminary design is created for the new system.

3. A first prototype of the new system is constructed from the preliminary design. This is usually a
scaled-down system, and represents an approximation of the characteristics of the final product.
 4. A second prototype is evolved by a fourfold procedure: (1) evaluating the first prototype in terms of
its strengths, weaknesses, and risks; (2) defining the requirements of the second prototype; (3)
planning and designing the second prototype; (4) constructing and testing the second prototype.

5. At the customer's option, the entire project can be aborted if the risk is deemed too great. Risk factors
might involve development cost overruns, operating-cost miscalculation, or any other factor that
could, in the customer's judgment, result in a less-than-satisfactory final product.

6. The existing prototype is evaluated in the same manner as was the previous prototype, and, if
necessary, another prototype is developed from it according to the fourfold procedure outlined
above.

7. The preceding steps are iterated until the customer is satisfied that the refined prototype represents
the final product desired.

8. The final system is constructed, based on the refined prototype.

9. The final system is thoroughly evaluated and tested. Routine maintenance is carried out on a
continuing basis to prevent large-scale failures and to minimize downtime.

Advantages

1. Estimates (i.e. budget, schedule, etc.) become more realistic as work progresses, because important
issues are discovered earlier.

2. It is more able to cope with the (nearly inevitable) changes that software development generally
entails.

3. Software engineers (who can get restless with protracted design processes) can get their hands in
and start working on a project earlier.

Disadvantages

1. Highly customized limiting re-usability

2. Applied differently for each application

3. Risk of not meeting budget or schedule

4. Risk of not meeting budget or schedule