Software testing

Software testing is an investigation conducted to provide stakeholders with information

about the quality of the product or service under test.[1] Software testing can also provide an
objective, independent view of the software to allow the business to appreciate and
understand the risks of software implementation. Test techniques include the process of
executing a program or application with the intent of finding software bugs (errors or other
defects).

Software testing involves the execution of a software component or system component to

evaluate one or more properties of interest. In general, these properties indicate the extent
to which the component or system under test:

 meets the requirements that guided its design and development,

 responds correctly to all kinds of inputs,
 performs its functions within an acceptable time,
 is sufficiently usable,
 can be installed and run in its intended environments, and
 achieves the general result its stakeholders desire.

As the number of possible tests for even simple software components is practically infinite,
all software testing uses some strategy to select tests that are feasible for the available time
and resources. As a result, software testing typically (but not exclusively) attempts to
execute a program or application with the intent of finding software bugs (errors or other
defects). The job of testing is an iterative process as when one bug is fixed, it can illuminate
other, deeper bugs, or can even create new ones.

Software testing can provide objective, independent information about the quality of software
and risk of its failure to users and/or sponsors.[1]

Software testing can be conducted as soon as executable software (even if partially

complete) exists. The overall approach to software development often determines when and
how testing is conducted. For example, in a phased process, most testing occurs after
system requirements have been defined and then implemented in testable programs. In
contrast, under an Agile approach, requirements, programming, and testing are often done
concurrently.

1. Overview

Although testing can determine the correctness of software under the assumption of some
specific hypotheses (see hierarchy of testing difficulty below), testing cannot identify all the
defects within software. Instead, it furnishes a criticism or comparison that compares the
state and behavior of the product against oracles—principles or mechanisms by which
someone might recognize a problem. These oracles may include (but are not limited to)

1
specifications, contracts,[3] comparable products, past versions of the same product,
inferences about intended or expected purpose, user or customer expectations, relevant
standards, applicable laws, or other criteria.

A primary purpose of testing is to detect software failures so that defects may be discovered
and corrected. Testing cannot establish that a product functions properly under all conditions
but can only establish that it does not function properly under specific conditions. [4] The
scope of software testing often includes examination of code as well as execution of that
code in various environments and conditions as well as examining the aspects of code: does
it do what it is supposed to do and do what it needs to do. In the current culture of software
development, a testing organization may be separate from the development team. There are
various roles for testing team members. Information derived from software testing may be
used to correct the process by which software is developed.[5]

Every software product has a target audience. For example, the audience for video game
software is completely different from banking software. Therefore, when an organization
develops or otherwise invests in a software product, it can assess whether the software
product will be acceptable to its end users, its target audience, its purchasers and other
stakeholders. Software testing is the process of attempting to make this assessment.

1.1. Defects and failures

Not all software defects are caused by coding errors. One common source of expensive
defects is requirement gaps, e.g., unrecognized requirements which result in errors of
omission by the program designer.[6] Requirement gaps can often be non-functional
requirements such as testability, scalability, maintainability, usability, performance and
security. Software faults occur through the following processes. A programmer makes
an error (mistake), which results in a defect (fault, bug) in the software source code. If this
defect is executed, in certain situations the system will produce wrong results, causing
a failure.[7] Not all defects will necessarily result in failures. For example, defects in dead
code will never result in failures. A defect can turn into a failure when the environment is
changed. Examples of these changes in environment include the software being run on a
newcomputer hardware platform, alterations in source data, or interacting with different
software.[7] A single defect may result in a wide range of failure symptoms.

1.2. Input combinations and preconditions

A fundamental problem with software testing is that testing under all combinations of inputs
and preconditions (initial state) is not feasible, even with a simple product.[4][8] This means
that the number of defects in a software product can be very large and defects that occur
infrequently are difficult to find in testing. More significantly, non-functionaldimensions of
quality (how it is supposed to be versus what it is supposed to do)—

2
usability, scalability, performance, compatibility, reliability—can be highly subjective;
something that constitutes sufficient value to one person may be intolerable to another.

Software developers can't test everything, but they can use combinatorial test design to
identify the minimum number of tests needed to get the coverage they want. Combinatorial
test design enables users to get greater test coverage with fewer tests. Whether they are
looking for speed or test depth, they can use combinatorial test design methods to build
structured variation into their test cases.[9] Note that "coverage", as used here, is referring to
combinatorial coverage, not requirements coverage.

2. Testing methods

2.1. Static vs. dynamic testing

There are many approaches available in software testing. Reviews, walkthroughs,

or inspections are referred to as static testing, whereas actually executing programmed code
with a given set of test cases is referred to as dynamic testing. Static testing is often implicit,
as proofreading, plus when programming tools/text editors check source code structure or
compilers (pre-compilers) check syntax and data flow as static program analysis. Dynamic
testing takes place when the program itself is run. Dynamic testing may begin before the
program is 100% complete in order to test particular sections of code and are applied to
discrete functions or modules. Typical techniques for this are either using stubs/drivers or
execution from a debugger environment.

Static testing involves verification, whereas dynamic testing involves validation. Together
they help improve software quality. Among the techniques for static analysis, mutation
testing can be used to ensure the test-cases will detect errors which are introduced by
mutating the source code.

2.2.The box approach

Software testing methods are traditionally divided into white- and black-box testing. These
two approaches are used to describe the point of view that a test engineer takes when
designing test cases.

2.2.1. White-box testing

White-box testing (also known as clear box testing, glass box testing, transparent box
testing and structural testing, by seeing the source code) tests internal structures or
workings of a program, as opposed to the functionality exposed to the end-user. In white-box
testing an internal perspective of the system, as well as programming skills, are used to
design test cases. The tester chooses inputs to exercise paths through the code and
determine the appropriate outputs. This is analogous to testing nodes in a circuit, e.g. in-
circuit testing (ICT).

3
While white-box testing can be applied at the unit, integration and system levels of the
software testing process, it is usually done at the unit level. It can test paths within a unit,
paths between units during integration, and between subsystems during a system–level test.
Though this method of test design can uncover many errors or problems, it might not detect
unimplemented parts of the specification or missing requirements.

Techniques used in white-box testing include:

 API testing – testing of the application using public and private APIs (application
programming interfaces)
 Code coverage – creating tests to satisfy some criteria of code coverage (e.g., the test
designer can create tests to cause all statements in the program to be executed at least
once)
 Fault injection methods – intentionally introducing faults to gauge the efficacy of testing
strategies
 Mutation testing methods
 Static testing methods

Code coverage tools can evaluate the completeness of a test suite that was created with any
method, including black-box testing. This allows the software team to examine parts of a
system that are rarely tested and ensures that the most important function points have been
tested.[22] Code coverage as a software metric can be reported as a percentage for:

 Function coverage, which reports on functions executed

 Statement coverage, which reports on the number of lines executed to complete
the test
 Decision coverage, which reports on whether both the True and the False branch
of a given test has been executed

100% statement coverage ensures that all code paths or branches (in terms of control
flow) are executed at least once. This is helpful in ensuring correct functionality, but not
sufficient since the same code may process different inputs correctly or incorrectly.

2.2.2. Black-box testing

Black box diagram

Black-box testing treats the software as a "black box", examining functionality without any
knowledge of internal implementation, without seeing the source code. The testers are only
aware of what the software is supposed to do, not how it does it.[23] Black-box testing
methods include: equivalence partitioning, boundary value analysis, all-pairs testing, state
transition tables, decision table testing, fuzz testing, model-based testing, use
case testing, exploratory testing and specification-based testing.

4
Specification-based testing aims to test the functionality of software according to the
applicable requirements.[24] This level of testing usually requires thorough test cases to be
provided to the tester, who then can simply verify that for a given input, the output value (or
behavior), either "is" or "is not" the same as the expected value specified in the test case.
Test cases are built around specifications and requirements, i.e., what the application is
supposed to do. It uses external descriptions of the software, including specifications,
requirements, and designs to derive test cases. These tests can be functional or non-
functional, though usually functional.

Specification-based testing may be necessary to assure correct functionality, but it is

insufficient to guard against complex or high-risk situations.[25]

One advantage of the black box technique is that no programming knowledge is required.
Whatever biases the programmers may have had, the tester likely has a different set and
may emphasize different areas of functionality. On the other hand, black-box testing has
been said to be "like a walk in a dark labyrinth without a flashlight." Because they do not
examine the source code, there are situations when a tester writes many test cases to check
something that could have been tested by only one test case, or leaves some parts of the
program untested.

This method of test can be applied to all levels of software

testing: unit, integration, system and acceptance. It typically comprises most if not all testing
at higher levels, but can also dominate unit testing as well.

2.2.2.1. Visual testing

The aim of visual testing is to provide developers with the ability to examine what was
happening at the point of software failure by presenting the data in such a way that the
developer can easily find the information she or he requires, and the information is
expressed clearly.[27][28]

At the core of visual testing is the idea that showing someone a problem (or a test failure),
rather than just describing it, greatly increases clarity and understanding. Visual testing
therefore requires the recording of the entire test process – capturing everything that occurs
on the test system in video format. Output videos are supplemented by real-time tester input
via picture-in-a-picture webcam and audio commentary from microphones.

Visual testing provides a number of advantages. The quality of communication is increased

drastically because testers can show the problem (and the events leading up to it) to the
developer as opposed to just describing it and the need to replicate test failures will cease to
exist in many cases. The developer will have all the evidence he or she requires of a test
failure and can instead focus on the cause of the fault and how it should be fixed.

Visual testing is particularly well-suited for environments that deploy agile methods in their
development of software, since agile methods require greater communication between
testers and developers and collaboration within small teams.[citation needed]

5
Ad hoc testing and exploratory testing are important methodologies for checking software
integrity, because they require less preparation time to implement, while the important bugs
can be found quickly. In ad hoc testing, where testing takes place in an improvised,
impromptu way, the ability of a test tool to visually record everything that occurs on a system
becomes very important in order to document the steps taken to uncover the bug. [clarification
needed][citation needed]

Visual testing is gathering recognition in customer acceptance and usability testing, because
the test can be used by many individuals involved in the development process.[citation
needed]
For the customer, it becomes easy to provide detailed bug reports and feedback, and
for program users, visual testing can record user actions on screen, as well as their voice
and image, to provide a complete picture at the time of software failure for the developers.

2.2.3. Grey-box testing

Grey-box testing (American spelling: gray-box testing) involves having knowledge of
internal data structures and algorithms for purposes of designing tests, while executing those
tests at the user, or black-box level. The tester is not required to have full access to the
software's source code.[29][not in citation given] Manipulating input data and formatting output do not
qualify as grey-box, because the input and output are clearly outside of the "black box" that
we are calling the system under test. This distinction is particularly important when
conducting integration testing between two modules of code written by two different
developers, where only the interfaces are exposed for test.

However, tests that require modifying a back-end data repository such as a database or a
log file does qualify as grey-box, as the user would not normally be able to change the data
repository in normal production operations.[citation needed]
Grey-box testing may also
include reverse engineering to determine, for instance, boundary values or error messages.

By knowing the underlying concepts of how the software works, the tester makes better-
informed testing choices while testing the software from outside. Typically, a grey-box tester
will be permitted to set up an isolated testing environment with activities such as seeding
a database. The tester can observe the state of the product being tested after performing
certain actions such as executing SQL statements against the database and then executing
queries to ensure that the expected changes have been reflected. Grey-box testing
implements intelligent test scenarios, based on limited information. This will particularly apply
to data type handling, exception handling, and so on.[30]

6
 3. Testing levels
There are generally four recognized levels of tests: unit testing, integration testing,
component interface testing, and system testing. Tests are frequently grouped by where they
are added in the software development process, or by the level of specificity of the test. The
main levels during the development process as defined by the SWEBOK guide are unit-,
integration-, and system testing that are distinguished by the test target without implying a
specific process model.[31] Other test levels are classified by the testing objective.[31]

3.1. Unit testing

Unit testing, also known as component testing, refers to tests that verify the functionality of a
specific section of code, usually at the function level. In an object-oriented environment, this
is usually at the class level, and the minimal unit tests include the constructors and
destructors.[32]

These types of tests are usually written by developers as they work on code (white-box
style), to ensure that the specific function is working as expected. One function might have
multiple tests, to catch corner cases or other branches in the code. Unit testing alone cannot
verify the functionality of a piece of software, but rather is used to ensure that the building
blocks of the software work independently from each other.

Unit testing is a software development process that involves synchronized application of a

broad spectrum of defect prevention and detection strategies in order to reduce software
development risks, time, and costs. It is performed by the software developer or engineer
during the construction phase of the software development lifecycle. Rather than replace
traditional QA focuses, it augments it. Unit testing aims to eliminate construction errors
before code is promoted to QA; this strategy is intended to increase the quality of the
resulting software as well as the efficiency of the overall development and QA process.

Depending on the organization's expectations for software development, unit testing might
include static code analysis, data flow analysis, metrics analysis, peer code reviews, code
coverage analysis and other software verification practices.

3.2. Integration testing

Integration testing is any type of software testing that seeks to verify the interfaces between
components against a software design. Software components may be integrated in an
iterative way or all together ("big bang"). Normally the former is considered a better practice
since it allows interface issues to be located more quickly and fixed.

Integration testing works to expose defects in the interfaces and interaction between
integrated components (modules). Progressively larger groups of tested software
components corresponding to elements of the architectural design are integrated and tested
until the software works as a system.[33]

7
 3.3. Component interface testing

The practice of component interface testing can be used to check the handling of data
passed between various units, or subsystem components, beyond full integration testing
between those units.[34][35] The data being passed can be considered as "message packets"
and the range or data types can be checked, for data generated from one unit, and tested
for validity before being passed into another unit. One option for interface testing is to keep a
separate log file of data items being passed, often with a timestamp logged to allow analysis
of thousands of cases of data passed between units for days or weeks. Tests can include
checking the handling of some extreme data values while other interface variables are
passed as normal values.[34] Unusual data values in an interface can help explain
unexpected performance in the next unit. Component interface testing is a variation of black-
box testing,[35] with the focus on the data values beyond just the related actions of a
subsystem component.

3.4. System testing

System testing, or end-to-end testing, tests a completely integrated system to verify that it
meets its requirements.[36] For example, a system test might involve testing a logon interface,
then creating and editing an entry, plus sending or printing results, followed by summary
processing or deletion (or archiving) of entries, then logoff.

3.5. Operational Acceptance testing

Operational Acceptance is used to conduct operational readiness (pre-release) of a product,

service or system as part of a quality management system. OAT is a common type of non-
functional software testing, used mainly in software development and software
maintenance projects. This type of testing focuses on the operational readiness of the
system to be supported, and/or to become part of the production environment. Hence, it is
also known as operational readiness testing (ORT) or Operations Readiness and
Assurance (OR&A) testing. Functional testing within OAT is limited to those tests which are
required to verify the non-functional aspects of the system.

In addition, the software testing should ensure that the portability of the system, as well as
working as expected, does not also damage or partially corrupt its operating environment or
cause other processes within that environment to become inoperative.[37]

8
 4. Testing types

4.1. Installation testing

An installation test assures that the system is installed correctly and working at actual
customer's hardware. Installation testing is a kind of quality assurance work in
the software industry that focuses on what customers will need to do to install and set up the
new software successfully. The testing process may involve full, partial or upgrades
install/uninstall processes.

This testing is typically performed in Operational Acceptance testing, by a software

testing engineer in conjunction with the configuration manager. Implementation testing is
usually defined as testing which places a compiled version of code into the testing or pre-
production environment, from which it may or may not progress into production. This
generally takes place outside of the software development environment to limit code
corruption from other future or past releases (or from the use of the wrong version of
dependencies such as shared libraries) which may reside on the development environment.

The simplest installation approach is to run an install program, sometimes called package
software. This package software typically uses a setup program which acts as a multi-
configuration wrapper and which may allow the software to be installed on a variety of
machine and/or operating environments. Every possible configuration should receive an
appropriate level of testing so that it can be released to customers with confidence.

In distributed systems, particularly where software is to be released into an already live

target environment (such as an operational website) installation (or software deploymentas it
is sometimes called) can involve database schema changes as well as the installation of
new software. Deployment plans in such circumstances may include back-out procedures
whose use is intended to roll the target environment back if the deployment is unsuccessful.
Ideally, the deployment plan itself should be tested in an environment that is a replica of the
live environment. A factor that can increase the organizational requirements of such an
exercise is the need to synchronize the data in the test deployment environment with that in
the live environment with minimum disruption to live operation. This type of implementation
may include testing of the processes which take place during the installation or upgrade of a
multi-tier application. This type of testing is commonly compared to a dress rehearsal or may
even be called a ―dry run‖.

9
 4.2. Compatibility testing

A common cause of software failure (real or perceived) is a lack of its compatibility with
other application software, operating systems (or operating system versions, old or new), or
target environments that differ greatly from the original (such as a terminal or GUI application
intended to be run on the desktop now being required to become a web application, which
must render in a web browser). For example, in the case of a lack of backward compatibility,
this can occur because the programmers develop and test software only on the latest
version of the target environment, which not all users may be running. This results in the
unintended consequence that the latest work may not function on earlier versions of the
target environment, or on older hardware that earlier versions of the target environment was
capable of using. Sometimes such issues can be fixed by proactively abstracting operating
system functionality into a separate program module or library.

Compatibility testing, part of software non-functional tests, is testing conducted on

the application to evaluate the application's compatibility with the computing environment.
Computing environment may contain some or all of the below mentioned elements:

 Computing capacity of Hardware Platform (IBM 360, HP 9000, etc.)..

 Bandwidth handling capacity of networking hardware
 Compatibility of peripherals (Printer, DVD drive, etc.)
 Operating systems (Linux, Windows, Mac etc.)
 Database (Oracle, SQL Server, MySQL, etc.)
 Other System Software (Web server, networking/ messaging tool, etc.)
 Browser compatibility (Chrome, Firefox, Netscape, Internet Explorer, Safari, etc.)

Browser compatibility testing, can be more appropriately referred to as user experience

testing. This requires that the web applications are tested on different web browsers, to
ensure the following:

 Users have the same visual experience irrespective of the browsers through which they
view the web application.
 In terms of functionality, the application must behave and respond the same way across
different browsers.
 Carrier compatibility (Verizon, Sprint, Orange, O2, AirTel, etc.)
 Backwards compatibility.
 Hardware (different phones)
 Different Compilers (compile the code correctly)
 Runs on multiple host/guest Emulators

10
 4.3. Smoke and sanity testing

Sanity testing determines whether it is reasonable to proceed with further testing.

Smoke testing consists of minimal attempts to operate the software, designed to determine
whether there are any basic problems that will prevent it from working at all. Such tests can
be used as build verification test.

4.4. Regression testing

Regression testing focuses on finding defects after a major code change has occurred.
Specifically, it seeks to uncover software regressions, as degraded or lost features, including
old bugs that have come back. Such regressions occur whenever software functionality that
was previously working correctly, stops working as intended. Typically, regressions occur as
an unintended consequence of program changes, when the newly developed part of the
software collides with the previously existing code. Common methods of regression testing
include re-running previous sets of test-cases and checking whether previously fixed faults
have re-emerged. The depth of testing depends on the phase in the release process and
the risk of the added features. They can either be complete, for changes added late in the
release or deemed to be risky, or be very shallow, consisting of positive tests on each
feature, if the changes are early in the release or deemed to be of low risk. Regression
testing is typically the largest test effort in commercial software development, [38] due to
checking numerous details in prior software features, and even new software can be
developed while using some old test-cases to test parts of the new design to ensure prior
functionality is still supported.

4.5. Acceptance testing

Acceptance testing can mean one of two things:

1. A smoke test is used as an acceptance test prior to introducing a new build to the
main testing process, i.e., before integration or regression.
2. Acceptance testing performed by the customer, often in their lab environment on
their own hardware, is known as user acceptance testing (UAT). Acceptance testing
may be performed as part of the hand-off process between any two phases of
development.

In engineering and its various subdisciplines, acceptance testing is a test conducted to

determine if the requirements of aspecification or contract are met. It may involve chemical
tests, physical tests, or performance tests. In systems engineering it may involve black-box
testing performed on a system (for example: a piece of software, lots of manufactured
mechanical parts, or batches of chemical products) prior to its delivery.[1]

In software testing the ISTQB defines acceptance as: formal testing with respect to user
needs, requirements, and business processes conducted to determine whether a system
satisfies the acceptance criteria and to enable the user, customers or other authorized entity

11
to determine whether or not to accept the system.[2] Acceptance testing is also known
as user acceptance testing (UAT), end-user testing, operational acceptance testing (OAT) or
field (acceptance) testing.

A smoke test may be used as an acceptance test prior to introducing a build of software to
the main testing process.

Process

The acceptance test suite may need to be performed multiple times, as all of the test cases
may not be executed within a single test iteration.[5]

The acceptance test suite is run using predefined acceptance test procedures to direct the
testers which data to use, the step-by-step processes to follow and the expected result
following execution. The actual results are retained for comparison with the expected
results.[5] If the actual results match the expected results for each test case, the test case is
said to pass. If the quantity of non-passing test cases does not breach the project's
predetermined threshold, the test suite is said to pass. If it does, the system may either be
rejected or accepted on conditions previously agreed between the sponsor and the
manufacturer.

The objective is to provide confidence that the developed product meets both the functional
and non-functional requirements. The purpose of conducting acceptance testing is that once
completed, and provided the acceptance criteria are met, it is expected the sponsors will
sign-off on the product development/enhancement as satisfying the defined requirements
(previously agreed between business and product provider/developer).

4.6. Alpha testing

Alpha testing is simulated or actual operational testing by potential users/customers or an

independent test team at the developers' site. Alpha testing is often employed for off-the-
shelf software as a form of internal acceptance testing, before the software goes to beta
testing.[39]
4.7. Beta testing

Beta testing comes after alpha testing and can be considered a form of external user
acceptance testing. Versions of the software, known as beta versions, are released to a
limited audience outside of the programming team known as beta testers. The software is
released to groups of people so that further testing can ensure the product has few faults
or bugs. Beta versions can be made available to the open public to increase
the feedback field to a maximal number of future users and to deliver value earlier, for an
extended or even infinite period of time (perpetual beta).[citation needed]

12
 4.8. Functional vs non-functional testing

Functional testing refers to activities that verify a specific action or function of the code.
These are usually found in the code requirements documentation, although some
development methodologies work from use cases or user stories. Functional tests tend to
answer the question of "can the user do this" or "does this particular feature work."

Non-functional testing refers to aspects of the software that may not be related to a specific
function or user action, such as scalability or other performance, behavior under
certain constraints, or security. Testing will determine the breaking point, the point at which
extremes of scalability or performance leads to unstable execution. Non-functional
requirements tend to be those that reflect the quality of the product, particularly in the
context of the suitability perspective of its users.

4.9. Continuous testing

Continuous testing is the process of executing automated tests as part of the software
delivery pipeline to obtain immediate feedback on the business risks associated with a
software release candidate.[40][41] Continuous testing includes the validation of both functional
requirements and non-functional requirements; the scope of testing extends from validating
bottom-up requirements or user stories to assessing the system requirements associated
with overarching business goals.[42][42][43][44]

Goals and benefits

The goal of continuous testing is to provide fast and continuous feedback regarding the level
of business risk in the latest build or release candidate.[2] This information can then be used
to determine if the software is ready to progress through the delivery pipeline at any given
time.

Since testing begins early and is executed continuously, application risks are exposed soon
after they are introduced.[5] Development teams can then prevent those problems from
progressing to the next stage of the SDLC. This reduces the time and effort that need to be
spent finding and fixing defects. As a result, it is possible to increase the speed and
frequency at which quality software (software that meets expectations for an acceptable
level of risk) is delivered, as well as decrease technical debt.

Moreover, when software quality efforts and testing are aligned with business expectations,
test execution produces a prioritized list of actionable tasks (rather than a potentially
overwhelming number of findings that require manual review). This helps teams focus their
efforts on the quality tasks that will have the greatest impact, based on their organization's
goals and priorities.[2]

13
 4.10. Destructive testing

Destructive testing attempts to cause the software or a sub-system to fail. It verifies that the
software functions properly even when it receives invalid or unexpected inputs, thereby
establishing the robustness of input validation and error-management routines. Software
fault injection, in the form of fuzzing, is an example of failure testing. Various commercial
non-functional testing tools are linked from the software fault injection page; there are also
numerous open-source and free software tools available that perform destructive testing.

In destructive testing, or (Destructive Physical Analysis DPA) tests are carried out to the
specimen's failure, in order to understand a specimen's structural performance or material
behaviour under different loads. These tests are generally much easier to carry out, yield
more information, and are easier to interpret than nondestructive testing. [1]Destructive
testing is most suitable, and economic, for objects which will be mass-produced, as the cost
of destroying a small number of specimens is negligible. It is usually not economical to do
destructive testing where only one or very few items are to be produced (for example, in the
case of a building). Analyzing and documenting the destructive failure mode is often
accomplished using a high-speed camera recording continuously (movie-loop) until the
failure is detected. Detecting the failure can be accomplished using a sound detector or
stress gauge which produces a signal to trigger the high-speed camera. These high-speed
cameras have advanced recording modes to capture almost any type of destructive
failure.[2] After the failure the high-speed camera will stop recording. The capture images can
be played back in slow motion showing precisely what happen before, during and after the
destructive event, image by image.

Some types of destructive testing:

 Stress tests
 Crash tests
 Hardness tests
 Metallographic tests

4.11. Software performance testing

Performance testing is generally executed to determine how a system or sub-system

performs in terms of responsiveness and stability under a particular workload. It can also
serve to investigate, measure, validate or verify other quality attributes of the system, such
as scalability, reliability and resource usage.

Load testing is primarily concerned with testing that the system can continue to operate
under a specific load, whether that be large quantities of data or a large number of users.
This is generally referred to as software scalability. The related load testing activity of when
performed as a non-functional activity is often referred to as endurance testing.Volume
testing is a way to test software functions even when certain components (for example a file

14
or database) increase radically in size. Stress testing is a way to test reliability under
unexpected or rare workloads. Stability testing (often referred to as load or endurance
testing) checks to see if the software can continuously function well in or above an
acceptable period.There is little agreement on what the specific goals of performance testing
are. The terms load testing, performance testing, scalability testing, and volume testing, are
often used interchangeably. Real-time software systems have strict timing constraints. To
test if timing constraints are met, real-time testing is used.

4.12. Usability testing

Usability testing is to check if the user interface is easy to use and understand. It is
concerned mainly with the use of the application. Usability testing is a technique used
in user-centered interaction design to evaluate a product by testing it on users. This can be
seen as an irreplaceable usability practice, since it gives direct input on how real users use
the system.[1] This is in contrast with usability inspection methods where experts use
different methods to evaluate a user interface without involving users.

Usability testing focuses on measuring a human-made product's capacity to meet its

intended purpose. Examples of products that commonly benefit from usability testing
arefoods, consumer products, web sites or web applications, computer interfaces,
documents, and devices. Usability testing measures the usability, or ease of use, of a
specific object or set of objects, whereas general human-computer interaction studies
attempt to formulate universal principles.

4.13. Accessibility testing

Accessibility refers to the design of products, devices, services, or environments for people
with disabilities.[1] The concept of accessible design ensures both "direct access" (i.e.
unassisted) and "indirect access" meaning compatibility with a person's assistive
technology (for example, computer screen readers).

Accessibility can be viewed as the "ability to access" and benefit from some system or entity.
The concept focuses on enabling access for people with disabilities, or special needs, or
enabling access through the use of assistive technology; however, research and
development in accessibility brings benefits to everyone.[2][3][4][5][6]

Accessibility is not to be confused with usability, which is the extent to which a product (such
as a device, service, or environment) can be used by specified users to achieve specified
goals with effectiveness, efficiency and satisfaction in a specified context of use.

Accessibility is strongly related to universal design which is the process of creating products
that are usable by people with the widest possible range of abilities, operating within the
widest possible range of situations. This is about making things accessible to all people
(whether they have a disability or not).

15
 4.14. Security testing

Security testing is essential for software that processes confidential data to prevent system
intrusion by hackers.

The International Organization for Standardization (ISO) defines this as a "type of testing
conducted to evaluate the degree to which a test item, and associated data and information,
are protected to that unauthorised persons or systems cannot use, read or modify them, and
authorized persons or systems are not denied access to them."[45] Security testing is a
process intended to reveal flaws in the security mechanisms of an information system that
protect data and maintain functionality as intended. Due to the logical limitations of security
testing, passing security testing is not an indication that no flaws exist or that the system
adequately satisfies the security requirements.Typical security requirements may include
specific elements of confidentiality, integrity, authentication, availability, authorization and
non-repudiation.

Confidentiality - A security measure which protects against the disclosure of information to

parties other than the intended recipient is by no means the only way of ensuring the
security.
Integrity - Integrity of information refers to protecting information from being modified by
unauthorized parties
- A measure intended to allow the receiver to determine that the information
provided by a system is correct.
- Integrity schemes often use some of the same underlying technologies as
confidentiality schemes, but they usually involve adding information to a communication, to
form the basis of an algorithmic check, rather than the encoding all of the communication.
- To check if the correct information is transferred from one application to other

Authentication - This might involve confirming the identity of a person, tracing the origins of
an artifact, ensuring that a product is what its packaging and labeling claims to be, or
assuring that a computer program is a trusted one.
Authorization - The process of determining that a requester is allowed to receive a service or
perform an operation.
-Access control is an example of authorization.

Availability - Assuring information and communications services will be ready for use when
expected.
- Information must be kept available to authorized persons when they need it

Non-repudiation - In reference to digital security, nonrepudiation means to ensure that a

transferred message has been sent and received by the parties claiming to have sent and
received the message. Nonrepudiation is a way to guarantee that the sender of a message
cannot later deny having sent the message and that the recipient cannot deny having
received the message.

16
 4.15. Internationalization and localization

The general ability of software to be internationalized and localized can be automatically

tested without actual translation, by using pseudolocalization. It will verify that the application
still works, even after it has been translated into a new language or adapted for a new
culture (such as different currencies or time zones).[46]

Actual translation to human languages must be tested, too. Possible localization failures
include:

 Software is often localized by translating a list of strings out of context, and the translator
may choose the wrong translation for an ambiguous source string.
 Technical terminology may become inconsistent if the project is translated by several
people without proper coordination or if the translator is imprudent.
 Literal word-for-word translations may sound inappropriate, artificial or too technical in
the target language.
 Untranslated messages in the original language may be left hard coded in the source
code.
 Some messages may be created automatically at run time and the resulting string may
be ungrammatical, functionally incorrect, misleading or confusing.
 Software may use a keyboard shortcut which has no function on the source
language's keyboard layout, but is used for typing characters in the layout of the target
language.
 Software may lack support for the character encoding of the target language.
 Fonts and font sizes which are appropriate in the source language may be inappropriate
in the target language; for example, CJK characters may become unreadable if the font
is too small.
 A string in the target language may be longer than the software can handle. This may
make the string partly invisible to the user or cause the software to crash or malfunction.
 Software may lack proper support for reading or writing bi-directional text.
 Software may display images with text that was not localized.
 Localized operating systems may have differently named system configuration
files and environment variables and different formats for date and currency.

4.16. Development testing

Development Testing is a software development process that involves synchronized

application of a broad spectrum of defect prevention and detection strategies in order to
reduce software development risks, time, and costs. It is performed by the software
developer or engineer during the construction phase of the software development lifecycle.
Rather than replace traditional QA focuses, it augments it. Development Testing aims to
eliminate construction errors before code is promoted to QA; this strategy is intended to
increase the quality of the resulting software as well as the efficiency of the overall
development and QA process.

Depending on the organization's expectations for software development, Development

Testing might include static code analysis, data flow analysis, metrics analysis, peer code
reviews, unit testing, code coverage analysis, traceability, and other software verification
practices.

17
 4.17. A/B testing

A/B testing is basically a comparison of two outputs, generally when only one variable has
changed: run a test, change one thing, run the test again, compare the results. This is more
useful with more small-scale situations, but very useful in fine-tuning any program. With
more complex projects, multivariant testing can be done.

In marketing and business intelligence, A/B testing is a term for a randomized

experiment with two variants, A and B, which are the control and variation in the controlled
experiment .[1]

As the name implies, two versions (A and B) are compared, which are identical except for
one variation that might affect a user's behavior. Version A might be the currently used
version (control), while version B is modified in some respect (treatment). For instance, on
an e-commerce website the purchase funnel is typically a good candidate for A/B testing, as
even marginal improvements in drop-off rates can represent a significant gain in sales.

4.18. Concurrent testing

In concurrent testing, the focus is more on what the performance is like when continuously
running with normal input and under normal operation as opposed to stress testing, or fuzz
testing. Memory leak is more easily found and resolved using this method, as well as more
basic faults.

Concurrent testing is a software testing activity that determines the stability of a system or
application under test during normal activity. Concurrent testing is the exercise of running
continuous testing with functional testing concurrently in order to discover defects that would
not otherwise be detected without the additional activity. Concurrent tests commonly put a
greater emphasis on robustness, performance, and system integration with production-like
activity, which should determine correct behavior of the system under normal circumstances.

Concurrent test vs. Stress test;

Stress testing tries to break the system under test by overwhelming its resources or by
taking resources away from it (in which case it is sometimes called negative testing). The
main purpose of this process is to make sure that the system fails and recovers gracefully—
a quality known as recoverability.

Concurrent testing implies a controlled environment staying at a constant level of activity.

Stress testing focuses on more random events, chaos and unpredictability.

18
 4.19. Conformance testing or type testing

In software testing, conformance testing verifies that a product performs according to its
specified standards. Compilers, for instance, are extensively tested to determine whether
they meet the recognized standard for that language.

5. Testing process

5.1. Traditional waterfall development model

A common practice of software testing is that testing is performed by an independent group

of testers after the functionality is developed, before it is shipped to the customer. [47]This
practice often results in the testing phase being used as a project buffer to compensate for
project delays, thereby compromising the time devoted to testing.[48]

Another practice is to start software testing at the same moment the project starts and it is a
continuous process until the project finishes.[49]

5.2. Agile or Extreme development model

In contrast, some emerging software disciplines such as extreme programming and the agile
software development movement, adhere to a "test-driven software development" model. In
this process, unit tests are written first, by the software engineers (often with pair
programming in the extreme programming methodology). Of course these tests fail initially;
as they are expected to. Then as code is written it passes incrementally larger portions of
the test suites. The test suites are continuously updated as new failure conditions and corner
cases are discovered, and they are integrated with any regression tests that are developed.
Unit tests are maintained along with the rest of the software source code and generally
integrated into the build process (with inherently interactive tests being relegated to a
partially manual build acceptance process). The ultimate goal of this test process is to
achieve continuous integration where software updates can be published to the public
frequently. This methodology increases the testing effort done by development, before
reaching any formal testing team. In some other development models, most of the test
execution occurs after the requirements have been defined and the coding process has
been completed.

19
 5.3. Top-down and bottom-up

Bottom Up Testing is an approach to integrated testing where the lowest level components
(modules, procedures, and functions) are tested first, then integrated and used to facilitate
the testing of higher level components. After the integration testing of lower level integrated
modules, the next level of modules will be formed and can be used for integration testing.
The process is repeated until the components at the top of the hierarchy are tested. This
approach is helpful only when all or most of the modules of the same development level are
ready.[citation needed] This method also helps to determine the levels of software developed and
makes it easier to report testing progress in the form of a percentage. [citation needed]

Top Down Testing is an approach to integrated testing where the top integrated modules
are tested and the branch of the module is tested step by step until the end of the related
module. In both, method stubs and drivers are used to stand-in for missing components and
are replaced as the levels are completed.

5.4. A sample testing cycle

Although variations exist between organizations, there is a typical cycle for testing.[52] The
sample below is common among organizations employing the Waterfall developmentmodel.
The same practices are commonly found in other development models, but might not be as
clear or explicit.

 Requirements analysis: Testing should begin in the requirements phase of

the software development life cycle. During the design phase, testers work to determine
what aspects of a design are testable and with what parameters those tests work.
 Test planning: Test strategy, test plan, testbed creation. Since many activities will be
carried out during testing, a plan is needed.
 Test development: Test procedures, test scenarios, test cases, test datasets, test
scripts to use in testing software.
 Test execution: Testers execute the software based on the plans and test documents
then report any errors found to the development team.
 Test reporting: Once testing is completed, testers generate metrics and make final
reports on their test effort and whether or not the software tested is ready for release.
 Test result analysis: Or Defect Analysis, is done by the development team usually
along with the client, in order to decide what defects should be assigned, fixed, rejected
(i.e. found software working properly) or deferred to be dealt with later.
 Defect Retesting: Once a defect has been dealt with by the development team, it is
retested by the testing team. AKA Resolution testing.
 Regression testing: It is common to have a small test program built of a subset of tests,
for each integration of new, modified, or fixed software, in order to ensure that the latest
delivery has not ruined anything, and that the software product as a whole is still working
correctly.
 Test Closure: Once the test meets the exit criteria, the activities such as capturing the
key outputs, lessons learned, results, logs, documents related to the project are archived
and used as a reference for future projects.

20