Software Quality Management

Lecture 5: Test process and 7 testing

principles

Produced by VTT in Aug 2020

 Contents

1. Fundamental test process

2. Generic testing process
3. Test plan and test cases in the real world
4. Test document standards
5. Validation vs. Verification
6. 7 testing principles

Test process + 7 testing principles 2

 Fundamental test process

 Depend on the system and software context and the

level of risk associated with the software, the activities
within the fundamental test process can be divided
into the following basic steps:
o Test planning and control
o Test analysis and design
o Test implementation and execution
o Evaluating exit criteria and reporting
o Test closure activities

Test process + 7 testing principles 3

 Fundamental test process (cont’d)
Test planning
During test planning, we make sure we understand the goals and
objectives of the customers, stakeholders, and the project, and the
risks which testing is intended to address. Six major tasks of TP are:
 Determine the scope and risks and identify the objectives of
testing: we consider what software, components, systems or
other products are in scope for testing
 Determine the test approach (techniques, test items, coverage,
identifying and interfacing with the teams involved in testing): we
consider how we will carry out the testing, the techniques to use,
what needs testing and how extensively
 Implement the test policy and/or the test strategy: we mentioned
that there may be an organization or program policy and strategy
for testing.
Test process + 7 testing principles 4
 Fundamental test process (cont’d)

Test planning
 Determine the required test resources (e.g. people, test
environment, PCs): from the planning we have already done we
can now go into detail
 Schedule test analysis and design tasks, test implementation,
execution and evaluation: we will need a schedule of all the tasks
and activities, so that we can track them and make sure we can
complete the testing on time.
 Determine the exit criteria: we need to set criteria such as
coverage criteria (for example, the percentage of statements in
the software that must be executed during testing) that will help
us track whether we are completing the test activities correctly

Test process + 7 testing principles 5

 Fundamental test process (cont’d)

Test control
Test control is an activity to control and manage progress against the
plan, and to report to PM or customer on the current status of testing
including any changes or deviation from the plan. Five major task of
test control are:
 Measure and analyze the results of review or testing: how many
of them have been done, track for how many passed, how many
failed, along with the number, type and importance of the defects
reported
 Monitor and document progress, test coverage and exit criteria:
inform the project team how much testing has been done, what
the results are, what conclusions and risk assessment we have
made

Test process + 7 testing principles 6

 Fundamental test process (cont’d)

Test control
 Provide information on testing: expect to make regular and
exceptional reports to the PM, customers, stakeholders …to help
them make informed decisions about project status
 Initiate corrective actions: tighten exit criteria for defects fixed,
ask for more effort to be put into debugging, prioritize defects to
be fixed at first
 Make decisions: to continue testing, to stop testing, to release the
software or to retain it for further work

Test process + 7 testing principles 7

 Fundamental test process (cont’d)

Analysis and design

Test analysis and design is the activity where general testing
objectives are transformed into tangible test conditions and test
designs. During test analysis and design, we take general testing
objectives identified during planning and build test designs and test
procedures (scripts). Its major tasks are:
 Review the test basis (such as the product risk analysis,
requirements, architecture, design specifications, and interfaces),
examining the specifications for the software we are testing
 Identify test conditions based on analysis of test items, their
specifications, and what we know about their behavior and
structure.

Test process + 7 testing principles 8

 Fundamental test process (cont’d)

Analysis and design

 Design the tests, using techniques to help select representative
tests that relate to particular aspects of the software which carry
risks or which are of particular interest, based on the test
conditions and going into more detail.
 Evaluate testability of the requirements and system
 Design the test environment set-up and identify any required
infrastructure and tools. This includes testing tools and support
tools such as spreadsheets, word processors, project planning
tools, and non-IT tools and equipment - everything we need to
carry out our work.

Test process + 7 testing principles 9

 Fundamental test process (cont’d)
Implementation and execution
During test implementation and execution, we take the test conditions
and make them into test cases and set up the test environment.
 Implementation:
o Develop and prioritize our test cases and create test data for those
tests.
o Create test suites from the test cases for efficient test execution.
o Implement and verify the environment.
 Execution:
o Execute the test suites and individual test cases, following our test
procedures.
o Log the outcome of test execution and record the identities and
versions of the software under test, test tools
o Compare actual results with expected results.

Test process + 7 testing principles 10

 Fundamental test process (cont’d)
Evaluating exit criteria and reporting
Evaluating exit criteria is the activity where test execution is assessed
against the defined objectives. This should be done for each test
level, as for each we need to know whether we have done enough
testing.
 Evaluating exit criteria has the following major tasks:
o Check test logs against the exit criteria specified in test planning.
o Assess if more tests are needed or if the exit criteria specified
should be changed.
o Write a test summary report for stakeholders.

Test process + 7 testing principles 11

 Fundamental test process (cont’d)
Test closure activities
During test closure activities, we collect data from completed test
activities to consolidate experience, including checking, and analyzing
facts and numbers.
 Test closure activities include the following major tasks :
o Check which planned deliverables we actually delivered and ensure
all incident reports have been resolved through defect repair or
deferral.
o Finalize and archive testware, such as scripts, the test environment,
and any other test infrastructure, for later reuse.
o Hand over testware to the maintenance organization who will
support the software and make any bug fixes or maintenance
changes, for use in confirmation testing and regression testing.
o Evaluate how the testing went and analyze lessons learned for
future releases and projects.

Test process + 7 testing principles 12

 Generic testing process
Set reliability/coverage goals

Planning & Preparation

- goal setting Reliability/
Entry - information gathering No coverage Yes
Execution
- model construction Test cases & Feedback & goals Exit
- test cases procedure adjustments satisfied?
- test procedure Measurements

Defect Analysis/
Selected handling modeling
Analysis &
measurements results
Follow-up
& models

(Tian J., 2005)

Test process + 7 testing principles 13

 Test plan and test cases in practice
Development phases Test plan & test cases
Preliminary Draft test plan + write test cases
investigation + for post-implementation
Analysis
Update test plan + write test cases
for final construction
Design
Update test plan + write test cases
for preliminary construction
Preliminary Execute test cases for preliminary
construction construction

Final Execute test cases for final

construction construction

Execute test cases for post

Installation
implementation

Test process + 7 testing principles 14

 Test document standards

Chuẩn tài liệu IEEE #829-1998

1. Test plan
2. Test design specification
3. Test case specification
4. Test procedure specification
5. Test log specification
6. Test incident report
7. Test summary report

Test process + 7 testing principles 15

 Validation vs. Verification

 Validation checks the conformance to quality

expectation of customers or users in the form of whether
the expected functions or features are presented or not.

 "Are we building the product right?“

 Verification checks the conformance of software product

implementation against its specifications to see if it is
implemented correctly.

 "Are we building the right product?“

(Tian J., 2005)

Test process + 7 testing principles 16

 7 testing principles
Testing can show that defects are present, but cannot
1. Testing shows prove that there are no defects. Testing reduces the
presence of defects probability of undiscovered defects remaining in the
software but, even if no defects are found, it is not a
proof of correctness.
Testing everything (all combinations of inputs and
2. Exhaustive preconditions) is not feasible. Instead of exhaustive
testing is testing, we use risks and priorities to focus testing
impossible efforts.
Testing activities should start as early as possible in the
3. Early testing software or system development life cycle and should be
focused on defined objectives.
A small number of modules contain most of the defects
4. Defect clustering discovered during prerelease testing or show the most
operational failures.

Test process + 7 testing principles 17

 7 testing principles (cont’d)
If the same tests are repeated over and over again,
eventually the same set of test cases will no longer find
any new bugs. To overcome this 'pesticide paradox', the
5. Pesticide paradox test cases need to be regularly reviewed and revised,
and new and different tests need to be written to exercise
different parts of the software or system to potentially find
more defects.
Testing is done differently in different contexts. For
6. Testing is context
example, safety-critical software is tested differently from
dependent
an e-commerce site.
Finding and fixing defects does not help if the system built
7. Absence-of-errors
is unusable and does not fulfill the users' needs and
fallacy
expectations

Test process + 7 testing principles 18