Software Quality Assurance

Quality
• The American Heritage Dictionary defines quality as
“a characteristic or attribute of something.”
• For software, two kinds of quality may been countered:
• Quality of design encompasses requirements,
specifications, and the design of the system.
• Quality of conformance is an issue focused primarily on
implementation.
• user satisfaction = compliant product + good
• quality + delivery within budget and schedule
 Software Quality
• Conformance to explicitly stated functional
and performance requirements, explicitly
documented development standards, and
implicit characteristics that are expected of all
professionally developed software.
 Elements of SQA
• Standards
• Reviews and Audits
• Testing
• Error/defect collection and analysis
• Change management
• Education
• Vendor management
• Security management
• Safety
• Risk management
 Role of the SQA Group-I
• Prepares an SQA plan for a project.
– The plan identifies
• evaluations to be performed
• audits and reviews to be performed
• standards that are applicable to the project
• procedures for error reporting and tracking
• documents to be produced by the SQA group
• amount of feedback provided to the software project team
• Participates in the development of the project’s
software process description.
– The SQA group reviews the process description for
compliance with organizational policy, internal software
standards, externally imposed standards (e.g., ISO-9001),
and other parts of the software project plan.
 Role of the SQA Group-II
• Reviews software engineering activities to verify compliance with the
defined software process.
– identifies, documents, and tracks deviations from the process and verifies that
corrections have been made.
• Audits designated software work products to verify compliance with those
defined as part of the software process.
– reviews selected work products; identifies, documents, and tracks deviations;
verifies that corrections have been made
– periodically reports the results of its work to the project manager.
• Ensures that deviations in software work and work products are
documented and handled according to a documented procedure.
• Records any noncompliance and reports to senior management.
– Noncompliance items are tracked until they are resolved.
 SQA Goals
• Requirements quality. The correctness, completeness, and
consistency of the requirements model will have a strong influence
on the quality of all work products that follow.
• Design quality. Every element of the design model should be
assessed by the software team to ensure that it exhibits high quality
and that the design itself conforms to requirements.
• Code quality. Source code and related work products (e.g., other
descriptive information) must conform to local coding standards
and exhibit characteristics that will facilitate maintainability.
• Quality control effectiveness. A software team should apply
limited resources in a way that has the highest likelihood of
achieving a high quality result.
Statistical SQA
Product Collect information on all defects
& Process Find the causes of the defects
Move to provide fixes for the process

measurement

... an understanding of how

to improve quality ...
 Statistical SQA
• Information about software errors and defects is collected and
categorized.
• An attempt is made to trace each error and defect to its underlying
cause (e.g., non-conformance to specifications, design error,
violation of standards, poor communication with the customer).
• Using the Pareto principle (80 percent of the defects can be traced to
20 percent of all possible causes), isolate the 20 percent (the vital
few).
• Once the vital few causes have been identified, move to correct the
problems that have caused the errors and defects.
 Software Reviews
• Software reviews are a filter for the software engineering
process.
• Reviews are applied at various points during software
development and serve to uncover errors and defects that
can then be removed.
• Software reviews purify the software engineering
activities that we have called analysis , design and
coding.
• A review is a way of using the diversity of a group of people
to:
– Point out needed improvements in the product of a
single person or team;
– Confirm those parts of a product in which improvement
is
either not desired or not needed;
– Achieve technical work of more uniform, or at least more
predictable, quality than can be achieved without
reviews, in order to make technical work more
manageable.
• There are different types of review with different objectives
– Inspections for defect removal (product);
– Reviews for progress assessment (product and process);
– Quality reviews (product and standards)
11
 Software
Reviews
• A group examines part or all of a process or system and its
documentation to find potential problems.
• Software or documents may be 'signed off' at
a review which signifies that progress to the
next development stage has been approved
by management.
• ).

12
 Quality Reviews
• A group of people carefully examine part or all
of a software system and its associated documentation.
• Code, designs, specifications, test plans, standards, etc. can
all be reviewed.
• Software or documents may be 'signed off' at a review
which signifies that progress to the next development
stage has been approved by management.

13
Software Reviews
Process

14
Reviews

... there is no particular

reason why your friend and
colleague
cannot also be your sternest
critic.
Jerry Weinberg

1
5
What Are Reviews?
 a meeting conducted by technical
people for technical people
 a technical assessment of a work
product created during the software
engineering process
 a software quality assurance
mechanism
 a training ground

1
6
What Reviews Are Not
 A project summary or progress assessment
 A meeting intended solely to impart
information
 A mechanism for political or personal
reprisal!

1
7
What Do We Look For?
 Errors and defects
 Error—a quality problem found before the software is released
to end users
 Defect—a quality problem found only after the software has been
released to end-users
 We make this distinction because errors and defects have very
different economic, business, psychological, and human
impact
 However, the temporal distinction made between errors and
defects in this book is not mainstream thinking

1
8
 Defect Amplification
 A defect amplification model [IBM81] can be used to illustrate
the generation and detection of errors during the design and
code generation actions of a software process.

Defects Detection
Errors from Errors passed through
Previous step Percent Errors passed
Amplified errors 1:x Efficiency To next step

Newly generated errors

Development step

1
9
 Defect Amplification
 In the example provided in SEPA, Section
15.2,
 a software process that does NOT include reviews,
• yields 94 errors at the beginning of testing and
• Releases 12 latent defects to the field
 a software process that does include reviews,
• yields 24 errors at the beginning of testing and
• releases 3 latent defects to the field
 A cost analysis indicates that the process with NO
reviews costs approximately 3 times more than the
process with reviews, taking the cost of correcting
the latent defects into account

2
0
 Metrics
 The total review effort and the total number of
errors discovered are defined as:
•E review = Ep + Ea + Er

 • Err tot = Errminor + Errmajor

Defect density represents the errors found per

unit of work product reviewed.
 • Defect density = Err tot / WPS

where …

2
1
Metrics
 Preparation effort, Ep—the effort (in person-hours) required to
review a work product prior to the actual review meeting
 Assessment effort, Ea— the effort (in person-hours) that is
expending during the actual review
 Rework effort, Er— the effort (in person-hours) that is dedicated
to the correction of those errors uncovered during the review

Work product size, WPS—a measure of the size of the work
product that has been reviewed (e.g., the number of UML
models, or the number of document pages, or the number of
lines of code)
 Minor errors found, Errminor—the number of errors found that can
be categorized as minor (requiring less than some pre-specified
effort to correct)

Major errors found, Errmajor— the number of errors found that
can be categorized as major (requiring more than some pre-
specified effort to correct)
An Example—I
 If past history indicates that
 the average defect density for a requirements model
is 0.6 errors per page, and a new requirement
model is 32 pages long,
 a rough estimate suggests that your software team
will find about 19 or 20 errors during the review of
the document.
 If you find only 6 errors, you’ve done an extremely
good job in developing the requirements model or
your review approach was not thorough enough.

23
An Example—II
 The effort required to correct a minor model error (immediately after
the review) was found to require 4 person-hours.
 The effort required for a major requirement error was found to be 18
person-hours.
 Examining the review data collected, you find that minor errors occur
about 6 times more frequently than major errors. Therefore, you can
estimate that the average effort to find and correct a requirements error
during review is about 6 person-hours.
 Requirements related errors uncovered during testing require an
average of 45 person-hours to find and correct. Using the averages
noted, we get:
 Effort saved per error = Etesting – Ereviews
 45 – 6 = 30 person-hours/error
 Since 22 errors were found during the review of the requirements
model, a saving of about 660 person-hours of testing effort would be
achieved. And that’s just for requirements-related errors.
Overall
 Effort expended with and without reviews

with reviews
Reference Model

26
Informal Reviews
 Informal reviews include:
 a simple desk check of a software engineering work
product with a colleague
 a casual meeting (involving more than 2 people) for the
purpose of reviewing a work product, or
 the review-oriented aspects of pair programming
 pair programming encourages continuous review as
a work product (design or code) is created.
 The benefit is immediate discovery of errors and better
work product quality as a consequence.

27
Formal Technical Reviews
 The objectives of an FTR are:
 to uncover errors in function, logic, or implementation for
any representation of the software
 to verify that the software under review meets its
requirements
 to ensure that the software has been represented according
to predefined standards
 to achieve software that is developed in a uniform manner
 to make projects more manageable
 The FTR is actually a class of reviews that includes
walkthroughs and inspections.
 The Review Meeting
• The duration of the review meeting should be less than two
hours. Between three and five people (typically) should be
involved in the review.
• Advance preparation should occur but should require no more
than two hours of work for each person.
• Focus is on a work product (e.g., a portion of a requirements
model, a detailed component design, source code for a
component)
The Players
review
standards bearer (SQA)
leader
producer

maintenance
oracle

recorder reviewer
user rep
Conducting the Review
 Review the product, not the producer.
 Set an agenda and maintain it.
 Limit debate and rebuttal.
 Enunciate problem areas, but don't attempt to solve every
problem noted.
 Take written notes.
 Limit the number of participants and insist upon advance
preparation.
 Develop a checklist for each product that is likely to be
reviewed.
 Allocate resources and schedule time for FTRs.
 Conduct meaningful training for all
 reviewers.
Review your early reviews.
 Review Options
Matrix
IPR* WT IN RRR
trained leader no yes yes
agenda established yes maybe yes yes
reviewers prepare in advance yes maybe yes yes
producer presents product yes maybe no no
“reader” presents product yes no no yes no
recorder takes notes maybe yes yes yes
checklists used to find errors no no yes no
errors categorized as found no no yes no
issues list created no yes yes
team must sign-off on result yes yes maybe
no
yes

I P*R—informal peer review WT—

33
Walkthrough IN— Inspection RRR—round
Sample-Driven Reviews (SDRs)
 SDRs attempt to quantify those work products that are
primary targets for full FTRs.
To accomplish this …
 Inspect a fraction ai of each software work product, i.

Record the number of faults, fi found within ai.

 Develop a gross estimate of the number of faults within
work product i by multiplying fi by 1/ai.

Sort the work products in descending order according to
the gross estimate of the number of faults in each.

Focus available review resources on those work
products that have the highest estimated number of
faults.

34
Metrics Derived fromReviews
inspection time per page of documentation
inspection time per KLOC or FP
inspection effort per KLOC or FP
errors uncovered per reviewer hour
errors uncovered per preparation hour
errors uncovered per SE task (e.g., design)
number of minor errors (e.g., typos)
number of major errors
(e.g., nonconformance to req.)
number of errors found during
preparation
 Six-Sigma for Software Engineering
• The term “six sigma” is derived from six standard deviations—3.4 instances
(defects) per million occurrences—implying an extremely high quality standard.
• The Six Sigma methodology defines three core steps:
– Define customer requirements and deliverables and project goals via well-defined
methods of customer communication
– Measure the existing process and its output to determine current quality performance
(collect defect metrics)
– Analyze defect metrics and determine the vital few causes.
– Improve the process by eliminating the root causes of defects.
– Control the process to ensure that future work does not reintroduce the causes of
defects.
 Software Reliability
• A simple measure of reliability is mean-time-
between-failure (MTBF), where
MTBF = MTTF + MTTR

• The acronyms MTTF and MTTR are mean-time-to-

failure and mean-time-to-repair, respectively.
• Software availability is the probability that a
program is operating according to requirements
at a given point in time and is defined as
Availability = [MTTF/(MTTF + MTTR)] x 100%
 Software Safety
• Software safety is a software quality assurance
activity that focuses on the identification and
assessment of potential hazards that may
affect software negatively and cause an entire
system to fail.
• If hazards can be identified early in the
software process, software design features
can be specified that will either eliminate or
control potential hazards.
 ISO 9001:2000 Standard
• ISO 9001:2000 is the quality assurance standard that applies to software
engineering.
• The standard contains 20 requirements that must be present for an
effective quality assurance system.
• The requirements delineated by ISO 9001:2000 address topics such as
– management responsibility, quality system, contract review, design control,
document and data control, product identification and traceability, process
control, inspection and testing, corrective and preventive action, control of
quality records, internal quality audits, training, servicing, and statistical
techniques.