Measure Quality On The Way in - Not Just On The Way Out: Author: Jan Fish Division: Philips Lifeline IT September, 2008

Measure Quality on the Way In –
Not Just on the Way Out
Author: Jan Fish

Email: jan.fish@philips.com
Division: Philips Lifeline IT
September, 2008
For Your Consideration
• Traditional Measurements for Test Organizations
• Value Added Opportunities
• 4 Phase Approach
– Target versus Actual Progress
– Bug Patterns within Builds / Deployments
– Workload Assessment for Outstanding Bugs
– Bug Injection Points and Bug Removal Points
• Measure Quality on the Way In; Not Just on the Way

Out
2
Traditional Measurements for Test Organizations
TEST METRICS:
 Standard of measurement
 Gauge effectiveness and efficiency
 Gathered and interpreted throughout the test effort
 Objectively measure success
PHILOSOPHY:
 Keep it simple
 Make it meaningful
 Track it
 Use it
3
Traditional Measurements for Test Organizations
BASE METRICS may include numbers and/or percentages for:
- Test cases created - Test cases in review

- Test cases to be executed - Test cases re-executed
- Test cases executed - Total executes
- Test cases passed - Total test cases passed
- Test cases failed - Total test cases failed
- Test cases blocked - Defect removal cost
- Bad test cases - Bad fixes
- Defects corrected - Test effectiveness (QA /
QA + Prod)
4
Value Added Opportunities
Forecast, Track and Respond:
 Inputs: Number of Test Cases, Testers and Cycle Time
First Run Failure Rate:
 Inputs: Failed Test Cases / Executed Test Cases
What is the Pattern:
 Inputs: Known Patterns vs. Current Pattern
Workload Distribution:
 Inputs: Number and Type of Bugs in DEV, QA and
Resolved
Bug Injection and Removal Points:
 Inputs: Error Created and Error Found
Plan It, Track It and Graph It:
 Inputs: Actual Progress to Targeted Goal
5
Target vs. Actual – Existing Project
Target Testcases Testcases Testcases Failure Rate
Date Executed Passed Failed (%)
Plan Actual Plan Actual Plan Actual Forecast Actual
Week 1 30 19 24 16 6 3 20.0% 15.8%
Week 8 275 110 384 59 11 46 20.0% 41.8%
Week 14 20 6 8 0 2 6 10.0% 100%

Running
total 475 201 431 102 19 87 4.0% 43.3%
43% : target
6
Target Planned vs. Passed vs. Failed Test Cases
7
New and Improved Target versus Actual
Test Cases Test Cases Test Cases
Target Date Retests Failure Rate (%) % Done
Executed Passed Failed
Target Actual Target Actual Target Actual Pending Done Forecast Actual Target Actual
Smoke Tests
6/18 3 3 3 2 0 1 1 0 0% 33%
Running total 3 3 3 2 0 1 1 0 0% 33% 9% 9%

Functional Tests
6/19 4 3 3 0 1 3 4 0 25% 100%
Running total 7 6 6 2 1 4 4 0 14% 67% 21% 18%

6/20 6 7 5 6 1 1 5 0 17% 14%
Running total 13 13 11 8 2 5 5 0 15% 38% 38% 38%

Exploratory Tests
6/23 2 2 2 2 0 0 3 2 0% 0%
Running total 15 15 10 10 2 5 3 2 13% 33% 35% 44%

Load Tests
6/24 3 3 3 3 0 0 0 3 0% 0%
Running total 18 18 10 13 2 5 0 5 11% 28% 35% 53%

Regression
6/25 15 15 15 14 0 1 1 0 0% 7%
Running total 33 33 25 27 2 6 1 5 6% 18% 79% 97%

6/26 1 1 1 1 0 0 0 1 0% 0%
Running total 34 34 26 28 2 6 0 6 6% 18% 82% 100%

8
New and Improved Chart
9
Target vs. Actual Math
• Target Test Cases Executed math = Target Test Cases Passed + Target Test
Cases Failed
• Actual Test Cases Executed math = Actual Test Cases Passed + Actual Test
Cases Failed
• Pending Retests math = Previous Pending Retests + Actual Test Cases Failed -
Retests Done
• Running Total math = Actual Values + Previous Reported Values
• Forecast Failure Rate math = Target Test Case Failed / Target Test Cases
Executed
• Actual Failure Rate math = Actual Test Case Failed / Actual Test Cases
Executed
• Forecast % Done math = Current Target (Passed + Failed) / Final Target Test
Cases Executed
• Actual % Done math = Current Actual (Passed + Failed) / Final Target Test
Cases Executed
10
The Added Values of Target vs. Actual
TRACK to plan and REACT immediately
 Is quality built in
 Is the build / deployment truly ready for test
 Is the test resource on schedule with each type of test
execution
 Functional
 Regression
 Load / Performance / Security
 Bug Fix Validation
11
PREDICT how many test cases should be run in a given time

period
To date, our First Run Failure Rates are:
 7% - 12% maintenance of existing functionality

 20% - 35% added functionality to existing Application
 30% - 47% new Application developed on-site
 10% - 25% new Application developed off-site
12
ADJUST time and / or staff on an immediate basis
 Substantiate “Gut Feel”

 Demonstrate facts
 Moderate re-work and know if it fits the schedule
 Adjust the plan and determine the level of effort well
before the last quadrant of the test cycle
13
SET realistic, track-based entrance criteria (as agreed upon
with upstream partners) for what is or is not acceptable
quality at the start of the project
USE the results to document expectations into project

contracts (internal or external)
 Establish what constitutes ”acceptable quality level”
 Set contract conditions for scaled fees based on
exceeding, meeting or failing quality objectives
14
PUBLISH and POST printed copy in a common area
 Eliminate the arguments

 Eliminate the negative chatter
 Document facts; don’t point fingers
 Grid each application/area Failure Rates in like-sets
 Review to determine likely process improvements
 Continue to track implemented improvement and assess
if it had a positive or negative impact
15
Bug Pattern Recognition – Sample 1
Build Date Critical High Medium Low Total Running
Name Total
b1 02/01 2 15 30 13 60 60
02/06 0 10 27 23 60 120
02/13 1 5 10 28 44 164
b2 02/15 10 3 0 0 13 177
02/20 5 15 6 0 26 203
02/22 0 23 12 14 49 252
02/26 0 5 13 7 25 277
b3 02/29 27 12 0 0 39 316
TOTALS 45 88 98 85 316 316
16
Pattern Recognition – Existing Project 2
Build
Date Date C H M L TT Run TT
02/04/2008 02/05/2008 0 0
02/07/2008 1 1 1
02/11/2008 02/12/2008 2 2 3
02/14/2008 0 3
02/19/2008 02/20/2008 1 1 4
02/22/2008 2 2 6
02/25/2008 02/26/2008 1 1 7
02/28/2008 1 1 8
TOTAL 7 1 0 0 8 8
17
The Added Values of Pattern Recognition
 INSTITUTE a process change and know the effect
 INCREASE the precision of estimates
 IDENTIFY trends and observations supporting or

hindering test and project plan
 INFORM team and management of patterns found in a

crisp, clean and simple manner such that the whole team
can make “next step” decisions
18
Bug, Bug, who‘s got the
Bug?
Bugs in Dev Bugs in QA
Pending Pending QA
Resolved
Development Fix Retest
C H M L TT Rev Fix TT C Rej Fut TT
11/02 0 1 2 0 3 0 0 0 0 0 0 0
11/19 0 2 2 0 4 4 7 11 4 2 0 6
11/27
19 1 10 9 2 22 3 5 8 9 2 0 11
The Added Values of Bug Reporting by Group
 WORKLOADS can be easily identified by management and

team members
 INFORMED DECISIONS can be made on next steps

without drilling into details (who, when, what, how and
where)
 RE-WORK can be tracked and compared to planned work
 PROGRESS toward resolving outstanding bugs
20
Bug Injection/Removal Points
21
Added Values of Bug Injection & Removal Points
 BASIS for estimating the number of bugs to be found at each
phase of the Software Development Lifecycle
 IDENTIFIED work cycles that would benefit from process
improvements and inspection points
 VALIDATE that improvements work
 DEMONSTRATE that quality must be the goal of all team
members, not just the responsibility of the test organization
“Quality is never an accident; it is always the result of

intelligent effort.” *
*John Ruskin, English writer and critic of art, architecture and society. 1819 - 1900
22
Measure Quality on the Way In; Not Just on the Way Out
• PEOPLE do not intentionally make a bad plan but they may
not be able to quickly adjust the plan to current
circumstances and abate risks
• PEOPLE do look for paths of least resistance and, if it is
easy to blame others, they will
• MEASURING quality once product is in production tells the
tale of what was not trapped and fixed but not the tale of
quality before production
• QUALITY can not be tested into a product but you can
measure quality coming into your test organization
• FOCUS on the level of quality at the project level
23

Measure Quality On The Way in - Not Just On The Way Out: Author: Jan Fish Division: Philips Lifeline IT September, 2008

Uploaded by

Copyright:

Available Formats

Measure Quality On The Way in - Not Just On The Way Out: Author: Jan Fish Division: Philips Lifeline IT September, 2008

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Measure Quality On The Way in - Not Just On The Way Out: Author: Jan Fish Division: Philips Lifeline IT September, 2008

Uploaded by

Copyright:

Available Formats

Measure Quality on the Way In –

Not Just on the Way Out

Author: Jan Fish

• Value Added Opportunities

• Measure Quality on the Way In; Not Just on the Way

- Test cases created - Test cases in review

Plan Actual Plan Actual Plan Actual Forecast Actual

Week 1 30 19 24 16 6 3 20.0% 15.8%

Week 8 275 110 384 59 11 46 20.0% 41.8%

Week 14 20 6 8 0 2 6 10.0% 100%

Running total 3 3 3 2 0 1 1 0 0% 33% 9% 9%

6/19 4 3 3 0 1 3 4 0 25% 100%

Running total 7 6 6 2 1 4 4 0 14% 67% 21% 18%

Running total 13 13 11 8 2 5 5 0 15% 38% 38% 38%

Running total 15 15 10 10 2 5 3 2 13% 33% 35% 44%

Running total 18 18 10 13 2 5 0 5 11% 28% 35% 53%

Running total 33 33 25 27 2 6 1 5 6% 18% 79% 97%

Running total 34 34 26 28 2 6 0 6 6% 18% 82% 100%

• Running Total math = Actual Values + Previous Reported Values

TRACK to plan and REACT immediately

PREDICT how many test cases should be run in a given time

To date, our First Run Failure Rates are:

 7% - 12% maintenance of existing functionality

ADJUST time and / or staff on an immediate basis

 Substantiate “Gut Feel”

USE the results to document expectations into project

PUBLISH and POST printed copy in a common area

 Eliminate the arguments

TOTALS 45 88 98 85 316 316

 INCREASE the precision of estimates

 IDENTIFY trends and observations supporting or

 INFORM team and management of patterns found in a

Bugs in Dev Bugs in QA

C H M L TT Rev Fix TT C Rej Fut TT

 WORKLOADS can be easily identified by management and

 INFORMED DECISIONS can be made on next steps

 RE-WORK can be tracked and compared to planned work

 PROGRESS toward resolving outstanding bugs

“Quality is never an accident; it is always the result of

You might also like