Challenge your Performance Test Estimation Effort Using Test Factors

Oct 2006

Ramya, Honeywell Technology Solutions Pvt Ltd, Ramya.Moorthy@honeywell.com

ABSTRACT
Performance Test effort estimation is the initial challenge we face in the Performance Test Lifecycle. As Performance Testing is more like an investigation activity which may take any direction, the ad-hoc estimation methodologies used for performance test estimation doesnt seem fruitful. Because of the dynamisms and varying scope, there are lots of issues in the test effort estimation. Based on our experience in the performance testing projects on the web based applications, we conjecture that the Performance Test Estimation approach using Test Factors could be more reliable. Our approach is based on the consideration of the various test factors involved in scripting, execution and analysis, thereby eliminating the use of ad-hoc estimation methodologies and it is architecture / technology independent.

This paper describes how to do effort estimation for a performance testing project. The paper focuses on the pre-requisites and the factors which need to be analyzed in order to arrive at the performance estimation model along with the details about how to use this estimation model. This approach is being implemented in Honeywell for the estimation of Performance Testing effort on web applications and is proven to be ninety-five percent accurate. Its been used to estimate ten projects falling under simple, medium and high complex project categories. The case study illustrates the application of this technique to arrive at the estimation for a typical web application.

The readability of the paper covers Performance Test Engineers, Test Leads, Test Managers and others with the domain knowledge of performance testing.

1.0 Introduction
Organizations generally like to make the headlines with their success stories and not about the failure web sites due to unexpected user load. Industry study reveals that poor performing IT applications cost industrialized nations almost $45 billion annually. Most of the organizations understand the importance of performance testing and the impact of bypassing it. How much time would it take to do the performance testing of an application? is a million dollar question for all organizations wherein the Software Performance Engineering (SPE) is not in place from the start of the project. Most of the organizations plan for the performance testing of the applications towards the end of the project, though it is of great importance to decide whether the performance bottlenecks are addressed before the go-alive dates.

Scott Barber rightly compares the Performance testing to crime investigation. It is tough to arrive at the effort estimation before investigating the system performance. This paper describes how to approach the performance test estimation for web applications and illustrates it with a case study. Its assumed that the reader is familiar with the performance test activities involved.

1.1 Issues in the Conventional Estimation Model

Performance Test Engineers use their past experience to estimate & schedule the performance test effort. Each organization follows its own way of estimation based on the type & scope of the performance testing projects.

Most of the Organizations use ROM (Rough Order of Magnitude) estimation based on the historical data & technical risk factors. For them, the test estimation effort is validated through the following breakup.

The Performance Test Planning phase contributes to 20% of the total effort. The Test Script generation contributes to 30% of the total effort. The Test Execution activities contribute to 45% of the total effort. The Performance Test Reporting contributes to 5% of the total effort. Various activities are carried out during each phase of performance testing as listed in the section 1.3. The importance of the activities carried out in each of the phases depends on the scope of the project which needs to be tested for its performance. In most of the cases, replanning is frequently required during the Test Execution Phase because of its dynamic nature. Projects planned for specific tests may end up having multiple rounds of unplanned tests because of the last minute surprises.

1.2 Overview of Test Factor Estimation Model

This estimation model is based on the Test Factors identified for each phase of the performance test lifecycle. This model sub divides the above explained four lifecycle phases into six phases namely the Test Strategy, Scripting, Environment Setup, Baselining & Benchmarking, Test Execution and Test Reporting for calculating the breakup efforts individually.

The Test Factors of a phase is associated with three values namely Importance Factor, Test Scope Factor and Test Complexity Factor. The Importance Factor is already defined (set as part of the estimation model) & it remains constant throughout the estimation. The Test Scope Factor needs to be set by the estimator based on the project. The Test Complexity

Factor is derived out as the product of the Importance Factor & the Test Complexity Factor. The Total Complexity Factor of the phase is the sum of the individual Test Complexity Factors of the phase divided by the number of test factors of the phase.

These three values are represented in specific way for each of the above mentioned six phases. The Importance Factor [IF] for all the test factors of each phase is represented with a specific superscript along with IF. For example, the Importance Factor for all the test factors in Test Strategy phase is represented as IFts. The Test Scope Factor [TSF] for all the test factors of each phase is represented with a specific superscript along with TSF. For example, the Test Scope Factor for all the test factors in Scripting phase is represented as TSFs. The Test Complexity Factor for all the test factors of each phase is represented with a prefix along with CF. For example, the Complexity Factor for all the test factors in Environment Setup phase is represented as ESCF.

The sum of the product of all Importance Factor [IF] and the Test Scope Factor [TSF] of a phase divided by the number of test factors available in a phase gives the Total Complexity Factor of the phase which is the total effort required for completing a phase in hours.

1.3 Assumptions in the Estimation Model

The activities performed in each phase of the performance test lifecycle are baselined as per the Organizations Testing process. The Test Factor estimation model doesnt cover the effort required for the initial system understanding phase and Review & Rework effort. The model also assumes that the project focal will be in the Practitioner Level in the competency framework. This estimation model holds good for performance testing projects which has the following activities. User Modeling Traffic Analysis Workload Modeling Performance Test Strategy development Generation of test scripts Test Environment Setup Test Data Generation Baselining & Benchmarking Load, Stress, Volume, Scalability & Isolation Tests Test Reporting

2.0 Test Factor Estimation Model

2.1 What is a Test Factor
In the Performance Test Lifecycle, all the phases discussed in the earlier section are driven by various factors which affect the effort required to complete a specific phase. These Factors are called Test Factors. These Factors have an impact on the effort required to complete the activities in a specific phase. For example, the effort required for a project to do its test planning is driven by the factors like: Effort required for User Modeling, Workload Modeling, Traffic Analysis, Feasibility Analysis of Monitors, etc. The scope & importance of these factors of the project determines the complexity & thereby the effort required to complete the phase.

2.2 What is a Importance Factor [IF]

The Test Factors are associated with an Importance Factor [IF] which is derived based on the impact of the factor in deciding the effort required for completing the phase. The Importance Factor [IF] values take any of the 3 values H, M, L which specifies H - High Impact, M - Medium Impact and L - Less Impact. The IF for the test factors can vary with respect to the organization and it is constant.

2.3 What is a Test Scope Factor [TSF]

Each Test Factor which is associated with an Importance Factor [IF] will have a Test Scope Factor [TSF] derived based on the scope of the Project. Based on the importance of the Test Factor in the scope of performance testing project (for which estimation needs to be done), the Test Scope Factor [TSF] values are decided. The Test Scope Factor [TSF] takes any one of the values 9, 5, 1, 0 which specifies 9 - Higher Scope, 5 - Medium Scope, 1 Lesser Scope, 0 Not Applicable. The Test Scope Factor [TSF] values needs to be entered by the estimator based on scope of the Performance Testing Project.

2.4 What is a Complexity Factor [CF]

Each Test Factor which is associated with an Importance Factor [IF] will be provided with the Test Scope Factor [TSF] by the estimator. The Product of Importance Factor [IF] and Test Scope Factor [TSF] provides the Complexity Factor for each of the test factors. The Total Complexity Factor is the sum of the products of Importance Factor [IF] and the Test

Scope Factor [TSF]. The Complexity Factor naming conventions with respect to each phase is available in the Figure3.

2.5 Effort Estimation for Test Strategy

The Test factors which affect the Test Strategy phase & the IFts which determines the impact of the factors in completing the phase are listed in the table1. Determine the TSFts [Test Scope factor] for each of these Test Factors & calculate the Test Strategy Complexity Factor [TSCF] against each factor. The Total TSCF is the sum of the products of each IFts & its corresponding TSFts. The Total TSCF value represents the effort required for the Test Strategy phase in hours. The TSCF value can be calculated using the below formulae Total TSCF = 4 + (sum (prod (IFts, TSFts))/6)

2.6 Effort Estimation for Script Design

The Test factors which affect the Scripting phase and the IFs which determine the impact of the factors in completing the phase are listed in the table2.

Determine the TSFs [Test Scope factor] for each of these Test Factors & calculate the Scripting Complexity Factor [SCF] against each factor. The Total SCF is the sum of the products of each IFts & its corresponding TSFs.The Total SCF value represents the effort required for a Script in hours. In the same way, calculate the SCF value for each of the script. The SCF value can be calculated using the below formulae Total SCF = sum (prod (IFs, TSFs))/7 + [sum (prod (IFs, TSFs))/7] .

2.7 Effort Estimation for Environment Setup

The Test factors which affect the Test Environment Setup phase and the IFes which determines the impact of the factors in completing the phase are listed in the table3. Determine the TSFes [Test Scope factor] for each of these Test Factors & calculate the Environment Setup Complexity Factor [ESCF] against each factor. The Total ESCF is the sum of the products of each IFes & its corresponding TSFes.The Total ESCF value represents the effort required for the Test environment setup in hours. The Co-ordination effort factor should not be counted while counting the number of test factors while calculating the Environment Setup Complexity Factor [ESCF] of the phase. The ESCF value can be calculated using the below formulae Total ESCF = sum (prod (IFes, TSFes))/4

2.8 Effort Estimation for Baseline & Benchmarking

The Test factors which affect the Baseline & Benchmarking phase and the IFbb which determines the impact of the factors in completing the phase are listed in the table4. Determine the TSFbb [Test Scope factor] for each of these Test Factors & calculate the Environment Setup Complexity Factor [BBCF] against each factor. The Total BBCF is the sum of the products of each IFbb & its corresponding TSFbb. The Total BBCF value represents the effort required for the Baseline & Benchmarking in hours. The BBCF value can be calculated using the below formulae Total BBCF = sum (prod (IFbb, TSFbb))/2

2.9 Effort Estimation for Test Execution

The Test factors which affect the Test Execution phase and the IFte which determines the impact of the factors in completing the phase are listed in the table5. Determine the TSFte [Test Scope factor] for each of these Test Factors & calculate the Test Execution Complexity Factor [TECF] against each factor. The Total TECF is the sum of the products of each IFte & its corresponding TSFte .The factors for Isolation Tests are separately identified & it is added along with the calculated test execution effort if there is a scope for isolation tests. The Total TECF value represents the effort required for the Test execution in hours. The Co-ordination effort factor should not be counted while counting the number of test factors while calculating the Test Execution Complexity Factor [TECF] of the phase. The TECF value can be calculated using the below formulae Total TECF = (sum (prod (IFte, TSFte))/7) + (sum (prod (IFte, TSFte))/3)

2.10 Effort Estimation for Test Reporting

The Test factors which affect the Test Reporting phase and the IFtr which determines the impact of the factors in completing the phase are listed in the table6. Determine the TSFtr [Test Scope factor] for each of these Test Factors & calculate the Test Reporting Complexity Factor [TRCF] against each factor. The Total TRCF is the sum of the products of each IFtr & its corresponding TSFtr. The Total TRCF value represents the effort required for the Test Reporting in hours. The TRCF value can be calculated using the below formulae Total TRCF = 4 + sum (prod (IFtr, TSFtr))/3

2.11 Final Performance Test Effort Calculation

The Final Performance Test Effort is the sum of the Complexity Factors of each of the six test phases along with the 10% of the estimated effort for Project Complexities and 10% of the estimated effort for management activities. These 2 efforts depend on the Organization.

Final Performance Test Effort = TSCF + SCF + ESCF + BBCF + TECF + TRCF + 10% of estimated effort (for Project Complexity) + 10% of estimated effort (Management activities).

3.0 Case Study

The example discussed in this section is the estimation done for a Honeywell intranet application. The estimation is done after understanding the system & doing the feasibility study on the scripting using Mercurys Load Runner 8.0 tool. 1. Estimate the Test Strategy development effort. Total TSCF = 4 + (sum (prod (IFts, TSFts))/6)

2. Estimate the Scripting effort Total SCF = [sum (prod (IFs, TSFs))/7*3] + [sum (prod (IFs, TSFs))/7 * 4]

3. Estimate Test Environment Setup effort Total ESCF = sum (prod (IFes, TSFes))/4

4. Estimate Baseline & Benchmarking effort Total BBCF = sum (prod (IFbb, TSFbb))/2

5. Estimate Test Execution effort Total TECF = (sum (prod (IFte, TSFte))/7) + (sum (prod (IFte, TSFte))/7) + (sum (prod (IFte, TSFte))/3)

6. Estimate Test Reporting effort Total TRCF = 4 + sum (prod (IFtr, TSFtr))/3

7. Final Test Effort Calculation Final Performance Test Effort = TSCF + SCF + ESCF + BBCF + TECF + TRCF + 10% of estimated effort (for Project Complexity) + 10% of estimated effort (Management activities). Final Performance Test Effort = 403 + 40.3 + 40.3 = 483.6 hours

4.0 Limitations
The estimation model based on Test Factors has the following limitations. 1. If the Performance Test Focal is not in the Practitioner Level of competency in Scripting, Test Execution & Analysis, then the estimated effort will not be accurate. 2. If the project includes any other specific activity other than the activities mentioned in the assumptions, then the estimated effort will not be accurate. For addition of any new activity, the test factors that would impact the efforts need to be identified. 3. The estimation will not cover the extra effort required, when there is a slippage from the external teams / customer for providing their inputs / tasks dependent on them.

5.0 Summary & Conclusions

This paper has described the application of the estimation technique for performance testing projects. Knowledge of the application & analyzing the feasibility for carrying out scripting/test execution play a vital role in this estimation technique. For each of the performance test lifecycle phases, rate the performance test project scope against each of the factors available. This provides the Complexity Factor for each of the factors. Sum of Products of all the Complexity Factors of each phase divided by the number of test factors of the phase will provide the total efforts required for the specific phase. The Sum of Complexity Factors of all phases provides the total test effort in hours. Add 10% of estimated effort for management activities & 10% of estimated effort for project complexities to get the total Performance Test Effort in hours.

Author Biography
I am an energetic Software Test Engineer with 4 years of extensive experience in System, Integration & Performance testing. I love to

work in a challenging environment which synergies my conceptual & technical skills coupled with hard work & determination. I turned out to be a Software Tester from being a developer in Microsoft platforms because of my passion towards testing. I have been working in J2EE, .NET & Client-Server applications as a system tester. I have more interest towards Model Based Testing approaches & Functional Test Automation using scripting languages like Ruby & Perl. I am working in the Performance Engineering field for the past 1+ year. As a performance engineer my responsibilities would include deriving the application usage model, generation of performance test scripts & bottleneck analysis. My other interests would include Security Testing.