« 

BEST TEST PRACTICES FOR »

« INTRANET APPLICATIONS»

Nom du document 41314048.doc

Version 001
Status du document Draft
Auteur H 十七 N 十七十七五 L
Date de création 28/03/2006
Nombre de pages 24
 TECHNOLOGIES DE L'INFORMATION

1. INTRODUCTION 4
1.1. DOCUMENT SCOPE 4
1.2. REFERENCES 4
1.3. GLOSSARY 4
1.4. CONVENTIONS 5
2. BEST TEST PRACTICES 6
2.1. FOREWORD 6
2.2. REQUIREMENTS 6
2.3. TEST CASE DESIGN 7
2.4. QUALITY FEATURES 9
2.5. LOAD TESTING 12
2.6. AUTOMATION TESTING 13
2.7. REPORTS 14
2.8. USER ACCEPTANCE TESTING 14
2.9. ISSUE MANAGEMENT 16
3. GLOBAL TESTS RESULTS 17
3.1. INTRODUCTION 17
3.2. RESSOURCES 17
3.3. STATISTICS 18
3.4. COMMENTS 19
4. DEPLOYMENT 20
4.1. INTRODUCTION 20
4.2. BY ROOT CAUSE 20
4.3. BY IMPORTANCE 21
4.4. COMMENTS 21
5. CONCLUSIONS 22
5.1. INTRODUCTION 22
5.2. IMPROVEMENTS 22

Best Test Practices for - Intranet Applications Page 2 / 24§

H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

CHANGES HISTORY

Version Date Related Sections Description Author(s)

001 23 March 2006 Document creation Fabiano Gaiga

002 12 May 2006 Document reviewed René Maes

Best Test Practices for - Intranet Applications Page 3 / 24§

H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

1. INTRODUCTION

1.1. Document scope

This document is a collection of formal test practices put in place in a first attempt to validate an
intranet application at the customer organization. The intranet application is named TOP and the end
users belong to the department verifying incoming documents in a large workflow approach.

By test practices, we mean test practices related to black box testing only, so unit and integration
testing is not discussed here.

The document is focused on an intranet application because it is one of the most popular application
style that we actually see today. In our perception, an intranet application implies the following
assumptions:

Feature Description
Security The security is set at an average; I mean the identification process is necessary but no
possible connection is allowed from outside but only from different geographical
locations.
Ergonomics The GUI is driven by HTML screens.
Client The client software running the application is Internet Explorer or at minimum the
browser recommended at corporate level.
Asynchronous The information flow respects the http protocol with forms sent to the server and
mode reply screens sent to the client. . No information is refreshed on screen or rebuilt
dynamically within the browser.

1.2. References
Domain/File name Comment Reference
Chartes_Site_Internet Charte de normalisation de la présence de eLuxembourg Project
l'état Luxembourgeois
Guide ergonomie sites CNRS Ergonomics recommendations CNRS/DSI/QUAL/WEB/GUIDE
web ERGO
Musée national Web sites related to the cultural department www.mnha.lu
d’histoire et d’art
(Luxembourg)
PE_Guide_Ergonomique Internal ergonomics recommendations
UML This is a very useful web site explaining in http://www.agilemodeling.com/s
details the rules and semantic of all UML tyle/
diagrams.
Security Site more related to Internet applications www.cert.org
but can be interesting to understand
specific techniques or threats.
Automation Product presentation, user guides. www.Fitnesse.org
Test forum Very good forum (a bit slow), splitted by www.qaforums.com/cgi-
test domains. bin/forums/ultimatebb.cgi
Test tool Best pairs www.satisfice.com/tools.shtml
Test tool Orthogonal matrix www.stsc.hill.af.mil/consulting/s
w_testing/improvement/cst.html

1.3. Glossary
Term Description
LDAP Lightweight Directory Access Protocol. Service giving access to directories
content. This service was used to manage the members of the customer
directory
Best Test Practices for - Intranet Applications Page 4 / 24§
H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

TO Tabling Office. This is the customer. The tabling office is responsible to

verify law proposal documents before sending them to translation services.
Documents are validated from a linguistic and juridical point of view.
TOP Tabling Office Portal. This is the application name. As the names suggests,
the first idea was to implement a portal, or a central point of information,
for the TO, later and with better definition of the requirements, the portal
became an interactive application.
Static test Is a form of software testing where the actual software isn't used. Instead
methods like code review, inspection and walk-through are used.
IE Internet Explorer
URL Unified resource Locator
Test campaign The execution phase of Testing. A test campaign includes all the test cases
designed for a specified version of the application.

1.4. Conventions

Icon conventions
Important information.
General information

Best Test Practices for - Intranet Applications Page 5 / 24§

H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

2. BEST TEST PRACTICES

2.1. Foreword

This is the first formal test approach introduced and implemented at the customer organization, in the
department of information distribution.

This first experimental approach was put in place to improve specific aspects that were causing
recurrent problems in different projects of this department:

 Applications refused at User acceptance level: This is the worst scenario for the IT
department in any organization.
 Poor application quality: For live applications, the general feeling from end users, is that the
applications are not user friendly, difficult to use, and bugs are frequent. We will not go into
details here but we can give many examples. Reading Error: Reference source not found, will
make you understand what we mean.
 Too much effort dedicated to support: This is a consequence of the second point, if the
applications are not reaching sufficient quality level, one can expect a strong negative
feedback from end users and a lot of time spent on solving related issues.

Different aspects of testing will be addressed in next chapters and they will be described in
chronological order as you would expect of any IT project: from inception to deployment.

However, when work was started on the project, inception deliverables were already produced and
the coding phase was underway with first iteration almost completed. Due to this context and to
limited technology background, there was no white box testing or unit testing foreseen in the Test
plan.
2.2. Requirements

2.2.1. Approach

For those deliverables, a static test approach was completed. In this type of approach, one needs a
good understanding of the different artefacts during this phase. For instance in this project, it was
necessary to have a good understanding of the UML deliverables.

A fair comprehension is needed to be able to:

 Validate the consistency across different documents

 Remove black holes in the analysis

 Point out ambiguous requirements

The three points above are crucial as this is where you can, at this early stage of the project, reduce
communication errors between users, analysts, and developers.

The last but not least important aspect of static testing is to check if requirements are testable.
Which means that you should always ask yourself “How will I test this requirement?”. For example, in
the TOP project, it was requested to have an error management in case of issues coming from external
applications; as we were not able to control the output of those applications, especially in the Test
environment, this was a major risk because no tests were actually feasible.

Static testing is a good start to identify Tests scenarios and high level definition of tests cases.
Best Test Practices for - Intranet Applications Page 6 / 24§
H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

2.2.2. Limitations

According to the technology used by Intranet applications, specific requirements might be impossible to
put in place and static test should help to highlight potential issues. This always depends on weight of
requirement and cost of development but several examples of requirements are more meaningful:

 Possibility to select multiple items across multiple pages: This is not easy to implement due to
the single “form” approach and web container structure of an Intranet application.

 In case of multiple forms entry, the information should be passed through the various forms in
the various steps of the data entry process. Again, this is not easy to implement due to the single
“form” approach. Hidden fields or session cookies are usually used for this purpose.

We don’t say this is impossible to implement but those requirements have their costs and it is our
duty to raise the flag.

2.3. Test Case Design

2.3.1. Introduction

Based on the requirements deliverables, we will explain how to use them and how to approach them
to get the most added value in the Test case definition process.

The purpose of this chapter is not to detail every possible design techniques but to highlight the most
interesting ones applied to this project.

2.3.2. General approach

Today, UML is the leading format of deliverables and it is very important to clearly understand the
subtleties of those documents. We will review the main UML deliverables and how to use them for a
Test design approach.

Deliverable Test case approach

Use case diagram Poor added value for Test case design. It is just a table of content of the application
and it helps only to prepare the list of the main Test scenarios. The actors triggering the
use cases should have a clear definition and must be supported by the application in
terms of rights definition

Use case description By use case description, we mean the text document, and also optional related
documents, such as business rules and screen layouts with data description.

The level of description of a use case can vary widely because one can use this tool to
describe a business process or a class method.

Even if we exclude low level use case descriptions, a use case description is a set of
steps describing the interaction between an actor and the system to be built. A step in
the use case description is either a step in a test case or an expected result, depending
if the step is executed by the actor or by the system.

Sometimes the use case description includes an activity diagram. If the activity diagram
is not included, We strongly recommend drawing one because this will help to identify
Best Test Practices for - Intranet Applications Page 7 / 24§
H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

all possible paths of the use case. This also constitutes a good check on the use case
description itself because it will help identify missing alternative paths.

Business rules: For each business rule defined in the use case description, the rule
should be checked either as an expected result or as a pre-condition of a test case.

Screen layout: This document is used as a part of the general validations of a test
scenario.

Activity diagram Depending at which level they are provided, activity diagrams can be used to describe:

 A use case: This has already been discussed in the previous topic.

 Screen navigation: Used to describe the possible navigation paths in the

application. This is helpful to identify in an exhaustive way, for each screen,
what are the navigation options.

 Process: In this context, we mean the description of business activities that the
application will support. This will help build acceptance scenarios. This topic
will be discussed in chapter 2.8.

State diagram This kind of diagram is used as a cross-check versus test scenarios; each transition
without guards is translated into a Test scenario, and a transition including a guard is
translated into a Test case of a Test scenario. The progression of states may help to
build the acceptance scenarios; see chapter 2.8.

Sequence diagram Here, like the activity diagram, it will depend at which level they are provided. To
make a long story short, the sequence diagrams describing object methods can be
dropped as we are not addressing white box testing.

In some cases, they are useful because they represents:

 Interactions between different systems; used in work-flow or applications

integration context. This will help describe, either the expected results, or the
pre-conditions triggering events in test cases.

 Interactions between actors: Actors seen here more as human actors and used
to describe business activities. This kind of diagram is then used in acceptance
scenarios; see chapter 2.8.

2.3.3. Concurrent access

As the number of users could reach 150, it was more than recommended to test concurrent access. To
prepare our test cases, we decide to build a matrix.

First, we had to identify, on the first axis, the major entities or business objects controlled by the
application. On the second axis, we had to list all end-user functionalities provided by the application.
On each intersection point, we indicated the type of access (Read, Write). Qualified test cases for
concurrent access where combinations of two functionalities with Write access on the same business
object.

The very first solutions proposed by Development to manage concurrent access were rejected:
Development tried to use the default behaviours proposed by Oracle (pessimistic and optimistic mode)
but a home-made had to be set-up to satisfy the following behaviour:

Best Test Practices for - Intranet Applications Page 8 / 24§

H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

 A resource is never locked by anybody,

 If a business object changed between the display of information and the action performed by
the end-user, the end-user is warned that somebody else changed the data and he is able
either to retrieve the latest information or to overwrite it with his own set of data.

 If a business object has been deleted between the display of information and the action
performed by the end-user, the end-user is warned that somebody else deleted the business
object and he can’t perform anything more.

Try to run your concurrent test cases on different physical PC’s, this will respect real situations of
concurrent accesses.

Another situation, maybe uncommon but feasible, is the same user on the same PC and with multiple
IE opened; this situation is also worth testing!

2.4. Quality features

2.4.1. Introduction

Based on the ISO/IEC 9126, we focused our test effort on those quality features

You should always have in mind this rule will follow you thru the whole test lifecycle:

You should always check that a functionality satisfies what it was requested for but also that
the functionality does not do what it was not requested to do.

Example: A requirement states that an email warning should be sent if a task is not completed before
the deadline. Based on that requirement, you should check that the email warning is sent when
needed but also that no warnings are sent in case of completed, closed, or refused task.

2.4.2. Reliability

There are several means to easily put an Intranet application under stress;

 Use Back and Forward keys to see how the application supports them. Even if you hide those
keys within IE, it is always possible to reactivate them and is something rather easy to enable
even for an end-user.

 Proper error management should be put in place to avoid obscure error messages for end-users
such as HTTP 500, 401, 403 …. These must be trapped and translated into a meaningful end
user message to avoid useless questions.

 Avoid the usual or regular context: Try to activate buttons without selecting anything or for
example try to activate the “Update” button without updating anything. This can be very important
because in our case, we had a requirement asking to send an email to all participants in the
verification process in case of update.

 If the application is using cache information, try to build test cases that interact with the cache
content and could create potential issues.

Best Test Practices for - Intranet Applications Page 9 / 24§

H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

2.4.3. Security

Check if you can bypass the identification process by experimenting with the following points:

 Open multiple sessions of the browser by using Ctrl-N (Open in new window); this short-cut
creates a new browser window but shares the session file creates by the first browser window. If
the application must support a time limited connection, it is interesting to test it against “new
browser windows”. If you have any doubt on your session file validity, flush the browser cache
(Ctrl-F5 in IE).

 On the opposite, if you double-click on the browser icon, you should be requested with an
identification process because you have an independent session file.

 Try to open the application by pasting an url to your favourite application web page. Your
application should return an error message stating that the page is not more valid.

 As we are dealing with an Intranet application, normally you are not requested to test the
application against attacks. However, it is maybe interesting to know about several techniques
such as the SQL Injection. It is a technique where you try to input SQL statements thru
alphanumeric fields. Example of a search field: '%JOHN';DELETE * FROM ....'
It has been proved that in many cases hackers are also insiders.

 In case of modal window, try to activate hyperlink, icons within this modal window to see if you
are not able to open a new IE window without authentication.

2.4.4. Accuracy

Here are some tricks that we found very useful and raised an important number of issues in the
application:

 In the fields, especially the alphanumeric one, try to insert spaces at the beginning, in the
middle and at the end. For the beginning and the end, the application should trim the spaces
except if stated clearly in the requirements. The spaces in the middle should be kept.

 Also in the alphanumeric fields, check if the following characters are well stored: the quote
(Alt-039), the percentage (Alt-037), the underscore (Alt-095), the double quote (Alt-034) and
the backslash (Alt-092).
The percentage and the underscore should be checked as well against search fields where
you can get weird results; indeed the underscore and the percentage are joker characters for
an Oracle database. So if you are not dealing with an Oracle database, try to get the joker
characters for your specific database.

 Always validate the correct dependency between fields on the same form if business rules are
set up; also try to change repeatedly the interdependencies on the screen to check if the rules
are reapplied. Indeed, sometimes the rules are correctly set up when you enter the form the
first time but after several changes, it can become inconsistent.

 Check that when you modify a field, the field remains modified even if other options (buttons,
icons,..) are activated on the same form except if specific rules need to be applied.

 Where you are dealing with list of items, try to get empty lists and check how your application
reacts to this unusual situation.
Best Test Practices for - Intranet Applications Page 10 / 24§
H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

 Try to use the application for what it is not intended to do: I don’t mean to cheat via the code
but only with the options provided on screen. Examples: Try to search without any search
criteria; confirm an update action without any updated fields, try to enter a date only when a
date/time is requested

2.4.5. User-friendliness

Even for an intranet it is possible to apply several rules applied to internet web sites:

 Respect the standards in the organization if any.

 Always know where the user is: This topic is valid for all the screens but was extended to the
application itself because it was necessary to add information regarding the current version
under test, and the environment where the test was executed.
 Be consistent in style, colours, fonts, icons (use CSS style sheets)
 Be consistent with labels: A field should always have the same label in all forms, emails,
reports. It should always be translated the same way (if needed). It should always appear on
the same position on each form: I know this is not literally possible but I mean always with the
same group of information logically interrelated. (NDT: by extension a field should have the
same name as the field in the database.... more easy for debug).
 Window and pop-up behaviour: Even if other windows are appearing during the application
execution, it should always be impossible to open new application connections thru those pop-
up windows because, otherwise, the number of Internet Explorer sessions would explode. It is
also necessary to validate that it is impossible to open at the same time, the same pop-up
window multiple times. Those recommendations are made to avoid user confusion with
multiple windows.
 For the different interfaces, use the business words and the words used by the actors
identified in the analysis. Indeed, the same concept is called differently by different actors in
the same organization and even in the same department.
 No horizontal and vertical scrolling at the same time to avoid excessive use of the mouse.
 No more than 10% of screen height for vertical scrolling for the same reason as the previous
point. This means in our case, a maximum of 10 items per page.
 Screen resolution of 1280 X 800 to maximize the area available for information and again to
optimize the use of the mouse.
 When options are proposed to the user, always start on the left with the less used option to
end on the right with the most used.
 Use navigation keys when list of items are displayed; provide sort and search facilities to help
filter the information. Also provide information on how many items were retrieved and which
items the current navigation is showing.
 No animation
 Limit the number of options available per page (maximum 7) to avoid confusion on screen and
avoid manipulation errors by the user.
 Integrated with the office environment: Ability to cut & paste information from and to the
application. Try to cut & paste a word document or to cut & paste a figure from Excel or a
reference from Outlook ( this should be true for any Office application and not just MS
ones).
 Limit the number of clicks to access the information (maximum of 3)

Best Test Practices for - Intranet Applications Page 11 / 24§

H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

 Display always the latest data status: As we are in a multi-user environment and as we are
using a web-application that displays pages which can become obsolete, provide a refresh
option (see 5.2.4 for more details).
 If space is lacking on the screen, it may be possible to use the IE F11 key to increase the
screen region.

2.5. Load testing

2.5.1. Introduction

Load and performance testing was needed to qualify the application for the production environment
deployment. The requirements needed by the production environment were the following:

Domain Limitation

CPU used Maximum of 20 % with peaks of 30% allowed during 10 sec

Memory Depends on the server memory amount but 256 mg seems reasonable; if
extra memory is needed, this generally means that there is a performance
issue.

Threads A maximum of one thread per concurrent user is allowed.

Static page response time Maximum 2 sec.

Dynamic page response time Maximum 10 sec.

External file (pdf, doc,…) Maximum 5 sec

response time

2.5.2. Design

An analysis phase was required to identify the major functions that would require the most from system
resources such as CPU, disk access, database access. The analysis was based on the frequency and
the amount of data manipulated by a function.

For each test case used, use a different set of data; this will avoid non requested caching effects.
Before starting the execution, always flush the memory before starting the load testing phase.

2.5.3. Conclusions

Those are the final results that were able to qualify the application in production. We needed three
rounds to do it. See conclusions for more details.

Domain Actual result

Average response time < 5 sec
CPU used < 30 %
Memory < 230 M

Those are the main adjustments put in place to satisfy the requirements:
Best Test Practices for - Intranet Applications Page 12 / 24§
H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

Issue Solution
Performance issue (CPU > 30 % and Memory <= The PDF file is now prepared when the link is
350 M): Avoid preparing additional information in activated, so only at user request.
advance. In our case, for each item in a list, a
PDF work file was prepared when needed.
Performance issue to display list of results The queries were revisited to produce directly a
list of results from a single query instead of
individual queries producing the list. There was
no need to redesign the database or to de-
normalize the database.
Global performance issue Frequently accessed information was loaded in
cache memory

2.6. Automation testing

2.6.1. Introduction

There was a requirement to use open-source software and according various recommendations, we
decided to install the Fitness automation tool. .

This software is supposed to support user acceptance testing because on one hand, you’ve got a User
defining his own set of test cases with let’s say Microsoft Excel. (NDT: what do you have on the other
hand?) @@@@@@
2.6.2. Purpose(s)

Domains covered Description

Smoke Test A basic set of tests is used to qualify the new application version for testing. The set
mainly covers:
 Check the connection/disconnection
 Check all the actions possible on each screen (activation of buttons, hyperlinks,
icons)
 Check the navigation for each menu option
Regression test The set of previous test cases manually run and representing the previous iterations of
the application; not all manual test cases were translated to the regression set; see
domains not covered.

It does covers all aspects that were very annoying to test manually because ,
depending on the architecture put in place, basic principles of OO such as re-usability
are not always well supported. This means, for example, that all field dates should be
checked for their format on each page.

Domains not Description

covered
Concurrent This is due to a Fitness limitation; it is not possible to run two threads at the same
access test time. With Fitness, you can build test suites or a set of test cases but those are run
successively.

2.6.3. Observations

Regression testing implies the most effort in automation because:

 It needs to include the latest functionalities previously tested manually

Best Test Practices for - Intranet Applications Page 13 / 24§
H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

 Revision is needed in case of new functionalities impacting the running set of test cases.

2.7. Reports

2.7.1. Introduction

We didn’t think it to be to difficult to test reports but basically we had two kinds of reports in the
application:

1. General information: This kind of report doesn't make any calculation and just list information
or a group of information. The report in this case is just another output mode to information
that is already displayed on the screen. The controls to be done are rather limited because, as
said above, it must be a "printed" confirmation of visual information.

2. Statistics: Here, we have two big families of validations to apply: data extracted based on
criteria and calculation resulting from the extraction.

2.7.2. Approach

In our application, a statistic report is made up of three components:

1. A filter screen including almost 10 different criteria.

2. A list of detailed items based on the filter.

3. A summary or a list of total fields based on the details.

The first component was already a difficult matter because some criteria were list of predefined
values and some criteria were exclusive with other ones. To test each criterion individually was not an
issue if you know your data set repository but the main issue was to prepare a set of meaningful filters
based on combinations of criteria with imbedded rules.

There are two mathematic approaches to solve this type of issue:

1. Best pairs: Method that helps you create test cases that pair each value of each of variables
with each value of each other variable at least once. One needs to identify all meaningful
values for each criterion and put them in a flat file. A small utility will read those values and
create sensible test combinations. See point 1.4 for more details.

2. Orthogonal matrix: Almost same approach but with a different tool (Excel spreadsheet). The
idea here is to create a minimum set of test cases based on multiple combinations of
parameters.

To validate the second and the third component, you need rather strong SQL skills to simulate the
extraction and the calculation by your own means.

2.8. User Acceptance testing

2.8.1. Introduction

Best Test Practices for - Intranet Applications Page 14 / 24§

H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

The scope of user acceptance testing is to assess the application quality by end-users thru Business
scenarios. If the quality is acceptable; this is done by evaluating the defined quality requirements
against the application, the application received the sign-off for deployment.

In this case, our role was to prepare and design the business scenarios for end-users; those scenarios
were afterwards validated by the same end-users before actual execution.

2.8.2. Approach

Here under are the techniques that we had to put in place to satisfy this important step because it
was the first real confrontation with the end-users.

 End to end scenarios: Generally speaking, an application represents a business process or a

portion of it and it is included in global picture where you have inward and outward data
flows.
An acceptance scenario should always starts from an initial business trigger that can be
digital or manual.
From that trigger, the story goes on and ends with a final business trigger again either
digital or manual.
To help defining the business scenarios, it is interesting to write a high level Use case with
main and alternate flows supported by an activity diagram. See 2.3.2 for more details.

 Data set preparation: To support the scenarios, data should be prepared. For each
scenario, a different data set should be defined and should be used in the acceptance
scenario description.

 User rights: Define clearly who does what and how the actors involved in the user
acceptance process are defined in the system: which user rights are granted to whom.

 Right Actors: identify the right actors needed to play the different scenarios and verify
that people participating are aware of the real job. Sometimes an actor replaces
somebody not invited or not identified because he is supposed to know the job; this
approach can cost you a big disappointment when the final evaluation of the acceptance is
provided.

2.8.3. Observations

 Don’t forget that when a new functionality is implemented, the set of existing business scenarios
is supposed to be revisited to check if there is an impact.

 The set of business scenarios is also used by the Test team to validate a new iteration of the
application because the regression test set doesn’t have the business view and there is a need
of extra skills to validate the application from and end-user point of view.

 Always invite the right actors when doing your acceptance testing with the end-users: try to make
them play their real role even if they are representing a group of users.

 Lesson learned: We made a big mistake during a user Acceptance workshop: we didn’t identify
correctly a type of actor and this type was included in another one that looked almost the same.
That mistake cost us a lot because we had to rebuild part of the application to support this new
actor.

Best Test Practices for - Intranet Applications Page 15 / 24§

H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

2.9. Issue management

2.9.1. Introduction

There is no good communication between Test and Development without an Issue reporting tool. Again,
we found JIRA in the library of open source tools. An official enterprise licence was then bought. This tool
is now used at department level.

JIRA is used in conjunction with Outlook and is supported by an Oracle database. Users are defined and
belong to groups defining a set of rights. Forms and work-flows can be adapted.

2.9.2. Work-flow

This is the typical work-flow followed for a bug discovered in the application:

1. The bug discoverer (generally a testing member or a developer) inputs the bug details.
The bug is assigned to the development leader.

2. The development leader assigns the bug to a developer.

3. The developer fixed the issue. The issue will be included in a delivery note.

4. When there is a new delivery, one of the activities of the testing team is to browse the
delivery note. Each issue fixed is then checked. According to the test results, either the
issue is definitely closed or it is reopened.

Remark: The end-users are not using the bug tracking application because it is too complex. The
support input the issues for them.

2.9.3. Extensions
JIRA is not only used to input bugs but also for different types of issues:

Issue type Description

Improvement Development needed to fully support the business
Personal Activities Activities related to a personal role within the project.
User support Issues in production that need a follow-up of activities to solve them.

Best Test Practices for - Intranet Applications Page 16 / 24§

H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

3. GLOBAL TESTS RESULTS

3.1. Introduction

Those results represent only bugs found during the various versions before the first deployment in
production; improvements are not taken into account. The application started from version 0.1.0 and
finished at version 0.4.1 before going into production; this means in duration tests covering the period
from the 6/12/2004 to 25/01/2006.

A total of 543 bugs were discovered. Improvements are not included; some could have been part of
the statistics because they cover user-friendliness or functionality qualities but it was too difficult to
sort them out.

The test basis is including the following components:

Component Description
Administration  Management of user rights and group of users
 Management of system parameters
Staff  Management of members information detail
administration  Management of group membership
Reception Management of new requests
Distribution of new requests
Management of on-going requests
Verification Acceptance/refusal of verification tasks
Completion of verification tasks
Take over of verification tasks
Reporting Statistics on requests
Integration Data sent to mail servers
Available views on external databases
Work-flow circuits
LDAP service

3.2. Ressources

Role Mission Number of days

Test manager (full time) 220
 Test strategy and test planning

 Set up the communication tools

 Design the test scenarios and test cases

 Execution of the test cases and bug

reporting

 Reporting to the project leader

 Design of test cases based on test

scenarios

 Execution of the test cases and bug

reporting
1 Tester (1/10 of the 24
time)  Design of test cases based on test
Best Test Practices for - Intranet Applications Page 17 / 24§
H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

scenarios

 Execution of the test cases and bug

reporting
1 Automation Tester 140
 Set up and customization of automation
tools

 Build automated Test suite for smoke Test

 Build automated Test suite for regression

Test

 Control execution of automated Test suite

and bug reporting
Total 384 days

Ratio bug/day 1.41

543 bugs / 384 days

3.3. Statistics

Criticity distribution

250

200

150
Bugs

100

50

0
Blocker Critical Major Minor Trivial
Criticity

Best Test Practices for - Intranet Applications Page 18 / 24§

H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

Time distribution

250

200

150
Bugs

100

50

0
From 1st Qtr 2005 2nd Qtr 2005 3rd Qtr 2005 4th Qtr 2005 Till 25/01/2006
6/12/2004
Time

3.4. Comments

 At the second delivery in test environment, we had to stop the testing process because we had
multiple blocking errors. At that moment, we decided to formalize the smoke test.

 The majority of bugs discovered occurred during the second quarter of 2005 because this
coincided with major functions delivery.

 Developers were more cautious on what they were delivering because they understood the
value of testing; this situation has still to improve, please see the conclusions.

Best Test Practices for - Intranet Applications Page 19 / 24§

H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

4. DEPLOYMENT

4.1. Introduction

After a month and a half of deployment in production, we can already draw conclusions based on the
support calls received during this period. The period starts from the 13 th of February 2006 till the 31 st
March 2006. The total of issues reported during this period is 44.

4.2. By root cause

Root cause analysis

3 1
5 Integration issue
18 Lack of Analysis
Lack of TO Info
Improvement
Security issue
7 Bug
Configuration
7 3

Root cause Number Description

Integration issue 18 Issue related to production environment and external applications
communicating with the TOP application.
Lack of Analysis 3 Point missed by the analysis team during inception.
Lack of TO info 7 Information not provided by key users team.
Improvement 7 Functionality to be added to help user’s work.
Security issue 5 Issue related to the access and rights of the application.
Bug 3 A default in the application not detected by the Test process.
Configuration 1 Issue related to the configuration of user’s platform

Best Test Practices for - Intranet Applications Page 20 / 24§

H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

4.3. By importance

Root cause split by importance

100%
90%
80%
70%
Trivial
Importance

60%
Minor
50%
Major
40%
Critical
30%
20%
10%
0%
t n
e i s fo en ue
g io
su lys in Bu at
n
is a TO ve
m iss ur
io An of ro it y fig
rat of p ur n
te
g ck ck Im
Se
c Co
In La La

Root causes

Value Definition
Critical Issue preventing the normal system operation
Major Issue affecting the information correctness or preventing the execution of
secondary level functionalities.
Minor Issue affecting the normal working procedure but a workaround is available.
Trivial Issue affecting the information or screen layout.

4.4. Comments

 The most important number of issues came from the integration: part of them will be removed by
providing a better protection on external events.

 It was very difficult to simulate the input of external applications because we had no control on
those applications (see 2.2). We had a very big surprise in production: between our application
and another one, there is a queue of messages to communicate; when we started in production,
we had to process a big bunch of messages of that queue to be up to date. The surprise came
from the fact that messages were sent to our application in LIFO mode instead of FIFO, so we
were building information in the opposite order. This point will be address in a next version by
creating a stub or a simulator of external applications. (NDT: exacty what needed to be done in
the first place...)

 The three bugs were all related to the same item; a test case did exist but did not discover these
bugs due to inaccurate data definition.

Best Test Practices for - Intranet Applications Page 21 / 24§

H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

5. CONCLUSIONS

5.1. Introduction

After the first experience of testing within this customer organization, we can now emphasize on
improvements that are either already in place or under sound observation.

According to the experience of this project, if those issues had not been managed or carefully
followed, the application would have encountered more severe failures.

5.2. Improvements

5.2.1. Team

The team was organized with following roles:

 1 Project Leader
 1 Functional Analyst
 2 Developers
 1 Test leader

At the beginning of the project and according to the habits in place at the customer, a Team
dedicated to Testing was something new to the department. After several rounds of delivery/test/bug
reporting, the role of Testing was understood by this department as a necessity and a risk protection.

However, Development is still considering Testing as a watchdog and not as a quality assurance; this
means that the development doesn't try to build quality proactively in its own stage and he is
expecting reactive controls . Changing this attitude is the next challenge for Testing.

Implementation of better quality at design and development level is a company challenge and should
receive high sponsorship, as well from IT as from the Business. Projects will be shorter, cost less, and
be more close to Business requirements, with more easy acceptance from end-users, if the duality and
complementarity of Development/Testing is understood and accepted by all. Testing is not a watch
dog, it is there to help development respect business requirements and insure overall quality. And by
this improve the image of IT toward the Business.

5.2.2. Unit test

According to the analysis of bug reports results, introducing quality at the lowest level is the next
challenge to reduce the number of bugs discovered during the test campaigns. As a starter, the testing
team produced a document explaining several techniques for the implementation of Unit testing.

As this document remained mainly “abstract” and not directly related to the development tools used,
the document was not used. The development team was anxious not to loose time by using those
techniques even if it was several times explained that the time needed to implement unit testing
should be gained by less bug fixing time.

The solution set in place to improve the situation is the hiring of a unit tester well aware of the
development techniques. The improvements expected won’t come immediately because it is difficult
to re-factor was has already been produced but the main challenge will come with the next release
when it will be possible to include quality directly in the code.

5.2.3. Acceptance test

Best Test Practices for - Intranet Applications Page 22 / 24§

H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

At the end of acceptance testing phase, we encountered one major failure. The acceptance phase was
built on three steps that had all to be accepted:

1. Set of Acceptance scenarios: We had played a set of the most representative business
scenarios with a reduced set of actors.

2. Week of training: The 150 final users were trained. The workshops approach was
taken ; in order to given them the ability to have real hands-on on the application
according to pre-defined scenarios. During the workshops, a feedback was collected.

3. Play in pre-production environment: This was the last step before going live. During
this phase, the acceptance process was halted. Mainly for the following reasons:
a) The application was not able to support all the actors in the process.
b) The application was a bit too complex to use; this was already a feedback received
during the workshops but the process was not halted at that time.
c) Not all the cases were totally supported by the application.

So the temporary conclusion was that if the application was not buggy it was nevertheless not
supporting the business entirely as expected. Decision was taken to delay deployment in production.

To be able to deliver, we had to adjust the application content in the following manner :

1. Reduce the scope of the application by eliminating functionalities not fully supporting
the business. Those features will have to be revisited later with a more in depth
analysis. Clearly, the analysis and the end-users missed the point here.

2. Invite the missing actors to re-factor the application remainder and support their
business. Aging, the analysis missed the point by overlooking some actors.

3. Improve the user interface to reduce confusion by revisiting the labels and improve
usability by reducing options and choices.

After those corrections and adjustments, we were definitely on the right track and ready for the
second and final acceptance step.

5.2.4. Application

Those are improvements that we had to implement and that were urgently requested because the
acceptance of the application was in danger.

Issue description Solution Moment of involvement

Reduce the time lag between  A refresh option has been added During the second
the information on screen and in all the screens where the acceptance phase
the current situation in the information was suppose to
database. Due to the Intranet evolve.
technology, there can be a big  A time limit of 30 minutes has
difference between the two been defined after the
situations connection; every attempt after
this limit triggers a reconnection
process.

Emails provided by the Improve the readability of the Early in production

application were bypassed by content; more space between the
specific actors. lines, alignment, use of the
Best Test Practices for - Intranet Applications Page 23 / 24§
H 十七 N 十七十七五 L 41314048.doc
 TECHNOLOGIES DE L'INFORMATION

application colours.

As we can see, the user friendliness of an application can become easily a point of failure. We need to
view the application that we are building as a trade-off between the use of an application by end-
users and the added value that is brought to them. If the application doesn’t help me in my daily work
why do I have to bother using it?

We need to understand that nowadays, end-users have different possibilities or tools to achieve their
business and we need to prove that the solution we are building is better than their previous one. We
are not anymore in those times, were the solution we were providing was the only one and so the best
one.

------------------------------------------- Fin du document --------------------------------------------

Best Test Practices for - Intranet Applications Page 24 / 24§

H 十七 N 十七十七五 L 41314048.doc