Simula Research Laboratory, Technical Report 2012-20 October 2012

Automated Selection of Test Cases using Feature

Model for Product Lines: An Industrial Case Study
Shuai Wang, Arnaud Gotlieb, Shaukat Ali, Marius Liaaen
Certus Software V&V Center CISCO Systems
Simula Research Laboratory Oslo, Norway
Oslo, Norway marliaae@cisco.com
{shuai,arnaud,shaukat}@simula.no

Abstract— Automated test case selection for a new product in a Test case selection is important for product line testing
software product line is challenging due to several reasons. since the number of all possible products derived from the
Firstly, the selection of relevant test cases requires modeling the product line is very huge and it is difficult to obtain a set of
variability within the product line. Secondly, existing test cases relevant test cases for a specific product [2]. Efficient test
needs to be associated to this modeling in order to perform
selection strategies can reduce testing time and at the same time
automated selection. The objective of such an automated selection
process is to increase the test coverage of product functionalities, improve testing efficiency [4]. In recent years, more and more
while decreasing the overall test effort. With this objective in researchers have spent significant efforts on fully automated
mind in the context of an ambitious collaborative project with strategies for test case selection, which have proven to be
Cisco Systems (Norway), we proposed a systematic and efficient as compared to manual strategies [5,6].
automated approach of test case selection for Cisco’s Video In the context of a large initiative, funded by the Research
Conference System (VCS) Product Line. Our approach captures Council of Norway, we created the Certus Software V&V
commonalties and variabilities of the VCS product line using a Center with several industrial and public administration user
Feature Model (FM) and captures the domain knowledge of test partners in 2011. As part of Certus, we started collaborating
experts using a Component Family Model (CFM). Based on these
models, a set of relevant test cases can be automatically selected
with Cisco Systems in Norway on a research project about
when a new product is to be tested. A tool called IPT (Import variability testing. We observed that a current practice for the
Plugin and Transformation) was developed to build our CFM testing of VCSs is to select test cases manually, when a new
automatically and our approach has been evaluated on four product needs to be tested. Due to the increasing complexity of
products of the VCS product line in Cisco. This paper also functionalities and diversity of products, manual selection
reports on this evaluation and presents the findings of a poses several challenges [8]. Firstly, manual test case selection
qualitative study we conducted via carefully designed is a process mainly driven by the expertise of test engineers and,
questionnaire to understand the benefits of our approach in an as such it is not an objective and reproducible process.
industrial testing set-up. Finally, we present initial lessons learnt
Secondly, manual selection may sometimes result in a set of
from our collaboration with Cisco, which can serve as guidelines
for other practitioners. test cases that have low testing functionality coverage, because
a focus has been placed on the testing of specific functionalities
Keywords—Test case selection; Product line; Feature model; Finally, no guideline or methodology is provided to train
Component Family model recently hired engineers to select test cases. This means in
particular that the current practice of test selection is not really
I. INTRODUCTION scalable whenever more and more products come into play.
Software product line engineering (SPLE) is an efficient To cope with the above-mentioned challenges, we
process to represent and customize a large number of similar formalized the process of test case selection and proposed an
products [1]. It is used to achieve a number of goals, such as industry-strength approach supported by a commercial tool to
reducing development time and costs, speeding up product automate this process, which offers the following benefits.
time-to-market, improving the quality of products belonging to Firstly, our approach captures variabilities and commonalities
the product line [2]. of a VCS product line with a Feature Model (FM) [1].
Using SPLE, Video Conference Systems (VCSs), which Secondly, we capture the domain and expert knowledge of test
offer means to provide high-quality face-to-face meetings, can engineers on VCS testing using Component Family Model
be designed and developed very efficiently. Commercial VCSs (CFM), which can represent how products are assembled,
such as the ones developed by Cisco Systems (Norway), need generated or tested in a product line [9]. These two models are
to be tested in a thorough way since they provide many high- then linked together, in order to automate the selection of test
quality services such as call establishment, high-resolution cases when a new product needs to be tested. At the same time,
video and convenient communication control [3]. test cases change very frequently and it is not practically
possible to build CFM for test structure manually. Hence, we
Simula Research Laboratory, Technical Report 2012-20 October 2012
also developed a tool named IPT (Import Plugin and two kinds of such constraints, namely the require and the
Transformation, an extension of Pure:: Variants [9,10]) to cope mutually exclusive constraints. A require relation among two
with this problem. Thirdly, we provide a complete and scalable features (a source and a target) means that if the source feature
process to systematically select test cases in order to help new is included into the current selection, then the targeted feature
engineers to develop test scripts for VCS. Finally, our process must also be included. A mutually exclusive relation has the
is automated hence it reduces the amount of time and effort opposite meaning, saying that if a feature is included then
required for test case selection. another feature cannot be included in a given configuration [8].
To sum up, the paper describes: 2) Component Family Model (CFM)
• Our collaboration process with Cisco; A CFM is used to represent how products are assembled
• A systematic and automated approach for test case and generated in a PL by modeling relations among software
selection; architectural elements. It has a hierarchical structure including
• Our implementation of automated test case selection items such as components and parts. Because of automatic
for the VCS PL; product generation from a valid selected feature model in PL,
• A quantitative and qualitative evaluation of the these items can be organized and used with relevant
proposed approach; information about the concrete architecture [9].
• A set of experiences and lessons learnt about our CFM can be represented as a 4-tuple (components, parts,
industrial collaboration. source elements, restrictions). Components are named entities
The rest of the paper is structured as follows: Section II organized into a tree-like structure that can be of any depth.
introduces relevant background of Feature Model (FM) and Each component represents one or more functional elements of
Component Family Model (CFM). Section III describes the the products in PL (e.g. C functions, Java classes or
automated process of selection including problem refinement, documentation files). Parts are named and typed entities. Each
solution realization, and test case selection. In particular, the part belongs to a component and contains one or more source
IPT tool is detailed in this section. Section IV presents elements. A part can be associated with given programming
quantitative and qualitative evaluations and relevant analysis language features, classes or objects, but it can also be
based on our proposed approach. Section V introduces a set of associated with other key elements. A source element is an
experiences and lessons learnt through our collaboration with unnamed but typed entity. Source elements are usually used to
determine how the source code for the specified element is
Cisco. Related work is presented in Section VI, while section
generated. Restrictions specify conditions under which a
VII concludes the paper.
component, part or source element may be excluded from a
II. BACKGROUND ABOUT FEATURE MODEL AND COMPONENT final selection [9,10].
FAMILY MODEL
III. PROPOSED AUTOMATED PROCESS
In this section, we briefly introduce some background about
This section presents the main steps of our automated test
Feature Model (FM) and Component Family Model (CFM).
case selection process as shown in Fig. 1. These steps include:
1) Feature Model (FM)
problem refinement, solution realization, and test selection.
Feature modeling is a hierarchical modeling approach for
A. Problem Refinement
capturing commonalities and variabilities in PL [11]. It can be
Activity A1 as shown in Fig. 1 describes how we identified
used to document similarities and differences among a large
an interesting research problem at Cisco and investigated a
number of products. Depending of the context, a feature in FM
suitable solution through our collaboration.
can be a requirement, a technical function, or a configuration
option. Since FM has a hierarchical structure, complete 1) Detailed Problem Clarification
configurations from a PL can be extracted from a FM through Test case selection of VCS PL is currently performed
feature selection [12]. manually during test plan design and test execution scheduling.
FM can be represented as a triple (features, relations, Through several meetings with our partner, we conducted a
constraints) with four different types of relations among careful domain analysis of VCSs, and investigated how VCSs
features, namely mandatory, optional, alternative and or. A are implemented. Based on this analysis, we understood that
mandatory relation between two features specifies that, if the the main question raised by our partner was “How to improve
father feature is included in the current selection, then the child the current test selection process for newly designed VCS
feature must also be included. An optional relation is also used products?” To respond to this question, we investigated the
between two features specifies that, if the father feature is problem in detail as activity A1.1 shown in Fig. 1. We arranged
included in the current selection, then selection of the child six meetings with Cisco’s test engineers with the objective to
understand the structure of the test case repository, composed
feature is optional. An alternative relation between a feature
of more than 2000 test cases. When a new product needs to be
and a set of children features is used when the selection of only
tested, test engineers manually select test cases from the test
one of the children is required, not less, not more. An or
repository, which is a tedious and time-consuming task. We
relation is used when at least one feature must be selected. In
addition, FM contains cross-tree constraints which are identified that more than half one hour was required in average
supplementary relations among unrelated features. There are to perform this selection for a single product.
Simula Research Laboratory, Technical Report 2012-20 October 2012

Fig. 1. Activities of proposed automated process

Some of the researchers involved in this collaboration Each testing functionality is associated with a feature in our
worked very closely (i.e., on a day-to-day basis) with the FM_T, represented and organized as nodes of tree, such as
testing team at Cisco for two weeks. The goal was to benefit “presentation” used to test the quality of presentations during
from the experience of test engineers to understand how to conference meetings, “network” used to test the quality of
perform test case selection from the test repository. We also network during conference meetings. The variants of the VCS
took this experience as an opportunity to understand the real PL can be described by performing different selections on the
test selection challenge to be solved. features of FM_T.
2) Proposed Solution Since there were no specification or description documents
To solve the above-mentioned problem, activity A1.2 led us to explain what should be a feature or how the FM_T should be
to introduce a solution based on Feature Model (FM) [1] and organized. So at first, there were five meetings arranged to
Component Family Model (CFM) [9,10]. As described in the discuss about the structure of FM_T, especially the meaning of
background section (Section II), FM provides a good features in the VCS domain. Different people have different
hierarchical structure and sufficient notations to capture the perspectives about the features and FM_T. For instance,
commonalities and variabilities for VCS PL. Meanwhile, hardware platforms were considered as features initially
according to our knowledge, CFM can be adapted to represent because test cases needed to know in which platform they
the overall structure of test cases and the notations of CFM are would be executed. These platforms were represented as
sufficient to describe the test structure of VCSs. In addition, features in the first version of FM_T. However, as discussions
restrictions can associate test cases with testing functionalities went further, hardware platforms became not suitable to be
of the VCS PL. Therefore, we proposed an automated process considered as features because VCS testing just focused on
of test cases using FM and CFM. software systems and it was always difficult to distinguish
We designed the Cisco VCS PL using a FM. Since our hardware platforms and software products. So we removed
context is VCS PL testing, we will call our FM as FM for hardware features from the second version of FM_T. Until now,
testing---FM_T in the rest of the paper. Meanwhile, CFM is not we have already had three versions of FM_T.
used to represent software architecture (a commonly used The testing of a VCS product requires the following
purpose of CFM), but rather to model the hierarchical structure information:
of test cases. Since our context is VCS testing, we will refer • Testing states such as “Ready” and “Standby”. The
CFM as CFM for testing---CFM_T. “Ready” state tells that a system is ready to be tested
B. Solution Realization and “Standby” describes that the system needs some
Activity A2 as shown in Fig. 1 is about solution realization. conditions or operations to wake up and transit into the
We used a FM_T to capture the commonalities and variabilities “Ready” state;
for the VCS PL and a CFM_T to model the overall structure of • Testing functionalities such as “Presentation” and
test cases. Our FM_T and CFM_T were built within “Phonebook”;
Pure::Variants (P::V) [11], a software modeling tool that allows • Testing parameters such as “Video Protocols”, “Audio
users to design feature model, component family model and Protocols”.
restrictions.
1) Feature Model for Testing (FM_T)
In our work, based on the implementation of VCSs and
domain expertise knowledge, we started with modeling eleven
distinct VCS products from Cisco.
Simula Research Laboratory, Technical Report 2012-20 October 2012
In order to meet the VCS testing domain, our FM_T means to limit and control the number of potential products
extends the meaning of feature model and represents testing represented by the FM_T.
states, testing functionalities and testing parameters as features, The modeling part of the VCS PL is thus currently
but different dimensions of testing features, which describe undertaken with the collaboration of Cisco test engineers. In
testing states of VCSs, functionalities needed to be tested and total, our FM_T contains 114 features (54 mandatory features
parameters needed to be configured for VCS testing. and 60 optional features) and 35 require relations. Besides,
Figure 2 shows the highest level of the latest version of according to the VCS expertise knowledge, we need to
FM_T where three essential VCS features are represented, mention that the FM_T for the VCS PL stays mostly stable
namely “Testing Sates” for testing states, “Testing Features” and thus is one time manual effort.
for testing functionalities and “Testing Parameters” for testing 2) Component Family Model for Testing (CFM_T)
parameters (Exclamation marks represent mandatory features). Test plans are usually composed of many test cases and
When performing feature selection for a VCS product, firstly, test engineers spend significant time organizing test cases
testing state of the VCS product needs to be identified by within these plans. In order to model the structure of test cases
selecting relevant features inside “States”. Secondly, selection and to automatically obtain relevant test cases for test plans,
of testing functionalities is performed inside “Testing Features” we proposed to capture the overall structure of test cases with
to describe which functionalities will be tested for the VCS a CFM_T.
product. At last, all needed test parameters can be selected First of all, we need to define what the test structure is in
inside “Testing Parameters” based on the testing requirements the context of VCS PL. Based on our experience and
of the VCS product. collaboration with Cisco, we found that the test structure in
VCS testing is composed by test suites, test tasks, and test
cases. A test suite is a collection of test tasks that belong
logically together such as ”Audio” and ”Video”. A test task is
a collection of test cases that has a common test resource
requirement such as ”Image Layouts” and ”Video controller
functionality” under ”Video”. Each test case is a test script
with a set of parameters, which can be run on different VCSs.
Our CFM_T is represented as a triple (components, parts,
Fig. 2. FM_T of the VCS PL (high-level) restrictions). A component represents a test suite or a test task
Fig. 3 shows a part of FM_T related with “Testing which can be regarded as a named set of test cases and it is
Features” (Question marks present optional feature). From Fig. hierarchically decomposed into further components or into
3, we can see that there is a set of mandatory or optional parts. A part represents a test case, which belongs to a test
testing functionalities in the VCS testing. For instance, there is task. A restriction constrains a relation between a component
mandatory relation between “Testing Features” and“Video” or a part in CFM_T and a feature in FM_T. A component or a
part is part of the resulting selection iff the relevant feature is
because any VCS product must contain “Video” functionality.
included in the set of feature selection. Restrictions play a key
On the contrary, there is an optional relation between “Testing
role that can be used to link FM_T and CFM_T. Using
Features” and “Phonebook” because a VCS product may
restrictions allows us to establish links between the structure
choose to contain the optional functionality “Phonebook”
of the test cases and the VCS PL.
according to different individual preferences of requirements.
Fig. 4 represents the highest level of the CFM_T where
Meanwhile, a set of cross-tree constraints are added into
each component is represented by its name, while Fig. 5
the FM_T since there are a set of internal relations among
represents the subtree of the ”Video” component. These names
specific functionalities. For instance, as shown in Fig. 3, the
have usually been selected based on the functionalities of
“CameraControl” feature requires the “Cameras” feature since
VCSs to be tested. From Fig. 4 and Fig. 5, we can see that a
camera controlling only makes sense when a VCS product
large number of test cases can be categorized according to
includes one or more cameras. These cross-tree relations are to
different test suites and test tasks.
be determined by the VCS domain knowledge and VCSs
In the CFM_T, each component or part can have any
information, which can be used to avoid invalid description of
number of restrictions. A component or part is contained in the
products. Note that cross-tree constraints are a convenient

Fig. 3. Graphical structure of the feature "Testing Features"

Simula Research Laboratory, Technical Report 2012-20 October 2012

Fig. 4. CFM_T for the test cases (high-level)

final selection iff its parent is included and its restrictions Resolution/Framerate" (test id: 1266) includes two tags,
evaluate to true. We used the restriction hasfeature() in P::V to which are “video” and “resolution”. This means that the test
link CFM_T and FM_T. For instance, we assigned the case is related with testing functionalities “Video” and
restriction hasfeature(‘Video’) to the part (‘SIP: “Resolution”, which also exists as features in FM_T. Using
Resolution/Framerate’) because test case “SIP: tags, restrictions between FM_T and CFM_T can be created
Resolution/Framerate” is related with the testing functionality efficiently. In general, a CFM_T with restrictions can be built
“Video”. It means during automated test case selection, a test based on the xml file. To support this, a tool called IPT was
case “SIP: Resolution/Framerate” can be included into the developed based on P::V (an extension of Pure::Variants).
final selection of test cases iff the feature “Video” is in the The architecture of the IPT is shown as Fig. 6. IPT
selection set of features. According to thorough discussions includes two functionality components, namely Import Plugin
with Cisco test engineers, a large number of restrictions (more and Transformation. There are five sub components within the
than 2000) are added into CFM_T. Import Plugin：
• FM_T Parser: This component parses FM_T to create a
map list between the names of features and the names of
tags since the names of tags and features should match to
build restrictions when importing the xml file;
• Test Structure Parser: This component parses the xml file
from test database to obtain all elements for building
CFM_T such as components and parts. Meanwhile, tags
information will also be created in order to build
restrictions;
• CFM_T Builder: This component builds the CFM_T
including components, parts and the structure of CFM_T
Fig. 5. Graphical structure of "Video" (part of it) using the elements from Test Structure Parser;
• Restriction Builder: This component builds the
However, test cases in Cisco’s test database change very restrictions for all test cases using tags information from
frequently. Manually building and updating CFM could be Test Structure Parser and the map list from FM_T Parser;
very time-consuming. To address this, we developed a tool
• CFM_T Assembler: This component assembles
called IPT to automatically build CFM_T with restrictions.
components, parts and restrictions for CFM_T from
a) IPT
CFM_T Builder and Restrictions Builder to obtain a
In Cisco, test cases can be exported from the test database
complete CFM_T.
as an xml file. The file can be put into test execution tool and
The second part of IPT is the Variant Transformation.
relevant test cases from the database are executed. All
When there is a new VCS product that needs to be tested, a set
necessary information for executing test cases is included in
of relevant test cases can be obtained in the CFM_T after
the file, such as all needed resource information (e.g.,
performing the selection of features in the FM_T using
hardware ID and software ID), test ID, max run time, test and
Pure::Variants. The Variant Transformation of IPT assist in
test script’s information. The most relevant information
generating an xml file including all selected test cases, which
included in this file is Tags, which is used to identify testing
can be used for further execution.
functionalities of test cases. For instance, the test case "SIP:

Fig. 6. Architecture of IPT

Simula Research Laboratory, Technical Report 2012-20 October 2012
C. Test Case Selection FM_T and CFM_T;
Activity A3 as shown in Fig. 1 presents test case selection 2) Number of People Required: Our proposed process
process when a new VCS product needs to be tested. requires only one person to perform the selection less
• Firstly, as activity A3.1 shown, based on the expertise than the manual process (at least two persons);
knowledge and system information, test engineers 3) Feature Coverage: Our automated process can cover
analyze the testing requirements and obtain a set of all selected features as long as there are test cases
features which can represent the testing requirements; related with the selected feature in the test database,
• Secondly, as activity A3.2 shown, test engineers which has more feature coverage than the current
perform the selection of obtained features and a set of manual selection process.
corresponding test cases will be selected in the CFM_T
B. Qualitative Validation for the Automated Process
automatically;
• Finally, as Activity A3.3 shown, test engineers To get initial feedback from Cisco, a detailed questionnaire
transform the selected test cases into an xml file, which was designed to solicit the views of those who were directly
includes all necessary information for further execution. involved in our proposed process. The objective of this
questionnaire was to assess the proposed automated approach
IV. EVALUATION ABOUT THE AUTOMATED PROCESS of test case selection for VCS PL in addition to collecting
This work was driven by needs of our industrial partner--- views about the current process.
Cisco with the goal to elaborate a practical process that assists This questionnaire is divided into six parts, which are
test engineers in test case selection. In this section, we related with job responsibility, working experience, views on
performed quantitative and qualitative evaluations to validate the current manual selection process, views on feature model
our automated. for testing (FM_T), views on component family model for
testing (CFM_T) and overall experiences on the proposed
A. Quantitative Validation for the Automated Process automated process. There are four main people (4 out of 7
The proposed automated process was tested in four existing number of testing team) involved this work, who are either
VCS products, namely, C20, C40, C60 and C90. Through quality assurance (QA) managers or test engineers. Therefore,
analysis of systems information and discussions with test we distributed four copies of questionnaire and got feedback
engineers, we chose a set of features related with each VCS from all of them (The detailed questionnaire is shown in
product. Then, we computed the number of test cases, how Appendix A and all results for the questionnaire are shown in
much time needed for each product and how many features Appendix B).
covered by the selected test cases, which are shown in Table 1. Firstly, a set of basic relevant information about involved
However, from Cisco’s side, it is difficult to obtain such people is collected and all of the basic information shows that
quantitative data about the current manual process since the the involved people have significant experience in VCS testing
data is not stored. But through discussions with the test and their feedback provide a valuable view on the proposed
engineers, a rough estimated qualitative data was obtained, process. In the following sections, summarized results and
which is shown as below. analysis are presented.
• More than half an hour is used for each manual 1) Current Manual Selection Process
selection of a VCS product; For all respondents, results can be summarized as below.
• At least two persons are required to perform manual • 2 out of 4 strongly agree and 1 out of 4 agree that the
selection of a VCS product; current manual selection process is time-consuming;
• Manual selection of test cases does not cover all • 3 out of 4 strongly agree and 1 out of 4 agree that the
needed test features. current manual selection process is error-prone;
Therefore, a set of quantitative data can be summarized • 2 out of 4 strongly agree and 2 out of 4 agree that the
based on the above qualitative data, which is also shown as current manual selection process largely depends on
Table 1. From Table 1, we concluded that: the expertise and domain knowledge;
1) Time: Using our process and the tool, the time used • 2 out of 4 strongly agree and 2 out of 4 agree that the
for test cases selection is reduced on average of 65%. current manual selection process can not cover all
Most of the time required for our process is to analyze testing requirements;
the product and obtain a set of features after building

Product #Features # Test Time Time # Persons # Persons Feature Feature

cases (Manual) (Now) (Manual) (Now) coverage coverage
(Manual) (now)

C20 17 138 >=30mins 7mins >=2 1 <100% 100%

C40 25 167 >=30mins 9.5mins >=2 1 <100% 100%
C60 32 192 >=30mins 11mins >=2 1 <100% 100%
C90 43 239 >=30mins 14mins >=2 1 <100% 100%

Table 1. Comparisons between the current process and our proposed process
Simula Research Laboratory, Technical Report 2012-20 October 2012
• All 4 respondents strongly agree that the current from the test database for representing the overall
manual selection process result in redundant test cases. structure of test cases;
Based on these results, we concluded that: • 2 out of 4 strongly agree, 1 out of 4 agree and 1 out of
4 have no opinion that it is easy to obtain and add tags
• The current manual selection of test cases for a information into the XML file for representing the
specific VCS product is time-consuming and error-
overall structure of test cases.
prone;
Based on these results, we concluded that:
• VCS expertise and domain knowledge is largely • The notations of CFM_T for the structure of test cases
required when performing the manual selection of test are not difficult to understand in the VCS testing.
cases. Different test engineers might get a different set However, more evaluations and improvements are still
of selected test cases; required;
• It is agreed that the selection of test cases manually do • In the VCS PL testing, CFM_T is sufficient to
not have a good capability to cover all testing represent the overall structure of test cases;
requirements; • It is easy to obtain the xml file, which is used to
• In particular, manual selection of test cases results in a represent test cases from the test database. Meanwhile,
large number of redundant test cases, which requires it does not require a lot of effort to obtain and add tags
more time for scheduling and execution. information into the xml file. So using the file and IPT
2) Our Proposed Process tool, CFM_T with restrictions can be built efficiently
In the section below, we provide results of the since required effort of building CFM_T is only to
questionnaire related to our proposed process. describe test cases and relations with the features in
a) Feature Model for Testing (FM_T) FM_T.
For all respondents, results can be summarized as below: 3) Overall Experience
• All 4 respondents agree that it is easy to understand the For involved respondents, 1 out of 4 strongly agree and 3
notations of FM_T; out of 4 agree that the proposed automated test case selection
• 3 out of 4 agree and 1 out of 4 have no opinion that process can improve the current test case selection process for
FM_T is sufficient to represent all functionalities of the VCS PL in the future. That means test engineers in Cisco
VCS PL; believe that our proposed process can bring practical
• 2 out of 4 agree and 2 out of 4 have no opinion that it is improvements for test selection in VCS testing as compared to
easy to understand and use the P::V tool for building the current manual process of selection.
FM_T;
V. EXPERIENCE AND LESSONS LEARNT
• 3 out of 4 agree and 1 out of 4 have no opinion that it is
easy to build and revise a FM_T for the VCS PL; This section presents our experience and lessons learnt.
• 2 out of 4 strongly agree, 1 out of 4 agree and 1 out of Firstly, we present our experience related to domain and
4 have no opinion that the functionalities of the VCS problem understanding. Secondly, we present our experience
PL do not change significantly and FM_T does not related to the selected solution and its realization. Thirdly, our
need to be changed frequently once it is built. experiences related to the evaluation of our approach are
Based on these results, we concluded that: explained and lastly, overall experience of our collaboration
• From industrial perspective, it is not difficult to with Cisco is presented.
understand the notations of FM_T; 1) Domain and Problem Understanding
• It is agreed that FM_T is sufficient to represent all Initially, several meetings were arranged with the testing
testing functionalities for the VCS PL; group of various VCS products to get familiarized with the
• It does not require a lot of effort to build FM_T for the VCS domain. On average, three test engineers from different
VCS PL using P::V (Only two people are involved sub-groups were invited in the meetings to get different
into the process of building FM_T); perspectives on the domain and testing problems. During the
first meeting, we learnt that the first testing problem was
• The testing functionalities of the VCS PL stays stable
manual selection of test cases.
and do not change frequently. So FM_T can be built
To get acquainted with the domain, current testing process,
manually and it does not require too many efforts for
tools and technologies, first author spent around two weeks in
test engineers to revise FM_T.
Cisco. During this period, several meetings were conducted
b) Component Family Model for Testing (CFM_T)
with different test engineers. Each meeting lasted on average
For all respondents, results can be summarized as below:
for one hour. In addition, the first author got the opportunity to
• 3 out of 4 agree and 1 out of 4 disagree that it is easy to get hands on experience with the current testing process.
understand the notations of CFM_T; Based on our experience in this step, we concluded that:
• 1 out of 4 strongly agree and 3 out of 4 agree that • Better understanding of specific domain knowledge can
CFM_T is sufficient to represent the test case structure; make a great contribution on clarifying an industrial
• 2 out of 4 strongly agree, 1 out of 4 agree and 1 out of problem;
4 have no opinion that it is easy to obtain the XML file
Simula Research Laboratory, Technical Report 2012-20 October 2012
• Being involved into the industrial process helps a lot to have more capabilities for different detailed problems
understand a problem and seeking potential solutions existing in industry;
for the problem.
2) Solution Clarification and Realization 3) Evaluation of the Solution
After clarifying the problem, the next step is to find a In this sub section, we summarize some experiences during
suitable solution and realize the solution. These steps pose the process of evaluation.
several challenges. After the realization of solution, a set of evaluations was
First challenge was that we needed to find a good way to performed. However, in our case, it is difficult to obtain
propose our selected solution to the test engineers of Cisco. enough quantitative data from Cisco for the current manual
Finding a suitable solution is not a difficult task, but selection. For instance, we did not know how much time was
presenting it to the industrial people, and agreeing on an taken, how many people were involved for manual selection,
optimal solution is difficult since people have different and how many test cases were selected for a specific product.
opinions about solutions than the ones originated from Through three meetings with Cisco, we learnt that some
research. For instance, the Cisco’s test engineers had difficult qualitative data could be captured, such as more than half an
to adapting to FM. But FM has proven to be a good modeling hour for each testing time of the manual selection and at least
approach to capture commonalities and variabilities in product two persons involved for each selection process.
lines as reported in many papers [1,12,13]. Meanwhile, in our automated process, we can record the
To deal with this, we explained the notations of FM using time and the number of test cases quantitatively based on a
the concepts from their domain and built a demo FM for the specific product. As the first part of Section IV shown,
VCS PL to show that VCS PL can be represented in a very through comparison and analysis, we find that it is still
clear hierarchy and different VCS products can be described reasonable to perform the evaluation because there are
by a set of features using FM. Meanwhile, using constraints in significant differences between the quantitative data from our
FM, invalid description for VCSs can be avoided. proposed process and qualitative data from the manual process.
Doing so increased the interest of industrial people. Similar Based on our experience, we learnt that:
process was adapted to CFM. CFM can be adapted for • It is always difficult to get enough quantitative data
modeling test structure in a good hierarchy and restrictions from industrial partner to perform evaluation;
assigned in CFM can link FM and CFM, which associate test • It is reasonable to perform evaluation using a set of
cases with features in FM_T efficiently. quantitative and qualitative data if the differences
Based on our experience we concluded that: among these data are significant such as the difference
• Associating specific domain concepts with potential between less than 20mins for the quantitative data in
solutions can help industrial partner to promote the our proposed process and more than 30mins for the
understanding of the solution. qualitative data in the current manual process (at least
After the clarification of the solution, the solution was one-third time is reduced).
realized. When building FM_T for the whole VCS product It is well known that it is difficult to perform evaluation
line, we did not have any specifications or description about proposed process before putting into a long-term use.
documents. What we had were only system information for However, some particular evaluations can still be performed,
distinct products and expertise knowledge of test engineers. such as interview and questionnaire. In our case, we use
To address this challenge, we arranged seven meetings and questionnaire to perform partial evaluation. Some lessons
each one lasted 1.5 hours in order to discuss the clear structure learnt from questionnaire are as below.
and notations of FM in the domain of VCS. During the • Before designing questionnaire, a clear objective needs
meetings, several test engineers explained what the features to be made;
should be and how the FM should be organized from their • It is better to make short questions and provide 3-4
perspectives. We recorded all the information we could gain options to choose and it is more understandable to add
from the meetings. The common views were handled easily, some descriptions in the beginning and divide
but it was difficult to deal with different views about the FM. questionnaire into separate parts;
After several discussions, we handled different views by • Before giving the questionnaire to relevant people, it is
extending the meaning of FM. For conflicting views, all also better to hold a meeting to explain additional
people presented their evidence to see whether an agreement details about the questionnaire for an overall
could be made. For views could not be agreed on, we recorded understanding on the objective of the questionnaire and
them (with detailed description) in a paper for the future how to perform it.
discussions. 4) Overall Experiences and Lessons Learnt
Similar principle can be adapted to CFM of the overall test In this sub section, we concluded a set of learnt lessons
structure. from the overall process based on our experience of
Based on our experience, we learnt: collaboration with Cisco. These are discussed as follows:
• Extending the existing modeling approaches (instead of • Automation is the key to successful industrial adoption;
developing an approach from the scratch) such as • People from industry are more inclined towards
feature model can meet more specific requirements and commercial tools than open source tools. Main reasons
Simula Research Laboratory, Technical Report 2012-20 October 2012
include: high usability, better stability and reliability, VII.CONCLUSION AND FUTURE WORK
and more technical support of commercial tools; In this paper, we proposed a systematic and automated
• A good user guide and a clear demonstration process approach for test case selection in the context of Cisco’s Video
are necessary when introducing a tool to industrial Conferencing Systems (VCS) Product Line (PL). Our approach
partner for their good understanding. captures commonalities and variabilities of the VCS PL using
Feature Model for Testing (FM_T) and domain knowledge and
VI. RELATED WORK
expertise of test engineers about test cases in Component
Software product line testing is a relatively new, but intense Family Model for Testing (CFM_T). Moreover, based on the
field of research since product line engineering has shown proposed approach, an assisting tool called IPT was developed
significant benefits in both academia and industry [2,4,15,25]. to automatically build CFM_T for test cases.
In [2], McGregor presented a set of activities, which can be We evaluated our work using both quantitative and
used to address testing individual assets (e.g., verification of qualitative analysis. Through quantitative evaluation, results
consistency between requirements and specifications) and showed that our proposed approach could reduce the selection
testing artifacts (e.g., test-case derivation and test suite design) time and effort, and increase the coverage of testing
that represent complete products in the context of product line functionalities as compared to the existing manual process.
[4]. In [7], based on a relational model capturing variability in Through qualitative analysis, to assess the applicability of the
product line, Cohen defined a family of cumulative coverage approach in the industry, we collected subjective opinions of
criteria to collect test coverage information, which was also test engineers at Cisco via carefully designed questionnaire.
used to map with existing combinatorial testing approaches for Results confirmed the mentioned challenges about the current
supporting product line testing. In [6], Nebut presented to manual selection of test cases in Cisco and showed that the test
describe functional variation points for product line at engineers are positive about the approach and believe that the
requirements level using extended UML user case notations to approach can bring significant practical advantages in the
automatically generate behaviors for a specific product for test future.
case generation. In [15], Muccine proposed associating Our paper also shared a set of initial experiences and
regression testing with product line by comparing code lessons learnt through our industrial collaboration with Cisco.
execution with the architectural design, which can be used as We hope that all these experiences and lessons can give a
guidelines for adapting existing techniques for regression meaningful guide for other researchers who want to adapt FM
testing to product line testing. However, most of these works and CFM in product line testing, especially for test selection.
are only proposals and only provide guidelines without In the future, we plan to conduct more experiments on other
specific testing process, which are not systematic. VCS products. We also want to extend our work to implement
Comparing our work with regression testing, regression test sophisticated test minimization techniques to even reduce the
selection aims at identifying a set of relevant test cases when number of selected test cases, while achieving high coverage
changes are made to existing software [7]. Various types of such as pairwise coverage of testing features. We also plan to
such techniques are proposed in the literature, but mostly link FM_T with behavior models (e.g., UML state machines) to
around the following two aspects: selection based on code generate new test cases that have high fault detection.
changes [16,17,18] and selection based on specification
changes [19,20]. In addition, several thorough survey papers REFERENCES
have been published in the literature [5,21]. Although [1] D. Benavides, S. Segura and A. R. Cortés, “Automated analysis
techniques for regression test selection have been evaluated in of feature models 20 years later: A literature review,”
many previous works [22,23,24], there is no enough evidence Information Systems (35), pp. 615–636, 2010.
to prove that these techniques still work well if being adapted [2] J. D. McGregor, “Testing a Software Product Line,” Technical
in the context of product line. In comparison, our work focuses Report, CMU/SEI-2001-TR-022, ESC-TR-2001-022, 2001.
on test selection for product lines and existing test cases, [3] Cisco Systems, Cisco telepresence codec c90, Data sheet,
regression test selection aims at seeking a selection solution available from http://www.cisco.com. 2010.
for existing software systems, which are subject to change. In [4] E. Engström, “Regression Test Selection and Product Line
our process, effort is spent on building reasonable models for System Testing,” Proc. Third International Conference on
product line and the structure of test cases, and making links Software Testing, Verification and Validation (ICST), pp. 512-
between them. In contrast to regression testing, where focus is 515, 2010.
on testing changed functionality of an existing software [5] E. Engström, P. Runeson and M. Skoglund, “A systematic
system, our work is applicable when a new product is to be review on regression test selection techniques,” Information and
tested. Software Technology (IST), 52(1), 2010.
Unfortunately, according our knowledge, we did not find [6] C. Nebut, Y. Le Traon and J. M. Jézéquel, “System Testing of
any relevant literatures about adapting FM and CFM in Product Lines: From Requirements to Test Cases,” in Software
product line testing, especially for test selection. Product Lines, Research Issues in Engineering and Management
pp. 447–477, Springer 2006.
[7] M.B. Cohen, M. B. Dwyer and J. Shi, “Coverage and Adequacy
in Software Product Line Testing,” Proc. ACM ISSTA
Simula Research Laboratory, Technical Report 2012-20 October 2012
Workshop on Role of Software Architecture for Testing and [17] J. Hartmann and D.J. Robson, “Techniques for selective
Analysis (ROSATEA 06), pp. 53-63, New York, 2006. revalidation,” IEEE Software 7(1), pp. 31–36, 1990.
[8] S. Wang, A. Gotlieb, M. Liaaen and L. Briand, “Automatic [18] M.J. Harrold and M.L. Souffa, “An incremental approach to unit
selection of test execution plans from a Video Conference testing during maintenance,” Proc. IEEE International
System Product Line,” Proc. ACM MODELS Workshop Conference on Software Maintenance (ICSM), pp. 362–367,
VARiability for You (VARY’ 12), pp. 30-35, Australia, 2012. 1988.
[9] Pure systems GmbH, “Variant management with pure::variants,” [19] A. Orso, M.J. Harrold, D. Rosenblum, G. Rothermel, M.L. Soffa
technical white paper, available from http://web.pure- and H. Do, “Using component metacontent to support the
systems.com, 2006. regression testing of component- based software,” Proc. IEEE
[10] Pure systems GmbH, “Pure::Variants User’s Guide,” available International Conference on Software Maintenance (ICSM), pp.
from http://web.pure-systems.com, 2011. 716–725, 2001.
[11] J. White, D. Benavides, D. C. Schmidt, P. Trinidad, B. [20] C. Yanping, L.P. Robert and D.P. Sims, “Specification-based
Dougherty and A. Ruiz-Cortés, “Automated diagnosis of feature regression test selection with risk analysis,” Proc. Conference of
model configurations,” Joural of Systems and Software (JSS), the Centre for Advanced Studies on Collaborative research, IBM
83(7), pp. 1094–1107, 2010. Press, 2002.
[12] K. Czarnecki, C. Kim and K. Kalleberg, “Feature models are [21] S.Yoo and M.Harman, “Regression testing minimization,
views on ontologies,” Proc. 10th International Software Product selection and prioritization: a survey.” Software: Testing,
Line Conference (SPLC), pp. 41–51, August 2006. Verification and Reliability 22(2), pp. 67-120, 2012.
[13] D. Beuche, H. Papa jewski and W. Schröder-Preikschat; [22] J. Bible, G. Rothermel and D.S. Rosenblum, “A comparative
“Variability management with feature models,” Science of study of coarse- and fine- grained safe regression test-selection
Computer Programming 53(3), pp. 333–352, 2004. techniques,” ACM Transactions on Software Engineering and
Methodology 10 (2), pp. 149–183, 2001.
[14] K. Pohl, G. Böckle, and F. van der Linden, “Software Product
Line Engineering: Foundations, Principles, and Techniques,” [23] T.L. Graves, M.J. Harrold, J.M. Kim, A. Porter and G.
Springer, Heidelberg, 2005. Rothermel, “An empirical study of regression test selection
techniques,” ACM Transactions on Software Engineering and
[15] H. Muccini and A. van der Hoek, “Towards Testing Product
Methodology 10 (2), pp. 184–208, 2001.
Line Architectures,” Electronic Notes in Theoretical Computer
Science 82 No. 6, 2003. [24] N. Mansour, R. Bahsoon and G. Baradhi, “Empirical
comparison of regression test selection algorithms,” The Journal
[16] Y.F. Chen, D.S. Rosenblum and K.P. Vo, “Test tube: a system
of Systems and Software 57 (1), pp. 79– 90, 2001.
for selective regression testing,” Proc. IEEE International
Conference on Software Engineering (ICSE), pp. 211–220, Los [25] E. Uzuncaova, D. Garcia, S. Khurshid and D. Batory, “Testing
Alamitos, CA, USA, 1994. software product lines using incremental test generations,” Proc.
IEEE International Symposium on Software Reliability
Engineering (ISSRE), pp. 249–258, 2008.
Simula Research Laboratory, Technical Report 2012-20 October 2012

Appendix A: Questionnaire for Cisco

S URVEY OF CISCO’ S C URRENT AND P ROPOSED T EST S ELECTION P ROCESS

Version: 1.0

(Estimated time to complete the survey: 10-15 minutes)

Dear participants,

Thank you for taking time to contribute to this survey. The objective of our survey is to seek potential improvements to the
current test case selection process at CISCO. This survey aims to solicit the views of those who are either directly involved in this
process or are affected by it.

The questions were derived from preliminary discussions with CISCO test engineers. At this time, we aim to verify and validate
our initial solution of test case selection for VCS (Video Conference System) product line in addition to collecting views about
current manual test case selection process.

Conclusions from any survey are only as good as the data provided by the respondents. So, please answer all the questions frankly
and accurately based on your personal knowledge and experience.

Confidentiality and Anonymity: This survey is NOT intended to assess individual knowledge and performance. Consequently,
individual responses will be kept strictly confidential. The results will be collected, anonymized, and reported in a summary form.
And, you will be notified when the analysis results are finalized.

If you have any question, feel free to contact:

Shuai Wang, Shaukat Ali
shuai@simula.no, shaukat@simula.no
Certus V&V Lab, Simula Research Laboratory
 Simula Research Laboratory, Technical Report 2012-20 October 2012

• Basic Information

1. YOUR RESPONSIBILITY
Question A1: Job function
Quality Assurance (QA) manager
Testing engineer
Other: please specify - .________________________

Question A2: I am working on the SATURN project.

YES
Other: please specify - .________________________

2. YOUR EXPERIENCE
This block of questions will help us to understand your level of involvement with the automated process.

Question B1: How would you assess your knowledge and experience in testing?

No experience
I have some experience but I am not an expert
I am highly knowledgeable but it is not my primary responsibility
I am an expert

Question B2: How long have you been involved in testing VCS products?

Less than 1 year

2-5 years
5-10 years
More than 10 years

Question B3: How many discussions have you been involved in the process of automated selection of test cases for VCS products?

Not involved
Less than 3 discussions
3-5 discussions
5-8 discussions
More than 8 discussions

3. CURRENT MANUAL SELECTION PROCESS

Question C1: It is time-consuming to select relevant test cases for a specific product.
Strongly agree
Agree
No opinion
Disagree
Strongly disagree

Question C2: It is error-prone to select relevant test cases for a specific product.
Strongly agree
Agree
No opinion
Disagree
Strongly disagree
 Simula Research Laboratory, Technical Report 2012-20 October 2012
Question C3: The current manual selection largely depends on the expertise and domain knowledge of a tester.
Strongly agree
Agree
No opinion
Disagree
Strongly disagree

Question C4: The selected test cases do not cover all testing requirements.
Strongly agree
Agree
No opinion
Disagree
Strongly disagree

Question C5: Some redundant test cases exist in the current manual selection of test cases.
Strongly agree
Agree
No opinion
Disagree
Strongly disagree

4. FEATURE MODEL FOR TESTING (FM_T)

Question D1: It is easy to understand the notations of FM_T.

Strongly agree
Agree
No opinion
Disagree
Strongly disagree

Question D2: FM_T is sufficient to represent all functionalities of a VCS product line.
Strongly agree
Agree
No opinion
Disagree
Strongly disagree

Question D3: It is easy to understand and use the Pure::Variants tool for building FM_T.
Strongly agree
Agree
No opinion
Disagree
Strongly disagree

Question D4: It is easy to build and revise a FM_T for a VCS product line.
Strongly agree
Agree
No opinion
Disagree
Strongly disagree
 Simula Research Laboratory, Technical Report 2012-20 October 2012
Question D5: The functionalities of a VCS product line do not change significantly. This means that a FM_T doesn’t need to be
changed significantly once it is built.
Strongly agree
Agree
No opinion
Disagree
Strongly disagree

5. COMPONENT FAMILY MODEL FOR TESTING (CFM_T)

Question E1: It is easy to understand the notations of CFM_T for VCSs.
Strongly agree
Agree
No opinion
Disagree
Strongly disagree

Question E2: A CFM_T is sufficient to represent test case structure.

Strongly agree
Agree
No opinion
Disagree
Strongly disagree

Question E3: It is easy to obtain the XML file from test database for representing the overall test case structure.
Strongly agree
Agree
No opinion
Disagree
Strongly disagree

Question E4: It is easy to obtain and add tags information into the XML file representing the overall test case structure.
Strongly agree
Agree
No opinion
Disagree
Strongly disagree

6. OVERALL EXPERIENCE
Question F1: The proposed automated test case selection process can significantly improve the current testing process for VCS
product line in the future.
Strongly agree
Agree
No opinion
Disagree
Strongly disagree

Question F2: Please feel free to provide any additional comments for the automated test case selection process.

THANK YOU VERY MUCH FOR YOUR HELP WITH THIS SURVEY!
Simula Research Laboratory, Technical Report 2012-20 October 2012
Appendix B: Results for the questionnaire a) FEATURE MODEL FOR TESTING (FM_T)

1) Basic information (Part 1,2 of questionnaire)

For all respondents,
• Two of them are QA managers, one is a test engineer
and one play both two roles, and they are all working on
the SATURN project, which is related with the VCS PL
testing;
• Two of them assess themselves experts in testing based
on the knowledge and experience, one is highly
knowledgeable but testing is not her primary
responsibility and one assess him having some
experience but not an expert in testing;
• Two of them are working for 5-10 years, one is working
for more than 10 years and one is just working for 1 year;
• Three of them have been involved in the discussions
related with our proposed process for more than 8 times
Fig. 8. Views on the FM_T
and one of them has been involved for 5-8 times.
No. Strongly Agree No Disagree Strongly
2) Current Manual Selection Process (Part 3 of
agree opinion Disagree
questionnaire) QD1 0 4 0 0 0
QD2 0 3 1 0 0
QD3 0 2 2 0 0
QD4 0 3 1 0 0
QD5 2 1 1 0 0
Table 3. Views on the FM_T

b) COMPONENT FAMILY MODEL FOR TESTING (CFM_T)

Fig. 7. Views on the current manual selection process

No. Strongly Agree No Disagree Strongly

agree opinion Disagree
QC1 2 2 0 0 0
QC2 3 1 0 0 0
QC3 2 2 0 0 0
Fig. 9. Views on the CFM_T
QC4 2 2 0 0 0
QC5 4 0 0 0 0 No. Strongly Agree No Disagree Strongly
Table 2. Views on the current manual selection process agree opinion Disagree
QE1 0 3 0 1 0
3) Our Proposed Process (Part 4,5 of questionnaire) QE2 1 3 0 0 0
QE3 2 1 1 0 0
QC4 2 1 1 0 0
Table 4. Views on the CFM_T