Scientific foundations to the multilevel method

IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 6, NO. 3, JUNE 1998 337 Scientific Foundations to the Multilevel Method B. Neelakantan and S. V. Raghavan, Senior Member, IEEE Abstract— Conformance testing of protocols is the process of checking whether an implementation under test conforms to the standards. In this paper we prove that the multilevel method (M method) [which splits the specification graph into a basic subgraph and one or more higher level subgraph(s)] is capable of diagnosing any number of faults in the higher level subgraph(s), if the basic subgraph is error-free. Heuristics for obtaining the basic subgraph and higher level subgraphs from the specification graph are also given. The advantage of the M method is that it has error recovery and, in addition, it does not assume the presence of reliable reset in the implementation under test. We additionally propose an incremental test sequence generation method, which produces a shorter test sequence and whose fault coverage is same as that of the M method. Finally, we compare the M method with other fault detection and fault diagnostic methods. Index Terms—Black box testing, finite state machines, protocol conformance testing, test methods, test sequence. I. INTRODUCTION ROTOCOL conformance testing has been an active area of research in the field of computer networks and distributed computing. From the time that the International Standards Organization (ISO) started developing standards for conformance testing of network protocols, a lot of research work has been undertaken in this field [10]. Conformance testing involves testing both the capabilities and the behavior of an implementation and checking what is observed against the conformance requirements in the relevant recommendations and against what the implementor states as the implementation capabilities [10], [20]. This is done by applying a test sequence to the implementation under test (IUT) and giving a verdict by comparing the observed output with the expected output. Conformance testing does not assess the performance, robustness, or reliability of an implementation [10], [20]. Network protocols can be specified as deterministic finite state machines (DFSM). The IUT is assumed to be a black box modeled as a DFSM. The internal states of the IUT are not visible and can be identified only by analyzing its input/output characteristics. A sequence of inputs, called the test sequence, when applied to the IUT, should be able to verify if all of the states that are present in the specification are also present in the IUT. In addition to that, it should also verify if all of the specified transitions have same head and tail states with the same input and output characteristics. As the complexity of the protocols increases, the lengths of the test sequences also increase. This necessitates the devel- P Manuscript received September 29, 1994; revised October 11, 1995; approved by IEEE/ACM TRANSACTIONS ON NETWORKING Editor K. Sabnani. The authors are with the Network Systems Laboratory, Department of Computer Science and Engineering, Indian Institute of Technology, Madras 600 036 India (e-mail: svr@shiva.iitm.ernet.in). Publisher Item Identifier S 1063-6692(98)04077-1. opment of systematic methods for test sequence generation. Various test sequence generation methods like transition tour (T), distinguishing sequence (D), characterizing sequence (W), unique input–output (U), extended transition tour (E), regular checking (R), fault resolution (FR), diagnostic approach (DA), and multilevel (M) methods are proposed in the literature [3], [7], [9], [13], [15], [16], [18], [19], [21]. The test sequence generation methods (T, D, W, U, E, R, FR, and DA methods) take the entire specification as a single entity for the test sequence generation. Since testing the entire protocol as a single entity is complex in nature, the method of splitting the given graph (i.e., the given DFSM) into smaller subgraphs and applying test sequences on them was advocated by [14] and [15]. This approach, called the M method, can detect and locate any number of faults in the IUT, if the basic subgraph is error-free. The key steps in the M method are: • identification of the basic subgraph and higher level subgraphs; • testing the basic subgraph by a method that is capable of detecting faulty IUT even in the presence of multiple faults; • testing the higher level subgraphs by a method which should have fault diagnosis capability. In [14] and [15] the identification of the subgraphs at various levels was more of an art than a science. In this paper we provide a formal approach to the M method. In addition to that, we propose an incremental test sequence generation method, which is capable of diagnosing any number of faults in the higher level subgraph if the basic subgraph is error-free. The added advantage of this method is that the length of the test sequence is usually less than that of the M method. If the basic subgraph is the same or close to the specification graph, then the advantage of this method will be somewhat lost. The rest of the paper is organized as follows. Section II introduces the terminologies that are used in the rest of the paper. Section III deals with the state-of-the-art in conformance testing of protocols. Section IV briefly outlines the R method [16]. Section V discusses the M method in detail. Section VI compares the M method with other testing methods and brings out the advantages and shortcomings of the M method. Section VII discusses the incremental test sequence generation method to protocol testing. Section VIII summarizes the contributions of this paper. II. PRELIMINARIES A DFSM is a quintuple , where , , and are finite nonempty sets of inputs, outputs, and states, respectively, is the state transition 1063–6692/98$10.00  1998 IEEE 338 . function, and is the output function such that is the set containing all pairs of The Cartesian product elements . The state transition function associates an element from , called the with each pair next state. The output function associates with each pair of an element from [12]. We use FSM to denote DFSM in the rest of the paper. A completely specified FSM is one in which there exists for every input element a permissible output at every state. Otherwise, the FSM is said to be partially specified. If any state input combination is not defined in the specification, it is assumed that the protocol entity ignores that input and remains in the same state, producing null output. This is called the completeness assumption [21]. of the FSM there exists If for every pair of states and an input sequence which takes the FSM from to , then is the FSM is said to be strongly connected. A machine minimal if the number of states of is less than or equal for any Mealy machine to the number of states of equivalent to . An FSM can be represented by a directed graph , where the set of nodes represents the set of specified states of the FSM and is the set of labeled directed edges, defined as follows. There is a , from to ( ) labeled directed edge , if and only if there is a transition from to . Since the FSM is deterministic, for each node there are no two with the same input labels. edges leaving The number of edges leaving (entering) a node is termed its outdegree (indegree) [8]. A test subsequence is a sequence of input symbols, testing the partial behavior of the specification. An input sequence is said to be a homing sequence if the state of the machine after application of the input can be uniquely determined from the machine’s response, regardless of the initial state [12]. A recovery subsequence takes the FSM from the home state to a state from where the next test subsequence starts. A faulty transition in an IUT is directly reachable by a test subsequence if no other fault is encountered until the input corresponding to edge under test is applied [3]. A machine is -distinguishable if, for each pair of states, there is at least one input sequence of length which, when applied to the pair, yields a different output sequence [7]. Some test sequence generation algorithms assume that a reliable reset is present in the IUT, which, when applied to the IUT, takes it to the start state, irrespective of the state in which the IUT exists. A. The FSM Fault Model The conformance testing methods consider the specification and the IUT as an FSM and center around the idea of applying to the FSM. If the observed a sequence of inputs output of the IUT is same as the expected output, then the IUT is said to have passed the test. However, if the observed output differs from the expected output, a fault is said to have occurred. Faults in an IUT can be classified as follows [4]. Output Fault: A transition has an output fault if, for the corresponding state and received input, the IUT provides an output different from the expected output. IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 6, NO. 3, JUNE 1998 Transfer Fault: A transition has a transfer fault if, for the corresponding state and received input, the IUT enters a different state other than the one specified by the next-state function. III. STATE OF THE ART The protocol test sequence methods can be classified into fault detection methods and fault diagnosis methods. The aim of a fault detection method is to find out whether the IUT is faulty or not. The test result analysis does not give any indication of the location of the fault. On the other hand, fault diagnosis methods, in addition to fault detection, give the exact location of the fault. A. Fault Detection Methods Transition Tour Method: Transition tour is an input sequence which takes the FSM from its initial state, traverses every transition at least once, and returns back to its initial state [18]. This method does not verify intermediate states and, hence, it does not have the capability of detecting transfer faults. However, it can detect output faults. Extended Transition Tour Method: A state-based approach is used in this method (E method), unlike the transition-based approach of existing methods. The principle behind the E method is to verify that the behavior corresponding to each incoming edge and outgoing edge of every state in the implementation is similar to that of the specification [13]. To achieve this, each incoming edge of the node is followed by every outgoing edge of the node at least once in the test sequence. This method can detect output faults. If the specification is onedistinguishable, then it can detect transfer faults. Otherwise, the E method may fail to detect transfer faults. Distinguishing Sequence Method: This method assumes that the specification graph is minimal, strongly connected, and completely specified, and that it possesses a distinguishing sequence (DS). An input string is said to be a DS of a machine , if the output string produced by in response to the input string is different for each initial state [9]. This method can detect both output and transfer faults. Characterizing Sequence Method: This method derives and the set . The set two sets of input sequences: the set consists of input sequences that can distinguish between every pair of states in the specification. is transition cover set. The set of test subsequences are obtained by concatenating [7]. This method can detect output and the two sets and transfer faults. Unique Input/Output Sequence Method: A unique input/ is denoted by output (UIO) sequence for a state , is an input/output sequence with the originating state such that there is no for which is specified input/output sequence for originating state [21]. For a state , in an FSM which does not possess a UIO sequence, a from the remaining states signature which distinguishes one by one is used. The UIO method can detect output errors. If any two states in the IUT have same UIO sequence, then the UIO method may fail to detect transfer errors [5]. A state may have more than one D, W, and UIO sequence [22], [24]. NEELAKANTAN AND RAGHAVAN: SCIENTIFIC FOUNDATIONS TO MULTILEVEL METHOD B. Fault Diagnosis Methods Fault Resolution Approach to Protocol Testing: This , the method (FR method) assumes that for each transition should not occur in the UIO portion label corresponding to does of the test sequence. This ensures that the erroneous not mask the error in the UIO sequence. Also, the IUT is assumed to have either output fault or transfer fault in at most one edge. This method assumes the presence of reliable reset in the IUT. A detailed algorithm is given in [19]. Diagnostic Approach to Protocol Testing: The basic assumption in this method (DA method) is that all of the faulty transitions are directly reachable by a test subsequence. The presence of reliable reset in the IUT is assumed. Initial testing is done by any one of the D, W, and U methods. The observed outputs are compared with the expected ones. The difference in outputs, called symptoms, are identified for all of the test subsequences. A set of edges where the fault might be present is guessed based on the symptoms. Then, a set of fault hypotheses for each test subsequence is constructed. Fault hypothesis is the assignment of a specific fault to all of the suspected edges. A new specification graph is constructed as suggested by the fault hypothesis and if the new observed output is same as the output of the faulty IUT, then the fault hypothesis explains all of the faults. Each fault hypothesis is a diagnostic candidate. This method may produce a number of diagnostic candidates. To reduce the number of diagnostic candidates, additional testing must be done. A detailed algorithm is given in [2] and [3]. IV. REGULAR CHECKING METHOD The test sequence generated based on the R method has better fault coverage than the methods T, D, W, U, and E. The methods (D, W, U, and E) combine the test subsequences without imposing any constraint on ordering the test subsequences. In [16] a detailed study has been made on how the ordering of different test subsequences affects the fault detection capability. Different cases were identified where one fault hides another fault. Care was taken so that all such hiding faults were detected. Hence, the R method is capable of detecting multiple faults in the IUT. A brief outline of the R method is given here. Let the specification have states and edges. For each edge , the states are identified. A effect of transfer faults on all test subsequence is generated which will detect all such errors. This step is repeated for all of the edges in the specification. Finally all of the test subsequences are combined in an optimal way to obtain the test sequence. Combining different test subsequences is made as per ordering constraints imposed by the structure of the specification graph. A description of the R method and a detailed analysis of how the ordering can affect the fault coverage can be found in [16]. V. MULTILEVEL METHOD The M method assumes that the specification is minimal, strongly connected, and completely specified. If the specification is not completely specified, the IUT should satisfy the completeness assumption. The IUT is assumed to have the same number of states as the specification. 339 To test the entire IUT, all of the edges in the specification should be tested. For each edge, a test subsequence can be generated. The test subsequence has three parts: a preamble, an edge under test, and a postamble. The preamble is an input sequence which will take the IUT to the head state of the edge under test. Due to limited controllability of the IUT, the preamble will be long. The postamble is the characterizing sequence (any one of D, W, and U subsequences) for the tail state of the edge under test. Due to limited observability of the IUT, the characterizing sequence will also be long. For the applied test subsequence, if any fault is detected, the fault may be due to an output and/or transfer fault in one or more of the following: preamble, edge under test, and postamble. Fault may also be observed if the test subsequence does not start from the state specified by the test subsequence. Thus, it is very difficult to locate the faults once detected. Suppose that the edges corresponding to the preamble and the postamble of each test subsequence are error-free. If the testing starts at the state specified by the test subsequence, it is possible to provide complete fault diagnosis for those edges. An output fault will be detected as soon as the input corresponding to the edge under test is applied. Transfer fault will be detected only when the input corresponding to the postamble portion of the test subsequence is applied, as the IUT would have moved to a different state and the characterizing sequence will detect the fault. If both output and transfer faults are present in the edge under test, the output fault will be detected by the application of the input corresponding to the edge under test and the transfer fault will be detected by the application of the characterizing sequence [13]–[15]. Once a fault is detected, appropriate steps must be taken to recover from it. For output fault, further testing can continue as the IUT will be in the state specified by the test sequence. However, for transfer fault, the IUT would have moved to a different state, the identity of which is unknown. A homing and recovery subsequence should be applied to take back the IUT to the required state. It is imperative that the FSM recovers back correctly after an error as, otherwise, further testing will lead to wrong conclusions. Hence, homing and recovery subsequences are derived using only those edges that have been verified to be correct. The M method addresses all of the aforementioned issues of testing, namely, controllability, observability, fault diagnosis, and error recovery [15]. denoted by Definition 1: The basic subgraph of has the minimum number of inputs among all strongly connected spanning subgraphs (SCSS’s) of . By number of inputs, we mean the cardinality of the input set of the DFSM corresponding to . Definition 2: Let denote the set of inputs from the . Let , where specification that are not present in , be nonempty sets of inputs such that . For any two and , for all , , and . The inputs in are called in are called levelthe level- inputs and the edges edges, where and . Definition 3: The inputs in the basic subgraph are also called level-0 inputs. 340 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 6, NO. 3, JUNE 1998 Definition 4: The maximal level- subgraph of the specification graph is a subgraph of the . specification, such that Definition 5: The minimal level- subgraph of the specification graph is a subgraph of the specification, . such that be the test sequence for level- inputs, Definition 6: Let obtained from the maximal level- subgraph. Let be the test sequence for level- inputs, obtained from the minimal level- subgraph. Definition 7: The level- subgraph is defined to be a maximal level- subgraph if there are no faults in all of the levelsubgraphs , and the minimal level- subgraph otherwise. The M method has two phases. In the first phase the basic subgraph and the higher level subgraphs are obtained from the specification. For testing the basic subgraph, we should use a test method which has better fault detection capability even in the presence of multiple faults. For testing the level- inputs, the preamble and the postamble should be selected from the inputs. The second phase does testing level-0 to levelof the IUT. If any error is detected during the testing, an appropriate error recovery sequence should be applied before further testing continues. We say that an IUT is erroneous if it has erroneous transitions. During the test sequence generation for level- inputs, we do not know whether the IUT has any erroneous transitions in the previous level inputs. Suppose each level has at most edges, the number of different erroneous IUT is at most , and the number of different test sequences required to . As take care of all these erroneous IUT will be at most error can be present in any level, it is not feasible to generate that large a number of test sequences before testing. Hence, if , then there is any error in some level- inputs, where the test sequence corresponding to the minimal basic subgraph should be applied for testing level- inputs. The framework of the multilevel test sequence generation phase is as given here. The input to the algorithm is the specification graph. The outputs of the algorithm are the higher level subgraphs and their corresponding test sequences. be the specification graph. 1) Let . Generate test sequence for . 2) Compute 3) Generate homing and recovery subsequence for . . 4) Let be the number of higher level subgraphs. Let , do the following. 5) While and . a) Construct the subgraphs b) Generate homing and recovery subsequence . c) Generate test sequence , respectively. and for and . d) Let 6) End. We will now provide the framework of the testing phase. In this phase the inputs are the test sequence and the IUT. The outputs are the faulty edges and the nature of faults. 1) Apply the test sequence for the . If has faulty transitions, halt. Otherwise continue testing. . Let , where 2) Initialize is the number of higher level subgraphs. Initialize the to to false. Boolean array , do the following. 3) While , then apply the test sequence ; a) If . else apply the test sequence , b) If any transfer fault is detected and if ; else apply . Furthen apply ther testing can be continued after error recovery. c) If any error is detected in level- subgraph, then . . d) 4) Report all of the faulty transitions. For output error there is no need for extra test sequences. For transfer error, if W or U sequences are used as characterizing sequences, test then to know the faulty tail state, at most subsequences are required. If D sequences are used as characterizing sequences, further testing is not required. Theorem 1: The M method can locate any number of faults present in the higher level subgraphs without reliable reset in is error-free; 2) is an SCSS; 3) the IUT if: 1) has a characterizing sequence for all states; and 4) the . preamble and postamble are selected from be the specification. Let be Proof: Let the basic subgraph, i.e., . Let . contains all of the edges to be tested. The set , apply a test subsequence To test an edge where is the preamble to bring the graph to the head is the characterizing state of the edge under test and be sequence of the tail state of the edge under test. Let the expected output derived from the specification and be the observed output of the IUT. satisfies the four conditions given in the Assume that statement of the theorem and testing starts from the specified , generate the test subsequence by start state. For each . No edges selecting the preamble and postamble from are faulty, as they are selected from . in . Hence, if at all any fault is observed it should be only in are given in Different types of faults that can be present in Section II-A. We consider all possible combinations of those faults here. . Hence, • In the case of output fault, output faults will be detected. • In the case of transfer fault, the IUT will be in some . will detect the transfer fault. This state , the two states is because if and are equivalent in the IUT, and this contradicts our assumption that each state in the basic subgraph has a characterizing sequence. Hence, the transfer fault will be detected. • In the case of both output and transfer faults, and . Hence, the fault will be detected. NEELAKANTAN AND RAGHAVAN: SCIENTIFIC FOUNDATIONS TO MULTILEVEL METHOD is missing, the IUT will remain in the same state by the completeness assumption. The characterizing will detect the error, since sequence of . Once a fault is identified, the state in which the IUT remains can have a head state is unknown. Since the edges in (as the IUT is assumed to be completely from all specified), the preamble must be capable of taking the IUT from any state to the required state. A recovery subsequence exists for a strongly connected graph which can take the IUT to the required state. As all of these edges are from the basic subgraph, the M method can recover from errors and further testing can be continued. Hence, the theorem is proven. • If A. Heuristics for Obtaining the Basic Subgraph If a particular input is applied to the IUT, its effect will be observed at all of the states where the input is applicable. By taking a subset of inputs in such a way that we get an SCSS of the specification, where each state has a characterizing sequence, testing of this subgraph can be done first. If all of the edges in this SCSS are error-free, then it has an errorfree characterizing sequence, transfer sequence, and homing sequence. On the other hand, if we take a minimum number of edges in constructing the SCSS, it will have at least as many inputs . Even though the SCSS has a minimum number as in of edges, all other edges which have same input label in the specification, but not included in the SCSS, will respond to the input if the input is applied to the IUT. In effect, we are taking all of those edges, when we intend to take a minimum number of edges. Hence, an SCSS which has a minimum number of instead of an SCSS with inputs should be taken as a minimum number of edges. , given , is -hard. Theorem 2: Identification of , Proof: Given a labeled directed graph let be the problem of identification of . Let be the problem of finding an SCSS with minimum number of is -hard [11]. Assume that edges. It is known that is polynomial solvable. Now, label each edge of uniquely from the set where and let the new on is the same graph be denoted by . The problem on . This contradicts the fact that is as the problem -hard. Hence, the problem of identification of is -hard. As the problem of identifying a is -hard, an approximate (polynomial) algorithm for identifying a basic subgraph is given in this section. Most of the real-world protocols are not completely specified and may have self loops. All self loops from the specification can be removed without loss of generality while . Some of the nodes in the specification may deriving have indegree and/or outdegree as one. All such inputs are . These inputs identified as they must be present in the are called essential inputs and without them the basic subgraph will not be an SCSS of the specification. This algorithm identifies a number of SCSS’s of the specification. The input to the algorithm is the specification graph and the output is the SCSS. 341 1) Let be the upper limit decided by the tester, beyond . which there is no need to find a minimal are the inputs that are in the specifi2) is the number of possible input events. cation, where Remove all of the essential inputs from this list and let . Hence, the new input list be is the number of essential inputs. Let . , then go to step 7. 3) If with inputs, 4) For each subset of construct a graph with edges corresponding to these inputs and essential inputs. If the graph is an SCSS of specification, then store the input set. 5) If any SCSS is found, then go to step 8. 6) Increment by one. Go to step 3. , then keep adding the input events 7) If there is no in any random order until an SCSS of the specification is obtained. 8) End. be the number of nodes. Let Time Complexity: Let be the number of possible input events. Let be the number of essential inputs. Step 2 takes time, as it can be implemented by testing every vertex whether it has indegree or outdegree one. Now we will analyze the time taken by step 4 in the th iteration of the loop from steps 3 to 6. In step 4 the graph under consideration has input events. In the edges corresponding to worst case there can be an outgoing edge for each input from every vertex. Hence, the number of edges in the graph under . To test whether consideration can be at most is strongly connected can be a directed graph time. Hence, step 4 in iteration takes done in . The completion of the algorithm can take place at two points: one within the loop 3–6 and another in step 7. If the algorithm terminates within the th iteration of steps 3–6, then the time taken is . If the algorithm terminates in step 7 after addition of input events in random order, then the time taken is . Hence, the worst-case complexity will be , in which case is identical to . Some of the SCSS’s that are obtained may not be minimal, because by removing some inputs from the specification it is possible that the resultant subgraph may have equivalent states. If the SCSS has equivalent states, an alternate basic subgraph must be chosen which has a characterizing sequence for all states. It should be noted that even if a single fault is detected in , then testing of higher level subgraphs cannot be the done. If the has essential functions of the specification and if any fault is detected in the IUT, then further testing can is chosen after be discontinued. Otherwise, an alternate doing test result analysis of the erroneous . Edges for the alternate subgraph can be chosen by not-included erroneous inputs to the new . 342 The number of inputs in the specification of most of the reallife protocols are bound by a constant. Also, they have a which has a smaller number of inputs. Hence, it is worthwhile , as it simplifies higher level testing. Also, to identify a it reduces the complexity of the testing process and helps in providing complete fault coverage for higher level inputs. B. Heuristics for Obtaining Higher Level Subgraphs will have approximate minimum number of input The events. As a result, the length of UIO sequences will be large and the number of multiple UIO (MUIO) sequences [22], [24] will be small. If we test all of the edges that are not in as one group, then the total length of the test sequence will be, in general, large. To reduce the total length of the test sequence, the length of UIO sequence should be small and the number of MUIO sequences should be large. This can be . achieved by adding more error-free inputs to the Definition 8: A new basic subgraph can be constructed by and error-free level- inputs. We call taking edges in a level- input error-free if all of the edges corresponding to level- inputs are error-free. Theorem 3: Any subgraph of the specification , which is and some set of error-free inputs of the the union of the IUT, can act as a new basic subgraph for testing the untested edges. Proof: We will show that by adding all edges corre, all of the properties sponding to the error-free inputs to mentioned in Theorem 1 are satisfied. • The added edges are error-free. Hence, the new basic subgraph is also error-free. is strongly connected. Since the new basic • The subgraph has been derived from by only adding edges and not removing edges, we see that the new basic subgraph is also strongly connected. • All of the added edges have different inputs than those . Hence, the characterizing of the edges in the is still the characterizing sequence sequence of the of new basic subgraph. In fact, the length of the characterizing sequences of any node in the new basic subgraph . Since will always be less than or equal to that of all of the newly added edges are error-free, there is no need for verification of these characterizing sequences. • A homing sequence exists for any strongly connected specification. Hence, by adding more inputs, the length of the homing sequence will always be less than or equal . to that of Hence, the theorem is proven. Observation 1: If level-0 to level- inputs are error-free, a new basic subgraph can be constructed by taking the inputs to level- inputs. from Observation 2: Suppose the IUT has been tested up to the level- inputs and if level- inputs have erroneous transitions, , those edges in level- inputs should not be where included in constructing the new basic subgraph. is The advantage of adding error-free level- inputs to that it reduces the total length of the test sequence (by reducing the length of premable and postamble). IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 6, NO. 3, JUNE 1998 This algorithm computes the higher level subgraphs. The inputs to the algorithm are the specification graph and various characterizing sequences. The output is the higher level subgraphs. 1) Let be the specification graph. Let be the basic subgraph. Let . Let . 2) For each input event in , do the following: a) find the average length of UIO sequences with this input added; b) find the average number of MUIO sequences with this input added. 3) Select the input which increases the number of MUIO sequences. 4) If two or more inputs have the same property, then select an input among those inputs which reduces the average length of UIO sequences. 5) If two or more inputs have the same effect, then find an input which reduces the length of the homing and recovery subsequence. subgraph. 6) This input will be chosen as a levelfrom Remove those edges corresponding to level the set . 7) Let . If the set is not empty, then go to step 2; else all of the subgraphs are obtained and, hence, stop. Time Complexity: It is assumed that the MUIO homing and recovery sequences are found initially using the methods presented in [12], [15], and [22]. After this, we can easily see time. that this algorithm takes C. Generation of the Test Sequence A lot of research has been done on how to concatenate the test subsequences in an optimal way [1], [22] with a minimum-length transfer subsequence. In general, to get a minimum-length test sequence, the length of the transfer sequence between any two subsequences should be minimum and the overlap between different test subsequences should be maximum. The methods [6], [17], [24] adopt these two observations to reduce the length of the test sequence. Overlapping is possible because some edges in the state identification subsequences are not from tested transitions and, hence, they are tested immediately. The aim of the methods in [1], [6], [17], [22], and [24] is to reduce the total length of the test sequence. All of these methods do not verify the uniqueness of the characterizing sequences in the IUT. As a result, the methods lose fault detection capability on some classes of IUT. Once an error is detected, further testing is meaningless as the IUT would have moved to some state, the identity of which is unknown. However, further testing can be continued by applying reset input and an appropriate transfer sequence. However, complete fault coverage is not possible in these methods. The M method, on the other hand, uses only tested transitions as characterizing sequences and, as a result, there is no scope for overlapping of test subsequences. Hence, to reduce the length of test sequence, shorter UIO sequences, MUIO sequences, and minimum-length transfer subsequences should NEELAKANTAN AND RAGHAVAN: SCIENTIFIC FOUNDATIONS TO MULTILEVEL METHOD be used. Also, the M method provides error recovery and complete fault coverage for higher level subgraphs. The algorithm for generating the complete test sequence from test subsequences of the higher level subgraphs is given now. The first three steps can be done before the testing starts. All of the other steps are applied when the testing is in progress. Algorithm for Generation of Test Sequence: 1) Obtain the test sequence for the basic subgraph by the R method, as it has better fault coverage among all fault detection methods, even in the presence of multiple faults. . The set contains error-free inputs. Initialize 2) Let . 3) Obtain the test subsequence for all of the higher level inputs based on MUIO method, by taking edges only from lower level inputs as preamble and postamble. 4) During the application of level- test subsequence, the following rules must be applied: a) if all of the edges that are in the transfer sequence is not in the set , then use the transfer sequence from the basic subgraph; b) if all of the edges that are in the characterizing sequence is not in the set , then apply the characterizing sequence from the basic subgraph. 5) If any error is detected in the level- subgraph, the error recovery subsequence is applied. The erroneous transition is recorded. 6) If all of the edges of the level- are error-free, add these . edges to . Let 7) If there are some more subgraphs to be tested, go to step 4. 8) Report all of the faults and stop. Most of the real-world protocols are not completely specified. To do weak conformance testing, the M method requires a basic subgraph which is completely specified. If this basic subgraph is error-free, weak conformance testing can be done for the remaining edges by taking them as level-1 inputs. If for some inputs, all of the state input combinations are defined, these inputs can be considered as higher level inputs and inputs. all of the remaining edges as levelTo add an input event to basic subgraph, all of the state input combinations of this input event should be tested. Otherwise, for the unspecified state input combinations, the IUT may behave in an unknown fashion. To have complete fault coverage, only those edges which are tested for all state input combinations should be used in the preamble and in characterizing sequences. It was pointed out that fault detection capability depends on the choice of the test subsequences and the way different test subsequences are sequenced to obtain the test sequence [23]. The reason attributed for that behavior was the arbitrariness in the algorithm to obtain the test sequence from the test subsequences. In the M method, test subsequences can be combined in any order without losing the fault diagnosis capability of the test sequence. This is because all of the characterizing sequences and the preamble consist of only 343 Fig. 1. The specification graph. Fig. 2. Basic subgraph. error-free transitions. Enough care is taken to start the test from the state specified by the test sequence. Whenever any fault is detected, the error recovery subsequence takes the IUT back to the state specified in the test sequence. Hence, the test subsequences can be sequenced without any fear of losing fault detection capability. The choice of the test subsequences also does not affect the fault detection capability, as all of the characterizing sequence are from error-free edges. D. Discussion and Example Consider the specification given in Fig. 1. It has three states , and ), three inputs ( , , and ), and two outputs ( , ( and ). Different characterizing sequences (D, W, and U sequences) and homing sequences are obtained. The D , , and . sequences of the specification are The input part of the UIO sequence is called unique input are , , sequence (UIS). Some of the UIS’s for state , , and . Similarly for state , UIS’s are , , , , and . For state , UIS’s are , , , , , and . , , and . The characterization sets are Three possible homing sequences for the specification graph , , and . are The basic subgraph is given in Fig. 2. It can be tested by and the the R method. The testing starts from the state test sequence is The next step is to generate higher level subgraphs. The inputs and produces more or less the same effect in reducing 344 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 6, NO. 3, JUNE 1998 Fig. 3. Level 1 subgraph. the length of UIO sequences. The selection of particular input is purely arbitrary and let input be selected. Fig. 3 gives the level-1 subgraph. For the edges – – , – – , and – – , the test subsequences are , , and , respectively. The level-2 subgraph is same as the specification graph. If level-1 inputs succeed the test, the level-2 test can use level-1 edges in the preamble and postamble. If not, the preamble and postamble should be taken from basic subgraph itself. If the basic subgraph alone is considered in deriving test sequence for level-2 subgraph, then for – – , the test subsequence . Similarly, for – – , it is and for is – – , it is . and it is The level-1 test sequence starts at state The level-2 test sequence (derived using the basic subgraph) and it is starts at state If level-1 edges are correct, then the length of the higher level test sequence can be reduced, as it is possible to have a shorter preamble and postamble. Assume that level-1 passes the test sequence; then the length of the test sequence for the level-2 subgraph is reduced by one The total length of the test sequence is 23 (five for the basic subgraph, nine for the level-1 subgraph, and nine for the level-2 subgraph). If level-1 inputs are used in deriving the test sequence for the level-2 subgraph, then the length of the test sequence is reduced by one. In the M method, if the is dense, then the length of the test sequence corresponding will increase. to VI. COMPARISON OF PROTOCOL TESTING METHODS The D, W, and U methods can be considered as special cases of the M method, where all of these methods have only the and level-1 subgraph. During phase-one testing of the D, W, and U methods, state identification and characterizing sequence verification are done first. If this phase succeeds the test, then it does not guarantee that all of the edges that are used in characterizing sequence are error-free, whereas in the M method if succeeds the test, then all of the edges that are in it are error-free. In phase-two testing, all of the edges are tested. Another basic difference between the M method and all other test sequence generation methods is that the M method tries to reduce the total number of inputs that can be present . As a result, the length of the characterizing in the sequence will be large. All other methods attempts to reduce the length of characterizing sequence. The M method does not assume the presence of reliable reset in the implementation. On the other hand, the DA and FR methods assume the presence of reliable reset in the implementation. Instead of reset, the M method uses a homing and recovery subsequence for recovering from error. The M method assumes that all faults are directly reachable from any state. The DA and FR methods assume that all of the faults are directly reachable from the start state. In the M method, if each input event has at most one faulty transition, then no basic subgraph will succeed. However, in such a situation it is possible that all faulty transitions are directly reachable. Under such circumstances, the DA method can provide fault location. The M and FR methods assume that the input corresponding to edge under test should not appear in the characterizing sequence. This eliminates the situation where error occurs in both the edge under test and the characterizing sequence. While the FR method can give fault diagnosis only for at most one fault, the M method can provide fault diagnosis for all edges in higher level subgraphs, provided the basic subgraph is error-free. The M method can report the erroneous transition once an error is detected, while the DA method [3] has to do further testing for locating error. This is because errors can be present in both the edge under test and the characterizing sequence. Hence, it provides a number of diagnostic candidates, and only by further testing can the DA method locate the faults. Also, the M method uses the intermediate test results effectively for testing the untested portions. If all possible basic subgraphs have at least one error, then the M method is not applicable. It should be investigated whether further testing can be done even if the basic subgraph has one or two errors. Consider a special case. If there are one or two errors in the basic subgraph and the erroneous transitions are identified, then we can consider the erroneous in the IUT as a new specification, and UIO’s for the new specification are obtained as suggested in [3]. The new UIO sequences can be used for higher level subgraph testing. The test sequences are derived based on the specification. Once standard test sequences are derived, the same can be used during development stages also. Initially the subgraph can be implemented, and the test corresponding to the sequence corresponding to that can be applied. The remaining subgraphs of the specification can be implemented in a phased manner and the correctness of the implementation can be verified by applying the corresponding test sequences. The upper bound of test sequence generation are given states and in Table I. It is assumed that the FSM has NEELAKANTAN AND RAGHAVAN: SCIENTIFIC FOUNDATIONS TO MULTILEVEL METHOD TABLE I UPPER BOUND ON THE LENGTH OF THE TEST SEQUENCE inputs. denotes the length of the D sequence. Similarly, and represent the length of the maximum UIO sequence and W sequence, respectively. If the W method is used, represents the maximum number of entries in the characterizing set. We assume that there is no overlap between test subsequences. The total length of the test sequence is the sum of the length of the test subsequence and the transfer subsequence. In the worst case, all test subsequences may need . is the length a transfer subsequence of length of the test sequence corresponding to basic subgraph. The will be vary depending upon the characterizing length of sequence used. VII. INCREMENTAL TESTING OF PROTOCOLS In the M method the test sequences are generated before the testing starts. For testing of the level- inputs, the test sequence . It was uses input from the basic subgraph up to level pointed out in Section V that the number of different erroneous , where is the IUT until the level- subgraph is at most number of edges in each higher level subgraph. So if any error , the characterizing is detected in some level where sequence and the preamble are taken from the basic subgraph. We show in this section that by generating test sequences for the higher level subgraph just before the start of level testing and by using the test results of previous level subgraph(s), the total length of the test sequence can be reduced considerably, even in the presence of errors. We assume that there is a basic subgraph which is error-free, as in the M method. Theorem 4: The union of the basic subgraph and tested level- inputs can act as the basic subgraph for untested edges, irrespective of whether the level- subgraph has error in it or not. be the specification graph and Proof: Let be the basic subgraph. Let be the level- subgraph. By Theorem 1, if all of the edges in are error-free, it can diagnose all of the faults in the levelsubgraph. If there is a transfer error, the faulty tail state can be identified. For output error, the faulty output can be obtained. Missing transitions are a combination of output and transfer faults. Hence, all of the faulty edges and the nature of faults can be found. Thus, the exact behavior of IUT with inputs corresponding to the basic subgraph and the levelsubgraph is known to the tester. So, these two subgraphs of IUT can together act as the new basic subgraph for the untested edges. We show that the new basic subgraph satisfies all of the properties mentioned in Theorem 1. is error-free. The level- edges, as are present • The in the IUT, are known to the tester. Hence, the union and level- edges can act as the new of edges in 345 specification. As the exact nature of IUT is known, this new basic subgraph can be considered as error-free. • The original basic subgraph is an SCSS of . Since the by only new basic subgraph has been derived from adding edges and not removing edges from , we see that the new basic subgraph is also strongly connected. • The added inputs are different from those inputs in the basic subgraph. Hence, the characterizing sequence of basic subgraph is still valid. In addition, by deriving a new set of characterizing sequences to this new basic subgraph, it is possible to get a reduced-length characterizing sequence. There is no need to verify these characterizing sequences in the IUT, as we are deriving it from edges that are tested in the IUT. • The homing and recovery sequence of the basic subgraph is still applicable for the new basic subgraph. A new set of homing and recovery subsequences can be derived for the new basic subgraph. It is likely that the length of these sequences can be reduced in the new basic subgraph. Hence, the theorem is proven. The algorithm for generating test sequences using an incremental test sequence generation method is given below. and higher level subgraphs as in the M 1) Identify method. and test it. If 2) Generate a test sequence for of the specification is error-free, then it can provide fault diagnosis for other edges; otherwise, halt. Let . 3) Generate a test sequence for the level- subgraph by and all level- inputs as it is taking inputs from . in IUT, where 4) Apply the level- test sequence for the level- subgraph. Identify all of the erroneous transitions. For output error, no further testing needs to be done. For transfer error, identify the faulty tail states. 5) The new basic subgraph is the union of level-0 to levelinputs as it is in the IUT, for the levelsubgraph. The homing sequence and recovery sequence should be computed for the new basic subgraph. . If , testing is over 6) Let and, hence, halt. Otherwise, go to step 3. The overhead in the incremental method is to generate test sequences before the test starts, but the reduction in the length of the test sequence will be very appreciable when the test sequence of the M method is compared with that of incremental method, where the higher level subgraph has many errors. VIII. CONCLUSION The M method assumes that the specification and the IUT are strongly connected, minimal, and completely specified. Both the specification and the IUT are assumed to have an equal number of states. If the IUT has a larger number of states than the specification, it can be detected during the testing of the basic subgraph. It has been proven in this paper that the M method is capable of locating faults in higher level is error-free. Also, it has been proven subgraph(s) if the is -hard. Heuristics to find that identification of the 346 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 6, NO. 3, JUNE 1998 the basic subgraph and other subgraph(s) were given. Testing of the basic subgraph should be done by a method which is capable of detecting multiple faults. Hence, the R method was used for testing the basic subgraph. Testing of higher level subgraphs was done using the MUIO method. Overlap between different test subsequences is not possible in the M method. As the M method does not assume the presence of a reliable reset in the IUT, it has wide applicability. Also, it splits the complex specification into many subgraphs and tests them separately. To do weak conformance testing using the M method, we need a basic subgraph which is completely specified. The M method was compared with other fault detection and fault diagnostic methods. An incremental test sequence generation method was presented which has the same fault coverage as the M method is same as or close but has a shorter test sequence. If to , then the advantages of the method will be somewhat lost. Further investigations can be done on the following problems. • It has been observed that one error may mask the presence of other errors, and a detailed study of that is done in [16]. Hence, the basic subgraph should be tested by a method which should detect any number of multiple faults. Since the success of the M method depends solely on the correctness of the basic subgraph, a method which guarantees to detect many combinations of output and transfer faults needs to be designed. • The possibility of applying the M method and the incremental method should be explored for special class of graphs. • The possibility of applying the M method should be investigated to test nondeterministic FSM’s. REFERENCES [1] A. V. Aho, A. T. Dahbura, D. Lee, and M. U. Uyar, “An optimization technique for protocol conformance test generation based on UIO sequences and rural Chinese postman tours,” Protocol Specification, Testing, and Verification VIII, pp. 75–86, 1988. [2] G. V. Bochmann and A. Ghedamsi, “Diagnostic tests for finite state machines,” Univ. Montreal, Montreal, P.Q., Canada, Tech. Rep. 807, 1992. [3] A. Ghedamsi, G. V. Bochmann, and R. Dssouli, “Multiple fault diagnostic tests for finite state machines,” in IEEE INFOCOM’93, San Francisco, CA, Mar. 1993, pp. 782–791. [4] G. V. Bochmann, R. Dssouli, A. Das, M. Dubuc, A. Ghedamsi, and G. Luo, “Fault models in testing,” in Int. Workshop Protocol Test Systems, 1991, pp. 17–32. [5] W. Y. L. Chan, S. T. Vuong, and M. R. Ito, “An improved protocol test generation procedure based on UIOS,” in SIGCOMM’89, Austin, TX, Sept. 1989, pp. 283–294. [6] M.-S. Chen, Y. Choi, and A. Kershenbaum, “Approaches utilizing segment overlap to minimize test sequences,” Protocol Specification, Testing, and Verification X, pp. 85–98, 1990. [7] T. S. Chow, “Testing software design modeled by finite state machines,” IEEE Trans. Software Eng., vol. SE-4, pp. 178–187, May 1978. [8] N. Deo, Graph Theory with Applications to Engineering and Computer Science. India: Prentice-Hall, 1984. [9] G. Gonenc, “A method for the design of fault detection experiments,” IEEE Trans. Comput., vol. C-19, pp. 551–558, June 1970. [10] “Information technology—OSI conformance testing methodology and framework—Part 1: General concepts,” International Organization for Standardization, 1989. [11] S. Khuller, B. Raghavachari, and N. Young, “Approximating the minimum equivalent digraph,” in Proc. Fifth Annu. ACM SIAM Symp. Discrete Algorithms, Arlington, VA, 1994, pp. 177–186. [12] Z. Kohavi, Switching and Finite Automata Theory. New York: McGraw-Hill, 1978. [13] R. S. Kripanandan, “Multi-level approach to protocol conformance testing,” M.Sc. thesis, Dep. Comput. Sci., Indian Inst. Technol., Madras, India, Sept. 1990. [14] R. S. Kripanandan and S. V. Raghavan, “Multilevel approach to protocol conformance testing,” in 10th Int. Conf. Computer Communication ICCC’90, New Delhi, India, 1990, pp. 772–778. , “Multilevel approach to protocol conformance testing,” Comput. [15] Networks ISDN Syst., vol. 27, pp. 591–610, 1995. [16] R. E. Miller and S. Paul, “Structural analysis of a protocol specification and generation of a maximal fault coverage conformance test sequence,” Univ. Maryland, College Park, MD, Tech. Rep., 1992. , “On the generation of minimal length conformance tests [17] for communication protocols,” IEEE Trans. Networking, vol. 1, pp. 116–129, Feb. 1993. [18] S. Naito and M. Tsunoyama, “Fault detection for sequential machines by transition tours,” in Proc. Fault Tolerant Computing Syst., 1981, pp. 238–243. [19] T. Ramalingam, A. Das, and K. Thulasiraman, “On conformance test and fault resolution protocols based on FSM model,” in Int. Conf. Computer Networks, Architecture, and Applications, Networks’92, Trivandrum, India, 1992, pp. 435–475. [20] D. Rayner, “OSI conference testing,” Comput. Networks ISDN Syst., vol. 14, pp. 79–98, 1987. [21] K. Sabnani and A. Dahbura, “A protocol test generation procedure,” Comput. Networks ISDN Syst., vol. 15, pp. 285–297, 1988. [22] Y. N. Shen, F. Lombardi, and A. T. Dabhura, “Protocol conformance testing using multiple UIO sequences,” IEEE Trans. Commun., vol. 40, pp. 1282–1287, Aug. 1992. [23] D. Sidhu and R. Vallurupalli, “On arbitrariness in protocol conformance test generation,” Univ. Maryland, College Park, MD, Tech. Rep. UMIACS-TR-90-28, CS-TR-2417, 1990. [24] B. Yang and H. Ural, “Protocol conformance test generation using multiple UIO sequences with overlapping,” in SIGCOMM’90, Philadelphia, PA, Sept. 1990, pp. 118–125. B. Neelakantan received the B.E. degree in computer science and engineering from Thiagarajar College of Engineering, Madurai, India, in 1992, and the M.S. (by Research) degree from the Department of Computer Science and Engineering, Indian Institute of Technology, Madras, India, in 1996. His research interests are networks, protocols, and graph theory. S. V. Raghavan (M’63–SM’92) is with the Department of Computer Science and Engineering, Indian Institute of Technology, Madras, India, as a Professor and the Chair of Engineering. For the last 20 years, he has been actively involved in research and development related to performance evaluation, networks, multimedia, and protocol engineering. He was one of the founding members of the Ernet (Education and Research in Computer Networking) in India, a joint initiative of the Government of India and United Nations Development Program. His current projects include development of multimedia integrated networking environment (MINE), encompassing issues arising out of mobile computing, high-speed networks, and operating systems and protocol support for handling multimedia information. Dr. Raghavan has served as Chair for numerous conferences and committees, and on the editorial boards of the Institution of Electronics and Telecommunication Engineers (IETE) Computer Communication Journal. He is a Fellow of the IETE and a Governor of the International Council for Computer Communication (ICCC).