Journal of chemical information and modeling, Jan 24, 2018
Molecular docking can account for receptor flexibility by combining the docking score over multip... more Molecular docking can account for receptor flexibility by combining the docking score over multiple rigid receptor conformations, such as snapshots from a molecular dynamics simulation. Here, we evaluate a number of common snapshot selection strategies using a quality metric from stratified sampling, the efficiency of stratification, which compares the variance of a selection strategy to simple random sampling. We also extend the metric to estimators of exponential averages (which involve an exponential transformation, averaging, and inverse transformation) and minima. For docking sets of over 500 ligands to four different proteins of varying flexibility, we observe that, for estimating ensemble averages and exponential averages, many clustering algorithms have similar performance trends: for a few snapshots (less than 25), medoids are the most efficient, while, for a larger number, optimal (the allocation that minimizes the variance) and proportional (to the size of each cluster) a...
10th International Symposium on Software Metrics, 2004. Proceedings.
Page 1. v Table of Contents 10th International Symposium on Software Metrics METRICS 2004 Forewor... more Page 1. v Table of Contents 10th International Symposium on Software Metrics METRICS 2004 Foreword from the Organizing Committee _____ ix Organizing Committee _____ x Program Committee_____ xi Keynote _____xii ...
IEEE Transactions on Evolutionary Computation, 2011
Most applications of Genetic Programming (GP) involve the creation of an entirely new function, p... more Most applications of Genetic Programming (GP) involve the creation of an entirely new function, program or expression to solve a specific problem. In this paper we propose a new approach that applies GP to improve existing software by optimising its nonfunctional properties such as execution time, memory usage or power consumption. In general, satisfying non-functional requirements is a difficult task and often achieved in part by optimising compilers. However, modern compilers are in general not always able to produce semantically equivalent alternatives that optimise non-functional properties, even if such alternatives are known to exist: this is usually due to the limited local nature of such optimisations. In this paper we discuss how best to combine and extend the existing evolutionary methods of GP, Multi-Objective Optimisation and coevolution in order to improve existing software. Given as input the implementation of a function, we attempt to evolve a semantically equivalent version, in this case optimised to reduce execution time subject to a given probability distribution of inputs. We demonstrate that our framework is able to produce non-obvious optimisations that compilers are not yet able to generate on eight example functions. We employ a coevolved population of test cases to encourage the preservation of the function's semantics. We exploit the original program both through seeding of the population in order to focus the search, and as an oracle for testing purposes. As well as discussing the issues that arise when attempting to improve software, we employ rigorous experimental method to provide interesting and practical insights to suggest how to address these issues.
Proceedings of the 13th annual conference on Genetic and evolutionary computation - GECCO '11, 2011
This paper introduces a messy-GA for transition coverage of Simulink/StateFlow models. We introdu... more This paper introduces a messy-GA for transition coverage of Simulink/StateFlow models. We introduce a tool that implements our approach and evaluate it on three benchmark embedded system Simulink models. Our messy-GA is able to achieve statistically significantly better coverage when compared to both random search and to a commercial tool for Simulink/StateFlow model Testing.
2nd International Symposium on Search Based Software Engineering, 2010
Despite the large number of publications on Search-Based Software Testing (SBST), there remain fe... more Despite the large number of publications on Search-Based Software Testing (SBST), there remain few publicly available tools. This paper introduces AUSTIN, a publicly available SBST tool for the C language. The paper validates the tool with an empirical study of its effectiveness and efficiency in achieving branch coverage compared to random testing and the Evolutionary Testing Framework (ETF), which is used inhouse by Daimler and others for Evolutionary Testing. The programs used in the study consist of eight non-trivial, realworld C functions drawn from three embedded automotive software modules. For the majority of the functions, AUSTIN is at least as effective (in terms of achieved branch coverage) as the ETF, and is considerably more efficient.
2009 Testing: Academic and Industrial Conference - Practice and Research Techniques, 2009
Whilst there is much evidence that both concolic and search based testing can outperform random t... more Whilst there is much evidence that both concolic and search based testing can outperform random testing, there has been little work demonstrating the effectiveness of either technique with complete real world software applications. As a consequence, many researchers have doubts not only about the scalability of both approaches but also their applicability to production code. This paper performs an empirical study applying a concolic tool, CUTE, and a search based tool, AUSTIN, to the source code of four large open source applications. Each tool is applied 'out of the box'; that is without writing additional code for special handling of any of the individual subjects, or by tuning the tools' parameters. Perhaps surprisingly, the results show that both tools can only obtain at best a modest level of code coverage. Several challenges remain for improving automated test data generators in order to achieve higher levels of code coverage.
Recently there has been an upsurge of interest in both, Search-Based Software Testing (SBST), and... more Recently there has been an upsurge of interest in both, Search-Based Software Testing (SBST), and Dynamic Symbolic Execution (DSE). Each of these two approaches has complementary strengths and weaknesses, making it a natural choice to explore the degree to which the strengths of one can be exploited to offset the weakness of the other. This paper introduces an augmented version of DSE that uses a SBST-based approach to handling floating point computations, which are known to be problematic for vanilla DSE. The approach has been implemented as a plug in for the Microsoft Pex DSE testing tool. The paper presents results from both, standard evaluation benchmarks, and two open source programs.
2006 22nd IEEE International Conference on Software Maintenance, 2006
One approach to supporting program comprehension involves binding concepts to source code. Previo... more One approach to supporting program comprehension involves binding concepts to source code. Previously proposed approaches to concept binding have enforced nonoverlapping boundaries. However, real-world programs may contain overlapping concepts. This paper presents techniques to allow boundary overlap in the binding of concepts to source code. In order to allow boundaries to overlap, the concept binding problem is reformulated as a search problem. It is shown that the search space of overlapping concept bindings is exponentially large, indicating the suitability of sampling-based search algorithms. Hill climbing and genetic algorithms are introduced for sampling the space. The paper reports on experiments that apply these algorithms to 21 COBOL II programs taken from the commercial financial services sector. The results show that the genetic algorithm produces significantly better solutions than both the hill climber and random search.
Proceedings of the First International Workshop on Software Test Output Validation - STOV '10, 2010
Due to the frequent non-existence of an automated oracle, test cases are often evaluated manually... more Due to the frequent non-existence of an automated oracle, test cases are often evaluated manually in practice. However, this fact is rarely taken into account by automatic test data generators, which seek to maximise a program's structural coverage only. The test data produced tends to be of a poor fit with the program's operational profile. As a result, each test case takes longer for a human to check, because the scenarios that arbitrary-looking data represent require time and effort to understand. This short paper proposes methods to extracting knowledge from programmers, source code and documentation and its incorporation into the automatic test data generation process so as to inject the realism required to produce test cases that are quick and easy for a human to comprehend and check. The aim is to reduce the so-called qualitative human oracle costs associated with automatic test data generation. The potential benefits of such an approach are demonstrated with a simple case study.
This paper describes work on the application of optimization techniques in software engineering. ... more This paper describes work on the application of optimization techniques in software engineering. These optimization techniques come from the operations research and metaheuristic computation research communities. The paper briefly reviews widely used optimization techniques and the key ingredients required for their successful application to software engineering, providing an overview of existing results in eight software engineering application domains. The paper also describes the benefits that are likely to accrue from the growing body of work in this area and provides a set of open problems, challenges and areas for future work.
Proceedings of the 2007 international symposium on Software testing and analysis - ISSTA '07, 2007
Evolutionary testing has been widely studied as a technique for automating the process of test ca... more Evolutionary testing has been widely studied as a technique for automating the process of test case generation. However, to date, there has been no theoretical examination of when and why it works. Furthermore, the empirical evidence for the effectiveness of evolutionary testing consists largely of small scale laboratory studies. This paper presents a first theoretical analysis of the scenarios in which evolutionary algorithms are suitable for structural test case generation. The theory is backed up by an empirical study that considers real world programs, the search spaces of which are several orders of magnitude larger than those previously considered.
2009 International Conference on Software Testing, Verification, and Validation Workshops, 2009
This paper introduces the concept of test suite latency. The more latent a test suite, the more i... more This paper introduces the concept of test suite latency. The more latent a test suite, the more it is possible to repeatedly select subsets that achieve a test goal (such as coverage) without reapplying test cases. Where a test case is reapplied it cannot reveal new information. The more a test suite is forced to reapply already applied test cases in order to achieve the test goal, the more it has become 'worn out'. Test suite latency is the flipside of wear out; the more latent a test suite, the less prone it is to wear out. The paper introduces a theory of test suite latency. It presents results from the empirical study of latency, highlighting the need for latency enhancement. The paper also introduces a strategy and algorithms for improving latency and an empirical study of their effectiveness. The results show that local search is effective at improving the latency of a test suite.
Proceedings of the 2007 international symposium on Software testing and analysis - ISSTA '07, 2007
This project will build on the previous work by Mark Harman and Shin Yoo on Pareto Efficient Test... more This project will build on the previous work by Mark Harman and Shin Yoo on Pareto Efficient Test Case Selection pertaining to two objective formulation. It will introduce the concept of Pareto Efficient Test Case selection and the benefits of this approach. This project will illustrate the benefits of Pareto efficient test case selection with empirical studies of two objective formulations.
Abstract The introduction of multiple remote testers to apply a test or checking sequence introdu... more Abstract The introduction of multiple remote testers to apply a test or checking sequence introduces the possibility of controllability and observability problems. These problems can require the use of external coordination message exchanges among testers. It is desirable ...
Automated test data generation has remained a topic of considerable interest for several decades ... more Automated test data generation has remained a topic of considerable interest for several decades because it lies at the heart of attempts to automate the process of Software Testing. This paper reports the results of an empirical study using the dynamic symbolic execution tool, CUTE, and a search based tool, AUSTIN on five non-trivial open source applications. The aim is to provide practitioners with an assessment of what can be achieved by existing techniques with little or no specialist knowledge and to provide researchers with baseline data against which to measure subsequent work. To achieve this, each tool is applied 'as is', with neither additional tuning nor supporting harnesses and with no adjustments applied to the subject programs under test. The mere fact that these tools can be applied 'out of the box' in this manner reflects the growing maturity of Automated Test Data Generation. However, as might be expected, the study reveals opportunities for improvement and suggests ways to hybridize these two approaches that have hitherto been developed entirely independently.
International Conference on Program Comprehension, 2007
Search based software engineering (SBSE) is an approach to software engineering in which search b... more Search based software engineering (SBSE) is an approach to software engineering in which search based optimization algorithms are used to identify optimal or near optimal solutions and to yield insight. SBSE techniques can cater for multiple, possibly competing objectives and/or constraints and applications where the potential solution space is large and complex. Such situations are common in software engineering, leading
Journal of chemical information and modeling, Jan 24, 2018
Molecular docking can account for receptor flexibility by combining the docking score over multip... more Molecular docking can account for receptor flexibility by combining the docking score over multiple rigid receptor conformations, such as snapshots from a molecular dynamics simulation. Here, we evaluate a number of common snapshot selection strategies using a quality metric from stratified sampling, the efficiency of stratification, which compares the variance of a selection strategy to simple random sampling. We also extend the metric to estimators of exponential averages (which involve an exponential transformation, averaging, and inverse transformation) and minima. For docking sets of over 500 ligands to four different proteins of varying flexibility, we observe that, for estimating ensemble averages and exponential averages, many clustering algorithms have similar performance trends: for a few snapshots (less than 25), medoids are the most efficient, while, for a larger number, optimal (the allocation that minimizes the variance) and proportional (to the size of each cluster) a...
10th International Symposium on Software Metrics, 2004. Proceedings.
Page 1. v Table of Contents 10th International Symposium on Software Metrics METRICS 2004 Forewor... more Page 1. v Table of Contents 10th International Symposium on Software Metrics METRICS 2004 Foreword from the Organizing Committee _____ ix Organizing Committee _____ x Program Committee_____ xi Keynote _____xii ...
IEEE Transactions on Evolutionary Computation, 2011
Most applications of Genetic Programming (GP) involve the creation of an entirely new function, p... more Most applications of Genetic Programming (GP) involve the creation of an entirely new function, program or expression to solve a specific problem. In this paper we propose a new approach that applies GP to improve existing software by optimising its nonfunctional properties such as execution time, memory usage or power consumption. In general, satisfying non-functional requirements is a difficult task and often achieved in part by optimising compilers. However, modern compilers are in general not always able to produce semantically equivalent alternatives that optimise non-functional properties, even if such alternatives are known to exist: this is usually due to the limited local nature of such optimisations. In this paper we discuss how best to combine and extend the existing evolutionary methods of GP, Multi-Objective Optimisation and coevolution in order to improve existing software. Given as input the implementation of a function, we attempt to evolve a semantically equivalent version, in this case optimised to reduce execution time subject to a given probability distribution of inputs. We demonstrate that our framework is able to produce non-obvious optimisations that compilers are not yet able to generate on eight example functions. We employ a coevolved population of test cases to encourage the preservation of the function's semantics. We exploit the original program both through seeding of the population in order to focus the search, and as an oracle for testing purposes. As well as discussing the issues that arise when attempting to improve software, we employ rigorous experimental method to provide interesting and practical insights to suggest how to address these issues.
Proceedings of the 13th annual conference on Genetic and evolutionary computation - GECCO '11, 2011
This paper introduces a messy-GA for transition coverage of Simulink/StateFlow models. We introdu... more This paper introduces a messy-GA for transition coverage of Simulink/StateFlow models. We introduce a tool that implements our approach and evaluate it on three benchmark embedded system Simulink models. Our messy-GA is able to achieve statistically significantly better coverage when compared to both random search and to a commercial tool for Simulink/StateFlow model Testing.
2nd International Symposium on Search Based Software Engineering, 2010
Despite the large number of publications on Search-Based Software Testing (SBST), there remain fe... more Despite the large number of publications on Search-Based Software Testing (SBST), there remain few publicly available tools. This paper introduces AUSTIN, a publicly available SBST tool for the C language. The paper validates the tool with an empirical study of its effectiveness and efficiency in achieving branch coverage compared to random testing and the Evolutionary Testing Framework (ETF), which is used inhouse by Daimler and others for Evolutionary Testing. The programs used in the study consist of eight non-trivial, realworld C functions drawn from three embedded automotive software modules. For the majority of the functions, AUSTIN is at least as effective (in terms of achieved branch coverage) as the ETF, and is considerably more efficient.
2009 Testing: Academic and Industrial Conference - Practice and Research Techniques, 2009
Whilst there is much evidence that both concolic and search based testing can outperform random t... more Whilst there is much evidence that both concolic and search based testing can outperform random testing, there has been little work demonstrating the effectiveness of either technique with complete real world software applications. As a consequence, many researchers have doubts not only about the scalability of both approaches but also their applicability to production code. This paper performs an empirical study applying a concolic tool, CUTE, and a search based tool, AUSTIN, to the source code of four large open source applications. Each tool is applied 'out of the box'; that is without writing additional code for special handling of any of the individual subjects, or by tuning the tools' parameters. Perhaps surprisingly, the results show that both tools can only obtain at best a modest level of code coverage. Several challenges remain for improving automated test data generators in order to achieve higher levels of code coverage.
Recently there has been an upsurge of interest in both, Search-Based Software Testing (SBST), and... more Recently there has been an upsurge of interest in both, Search-Based Software Testing (SBST), and Dynamic Symbolic Execution (DSE). Each of these two approaches has complementary strengths and weaknesses, making it a natural choice to explore the degree to which the strengths of one can be exploited to offset the weakness of the other. This paper introduces an augmented version of DSE that uses a SBST-based approach to handling floating point computations, which are known to be problematic for vanilla DSE. The approach has been implemented as a plug in for the Microsoft Pex DSE testing tool. The paper presents results from both, standard evaluation benchmarks, and two open source programs.
2006 22nd IEEE International Conference on Software Maintenance, 2006
One approach to supporting program comprehension involves binding concepts to source code. Previo... more One approach to supporting program comprehension involves binding concepts to source code. Previously proposed approaches to concept binding have enforced nonoverlapping boundaries. However, real-world programs may contain overlapping concepts. This paper presents techniques to allow boundary overlap in the binding of concepts to source code. In order to allow boundaries to overlap, the concept binding problem is reformulated as a search problem. It is shown that the search space of overlapping concept bindings is exponentially large, indicating the suitability of sampling-based search algorithms. Hill climbing and genetic algorithms are introduced for sampling the space. The paper reports on experiments that apply these algorithms to 21 COBOL II programs taken from the commercial financial services sector. The results show that the genetic algorithm produces significantly better solutions than both the hill climber and random search.
Proceedings of the First International Workshop on Software Test Output Validation - STOV '10, 2010
Due to the frequent non-existence of an automated oracle, test cases are often evaluated manually... more Due to the frequent non-existence of an automated oracle, test cases are often evaluated manually in practice. However, this fact is rarely taken into account by automatic test data generators, which seek to maximise a program's structural coverage only. The test data produced tends to be of a poor fit with the program's operational profile. As a result, each test case takes longer for a human to check, because the scenarios that arbitrary-looking data represent require time and effort to understand. This short paper proposes methods to extracting knowledge from programmers, source code and documentation and its incorporation into the automatic test data generation process so as to inject the realism required to produce test cases that are quick and easy for a human to comprehend and check. The aim is to reduce the so-called qualitative human oracle costs associated with automatic test data generation. The potential benefits of such an approach are demonstrated with a simple case study.
This paper describes work on the application of optimization techniques in software engineering. ... more This paper describes work on the application of optimization techniques in software engineering. These optimization techniques come from the operations research and metaheuristic computation research communities. The paper briefly reviews widely used optimization techniques and the key ingredients required for their successful application to software engineering, providing an overview of existing results in eight software engineering application domains. The paper also describes the benefits that are likely to accrue from the growing body of work in this area and provides a set of open problems, challenges and areas for future work.
Proceedings of the 2007 international symposium on Software testing and analysis - ISSTA '07, 2007
Evolutionary testing has been widely studied as a technique for automating the process of test ca... more Evolutionary testing has been widely studied as a technique for automating the process of test case generation. However, to date, there has been no theoretical examination of when and why it works. Furthermore, the empirical evidence for the effectiveness of evolutionary testing consists largely of small scale laboratory studies. This paper presents a first theoretical analysis of the scenarios in which evolutionary algorithms are suitable for structural test case generation. The theory is backed up by an empirical study that considers real world programs, the search spaces of which are several orders of magnitude larger than those previously considered.
2009 International Conference on Software Testing, Verification, and Validation Workshops, 2009
This paper introduces the concept of test suite latency. The more latent a test suite, the more i... more This paper introduces the concept of test suite latency. The more latent a test suite, the more it is possible to repeatedly select subsets that achieve a test goal (such as coverage) without reapplying test cases. Where a test case is reapplied it cannot reveal new information. The more a test suite is forced to reapply already applied test cases in order to achieve the test goal, the more it has become 'worn out'. Test suite latency is the flipside of wear out; the more latent a test suite, the less prone it is to wear out. The paper introduces a theory of test suite latency. It presents results from the empirical study of latency, highlighting the need for latency enhancement. The paper also introduces a strategy and algorithms for improving latency and an empirical study of their effectiveness. The results show that local search is effective at improving the latency of a test suite.
Proceedings of the 2007 international symposium on Software testing and analysis - ISSTA '07, 2007
This project will build on the previous work by Mark Harman and Shin Yoo on Pareto Efficient Test... more This project will build on the previous work by Mark Harman and Shin Yoo on Pareto Efficient Test Case Selection pertaining to two objective formulation. It will introduce the concept of Pareto Efficient Test Case selection and the benefits of this approach. This project will illustrate the benefits of Pareto efficient test case selection with empirical studies of two objective formulations.
Abstract The introduction of multiple remote testers to apply a test or checking sequence introdu... more Abstract The introduction of multiple remote testers to apply a test or checking sequence introduces the possibility of controllability and observability problems. These problems can require the use of external coordination message exchanges among testers. It is desirable ...
Automated test data generation has remained a topic of considerable interest for several decades ... more Automated test data generation has remained a topic of considerable interest for several decades because it lies at the heart of attempts to automate the process of Software Testing. This paper reports the results of an empirical study using the dynamic symbolic execution tool, CUTE, and a search based tool, AUSTIN on five non-trivial open source applications. The aim is to provide practitioners with an assessment of what can be achieved by existing techniques with little or no specialist knowledge and to provide researchers with baseline data against which to measure subsequent work. To achieve this, each tool is applied 'as is', with neither additional tuning nor supporting harnesses and with no adjustments applied to the subject programs under test. The mere fact that these tools can be applied 'out of the box' in this manner reflects the growing maturity of Automated Test Data Generation. However, as might be expected, the study reveals opportunities for improvement and suggests ways to hybridize these two approaches that have hitherto been developed entirely independently.
International Conference on Program Comprehension, 2007
Search based software engineering (SBSE) is an approach to software engineering in which search b... more Search based software engineering (SBSE) is an approach to software engineering in which search based optimization algorithms are used to identify optimal or near optimal solutions and to yield insight. SBSE techniques can cater for multiple, possibly competing objectives and/or constraints and applications where the potential solution space is large and complex. Such situations are common in software engineering, leading
Uploads
Papers by John Clark