2017 International Conference on Communication Technologies (ComTech)

Comparative Analysis of Automated Load Testing Tools:

Apache JMeter, Microsoft Visual Studio (TFS),
LoadRunner, Siege

Rabiya Abbas
Department of Software Engineering
Bahria University Islamabad, Pakistan
Dr. Shahid Nazir Bhatti
rabiyaabbas786@gmail.com Department of Software Engineering
Bahria University Islamabad, Pakistan
Zainab Sultan
Department of Software Engineering snbhattii.buic@bahria.edu.pk
Bahria University Islamabad, Pakistan
zaini.1984@gmail.com

Abstract-Software testing is the process of testing,

verifying and validating the user’s requirements.
During whole software development, testing is an
ongoing process. Black Box, White Box testing
and grey box testing are the three main types of
software testing. In Black box testing, user doesn’t
know intrinsic logics and design of system. In
white box testing, Tester knows the intrinsic logic
of code. In Grey box testing, Tester has little bit
knowledge about the internal structure, code and Fig. 1 Software testing
working of the system. It is commonly used in case
of Integration testing. Load testing is the type of Manual testing is executed by the testers. First of
testing which helps us to analyze the performance all a written test plan is created, and followed by
of the system under heavy load or under Zero testers that provides a guideline through different
load. With the help of a automated load Testing steps. But there are a lot of problems faced by testers
Tools, we can do it in better way. The intention for like it is very time taking and consuming, no
writing this research is to carry out a comparison reusability, has no scripting feature, much human
of four automated load testing tools i.e. Apache effort required, and still major or minor bugs remain
JMeter, HP LoadRunner, Microsoft Visual Studio unexplored. Therefore to cover all types of errors and
(TFS), Siege based on certain criteria i.e. test bugs automation testing has introduced that explores
scripts generation , plug-in support, result reports, all the issues exist in manual testing [11]. All
application support and cost . The main focus is Automation testing tools test software in less time,
to study and analyze these load testing tools and produce more reliable, repeatable, and reusable final
identify which tool is best and more efficient [10]. product.
We assume this comparative analysis can help in Load testing is activated when we steadily raise
selecting the most appropriate tool and motivates the load upon a system until it reaches a target load.
the use of open source load testing tools. Usually this is the maximum load, average or even
zero load. The goal of a load testing is to discover
Keywords— Testing, manual testing, automated functional and performance issues of a system under
testing, testing tools, load testing, stress test. load. Load testing is appropriate for testing the
performance of web site, and its framework[18].
I. INTRODUCTION The intention for writing this research is to carry
The objective of software testing is to find defects, out a comparison of four load testing tools i.e. Apache
errors and bugs in a software, system or product. JMeter, LoadRunner, Microsoft Visual Studio (TFS),
Software testing is characterized into manual testing Siege based on some parameters. This research paper
and automation testing. is divided into different sections. Section 1 consists of
978-1-5090-5984-3/17/$31.00 ©2017 IEEE. Personal use of this
material is permitted. Permission from IEEE must be obtained for all
other uses, in any current or future media, including reprinting/repub-
lishing this material for advertising or promotional purposes, creating
new collective works, for resale or redistribution to servers or lists, or
reuse of any copyrighted component of this work in other works.
39
introduction. Literature review is discussed in Section comparison they concluded that “Ranorex” is the best
2. Research methodology is presented in section 3 .In tool for web based applications [10].Monika Sharma,
section 4, we present evaluation study. In section 5, Vaishnavi S. Iyer, Sugandhi Subramanian, and
and comparison table of automated testing tools is Abhinandhan Shetty in their paper focuses on
presented. In section 6, results and analysis of study is comparing load testing tools-Apache JMeter, HP
presented and in section 7, on the basis of research, LoadRunner, WebLOAD, and The Grinder on the
conclusion is presented. basis of different parameters[11].

II. LITERATURE REVIEW III. RELATED WORK

Thissection presents the literature review of Comparison between load testing tools has been
research topic.Farmeena Khan Et.al in their paper done by many authors. Vandana E. [13] [20], have
describes and compares three main testing techniques: done comparative study of testing tools which are
“White box, Black box and Grey box testing”. jmeter and load runner. They described advantages
Authors’ presents the future changes in software and disadvantages of both tools and recommended
testing industry due to new technologies like SOA that Jmeter is much better than Load Runner because
(Service Oriented Architecture) & mobile it has clean UI that offers much simplicity. Bhatti
technologies etc[3]. Niranjanamurthy et .al in their Et.al [19], described number of load testing tools for
paper discusses testing terminologies used in testing, test web applications. The testing tools they discussed
levels of testing, analysis of automated and manual are Load Runner, NeoLoad, WAPT, Soasta Cloud
techniques, comparison of Selenium and QTP Tools, Test, LoadStorm, Loadster, Apache, JMeter,
comparison of “White box, Black box and Grey box HTTPERF, LoadUI, and LoadImpact. They analyzed
testing techniques” [2]. Taraq Hussain et al in their that among all tools to test a web application,
research mentions that testing can never completely NEOLOAD is best for load testing due to its visual
diagnoses all errors of a software but it provides programming and its script less design. Rina [21]
evaluation techniques which helps the tester to analyzed the NeoLoad, WAPT and Loadster tools on
diagnose a problem. After comparison they show that different browsers and compared the results of their
the White Box Testing is best suitable for software performance. One site has tested on above three tools
reliability[4]. “The Growth of Software Testing” is for performance. The comparison they done provides
written by “David Gelperin and Bill Hetzel”. In this a better understanding for selecting the best tool
paper, authors describes the evolution of different according to requirements and possibilities, however
software testing process models , their merits and they concluded that It is difficult to compare tools
demerits due to which some of these are failed . From because many parameters values are not considers in
1956 to present, different software testing models are all tools. Upadhyay [22]compared some specific
discussed; Changes in these models are evaluated[14]. performance testing tools for their usability and
Harpareet Kaur and Dr.Gagan Gupta in their paper effectiveness. WAPT and RANOXEX performance
discussed the two ways of testing: Manual testing and testing tools inferences, implications and results have
Automation testing. In this research paper, they been presented and discussed. Different attributes,
discuss the parameters of “Selenium 2.0.0, Quick Test their ability to compare the results, test cases
Professional 10.0, and TestComplete 9.0”. These three documentation ability and regression testing
tools comparison is based on different specification performance ability have been compared. Dart Et.al
and parameters. After analysis, researchers concluded compared software web tools in terms of their
that anyone can choose the testing tool on the basis of dynamic test generation ability [18]. A survey has
budget and nature of software that has to be tested. been presented on static and dynamic testing analysis.
Researchers found that QTP is more suitable among Sufiani [23] compared different performance testing
all that three tools[6]. tools response time and justified these differences
Neha Bhatia in her paper discussed manual testing include architecture and simulation mechanism.
and automation testing. If the requirements are
continuously changing and regression testing is IV. RESEARCH METHODOLOGY
needed to perform repeatedly, then automated testing Testing is an important and critical part of the
is more suitable in that environment. In this paper, SDLC. In recent times different automated software
researcher discussed different automation tools[7]. testing tools are available in market. Several studies
Neha Dubey et al in their paper compare and study are available in which comparisons of different testing
concepts and characteristics of two “software tools are done. According to our observations, there is
automated tools Ranorex and the Automated QA no comparative analysis on the load testing tools, such
TestComplete” that are based on some criteria. After as “JMeter, Siege, LoadRunner, and Microsoft Visual

40
Studio (TFS)”. In this paper, we compare these load
testing tools on the basis of different parameters. 3. Siege
A. A. Automated software testing tools “Siege” was developed and implemented by
A brief explanation and comprehensive account of “Jeffrey Fulmer” as Webmaster for Armstrong World
automated software testing tools is taken here in this Industries. It is a software load testing tool which is
section. very productive in detecting the performance of
system when load exists [13]. Siege executing
1. Apache JMeter commands and Test summary reports is shown in fig
4.
Apache JMeter is an “open-source testing tool”
developed by “Apache Software Foundation (ASF)”.
JMeter’s main function is to load test client/server.
Moreover, JMeter is used in regression testing by
generating test scripts [12]. JMeter provides offline
reporting of test results. JMeter test reports are shown
in fig 2.

Fig.4 Siege executing commands &test report summary

4. Microsoft visual studio (TFS)

“Team Foundation Server (TFS)” is a load

testing tool which facilitate with source code
management, Project management, Requirement
management, reporting, testing capabilities,
automated builds, lab management, [17] TFS test
Summary reports are shown in fig 5.

Fig.2. JMetertest reports

2. LoadRunner
HPE LoadRunner is an “automation testing tool”
from “Hewlett Packard” enterprise [15]. HP
LoadRunner software testing tool helps you to detect
and prevent from software performance issues by
identifying bottlenecks [16]. HP LoadRunner
Scripting and test report summary is shown in Fig 3.

Fig.5. TFS test summary report

V. EVALUATION STUDY
Now a day’s different open source and profitable
Load testing tools are available in the market. For this
comparative study, we are using the latest and running
versions of “Apache JMeter, LoadRunner, Microsoft
Visual Studio (TFS), and Siege”. Through these tools
Fig.3. HP LoadRunner scripting & test summary reports

41
we test the Bahria University Islamabad .It has built-in testing capabilities whether there are
campusWebsite (http://www.buic.edu.pk/) and for 100 parallel users or 1000s, it is easy to test according
Siege, we test telecommunication company website to user requirements but it can only supports
(www.telenor.pk). Comparison between these four Windows OS and it has high licensing cost. Apache
tools is made on the basis of list evaluation parameters JMeter is best option as it is free of cost (see fig 7 in
with the explanation. this). It takes more time on one time installation but it
has broad set of options for result analysis and it is
VI. RESULT AND ANALYSIS OF STUDY good for different tests to be run simultaneously .It
For assessment of the parameters, we use 3-point has several plugins which raise its testing capabilities.
scale in a graph i.e. 3, 2, 1 as Best, Average, and
Worst respectively. Different value for different TABLE I. EVALUATION PARAMETERS WITH EXPLANATION
parameters with selected automated tools is verified.
The calculated value of parameters is used for
conclusion and investigation of this comparative
study. The overall comparison based graph for these Appropriate Parameters Explanation
four automated load testing tools is shown in Fig 6. Recording Efficiency For handling the application
which is to be tested.

Generating scripts Generating the interrelated

capability scripts.

Test Result Reports Efficient investigation of test

generation scripts

Cross platform Operating systems on which

tool operates on

Easy to learn GUI(GRAPHICAL USER

INTERFACE)

Application support Applications that are supported

by testing tool
Fig.6. Comparison graph based on selected parameter results Scripting languages Languages that are used for
scripting
VII. RESULTS AND FINDINGS
Plugin support Testing tools either support
After comparison and analysis, we conclude that plugins or not
anyone can choose the testing tool on the basis of
Licensing Cost Reasonable or low cost
budget and nature of software that has to be tested.
Apache JMeter, LoadRunner, Microsoft Visual Studio
(TFS), Siege all four are good tools for test
automation. We take two different websites Bahria
University Islamabad campus Website
(http://www.buic.edu.pk/) and for Siege, we test
telecommunication company website
(www.telenor.pk) because Siege cannot test live sites.
Each tool has its own benefits and drawbacks too; a
detail analysis in this context is in Table I below. It is
to be noted for Telenor System we did have the access
code available but for the Bahria University we did
not have access to the code.
Siege can reduce the cost as it is open source but it
has limited options to be used as it is command line
tool and it sometimes generate inaccurate result.
HP LoadRunner is best for performance checking
when load found. It can handle multiple users at same Fig.7. Pie chart showing JMeter as highest used
time butit has some configuration or installation
problems across firewalls and its licensing cost is
high. Microsoft Visual Studio (TFS) is user friendly

42
 VIII. COMPARISON OF AUTOMATION TESTING
TOOLS
In this section, we compare the automation testing
tools. This comparison is beneficial for the
testers/researchers (technical stake holders) to choose
the more appropriate load test tool as per
requirements. Table II presents comparison of
automated testing tools i.e. Apache JMeter,
LoadRunner, Microsoft Visual Studio (TFS), and
Siege on the basis of different parameters
(characteristics).

TABLE II.COMPARISON OF WEB SERVICE TESTING TOOLS

FEATURES JMeter Siege LoadRunner Microsoft Visual

Studio(TFS)
Cost It is free or no renovation cost It is free of cost. License is expensive About $2K licensing cost
for this tool.
Application Static and dynamic resources, Only used for testing Websites and other Website & Other
support Web services and databases. websites. applications. applications.

Scripting Javascript ,BeanShell Skrit Citrix, ANSI C, .Net and PowerShell ,Perl
language Java

Cross Supports Windows Supports UNIX, AIX, Supports Microsoft Supports Windows 7,
Platforms PC/MAC/UNIX Platforms. BSD, Solaris. Windows and LINUX OS. Windows Vista, Windows
Server 2008 or later
operating systems.
Plugin support It has several plugins which Siege has small no. of It has several plugins which It has several plugins which
raise its testing capabilities. plugins. raise its testing capabilities. raise its testing capabilities.

Interface GUI Command Line Interface GUI GUI

Benefits It provides GUI and has many It has faster setup. It is best for performance It is simple to use.
features that can be used while checking where there is
testing. It is good for quick results. actual load. It has inherent testing
capabilities.
It has vast set of options for It can handle large no. of
result analysis. users at the same time. It uses graphical illustrations
in reports.
It is good for different tests to be It can also checks network
run simultaneously. and server resources for Whether there are 100
improving performance. parallel users or 1000s, it is
It gives accurate results. easy to test as per
It automatically trace
requirements.
client/server performance
while testing

Drawbacks JMeter takes more time to setup It has limited options to be It has some configuration or It only supports Windows
as it involves many steps. used as it is command line installation issues OS.
tool. across firewalls.
It sometimes generate It has high licensing cost.
inaccurate result.
Report JMeter supports dashboard Reports total no.of It allow user to convert In this, reports are generated
Generation report generation to get transactions ,server performance report into in SQL Server Reporting
graphical illustrations. response etc. word, excel, pdf etc. Services.

relies on the software development. Thus the quality

IX. CONCLUSION issue related to the software’s industry becomes more
Improving software quality and performance has important, apparent and more technical also
become a priority for almost every organization that considering the user’s requirements in this aspect.
Software systems have to ensure consistent and bug

43
free execution at a rapid pace every time they are used [9] Raj Kumar, Manjit Kaur, “Comparative Study of
especially in web based development. Automated Testing Tools: Test Complete and Quick
In this work we have performed a thorough and Testpro”, Intl. Journal of Computer Applications
(0975-8887), Volume 24, No. 1, June 2011.
comprehensive comparison and analysis using
different tools/ technologies available for testing (load [10] Monika Sharma, , Abhinandhan Shetty, Sugandhi
testing as case scenario). After a through analytical Subramanian, Vaishnavi S. Iyer,“A Comparative
Study on Load Testing Tools” , Int. Journal of
review of these different tools mentioned in sections Innovative Research in Computer and Communication
IV and V for Load testing, we summarize here that Engineering ,Vol. 4, Issue 2, February 2016
anyone can choose the testing tool but on the basis of
[11] Neha Dubey, Mrs. Savita Shiwani , “Studying and
budget, time and nature of software system under Comparing Automated Testing Tools; Ranorex and
consideration that has to be tested. Besides Each tool TestComplete” , (IJECS) , Volume 3, Issue 5, Pp.
have its own benefits and drawbacks, and have to 5916-5923
keep in queue when performing anyone of the [12] Sanjay Tyagi , Pooja Ahlawat , “A Comparative
mentioned testing strategies (or any other). Apache Analysis of Load Testing Tools Using Optimal
JMeter, LoadRunner, Microsoft Visual Studio (TFS), Response Rate” , (IJARCSSE), Volume 3, Issue 5,
Siege all four are good tools for test automation. But May 2013.
we have shown that JMeter provides better results [14] Dr.K..V..K .K Prasad , Software Testing Tools:
than any other tested tools (techniques) as it is a ratio Covering WinRunner, SilkTest, LoadRunner, JMeter,
scale methodology, and also includes a consistency TestDirector and QTP with Paperback , 2007 .
check. [15] Ibrahima Kalil Toure, Abdoulaye Diop, Shariq
In future work, with the access to code (for web Hussain and Zhaoshun Wang, “Web Service Testing
projects) the applications and values (attributes) of Tools: A Comparative Study”, IJCSI Int. Journal of
these tools can be estimated, especially in case of Computer Science Issues, Vol. 10, 2013.
stress testing while performing Load testing. As stress [16] Daniel A. Menasce, “Load Testing of Websites”,
testing evaluate the system when stressed to its limits http:/computer.org/internet, IEEE Internet Computing,
PP.70- 74, 2002.
over a short period of time and that following testing
is especially important for systems that usually [17] Li Xiao-jie, Zhang Hui-li and Zhang Shu.“Research of
Load Testing and Result Application Based on
operate below maximum capacity but are severely
LoadRunner”, National Conference on Information
stressed at certain times of peak demand. Technology and Computer Science, 2012.
[18] Sneha Khoria, Pragati Upadhyay, “Performance
Evaluation and Comparison of Testing Tools”, VSRD
REFERENCES Int. Journal of Compt. Science & IT, Vol. 2, 2012.
[1] Rapinder Singh ,Manprit Kaur ,”A review of software [19] Sinha M, and Arora A., “Web Application Testing: A
testing techniques”,( IJEEE), ISSN 0974-2174, Review on Techniques, Tools and State of Art”,
Volume 7, pp. 463, 2014. (IJSER), Volume 3, Issue 2, 2012.
[2] Nitesh S N, Niranjanamurthy M, Balaji Sriraman [20] [20] Vandana Chandel, Shilpa Patial Sonal Guleria,
,Nagesh S N, “Comparative Study of Software Testing “ComparativeStudy of Testing Tools: Apache JMeter
Techniques “, IJCSMC, Vol. 3, Issue. 10, October and Load Runner”,IJCCR, VOLUME 3 ISSUE 3 May
2014, pg.151 – 158. 2013.
[3] Farmeena Khan, Mohammad Ehmar Khan ,”A [21] Sandeep Bhatti, Raj Kumari, “Comparative Study of
comparative study of white box , black box , grey box LoadTesting Tools”, ijircce, Vol. 3, Issue 3, March
testing techniques” , (IJACSA) , Vol. 3, No.6, 2012. 2015.
[4] Taraq Hussain, Dr.Satyaveer Singh, “A Comparative [22] Rina, Sanjay Tyagi, “A Comparative Study of
Study of Software Testing Techniques Viz. White Box PerformanceTesting Tools”, Volume 3, Issue 5, May
Testing Black Box Testing and Grey Box Testing”, ( 2013.
IJAPRR) , ISSN 2350-1294. [23] Dr. S. M. Afroz, N. Elezabeth Rani and N. Indira
[5] Kamna Solanki, Jyoti, , “A Comparative Study of Priyadarshini, “Web Application– A Study on
Five Regression Testing Techniques: A Survey “, ComparingSoftware Testing Tools”, International
International Journal of Scientific & Technology Journal of ComputerScience and Telecommunications,
Research (IJSTR), Volume 3, Issue 8, August 2014. Volume 2, Issue 3, June2011.
[6] Neha Bhateja , “A Study on Various Software [24] Muhammad Dhiauddin Mohamed Suffiani, Fairul
Automation Testing Tools” ,(IJACSA), Volume 5, RizalFahrurazi, “Performance Testing: Analyzing
Issue 6, June 2015 . Differences ofResponse Time between Performance
[8] Richa Rattan, “Comparative Study Of Automation Testing Tools”, inproceeding of International
Testing Tools: Quick Test Professional & Selenium”, Conference on Computer &Inf. Science (ICCIS)
IJCSIT, Vol. 3, No. 6 June 2013. 2012.

44