International Journal of Computer Applications Technology and Research

Volume 6Issue 1, 19-22, 2017, ISSN:-23198656

Performance Lifecycle in Banking Domain

Vijay Datla

Abstract: Performance assurance and testing plays a key role in complex applications and is an essential element of the application
development life cycle. This case study is about integrating performance at a large national bank. Learn how custom monitoring, Six
Sigma techniques, performance testing, and daily production reports played an important role in identifying production issues. This
paper illustrates and examines the challenges and successes of performance planning, testing, analysis, and optimization after the
release of X Banks CRM application.

Keywords: Performance, Lifecycle, Banking, Vijay Datla, Vijay, Datla

The rollout plan for X Bank called for 500 branches every 6
1. INTRODUCTION months until reaching the goal of 2,000 branches.
Software Performance Lifecycle (SPL) is an approach that can The CRM application technologies consisted of an ASP front
be applied to all types of technologies and industries. This end on IIS Web servers, C++ middle tier on MO Series and an
solution allows the true possibilities of the system and Oracle database.
software under test to be examined. It allows for precise
planning and budgeting. The SPL approach can begin at any
stage of the Software Development Life Cycle (SDLC) and
2.1 Planning & Setup Phase
will mature as the wheel turns (or the life cycle progresses). The rst step in the SPL process is planning, this entails
However, the earlier performance is evaluated, the sooner planning for the entire process, creating a performance test
design and architectural flaws can be addressed and the faster plan, and setting up the performance environment. The initial
and cheaper the software development life cycle becomes. responsibilities of the two Performance Engineers was to
The wheel in this case is the development life cycle as a interact with the bank resources, business analysts,
whole, not just one application release but all releases from developers, system administrators, database administrators,
the start of the application. The SPL steps include planning, application engineers, and others to gather enough
testing, monitoring, analysis, tuning and optimization. These information to devise a performance test plan. To help create
steps will be discussed in conjunction with this case study. the performance test plan, we needed to fully understand the
The idea is to begin the SPL approach at any point on the application behavior at X Bank. First, we sat down with
wheel (or development life cycle). As the wheel turns the SPL business analysts to understand the major pain points and
approach will position itself to start earlier and earlier in the learn the application usage at the bank. We also made trips out
development life cycle for future release levels. In the to different branches and spoke to actual end users of the
example explained below, SPL started at the end of the first application to analyze their user experience and performance
release of the application to be tested. Due to the late concerns.
introduction, we ran into different issues and problems, but Next, we needed to get a better understanding of the database
we jumped on and started the performance lifecycle. The volumes in production to allow us to properly populate the
introduction of the SPL approach will save significant time perfom1ance test lab database. To do this we received
and money, while ensuring end user satisfaction. Our database row counts from the production database
organization used these set of techniques and procedures and administrator for the previous three months, and we used that
called it Software Performance Lifecycle (SPL) information in conjunction with the growth projections to
appropriately populate the performance test lab database.
All the information gathered during the planning phase
2. CASE STUDY enabled us to get a better understanding and positioned us to
This performance case study involves a major national bank create a performance test plan. The test plan included actual
with over 2,000 branches, ctionally named X Bank for this use case! business process steps, SLA goals, database sizing
study. The bank was facing performance issues with various information, performance lab specications, and exit criteria
portions of their CRIVI application. They were experiencing for this round.
high response times, degraded throughput, poor scaling Next we set out to create an onsite performance test lab. The
properties, and other issues. This caused un-acceptance from test lab included load testing servers, application servers, a
their end users and customer base. database server, an Integrated Architecture (IA) server, and a
During the first round of implementing performance at X host system. The perfom1ance lab environment mimicked
Bank, our responsibilities as consultants was to help elevate production in terms of the application servers but lacked in
their performance issues. We were in charge of managing and terms of number of CPUs on the IA Server. The load testing
executing the Software Performance Life Cycle, which software of choice was LoadRunner and Win Runner. We
included items such as planning, scripting, testing, analyzing, utilized one load testing controller, two load generators, and
tuning, and managing the performance lab. As the application two end user workstations. The workstations were used in
grew in size, so did the team. lt started with two Senior conjunction with WinRunner to truly understand end user
Performance Engineers and evolved to a Senior Performance experience under load. Lastly a custom dashboard application
Engineer, a Senior Application Developer, two Scripting was written to monitor application transaction response times
Resources, Environment Team Resource, and a part-time at the web and application tiers and to provide server statistics
Database Admin. information
The production environment consisted of five ACS The last step was to install the application on the servers and
(Application Combined Servers) and one NT database server. load the appropriate data volumes into the performance
database server. We used Perl scripts to generate the

www.ijcat.com 19
 International Journal of Computer Applications Technology and Research
Volume 6Issue 1, 19-22, 2017, ISSN:-23198656

appropriate volumes of test data, which gave us the exibility was the front-end time which does not include GUI rendering
to increase volume as needed. time (shown as dotted line below). As the front end time
includes the back-end times, the difference provided us with
just the web server response time, giving us another data point
for our analysis. The last two entries provided the exact
request size and response size of each transaction, thus
allowing us to verify the correct data sizes for the appropriate
business processes.

2.3 Tuning & Optimization Phase

The next steps in the SPL include tuning and optimization of
the application. Through an iterative testing cycle, which
included testing, database resets, monitoring, and analysis we
made signicant changes to the application. These changes
included items such as the login cache mechanism, which
Above picture shows Performance Setup originally cached unnecessary content, and changes to a third-
party party DLL that was affecting database queries. We also
2.2 Performance Testing & Monitoring changed the application conguration le to start the correct
number of application server instances, which in turn
Phase maximized the server resources.
At this point, we had devised a performance test plan, had an
understanding of the performance concerns, and created a
performance test lab. Now the next step was to begin
2.4 Results
performance testing and application system monitoring. We measured the round trip application response times a
several different ways. As we ran our tests we utilized
The rst step was to create test scripts for all the outlined
LoadRunner to give us end-to-end response time, which did
business process in our test plan using
not include GUI rendering. So we utilized WinRunner to get
LoadRunner. The scripting process included recording the
script, enhancing the script and running conguration true end-to-end response times which included GUI rendering
time. The way we approached this was by running a full load
audits. Enhancing the scripts included items such as
with LoadRunner and placing two workstations on a emulated
parameterization, correlation, verication, logic, extra
branch circuit running WinRunner. The WinRunner statistics
coding, and anything else that was required to mimic a real
end user. Recording, enhancing, and single playbacks were all provided response times that included the WAN emulation
Of a remote branch circuit, as well as GUI rendering. This in
performed under the LoadRunner Vugen utility. The Vugen
tum provided us with a true user experience and end-to-end
utility is LoadRunner's scripting engine. The next step was to
response times. Also to allow the business users to understand
run conguration audits, with multiple users, using the
LoadRunner Controller for all dened test scripts. The the feel of the application under load we had them walk
through the application while we were running a full load test.
LoadRunner Controller is the utility utilized to generate
The business users performed the business process steps from
virtual user traffic. The reason for running conguration
a lab that emulated remote branches. Lastly the business users
audits was to make sure issues, such as data dependencies and
concurrency problems, did not arise in multiple user mode. provided response times from the branch circuit lab when no
After we completed the conguration audits, we executed load was emulated on the system. This provided a true picture,
meaning the best we can expect from our end user response
load tests separately against each of the ve AC8 servers. The
times on system. The changes made allowed us to drastically
separate tests were conducted to make sure all servers were
behaving exactly the same in terms of response times, reduce response times for most transactions while increasing
throughput, and utilization. Next we ramped up to two servers the throughput of the application
Before we started SPL process at bank, they are facing
where we calculated BP (Business Process) throughput and
performance issues at hundreds of branches.
compared them to our goals. Other testing goals, used for
comparison, included concurrent virtual user mark, business Response times were high for key transactions, and marketing
process throughput for other business processes, and days were not acceptable by end users. X Bank did not truly
understand the application scaling properties, capacity
transaction response times. After the two AC5 server tests, we
planning, hardware sizing, nor was the bank able to identify
scaled up to ve AC8 servers, and were not able to meet all
the performance issues they were having. But after the rst
our throughput goals.
pass through of the SPL process of planning, testing,
monitoring, analysis, tuning, and optimization the bank was
The logs showed us which application transaction was being
meeting SLAs, had proper hardware sizing in place, decreased
executed, along with specic information of each transaction.
costs, and gain condence on marketing days.
lt showed the back-end response time which is depicted as a
solid line in the diagram below. The next entry in the log le

www.ijcat.com 20
 International Journal of Computer Applications Technology and Research
Volume 6Issue 1, 19-22, 2017, ISSN:-23198656

3. PRODUCTION MONITORING
After we rolled all the changes into production and completed
the rst iteration of SPL at X Bank, we began monitoring
production on a daily basis. The application production team
provided daily server statistics for all of production. We used
six sigma techniques such as regression analysis, processor
capabilities charts, Xbar-S charts and boxplots to aid in our
production monitoring. First, we performed daily regression
analysis on server processes to isolate any top consuming
processes. The regression analysis consists of gathering
process information from all servers, which was supplemented
by Perfmon, Minitab, and Perl. Perform is a windows
monitoring solution, Minitab is a statistical computing system,
and Perl, a scripting language, was used to parse and format
Perfmon data to t Minitab. Next, the formatted data was
imported into Minitab and a Minitab worksheet was created.
After the Minitab worksheet was created, a regression
analysis was performed to nd the top consuming processes
[M|Nll]3]. For this analysis, Processor Total was the Resp
processes, to be analyzed, were the Predictors (x variable).
See picture below.

After performing the above steps for all the production

servers, the analysis provided us with a list of top processes of
each production server. Below picture shows the high
consuming processes. The higher T value the higher
controlling factor the predictor, or the process.

After indicating the response and predictors in Figure 1 the

next step was to execute the regression analysis. The Output
created from the regression analysis is shown in Figure 2.
Figure 2 provides a value called R-Sq, the Higher the value of
R-Sq the more relevant this data set is to our analysis. ln our
example the R-Sq value was 9i'.2%, indicating our data
having a really good t to the model.
Below Picture shows Regression output, Processor vs
Processes

www.ijcat.com 21
 International Journal of Computer Applications Technology and Research
Volume 6Issue 1, 19-22, 2017, ISSN:-23198656

Below table shows list of top CPU consuming processes : utilization we are following outside our acceptable range 2274
times.

Below Picture shows Processor capability:

The CPU and memory graphs below displays a control chart

for subgroup means (an X chart) and a control chart for
subgroup standard deviations (an S chart) in the same graph
window. The X chart is drawn in the upper half of the screen;
the S chart in the lower half. Seeing both charts together
allows you to track both the process level and process We used the above tables and graphs to create daily
variation at the same time [M|Nll]3]. The x-axis of the graphs production reports. From the daily reports we created weekly
represents duration (time) while the y-axis represents sample and monthly trends to isolate any long-term problems.
mean or sample standard deviation. We primarily used this Specically, it helped us isolate a few high processes that
information to detect trends overtime. Minitab draws the were behaving differently in production than in our test lab.
average (center line), the upper control limit (UCL), and the The reason some the processes were behaving differently in
lower control limit (LCL) lines by default. production was due the fact that we could not test all business
CPU Processor control chart: processes during this round of testing and only limited our
testing to the business processes that produced 80% of the
volume. The 80% was used based on the rule of thumb
that20% of the transactions produce 80% of the volume. This
allowed us to get a good indication of production volume
without spending months scripting many different business
processes. It seemed that transactions that were a part of the
untested business processes were the top consuming processes
to show up in production, and not in the performance lab. Due
to our daily monitoring and reports, we were able to resolve
these types of issues prior to any production downtime. ln
conjunction, we utilized a custom dashboard application that
provided response time information at the web and application
Memory Usage during peak layer for every transaction. It also provided real-time server
and network statistics for all production servers. The
dashboard alerted the application support team if any
transaction or server network statistics breached the SLA
thresholds.

3.1 Conclusion
Start the SPL approach at any stage of the SDLC. The bank
faced performance issues in production, which caused un-
acceptance from end users, bank personal, and the possible
loss of future product upgrades. This could have cost millions
of dollars in product revenue and maintenance licenses. But it
did not, even though we started the SPL process of planning,
testing, monitoring, analysis, tuning, and optimization after
release l was in production. Since we were already on the
The processor capability graph [MINI03] below shows the wheel, we were able to include SPL earlier and earlier in the
processor distribution model around the CPU utilization Software Development Life Cycle as the product grew, or as
for the server on a given day. We used this information to see the wheel turned. The SPL approach saved signicant time
how many times (or parts per million, PPM) the data exceed and money, ensured production readiness, improved
our upper specications limit (USL). ln our case any data performance and scalability, and built condence.
point outside the 80% USL mark is considered defective
because the application degraded after the CPU utilization hit
80%. In this graph there are a few things to keep in mind, Left
4. REFERENCES
Boundary (LB), Upper Specication limit (USL), and parts [1] MINITAB Statistical Software
per million (PPM). Ln Graph 1, PPM > USL is 2274,
indicating that for every 1 million data points of CPU

www.ijcat.com 22