A/B Testing

About the Tutorial

A/B Testing is one of the best way to compare two or more versions of an application or a
web page. It enables you to determine which one of them performs better and can
generate better conversion rates. It is one of the easiest ways to analyze an application
or a web page to create a new version that is more effective.

This is a brief tutorial that covers the fundamentals of A/B Testing with suitable examples
to illustrate how you can put it into practice.

Audience
This tutorial has been designed to suit the requirements of all those professionals who are
working in the software testing domain. It provides sufficient insight into the concepts of
A/B Testing and how you can apply it to perform data analysis and maximize the
conversion ratio of any website.

Prerequisites
We assume that the readers of this tutorial have basic knowledge of HTML and some
experience of handling a website. In addition, it is going to help if the readers have an
elementary knowledge of Data Analysis and Conversion ratio of websites and mobile
applications.

Copyright and Disclaimer

 Copyright 2018 by Tutorials Point (I) Pvt. Ltd.

All the content and graphics published in this e-book are the property of Tutorials Point (I)
Pvt. Ltd. The user of this e-book is prohibited to reuse, retain, copy, distribute or republish
any contents or a part of contents of this e-book in any manner without written consent
of the publisher.

We strive to update the contents of our website and tutorials as timely and as precisely as
possible, however, the contents may contain inaccuracies or errors. Tutorials Point (I) Pvt.
Ltd. provides no guarantee regarding the accuracy, timeliness or completeness of our
website or its contents including this tutorial. If you discover any errors on our website or
in this tutorial, please notify us at contact@tutorialspoint.com

i
 A/B Testing

Table of Contents
About the Tutorial .................................................................................................................................... i

Audience .................................................................................................................................................. i

Prerequisites ............................................................................................................................................ i

Copyright and Disclaimer ......................................................................................................................... i

Table of Contents .................................................................................................................................... ii

1. A/B TESTING – OVERVIEW ................................................................................................... 1

2. A/B TESTING – HOW IT WORKS ........................................................................................... 2

Sampling of Data ..................................................................................................................................... 2

Confidence Intervals in A/B Testing......................................................................................................... 3

3. A/B TESTING – WHY TO USE? .............................................................................................. 4

4. A/B TESTING – WHAT TO TEST?........................................................................................... 5

5. A/B TESTING – PROCESS ...................................................................................................... 6

6. A/B TESTING – COLLECT DATA ............................................................................................. 8

Google Analytics / Mix Panel (Analytics Tool) ......................................................................................... 8

Mouse Flow / Crazy Egg (Replay Tools) ................................................................................................... 8

WebEngage (Survey Tools) ...................................................................................................................... 9

Other Tools – Chat, Email ...................................................................................................................... 10

7. A/B TESTING – IDENTIFY GOALS ........................................................................................ 11

A/B Testing – Generate Hypothesis ....................................................................................................... 11

8. A/B TESTING – CREATE VARIATIONS .................................................................................. 12

Visual Website Optimizer (VWO) .......................................................................................................... 12

Google Content Experiments ................................................................................................................. 13

Optimizely ............................................................................................................................................. 13

ii
 A/B Testing

9. A/B TESTING – RUN EXPERIMENT ...................................................................................... 14

Visual Website Optimizer ...................................................................................................................... 14

Google Content Experiments ................................................................................................................. 15

Content Experiments............................................................................................................................. 16

10. A/B TESTING – ANALYZE RESULTS...................................................................................... 19

11. A/B TESTING – TOOLS ........................................................................................................ 20

12. A/B TESTING – MULTIVARIATE .......................................................................................... 23

13. A/B TESTING – SEO ............................................................................................................ 25

14. A/B TESTING – INTERVIEW QUESTIONS ............................................................................. 27

iii
 1. A/B Testing – Overview A/B Testing

A/B Testing (also known as Split testing) defines a way to compare two versions of an
application or a web page that enables you to determine which one performs better. It is
one of the easiest ways to analyze an application or a web page to create a new version.
Thereafter, both these versions can be compared to find the conversion rate, which
further helps in finding the better performer of these two.

Example
Let us assume that there is a web page and all the traffic is directed to this page. Now as
a part of A/B Testing, you have made some minor changes like headlines, numbering, etc.
on the same page and half of its traffic is directed to the modified version of this web page.
Now you have version A and version B of the same web page and you can monitor the
visitor’s actions using statistics and analysis to determine the version that yields a higher
conversion rate.

A conversion rate is defined as the instance, when any visitor on your website performs a
desired action. A/B Testing enables you to determine the best online marketing
strategy for your business. Take a look at the following illustration. It shows that version
A yields a conversion rate of 15% and version B yields a conversion rate of 22%.

1
 2. A/B Testing – How it Works A/B Testing

You can monitor the visitor’s actions using statistics and analysis to determine the version
that yields a higher conversion rate. A/B Testing results are usually given in fancy
mathematical and statistical terms, but the meaning behind the numbers are actually
quite simple. There are two significant methods through which you can check conversion
rates using A/B Testing:

 Sampling of Data
 Confidence Intervals

Let us discuss these two methods in detail.

Sampling of Data
The number of samples depend on the number of tests performed. The count of conversion
rate is called a sample and the process of collecting these samples is called as sampling.

Example
Let us say you have two products A and B, you want to collect sample data as per its
demand in the market. You can ask a few people to choose from product A and B and then
request them to participate in a survey. As the number of participant’s increase, it will
start showing a realistic conversion rate.

There are various tools that can be used to determine the correct number of sample size.
One such free tool available is:

http://www.evanmiller.org/ab-testing/sample-size.html

2
 A/B Testing

Confidence Intervals in A/B Testing

Confidence interval is the measurement of deviation from the average on the multiple
number of samples. Let us assume that 22% of people prefer product A in the above
example, with ±2% of confidence interval. This interval indicates the upper and lower limit
of the people, who opt for Product A and is also called as margin of error. For best results
in this average survey, the margin of error should be as small as possible.

Example
Let us assume that in Product B, we have added a minor change and then performed A/B
Testing on these two products. Confidence interval product A and B are 10% with ±1%
and 20% with ±2% respectively. So this shows that a minor change has increased the
conversion rate. If we ignore the margin of error, conversion rate for test variation A is
10% and conversion rate for test variation B is 20%, i.e. a 10% increase in the test
variation.

Now, if we divide the difference by control variation rate 10% ÷ 10% = 1.0 = 100%, it
shows an improvement of 100%. Hence, we can say that A/B Testing is a technique based
on mathematical methods and analysis. There are various online tools that can be used to
calculate A/B significance.

http://getdatadriven.com/ab-significance-test

3
 3. A/B Testing – Why to Use? A/B Testing

A/B Testing is used to make business decisions based on the results derived from data,
instead of just making predictions. It allows you to create variations of your website or
app and then helps you to confirm or discard your decision to make changes.

This testing allows you to optimize your site or app in such a way that it increases the
conversion rates. A higher conversion rate means getting more value from your existing
users instead of having to pay more on traffic acquisition. A/B Testing can also help you
to change other factors in business like changing work culture, etc. It helps you to use
mathematical data and statistics to determine the direction of your product variations.

Either you are a designer, a business analyst, or a developer, A/B Testing provides a simple
way to use the power of data & statistics to reduce risks, improve results, and become
more data-driven in your work.

To run an A/B Test, you should consider the following points:

1. Always perform A/B Testing if there is probability to beat the original variation by
>5%.

2. Test should be run for a considerable amount of time, so that you should have
enough sample data to perform statistics and analysis.

A/B Testing also enables you to gain maximum from your existing traffic on a webpage.
The cost of increasing your conversion is minimal as compared to the cost of setting up
the traffic on your website. The ROI (return on investment) on A/B Testing is huge, as
a few minor changes on a website can result in a significant increase of the conversion
rate.

4
 A/B Testing

End of ebook preview

If you liked what you saw…

Buy it from our store @ https://store.tutorialspoint.com