Back to blog

A Beginner’s Guide to A/B Testing

---

App Builder Appy Pie, April 09, 2018: You have built an app and it seems to be on the right track to achieve a measure of success. Does this mean you are all set to put your feet up and simply let things happen on their own? You would of course want your app to keep doing better, and for that to happen you would have to make your app and app experience better. You might even have gotten some feedback and suggestions about how to make your app better and more desirable.

Though you might be getting some seemingly brilliant ideas about making your app better, but it is only natural that you are unsure about implementing these changes. Would these changes make your app better or worse? How do you allay these fears? To figure out whether a change in software is good or bad, a controlled experiment called A/B testing is carried out.

A/B testing is a lot more accurate way of figuring this out as compared to simply guess, assume, or judge from your own observation. A/B testing will actually give you some numbers to compare one version with the other. This way you get an unbiased and genuine idea about what’s going on with your app and then lets you make better informed decisions for the future of the app.

What is A/B Testing?

A/B testing or split testing as it is sometimes called, is the process of drawing comparison between two versions of an app, a landing page, a website etc. to understand the likes and dislikes of your target audience on the basis of the performance of the two versions under similar circumstances. The results that you get from these tests can then help you make tweaks and changes to the app in order to optimize conversion rates.

In this process two versions of the same app are showcased to two different samples of the target audience. The success of the A/B tests is greatly dependent on the size of the sample that you choose to test the app on. The credibility of the insights from these tests are directly proportional to the size of the sample. This means the bigger the sample size, the more confident you can be about the credibility of the test results. If the number of people responding to the test is too small, they cannot possibly represent the preferences of the wide audience you may be catering to.

Why should you use A/B Testing?

When you were conducting market research before beginning to build your app and were quite carefully analyzing the UI/UX principles, and were following the industry best practices, you were still leaving a lot of your app elements to guesswork!

Some of the most common assumptions include thinking that what works for others is going to work for us, that the results of our market research would apply to a majority of our target users, and so many more. However, it is imperative that we do make these assumptions, because if we do not, then we wouldn’t be able to do anything at all due to myriad doubts!

However, there are ways to turn “I guess we could…” into “I know we can…” and A/B testing or split testing the different elements of the app. When there is solid data to conclusively prove that Option A is better than option B, guesswork is taken out of the picture.

A/B testing has the potential to become the core principle of your conversion marketing strategy and offer to you, ways to carry out investigation and come up with solutions to increase the number of visitors.

There are several benefits or advantages that come with the process of A/B testing for mobile apps, but let’s talk about the meaty ones here

• Sale Upsurge: When you implement A/B testing, you have the freedom to explore a variety of screen layouts, and other defining components of the app, which give you a better understanding of conversion and sales. Minute elements like the placement of the “Buy” button and even the color of the button might do the trick and bring you more conversions and higher sales.
• Enhanced UI/UX: User experience is of prime importance when it comes to app development because it is finally the users that you are building the apps for. A/B testing carried out in an efficient manner would disclose what is it that the users need from the apps they want to use.
• Analytics & Ideas: A/B testing offers useful analytics for the developers and even some great ideas for push messaging and such aspects. The data received from testing can help you get some really valuable insights about different options for the app features.

Though these are some of the most obvious advantages that an app developer must be looking for, to make their app a success but there are not really many organizations that go for it. The budget and time restrictions often are the underlying reasons why companies go on to ignore the process. A/B testing your mobile app might seem to be a relatively simple thing to implement when you are reading about it, the actual execution part is actually quite challenging.

What can you A/B Test?

Almost everything that is a part of your marketing material, including the headlines, call to actions, content, images etc. can all be A/B tested. However, it is wise to prioritize the elements in your app that have the potential to influence conversions. Following is a list of elements that might be a good place to begin with:

• Email sign-up modals
• Search modals
• Copy & CTA on the Landing page
• Copy & CTA on the Homepage
• Flow of Sign-up or Account Creation
• Flow of Checking out
• All the CTAs for their text, design & placement
• Elements of Navigation
• Components of the Lead Form
• Copy & Presentation of any promotional element
• Search Results
• Structure of Pricing Plans
• Free vs Paid

The Real A/B Testing Process

The process of A/B testing isn’t something you do once and forget about it after that, rather, it is a continuous process that needs to be incorporated in your improvement cycle. According to the Shewhart Cycle of continuous improvement or processes and products there are four phases in the cycle – plan. Do, check, & act. The main idea here is to move your business closer to perfect operation. Perfection, however isn’t achieved in a day, which is why A/B testing is a repetitive program instead of being a one-off activity.

1) Gather Data

Once you have realized that you need to conduct A/B testing, your first step is in identifying what is it that you intend to improve through this testing, and then prioritize the list. Though there is an element of assumption in doing so, but it can’t possibly be a complete guesswork. The testing needs to be guided with some sort of evidence in form of data.

• Analytics
  Analytics is a convenient place to begin with data collection. As app or website owners, we have to anyway conduct a lot of data analyses, which gives us a robust knowledge of it, and we have easy access to it too. The idea here is to identify the pages that have problem including high pageviews & high exit, drop-off, or bounce rates, or the pages that you feel can do better.
• Heatmaps
  Analytics can only identify the problem pages, but they don’t really reveal what the real problem might be. Heatmaps can help you identify where people are clicking, how far down are the visitors scrolling down and there are a number of heatmapping tools available even for the smaller businesses.
• Surveys
  Surveys that ask direct & specific questions can offer you valuable insights into the areas on your app or website where people are facing troubles. The surveys can either be on-site for the visitors or be sent through emails to the actual customers.

2) Assign Goals

After you have collected your data and have managed to identify the pages that hamper your conversions or have the potential to increase your conversions, you must prioritize these pages and assign goals to them. When you have assigned clear goals, you are better equipped to decide which page performed better – the original one or the variation. Prioritizing allows you to test according to potential, importance & ease.

• Potential
  Typically the pages that are performing worst, and when you fix these pages, you are afforded the highest potential for improvement.
• Importance
  These are the pages with the highest traffic, but also have the highest bounce & exit rates, including those pages that have a higher price tag in your efforts to bring more visitors to them. When you make improvements to these pages, you increase conversions & thus lower the CAC or Customer Acquisition Cost.
• Ease
  In process of prioritizing your tests, one important factor to consider is ease. While some tests are easy to put in place, there are others that need some technical know-how.

In most cases, your goals will relate to increasing conversions, and decreasing drop-offs. However, conversions don’t always follow, after a visitor has only viewed just one page. As a part of running A/B tests you would often use Click Goals, Visit Goals, & Revenue Goals. It is impossible to run even a single A/B test without first identifying the goals.

3) Propose Hypotheses

Like we discussed earlier, though the A/B testing cannot be a complete guesswork, there still is a need to rely on some assumptions. A hypothesis helps put forth an assumption that changing a particular element on a certain page would mean higher engagement, increased conversions, or higher revenues. However, these assumptions need to be realistic and must be backed by some sort of evidence.

4) Build Different Versions

No matter which element you are testing out – a particular button, re-wording a particular copy or description of a product or service, changing banner image colors, or any other changes, any of these changes must have a robust justification at the root of it. It simply means that the moment you are asked about the reason you want to make a particular change, you should have a well-formed answer for it.

Now that you have gathered enough data, identified areas of potential, have managed to assign goals, and have proposed hypotheses for the changes, you’ve almost reached the stage of running the actual tests. Till this time, you had been relying on information that was already available on you, for this part, however you would have to probably rely on external tools.

5) Run the A/B Test

After building different versions or variations, clearly set some goals for them, you may also include custom targeting, and then run the actual testing. This is when you sit back and wait for the test duration to get over, so that you can begin analyzing the results. The A/B testing tool that you choose would ensure that the visitors to your page encounter one of the two variations randomly and then measures their interactions for further analysis.

6) Analyze the Results

After ending the tests, the findings need to be analyzed, and it is this phase of the test that is most important in the whole process of testing. This part of the process essentially lets you apply whatever you have learnt from the test results so that you can bring about some positive changes in your product.

As you carry out the examination and analysis of the results, make sure that you look at them in context of the goal and the hypotheses that you defined at the beginning. For example, if you were testing an increase in the revenue, the page that records higher revenue would be the winner, as compared to the one that had higher click-throughs or sign-ups. Whether the outcome supports your hypotheses or not, you must carry on with the testing. Even after you have identified one element that helps you increase your revenues, you still need to carry on testing other elements. After you have conducted several A/B tests you would then notice a pattern emerge that would indicate which changes yield better results. But one important thing to remember is that the percentage change or difference you might observe during the testing phase might not translate into a permanent change of the same scale.

Points to Consider for A/B Testing

After being convinced that you need A/B testing and having decided to go with it, and before you actually begin processing it there are a few things you might have to consider.

#1 Be Sure About What You Want to Measure

It is great that you want to make improvements, but before getting into the fine details, there is a need to develop a robust plan. Determine the exact page, element, or parameter that you want to improve, and on that basis, you would have to figure out the parameter that you’re going to measure for testing your hypotheses.

#2 Begin with A Single Variable

Initially pick just one element to play around with, in order to keep things simple. When the initial test works then you can go on to experiment with more tweaks. Starting small would always keep you in sight of the changes bringing about an impact. Through this kind of staggered testing you gain greater control over the results.

#3 Don’t Go Overboard with the Timeline

It is true that there is some patience that you would to exercise when conducting an A/B test, but letting the tests run for too long might cause skewing of the results, losing their validity. You would, of course want to run the test for a minimum of one or two weeks so that you receive substantial results but going beyond two months would only bring in too many variables into play.

#4 You Can’t Simply Stop Testing

To think that you would ever be ‘done’ with your app, is quite simply a myth! It is important that you keep working on it and your site to make sure that it is performing at its best. Your pages might all be performing quite well, but there is always scope for further improvement. Always look at your app as a perennial work in progress and never lose out in an opportunity to make any perceivable improvement.

How Long Should the Tests Run?

We did talk about not running the tests for too long, but how long should you ideally run them? Unfortunately, there is no fixed number when it comes to the duration of the tests. The pages that have a higher traffic need the tests to be run for a shorter duration, as a valid statistical significance is reached sooner. However, some conversion decisions might take a little longer to make. However, you must not give up when this happens, instead allow the test to run for some more days to see whether the results separate again. Apart from this, there is also a need to consider the timing and its effect on the site traffic and conversions. The numbers in the holiday season might be different from the ones in regular days.

As a general rule, tests would be allowed to run for a minimum of 10 days preferably including two weekends. However, the main factor to consider here is that the test must achieve a valid statistical significance in the period.

Conclusion

A/B testing is one of the most effective and efficient techniques in order to determine why a certain page, an email marketing campaign, or any other advertising campaign isn’t bearing the results you were hoping for. Though this article talks about quite a few facets of the testing method, but there is a lot more to it still. The underlying intent of this emerging technique is helping you showcase confirmed aims, and to set your app up for an exponential growth with the near perfect tests, analytics, and results.

Create Your Own App