The ABs of AB Testing

What is A/B Testing

Google has a phrase that they live by,"In God we trust; all others must bring data." In other words, it doesn’t matter your authority, you must be able to back up their claims with data. You better have valid proof that what you’re doing actually works. While Google may use the phrase, it was originally coined by Edwards Deming.

So how does this apply to web marketing and everything that is included in this broad umbrella? The answer lies in A/B testing and how it can help improve your websites analytics on many different fronts. Google even recommends doing these type of experiments stating, “experiments help you to make informed decisions about how to configure your ad settings, and can help you to increase your earnings.”

An A/B test involves testing two versions of a web page: an A version (the control) and a B version (the variation). The two versions are presented with live traffic at a random 50/50 and are measured based on different measuring methods.

I will review the basics, some good things to test, and how to interpret the results. A/B testing is obviously much more integrate than this blog post, but this blog post will be a great foundation for anyone who is interested in improving their analytics and conversion rates. A/B testing helps us determine if our basic intuition and common sense on a particular part of our website will actually work. A lot of times what we think is the best practice doesn’t work as well as a less obvious answer. Finally, what works in one market may not work very well in another.

How to get started:

  1. Create a testable hypothesis
    • Start an A/B test by identifying a goal for your company then determine which pages on your site contribute to the successful completion of that goal.
    • By definition, a hypothesis is a proposed statement made on the basis of limited evidence that can be proved or disproved and is used as a starting point for further investigation.
    • If the winning variation only applies to the test you just completed, you are missing out on an opportunity to get the most value out from your experiments.
  2. Have a good Control
    • This should go without saying, but one of the tests should be a control or what you have been using.
    • Your test should only being testing one thing at a time.
  3. Audience/Time
    • By this I mean a significant number of people should be viewing your A/B testing.
    • You should also run these tests as short of time as possible.
    • Use software to help
    • There are a lot of programs out there to help you track how your pages are doing.
    • You can even use software that sorts site visitors to your A/B pages randomly.

What to Test

There are a ton of things to test, some of these apply to emails and some of them apply to landing pages. I will warn you, it is very easy to fall into the trap of finding very small and insignificant items on your page to swap out and test. Always ask yourself is my return on investment worth it? Is the time I’m spending creating this test going to give me a big enough percentage boost in traffic?Here are some ideas:

  • Subject line (email)
  • Call to Action
  • Layouts
  • Images
  • Forms
  • The call to action’s (i.e. the button’s) wording, size, color and placement,
  • Headline or product description,
  • Form’s length and types of fields,
  • Layout and style of website,
  • Product pricing and promotional offers,
  • Images on landing and product pages,
  • Amount of text on the page (short vs. long)

Additional tips

Know how long to run a test before giving up. Giving up too early can cost you because you may have gotten meaningful results had you waited a little longer. Giving up too late isn’t good either, because poorly performing variations could cost you conversions and sales. Use a calculator (like this one) to determine exactly how long to run a test before giving up.

Show repeat visitors the same variations. Your tool should have a mechanism for remembering which variation a visitor has seen. This prevents blunders, such as showing a user a different price or a different promotional offer.

Make your A/B test consistent across the whole website. If you are testing a sign-up button that appears in multiple locations, then a visitor should see the same variation everywhere. Showing one variation on page 1 and another variation on page 2 will skew the results.

Do many A/B tests. Let’s face it: chances are, your first A/B test will turn out a lemon. But don’t despair. An A/B test can have only three outcomes: no result, a negative result or a positive result. The key to optimizing conversion rates is to do a ton of A/B tests, so that all positive results add up to a huge boost to your sales and achieved goals.

Extra Help

You will begin to see patterns among the different elements that increased performance. You can then use these patterns to better understand what works and what doesn't.

What else can you do right now to optimize performance and start producing happier customers? Stay ahead of the curve and make sure you have the data to back up your decisions. For more information regarding a/b testing and website experiments visit google's help section and here is a great blog from moz that wars of falling into a/b testing minutiae.