Run A/B Tests To Improve Your Website

Intro into A/B testing

Both page options should have the same URL. Displaying different options of the same web page to various people, you can track their behaviour and analyze the results: which one has higher conversion rate, avarage bill per client, amount of subscriptions, etc.

The idea behind A/B testing is straightforward: you don't have to argue with your team which page is better. Raw data from testing will clearly provide an answer.

But it's worth noticing that even though the idea of A/B testing is quite simple, it's very important to set the ground rules correctly, so that the results are accurate and comparable.

Check out how EA raised their sales by 43% with the help of A/B testing (Case Study)

As EA prepared to release the new version of SimCity, they launched a promotional offer to drive more game pre-orders. The offer was displayed as a banner at the top of the pre-order page – front-and-center for shoppers. But according to the team, the promotion was not driving as much growth in pre-orders as they had expected. The team decided to test some other options and see which design and banner layout will drive more revenue.

The First Option was to remove a promotional offer  from the page. Surprisingly, removing direct all-to-action drove 43.4% more purchases. Turned out people really just wanted to buy the game with no extra incentive necessary.

Most people believe that direct promotions drive sales, but for EA this turned out to be totally false. Testing gave them the information needed to maximize the revenue in a non-obvious way.

Sign up for Optimizely

• This service is very easy to install and easy-to-use. You don't have to be a programmer to get started with Optimizely.  
• We use Optimizely at Casual.pm, and we are really happy with the results so far.

  

• Vwo 
• Instapage (A/B testing for landing pages)
• Abtasty 
• Omniconvert 

Set your Optimizely account

1. Go to the Optimizely Home page. Click the Settings tab, then the Implementation sub-tab. 
2. Copy the line of code above the Snippet Details, exactly as you see it. Don’t add http/https to the URL or make any other modifications.

   

Add snippet to your source code

Here's what the Optimizely Snippet may look like on a page:

If you want to add the snippet to a CMS-powered website (such as WordPress, etc.) please, check out the following webpage.

Setting up the experiments

1. Slogan
2. Sign up Form
3. Above the fold

1. Create a variation
2. Set goals to evaluate the result: was the experiment successful or not
3. Run the tests and analyze the results

Test 1: "Slogan"

Your main tag line is essential in terms of website conversion. Thus, testing the main slogan or tagline should be on top of your agenda as it can lead to a substantial increase in sign ups for your services. In the end, your customers get a clear idea of what you do and how you can help them.

Create a variation

1. Navigate to the Home page and click "New Experiment" (this is where you will create a variation for your A/B test). Enter the URL of the website you’d like to test and name your experiment.

   

   Name your experiment:

   

   2. Using Visual editor create changes that you'd like to implement.

   

   3. In this variation we are going to substitute "Turn Your Projects into Awesome Visual Maps" with "Turn your boring task list into a visual project plan".

   

Set up URL targeting

Go to Options and select "URL targeting".

This is a place where you specify the address of a page that you are going to test. Usually, startups have just one landing page, but if you have a more complex website, then you can run multiple experiments on different pages at the same time.

Set audience and traffic allocation

Select the "Audience" button to identify user groups that will take part in the test campaign.

Experiments are set to be displayed to everyone by default. However, you can narrow down your audience using a diverse set of parametres.

1. Press "Options", then select "Traffic allocation". Take a look at the screenshot below.
2. By default, all traffic is divided into two equal groups: 50% of users see the original version of the page, and other 50% see the modified version.

   

   High-traffic websites can set the limit for their traffic that participates in experiment. 
   
   

Set Goals

Every experiment will lead to a conclusion that is based on gathered data. Setting the right goals is crucial to determine whether your experiment was successful or not. Click the "Set goals" icon:

Then set goals. In our case we want to test our slogan, whether it gets the customer interested. So our main focus is Engagement and Sign ups:

• Sign up button clicks
• Engagement
• Button click (to measure the general engagement on our Landing Page)

  

Q&A and publish

After setting up all components of your experiment, press "Save now" to secure your changes.

Then click "Variation" on your Editor page and select "Preview". Besides, you should perform an in-depth QA review before pushing your experiment live to the world. Create a QA checklist that outlines all of the changes you’ve made.

Your QA checklist should include: 

• All goals that were added, as well as how each is triggered 
• All functionality that’s been added (for example, a new button)
• Common use cases including expected user flows to and from the page
• Audiences that are eligible and ineligible for the test (e.g. certain location only)
• URLs of the tested pages
• Sample workflow to fire a goal (especially for custom events) 

Run your experiment and analyze the results

After some time passed you can stop your experiment and analyze the results. Waiting time depends on the amount of traffic you have on your site: the lower the traffic means you'll have to wait longer until there is a conclusive result on your test.

Go to Home page and press "Results" to see whether your test proved the value of any improvements made to the original version of the site.

Our test results show that Engagement has slightly increased, but signups stayed the same.

That's not much of achievement. Let's declare this experiment failed. However, it is important that we conducted an experiment and now operate with numbers instead of guesses. 

Implement changes (if needed)

Test 2: "Sign up Form"

Create a variation

To get started with this experiment, load your landing page into the editor.

Click one of the input fields and select Edit Element, then Edit Style. The "Edit Style" feature lets you easily make changes to the background color, border size, text fonts etc.

The Dimensions tab allows you to change the height and width of the input field. Once you have finished adjusting the sizes and input fields colors, click the text titles and repeat the process.

Click Options, then URL Targeting to set up the URL targeting conditions for this experiment. Since your experiment should only run on one page, simply paste your landing page URL.

Please note that we omit some steps (like Set Audience or Traffic Allocation), because they are basically the same in all experiments.

Examples:

Set Goals

Run experiment

Test 3: "Above the fold"

1. Understand what are you trying to sell.
2. What does the product do.
3. How it helps and delivers value to the customer. 
4. How you can buy the service. 

Create a variation

This is a good example:

Version 1:

Version 2:

Second version generated 47% more sign ups! Why? 

1. Sign up page is clear, it doesn't confuse or distract, there is no images. 
2. The headline is straightforward and communicates the value better. 

Set goals and run the experiment

Good luck!

1. Get started on web optimization
2. 10 Common Experiments and how to build them
3. A/B Testing Case Studies
4. Qualitative Questions That Lead to Great A/B Tests
5. 15 of the Best Website Homepage Design Examples
6. 12 Surprising A/B Test Results to Stop You Making Assumptions
7. Useful Value Proposition Examples (and How to Create a Good One)