The ABCs of A/B Testing

If you’ve ever wondered how your earnings would change if you swapped a banner ad for a leaderboard or if you changed your color palette to Open Air, now you can find out by using A/B testing. In this brief tutorial, we'll review the fundamentals of completing an A/B test.

Tutorial

Let's say you’ve got a gut feeling that a leaderboard ad would double your revenue compared to a banner. To prove it, you'll need some evidence, so you decide to perform an A/B Test.

First, you'll need to create two custom channels, one for each ad unit you're going to be testing. Then, create a leaderboard ad unit and a banner ad unit with identical settings, except for the ad format. Be sure to assign each ad unit the appropriate channel created above.

You can decide how you want to run your A/B test; for example, you can choose to run the test on only 5% of your traffic or only on your home page. No matter what you choose, you'll need to run both of your ad units at the same time. But how can you easily alternate between your two ad formats? The simplest way to do this is to use the JavaScript template below - it will rotate between the two ad units, showing each 50% of the time.

<script type="text/javascript">
    var random_number = Math.random();
    if (random_number < .5){
        //your first ad unit code goes here
    } else {
        //your second ad unit code goes here
    }
</script>
<script type="text/javascript"     src="http://pagead2.googlesyndication.com/pagead/show_ads.js"></script>

When using this template, remember to replace “//your first ad unit goes here” with your ad code inside the first set of <script></script> tags.

Keep in mind that generally, modifying your ad code is against AdSense program policies. However, we've checked with our policy folks and they've given our publishers permission to modify the code for use with this specific script for the purposes of A/B testing. Please be aware that the AdSense team isn't able to provide any support or troubleshooting help for this script or this sort of testing.

Once you’ve implemented the A/B testing code on your page, you'll be able to view the experiment results in your channel reports and compare the figures from the custom channels you created.

If you notice that one ad configuration clearly performs better over time, you can replace the testing script with the ad code for that ad unit to display it on a permanent basis. Or, update the script to test the performance of that ad unit against another ad configuration.

A few important things to remember when designing and running an A/B test:

1. One of the ad configurations in your test should be the configuration you're currently displaying. In order to understand the effects of each ad configuration, you need to compare the experimental results to a baseline. This baseline, also called the control, should be the ad configuration you are currently using.

2. You should only make one change to the ad configuration in each A/B experiment. The goal of A/B testing is to isolate the revenue impact or CTR change of one particular modification, so that you know exactly which change affected your revenue. By modifying the baseline ad code in only one way, you can be assured of this.

3. For one A/B test, all configurations must be running simultaneously. Since we need to ensure that all ad configurations are subject to the same conditions such as the same traffic patterns, all configurations must be running at the same time. The JavaScript template above is one way of implementing this.

If you’d like to read more about A/B testing, please visit our Help Center .

UPDATED template for accuracy

Posted by Tom Tunguz - AdSense Optimization Team