What is an A/B Test: A Review of the UTA Superpage Image A/B Test Results

From AP-Wiki
Jump to: navigation, search
A-B Testing.png

A/B testing, which you may also have heard referred to as split testing, is a method of website optimization in which the conversion rates of two versions of a page — version A and version B — are compared to one another using live traffic. Site visitors are bucketed randomly into one version or the other. By tracking the way visitors interact with the page they are shown — e.g. the selections they make, the videos they watch, the buttons they click, or whether or not they sign up to receive more information — analysts can determine which version of the page is most effective.

One of the most common ways A/B testing is utilized is to test two very different design directions against one another. For example, the current version of a university's home page might have in-text calls to action, while the new version might eliminate most text, but include a new top bar advertising the latest promotion - a new academic program, or a limited time offer (LTO), for example. After enough visitors have been funneled to both pages (in order to give the high enough level of confidence), the number of clicks on each page's version of the call to action can be compared. It is important to note that even though many design elements are changed in this kind of A/B tests, only the impact of the design as a whole of each page is tracked, not individual elements.

This A/B test was designed in order to compare the UTA superpage original image (A) and the Hero female image (B). Two different B versions were created - 'Model-Blonde' (B1) and 'Model-Brown' (B2). As the goal was to compare the results for practically 3 versions of the UTA superpage, the visitors to the UTA superpage were randomly directed to see one of the three images: A, B1 or B2.

UTA Superpage Original vs Hero Image(s)- A-B1-B2.png

The test start date was Dec. 1, 2016 and by the publishing of this article, it was running for 78 consecutive days. Here are the data, as provided by the Optimizely platform.

The total of unique visitors was 41,135. All three versions had a very close number of unique visitors each (which was expected, as the three versions were showed to visitors randomly. The chart with the results shows (below) that the Hero image 'Model-Blonde' outperformed the 'Model-Brown' in its conversion rates (in our tests it is most often page visits to requested information).

Variations Unique Visitors Unique Visitors % Conversion Rate
Original (control) 13,585 33.00% 7.28%
Model-Blonde 13,958 33.90% 7.27%
Model-Brown 13,592 33.00% 6.86%

As the conversion rates of the Original image and the Hero image 'Model-Blonde' were so close, there is no apparent winner of this A/B test. At the moment, it is basically a tie between the Original (control) image and the 'Model-Blonde' image. Some preliminary tests for other universities suggest that a good next step will be to test how the UTA Original (text only) image will perform against UTA campus images.