Most web developers realize the importance of A/B testing in ensuring their products are functional and user friendly.
We recently carried out testing on our site and found the results and lessons learned to be really useful. We think the case study shows how invaluable A/B testing is to our business. Hopefully you’ll find it interesting too.
Historically RevaHealth.com has always had a fairly unusual and unique User Interface. Whenever a user searched on our site, we would return a list of results on the left hand side on the page and when a user clicked on a search result, the brochure would appear on the right hand side. We built it this way because we thought that it would allow the user to be able to compare clinics quickly without having to reload the page.
We were never entirely happy with this interface. We worried that, because it was so unique, users wouldn’t know how to use it. Nonetheless we stuck with the basic premise for over a year and concentrated on fine tuning it.
When we finally secured funding we decided to look at it again. We solicited feedback from a lot of people including some leading design agencies. What we got back was nearly universal – everyone hated our UI. It was too cluttered, they didn’t know how to use it and it was too unusual. So, we decided we needed to change to more traditional approach, with search results on one page and brochure information on a second page.
Our original plan was to A/B test this new layout against the original design, but unfortunately we ran out of time on the project. We decided to push it out anyway and hoped that it would be an obvious improvement and that A/B testing would be unnecessary.
What happened surprised everyone – it was a disaster. Our bounce rate went from 45% to 51%, page views reduced by 25% and worst of all our conversion rate went from 7.2% down to 2.4%. IT WAS KILLING 75% OF OUR REVENUE! After a week of counting the cost we pulled it out and brought back the original design.
So that was the end of it then? Well … no. You see, when we looked at the stats for that week, we saw that there had been a dramatic increase in traffic from the UK. Not only that, but we hadn’t had time to tweak the new design, as we had done with the original design for over a year. So after licking our wounds we decided to try again, but this time with proper A/B testing.
We made some obvious changes to both designs and attempted to make sure that we were testing like with like. We split our traffic 50/50. Half our users saw the ‘narrow’ page layout (search results in a panel on the left hand side, brochures in a panel on right), and half saw the ‘wide’ version (search results fill the page, and brochures open on a new page).
Unfortunately for us, the results were much the same as when we pushed out our previous attempt at the start of the year. The wide version was generating nearly 25% fewer searches than the narrow version, and it was getting about 12% fewer brochure views too. So, once again we were disappointed and confused.
Conventional wisdom says not to surprise your users with an unusual or non-standard UI. Our UI was very unconventional, so, why did shifting to a more standard visual experience hurt our numbers so badly? Were our users really so unusual that they preferred something so different to most of the search sites out there? We couldn’t believe that, so we stared at the numbers until our eyes bled. And the answer, as is always the case with A/B testing, was in the numbers.
It turned out that the metric we were using at the start, the number of searches performed, wasn’t actually a proper measure of what we were interested in testing. A fall in the number of searches performed didn’t actually mean the newer version of the page was worse. In fact, around the same number of people ended up contacting clinics through both funnels, and since the number of searches performed in the new funnel was lower, users there were finding what they wanted faster!
So what’s the conclusion? And more importantly, how can you avoid some of the mistakes we made?
It’s very easy to get caught up on one simple metric and try to force it one way or another. Even a relatively simple interaction of users searching, narrowing, viewing and selecting is affected by many factors. It was too simple to look at our raw metrics and conclude that our users ‘like’ one UI over the other.
All you can do is refine and test, and refine and test again. After each test, look hard and deep at the numbers – all of the numbers. Because, while one change in the UI may seem simple, the effect it has on the system as a whole may be widespread and not so easy to understand. If you only focus on one metric changing, you may well miss the bigger picture.
Most importantly, make sure that the metric you are looking at is a proper measure of the action / reaction you are supposed to be testing!
Contributed by the RevaHealth Team.
Quintura (http://www.affiliates.quintura.com), an innovative provider of visual-based site search, analytics and monetization platform for online content publishers, today announced the awarding of the US patents. Quintura is the only fully-interactive, context-based method for search visualization on the Web.
When you enter a search term, 
Caronte Web
Federated Search
Highlight Health
MSearchGroove
Ramenos Blog
Reel SEO
Software Abstractions
The Genius Files
The Next Web