Why A/B Testing Is Not For You?

First of all, A/B testing is a great business tool. We will not argue with that!

However, a lot of businesses shouldn’t bother with A/B testing. Small businesses that are trying to grow, early age startups and other micro businesses simply don’t have the traffic and the transactions to accurately perform A/B tests. It takes a significant amount of traffic to provide accurate, measurable results.

If the traffic isn’t there yet, you can still optimize your site based on audience data you’ve gathered. But, A/B tests won’t be helpful for a while and they might produce false information.

Let’s see what can you do after you admit your site isn’t there yet.

A/B Testing in Early Stage SaaS Companies

Running an A/B test that directly compares a variation against a current experience lets you ask focused questions about changes to your website or an app. Then you can collect data about the impact of that change.


A/B testing allows individuals, teams, and companies to make careful changes to their user experiences while collecting data of the results. This allows them to construct hypotheses, and understand better Why certain elements of their experiences impact user behavior. Also, their opinion about the best experience for a given goal can be proven wrong through an A/B test.

This can be extremely helpful for a company that is looking for ways to make right data-driven decisions. But here is a key ingredient to A/B testing that most SaaS companies don’t have, volume!

The Traffic Volume Isn’t There

Here is the problem: to detect a 20% change in a target metric with 90% confidence, if it happens only to 3% of visitors, requires volume of 12,000 visitors per test. In other words, the A/B test requires 24,000 visitors to test just two variations.

There’s no doubt that A/B testing can be useful for businesses that want to improve their conversion rates. But, most early stage SaaS companies simply don’t have enough volume to achieve statistical significance. For many early-stage software companies achieving this kind of scale requires months or even quarters, an eternity in increasingly competitive markets.

You need to have sufficient volume for test groups, results and the differences between them. The statistical significance of your experiment should be over 95%. This way you can be sure that your results are valid and not based on chance.

As an early stage startup, you should not lean on quantitative but qualitative data.

Quantitative vs Qualitative Data


Qualitative data deals with descriptions, not with numbers directly. That makes them harder to analyze. First you must go through the data, understand it, “code” the data, then interpret and analyze. Quantitative data tells us that something is wrong, while qualitative data answers the question Why it’s wrong.

At small-scale, intuition and research trump statistical analysis. It’s not that statistical analysis doesn’t have its place. At scale, running large multivariate tests enables businesses like Google and others to consistently seek out a marginal 1 to 5% gain in product usage, performance, and revenue. But you need to be able to achieve that scale. And you won’t be able to do it before your product becomes popular.

Ways to Get Useful Data Without Traffic or Too Many Customers

You need to talk to people, again! What you’re actually doing is repeating phase of MVP (Minimum Viable Product) process you did before you even launched your product. Remember?

“Gathering real-world feedback from customers is a core concept of Customer Development as well as the Lean Startup.

But what information to collect?”  – Steve Blank

We’ve already written in our previous post Why should you use the MVP approach and get out of the building to test your idea. It’s a pretty similar with A/B testing when you’re not “big enough”, but you should analyze your website.

Form a Hypothesis

The CRO process is meant to have validated learning based on hypotheses. If you had a good process to follow, conversion optimization is a systematic, repeatable, teachable process. When it’s done right! Running tests without having a quality hypothesis is a waste of time. You need to complete proper conversion research to discover where the problems lie and then perform analysis to figure out what the problems might be. Finally, you’ll be coming up with a hypothesis to overcome site’s problems. After you have defined your problem and articulated your hypothesis, you can come up with specific split-testing variations.

Make sure you are getting actionable data from Google Analytics

Before you start working on your data, make sure you have defined and properly set up this four things in your Google Analytics:

  1. Business objectives
  2. Website goals
  3. KPIs
  4. Target metrics

A/B testing

Learn a bit more about your analytics

  • How your content is performing and whether it is keeping people engaged
  • How site elements or offers are performing
  • What people are doing on your site and the routes they typically take
  • Where people are landing, as well as where they’re leaving your site
  • Where your funnel is losing money

You have to approach your testing and analytics with a problem so you can find an answer in the data. This way, you can identify issues and confirm what aspects you need to change. Keep in mind the data won’t specifically tell you how to fix problems but it’s a great starting point to discover actionable insights. Without this data, and without the ability to interpret it, A/B testing is pointless.

Talk to your visitors

As I already mentioned, besides knowing what is going on, you need to know Why is it happening as well. Interviewing users about the Whys underpinning their points of view on price, reviewing the video of people exploring the product, and opinions about design.

You can find 5 to 10 people who represent your target audience and are not familiar with your site. Have them complete tasks regarding it and pay attention to these three critical areas:

  • Learn How they understand your site and what it is about
  • See if they have had confusion or difficulty when checking out any business critical tasks
  • Understand How they would like to buy and what matters to them

Qualitative surveys

The best way to get qualitative data from your visitors is through surveys. Your goal is to find out why visitors buy and why they leave without purchasing. You can use one of these survey tools:

  • Qualaroo – This tool is for when you need to survey current visitors of your site. Tool allows you to set up behavior-triggered surveys to find out why your customers are behaving the way they do.
  • Surveymonkey – SurveyMonkey has been around for a long time and offers a stable experience. Has great analytical tools so you can analyzing your responses precisely.
  • Google Docs – Free and easy to use. Data is collected into spreadsheets which make the data easy to analyze.
  • Typeform – The prettiest survey tool of them all.

User session replays

You don’t need a million visitors to record user sessions – this is almost like qualitative data. Use tools like Inspectlet, SessionCam, or Clicktale to record user sessions. Then watch your actual visitors interact with your site. Session replays are extremely useful for observing how people fill out forms on your site. You can configure event tracking at Google Analytics. But, it won’t provide the level of insight that user session replay videos do.

Use segmentation to get actionable data

By default, a lot of the data you get is useless in reality. The number of visits to your site doesn’t really give you any actionable information. To get actionable data for testing, you need to segment the data you have.

“Never report a metric (even the most hyped or your favorite KPI) without segmenting it a few levels deep. That is the only way to get deep insights into what that metric is really hiding or to see what valuable information you can use.”

Avinash Kaushik has outlined a simple strategy for segmenting your traffic.


The reason for the in-depth analysis is to get a clear understanding of what is happening on your website and find areas where A/B testing can bring the best results! Your time would be better spent with your analytics, where you can set up goal tracking, creating marketing campaigns, and developing your content offers and resources. Focus on growing your business for now. Learn more about your customers, make them happy. The A/B testing can come later once you have more to offer and traffic has grown substantially.


Leave a Reply

Your email address will not be published. Required fields are marked *