Why Bayesian Tests Beat Traditional Split Tests

If you've ever run an A/B test, you've probably used frequentist statistics. Here's what that means and why it's wrong for most testing.

Frequentist: the old way

You pick a sample size upfront. You let the test run until you hit that number. You check once at the end. You get a p-value.

Problems with this:

- You have to guess traffic volume in advance - Checking results early "invalidates" the test (which is a weird thing to say, but true in this framework) - Low-traffic sites wait months - The p-value doesn't tell you what you want to know

That last one is the big one. A p-value of 0.04 doesn't mean "96% chance B is better than A." It means something much more convoluted about the probability of observing this data if the null hypothesis were true. Nobody actually thinks in those terms.

Bayesian: what Helix uses

The test runs until the data is clear. Every new visitor updates the probability that each variant is best. You see live confidence, in real time, in the units you actually care about: "75% chance B wins."

Benefits:

- No fixed sample size — tests end when they're ready - Peeking early is fine - Low-traffic sites still work - The output matches your intuition

Thompson Sampling

Helix routes traffic with Thompson Sampling. Early on, when it doesn't know which variant is best, it splits traffic evenly. As one pulls ahead, it sends more traffic to the leader while still exploring the others.

This means losers get less traffic (so you lose less revenue during the test), winners get confirmed faster, and you don't waste impressions on obviously bad variants.

What this changes in practice

Check the dashboard anytime. You're not breaking anything.

A site with 50 visitors a day still works. It just takes longer — maybe 2-4 weeks instead of 2-4 days.

When the probability hits 95%, a winner gets declared automatically. That's the rule. You don't need to understand the math.

It's the difference between GPS and a printed map. Both get you there. One updates.