Why Bayesian?
EarlyCRO uses Bayesian statistics instead of traditional frequentist methods. Here's why this matters for you:
Frequentist (Traditional)
- - "95% confidence interval"
- - Requires fixed sample size
- - Can't peek at results early
- - p-values are confusing
Bayesian (EarlyCRO)
- + "92% chance B beats A"
- + Check results anytime
- + Intuitive probability statements
- + Faster decisions
Key Metrics Explained
Probability to Beat Control
This is the most important number. It tells you the probability that a variation's true conversion rate is higher than the control.
- 95%+: Strong winner - safe to implement
- 80-95%: Likely winner - consider risk tolerance
- 50-80%: Inconclusive - need more data
- <50%: Control is likely better
Expected Lift
The estimated percentage improvement over control. Shown with a credible interval (e.g., +5.2% [+2.1%, +8.3%]).
Credible Interval
The range where we're 95% confident the true value lies. Narrower intervals mean more certainty.
Reading Your Results
Here's how to interpret the results dashboard:
// Example Results
Control: 3.2% conversion rate (1,247 visitors, 40 conversions)
Variant B: 4.1% conversion rate (1,253 visitors, 51 conversions)
Probability B beats Control: 89.3%
Expected Lift: +28.1% [+4.2%, +56.3%]
In this example, there's an 89.3% chance that Variant B is genuinely better. The expected improvement is about 28%, but could be anywhere from 4% to 56% improvement. We'd recommend waiting for more data before declaring a winner.
Statistical Significance
Unlike frequentist statistics, Bayesian analysis doesn't have a binary "significant/not significant" threshold. Instead, you get a continuous probability that helps you make informed decisions.
Decision Framework
We recommend waiting until probability to beat control reaches 95% for most decisions. For high-stakes changes, consider waiting for 99%. For quick iterations, 90% may be acceptable.
Common Pitfalls to Avoid
Stopping too early
Even with Bayesian stats, early results can be misleading. Wait for at least 100 conversions per variant.
Ignoring the credible interval
A 95% winner with a wide interval (e.g., -5% to +40%) is less certain than it appears.
Testing too many variations
Each variation needs its own traffic. More variations = longer tests. Stick to 2-4 variations max.
Seasonality effects
Run tests for at least one full week to account for day-of-week variations in traffic.
The Math (Optional)
For the technically curious, here's how our Bayesian engine works:
- We model conversion rates using a Beta distribution
- Start with a uniform prior: Beta(1, 1)
- Update with observed data: Beta(conversions + 1, visitors - conversions + 1)
- Run 100,000 Monte Carlo samples to calculate probability to beat control
- Calculate credible intervals from the posterior distribution
// Simplified algorithm
posterior_A = Beta(conv_A + 1, visitors_A - conv_A + 1)
posterior_B = Beta(conv_B + 1, visitors_B - conv_B + 1)
// Monte Carlo simulation
wins = 0
for i in range(100000):
sample_A = posterior_A.sample()
sample_B = posterior_B.sample()
if sample_B > sample_A:
wins += 1
probability_B_beats_A = wins / 100000