Datalad calculator
Bayesian vs Frequentist Comparator
The same test result, read both ways, side by side. They are not two opinions about one question. They are careful answers to two different questions, and most arguments about which is right are really arguments about which question you meant to ask.
1
Your result
Only the Bayesian side uses this
Absolute lift in percentage points
Frequentist side only
2
The same data, two answers
Frequentist
If there were really no difference, how surprising would this result be?
0
p value
Control rate0
Variant rate0
Absolute lift0
Relative lift0
Confidence interval0
z statistic0
Bayesian
Given what we have seen, how likely is it that the variant is actually better?
0
chance the variant is better
Posterior control0
Posterior variant0
Most likely lift0
Credible interval0
Risk of shipping it0
Beats your threshold0
3
The two posteriors
Where the two humps overlap is the honest uncertainty. A p value compresses all of that into one number and then a threshold turns it into a yes or no, which is where most of the information goes missing.
4
[…] Bayesian vs Frequentist Comparator […]
[…] Bayesian vs Frequentist Comparator […]
[…] prior belief and what is worth shipping A “reading it” section for the interpretation Open the comparator Free · runs in your browser · no […]