Parent: Experimentation Brain
Destination: mwmsbrain.site
Page Type: Operational Validation Page
Source of Truth: MCR
Purpose
Statistical Confidence Framework is used to confirm whether test results are statistically reliable.
This page protects MWMS from making decisions based on insufficient data or random variation.
What This Page Does
- checks if enough data has been collected
- validates whether results are meaningful
- reduces risk of false positives
- reduces risk of false negatives
- supports evidence-based decisions
Data Volume Check
Confirm:
- sufficient number of clicks or views
- sufficient number of conversions
- test has run long enough
- results are not based on very small samples
Consistency Check
Assess:
- performance stability over time
- absence of extreme spikes or drops
- consistent conversion behaviour
- stable cost per result
Sample Reliability
Check:
- sample size is meaningful
- results are not dominated by a few events
- traffic distribution is consistent
- no bias in traffic source
Confidence Levels
Classify:
- High Statistical Confidence → enough data, stable results
- Medium Statistical Confidence → borderline data, needs more testing
- Low Statistical Confidence → not enough data or unstable
- No Statistical Confidence → unreliable data
Risk Indicators
Watch for:
- very low conversion count
- sudden spikes in performance
- inconsistent traffic quality
- short test duration
- uneven data distribution
Decision Support
Use statistical confidence with signal confidence:
- High + High → strong candidate for scaling
- Medium → continue testing
- Low → adjust and retest
- None → stop or restart
Next Step
Use alongside:
Signal Confidence Framework
Test Result And Decision Workflow
Rule
Do not trust results without enough data.
Statistical confidence protects against random outcomes being mistaken for real performance.
MCR remains the source of truth.