Parent: Experimentation Brain
Destination: mwmsbrain.site
Page Type: Operational Evaluation Page
Source of Truth: MCR
Purpose
Signal Confidence Framework is used to judge how reliable a test result actually is.
This prevents MWMS from scaling weak signals or rejecting opportunities too early.
What This Page Does
- evaluates strength of test signals
- separates noise from real performance
- protects against false positives and false negatives
- supports better scaling and kill decisions
Signal Quality Check
Assess:
- consistency of results
- stability of performance over time
- alignment with hypothesis
- clarity of conversion path
- absence of tracking issues
Confidence Factors
Check:
- sufficient data volume
- repeatable conversions
- stable CPA or performance metric
- no major outliers skewing results
- tracking reliability
- clean traffic quality
Confidence Levels
Classify the signal:
- High Confidence → strong, stable, repeatable signal
- Medium Confidence → promising but not fully proven
- Low Confidence → inconsistent or unclear
- No Confidence → no usable signal
False Signal Check
Watch for:
- one-off conversions
- random spikes
- poor traffic quality
- broken tracking
- funnel issues masking true performance
- short test duration
Decision Support
Use confidence level to guide decisions:
- High Confidence → consider scaling
- Medium Confidence → continue testing
- Low Confidence → adjust and retest
- No Confidence → stop or restart
Next Step
Use this with:
Test Result And Decision Workflow
Rule
A result is only as good as its confidence level.
Never scale low-confidence results.
MCR remains the source of truth.