Experiment Validity Readiness Checklist

How to read this checklist

A growth team is about to assign traffic to an A/B test. The variations are built and someone is asking for a green light. Use this checklist as the gate between "ready to build" and "ready to measure." The risk is not that the experiment fails to run -- it is that it runs, produces a number, and the team ships a decision built on measurement that was never trustworthy.

Sample and Exposure Gate

Traffic volume and stability determine whether the experiment can detect a real difference or will only produce noise that looks like signal.

What to check:

Decision rule: Hold when volume is too low, traffic is unstable, or the decision would rely on a tiny segment. A test that needs 14 days at 50/50 but launches at 90/10 on a holiday week will not produce a trustworthy read.

• Traffic forecast against minimum detectable effect
• Baseline conversion rate stability over the past 2-4 weeks
• Split allocation percentage across variants
• Minimum run time given current daily volume
• Exclusion rules that shrink the eligible audience
• Seasonality patterns that could distort the exposure window

Metric Ownership Gate

An experiment without a clearly owned primary metric invites post-hoc storytelling. When success criteria are ambiguous, teams pick whichever number moved favorably and call it a win.

What to check:

Decision rule: Hold when the primary metric is ambiguous, unowned, or disconnected from the decision.

• Primary KPI with an unambiguous event definition
• Guardrail metrics that flag collateral damage
• Dashboard location where the metric is already tracked
• Named metric owner who will interpret the result
• Caveats on attribution or measurement lag

QA and Assignment Gate

Assignment integrity is the foundation of causal inference. If users shift between variants or events do not fire consistently, the result is a reporting error dressed as evidence.

What to check:

Decision rule: Hold when QA is incomplete, assignment can change midstream, or tracking cannot be trusted. A broken mobile event means conversions are invisible on 40% of traffic, and the result reflects desktop-only behavior presented as universal.

• Control variant rendered and functional across browsers and devices
• Treatment variant rendered and functional across browsers and devices
• Assignment logic confirmed as sticky (no re-bucketing on revisit)
• Event firing validated in tag manager for both variants
• Rollback owner identified in case assignment breaks mid-test

Monitoring and Peeking Gate

Peeking at results without a pre-agreed rule is the fastest path to false positives. Early movement is expected noise, not signal. Teams that react to day-two lifts are confirming bias, not running experiments.

What to check:

Decision rule: Hold when the team plans to react to early movement without a pre-agreed rule. If the team channel celebrates a day-three lift and pressures a ship decision, the monitoring plan has already failed.

• Named monitoring owner responsible for anomaly detection
• Anomaly threshold that triggers investigation (not early calls)
• Minimum read date before any directional interpretation
• Stop rule for genuine harm (safety valve, not optimization)
• Communication channel for status updates that does not invite premature reads

Result Interpretation Gate

A result without interpretation rules produces a number but not a decision. "Variant B won" means nothing without the confidence caveat, business impact translation, and conditions under which you would not ship despite a positive signal.

What to check:

Decision rule: Hold when the result would be reported as a winner without caveats or business context. A significant lift on a metric nobody acts on is not a win -- it is a distraction that consumed test capacity.

• Outcome labels defined: win, loss, inconclusive, learning note
• Confidence caveat and what "close but not significant" means
• Business impact framing (what does the lift mean in revenue terms)
• Next action for each outcome (ship, iterate, retest, kill)
• Named reviewer and approval status gating implementation

Review checklist

Use these checks to keep the recommendation approval-gated before the team changes the page, campaign, workflow, or reporting setup.

• Traffic forecast supports minimum detectable effect within planned run time
• Baseline conversion rate is stable
• Split allocation matches power calculation
• Primary metric has a single unambiguous event definition
• Guardrail metrics named and dashboarded
• Metric owner identified
• Both variants pass QA on target devices and browsers
• Assignment logic is sticky and validated
• Events fire correctly for both variants
• Monitoring owner named with minimum-read-date commitment
• Stop rule exists for genuine harm only
• Outcome labels pre-defined (win/loss/inconclusive/learning)

Worked Example

The conversion lead is trying to test a new checkout flow, but the review has to confirm the evidence before changing the page, offer, or experiment decision. Traffic is 800 sessions/day, baseline conversion is 3.2%, target is a 10% relative lift.

At 50/50 split, minimum run time for 80% power is 28 days. The team planned 14 days. The primary metric event fires on confirmation page load, which also triggers on order-status revisits -- creating a double-counting risk.

Hold launch. Extend to 28 days minimum, redefine conversion event to first-visit confirmation only.

Sample gate fails on timeline; metric definition has a deduplication gap.

Approval boundary

Pass: All five gates have visible evidence, named owners, and no unresolved caveats. Fail: Any gate has missing evidence, an unnamed owner, or an unresolved caveat. The test may run, but it cannot produce a result the team should act on. Hold until the gap is closed.

Conversion quality and measurement confidence

Evidence to review: Conversion action, diagnostic event, downstream quality source, attribution caveat, and value signal.

• Separate decision-driving conversions from diagnostic events and caveated attribution signals.
• If conversion quality is unknown, keep the recommendation caveated until the downstream source is reviewed.
• Conversion quality and measurement confidence is supported by visible inputs and the caveat is clear.

Funnel math and scenario quality

Evidence to review: Traffic unit, stage conversion, offer value, expansion path, revenue timing, and confidence label.

• Separate observed inputs from assumptions before treating a scenario as decision evidence.
• If the model is sensitive to an assumed number, keep the recommendation as a scenario until the source is verified.
• Funnel math and scenario quality is supported by visible inputs and the caveat is clear.

Commerce and revenue quality

Evidence to review: Product performance, order quality, payment signal, cash timing, and margin or payback caveat.

• Connect campaign or funnel movement with commerce and payment context before judging quality.
• If revenue quality or cash timing is missing, avoid turning source movement into a payback conclusion.
• Commerce and revenue quality is supported by visible inputs and the caveat is clear.

Operating failure modes

Evidence to review: Implementation status, lead flow, delivery quality, follow-up owner, and customer-result feedback.

• Separate a funnel leak from an operating leak, such as no follow-up, no promotion, weak delivery, or no owner.
• If the operating owner or follow-up path is unclear, mark the recommendation as a process fix before a creative fix.
• Operating failure modes is supported by visible inputs and the caveat is clear.

Sample and exposure gate

Evidence to review: Traffic forecast, baseline conversion, split allocation, minimum run time, exclusion rules, and seasonality caveat.

• Confirm the test has enough stable exposure and conversion volume to support the decision.
• Hold when volume is too low, traffic is unstable, or the decision would rely on a tiny segment.
• Sample and exposure gate is supported by visible inputs and the caveat is clear.

Sample review note

10X should review Experiment Validity Readiness Checklist, compare the decision evidence with the caveats, and keep the next recommendation approval-gated until the reviewer accepts it.