A/B Testing Decision Quality Memo

How to read this report

A completed experiment needs an operator-facing decision memo. The growth team has results but someone needs to determine whether those results justify changing a page, product flow, or funnel step. Ship a false positive and you degrade conversion for weeks; sit on a real winner and you bleed revenue rerunning tests that already answered the question. Use this when the team needs to move from "we have data" to "we have an approved decision."

Outcome Label Quality

Every experiment produces metric movement, but movement is not a decision. The first job is to classify the result into exactly one label: win, loss, inconclusive, learning-only, or retest. Without that classification, teams default to "it went up, ship it" and skip the caveats that determine whether the lift is durable.

What to check:

Decision rule: If the result label is unclear, hold the decision and write a retest or learning note. Acting on ambiguous evidence is the most common source of wasted engineering cycles after experiments.

• Primary KPI movement direction and magnitude
• Guardrail metrics (did anything break while the target improved?)
• Confidence interval width and overlap
• Sample size relative to minimum detectable effect
• Anomaly notes (outages, traffic spikes, bot contamination)

Validity Caveat Visibility

A result can be statistically significant and still misleading. Instrumentation gaps, segment imbalance, seasonality, or QA issues can produce numbers that look decisive but collapse under scrutiny. The review must surface every caveat that could change the recommendation.

What to check:

Decision rule: If a caveat could reverse the decision, do not publish the recommendation as final. Mark the memo as caveated and route it for additional review before implementation begins.

• Instrumentation notes (were events firing correctly for the full duration?)
• Sample balance across segments and devices
• Segment-level results (did the lift hold for all key cohorts or only one?)
• Seasonality overlap (holiday traffic, end-of-quarter buying, promotional periods)
• QA issues or monitoring exceptions during the test window

Business Impact Translation

A positive result is not automatically worth implementing. The review must translate percentage lifts into realistic revenue numbers without hiding assumptions. Teams that skip this step approve changes based on relative improvement ("12% lift!") without checking whether the absolute gain justifies implementation cost.

What to check:

Decision rule: If the business case depends on an assumed number, keep it as a scenario rather than a mandate. This prevents the team from committing engineering resources to a change whose value depends on unvalidated projections.

• Current traffic volume at the test point
• Observed conversion movement (not projected)
• Revenue or value per conversion (observed, not averaged from unrelated periods)
• Confidence range (best, expected, worst case)
• Cost to implement at full scale
• Dependencies on other teams or systems

Decision and Learning Reuse

Every experiment produces two outputs: a decision (do this next) and a learning (carry this forward). Most teams conflate these, so learnings get lost in tickets and decisions get repeated because nobody documented the reasoning.

What to check:

Decision rule: If ownership or applicability is unclear, approve the learning note but hold execution. A learning without boundaries becomes dogma; a decision without an owner becomes a suggestion.

• Decision log entry (what was decided, by whom, on what date)
• Next action with explicit owner
• Learning summary in reusable language
• Applicability boundaries (where this applies and where it does not)

Conversion Quality and Measurement Confidence

Not all conversions carry equal decision weight. Diagnostic events (scroll depth, time on page) inform hypotheses but should not drive ship decisions. Attribution-caveated signals require downstream validation before they justify changes.

What to check:

Decision rule: If conversion quality is unknown, keep the recommendation caveated until the downstream source is reviewed. Shipping based on a proxy metric without confirming it connects to business value produces optimization theater.

• Conversion action type (transaction, signup, qualified lead vs. micro-conversion)
• Downstream quality source (does the conversion lead to real business value?)
• Attribution model and its known limitations

Review checklist

Use these checks to keep the recommendation approval-gated before the team changes the page, campaign, workflow, or reporting setup.

• Result classified into exactly one label (win, loss, inconclusive, learning-only, retest)
• All validity caveats surfaced and written into the memo
• Business impact calculated using observed numbers, not projections
• Assumptions clearly separated from observations
• Learning note written with applicability boundaries
• Next action has an explicit owner
• Approval status marked (approved, held, needs more evidence)
• Follow-up gated on reviewer sign-off

Worked Example

A checkout flow test shows a 9% lift in completion rate over three weeks with adequate sample size.

Label: win. However, the test ran during a promotional period and mobile showed no lift (desktop carried the result). Two caveats: seasonality overlap, segment imbalance.

Publish the winner to desktop only. Schedule mobile validation. Write impact note using conservative numbers. Log the learning: this layout works differently across device cohorts.

The business case should model only the desktop cohort at non-promotional traffic levels. The 9% overall lift overstates durable impact.

Approval boundary

Pass: The memo has a clear outcome label, all known caveats are visible, the business case uses observed inputs, and the next action has an owner and approval status. Fail: The outcome label is ambiguous, a caveat is missing or buried, the business case relies on unverified assumptions presented as facts, or the next action lacks an owner. Any of these means the memo should be held for revision.

Sample review note

10X should review A/B Testing Decision Quality Memo, compare the decision evidence with the caveats, and keep the next recommendation approval-gated until the reviewer accepts it.