Holdout test: designs, minimums and readouts

A holdout test changes spend in some areas and compares their signed cases with matched controls, under rules written down before it starts. It is how the Standard answers whether a channel creates cases or only takes credit for them.

Designs

• Step tests (preferred). Treatment areas step a channel’s budget down or up, for example 50% down or 30% up, while controls stay on plan. They answer the marginal question and keep markets open.
• Holdouts. Treatment areas switch a channel off: used for brand search, a Local Services Ads category, social prospecting or a Spanish program, with limits on duration and on how much of the market is held out.
• Matched markets and synthetic control. Treatment and control units are media markets, counties or ZIP clusters the platforms can target, chosen by pre-period fit with Meta’s open-source GeoLift (GitHub) and read against a synthetic-control counterfactual.
• Platform lift studies count only when we can see the design and the data, as with Google’s geo-based Conversion Lift, which can use the firm’s offline conversions (Google). Otherwise they’re labeled “platform-reported”.
• Never causal on their own: before-and-after comparisons without a control, and platform “modeled” incrementality we can’t audit.

Minimums

Power

A power analysis runs the planned design on the firm’s own history first. Results carry two-sided 90% intervals (alpha 0.10, GeoLift’s default, walkthrough), and the design needs at least 80% power at its minimum detectable effect, the conventional threshold (PMC). The minimum detectable effect may be no larger than the smallest effect that would change the decision. If no design within 8 weeks reaches that, we don’t run the test; we say so and propose a larger design or a different question.

Pre-registration

A dated one-page plan goes to the firm before launch: the question and expected direction; the primary metric (signed cases from leads created during treatment, from the firm’s records) and at most two secondary metrics; the design and how areas were assigned; the budget change; the dates; the minimum detectable effect and power; the analysis method; exclusions, integrity checks and stopping rules; and the decision rule for a win, a null or an inconclusive result. Any later change is logged and shown beside the original plan.

Reading the result

An interval that excludes zero in the opposite direction is reported as a reverse effect. Every readout shows the estimate and interval, incremental cost per signed case, attributed cases in the same areas, the incrementality factor, the pre-period fit, the integrity log and the decision.

Cite as: The Probative Standard v1.0, Holdout test. Last reviewed October 24, 2026, by the Probative editorial team.