A holdout test deliberately withholds marketing from a portion of your audience or market to compare outcomes against the exposed group. The difference between the two reveals the true causal effect of the activity. It is the practical mechanism behind most incrementality testing.

A holdout test feels wrong because you are deliberately not marketing to part of your market, and that is exactly why it produces honest answers. It is the only way to see what happens without the spend. Run one before you double a channel's budget, because it will either justify the increase or save you from an expensive mistake.

Example:

Running a campaign in most regions while withholding it from one comparable region shows whether results differ, isolating what the campaign actually caused.

How does a holdout test work?

You withhold marketing from a comparable segment or region, then compare its outcomes against the exposed group. The gap shows the activity's causal effect.

Why run a holdout test before scaling spend?

It reveals whether the channel genuinely causes additional revenue, so a budget increase is based on proven causation rather than attributed credit.