Incrementality Is the Question Your Agency Doesn't Want You to Ask

Quick Answer

Incrementality is the question of what your ad spend actually caused, versus what would have happened anyway. A brand can show high ROAS and zero incremental revenue if most of the attributed sales would have come through organic, branded search, email, or word of mouth. Testing incrementality is straightforward (geo holdouts, audience holdouts, or simple on/off tests), but most agencies do not run them because the results are usually unflattering. If you have never seen an incrementality test on your account, you are likely paying for revenue you were going to get anyway.

The TikTok Tide Problem

A brand we audited last year was running about $40K/month in Meta and Google. ROAS looked great. Then we noticed something: the same brand had a single product blowing up on TikTok organically. Sales were climbing every week with no paid push.

Meta and Google were both happily taking credit. Their dashboards looked like the campaigns were the driver. They were not. The campaigns were riding a tide that started somewhere else. When we ran a holdout test, the incremental ROAS on Meta came in at about 1.4x. The platform was reporting 4.2x. The gap was the TikTok tide, plus some baseline branded search demand that Meta retargeting was capturing without creating.

This is incrementality in one example. The question is never "did revenue happen after the ad ran?" The question is "would the revenue have happened anyway?" Almost no platform metric can answer that. The platforms have no incentive to find out.

What Incrementality Actually Measures

Imagine you run two versions of the same week. In one, your ads run normally. In the other, you turn the ads off for a defined slice of your audience or geography. The difference between the two is the incremental contribution of the ad spend.

That difference is almost always smaller than the platform claims. For a healthy direct-response account, incremental ROAS often runs 40-70% of platform-reported ROAS. For brand-heavy or organic-strong accounts, it can run lower than that. We have seen one brand where the incremental ROAS of Meta retargeting was actually negative once you accounted for the cost of the ads and the cannibalization of organic conversions. The agency had been reporting a 6x on that line item for nine months. The brand had been paying for the privilege of intercepting its own traffic.

This is not a reason to stop running ads. It is a reason to be honest about what they are doing. Some channels run at high incrementality (often top-of-funnel prospecting). Others run at low incrementality (often retargeting and branded search). The job is to know which is which and shift the mix accordingly.

How to Run an Incrementality Test on a Small Budget

You do not need a stats team or a six-figure budget to do this. Three approaches work even at $30K-$50K/month in spend.

Geo holdouts. Pick two or three matched markets. Turn ads off in one set, leave them on in the others. Run for two to four weeks. Compare revenue per market. For a brand spending $40K/month, a workable design is to pause Meta in three matched DMAs that together represent about 10% of historical revenue. Run for four weeks. Compare the paused DMAs to a control set of similar DMAs. If revenue in the paused markets drops by 35%, you have evidence that Meta is driving roughly that much incremental contribution. If revenue stays flat, Meta is largely non-incremental in those markets and probably elsewhere.

Audience holdouts. Inside Meta or Google, exclude a percentage of your retargeting audience from seeing ads. Compare conversion rates between the holdout and the exposed group. This works particularly well for retargeting, where the question of "would they have converted anyway?" is sharpest. A 10% holdout for four weeks usually gives you enough signal to decide whether retargeting is earning its budget. The trap is forgetting to maintain the holdout consistently. If your other ad sets bleed impressions into the holdout group, the test is contaminated.

Pause-and-watch. The crudest version, and still useful. Turn one campaign off for two weeks. See what happens to total brand revenue. If nothing moves, you have your answer. This is the test we run when a client is skeptical of the more rigorous designs. It is less defensible statistically, but it produces a moment of clarity that is hard to argue with. The downside is that you cannot isolate the cause as cleanly. You will know revenue did or did not move, but you may not know why.

None of these are perfect. All of them are better than guessing. The biggest failure mode is running one of them once, getting an inconclusive result, and deciding incrementality testing does not work. The right cadence is once per quarter for the top one or two channels, with a willingness to refine the design when the first run is noisy.

What to Do With the Result

Three common outcomes, three different responses.

If incremental ROAS comes in at 70%+ of platform-reported ROAS, the channel is doing real work and you can keep scaling with confidence. Most prospecting campaigns on Meta and Google fall here.

If incremental ROAS comes in at 30-70% of platform-reported ROAS, the channel is contributing but a meaningful portion of the attributed revenue would have happened anyway. The right move is to reset the ROAS target downward to account for the gap, and to start questioning whether the marginal dollar in that channel is still profitable. Often the right move is to redirect the bottom 20% of spend on that channel to a new channel or to top-of-funnel work.

If incremental ROAS comes in below 30% of platform-reported ROAS, the channel is largely non-incremental and the budget should be cut significantly or redirected. We see this most often on Meta retargeting for established brands with strong organic search and email programs. The retargeting is mostly intercepting buyers who were already on a path to convert.

The hardest part of incrementality testing is acting on the result. It is easier to commission the test than to actually cut a channel that has been a comfortable presence in the budget for two years. The brands that get the most value from incrementality testing are the ones who treat the result as a directive, not as input for a debate.

Why Agencies Skip These Tests

Three honest reasons.

The first is the result. Incrementality tests usually shrink the agency's apparent contribution. No agency wants to commission a report that lowers their own number.

The second is the conversation. Once you have an incremental ROAS, you can no longer hide behind "platform-attributed." The agency has to explain real returns on real spend, and that requires a more skilled team than most rosters have. Strategists who have spent their careers reporting platform numbers often struggle when the conversation shifts to incrementality, because the analytical vocabulary is different.

The third is the workflow. Running and analyzing a geo holdout takes hours of setup. Most agencies are billing by the retainer and do not want to absorb that cost unless the client demands it. The agencies that build incrementality testing into their standard offering can spread the cost across the client base. The ones who treat it as a one-off project tend to charge extra or skip it entirely.

We run an incrementality test on every new account in the first 90 days. The number we find is usually the most important number we share that quarter, and it sets the baseline for everything else. Once you know the incremental ROAS of each major channel, the budget conversation becomes a math problem instead of an argument.

A 12-Month Incrementality Program

For brands that want to operationalize this, here is the cadence we run:

Months 1-3: Initial baseline test on the highest-spending channel. Geo holdout for 4 weeks, results reviewed in week 5.

Months 4-6: Second-highest-spending channel gets the same treatment. By this point you have two incremental ROAS numbers and can start making mix decisions with real data.

Months 7-9: Retargeting holdout across Meta and Google. This is usually where the biggest surprises live, and the biggest budget reallocations come out.

Months 10-12: Re-test the original channel from months 1-3 to see how incrementality has shifted as the mix has changed. This is the test that proves the program is working.

The total cost of running this for a year is usually under $20K in absolute revenue impact (the dip during holdout weeks) for a brand spending $40K-$60K/month. The information value is usually 5-10x that, in the form of mix decisions that were impossible without it.

How to Ask for One

Send this to your agency:

Can we set up a basic incrementality test on our top-spending channel this quarter? I want to know what the campaigns are actually adding versus what would have happened anyway. A geo holdout or audience holdout would be fine. Can you put together a plan and a budget impact estimate by next Friday?

If they say it is not possible at your scale, get a second opinion. We have done it for brands spending under $20K/month.

FAQ

Will an incrementality test hurt my revenue? A well-designed test affects 10-20% of your audience for two to four weeks. The revenue impact is small and short-lived, and the information you get back changes how every future dollar gets spent.

What is the difference between incremental ROAS and platform ROAS? Platform ROAS counts every conversion the platform claims credit for. Incremental ROAS counts only the conversions that would not have happened without the ad spend. The first is what your agency reports. The second is what your business actually earned.

How often should I rerun an incrementality test? Once a quarter is enough for most brands. More often if you are making big changes to channel mix or creative strategy.

What is a good incremental ROAS? Depends on margin profile. For most eCom brands with 35-45% contribution margins, an incremental ROAS above 2.0x is healthy. Above 3.0x is strong. Below 1.5x is a flag.