How do I measure incrementality in ecommerce?

Attribution allocates credit for orders that happened. Incrementality asks a different question: how many of those orders would have happened anyway.

The distinction matters most where a channel talks to people who were already going to buy. Retargeting is the clearest case. An ad shown to someone who has your product in their cart will be credited with the sale, and a share of those people were coming back regardless. Branded search sits in the same category. So does any campaign heavily weighted towards existing customers.

This is why an account can report a strong return while the business grows slowly. The reported number is real in the sense that those orders happened and the ad was involved. It is misleading as a measure of what the spending caused.

The only way to find out is to withhold the activity from some group and compare what happens. No model, no attribution setting and no tagging scheme substitutes for that, because every one of them works from data where the ads always ran.

The good news is that the simplest version of the test is free and takes a fortnight.

Pick one channel or one campaign type, switch it off completely for a defined period, and watch total store revenue. Not attributed revenue. Total.

Choose the period carefully. It needs to be long enough to cover a full buying cycle for your products, which for most stores means at least two weeks and often four. It should avoid a sale, a holiday and any launch. And you should have a comparable period immediately before it, ideally with the same weekday pattern.

What you are looking for is whether total orders and total revenue fall by anything like the amount the channel claimed. If a campaign was reporting a large share of your revenue and total revenue barely moves when it stops, most of that revenue was being reallocated rather than created.

The weaknesses of this test are real and worth stating. Other things change during the same period, seasonality moves underneath you, and a single on and off cycle can mislead. Repeat it, ideally alternating on and off across several periods, before you act on a large budget decision.

Start with the channels where you most suspect overlap. That is usually retargeting and branded search.

A better test runs the channel and does not run it at the same time, in comparable places. This removes seasonality, because both groups experience the same week.

Split your market into two groups of regions, cities or postal areas that behave similarly. Match them on order volume and product mix rather than on population, and check the match against several months of history before you start.

Then run the channel normally in one group and pause it in the other. Compare total revenue per group over the test window against each group's own baseline. The difference between the changes is your estimate of the incremental effect.

This works well for channels that can be targeted geographically, which covers most paid media. It does not work for email or for anything delivered to a list that is not location based.

Size the test honestly. If your order volume in each group is small, normal week to week variation will be larger than the effect you are trying to detect, and you will get a confident looking answer that means nothing. Fewer, larger groups over a longer window beats a fine grained split you cannot read.

Several ad platforms offer built in experiments that hold back a random share of your audience and compare outcomes. These are convenient and they answer the question inside the platform's own measurement, which is both their strength and their limit.

Use them for comparing two versions of something, such as a campaign structure or a bidding approach, where you want a clean split with the seasonality removed. Treat their conclusions about total business impact more cautiously, because the platform measures the orders it can see.

Audience holdouts work well for email and SMS, where you control the list directly. Hold back a random share of the recipients from a flow or a campaign for a month, then compare revenue per recipient across the two groups. This is the most reliable incrementality test most stores have available to them, and it is often the most surprising, because a large part of the revenue attributed to abandoned cart and win-back messages turns out to be people who would have returned anyway.

Whatever the mechanism, the principle does not change. A group that got the activity, a comparable group that did not, and a comparison of outcomes rather than of credit.

Write down five things before you start, because a test decided after the fact is not a test.

What you are turning off or holding back, exactly. Which group is the control. How long it runs, chosen from your buying cycle rather than from impatience. Which single number you will judge it on, which should be total revenue or total gross profit. And what result would change your decision.

That last one is the important one. If no plausible outcome would change what you do, do not run the test.

During the test, change nothing else. No new creative, no price changes, no site releases, no extra email sends. One variable.

Afterwards, be honest about noise. If the change you measured is within the range your revenue normally moves week to week, the result is inconclusive, and saying so is a legitimate outcome. Run it again for longer rather than reading a story into the noise.

Repeat the important tests once or twice a year. A channel that was incremental when it was small can stop being incremental as it grows and starts overlapping with everything else you do.