Your ROAS is up.
Revenue is flat.
Both numbers are correct.
A practical guide to marketing measurement when tracking, attribution, and experiments are imperfect, which is always. Written by Reid Rhodes, a marketing data scientist working on measurement, incrementality, and growth decisions.
The gap, measured
- Reported CAC
- $59
- Incremental CAC
- $88
A six-week geo holdout across 40 markets put the lift at +7.2% (95% CI +4.0% to +10.5%). Reported CAC was off by a third.
Start with the question
What are you trying to understand?
The method follows from the question. Each of these opens to the evidence that answers it, and to what that evidence still will not tell you.
-
Can I trust my tracking? Observe
Audit identity, consent, and event definitions. Measure coverage loss by segment rather than as one accuracy number. Random loss costs you precision. Systematic loss costs you validity, and only the second one changes your conclusions.
What it will not tell you Give you the true conversion count. Better tracking narrows the gap between what you see and what happened. It never closes it.
Go deeper → -
Did this campaign create incremental growth? Estimate
A holdout: geo, user, or cluster, depending on what you can withhold cleanly. Settle the design before the estimator. If you cannot construct a credible control, no model recovers one for you.
What it will not tell you Tell you the effect at a spend level you did not test. A holdout measures the campaign you ran, not the one you are planning.
Go deeper → -
Why do my attribution models disagree? Describe
They are different rules for splitting credit across the same observed journeys. None of them estimates a counterfactual, so disagreement between them is the expected result.
What it will not tell you Be fixed by choosing a better model. No attribution model becomes causal by becoming more sophisticated.
Go deeper → -
How do paid search and SEO interact? Explain
Pause or geo-test branded paid search and measure total branded conversions, not paid conversions. You are asking how much paid captured demand that organic would have caught anyway.
What it will not tell you Be answered from channel reports. Both channels count their own conversions, and the overlap is invisible to each of them.
Go deeper → -
Can I use a marketing mix model? Explain
With two or more years of history, real variation in spend, and experiments to calibrate against, yes, for budget-level questions. MMM is a strong allocation tool and a weak causal one.
What it will not tell you Identify a causal effect on its own. Bayesian priors communicate uncertainty. They do not create identification that the variation in your data cannot support.
Go deeper → -
Where should the next dollar go? Decide
Marginal return, not average return. You want the slope of the response curve at your current spend, ideally anchored by an experiment run at that spend.
What it will not tell you Be read off a ROAS table. The channel with the best historical average is often the one closest to saturation.
Go deeper → -
Which outcome should my campaigns optimize toward? Decide
The deepest event that stays reliable, timely, and frequent enough to train on. Those three pull against each other, and the right answer moves as your volume grows.
What it will not tell you Be settled by picking the most valuable event. Optimizing toward a sparse, delayed signal produces worse delivery than optimizing toward a good proxy.
Go deeper → -
Who should receive a nurture campaign, or no action at all? Decide
The people whose outcome would change because you contacted them. That is an uplift question, and it needs a randomized holdout to train and validate against.
What it will not tell you Be answered by a propensity model. High propensity finds people likely to convert, which often means people who needed no contact at all.
Go deeper →
The progression
Measurement is a sequence, not a tool choice
Each rung depends on the one below it. Attribution and incrementality are not competing answers to the same question. They sit at different rungs and answer different questions.
01
Observe
What happened?
02
Describe
Where did conversions appear to come from?
03
Estimate
What changed because of marketing?
04
Explain
How did channels and outside factors contribute?
05
Decide
What should happen next?
06
Learn
What uncertainty should we reduce next?
Four sources of doubt
Why this is difficult
Only one of these is a statistics problem. Teams that treat all four that way buy better models and stay stuck.
Tracking uncertainty
Consent, identity resolution, and cross-device loss make the events you observe a biased sample of the events that happened.
Operational uncertainty
Campaigns launch late, audiences overlap, exclusions fail quietly. What ran is rarely quite what was designed.
Causal uncertainty
You never observe the counterfactual. Every estimate of it rests on assumptions you can check but cannot prove.
Decision uncertainty
A clean estimate still has to survive margin, capacity, seasonality, and what the business is willing to do.
Know what you are holding
Different questions require different evidence
Three of these seven support a causal claim. The rest are useful, often more useful day to day. Calling them causal is where measurement programs go wrong.
| Evidence | Answers | Causal? |
|---|---|---|
| Attribution | Which touchpoints preceded the conversions I can see? A rule for splitting credit across observed journeys. | Descriptive |
| Experiments | What changed because of this intervention? The counterfactual is built by design, not assumed. | Causal |
| Quasi-experiments | What changed, where randomization was not possible? Causal only if the identifying assumptions survive diagnostics. | Causal |
| Marketing mix models | How did channels and outside factors contribute over time? Allocation-grade evidence. Causal only once calibrated to experiments. | Descriptive |
| Predictive ML | Who is likely to convert, activate, or churn? Ranks people by likelihood, not by persuadability. | Descriptive |
| Causal ML | Whose outcome would change if we acted? Needs randomized variation to train and validate against. | Causal |
| Decision systems | Which action, if any, should each customer receive? Turns estimates into policy. Judged by holdout, not by accuracy. | Descriptive |
Beyond the click
Acquisition is the easy half
A click is the start of a customer, not the end of a campaign. The harder question is what happens next: which lead gets nurtured, which customer gets a retention offer, which account gets a rep's time, and who should receive nothing at all.
That path runs from rules to predictive models to uplift to constrained automated decisioning. Every step still needs logged decisions, holdouts, and consent. Automating a decision does not exempt it from measurement.
Explore the decisioning track →The optimization signal ladder
- Click Fast and dense, and almost never what you want to buy.
- Lead Cheap to generate, easy to generate badly.
- Qualified lead Usually the right target for sales-led programs.
- Activation The product-led equivalent. A strong proxy for value.
- Purchase Reliable, but often too sparse for delivery algorithms.
- Retention Slow. Good for evaluation, rarely for optimization.
- LTV The right target, at the wrong latency.
- Incremental value What you actually want. Requires a holdout.
The machinery
Methods are the easy part
Difference-in-differences is published and well understood. What is missing in most organisations is the plumbing that lets you apply it repeatedly, on a deadline, without breaking consent or colliding with another campaign. These are the components of a measurement operating system, in dependency order.
-
01
Foundation
Who exists, what may we do with them, and how do we describe a group.
1 of 3 written
-
02
Experimentation
Splitting an audience so the result can be believed later.
1 of 3 written
-
03
Measurement
Turning logged decisions into a standing readout.
1 planned
-
04
Decisioning
Choosing an action per customer, including no action.
2 planned
Four disciplines
Where the work lives
Attribution
Can we trust what we observe?
Tracking loss, identity, consent, event definitions, funnel grain, and attribution as descriptive evidence. Everything downstream inherits the flaws you accept here.
Incrementality
Did marketing change the outcome?
Counterfactuals, experiments, quasi-experiments, and the diagnostics that decide whether an estimate deserves to be believed. Causal ML sits here too, where the question moves from an average effect to whose effect differs.
Allocation
Where should the next dollar go?
Marginal rather than average returns, channel interaction, marketing mix modeling, and budget decisions made under real uncertainty.
Decisioning
What should happen after the click?
The optimization signal ladder, lifecycle experimentation, uplift modeling, and the systems that turn estimates into governed decisions. Where causal ML stops being an estimate and becomes a policy.
Start with these
Designing a Credible Geo Holdout
A paid social incrementality test carried end to end: market pairing, power, pre-trend diagnostics, a difference-in-differences estimate with its interval, incremental CAC, and what the design cannot settle.
Broad Nurture, Propensity, or Uplift?
One trial-to-paid nurture sequence, four targeting policies, and a randomized holdout. Compared on incremental conversions rather than model accuracy, which is the only comparison that changes the answer.
Attribution Is Not Incrementality
Attribution assigns credit for conversions you observed. Incrementality estimates the ones that would not have happened otherwise. The gap between them is where budgets go wrong.
Who writes this
Reid Rhodes, a marketing data scientist working on measurement, incrementality, and growth decisions. Everything here comes from answering these questions with imperfect data and a deadline, which is the only condition they ever come up in.
Worked examples use public or simulated data and say which, on every page. No result here is presented as a real company's unless it is one, and labeled that way.
What is written
- Lessons
- 1 of 22
- Worked examples
- 2 of 4
- System components
- 2 of 9
Written against planned. The roadmap is published so the shape is visible, not to suggest the unwritten parts exist.