--- name: run-incrementality-and-lift-tests description: >- Answers the question attribution windows can't: how many credited conversions would have happened anyway. Runs a Conversion Lift / A-B study via mads_create_ad_study (randomized control vs exposed) or a geo holdout (control markets dark, exposed markets live, compared on backend conversions) to measure true incremental lift instead of trusting last-click attributed ROAS. Covers sizing the test so both arms reach a readable conversion sample, the duration needed for a stable read, a clean control/exposed split, and reading the result, incremental conversions, cost per incremental conversion, statistical confidence. Names the case this matters most: retargeting and brand campaigns, where last-click over-credits people who'd have converted anyway. Reach for it whenever attributed ROAS is about to justify a budget call and no one has checked whether the credit is real. It ends at an incrementality read, the attribution-window mechanics being tested against live in configure-attribution-settings. --- # Run Incrementality and Conversion Lift Tests ## Purpose Attribution is a credit-assignment heuristic, not causal measurement. Last-click and last-touch attribution structurally over-credit bottom-funnel and retargeting touches, the person who already had their card out gets full credit for a purchase they may have made anyway. Holdout testing is the only way to see the counterfactual: what would have happened without the ad. This skill runs that test and reads the result against the attributed number it's meant to check. ## When to run - A retargeting or brand campaign shows strong attributed ROAS and its budget is about to be scaled on that number alone. - Deciding prospecting vs retargeting allocation, where attribution alone would over-favor retargeting. - Launching a new channel or tactic and leadership wants proof of causal impact, not just attributed conversions. - Periodically (quarterly) on the account's largest budget line, as a sanity check on the whole bidding/attribution stack. ## When NOT to run - Conversion volume is too low to reach a readable sample within a reasonable window, fix signal/volume first, or accept a longer test. - The question is which attribution window to use, not whether the credit is real → `configure-attribution-settings`. - No conversion tracking exists yet → `set-up-pixel-and-standard-events` / `implement-conversions-api` first; a lift test needs the same signal attribution needs. ## Prerequisites - Stable, verified conversion tracking (pixel + CAPI), a lift test on broken tracking measures noise, not lift. - Enough weekly conversion volume that a held-out control group still leaves a readable exposed sample (the campaign should already clear the ~50/week learning-phase floor per ad set). - A defined primary metric (Purchase, Lead) and its expected baseline conversion rate, to size the test. - Buy-in for a control group or geo holdout market that intentionally sees no ads for the test duration. ## Procedure 1. **Decide the test type.** Conversion Lift / A-B study (`mads_create_ad_study`) randomly splits individual users into control/exposed within the same geography, the default when the platform can run a clean random split. Geo holdout (some markets run the campaign, matched control markets don't) is the fallback when user-level randomization isn't possible (small accounts, cross-platform campaigns). 2. **Size the sample and duration.** A bigger expected lift needs a smaller sample to detect; a smaller expected lift needs more. HUMAN STEP: use the study-creation flow's estimated minimum duration/budget for the target confidence level, do not launch below its recommendation. 3. **Create the study.** `mads_create_ad_study` with the campaign(s) under test, the holdout percentage (control group size), and the study duration. Keep targeting, creative, and budget otherwise unchanged for the duration. 4. **Let it run undisturbed for the full duration.** Do not pause, budget-shift, or edit the tested campaign(s) mid-study, any edit resets the clean comparison the study depends on. 5. **Read the result.** Incremental conversions (exposed minus what control would have converted anyway), cost per incremental conversion (spend divided by incremental conversions, nearly always higher than attributed CPA), and the confidence interval Meta reports for the study. 6. **Compare cost-per-incremental against the attributed CPA.** A large gap (attributed CPA far below cost-per-incremental) means attribution was over-crediting, usually retargeting/brand touches getting credit for conversions that would have happened anyway. 7. **Act on the read.** IF lift is strong and cost-per-incremental sits close to attributed CPA THEN the channel is pulling real weight, scale with confidence. IF lift is weak or not significant THEN the budget is largely paying for conversions that would have happened anyway, reallocate toward prospecting or a channel with proven incrementality. ## Decision rules - IF the platform can randomize at the user level within budget/audience minimums THEN prefer a Conversion Lift study over a geo holdout, cleaner control for regional demand swings. - IF the campaign spans multiple platforms, or the account is below a study's minimums, THEN geo holdout is the fallback, with control/exposed markets matched on size and trend beforehand. - IF retargeting shows outsized attributed ROAS THEN it's the highest-priority candidate for a lift test, the segment last-click over-credits hardest. - IF the result isn't statistically significant THEN don't act on the point estimate, extend duration or increase tested budget rather than reading a noisy number as fact. - IF cost-per-incremental is unaffordable even though attributed CPA looked fine THEN the attributed number was misleading the budget decision, this is exactly the case the test exists to catch. - **Done means:** a completed study or holdout with incremental conversions, cost-per-incremental, and a confidence read reported back, and a budget decision made against that number, not the attributed one. ## Common failure modes - Editing the tested campaign mid-study (budget, creative, targeting), invalidates the comparison and forces a restart. - Ending the test early because "the trend looks clear", early reads on low volume are frequently wrong. - Testing a campaign too small to ever reach a readable sample, burning the test budget on an inconclusive result. - Treating attributed ROAS and incremental lift as the same number and being surprised when they diverge, they measure different things by design. - Running a geo holdout with poorly matched control/exposed markets and blaming the ad channel for what's actually a market difference. - Skipping this entirely and scaling retargeting purely on attributed ROAS, the single most common incrementality mistake on Meta. ## Related skills - Prerequisite: `set-up-pixel-and-standard-events`, `implement-conversions-api` (the signal the study measures against). - Contrast with: `configure-attribution-settings` (the credit-assignment mechanics this skill tests the truth of). - Feeds: `select-campaign-objective` and budget/scaling decisions elsewhere in the account, act on incremental CPA, not attributed CPA, when reallocating.