--- name: run-a-landing-page-a-b-test description: >- Runs a disciplined landing-page split test end to end: pick the page with the best volume-to-upside ratio, write a falsifiable hypothesis, change exactly one element, split traffic 50/50, hold the line for 14+ days and 100+ conversions per arm, then call a winner at 95% confidence and deploy it. VigilDog selects candidates and reads results via gads_run_gaql_query on the landing-page report plus ga4_run_report guardrails (bounce, time on page), and can run the split itself through gads_create_experiment or final-URL changes via gads_update_ad - all writes preview-first and applied only after user approval. Use it on pages that already pass baseline quality when CVR has plateaued or before scaling spend. Fix foundations first with improve-landing-page-experience; for campaign-setting experiments use run-a-campaign-experiment; for a non-test overhaul use audit-and-optimize-an-existing-landing-page. --- # Landing Page A/B Test ## Purpose Turns landing-page improvement from opinion into evidence: one page, one changed element, a controlled 50/50 split, and a statistical verdict. The discipline is the value - most "tests" die from peeking early, changing two things at once, or running on pages without the volume to ever reach a conclusion. ## When to run - A high-traffic page's CVR has been flat for 30+ days despite stable traffic quality. - After foundational repairs, to validate the next layer of improvements. - Before scaling budget into a campaign - squeeze the destination first, the lift multiplies the spend. - On a quarterly cadence per high-traffic page, one active test per page at a time. ## When NOT to run - The page fails baseline quality (Below Average landing-page experience, broken mobile, slow load) → improve-landing-page-experience first; testing on a broken page wastes the traffic. - You want to test bidding, budget, or targeting changes → run-a-campaign-experiment. - The page needs many fixes at once, not one measured change → audit-and-optimize-an-existing-landing-page. - The page gets fewer than ~30 conversions per month → the test will never conclude; make best-judgment changes instead and revisit when volume grows. ## Prerequisites - Test page rates Average or better on landing-page experience. - 1,000+ monthly visitors and 30+ monthly conversions on the page (hard volume gate). - 30 days of baseline CVR data and verified conversion tracking. - Ability to produce a variant page (LP builder or developer) - human resource. - No other test or major campaign change planned on this traffic during the window. ## Procedure 1. **Select the candidate.** Use `gads_run_gaql_query` on the landing-page report: clicks, conversions, CVR, CPA per final URL over 30 days. Rank pages by (traffic volume x CVR gap to the account average). High traffic + below-average CVR is the ideal candidate; low-traffic pages are disqualified regardless of upside. 2. **Record the baseline.** From the same query plus `ga4_run_report` (bounce rate, average time on page for the URL): visitors, conversions, CVR, CPA, bounce, time on page - a dated snapshot the verdict will be measured against. 3. **Confirm readiness.** Check every gate in Prerequisites. Any failure stops the test here. 4. **Choose one element, highest leverage first:** headline/offer framing → above-the-fold layout → form design → CTA copy/placement → proof type and position → hero format → section order. Top-of- list elements typically move CVR several times more than bottom-of-list ones. 5. **Write the hypothesis** in one sentence with a falsifiable prediction: the element changed, from what to what, the expected relative CVR lift, and the data-based reason you expect it. Define success (minimum +10% relative CVR at 95% confidence) and guardrails (bounce must not rise more than 5%; time on page must not collapse). 6. **HUMAN STEP (outside VigilDog):** build the variant page changing only the chosen element - identical layout, images, and form otherwise. Verify it loads correctly on mobile and desktop and that conversion tracking fires on both versions. 7. **Compute the required duration.** 100 conversions per arm minimum. Days needed = 100 / (daily page conversions / 2), and never less than 14 days regardless of volume, so both weekly cycles are captured. 8. **Set up the split.** Preferred: the LP builder's native split test (HUMAN STEP - it handles serving and stats). Via VigilDog instead: consult `gads_policy_guardrail` before the session's first Google Ads write, then use `gads_create_experiment` on the traffic-driving campaign with the treatment arm's ads pointing at the variant URL at a 50/50 split - or duplicate ads updated via `gads_update_ad` with the variant final URL. Every write runs preview (validate_only) first and is applied only after explicit user approval. 9. **Pre-launch check:** both URLs tracked, both forms/checkouts working, comparable load speed, no overlapping tests. Then launch. 10. **Monitor hands-off.** Day 1: confirm both arms receive traffic near 50/50 (`gads_run_gaql_query` by final URL). Days 2-3: watch for technical failures - broken tracking restarts the clock. Weekly: log any external events (promos, outages, seasonality). Do not touch either page, do not shift the split, do not stop because one arm "looks" ahead. 11. **Analyze only when both gates pass** - 14+ days AND 100+ conversions per arm. Pull final numbers per URL via `gads_run_gaql_query` and guardrails via `ga4_run_report`. Call the winner at p < 0.05 on CVR; check guardrails before celebrating. 12. **Deploy and document.** Winner = variant: make it the default, remove the split, update any final URLs via `gads_update_ad` (preview → user approval → apply), and re-verify tracking. Winner = control or inconclusive: remove the variant, keep control. Either way, record page, dates, element, hypothesis, numbers, verdict, and the learning - then queue the next element for next quarter. ## Decision rules - **Volume gates:** 1,000+ visitors/month, 30+ conversions/month to start; 100+ conversions per arm and 14+ days to conclude. With 100/arm you can reliably detect a ~15-20% relative lift; detecting ~10% takes 300-500 per arm. - **One variable per test.** Two changes = zero attributable learnings. - **Meaningful changes only.** Button-color-sized tweaks rarely clear the detectable-effect floor; test differences a visitor would actually notice. - **Early stop only for:** variant CVR down 50%+ sustained 7+ days (stop, keep control), technical breakage (fix, restart clock), or an external event that invalidates the traffic (stop, rerun later). Never for "it looks like it's winning." - **Guardrail conflict:** variant wins CVR but breaches a guardrail → judge the trade explicitly with the user; do not auto-deploy. - **Segment cuts (device, source, weekday) are hypothesis generators,** not verdict changers - the overall result decides. - **Post-deploy watch:** 7 and 14 days after rollout, confirm the lift holds at full traffic; a vanished lift means the test was contaminated. ## Common failure modes - **Peeking and stopping early.** Week-one streaks are noise; committing to the calculated duration is what makes the answer real. - **Testing on a trickle.** A 15-conversion-per-month page needs most of a year to conclude; the gate exists to stop that waste. - **Mid-test meddling.** Editing a page or shifting budget mid-run silently invalidates both arms; freeze everything that touches the test traffic. - **Cherry-picking a segment win.** "It won on mobile" after an overall loss is how bad variants ship; segments inform the next hypothesis only. - **Winner deployed, tracking broken.** URL swaps at deploy time are where pixels die; re-verify conversions fire after every rollout. ## Related skills - **Before:** improve-landing-page-experience (baseline quality gate), monitor-landing-page-performance (surfaces candidates), audit-and-optimize-an-existing-landing-page (fix fundamentals first). - **Parallel:** run-a-campaign-experiment (campaign-setting experiments use the same discipline), manage-experiment-backlog (queueing next tests). - **After:** monitor-landing-page-performance (confirm the lift persists in regular reviews).