MCP

A/B testing

TL;DR

  • The method is uncontroversial: change one variable, split the list into comparable halves, run both in the same window.
  • The sample size is where cold-email testing quietly fails. Published guidance suggests 100 prospects per variant [1]; detecting a realistic reply-rate difference needs thousands.
  • If you cannot reach that, do not run an A/B test. Make the change you believe in and judge it over several campaigns.

The method

Change one variable at a time — the subject line, or the CTA, or the opening, never two. Divide the prospects into at least two subgroups that resemble each other on everything that matters (segment, tier, company size, seniority), and send both variants in the same window so that timing and reputation are held constant [1].

That much is agreed and worth following. The problem is what comes next.

The arithmetic nobody publishes

Published guidance recommends "at least 100 prospects" per variant [1]. Here is what 100 buys you at realistic rates.

A rough rule of thumb for the per-arm sample needed to detect a difference between two proportions at conventional confidence and power is 16 × p × (1 − p) / d², where p is the baseline rate and d the absolute difference you want to detect.

At a 3.4% baseline reply rate [2], detecting a 50% relative improvement — 3.4% to 5.1%, a large effect nobody would sniff at — means d = 0.017 and needs roughly 2,200 prospects per variant. Even testing on opens, where the base rate is far higher (say 35%), detecting a 5-point difference needs around 1,450 per variant.

With 100 per variant you would need a difference of roughly 3.4% versus 12% before the result meant anything. Effects that large exist, but they come from changing the list or the offer, not from a subject line.

The uncomfortable conclusion: most cold-email A/B tests are theater. They produce a winner because one number is bigger, and the winner is noise. Two campaigns run at 100 per variant will frequently disagree.

What to do instead

Test big things. A different segment, a different offer, a fundamentally different message strategy — changes plausibly capable of moving the rate by half or more (Messaging frameworks). Small copy edits are not measurable at your volume, which is an argument for making them by judgment rather than pretending to measure them.

Accumulate across campaigns. The same variant tried across five campaigns builds a sample no single campaign can. This requires keeping definitions stable — Benchmarks.

Test on a leading indicator when you must, knowing it is a proxy. Opens are cheaper to power than replies and thoroughly corrupted (Tracking caveats), so a subject-line test on opens tells you about machines as much as humans.

Read the replies. Thirty replies read carefully will tell you more about why a campaign is failing than an underpowered split test ever will — and they are qualitative evidence, which is the right kind for small samples.

Pitfalls

  • Peeking and stopping early. Watching until one variant leads and declaring victory manufactures a winner from noise. Fix the sample and the window in advance.
  • Splitting by list order. Alphabetical or import-order splits correlate with company, region and data source. Randomize.
  • Changing infrastructure mid-test. A new mailbox, a volume change, a different send window all contaminate the comparison (Sending operations).
  • Testing during a deliverability problem. Both variants are then measuring placement.

References

  1. Woodpecker — Step-by-step guide to A/B testing cold emails and follow-ups (vendor guide, updated Jan 2025)
  2. Woodpecker — Cold email statistics (20M+ email corpus, updated Jun 2026)