Is your campaign failing, or is it just early?

How the Signal Check works, in plain words

1. Every email is a coin toss

Each email you send either gets a reply or it doesn't. The same goes for an out-of-office reply and for an unsubscribe. Your campaign has a true rate for each, but you never see it directly. You only see how the tosses came out so far. Statisticians call this a binomial model, and it is the whole foundation.

2. The main question: could this be chance?

Each signal has a threshold: the out-of-office rate a healthy inbox produces, the reply rate that would pay off for you, and the unsubscribe rate you can tolerate. The check then asks one question:

"Suppose the true rate sits exactly on the threshold. How likely is it that chance alone gives a result this bad, or worse?"

That likelihood is the p-value. Think of a coin: one tails in a row means nothing, but ten tails in a row from a fair coin happens about once in a thousand tries, so you start to doubt the coin. When the chance drops below 5%, the check stops calling it bad luck and gives a verdict: fail.

Each test looks in one direction only. For replies and out-of-office replies, the problem is too few. For unsubscribes, the problem is too many. Looking the wrong way would turn the verdict upside down.

3. The bar: where your true rate probably is

Next to each verdict you see a bar. It is the range your true rate plausibly sits in (a 90% interval), drawn against the threshold line. Reading it is simple:

• The whole bar on the bad side of the line: fail.
• The whole bar on the good side: pass.
• The bar crosses the line: no verdict yet. There is not enough data to tell.

The width of the bar shows how much data you have. After a few dozen emails it is very wide. After a few thousand it is narrow. That is why zero replies after 40 emails and zero replies after 4,000 mean completely different things.

The interval is calculated with the Clopper-Pearson method rather than the textbook "rate plus or minus two standard errors". With zero replies the textbook formula gives a range of 0% to 0%, claiming perfect certainty exactly when you know the least.

4. Built-in safeguards

• No verdict below 30 emails. Otherwise a single unsubscribe on the third email would condemn the whole campaign.
• "How many more?" When there is no verdict yet, the check estimates how many more emails would settle it if the current rate holds. "We don't know" becomes "check back after N more".
• Delivery first, then interest. An out-of-office reply is sent by the mail server, not by a person, and Gmail and Outlook don't send one for a message filed as spam. So the out-of-office rate measures one thing: whether your emails reach the inbox. If delivery fails, the check refuses to judge your reply rate, because an email nobody saw can't be judged on its writing.
• People prove delivery too. Nobody replies to or unsubscribes from an email in a spam folder. If replies and unsubscribes together come in at least as often as out-of-office replies should, delivery counts as proven even when auto-replies are scarce.

5. What the numbers don't say

• The emails are not truly independent. Hundreds of copies of one email are closer to one experiment repeated many times than to hundreds of separate experiments. So a "fail" means "this version of the email doesn't work", not "this audience is a dead end".
• A p-value is not the chance that your campaign is bad. A p-value of 0.002 means "if everything were fine, a result like this would turn up 0.2% of the time". It does not mean "99.8% sure the campaign is bad".

The same code runs inside Sellvance on every campaign. The full method, with the sample sizes each verdict needs, is in the Experiment Playbook.