How to Test an Email Verification Service: A Buyer's Benchmark

RocketVerifier sells email verification, and we are confident enough in our results to tell you exactly how to test us against everyone else. The method in this post works on every verifier, uses your own data, and does not depend on trusting any vendor’s homepage, including ours.

Most buyers compare verifiers by reading homepages. Every one of them quotes a high accuracy figure, and the figures cluster so tightly that they carry no information. The vendors are not necessarily lying. They are each measuring something different, on data you will never see.

The alternative takes an afternoon and a few dollars per vendor. You build a small list of addresses whose real outcome you already know, run it through each candidate, and score the results the same way. You can download the scorecard template (CSV) now and fill it in as you go.

Why published accuracy numbers do not compare

An accuracy percentage is a fraction. To compare two of them you need the same numerator rules and the same denominator. Verification vendors share neither.

There is no shared ground truth. Nobody publishes a standard test list of addresses with certified outcomes, and one would go stale within months as mailboxes are created and closed. Each vendor tests on whatever data it has, often its own historical results, which is circular.

Unknowns are excluded or counted differently. Some servers defer, block or time out, so every honest verifier returns some results as unknown. If a vendor computes accuracy only over the addresses it answered, it can raise its headline number by answering fewer. A vendor that leaves 15% of a list unresolved and is right on the rest can quote a higher figure than one that resolves 99% and is slightly less precise. The worked example below shows exactly this.

Catch-alls are often hidden. A catch-all domain accepts every address at the SMTP stage, so the server’s “yes” proves almost nothing. A vendor can report those as valid and count them as correct (the server did accept), as unknown and leave them out, or as their own category. Each choice moves the accuracy figure in a different direction, and the homepage rarely says which was made.

Add the fact that accuracy depends heavily on the list. A list of fresh Gmail signups and a list of scraped addresses at small manufacturers will produce very different numbers on the same engine. A vendor’s figure, however honestly computed, describes someone else’s list.

What a good test set looks like

The test set is a list of 500 to 2,000 addresses where you already know what the correct answer should be. You build it from your own sending history plus a few addresses you create. The mix below adds up to 1,000 and is the one used in the worked example.

Bucket	How to collect it	Rows	Correct result
Recent hard bounces	Export from your ESP: bounced in the last 90 days with a “user unknown” code such as 5.1.1	300	undeliverable
Seeded fakes	Random local parts on a domain you control that is not catch-all, plus a few at a .invalid domain	40	undeliverable
Recently engaged	Contacts who clicked, replied, logged in or bought in the last 30 days	450	Not undeliverable
Seeded real	Mailboxes you own and know exist	10	Not undeliverable
Typo domains	Engaged local parts moved to gmial.com, hotmial.com or a .con ending	50	Flagged as a typo or invalid
Disposable	Addresses at domains from a public disposable list	50	Flagged as disposable
Catch-all samples	Addresses at domains you know accept everything	100	Labelled catch-all (or risky)

A few details decide whether the answer key is trustworthy.

Filter hard bounces by code, not by your ESP’s label. Depending on how your ESP classifies failures, its hard bounce label can include policy rejections. Under RFC 3463, 5.1.1 means the mailbox does not exist, while 5.7.1 means the sender was refused. Only the first is a fact about the address. Keep rows like 550 5.1.1 and drop rows like 550 5.7.1.

Define engagement without opens. Apple’s Mail Privacy Protection downloads remote content in the background whether or not the recipient opens the message, so an open pixel no longer proves a human or a live mailbox. Clicks, replies, logins and purchases do.

Seeded fakes need a domain that rejects unknown users. Confirm it first: if your own mail server accepts any local part, those addresses belong in the catch-all bucket instead. The .invalid top-level domain is reserved by RFC 2606 for names that are guaranteed not to exist, which makes it a useful floor check. Any vendor that misses those has a problem.

Catch-all samples are the hardest bucket to label. The cleanest source is a spare domain of your own with catch-all routing switched on, seeded with random local parts. Next best is engaged contacts at business domains that at least two of the vendors you are testing flag as catch-all. That second source is partly circular, so treat it as lower confidence.

Pick disposable domains that are still active. The community disposable list states that it cannot guarantee every entry is still disposable. Spot-check a sample with the disposable email checker before adding it.

How to run the test fairly

The test is only useful if every vendor gets the same conditions. Most unfair comparisons come from running vendors days apart or leaking the answer key.

The six measurements

Score each bucket separately. A single blended accuracy figure is how vendors hide weaknesses, and you would be doing the same to yourself.

1. Agreement with known outcomes

For the known-bad rows (hard bounces and seeded fakes), count how many came back undeliverable. Separately count how many came back unknown and how many came back deliverable. A deliverable result on a row that hard-bounced last month is a miss you would have paid for in bounces.

For typo and disposable rows, count how many were flagged in any way. A gmial.com address labelled deliverable without a typo warning is a miss.

2. Unknown rate

The share of all rows with no real answer. Lower is better, but only when the false invalid rate stays low as well. A vendor can drive unknowns toward zero by guessing, and the guesses show up as errors in the other rows. Unknown means the server deferred, blocked or timed out. Our explainer on how email verification works covers why that is the honest result for those cases.

3. Catch-all labelling

For the catch-all bucket, count three outcomes: labelled catch-all (or risky with a catch-all reason), labelled deliverable, labelled unknown. A vendor that calls most catch-all addresses deliverable is not more accurate, it is hiding risk. If you send to them, some will bounce later or land in a shared inbox nobody reads. Better vendors also grade catch-alls by likelihood, which matters if catch-alls are a large share of your list. Catch-all emails: should you send to them? explains how to use those grades.

4. False invalids on known-good addresses

The number of engaged or seeded-real rows a vendor marked undeliverable. This is the costliest error, and the reason is asymmetry. A bounce costs you a little reputation and you see it in your ESP report. A false invalid deletes a real customer silently. You never mail them again and never learn it happened.

Put it in list terms. Suppose 60,000 addresses on a 100,000 list are good (an illustrative split). A 3.9% false invalid rate removes 2,340 real people. A 0.9% rate removes 540. That gap usually outweighs any difference in price per credit.

5. Speed

For bulk, record the time from upload to finished results. Be wary of a vendor that is much faster than the others, resolves almost everything and shows no retries. Greylisting servers ask the sender to come back later, so a vendor that never waits is answering those rows without a real reply. If you need the real-time API for a signup form, also time 100 single calls and note the typical and slowest response.

6. Cost per resolved address

Price per credit is the wrong number, because you are buying answers, not attempts. Compute what you paid per row that got a real answer:

rows charged      = all rows, or only resolved rows if unknowns are free
                    (minus duplicates, if duplicates are free)
amount billed     = price per credit x rows charged
cost per resolved = amount billed / rows resolved (not unknown)

When unknowns are billed, a vendor with a high unknown rate costs more per answer than its list price suggests. Our breakdown of email verification pricing covers the other ways sticker prices mislead, such as tier jumps and expiring credits.

A worked scoring example

Here is a complete scorecard for three made-up vendors on the 1,000-row test set above. These are illustrative numbers, built to show common patterns. They are not results from any real vendor, including us.

Illustrative Made-up numbers for a 1,000-address test. Not results from any real vendor.

Metric Vendor AVendor BVendor C

• Known-bad addresses caught

  340 rows: recent 5.1.1 hard bounces plus seeded fakes. Higher is better.

  Vendor A 97.1% 330 of 340 (rated good)

  Vendor B 99.1% 337 of 340 (rated good)

  Vendor C 86.8% 295, plus 40 unknown (rated weak)

• False invalids on known-good

  460 rows that recently clicked or replied. Lower is better. The costliest error.

  Vendor A 0.9% 4 of 460 (rated good)

  Vendor B 3.9% 18 of 460 (rated weak)

  Vendor C 0.4% 2 of 460 (rated good)

• Unknown rate

  All 1,000 rows. Read it next to the row above, never alone.

  Vendor A 3.6% 36 rows (rated for reference)

  Vendor B 0.5% 5 rows (rated for reference)

  Vendor C 14.9% 149 rows (rated weak)

• Catch-all samples labelled as catch-all

  100 rows on domains known to accept everything. Higher is better.

  Vendor A 88% 12 unknown (rated good)

  Vendor B 15% 85 called valid (rated weak)

  Vendor C 45% 55 unknown (rated mixed)

• Typos and disposables flagged

  100 rows. Higher is better.

  Vendor A 96% 4 called valid (rated good)

  Vendor B 98% 2 called valid (rated good)

  Vendor C 92% 4 valid, 4 unknown (rated mixed)

• Time to finish the file

  Upload to downloadable results, same day, same file.

  Vendor A 7 min retried deferrals (rated for reference)

  Vendor B 2 min no retries seen (rated for reference)

  Vendor C 25 min slow, still 149 unknown (rated mixed)

• Price per credit

  At the tier you would actually buy.

  Vendor A $0.0045 unknowns free (rated for reference)

  Vendor B $0.0030 unknowns charged (rated for reference)

  Vendor C $0.0040 unknowns charged (rated for reference)

• Cost per resolved address

  Amount billed divided by rows that got a real answer.

  Vendor A $0.0045 $4.34 / 964 (rated good)

  Vendor B $0.0030 $3.00 / 995 (rated good)

  Vendor C $0.0047 $4.00 / 851 (rated weak)

Vendor A

No standout number, no serious failure. Admits uncertainty and does not bill for it.

Vendor B

Cheapest and fastest. Also removes about 1 in 26 real contacts and calls most catch-alls valid.

Vendor C

Rarely wrong when it answers, but leaves 15% unresolved and costs the most per answer.

good mixed weak context, not a score

Three patterns show up here that also show up in real tests.

Vendor B looks best on the homepage metrics. It is the cheapest, the fastest, catches the most bad addresses and almost never returns unknown. It gets there by guessing. It marks 18 known-good contacts undeliverable and calls 85 of 100 catch-all addresses valid. A buyer comparing price and unknown rate would pick it and lose a few percent of their real customers.

Vendor C would advertise the highest accuracy. It is rarely wrong when it answers. But it answers less, so its headline figure depends entirely on what you do with its unknowns:

Vendor C on known-outcome rows, unknowns left out (703 of 710)

99.0%

Vendor C on known-outcome rows, unknowns left out (703 of 710)

Same results, unknowns counted as misses (703 of 800)

87.9%

Same results, unknowns counted as misses (703 of 800)

Vendor B, unknowns counted as misses (774 of 800)

96.8%

Vendor B, unknowns counted as misses (774 of 800)

Same results, two defensible formulas, an 11-point difference. Vendor C’s marketing page would print the first number. Scored the stricter way, it trails Vendor B, and it also costs the most per resolved address ($4.00 billed for 851 answers) because it charges for its 149 unknowns.

Vendor A wins without winning any single row. It is never the cheapest or fastest. It has no serious failure, does not bill for uncertainty, and labels catch-alls as catch-alls. For most senders, that profile is the one to buy.

How big a gap is real

With a few hundred rows per bucket, small differences are noise. Using the normal approximation from the NIST e-Handbook, a 97.1% catch rate on 340 known-bad rows has a 95% margin of about 1.8 points either way: 1.96 x √(0.971 x 0.029 / 340) ≈ 0.018. So Vendor A at 97.1% and Vendor B at 99.1% are close to a tie on that row. The false invalid gap (0.9% against 3.9% of 460) is well outside that margin. As a rule of thumb, treat differences under about 2 points as ties at this sample size, and use 2,000 rows if two finalists are that close. For rates near zero (a handful of errors), the same handbook recommends the Wilson or exact method instead.

What to read in the terms

Two vendors with identical results can still cost very different amounts over a year. Read these before you buy, and add them to the last columns of the scorecard.

Bounce guarantees deserve a careful read. Almost all of them, ours included, exclude catch-all, risky and unknown results, so they only cover the rows marked deliverable. That is reasonable, but it means the guarantee covers a smaller slice of your list the more of it lands in those buckets. Check the threshold too: a guarantee that only pays above a high bounce rate rarely pays. And remember that any payout is credits or money. It does not repair your sender reputation.

For data handling, Article 28 of the GDPR lists what a processor contract must contain, including that the processor deletes or returns personal data at the end of the service. Ask where the data is processed and how long lists stay on the vendor’s servers after a job finishes.

How to weigh the results for your use

There is no single winner across all senders. Rank the measurements by what your list is for.

• Opt-in marketing lists: false invalids first. These are people who asked to hear from you, and a wrongly deleted subscriber is lost revenue.
• Cold outbound: known-bad catch rate and catch-all labelling first. New sending domains have little reputation to absorb bounces, and B2B lists are full of catch-all domains. Our guide to verification for cold email goes further.
• Signup forms: API speed and typo detection first. A typo suggestion at the moment of signup recovers a customer, and a slow check costs conversions.

Then use cost per resolved address to break ties, not to pick the shortlist.

Running the test on us

This test rewards exactly what RocketVerifier is built for: real SMTP answers instead of guesses, catch-all verdicts instead of a bare flag, and no charge for results we cannot confirm. We verify with 99.5% accuracy, and here is what to expect when you put us through it.

We return four statuses: deliverable, undeliverable, risky and unknown, each with a sub-status that says why. In the dashboard and the current API version, catch-all addresses are risky/ catch_all with a verdict of send_safe, send_with_caution or avoid. API keys on the older version see catch-alls as unknown, so run the test through the dashboard or the current API version. Our method, including how we treat blocked and greylisted checks, is on the accuracy page.

On billing: unknown results and duplicate addresses in a bulk list are never charged, and pay-as-you-go credits never expire. A 1,000-credit pack, enough for the test, is $6; pricing has the rest. Our bounce guarantee goes further than most: connect SendGrid, Mailgun, Amazon SES or Postmark and every hard bounce on an address we marked deliverable is refunded automatically within 30 days. Data handling, retention and the DPA are on the security page.

If you want a head start on the shortlist, our comparison pages cover other verifiers one at a time. Then run the test, fill in the scorecard template, and see how we compare on your own data.