SaaStr Runs Email Deliverability Test Across 3 Vendors
SaaStr compared email deliverability across three vendors and over 500,000 emails, with its AI agent running the test. The results showed significant differences in deliverability between vendors.
Original reporting · saastr.com
Original: “We Ran an Email Deliverability Bake-Off Across 3 Vendors and 500,000+ Emails This Week. Our AI Agent Ran All of It.”
Read “We Ran an Email Deliverability Bake-Off Across 3 Vendors and 500,000+ Emails This Week.…” at SaaStr →
How this was written. Voyager Commercial wrote this article with a language model from SaaStr’s reporting (model: @cf/meta/llama-3.3-70b-instruct-fp8-fast); it describes that reporting rather than adding to it. The original, linked above, is the full story.
Key points from SaaStr’s report
- SaaStr conducted an email deliverability test across three vendors and over 500,000 emails, with its AI agent running the test.
- The test revealed significant differences in deliverability between vendors, with one vendor having a 50.5% open rate compared to 21.0% for the incumbent vendor.
- The use of an AI agent to run the test has changed how SaaStr buys software, allowing the company to test vendors at production volume and check vendor reporting against underlying records.
SaaStr has conducted vendor bake-offs for software purchases, but these comparisons used to be limited in scope. A few hundred test emails were sent, and the decision was often based on which vendor's representative was most responsive and what the team was already familiar with.
A recent test conducted by SaaStr compared email deliverability across three vendors and over 500,000 emails. The test was run by 10K, SaaStr's AI VP of Marketing, which caught every issue, ran multivariate diagnostics, and provided a clear recommendation.
The test involved sending identical content through each vendor at the same time and pulling delivery, open, click, bounce, complaint, and unsubscribe data from each vendor's dashboard. The results showed that one vendor had a significantly higher open rate, with 50.5% of emails being opened, compared to 21.0% for the incumbent vendor.
The test also revealed issues with the incumbent vendor's reporting, with 80,708 failed sends being reported as 'Completed'. The AI agent ran diagnostic sends to identify the cause of the issue, which was found to be a problem with the vendor's transport layer.
The AI agent scored the vendors on seven criteria, including completed delivery, inbox placement, spam complaints, unsubscribes, human clicks, sponsor reporting, and operator effort. The results showed significant differences between the vendors, with the winning vendor requiring less operator effort.
SaaStr plans to move its core newsletter sends to the winning vendor, ramping up the volume over several days. The company will also confirm unsubscribe sync in both directions and map other workflows that depend on the incumbent vendor before canceling anything.
The use of an AI agent to run the test has changed how SaaStr buys software, allowing the company to test vendors at production volume and check vendor reporting against underlying records. This approach is expected to reduce switching costs and make it easier to compare vendors.
SaaStr's experience has implications for B2B software vendors, as customers' agents can now run evaluations and test vendors at real volume. This may reduce the protection afforded by incumbency and make it easier for challengers to be considered.
The company plans to keep all core vendors in its stack in a running bake-off, with the AI agent running the evaluation and scoring vendors on the same criteria. This approach is expected to improve the buying process and reduce the risk of vendor lock-in.
Questions and answers
What was the purpose of SaaStr's email deliverability test?
The purpose of the test was to compare email deliverability across three vendors and over 500,000 emails, with the goal of identifying the most effective vendor.
How did SaaStr's AI agent contribute to the test?
The AI agent, 10K, ran the test, caught every issue, ran multivariate diagnostics, and provided a clear recommendation based on the results.
What were the implications of the test results for SaaStr's software purchasing process?
The test results changed how SaaStr buys software, allowing the company to test vendors at production volume, check vendor reporting against underlying records, and reduce switching costs.
