Making Broadcast Sends Bulletproof: Safe to Crash, Safe to Retry, Safe to Resume
Here is the failure every person who sends bulk email has thought about at least once. A broadcast is halfway through a list of forty thousand subscribers. The worker process sending it gets killed: a deploy, an out-of-memory kill, a server reboot, a network partition that times out the SMTP connection. Now you have two questions and no good answer to either. Which subscribers already got the email? And if you restart the job, who gets a second copy?
When you self-host your email platform, both of those questions are yours to answer. There is no provider dashboard deduplicating for you, no support team watching a queue. The software has to be correct on its own. In Broadcast v2.12.0 we reworked the send pipeline around three properties: safe to crash, safe to retry, safe to resume. This post walks through how each one actually works in the code, including a batching change from v2.0.1 that made the whole thing cheaper to run.
Broadcast is a Rails app. Background work runs on SolidQueue, backed by the same PostgreSQL database that holds everything else (I wrote about why in the self-hosting post linked at the end). That choice matters here, because the guarantees below are built out of database constraints and Active Job features, not a separate queue service.
A crash mid-send should never double-send
The core of idempotent delivery is a claim we take before an email goes out, not after. Every send that can be deduplicated (broadcasts and transactional emails) writes a row to an outbound_receipts table first, keyed on the thing being sent and the recipient address:
receipt = OutboundReceipt.create_or_find_by!( receipted_type: "Broadcast", receipted_id: broadcast.id, email: address.downcase ) return unless receipt.previously_new_record?
A partial unique index on (receipted_type, receipted_id, email) enforces that only one such row can ever exist. create_or_find_by! either inserts the row (this attempt owns the send) or finds the existing one (someone already claimed it). If the row was already there, previously_new_record? is false and we return without sending. That is the whole trick: the database, not application logic, decides who is allowed to send to a given address for a given broadcast.
Claiming before the send is the part that makes it safe. If the process dies between the claim and the SMTP handshake, the receipt is still there, so a retry sees the claim and skips. If the send itself raises before the email leaves, we release the claim we just created so a retry can legitimately try again. The failure mode we refused to allow was sending twice. Skipping a claimed-but-not-sent email is recoverable; sending a duplicate is not.
Temporary failures should retry themselves
Not every failure deserves the same response. A syntactically invalid address will never succeed no matter how many times you try it. A connection reset or a provider returning 429 almost certainly will. So the first thing we needed was one place that classifies an error as transient or permanent:
TRANSIENT_ERRORS = [ Net::OpenTimeout, Net::ReadTimeout, Net::SMTPServerBusy, Errno::ECONNRESET, OpenSSL::SSL::SSLError, OutboundMailer::TransientDeliveryError # provider 429 / 5xx ]
Anything not on the transient list is treated as permanent. That default is deliberate: an unrecognized error is far more likely to be a content or config problem than a passing blip, and retrying it just burns attempts.
Transactional emails (password resets, one-time codes) opt into framework-managed retries with a few attempts and a growing backoff. This is the honest part: before this work, transactional sends retried exactly zero times. A single SMTP hiccup could permanently drop a password-reset email, and the user just never got it. Now a transient failure retries, and because of the claim above, a retry that succeeds sends exactly once. When retries are exhausted the job re-raises so SolidQueue records a failed execution instead of pretending the email went out.
Broadcasts handle the same class of error differently, which leads into the next two sections.
One bad address should not sink the whole broadcast
The old broadcast loop had a bug I am not proud of: certain send errors would halt the entire broadcast. Send to thirty-nine thousand good addresses and one mailbox that hard-rejects, and depending on where it landed, the whole run could stop.
Now a permanent error clearly scoped to a single recipient (a rejected address) is logged, that recipient is marked failed, and the send keeps going. A permanent error that could be channel-wide (bad auth, a misconfigured sender, a suspended account) still halts the broadcast, on purpose, because continuing would just mark every remaining subscriber failed for a reason a human needs to fix.
The line we drew is fail-safe. Only errors we can positively identify as one-bad-address are treated as skip-and-continue. Anything ambiguous halts and surfaces an error, because the cost of guessing wrong in that direction (a human resumes a paused broadcast) is much lower than the other direction (silently burning your whole list on a config mistake).
An interrupted send should pick up where it left off
Idempotency stops duplicates, but on its own it would make you re-walk the entire list after a crash, taking the claim-skip path forty thousand times to find the few thousand you had left. Resumable sends fix that.
The broadcast job is an Active Job continuation. As it sends, it advances a cursor: the highest subscriber id it has committed. That cursor lives in a column on the broadcast, and it is written as part of the same batched write that records who was sent (more on batching next), so the cursor never gets ahead of reality. On resume, the job re-queries from the cursor instead of from the top:
subscribers = base_query.where("sequential_id > ?", broadcast.last_subscriber_sequential_id)
Two things can interrupt a send: the worker shutting down (a deploy signals the job, which checkpoints and requeues) and a transient send error (which requeues and resumes rather than blocking a worker for hours on a single unreachable recipient, which is what the old loop did). Either way it comes back and continues from the cursor. The idempotency claim covers the one subscriber sitting exactly on the boundary, so even if a send committed but the cursor had not caught up, that person cannot be sent to twice.
There is a giving-up condition too. A send that keeps failing does not resume forever. After a bounded number of resumptions the job stops and the broadcast is marked failed, so it is visible rather than stuck in “sending” for eternity. An earlier version of this code had exactly that stuck-forever bug, where a dead job left a broadcast that looked resumable but never moved.
Batching the database writes
Earlier, in v2.0.1, we changed how the send loop talks to Postgres. The naive version wrote a row per email as it went: one insert for the recipient record, another for the history entry, counter bumps, a cursor update, all per subscriber. On a large list that is a lot of tiny round-trips.
Now those writes accumulate in memory and flush in bulk with insert_all and update_all. The flush size adapts to the list: small lists flush every few emails so the progress bar still moves, large lists flush in bigger chunks to cut overhead.
BroadcastRecipient.insert_all(pending_recipient_rows) Subscriber.where(id: sent_ids).update_all(last_email_sent_at: now) broadcast.update_column(:last_subscriber_sequential_id, max_sequential_id)
The resume cursor is updated inside this same flush, which is what ties batching and resumability together: whatever the cursor says was sent really was sent and committed, so an interrupt can never lose or repeat the boundary subscriber.
When you hit the rate limit, you can clear it
Broadcast throttles each email server to an hourly limit. If a send hits that ceiling, it waits. Sometimes you know the count is stale (you raised your provider quota, or you were testing), and waiting out the window is pointless. v2.12.0 adds a manual reset:
def reset_rate_limit! update!(rate_limit_reset_at: Time.current) end # the counting window respects the reset cutoff = [1.hour.ago, rate_limit_reset_at].compact.max outbound_receipts.where("created_at > ?", cutoff).count
The reset does not delete any receipt rows. It moves the counting window forward so only emails sent after the reset count against the limit, and a broadcast that was waiting picks back up on its next check.
Why this matters more when you self-host
With a hosted email service, a lot of this is someone else’s problem. Their ops team handles the crash, the retry, the deduplication, and you never see it. When Broadcast runs on your server, that safety net is the software itself. There is no second system to catch a mistake. The correctness has to be built into the structure.
That is also why I lean on database constraints and Active Job instead of clever application code. A partial unique index cannot be talked out of its guarantee by a race between two workers. A continuation resumes whether the interruption was a deploy or a kernel OOM kill. The fewer places correctness depends on a process staying alive, the fewer ways a self-hosted install can surprise its owner. None of this is glamorous, and most of the time you will never notice it working. That is the point.
More on the architecture underneath this: Building Self-Hosting Rails Applications covers the one-database, SolidQueue setup, and E2E Email Provider Testing covers how we verify emails actually arrive before you do.
