NEW YEAR SALE | 40% OFF with code NEWYEAR2026 | Ends Feb 1, 2026

· by Simon Chiu

E2E Email Provider Testing in Rails CI: Verifying Emails Actually Arrive

When you’re building an email marketing platform like Broadcast, you integrate with multiple email providers—AWS SES, Postmark, SendGrid, Mailgun. Each has their own API quirks, SMTP configurations, and authentication methods.

Unit tests mock these integrations. Integration tests verify API calls succeed. But neither answers the question that actually matters: did the email arrive?

This post details how we built end-to-end email provider tests that send real emails through real providers and verify they’re actually delivered.

The Problem

Testing email delivery has always been awkward:

  1. Mocking hides bugs — Your test passes, but you typo’d an API parameter. Production breaks.
  2. API success ≠ delivery — The provider accepted your email. Did it arrive? Did spam filters eat it?
  3. SMTP is stateless — You get a 250 OK. The email vanishes into the void.
  4. Providers differ — SendGrid uses apikey as the literal SMTP username. Postmark uses the same token for API and SMTP. AWS SES has separate SMTP credentials. Each is a landmine.

We wanted tests that would catch real integration failures before they hit production.

The Architecture

The solution has three parts:

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────────┐
│   Rails CI      │────▶│  Email Provider  │────▶│  Cloudflare Email   │
│   (GitHub Actions)    │  (SES/Postmark/etc)    │  Routing            │
└─────────────────┘     └──────────────────┘     └──────────┬──────────┘
        │                                                    │
        │                                                    ▼
        │                                        ┌─────────────────────┐
        │                                        │  Cloudflare Worker  │
        │                                        │  (parse & store)    │
        │                                        └──────────┬──────────┘
        │                                                    │
        │              ┌─────────────────────┐              │
        └─────────────▶│  HTTP API           │◀─────────────┘
           poll/verify │  (fetch emails)     │   store in KV
                       └─────────────────────┘
  1. Test sends email through a real provider to a test address
  2. Cloudflare Email Routing catches all mail to *@test.yourdomain.com
  3. Cloudflare Worker parses the email and stores it in KV
  4. Test polls the Worker’s HTTP API until the email arrives (or times out)

Part 1: The Email Receiver (Cloudflare Worker)

Cloudflare’s Email Routing can forward emails to Workers. The Worker parses incoming mail with postal-mime and stores it in KV:

// Simplified email handler
async function handleEmail(message: EmailMessage, env: Env): Promise<void> {
  const rawEmail = await streamToArrayBuffer(message.raw);
  const parser = new PostalMime();
  const parsed = await parser.parse(rawEmail);

  const emailId = generateEmailId();
  const testId = extractTestId(message.to); // test-{id}@domain.com -> id

  const storedEmail = {
    id: emailId,
    from: message.from,
    to: message.to,
    subject: parsed.subject,
    text: parsed.text,
    html: parsed.html,
    headers: buildHeadersObject(parsed.headers),
    receivedAt: new Date().toISOString(),
  };

  // Store with 1-hour TTL
  await env.EMAILS.put(`email:${emailId}`, JSON.stringify(storedEmail), {
    expirationTtl: 3600,
  });

  // Index by test ID for easy lookup
  if (testId) {
    const listKey = `test:${testId}`;
    const existing = await env.EMAILS.get(listKey);
    const emailIds = existing ? JSON.parse(existing) : [];
    emailIds.push(emailId);
    await env.EMAILS.put(listKey, JSON.stringify(emailIds), {
      expirationTtl: 3600,
    });
  }
}

The key insight: embed a unique test ID in the recipient address ([email protected]). This lets concurrent CI runs retrieve only their emails.

The Worker also exposes HTTP endpoints:

  • GET /test/:testId/wait?timeout=60 — Long-poll until email arrives
  • GET /test/:testId — Get all emails for a test
  • DELETE /test/:testId — Cleanup after test

The /wait endpoint is crucial for CI. Email delivery takes 1-60+ seconds depending on the provider. Rather than sleep-and-pray, we poll:

// Wait endpoint (simplified)
const startTime = Date.now();
const maxWait = Math.min(timeout, 55) * 1000; // CF has 60s limit

while (Date.now() - startTime < maxWait) {
  const emails = await getEmailsForTest(testId);
  if (emails.length >= minCount) {
    return jsonResponse({ emails, count: emails.length });
  }
  await new Promise(resolve => setTimeout(resolve, 500));
}

return jsonResponse({ emails: [], timedOut: true }, 408);

Part 2: The Rails Test Harness

On the Rails side, we created a client to talk to the Worker:

class EmailTestReceiver
  def initialize(base_url:, api_key: nil, domain:)
    @base_url = base_url
    @api_key = api_key
    @domain = domain
  end

  def test_email_address(test_id)
    "test-#{test_id}@#{@domain}"
  end

  def wait_for_email(test_id, timeout: 60)
    uri = URI("#{@base_url}/test/#{test_id}/wait?timeout=#{timeout}")
    # ... HTTP request with timeout + 10s buffer
  end

  def cleanup(test_id)
    # DELETE /test/:testId
  end
end

Each test generates a unique ID, sends to test-{id}@test.yourdomain.com, then waits:

class PostmarkApiE2ETest < EmailProviderE2ETest
  def test_send_email_via_postmark_api
    config = credentials.postmark_api

    email_server = build_mock_email_server(
      vendor: 'postmark',
      postmark_api_token: config.token
    )

    service = PostmarkApiService.new(email_server)

    result = service.send_email(
      to: @recipient,           # test-{uuid}@test.yourdomain.com
      from: sender_address,
      subject: test_subject,    # includes test ID for verification
      body: test_body_text,
      html_body: test_body_html,
      message_stream: 'outbound'
    )

    assert result[:success], "Send failed: #{result[:error]}"

    # Wait up to 90 seconds for delivery
    email = assert_email_received(timeout: 90)

    # Verify content arrived intact
    assert_includes email['subject'], @test_id
    assert_includes email['html'], @test_id
  end
end

Part 3: Rails Credentials for Provider Config

We didn’t want environment variables scattered everywhere. Rails credentials keep everything organized:

# config/credentials/test.yml.enc
e2e:
  email_test_receiver:
    url: https://email-receiver.workers.dev
    domain: test.yourdomain.com

  sender:
    email: [email protected]
    name: E2E Test

  postmark_api:
    enabled: true
    token: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    message_stream: outbound

  aws_ses_api:
    enabled: false  # Toggle providers on/off
    access_key_id: ...
    secret_access_key: ...
    region: us-east-1

  # ... more providers

The enabled flag lets you selectively run tests. Not every environment needs every provider configured.

Tests skip gracefully when disabled:

def skip_unless_provider_enabled(provider_key)
  config = credentials.send(provider_key)
  skip "#{provider_key} not configured" unless config
  skip "#{provider_key} is disabled" unless config.enabled
end

Part 4: GitHub Actions Integration

The workflow runs nightly and on-demand:

name: E2E Email Provider Tests

on:
  workflow_dispatch:  # Manual trigger
  schedule:
    - cron: '0 5 * * *'  # Midnight EST

jobs:
  e2e-email-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 15

    steps:
      - uses: actions/checkout@v4

      - uses: ruby/setup-ruby@v1
        with:
          bundler-cache: true

      - name: Set up test credentials
        run: |
          mkdir -p config/credentials
          echo "${{ secrets.RAILS_TEST_KEY }}" > config/credentials/test.key

      - name: Run E2E tests
        env:
          E2E_EMAIL_TESTS: "1"
        run: bin/rails e2e:email_providers

One secret (RAILS_TEST_KEY) unlocks all the provider credentials. The encrypted test.yml.enc lives in the repo.

What This Catches

Since deploying these tests, we’ve caught:

  • Postmark message stream typos — broadcasts vs broadcast (singular matters!)
  • AWS SES region mismatches — Credentials for us-east-1, config said us-west-2
  • SendGrid username confusion — It’s literally the string apikey, not your actual API key
  • Header encoding issues — UTF-8 subjects that broke on one provider but not others

Each would’ve been a production incident. Now they’re CI failures.

The Costs

This setup is nearly free:

  • Cloudflare Email Routing — Free for up to 200 addresses
  • Cloudflare Workers — Free tier covers 100K requests/day
  • Cloudflare KV — Free tier covers 100K reads/day
  • Email sends — Each provider has free tiers; tests send maybe 10 emails per run

Try It Yourself

The general pattern works for any email-sending application:

  1. Set up a test domain with Cloudflare Email Routing
  2. Deploy a Worker to receive and store emails
  3. Write tests that send to test-{unique-id}@yourdomain.com
  4. Poll the Worker until email arrives or timeout

The test ID in the address is the key trick—it makes concurrent test runs work without collision.

For Broadcast users: these tests run nightly against our own infrastructure, so we know the email providers work before you discover they don’t.

Questions or war stories about email testing? Find me on Twitter/X.

← Back to all posts