Performance Tuning

Broadcast’s stock configuration is sized for the System requirements baseline: a small server with one or two cores. On a larger machine the stack does not grow to fill it on its own. This page is for advanced users who want to size the web tier, the job tier, and the database connection limit to match their hardware.

Note

None of this is required. A default installation runs correctly on any supported server. Tune only when you have a specific symptom (a slow dashboard on a machine that looks idle, or connection errors under load) and change one thing at a time.

How the pieces fit

Broadcast runs three services: the app (the web interface and API, served by Puma), the job worker (sending, sequences, imports, run by Solid Queue), and PostgreSQL. Each app or job process opens its own database connections, so the three settings below are coupled through one shared resource: PostgreSQL’s connection limit.

Setting Controls Default
WEB_CONCURRENCY Number of web (Puma) processes 1 process
RAILS_MAX_THREADS Threads per web process 10
JOB_CONCURRENCY Number of background job processes CPU cores minus 1
max_connections PostgreSQL’s connection limit 100

The first three are environment variables read by the application. The last is a PostgreSQL server setting.

Where the settings live

On an automatic installation, the application reads its environment from /opt/broadcast/app/.env. Both the app and the job container load this file, so a variable set here applies to both. Add or change lines in that file:

WEB_CONCURRENCY=3
JOB_CONCURRENCY=6

Changes take effect on the next restart:

cd /opt/broadcast
./broadcast.sh restart

Warning

restart stops every service, including a broadcast that is part-way through sending. Sending resumes where it left off after the restart, but check the dashboard for anything in progress and pick a quiet moment.

On a Manual Installation, set the same variables wherever your compose file sources the application’s environment.

Web processes: WEB_CONCURRENCY

Ruby runs one thread of Ruby code at a time per process, however many threads you configure. Threads only help while a request waits on the database or the network. So with the default single process, the web interface can use at most about one CPU core, no matter how many the server has. On a busy multi-core server this shows up as a dashboard, subscriber search, or public opt-in form that feels slow while the machine looks bored.

WEB_CONCURRENCY starts that many web processes, each with its own threads:

Value Effect
unset or 0 One process (the default)
2 or more That many processes, with the application loaded once and shared between them to save memory
auto One process per CPU core

Leave cores for the job tier: auto uses every core for the web interface, which is rarely what you want on a server that also sends email. A good starting point is a third to a half of your cores, rounded down, and never more than you have.

Every web process is a full copy of the application. Budget roughly 500 MB of RAM for each one.

Threads per process: RAILS_MAX_THREADS

Each web process serves requests from a pool of this many threads. The default of 10 is a sensible balance; raising it trades response time for throughput and rarely helps.

The value is capped in practice at 25. The application’s production database pool is 25 connections per process, so a process with more threads than that will queue on the pool and eventually raise ActiveRecord::ConnectionTimeoutError under load. Stay at or below 25.

Job processes: JOB_CONCURRENCY

Background work (sending broadcasts, running sequences, imports, webhooks) runs in the job container across this many processes, each with three threads. When unset, Broadcast uses one fewer than the number of CPU cores, with a minimum of one.

Set it explicitly when you add web processes, so the two tiers do not compete for the same cores. More job processes send faster only up to the rate limits of your email provider; beyond that they add database load for no gain.

Each job process is also a full copy of the application, so budget the same roughly 500 MB each.

The connection limit: max_connections

This is the setting that ties the others together. Every web and job process holds database connections, and PostgreSQL refuses new ones past its limit. The stock installation leaves that limit at PostgreSQL’s default of 100, which is plenty for one web process and a handful of job processes, and not enough once you scale either tier up.

When the limit is hit you will see one of these in the logs:

  • FATAL: sorry, too many clients already (PostgreSQL)
  • ActiveRecord::ConnectionNotEstablished or PG::ConnectionBad (application)

Estimating what you need

A conservative rule of thumb, counting connections each process may hold at once:

  • each web process: RAILS_MAX_THREADS plus 2
  • each job process: 5
  • plus 10 for the job supervisor, dispatcher, and maintenance tasks

Keep the total under about 80% of max_connections. For example, on an 8-core, 16 GB server with WEB_CONCURRENCY=3, the default 10 threads, and JOB_CONCURRENCY=5:

web:   3 × (10 + 2) = 36
jobs:  5 × 5        = 25
fixed:                10
total                 71

That is over 70% of the default 100, so raise the limit to 200 before scaling further.

Raising the limit

Set it through the override file described in Customizing Docker Services. PostgreSQL takes its settings as command-line flags, and Compose replaces the command wholesale rather than merging it, so repeat the stock flag alongside your new one. Create or edit /opt/broadcast/docker-compose.override.yml:

services:
  postgres:
    command:
      - postgres
      - -c
      - idle_in_transaction_session_timeout=10min
      - -c
      - max_connections=200

Then restart, and confirm the new value took effect:

cd /opt/broadcast
./broadcast.sh restart
docker exec postgres psql -U broadcast -c "SHOW max_connections;"

Warning

Each PostgreSQL connection costs memory whether or not it is busy. Raising max_connections on a small server can push it into swap under load, which is worse than the error it was meant to prevent. Raise it only as far as your estimate needs, and keep at least 2 GB of RAM free for PostgreSQL and the operating system after budgeting for your web and job processes. A value above 500 is almost never right for a single Broadcast server.

To see how many connections are actually in use at any moment:

docker exec postgres psql -U broadcast -c "SELECT count(*) FROM pg_stat_activity;"

Worked example

An 8-core, 16 GB server, sending through a provider with generous rate limits, where the dashboard felt slow during large sends.

/opt/broadcast/app/.env:

WEB_CONCURRENCY=3
JOB_CONCURRENCY=5

/opt/broadcast/docker-compose.override.yml:

services:
  postgres:
    command:
      - postgres
      - -c
      - idle_in_transaction_session_timeout=10min
      - -c
      - max_connections=200

Memory check: 3 web + 5 job = 8 processes at about 500 MB each is 4 GB, leaving 12 GB for PostgreSQL, the OS, and headroom. Connections: 71 of 200.

Verifying

After a restart, confirm each tier came up the way you configured it:

cd /opt/broadcast
./broadcast.sh logs app     # look for "Workers: 3" and a "Worker N ... booted" line for each
./broadcast.sh logs job     # look for the worker processes starting

Both commands follow the log; press Ctrl-C to stop. See Monitoring and Logs for more on reading them.

Going back

Remove the lines from app/.env (or the override file) and restart. The stock behaviour returns immediately: one web process, one job process per core minus one, and PostgreSQL’s default limit. There is nothing else to undo.

Last updated

Was this page helpful?

Thanks for your feedback!

Thanks for letting us know. We'll work on improving this page.