Performance Tuning

Broadcast’s stock configuration is sized for the System requirements baseline: a small server with one or two cores. On a larger machine the stack does not grow to fill it on its own. This page is for advanced users who want to size the web tier, the job tier, and the database connection limit to match their hardware.

How the pieces fit

Broadcast runs three services: the app (the web interface and API, served by Puma), the job worker (sending, sequences, imports, run by Solid Queue), and PostgreSQL. Each app or job process opens its own database connections, so the three settings below are coupled through one shared resource: PostgreSQL’s connection limit.

The first three are environment variables read by the application. Small servers have one more choice, the job server’s memory mode, which you set on the Jobs page (see Low memory mode below). The last is a PostgreSQL server setting.

Where the settings live

On an automatic installation, the application reads its environment from /opt/broadcast/app/.env. Both the app and the job container load this file, so a variable set here applies to both. Add or change lines in that file:

WEB_CONCURRENCY=3
JOB_CONCURRENCY=6

Changes take effect on the next restart:

cd /opt/broadcast
./broadcast.sh restart

On a Manual Installation, set the same variables wherever your compose file sources the application’s environment.

Web processes: WEB_CONCURRENCY

Ruby runs one thread of Ruby code at a time per process, however many threads you configure. Threads only help while a request waits on the database or the network. So with the default single process, the web interface can use at most about one CPU core, no matter how many the server has. On a busy multi-core server this shows up as a dashboard, subscriber search, or public opt-in form that feels slow while the machine looks bored.

WEB_CONCURRENCY starts that many web processes, each with its own threads:

Leave cores for the job tier: auto uses every core for the web interface, which is rarely what you want on a server that also sends email. A good starting point is a third to a half of your cores, rounded down, and never more than you have.

Every web process is a full copy of the application. Budget roughly 500 MB of RAM for each one.

Threads per process: RAILS_MAX_THREADS

Each web process serves requests from a pool of this many threads. The default of 10 is a sensible balance; raising it trades response time for throughput and rarely helps.

The value is capped in practice at 25. The application’s production database pool is 25 connections per process, so a process with more threads than that will queue on the pool and eventually raise ActiveRecord::ConnectionTimeoutError under load. Stay at or below 25.

Job processes: JOB_CONCURRENCY

Background work (sending broadcasts, running sequences, imports, webhooks) runs in the job container across this many processes, each with three threads. When unset, Broadcast uses one fewer than the number of CPU cores, with a minimum of one.

Set it explicitly when you add web processes, so the two tiers do not compete for the same cores. More job processes send faster only up to the rate limits of your email provider; beyond that they add database load for no gain.

Each job process is also a full copy of the application, so budget the same roughly 500 MB each.

The job container also runs one more process, with one thread, for transactional email only. Login codes and password resets then never wait for a free worker. This process always runs, on top of JOB_CONCURRENCY, and you cannot turn it off. Count it like a job process: about 500 MB in your budget, though it usually uses about 300 MB.

In Low memory mode (below), JOB_CONCURRENCY has no effect.

Low memory mode

The job server has two memory modes. You choose one on the Jobs page (Application > Jobs), in the Job server memory mode panel at the top.

In Low memory mode, all background work shares one process: three worker threads, plus one thread for transactional email only, so login codes and password resets still never wait for a broadcast.

On a 2 GB test server, during a send to 8,000 recipients with an import running at the same time, the job container peaked at about 690 MB in Standard mode and about 380 MB in Low memory mode. The server’s lowest free memory went from about 570 MB to about 1,100 MB. Both modes sent at the same speed.

The trade-offs:

• JOB_CONCURRENCY has no effect. Low memory mode always runs one process, so it does not suit a server with many cores and a fast email provider.
• All jobs share one process. One slow or CPU-heavy job, such as a large import, can slow the others a little.
• If that one process stops, all background work stops until the job container restarts. Docker restarts it on its own.

Changing the mode

The job server reads the mode when it starts, so a change applies after the job server restarts. The web interface does not restart.

• Automatic installation: select the mode and click Save and restart job runner. Broadcast restarts only the job container, within a minute or two. While a broadcast is sending or a job is running, Broadcast refuses the change and tells you why. Try again when the work is finished.
• Manual installation: select the mode and click Save, then restart the job container yourself, at a quiet moment:

  docker compose restart job

Until the restart, the panel shows Restart pending. It always shows the mode the job server actually runs in.

Setting the mode in app/.env

The SOLID_QUEUE_SUPERVISOR_MODE environment variable also sets the mode: async is Low memory and fork is Standard. When it is set, it decides the mode, and the Jobs page shows the setting without the choice. Remove the line and restart the job container to manage the mode on the Jobs page again.

SOLID_QUEUE_SUPERVISOR_MODE=async

If Solid Queue runs inside the web server (SOLID_QUEUE_IN_PUMA=true, see Manual Installation), the Jobs page cannot change the mode. Use this variable and restart Broadcast.

To confirm the mode after a restart, look for this line in the job log:

./broadcast.sh logs job     # "Job runner: async mode (source: dashboard)"

The connection limit: max_connections

This is the setting that ties the others together. Every web and job process holds database connections, and PostgreSQL refuses new ones past its limit. The stock installation leaves that limit at PostgreSQL’s default of 100, which is plenty for one web process and a handful of job processes, and not enough once you scale either tier up.

When the limit is hit you will see one of these in the logs:

• FATAL: sorry, too many clients already (PostgreSQL)
• ActiveRecord::ConnectionNotEstablished or PG::ConnectionBad (application)

Estimating what you need

A conservative rule of thumb, counting connections each process may hold at once:

• each web process: RAILS_MAX_THREADS plus 2
• each job process: 5. Count JOB_CONCURRENCY plus 1, for the transactional email process
• plus 10 for the job supervisor, dispatcher, and maintenance tasks

Keep the total under about 80% of max_connections. For example, on an 8-core, 16 GB server with WEB_CONCURRENCY=3, the default 10 threads, and JOB_CONCURRENCY=5:

web:   3 × (10 + 2) = 36
jobs:  (5 + 1) × 5  = 30
fixed:                10
total                 76

That is over 70% of the default 100, so raise the limit to 200 before scaling further.

Raising the limit

Set it through the override file described in Customizing Docker Services. PostgreSQL takes its settings as command-line flags, and Compose replaces the command wholesale rather than merging it, so repeat the stock flag alongside your new one. Create or edit /opt/broadcast/docker-compose.override.yml:

services:
  postgres:
    command:
      - postgres
      - -c
      - idle_in_transaction_session_timeout=10min
      - -c
      - max_connections=200

Then restart, and confirm the new value took effect:

cd /opt/broadcast
./broadcast.sh restart
docker exec postgres psql -U broadcast -c "SHOW max_connections;"

To see how many connections are actually in use at any moment:

docker exec postgres psql -U broadcast -c "SELECT count(*) FROM pg_stat_activity;"

Worked example

An 8-core, 16 GB server, sending through a provider with generous rate limits, where the dashboard felt slow during large sends.

/opt/broadcast/app/.env:

WEB_CONCURRENCY=3
JOB_CONCURRENCY=5

/opt/broadcast/docker-compose.override.yml:

services:
  postgres:
    command:
      - postgres
      - -c
      - idle_in_transaction_session_timeout=10min
      - -c
      - max_connections=200

Memory check: 3 web + 5 job processes + 1 transactional process = 9 processes at about 500 MB each is 4.5 GB, leaving 11.5 GB for PostgreSQL, the OS, and headroom. Connections: 76 of 200.

Verifying

After a restart, confirm each tier came up the way you configured it:

cd /opt/broadcast
./broadcast.sh logs app     # look for "Workers: 3" and a "Worker N ... booted" line for each
./broadcast.sh logs job     # look for the worker processes starting

Both commands follow the log; press Ctrl-C to stop. See Monitoring and Logs for more on reading them.

Going back

Remove the lines from app/.env (or the override file) and restart. The stock behaviour returns immediately: one web process, one job process per core minus one plus the transactional email process, and PostgreSQL’s default limit. There is nothing else to undo. To leave Low memory mode, select Standard on the Jobs page and restart the job server as described above.