Monitoring and Logs

When something looks wrong (a broadcast that never finished, an email server that stopped accepting mail, a webhook that isn’t firing) Broadcast has four places to look. They all live under Application → Monitoring in the sidebar:

All four require the system settings permission. Users without it don’t see the Application section at all.

Error Logs

Error Logs is the record of failures Broadcast noticed on its own: a rejected SMTP login, a webhook endpoint that kept returning 502, an import that skipped rows, a DNS record that drifted from its baseline.

Each entry carries a type and a severity.

Severities run Info, Warning, Error, Critical. Critical entries are the ones worth interrupting your day for: a failed SMTP login means mail is not going out.

Filtering and acknowledging

The filter bar narrows by type, by severity, and to unacknowledged entries only. Acknowledge marks a single entry as handled; Acknowledge All marks everything matching the current filter, so filter first if you only mean to clear one category.

Acknowledging is bookkeeping, not repair: it records that a human looked at the entry. Nothing is deleted.

Clicking a row opens the full entry, including the untruncated message and the channel it belongs to:

Some features write here deliberately. Domain authentication drift, for example, raises an entry after two consecutive failed checks, see Domain DNS Health.

Application Logs

Application Logs streams your installation’s log output into the browser, live. Use it when an Error Log entry doesn’t explain enough, or when you want to watch something happen as you trigger it.

The controls:

• Start Tailing / Stop: begin or end streaming.
• Download: save what’s on screen as a text file (see below).
• Clear: empty the on-screen buffer. It does not touch the log file.
• Auto-scroll: keep the newest line in view; uncheck it to read back without being yanked to the bottom.

Four behaviours are worth knowing:

• You get history immediately. On connect, Broadcast sends roughly the last 200 lines before it starts following the file, so a quiet installation shows its recent past instead of an empty box.
• Streaming stops itself after 2000 lines. This keeps a busy server from flooding your browser. Click Start Tailing to continue.
• A stale log file says so. If the file hasn’t been written to in over a minute, the viewer posts a notice that the host log streamer may be starting up or stopped. Without it, a stalled streamer and an idle app look identical.
• Closing the tab does not stop the stream. The server keeps writing the stream to /opt/broadcast/logs/application.log until you click Stop. When that file passes 50 MB, the server empties it and the viewer continues with the new lines. When no stream runs, the server removes the file.

To search what you’ve captured, use your browser’s own find (Cmd+F / Ctrl+F).

Sending logs to support

Download packages the captured lines into a broadcast-logs-<timestamp>.txt file. Broadcast writes a short header into it first: your version, the environment, whether the installation is managed, the capture time, and the line count, which is exactly the context support would otherwise have to ask you for.

The workflow when reporting a problem:

1. Click Start Tailing.
2. Reproduce the problem.
3. Click Download.
4. Email the file to support@sendbroadcast.net with a description of what you saw. The page’s footer link opens a pre-addressed message.

Log level

Broadcast logs at info: one line for each request and job, plus warnings and errors, without SQL statements. To change the level, set RAILS_LOG_LEVEL in the app .env file (/opt/broadcast/app/.env on an automatic installation):

RAILS_LOG_LEVEL=debug

An empty or unknown value means info. The containers read the file when they start. On an automatic installation, run cd /opt/broadcast && sudo ./broadcast.sh restart. On a manual installation, run docker compose -f docker-compose.manual.yml up -d.

Up to version 2.50.2, Broadcast ignored RAILS_LOG_LEVEL and always logged at debug. Upgrade to fix this. See Logs and disk space if the logs already use a lot of disk space.

Jobs

Almost everything slow in Broadcast (sending a broadcast, running a sequence step, delivering a webhook, checking DNS) happens in a background job. The Jobs dashboard is where you confirm that work is actually moving.

Job statistics count jobs by state: Pending, Scheduled, In Progress, Blocked, Failed, Finished today, and Total finished.

Queues lists each queue with its size, latency, and status. Latency is the useful number: a queue with a size of zero is idle, but a queue with steady size and growing latency is falling behind. Each queue can be paused, resumed, or cleared, and there are pause-all and resume-all controls.

Workers lists the processes doing the work, with a last-heartbeat time. If there are no workers, or every heartbeat is stale, nothing is being processed no matter how healthy the queues look. Dead workers can be pruned.

Recent failed jobs shows what gave up. Individual jobs can be retried or discarded, and the failed-jobs view supports retrying or discarding in bulk. Retry is safe for email sends: delivery is idempotent, so a retried broadcast job will not re-send to recipients it already delivered to.

Job server memory mode, at the top of the page, chooses between Standard (one process for each worker, the default) and Low memory (the whole job server in one process, for small servers). It shows the mode the job server actually runs in, and Restart pending until a change applies. See Performance Tuning for when to use Low memory mode.

Restart Job Server restarts the background worker process. Reach for it when workers are gone or wedged, not as a routine fix.

Logs and disk space

Broadcast keeps its logs in these places:

To see how much space the logs use on an automatic installation:

journalctl --disk-usage
du -sh /opt/broadcast/logs

To free journal space now, keep only the newest 1 GB:

sudo journalctl --vacuum-size=1G

./broadcast.sh upgrade from version 2.50.2 or earlier sets the log level to info and limits application.log without manual steps. Debug lines that are already in the journal stay there until journald removes old entries, or until you run the command above.

A troubleshooting order that works

1. Error Logs: has Broadcast already identified the problem? Filter to Critical first.
2. Jobs: are workers alive and queues draining? A stuck broadcast is far more often a stalled worker than a bug.
3. Application Logs: tail while reproducing the problem, to see the detail behind either of the above.
4. Download and send to support if the cause still isn’t clear.

Related: Domain DNS Health for authentication-record monitoring, Email Server Rate Limits for deferred sends, and Backup & Recovery for backup and restore.