# Keep it running: upgrades, backups, monitoring

> Upgrade Broadcast safely, make backups with an offsite copy, restore them on a new server, and find problems fast with Error Logs and Jobs.

Source: https://sendbroadcast.net/university/getting-started/upgrades-backups-monitoring

A self-hosted install is yours to keep healthy. The good news is that the routine is short. In this lesson you learn to upgrade without breaking a send, to keep a backup that survives the loss of the server, and to find the cause of a problem in minutes.

## Upgrades

Broadcast checks for a new version every hour. When one is out, you see an **Update available** notice in the bottom-right corner of the dashboard. Nothing upgrades by itself.

Warning

Make a backup before every upgrade.

To upgrade from the dashboard:

1. Make a backup (see below).
2. Go to **Application → Check for updates**.
3. Click **Check for updates** again to ask the upgrade server.
4. Click **Upgrade now** and confirm.

The upgrade takes 5 to 10 minutes, usually less. The dashboard is unavailable for a few minutes while the new version starts and runs its database updates. The first time you load it after, a **What's new** dialog shows the highlights of every version you skipped.

![The What is new dialog showing a highlight about backup storage with a Set up backup storage button and a Next button](https://sendbroadcast.net/assets/docs/upgrading/whats-new-wizard-7ac08aba.png)

The dialog after an upgrade. It covers every version you skipped.

Broadcast protects sends in progress. It will not start an upgrade while a broadcast is queueing or sending, or while a background job is part way through, because stopping the containers would cut that work off mid-batch. You see **Upgrade not started** with what is running. Wait for it to finish and try again. Work that is only queued does not block the upgrade: it waits in the database and runs after the restart.

From the server, the same upgrade is one command from `/opt/broadcast`:

```bash
./broadcast.sh upgrade
```

It runs the same safety check first. `./broadcast.sh preflight` runs only the check. If an upgrade step fails, the previous version starts again automatically. See [Upgrading](/docs/upgrading) and the [CLI reference](/docs/cli-reference).

## Backups

A backup holds your database: subscribers, broadcasts, sequences, templates and settings. To make one:

1. Go to **Application → Backups**.
2. Click **Create backup**.
3. Wait a few minutes, then download the file if you want a copy.

Or run `./broadcast.sh backup_database` from `/opt/broadcast`. The backup goes to `/opt/broadcast/db/backups/` with a SHA-256 checksum file next to it.

Three facts shape your backup plan. First, scheduled backups are not set up by default. Add a cron job if you want a daily backup, for example at 2 AM:

```bash
0 2 * * * /opt/broadcast/broadcast.sh backup_database >> /opt/broadcast/logs/cron/backup.log 2>&1
```

Second, the server keeps only the newest backup. Each new backup deletes the one before it from `/opt/broadcast/db/backups/`, so a daily cron job does not build a history on the server.

Third, the local backup is on the same disk as the database. If the server dies, the backup dies with it. An offsite copy fixes both problems. On the Backups page, click **Configure** under **Backup destinations**, enter an S3-compatible bucket (AWS S3, Cloudflare R2, Backblaze B2, Wasabi or MinIO), click **Test connection**, then save and turn on uploads. Every new backup now uploads in the background, and **Remote copies to keep** decides how many older backups stay in the bucket. Use credentials scoped to that one bucket.

### Backup on the server only

- Keeps only the newest backup
- On the same disk as the database
- Lost if the server is lost

### Backup with an offsite copy

- Copy goes to an S3-compatible bucket
- Builds a history in the bucket
- Survives the loss of the server

What each setup survives.

![The Backups page with a Create Backup button, local disk and S3-compatible destinations, and a backup history with upload status](https://sendbroadcast.net/assets/docs/backups/backups-page-destinations-53a80270.png)

The Backups page shows each destination, and whether each backup reached the bucket.

If you keep uploaded images on local storage, they live in `/opt/broadcast/app/storage/` and are not in the database backup. Copy them separately.

## Restores

A restore replaces all data in the database. You use it to move to a new server, or to recover from a disaster. The steps:

1. Install Broadcast on the target server with the usual installer.
2. Copy the backup file, and its `.sha256` file, to `/opt/broadcast/`.
3. Run `sudo ./broadcast.sh restore .tar.gz` and type `yes`. 4. Log in with the credentials from the backup and check your channels and subscribers. The restore checks the file against its checksum and refuses a corrupt backup before it touches anything. An older backup on a newer install is fine: the migrations run after the restore. A newer backup on an older install is blocked. Upgrade first. Test a restore now and then, so you know it works before you need it. See [Backup and recovery](/docs/backup). ## Monitoring When something looks wrong, open **Application → Monitoring**. Work through it in this order: 1. **Error Logs.** Broadcast already decided these are problems: a rejected SMTP login, a webhook endpoint that keeps failing, a DNS record that drifted. Filter to **Critical** first. **Acknowledge** records that a person looked. It fixes nothing. 2. **Jobs.** Almost everything slow runs as a background job. If there are no workers, or every heartbeat is old, nothing moves. A queue with growing latency is falling behind. A stuck broadcast is far more often a stalled worker than a bug. 3. **Application Logs.** Watch the live log while you repeat the problem. 4. If the cause is still not clear, send the logs to support. On **Application Logs**, click **Start Tailing**, repeat the problem, click **Download**, and email the file.

![The Error Logs table with SMTP, Configuration, Rate Limit, Webhook, Import and API entries, each with a severity badge](https://sendbroadcast.net/assets/docs/monitoring/error-logs-index-e337a09e.png)

Error Logs: start here, and filter to Critical first.

![The Jobs Dashboard with job statistics, queues with latency, and active workers with their last heartbeat](https://sendbroadcast.net/assets/docs/monitoring/jobs-dashboard-779f22ff.png)

Jobs: check that workers have a recent heartbeat and that queue latency is not growing.

 On the server, `./broadcast.sh diagnose` collects a support bundle. Run it **before** you restart anything, because a restart destroys the container logs that explain what happened. See [Monitoring and logs](/docs/monitoring-and-logs). ## What you learned - Back up, then upgrade from **Check for updates** or with `./broadcast.sh upgrade`. Running sends block an upgrade. - Schedule backups with cron and upload them to an S3-compatible bucket. - A restore checks the checksum and refuses a newer backup on an older install. - Look at Error Logs first, then Jobs, then Application Logs.

## Quiz
1. You click Upgrade now while a broadcast is sending. What happens?
   - The upgrade starts and the broadcast restarts from the beginning
   - The dashboard shows Upgrade not started and changes nothing (correct)
   - The broadcast is aborted so the upgrade can run
   Broadcast will not start an upgrade while a broadcast is queueing or sending, or while a job is part way through.
2. Can you restore a backup from a newer version onto an older installation?
   - Yes, the restore downgrades the schema
   - Yes, if you pass --yes
   - No, upgrade the installation to the backup's version first (correct)
   A newer backup on an older installation is blocked. Older backups on newer installations are supported.
3. Where should you look first when something seems wrong?
   - Application Logs
   - Host System
   - Jobs
   - Error Logs (correct)
   Error Logs is a curated list of problems Broadcast already found, so it is almost always the fastest place to start.
