Back up and restore Softwarden

Status: written November 3, 2026 (1b E30-3, E30-4, US-1b-12, D67, D72, 1B-D2, 1B-D3); the worker’s nightly backup and restore.sh --rehearse added November 7, 2026. The worker backup was run for real in the image against PostgreSQL 17, and its set was decrypted, restored into a scratch database and checked (migrate --check, upgrade_check --manifest, verify_attachments) by hand; restore.sh --rehearse itself has not yet run end to end (the CI job of the next package runs it). The scripts were tested on a development machine against synthetic data; the nightly timer and a restore drill have not been run on a pilot VM, and nothing is deployed. Record each drill in the log at the end. Targets: RPO 24 hours (at most one night of work lost), RTO 1 business day (back in service the next working day). Rule (D72): no real workplace data is loaded until a restore drill has passed, the pilot is approved in writing, and the employer has named a data contact. See Before real data.

Organization exports aren’t in the media archive. media/exports/ (organizations’ data exports and nightly tenant backups) is left out by both the worker and ops/backup.sh: those zips are rebuilt from the database and the attachments, are short-lived, and would multiply every set by the retained tenant backups. After a restore their rows remain but the files are gone, so a download answers 404 until the next nightly backup or a new export. verify_attachments and upgrade_check don’t look at them.

Two ways to take backups, one format. Since Phase 9 the worker takes them (the default: nothing to set up, status on the System page). The host scripts (ops/backup.sh with its systemd timer) remain an alternative, for a destination only the host can reach. Both write the same encrypted files, so ops/restore.sh restores a set from either.

The worker’s nightly backup (default)

Every night at 02:30 UTC (SOFTWARDEN_BACKUP_CRON, five-field cron in UTC) the backup.nightly schedule queues one backup.run job, which the worker runs in its long-job lane (apps/api/core/backup.py). It writes into the softwarden-backups volume, /data/backups in the worker, as the image user (uid 10001):

/data/backups/daily/softwarden-20261107T023000Z/
    softwarden-20261107T023000Z.manifest.json.enc   upgrade_check --write-manifest, written just before the dump
    softwarden-20261107T023000Z.dump.enc            pg_dump -Fc
    softwarden-20261107T023000Z.media.tar.enc       the attachment files (media/, without .incoming)
    softwarden-20261107T023000Z.sha256              sha256sum lines for the three files
/data/backups/weekly/softwarden-20261108T023000Z/  a Sunday's (UTC) set, hard-linked from daily/

• Encrypted the same way as the host script: openssl enc -aes-256-cbc -pbkdf2 -iter 200000 -salt -pass file:<key>. pg_dump writes straight into openssl and the media tar is streamed into it, so no readable dump touches the disk. The key goes to openssl by file and the database password to pg_dump through a private 0600 PGPASSFILE: never on a command line, in a job’s arguments, in a log line or on the System page.
• Checked before it is published: the dump is decrypted in full and listed with pg_restore --list; the media archive is decrypted and read entry by entry.
• Published atomically: a set is built in daily/.in-progress-<id>/, the .sha256 is written last, then the folder is renamed into place. A run that fails publishes nothing (its folder is removed; a crash’s leftover goes at the next run). A set without its .sha256, or whose .sha256 doesn’t list every file, is incomplete and never counted.
• One at a time: the run holds a PostgreSQL advisory lock, so the nightly job and backup_now --inline can’t overlap. Two sets in the same second get -2, -3 suffixes. A worker stopping mid-run (or losing its job lease) kills the tools and publishes nothing.
• Media consistency: the dump first, then the files. A file deleted after the dump stays on the volume for SOFTWARDEN_ATTACHMENT_RETENTION_DAYS (7), so every file the dump names is in the archive.
• Disk safety: the softwarden-backups volume is on the same disk as the database by default, and a full disk stops PostgreSQL too. Before writing, a run checks the destination’s free space and refuses when free space minus an estimate (the last set’s size × 1.2; on the first run the database size plus the attachment files) would fall below a floor: SOFTWARDEN_BACKUP_MIN_FREE_MB, or by default the larger of 2 GB and 10 % of the disk. The refusal is a recorded failure (backup_failed on the System page). Move the sets to another disk (below) rather than lowering the floor.
• Only into a mounted volume: a run refuses a destination that isn’t a mount point (SOFTWARDEN_BACKUP_REQUIRE_MOUNT, on by default). The app container has no backup volume, so backup_now there would write into the container’s own layer and lose the set with it: run it in the worker.
• Retention: after each good set the newest 14 daily and 8 weekly complete sets are kept (SOFTWARDEN_BACKUP_KEEP_DAILY, SOFTWARDEN_BACKUP_KEEP_WEEKLY); the newest complete set is never deleted.

Status. Admin > System shows the last good backup’s time, age, size and set name, and the last failure (GET /api/v1/system, backup). After 26 hours without a good backup (SOFTWARDEN_BACKUP_STALE_HOURS) it shows the backup_stale warning; a run that failed after the last good one shows backup_failed. Who sees this: the installation’s operator only (PRE_RELEASE_REVIEW F-22), that is platform staff (also at GET /api/v1/platform/system) and on a self-hosted installation only (not SOFTWARDEN_CLOUD), the Admins of its first organization (the one bootstrap_admin made). The Admins of any other organization get the System page’s organization view, with no backup block and no backup warnings. It still names only the stage and a tool’s exit status (for example pg_dump exited with status 1.); the tool’s own message, secrets removed, is in the worker log (docker compose logs worker | grep backup). If a Sunday set can’t also be kept as the week’s, the daily set still counts and the log says why. Warnings never fail /readyz. The job’s own row has the set, size and duration.

Take one now (before an upgrade, or to test):

docker compose exec worker python manage.py backup_now --inline   # in this process; exit 0 only if the set was published
docker compose exec worker python manage.py backup_now             # queue the job and wait for the worker

The key. The init service generates backup_key in the softwarden-secrets volume on first start (or on the first start after upgrading to Phase 9), unless SOFTWARDEN_BACKUP_KEY_FILE names your own key file (a path inside the containers; mount it into worker and app yourself, read-only, mode 600, owned by uid 10001). Lose the key and every backup is unreadable, and the generated key lives on the same host as the backups. Until an Admin confirms a copy is off the host, the System page shows “Backup key is stored on this host: copy it somewhere safe” (backup_key_unacknowledged). Copy it, store it as the key section says, then acknowledge it on the System page (POST /api/v1/system/backup/acknowledge-key, an Admin who sees the installation view (others get 403), or platform staff at POST /api/v1/platform/system/backup/acknowledge-key, audited as backup.key_acknowledged in every organization, since the key is the installation’s (with the Admin named only in their own organization, and as via: another_admin in the others); a new key brings the warning back):

docker compose cp worker:/data/secrets/backup_key ./backup.key   # then move it off the host and delete this copy

Getting sets off the host. The volume is on the same disk as the database. Copy the newest set somewhere else each day (docker compose cp worker:/data/backups/daily/<set> /mnt/offsite/), or put the sets on another disk from the start: in a compose.override.yaml, replace the worker’s softwarden-backups:/data/backups with a bind mount of a directory owned by uid 10001 (sudo install -d -m 700 -o 10001 -g 10001 /mnt/softwarden-backups, then - /mnt/softwarden-backups:/data/backups). The worker refuses a destination it can’t write, and the System page shows why.

Restoring a worker set. Copy the set’s folder and the key to the host (as above), then run ops/restore.sh --yes <folder>/<set>.dump.enc exactly as for a host-script backup (Restore), or prove it first with --rehearse. CI does that sequence on the built stack for every change (ops/ci/inapp-backup-rehearsal.sh: backup_now --inline, the set and key copied out, restore.sh --rehearse); run it on a throwaway stack the same way.

What is backed up

• The database. Uploaded import files and their raw rows live in PostgreSQL until the 30-day purge (1B-D2). One pg_dump -Fc holds every record, review, renewal, import run, attachment row and the audit log.
• The attachment files: the softwarden-media volume, /data/media in app and worker. Attachment rows are in the dump; their files are not, so ops/backup.sh archives the volume right after the dump (tar in a one-off container of the app image, streamed straight into openssl with the same key and parameters: no readable copy touches the disk). A dump and the archive taken after it always agree, because a deleted attachment’s file stays on the volume for SOFTWARDEN_ATTACHMENT_RETENTION_DAYS (7, longer than any backup run) before the nightly attachments.sweep removes it. --no-media skips it, for tests only. docker compose down -v deletes the volume with the others.
• A manifest. upgrade_check --write-manifest records what must survive (reviews, renewal completions, audit events; every record, contract and membership by id). It is what proves a restore worked.
• Not in the dump: the secrets. The softwarden-secrets volume holds the generated Django key, database password and credential encryption key (/data/secrets/credential_keys), and .env holds any you set yourself (including the pre-Phase-1 SOFTWARDEN_SECRETS_KEY). Copy the credential key off the host when it is first shown (bootstrap_admin prints it once) and after every rotation; copy .env whenever it changes. Without the credential key a restored database can’t read any stored secret (connector client secrets must be entered again); without the database password the app can’t connect to the old volume (a restore into a fresh database uses the new one). docker compose down -v deletes the secrets volume with the database volume.
• Not in the dump: the images. Compose rebuilds or reloads them. SOFTWARDEN_IMAGE in .env names the release to run.

The host script (ops/backup.sh) produces, each night, under $BACKUP_DEST/daily/:

On Sundays the same files are also copied to $BACKUP_DEST/weekly/.

Retention

14 daily and 8 weekly (Sunday) backups. After each run the script deletes the older ones, with their manifest and checksum files. That is about two months of history; each backup is about the size of the compressed database. Change BACKUP_KEEP_DAILY / BACKUP_KEEP_WEEKLY only with the employer’s agreement.

The key (read this first)

Backups are encrypted with openssl enc -aes-256-cbc -pbkdf2 -iter 200000 -salt and a key file. Lose the key and every backup is unreadable.

• Generate it once: sudo install -d -m 700 /etc/softwarden && openssl rand -base64 48 | sudo tee /etc/softwarden/backup.key >/dev/null && sudo chmod 600 /etc/softwarden/backup.key.
• Keep a second copy off the host and away from the backups: the employer’s password manager or sealed vault, held by the named data contact and one other person. A backup share that also holds the key protects nothing.
• It never goes in the repository, in .env, or in BACKUP_DEST. The scripts refuse to run if the key file is missing, empty, readable by group or others, or inside the project directory. (Under Git Bash on Windows, for the upgrade rehearsal ops/ci/restore-previous.sh, the mode check is skipped with a warning only when SOFTWARDEN_ALLOW_NOACL_KEY=1 is set, which that script does: NTFS keeps no Unix modes, so every file looks like 644 there. Without it the scripts refuse there too. Servers are Linux, where it always applies.)
• Rotating it: take a new backup with the new key, keep the old key until the last backup made with it is pruned (about 8 weeks).

Set up the host script (alternative; 30 min, once)

1. Pick the destination. A mounted share or any folder the employer provides, on a different disk from the VM’s own. Check the user that will run the job can write there.

2. Create the key as above, and put the off-host copy away now, before the first backup.

3. Write the settings. Either set the variables in the systemd unit (step 5) or copy the example:

   cd ~/softwarden
   cp ops/backup.env.example ops/backup.env
   chmod 600 ops/backup.env
   # edit BACKUP_DEST and BACKUP_KEY_FILE

   Variable	Meaning	Default
   BACKUP_DEST	where backups go	required
   BACKUP_KEY_FILE	the key file	required
   SOFTWARDEN_DIR	the Compose project directory	the current directory
   BACKUP_KEEP_DAILY / BACKUP_KEEP_WEEKLY	how many to keep	14 / 8

   ops/backup.env holds paths, not secrets, but keep it out of the repository. Variables already set in the environment win over the file.

4. Run it once by hand from the project directory with the stack running (app and db up):

   cd ~/softwarden
   ops/backup.sh

   One line per step, ending backup complete. The exit code is 0 only if every step passed. Check $BACKUP_DEST/daily/ for the three files.

5. Schedule it. Edit the three paths marked CHANGE in ops/softwarden-backup.service, then:

   sudo cp ops/softwarden-backup.service ops/softwarden-backup.timer /etc/systemd/system/
   sudo systemctl daemon-reload
   sudo systemctl enable --now softwarden-backup.timer
   systemctl list-timers softwarden-backup      # next run about 02:30, up to 15 minutes later

   The service user needs the docker group, read access to the key file, and write access to BACKUP_DEST.

6. Test the timer’s path: sudo systemctl start softwarden-backup.service, then journalctl -u softwarden-backup -n 20.

The install guide points here from its “Set up nightly backups” step.

The host script’s nightly job

ops/backup.sh does, in order, and stops with a non-zero exit on the first failure:

1. checks the key file and destination, and that the database is up;
2. writes the manifest with upgrade_check --write-manifest and encrypts it;
3. streams pg_dump -Fc through docker compose exec -T db straight into openssl, so no readable dump touches the disk;
4. decrypts the result and lists its contents with pg_restore -l to prove it’s usable;
5. writes the checksum file for the dump (and manifest), so a later failure can’t leave the dump without one;
6. streams a tar of the attachment volume (a one-off docker compose run of the app image) straight into openssl, then decrypts it and lists it with tar -t to prove it’s usable, and rewrites the checksum file to include it; a file that changed while it was read (an upload during the backup) is kept as read and is not an error (--warning=no-file-changed, tar’s exit 1); on Sundays copies everything to weekly/;
7. prunes to 14 daily and 8 weekly (each with its manifest, media archive and checksums).

Nothing secret is printed. Output goes to the journal (the worker’s backup shows on the System page; this one doesn’t). Someone has to look: a missed night is invisible unless a person checks. Every Monday, run systemctl status softwarden-backup.service and ls -l $BACKUP_DEST/daily | tail, or have the employer’s monitoring alert when the newest file is over 26 hours old. Until that is set up, the IT coordinator owns the weekly check.

Before an upgrade, run ops/backup.sh by hand first. It replaces the pg_dump shortcut in the upgrade runbook.

Restore

This replaces the database. For a real recovery, tell the IT staff first; work done after the backup is lost.

cd ~/softwarden
docker compose up -d db
BACKUP_KEY_FILE=/etc/softwarden/backup.key \
  ops/restore.sh --yes $BACKUP_DEST/daily/softwarden-<date>.dump.enc

It refuses without --yes. It then:

1. checks the checksum file, and that the key opens the backup;
2. stops app and worker, drops and recreates the softwarden database, and runs pg_restore;
3. restores the attachment files, only with the default database. Before stopping anything it checks that the checksum file lists the media archive and that the archive decrypts in full and reads as a tar (otherwise it stops and changes nothing; --no-media restores the database alone). With app and worker stopped it extracts the archive into a staging folder inside the volume (/data/media/.incoming) and, only once that has succeeded, removes the old files and moves the staged ones into place, as the app user, keeping owner and mode. A failed or cut-off stream never deletes the current files; a leftover .incoming is cleared by the next attempt. A backup made before Phase 2 has no archive, and the script says so;
4. starts app and worker, waits for app to be healthy, runs upgrade_check --manifest with the manifest saved next to the backup, and then verify_attachments, which checks every attachment file’s size and SHA-256 and fails the restore naming any that is missing or changed.

Every line should start with ✓ (OK on a console that can’t show the check mark, such as a Windows console on cp1252), ending “All upgrade checks passed.” Then sign in and open Needs attention. Put the previous .env back first if the host is new.

Attachment files. With --database NAME the attachment volume is left alone (it belongs to the live database). To check files by hand at any time: docker compose exec app python manage.py verify_attachments (non-zero exit on a problem). Files that no row references (from a later archive) are harmless; the sweep removes them after the retention window.

--database NAME restores into a different database (for example softwarden_restore_test, to try a backup safely on a live stack); the manifest check is skipped then, because the app only looks at its own database. --no-manifest skips it; use that only when testing.

Rehearsal: restore into a scratch database

--rehearse proves a backup restores without touching anything live: no --yes, no service stopped, the live database and attachment volume untouched. It is non-interactive and exits non-zero on the first failed step, so CI can run it.

docker compose up -d db
BACKUP_KEY_FILE=/etc/softwarden/backup.key ops/restore.sh --rehearse ./softwarden-20261107T023000Z/softwarden-20261107T023000Z.dump.enc

It checks the .sha256 (required) and that the key opens the dump and the media archive, restores into a scratch database (softwarden_rehearsal, or a --database NAME that doesn’t exist yet: a rehearsal refuses to reuse any other existing database, since it drops its scratch database), then runs, in one-off app containers pointed at that database: migrate --check (every migration of this image applied), upgrade_check --manifest with the manifest saved in the set (nothing lost), and verify_attachments against the media archive unpacked inside the container (every attachment file present and matching its size and SHA-256), and finally counts the stored secrets this stack’s credential key can’t open (any at all fails the rehearsal: the set’s secrets were encrypted under a key this stack doesn’t have). The scratch database is dropped at the end, pass or fail. --no-media rehearses the database alone.

Upgrade rehearsal. A backup of the previous release restored under the new image has migrations to apply: add --migrate to run migrate on the scratch database first, then the same checks. This is what the CI job runs against the previous tagged release.

Linux only, like the other scripts (bash, GNU coreutils, docker compose). The one-off containers unpack the media archive into their own /tmp, so Docker’s storage needs that much free space.

Restore drill (before real data, then quarterly)

D67 asks for a restore to a fresh Compose stack before real data is loaded, then every quarter, on the version currently running. Target: finished inside a working day.

1. Bring three things to a spare VM (or a second project directory with its own volumes), each from its off-host copy, never from the live host: the latest nightly backup files, the backup key, and the credential key (credential_keys, the key bootstrap_admin printed once; on a default install it lives in the softwarden-secrets volume, not in .env, and no backup contains it). Bring a copy of .env too if you keep one.
2. Follow Install up to step 4 with the same SOFTWARDEN_IMAGE and that .env, plus SOFTWARDEN_CREDENTIAL_KEYS=<the old credential key> in it (several keys: comma-separated, newest first). Without it the new stack generates a key of its own, and every stored secret in the restored data (connection client secrets, SMTP and SSO secrets, webhook and Teams URLs) is unreadable. Don’t run bootstrap_admin. A rehearsal on the live stack (ops/restore.sh --rehearse <set>) is a quick check between drills; it doesn’t replace the drill.
3. Start only the database, then restore: docker compose up -d db, then ops/restore.sh --yes <backup>. Time it from the start of step 1. The script ends by counting stored secrets this stack’s credential key can’t open; any at all fails it (exit 1) with a message saying which key is missing.
4. Confirm the manifest check passes (every line ✓) and the script ends “restore complete; the manifest check, the attachment check and the stored-secrets check passed”. Run docker compose exec app python manage.py doctor: no stored_secrets warning. Sign in with an existing Admin, check that Administration › System status lists no “Secrets unreadable” warning, open Needs attention, open one record with its reviews, and run Test connection on one discovery connection or send a test email (either reads a stored secret).
5. Record the date, the backup used, the time taken and the result in the log below, and schedule the next drill for three months out. A pass means every check is ✓, no stored_secrets warning, and sign-in works. Data that restores without its secrets is a failed drill: the off-host credential key copy is missing or stale.
6. Tear down the drill stack (docker compose down -v) and delete the copied backup files from the spare machine.

A failed drill is a finding, not a mark against anyone: fix the cause, write it in Notes, and run it again.

Before real data

Real workplace data is not loaded until all three are true (D72). Until then, synthetic data only.

1. A restore drill has passed (the log below has a row with result “pass”), and the nightly job has run successfully for at least a week.
2. Written pilot approval from the employer (D02).
3. A named employer data contact who holds the off-host key copy, is told when a backup fails, and decides what happens to the data if the pilot ends.

If any is missing, don’t import. Say so, and point to this page.

Log

Drills and notable restores. A pass means every manifest check is ✓, no stored_secrets warning, and sign-in works.

Next drill due: _____ (quarterly, and before real data).