Skip to content

09-backups.md

Latest commit

 

History

History
188 lines (125 loc) · 4.67 KB

09-backups.md

File metadata and controls

188 lines (125 loc) · 4.67 KB

09 — Backups

Backups are a first-class concern in this stack. Data loss in Paperless-ngx or Vaultwarden would be severe and potentially irreversible.

Backup Strategy

Layer Method Schedule Location
Postgres database pg_dump via cron Nightly Local volume + offsite
Docker volumes docker-volume-backup sidecar Nightly Local volume + offsite
OCI boot volumes OCI Block Volume Backup Policy Weekly (Bronze) OCI snapshot
Caddy certificates Included in volume backup Nightly With volume backup

Postgres Backups (VM2)

A nightly pg_dump exports the Paperless database to a compressed file.

Manual pg_dump

cd ~/homelab-cloud-stack/vm2/stack

docker compose exec postgres pg_dump \
  -U ${POSTGRES_USER} \
  -d ${POSTGRES_DB} \
  | gzip > ~/backups/paperless_$(date +%Y%m%d).sql.gz

Automated via Cron

Add to the deploy user's crontab (crontab -e):

# Nightly Postgres backup at 02:00
0 2 * * * cd ~/homelab-cloud-stack/vm2/stack && \
  docker compose exec -T postgres pg_dump \
  -U paperless paperless \
  | gzip > ~/backups/paperless_$(date +\%Y\%m\%d).sql.gz

Create the backup directory:

mkdir -p ~/backups

Docker Volume Backups

The offen/docker-volume-backup image handles volume backup to a local or remote destination.

It is included as a service in vm2/stack/docker-compose.yml. Configure it via the environment variables in .env:

BACKUP_CRON_EXPRESSION=0 3 * * *     # 03:00 nightly
BACKUP_RETENTION_DAYS=14             # keep 14 days of backups

Volume backups are stored in a separate backup_archive volume mounted at /archive inside the backup container.

To manually trigger a backup:

docker compose exec backup backup

Offsite Replication (Recommended)

For true durability, replicate backups offsite. Options:

OCI Object Storage (Free Tier)

OCI provides 20 GB of Object Storage on Always-Free. Use rclone to sync backups:

# Install rclone
sudo apt install -y rclone

# Configure OCI Object Storage as a remote
rclone config
# Choose: S3-compatible, use OCI endpoints

# Sync backup directory
rclone sync ~/backups oci-bucket:homelab-backups/

Add to crontab after the pg_dump job:

30 2 * * * rclone sync ~/backups oci-bucket:homelab-backups/ --log-file ~/backups/rclone.log

Restic to Any S3-Compatible Endpoint

# Initialize restic repository
restic -r s3:https://<endpoint>/<bucket> init

# Backup volumes
restic -r s3:https://<endpoint>/<bucket> backup /var/lib/docker/volumes/

# Prune old snapshots (keep 30 days)
restic -r s3:https://<endpoint>/<bucket> forget --keep-daily 30 --prune

OCI Block Volume Backup Policy

In the OCI console, assign a backup policy to each VM's boot volume:

Block Storage → Boot Volumes → [VM boot volume] → Assign Backup Policy → Bronze

Bronze policy: weekly incremental, retained for 1 year. This provides a full-system restore point at the infrastructure level.

Restore Procedures

Restore Postgres from pg_dump

# On VM2
cd ~/homelab-cloud-stack/vm2/stack

# Stop the webserver and worker (not the database)
docker compose stop paperless-web paperless-worker

# Drop and recreate the database
docker compose exec postgres psql -U paperless -c "DROP DATABASE paperless;"
docker compose exec postgres psql -U paperless -c "CREATE DATABASE paperless;"

# Restore from dump
gunzip -c ~/backups/paperless_20250101.sql.gz \
  | docker compose exec -T postgres psql -U paperless paperless

# Restart services
docker compose start paperless-web paperless-worker

Restore Docker Volumes

If a volume was lost or corrupted:

# Stop the affected service
docker compose stop memos

# Remove the damaged volume
docker volume rm vm1_stack_memos_data

# Restore from backup tar (created by docker-volume-backup)
docker run --rm \
  -v vm1_stack_memos_data:/volume \
  -v ~/backups:/backup \
  alpine tar xf /backup/memos_data_backup.tar.gz -C /volume

# Restart
docker compose start memos

Full VM Restore from OCI Snapshot

  1. In OCI Console: Block Storage → Boot Volume Backups → [backup] → Restore Boot Volume
  2. Launch a new instance with the restored boot volume
  3. Update DNS A records to the new instance's public IP
  4. Update the NSG SSH ingress source CIDR to your current IP if needed

Verifying Backups

Run restore tests periodically. At minimum:

  • Monthly: Test a Postgres restore on a temporary VM
  • Quarterly: Test a full volume restore

A backup that has never been tested is not a backup.

Continue Reading

→ 10 — Troubleshooting