d13x.net Disaster Recovery Plan

Recovery Files

• promote-backup-server.sh - run this on the standby server to restore and promote.
• RECOVERY-STEPS.txt - quick text checklist.
• web1-usb-backup.sh - backup script running on primary.
• web1-usb-backup.service
• web1-usb-backup.timer

Primary Server Backup Status

• Backup target mount: /mnt/backup-usb
• Snapshots: /mnt/backup-usb/snapshots (latest symlink maintained)
• DB dumps: /mnt/backup-usb/database/latest-all-databases.sql.gz
• Timer: web1-usb-backup.timer nightly at 02:20 local time (with randomized delay)

Failover Procedure

1. Plug the USB backup drive (BACKUP_USB) into the standby server.
2. Boot standby server and log in with sudo access.
3. Run DRY RUN first (no writes):

sudo -i
DRY_RUN=1 /mnt/backup-usb/runbooks/promote-backup-server.sh

4. Run real restore:

/mnt/backup-usb/runbooks/promote-backup-server.sh

5. Verify services:

systemctl is-active mariadb php8.3-fpm nginx
curl -k -I https://127.0.0.1

6. Move DNS/public routing to standby server.
7. Validate externally: site home page, /status, SSL certificate, and app/database behavior.

Operational Notes

• Risk: real promotion run overwrites nginx/php/systemd config and restores MariaDB data from latest backup.
• Use same Ubuntu major version and matching package versions on standby.
• To restore a specific snapshot: SNAPSHOT_NAME=YYYYMMDD-HHMMSS /mnt/backup-usb/runbooks/promote-backup-server.sh (or pass timestamp as first argument).