pobsync/README.md

pobsync

pobsync is a pull-based backup service. It runs on a central backup server and pulls data from remote machines via rsync over SSH.

The refactor direction is SQL-first:

• Django is the management layer and source of truth.
• SQLite is the default database; MariaDB is optional.
• Backups still use the existing rsync snapshot engine internally.
• Scheduling is handled by a Django scheduler service, not host cron.
• Legacy YAML import/export exists only for migration and inspection.

Requirements

On the backup server or in the container:

• Python 3.11+
• rsync
• ssh
• SSH key-based access from the backup server to remotes
• systemd for the recommended production deployment

Local Development

python3 -m venv .venv
. .venv/bin/activate
python3 -m pip install -e .
mkdir -p var
python3 manage.py migrate
python3 manage.py createsuperuser
python3 manage.py runserver

The admin is available at:

• http://127.0.0.1:8000/
• http://127.0.0.1:8000/admin/

Staff-only JSON endpoints are available at:

• http://127.0.0.1:8000/api/
• http://127.0.0.1:8000/api/status/

SQL-First Setup

Create global config:

pobsync configure-global --backup-root /mnt/backups/pobsync

Create a host config:

pobsync configure-host <host> --address <host-or-ip>

Run a backup:

pobsync backup <host> --prune

Create or update a schedule:

pobsync schedule <host> --cron "15 2 * * *" --prune

Run the scheduler:

pobsync scheduler --loop --interval 60

Plan or apply retention manually:

pobsync retention <host>
pobsync retention <host> --apply --yes --max-delete 10

Discover snapshots already present on disk:

pobsync discover-snapshots --host <host>

The pobsync executable is a thin wrapper around Django management commands. Direct Django access is also available:

pobsync django check
python3 manage.py run_pobsync_backup <host> --prune

Migration Helpers

Import existing legacy YAML configs:

python3 manage.py import_pobsync_configs --prefix /opt/pobsync

Export SQL config to legacy runtime YAML for inspection or one-off compatibility:

python3 manage.py export_pobsync_configs --prefix /opt/pobsync

These commands are migration helpers, not the normal operating model.

Production With Systemd

The recommended production deployment is native systemd services on the backup server. This avoids Docker friction around SSH, filesystems, large backup mounts, and host-level service logs.

Recommended layout:

/opt/pobsync/app          # git checkout
/opt/pobsync/venv         # Python virtualenv
/etc/pobsync/pobsync.env  # settings and secrets
/var/lib/pobsync          # SQLite database, state, runtime SSH key files, static files
/backups                  # backup storage, or set POBSYNC_BACKUP_ROOT to another absolute path

Install OS packages first:

apt install python3 python3-venv rsync openssh-client

From a checked-out copy of this repository, run:

sudo scripts/install-systemd

By default the installer copies the checkout to /opt/pobsync/app, creates /opt/pobsync/venv, writes /etc/pobsync/pobsync.env, creates /var/lib/pobsync and /backups, installs dependencies, runs migrations, collects static files, and starts the services.

Common overrides:

sudo scripts/install-systemd \
  --app-dir /opt/pobsync/app \
  --backup-root /mnt/backups/pobsync \
  --allowed-hosts backup.example.com,localhost,127.0.0.1 \
  --csrf-trusted-origins https://backup.example.com

Use --force-env when you intentionally want the installer to rewrite an existing /etc/pobsync/pobsync.env.

The installer creates or updates:

• pobsync-web.service for Gunicorn on 127.0.0.1:8010
• pobsync-worker.service for queued backup runs
• pobsync-scheduler.service for SQL-backed schedules
• /etc/pobsync/pobsync.env if it does not exist

Edit /etc/pobsync/pobsync.env before exposing the service:

POBSYNC_DJANGO_ALLOWED_HOSTS=backup.example.com,localhost,127.0.0.1
POBSYNC_DJANGO_CSRF_TRUSTED_ORIGINS=https://backup.example.com
POBSYNC_BACKUP_ROOT=/backups
POBSYNC_WEB_BIND=127.0.0.1:8010

Restart after changes:

sudo systemctl restart pobsync-web pobsync-worker pobsync-scheduler

Check service state and logs:

systemctl status pobsync-web pobsync-worker pobsync-scheduler
journalctl -u pobsync-worker -f

The Django UI also has a staff-only /self-check/ page that verifies runtime settings, required binaries, writable paths, database connectivity, global config state, and systemd service state when systemd is available.

Update an existing native install:

git pull
sudo scripts/install-systemd --app-dir /opt/pobsync/app

Use an existing reverse proxy by forwarding to http://127.0.0.1:8010. To install a simple nginx site file as a starting point:

sudo scripts/install-systemd --with-nginx --server-name backup.example.com

Docker With SQLite

Docker Compose is still useful for local development and disposable test installs. Native systemd is preferred for production backup servers.

docker compose up --build web

This starts Django on:

• http://127.0.0.1:8010/
• http://127.0.0.1:8010/admin/
• http://127.0.0.1:8010/api/
• http://127.0.0.1:8010/api/status/

Run the scheduler alongside the web admin:

docker compose up --build web scheduler worker

The web service runs Django through Gunicorn and serves static files with WhiteNoise. The container persists /opt/pobsync and the SQLite database in Docker volumes. Backup data is always available at /backups inside the containers. By default this uses ./backups on the host. Override the host-side mount with POBSYNC_BACKUP_ROOT:

POBSYNC_BACKUP_ROOT=/mnt/backups/pobsync docker compose up --build web scheduler worker

The Django setup UI keeps the backup root fixed at /backups; only the Docker mount decides which host directory that points to.

Django-Managed SSH Keys

SSH keys can be managed from the Django UI at /ssh-credentials/. Add a private key there, optionally paste known_hosts entries, and select the credential either as the global default or as a per-host override.

When a backup starts, the worker writes the selected key to $POBSYNC_HOME/state/ssh-credentials/<id>/identity with 0600 permissions and injects IdentityFile into the rsync SSH command. If known_hosts is configured, the worker also writes a matching known_hosts file and injects UserKnownHostsFile.

Docker With MariaDB

docker compose --profile mariadb up --build web-mariadb

With the scheduler:

docker compose --profile mariadb up --build web-mariadb scheduler-mariadb worker-mariadb

SQLite remains the default because it is enough for a single backup server and keeps deployment simple.

Current Architecture

The public command surface is Django-first. The old YAML/cron CLI has been retired from the pobsync entrypoint. Discovered snapshots are stored in SnapshotRecord, including the base snapshot metadata and a nullable SQL link to the base record when it is known. The Django retention command plans from SnapshotRecord instead of rediscovering snapshots from the filesystem. Post-backup pruning from Django also uses the SQL retention service after the completed snapshot is recorded. Staff-only JSON endpoints expose service status, hosts, snapshots, and backup runs for lightweight inspection. Staff-only dashboard views expose the same operational state through Django templates. Host pages include a safe snapshot discovery action that records existing snapshots into SQL. Host pages also include a read-only SQL retention plan view before any destructive pruning action. Schedules can be created or updated from host pages using the same SQL-backed scheduler model. Host config can be edited from host pages while keeping host identity stable.

The remaining internal engine code still contains reusable backup primitives:

• snapshot naming and metadata
• rsync command construction and execution
• retention planning and pruning
• host locking

Next refactor targets:

• Move more snapshot lifecycle details into typed domain objects.
• Replace remaining dictionary-shaped config at engine boundaries.
• Remove legacy YAML import/export once production migration no longer needs it.