# pobsync `pobsync` is a pull-based backup service. It runs on a central backup server and pulls data from remote machines via rsync over SSH. The refactor direction is SQL-first: - Django is the management layer and source of truth. - SQLite is the default database; MariaDB is optional. - Backups still use the existing rsync snapshot engine internally. - Scheduling is handled by a Django scheduler service, not host cron. - Legacy YAML import/export exists only for migration and inspection. ## Requirements On the backup server or in the container: - Python 3.11+ - rsync - ssh - SSH key-based access from the backup server to remotes - systemd for the recommended production deployment ## Local Development ``` python3 -m venv .venv . .venv/bin/activate python3 -m pip install -e . mkdir -p var python3 manage.py migrate python3 manage.py createsuperuser python3 manage.py runserver ``` The admin is available at: - http://127.0.0.1:8000/ - http://127.0.0.1:8000/admin/ Staff-only JSON endpoints are available at: - http://127.0.0.1:8000/api/ - http://127.0.0.1:8000/api/status/ ## SQL-First Setup Create global config: ``` pobsync configure-global --backup-root /mnt/backups/pobsync ``` Create a host config: ``` pobsync configure-host --address ``` Run a backup: ``` pobsync backup --prune ``` Create or update a schedule: ``` pobsync schedule --cron "15 2 * * *" --prune ``` Run the scheduler: ``` pobsync scheduler --loop --interval 60 ``` Plan or apply retention manually: ``` pobsync retention pobsync retention --apply --yes --max-delete 10 ``` Discover snapshots already present on disk: ``` pobsync discover-snapshots --host ``` The `pobsync` executable is a thin wrapper around Django management commands. Direct Django access is also available: ``` pobsync django check python3 manage.py run_pobsync_backup --prune ``` ## Migration Helpers Import existing legacy YAML configs: ``` python3 manage.py import_pobsync_configs --prefix /opt/pobsync ``` Export SQL config to legacy runtime YAML for inspection or one-off compatibility: ``` python3 manage.py export_pobsync_configs --prefix /opt/pobsync ``` These commands are migration helpers, not the normal operating model. ## Production With Systemd The recommended production deployment is native systemd services on the backup server. This avoids Docker friction around SSH, filesystems, large backup mounts, and host-level service logs. Recommended layout: ``` /opt/pobsync/app # git checkout /opt/pobsync/venv # Python virtualenv /etc/pobsync/pobsync.env # settings and secrets /var/lib/pobsync # SQLite database, state, runtime SSH key files, static files /backups # backup storage, or set POBSYNC_BACKUP_ROOT to another absolute path ``` Install OS packages first: ``` apt install python3 python3-venv rsync openssh-client ``` From a checked-out copy of this repository, run: ``` sudo scripts/install-systemd ``` By default the installer copies the checkout to `/opt/pobsync/app`, creates `/opt/pobsync/venv`, writes `/etc/pobsync/pobsync.env`, creates `/var/lib/pobsync` and `/backups`, installs dependencies, runs migrations, collects static files, and starts the services. Common overrides: ``` sudo scripts/install-systemd \ --app-dir /opt/pobsync/app \ --backup-root /mnt/backups/pobsync \ --allowed-hosts backup.example.com,localhost,127.0.0.1 \ --csrf-trusted-origins https://backup.example.com ``` Use `--force-env` when you intentionally want the installer to rewrite an existing `/etc/pobsync/pobsync.env`. The installer creates or updates: - `pobsync-web.service` for Gunicorn on `127.0.0.1:8010` - `pobsync-worker.service` for queued backup runs - `pobsync-scheduler.service` for SQL-backed schedules - `/etc/pobsync/pobsync.env` if it does not exist Edit `/etc/pobsync/pobsync.env` before exposing the service: ``` POBSYNC_DJANGO_ALLOWED_HOSTS=backup.example.com,localhost,127.0.0.1 POBSYNC_DJANGO_CSRF_TRUSTED_ORIGINS=https://backup.example.com POBSYNC_BACKUP_ROOT=/backups POBSYNC_WEB_BIND=127.0.0.1:8010 ``` Restart after changes: ``` sudo systemctl restart pobsync-web pobsync-worker pobsync-scheduler ``` Check service state and logs: ``` systemctl status pobsync-web pobsync-worker pobsync-scheduler journalctl -u pobsync-worker -f ``` The Django UI also has a staff-only `/self-check/` page that verifies runtime settings, required binaries, writable paths, database connectivity, global config state, and systemd service state when systemd is available. Update an existing native install: ``` git pull sudo scripts/install-systemd --app-dir /opt/pobsync/app ``` Use an existing reverse proxy by forwarding to `http://127.0.0.1:8010`. To install a simple nginx site file as a starting point: ``` sudo scripts/install-systemd --with-nginx --server-name backup.example.com ``` ## Docker With SQLite Docker Compose is still useful for local development and disposable test installs. Native systemd is preferred for production backup servers. ``` docker compose up --build web ``` This starts Django on: - http://127.0.0.1:8010/ - http://127.0.0.1:8010/admin/ - http://127.0.0.1:8010/api/ - http://127.0.0.1:8010/api/status/ Run the scheduler alongside the web admin: ``` docker compose up --build web scheduler worker ``` The web service runs Django through Gunicorn and serves static files with WhiteNoise. The container persists `/opt/pobsync` and the SQLite database in Docker volumes. Backup data is always available at `/backups` inside the containers. By default this uses `./backups` on the host. Override the host-side mount with `POBSYNC_BACKUP_ROOT`: ``` POBSYNC_BACKUP_ROOT=/mnt/backups/pobsync docker compose up --build web scheduler worker ``` The Django setup UI keeps the backup root fixed at `/backups`; only the Docker mount decides which host directory that points to. ## Django-Managed SSH Keys SSH keys can be managed from the Django UI at `/ssh-credentials/`. Add a private key there, optionally paste `known_hosts` entries, and select the credential either as the global default or as a per-host override. When a backup starts, the worker writes the selected key to `$POBSYNC_HOME/state/ssh-credentials//identity` with `0600` permissions and injects `IdentityFile` into the rsync SSH command. If `known_hosts` is configured, the worker also writes a matching `known_hosts` file and injects `UserKnownHostsFile`. ## Docker With MariaDB ``` docker compose --profile mariadb up --build web-mariadb ``` With the scheduler: ``` docker compose --profile mariadb up --build web-mariadb scheduler-mariadb worker-mariadb ``` SQLite remains the default because it is enough for a single backup server and keeps deployment simple. ## Current Architecture The public command surface is Django-first. The old YAML/cron CLI has been retired from the `pobsync` entrypoint. Discovered snapshots are stored in `SnapshotRecord`, including the base snapshot metadata and a nullable SQL link to the base record when it is known. The Django retention command plans from `SnapshotRecord` instead of rediscovering snapshots from the filesystem. Post-backup pruning from Django also uses the SQL retention service after the completed snapshot is recorded. Staff-only JSON endpoints expose service status, hosts, snapshots, and backup runs for lightweight inspection. Staff-only dashboard views expose the same operational state through Django templates. Host pages include a safe snapshot discovery action that records existing snapshots into SQL. Host pages also include a read-only SQL retention plan view before any destructive pruning action. Schedules can be created or updated from host pages using the same SQL-backed scheduler model. Host config can be edited from host pages while keeping host identity stable. The remaining internal engine code still contains reusable backup primitives: - snapshot naming and metadata - rsync command construction and execution - retention planning and pruning - host locking Next refactor targets: - Move more snapshot lifecycle details into typed domain objects. - Replace remaining dictionary-shaped config at engine boundaries. - Remove legacy YAML import/export once production migration no longer needs it.