hippogrief/pobsync
1
0
Fork 0
Code Issues 21 Pull Requests Actions Packages Projects Releases 3 Wiki Activity

(config) Install native runtime requirements and split docs

Browse Source 
Teach the systemd installer to install required Debian/Ubuntu
packages by default, with an opt-out for users who manage system
dependencies themselves.

Add explicit MariaDB install-extra handling so native installs can
pull in both the Python extra and required client build packages.

Slim down the README to production setup and operations, and move
development, Docker, migration helper, and architecture notes into
docs/development.md.
This commit is contained in:
Peter van Arkel
2026-05-19 18:15:34 +02:00
parent 372a857f15
commit b1789d8621
3 changed files with 339 additions and 224 deletions
Download Patch File Download Diff File Expand all files Collapse all files

326
README.md
View File

@@ -1,134 +1,30 @@
# pobsync # pobsync
`pobsync` is a pull-based backup service. It runs on a central backup server and pulls data from remote machines via rsync over SSH. `pobsync` is a pull-based backup service. It runs on a central backup server and pulls data from remote machines via
rsync over SSH.
The refactor direction is SQL-first: The current refactor is SQL-first:
- Django is the management layer and source of truth. - Django is the management layer and source of truth.
- SQLite is the default database; MariaDB is optional. - SQLite is the default database; MariaDB is optional.
- Backups still use the existing rsync snapshot engine internally. - Backups use the existing rsync snapshot engine internally.
- Scheduling is handled by a Django scheduler service, not host cron. - Scheduling is handled by a Django scheduler service, not host cron.
- Legacy YAML import/export exists only for migration and inspection. - SSH keys can be managed from Django and selected globally or per host.
## Requirements ## Recommended Production Install
On the backup server or in the container: The recommended production deployment is native systemd services on the backup server. Docker Compose remains available
for development and disposable test installs, but native systemd avoids Docker friction around SSH, filesystem mounts,
- Python 3.11+ large backup storage, and host-level service logs.
- rsync
- ssh
- SSH key-based access from the backup server to remotes
- systemd for the recommended production deployment
## Local Development
```
python3 -m venv .venv
. .venv/bin/activate
python3 -m pip install -e .
mkdir -p var
python3 manage.py migrate
python3 manage.py createsuperuser
python3 manage.py runserver
```
The admin is available at:
- http://127.0.0.1:8000/
- http://127.0.0.1:8000/admin/
Staff-only JSON endpoints are available at:
- http://127.0.0.1:8000/api/
- http://127.0.0.1:8000/api/status/
## SQL-First Setup
Create global config:
```
pobsync configure-global --backup-root /mnt/backups/pobsync
```
Create a host config:
```
pobsync configure-host <host> --address <host-or-ip>
```
Run a backup:
```
pobsync backup <host> --prune
```
Create or update a schedule:
```
pobsync schedule <host> --cron "15 2 * * *" --prune
```
Run the scheduler:
```
pobsync scheduler --loop --interval 60
```
Plan or apply retention manually:
```
pobsync retention <host>
pobsync retention <host> --apply --yes --max-delete 10
```
Discover snapshots already present on disk:
```
pobsync discover-snapshots --host <host>
```
The `pobsync` executable is a thin wrapper around Django management commands. Direct Django access is also available:
```
pobsync django check
python3 manage.py run_pobsync_backup <host> --prune
```
## Migration Helpers
Import existing legacy YAML configs:
```
python3 manage.py import_pobsync_configs --prefix /opt/pobsync
```
Export SQL config to legacy runtime YAML for inspection or one-off compatibility:
```
python3 manage.py export_pobsync_configs --prefix /opt/pobsync
```
These commands are migration helpers, not the normal operating model.
## Production With Systemd
The recommended production deployment is native systemd services on the backup server. This avoids Docker friction around
SSH, filesystems, large backup mounts, and host-level service logs.
Recommended layout: Recommended layout:
``` ```
/opt/pobsync/app # git checkout /opt/pobsync/app # installed app checkout
/opt/pobsync/venv # Python virtualenv /opt/pobsync/venv # Python virtualenv
/etc/pobsync/pobsync.env # settings and secrets /etc/pobsync/pobsync.env # settings and secrets
/var/lib/pobsync # SQLite database, state, runtime SSH key files, static files /var/lib/pobsync # SQLite database, state, runtime SSH key files, static files
/backups # backup storage, or set POBSYNC_BACKUP_ROOT to another absolute path /backups # backup storage, or set another absolute path
```
Install OS packages first:
```
apt install python3 python3-venv rsync openssh-client
``` ```
From a checked-out copy of this repository, run: From a checked-out copy of this repository, run:
@@ -137,43 +33,42 @@ From a checked-out copy of this repository, run:
sudo scripts/install-systemd sudo scripts/install-systemd
``` ```
By default the installer copies the checkout to `/opt/pobsync/app`, creates `/opt/pobsync/venv`, writes The installer will, by default:
`/etc/pobsync/pobsync.env`, creates `/var/lib/pobsync` and `/backups`, installs dependencies, runs migrations, collects
static files, and starts the services. - install required Debian/Ubuntu OS packages with `apt-get`
- copy the checkout to `/opt/pobsync/app`
- create `/opt/pobsync/venv`
- write `/etc/pobsync/pobsync.env` if it does not exist
- create `/var/lib/pobsync`, `/var/log/pobsync`, and the backup root
- install Python dependencies
- run migrations and collect static files
- install and start `pobsync-web`, `pobsync-worker`, and `pobsync-scheduler`
Common overrides: Common overrides:
``` ```
sudo scripts/install-systemd \ sudo scripts/install-systemd \
--app-dir /opt/pobsync/app \
--backup-root /mnt/backups/pobsync \ --backup-root /mnt/backups/pobsync \
--allowed-hosts backup.example.com,localhost,127.0.0.1 \ --allowed-hosts backup.example.com,localhost,127.0.0.1 \
--csrf-trusted-origins https://backup.example.com --csrf-trusted-origins https://backup.example.com
``` ```
Use `--force-env` when you intentionally want the installer to rewrite an existing `/etc/pobsync/pobsync.env`. Use `--no-install-os-packages` if you want to manage system packages yourself. Use `--force-env` only when you want the
installer to rewrite an existing `/etc/pobsync/pobsync.env`.
The installer creates or updates: For MariaDB support, add:
- `pobsync-web.service` for Gunicorn on `127.0.0.1:8010`
- `pobsync-worker.service` for queued backup runs
- `pobsync-scheduler.service` for SQL-backed schedules
- `/etc/pobsync/pobsync.env` if it does not exist
Edit `/etc/pobsync/pobsync.env` before exposing the service:
``` ```
POBSYNC_DJANGO_ALLOWED_HOSTS=backup.example.com,localhost,127.0.0.1 sudo scripts/install-systemd --install-extras mariadb
POBSYNC_DJANGO_CSRF_TRUSTED_ORIGINS=https://backup.example.com
POBSYNC_BACKUP_ROOT=/backups
POBSYNC_WEB_BIND=127.0.0.1:8010
``` ```
Restart after changes: ## Services
``` The installer creates:
sudo systemctl restart pobsync-web pobsync-worker pobsync-scheduler
``` - `pobsync-web.service`: Gunicorn Django control panel on `127.0.0.1:8010`
- `pobsync-worker.service`: queued backup worker
- `pobsync-scheduler.service`: SQL-backed schedule dispatcher
Check service state and logs: Check service state and logs:
@@ -182,103 +77,88 @@ systemctl status pobsync-web pobsync-worker pobsync-scheduler
journalctl -u pobsync-worker -f journalctl -u pobsync-worker -f
``` ```
The Django UI also has a staff-only `/self-check/` page that verifies runtime settings, required binaries, writable Restart after configuration changes:
paths, database connectivity, global config state, and systemd service state when systemd is available.
Update an existing native install: ```
sudo systemctl restart pobsync-web pobsync-worker pobsync-scheduler
```
## Reverse Proxy
Use an existing reverse proxy by forwarding to:
```
http://127.0.0.1:8010
```
To install a starter nginx site file:
```
sudo scripts/install-systemd --with-nginx --server-name backup.example.com
```
For HTTPS behind a reverse proxy, set:
```
POBSYNC_DJANGO_ALLOWED_HOSTS=backup.example.com,localhost,127.0.0.1
POBSYNC_DJANGO_CSRF_TRUSTED_ORIGINS=https://backup.example.com
```
## Django UI
After install, open the control panel through your reverse proxy or directly at:
```
http://127.0.0.1:8010/
```
Create a superuser if needed:
```
sudo -u pobsync /opt/pobsync/venv/bin/python /opt/pobsync/app/manage.py createsuperuser
```
The UI includes:
- dashboard and host detail pages
- global and per-host config forms
- schedule editing
- manual backup queueing
- snapshot discovery
- SQL retention planning and apply flow
- Django-managed SSH keys
- `/self-check/` for runtime checks
## SSH Keys
SSH keys can be managed from `/ssh-credentials/`. Add a private key, optionally paste `known_hosts` entries, and select
the credential either as the global default or as a per-host override.
When a backup starts, the worker writes the selected key to:
```
$POBSYNC_HOME/state/ssh-credentials/<id>/identity
```
The key file is written with `0600` permissions and injected into the rsync SSH command with `IdentityFile`.
## Updates
From a fresh checkout or the existing app directory:
``` ```
git pull git pull
sudo scripts/install-systemd --app-dir /opt/pobsync/app sudo scripts/install-systemd --app-dir /opt/pobsync/app
``` ```
Use an existing reverse proxy by forwarding to `http://127.0.0.1:8010`. To install a simple nginx site file as a Then check:
starting point:
``` ```
sudo scripts/install-systemd --with-nginx --server-name backup.example.com systemctl status pobsync-web pobsync-worker pobsync-scheduler
``` ```
## Docker With SQLite ## Development
Docker Compose is still useful for local development and disposable test installs. Native systemd is preferred for Development, Docker, migration helper commands, and architecture notes live in:
production backup servers.
``` - [docs/development.md](docs/development.md)
docker compose up --build web
```
This starts Django on:
- http://127.0.0.1:8010/
- http://127.0.0.1:8010/admin/
- http://127.0.0.1:8010/api/
- http://127.0.0.1:8010/api/status/
Run the scheduler alongside the web admin:
```
docker compose up --build web scheduler worker
```
The web service runs Django through Gunicorn and serves static files with WhiteNoise. The container persists `/opt/pobsync`
and the SQLite database in Docker volumes.
Backup data is always available at `/backups` inside the containers. By default this uses `./backups` on the host.
Override the host-side mount with `POBSYNC_BACKUP_ROOT`:
```
POBSYNC_BACKUP_ROOT=/mnt/backups/pobsync docker compose up --build web scheduler worker
```
The Django setup UI keeps the backup root fixed at `/backups`; only the Docker mount decides which host directory
that points to.
## Django-Managed SSH Keys
SSH keys can be managed from the Django UI at `/ssh-credentials/`. Add a private key there, optionally paste
`known_hosts` entries, and select the credential either as the global default or as a per-host override.
When a backup starts, the worker writes the selected key to `$POBSYNC_HOME/state/ssh-credentials/<id>/identity`
with `0600` permissions and injects `IdentityFile` into the rsync SSH command. If `known_hosts` is configured, the
worker also writes a matching `known_hosts` file and injects `UserKnownHostsFile`.
## Docker With MariaDB
```
docker compose --profile mariadb up --build web-mariadb
```
With the scheduler:
```
docker compose --profile mariadb up --build web-mariadb scheduler-mariadb worker-mariadb
```
SQLite remains the default because it is enough for a single backup server and keeps deployment simple.
## Current Architecture
The public command surface is Django-first. The old YAML/cron CLI has been retired from the `pobsync` entrypoint.
Discovered snapshots are stored in `SnapshotRecord`, including the base snapshot metadata and a nullable SQL link to the
base record when it is known.
The Django retention command plans from `SnapshotRecord` instead of rediscovering snapshots from the filesystem.
Post-backup pruning from Django also uses the SQL retention service after the completed snapshot is recorded.
Staff-only JSON endpoints expose service status, hosts, snapshots, and backup runs for lightweight inspection.
Staff-only dashboard views expose the same operational state through Django templates.
Host pages include a safe snapshot discovery action that records existing snapshots into SQL.
Host pages also include a read-only SQL retention plan view before any destructive pruning action.
Schedules can be created or updated from host pages using the same SQL-backed scheduler model.
Host config can be edited from host pages while keeping host identity stable.
The remaining internal engine code still contains reusable backup primitives:
- snapshot naming and metadata
- rsync command construction and execution
- retention planning and pruning
- host locking
Next refactor targets:
- Move more snapshot lifecycle details into typed domain objects.
- Replace remaining dictionary-shaped config at engine boundaries.
- Remove legacy YAML import/export once production migration no longer needs it.

185
docs/development.md Normal file
View File

@@ -0,0 +1,185 @@
# Development Notes
This document contains development and optional Docker workflows. The recommended production path is the native
systemd installer documented in the README.
## Local Development
```
python3 -m venv .venv
. .venv/bin/activate
python3 -m pip install -e .
mkdir -p var
python3 manage.py migrate
python3 manage.py createsuperuser
python3 manage.py runserver
```
The admin is available at:
- http://127.0.0.1:8000/
- http://127.0.0.1:8000/admin/
Staff-only JSON endpoints are available at:
- http://127.0.0.1:8000/api/
- http://127.0.0.1:8000/api/status/
## Running Tests
The project test suite is currently run through the Docker image so the runtime dependencies match deployment:
```
docker compose build web scheduler worker
docker compose run --rm web python manage.py test pobsync_backend --verbosity 2
```
## SQL-First Commands
The Django UI is the normal operating surface, but the `pobsync` entrypoint remains useful for setup and inspection.
Create global config:
```
pobsync configure-global --backup-root /mnt/backups/pobsync
```
Create a host config:
```
pobsync configure-host <host> --address <host-or-ip>
```
Run a backup:
```
pobsync backup <host> --prune
```
Create or update a schedule:
```
pobsync schedule <host> --cron "15 2 * * *" --prune
```
Run the scheduler:
```
pobsync scheduler --loop --interval 60
```
Plan or apply retention manually:
```
pobsync retention <host>
pobsync retention <host> --apply --yes --max-delete 10
```
Discover snapshots already present on disk:
```
pobsync discover-snapshots --host <host>
```
The `pobsync` executable is a thin wrapper around Django management commands. Direct Django access is also available:
```
pobsync django check
python3 manage.py run_pobsync_backup <host> --prune
```
## Migration Helpers
Import existing legacy YAML configs:
```
python3 manage.py import_pobsync_configs --prefix /opt/pobsync
```
Export SQL config to legacy runtime YAML for inspection or one-off compatibility:
```
python3 manage.py export_pobsync_configs --prefix /opt/pobsync
```
These commands are migration helpers, not the normal operating model.
## Docker With SQLite
Docker Compose is useful for local development and disposable test installs. Native systemd is preferred for production
backup servers.
```
docker compose up --build web
```
This starts Django on:
- http://127.0.0.1:8010/
- http://127.0.0.1:8010/admin/
- http://127.0.0.1:8010/api/
- http://127.0.0.1:8010/api/status/
Run the scheduler alongside the web admin:
```
docker compose up --build web scheduler worker
```
The web service runs Django through Gunicorn and serves static files with WhiteNoise. The container persists
`/opt/pobsync` and the SQLite database in Docker volumes.
Backup data is always available at `/backups` inside the containers. By default this uses `./backups` on the host.
Override the host-side mount with `POBSYNC_BACKUP_ROOT`:
```
POBSYNC_BACKUP_ROOT=/mnt/backups/pobsync docker compose up --build web scheduler worker
```
## Docker With MariaDB
```
docker compose --profile mariadb up --build web-mariadb
```
With the scheduler:
```
docker compose --profile mariadb up --build web-mariadb scheduler-mariadb worker-mariadb
```
SQLite remains the default because it is enough for a single backup server and keeps deployment simple.
For native systemd installs with MariaDB client support, run the installer with:
```
sudo scripts/install-systemd --install-extras mariadb
```
## Current Architecture
The public command surface is Django-first. The old YAML/cron CLI has been retired from the `pobsync` entrypoint.
Discovered snapshots are stored in `SnapshotRecord`, including the base snapshot metadata and a nullable SQL link to the
base record when it is known.
The Django retention command plans from `SnapshotRecord` instead of rediscovering snapshots from the filesystem.
Post-backup pruning from Django also uses the SQL retention service after the completed snapshot is recorded.
Staff-only JSON endpoints expose service status, hosts, snapshots, and backup runs for lightweight inspection.
Staff-only dashboard views expose the same operational state through Django templates.
Host pages include a safe snapshot discovery action that records existing snapshots into SQL.
Host pages also include a read-only SQL retention plan view before any destructive pruning action.
Schedules can be created or updated from host pages using the same SQL-backed scheduler model.
Host config can be edited from host pages while keeping host identity stable.
The remaining internal engine code still contains reusable backup primitives:
- snapshot naming and metadata
- rsync command construction and execution
- retention planning and pruning
- host locking
Next refactor targets:
- Move more snapshot lifecycle details into typed domain objects.
- Replace remaining dictionary-shaped config at engine boundaries.
- Remove legacy YAML import/export once production migration no longer needs it.

52
scripts/install-systemd
View File

@@ -14,6 +14,7 @@ CSRF_TRUSTED_ORIGINS=${POBSYNC_CSRF_TRUSTED_ORIGINS:-}
BACKUP_ROOT=${POBSYNC_BACKUP_ROOT:-/backups} BACKUP_ROOT=${POBSYNC_BACKUP_ROOT:-/backups}
WEB_BIND=${POBSYNC_WEB_BIND:-127.0.0.1:8010} WEB_BIND=${POBSYNC_WEB_BIND:-127.0.0.1:8010}
FORCE_ENV=0 FORCE_ENV=0
INSTALL_OS_PACKAGES=1
WITH_NGINX=0 WITH_NGINX=0
while [ "$#" -gt 0 ]; do while [ "$#" -gt 0 ]; do
@@ -62,6 +63,14 @@ while [ "$#" -gt 0 ]; do
FORCE_ENV=1 FORCE_ENV=1
shift shift
;; ;;
--no-install-os-packages)
INSTALL_OS_PACKAGES=0
shift
;;
--install-extras)
INSTALL_EXTRAS=$2
shift 2
;;
--with-nginx) --with-nginx)
WITH_NGINX=1 WITH_NGINX=1
shift shift
@@ -82,6 +91,29 @@ if [ "$(id -u)" -ne 0 ]; then
exit 1 exit 1
fi fi
install_os_packages() {
if [ "$INSTALL_OS_PACKAGES" -ne 1 ]; then
return
fi
if command -v apt-get >/dev/null 2>&1; then
packages="python3 python3-venv python3-pip rsync openssh-client"
if [ "$WITH_NGINX" -eq 1 ]; then
packages="$packages nginx"
fi
if [ "$INSTALL_EXTRAS" = "mariadb" ] || [ "$INSTALL_EXTRAS" = "[mariadb]" ] || [ "$INSTALL_EXTRAS" = ".[mariadb]" ]; then
packages="$packages default-libmysqlclient-dev build-essential pkg-config"
fi
apt-get update
apt-get install -y --no-install-recommends $packages
return
fi
echo "No supported package manager found; install python3, python3-venv, rsync, and openssh-client manually." >&2
}
install_os_packages
if ! command -v python3 >/dev/null 2>&1; then if ! command -v python3 >/dev/null 2>&1; then
echo "python3 is required." >&2 echo "python3 is required." >&2
exit 1 exit 1
@@ -127,7 +159,25 @@ fi
python3 -m venv "$VENV_DIR" python3 -m venv "$VENV_DIR"
"$VENV_DIR/bin/python" -m pip install --upgrade pip "$VENV_DIR/bin/python" -m pip install --upgrade pip
"$VENV_DIR/bin/python" -m pip install -e "$APP_DIR$INSTALL_EXTRAS" case "$INSTALL_EXTRAS" in
"")
pip_target=$APP_DIR
;;
mariadb)
pip_target="$APP_DIR[mariadb]"
;;
\[*\])
pip_target="$APP_DIR$INSTALL_EXTRAS"
;;
.\[*\])
pip_target="$APP_DIR${INSTALL_EXTRAS#.}"
;;
*)
echo "Unsupported install extras: $INSTALL_EXTRAS" >&2
exit 2
;;
esac
"$VENV_DIR/bin/python" -m pip install -e "$pip_target"
if [ ! -f "$ENV_FILE" ] || [ "$FORCE_ENV" -eq 1 ]; then if [ ! -f "$ENV_FILE" ] || [ "$FORCE_ENV" -eq 1 ]; then
secret=$("$VENV_DIR/bin/python" -c "import secrets; print(secrets.token_urlsafe(48))") secret=$("$VENV_DIR/bin/python" -c "import secrets; print(secrets.token_urlsafe(48))")