diff --git a/README.md b/README.md index 600a153..5b2fd13 100644 --- a/README.md +++ b/README.md @@ -1,134 +1,30 @@ # pobsync -`pobsync` is a pull-based backup service. It runs on a central backup server and pulls data from remote machines via rsync over SSH. +`pobsync` is a pull-based backup service. It runs on a central backup server and pulls data from remote machines via +rsync over SSH. -The refactor direction is SQL-first: +The current refactor is SQL-first: - Django is the management layer and source of truth. - SQLite is the default database; MariaDB is optional. -- Backups still use the existing rsync snapshot engine internally. +- Backups use the existing rsync snapshot engine internally. - Scheduling is handled by a Django scheduler service, not host cron. -- Legacy YAML import/export exists only for migration and inspection. +- SSH keys can be managed from Django and selected globally or per host. -## Requirements +## Recommended Production Install -On the backup server or in the container: - -- Python 3.11+ -- rsync -- ssh -- SSH key-based access from the backup server to remotes -- systemd for the recommended production deployment - -## Local Development - -``` -python3 -m venv .venv -. .venv/bin/activate -python3 -m pip install -e . -mkdir -p var -python3 manage.py migrate -python3 manage.py createsuperuser -python3 manage.py runserver -``` - -The admin is available at: - -- http://127.0.0.1:8000/ -- http://127.0.0.1:8000/admin/ - -Staff-only JSON endpoints are available at: - -- http://127.0.0.1:8000/api/ -- http://127.0.0.1:8000/api/status/ - -## SQL-First Setup - -Create global config: - -``` -pobsync configure-global --backup-root /mnt/backups/pobsync -``` - -Create a host config: - -``` -pobsync configure-host --address -``` - -Run a backup: - -``` -pobsync backup --prune -``` - -Create or update a schedule: - -``` -pobsync schedule --cron "15 2 * * *" --prune -``` - -Run the scheduler: - -``` -pobsync scheduler --loop --interval 60 -``` - -Plan or apply retention manually: - -``` -pobsync retention -pobsync retention --apply --yes --max-delete 10 -``` - -Discover snapshots already present on disk: - -``` -pobsync discover-snapshots --host -``` - -The `pobsync` executable is a thin wrapper around Django management commands. Direct Django access is also available: - -``` -pobsync django check -python3 manage.py run_pobsync_backup --prune -``` - -## Migration Helpers - -Import existing legacy YAML configs: - -``` -python3 manage.py import_pobsync_configs --prefix /opt/pobsync -``` - -Export SQL config to legacy runtime YAML for inspection or one-off compatibility: - -``` -python3 manage.py export_pobsync_configs --prefix /opt/pobsync -``` - -These commands are migration helpers, not the normal operating model. - -## Production With Systemd - -The recommended production deployment is native systemd services on the backup server. This avoids Docker friction around -SSH, filesystems, large backup mounts, and host-level service logs. +The recommended production deployment is native systemd services on the backup server. Docker Compose remains available +for development and disposable test installs, but native systemd avoids Docker friction around SSH, filesystem mounts, +large backup storage, and host-level service logs. Recommended layout: ``` -/opt/pobsync/app # git checkout +/opt/pobsync/app # installed app checkout /opt/pobsync/venv # Python virtualenv /etc/pobsync/pobsync.env # settings and secrets /var/lib/pobsync # SQLite database, state, runtime SSH key files, static files -/backups # backup storage, or set POBSYNC_BACKUP_ROOT to another absolute path -``` - -Install OS packages first: - -``` -apt install python3 python3-venv rsync openssh-client +/backups # backup storage, or set another absolute path ``` From a checked-out copy of this repository, run: @@ -137,43 +33,42 @@ From a checked-out copy of this repository, run: sudo scripts/install-systemd ``` -By default the installer copies the checkout to `/opt/pobsync/app`, creates `/opt/pobsync/venv`, writes -`/etc/pobsync/pobsync.env`, creates `/var/lib/pobsync` and `/backups`, installs dependencies, runs migrations, collects -static files, and starts the services. +The installer will, by default: + +- install required Debian/Ubuntu OS packages with `apt-get` +- copy the checkout to `/opt/pobsync/app` +- create `/opt/pobsync/venv` +- write `/etc/pobsync/pobsync.env` if it does not exist +- create `/var/lib/pobsync`, `/var/log/pobsync`, and the backup root +- install Python dependencies +- run migrations and collect static files +- install and start `pobsync-web`, `pobsync-worker`, and `pobsync-scheduler` Common overrides: ``` sudo scripts/install-systemd \ - --app-dir /opt/pobsync/app \ --backup-root /mnt/backups/pobsync \ --allowed-hosts backup.example.com,localhost,127.0.0.1 \ --csrf-trusted-origins https://backup.example.com ``` -Use `--force-env` when you intentionally want the installer to rewrite an existing `/etc/pobsync/pobsync.env`. +Use `--no-install-os-packages` if you want to manage system packages yourself. Use `--force-env` only when you want the +installer to rewrite an existing `/etc/pobsync/pobsync.env`. -The installer creates or updates: - -- `pobsync-web.service` for Gunicorn on `127.0.0.1:8010` -- `pobsync-worker.service` for queued backup runs -- `pobsync-scheduler.service` for SQL-backed schedules -- `/etc/pobsync/pobsync.env` if it does not exist - -Edit `/etc/pobsync/pobsync.env` before exposing the service: +For MariaDB support, add: ``` -POBSYNC_DJANGO_ALLOWED_HOSTS=backup.example.com,localhost,127.0.0.1 -POBSYNC_DJANGO_CSRF_TRUSTED_ORIGINS=https://backup.example.com -POBSYNC_BACKUP_ROOT=/backups -POBSYNC_WEB_BIND=127.0.0.1:8010 +sudo scripts/install-systemd --install-extras mariadb ``` -Restart after changes: +## Services -``` -sudo systemctl restart pobsync-web pobsync-worker pobsync-scheduler -``` +The installer creates: + +- `pobsync-web.service`: Gunicorn Django control panel on `127.0.0.1:8010` +- `pobsync-worker.service`: queued backup worker +- `pobsync-scheduler.service`: SQL-backed schedule dispatcher Check service state and logs: @@ -182,103 +77,88 @@ systemctl status pobsync-web pobsync-worker pobsync-scheduler journalctl -u pobsync-worker -f ``` -The Django UI also has a staff-only `/self-check/` page that verifies runtime settings, required binaries, writable -paths, database connectivity, global config state, and systemd service state when systemd is available. +Restart after configuration changes: -Update an existing native install: +``` +sudo systemctl restart pobsync-web pobsync-worker pobsync-scheduler +``` + +## Reverse Proxy + +Use an existing reverse proxy by forwarding to: + +``` +http://127.0.0.1:8010 +``` + +To install a starter nginx site file: + +``` +sudo scripts/install-systemd --with-nginx --server-name backup.example.com +``` + +For HTTPS behind a reverse proxy, set: + +``` +POBSYNC_DJANGO_ALLOWED_HOSTS=backup.example.com,localhost,127.0.0.1 +POBSYNC_DJANGO_CSRF_TRUSTED_ORIGINS=https://backup.example.com +``` + +## Django UI + +After install, open the control panel through your reverse proxy or directly at: + +``` +http://127.0.0.1:8010/ +``` + +Create a superuser if needed: + +``` +sudo -u pobsync /opt/pobsync/venv/bin/python /opt/pobsync/app/manage.py createsuperuser +``` + +The UI includes: + +- dashboard and host detail pages +- global and per-host config forms +- schedule editing +- manual backup queueing +- snapshot discovery +- SQL retention planning and apply flow +- Django-managed SSH keys +- `/self-check/` for runtime checks + +## SSH Keys + +SSH keys can be managed from `/ssh-credentials/`. Add a private key, optionally paste `known_hosts` entries, and select +the credential either as the global default or as a per-host override. + +When a backup starts, the worker writes the selected key to: + +``` +$POBSYNC_HOME/state/ssh-credentials//identity +``` + +The key file is written with `0600` permissions and injected into the rsync SSH command with `IdentityFile`. + +## Updates + +From a fresh checkout or the existing app directory: ``` git pull sudo scripts/install-systemd --app-dir /opt/pobsync/app ``` -Use an existing reverse proxy by forwarding to `http://127.0.0.1:8010`. To install a simple nginx site file as a -starting point: +Then check: ``` -sudo scripts/install-systemd --with-nginx --server-name backup.example.com +systemctl status pobsync-web pobsync-worker pobsync-scheduler ``` -## Docker With SQLite +## Development -Docker Compose is still useful for local development and disposable test installs. Native systemd is preferred for -production backup servers. +Development, Docker, migration helper commands, and architecture notes live in: -``` -docker compose up --build web -``` - -This starts Django on: - -- http://127.0.0.1:8010/ -- http://127.0.0.1:8010/admin/ -- http://127.0.0.1:8010/api/ -- http://127.0.0.1:8010/api/status/ - -Run the scheduler alongside the web admin: - -``` -docker compose up --build web scheduler worker -``` - -The web service runs Django through Gunicorn and serves static files with WhiteNoise. The container persists `/opt/pobsync` -and the SQLite database in Docker volumes. -Backup data is always available at `/backups` inside the containers. By default this uses `./backups` on the host. -Override the host-side mount with `POBSYNC_BACKUP_ROOT`: - -``` -POBSYNC_BACKUP_ROOT=/mnt/backups/pobsync docker compose up --build web scheduler worker -``` - -The Django setup UI keeps the backup root fixed at `/backups`; only the Docker mount decides which host directory -that points to. - -## Django-Managed SSH Keys - -SSH keys can be managed from the Django UI at `/ssh-credentials/`. Add a private key there, optionally paste -`known_hosts` entries, and select the credential either as the global default or as a per-host override. - -When a backup starts, the worker writes the selected key to `$POBSYNC_HOME/state/ssh-credentials//identity` -with `0600` permissions and injects `IdentityFile` into the rsync SSH command. If `known_hosts` is configured, the -worker also writes a matching `known_hosts` file and injects `UserKnownHostsFile`. - -## Docker With MariaDB - -``` -docker compose --profile mariadb up --build web-mariadb -``` - -With the scheduler: - -``` -docker compose --profile mariadb up --build web-mariadb scheduler-mariadb worker-mariadb -``` - -SQLite remains the default because it is enough for a single backup server and keeps deployment simple. - -## Current Architecture - -The public command surface is Django-first. The old YAML/cron CLI has been retired from the `pobsync` entrypoint. -Discovered snapshots are stored in `SnapshotRecord`, including the base snapshot metadata and a nullable SQL link to the -base record when it is known. -The Django retention command plans from `SnapshotRecord` instead of rediscovering snapshots from the filesystem. -Post-backup pruning from Django also uses the SQL retention service after the completed snapshot is recorded. -Staff-only JSON endpoints expose service status, hosts, snapshots, and backup runs for lightweight inspection. -Staff-only dashboard views expose the same operational state through Django templates. -Host pages include a safe snapshot discovery action that records existing snapshots into SQL. -Host pages also include a read-only SQL retention plan view before any destructive pruning action. -Schedules can be created or updated from host pages using the same SQL-backed scheduler model. -Host config can be edited from host pages while keeping host identity stable. - -The remaining internal engine code still contains reusable backup primitives: - -- snapshot naming and metadata -- rsync command construction and execution -- retention planning and pruning -- host locking - -Next refactor targets: - -- Move more snapshot lifecycle details into typed domain objects. -- Replace remaining dictionary-shaped config at engine boundaries. -- Remove legacy YAML import/export once production migration no longer needs it. +- [docs/development.md](docs/development.md) diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..b1f4ef6 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,185 @@ +# Development Notes + +This document contains development and optional Docker workflows. The recommended production path is the native +systemd installer documented in the README. + +## Local Development + +``` +python3 -m venv .venv +. .venv/bin/activate +python3 -m pip install -e . +mkdir -p var +python3 manage.py migrate +python3 manage.py createsuperuser +python3 manage.py runserver +``` + +The admin is available at: + +- http://127.0.0.1:8000/ +- http://127.0.0.1:8000/admin/ + +Staff-only JSON endpoints are available at: + +- http://127.0.0.1:8000/api/ +- http://127.0.0.1:8000/api/status/ + +## Running Tests + +The project test suite is currently run through the Docker image so the runtime dependencies match deployment: + +``` +docker compose build web scheduler worker +docker compose run --rm web python manage.py test pobsync_backend --verbosity 2 +``` + +## SQL-First Commands + +The Django UI is the normal operating surface, but the `pobsync` entrypoint remains useful for setup and inspection. + +Create global config: + +``` +pobsync configure-global --backup-root /mnt/backups/pobsync +``` + +Create a host config: + +``` +pobsync configure-host --address +``` + +Run a backup: + +``` +pobsync backup --prune +``` + +Create or update a schedule: + +``` +pobsync schedule --cron "15 2 * * *" --prune +``` + +Run the scheduler: + +``` +pobsync scheduler --loop --interval 60 +``` + +Plan or apply retention manually: + +``` +pobsync retention +pobsync retention --apply --yes --max-delete 10 +``` + +Discover snapshots already present on disk: + +``` +pobsync discover-snapshots --host +``` + +The `pobsync` executable is a thin wrapper around Django management commands. Direct Django access is also available: + +``` +pobsync django check +python3 manage.py run_pobsync_backup --prune +``` + +## Migration Helpers + +Import existing legacy YAML configs: + +``` +python3 manage.py import_pobsync_configs --prefix /opt/pobsync +``` + +Export SQL config to legacy runtime YAML for inspection or one-off compatibility: + +``` +python3 manage.py export_pobsync_configs --prefix /opt/pobsync +``` + +These commands are migration helpers, not the normal operating model. + +## Docker With SQLite + +Docker Compose is useful for local development and disposable test installs. Native systemd is preferred for production +backup servers. + +``` +docker compose up --build web +``` + +This starts Django on: + +- http://127.0.0.1:8010/ +- http://127.0.0.1:8010/admin/ +- http://127.0.0.1:8010/api/ +- http://127.0.0.1:8010/api/status/ + +Run the scheduler alongside the web admin: + +``` +docker compose up --build web scheduler worker +``` + +The web service runs Django through Gunicorn and serves static files with WhiteNoise. The container persists +`/opt/pobsync` and the SQLite database in Docker volumes. + +Backup data is always available at `/backups` inside the containers. By default this uses `./backups` on the host. +Override the host-side mount with `POBSYNC_BACKUP_ROOT`: + +``` +POBSYNC_BACKUP_ROOT=/mnt/backups/pobsync docker compose up --build web scheduler worker +``` + +## Docker With MariaDB + +``` +docker compose --profile mariadb up --build web-mariadb +``` + +With the scheduler: + +``` +docker compose --profile mariadb up --build web-mariadb scheduler-mariadb worker-mariadb +``` + +SQLite remains the default because it is enough for a single backup server and keeps deployment simple. + +For native systemd installs with MariaDB client support, run the installer with: + +``` +sudo scripts/install-systemd --install-extras mariadb +``` + +## Current Architecture + +The public command surface is Django-first. The old YAML/cron CLI has been retired from the `pobsync` entrypoint. +Discovered snapshots are stored in `SnapshotRecord`, including the base snapshot metadata and a nullable SQL link to the +base record when it is known. + +The Django retention command plans from `SnapshotRecord` instead of rediscovering snapshots from the filesystem. +Post-backup pruning from Django also uses the SQL retention service after the completed snapshot is recorded. +Staff-only JSON endpoints expose service status, hosts, snapshots, and backup runs for lightweight inspection. +Staff-only dashboard views expose the same operational state through Django templates. +Host pages include a safe snapshot discovery action that records existing snapshots into SQL. +Host pages also include a read-only SQL retention plan view before any destructive pruning action. +Schedules can be created or updated from host pages using the same SQL-backed scheduler model. +Host config can be edited from host pages while keeping host identity stable. + +The remaining internal engine code still contains reusable backup primitives: + +- snapshot naming and metadata +- rsync command construction and execution +- retention planning and pruning +- host locking + +Next refactor targets: + +- Move more snapshot lifecycle details into typed domain objects. +- Replace remaining dictionary-shaped config at engine boundaries. +- Remove legacy YAML import/export once production migration no longer needs it. diff --git a/scripts/install-systemd b/scripts/install-systemd index da2c0bf..21ca52d 100755 --- a/scripts/install-systemd +++ b/scripts/install-systemd @@ -14,6 +14,7 @@ CSRF_TRUSTED_ORIGINS=${POBSYNC_CSRF_TRUSTED_ORIGINS:-} BACKUP_ROOT=${POBSYNC_BACKUP_ROOT:-/backups} WEB_BIND=${POBSYNC_WEB_BIND:-127.0.0.1:8010} FORCE_ENV=0 +INSTALL_OS_PACKAGES=1 WITH_NGINX=0 while [ "$#" -gt 0 ]; do @@ -62,6 +63,14 @@ while [ "$#" -gt 0 ]; do FORCE_ENV=1 shift ;; + --no-install-os-packages) + INSTALL_OS_PACKAGES=0 + shift + ;; + --install-extras) + INSTALL_EXTRAS=$2 + shift 2 + ;; --with-nginx) WITH_NGINX=1 shift @@ -82,6 +91,29 @@ if [ "$(id -u)" -ne 0 ]; then exit 1 fi +install_os_packages() { + if [ "$INSTALL_OS_PACKAGES" -ne 1 ]; then + return + fi + + if command -v apt-get >/dev/null 2>&1; then + packages="python3 python3-venv python3-pip rsync openssh-client" + if [ "$WITH_NGINX" -eq 1 ]; then + packages="$packages nginx" + fi + if [ "$INSTALL_EXTRAS" = "mariadb" ] || [ "$INSTALL_EXTRAS" = "[mariadb]" ] || [ "$INSTALL_EXTRAS" = ".[mariadb]" ]; then + packages="$packages default-libmysqlclient-dev build-essential pkg-config" + fi + apt-get update + apt-get install -y --no-install-recommends $packages + return + fi + + echo "No supported package manager found; install python3, python3-venv, rsync, and openssh-client manually." >&2 +} + +install_os_packages + if ! command -v python3 >/dev/null 2>&1; then echo "python3 is required." >&2 exit 1 @@ -127,7 +159,25 @@ fi python3 -m venv "$VENV_DIR" "$VENV_DIR/bin/python" -m pip install --upgrade pip -"$VENV_DIR/bin/python" -m pip install -e "$APP_DIR$INSTALL_EXTRAS" +case "$INSTALL_EXTRAS" in + "") + pip_target=$APP_DIR + ;; + mariadb) + pip_target="$APP_DIR[mariadb]" + ;; + \[*\]) + pip_target="$APP_DIR$INSTALL_EXTRAS" + ;; + .\[*\]) + pip_target="$APP_DIR${INSTALL_EXTRAS#.}" + ;; + *) + echo "Unsupported install extras: $INSTALL_EXTRAS" >&2 + exit 2 + ;; +esac +"$VENV_DIR/bin/python" -m pip install -e "$pip_target" if [ ! -f "$ENV_FILE" ] || [ "$FORCE_ENV" -eq 1 ]; then secret=$("$VENV_DIR/bin/python" -c "import secrets; print(secrets.token_urlsafe(48))")