diff --git a/README.md b/README.md index 4743a6b..efe37c6 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ The refactor direction is SQL-first: - Django is the management layer and source of truth. - SQLite is the default database; MariaDB is optional. - Backups still use the existing rsync snapshot engine internally. -- Scheduling is handled by a Django/Docker scheduler process, not host cron. +- Scheduling is handled by a Django scheduler service, not host cron. - Legacy YAML import/export exists only for migration and inspection. ## Requirements @@ -18,6 +18,7 @@ On the backup server or in the container: - rsync - ssh - SSH key-based access from the backup server to remotes +- systemd for the recommended production deployment ## Local Development @@ -109,8 +110,83 @@ python3 manage.py export_pobsync_configs --prefix /opt/pobsync These commands are migration helpers, not the normal operating model. +## Production With Systemd + +The recommended production deployment is native systemd services on the backup server. This avoids Docker friction around +SSH, filesystems, large backup mounts, and host-level service logs. + +Recommended layout: + +``` +/opt/pobsync/app # git checkout +/opt/pobsync/venv # Python virtualenv +/etc/pobsync/pobsync.env # settings and secrets +/var/lib/pobsync # SQLite database, state, runtime SSH key files, static files +/backups # backup storage, or set POBSYNC_BACKUP_ROOT to another absolute path +``` + +Install OS packages first: + +``` +apt install python3 python3-venv rsync openssh-client +``` + +Clone or update the app at `/opt/pobsync/app`, then run: + +``` +cd /opt/pobsync/app +sudo scripts/install-systemd +``` + +The installer creates: + +- `pobsync-web.service` for Gunicorn on `127.0.0.1:8010` +- `pobsync-worker.service` for queued backup runs +- `pobsync-scheduler.service` for SQL-backed schedules +- `/etc/pobsync/pobsync.env` if it does not exist + +Edit `/etc/pobsync/pobsync.env` before exposing the service: + +``` +POBSYNC_DJANGO_ALLOWED_HOSTS=backup.example.com,localhost,127.0.0.1 +POBSYNC_DJANGO_CSRF_TRUSTED_ORIGINS=https://backup.example.com +POBSYNC_BACKUP_ROOT=/backups +POBSYNC_WEB_BIND=127.0.0.1:8010 +``` + +Restart after changes: + +``` +sudo systemctl restart pobsync-web pobsync-worker pobsync-scheduler +``` + +Check service state and logs: + +``` +systemctl status pobsync-web pobsync-worker pobsync-scheduler +journalctl -u pobsync-worker -f +``` + +Update an existing native install: + +``` +cd /opt/pobsync/app +git pull +sudo scripts/install-systemd +``` + +Use an existing reverse proxy by forwarding to `http://127.0.0.1:8010`. To install a simple nginx site file as a +starting point: + +``` +sudo scripts/install-systemd --with-nginx --server-name backup.example.com +``` + ## Docker With SQLite +Docker Compose is still useful for local development and disposable test installs. Native systemd is preferred for +production backup servers. + ``` docker compose up --build web ``` @@ -140,55 +216,14 @@ POBSYNC_BACKUP_ROOT=/mnt/backups/pobsync docker compose up --build web scheduler The Django setup UI keeps the backup root fixed at `/backups`; only the Docker mount decides which host directory that points to. -## Remote Server Deployment - -For a single backup server, use Docker Compose with the SQLite services and put a reverse proxy such as Caddy, nginx, -or Traefik in front of `web`. - -Create a `.env` from the example: - -``` -cp .env.example .env -``` - -Set at least: - -``` -POBSYNC_BACKUP_ROOT=/mnt/backups/pobsync -POBSYNC_DJANGO_ALLOWED_HOSTS=backup.example.com,localhost,127.0.0.1 -POBSYNC_DJANGO_SECRET_KEY= -POBSYNC_DJANGO_DEBUG=0 -POBSYNC_WEB_BIND=127.0.0.1 -``` - -Deploy or update: - -``` -git pull -docker compose build web scheduler worker -docker compose up -d --force-recreate web scheduler worker -docker compose exec web python manage.py migrate -``` - -Check service state: - -``` -docker compose ps -docker compose logs --tail=100 worker -docker compose logs --tail=100 scheduler -``` - -`web`, `scheduler`, and `worker` use `restart: unless-stopped` and Docker healthchecks. If `POBSYNC_WEB_BIND` is -`127.0.0.1`, expose the app through your reverse proxy instead of directly publishing it to the internet. - ## Django-Managed SSH Keys SSH keys can be managed from the Django UI at `/ssh-credentials/`. Add a private key there, optionally paste `known_hosts` entries, and select the credential either as the global default or as a per-host override. -When a backup starts, the worker writes the selected key to `/opt/pobsync/state/ssh-credentials//identity` -inside the container with `0600` permissions and injects `IdentityFile` into the rsync SSH command. If `known_hosts` -is configured, the worker also writes a matching `known_hosts` file and injects `UserKnownHostsFile`. +When a backup starts, the worker writes the selected key to `$POBSYNC_HOME/state/ssh-credentials//identity` +with `0600` permissions and injects `IdentityFile` into the rsync SSH command. If `known_hosts` is configured, the +worker also writes a matching `known_hosts` file and injects `UserKnownHostsFile`. ## Docker With MariaDB diff --git a/deploy/nginx/pobsync.conf b/deploy/nginx/pobsync.conf new file mode 100644 index 0000000..aff93f1 --- /dev/null +++ b/deploy/nginx/pobsync.conf @@ -0,0 +1,15 @@ +server { + listen 80; + server_name @POBSYNC_SERVER_NAME@; + + client_max_body_size 16m; + + location / { + proxy_pass http://127.0.0.1:8010; + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } +} diff --git a/deploy/pobsync.env.example b/deploy/pobsync.env.example new file mode 100644 index 0000000..076b7c0 --- /dev/null +++ b/deploy/pobsync.env.example @@ -0,0 +1,15 @@ +POBSYNC_DJANGO_DEBUG=0 +POBSYNC_DJANGO_SECRET_KEY=change-me-to-a-long-random-secret +POBSYNC_DJANGO_ALLOWED_HOSTS=backup.example.com,localhost,127.0.0.1 +POBSYNC_DJANGO_CSRF_TRUSTED_ORIGINS=https://backup.example.com + +POBSYNC_HOME=/var/lib/pobsync +POBSYNC_BACKUP_ROOT=/backups +POBSYNC_SQLITE_PATH=/var/lib/pobsync/pobsync.sqlite3 +POBSYNC_STATIC_ROOT=/var/lib/pobsync/static + +POBSYNC_WEB_BIND=127.0.0.1:8010 +POBSYNC_GUNICORN_WORKERS=2 +POBSYNC_GUNICORN_TIMEOUT=120 +POBSYNC_WORKER_INTERVAL=15 +POBSYNC_SCHEDULER_INTERVAL=60 diff --git a/deploy/systemd/pobsync-scheduler.service b/deploy/systemd/pobsync-scheduler.service new file mode 100644 index 0000000..a3a5dcf --- /dev/null +++ b/deploy/systemd/pobsync-scheduler.service @@ -0,0 +1,17 @@ +[Unit] +Description=pobsync schedule dispatcher +After=network-online.target pobsync-web.service +Wants=network-online.target + +[Service] +Type=simple +User=@POBSYNC_USER@ +Group=@POBSYNC_GROUP@ +WorkingDirectory=@POBSYNC_APP_DIR@ +EnvironmentFile=@POBSYNC_ENV_FILE@ +ExecStart=/bin/sh -c 'exec @POBSYNC_VENV_DIR@/bin/python manage.py run_pobsync_scheduler --loop --interval "${POBSYNC_SCHEDULER_INTERVAL:-60}"' +Restart=on-failure +RestartSec=5 + +[Install] +WantedBy=multi-user.target diff --git a/deploy/systemd/pobsync-web.service b/deploy/systemd/pobsync-web.service new file mode 100644 index 0000000..3f99b0b --- /dev/null +++ b/deploy/systemd/pobsync-web.service @@ -0,0 +1,19 @@ +[Unit] +Description=pobsync Django control panel +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +User=@POBSYNC_USER@ +Group=@POBSYNC_GROUP@ +WorkingDirectory=@POBSYNC_APP_DIR@ +EnvironmentFile=@POBSYNC_ENV_FILE@ +ExecStartPre=@POBSYNC_VENV_DIR@/bin/python manage.py migrate --noinput +ExecStartPre=@POBSYNC_VENV_DIR@/bin/python manage.py collectstatic --noinput --clear +ExecStart=/bin/sh -c 'exec @POBSYNC_VENV_DIR@/bin/gunicorn pobsync_server.wsgi:application --bind "${POBSYNC_WEB_BIND:-127.0.0.1:8010}" --workers "${POBSYNC_GUNICORN_WORKERS:-2}" --timeout "${POBSYNC_GUNICORN_TIMEOUT:-120}"' +Restart=on-failure +RestartSec=5 + +[Install] +WantedBy=multi-user.target diff --git a/deploy/systemd/pobsync-worker.service b/deploy/systemd/pobsync-worker.service new file mode 100644 index 0000000..a1c4837 --- /dev/null +++ b/deploy/systemd/pobsync-worker.service @@ -0,0 +1,17 @@ +[Unit] +Description=pobsync queued backup worker +After=network-online.target pobsync-web.service +Wants=network-online.target + +[Service] +Type=simple +User=@POBSYNC_USER@ +Group=@POBSYNC_GROUP@ +WorkingDirectory=@POBSYNC_APP_DIR@ +EnvironmentFile=@POBSYNC_ENV_FILE@ +ExecStart=/bin/sh -c 'exec @POBSYNC_VENV_DIR@/bin/python manage.py run_pobsync_worker --loop --interval "${POBSYNC_WORKER_INTERVAL:-15}"' +Restart=on-failure +RestartSec=5 + +[Install] +WantedBy=multi-user.target diff --git a/scripts/install-systemd b/scripts/install-systemd new file mode 100755 index 0000000..be385fe --- /dev/null +++ b/scripts/install-systemd @@ -0,0 +1,123 @@ +#!/bin/sh +set -eu + +APP_DIR=${POBSYNC_APP_DIR:-$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)} +VENV_DIR=${POBSYNC_VENV_DIR:-/opt/pobsync/venv} +ENV_FILE=${POBSYNC_ENV_FILE:-/etc/pobsync/pobsync.env} +SERVICE_USER=${POBSYNC_SERVICE_USER:-pobsync} +SERVICE_GROUP=${POBSYNC_SERVICE_GROUP:-pobsync} +INSTALL_EXTRAS=${POBSYNC_INSTALL_EXTRAS:-} +SERVER_NAME=${POBSYNC_SERVER_NAME:-_} +WITH_NGINX=0 + +while [ "$#" -gt 0 ]; do + case "$1" in + --with-nginx) + WITH_NGINX=1 + shift + ;; + --server-name) + SERVER_NAME=$2 + shift 2 + ;; + *) + echo "Unknown argument: $1" >&2 + exit 2 + ;; + esac +done + +if [ "$(id -u)" -ne 0 ]; then + echo "Run this installer as root." >&2 + exit 1 +fi + +if ! command -v python3 >/dev/null 2>&1; then + echo "python3 is required." >&2 + exit 1 +fi + +if ! command -v rsync >/dev/null 2>&1; then + echo "rsync is required." >&2 + exit 1 +fi + +if ! command -v ssh >/dev/null 2>&1; then + echo "openssh-client is required." >&2 + exit 1 +fi + +if ! getent group "$SERVICE_GROUP" >/dev/null 2>&1; then + groupadd --system "$SERVICE_GROUP" +fi + +if ! id "$SERVICE_USER" >/dev/null 2>&1; then + useradd --system --home /var/lib/pobsync --shell /usr/sbin/nologin --gid "$SERVICE_GROUP" "$SERVICE_USER" +fi + +mkdir -p /etc/pobsync /var/lib/pobsync /var/log/pobsync "$(dirname "$VENV_DIR")" +chown "$SERVICE_USER:$SERVICE_GROUP" /var/lib/pobsync /var/log/pobsync +chmod 0750 /var/lib/pobsync /var/log/pobsync + +python3 -m venv "$VENV_DIR" +"$VENV_DIR/bin/python" -m pip install --upgrade pip +"$VENV_DIR/bin/python" -m pip install -e "$APP_DIR$INSTALL_EXTRAS" + +if [ ! -f "$ENV_FILE" ]; then + secret=$("$VENV_DIR/bin/python" -c "import secrets; print(secrets.token_urlsafe(48))") + cat > "$ENV_FILE" < "$dest" + chmod 0644 "$dest" +} + +install_unit "$APP_DIR/deploy/systemd/pobsync-web.service" /etc/systemd/system/pobsync-web.service +install_unit "$APP_DIR/deploy/systemd/pobsync-worker.service" /etc/systemd/system/pobsync-worker.service +install_unit "$APP_DIR/deploy/systemd/pobsync-scheduler.service" /etc/systemd/system/pobsync-scheduler.service + +systemctl daemon-reload +"$VENV_DIR/bin/python" "$APP_DIR/manage.py" migrate --noinput +"$VENV_DIR/bin/python" "$APP_DIR/manage.py" collectstatic --noinput --clear +systemctl enable --now pobsync-web.service pobsync-worker.service pobsync-scheduler.service + +if [ "$WITH_NGINX" -eq 1 ]; then + if ! command -v nginx >/dev/null 2>&1; then + echo "nginx is not installed; skipping nginx config." >&2 + else + sed "s|@POBSYNC_SERVER_NAME@|$SERVER_NAME|g" "$APP_DIR/deploy/nginx/pobsync.conf" > /etc/nginx/sites-available/pobsync.conf + ln -sf /etc/nginx/sites-available/pobsync.conf /etc/nginx/sites-enabled/pobsync.conf + nginx -t + systemctl reload nginx + fi +fi + +systemctl --no-pager --full status pobsync-web.service pobsync-worker.service pobsync-scheduler.service || true diff --git a/src/pobsync_server/settings.py b/src/pobsync_server/settings.py index b254510..24d49d4 100644 --- a/src/pobsync_server/settings.py +++ b/src/pobsync_server/settings.py @@ -12,6 +12,9 @@ DEBUG = os.getenv("POBSYNC_DJANGO_DEBUG", "0").lower() in {"1", "true", "yes", " _allowed_hosts = os.getenv("POBSYNC_DJANGO_ALLOWED_HOSTS", "localhost,127.0.0.1") ALLOWED_HOSTS = [host.strip() for host in _allowed_hosts.split(",") if host.strip()] +_csrf_trusted_origins = os.getenv("POBSYNC_DJANGO_CSRF_TRUSTED_ORIGINS", "") +CSRF_TRUSTED_ORIGINS = [origin.strip() for origin in _csrf_trusted_origins.split(",") if origin.strip()] + INSTALLED_APPS = [ "django.contrib.admin", "django.contrib.auth", @@ -95,4 +98,4 @@ STORAGES = { DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField" POBSYNC_HOME = os.getenv("POBSYNC_HOME", "/opt/pobsync") -POBSYNC_BACKUP_ROOT = "/backups" +POBSYNC_BACKUP_ROOT = os.getenv("POBSYNC_BACKUP_ROOT", "/backups")