Peter van Arkel
100215bf11
Allow retention planning and pruning to use the same ConfigSource abstraction as scheduled backups. This removes the remaining SQL-to-YAML export dependency from Django backup runs with pruning, keeping YAML only as a legacy CLI compatibility path.
204 lines
5.0 KiB
Markdown
204 lines
5.0 KiB
Markdown
# pobsync
|
|
|
|
`pobsync` is a pull-based backup tool that runs on a central backup server and pulls data from remote servers via rsync over SSH.
|
|
|
|
Key points:
|
|
|
|
- All backup data lives on the backup server.
|
|
- Snapshots are rsync-based and use hardlinking (--link-dest) for space efficiency.
|
|
- Designed for scheduled runs (cron) and manual runs.
|
|
- Minimal external dependencies (currently only PyYAML).
|
|
|
|
## Requirements
|
|
|
|
On the backup server:
|
|
|
|
- Python 3
|
|
- rsync
|
|
- ssh
|
|
- SSH key-based access from the backup server to remotes
|
|
|
|
## Canonical installation (no venv, repo used only for deployment)
|
|
|
|
This project uses a simple and explicit deployment model:
|
|
|
|
- The git clone is only used as a deployment input (and later for updates).
|
|
- Runtime code is deployed into /opt/pobsync/lib.
|
|
- The canonical entrypoint is /opt/pobsync/bin/pobsync.
|
|
|
|
### Install
|
|
|
|
```git clone https://code.hosting.hippogrief.nl/hippogrief/pobsync.git
|
|
cd pobsync
|
|
sudo ./scripts/deploy --prefix /opt/pobsync
|
|
|
|
pobsync install --backup-root /mnt/backups/pobsync (install default configurations)
|
|
pobsync doctor (check if the installation was done correctly)
|
|
```
|
|
|
|
### Update
|
|
|
|
```
|
|
cd /path/to/pobsync
|
|
git pull
|
|
|
|
sudo ./scripts/deploy --prefix /opt/pobsync
|
|
sudo /opt/pobsync/bin/pobsync doctor
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Global configuration is stored at:
|
|
|
|
- /opt/pobsync/config/global.yaml
|
|
|
|
Per-host configuration files are stored at:
|
|
|
|
- /opt/pobsync/config/hosts/<host>.yaml
|
|
|
|
## Some useful commands to get you started
|
|
|
|
Create a new host configuration:
|
|
|
|
`pobsync init-host <host>`
|
|
|
|
List configured remotes:
|
|
|
|
`pobsync list-remotes`
|
|
|
|
Inspect the effective configuration for a host:
|
|
|
|
`pobsync show-config <host>`
|
|
|
|
## Running backups
|
|
|
|
Run a scheduled backup for a host:
|
|
|
|
`pobsync run-scheduled <host>`
|
|
|
|
Optionally apply retention pruning after the run:
|
|
|
|
`pobsync run-scheduled <host> --prune`
|
|
|
|
## Scheduling (cron)
|
|
|
|
Create a cron schedule (writes into /etc/cron.d/pobsync by default):
|
|
|
|
`pobsync schedule create <host> --daily 02:15 --prune`
|
|
|
|
List existing schedules:
|
|
|
|
`pobsync schedule list`
|
|
|
|
Remove a schedule:
|
|
|
|
`pobsync schedule remove <host>`
|
|
|
|
Cron output is redirected to:
|
|
|
|
- /var/log/pobsync/<host>.cron.log
|
|
|
|
## Development (optional)
|
|
|
|
For development purposes you can still use an editable install, this is why pyproject.toml still exists. On systems with an externally managed Python installation, create a virtualenv first.
|
|
|
|
```
|
|
python3 -m venv .venv
|
|
. .venv/bin/activate
|
|
python3 -m pip install -e .
|
|
pobsync --help
|
|
```
|
|
|
|
For production use, always use the canonical entrypoint:
|
|
|
|
/opt/pobsync/bin/pobsync
|
|
|
|
## Django backend (early refactor layer)
|
|
|
|
The Django backend is a management layer around the existing pobsync engine. The current CLI remains the source of truth for executing backups; Django stores configs, schedules, backup runs, and snapshot metadata so the project can grow toward a web/admin/API surface without rewriting rsync behavior in one risky step.
|
|
|
|
### Local SQLite development
|
|
|
|
```
|
|
python3 -m venv .venv
|
|
. .venv/bin/activate
|
|
python3 -m pip install -e .
|
|
mkdir -p var
|
|
python3 manage.py migrate
|
|
python3 manage.py createsuperuser
|
|
python3 manage.py runserver
|
|
```
|
|
|
|
The admin is available at:
|
|
|
|
- http://127.0.0.1:8000/admin/
|
|
|
|
Import existing YAML configs into the database:
|
|
|
|
```
|
|
python3 manage.py import_pobsync_configs --prefix /opt/pobsync
|
|
```
|
|
|
|
Run a backup through Django while still using the existing pobsync engine:
|
|
|
|
```
|
|
python3 manage.py run_pobsync_backup <host> --prefix /opt/pobsync --prune
|
|
```
|
|
|
|
The Django backup command reads backup and retention config from SQL directly. Runtime YAML export is kept as a compatibility tool for older CLI flows during the transition.
|
|
|
|
Export database configs to the runtime YAML files consumed by the current engine:
|
|
|
|
```
|
|
python3 manage.py export_pobsync_configs --prefix /opt/pobsync
|
|
```
|
|
|
|
Run due schedules from the database:
|
|
|
|
```
|
|
python3 manage.py run_pobsync_scheduler --loop --interval 60
|
|
```
|
|
|
|
### Docker with SQLite
|
|
|
|
```
|
|
docker compose up --build web
|
|
```
|
|
|
|
This starts Django on:
|
|
|
|
- http://127.0.0.1:8000/admin/
|
|
|
|
The container persists `/opt/pobsync` and the SQLite database in Docker volumes.
|
|
|
|
Run the Django scheduler alongside the web admin:
|
|
|
|
```
|
|
docker compose up --build web scheduler
|
|
```
|
|
|
|
### Docker with MariaDB
|
|
|
|
```
|
|
docker compose --profile mariadb up --build web-mariadb
|
|
```
|
|
|
|
With the scheduler:
|
|
|
|
```
|
|
docker compose --profile mariadb up --build web-mariadb scheduler-mariadb
|
|
```
|
|
|
|
The MariaDB profile is optional. SQLite remains the default because it is enough for a single backup server and keeps deployment simple.
|
|
|
|
### Refactor direction
|
|
|
|
Recommended next steps:
|
|
|
|
- Continue moving config reading/writing behind repository interfaces so YAML export can eventually disappear.
|
|
- Record more engine-side run details into `BackupRun` and `SnapshotRecord`.
|
|
- Treat SQL as the source of truth and export YAML only as a compatibility layer for the current engine.
|
|
- Run schedules from Django/Docker instead of writing host cron files.
|
|
- Add a snapshot discovery command that syncs existing snapshot metadata into `SnapshotRecord`.
|
|
- Add tests around retention, scheduling, and config merge before deeper internal reshaping.
|