62 Commits

Author SHA1 Message Date
51142081c9 Merge pull request '(release) Prepare 1.2.0' (#74) from release-1.2 into master
Reviewed-on: #74
2026-05-28 22:19:23 +02:00
02616eebbc (release) Prepare 1.2.0
Bump the package version to 1.2.0 and document the operations-focused release
with updater, readonly access, notifications, bandwidth controls, live progress,
and more robust retention cleanup.

Make the CLI version test assert against the package version so future release
bumps do not require changing a hardcoded expected value.
2026-05-28 22:19:03 +02:00
a61e3d8302 Merge pull request '(feature) Add staff updater page' (#73) from issue-39-updater-ui into master
Reviewed-on: #73
2026-05-28 22:11:07 +02:00
0450f8bdb0 (feature) Add staff updater page
Add a Django updater view for checking configured Gitea releases, inspecting
the installed git checkout, fetching tags, pulling the current branch, and
running the configured native systemd update command.

Document the updater environment settings and keep the page staff-only so
readonly status users cannot trigger deployment actions.
2026-05-28 22:10:45 +02:00
b4833560b5 Merge pull request '(feature) Add read-only access level to control panel' (#72) from issue-40-access-levels into master
Reviewed-on: #72
2026-05-28 22:00:45 +02:00
81ee848f5f (feature) Add read-only access level to control panel
Introduce a central access policy that lets authenticated non-staff users view
backup status pages while keeping credentials, logs, configs, and mutating
actions staff-only.

Hide sensitive navigation and host controls for read-only users, expose only
the status API to authenticated viewers, and document the two access levels.
2026-05-28 22:00:16 +02:00
7f2bbe4d20 Merge pull request '(bugfix) Enable rsync progress output for live real runs' (#71) from issue-64-rsync-progress-live-runs into master
Reviewed-on: #71
2026-05-28 21:43:02 +02:00
29f455a153 (bugfix) Enable rsync progress output for live real runs
Default queued and management-command backups to verbose rsync output so live
run views show progress for long-running real backups, matching dry-run
visibility.

Add a quiet-rsync escape hatch for operators who intentionally want less noisy
real-run logs.
2026-05-28 21:42:40 +02:00
41ceab5a40 Merge pull request '(bugfix) Measure incomplete snapshot data from disk' (#70) from issue-69-data-overview-incomplete-sizes into master
Reviewed-on: #70
2026-05-28 21:34:22 +02:00
2ad119e214 (bugfix) Measure incomplete snapshot data from disk
Use filesystem usage for incomplete snapshots instead of trusting potentially
stale metadata, and expose unique non-hardlinked data totals for completed
snapshots.

Update dashboard and host storage summaries so incomplete data is visible and
complete snapshot totals distinguish allocated and unique data.
2026-05-28 21:33:26 +02:00
eb121453c8 Merge pull request '(feature) Add run completion notifications' (#68) from issue-50-run-notifications into master
Reviewed-on: #68
2026-05-28 21:20:59 +02:00
67ffd6101b (feature) Add run completion notifications
Add email and webhook notification targets with delivery tracking, and send
notifications when backup runs reach a terminal status.

Expose notification target management in the Django UI and keep delivery
failures recorded without failing the backup worker.
2026-05-28 21:20:38 +02:00
1f5c4e0756 Merge pull request '(feature) Require review before incomplete cleanup' (#67) from issue-47-reviewed-incomplete-cleanup into master
Reviewed-on: #67
2026-05-28 21:08:21 +02:00
b5e87abad2 (feature) Require review before incomplete cleanup
Require incomplete snapshots to be marked reviewed before the cleanup action
can delete them, and show review state in the retention plan UI.

Keep cleanup confirmation counts scoped to reviewed incomplete snapshots and
add coverage for blocked, reviewed, and deletion flows.
2026-05-28 21:07:59 +02:00
fc6df89370 Merge pull request '(bugfix) Make snapshot pruning robust to archived permissions' (#66) from issue-47-65-robust-prune-cleanup into master
Reviewed-on: #66
2026-05-28 20:59:23 +02:00
3893df4640 (bugfix) Make snapshot pruning robust to archived permissions
Repair user permissions inside snapshot trees before deleting them so
retention prune and incomplete cleanup can remove directories copied with
restrictive rsync archive modes.

Add path validation for scheduled/manual snapshot deletes and cover
non-traversable nested directories in retention tests.
2026-05-28 20:58:37 +02:00
f86c67aeee Merge pull request '## Summary' (#63) from bugfix/62-incomplete-backup-data-totals into master
Reviewed-on: #63
2026-05-23 01:36:18 +02:00
7dc4c1df84 ## Summary
- Fall back to filesystem measurement when snapshot storage metadata is missing.
- Prefer `data/` inside snapshot directories so incomplete snapshot metadata/log files are not counted as backup data.
- Add stats and dashboard rendering coverage for incomplete snapshots without recorded storage metadata.

## Tests
- .venv/bin/python manage.py test src.pobsync_backend.tests.test_stats_summary src.pobsync_backend.tests.test_views.ViewTests.test_dashboard_host_cards_measure_incomplete_data_without_snapshot_metadata --verbosity 2
- .venv/bin/python manage.py check
- git diff --check
- .venv/bin/python manage.py test src.pobsync_backend --verbosity 2

Closes #62
2026-05-23 01:35:38 +02:00
10e0293559 Merge pull request '(feature) Show backup data totals by snapshot kind' (#61) from issue-53-host-backup-data-totals into master
Reviewed-on: #61
2026-05-23 01:28:18 +02:00
9dd690bb3b (feature) Show backup data totals by snapshot kind
Aggregate snapshot storage metadata by snapshot kind so operators can see
scheduled, manual, incomplete, and total backup data.

Surface the totals per host and across all hosts on the dashboard, using
allocated snapshot size from recorded backup metadata without rescanning
backup storage.
2026-05-23 01:27:51 +02:00
8740b75841 Merge pull request '## Summary' (#60) from issue-48-49-hosts-page-controls into master
Reviewed-on: #60
2026-05-23 01:14:48 +02:00
ce1cb9d157 ## Summary
- Add a dedicated `/hosts/` page with host cards and enabled/disabled filtering.
- Link the dashboard Hosts metric and top navigation to the new page.
- Add host enable/disable plus schedule and scheduled-retention pause/resume actions.

## Tests
- `.venv/bin/python manage.py test src.pobsync_backend.tests.test_views.ViewTests.test_base_navigation_groups_primary_and_system_links src.pobsync_backend.tests.test_views.ViewTests.test_dashboard_renders_hosts_and_latest_runs src.pobsync_backend.tests.test_views.ViewTests.test_dashboard_hosts_live_returns_hosts_partial src.pobsync_backend.tests.test_views.ViewTests.test_hosts_list_renders_host_cards_and_controls src.pobsync_backend.tests.test_views.ViewTests.test_hosts_list_filters_by_enabled_state src.pobsync_backend.tests.test_views.ViewTests.test_update_host_state_toggles_host_schedule_and_retention --verbosity 2`
- `.venv/bin/python manage.py check`
- `.venv/bin/python manage.py test src.pobsync_backend --verbosity 2`

Closes #48
Closes #49
2026-05-23 01:13:32 +02:00
8e83fee7b5 Merge pull request '## Summary' (#59) from issue-55-writable-test-state into master
Reviewed-on: #59
2026-05-23 01:07:26 +02:00
a6d6468da8 ## Summary
- Override `POBSYNC_HOME` in the filesystem SSH credential test.
- Keep credential materialization tests self-contained and writable in local development.
- Leave production runtime defaults unchanged.

## Tests
- `.venv/bin/python manage.py test src.pobsync_backend.tests.test_django_config_source --verbosity 2`
- `.venv/bin/python manage.py test src.pobsync_backend --verbosity 2`

Closes #55
2026-05-23 01:06:22 +02:00
b87203c538 Merge pull request '## Summary' (#58) from issue-51-bandwidth-limit into master
Reviewed-on: #58
2026-05-23 01:02:28 +02:00
515330c436 ## Summary
- Add per-host rsync bandwidth limit overrides with inherit/unlimited semantics.
- Store the effective bwlimit in run metadata/results and show it in host/run detail views.
- Document recommended starting values for VPN and remote backups.

## Tests
- `.venv/bin/python manage.py makemigrations --check --dry-run`
- `.venv/bin/python manage.py test src.pobsync_backend.tests.test_django_config_source.DjangoConfigSourceTests.test_returns_effective_config_from_database src.pobsync_backend.tests.test_django_config_source.DjangoConfigSourceTests.test_host_can_disable_global_rsync_bandwidth_limit src.pobsync_backend.tests.test_configure_commands.ConfigureCommandsTests.test_configure_host_uses_global_retention_defaults src.pobsync_backend.tests.test_run_scheduled_config_source.RunScheduledConfigSourceTests.test_dry_run_applies_configured_bandwidth_limit src.pobsync_backend.tests.test_run_scheduled_config_source.RunScheduledConfigSourceTests.test_real_run_can_request_verbose_output_args --verbosity 2`
- `.venv/bin/python manage.py test src.pobsync_backend.tests.test_views.ViewTests.test_create_host_config_form_creates_host src.pobsync_backend.tests.test_views.ViewTests.test_host_detail_renders_effective_config_preview src.pobsync_backend.tests.test_views.ViewTests.test_run_detail_renders_result_payload src.pobsync_backend.tests.test_views.ViewTests.test_host_config_form_updates_host_config --verbosity 2`
- `.venv/bin/python manage.py check`

Closes #51
2026-05-23 00:59:55 +02:00
fdf401a0be Merge pull request '(refactor) Unify run progress panels' (#57) from issue-52-live-normal-runs into master
Reviewed-on: #57
2026-05-23 00:48:22 +02:00
3b77f2e5d0 (refactor) Unify run progress panels
Use a shared Run Progress presentation for dry-runs and normal backup
runs so live run feedback is consistent across run types.

Keep mode-specific metrics while aligning status, mode, log, and warning
layout.

Refs #52
2026-05-23 00:46:52 +02:00
df9ec5b04c Merge pull request 'issue-54-worker-rsync-state' (#56) from issue-54-worker-rsync-state into master
Reviewed-on: #56
2026-05-23 00:39:47 +02:00
5788f53854 (bugfix) Keep rsync runner callback optional
Only pass the process_started hook when live run state tracking is active,
so existing rsync call sites and tests without that hook remain compatible.

Refs #54
2026-05-23 00:31:24 +02:00
28da9c4096 (bugfix) Track rsync process state for running backups
Record rsync process pid and execution phase while normal backup runs are
active so the worker can reconcile stale running rows when rsync has
already disappeared.

Keep finalizing runs out of the missing-process path to avoid marking
slow post-rsync stats collection as a failed transfer.

Closes #54
2026-05-23 00:26:22 +02:00
6eb1b4add3 (bugfix) Reconcile real rsync failures from worker logs
Record live rsync log paths for normal backup runs so the worker can
recover stale running state after terminal rsync errors.

Treat rsync vanished-file exit code 24 as a warning and keep the
completed snapshot instead of failing the run into incomplete state.

Closes #54
2026-05-23 00:23:14 +02:00
8633cbea26 Merge pull request '(bugfix) Quote remote preflight shell commands' (#46) from issue-45-preflight-shell-quoting into master
Reviewed-on: #46
2026-05-21 15:45:46 +02:00
3fb8209aef (bugfix) Quote remote preflight shell commands
Pass remote rsync and source-root preflight checks as a single quoted
shell command to SSH so the remote shell evaluates command -v and test
expressions reliably.

Refs #45
2026-05-21 15:44:46 +02:00
833edb2466 Merge pull request '(release) Prepare pobsync 1.1.0' (#44) from release/1.1.0 into master
Reviewed-on: #44
2026-05-21 15:24:55 +02:00
c7e9e69345 (release) Prepare pobsync 1.1.0
Bump the package and runtime version to 1.1.0, add release notes for
the UI-focused control panel update, and refresh version assertions for
the CLI entrypoint.

Refs 1.1
2026-05-21 15:23:50 +02:00
e79d871f36 Merge pull request 'issue-36-lightweight-live-refresh' (#43) from issue-36-lightweight-live-refresh into master
Reviewed-on: #43
2026-05-21 15:18:46 +02:00
ad45fbe46e (feature) Add live refresh for dashboard status panels
Split dashboard priority and host status sections into server-rendered
partials and wire them into the shared refresh hook so operational state
updates without a full page reload.

Refs #36
2026-05-21 15:17:11 +02:00
3cac7b61ac (feature) Add live refresh for run detail status
Add a server-rendered run detail partial and a small vanilla JavaScript
refresh hook so active backup runs update status, controls, timing, and
rsync log output without a full page reload.

Document the Django-template-first refresh pattern for future control
panel work.

Refs #36
2026-05-21 15:10:37 +02:00
1d6c21764b Merge pull request 'issue-37-clean-primary-navigation' (#42) from issue-37-clean-primary-navigation into master
Reviewed-on: #42
2026-05-21 15:02:02 +02:00
6f392bef65 (ui) Highlight current navigation section
Mark the active primary or system navigation link with aria-current and a
subtle visual state so staff users can see where they are in the control
panel without making Admin a primary app route.

Refs #37
2026-05-21 15:01:14 +02:00
6035c547ae (ui) Split primary and system navigation
Move operator routes into a primary navigation group and demote admin,
self-check, changelog, and status API links into a secondary system group
so the header reflects the control panel workflow more clearly.

Refs #37
2026-05-21 14:56:30 +02:00
a3a8fea071 Merge pull request 'issue-38-dashboard-responsive-layout' (#41) from issue-38-dashboard-responsive-layout into master
Reviewed-on: #41
2026-05-21 14:51:34 +02:00
0e2f48ab65 (ui) Remove dashboard panel overflow traps
Let dashboard-specific panels size naturally instead of inheriting generic
panel overflow behavior, and tighten activity and host card sizing so long
content wraps without creating internal scrollbars.

Refs #38
2026-05-21 14:50:05 +02:00
b55950e24a (ui) Add dashboard section responsive hooks
Give dashboard summary, trends, and host sections dedicated layout hooks
and tighten their responsive behavior so metrics and host cards remain
readable on narrower screens.

Refs #38
2026-05-21 14:46:23 +02:00
025cd0336c (ui) Harden dashboard responsive layout
Rework the dashboard priority grid to avoid cramped four-column layouts,
prevent stretched empty panels, and make long host and snapshot text wrap
safely across dashboard cards.

Refs #38
2026-05-21 14:43:24 +02:00
4c76ae9f52 Merge pull request 'Polish forms and action flows' (#35) from issue-25-forms-action-flows into master
Reviewed-on: #35
2026-05-21 14:28:20 +02:00
7a552715fe (ui) Clarify run action flows
Move run cancellation and review actions out of the page header into
dedicated action panels with clearer operator copy and consistent form
button styling.

Refs #25
2026-05-21 14:25:26 +02:00
0f0de5dc30 (ui) Standardize list filter actions
Give run, snapshot, schedule, purged snapshot, and log filters the same
responsive form layout with consistent Apply/Clear actions.

Refs #25
2026-05-21 14:22:11 +02:00
1604f0f6f4 (ui) Clarify destructive action flows
Make retention apply, incomplete cleanup, and SSH key deletion visibly
destructive with warning copy, danger styling, and consistent cancel actions
while keeping the existing confirmation requirements intact.

Refs #25
2026-05-21 14:17:07 +02:00
af548f11c4 (ui) Standardize primary form actions
Add shared form action styling and consistent Cancel links across config,
schedule, and SSH key forms so create/edit flows behave predictably.

Refs #25
2026-05-21 14:13:05 +02:00
c0eca3da55 Merge pull request 'issue-26-host-detail-control-page' (#34) from issue-26-host-detail-control-page into master
Reviewed-on: #34
2026-05-21 14:05:51 +02:00
212813e066 (ui) Make host diagnostics easier to scan
Replace the raw host check table with diagnostic cards and group effective
runtime config into operator-focused sections for backup target, connection,
and selection/retention details.

Refs #26
2026-05-21 14:03:43 +02:00
ab5291b8d3 (ui) Show host runs and snapshots as record cards
Replace the database-style Latest Runs and Snapshots tables on the host
detail page with scannable record cards and host-filtered View all links.

Refs #26
2026-05-21 13:59:16 +02:00
1929196287 (ui) Group host detail actions by context
Move host-level actions out of the page header and into the panels where
operators expect them: configuration, connection preflight, and snapshot
storage. This keeps the host control page calmer while preserving the same
actions.

Refs #26
2026-05-21 13:53:10 +02:00
9e75273fc5 (ui) Promote host detail operator controls
Add a first-screen host control workspace with status, backup actions,
schedule state, and current activity so the host detail page behaves as the
primary operator page instead of starting with raw configuration blocks.

Refs #26
2026-05-21 13:40:37 +02:00
5dd6ebb3db Merge pull request 'issue-27-dashboard-information-architecture' (#33) from issue-27-dashboard-information-architecture into master
Reviewed-on: #33
2026-05-21 13:29:18 +02:00
864a40e862 (ui) Surface storage pressure in dashboard priorities
Move backup root usage, runway, daily new data, and available capacity into
the top dashboard priority area so storage risk is visible before deeper trend
details.

Refs #27
2026-05-21 13:27:39 +02:00
9412feaa58 (ui) Rework dashboard around operator priorities
Move required actions, upcoming scheduled work, and recent run activity to the
top of the dashboard so the first screen answers what needs attention next.
Keep summary metrics, trends, and host cards as supporting drill-down content.

Refs #27
2026-05-21 13:21:09 +02:00
0fe2aa439f Merge pull request '(ui) Add review actions to filtered run lists' (#32) from issue-22-operational-review-actions into master
Reviewed-on: #32
2026-05-21 13:12:20 +02:00
fe4ae9d147 (ui) Add review actions to filtered run lists
Add inline Mark reviewed actions for failed and warning runs on the run list,
preserving active filters after review so Operational Status drill-downs can
be cleared without opening every run detail page.

Refs #22
2026-05-21 13:07:45 +02:00
0a3a3448d6 Merge pull request 'Make dashboard cards link to operational lists' (#31) from issue-23-dashboard-list-pages into master
Reviewed-on: #31
2026-05-21 12:49:15 +02:00
64 changed files with 5163 additions and 821 deletions

View File

@@ -1,5 +1,56 @@
# Changelog # Changelog
## 1.2.0 - 2026-05-28
Operations-focused release for more reliable production backups and maintenance.
### Added
- Staff-only updater page for checking configured Gitea releases, inspecting the installed git checkout, fetching tags, pulling the current branch, and running the native systemd updater.
- Read-only control panel access level for authenticated non-staff users, with status pages visible and credentials, logs, configs, retention, and mutating actions kept staff-only.
- Run completion notifications for email and webhooks, including recorded delivery history per run and target.
- Dedicated hosts page with host cards, enabled/disabled filtering, and quick host/schedule/retention state controls.
- Per-host rsync bandwidth limit overrides with inherit, unlimited, and explicit limit semantics.
- Backup data totals by snapshot kind on dashboard and host detail pages, including unique/non-hardlinked data totals.
### Changed
- Real backup runs now default to verbose rsync progress output so the live run view behaves consistently with dry-runs.
- Run progress panels are shared between dry-runs and real runs for more consistent status, timing, cancellation, and log display.
- Incomplete snapshot cleanup now requires operator review before deletion.
- Incomplete snapshot size reporting now prefers on-disk measurement when metadata is stale or missing.
- Installer and environment examples now include optional updater configuration.
### Fixed
- Remote preflight shell commands are now quoted correctly, including roots such as `/`.
- Worker reconciliation now detects real rsync failures and stale/running process state more reliably.
- Retention pruning and incomplete cleanup can delete snapshots containing restrictive directory modes preserved by rsync archive mode.
- Snapshot data summaries no longer count incomplete metadata/log files as backup data when measuring from disk.
- Filesystem SSH credential tests use writable test state without changing production defaults.
## 1.1.0 - 2026-05-21
UI-focused release for the Django control panel.
### Added
- Dedicated list pages for runs, snapshots, schedules, purged snapshots, and changelog navigation.
- Dashboard priority panels for required action, next scheduled work, recent activity, and storage pressure.
- Dashboard host cards with clearer backup activity, snapshot health, next run, and retention status.
- Lightweight live refresh for active run detail pages, including status, timing, controls, and rsync log output.
- Lightweight live refresh for dashboard priority and host status sections.
- Current-page navigation states for primary and system navigation.
- Responsive dashboard behavior for narrower screens.
### Changed
- Reworked the primary navigation around day-to-day operator workflows and moved admin/system links out of the main path.
- Simplified legacy-facing labels and removed source-of-truth wording that no longer applies to the Django-first model.
- Improved run and snapshot detail pages with clearer links between backup runs, snapshots, logs, and review actions.
- Improved dashboard spacing and card layouts to reduce cramped or overlapping text.
- Documented the Django-template-first partial refresh pattern for future UI work.
## 1.0.0 - 2026-05-21 ## 1.0.0 - 2026-05-21
Initial stable release of the Django-first pobsync control panel. Initial stable release of the Django-first pobsync control panel.

View File

@@ -131,6 +131,11 @@ Create a superuser if needed:
sudo -u pobsync pobsync-manage createsuperuser sudo -u pobsync pobsync-manage createsuperuser
``` ```
The control panel supports two access levels. Django staff users can manage hosts, SSH keys, configs, retention,
notifications, logs, and administrative actions. Normal authenticated users can view backup status pages such as the
dashboard, hosts, runs, snapshots, schedules, purged history, changelog, and `/api/status/`, but cannot see SSH
credentials or run mutating actions.
For other Django management commands on native installs, use `pobsync-manage` so the production environment file is For other Django management commands on native installs, use `pobsync-manage` so the production environment file is
loaded before Django starts: loaded before Django starts:
@@ -153,10 +158,24 @@ The UI includes:
- Django-managed SSH keys - Django-managed SSH keys
- `/self-check/` for runtime checks - `/self-check/` for runtime checks
- `/logs/` for filtered pobsync service logs - `/logs/` for filtered pobsync service logs
- `/updater/` for checking Gitea releases, pulling the git checkout, and running the native updater
## Bandwidth Limits
Global config can set an rsync bandwidth limit in KB/s. The default `0` means unlimited. Each host can inherit the
global value, set `0` to explicitly run unlimited, or set its own limit for slower remote links.
For VPN-backed or remote backups, start conservatively and adjust after watching normal traffic:
- `2500` KB/s is roughly 20 Mbit/s
- `5000` KB/s is roughly 40 Mbit/s
- `10000` KB/s is roughly 80 Mbit/s
pobsync passes the effective value to rsync as `--bwlimit=<KB/s>` and shows it on the host detail and run detail pages.
## Restoring Data ## Restoring Data
pobsync 1.0 treats restores as an explicit manual operation. The control panel shows restore guidance on each snapshot pobsync treats restores as an explicit manual operation. The control panel shows restore guidance on each snapshot
detail page, but it does not run restore commands for you yet. That is deliberate: restores should be inspected and detail page, but it does not run restore commands for you yet. That is deliberate: restores should be inspected and
tested before data is copied back into a live system. tested before data is copied back into a live system.
@@ -225,6 +244,21 @@ The updater is a thin wrapper around the installer for normal production deploys
Python dependencies, runs migrations, collects static files, and restarts the systemd services so new Django code is Python dependencies, runs migrations, collects static files, and restarts the systemd services so new Django code is
loaded. loaded.
The Django control panel also exposes an `/updater/` page for staff users. It can check a Gitea releases endpoint, run
`git fetch`, run a fast-forward-only pull for the installed branch, and invoke the configured native update command.
Configure these optional environment variables in `/etc/pobsync/pobsync.env`:
```
POBSYNC_UPDATE_RELEASES_URL=https://code.example.test/api/v1/repos/owner/pobsync/releases
POBSYNC_UPDATE_RELEASES_TOKEN=
POBSYNC_UPDATE_GIT_REMOTE=origin
POBSYNC_UPDATE_COMMAND=sudo -n scripts/update-systemd
```
If the web service runs as the `pobsync` user, `POBSYNC_UPDATE_COMMAND` needs a matching sudoers rule or a different
operator-approved command. Without that, the page still shows update status and command output, but the native update
action will fail with a permission error instead of silently doing the wrong thing.
Use the full installer again when you intentionally want to change install-time settings, install OS packages, enable Use the full installer again when you intentionally want to change install-time settings, install OS packages, enable
nginx, or rewrite the environment file: nginx, or rewrite the environment file:

View File

@@ -16,3 +16,11 @@ POBSYNC_GUNICORN_WORKERS=2
POBSYNC_GUNICORN_TIMEOUT=120 POBSYNC_GUNICORN_TIMEOUT=120
POBSYNC_WORKER_INTERVAL=15 POBSYNC_WORKER_INTERVAL=15
POBSYNC_SCHEDULER_INTERVAL=60 POBSYNC_SCHEDULER_INTERVAL=60
# Optional UI updater integration.
# Point this at the Gitea releases API endpoint, for example:
# https://code.example.test/api/v1/repos/owner/pobsync/releases
POBSYNC_UPDATE_RELEASES_URL=
POBSYNC_UPDATE_RELEASES_TOKEN=
POBSYNC_UPDATE_GIT_REMOTE=origin
POBSYNC_UPDATE_COMMAND=sudo -n scripts/update-systemd

View File

@@ -50,6 +50,16 @@ python3 manage.py showmigrations pobsync_backend
The short `pobsync` aliases are limited to operational actions that are useful while debugging a running install. The short `pobsync` aliases are limited to operational actions that are useful while debugging a running install.
Configuration aliases are intentionally not public commands; use the Django UI or explicit management commands instead. Configuration aliases are intentionally not public commands; use the Django UI or explicit management commands instead.
## UI Refresh Pattern
The control panel stays Django-template-first. Pages that need live status should expose a small server-rendered partial
view and opt into refresh with `data-refresh-url` and `data-refresh-interval` on the container that should be replaced.
The shared script in `base.html` polls only those explicit regions, skips refreshes while the browser tab is hidden, and
lets the partial response turn polling off with the `X-Pobsync-Refresh-Active: false` header.
Use this for operational status surfaces such as running backup details. Avoid refreshing form-heavy sections while an
operator might be typing.
Worker and scheduler commands are normally run by systemd services: Worker and scheduler commands are normally run by systemd services:
``` ```

View File

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "pobsync" name = "pobsync"
version = "1.0.0" version = "1.2.0"
description = "Pull-based rsync backup tool with hardlinked snapshots" description = "Pull-based rsync backup tool with hardlinked snapshots"
requires-python = ">=3.11" requires-python = ">=3.11"
dependencies = [ dependencies = [

View File

@@ -472,6 +472,10 @@ POBSYNC_GUNICORN_WORKERS=2
POBSYNC_GUNICORN_TIMEOUT=120 POBSYNC_GUNICORN_TIMEOUT=120
POBSYNC_WORKER_INTERVAL=15 POBSYNC_WORKER_INTERVAL=15
POBSYNC_SCHEDULER_INTERVAL=60 POBSYNC_SCHEDULER_INTERVAL=60
POBSYNC_UPDATE_RELEASES_URL=
POBSYNC_UPDATE_RELEASES_TOKEN=
POBSYNC_UPDATE_GIT_REMOTE=origin
POBSYNC_UPDATE_COMMAND=sudo -n scripts/update-systemd
EOF EOF
chmod 0640 "$ENV_FILE" chmod 0640 "$ENV_FILE"
chown "root:$SERVICE_GROUP" "$ENV_FILE" chown "root:$SERVICE_GROUP" "$ENV_FILE"

View File

@@ -1,2 +1,2 @@
__all__ = ["__version__"] __all__ = ["__version__"]
__version__ = "1.0.0" __version__ = "1.2.0"

View File

@@ -23,6 +23,7 @@ from ..util import ensure_dir, realpath_startswith, sanitize_host, write_yaml_at
DEFAULT_DRY_RUN_TIMEOUT_SECONDS = 900 DEFAULT_DRY_RUN_TIMEOUT_SECONDS = 900
RSYNC_PARTIAL_VANISHED_EXIT_CODE = 24
def dry_run_log_path(host: str, run_id: int | None = None) -> Path: def dry_run_log_path(host: str, run_id: int | None = None) -> Path:
@@ -72,6 +73,24 @@ def classify_rsync_failure(exit_code: int | None, log_tail: list[str]) -> dict[s
} }
def classify_rsync_warning(exit_code: int | None, log_tail: list[str]) -> dict[str, str] | None:
joined_tail = "\n".join(log_tail).lower()
if exit_code == RSYNC_PARTIAL_VANISHED_EXIT_CODE:
return {
"category": "vanished",
"message": "Some source files vanished during rsync.",
"hint": "This is common on live systems. The snapshot was kept, but review the rsync log if this happens often.",
}
if exit_code in (None, RSYNC_PARTIAL_VANISHED_EXIT_CODE) and (
"file has vanished" in joined_tail or "vanished before it could be transferred" in joined_tail
):
return {
"category": "vanished",
"message": "Some source files vanished during rsync.",
"hint": "This is common on live systems. The snapshot was kept, but review the rsync log if this happens often.",
}
return None
def _collect_run_stats( def _collect_run_stats(
*, *,
log_path: Path, log_path: Path,
@@ -158,6 +177,7 @@ def run_scheduled(
run_id: int | None = None, run_id: int | None = None,
cancel_check: Callable[[], bool] | None = None, cancel_check: Callable[[], bool] | None = None,
verbose_output: bool = False, verbose_output: bool = False,
state_callback: Callable[[dict[str, Any]], None] | None = None,
) -> dict[str, Any]: ) -> dict[str, Any]:
host = sanitize_host(host) host = sanitize_host(host)
@@ -258,6 +278,7 @@ def run_scheduled(
"exit_code": result.exit_code, "exit_code": result.exit_code,
"command": result.command, "command": result.command,
"log_tail": log_tail, "log_tail": log_tail,
"bwlimit_kbps": bwlimit_kbps,
}, },
} }
if result.exit_code != 0: if result.exit_code != 0:
@@ -316,20 +337,65 @@ def run_scheduled(
"ended_at": None, "ended_at": None,
"duration_seconds": None, "duration_seconds": None,
"base": _base_meta_from_path(base_dir, link_dest), "base": _base_meta_from_path(base_dir, link_dest),
"rsync": {"exit_code": None, "command": cmd, "stats": {}}, "rsync": {"exit_code": None, "command": cmd, "stats": {}, "bwlimit_kbps": bwlimit_kbps},
"overrides": {"includes": [], "excludes": [], "base": None}, "overrides": {"includes": [], "excludes": [], "base": None},
} }
log_path.touch(exist_ok=True) log_path.touch(exist_ok=True)
write_yaml_atomic(meta_path, meta) write_yaml_atomic(meta_path, meta)
if state_callback is not None:
state_callback(
{
"status": "running",
"phase": "preparing",
"snapshot": str(incomplete_dir),
"log": str(log_path),
"rsync": {"command": cmd, "exit_code": None, "bwlimit_kbps": bwlimit_kbps},
}
)
result = run_rsync(cmd, log_path=log_path, timeout_seconds=timeout_seconds, cancel_check=cancel_check) def process_started(pid: int, pgid: int) -> None:
if state_callback is None:
return
state_callback(
{
"status": "running",
"phase": "rsync",
"snapshot": str(incomplete_dir),
"log": str(log_path),
"rsync": {"command": cmd, "exit_code": None, "pid": pid, "pgid": pgid, "bwlimit_kbps": bwlimit_kbps},
}
)
run_rsync_kwargs: dict[str, Any] = {
"log_path": log_path,
"timeout_seconds": timeout_seconds,
"cancel_check": cancel_check,
}
if state_callback is not None:
run_rsync_kwargs["process_started"] = process_started
result = run_rsync(cmd, **run_rsync_kwargs)
log_tail = _read_log_tail(log_path)
warning = classify_rsync_warning(result.exit_code, log_tail)
successful_or_warning = result.exit_code == 0 or warning is not None
if state_callback is not None:
state_callback(
{
"status": "running",
"phase": "finalizing",
"snapshot": str(incomplete_dir),
"log": str(log_path),
"rsync": {"command": cmd, "exit_code": result.exit_code, "log_tail": log_tail, "bwlimit_kbps": bwlimit_kbps},
}
)
end_ts = utc_now() end_ts = utc_now()
meta["ended_at"] = format_iso_z(end_ts) meta["ended_at"] = format_iso_z(end_ts)
meta["duration_seconds"] = int((end_ts - ts).total_seconds()) meta["duration_seconds"] = int((end_ts - ts).total_seconds())
meta["rsync"]["exit_code"] = result.exit_code meta["rsync"]["exit_code"] = result.exit_code
meta["status"] = "cancelled" if result.cancelled else ("success" if result.exit_code == 0 else "failed") meta["status"] = "cancelled" if result.cancelled else ("warning" if warning else ("success" if result.exit_code == 0 else "failed"))
if warning is not None:
meta["warning"] = warning
meta["stats"] = _collect_run_stats( meta["stats"] = _collect_run_stats(
log_path=log_path, log_path=log_path,
backup_root=Path(backup_root), backup_root=Path(backup_root),
@@ -349,8 +415,7 @@ def run_scheduled(
"error": "rsync.log missing after execution", "error": "rsync.log missing after execution",
} }
if result.exit_code != 0: if not successful_or_warning:
log_tail = _read_log_tail(log_path)
return { return {
"ok": False, "ok": False,
"dry_run": False, "dry_run": False,
@@ -366,6 +431,7 @@ def run_scheduled(
"exit_code": result.exit_code, "exit_code": result.exit_code,
"command": result.command, "command": result.command,
"log_tail": log_tail, "log_tail": log_tail,
"bwlimit_kbps": bwlimit_kbps,
}, },
"failure": classify_rsync_failure(result.exit_code, log_tail), "failure": classify_rsync_failure(result.exit_code, log_tail),
} }
@@ -404,7 +470,9 @@ def run_scheduled(
"snapshot": str(final_dir), "snapshot": str(final_dir),
"base": str(base_dir) if base_dir else None, "base": str(base_dir) if base_dir else None,
"log": str(final_log_path), "log": str(final_log_path),
"rsync": {"exit_code": result.exit_code}, "status": meta["status"],
"warning": warning,
"rsync": {"exit_code": result.exit_code, "command": result.command, "log_tail": log_tail, "bwlimit_kbps": bwlimit_kbps},
"verbose_output": bool(verbose_output), "verbose_output": bool(verbose_output),
"duration_seconds": meta["duration_seconds"], "duration_seconds": meta["duration_seconds"],
"stats": meta["stats"], "stats": meta["stats"],

View File

@@ -110,6 +110,7 @@ GLOBAL_SCHEMA = Schema(
HOST_RSYNC_SCHEMA = Schema( HOST_RSYNC_SCHEMA = Schema(
fields={ fields={
"bwlimit_kbps": FieldSpec(int, required=False, min_value=0),
"extra_args": FieldSpec(list, required=False, default=[], item=FieldSpec(str)), "extra_args": FieldSpec(list, required=False, default=[], item=FieldSpec(str)),
}, },
allow_unknown=False, allow_unknown=False,

View File

@@ -82,6 +82,7 @@ def run_rsync(
log_path: Path, log_path: Path,
timeout_seconds: int, timeout_seconds: int,
cancel_check: Callable[[], bool] | None = None, cancel_check: Callable[[], bool] | None = None,
process_started: Callable[[int, int], None] | None = None,
) -> RsyncResult: ) -> RsyncResult:
""" """
Run rsync and always write stdout/stderr to log_path. Run rsync and always write stdout/stderr to log_path.
@@ -95,6 +96,8 @@ def run_rsync(
with log_path.open("ab") as f: with log_path.open("ab") as f:
process = subprocess.Popen(command, stdout=f, stderr=subprocess.STDOUT, start_new_session=True) process = subprocess.Popen(command, stdout=f, stderr=subprocess.STDOUT, start_new_session=True)
if process_started is not None:
process_started(process.pid, os.getpgid(process.pid))
started = time.monotonic() started = time.monotonic()
while True: while True:
exit_code = process.poll() exit_code = process.poll()

View File

@@ -0,0 +1,39 @@
from __future__ import annotations
from collections.abc import Callable
from functools import wraps
from typing import Any
from django.contrib.auth.decorators import login_required
from django.core.exceptions import PermissionDenied
from django.http import HttpRequest, HttpResponse
def can_view_status(user) -> bool:
return bool(user.is_authenticated)
def can_manage_control_panel(user) -> bool:
return bool(user.is_authenticated and user.is_staff)
def status_view_required(view_func: Callable[..., HttpResponse]) -> Callable[..., HttpResponse]:
return login_required(view_func)
def control_panel_admin_required(view_func: Callable[..., HttpResponse]) -> Callable[..., HttpResponse]:
@login_required
@wraps(view_func)
def wrapper(request: HttpRequest, *args: Any, **kwargs: Any) -> HttpResponse:
if not can_manage_control_panel(request.user):
raise PermissionDenied
return view_func(request, *args, **kwargs)
return wrapper
def access_context(request: HttpRequest) -> dict[str, Any]:
return {
"can_view_status": can_view_status(request.user),
"can_manage_control_panel": can_manage_control_panel(request.user),
}

View File

@@ -6,7 +6,17 @@ from django.urls import reverse
from django.utils.html import format_html from django.utils.html import format_html
from django.utils.http import urlencode from django.utils.http import urlencode
from .models import BackupRun, GlobalConfig, HostConfig, PurgedSnapshot, ScheduleConfig, SnapshotRecord, SshCredential from .models import (
BackupRun,
GlobalConfig,
HostConfig,
NotificationDelivery,
NotificationTarget,
PurgedSnapshot,
ScheduleConfig,
SnapshotRecord,
SshCredential,
)
@admin.register(SshCredential) @admin.register(SshCredential)
@@ -73,7 +83,7 @@ class HostConfigAdmin(admin.ModelAdmin):
(None, {"fields": ("host", "address", "enabled")}), (None, {"fields": ("host", "address", "enabled")}),
("SSH override", {"fields": ("ssh_credential", "ssh_user", "ssh_port")}), ("SSH override", {"fields": ("ssh_credential", "ssh_user", "ssh_port")}),
("Source", {"fields": ("source_root", "includes", "excludes_add", "excludes_replace")}), ("Source", {"fields": ("source_root", "includes", "excludes_add", "excludes_replace")}),
("Rsync override", {"fields": ("rsync_extra_args",)}), ("Rsync override", {"fields": ("rsync_extra_args", "rsync_bwlimit_kbps")}),
("Retention", {"fields": ("retention_daily", "retention_weekly", "retention_monthly", "retention_yearly")}), ("Retention", {"fields": ("retention_daily", "retention_weekly", "retention_monthly", "retention_yearly")}),
("Runtime state", {"fields": ("config",), "classes": ("collapse",)}), ("Runtime state", {"fields": ("config",), "classes": ("collapse",)}),
("Timestamps", {"fields": ("created_at", "updated_at"), "classes": ("collapse",)}), ("Timestamps", {"fields": ("created_at", "updated_at"), "classes": ("collapse",)}),
@@ -136,6 +146,38 @@ class BackupRunAdmin(admin.ModelAdmin):
return format_html('<a href="{}">{}</a>', url, obj.snapshot.dirname) return format_html('<a href="{}">{}</a>', url, obj.snapshot.dirname)
@admin.register(NotificationTarget)
class NotificationTargetAdmin(admin.ModelAdmin):
list_display = ("name", "channel", "enabled", "last_status", "last_sent_at", "updated_at")
list_filter = ("enabled", "channel", "last_status")
search_fields = ("name", "email_to", "webhook_url", "notes")
readonly_fields = ("created_at", "updated_at", "last_status", "last_error", "last_sent_at")
fieldsets = (
(None, {"fields": ("name", "enabled", "channel", "statuses")}),
("Email", {"fields": ("email_to",)}),
("Webhook", {"fields": ("webhook_url", "webhook_headers")}),
("State", {"fields": ("last_status", "last_error", "last_sent_at"), "classes": ("collapse",)}),
("Notes", {"fields": ("notes",), "classes": ("collapse",)}),
("Timestamps", {"fields": ("created_at", "updated_at"), "classes": ("collapse",)}),
)
@admin.register(NotificationDelivery)
class NotificationDeliveryAdmin(admin.ModelAdmin):
list_display = ("target", "run", "status", "created_at")
list_filter = ("status", "target__channel", "created_at")
search_fields = ("target__name", "run__host__host", "error")
readonly_fields = ("target", "run", "status", "error", "payload", "created_at")
list_select_related = ("target", "run", "run__host")
date_hierarchy = "created_at"
def has_add_permission(self, request) -> bool:
return False
def has_change_permission(self, request, obj=None) -> bool:
return False
@admin.register(SnapshotRecord) @admin.register(SnapshotRecord)
class SnapshotRecordAdmin(admin.ModelAdmin): class SnapshotRecordAdmin(admin.ModelAdmin):
list_display = ("host", "kind", "dirname", "status", "base_link", "backup_run_count_link", "started_at", "discovered_at") list_display = ("host", "kind", "dirname", "status", "base_link", "backup_run_count_link", "started_at", "discovered_at")

View File

@@ -2,16 +2,16 @@ from __future__ import annotations
from typing import Any from typing import Any
from django.contrib.admin.views.decorators import staff_member_required
from django.db import connection from django.db import connection
from django.db.models import Count from django.db.models import Count
from django.http import JsonResponse from django.http import JsonResponse
from django.utils import timezone from django.utils import timezone
from .models import BackupRun, HostConfig, ScheduleConfig, SnapshotRecord from .models import BackupRun, HostConfig, ScheduleConfig, SnapshotRecord
from .access import control_panel_admin_required, status_view_required
@staff_member_required @control_panel_admin_required
def api_index(request) -> JsonResponse: def api_index(request) -> JsonResponse:
return JsonResponse( return JsonResponse(
{ {
@@ -26,7 +26,7 @@ def api_index(request) -> JsonResponse:
) )
@staff_member_required @status_view_required
def status(request) -> JsonResponse: def status(request) -> JsonResponse:
latest_run = BackupRun.objects.select_related("host", "snapshot").order_by("-created_at").first() latest_run = BackupRun.objects.select_related("host", "snapshot").order_by("-created_at").first()
latest_schedule = ScheduleConfig.objects.select_related("host").order_by("-last_started_at", "-updated_at").first() latest_schedule = ScheduleConfig.objects.select_related("host").order_by("-last_started_at", "-updated_at").first()
@@ -55,7 +55,7 @@ def status(request) -> JsonResponse:
) )
@staff_member_required @control_panel_admin_required
def hosts(request) -> JsonResponse: def hosts(request) -> JsonResponse:
host_qs = ( host_qs = (
HostConfig.objects.annotate(snapshot_count=Count("snapshots", distinct=True), run_count=Count("runs", distinct=True)) HostConfig.objects.annotate(snapshot_count=Count("snapshots", distinct=True), run_count=Count("runs", distinct=True))
@@ -65,7 +65,7 @@ def hosts(request) -> JsonResponse:
return JsonResponse({"ok": True, "hosts": [_host_payload(host) for host in host_qs]}) return JsonResponse({"ok": True, "hosts": [_host_payload(host) for host in host_qs]})
@staff_member_required @control_panel_admin_required
def snapshots(request) -> JsonResponse: def snapshots(request) -> JsonResponse:
snapshot_qs = SnapshotRecord.objects.select_related("host", "base").order_by("host__host", "-started_at", "dirname") snapshot_qs = SnapshotRecord.objects.select_related("host", "base").order_by("host__host", "-started_at", "dirname")
host_filter = request.GET.get("host") host_filter = request.GET.get("host")
@@ -78,7 +78,7 @@ def snapshots(request) -> JsonResponse:
return JsonResponse({"ok": True, "snapshots": [_snapshot_payload(snapshot) for snapshot in snapshot_qs[:limit]]}) return JsonResponse({"ok": True, "snapshots": [_snapshot_payload(snapshot) for snapshot in snapshot_qs[:limit]]})
@staff_member_required @control_panel_admin_required
def runs(request) -> JsonResponse: def runs(request) -> JsonResponse:
run_qs = BackupRun.objects.select_related("host", "snapshot").order_by("-created_at") run_qs = BackupRun.objects.select_related("host", "snapshot").order_by("-created_at")
host_filter = request.GET.get("host") host_filter = request.GET.get("host")

View File

@@ -8,9 +8,16 @@ from pathlib import Path
from django.db import transaction from django.db import transaction
from django.utils import timezone from django.utils import timezone
from pobsync.commands.run_scheduled import DEFAULT_DRY_RUN_TIMEOUT_SECONDS, classify_rsync_failure, dry_run_log_path, run_scheduled from pobsync.commands.run_scheduled import (
DEFAULT_DRY_RUN_TIMEOUT_SECONDS,
classify_rsync_failure,
classify_rsync_warning,
dry_run_log_path,
run_scheduled,
)
from pobsync_backend.config_source import DjangoConfigSource from pobsync_backend.config_source import DjangoConfigSource
from pobsync_backend.models import BackupRun, HostConfig from pobsync_backend.models import BackupRun, HostConfig
from pobsync_backend.notifications import notify_backup_run_completed
from pobsync_backend.retention import run_sql_retention_apply from pobsync_backend.retention import run_sql_retention_apply
from pobsync_backend.snapshot_discovery import infer_snapshot_kind, upsert_snapshot_record from pobsync_backend.snapshot_discovery import infer_snapshot_kind, upsert_snapshot_record
@@ -20,7 +27,7 @@ def queue_backup_run(
host: HostConfig, host: HostConfig,
run_type: str = BackupRun.RunType.MANUAL, run_type: str = BackupRun.RunType.MANUAL,
dry_run: bool = False, dry_run: bool = False,
verbose_output: bool = False, verbose_output: bool = True,
prune: bool = False, prune: bool = False,
prune_max_delete: int = 10, prune_max_delete: int = 10,
prune_protect_bases: bool = False, prune_protect_bases: bool = False,
@@ -66,6 +73,7 @@ def execute_backup_run(
run_id=run.id, run_id=run.id,
cancel_check=lambda: _run_cancel_requested(run.id), cancel_check=lambda: _run_cancel_requested(run.id),
verbose_output=bool(dry_run or verbose_output), verbose_output=bool(dry_run or verbose_output),
state_callback=lambda state: _record_running_state(run.id, state),
) )
except Exception as exc: except Exception as exc:
run.refresh_from_db() run.refresh_from_db()
@@ -78,11 +86,14 @@ def execute_backup_run(
"type": type(exc).__name__, "type": type(exc).__name__,
} }
run.save(update_fields=["status", "ended_at", "result"]) run.save(update_fields=["status", "ended_at", "result"])
notify_backup_run_completed(run)
raise raise
run.refresh_from_db() run.refresh_from_db()
if result.get("cancelled") or run.status == BackupRun.Status.CANCELLED: if result.get("cancelled") or run.status == BackupRun.Status.CANCELLED:
run.status = BackupRun.Status.CANCELLED run.status = BackupRun.Status.CANCELLED
elif result.get("status") == BackupRun.Status.WARNING:
run.status = BackupRun.Status.WARNING
else: else:
run.status = BackupRun.Status.SUCCESS if result.get("ok") else BackupRun.Status.FAILED run.status = BackupRun.Status.SUCCESS if result.get("ok") else BackupRun.Status.FAILED
run.ended_at = timezone.now() run.ended_at = timezone.now()
@@ -142,6 +153,7 @@ def execute_backup_run(
"result", "result",
], ],
) )
notify_backup_run_completed(run)
return run return run
@@ -201,11 +213,100 @@ def _run_cancel_requested(run_id: int) -> bool:
return False return False
def _record_running_state(run_id: int, state: dict[str, object]) -> None:
try:
run = BackupRun.objects.only("id", "status", "result", "snapshot_path", "rsync_exit_code").get(id=run_id)
except BackupRun.DoesNotExist:
return
if run.status != BackupRun.Status.RUNNING:
return
result = run.result if isinstance(run.result, dict) else {}
execution = result.get("execution") if isinstance(result.get("execution"), dict) else {}
rsync = result.get("rsync") if isinstance(result.get("rsync"), dict) else {}
incoming_rsync = state.get("rsync") if isinstance(state.get("rsync"), dict) else {}
log_path = state.get("log")
snapshot_path = state.get("snapshot")
phase = state.get("phase")
if isinstance(phase, str) and phase:
execution["phase"] = phase
if isinstance(log_path, str) and log_path:
execution["log"] = log_path
if isinstance(snapshot_path, str) and snapshot_path:
execution["snapshot"] = snapshot_path
run.snapshot_path = snapshot_path
if incoming_rsync:
result["rsync"] = {**rsync, **incoming_rsync}
exit_code = incoming_rsync.get("exit_code")
if isinstance(exit_code, int):
run.rsync_exit_code = exit_code
result["execution"] = {
**execution,
"worker_pid": os.getpid(),
"worker_host": socket.gethostname(),
"heartbeat_at": timezone.now().isoformat(),
}
run.result = result
run.save(update_fields=["snapshot_path", "rsync_exit_code", "result"])
def _reconcile_running_run(*, run: BackupRun, grace_seconds: int, stale_worker_seconds: int) -> bool: def _reconcile_running_run(*, run: BackupRun, grace_seconds: int, stale_worker_seconds: int) -> bool:
result = run.result if isinstance(run.result, dict) else {} result = run.result if isinstance(run.result, dict) else {}
requested = result.get("requested") if isinstance(result.get("requested"), dict) else {} requested = result.get("requested") if isinstance(result.get("requested"), dict) else {}
log_path = _execution_log_path(result)
log_tail = _read_log_tail(log_path) if log_path is not None else []
terminal_log = _terminal_rsync_log(log_tail)
exit_code = _exit_code_from_log(log_tail)
stale_worker = _running_worker_timed_out(run=run, stale_worker_seconds=stale_worker_seconds) stale_worker = _running_worker_timed_out(run=run, stale_worker_seconds=stale_worker_seconds)
if not requested.get("dry_run"): if not requested.get("dry_run"):
if terminal_log:
failure = classify_rsync_failure(exit_code or 255, log_tail)
result.update(
{
"ok": False,
"host": run.host.host,
"log": str(log_path) if log_path else "",
"failure": failure,
"rsync": {
**(result.get("rsync") if isinstance(result.get("rsync"), dict) else {}),
"exit_code": exit_code or 255,
"log_tail": log_tail,
},
}
)
run.status = BackupRun.Status.FAILED
run.ended_at = timezone.now()
run.rsync_exit_code = exit_code or 255
run.result = result
run.save(update_fields=["status", "ended_at", "rsync_exit_code", "result"])
notify_backup_run_completed(run)
return True
if _running_rsync_process_missing(run=run, grace_seconds=grace_seconds):
result.update(
{
"ok": False,
"host": run.host.host,
"log": str(log_path) if log_path else "",
"failure": {
"category": "rsync_process",
"message": "The rsync process is no longer running while the backup is still marked running.",
"hint": "Check the rsync log and pobsync-worker.service logs before retrying the backup.",
},
"rsync": {
**(result.get("rsync") if isinstance(result.get("rsync"), dict) else {}),
"exit_code": exit_code or 255,
"log_tail": log_tail,
},
}
)
run.status = BackupRun.Status.FAILED
run.ended_at = timezone.now()
run.rsync_exit_code = exit_code or 255
run.result = result
run.save(update_fields=["status", "ended_at", "rsync_exit_code", "result"])
notify_backup_run_completed(run)
return True
if stale_worker: if stale_worker:
result.update( result.update(
{ {
@@ -222,17 +323,15 @@ def _reconcile_running_run(*, run: BackupRun, grace_seconds: int, stale_worker_s
run.ended_at = timezone.now() run.ended_at = timezone.now()
run.result = result run.result = result
run.save(update_fields=["status", "ended_at", "result"]) run.save(update_fields=["status", "ended_at", "result"])
notify_backup_run_completed(run)
return True return True
return False return False
log_path = _execution_log_path(result)
log_tail = _read_log_tail(log_path) if log_path is not None else []
terminal_log = _terminal_rsync_log(log_tail)
timed_out = _running_dry_run_timed_out(run=run, grace_seconds=grace_seconds) timed_out = _running_dry_run_timed_out(run=run, grace_seconds=grace_seconds)
if not terminal_log and not timed_out and not stale_worker: if not terminal_log and not timed_out and not stale_worker:
return False return False
exit_code = _exit_code_from_log(log_tail) or (124 if timed_out or stale_worker else 255) exit_code = exit_code or (124 if timed_out or stale_worker else 255)
failure = classify_rsync_failure(exit_code, log_tail) failure = classify_rsync_failure(exit_code, log_tail)
if stale_worker and not terminal_log: if stale_worker and not terminal_log:
failure = { failure = {
@@ -260,6 +359,7 @@ def _reconcile_running_run(*, run: BackupRun, grace_seconds: int, stale_worker_s
run.rsync_exit_code = exit_code run.rsync_exit_code = exit_code
run.result = result run.result = result
run.save(update_fields=["status", "ended_at", "rsync_exit_code", "result"]) run.save(update_fields=["status", "ended_at", "rsync_exit_code", "result"])
notify_backup_run_completed(run)
return True return True
@@ -305,6 +405,9 @@ def _read_log_tail(log_path: Path | None, *, max_lines: int = 40) -> list[str]:
def _terminal_rsync_log(log_tail: list[str]) -> bool: def _terminal_rsync_log(log_tail: list[str]) -> bool:
warning = classify_rsync_warning(_exit_code_from_log(log_tail), log_tail)
if warning is not None:
return False
return any(line.startswith("rsync error:") for line in log_tail) return any(line.startswith("rsync error:") for line in log_tail)
@@ -312,6 +415,8 @@ def _exit_code_from_log(log_tail: list[str]) -> int | None:
for line in reversed(log_tail): for line in reversed(log_tail):
if "code 255" in line: if "code 255" in line:
return 255 return 255
if "code 24" in line:
return 24
if "code 124" in line: if "code 124" in line:
return 124 return 124
if "code 12" in line: if "code 12" in line:
@@ -342,6 +447,33 @@ def _running_worker_timed_out(*, run: BackupRun, stale_worker_seconds: int) -> b
return timezone.now() >= heartbeat_at + timedelta(seconds=stale_worker_seconds) return timezone.now() >= heartbeat_at + timedelta(seconds=stale_worker_seconds)
def _running_rsync_process_missing(*, run: BackupRun, grace_seconds: int) -> bool:
if grace_seconds <= 0:
return False
result = run.result if isinstance(run.result, dict) else {}
execution = result.get("execution") if isinstance(result.get("execution"), dict) else {}
if execution.get("phase") != "rsync":
return False
rsync = result.get("rsync") if isinstance(result.get("rsync"), dict) else {}
pid = rsync.get("pid")
if not isinstance(pid, int) or pid <= 0:
return False
heartbeat_at = _parse_iso_datetime(execution.get("heartbeat_at")) or run.started_at
if heartbeat_at is None or timezone.now() < heartbeat_at + timedelta(seconds=grace_seconds):
return False
return not _process_exists(pid)
def _process_exists(pid: int) -> bool:
try:
os.kill(pid, 0)
except ProcessLookupError:
return False
except PermissionError:
return True
return True
def _parse_iso_datetime(value: object): def _parse_iso_datetime(value: object):
if not isinstance(value, str) or not value: if not isinstance(value, str) or not value:
return None return None

View File

@@ -68,8 +68,12 @@ def _host_runtime_data(host_config: HostConfig) -> dict[str, Any]:
data["excludes_replace"] = list(host_config.excludes_replace or []) data["excludes_replace"] = list(host_config.excludes_replace or [])
else: else:
data["excludes_add"] = list(host_config.excludes_add or []) data["excludes_add"] = list(host_config.excludes_add or [])
if host_config.rsync_extra_args or host_config.rsync_bwlimit_kbps is not None:
data["rsync"] = {}
if host_config.rsync_extra_args: if host_config.rsync_extra_args:
data["rsync"] = {"extra_args": list(host_config.rsync_extra_args or [])} data["rsync"]["extra_args"] = list(host_config.rsync_extra_args or [])
if host_config.rsync_bwlimit_kbps is not None:
data["rsync"]["bwlimit_kbps"] = host_config.rsync_bwlimit_kbps
return validate_dict(data, HOST_SCHEMA, path="host") return validate_dict(data, HOST_SCHEMA, path="host")

View File

@@ -0,0 +1,9 @@
from __future__ import annotations
from django.http import HttpRequest
from .access import access_context
def pobsync_access(request: HttpRequest) -> dict[str, object]:
return access_context(request)

View File

@@ -9,7 +9,7 @@ from tempfile import TemporaryDirectory
from django import forms from django import forms
from django.conf import settings from django.conf import settings
from .models import GlobalConfig, HostConfig, ScheduleConfig, SshCredential from .models import BackupRun, GlobalConfig, HostConfig, NotificationTarget, ScheduleConfig, SshCredential
from .scheduler import parse_cron_expr from .scheduler import parse_cron_expr
@@ -60,6 +60,7 @@ class HostConfigForm(forms.ModelForm):
"excludes_add", "excludes_add",
"excludes_replace", "excludes_replace",
"rsync_extra_args", "rsync_extra_args",
"rsync_bwlimit_kbps",
"retention_daily", "retention_daily",
"retention_weekly", "retention_weekly",
"retention_monthly", "retention_monthly",
@@ -70,6 +71,7 @@ class HostConfigForm(forms.ModelForm):
"ssh_user": "Leave empty to use the global SSH user.", "ssh_user": "Leave empty to use the global SSH user.",
"ssh_port": "Leave empty to use the global SSH port.", "ssh_port": "Leave empty to use the global SSH port.",
"source_root": "Leave empty to use the global default source root.", "source_root": "Leave empty to use the global default source root.",
"rsync_bwlimit_kbps": "Leave empty to inherit the global limit. Use 0 for unlimited on this host.",
} }
@@ -112,6 +114,7 @@ class GlobalConfigForm(forms.ModelForm):
help_texts = { help_texts = {
"name": "Usually 'default'. The backup engine currently reads the default config.", "name": "Usually 'default'. The backup engine currently reads the default config.",
"default_ssh_credential": "Optional. Used by hosts without their own SSH credential.", "default_ssh_credential": "Optional. Used by hosts without their own SSH credential.",
"rsync_bwlimit_kbps": "Rsync bandwidth limit in KB/s. Use 0 for unlimited.",
"default_source_root": "Used by hosts without a custom source root.", "default_source_root": "Used by hosts without a custom source root.",
"default_destination_subdir": "Optional subdirectory below each snapshot.", "default_destination_subdir": "Optional subdirectory below each snapshot.",
} }
@@ -150,6 +153,62 @@ class ManualBackupForm(forms.Form):
) )
class NotificationTargetForm(forms.ModelForm):
TERMINAL_STATUS_CHOICES = (
(BackupRun.Status.SUCCESS, BackupRun.Status.SUCCESS.label),
(BackupRun.Status.WARNING, BackupRun.Status.WARNING.label),
(BackupRun.Status.FAILED, BackupRun.Status.FAILED.label),
(BackupRun.Status.CANCELLED, BackupRun.Status.CANCELLED.label),
)
statuses = forms.MultipleChoiceField(
choices=TERMINAL_STATUS_CHOICES,
widget=forms.CheckboxSelectMultiple,
initial=[choice[0] for choice in TERMINAL_STATUS_CHOICES],
help_text="Send notifications for these terminal run statuses.",
)
email_to = forms.CharField(
widget=forms.Textarea,
required=False,
help_text="One recipient per line, or comma-separated.",
)
webhook_headers = forms.JSONField(
required=False,
widget=forms.Textarea(attrs={"rows": 4}),
help_text='Optional JSON object with extra headers, for example {"Authorization": "Bearer ..."}.',
)
class Meta:
model = NotificationTarget
fields = (
"name",
"enabled",
"channel",
"statuses",
"email_to",
"webhook_url",
"webhook_headers",
"notes",
)
widgets = {
"notes": forms.Textarea,
}
def clean(self):
cleaned_data = super().clean()
channel = cleaned_data.get("channel")
if channel == NotificationTarget.Channel.EMAIL and not cleaned_data.get("email_to", "").strip():
self.add_error("email_to", "Email targets need at least one recipient.")
if channel == NotificationTarget.Channel.WEBHOOK and not cleaned_data.get("webhook_url"):
self.add_error("webhook_url", "Webhook targets need a URL.")
return cleaned_data
def clean_email_to(self) -> str:
value = self.cleaned_data.get("email_to", "")
recipients = [line.strip() for line in value.replace(",", "\n").splitlines() if line.strip()]
return "\n".join(recipients)
class SshCredentialForm(forms.ModelForm): class SshCredentialForm(forms.ModelForm):
private_key_file = forms.FileField( private_key_file = forms.FileField(
required=False, required=False,

View File

@@ -22,6 +22,12 @@ class Command(BaseCommand):
parser.add_argument("--exclude-add", action="append", default=[]) parser.add_argument("--exclude-add", action="append", default=[])
parser.add_argument("--exclude-replace", action="append", default=None) parser.add_argument("--exclude-replace", action="append", default=None)
parser.add_argument("--rsync-extra-arg", action="append", default=[]) parser.add_argument("--rsync-extra-arg", action="append", default=[])
parser.add_argument(
"--rsync-bwlimit-kbps",
type=int,
default=None,
help="Host rsync bandwidth limit in KB/s. Omit to inherit global; set 0 for unlimited.",
)
parser.add_argument("--retention", default=None) parser.add_argument("--retention", default=None)
parser.add_argument("--disabled", action="store_true") parser.add_argument("--disabled", action="store_true")
parser.add_argument("--force", action="store_true", help="Update existing host") parser.add_argument("--force", action="store_true", help="Update existing host")
@@ -42,6 +48,7 @@ class Command(BaseCommand):
"excludes_add": [] if options["exclude_replace"] is not None else list(options["exclude_add"]), "excludes_add": [] if options["exclude_replace"] is not None else list(options["exclude_add"]),
"excludes_replace": options["exclude_replace"], "excludes_replace": options["exclude_replace"],
"rsync_extra_args": list(options["rsync_extra_arg"]), "rsync_extra_args": list(options["rsync_extra_arg"]),
"rsync_bwlimit_kbps": options["rsync_bwlimit_kbps"],
"retention_daily": retention["daily"], "retention_daily": retention["daily"],
"retention_weekly": retention["weekly"], "retention_weekly": retention["weekly"],
"retention_monthly": retention["monthly"], "retention_monthly": retention["monthly"],

View File

@@ -19,6 +19,7 @@ class Command(BaseCommand):
parser.add_argument("--prefix", default=settings.POBSYNC_HOME, help="Runtime state root") parser.add_argument("--prefix", default=settings.POBSYNC_HOME, help="Runtime state root")
parser.add_argument("--dry-run", action="store_true", help="Run rsync --dry-run") parser.add_argument("--dry-run", action="store_true", help="Run rsync --dry-run")
parser.add_argument("--verbose-rsync", action="store_true", help="Write itemized rsync output to the run log") parser.add_argument("--verbose-rsync", action="store_true", help="Write itemized rsync output to the run log")
parser.add_argument("--quiet-rsync", action="store_true", help="Skip default rsync progress output for real runs")
parser.add_argument("--prune", action="store_true", help="Apply retention after a successful run") parser.add_argument("--prune", action="store_true", help="Apply retention after a successful run")
parser.add_argument("--prune-max-delete", type=int, default=10) parser.add_argument("--prune-max-delete", type=int, default=10)
parser.add_argument("--prune-protect-bases", action="store_true") parser.add_argument("--prune-protect-bases", action="store_true")
@@ -32,6 +33,7 @@ class Command(BaseCommand):
except HostConfig.DoesNotExist as exc: except HostConfig.DoesNotExist as exc:
raise CommandError(f"Missing enabled host {host_name!r}") from exc raise CommandError(f"Missing enabled host {host_name!r}") from exc
verbose_output = bool(options["dry_run"] or options["verbose_rsync"] or not options["quiet_rsync"])
run = BackupRun.objects.create( run = BackupRun.objects.create(
host=host, host=host,
run_type=BackupRun.RunType.MANUAL if options["manual"] else BackupRun.RunType.SCHEDULED, run_type=BackupRun.RunType.MANUAL if options["manual"] else BackupRun.RunType.SCHEDULED,
@@ -39,7 +41,7 @@ class Command(BaseCommand):
result={ result={
"requested": { "requested": {
"dry_run": bool(options["dry_run"]), "dry_run": bool(options["dry_run"]),
"verbose_output": bool(options["dry_run"] or options["verbose_rsync"]), "verbose_output": verbose_output,
"prune": bool(options["prune"]), "prune": bool(options["prune"]),
"prune_max_delete": int(options["prune_max_delete"]), "prune_max_delete": int(options["prune_max_delete"]),
"prune_protect_bases": bool(options["prune_protect_bases"]), "prune_protect_bases": bool(options["prune_protect_bases"]),
@@ -50,7 +52,7 @@ class Command(BaseCommand):
run=run, run=run,
prefix=paths.home, prefix=paths.home,
dry_run=bool(options["dry_run"]), dry_run=bool(options["dry_run"]),
verbose_output=bool(options["dry_run"] or options["verbose_rsync"]), verbose_output=verbose_output,
prune=bool(options["prune"]), prune=bool(options["prune"]),
prune_max_delete=int(options["prune_max_delete"]), prune_max_delete=int(options["prune_max_delete"]),
prune_protect_bases=bool(options["prune_protect_bases"]), prune_protect_bases=bool(options["prune_protect_bases"]),

View File

@@ -0,0 +1,18 @@
# Generated by Django 5.2.14 on 2026-05-22 22:52
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('pobsync_backend', '0013_purgedsnapshot'),
]
operations = [
migrations.AddField(
model_name='hostconfig',
name='rsync_bwlimit_kbps',
field=models.PositiveIntegerField(blank=True, null=True),
),
]

View File

@@ -0,0 +1,54 @@
# Generated by Django 5.2.14 on 2026-05-28 19:11
import django.db.models.deletion
import pobsync_backend.models
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('pobsync_backend', '0014_host_bwlimit_override'),
]
operations = [
migrations.CreateModel(
name='NotificationTarget',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('created_at', models.DateTimeField(auto_now_add=True)),
('updated_at', models.DateTimeField(auto_now=True)),
('name', models.CharField(max_length=128, unique=True)),
('enabled', models.BooleanField(default=True)),
('channel', models.CharField(choices=[('email', 'Email'), ('webhook', 'Webhook')], max_length=16)),
('statuses', models.JSONField(blank=True, default=pobsync_backend.models.default_notification_statuses)),
('email_to', models.TextField(blank=True)),
('webhook_url', models.URLField(blank=True, max_length=1024)),
('webhook_headers', models.JSONField(blank=True, default=dict)),
('notes', models.TextField(blank=True)),
('last_status', models.CharField(blank=True, max_length=16)),
('last_error', models.TextField(blank=True)),
('last_sent_at', models.DateTimeField(blank=True, null=True)),
],
options={
'ordering': ['name'],
},
),
migrations.CreateModel(
name='NotificationDelivery',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('status', models.CharField(choices=[('sent', 'Sent'), ('failed', 'Failed'), ('skipped', 'Skipped')], max_length=16)),
('error', models.TextField(blank=True)),
('payload', models.JSONField(blank=True, default=dict)),
('created_at', models.DateTimeField(auto_now_add=True)),
('run', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='notification_deliveries', to='pobsync_backend.backuprun')),
('target', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='deliveries', to='pobsync_backend.notificationtarget')),
],
options={
'verbose_name_plural': 'notification deliveries',
'ordering': ['-created_at', 'target__name'],
'constraints': [models.UniqueConstraint(fields=('target', 'run'), name='unique_notification_delivery_per_target_run')],
},
),
]

View File

@@ -63,6 +63,7 @@ class HostConfig(TimestampedModel):
excludes_add = models.JSONField(default=list, blank=True) excludes_add = models.JSONField(default=list, blank=True)
excludes_replace = models.JSONField(null=True, blank=True) excludes_replace = models.JSONField(null=True, blank=True)
rsync_extra_args = models.JSONField(default=list, blank=True) rsync_extra_args = models.JSONField(default=list, blank=True)
rsync_bwlimit_kbps = models.PositiveIntegerField(null=True, blank=True)
retention_daily = models.PositiveIntegerField(default=14) retention_daily = models.PositiveIntegerField(default=14)
retention_weekly = models.PositiveIntegerField(default=8) retention_weekly = models.PositiveIntegerField(default=8)
retention_monthly = models.PositiveIntegerField(default=12) retention_monthly = models.PositiveIntegerField(default=12)
@@ -134,6 +135,63 @@ class BackupRun(models.Model):
return f"{self.host} {self.run_type} {self.status}" return f"{self.host} {self.run_type} {self.status}"
def default_notification_statuses() -> list[str]:
return [
BackupRun.Status.SUCCESS,
BackupRun.Status.WARNING,
BackupRun.Status.FAILED,
BackupRun.Status.CANCELLED,
]
class NotificationTarget(TimestampedModel):
class Channel(models.TextChoices):
EMAIL = "email", "Email"
WEBHOOK = "webhook", "Webhook"
name = models.CharField(max_length=128, unique=True)
enabled = models.BooleanField(default=True)
channel = models.CharField(max_length=16, choices=Channel.choices)
statuses = models.JSONField(default=default_notification_statuses, blank=True)
email_to = models.TextField(blank=True)
webhook_url = models.URLField(max_length=1024, blank=True)
webhook_headers = models.JSONField(default=dict, blank=True)
notes = models.TextField(blank=True)
last_status = models.CharField(max_length=16, blank=True)
last_error = models.TextField(blank=True)
last_sent_at = models.DateTimeField(null=True, blank=True)
class Meta:
ordering = ["name"]
def __str__(self) -> str:
return self.name
class NotificationDelivery(models.Model):
class Status(models.TextChoices):
SENT = "sent", "Sent"
FAILED = "failed", "Failed"
SKIPPED = "skipped", "Skipped"
target = models.ForeignKey(NotificationTarget, on_delete=models.CASCADE, related_name="deliveries")
run = models.ForeignKey(BackupRun, on_delete=models.CASCADE, related_name="notification_deliveries")
status = models.CharField(max_length=16, choices=Status.choices)
error = models.TextField(blank=True)
payload = models.JSONField(default=dict, blank=True)
created_at = models.DateTimeField(auto_now_add=True)
class Meta:
constraints = [
models.UniqueConstraint(fields=["target", "run"], name="unique_notification_delivery_per_target_run"),
]
ordering = ["-created_at", "target__name"]
verbose_name_plural = "notification deliveries"
def __str__(self) -> str:
return f"{self.target} run {self.run_id} {self.status}"
class SnapshotRecord(models.Model): class SnapshotRecord(models.Model):
class Kind(models.TextChoices): class Kind(models.TextChoices):
SCHEDULED = "scheduled", "Scheduled" SCHEDULED = "scheduled", "Scheduled"

View File

@@ -0,0 +1,168 @@
from __future__ import annotations
import json
import urllib.error
import urllib.request
from dataclasses import dataclass
from typing import Any
from django.conf import settings
from django.core.mail import send_mail
from django.utils import timezone
from .models import BackupRun, NotificationDelivery, NotificationTarget
TERMINAL_RUN_STATUSES = {
BackupRun.Status.SUCCESS,
BackupRun.Status.WARNING,
BackupRun.Status.FAILED,
BackupRun.Status.CANCELLED,
}
@dataclass(frozen=True)
class DeliveryResult:
target: NotificationTarget
delivery: NotificationDelivery
sent: bool
def notify_backup_run_completed(run: BackupRun) -> list[DeliveryResult]:
if run.status not in TERMINAL_RUN_STATUSES:
return []
targets = [target for target in NotificationTarget.objects.filter(enabled=True) if _target_wants_status(target, run.status)]
return [_notify_target(target=target, run=run) for target in targets]
def _target_wants_status(target: NotificationTarget, status: str) -> bool:
statuses = target.statuses
if not isinstance(statuses, list):
return False
return status in {str(item) for item in statuses}
def _notify_target(*, target: NotificationTarget, run: BackupRun) -> DeliveryResult:
payload = _run_payload(run)
delivery, created = NotificationDelivery.objects.get_or_create(
target=target,
run=run,
defaults={
"status": NotificationDelivery.Status.SKIPPED,
"payload": payload,
},
)
if not created:
return DeliveryResult(target=target, delivery=delivery, sent=False)
try:
if target.channel == NotificationTarget.Channel.EMAIL:
_send_email(target=target, run=run, payload=payload)
elif target.channel == NotificationTarget.Channel.WEBHOOK:
_send_webhook(target=target, payload=payload)
else:
raise ValueError(f"Unsupported notification channel: {target.channel}")
except Exception as exc:
delivery.status = NotificationDelivery.Status.FAILED
delivery.error = str(exc)
delivery.save(update_fields=["status", "error"])
target.last_status = NotificationDelivery.Status.FAILED
target.last_error = str(exc)
target.save(update_fields=["last_status", "last_error", "updated_at"])
return DeliveryResult(target=target, delivery=delivery, sent=False)
delivery.status = NotificationDelivery.Status.SENT
delivery.save(update_fields=["status"])
target.last_status = NotificationDelivery.Status.SENT
target.last_error = ""
target.last_sent_at = timezone.now()
target.save(update_fields=["last_status", "last_error", "last_sent_at", "updated_at"])
return DeliveryResult(target=target, delivery=delivery, sent=True)
def _send_email(*, target: NotificationTarget, run: BackupRun, payload: dict[str, Any]) -> None:
recipients = [line.strip() for line in target.email_to.replace(",", "\n").splitlines() if line.strip()]
if not recipients:
raise ValueError("Email notification target has no recipients.")
subject = f"pobsync {run.status}: {run.host.host} run {run.id}"
message = _email_message(payload)
from_email = getattr(settings, "DEFAULT_FROM_EMAIL", "") or "pobsync@localhost"
sent = send_mail(subject, message, from_email, recipients, fail_silently=False)
if sent == 0:
raise ValueError("Django email backend reported zero sent messages.")
def _send_webhook(*, target: NotificationTarget, payload: dict[str, Any]) -> None:
if not target.webhook_url:
raise ValueError("Webhook notification target has no URL.")
headers = {"Content-Type": "application/json", **_string_headers(target.webhook_headers)}
request = urllib.request.Request(
target.webhook_url,
data=json.dumps(payload).encode("utf-8"),
headers=headers,
method="POST",
)
try:
with urllib.request.urlopen(request, timeout=10) as response:
if response.status >= 400:
raise ValueError(f"Webhook returned HTTP {response.status}.")
except urllib.error.HTTPError as exc:
raise ValueError(f"Webhook returned HTTP {exc.code}.") from exc
def _string_headers(headers: object) -> dict[str, str]:
if not isinstance(headers, dict):
return {}
return {str(key): str(value) for key, value in headers.items() if str(key).strip()}
def _run_payload(run: BackupRun) -> dict[str, Any]:
result = run.result if isinstance(run.result, dict) else {}
failure = result.get("failure") if isinstance(result.get("failure"), dict) else {}
prune = result.get("prune") if isinstance(result.get("prune"), dict) else {}
return {
"event": "backup_run.completed",
"run": {
"id": run.id,
"host": run.host.host,
"type": run.run_type,
"status": run.status,
"started_at": run.started_at.isoformat() if run.started_at else None,
"ended_at": run.ended_at.isoformat() if run.ended_at else None,
"snapshot": run.snapshot_path,
"rsync_exit_code": run.rsync_exit_code,
},
"failure": {
"category": failure.get("category"),
"message": failure.get("message") or result.get("error"),
"hint": failure.get("hint"),
},
"prune": {
"ok": prune.get("ok") if prune else None,
"error": prune.get("error") if prune else "",
},
}
def _email_message(payload: dict[str, Any]) -> str:
run = payload["run"]
lines = [
f"Host: {run['host']}",
f"Run: {run['id']}",
f"Type: {run['type']}",
f"Status: {run['status']}",
f"Started: {run['started_at'] or '-'}",
f"Ended: {run['ended_at'] or '-'}",
f"Snapshot: {run['snapshot'] or '-'}",
f"Rsync exit code: {run['rsync_exit_code'] if run['rsync_exit_code'] is not None else '-'}",
]
failure = payload.get("failure") if isinstance(payload.get("failure"), dict) else {}
if failure.get("message"):
lines.extend(["", f"Failure: {failure['message']}"])
prune = payload.get("prune") if isinstance(payload.get("prune"), dict) else {}
if prune.get("error"):
lines.extend(["", f"Retention: {prune['error']}"])
return "\n".join(lines)

View File

@@ -97,9 +97,7 @@ def run_remote_preflight(host: HostConfig, *, timeout_seconds: int = 20) -> dict
*ssh_cmd, *ssh_cmd,
"-oBatchMode=yes", "-oBatchMode=yes",
target, target,
"sh", _remote_shell_command(f"command -v {shlex.quote(rsync_binary)} >/dev/null"),
"-lc",
f"command -v {shlex.quote(rsync_binary)} >/dev/null",
], ],
timeout_seconds=timeout_seconds, timeout_seconds=timeout_seconds,
), ),
@@ -109,9 +107,7 @@ def run_remote_preflight(host: HostConfig, *, timeout_seconds: int = 20) -> dict
*ssh_cmd, *ssh_cmd,
"-oBatchMode=yes", "-oBatchMode=yes",
target, target,
"sh", _remote_shell_command(f"test -e {shlex.quote(source_root)} && test -r {shlex.quote(source_root)}"),
"-lc",
f"test -e {shlex.quote(source_root)} && test -r {shlex.quote(source_root)}",
], ],
timeout_seconds=timeout_seconds, timeout_seconds=timeout_seconds,
), ),
@@ -129,6 +125,10 @@ def run_remote_preflight(host: HostConfig, *, timeout_seconds: int = 20) -> dict
return result return result
def _remote_shell_command(script: str) -> str:
return f"sh -lc {shlex.quote(script)}"
def effective_host_config_preview(host: HostConfig, global_config: GlobalConfig) -> dict[str, Any]: def effective_host_config_preview(host: HostConfig, global_config: GlobalConfig) -> dict[str, Any]:
config = build_effective_config(global_config_object_data(global_config), host_config_object_data(host)) config = build_effective_config(global_config_object_data(global_config), host_config_object_data(host))
credential = host.ssh_credential or global_config.default_ssh_credential credential = host.ssh_credential or global_config.default_ssh_credential

View File

@@ -23,7 +23,7 @@ def run_sql_retention_plan(*, host: str, kind: str, protect_bases: bool) -> dict
host_config = _enabled_host_config(host) host_config = _enabled_host_config(host)
retention = _retention_for_host(host_config) retention = _retention_for_host(host_config)
snapshots = _snapshots_for_retention(host_config=host_config, kind=kind) snapshots = _snapshots_for_retention(host_config=host_config, kind=kind)
incomplete_snapshots = _incomplete_snapshots_for_host(host_config) incomplete_items = _incomplete_snapshot_items_for_host(host_config)
plan = build_retention_plan( plan = build_retention_plan(
snapshots=snapshots, snapshots=snapshots,
@@ -49,10 +49,9 @@ def run_sql_retention_plan(*, host: str, kind: str, protect_bases: bool) -> dict
"keep": sorted(keep), "keep": sorted(keep),
"keep_items": [_snapshot_to_item(snapshot, reasons=reasons.get(snapshot.dirname, [])) for snapshot in keep_items], "keep_items": [_snapshot_to_item(snapshot, reasons=reasons.get(snapshot.dirname, [])) for snapshot in keep_items],
"delete": [_snapshot_to_item(snapshot, reasons=["outside retention policy"]) for snapshot in delete], "delete": [_snapshot_to_item(snapshot, reasons=["outside retention policy"]) for snapshot in delete],
"incomplete": [ "incomplete": incomplete_items,
_snapshot_to_item(snapshot, reasons=["incomplete snapshot; excluded from retention cleanup"]) "incomplete_reviewed_count": sum(1 for item in incomplete_items if item["reviewed"]),
for snapshot in incomplete_snapshots "incomplete_unreviewed_count": sum(1 for item in incomplete_items if not item["reviewed"]),
],
"reasons": reasons, "reasons": reasons,
} }
@@ -103,6 +102,7 @@ def run_sql_retention_apply(
raise ConfigError(f"Refusing to delete unsupported snapshot kind: {snap_kind!r}") raise ConfigError(f"Refusing to delete unsupported snapshot kind: {snap_kind!r}")
path = _snapshot_delete_path(path=Path(snap_path), dirname=dirname) path = _snapshot_delete_path(path=Path(snap_path), dirname=dirname)
_validate_snapshot_delete_path(host=host, kind=snap_kind, path=path, dirname=dirname)
reason = str(item.get("reason") or "outside retention policy") reason = str(item.get("reason") or "outside retention policy")
if not path.exists(): if not path.exists():
actions.append(f"skip missing {snap_kind}/{dirname}") actions.append(f"skip missing {snap_kind}/{dirname}")
@@ -163,9 +163,15 @@ def run_incomplete_cleanup(
def _do_cleanup() -> dict[str, Any]: def _do_cleanup() -> dict[str, Any]:
host_config = _enabled_host_config(host) host_config = _enabled_host_config(host)
unreviewed_count = _unreviewed_incomplete_count(host_config)
if unreviewed_count:
raise ConfigError(
f"Refusing to delete {unreviewed_count} incomplete snapshot(s) that have not been reviewed."
)
incomplete_list = [ incomplete_list = [
_snapshot_to_item(snapshot, reasons=["manual incomplete cleanup"]) _snapshot_to_item(snapshot, reasons=["manual incomplete cleanup"])
for snapshot in _incomplete_snapshots_for_host(host_config) for snapshot in _reviewed_incomplete_snapshots_for_host(host_config)
] ]
if max_delete == 0 and len(incomplete_list) > 0: if max_delete == 0 and len(incomplete_list) > 0:
raise ConfigError("Incomplete cleanup blocked by --max-delete=0") raise ConfigError("Incomplete cleanup blocked by --max-delete=0")
@@ -252,15 +258,39 @@ def _snapshots_for_retention(*, host_config: HostConfig, kind: str) -> list[Snap
return [_snapshot_from_record(record) for record in records] return [_snapshot_from_record(record) for record in records]
def _incomplete_snapshots_for_host(host_config: HostConfig) -> list[Snapshot]: def _incomplete_snapshot_items_for_host(host_config: HostConfig) -> list[dict[str, Any]]:
records = ( records = (
SnapshotRecord.objects.filter(host=host_config, kind=SnapshotRecord.Kind.INCOMPLETE) SnapshotRecord.objects.filter(host=host_config, kind=SnapshotRecord.Kind.INCOMPLETE)
.select_related("base") .select_related("base")
.order_by("-started_at", "dirname") .order_by("-started_at", "dirname")
) )
return [
_snapshot_record_to_item(record, reasons=["incomplete snapshot; excluded from retention cleanup"])
for record in records
]
def _reviewed_incomplete_snapshots_for_host(host_config: HostConfig) -> list[Snapshot]:
records = (
SnapshotRecord.objects.filter(
host=host_config,
kind=SnapshotRecord.Kind.INCOMPLETE,
reviewed_at__isnull=False,
)
.select_related("base")
.order_by("-started_at", "dirname")
)
return [_snapshot_from_record(record) for record in records] return [_snapshot_from_record(record) for record in records]
def _unreviewed_incomplete_count(host_config: HostConfig) -> int:
return SnapshotRecord.objects.filter(
host=host_config,
kind=SnapshotRecord.Kind.INCOMPLETE,
reviewed_at__isnull=True,
).count()
def _snapshot_from_record(record: SnapshotRecord) -> Snapshot: def _snapshot_from_record(record: SnapshotRecord) -> Snapshot:
return Snapshot( return Snapshot(
kind=record.kind, kind=record.kind,
@@ -300,6 +330,14 @@ def _snapshot_to_item(snapshot: Snapshot, *, reasons: list[str]) -> dict[str, An
} }
def _snapshot_record_to_item(record: SnapshotRecord, *, reasons: list[str]) -> dict[str, Any]:
item = _snapshot_to_item(_snapshot_from_record(record), reasons=reasons)
item["reviewed"] = record.reviewed_at is not None
item["reviewed_at"] = record.reviewed_at.isoformat() if record.reviewed_at else ""
item["reviewed_by"] = record.reviewed_by
return item
def _snapshot_delete_path(*, path: Path, dirname: str) -> Path: def _snapshot_delete_path(*, path: Path, dirname: str) -> Path:
if path.name == "data" and path.parent.name == dirname: if path.name == "data" and path.parent.name == dirname:
return path.parent return path.parent
@@ -339,14 +377,55 @@ def _validate_incomplete_delete_path(*, host: str, path: Path, dirname: str) ->
raise ConfigError(f"Refusing to delete incomplete snapshot outside host backup root: {path}") raise ConfigError(f"Refusing to delete incomplete snapshot outside host backup root: {path}")
def _validate_snapshot_delete_path(*, host: str, kind: str, path: Path, dirname: str) -> None:
if kind not in {SnapshotRecord.Kind.SCHEDULED, SnapshotRecord.Kind.MANUAL}:
raise ConfigError(f"Refusing to delete unsupported snapshot kind: {kind!r}")
path_parts = path.parts
if path.name != dirname or kind not in path_parts or host not in path_parts:
raise ConfigError(f"Refusing to delete unexpected snapshot path: {path}")
kind_index = path_parts.index(kind)
if kind_index == 0 or path_parts[kind_index - 1] != host:
raise ConfigError(f"Refusing to delete snapshot outside host backup root: {path}")
def _remove_snapshot_tree(path: Path) -> None: def _remove_snapshot_tree(path: Path) -> None:
_make_directories_user_writable(path) _make_snapshot_tree_user_removable(path)
shutil.rmtree(path) shutil.rmtree(path, onexc=_retry_remove_with_user_permissions)
def _make_directories_user_writable(path: Path) -> None: def _make_snapshot_tree_user_removable(path: Path) -> None:
for directory in [path, *[child for child in path.rglob("*") if child.is_dir() and not child.is_symlink()]]: stack = [path]
mode = directory.stat().st_mode while stack:
if mode & stat.S_IWUSR: directory = stack.pop()
if directory.is_symlink():
continue continue
directory.chmod(mode | stat.S_IWUSR) _make_path_user_removable(directory)
try:
children = list(directory.iterdir())
except OSError:
continue
for child in children:
if child.is_dir() and not child.is_symlink():
stack.append(child)
def _retry_remove_with_user_permissions(function: Any, path: str, excinfo: BaseException) -> None:
failed_path = Path(path)
_make_path_user_removable(failed_path)
function(path)
def _make_path_user_removable(path: Path) -> None:
try:
mode = path.stat().st_mode
except OSError:
return
wanted = stat.S_IRUSR | stat.S_IWUSR
if path.is_dir() and not path.is_symlink():
wanted |= stat.S_IXUSR
if mode & wanted == wanted:
return
try:
path.chmod(mode | wanted)
except OSError:
return

View File

@@ -5,12 +5,13 @@ from typing import Any, Iterable
from django.utils import timezone from django.utils import timezone
from pobsync.run_stats import filesystem_capacity from pobsync.run_stats import filesystem_capacity, tree_usage
from .models import BackupRun, GlobalConfig, HostConfig, SnapshotRecord from .models import BackupRun, GlobalConfig, HostConfig, SnapshotRecord
def collect_dashboard_stats(*, hosts: Iterable[HostConfig], global_config: GlobalConfig | None) -> dict[str, Any]: def collect_dashboard_stats(*, hosts: Iterable[HostConfig], global_config: GlobalConfig | None) -> dict[str, Any]:
hosts = list(hosts)
runs = list( runs = list(
BackupRun.objects.select_related("host", "snapshot") BackupRun.objects.select_related("host", "snapshot")
.filter(status__in=_COMPLETED_BACKUP_STATUSES) .filter(status__in=_COMPLETED_BACKUP_STATUSES)
@@ -21,6 +22,7 @@ def collect_dashboard_stats(*, hosts: Iterable[HostConfig], global_config: Globa
for host in hosts: for host in hosts:
host.stats_summary = collect_host_stats(host=host) host.stats_summary = collect_host_stats(host=host)
backup_data = _sum_backup_data_by_kind(host.stats_summary["backup_data"] for host in hosts)
literal_values = [_int_at(run, "rsync", "literal_data_bytes") for run in real_runs] literal_values = [_int_at(run, "rsync", "literal_data_bytes") for run in real_runs]
literal_values = [value for value in literal_values if value is not None] literal_values = [value for value in literal_values if value is not None]
@@ -51,6 +53,7 @@ def collect_dashboard_stats(*, hosts: Iterable[HostConfig], global_config: Globa
"estimated_runs_until_full": int(available / avg_literal) if available and avg_literal else None, "estimated_runs_until_full": int(available / avg_literal) if available and avg_literal else None,
"estimated_days_until_full": int(available / daily_literal) if available and daily_literal else None, "estimated_days_until_full": int(available / daily_literal) if available and daily_literal else None,
"capacity": capacity, "capacity": capacity,
"backup_data": backup_data,
} }
@@ -61,6 +64,7 @@ def collect_host_stats(*, host: HostConfig, limit: int = 8) -> dict[str, Any]:
trend_runs = [run for run in completed_real_runs if run["has_stats"]][:limit] trend_runs = [run for run in completed_real_runs if run["has_stats"]][:limit]
latest_snapshot = host.snapshots.order_by("-started_at", "-discovered_at", "-id").first() latest_snapshot = host.snapshots.order_by("-started_at", "-discovered_at", "-id").first()
latest_snapshot_stats = _snapshot_summary(latest_snapshot) if latest_snapshot else {} latest_snapshot_stats = _snapshot_summary(latest_snapshot) if latest_snapshot else {}
backup_data = _backup_data_by_kind(host)
literal_values = [_int_at(run, "rsync", "literal_data_bytes") for run in trend_runs] literal_values = [_int_at(run, "rsync", "literal_data_bytes") for run in trend_runs]
literal_values = [value for value in literal_values if value is not None] literal_values = [value for value in literal_values if value is not None]
@@ -75,6 +79,7 @@ def collect_host_stats(*, host: HostConfig, limit: int = 8) -> dict[str, Any]:
"latest_good_run": _first_run_with_status(real_runs, {BackupRun.Status.SUCCESS}), "latest_good_run": _first_run_with_status(real_runs, {BackupRun.Status.SUCCESS}),
"latest_problem_run": _first_run_with_status(real_runs, {BackupRun.Status.WARNING, BackupRun.Status.FAILED}), "latest_problem_run": _first_run_with_status(real_runs, {BackupRun.Status.WARNING, BackupRun.Status.FAILED}),
"latest_snapshot": latest_snapshot_stats, "latest_snapshot": latest_snapshot_stats,
"backup_data": backup_data,
"avg_literal_data_bytes": _average(literal_values), "avg_literal_data_bytes": _average(literal_values),
"avg_daily_literal_data_bytes": _average_daily_literal(trend_runs), "avg_daily_literal_data_bytes": _average_daily_literal(trend_runs),
"total_literal_data_bytes": sum(literal_values), "total_literal_data_bytes": sum(literal_values),
@@ -102,6 +107,65 @@ def _run_summary(run: BackupRun) -> dict[str, Any]:
} }
def _backup_data_by_kind(host: HostConfig) -> dict[str, Any]:
rows: dict[str, dict[str, int]] = {
SnapshotRecord.Kind.SCHEDULED: _empty_snapshot_data_row(),
SnapshotRecord.Kind.MANUAL: _empty_snapshot_data_row(),
SnapshotRecord.Kind.INCOMPLETE: _empty_snapshot_data_row(),
}
total = _empty_snapshot_data_row()
for snapshot in host.snapshots.all():
summary = _snapshot_summary(snapshot)
row = rows.setdefault(snapshot.kind, _empty_snapshot_data_row())
allocated = summary.get("allocated_size_bytes") or summary.get("apparent_size_bytes") or 0
apparent = summary.get("apparent_size_bytes") or 0
unique_apparent = summary.get("unique_apparent_size_bytes") or 0
row["count"] += 1
row["allocated_size_bytes"] += int(allocated)
row["apparent_size_bytes"] += int(apparent)
row["unique_apparent_size_bytes"] += int(unique_apparent)
total["count"] += 1
total["allocated_size_bytes"] += int(allocated)
total["apparent_size_bytes"] += int(apparent)
total["unique_apparent_size_bytes"] += int(unique_apparent)
return {
"scheduled": rows[SnapshotRecord.Kind.SCHEDULED],
"manual": rows[SnapshotRecord.Kind.MANUAL],
"incomplete": rows[SnapshotRecord.Kind.INCOMPLETE],
"total": total,
}
def _empty_snapshot_data_row() -> dict[str, int]:
return {
"count": 0,
"allocated_size_bytes": 0,
"apparent_size_bytes": 0,
"unique_apparent_size_bytes": 0,
}
def _sum_backup_data_by_kind(rows: Iterable[dict[str, dict[str, int]]]) -> dict[str, dict[str, int]]:
total_rows: dict[str, dict[str, int]] = {
"scheduled": _empty_snapshot_data_row(),
"manual": _empty_snapshot_data_row(),
"incomplete": _empty_snapshot_data_row(),
"total": _empty_snapshot_data_row(),
}
for row in rows:
for kind, values in row.items():
total_row = total_rows.setdefault(kind, _empty_snapshot_data_row())
total_row["count"] += values.get("count", 0)
total_row["allocated_size_bytes"] += values.get("allocated_size_bytes", 0)
total_row["apparent_size_bytes"] += values.get("apparent_size_bytes", 0)
total_row["unique_apparent_size_bytes"] += values.get("unique_apparent_size_bytes", 0)
return total_rows
def _snapshot_summary(snapshot: SnapshotRecord | None) -> dict[str, Any]: def _snapshot_summary(snapshot: SnapshotRecord | None) -> dict[str, Any]:
if snapshot is None: if snapshot is None:
return {} return {}
@@ -109,18 +173,43 @@ def _snapshot_summary(snapshot: SnapshotRecord | None) -> dict[str, Any]:
stats = metadata.get("stats") if isinstance(metadata.get("stats"), dict) else {} stats = metadata.get("stats") if isinstance(metadata.get("stats"), dict) else {}
storage = stats.get("storage") if isinstance(stats.get("storage"), dict) else {} storage = stats.get("storage") if isinstance(stats.get("storage"), dict) else {}
snapshot_storage = storage.get("snapshot") if isinstance(storage.get("snapshot"), dict) else {} snapshot_storage = storage.get("snapshot") if isinstance(storage.get("snapshot"), dict) else {}
if snapshot.kind == SnapshotRecord.Kind.INCOMPLETE:
snapshot_storage = _snapshot_storage_from_filesystem(snapshot)
else:
has_recorded_size = (
_int_at(snapshot_storage, "allocated_size_bytes") is not None
or _int_at(snapshot_storage, "apparent_size_bytes") is not None
)
if not has_recorded_size:
snapshot_storage = _snapshot_storage_from_filesystem(snapshot)
apparent_size = _int_at(snapshot_storage, "apparent_size_bytes")
hardlinked_apparent = _int_at(snapshot_storage, "hardlinked_apparent_size_bytes") or 0
return { return {
"id": snapshot.id, "id": snapshot.id,
"dirname": snapshot.dirname, "dirname": snapshot.dirname,
"kind": snapshot.kind, "kind": snapshot.kind,
"status": snapshot.status, "status": snapshot.status,
"started_at": snapshot.started_at, "started_at": snapshot.started_at,
"apparent_size_bytes": _int_at(snapshot_storage, "apparent_size_bytes"), "apparent_size_bytes": apparent_size,
"allocated_size_bytes": _int_at(snapshot_storage, "allocated_size_bytes"), "allocated_size_bytes": _int_at(snapshot_storage, "allocated_size_bytes"),
"hardlinked_files": _int_at(snapshot_storage, "hardlinked_files"), "hardlinked_files": _int_at(snapshot_storage, "hardlinked_files"),
"hardlinked_apparent_size_bytes": hardlinked_apparent,
"unique_apparent_size_bytes": max((apparent_size or 0) - hardlinked_apparent, 0),
} }
def _snapshot_storage_from_filesystem(snapshot: SnapshotRecord) -> dict[str, Any]:
if not snapshot.path:
return {}
snapshot_path = Path(snapshot.path)
data_path = snapshot_path / "data"
if snapshot_path.name == "data":
return tree_usage(snapshot_path)
if data_path.exists():
return tree_usage(data_path)
return tree_usage(snapshot_path)
def _is_real_run(run: BackupRun) -> bool: def _is_real_run(run: BackupRun) -> bool:
result = run.result if isinstance(run.result, dict) else {} result = run.result if isinstance(run.result, dict) else {}
if result.get("dry_run") is True: if result.get("dry_run") is True:

View File

@@ -80,6 +80,28 @@
padding-left: 0; padding-left: 0;
} }
nav strong a:hover { background: transparent; } nav strong a:hover { background: transparent; }
nav a[aria-current="page"] {
background: #eaf3fb;
color: var(--link-strong);
font-weight: 720;
}
.nav-primary,
.nav-secondary {
align-items: center;
display: flex;
flex-wrap: wrap;
gap: 4px;
}
.nav-secondary {
justify-content: flex-end;
}
.nav-secondary a {
font-size: 13px;
}
.nav-user {
margin-left: 6px;
white-space: nowrap;
}
nav .spacer { flex: 1; } nav .spacer { flex: 1; }
main { main {
max-width: 1180px; max-width: 1180px;
@@ -182,6 +204,19 @@
.metric-link:focus-visible { .metric-link:focus-visible {
outline: 3px solid #93c5fd; outline: 3px solid #93c5fd;
outline-offset: 2px; outline-offset: 2px;
}
.dashboard-summary-grid {
grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));
}
.dashboard-summary-grid .metric {
min-height: 78px;
}
.dashboard-summary-grid .metric .value {
font-size: 25px;
}
.dashboard-trends-panel,
.dashboard-hosts-panel {
overflow: visible;
} }
.panel { .panel {
margin-bottom: 18px; margin-bottom: 18px;
@@ -240,7 +275,31 @@
.status.skipped { color: var(--muted); background: #f7f9fb; } .status.skipped { color: var(--muted); background: #f7f9fb; }
.stack { display: grid; gap: 5px; } .stack { display: grid; gap: 5px; }
.stack.spaced { margin-bottom: 14px; } .stack.spaced { margin-bottom: 14px; }
.detail-list {
display: grid;
gap: 8px 14px;
grid-template-columns: minmax(120px, max-content) minmax(0, 1fr);
margin: 0 0 12px;
}
.detail-list dt {
color: var(--muted);
font-size: 12px;
font-weight: 750;
text-transform: uppercase;
}
.detail-list dd {
margin: 0;
min-width: 0;
overflow-wrap: anywhere;
}
.two-col { display: grid; gap: 18px; grid-template-columns: minmax(0, 1fr) minmax(0, 1fr); } .two-col { display: grid; gap: 18px; grid-template-columns: minmax(0, 1fr) minmax(0, 1fr); }
.panel-grid {
display: grid;
gap: 18px;
grid-template-columns: repeat(auto-fit, minmax(280px, 1fr));
margin-bottom: 18px;
}
.panel-grid .panel { margin-bottom: 0; }
.actions { .actions {
align-items: center; align-items: center;
display: flex; display: flex;
@@ -273,6 +332,22 @@
} }
button.secondary:hover, button.secondary:hover,
.button-link.secondary:hover { background: #eef3f8; } .button-link.secondary:hover { background: #eef3f8; }
button.danger,
.button-link.danger {
background: var(--failed);
border-color: var(--failed);
color: #fff;
}
button.danger:hover,
.button-link.danger:hover {
background: #842828;
border-color: #842828;
}
button.compact,
.button-link.compact {
font-size: 12px;
padding: 5px 8px;
}
button:disabled { button:disabled {
background: #d8dee6; background: #d8dee6;
border-color: #d8dee6; border-color: #d8dee6;
@@ -280,9 +355,196 @@
cursor: not-allowed; cursor: not-allowed;
} }
.inline-form { margin: 0; } .inline-form { margin: 0; }
.status-overview { .dashboard-priority-grid {
align-items: start;
display: grid;
gap: 14px;
grid-template-columns: repeat(2, minmax(0, 1fr));
margin-bottom: 20px;
}
.priority-panel {
display: grid;
gap: 12px;
margin-bottom: 0;
min-width: 0;
overflow: visible;
}
.priority-panel > h2:first-child {
flex-wrap: wrap;
margin-bottom: 0;
}
.action-list,
.activity-list,
.schedule-list {
display: grid; display: grid;
gap: 8px; gap: 8px;
}
.record-list {
display: grid;
gap: 10px;
}
.record-card {
border: 1px solid var(--border);
border-radius: var(--radius);
display: grid;
gap: 10px;
padding: 12px;
}
.record-card-header {
align-items: start;
display: flex;
gap: 12px;
justify-content: space-between;
}
.record-title {
display: grid;
gap: 3px;
min-width: 0;
}
.record-title a {
font-weight: 750;
overflow-wrap: anywhere;
}
.record-facts {
display: grid;
gap: 8px 16px;
grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));
}
.record-fact {
display: grid;
gap: 2px;
min-width: 0;
}
.record-fact .label {
color: var(--muted);
font-size: 11px;
font-weight: 700;
letter-spacing: 0.04em;
text-transform: uppercase;
}
.record-fact strong,
.record-fact span {
overflow-wrap: anywhere;
}
.action-row,
.activity-row,
.schedule-row {
align-items: start;
border: 1px solid var(--border);
border-radius: 7px;
color: inherit;
display: grid;
gap: 9px;
padding: 10px;
text-decoration: none;
transition: border-color 0.15s ease, background 0.15s ease, box-shadow 0.15s ease;
}
.action-row,
.activity-row {
grid-template-columns: max-content minmax(0, 1fr);
}
.activity-row .status {
justify-self: start;
}
.schedule-row {
grid-template-columns: minmax(0, 1fr) max-content;
}
.action-row:hover,
.activity-row:hover,
.schedule-row:hover {
background: var(--panel-subtle);
border-color: var(--border-strong);
box-shadow: var(--shadow-sm);
}
.action-row.failed { border-color: #e8b4b4; background: #fff7f7; }
.action-row.warning { border-color: #e7cf8a; background: #fffaf0; }
.action-row span:not(.status),
.activity-row span:not(.status),
.schedule-row span {
display: grid;
gap: 2px;
min-width: 0;
}
.action-row strong,
.action-row .muted,
.activity-row strong,
.activity-row .muted,
.schedule-row strong,
.schedule-row .muted {
overflow-wrap: anywhere;
}
.schedule-time {
justify-items: end;
text-align: right;
}
.storage-priority {
display: grid;
gap: 12px;
}
.storage-priority .label {
color: var(--muted);
font-size: 12px;
font-weight: 650;
text-transform: uppercase;
}
.storage-priority .value {
font-size: 27px;
font-weight: 760;
line-height: 1.15;
margin-top: 4px;
}
.storage-priority-facts {
display: grid;
gap: 8px;
}
.storage-priority-facts > div {
align-items: baseline;
border-top: 1px solid var(--border);
display: flex;
gap: 10px;
justify-content: space-between;
padding-top: 8px;
}
.storage-priority-facts strong {
text-align: right;
overflow-wrap: anywhere;
}
.host-control-grid {
display: grid;
gap: 14px;
grid-template-columns: minmax(280px, 1.25fr) repeat(3, minmax(220px, 1fr));
margin-bottom: 20px;
}
.host-control-panel {
display: grid;
gap: 12px;
margin-bottom: 0;
}
.host-control-panel > h2:first-child { margin-bottom: 0; }
.host-control-primary {
display: grid;
gap: 8px;
}
.host-control-meta {
display: grid;
gap: 6px;
}
.host-control-meta > div {
align-items: baseline;
border-top: 1px solid var(--border);
display: flex;
gap: 10px;
justify-content: space-between;
padding-top: 7px;
}
.host-control-meta .label {
color: var(--muted);
font-size: 12px;
font-weight: 650;
text-transform: uppercase;
}
.host-control-meta strong {
text-align: right;
} }
.status-summary { .status-summary {
align-items: center; align-items: center;
@@ -307,12 +569,6 @@
box-shadow: var(--shadow-sm); box-shadow: var(--shadow-sm);
transform: translateY(-1px); transform: translateY(-1px);
} }
.status-summary .summary-action {
color: var(--muted-strong);
font-size: 12px;
font-weight: 650;
margin-left: auto;
}
.operator-state { .operator-state {
align-items: center; align-items: center;
display: flex; display: flex;
@@ -320,6 +576,15 @@
gap: 8px; gap: 8px;
margin-bottom: 14px; margin-bottom: 14px;
} }
.refresh-controls {
align-items: center;
display: flex;
gap: 14px;
justify-content: space-between;
}
.refresh-controls h2 {
margin-bottom: 4px;
}
.trend-bars { .trend-bars {
display: grid; display: grid;
gap: 5px; gap: 5px;
@@ -389,6 +654,7 @@
border: 1px solid var(--border); border: 1px solid var(--border);
border-radius: var(--radius); border-radius: var(--radius);
box-shadow: var(--shadow-sm); box-shadow: var(--shadow-sm);
min-width: 0;
padding: 16px; padding: 16px;
} }
.host-card:hover { .host-card:hover {
@@ -422,8 +688,8 @@
} }
.host-card-layout { .host-card-layout {
display: grid; display: grid;
gap: 24px; gap: 18px;
grid-template-columns: minmax(0, 2fr) minmax(260px, 1fr); grid-template-columns: minmax(0, 1.7fr) minmax(240px, 0.9fr);
} }
.host-card-section { .host-card-section {
align-content: start; align-content: start;
@@ -440,7 +706,7 @@
.host-card-timeline { .host-card-timeline {
display: grid; display: grid;
gap: 16px 22px; gap: 16px 22px;
grid-template-columns: repeat(auto-fit, minmax(210px, 1fr)); grid-template-columns: repeat(auto-fit, minmax(180px, 1fr));
} }
.host-card-stats { .host-card-stats {
align-content: start; align-content: start;
@@ -466,6 +732,10 @@
.host-card-item .value { .host-card-item .value {
overflow-wrap: anywhere; overflow-wrap: anywhere;
} }
.host-card-item .value a {
overflow-wrap: anywhere;
word-break: break-word;
}
.host-card-stat { .host-card-stat {
display: grid; display: grid;
gap: 3px; gap: 3px;
@@ -496,6 +766,18 @@
margin-top: 14px; margin-top: 14px;
padding: 10px; padding: 10px;
} }
.host-card-warning > * {
min-width: 0;
}
.host-card-actions {
align-items: center;
border-top: 1px solid var(--border);
display: flex;
flex-wrap: wrap;
gap: 8px;
margin-top: 14px;
padding-top: 12px;
}
.messages { display: grid; gap: 8px; margin-bottom: 18px; } .messages { display: grid; gap: 8px; margin-bottom: 18px; }
.message { .message {
background: var(--panel); background: var(--panel);
@@ -507,6 +789,30 @@
.message.error { border-color: #e8b4b4; background: #fff0f0; color: var(--failed); } .message.error { border-color: #e8b4b4; background: #fff0f0; color: var(--failed); }
.message.warning { border-color: #e7cf8a; background: #fff8df; color: var(--running); } .message.warning { border-color: #e7cf8a; background: #fff8df; color: var(--running); }
.form-grid { display: grid; gap: 15px; max-width: 720px; } .form-grid { display: grid; gap: 15px; max-width: 720px; }
.filter-form {
align-items: end;
display: grid;
gap: 15px;
grid-template-columns: repeat(auto-fit, minmax(180px, 1fr));
max-width: none;
}
.form-actions {
align-items: center;
border-top: 1px solid var(--border);
display: flex;
flex-wrap: wrap;
gap: 10px;
margin-top: 4px;
padding-top: 15px;
}
.form-actions .button-link.secondary { margin-left: auto; }
.filter-form .form-actions {
border-top: 0;
justify-content: flex-end;
margin-top: 0;
padding-top: 0;
}
.filter-form .form-actions .button-link.secondary { margin-left: 0; }
.field { display: grid; gap: 6px; } .field { display: grid; gap: 6px; }
.field label { font-weight: 700; } .field label { font-weight: 700; }
.field input[type="text"], .field input[type="number"], .field select, .field textarea { .field input[type="text"], .field input[type="number"], .field select, .field textarea {
@@ -552,34 +858,104 @@
padding: 8px 0; padding: 8px 0;
} }
nav strong { flex-basis: 100%; margin-right: 0; } nav strong { flex-basis: 100%; margin-right: 0; }
.nav-secondary {
justify-content: flex-start;
}
.nav-user {
margin-left: 0;
}
nav .spacer { display: none; } nav .spacer { display: none; }
.page-header { .page-header {
align-items: stretch; align-items: stretch;
display: grid; display: grid;
} }
.page-header .actions { justify-content: flex-start; } .page-header .actions { justify-content: flex-start; }
.two-col { grid-template-columns: 1fr; } .two-col,
.panel-grid { grid-template-columns: 1fr; }
.refresh-controls {
align-items: stretch;
display: grid;
}
.dashboard-priority-grid { grid-template-columns: 1fr; }
.host-control-grid { grid-template-columns: 1fr; }
.schedule-row { grid-template-columns: 1fr; }
.schedule-time { justify-items: start; text-align: left; }
.form-actions .button-link.secondary { margin-left: 0; }
.host-card-header { display: grid; } .host-card-header { display: grid; }
.host-card-status { justify-content: flex-start; max-width: none; } .host-card-status { justify-content: flex-start; max-width: none; }
.host-card-layout { grid-template-columns: 1fr; } .host-card-layout { grid-template-columns: 1fr; }
.host-card-stats { grid-template-columns: repeat(auto-fit, minmax(140px, 1fr)); } .host-card-stats { grid-template-columns: repeat(auto-fit, minmax(140px, 1fr)); }
.insight-grid { grid-template-columns: 1fr; } .insight-grid { grid-template-columns: 1fr; }
} }
@media (max-width: 1100px) {
.dashboard-priority-grid {
grid-template-columns: 1fr;
}
.dashboard-summary-grid {
grid-template-columns: repeat(3, minmax(0, 1fr));
}
.host-card-layout {
grid-template-columns: 1fr;
}
.host-card-status {
justify-content: flex-start;
max-width: none;
}
.schedule-row {
grid-template-columns: 1fr;
}
.schedule-time {
justify-items: start;
text-align: left;
}
}
@media (max-width: 560px) {
.dashboard-summary-grid {
grid-template-columns: repeat(2, minmax(0, 1fr));
}
.metric {
min-height: 76px;
padding: 12px;
}
.metric .value {
font-size: 24px;
}
.host-card {
padding: 13px;
}
.host-card-stats {
grid-template-columns: 1fr;
}
}
</style> </style>
</head> </head>
<body> <body>
<header> <header>
<nav> <nav>
<strong><a href="{% url 'dashboard' %}">pobsync</a></strong> <strong><a href="{% url 'dashboard' %}">pobsync</a></strong>
<a href="{% url 'admin:index' %}">Admin</a> <span class="nav-primary" aria-label="Primary navigation">
<a href="{% url 'ssh_credentials' %}">SSH Keys</a> <a href="{% url 'dashboard' %}" {% if request.resolver_match.url_name == "dashboard" %}aria-current="page"{% endif %}>Dashboard</a>
<a href="{% url 'self_check' %}">Self Check</a> <a href="{% url 'hosts_list' %}" {% if request.resolver_match.url_name == "hosts_list" or request.resolver_match.url_name == "host_detail" or request.resolver_match.url_name == "create_host_config" or request.resolver_match.url_name == "edit_host_config" or request.resolver_match.url_name == "edit_host_schedule" %}aria-current="page"{% endif %}>Hosts</a>
<a href="{% url 'logs' %}">Logs</a> {% if can_manage_control_panel %}
<a href="{% url 'purged_snapshots' %}">Purged</a> <a href="{% url 'ssh_credentials' %}" {% if request.resolver_match.url_name == "ssh_credentials" or request.resolver_match.url_name == "create_ssh_credential" or request.resolver_match.url_name == "generate_ssh_credential" or request.resolver_match.url_name == "edit_ssh_credential" %}aria-current="page"{% endif %}>SSH Keys</a>
<a href="{% url 'changelog' %}">Changelog</a> <a href="{% url 'notification_targets' %}" {% if request.resolver_match.url_name == "notification_targets" or request.resolver_match.url_name == "create_notification_target" or request.resolver_match.url_name == "edit_notification_target" %}aria-current="page"{% endif %}>Notifications</a>
<a href="/api/status/">Status API</a> <a href="{% url 'logs' %}" {% if request.resolver_match.url_name == "logs" %}aria-current="page"{% endif %}>Logs</a>
<a href="{% url 'updater' %}" {% if request.resolver_match.url_name == "updater" %}aria-current="page"{% endif %}>Updater</a>
{% endif %}
<a href="{% url 'purged_snapshots' %}" {% if request.resolver_match.url_name == "purged_snapshots" %}aria-current="page"{% endif %}>Purged</a>
</span>
<span class="spacer"></span> <span class="spacer"></span>
<span class="muted">{{ request.user.username }}</span> <span class="nav-secondary" aria-label="System navigation">
{% if can_manage_control_panel %}
<a href="{% url 'self_check' %}" {% if request.resolver_match.url_name == "self_check" %}aria-current="page"{% endif %}>Self Check</a>
{% endif %}
<a href="{% url 'changelog' %}" {% if request.resolver_match.url_name == "changelog" %}aria-current="page"{% endif %}>Changelog</a>
<a href="/api/status/">Status API</a>
{% if can_manage_control_panel %}
<a href="{% url 'admin:index' %}">Admin</a>
{% endif %}
</span>
<span class="muted nav-user">{{ request.user.username }}</span>
</nav> </nav>
</header> </header>
<main> <main>
@@ -592,5 +968,54 @@
{% endif %} {% endif %}
{% block content %}{% endblock %} {% block content %}{% endblock %}
</main> </main>
<script>
(() => {
const updateRefreshControls = (region) => {
const toggle = document.querySelector(`[data-refresh-toggle][data-refresh-target="${region.id}"]`);
const state = document.querySelector(`[data-refresh-state="${region.id}"]`);
const paused = region.dataset.refreshPaused === "true";
const active = region.dataset.refreshActive === "true";
if (state) state.textContent = paused ? "paused" : (active ? "on" : "off");
if (toggle) {
toggle.textContent = paused ? "Resume refresh" : "Pause refresh";
toggle.disabled = !active && !paused;
}
};
const refreshRegion = async (region) => {
if (region.dataset.refreshActive !== "true" || region.dataset.refreshPaused === "true" || document.hidden) return;
try {
const response = await fetch(region.dataset.refreshUrl, {
credentials: "same-origin",
headers: { "X-Requested-With": "XMLHttpRequest" },
});
if (!response.ok) return;
region.innerHTML = await response.text();
const refreshActive = response.headers.get("X-Pobsync-Refresh-Active");
if (refreshActive) region.dataset.refreshActive = refreshActive;
updateRefreshControls(region);
} catch (error) {
// Keep the current server-rendered content visible if a refresh fails.
}
};
document.addEventListener("click", (event) => {
const toggle = event.target.closest("[data-refresh-toggle]");
if (!toggle) return;
const region = document.getElementById(toggle.dataset.refreshTarget);
if (!region) return;
const paused = region.dataset.refreshPaused === "true";
region.dataset.refreshPaused = paused ? "false" : "true";
if (paused && region.dataset.refreshActive === "true") refreshRegion(region);
updateRefreshControls(region);
});
document.querySelectorAll("[data-refresh-url]").forEach((region) => {
const interval = Number.parseInt(region.dataset.refreshInterval || "5000", 10);
updateRefreshControls(region);
window.setInterval(() => refreshRegion(region), Number.isFinite(interval) ? interval : 5000);
});
})();
</script>
</body> </body>
</html> </html>

View File

@@ -9,12 +9,15 @@
<h1>Dashboard</h1> <h1>Dashboard</h1>
<div class="page-subtitle">Backup health, required action, storage pressure, and recent activity in one place.</div> <div class="page-subtitle">Backup health, required action, storage pressure, and recent activity in one place.</div>
</div> </div>
{% if can_manage_control_panel %}
<section class="actions" aria-label="Dashboard actions"> <section class="actions" aria-label="Dashboard actions">
<a class="button-link" href="{% url 'create_host_config' %}">New host</a> <a class="button-link" href="{% url 'create_host_config' %}">New host</a>
<a class="button-link secondary" href="{% url 'edit_global_config' %}">{% if global_config %}Edit global config{% else %}Create global config{% endif %}</a> <a class="button-link secondary" href="{% url 'edit_global_config' %}">{% if global_config %}Edit global config{% else %}Create global config{% endif %}</a>
</section> </section>
{% endif %}
</header> </header>
{% if can_manage_control_panel %}
{% if not global_config or not counts.hosts %} {% if not global_config or not counts.hosts %}
<section class="panel"> <section class="panel">
<h2>Setup</h2> <h2>Setup</h2>
@@ -31,80 +34,30 @@
{% endif %} {% endif %}
</section> </section>
{% endif %} {% endif %}
{% endif %}
<section class="grid" aria-label="Summary"> <div
<a class="metric metric-link" href="#hosts"><div class="label">Hosts</div><div class="value">{{ counts.enabled_hosts }}/{{ counts.hosts }}</div></a> data-refresh-url="{% url 'dashboard_priority_live' %}"
data-refresh-interval="10000"
data-refresh-active="true"
aria-live="polite"
>
{% include "pobsync_backend/partials/dashboard_priority.html" %}
</div>
<section class="grid dashboard-summary-grid" aria-label="Summary">
<a class="metric metric-link" href="{% url 'hosts_list' %}"><div class="label">Hosts</div><div class="value">{{ counts.enabled_hosts }}/{{ counts.hosts }}</div></a>
<a class="metric metric-link" href="{% url 'schedules_list' %}"><div class="label">Schedules</div><div class="value">{{ counts.enabled_schedules }}/{{ counts.schedules }}</div></a> <a class="metric metric-link" href="{% url 'schedules_list' %}"><div class="label">Schedules</div><div class="value">{{ counts.enabled_schedules }}/{{ counts.schedules }}</div></a>
<a class="metric metric-link" href="{% url 'snapshots_list' %}"><div class="label">Snapshots</div><div class="value">{{ counts.snapshots }}</div></a> <a class="metric metric-link" href="{% url 'snapshots_list' %}"><div class="label">Snapshots</div><div class="value">{{ counts.snapshots }}</div></a>
<a class="metric metric-link" href="{% url 'runs_list' %}"><div class="label">Runs</div><div class="value">{{ counts.runs }}</div></a> <a class="metric metric-link" href="{% url 'runs_list' %}"><div class="label">Runs</div><div class="value">{{ counts.runs }}</div></a>
<a class="metric metric-link {% if counts.queued_runs %}queued{% endif %}" href="{% url 'runs_list' %}?status=queued"><div class="label">Queued</div><div class="value">{{ counts.queued_runs }}</div></a>
<a class="metric metric-link {% if counts.running_runs %}running{% endif %}" href="{% url 'runs_list' %}?status=running"><div class="label">Running</div><div class="value">{{ counts.running_runs }}</div></a>
<a class="metric metric-link {% if counts.warning_runs %}warning{% endif %}" href="{% url 'runs_list' %}?status=warning&amp;review=needed"><div class="label">Warnings</div><div class="value">{{ counts.warning_runs }}</div></a> <a class="metric metric-link {% if counts.warning_runs %}warning{% endif %}" href="{% url 'runs_list' %}?status=warning&amp;review=needed"><div class="label">Warnings</div><div class="value">{{ counts.warning_runs }}</div></a>
<a class="metric metric-link {% if counts.failed_runs %}failed{% endif %}" href="{% url 'runs_list' %}?status=failed&amp;review=needed"><div class="label">Failed</div><div class="value">{{ counts.failed_runs }}</div></a> <a class="metric metric-link {% if counts.failed_runs %}failed{% endif %}" href="{% url 'runs_list' %}?status=failed&amp;review=needed"><div class="label">Failed</div><div class="value">{{ counts.failed_runs }}</div></a>
</section> </section>
<section class="panel"> <section class="panel dashboard-trends-panel">
<h2>Operational Status</h2>
{% if counts.failed_runs or counts.warning_runs or counts.running_runs or counts.queued_runs %}
<div class="status-overview">
{% if counts.failed_runs %}
<a class="status-summary failed" href="{% url 'runs_list' %}?status=failed&amp;review=needed">
<span class="status failed">failed</span>
<strong>{{ counts.failed_runs }} failed run{{ counts.failed_runs|pluralize }} need{{ counts.failed_runs|pluralize:"s," }} review.</strong>
<span class="summary-action">Review failed runs</span>
</a>
{% endif %}
{% if counts.warning_runs %}
<a class="status-summary warning" href="{% url 'runs_list' %}?status=warning&amp;review=needed">
<span class="status warning">warning</span>
<strong>{{ counts.warning_runs }} run{{ counts.warning_runs|pluralize }} completed with warnings.</strong>
<span class="summary-action">Review warnings</span>
</a>
{% endif %}
{% if counts.running_runs %}
<a class="status-summary running" href="{% url 'runs_list' %}?status=running">
<span class="status running">running</span>
<strong>{{ counts.running_runs }} backup run{{ counts.running_runs|pluralize }} in progress.</strong>
<span class="summary-action">View running runs</span>
</a>
{% endif %}
{% if counts.queued_runs %}
<a class="status-summary queued" href="{% url 'runs_list' %}?status=queued">
<span class="status queued">queued</span>
<strong>{{ counts.queued_runs }} backup run{{ counts.queued_runs|pluralize }} waiting for the worker.</strong>
<span class="summary-action">View queued runs</span>
</a>
{% endif %}
</div>
{% elif counts.hosts %}
<p><span class="status ok">ok</span> No queued, running, or unreviewed warning/failed runs.</p>
{% else %}
<p class="muted">Add a host to start tracking backup status here.</p>
{% endif %}
</section>
<section class="panel" id="hosts">
<h2>Backup Trends</h2> <h2>Backup Trends</h2>
{% if stats_summary.runs_sampled %} {% if stats_summary.runs_sampled %}
<div class="insight-grid" aria-label="Backup trends"> <div class="insight-grid" aria-label="Backup trends">
<div class="insight-main">
<div class="label">Storage Used</div>
<div class="value">
{% if stats_summary.capacity.used_percent is not None %}
{{ stats_summary.capacity.used_percent|floatformat:1 }}%
{% else %}
unknown
{% endif %}
</div>
{% if stats_summary.capacity.used_percent is not None %}
<div class="storage-meter" aria-label="Backup root storage usage">
<span style="width: {{ stats_summary.capacity.used_percent|floatformat:0 }}%"></span>
</div>
{% endif %}
<div class="muted">
{{ stats_summary.capacity.available_bytes|filesizeformat }} available from the backup root.
</div>
</div>
<div class="insight-item"> <div class="insight-item">
<div class="label">Runway</div> <div class="label">Runway</div>
<div class="value"> <div class="value">
@@ -145,155 +98,13 @@
{% endif %} {% endif %}
</section> </section>
<section class="panel"> <div
<h2>Hosts</h2> data-refresh-url="{% url 'dashboard_hosts_live' %}"
<div class="host-list"> data-refresh-interval="15000"
{% for host in hosts %} data-refresh-active="true"
<article class="host-card"> aria-live="polite"
<div class="host-card-header"> >
<div class="host-card-title"> {% include "pobsync_backend/partials/dashboard_hosts.html" %}
<a href="{% url 'host_detail' host.host %}">{{ host.host }}</a>
<span class="muted">{{ host.address }}</span>
</div> </div>
<div class="host-card-status">
<span class="status {% if host.enabled %}ok{% else %}skipped{% endif %}">{{ host.enabled|yesno:"enabled,disabled" }}</span>
{% if host.queued_run_count %}
<span class="status queued">queued {{ host.queued_run_count }}</span>
{% endif %}
{% if host.running_run_count %}
<span class="status running">running {{ host.running_run_count }}</span>
{% endif %}
{% if host.warning_run_count %}
<span class="status warning">warning {{ host.warning_run_count }}</span>
{% endif %}
{% if host.failed_run_count %}
<span class="status failed">failed {{ host.failed_run_count }}</span>
{% endif %}
</div>
</div>
<div class="host-card-layout">
<div class="host-card-section">
<div class="host-card-section-title">Backup activity</div>
<div class="host-card-timeline">
<div class="host-card-item">
<div class="label">Latest Snapshot</div>
<div class="value">
{% if host.latest_snapshot %}
<a href="{% url 'snapshot_detail' host.latest_snapshot.id %}">{{ host.latest_snapshot.dirname }}</a>
<div class="muted">{{ host.latest_snapshot.kind }} {{ host.latest_snapshot.status }}</div>
{% else %}
<span class="muted">none</span>
{% endif %}
</div>
</div>
<div class="host-card-item">
<div class="label">Last Good Backup</div>
<div class="value">
{% if host.stats_summary.latest_good_run.id %}
<a href="{% url 'run_detail' host.stats_summary.latest_good_run.id %}">Run {{ host.stats_summary.latest_good_run.id }}</a>
<div class="muted">{{ host.stats_summary.latest_good_run.run_type }} {{ host.stats_summary.latest_good_run.duration_seconds|default:"" }}{% if host.stats_summary.latest_good_run.duration_seconds is not None %}s{% endif %}</div>
{% else %}
<span class="muted">none</span>
{% endif %}
</div>
</div>
<div class="host-card-item">
<div class="label">Latest Issue</div>
<div class="value">
{% if host.stats_summary.latest_problem_run.id %}
<a href="{% url 'run_detail' host.stats_summary.latest_problem_run.id %}">Run {{ host.stats_summary.latest_problem_run.id }}</a>
<div><span class="status {{ host.stats_summary.latest_problem_run.status }}">{{ host.stats_summary.latest_problem_run.status }}</span></div>
<div class="muted">{{ host.stats_summary.latest_problem_run.run_type }} {{ host.stats_summary.latest_problem_run.duration_seconds|default:"" }}{% if host.stats_summary.latest_problem_run.duration_seconds is not None %}s{% endif %}</div>
{% else %}
<span class="muted">none</span>
{% endif %}
</div>
</div>
<div class="host-card-item">
<div class="label">Next Run</div>
<div class="value">
{% if host.next_run_at %}
{{ host.next_run_at|date:"Y-m-d H:i T" }}
<div class="muted">{{ scheduler_timezone }}</div>
{% else %}
<span class="muted">none</span>
{% endif %}
</div>
</div>
</div>
</div>
<div class="host-card-section">
<div class="host-card-section-title">Snapshot health</div>
<div class="host-card-stats">
<div class="host-card-stat">
<div class="label">Snapshots</div>
<div class="value">{{ host.snapshot_count }}</div>
</div>
<div class="host-card-stat">
<div class="label">Runs</div>
<div class="value">{{ host.run_count }}</div>
</div>
<div class="host-card-stat">
<div class="label">New Data</div>
<div class="value">{{ host.stats_summary.latest_run.rsync.literal_data_bytes|filesizeformat }}</div>
</div>
<div class="host-card-stat">
<div class="label">Retention</div>
<div class="value">d{{ host.retention_daily }} w{{ host.retention_weekly }} m{{ host.retention_monthly }} y{{ host.retention_yearly }}</div>
</div>
</div>
</div>
</div>
{% if host.retention_warning.has_warning %}
<div class="host-card-warning">
<span class="status warning">retention</span>
{% if host.retention_warning.prune_exceeded %}
Scheduled prune would delete {{ host.retention_warning.delete_count }} snapshot(s), above max {{ host.retention_warning.max_delete }}.
{% endif %}
{% if host.retention_warning.incomplete_count %}
{{ host.retention_warning.incomplete_count }} incomplete snapshot(s) need review.
<form method="post" action="{% url 'resolve_host_incomplete_reviews' host.host %}">
{% csrf_token %}
<button type="submit" class="secondary">Mark reviewed</button>
</form>
{% endif %}
{% if host.retention_warning.error %}
{{ host.retention_warning.error }}
{% endif %}
</div>
{% endif %}
</article>
{% empty %}
<p class="muted">No hosts configured yet.</p>
{% endfor %}
</div>
</section>
<section class="panel">
<h2>Latest Runs <a class="button-link secondary" href="{% url 'runs_list' %}">View all</a></h2>
<table>
<thead>
<tr>
<th>Host</th>
<th>Status</th>
<th>Started</th>
<th>Ended</th>
<th>Snapshot</th>
</tr>
</thead>
<tbody>
{% for run in latest_runs %}
<tr>
<td><a href="{% url 'host_detail' run.host.host %}">{{ run.host.host }}</a></td>
<td><a href="{% url 'run_detail' run.id %}"><span class="status {{ run.status }}">{{ run.status }}</span></a></td>
<td>{{ run.started_at|default:"" }}</td>
<td>{{ run.ended_at|default:"" }}</td>
<td>{% if run.snapshot %}<a href="{% url 'snapshot_detail' run.snapshot.id %}">{{ run.snapshot.dirname }}</a>{% else %}<span class="muted">{{ run.snapshot_path }}</span>{% endif %}</td>
</tr>
{% empty %}
<tr><td colspan="5" class="muted">No backup runs recorded yet.</td></tr>
{% endfor %}
</tbody>
</table>
</section>
{% endblock %} {% endblock %}

View File

@@ -33,8 +33,9 @@
</div> </div>
{% endfor %} {% endfor %}
<div class="actions"> <div class="form-actions">
<button type="submit">Save global config</button> <button type="submit">Save global config</button>
<a class="button-link secondary" href="{% url 'dashboard' %}">Cancel</a>
</div> </div>
</form> </form>
</section> </section>

View File

@@ -9,70 +9,8 @@
<h1>{{ host.host }}</h1> <h1>{{ host.host }}</h1>
<div class="page-subtitle">{{ host.address }} · {{ host.enabled|yesno:"enabled,disabled" }}</div> <div class="page-subtitle">{{ host.address }} · {{ host.enabled|yesno:"enabled,disabled" }}</div>
</div> </div>
<section class="actions" aria-label="Host actions">
<a class="button-link" href="{% url 'edit_host_config' host.host %}">Edit config</a>
<form method="post" action="{% url 'discover_host_snapshots' host.host %}">
{% csrf_token %}
<button type="submit">Discover snapshots</button>
</form>
<a class="button-link" href="{% url 'host_retention_plan' host.host %}">Plan retention</a>
<a class="button-link" href="{% url 'edit_host_schedule' host.host %}">Edit schedule</a>
<form method="post" action="{% url 'prepare_host_directories' host.host %}">
{% csrf_token %}
<button type="submit" class="secondary">Prepare directories</button>
</form>
<form method="post" action="{% url 'scan_host_known_key' host.host %}">
{% csrf_token %}
<button type="submit" class="secondary">Scan SSH host key</button>
</form>
<form method="post" action="{% url 'run_host_preflight' host.host %}">
{% csrf_token %}
<button type="submit" class="secondary">Run connection preflight</button>
</form>
</section>
</header> </header>
<section class="grid" aria-label="Host summary">
<div class="metric"><div class="label">Snapshots</div><div class="value">{{ counts.snapshots }}</div></div>
<div class="metric"><div class="label">Runs</div><div class="value">{{ counts.runs }}</div></div>
<div class="metric"><div class="label">Queued</div><div class="value">{{ counts.queued_runs }}</div></div>
<div class="metric"><div class="label">Running</div><div class="value">{{ counts.running_runs }}</div></div>
<div class="metric"><div class="label">Failed Runs</div><div class="value">{{ counts.failed_runs }}</div></div>
<div class="metric"><div class="label">Incomplete</div><div class="value">{{ counts.incomplete_snapshots }}</div></div>
</section>
<div class="two-col">
<section class="panel">
<h2>Config</h2>
<div class="stack">
<div><strong>Address:</strong> {{ host.address }}</div>
<div><strong>Enabled:</strong> {{ host.enabled|yesno:"yes,no" }}</div>
<div><strong>SSH key:</strong> {{ host.ssh_credential|default:"global default" }}</div>
<div><strong>SSH:</strong> {{ host.ssh_user|default:"global" }}{% if host.ssh_port %}:{{ host.ssh_port }}{% endif %}</div>
<div><strong>Backup source:</strong> {{ host.source_root|default:"global default" }}</div>
<div><strong>Retention:</strong> daily {{ host.retention_daily }}, weekly {{ host.retention_weekly }}, monthly {{ host.retention_monthly }}, yearly {{ host.retention_yearly }}</div>
</div>
</section>
<section class="panel">
<h2>Schedule</h2>
{% if schedule %}
<div class="stack">
<div><strong>Schedule expression:</strong> {{ schedule.cron_expr }}</div>
<div class="muted">Evaluated by the pobsync scheduler service.</div>
<div><strong>Enabled:</strong> {{ schedule.enabled|yesno:"yes,no" }}</div>
<div><strong>Next run:</strong> {% if next_run_at %}{{ next_run_at|date:"Y-m-d H:i T" }} <span class="muted">{{ scheduler_timezone }}</span>{% endif %}</div>
<div><strong>Prune:</strong> {{ schedule.prune|yesno:"yes,no" }}</div>
<div><strong>Last status:</strong> {{ schedule.last_status|default:"" }}</div>
<div><strong>Last started:</strong> {{ schedule.last_started_at|default:"" }}</div>
<div><strong>Last finished:</strong> {{ schedule.last_finished_at|default:"" }}</div>
</div>
{% else %}
<p class="muted">No schedule configured.</p>
{% endif %}
</section>
</div>
{% if retention_warning.has_warning %} {% if retention_warning.has_warning %}
<section class="panel highlight warning"> <section class="panel highlight warning">
<h2>Retention Warnings</h2> <h2>Retention Warnings</h2>
@@ -89,11 +27,13 @@
{{ retention_warning.incomplete_count }} incomplete snapshot(s) exist. Retention does not delete incomplete {{ retention_warning.incomplete_count }} incomplete snapshot(s) exist. Retention does not delete incomplete
snapshots automatically; inspect them before cleanup. snapshots automatically; inspect them before cleanup.
</div> </div>
{% if can_manage_control_panel %}
<form method="post" action="{% url 'resolve_host_incomplete_reviews' host.host %}"> <form method="post" action="{% url 'resolve_host_incomplete_reviews' host.host %}">
{% csrf_token %} {% csrf_token %}
<button type="submit" class="secondary">Mark incomplete reviewed</button> <button type="submit" class="secondary">Mark incomplete reviewed</button>
</form> </form>
{% endif %} {% endif %}
{% endif %}
{% if retention_warning.error %} {% if retention_warning.error %}
<div>{{ retention_warning.error }}</div> <div>{{ retention_warning.error }}</div>
{% endif %} {% endif %}
@@ -101,60 +41,177 @@
</section> </section>
{% endif %} {% endif %}
{% if effective_config %} <section class="host-control-grid" aria-label="Host control workspace">
<section class="panel"> <article class="panel host-control-panel">
<h2>Effective Config</h2> <h2>Host Status</h2>
<div class="two-col"> <div class="host-control-primary">
<div class="stack">
<div><strong>Backup source:</strong> {{ effective_config.source_root }}</div>
<div><strong>Destination subdir:</strong> {{ effective_config.destination_subdir|default:"none" }}</div>
<div><strong>SSH:</strong> {{ effective_config.ssh.user }}@{{ host.address }}:{{ effective_config.ssh.port }}</div>
<div><strong>SSH key:</strong> {{ effective_config.ssh.credential|default:"none selected" }}</div>
<div><strong>SSH options:</strong> {{ effective_config.ssh.options|join:" " }}</div>
<div><strong>Rsync binary:</strong> {{ effective_config.rsync.binary }}</div>
<div><strong>Rsync args:</strong> {{ effective_config.rsync.args|join:" " }}</div>
<div><strong>Timeout:</strong> {{ effective_config.rsync.timeout_seconds }}s</div>
<div><strong>Bandwidth limit:</strong> {{ effective_config.rsync.bwlimit_kbps }} KB/s</div>
<div> <div>
<strong>Retention:</strong> {% if host.enabled %}
d{{ effective_config.retention.daily }} <span class="status ok">enabled</span>
w{{ effective_config.retention.weekly }}
m{{ effective_config.retention.monthly }}
y{{ effective_config.retention.yearly }}
</div>
</div>
<div class="stack">
<div><strong>Includes:</strong> {{ effective_config.includes|length }}</div>
{% if effective_config.includes %}
<pre>{{ effective_config.includes|join:"&#10;" }}</pre>
{% else %} {% else %}
<div class="muted">No include rules configured.</div> <span class="status failed">disabled</span>
{% endif %} {% endif %}
<div><strong>Excludes:</strong> {{ effective_config.excludes|length }}</div> <span class="muted">{{ host.address }}</span>
{% if effective_config.excludes %} </div>
<pre>{{ effective_config.excludes|join:"&#10;" }}</pre> {% if active_run %}
<a class="status-summary {{ active_run.status }}" href="{% url 'run_detail' active_run.id %}">
<span class="status {{ active_run.status }}">{{ active_run.status }}</span>
<strong>Run {{ active_run.id }} is active.</strong>
</a>
{% elif counts.failed_runs %}
<a class="status-summary failed" href="{% url 'runs_list' %}?host={{ host.host }}&amp;status=failed&amp;review=needed">
<span class="status failed">failed</span>
<strong>{{ counts.failed_runs }} failed run{{ counts.failed_runs|pluralize }} need review.</strong>
</a>
{% elif retention_warning.has_warning %}
<span class="status-summary warning">
<span class="status warning">warning</span>
<strong>Retention needs attention.</strong>
</span>
{% else %} {% else %}
<div class="muted">No exclude rules configured.</div> <span class="status-summary success">
<span class="status ok">ok</span>
<strong>No active blockers for this host.</strong>
</span>
{% endif %} {% endif %}
</div> </div>
<div class="host-control-meta">
<div><span class="label">Snapshots</span><strong>{{ counts.snapshots }}</strong></div>
<div><span class="label">Runs</span><strong>{{ counts.runs }}</strong></div>
<div><span class="label">Incomplete</span><strong>{{ counts.incomplete_snapshots }}</strong></div>
</div> </div>
</article>
{% if can_manage_control_panel %}
<article class="panel host-control-panel">
<h2>Backup Control</h2>
<div class="operator-state">
{% if active_run %}
<span class="status {{ active_run.status }}">{{ active_run.status }}</span>
<a href="{% url 'run_detail' active_run.id %}">Run {{ active_run.id }}</a>
{% elif has_global_config and host.enabled %}
<span class="status {{ backup_gate.state }}">{{ backup_gate.state }}</span>
<span class="muted">{{ backup_gate.message }}</span>
{% elif not host.enabled %}
<span class="status failed">disabled</span>
{% elif not has_global_config %}
<span class="status failed">missing global config</span>
{% endif %}
</div>
<section class="actions inline" aria-label="Quick backup actions">
<form class="inline-form" method="post" action="{% url 'queue_manual_backup' host.host %}">
{% csrf_token %}
<input type="hidden" name="dry_run" value="on">
<input type="hidden" name="verbose_output" value="on">
<input type="hidden" name="prune_max_delete" value="10">
<button type="submit" class="secondary" {% if not can_queue_dry_run %}disabled{% endif %}>Queue dry-run</button>
</form>
<form class="inline-form" method="post" action="{% url 'queue_manual_backup' host.host %}">
{% csrf_token %}
<input type="hidden" name="verbose_output" value="on">
<input type="hidden" name="prune_max_delete" value="10">
<button type="submit" {% if not can_queue_real_backup %}disabled{% endif %}>Queue backup</button>
</form>
</section> </section>
{% if active_run %}
<p class="muted">Wait for the active run to finish, or cancel it from the run detail page.</p>
{% elif not can_queue_dry_run or not can_queue_real_backup %}
{% if not has_global_config %}
<p class="muted">Create the default global config before queueing backups.</p>
{% elif not host.enabled %}
<p class="muted">Enable this host before queueing backups.</p>
{% elif backup_gate.real_blockers %}
<p class="muted">Real backups are blocked by failed preflight checks. Dry-runs may still be available when storage-only checks fail.</p>
{% endif %}
{% endif %}
</article>
{% endif %} {% endif %}
<section class="panel"> <article class="panel host-control-panel">
<h2>Snapshot Discovery</h2> <h2>
<div class="stack"> Schedule
<div><strong>Backup root:</strong> {{ discovery.backup_root|default:"" }}</div> {% if can_manage_control_panel %}
<div><strong>Host root:</strong> {{ discovery.host_root|default:"" }}</div> <a class="button-link secondary compact" href="{% url 'edit_host_schedule' host.host %}">Edit schedule</a>
<div><strong>Status:</strong> {{ discovery.message }}</div> {% endif %}
{% if discovery.kind_counts %} </h2>
<div><strong>On disk:</strong> {% if schedule %}
scheduled {{ discovery.kind_counts.scheduled|default:0 }}, <div class="host-control-meta">
manual {{ discovery.kind_counts.manual|default:0 }}, <div><span class="label">Schedule expression</span><strong>{{ schedule.cron_expr }}</strong></div>
incomplete {{ discovery.kind_counts.incomplete|default:0 }} <div><span class="label">Next run</span><strong>{% if next_run_at %}{{ next_run_at|date:"Y-m-d H:i T" }}{% else %}none{% endif %}</strong></div>
<div><span class="label">Timezone</span><strong>{{ scheduler_timezone }}</strong></div>
<div><span class="label">Prune</span><strong>{{ schedule.prune|yesno:"yes,no" }}</strong></div>
<div><span class="label">Last status</span><strong>{{ schedule.last_status|default:"none" }}</strong></div>
</div>
<p class="muted">Evaluated by the pobsync scheduler service.</p>
{% else %}
<p class="muted">No schedule configured.</p>
{% if can_manage_control_panel %}
<a class="button-link secondary compact" href="{% url 'edit_host_schedule' host.host %}">Add schedule</a>
{% endif %}
{% endif %}
</article>
<article class="panel host-control-panel">
<h2>Current Activity</h2>
{% if latest_runs %}
{% with run=latest_runs.0 %}
<a class="activity-row" href="{% url 'run_detail' run.id %}">
<span class="status {{ run.status }}">{{ run.status }}</span>
<span>
<strong>Run {{ run.id }}</strong>
<span class="muted">{{ run.run_type }} · {{ run.started_at|default:run.created_at }}</span>
</span>
</a>
{% endwith %}
{% else %}
<p class="muted">No backup runs recorded for this host.</p>
{% endif %}
{% if stats_summary.latest_run.duration_seconds is not None %}
<div class="host-control-meta">
<div><span class="label">Latest duration</span><strong>{{ stats_summary.latest_run.duration_seconds }}s</strong></div>
<div><span class="label">New data</span><strong>{{ stats_summary.latest_run.rsync.literal_data_bytes|filesizeformat }}</strong></div>
</div> </div>
{% endif %} {% endif %}
</article>
</section>
<section class="grid" aria-label="Host summary">
<div class="metric"><div class="label">Snapshots</div><div class="value">{{ counts.snapshots }}</div></div>
<div class="metric"><div class="label">Runs</div><div class="value">{{ counts.runs }}</div></div>
<div class="metric"><div class="label">Queued</div><div class="value">{{ counts.queued_runs }}</div></div>
<div class="metric"><div class="label">Running</div><div class="value">{{ counts.running_runs }}</div></div>
<div class="metric"><div class="label">Failed Runs</div><div class="value">{{ counts.failed_runs }}</div></div>
<div class="metric"><div class="label">Incomplete</div><div class="value">{{ counts.incomplete_snapshots }}</div></div>
</section>
<section class="panel">
<h2>Backup Data</h2>
<section class="grid" aria-label="Host backup data totals">
<div class="metric">
<div class="label">Scheduled</div>
<div class="value">{{ stats_summary.backup_data.scheduled.allocated_size_bytes|filesizeformat }}</div>
<div class="muted">unique {{ stats_summary.backup_data.scheduled.unique_apparent_size_bytes|filesizeformat }}</div>
</div> </div>
<div class="metric">
<div class="label">Manual</div>
<div class="value">{{ stats_summary.backup_data.manual.allocated_size_bytes|filesizeformat }}</div>
<div class="muted">unique {{ stats_summary.backup_data.manual.unique_apparent_size_bytes|filesizeformat }}</div>
</div>
<div class="metric">
<div class="label">Incomplete</div>
<div class="value">{{ stats_summary.backup_data.incomplete.allocated_size_bytes|filesizeformat }}</div>
<div class="muted">measured from disk</div>
</div>
<div class="metric">
<div class="label">Total</div>
<div class="value">{{ stats_summary.backup_data.total.allocated_size_bytes|filesizeformat }}</div>
<div class="muted">unique {{ stats_summary.backup_data.total.unique_apparent_size_bytes|filesizeformat }}</div>
</div>
</section>
<p class="muted">
Main totals use allocated snapshot size. Unique values estimate non-hardlinked visible data; incomplete
snapshots are measured from disk because their metadata can be stale.
</p>
</section> </section>
{% if stats_summary.runs %} {% if stats_summary.runs %}
@@ -206,6 +263,7 @@
</section> </section>
{% endif %} {% endif %}
{% if can_manage_control_panel %}
<section class="panel"> <section class="panel">
<h2>Host Check</h2> <h2>Host Check</h2>
<section class="grid" aria-label="Host check summary"> <section class="grid" aria-label="Host check summary">
@@ -214,104 +272,210 @@
<div class="metric"><div class="label">Failed</div><div class="value">{{ host_check_summary.failed }}</div></div> <div class="metric"><div class="label">Failed</div><div class="value">{{ host_check_summary.failed }}</div></div>
<div class="metric"><div class="label">Skipped</div><div class="value">{{ host_check_summary.skipped }}</div></div> <div class="metric"><div class="label">Skipped</div><div class="value">{{ host_check_summary.skipped }}</div></div>
</section> </section>
<table> <div class="record-list">
<thead>
<tr>
<th>Status</th>
<th>Check</th>
<th>Message</th>
<th>Detail</th>
</tr>
</thead>
<tbody>
{% for check in host_checks %} {% for check in host_checks %}
<tr> <article class="record-card">
<td><span class="status {{ check.status }}">{{ check.status }}</span></td> <div class="record-card-header">
<td>{{ check.name }}</td> <div class="record-title">
<td>{{ check.message }}</td> <strong>{{ check.name }}</strong>
<td class="muted">{{ check.detail }}</td> <span class="muted">{{ check.message }}</span>
</tr> </div>
<span class="status {{ check.status }}">{{ check.status }}</span>
</div>
{% if check.detail %}
<div class="record-fact">
<span class="label">Detail</span>
<span class="muted">{{ check.detail }}</span>
</div>
{% endif %}
</article>
{% endfor %} {% endfor %}
</tbody> </div>
</table> </section>
{% endif %}
<div class="panel-grid">
<section class="panel">
<h2>Configuration</h2>
<div class="host-control-meta">
<div><span class="label">Address</span><strong>{{ host.address }}</strong></div>
{% if can_manage_control_panel %}
<div><span class="label">SSH key</span><strong>{{ host.ssh_credential|default:"global default" }}</strong></div>
<div><span class="label">SSH</span><strong>{{ host.ssh_user|default:"global" }}{% if host.ssh_port %}:{{ host.ssh_port }}{% endif %}</strong></div>
{% endif %}
<div><span class="label">Backup source</span><strong>{{ host.source_root|default:"global default" }}</strong></div>
<div><span class="label">Retention</span><strong>d{{ host.retention_daily }} w{{ host.retention_weekly }} m{{ host.retention_monthly }} y{{ host.retention_yearly }}</strong></div>
</div>
{% if can_manage_control_panel %}
<div class="actions inline">
<a class="button-link secondary compact" href="{% url 'edit_host_config' host.host %}">Edit config</a>
<a class="button-link secondary compact" href="{% url 'host_retention_plan' host.host %}">Plan retention</a>
</div>
{% endif %}
</section> </section>
{% if can_manage_control_panel %}
<section class="panel">
<h2>Connection Preflight &amp; SSH</h2>
{% if last_preflight %} {% if last_preflight %}
<section class="panel"> <div class="host-control-meta">
<h2>Connection Preflight</h2> <div>
<div class="stack spaced"> <span class="label">Preflight</span>
<div><strong>Status:</strong> <span class="status {% if last_preflight.ok %}ok{% else %}failed{% endif %}">{% if last_preflight.ok %}ok{% else %}failed{% endif %}</span></div> <strong>
<div><strong>Target:</strong> {{ last_preflight.target }}</div> <span class="status {% if last_preflight.ok %}ok{% else %}failed{% endif %}">
<div><strong>Backup source:</strong> {{ last_preflight.source_root }}</div> {% if last_preflight.ok %}ok{% else %}failed{% endif %}
<div><strong>Remote rsync:</strong> {{ last_preflight.rsync_binary }}</div> </span>
</strong>
</div> </div>
<table> <div><span class="label">Target</span><strong>{{ last_preflight.target }}</strong></div>
<thead> <div><span class="label">Backup source</span><strong>{{ last_preflight.source_root }}</strong></div>
<tr> <div><span class="label">Remote rsync</span><strong>{{ last_preflight.rsync_binary }}</strong></div>
<th>Status</th> </div>
<th>Check</th> {% else %}
<th>Message</th> <p class="muted">No connection preflight recorded yet.</p>
<th>Detail</th> {% endif %}
</tr> <div class="actions inline">
</thead> <form method="post" action="{% url 'run_host_preflight' host.host %}">
<tbody> {% csrf_token %}
<button type="submit" class="secondary compact">Run connection preflight</button>
</form>
<form method="post" action="{% url 'scan_host_known_key' host.host %}">
{% csrf_token %}
<button type="submit" class="secondary compact">Scan SSH host key</button>
</form>
</div>
{% if last_preflight.checks %}
<div class="activity-list">
{% for check in last_preflight.checks %} {% for check in last_preflight.checks %}
<tr> <div class="activity-row">
<td><span class="status {% if check.ok %}ok{% else %}failed{% endif %}">{% if check.ok %}ok{% else %}failed{% endif %}</span></td> <span class="status {% if check.ok %}ok{% else %}failed{% endif %}">
<td>{{ check.name }}</td> {% if check.ok %}ok{% else %}failed{% endif %}
<td>{{ check.message }}</td> </span>
<td class="muted">{{ check.detail }}</td> <span>
</tr> <strong>{{ check.name }}</strong>
<span class="muted">{{ check.message }}{% if check.detail %} · {{ check.detail }}{% endif %}</span>
</span>
</div>
{% endfor %} {% endfor %}
</tbody> </div>
</table> {% endif %}
</section> </section>
{% endif %} {% endif %}
<section class="panel"> <section class="panel">
<h2>Backup Control</h2> <h2>Snapshot Storage</h2>
<div class="operator-state"> <div class="host-control-meta">
{% if active_run %} <div><span class="label">Backup root</span><strong>{{ discovery.backup_root|default:"" }}</strong></div>
<span class="status {{ active_run.status }}">{{ active_run.status }}</span> <div><span class="label">Host root</span><strong>{{ discovery.host_root|default:"" }}</strong></div>
<a href="{% url 'run_detail' active_run.id %}">Run {{ active_run.id }}</a> <div><span class="label">Status</span><strong>{{ discovery.message }}</strong></div>
{% elif has_global_config and host.enabled %} {% if discovery.kind_counts %}
<span class="status {{ backup_gate.state }}">{{ backup_gate.state }}</span> <div>
<span class="muted">{{ backup_gate.message }}</span> <span class="label">On disk</span>
{% elif not host.enabled %} <strong>
<span class="status failed">disabled</span> scheduled {{ discovery.kind_counts.scheduled|default:0 }},
{% elif not has_global_config %} manual {{ discovery.kind_counts.manual|default:0 }},
<span class="status failed">missing global config</span> incomplete {{ discovery.kind_counts.incomplete|default:0 }}
</strong>
</div>
{% endif %} {% endif %}
</div> </div>
{% if can_manage_control_panel %}
<section class="actions inline" aria-label="Quick backup actions"> <div class="actions inline">
<form class="inline-form" method="post" action="{% url 'queue_manual_backup' host.host %}"> <form method="post" action="{% url 'discover_host_snapshots' host.host %}">
{% csrf_token %} {% csrf_token %}
<input type="hidden" name="dry_run" value="on"> <button type="submit" class="secondary compact">Discover snapshots</button>
<input type="hidden" name="verbose_output" value="on">
<input type="hidden" name="prune_max_delete" value="10">
<button type="submit" class="secondary" {% if not can_queue_dry_run %}disabled{% endif %}>Queue dry-run</button>
</form> </form>
<form class="inline-form" method="post" action="{% url 'queue_manual_backup' host.host %}"> <form method="post" action="{% url 'prepare_host_directories' host.host %}">
{% csrf_token %} {% csrf_token %}
<input type="hidden" name="prune_max_delete" value="10"> <button type="submit" class="secondary compact">Prepare directories</button>
<button type="submit" {% if not can_queue_real_backup %}disabled{% endif %}>Queue backup</button>
</form> </form>
</div>
{% endif %}
</section> </section>
</div>
{% if active_run %} {% if effective_config %}
<p class="muted">Wait for the active run to finish, or cancel it from the run detail page.</p> <section class="panel">
{% elif not can_queue_dry_run or not can_queue_real_backup %} <h2>Effective Config</h2>
{% if not has_global_config %} <p class="muted">Runtime settings after global defaults and host overrides are combined.</p>
<p class="muted">Create the default global config before queueing backups.</p> <div class="record-list">
{% elif not host.enabled %} <article class="record-card">
<p class="muted">Enable this host before queueing backups.</p> <div class="record-card-header">
{% elif backup_gate.real_blockers %} <div class="record-title">
<p class="muted">Real backups are blocked by failed preflight checks. Dry-runs may still be available when storage-only checks fail.</p> <strong>Backup target</strong>
<span class="muted">Source and destination used by rsync.</span>
</div>
</div>
<div class="record-facts">
<div class="record-fact"><span class="label">Backup source:</span><strong>{{ effective_config.source_root }}</strong></div>
<div class="record-fact"><span class="label">Destination subdir:</span><strong>{{ effective_config.destination_subdir|default:"none" }}</strong></div>
</div>
</article>
<article class="record-card">
<div class="record-card-header">
<div class="record-title">
<strong>Connection</strong>
<span class="muted">SSH and rsync execution settings.</span>
</div>
</div>
<div class="record-facts">
<div class="record-fact"><span class="label">SSH:</span><strong>{{ effective_config.ssh.user }}@{{ host.address }}:{{ effective_config.ssh.port }}</strong></div>
<div class="record-fact"><span class="label">SSH key:</span><strong>{{ effective_config.ssh.credential|default:"none selected" }}</strong></div>
<div class="record-fact"><span class="label">SSH options:</span><span>{{ effective_config.ssh.options|join:" " }}</span></div>
<div class="record-fact"><span class="label">Rsync binary:</span><strong>{{ effective_config.rsync.binary }}</strong></div>
<div class="record-fact"><span class="label">Rsync args:</span><span>{{ effective_config.rsync.args|join:" " }}</span></div>
<div class="record-fact"><span class="label">Timeout:</span><strong>{{ effective_config.rsync.timeout_seconds }}s</strong></div>
<div class="record-fact">
<span class="label">Bandwidth limit:</span>
<strong>{% if effective_config.rsync.bwlimit_kbps %}{{ effective_config.rsync.bwlimit_kbps }} KB/s{% else %}unlimited{% endif %}</strong>
</div>
</div>
</article>
<article class="record-card">
<div class="record-card-header">
<div class="record-title">
<strong>Selection &amp; retention</strong>
<span class="muted">Include/exclude rules and retention counts.</span>
</div>
</div>
<div class="record-facts">
<div class="record-fact">
<span class="label">Retention:</span>
<strong>
d{{ effective_config.retention.daily }}
w{{ effective_config.retention.weekly }}
m{{ effective_config.retention.monthly }}
y{{ effective_config.retention.yearly }}
</strong>
</div>
<div class="record-fact"><span class="label">Includes:</span><strong>{{ effective_config.includes|length }}</strong></div>
<div class="record-fact"><span class="label">Excludes:</span><strong>{{ effective_config.excludes|length }}</strong></div>
</div>
<div class="two-col">
<div class="stack">
{% if effective_config.includes %}
<pre>{{ effective_config.includes|join:"&#10;" }}</pre>
{% else %}
<div class="muted">No include rules configured.</div>
{% endif %} {% endif %}
</div>
<div class="stack">
{% if effective_config.excludes %}
<pre>{{ effective_config.excludes|join:"&#10;" }}</pre>
{% else %}
<div class="muted">No exclude rules configured.</div>
{% endif %}
</div>
</div>
</article>
</div>
</section>
{% endif %} {% endif %}
<h3>Advanced Options</h3> {% if can_manage_control_panel %}
<section class="panel">
<h2>Backup Options</h2>
<p class="muted">Use this when the quick actions above need a custom label, include/exclude override, or prune limit.</p>
<form method="post" action="{% url 'queue_manual_backup' host.host %}" class="form-grid"> <form method="post" action="{% url 'queue_manual_backup' host.host %}" class="form-grid">
{% csrf_token %} {% csrf_token %}
{{ manual_backup_form.non_field_errors }} {{ manual_backup_form.non_field_errors }}
@@ -325,65 +489,96 @@
</div> </div>
{% endfor %} {% endfor %}
<div class="actions"> <div class="form-actions">
<button type="submit" {% if not can_queue_dry_run and not can_queue_real_backup %}disabled{% endif %}>Queue with options</button> <button type="submit" {% if not can_queue_dry_run and not can_queue_real_backup %}disabled{% endif %}>Queue with options</button>
</div> </div>
</form> </form>
</section> </section>
{% endif %}
<section class="panel"> <section class="panel">
<h2>Latest Runs</h2> <h2>Latest Runs <a class="button-link secondary compact" href="{% url 'runs_list' %}?host={{ host.host }}">View all</a></h2>
<table> <div class="record-list">
<thead>
<tr>
<th>Status</th>
<th>Started</th>
<th>Ended</th>
<th>Snapshot</th>
<th>Base</th>
</tr>
</thead>
<tbody>
{% for run in latest_runs %} {% for run in latest_runs %}
<tr> <article class="record-card">
<td><a href="{% url 'run_detail' run.id %}"><span class="status {{ run.status }}">{{ run.status }}</span></a></td> <div class="record-card-header">
<td>{{ run.started_at|default:"" }}</td> <div class="record-title">
<td>{{ run.ended_at|default:"" }}</td> <a href="{% url 'run_detail' run.id %}">Run {{ run.id }}</a>
<td>{% if run.snapshot %}<a href="{% url 'snapshot_detail' run.snapshot.id %}">{{ run.snapshot.dirname }}</a>{% else %}<span class="muted">{{ run.snapshot_path }}</span>{% endif %}</td> <span class="muted">{{ run.run_type }}{% if run.result.duration_seconds %} · {{ run.result.duration_seconds }}s{% endif %}</span>
<td>{{ run.base_path|default:"" }}</td> </div>
</tr> <span class="status {{ run.status }}">{{ run.status }}</span>
</div>
<div class="record-facts">
<div class="record-fact">
<span class="label">Started</span>
<strong>{{ run.started_at|default:run.created_at }}</strong>
</div>
<div class="record-fact">
<span class="label">Ended</span>
<strong>{{ run.ended_at|default:"running or queued" }}</strong>
</div>
<div class="record-fact">
<span class="label">Snapshot</span>
{% if run.snapshot %}
<strong><a href="{% url 'snapshot_detail' run.snapshot.id %}">{{ run.snapshot.dirname }}</a></strong>
{% elif run.snapshot_path %}
<span class="muted">{{ run.snapshot_path }}</span>
{% else %}
<span class="muted">none</span>
{% endif %}
</div>
<div class="record-fact">
<span class="label">Base</span>
<span class="muted">{{ run.base_path|default:"none" }}</span>
</div>
</div>
</article>
{% empty %} {% empty %}
<tr><td colspan="5" class="muted">No backup runs recorded for this host.</td></tr> <p class="muted">No backup runs recorded for this host.</p>
{% endfor %} {% endfor %}
</tbody> </div>
</table>
</section> </section>
<section class="panel"> <section class="panel">
<h2>Snapshots</h2> <h2>Snapshots <a class="button-link secondary compact" href="{% url 'snapshots_list' %}?host={{ host.host }}">View all</a></h2>
<table> <div class="record-list">
<thead>
<tr>
<th>Kind</th>
<th>Status</th>
<th>Started</th>
<th>Dirname</th>
<th>Base</th>
</tr>
</thead>
<tbody>
{% for snapshot in snapshots %} {% for snapshot in snapshots %}
<tr> <article class="record-card">
<td>{{ snapshot.kind }}</td> <div class="record-card-header">
<td>{{ snapshot.status }}</td> <div class="record-title">
<td>{{ snapshot.started_at|default:"" }}</td> <a href="{% url 'snapshot_detail' snapshot.id %}">{{ snapshot.dirname }}</a>
<td><a href="{% url 'snapshot_detail' snapshot.id %}">{{ snapshot.dirname }}</a></td> <span class="muted">{{ snapshot.kind }}</span>
<td>{% if snapshot.base %}<a href="{% url 'snapshot_detail' snapshot.base.id %}">{{ snapshot.base.dirname }}</a>{% else %}{{ snapshot.base_dirname }}{% endif %}</td> </div>
</tr> <span class="status {{ snapshot.status }}">{{ snapshot.status }}</span>
</div>
<div class="record-facts">
<div class="record-fact">
<span class="label">Started</span>
<strong>{{ snapshot.started_at|default:"unknown" }}</strong>
</div>
<div class="record-fact">
<span class="label">Ended</span>
<strong>{{ snapshot.ended_at|default:"unknown" }}</strong>
</div>
<div class="record-fact">
<span class="label">Base</span>
{% if snapshot.base %}
<strong><a href="{% url 'snapshot_detail' snapshot.base.id %}">{{ snapshot.base.dirname }}</a></strong>
{% elif snapshot.base_dirname %}
<span class="muted">{{ snapshot.base_dirname }}</span>
{% else %}
<span class="muted">none</span>
{% endif %}
</div>
<div class="record-fact">
<span class="label">Path</span>
<span class="muted">{{ snapshot.path|default:"not recorded" }}</span>
</div>
</div>
</article>
{% empty %} {% empty %}
<tr><td colspan="5" class="muted">No snapshots discovered for this host.</td></tr> <p class="muted">No snapshots discovered for this host.</p>
{% endfor %} {% endfor %}
</tbody> </div>
</table>
</section> </section>
{% endblock %} {% endblock %}

View File

@@ -33,8 +33,13 @@
</div> </div>
{% endfor %} {% endfor %}
<div class="actions"> <div class="form-actions">
<button type="submit">{% if host %}Save config{% else %}Create host{% endif %}</button> <button type="submit">{% if host %}Save config{% else %}Create host{% endif %}</button>
{% if host %}
<a class="button-link secondary" href="{% url 'host_detail' host.host %}">Cancel</a>
{% else %}
<a class="button-link secondary" href="{% url 'dashboard' %}">Cancel</a>
{% endif %}
</div> </div>
</form> </form>
</section> </section>

View File

@@ -0,0 +1,45 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}Hosts | pobsync{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Inventory</div>
<h1>Hosts</h1>
<div class="page-subtitle">Configured backup targets, schedules, retention state, and host-level controls.</div>
</div>
{% if can_manage_control_panel %}
<section class="actions" aria-label="Host actions">
<a class="button-link" href="{% url 'create_host_config' %}">New host</a>
</section>
{% endif %}
</header>
<section class="grid dashboard-summary-grid" aria-label="Host summary">
<a class="metric metric-link" href="{% url 'hosts_list' %}"><div class="label">Showing</div><div class="value">{{ counts.hosts }}</div></a>
<a class="metric metric-link" href="{% url 'hosts_list' %}?enabled=yes"><div class="label">Enabled</div><div class="value">{{ counts.enabled_hosts }}</div></a>
<a class="metric metric-link" href="{% url 'hosts_list' %}?enabled=no"><div class="label">Disabled</div><div class="value">{{ counts.disabled_hosts }}</div></a>
<a class="metric metric-link" href="{% url 'dashboard' %}"><div class="label">Total</div><div class="value">{{ total_count }}</div></a>
</section>
<section class="panel">
<h2>Filters</h2>
<form class="filter-form" method="get">
<div class="field">
<label for="enabled">Host state</label>
<select id="enabled" name="enabled">
<option value="" {% if selected_enabled == "" %}selected{% endif %}>All hosts</option>
<option value="yes" {% if selected_enabled == "yes" %}selected{% endif %}>Enabled only</option>
<option value="no" {% if selected_enabled == "no" %}selected{% endif %}>Disabled only</option>
</select>
</div>
<div class="form-actions">
<button type="submit">Apply filters</button>
<a class="button-link secondary" href="{% url 'hosts_list' %}">Reset</a>
</div>
</form>
</section>
{% include "pobsync_backend/partials/dashboard_hosts.html" %}
{% endblock %}

View File

@@ -16,7 +16,7 @@
<section class="panel"> <section class="panel">
<h2>Filter</h2> <h2>Filter</h2>
<form method="get" class="form-grid"> <form method="get" class="filter-form">
<div class="field"> <div class="field">
<label for="unit">Unit</label> <label for="unit">Unit</label>
<select id="unit" name="unit"> <select id="unit" name="unit">
@@ -54,8 +54,9 @@
<label for="q">Message contains</label> <label for="q">Message contains</label>
<input id="q" name="q" value="{{ query }}"> <input id="q" name="q" value="{{ query }}">
</div> </div>
<div class="actions"> <div class="form-actions">
<button type="submit">Filter logs</button> <button type="submit">Filter logs</button>
<a class="button-link secondary" href="{% url 'logs' %}">Clear</a>
</div> </div>
</form> </form>
</section> </section>

View File

@@ -0,0 +1,38 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}{{ title }} | pobsync{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Reports</div>
<h1>{{ title }}</h1>
<div class="page-subtitle">Choose which completed backup statuses should trigger an email or webhook report.</div>
</div>
<section class="actions" aria-label="Notification target form actions">
<a class="button-link" href="{% url 'notification_targets' %}">Back to notifications</a>
</section>
</header>
<section class="panel">
<h2>{% if target %}Edit Target{% else %}Create Target{% endif %}</h2>
<form method="post" class="form-grid">
{% csrf_token %}
{{ form.non_field_errors }}
{% for field in form %}
<div class="field">
{{ field.errors }}
<label for="{{ field.id_for_label }}">{{ field.label }}</label>
{{ field }}
{% if field.help_text %}<div class="helptext">{{ field.help_text }}</div>{% endif %}
</div>
{% endfor %}
<div class="form-actions">
<button type="submit">{{ submit_label }}</button>
<a class="button-link secondary" href="{% url 'notification_targets' %}">Cancel</a>
</div>
</form>
</section>
{% endblock %}

View File

@@ -0,0 +1,91 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}Notifications | pobsync{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Reports</div>
<h1>Notifications</h1>
<div class="page-subtitle">Send email or webhook reports when backup runs finish.</div>
</div>
<section class="actions" aria-label="Notification actions">
<a class="button-link" href="{% url 'create_notification_target' %}">New target</a>
<a class="button-link secondary" href="{% url 'dashboard' %}">Back to dashboard</a>
</section>
</header>
<section class="panel">
<h2>Targets</h2>
<table>
<thead>
<tr>
<th>Name</th>
<th>Channel</th>
<th>Status</th>
<th>Events</th>
<th>Destination</th>
<th>Last delivery</th>
<th>Actions</th>
</tr>
</thead>
<tbody>
{% for target in targets %}
<tr>
<td><a href="{% url 'edit_notification_target' target.id %}">{{ target.name }}</a></td>
<td>{{ target.get_channel_display }}</td>
<td><span class="status {% if target.enabled %}ok{% else %}skipped{% endif %}">{{ target.enabled|yesno:"enabled,disabled" }}</span></td>
<td>{{ target.statuses|join:", " }}</td>
<td>
{% if target.channel == "email" %}
{{ target.email_to|linebreaksbr }}
{% else %}
<code>{{ target.webhook_url|truncatechars:70 }}</code>
{% endif %}
</td>
<td>
{% if target.last_status %}
<span class="status {{ target.last_status }}">{{ target.last_status }}</span>
{% if target.last_error %}<div class="muted">{{ target.last_error|truncatechars:90 }}</div>{% endif %}
{% if target.last_sent_at %}<div class="muted">{{ target.last_sent_at }}</div>{% endif %}
{% else %}
<span class="muted">none</span>
{% endif %}
</td>
<td><a class="button-link secondary" href="{% url 'edit_notification_target' target.id %}">Edit</a></td>
</tr>
{% empty %}
<tr><td colspan="7" class="muted">No notification targets configured yet.</td></tr>
{% endfor %}
</tbody>
</table>
</section>
<section class="panel">
<h2>Recent Deliveries</h2>
<table>
<thead>
<tr>
<th>Target</th>
<th>Run</th>
<th>Status</th>
<th>Created</th>
<th>Error</th>
</tr>
</thead>
<tbody>
{% for delivery in deliveries %}
<tr>
<td>{{ delivery.target.name }}</td>
<td><a href="{% url 'run_detail' delivery.run.id %}">Run {{ delivery.run.id }}</a> {{ delivery.run.host.host }}</td>
<td><span class="status {{ delivery.status }}">{{ delivery.status }}</span></td>
<td>{{ delivery.created_at }}</td>
<td class="muted">{{ delivery.error|default:"" }}</td>
</tr>
{% empty %}
<tr><td colspan="5" class="muted">No notification deliveries recorded yet.</td></tr>
{% endfor %}
</tbody>
</table>
</section>
{% endblock %}

View File

@@ -0,0 +1,178 @@
<section class="panel dashboard-hosts-panel" id="hosts">
<h2>Hosts</h2>
<div class="host-list">
{% for host in hosts %}
<article class="host-card">
<div class="host-card-header">
<div class="host-card-title">
<a href="{% url 'host_detail' host.host %}">{{ host.host }}</a>
<span class="muted">{{ host.address }}</span>
</div>
<div class="host-card-status">
<span class="status {% if host.enabled %}ok{% else %}skipped{% endif %}">{{ host.enabled|yesno:"enabled,disabled" }}</span>
{% if host.queued_run_count %}
<span class="status queued">queued {{ host.queued_run_count }}</span>
{% endif %}
{% if host.running_run_count %}
<span class="status running">running {{ host.running_run_count }}</span>
{% endif %}
{% if host.warning_run_count %}
<span class="status warning">warning {{ host.warning_run_count }}</span>
{% endif %}
{% if host.failed_run_count %}
<span class="status failed">failed {{ host.failed_run_count }}</span>
{% endif %}
{% if show_host_controls %}
{% if host.schedule %}
<span class="status {% if host.schedule.enabled %}ok{% else %}skipped{% endif %}">schedule {{ host.schedule.enabled|yesno:"on,paused" }}</span>
<span class="status {% if host.schedule.prune %}ok{% else %}skipped{% endif %}">retention {{ host.schedule.prune|yesno:"on,paused" }}</span>
{% else %}
<span class="status skipped">no schedule</span>
{% endif %}
{% endif %}
</div>
</div>
<div class="host-card-layout">
<div class="host-card-section">
<div class="host-card-section-title">Backup activity</div>
<div class="host-card-timeline">
<div class="host-card-item">
<div class="label">Latest Snapshot</div>
<div class="value">
{% if host.latest_snapshot %}
<a href="{% url 'snapshot_detail' host.latest_snapshot.id %}">{{ host.latest_snapshot.dirname }}</a>
<div class="muted">{{ host.latest_snapshot.kind }} {{ host.latest_snapshot.status }}</div>
{% else %}
<span class="muted">none</span>
{% endif %}
</div>
</div>
<div class="host-card-item">
<div class="label">Last Good Backup</div>
<div class="value">
{% if host.stats_summary.latest_good_run.id %}
<a href="{% url 'run_detail' host.stats_summary.latest_good_run.id %}">Run {{ host.stats_summary.latest_good_run.id }}</a>
<div class="muted">{{ host.stats_summary.latest_good_run.run_type }} {{ host.stats_summary.latest_good_run.duration_seconds|default:"" }}{% if host.stats_summary.latest_good_run.duration_seconds is not None %}s{% endif %}</div>
{% else %}
<span class="muted">none</span>
{% endif %}
</div>
</div>
<div class="host-card-item">
<div class="label">Latest Issue</div>
<div class="value">
{% if host.stats_summary.latest_problem_run.id %}
<a href="{% url 'run_detail' host.stats_summary.latest_problem_run.id %}">Run {{ host.stats_summary.latest_problem_run.id }}</a>
<div><span class="status {{ host.stats_summary.latest_problem_run.status }}">{{ host.stats_summary.latest_problem_run.status }}</span></div>
<div class="muted">{{ host.stats_summary.latest_problem_run.run_type }} {{ host.stats_summary.latest_problem_run.duration_seconds|default:"" }}{% if host.stats_summary.latest_problem_run.duration_seconds is not None %}s{% endif %}</div>
{% else %}
<span class="muted">none</span>
{% endif %}
</div>
</div>
<div class="host-card-item">
<div class="label">Next Run</div>
<div class="value">
{% if host.next_run_at %}
{{ host.next_run_at|date:"Y-m-d H:i T" }}
<div class="muted">{{ scheduler_timezone }}</div>
{% else %}
<span class="muted">none</span>
{% endif %}
</div>
</div>
</div>
</div>
<div class="host-card-section">
<div class="host-card-section-title">Snapshot health</div>
<div class="host-card-stats">
<div class="host-card-stat">
<div class="label">Snapshots</div>
<div class="value">{{ host.snapshot_count }}</div>
</div>
<div class="host-card-stat">
<div class="label">Runs</div>
<div class="value">{{ host.run_count }}</div>
</div>
<div class="host-card-stat">
<div class="label">New Data</div>
<div class="value">{{ host.stats_summary.latest_run.rsync.literal_data_bytes|filesizeformat }}</div>
</div>
<div class="host-card-stat">
<div class="label">Retention</div>
<div class="value">d{{ host.retention_daily }} w{{ host.retention_weekly }} m{{ host.retention_monthly }} y{{ host.retention_yearly }}</div>
</div>
<div class="host-card-stat">
<div class="label">Scheduled data</div>
<div class="value">{{ host.stats_summary.backup_data.scheduled.allocated_size_bytes|filesizeformat }}</div>
<div class="muted">unique {{ host.stats_summary.backup_data.scheduled.unique_apparent_size_bytes|filesizeformat }}</div>
</div>
<div class="host-card-stat">
<div class="label">Manual data</div>
<div class="value">{{ host.stats_summary.backup_data.manual.allocated_size_bytes|filesizeformat }}</div>
<div class="muted">unique {{ host.stats_summary.backup_data.manual.unique_apparent_size_bytes|filesizeformat }}</div>
</div>
<div class="host-card-stat">
<div class="label">Incomplete data</div>
<div class="value">{{ host.stats_summary.backup_data.incomplete.allocated_size_bytes|filesizeformat }}</div>
<div class="muted">measured from disk</div>
</div>
<div class="host-card-stat">
<div class="label">Total data</div>
<div class="value">{{ host.stats_summary.backup_data.total.allocated_size_bytes|filesizeformat }}</div>
<div class="muted">unique {{ host.stats_summary.backup_data.total.unique_apparent_size_bytes|filesizeformat }}</div>
</div>
</div>
</div>
</div>
{% if host.retention_warning.has_warning %}
<div class="host-card-warning">
<span class="status warning">retention</span>
{% if host.retention_warning.prune_exceeded %}
Scheduled prune would delete {{ host.retention_warning.delete_count }} snapshot(s), above max {{ host.retention_warning.max_delete }}.
{% endif %}
{% if host.retention_warning.incomplete_count %}
{{ host.retention_warning.incomplete_count }} incomplete snapshot(s) need review.
<form method="post" action="{% url 'resolve_host_incomplete_reviews' host.host %}">
{% csrf_token %}
<button type="submit" class="secondary">Mark reviewed</button>
</form>
{% endif %}
{% if host.retention_warning.error %}
{{ host.retention_warning.error }}
{% endif %}
</div>
{% endif %}
{% if show_host_controls %}
<div class="host-card-actions">
<a class="button-link compact secondary" href="{% url 'host_detail' host.host %}">Open</a>
<a class="button-link compact secondary" href="{% url 'edit_host_config' host.host %}">Edit config</a>
<a class="button-link compact secondary" href="{% url 'edit_host_schedule' host.host %}">{% if host.schedule %}Edit schedule{% else %}Create schedule{% endif %}</a>
<form class="inline-form" method="post" action="{% url 'update_host_state' host.host %}">
{% csrf_token %}
<input type="hidden" name="next" value="{{ request.get_full_path }}">
<input type="hidden" name="action" value="{% if host.enabled %}disable_host{% else %}enable_host{% endif %}">
<button class="compact {% if host.enabled %}secondary{% endif %}" type="submit">{{ host.enabled|yesno:"Disable host,Enable host" }}</button>
</form>
{% if host.schedule %}
<form class="inline-form" method="post" action="{% url 'update_host_state' host.host %}">
{% csrf_token %}
<input type="hidden" name="next" value="{{ request.get_full_path }}">
<input type="hidden" name="action" value="{% if host.schedule.enabled %}disable_schedule{% else %}enable_schedule{% endif %}">
<button class="compact secondary" type="submit">{{ host.schedule.enabled|yesno:"Pause schedule,Resume schedule" }}</button>
</form>
<form class="inline-form" method="post" action="{% url 'update_host_state' host.host %}">
{% csrf_token %}
<input type="hidden" name="next" value="{{ request.get_full_path }}">
<input type="hidden" name="action" value="{% if host.schedule.prune %}disable_prune{% else %}enable_prune{% endif %}">
<button class="compact secondary" type="submit">{{ host.schedule.prune|yesno:"Pause retention,Resume retention" }}</button>
</form>
{% endif %}
</div>
{% endif %}
</article>
{% empty %}
<p class="muted">No hosts configured yet.</p>
{% endfor %}
</div>
</section>

View File

@@ -0,0 +1,152 @@
<section class="dashboard-priority-grid" aria-label="Operator priorities">
<article class="panel priority-panel dashboard-panel-required">
<h2>Required Action</h2>
{% if action_items %}
<div class="action-list">
{% for item in action_items %}
<a class="action-row {{ item.status }}" href="{{ item.url }}">
<span class="status {{ item.status }}">{{ item.label }}</span>
<span>
<strong>{{ item.host.host }}</strong>
<span class="muted">{{ item.message }}</span>
</span>
</a>
{% endfor %}
</div>
{% elif counts.hosts %}
<p><span class="status ok">ok</span> No queued, running, unreviewed warning/failed runs, or retention warnings.</p>
{% else %}
<p class="muted">Add a host to start tracking backup status here.</p>
{% endif %}
{% if counts.running_runs or counts.queued_runs %}
<div class="operator-state">
{% if counts.running_runs %}
<a class="status-summary running" href="{% url 'runs_list' %}?status=running">
<span class="status running">running</span>
<strong>{{ counts.running_runs }} backup run{{ counts.running_runs|pluralize }} in progress.</strong>
</a>
{% endif %}
{% if counts.queued_runs %}
<a class="status-summary queued" href="{% url 'runs_list' %}?status=queued">
<span class="status queued">queued</span>
<strong>{{ counts.queued_runs }} backup run{{ counts.queued_runs|pluralize }} waiting.</strong>
</a>
{% endif %}
</div>
{% endif %}
</article>
<article class="panel priority-panel dashboard-panel-schedules">
<h2>Next Scheduled Work <a class="button-link secondary compact" href="{% url 'schedules_list' %}">View all</a></h2>
{% if next_schedule_rows %}
<div class="schedule-list">
{% for row in next_schedule_rows %}
<a class="schedule-row" href="{% url 'host_detail' row.schedule.host.host %}">
<span>
<strong>{{ row.schedule.host.host }}</strong>
<span class="muted">{{ row.schedule.cron_expr }}</span>
</span>
<span class="schedule-time">
{% if row.next_run_at %}
{{ row.next_run_at|date:"Y-m-d H:i T" }}
<span class="muted">{{ scheduler_timezone }}</span>
{% else %}
<span class="muted">not due</span>
{% endif %}
</span>
</a>
{% endfor %}
</div>
{% else %}
<p class="muted">No enabled schedules yet.</p>
{% endif %}
</article>
<article class="panel priority-panel dashboard-panel-activity">
<h2>Recent Activity <a class="button-link secondary compact" href="{% url 'runs_list' %}">View all</a></h2>
{% if recent_runs %}
<div class="activity-list">
{% for run in recent_runs %}
<a class="activity-row" href="{% url 'run_detail' run.id %}">
<span class="status {{ run.status }}">{{ run.status }}</span>
<span>
<strong>Run {{ run.id }}</strong>
<span class="muted">{{ run.host.host }} · {{ run.run_type }} · {{ run.started_at|default:run.created_at }}</span>
</span>
</a>
{% endfor %}
</div>
{% else %}
<p class="muted">No backup runs recorded yet.</p>
{% endif %}
</article>
<article class="panel priority-panel dashboard-panel-storage">
<h2>Storage Pressure</h2>
{% if stats_summary.runs_sampled %}
<div class="storage-priority">
<div>
<div class="label">Backup root used</div>
<div class="value">
{% if stats_summary.capacity.used_percent is not None %}
{{ stats_summary.capacity.used_percent|floatformat:1 }}%
{% else %}
unknown
{% endif %}
</div>
{% if stats_summary.capacity.used_percent is not None %}
<div class="storage-meter" aria-label="Backup root storage usage">
<span style="width: {{ stats_summary.capacity.used_percent|floatformat:0 }}%"></span>
</div>
{% endif %}
</div>
<div class="storage-priority-facts">
<div>
<span class="label">Runway</span>
<strong>
{% if stats_summary.estimated_days_until_full %}
{{ stats_summary.estimated_days_until_full }} days
{% elif stats_summary.estimated_runs_until_full %}
{{ stats_summary.estimated_runs_until_full }} runs
{% else %}
unknown
{% endif %}
</strong>
</div>
<div>
<span class="label">New data</span>
<strong>{{ stats_summary.avg_daily_literal_data_bytes|filesizeformat }}/day</strong>
</div>
<div>
<span class="label">Available</span>
<strong>{{ stats_summary.capacity.available_bytes|filesizeformat }}</strong>
</div>
</div>
</div>
{% else %}
<p class="muted">Storage pressure appears after the first completed backup with stats.</p>
{% endif %}
<div class="storage-priority-facts">
<div>
<span class="label">Scheduled data</span>
<strong>{{ stats_summary.backup_data.scheduled.allocated_size_bytes|filesizeformat }}</strong>
<span class="muted">unique {{ stats_summary.backup_data.scheduled.unique_apparent_size_bytes|filesizeformat }}</span>
</div>
<div>
<span class="label">Manual data</span>
<strong>{{ stats_summary.backup_data.manual.allocated_size_bytes|filesizeformat }}</strong>
<span class="muted">unique {{ stats_summary.backup_data.manual.unique_apparent_size_bytes|filesizeformat }}</span>
</div>
<div>
<span class="label">Incomplete data</span>
<strong>{{ stats_summary.backup_data.incomplete.allocated_size_bytes|filesizeformat }}</strong>
<span class="muted">measured from disk</span>
</div>
<div>
<span class="label">Total snapshot data</span>
<strong>{{ stats_summary.backup_data.total.allocated_size_bytes|filesizeformat }}</strong>
<span class="muted">unique {{ stats_summary.backup_data.total.unique_apparent_size_bytes|filesizeformat }}</span>
</div>
</div>
</article>
</section>

View File

@@ -0,0 +1,222 @@
<section class="grid" aria-label="Run summary">
<div class="metric"><div class="label">Host</div><div class="value">{{ run.host.host }}</div></div>
<div class="metric"><div class="label">Status</div><div class="value"><span class="status {{ run.status }}">{{ run.status }}</span></div></div>
<div class="metric"><div class="label">Type</div><div class="value">{{ run.run_type }}</div></div>
<div class="metric"><div class="label">Rsync</div><div class="value">{{ run.rsync_exit_code|default:"" }}</div></div>
</section>
{% if can_cancel %}
<section class="panel highlight warning">
<h2>Run Control</h2>
<p>
Cancelling a queued run stops it immediately. Cancelling a running run asks the worker to stop
and records the cancellation request on this run.
</p>
<form method="post" action="{% url 'cancel_run' run.id %}">
{% csrf_token %}
<div class="form-actions">
<button type="submit" class="danger">Cancel run</button>
</div>
</form>
</section>
{% endif %}
{% if failure %}
<section class="panel highlight failed">
<h2>Failure</h2>
<div class="stack">
<div><strong>Category:</strong> {{ failure.category|default:"unknown" }}</div>
<div><strong>Summary:</strong> {{ failure_summary }}</div>
<div><strong>Hint:</strong> {{ failure.hint|default:"" }}</div>
</div>
</section>
{% endif %}
{% if can_manage_control_panel and run.status == "failed" or can_manage_control_panel and run.status == "warning" %}
{% if not run.reviewed_at %}
<section class="panel highlight warning">
<h2>Review Required</h2>
<p>Mark this run as reviewed after you have checked the failure or warning and no longer need it in the action queue.</p>
<form method="post" action="{% url 'resolve_run_review' run.id %}">
{% csrf_token %}
<div class="form-actions">
<button type="submit" class="secondary">Mark reviewed</button>
</div>
</form>
</section>
{% endif %}
{% endif %}
{% if run.reviewed_at %}
<section class="panel highlight success">
<h2>Review</h2>
<div class="stack">
<div><strong>Reviewed:</strong> {{ run.reviewed_at }}</div>
<div><strong>Reviewed by:</strong> {{ run.reviewed_by|default:"unknown" }}</div>
</div>
</section>
{% endif %}
{% if dry_run_summary %}
<section class="panel highlight {{ dry_run_summary.highlight_class }}">
<h2>Run Progress</h2>
<section class="grid" aria-label="Run progress">
<div class="metric"><div class="label">Status</div><div class="value">{{ dry_run_summary.status }}</div></div>
<div class="metric">
<div class="label">Mode</div>
<div class="value">dry run</div>
</div>
<div class="metric"><div class="label">Files Seen</div><div class="value">{{ dry_run_summary.files_seen|default:"unknown" }}</div></div>
<div class="metric"><div class="label">Would Transfer</div><div class="value">{{ dry_run_summary.files_would_transfer|default:"unknown" }}</div></div>
<div class="metric"><div class="label">Transfer Estimate</div><div class="value">{{ dry_run_summary.transfer_estimate_bytes|filesizeformat }}</div></div>
<div class="metric"><div class="label">Total Size</div><div class="value">{{ dry_run_summary.total_file_size_bytes|filesizeformat }}</div></div>
<div class="metric"><div class="label">Link-Dest Saving</div><div class="value">{{ dry_run_summary.link_dest_estimated_savings_bytes|filesizeformat }}</div></div>
</section>
<div class="stack">
{% if dry_run_summary.duration_seconds is not None %}
<div><strong>Duration:</strong> {{ dry_run_summary.duration_seconds }}s</div>
{% endif %}
<div>
<strong>Log:</strong>
{% if dry_run_summary.log_available and can_manage_control_panel %}
<a href="{% url 'run_rsync_log' run.id %}">Open full rsync log</a>
{% elif rsync_log_path %}
<span class="muted">{{ rsync_log_path }} (missing)</span>
{% else %}
<span class="muted">not recorded yet</span>
{% endif %}
</div>
{% if dry_run_summary.warnings %}
<div><strong>Warnings:</strong></div>
<ul>
{% for warning in dry_run_summary.warnings %}
<li>{{ warning }}</li>
{% endfor %}
</ul>
{% else %}
<div><strong>Warnings:</strong> none recorded</div>
{% endif %}
</div>
</section>
{% endif %}
{% if live_progress %}
<section class="panel highlight running">
<h2>Run Progress</h2>
<section class="grid" aria-label="Run progress">
<div class="metric">
<div class="label">Status</div>
<div class="value">{{ run.status }}</div>
</div>
<div class="metric">
<div class="label">Mode</div>
<div class="value">backup</div>
</div>
<div class="metric">
<div class="label">Phase</div>
<div class="value">{{ live_progress.phase }}</div>
</div>
<div class="metric">
<div class="label">Rsync PID</div>
<div class="value">{{ live_progress.rsync_pid|default:"" }}</div>
</div>
<div class="metric">
<div class="label">Log Updated</div>
<div class="value">
{% if live_progress.log.exists %}
{{ live_progress.log.seconds_since_modified }}s ago
{% else %}
missing
{% endif %}
</div>
</div>
<div class="metric">
<div class="label">Log Size</div>
<div class="value">{{ live_progress.log.size_bytes|filesizeformat }}</div>
</div>
{% if live_progress.snapshot.exists %}
<div class="metric">
<div class="label">Data Files</div>
<div class="value">{% if live_progress.snapshot.scan_limited %}at least {% endif %}{{ live_progress.snapshot.files }}</div>
</div>
<div class="metric">
<div class="label">Data Size</div>
<div class="value">{% if live_progress.snapshot.scan_limited %}at least {% endif %}{{ live_progress.snapshot.apparent_size_bytes|filesizeformat }}</div>
</div>
{% endif %}
</section>
<div class="stack">
{% if live_progress.snapshot.path %}
<div><strong>Snapshot path:</strong> {{ live_progress.snapshot.path }}</div>
{% endif %}
{% if live_progress.snapshot.scan_limited %}
<div class="muted">Progress scan was capped to keep the UI responsive.</div>
{% endif %}
{% if live_progress.log.path %}
<div>
<strong>Log:</strong>
{% if live_progress.log.exists and can_manage_control_panel %}
<a href="{% url 'run_rsync_log' run.id %}">Open full rsync log</a>
{% else %}
<span class="muted">{{ live_progress.log.path }} (missing)</span>
{% endif %}
</div>
<div><strong>Log path:</strong> {{ live_progress.log.path }}</div>
{% endif %}
<div><strong>Warnings:</strong> none recorded</div>
</div>
</section>
{% endif %}
<div class="two-col">
<section class="panel">
<h2>Timing</h2>
<div class="stack">
<div><strong>Created:</strong> {{ run.created_at }}</div>
<div><strong>Started:</strong> {{ run.started_at|default:"" }}</div>
<div><strong>Ended:</strong> {{ run.ended_at|default:"" }}</div>
{% if execution %}
<div><strong>Worker:</strong> {{ execution.worker_host|default:"unknown" }}{% if execution.worker_pid %} pid {{ execution.worker_pid }}{% endif %}</div>
<div><strong>Worker heartbeat:</strong> {{ execution.heartbeat_at|default:"" }}</div>
{% endif %}
</div>
</section>
<section class="panel">
<h2>Snapshot</h2>
<div class="stack">
<div><strong>Snapshot:</strong> {% if run.snapshot %}<a href="{% url 'snapshot_detail' run.snapshot.id %}">{{ run.snapshot.dirname }}</a>{% else %}{{ run.snapshot_path|default:"" }}{% endif %}</div>
<div><strong>Base:</strong> {{ run.base_path|default:"" }}</div>
<div>
<strong>Rsync log:</strong>
{% if rsync_log_exists and can_manage_control_panel %}
<a href="{% url 'run_rsync_log' run.id %}">{{ rsync_log_path }}</a>
{% elif rsync_log_path %}
<span class="muted">{{ rsync_log_path }} (missing)</span>
{% else %}
<span class="muted">none</span>
{% endif %}
</div>
</div>
</section>
</div>
<section class="panel">
<h2>Rsync Log</h2>
<div class="stack spaced">
{% if rsync_log_exists and can_manage_control_panel %}
<div><a href="{% url 'run_rsync_log' run.id %}">Open full rsync log</a></div>
<div class="muted">{{ rsync_log_path }}</div>
{% elif rsync_log_path %}
<div class="muted">{{ rsync_log_path }} (missing)</div>
{% else %}
<div class="muted">No rsync log path recorded yet.</div>
{% endif %}
</div>
{% if rsync_log_tail %}
<pre>{% for line in rsync_log_tail %}{{ line }}{% if not forloop.last %}
{% endif %}{% endfor %}</pre>
{% else %}
<p class="muted">No recent rsync log output recorded yet.</p>
{% endif %}
</section>

View File

@@ -16,7 +16,7 @@
<section class="panel"> <section class="panel">
<h2>Filters</h2> <h2>Filters</h2>
<form method="get" class="form-grid"> <form method="get" class="filter-form">
<div class="field"> <div class="field">
<label for="host">Host</label> <label for="host">Host</label>
<select id="host" name="host"> <select id="host" name="host">
@@ -35,7 +35,7 @@
{% endfor %} {% endfor %}
</select> </select>
</div> </div>
<div class="actions"> <div class="form-actions">
<button type="submit">Apply filters</button> <button type="submit">Apply filters</button>
<a class="button-link secondary" href="{% url 'purged_snapshots' %}">Clear</a> <a class="button-link secondary" href="{% url 'purged_snapshots' %}">Clear</a>
</div> </div>

View File

@@ -45,8 +45,9 @@
snapshots automatically because they can indicate an interrupted backup that should be inspected first. snapshots automatically because they can indicate an interrupted backup that should be inspected first.
</p> </p>
<p> <p>
After inspection, use the dedicated cleanup form below to delete only incomplete snapshot directories and their {{ incomplete_unreviewed_count }} still need review. After inspection, mark them reviewed and use the dedicated
tracking records. Successful scheduled and manual snapshots are not touched by this cleanup. cleanup form below to delete only incomplete snapshot directories and their tracking records. Successful
scheduled and manual snapshots are not touched by this cleanup.
</p> </p>
</section> </section>
{% endif %} {% endif %}
@@ -104,8 +105,12 @@
</section> </section>
{% if plan.delete %} {% if plan.delete %}
<section class="panel"> <section class="panel highlight warning">
<h2>Apply Retention</h2> <h2>Apply Retention</h2>
<p class="muted">
This permanently deletes the snapshot directories listed in Would Delete. Confirm the host and delete count
before applying the plan.
</p>
<form method="post" action="{% url 'apply_host_retention' host.host %}" class="form-grid"> <form method="post" action="{% url 'apply_host_retention' host.host %}" class="form-grid">
{% csrf_token %} {% csrf_token %}
{{ apply_form.non_field_errors }} {{ apply_form.non_field_errors }}
@@ -138,8 +143,9 @@
<div class="helptext">{{ apply_form.confirm_delete_count.help_text }}</div> <div class="helptext">{{ apply_form.confirm_delete_count.help_text }}</div>
</div> </div>
<div class="actions"> <div class="form-actions">
<button type="submit">Apply retention</button> <button type="submit" class="danger">Apply retention</button>
<a class="button-link secondary" href="{% url 'host_retention_plan' host.host %}?kind={{ kind }}">Cancel</a>
</div> </div>
</form> </form>
</section> </section>
@@ -182,6 +188,7 @@
<th>Dirname</th> <th>Dirname</th>
<th>Started</th> <th>Started</th>
<th>Status</th> <th>Status</th>
<th>Review</th>
<th>Reason</th> <th>Reason</th>
<th>Path</th> <th>Path</th>
</tr> </tr>
@@ -192,6 +199,14 @@
<td>{{ snapshot.dirname }}</td> <td>{{ snapshot.dirname }}</td>
<td>{{ snapshot.dt }}</td> <td>{{ snapshot.dt }}</td>
<td>{{ snapshot.status|default:"" }}</td> <td>{{ snapshot.status|default:"" }}</td>
<td>
{% if snapshot.reviewed %}
<span class="status ok">reviewed</span>
<span class="muted">{{ snapshot.reviewed_by|default:"unknown" }}</span>
{% else %}
<span class="status warning">needs review</span>
{% endif %}
</td>
<td>{{ snapshot.reason }}</td> <td>{{ snapshot.reason }}</td>
<td class="muted">{{ snapshot.path }}</td> <td class="muted">{{ snapshot.path }}</td>
</tr> </tr>
@@ -200,6 +215,20 @@
</table> </table>
<h3>Cleanup Incomplete Snapshots</h3> <h3>Cleanup Incomplete Snapshots</h3>
{% if incomplete_unreviewed_count %}
<p class="muted">
Cleanup is blocked until all incomplete snapshots are reviewed. This extra step makes it explicit that the
interrupted backup was inspected before deletion.
</p>
<form method="post" action="{% url 'resolve_host_incomplete_reviews' host.host %}" class="actions inline">
{% csrf_token %}
<button type="submit" class="secondary">Mark incomplete snapshots reviewed</button>
</form>
{% else %}
<p class="muted">
This deletes only reviewed incomplete snapshot directories and their tracking records. Successful manual and
scheduled snapshots are not touched.
</p>
<form method="post" action="{% url 'cleanup_host_incomplete_snapshots' host.host %}" class="form-grid"> <form method="post" action="{% url 'cleanup_host_incomplete_snapshots' host.host %}" class="form-grid">
{% csrf_token %} {% csrf_token %}
{{ incomplete_cleanup_form.non_field_errors }} {{ incomplete_cleanup_form.non_field_errors }}
@@ -225,10 +254,12 @@
<div class="helptext">{{ incomplete_cleanup_form.confirm_delete_count.help_text }}</div> <div class="helptext">{{ incomplete_cleanup_form.confirm_delete_count.help_text }}</div>
</div> </div>
<div class="actions"> <div class="form-actions">
<button type="submit">Delete incomplete snapshots</button> <button type="submit" class="danger">Delete incomplete snapshots</button>
<a class="button-link secondary" href="{% url 'host_retention_plan' host.host %}?kind={{ kind }}">Cancel</a>
</div> </div>
</form> </form>
{% endif %}
</section> </section>
{% endif %} {% endif %}
{% endblock %} {% endblock %}

View File

@@ -11,121 +11,28 @@
</div> </div>
<section class="actions" aria-label="Run actions"> <section class="actions" aria-label="Run actions">
<a class="button-link" href="{% url 'host_detail' run.host.host %}">Back to host</a> <a class="button-link" href="{% url 'host_detail' run.host.host %}">Back to host</a>
{% if can_cancel %}
<form method="post" action="{% url 'cancel_run' run.id %}">
{% csrf_token %}
<button type="submit" class="secondary">Cancel run</button>
</form>
{% endif %}
{% if run.status == "failed" or run.status == "warning" %}
{% if not run.reviewed_at %}
<form method="post" action="{% url 'resolve_run_review' run.id %}">
{% csrf_token %}
<button type="submit" class="secondary">Mark reviewed</button>
</form>
{% endif %}
{% endif %}
</section> </section>
</header> </header>
<section class="grid" aria-label="Run summary"> {% if can_auto_refresh %}
<div class="metric"><div class="label">Host</div><div class="value">{{ run.host.host }}</div></div> <section class="panel refresh-controls" aria-label="Live refresh controls">
<div class="metric"><div class="label">Status</div><div class="value"><span class="status {{ run.status }}">{{ run.status }}</span></div></div>
<div class="metric"><div class="label">Type</div><div class="value">{{ run.run_type }}</div></div>
<div class="metric"><div class="label">Rsync</div><div class="value">{{ run.rsync_exit_code|default:"" }}</div></div>
</section>
{% if failure %}
<section class="panel highlight failed">
<h2>Failure</h2>
<div class="stack">
<div><strong>Category:</strong> {{ failure.category|default:"unknown" }}</div>
<div><strong>Summary:</strong> {{ failure_summary }}</div>
<div><strong>Hint:</strong> {{ failure.hint|default:"" }}</div>
</div>
</section>
{% endif %}
{% if run.reviewed_at %}
<section class="panel highlight success">
<h2>Review</h2>
<div class="stack">
<div><strong>Reviewed:</strong> {{ run.reviewed_at }}</div>
<div><strong>Reviewed by:</strong> {{ run.reviewed_by|default:"unknown" }}</div>
</div>
</section>
{% endif %}
{% if dry_run_summary %}
<section class="panel highlight {{ dry_run_summary.highlight_class }}">
<h2>Dry Run Summary</h2>
<section class="grid" aria-label="Dry run summary">
<div class="metric"><div class="label">Status</div><div class="value">{{ dry_run_summary.status }}</div></div>
<div class="metric"><div class="label">Files Seen</div><div class="value">{{ dry_run_summary.files_seen|default:"unknown" }}</div></div>
<div class="metric"><div class="label">Would Transfer</div><div class="value">{{ dry_run_summary.files_would_transfer|default:"unknown" }}</div></div>
<div class="metric"><div class="label">Transfer Estimate</div><div class="value">{{ dry_run_summary.transfer_estimate_bytes|filesizeformat }}</div></div>
<div class="metric"><div class="label">Total Size</div><div class="value">{{ dry_run_summary.total_file_size_bytes|filesizeformat }}</div></div>
<div class="metric"><div class="label">Link-Dest Saving</div><div class="value">{{ dry_run_summary.link_dest_estimated_savings_bytes|filesizeformat }}</div></div>
</section>
<div class="stack">
{% if dry_run_summary.duration_seconds is not None %}
<div><strong>Duration:</strong> {{ dry_run_summary.duration_seconds }}s</div>
{% endif %}
<div> <div>
<strong>Log:</strong> <h2>Live Updates</h2>
{% if dry_run_summary.log_available %} <p class="muted">Auto-refresh is <strong data-refresh-state="run-live-region">on</strong> while this run is active.</p>
<a href="{% url 'run_rsync_log' run.id %}">Open full rsync log</a>
{% elif rsync_log_path %}
<span class="muted">{{ rsync_log_path }} (missing)</span>
{% else %}
<span class="muted">not recorded yet</span>
{% endif %}
</div>
{% if dry_run_summary.warnings %}
<div><strong>Warnings:</strong></div>
<ul>
{% for warning in dry_run_summary.warnings %}
<li>{{ warning }}</li>
{% endfor %}
</ul>
{% else %}
<div><strong>Warnings:</strong> none recorded</div>
{% endif %}
</div> </div>
<button type="button" class="secondary" data-refresh-toggle data-refresh-target="run-live-region">Pause refresh</button>
</section> </section>
{% endif %} {% endif %}
<div class="two-col"> <div
<section class="panel"> id="run-live-region"
<h2>Timing</h2> data-refresh-url="{% url 'run_detail_live' run.id %}"
<div class="stack"> data-refresh-interval="5000"
<div><strong>Created:</strong> {{ run.created_at }}</div> data-refresh-active="{{ can_auto_refresh|yesno:'true,false' }}"
<div><strong>Started:</strong> {{ run.started_at|default:"" }}</div> data-refresh-paused="false"
<div><strong>Ended:</strong> {{ run.ended_at|default:"" }}</div> aria-live="polite"
{% if execution %} >
<div><strong>Worker:</strong> {{ execution.worker_host|default:"unknown" }}{% if execution.worker_pid %} pid {{ execution.worker_pid }}{% endif %}</div> {% include "pobsync_backend/partials/run_detail_live.html" %}
<div><strong>Worker heartbeat:</strong> {{ execution.heartbeat_at|default:"" }}</div>
{% endif %}
</div>
</section>
<section class="panel">
<h2>Snapshot</h2>
<div class="stack">
<div><strong>Snapshot:</strong> {% if run.snapshot %}<a href="{% url 'snapshot_detail' run.snapshot.id %}">{{ run.snapshot.dirname }}</a>{% else %}{{ run.snapshot_path|default:"" }}{% endif %}</div>
<div><strong>Base:</strong> {{ run.base_path|default:"" }}</div>
<div>
<strong>Rsync log:</strong>
{% if rsync_log_exists %}
<a href="{% url 'run_rsync_log' run.id %}">{{ rsync_log_path }}</a>
{% elif rsync_log_path %}
<span class="muted">{{ rsync_log_path }} (missing)</span>
{% else %}
<span class="muted">none</span>
{% endif %}
</div>
</div>
</section>
</div> </div>
{% if requested %} {% if requested %}
@@ -143,6 +50,10 @@
<section class="panel"> <section class="panel">
<h2>Rsync Command</h2> <h2>Rsync Command</h2>
<p class="muted">
<strong>Bandwidth limit:</strong>
{% if rsync_bwlimit_kbps %}{{ rsync_bwlimit_kbps }} KB/s{% else %}unlimited{% endif %}
</p>
{% if rsync_command %} {% if rsync_command %}
<pre>{% for part in rsync_command %}{{ part }}{% if not forloop.last %} <pre>{% for part in rsync_command %}{{ part }}{% if not forloop.last %}
{% endif %}{% endfor %}</pre> {% endif %}{% endfor %}</pre>
@@ -151,26 +62,6 @@
{% endif %} {% endif %}
</section> </section>
<section class="panel">
<h2>Rsync Log</h2>
<div class="stack spaced">
{% if rsync_log_exists %}
<div><a href="{% url 'run_rsync_log' run.id %}">Open full rsync log</a></div>
<div class="muted">{{ rsync_log_path }}</div>
{% elif rsync_log_path %}
<div class="muted">{{ rsync_log_path }} (missing)</div>
{% else %}
<div class="muted">No rsync log path recorded yet.</div>
{% endif %}
</div>
{% if rsync_log_tail %}
<pre>{% for line in rsync_log_tail %}{{ line }}{% if not forloop.last %}
{% endif %}{% endfor %}</pre>
{% else %}
<p class="muted">No recent rsync log output recorded yet.</p>
{% endif %}
</section>
{% if stats %} {% if stats %}
<section class="panel"> <section class="panel">
<h2>Stats</h2> <h2>Stats</h2>

View File

@@ -16,7 +16,7 @@
<section class="panel"> <section class="panel">
<h2>Filters</h2> <h2>Filters</h2>
<form method="get" class="form-grid"> <form method="get" class="filter-form">
<div class="field"> <div class="field">
<label for="status">Status</label> <label for="status">Status</label>
<select id="status" name="status"> <select id="status" name="status">
@@ -52,7 +52,7 @@
<option value="reviewed" {% if selected_review == "reviewed" %}selected{% endif %}>Reviewed</option> <option value="reviewed" {% if selected_review == "reviewed" %}selected{% endif %}>Reviewed</option>
</select> </select>
</div> </div>
<div class="actions"> <div class="form-actions">
<button type="submit">Apply filters</button> <button type="submit">Apply filters</button>
<a class="button-link secondary" href="{% url 'runs_list' %}">Clear</a> <a class="button-link secondary" href="{% url 'runs_list' %}">Clear</a>
</div> </div>
@@ -95,7 +95,22 @@
<span class="muted">none</span> <span class="muted">none</span>
{% endif %} {% endif %}
</td> </td>
<td>{% if run.reviewed_at %}reviewed{% elif run.status == "failed" or run.status == "warning" %}<span class="status warning">needed</span>{% else %}<span class="muted">none</span>{% endif %}</td> <td>
{% if run.reviewed_at %}
reviewed
{% elif run.status == "failed" or run.status == "warning" %}
<div class="stack">
<span class="status warning">needed</span>
<form class="inline-form" method="post" action="{% url 'resolve_run_review' run.id %}">
{% csrf_token %}
<input type="hidden" name="next" value="{{ request.get_full_path }}">
<button type="submit" class="secondary compact">Mark reviewed</button>
</form>
</div>
{% else %}
<span class="muted">none</span>
{% endif %}
</td>
</tr> </tr>
{% empty %} {% empty %}
<tr><td colspan="9" class="muted">No runs matched the current filter.</td></tr> <tr><td colspan="9" class="muted">No runs matched the current filter.</td></tr>

View File

@@ -30,8 +30,9 @@
</div> </div>
{% endfor %} {% endfor %}
<div class="actions"> <div class="form-actions">
<button type="submit">Save schedule</button> <button type="submit">Save schedule</button>
<a class="button-link secondary" href="{% url 'host_detail' host.host %}">Cancel</a>
</div> </div>
</form> </form>
</section> </section>

View File

@@ -16,7 +16,7 @@
<section class="panel"> <section class="panel">
<h2>Filters</h2> <h2>Filters</h2>
<form method="get" class="form-grid"> <form method="get" class="filter-form">
<div class="field"> <div class="field">
<label for="host">Host</label> <label for="host">Host</label>
<select id="host" name="host"> <select id="host" name="host">
@@ -42,7 +42,7 @@
<option value="no" {% if selected_prune == "no" %}selected{% endif %}>Prune disabled</option> <option value="no" {% if selected_prune == "no" %}selected{% endif %}>Prune disabled</option>
</select> </select>
</div> </div>
<div class="actions"> <div class="form-actions">
<button type="submit">Apply filters</button> <button type="submit">Apply filters</button>
<a class="button-link secondary" href="{% url 'schedules_list' %}">Clear</a> <a class="button-link secondary" href="{% url 'schedules_list' %}">Clear</a>
</div> </div>

View File

@@ -16,7 +16,7 @@
<section class="panel"> <section class="panel">
<h2>Filters</h2> <h2>Filters</h2>
<form method="get" class="form-grid"> <form method="get" class="filter-form">
<div class="field"> <div class="field">
<label for="host">Host</label> <label for="host">Host</label>
<select id="host" name="host"> <select id="host" name="host">
@@ -44,7 +44,7 @@
{% endfor %} {% endfor %}
</select> </select>
</div> </div>
<div class="actions"> <div class="form-actions">
<button type="submit">Apply filters</button> <button type="submit">Apply filters</button>
<a class="button-link secondary" href="{% url 'snapshots_list' %}">Clear</a> <a class="button-link secondary" href="{% url 'snapshots_list' %}">Clear</a>
</div> </div>

View File

@@ -41,8 +41,9 @@
</div> </div>
{% endfor %} {% endfor %}
<div class="actions"> <div class="form-actions">
<button type="submit">{% if credential %}Save SSH key{% else %}Create SSH key{% endif %}</button> <button type="submit">{% if credential %}Save SSH key{% else %}Create SSH key{% endif %}</button>
<a class="button-link secondary" href="{% url 'ssh_credentials' %}">Cancel</a>
</div> </div>
</form> </form>
</section> </section>
@@ -64,7 +65,10 @@
<label for="confirm_name">Confirm key name</label> <label for="confirm_name">Confirm key name</label>
<input id="confirm_name" name="confirm_name" type="text" {% if credential.hosts.exists or credential.global_configs.exists %}disabled{% endif %}> <input id="confirm_name" name="confirm_name" type="text" {% if credential.hosts.exists or credential.global_configs.exists %}disabled{% endif %}>
</div> </div>
<div class="form-actions">
<button type="submit" class="danger" {% if credential.hosts.exists or credential.global_configs.exists %}disabled{% endif %}>Delete SSH key</button> <button type="submit" class="danger" {% if credential.hosts.exists or credential.global_configs.exists %}disabled{% endif %}>Delete SSH key</button>
<a class="button-link secondary" href="{% url 'ssh_credentials' %}">Cancel</a>
</div>
</form> </form>
</section> </section>
{% endif %} {% endif %}

View File

@@ -29,8 +29,9 @@
</div> </div>
{% endfor %} {% endfor %}
<div class="actions"> <div class="form-actions">
<button type="submit">Generate SSH key</button> <button type="submit">Generate SSH key</button>
<a class="button-link secondary" href="{% url 'ssh_credentials' %}">Cancel</a>
</div> </div>
</form> </form>
</section> </section>

View File

@@ -0,0 +1,122 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}Updater | pobsync{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Operations</div>
<h1>Updater</h1>
<div class="page-subtitle">Check Gitea releases, pull the installed git checkout, and run the native systemd updater.</div>
</div>
<section class="actions" aria-label="Updater actions">
<a class="button-link secondary" href="{% url 'dashboard' %}">Back to dashboard</a>
</section>
</header>
<section class="panel-grid">
<section class="panel">
<h2>Installed App</h2>
<dl class="detail-list">
<dt>Version</dt>
<dd>{{ status.installed_version }}</dd>
<dt>Git branch</dt>
<dd>{{ status.git.branch|default:"unknown" }}</dd>
<dt>Git commit</dt>
<dd>{{ status.git.commit|default:"unknown" }}</dd>
<dt>Git describe</dt>
<dd>{{ status.git.describe|default:"unknown" }}</dd>
<dt>App directory</dt>
<dd>{{ status.app_dir }}</dd>
</dl>
</section>
<section class="panel">
<h2>Release Check</h2>
<dl class="detail-list">
<dt>Status</dt>
<dd>
{% if status.update_available == True %}
<span class="status warning">update available</span>
{% elif status.update_available == False %}
<span class="status ok">up to date</span>
{% elif status.release_check_configured %}
<span class="status skipped">not checked</span>
{% else %}
<span class="status skipped">not configured</span>
{% endif %}
</dd>
<dt>Latest release</dt>
<dd>
{% if status.latest_release %}
{% if status.latest_release.html_url %}
<a href="{{ status.latest_release.html_url }}">
{% if status.latest_release.tag_name %}{{ status.latest_release.tag_name }}{% else %}{{ status.latest_release.name }}{% endif %}
</a>
{% else %}
{% if status.latest_release.tag_name %}{{ status.latest_release.tag_name }}{% else %}{{ status.latest_release.name }}{% endif %}
{% endif %}
{% else %}
none
{% endif %}
</dd>
<dt>Release endpoint</dt>
<dd>{% if status.release_check_configured %}configured{% else %}set POBSYNC_UPDATE_RELEASES_URL{% endif %}</dd>
</dl>
{% if status.release_error %}
<p class="status failed">{{ status.release_error }}</p>
{% endif %}
<form method="post" class="actions inline">
{% csrf_token %}
<button type="submit" name="action" value="check_release">Check releases</button>
</form>
</section>
</section>
<section class="panel">
<h2>Update Actions</h2>
<p class="muted">Run these from the installed checkout. The native updater may require a sudoers rule for the pobsync service user.</p>
<dl class="detail-list">
<dt>Git remote</dt>
<dd>{{ status.git_remote }}</dd>
<dt>Update command</dt>
<dd><code>{{ status.update_command }}</code></dd>
</dl>
<div class="actions">
<form method="post" class="inline-form">
{% csrf_token %}
<button class="secondary" type="submit" name="action" value="git_fetch">Fetch releases</button>
</form>
<form method="post" class="inline-form">
{% csrf_token %}
<button class="secondary" type="submit" name="action" value="git_pull">Pull current branch</button>
</form>
<form method="post" class="inline-form">
{% csrf_token %}
<button type="submit" name="action" value="run_update">Run native updater</button>
</form>
</div>
</section>
{% if action_result %}
<section class="panel">
<h2>Last Action Result</h2>
<dl class="detail-list">
<dt>Status</dt>
<dd><span class="status {% if action_result.ok %}ok{% else %}failed{% endif %}">{% if action_result.ok %}ok{% else %}failed{% endif %}</span></dd>
<dt>Exit code</dt>
<dd>{{ action_result.exit_code }}</dd>
<dt>Command</dt>
<dd><code>{{ action_result.command|join:" " }}</code></dd>
</dl>
{% if action_result.stdout %}
<h3>Stdout</h3>
<pre>{{ action_result.stdout }}</pre>
{% endif %}
{% if action_result.stderr %}
<h3>Stderr</h3>
<pre>{{ action_result.stderr }}</pre>
{% endif %}
</section>
{% endif %}
{% endblock %}

View File

@@ -18,6 +18,12 @@ class ApiTests(TestCase):
is_staff=True, is_staff=True,
is_superuser=True, is_superuser=True,
) )
self.readonly_user = user_model.objects.create_user(
username="viewer",
password="secret",
is_staff=False,
is_superuser=False,
)
def test_api_requires_staff_login(self) -> None: def test_api_requires_staff_login(self) -> None:
response = self.client.get("/api/hosts/") response = self.client.get("/api/hosts/")
@@ -25,6 +31,15 @@ class ApiTests(TestCase):
self.assertEqual(response.status_code, 302) self.assertEqual(response.status_code, 302)
self.assertIn("/admin/login/", response["Location"]) self.assertIn("/admin/login/", response["Location"])
def test_readonly_user_can_access_status_endpoint_only(self) -> None:
self.client.force_login(self.readonly_user)
status_response = self.client.get("/api/status/")
hosts_response = self.client.get("/api/hosts/")
self.assertEqual(status_response.status_code, 200)
self.assertEqual(hosts_response.status_code, 403)
def test_hosts_endpoint_returns_counts_and_schedule(self) -> None: def test_hosts_endpoint_returns_counts_and_schedule(self) -> None:
self.client.force_login(self.staff_user) self.client.force_login(self.staff_user)
host = HostConfig.objects.create(host="web-01", address="web-01.example.test") host = HostConfig.objects.create(host="web-01", address="web-01.example.test")

View File

@@ -11,7 +11,7 @@ from django.utils import timezone
from pobsync.util import write_yaml_atomic from pobsync.util import write_yaml_atomic
from pobsync_backend.backup_runner import queue_backup_run, reconcile_running_runs from pobsync_backend.backup_runner import queue_backup_run, reconcile_running_runs
from pobsync_backend.management.commands.run_pobsync_worker import Command from pobsync_backend.management.commands.run_pobsync_worker import Command
from pobsync_backend.models import BackupRun, GlobalConfig, HostConfig, SnapshotRecord from pobsync_backend.models import BackupRun, GlobalConfig, HostConfig, NotificationDelivery, NotificationTarget, SnapshotRecord
class BackupWorkerTests(TestCase): class BackupWorkerTests(TestCase):
@@ -39,13 +39,20 @@ class BackupWorkerTests(TestCase):
}, },
) )
def test_queue_backup_run_can_request_verbose_output(self) -> None: def test_queue_backup_run_enables_verbose_output_by_default(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test") host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
run = queue_backup_run(host=host, verbose_output=True) run = queue_backup_run(host=host)
self.assertTrue(run.result["requested"]["verbose_output"]) self.assertTrue(run.result["requested"]["verbose_output"])
def test_queue_backup_run_can_disable_verbose_output(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
run = queue_backup_run(host=host, verbose_output=False)
self.assertFalse(run.result["requested"]["verbose_output"])
def test_worker_executes_next_queued_run(self) -> None: def test_worker_executes_next_queued_run(self) -> None:
with TemporaryDirectory() as tmp: with TemporaryDirectory() as tmp:
backup_root = Path(tmp) / "backups" backup_root = Path(tmp) / "backups"
@@ -85,6 +92,73 @@ class BackupWorkerTests(TestCase):
self.assertEqual(SnapshotRecord.objects.count(), 1) self.assertEqual(SnapshotRecord.objects.count(), 1)
self.assertEqual(run.snapshot, SnapshotRecord.objects.get()) self.assertEqual(run.snapshot, SnapshotRecord.objects.get())
def test_worker_records_warning_status_from_completed_run(self) -> None:
with TemporaryDirectory() as tmp:
backup_root = Path(tmp) / "backups"
GlobalConfig.objects.create(name="default", backup_root=str(backup_root))
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
snapshot_dir = backup_root / host.host / "scheduled" / "20260519-021500Z__ABCDEFGH"
meta_dir = snapshot_dir / "meta"
meta_dir.mkdir(parents=True)
write_yaml_atomic(meta_dir / "meta.yaml", {"status": "warning", "started_at": "2026-05-19T02:15:00Z"})
run = queue_backup_run(host=host)
with patch("pobsync_backend.backup_runner.run_scheduled") as run_scheduled:
run_scheduled.return_value = {
"ok": True,
"status": "warning",
"dry_run": False,
"host": host.host,
"snapshot": str(snapshot_dir),
"base": None,
"warning": {"category": "vanished"},
"rsync": {"exit_code": 24},
}
count = Command()._run_once(prefix=Path(tmp) / "home")
self.assertEqual(count, 1)
run.refresh_from_db()
self.assertEqual(run.status, BackupRun.Status.WARNING)
self.assertEqual(run.rsync_exit_code, 24)
self.assertEqual(run.result["warning"]["category"], "vanished")
def test_worker_sends_notification_after_completed_run(self) -> None:
with TemporaryDirectory() as tmp:
backup_root = Path(tmp) / "backups"
GlobalConfig.objects.create(name="default", backup_root=str(backup_root))
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
NotificationTarget.objects.create(
name="ops",
channel=NotificationTarget.Channel.EMAIL,
email_to="ops@example.test",
)
snapshot_dir = backup_root / host.host / "scheduled" / "20260519-021500Z__ABCDEFGH"
meta_dir = snapshot_dir / "meta"
meta_dir.mkdir(parents=True)
write_yaml_atomic(meta_dir / "meta.yaml", {"status": "success", "started_at": "2026-05-19T02:15:00Z"})
run = queue_backup_run(host=host)
with (
patch("pobsync_backend.backup_runner.run_scheduled") as run_scheduled,
patch("pobsync_backend.notifications.send_mail", return_value=1) as send_mail,
):
run_scheduled.return_value = {
"ok": True,
"dry_run": False,
"host": host.host,
"snapshot": str(snapshot_dir),
"base": None,
"rsync": {"exit_code": 0},
}
Command()._run_once(prefix=Path(tmp) / "home")
run.refresh_from_db()
self.assertEqual(run.status, BackupRun.Status.SUCCESS)
self.assertEqual(NotificationDelivery.objects.get(run=run).status, NotificationDelivery.Status.SENT)
send_mail.assert_called_once()
def test_worker_refreshes_heartbeat_while_run_is_active(self) -> None: def test_worker_refreshes_heartbeat_while_run_is_active(self) -> None:
with TemporaryDirectory() as tmp: with TemporaryDirectory() as tmp:
GlobalConfig.objects.create(name="default", backup_root=str(Path(tmp) / "backups")) GlobalConfig.objects.create(name="default", backup_root=str(Path(tmp) / "backups"))
@@ -116,6 +190,44 @@ class BackupWorkerTests(TestCase):
run_scheduled.side_effect = fake_run_scheduled run_scheduled.side_effect = fake_run_scheduled
Command()._run_once(prefix=Path(tmp) / "home") Command()._run_once(prefix=Path(tmp) / "home")
def test_worker_records_real_run_log_path_while_running(self) -> None:
with TemporaryDirectory() as tmp:
GlobalConfig.objects.create(name="default", backup_root=str(Path(tmp) / "backups"))
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
run = queue_backup_run(host=host)
snapshot_dir = Path(tmp) / "backups" / host.host / ".incomplete" / "20260519-021500Z__ABCDEFGH"
log_path = snapshot_dir / "meta" / "rsync.log"
with patch("pobsync_backend.backup_runner.run_scheduled") as run_scheduled:
def fake_run_scheduled(**kwargs):
kwargs["state_callback"](
{
"status": "running",
"phase": "rsync",
"snapshot": str(snapshot_dir),
"log": str(log_path),
"rsync": {"command": ["rsync"], "exit_code": None, "pid": 1234, "pgid": 1234},
}
)
run.refresh_from_db()
self.assertEqual(run.snapshot_path, str(snapshot_dir))
self.assertEqual(run.result["execution"]["phase"], "rsync")
self.assertEqual(run.result["execution"]["log"], str(log_path))
self.assertEqual(run.result["execution"]["snapshot"], str(snapshot_dir))
self.assertEqual(run.result["rsync"]["command"], ["rsync"])
self.assertEqual(run.result["rsync"]["pid"], 1234)
return {
"ok": True,
"dry_run": False,
"host": host.host,
"snapshot": "",
"base": None,
"rsync": {"exit_code": 0},
}
run_scheduled.side_effect = fake_run_scheduled
Command()._run_once(prefix=Path(tmp) / "home")
def test_worker_reconciles_stale_real_run_after_heartbeat_timeout(self) -> None: def test_worker_reconciles_stale_real_run_after_heartbeat_timeout(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test") host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
run = queue_backup_run(host=host) run = queue_backup_run(host=host)
@@ -136,6 +248,97 @@ class BackupWorkerTests(TestCase):
self.assertEqual(run.result["failure"]["category"], "worker") self.assertEqual(run.result["failure"]["category"], "worker")
self.assertIn("heartbeat stopped", run.result["failure"]["message"]) self.assertIn("heartbeat stopped", run.result["failure"]["message"])
def test_worker_reconciles_real_run_with_terminal_broken_pipe_log(self) -> None:
with TemporaryDirectory() as tmp:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
run = queue_backup_run(host=host)
log_path = Path(tmp) / "backups" / host.host / ".incomplete" / "20260519-021500Z__ABCDEFGH" / "meta" / "rsync.log"
log_path.parent.mkdir(parents=True, exist_ok=True)
log_path.write_text(
"rsync error: unexplained error (code 255) at rsync.c(716) [generator=3.4.1]\n"
"rsync error: received SIGUSR1 (code 19) at main.c(1600) [receiver=3.4.1]\n"
"rsync: [generator] write error: Broken pipe (32)\n",
encoding="utf-8",
)
run.status = BackupRun.Status.RUNNING
run.started_at = timezone.now()
run.result["execution"] = {"log": str(log_path)}
run.save(update_fields=["status", "started_at", "result"])
reconciled = reconcile_running_runs()
self.assertEqual(reconciled, 1)
run.refresh_from_db()
self.assertEqual(run.status, BackupRun.Status.FAILED)
self.assertEqual(run.rsync_exit_code, 255)
self.assertEqual(run.result["failure"]["category"], "transport")
self.assertIn("Broken pipe", "\n".join(run.result["rsync"]["log_tail"]))
def test_worker_reconciles_real_run_when_rsync_process_disappears(self) -> None:
with TemporaryDirectory() as tmp:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
run = queue_backup_run(host=host)
log_path = Path(tmp) / "backups" / host.host / ".incomplete" / "20260519-021500Z__ABCDEFGH" / "meta" / "rsync.log"
log_path.parent.mkdir(parents=True, exist_ok=True)
log_path.write_text("sending incremental file list\n", encoding="utf-8")
run.status = BackupRun.Status.RUNNING
run.started_at = timezone.now() - timedelta(minutes=10)
run.result["execution"] = {
"phase": "rsync",
"log": str(log_path),
"heartbeat_at": (timezone.now() - timedelta(minutes=10)).isoformat(),
}
run.result["rsync"] = {"pid": 999999, "pgid": 999999, "command": ["rsync"]}
run.save(update_fields=["status", "started_at", "result"])
reconciled = reconcile_running_runs(grace_seconds=300, stale_worker_seconds=24 * 60 * 60)
self.assertEqual(reconciled, 1)
run.refresh_from_db()
self.assertEqual(run.status, BackupRun.Status.FAILED)
self.assertEqual(run.result["failure"]["category"], "rsync_process")
self.assertEqual(run.rsync_exit_code, 255)
def test_worker_does_not_reconcile_missing_rsync_process_during_finalizing_phase(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
run = queue_backup_run(host=host)
run.status = BackupRun.Status.RUNNING
run.started_at = timezone.now() - timedelta(minutes=10)
run.result["execution"] = {
"phase": "finalizing",
"heartbeat_at": (timezone.now() - timedelta(minutes=10)).isoformat(),
}
run.result["rsync"] = {"pid": 999999, "pgid": 999999, "command": ["rsync"], "exit_code": 0}
run.save(update_fields=["status", "started_at", "result"])
reconciled = reconcile_running_runs(grace_seconds=300, stale_worker_seconds=24 * 60 * 60)
self.assertEqual(reconciled, 0)
run.refresh_from_db()
self.assertEqual(run.status, BackupRun.Status.RUNNING)
def test_worker_does_not_fail_real_run_for_vanished_file_warning_log(self) -> None:
with TemporaryDirectory() as tmp:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
run = queue_backup_run(host=host)
log_path = Path(tmp) / "backups" / host.host / ".incomplete" / "20260519-021500Z__ABCDEFGH" / "meta" / "rsync.log"
log_path.parent.mkdir(parents=True, exist_ok=True)
log_path.write_text(
"file has vanished: \"/var/lib/app/session\"\n"
"rsync warning: some files vanished before they could be transferred (code 24) at main.c(1338) [sender=3.4.1]\n",
encoding="utf-8",
)
run.status = BackupRun.Status.RUNNING
run.started_at = timezone.now()
run.result["execution"] = {"log": str(log_path)}
run.save(update_fields=["status", "started_at", "result"])
reconciled = reconcile_running_runs()
self.assertEqual(reconciled, 0)
run.refresh_from_db()
self.assertEqual(run.status, BackupRun.Status.RUNNING)
def test_worker_records_dry_run_log_path_while_running(self) -> None: def test_worker_records_dry_run_log_path_while_running(self) -> None:
with TemporaryDirectory() as tmp: with TemporaryDirectory() as tmp:
GlobalConfig.objects.create(name="default", backup_root=str(Path(tmp) / "backups")) GlobalConfig.objects.create(name="default", backup_root=str(Path(tmp) / "backups"))

View File

@@ -42,6 +42,7 @@ class ConfigureCommandsTests(TestCase):
address="web-01.example.test", address="web-01.example.test",
exclude_add=["/tmp/***"], exclude_add=["/tmp/***"],
rsync_extra_arg=["--delete"], rsync_extra_arg=["--delete"],
rsync_bwlimit_kbps=4096,
stdout=out, stdout=out,
) )
@@ -49,10 +50,12 @@ class ConfigureCommandsTests(TestCase):
self.assertEqual(host.retention_daily, 5) self.assertEqual(host.retention_daily, 5)
self.assertEqual(host.excludes_add, ["/tmp/***"]) self.assertEqual(host.excludes_add, ["/tmp/***"])
self.assertEqual(host.rsync_extra_args, ["--delete"]) self.assertEqual(host.rsync_extra_args, ["--delete"])
self.assertEqual(host.rsync_bwlimit_kbps, 4096)
effective = DjangoConfigSource().effective_config_for_host("web-01") effective = DjangoConfigSource().effective_config_for_host("web-01")
self.assertEqual(effective["retention"]["yearly"], 2) self.assertEqual(effective["retention"]["yearly"], 2)
self.assertEqual(effective["excludes_effective"], ["/tmp/***"]) self.assertEqual(effective["excludes_effective"], ["/tmp/***"])
self.assertEqual(effective["rsync"]["bwlimit_kbps"], 4096)
def test_configure_schedule_creates_sql_schedule(self) -> None: def test_configure_schedule_creates_sql_schedule(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test") host = HostConfig.objects.create(host="web-01", address="web-01.example.test")

View File

@@ -5,6 +5,7 @@ from unittest.mock import patch
from django.test import SimpleTestCase from django.test import SimpleTestCase
from pobsync import __version__
from pobsync.cli import main from pobsync.cli import main
@@ -15,7 +16,7 @@ class ConsoleEntrypointTests(SimpleTestCase):
exit_code = main(["--version"]) exit_code = main(["--version"])
self.assertEqual(exit_code, 0) self.assertEqual(exit_code, 0)
self.assertEqual(stdout.getvalue().strip(), "pobsync 1.0.0") self.assertEqual(stdout.getvalue().strip(), f"pobsync {__version__}")
def test_maps_backup_alias_to_django_command(self) -> None: def test_maps_backup_alias_to_django_command(self) -> None:
with patch("pobsync.cli.execute_from_command_line") as execute: with patch("pobsync.cli.execute_from_command_line") as execute:

View File

@@ -17,6 +17,7 @@ class DjangoConfigSourceTests(TestCase):
backup_root="/backups", backup_root="/backups",
rsync_args=["--archive"], rsync_args=["--archive"],
rsync_extra_args=["--numeric-ids"], rsync_extra_args=["--numeric-ids"],
rsync_bwlimit_kbps=10000,
excludes_default=["/proc/***"], excludes_default=["/proc/***"],
retention_daily=7, retention_daily=7,
retention_weekly=4, retention_weekly=4,
@@ -28,6 +29,7 @@ class DjangoConfigSourceTests(TestCase):
address="web-01.example.test", address="web-01.example.test",
excludes_add=["/tmp/***"], excludes_add=["/tmp/***"],
rsync_extra_args=["--delete"], rsync_extra_args=["--delete"],
rsync_bwlimit_kbps=2500,
retention_daily=7, retention_daily=7,
retention_weekly=4, retention_weekly=4,
retention_monthly=3, retention_monthly=3,
@@ -46,6 +48,24 @@ class DjangoConfigSourceTests(TestCase):
self.assertEqual(cfg["address"], "web-01.example.test") self.assertEqual(cfg["address"], "web-01.example.test")
self.assertEqual(cfg["excludes_effective"], ["/proc/***", "/tmp/***"]) self.assertEqual(cfg["excludes_effective"], ["/proc/***", "/tmp/***"])
self.assertEqual(cfg["rsync"]["args_effective"], ["--archive", "--numeric-ids", "--delete"]) self.assertEqual(cfg["rsync"]["args_effective"], ["--archive", "--numeric-ids", "--delete"])
self.assertEqual(cfg["rsync"]["bwlimit_kbps"], 2500)
def test_host_can_disable_global_rsync_bandwidth_limit(self) -> None:
GlobalConfig.objects.create(
name="default",
backup_root="/backups",
rsync_args=["--archive"],
rsync_bwlimit_kbps=5000,
)
HostConfig.objects.create(
host="web-01",
address="web-01.example.test",
rsync_bwlimit_kbps=0,
)
cfg = DjangoConfigSource().effective_config_for_host("web-01")
self.assertEqual(cfg["rsync"]["bwlimit_kbps"], 0)
def test_materializes_global_ssh_credential_for_runtime_config(self) -> None: def test_materializes_global_ssh_credential_for_runtime_config(self) -> None:
credential = SshCredential.objects.create( credential = SshCredential.objects.create(
@@ -113,6 +133,7 @@ class DjangoConfigSourceTests(TestCase):
) )
HostConfig.objects.create(host="web-01", address="web-01.example.test") HostConfig.objects.create(host="web-01", address="web-01.example.test")
with override_settings(POBSYNC_HOME=str(Path(tmp) / "home")):
cfg = DjangoConfigSource().effective_config_for_host("web-01") cfg = DjangoConfigSource().effective_config_for_host("web-01")
self.assertIn(f"-oIdentityFile={identity_file}", cfg["ssh"]["options"]) self.assertIn(f"-oIdentityFile={identity_file}", cfg["ssh"]["options"])

View File

@@ -0,0 +1,125 @@
from __future__ import annotations
from datetime import timedelta
from unittest.mock import Mock, patch
from django.test import TestCase
from django.utils import timezone
from pobsync_backend.models import BackupRun, HostConfig, NotificationDelivery, NotificationTarget
from pobsync_backend.notifications import notify_backup_run_completed
class NotificationTests(TestCase):
def test_email_notification_is_sent_for_matching_status(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
target = NotificationTarget.objects.create(
name="ops",
channel=NotificationTarget.Channel.EMAIL,
statuses=[BackupRun.Status.FAILED],
email_to="ops@example.test",
)
run = BackupRun.objects.create(
host=host,
status=BackupRun.Status.FAILED,
run_type=BackupRun.RunType.MANUAL,
started_at=timezone.now() - timedelta(minutes=5),
ended_at=timezone.now(),
rsync_exit_code=12,
result={"failure": {"message": "rsync failed"}},
)
with patch("pobsync_backend.notifications.send_mail", return_value=1) as send_mail:
results = notify_backup_run_completed(run)
self.assertEqual(len(results), 1)
self.assertTrue(results[0].sent)
send_mail.assert_called_once()
subject, message, _from_email, recipients = send_mail.call_args.args
self.assertEqual(subject, f"pobsync failed: web-01 run {run.id}")
self.assertIn("Failure: rsync failed", message)
self.assertEqual(recipients, ["ops@example.test"])
delivery = NotificationDelivery.objects.get(target=target, run=run)
self.assertEqual(delivery.status, NotificationDelivery.Status.SENT)
target.refresh_from_db()
self.assertEqual(target.last_status, NotificationDelivery.Status.SENT)
self.assertEqual(target.last_error, "")
self.assertIsNotNone(target.last_sent_at)
def test_webhook_notification_posts_payload(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
target = NotificationTarget.objects.create(
name="discord",
channel=NotificationTarget.Channel.WEBHOOK,
webhook_url="https://hooks.example.test/pobsync",
webhook_headers={"X-Token": "secret"},
)
run = BackupRun.objects.create(host=host, status=BackupRun.Status.SUCCESS, run_type=BackupRun.RunType.SCHEDULED)
response = Mock()
response.status = 204
response.__enter__ = Mock(return_value=response)
response.__exit__ = Mock(return_value=False)
with patch("pobsync_backend.notifications.urllib.request.urlopen", return_value=response) as urlopen:
notify_backup_run_completed(run)
request = urlopen.call_args.args[0]
self.assertEqual(request.full_url, "https://hooks.example.test/pobsync")
self.assertEqual(request.get_method(), "POST")
self.assertEqual(request.headers["X-token"], "secret")
self.assertIn(f'"id": {run.id}', request.data.decode("utf-8"))
self.assertEqual(NotificationDelivery.objects.get(target=target, run=run).status, NotificationDelivery.Status.SENT)
def test_notification_filters_statuses(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
NotificationTarget.objects.create(
name="failures-only",
channel=NotificationTarget.Channel.EMAIL,
statuses=[BackupRun.Status.FAILED],
email_to="ops@example.test",
)
run = BackupRun.objects.create(host=host, status=BackupRun.Status.SUCCESS)
with patch("pobsync_backend.notifications.send_mail") as send_mail:
results = notify_backup_run_completed(run)
self.assertEqual(results, [])
send_mail.assert_not_called()
self.assertEqual(NotificationDelivery.objects.count(), 0)
def test_notification_delivery_is_idempotent_per_run_and_target(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
target = NotificationTarget.objects.create(
name="ops",
channel=NotificationTarget.Channel.EMAIL,
email_to="ops@example.test",
)
run = BackupRun.objects.create(host=host, status=BackupRun.Status.WARNING)
with patch("pobsync_backend.notifications.send_mail", return_value=1) as send_mail:
notify_backup_run_completed(run)
notify_backup_run_completed(run)
self.assertEqual(NotificationDelivery.objects.filter(target=target, run=run).count(), 1)
send_mail.assert_called_once()
def test_failed_delivery_is_recorded_without_raising(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
target = NotificationTarget.objects.create(
name="broken",
channel=NotificationTarget.Channel.EMAIL,
email_to="ops@example.test",
)
run = BackupRun.objects.create(host=host, status=BackupRun.Status.FAILED)
with patch("pobsync_backend.notifications.send_mail", side_effect=RuntimeError("smtp down")):
results = notify_backup_run_completed(run)
self.assertEqual(len(results), 1)
self.assertFalse(results[0].sent)
delivery = NotificationDelivery.objects.get(target=target, run=run)
self.assertEqual(delivery.status, NotificationDelivery.Status.FAILED)
self.assertEqual(delivery.error, "smtp down")
target.refresh_from_db()
self.assertEqual(target.last_status, NotificationDelivery.Status.FAILED)
self.assertEqual(target.last_error, "smtp down")

View File

@@ -39,12 +39,16 @@ class RunBackupRecordsSnapshotTests(TestCase):
"host": host.host, "host": host.host,
"snapshot": str(snapshot_dir), "snapshot": str(snapshot_dir),
"base": None, "base": None,
"verbose_output": True,
"rsync": {"exit_code": 0}, "rsync": {"exit_code": 0},
} }
call_command("run_pobsync_backup", host.host, prefix=str(Path(tmp) / "home"), stdout=StringIO()) call_command("run_pobsync_backup", host.host, prefix=str(Path(tmp) / "home"), stdout=StringIO())
run_scheduled.assert_called_once()
self.assertTrue(run_scheduled.call_args.kwargs["verbose_output"])
self.assertEqual(BackupRun.objects.count(), 1) self.assertEqual(BackupRun.objects.count(), 1)
run = BackupRun.objects.get() run = BackupRun.objects.get()
self.assertTrue(run.result["verbose_output"])
self.assertEqual(SnapshotRecord.objects.count(), 1) self.assertEqual(SnapshotRecord.objects.count(), 1)
record = SnapshotRecord.objects.get() record = SnapshotRecord.objects.get()
self.assertEqual(run.snapshot, record) self.assertEqual(run.snapshot, record)
@@ -52,6 +56,45 @@ class RunBackupRecordsSnapshotTests(TestCase):
self.assertEqual(record.kind, "scheduled") self.assertEqual(record.kind, "scheduled")
self.assertEqual(record.status, "success") self.assertEqual(record.status, "success")
def test_backup_command_can_skip_default_verbose_rsync_output(self) -> None:
with TemporaryDirectory() as tmp:
backup_root = Path(tmp) / "backups"
GlobalConfig.objects.create(name="default", backup_root=str(backup_root))
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
snapshot_dir = backup_root / host.host / "scheduled" / "20260519-021500Z__ABCDEFGH"
meta_dir = snapshot_dir / "meta"
meta_dir.mkdir(parents=True)
write_yaml_atomic(
meta_dir / "meta.yaml",
{
"status": "success",
"started_at": "2026-05-19T02:15:00Z",
"ended_at": "2026-05-19T02:16:00Z",
},
)
with patch("pobsync_backend.backup_runner.run_scheduled") as run_scheduled:
run_scheduled.return_value = {
"ok": True,
"dry_run": False,
"host": host.host,
"snapshot": str(snapshot_dir),
"base": None,
"verbose_output": False,
"rsync": {"exit_code": 0},
}
call_command(
"run_pobsync_backup",
host.host,
prefix=str(Path(tmp) / "home"),
quiet_rsync=True,
stdout=StringIO(),
)
run_scheduled.assert_called_once()
self.assertFalse(run_scheduled.call_args.kwargs["verbose_output"])
self.assertFalse(BackupRun.objects.get().result["verbose_output"])
def test_prune_uses_sql_retention_after_snapshot_record_is_created(self) -> None: def test_prune_uses_sql_retention_after_snapshot_record_is_created(self) -> None:
with TemporaryDirectory() as tmp: with TemporaryDirectory() as tmp:
backup_root = Path(tmp) / "backups" backup_root = Path(tmp) / "backups"

View File

@@ -12,8 +12,9 @@ from pobsync.rsync import RsyncResult
class FakeConfigSource: class FakeConfigSource:
def __init__(self, backup_root: str = "/tmp/pobsync-test-backups") -> None: def __init__(self, backup_root: str = "/tmp/pobsync-test-backups", bwlimit_kbps: int = 0) -> None:
self.backup_root = backup_root self.backup_root = backup_root
self.bwlimit_kbps = bwlimit_kbps
def effective_config_for_host(self, host: str) -> dict: def effective_config_for_host(self, host: str) -> dict:
return { return {
@@ -25,7 +26,7 @@ class FakeConfigSource:
"binary": "rsync", "binary": "rsync",
"args_effective": ["--archive"], "args_effective": ["--archive"],
"timeout_seconds": 0, "timeout_seconds": 0,
"bwlimit_kbps": 0, "bwlimit_kbps": self.bwlimit_kbps,
}, },
"source_root": "/", "source_root": "/",
"includes": [], "includes": [],
@@ -54,6 +55,21 @@ class RunScheduledConfigSourceTests(SimpleTestCase):
self.assertEqual(result["host"], "web-01") self.assertEqual(result["host"], "web-01")
run_rsync.assert_called_once() run_rsync.assert_called_once()
def test_dry_run_applies_configured_bandwidth_limit(self) -> None:
with patch("pobsync.commands.run_scheduled.run_rsync") as run_rsync:
run_rsync.return_value = RsyncResult(exit_code=0, command=["rsync", "--bwlimit=4096"])
result = run_scheduled(
prefix=Path("/missing-prefix"),
host="web-01",
dry_run=True,
config_source=FakeConfigSource(bwlimit_kbps=4096),
)
command = run_rsync.call_args.args[0]
self.assertIn("--bwlimit=4096", command)
self.assertEqual(result["rsync"]["bwlimit_kbps"], 4096)
def test_failed_dry_run_includes_log_tail(self) -> None: def test_failed_dry_run_includes_log_tail(self) -> None:
def fake_run_rsync(command, log_path, timeout_seconds, cancel_check=None): def fake_run_rsync(command, log_path, timeout_seconds, cancel_check=None):
log_path.parent.mkdir(parents=True, exist_ok=True) log_path.parent.mkdir(parents=True, exist_ok=True)
@@ -186,11 +202,13 @@ class RunScheduledConfigSourceTests(SimpleTestCase):
host="web-01", host="web-01",
dry_run=False, dry_run=False,
verbose_output=True, verbose_output=True,
config_source=FakeConfigSource(backup_root=str(Path(tmp) / "backups")), config_source=FakeConfigSource(backup_root=str(Path(tmp) / "backups"), bwlimit_kbps=2048),
) )
command = run_rsync.call_args.args[0] command = run_rsync.call_args.args[0]
self.assertTrue(result["ok"]) self.assertTrue(result["ok"])
self.assertIn("--bwlimit=2048", command)
self.assertEqual(result["rsync"]["bwlimit_kbps"], 2048)
self.assertIn("--stats", command) self.assertIn("--stats", command)
self.assertIn("--itemize-changes", command) self.assertIn("--itemize-changes", command)
self.assertIn("--info=flist2,progress2,stats2", command) self.assertIn("--info=flist2,progress2,stats2", command)
@@ -256,6 +274,71 @@ class RunScheduledConfigSourceTests(SimpleTestCase):
self.assertIn("stats:", meta_text) self.assertIn("stats:", meta_text)
self.assertIn("files_total: 10", meta_text) self.assertIn("files_total: 10", meta_text)
def test_real_run_reports_running_state_callback_before_rsync_returns(self) -> None:
states = []
def fake_run_rsync(command, log_path, timeout_seconds, cancel_check=None, process_started=None):
self.assertEqual(len(states), 1)
self.assertEqual(states[0]["status"], "running")
self.assertEqual(states[0]["phase"], "preparing")
self.assertEqual(states[0]["log"], str(log_path))
self.assertEqual(states[0]["rsync"]["command"], command)
self.assertIsNotNone(process_started)
process_started(1234, 1234)
self.assertEqual(len(states), 2)
self.assertEqual(states[1]["phase"], "rsync")
self.assertEqual(states[1]["rsync"]["pid"], 1234)
self.assertEqual(states[1]["rsync"]["pgid"], 1234)
log_path.write_text("Number of files: 1\n", encoding="utf-8")
return RsyncResult(exit_code=0, command=command)
with TemporaryDirectory() as tmp:
with patch("pobsync.commands.run_scheduled.run_rsync", side_effect=fake_run_rsync):
run_scheduled(
prefix=Path(tmp) / "home",
host="web-01",
dry_run=False,
config_source=FakeConfigSource(backup_root=str(Path(tmp) / "backups")),
state_callback=states.append,
)
self.assertEqual(len(states), 3)
self.assertIn("/.incomplete/", states[0]["snapshot"])
self.assertEqual(states[2]["phase"], "finalizing")
self.assertEqual(states[2]["rsync"]["exit_code"], 0)
def test_real_run_keeps_snapshot_with_warning_for_vanished_files(self) -> None:
def fake_run_rsync(command, log_path, timeout_seconds, cancel_check=None):
log_path.write_text(
"file has vanished: \"/var/lib/app/session\"\n"
"rsync warning: some files vanished before they could be transferred (code 24) at main.c(1338) [sender=3.4.1]\n",
encoding="utf-8",
)
data_dir = log_path.parent.parent / "data"
data_dir.mkdir(parents=True, exist_ok=True)
(data_dir / "payload.txt").write_text("payload", encoding="utf-8")
return RsyncResult(exit_code=24, command=command)
with TemporaryDirectory() as tmp:
backup_root = Path(tmp) / "backups"
with patch("pobsync.commands.run_scheduled.run_rsync", side_effect=fake_run_rsync):
result = run_scheduled(
prefix=Path(tmp) / "home",
host="web-01",
dry_run=False,
config_source=FakeConfigSource(backup_root=str(backup_root)),
)
snapshot = Path(result["snapshot"])
self.assertTrue((snapshot / "data" / "payload.txt").exists())
self.assertTrue(result["ok"])
self.assertEqual(result["status"], "warning")
self.assertEqual(result["rsync"]["exit_code"], 24)
self.assertEqual(result["warning"]["category"], "vanished")
self.assertEqual(snapshot.parent.name, "scheduled")
self.assertIn("file has vanished", "\n".join(result["rsync"]["log_tail"]))
def test_dry_run_reports_cancelled_rsync(self) -> None: def test_dry_run_reports_cancelled_rsync(self) -> None:
def fake_run_rsync(command, log_path, timeout_seconds, cancel_check=None): def fake_run_rsync(command, log_path, timeout_seconds, cancel_check=None):
self.assertTrue(cancel_check()) self.assertTrue(cancel_check())

View File

@@ -154,6 +154,68 @@ class SqlRetentionTests(TestCase):
], ],
) )
def test_apply_deletes_snapshot_with_non_traversable_nested_directory(self) -> None:
with TemporaryDirectory() as tmp:
prefix = Path(tmp) / "home"
host = HostConfig.objects.create(
host="web-01",
address="web-01.example.test",
retention_daily=0,
retention_weekly=0,
retention_monthly=0,
retention_yearly=0,
)
old_dir = Path(tmp) / "backups" / host.host / "scheduled" / "20260518-021500Z__OLD"
restricted_dir = old_dir / "data" / "var" / "lib" / "snapd" / "void"
restricted_dir.mkdir(parents=True)
restricted_dir.joinpath("state").write_text("preserved permissions\n")
restricted_dir.chmod(0)
new_dir = Path(tmp) / "backups" / host.host / "scheduled" / "20260519-021500Z__NEW"
new_dir.mkdir(parents=True)
old = self._snapshot(host, old_dir.name, path=str(old_dir))
self._snapshot(host, new_dir.name, path=str(new_dir))
result = run_sql_retention_apply(
prefix=prefix,
host=host.host,
kind="scheduled",
protect_bases=False,
yes=True,
max_delete=1,
acquire_lock=False,
)
self.assertFalse(old_dir.exists())
self.assertFalse(SnapshotRecord.objects.filter(pk=old.pk).exists())
self.assertEqual(result["deleted"][0]["dirname"], old.dirname)
def test_apply_rejects_scheduled_snapshot_path_outside_host_kind_directory(self) -> None:
host = HostConfig.objects.create(
host="web-01",
address="web-01.example.test",
retention_daily=0,
retention_weekly=0,
retention_monthly=0,
retention_yearly=0,
)
self._snapshot(
host,
"20260518-021500Z__OLD",
path="/backups/web-01/manual/20260518-021500Z__OLD",
)
self._snapshot(host, "20260519-021500Z__NEW")
with self.assertRaisesRegex(ConfigError, "unexpected snapshot path"):
run_sql_retention_apply(
prefix=Path("/tmp/pobsync-test"),
host=host.host,
kind="scheduled",
protect_bases=False,
yes=True,
max_delete=1,
acquire_lock=False,
)
def test_apply_respects_max_delete(self) -> None: def test_apply_respects_max_delete(self) -> None:
host = HostConfig.objects.create( host = HostConfig.objects.create(
host="web-01", host="web-01",
@@ -192,6 +254,8 @@ class SqlRetentionTests(TestCase):
path=str(incomplete_dir), path=str(incomplete_dir),
status="failed", status="failed",
started_at=datetime(2026, 5, 19, 3, 15, tzinfo=timezone.utc), started_at=datetime(2026, 5, 19, 3, 15, tzinfo=timezone.utc),
reviewed_at=datetime(2026, 5, 19, 4, 15, tzinfo=timezone.utc),
reviewed_by="admin",
) )
result = run_incomplete_cleanup( result = run_incomplete_cleanup(
@@ -213,6 +277,58 @@ class SqlRetentionTests(TestCase):
self.assertEqual(purged.action, PurgedSnapshot.Action.INCOMPLETE_CLEANUP) self.assertEqual(purged.action, PurgedSnapshot.Action.INCOMPLETE_CLEANUP)
self.assertEqual(purged.reason, "manual incomplete cleanup") self.assertEqual(purged.reason, "manual incomplete cleanup")
def test_incomplete_cleanup_deletes_non_traversable_nested_directory(self) -> None:
with TemporaryDirectory() as tmp:
prefix = Path(tmp) / "home"
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
incomplete_dir = Path(tmp) / "backups" / host.host / ".incomplete" / "20260519-031500Z__BROKEN01"
restricted_dir = incomplete_dir / "data" / "var" / "lib" / "snapd" / "void"
restricted_dir.mkdir(parents=True)
restricted_dir.joinpath("state").write_text("interrupted\n")
restricted_dir.chmod(0)
record = SnapshotRecord.objects.create(
host=host,
kind=SnapshotRecord.Kind.INCOMPLETE,
dirname=incomplete_dir.name,
path=str(incomplete_dir),
status="failed",
started_at=datetime(2026, 5, 19, 3, 15, tzinfo=timezone.utc),
reviewed_at=datetime(2026, 5, 19, 4, 15, tzinfo=timezone.utc),
reviewed_by="admin",
)
result = run_incomplete_cleanup(
prefix=prefix,
host=host.host,
yes=True,
max_delete=1,
acquire_lock=False,
)
self.assertFalse(incomplete_dir.exists())
self.assertFalse(SnapshotRecord.objects.filter(pk=record.pk).exists())
self.assertEqual(result["deleted"][0]["dirname"], incomplete_dir.name)
def test_incomplete_cleanup_requires_reviewed_snapshots(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
SnapshotRecord.objects.create(
host=host,
kind=SnapshotRecord.Kind.INCOMPLETE,
dirname="20260519-031500Z__BROKEN01",
path=f"/backups/{host.host}/.incomplete/20260519-031500Z__BROKEN01",
status="failed",
started_at=datetime(2026, 5, 19, 3, 15, tzinfo=timezone.utc),
)
with self.assertRaisesRegex(ConfigError, "have not been reviewed"):
run_incomplete_cleanup(
prefix=Path("/tmp/pobsync-test"),
host=host.host,
yes=True,
max_delete=1,
acquire_lock=False,
)
def test_incomplete_cleanup_respects_max_delete(self) -> None: def test_incomplete_cleanup_respects_max_delete(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test") host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
SnapshotRecord.objects.create( SnapshotRecord.objects.create(
@@ -222,6 +338,8 @@ class SqlRetentionTests(TestCase):
path=f"/backups/{host.host}/.incomplete/20260519-031500Z__BROKEN01", path=f"/backups/{host.host}/.incomplete/20260519-031500Z__BROKEN01",
status="failed", status="failed",
started_at=datetime(2026, 5, 19, 3, 15, tzinfo=timezone.utc), started_at=datetime(2026, 5, 19, 3, 15, tzinfo=timezone.utc),
reviewed_at=datetime(2026, 5, 19, 4, 15, tzinfo=timezone.utc),
reviewed_by="admin",
) )
with self.assertRaisesRegex(ConfigError, "blocked by --max-delete=0"): with self.assertRaisesRegex(ConfigError, "blocked by --max-delete=0"):
@@ -242,6 +360,8 @@ class SqlRetentionTests(TestCase):
path=f"/backups/{host.host}/scheduled/20260519-031500Z__BROKEN01", path=f"/backups/{host.host}/scheduled/20260519-031500Z__BROKEN01",
status="failed", status="failed",
started_at=datetime(2026, 5, 19, 3, 15, tzinfo=timezone.utc), started_at=datetime(2026, 5, 19, 3, 15, tzinfo=timezone.utc),
reviewed_at=datetime(2026, 5, 19, 4, 15, tzinfo=timezone.utc),
reviewed_by="admin",
) )
with self.assertRaisesRegex(ConfigError, "unexpected incomplete snapshot path"): with self.assertRaisesRegex(ConfigError, "unexpected incomplete snapshot path"):

View File

@@ -0,0 +1,195 @@
from __future__ import annotations
from datetime import datetime, timezone
from pathlib import Path
from tempfile import TemporaryDirectory
from django.test import TestCase
from pobsync.run_stats import tree_usage
from pobsync_backend.models import HostConfig, SnapshotRecord
from pobsync_backend.stats_summary import collect_dashboard_stats, collect_host_stats
class StatsSummaryTests(TestCase):
def test_collect_dashboard_stats_sums_backup_data_across_hosts(self) -> None:
web = HostConfig.objects.create(host="web-01", address="web-01.example.test")
db = HostConfig.objects.create(host="db-01", address="db-01.example.test")
self._snapshot(web, "20260519-021500Z__SCHED01", SnapshotRecord.Kind.SCHEDULED, allocated=100)
self._snapshot(web, "20260519-031500Z__MANUAL1", SnapshotRecord.Kind.MANUAL, allocated=200)
self._snapshot(db, "20260519-041500Z__SCHED02", SnapshotRecord.Kind.SCHEDULED, allocated=300)
with TemporaryDirectory() as tmp:
incomplete_usage = self._incomplete_snapshot_on_disk(
db,
Path(tmp),
"20260519-051500Z__BROKEN1",
)
stats = collect_dashboard_stats(hosts=[web, db], global_config=None)
self.assertEqual(stats["backup_data"]["scheduled"]["count"], 2)
self.assertEqual(stats["backup_data"]["scheduled"]["allocated_size_bytes"], 400)
self.assertEqual(stats["backup_data"]["manual"]["allocated_size_bytes"], 200)
self.assertEqual(stats["backup_data"]["incomplete"]["allocated_size_bytes"], incomplete_usage["allocated_size_bytes"])
self.assertEqual(stats["backup_data"]["total"]["count"], 4)
self.assertEqual(stats["backup_data"]["total"]["allocated_size_bytes"], 600 + incomplete_usage["allocated_size_bytes"])
self.assertEqual(stats["backup_data"]["total"]["unique_apparent_size_bytes"], 1200 + incomplete_usage["apparent_size_bytes"])
def test_collect_host_stats_sums_backup_data_by_snapshot_kind(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
self._snapshot(host, "20260519-021500Z__SCHED01", SnapshotRecord.Kind.SCHEDULED, allocated=100)
self._snapshot(host, "20260519-031500Z__SCHED02", SnapshotRecord.Kind.SCHEDULED, allocated=200)
self._snapshot(host, "20260519-041500Z__MANUAL1", SnapshotRecord.Kind.MANUAL, allocated=300)
with TemporaryDirectory() as tmp:
incomplete_usage = self._incomplete_snapshot_on_disk(
host,
Path(tmp),
"20260519-051500Z__BROKEN1",
)
stats = collect_host_stats(host=host)
self.assertEqual(stats["backup_data"]["scheduled"]["count"], 2)
self.assertEqual(stats["backup_data"]["scheduled"]["allocated_size_bytes"], 300)
self.assertEqual(stats["backup_data"]["manual"]["allocated_size_bytes"], 300)
self.assertEqual(stats["backup_data"]["incomplete"]["allocated_size_bytes"], incomplete_usage["allocated_size_bytes"])
self.assertEqual(stats["backup_data"]["total"]["count"], 4)
self.assertEqual(stats["backup_data"]["total"]["allocated_size_bytes"], 600 + incomplete_usage["allocated_size_bytes"])
self.assertEqual(stats["backup_data"]["total"]["unique_apparent_size_bytes"], 1200 + incomplete_usage["apparent_size_bytes"])
def test_collect_host_stats_falls_back_to_filesystem_usage_for_snapshots_without_metadata(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
with TemporaryDirectory() as tmp:
incomplete_dir = Path(tmp) / host.host / ".incomplete" / "20260519-051500Z__BROKEN1"
data_dir = incomplete_dir / "data"
meta_dir = incomplete_dir / "meta"
data_dir.mkdir(parents=True)
meta_dir.mkdir()
data_dir.joinpath("partial-file").write_text("interrupted backup data\n", encoding="utf-8")
meta_dir.joinpath("rsync.log").write_text("not part of the backup data total\n", encoding="utf-8")
expected_usage = tree_usage(data_dir)
SnapshotRecord.objects.create(
host=host,
kind=SnapshotRecord.Kind.INCOMPLETE,
dirname=incomplete_dir.name,
path=str(incomplete_dir),
status="failed",
metadata={},
)
stats = collect_host_stats(host=host)
self.assertEqual(stats["backup_data"]["incomplete"]["count"], 1)
self.assertEqual(
stats["backup_data"]["incomplete"]["allocated_size_bytes"],
expected_usage["allocated_size_bytes"],
)
self.assertEqual(
stats["backup_data"]["incomplete"]["apparent_size_bytes"],
expected_usage["apparent_size_bytes"],
)
self.assertEqual(
stats["backup_data"]["total"]["allocated_size_bytes"],
expected_usage["allocated_size_bytes"],
)
def test_collect_host_stats_measures_incomplete_data_from_disk_even_with_stale_metadata(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
with TemporaryDirectory() as tmp:
incomplete_dir = Path(tmp) / host.host / ".incomplete" / "20260519-051500Z__BROKEN1"
data_dir = incomplete_dir / "data"
data_dir.mkdir(parents=True)
data_dir.joinpath("partial-file").write_text("interrupted backup data\n", encoding="utf-8")
expected_usage = tree_usage(data_dir)
SnapshotRecord.objects.create(
host=host,
kind=SnapshotRecord.Kind.INCOMPLETE,
dirname=incomplete_dir.name,
path=str(incomplete_dir),
status="failed",
metadata={
"stats": {
"storage": {
"snapshot": {
"apparent_size_bytes": 0,
"allocated_size_bytes": 0,
}
}
}
},
)
stats = collect_host_stats(host=host)
self.assertEqual(
stats["backup_data"]["incomplete"]["allocated_size_bytes"],
expected_usage["allocated_size_bytes"],
)
self.assertGreater(stats["backup_data"]["incomplete"]["apparent_size_bytes"], 0)
def test_collect_host_stats_reports_non_hardlinked_snapshot_data(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
self._snapshot_with_sizes(
host,
"20260519-021500Z__SCHED01",
SnapshotRecord.Kind.SCHEDULED,
allocated=1_200,
apparent=2_000,
hardlinked_apparent=1_500,
)
stats = collect_host_stats(host=host)
self.assertEqual(stats["backup_data"]["scheduled"]["apparent_size_bytes"], 2_000)
self.assertEqual(stats["backup_data"]["scheduled"]["unique_apparent_size_bytes"], 500)
self.assertEqual(stats["backup_data"]["total"]["unique_apparent_size_bytes"], 500)
def _snapshot(self, host: HostConfig, dirname: str, kind: str, *, allocated: int) -> SnapshotRecord:
return self._snapshot_with_sizes(host, dirname, kind, allocated=allocated)
def _incomplete_snapshot_on_disk(self, host: HostConfig, root: Path, dirname: str) -> dict:
incomplete_dir = root / host.host / ".incomplete" / dirname
data_dir = incomplete_dir / "data"
data_dir.mkdir(parents=True)
data_dir.joinpath("partial-file").write_text("interrupted backup data\n", encoding="utf-8")
usage = tree_usage(data_dir)
SnapshotRecord.objects.create(
host=host,
kind=SnapshotRecord.Kind.INCOMPLETE,
dirname=dirname,
path=str(incomplete_dir),
status="failed",
)
return usage
def _snapshot_with_sizes(
self,
host: HostConfig,
dirname: str,
kind: str,
*,
allocated: int,
apparent: int | None = None,
hardlinked_apparent: int = 0,
) -> SnapshotRecord:
started_at = datetime.strptime(dirname.split("__", 1)[0], "%Y%m%d-%H%M%SZ").replace(tzinfo=timezone.utc)
apparent_size = apparent if apparent is not None else allocated * 2
return SnapshotRecord.objects.create(
host=host,
kind=kind,
dirname=dirname,
path=f"/backups/{host.host}/{kind}/{dirname}",
status="success",
started_at=started_at,
metadata={
"stats": {
"storage": {
"snapshot": {
"apparent_size_bytes": apparent_size,
"allocated_size_bytes": allocated,
"hardlinked_apparent_size_bytes": hardlinked_apparent,
}
}
}
},
)

View File

@@ -0,0 +1,72 @@
from __future__ import annotations
import json
from unittest.mock import MagicMock, patch
from django.test import SimpleTestCase, override_settings
from pobsync_backend import updater
class UpdaterTests(SimpleTestCase):
@override_settings(
POBSYNC_UPDATE_RELEASES_URL="https://code.example.test/api/v1/repos/owner/pobsync/releases",
POBSYNC_UPDATE_RELEASES_TOKEN="secret",
)
def test_fetch_latest_release_reads_first_gitea_release(self) -> None:
response = MagicMock()
response.__enter__.return_value.read.return_value = json.dumps(
[
{
"tag_name": "v1.2.0",
"name": "1.2.0",
"html_url": "https://code.example.test/releases/v1.2.0",
}
]
).encode("utf-8")
with patch("pobsync_backend.updater.urlopen", return_value=response) as urlopen:
release = updater.fetch_latest_release()
self.assertEqual(release["tag_name"], "v1.2.0")
request = urlopen.call_args.args[0]
self.assertEqual(request.full_url, "https://code.example.test/api/v1/repos/owner/pobsync/releases")
self.assertEqual(request.headers["Authorization"], "token secret")
@override_settings(POBSYNC_UPDATE_RELEASES_URL="")
def test_collect_update_status_reports_unconfigured_release_check(self) -> None:
with patch("pobsync_backend.updater._git_status", return_value={"branch": "master"}):
status = updater.collect_update_status(check_release=True)
self.assertFalse(status["release_check_configured"])
self.assertEqual(status["release_error"], "POBSYNC_UPDATE_RELEASES_URL is not configured.")
self.assertIsNone(status["update_available"])
@override_settings(POBSYNC_UPDATE_GIT_REMOTE="upstream")
def test_run_git_fetch_uses_configured_remote(self) -> None:
completed = MagicMock(returncode=0, stdout="ok", stderr="")
with patch("pobsync_backend.updater.subprocess.run", return_value=completed) as run:
result = updater.run_git_fetch()
self.assertTrue(result.ok)
self.assertEqual(result.command, ["git", "fetch", "--tags", "--prune", "upstream"])
run.assert_called_once()
@override_settings(POBSYNC_UPDATE_GIT_REMOTE="origin")
def test_run_git_pull_rejects_detached_checkout(self) -> None:
with patch("pobsync_backend.updater._git_current_branch", return_value=""):
result = updater.run_git_pull()
self.assertFalse(result.ok)
self.assertEqual(result.exit_code, 2)
self.assertIn("not on a branch", result.stderr)
@override_settings(POBSYNC_UPDATE_COMMAND="sudo -n scripts/update-systemd --verbose")
def test_run_native_update_splits_configured_command(self) -> None:
completed = MagicMock(returncode=1, stdout="", stderr="sudo failed")
with patch("pobsync_backend.updater.subprocess.run", return_value=completed):
result = updater.run_native_update()
self.assertFalse(result.ok)
self.assertEqual(result.command, ["sudo", "-n", "scripts/update-systemd", "--verbose"])
self.assertEqual(result.stderr, "sudo failed")

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,149 @@
from __future__ import annotations
import json
import shlex
import subprocess
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
from django.conf import settings
from pobsync import __version__
@dataclass(frozen=True)
class CommandResult:
command: list[str]
exit_code: int
stdout: str
stderr: str
@property
def ok(self) -> bool:
return self.exit_code == 0
def collect_update_status(*, check_release: bool = False) -> dict[str, Any]:
app_dir = Path(settings.BASE_DIR)
status: dict[str, Any] = {
"app_dir": app_dir,
"installed_version": __version__,
"release_check_configured": bool(settings.POBSYNC_UPDATE_RELEASES_URL),
"update_command": settings.POBSYNC_UPDATE_COMMAND,
"git_remote": settings.POBSYNC_UPDATE_GIT_REMOTE,
"git": _git_status(app_dir),
"latest_release": None,
"release_error": "",
"update_available": None,
}
if check_release:
if not settings.POBSYNC_UPDATE_RELEASES_URL:
status["release_error"] = "POBSYNC_UPDATE_RELEASES_URL is not configured."
else:
try:
latest_release = fetch_latest_release()
status["latest_release"] = latest_release
status["update_available"] = _version_key(latest_release.get("tag_name", "")) != _version_key(__version__)
except (HTTPError, URLError, TimeoutError, json.JSONDecodeError, ValueError) as exc:
status["release_error"] = str(exc)
return status
def fetch_latest_release() -> dict[str, Any]:
request = Request(settings.POBSYNC_UPDATE_RELEASES_URL, headers={"Accept": "application/json"})
if settings.POBSYNC_UPDATE_RELEASES_TOKEN:
request.add_header("Authorization", f"token {settings.POBSYNC_UPDATE_RELEASES_TOKEN}")
with urlopen(request, timeout=10) as response:
payload = json.loads(response.read().decode("utf-8"))
if isinstance(payload, list):
if not payload:
raise ValueError("No releases were returned.")
release = payload[0]
elif isinstance(payload, dict):
release = payload
else:
raise ValueError("Release endpoint returned an unexpected payload.")
if not isinstance(release, dict):
raise ValueError("Release endpoint returned an unexpected release entry.")
return release
def run_git_fetch() -> CommandResult:
remote = settings.POBSYNC_UPDATE_GIT_REMOTE
return _run_command(["git", "fetch", "--tags", "--prune", remote])
def run_git_pull() -> CommandResult:
remote = settings.POBSYNC_UPDATE_GIT_REMOTE
branch = _git_current_branch(Path(settings.BASE_DIR))
if not branch:
return CommandResult(
command=["git", "pull", "--ff-only", remote],
exit_code=2,
stdout="",
stderr="Cannot pull automatically because the installed checkout is not on a branch.",
)
return _run_command(["git", "pull", "--ff-only", remote, branch])
def run_native_update() -> CommandResult:
return _run_command(shlex.split(settings.POBSYNC_UPDATE_COMMAND))
def _run_command(command: list[str]) -> CommandResult:
completed = subprocess.run(
command,
cwd=settings.BASE_DIR,
capture_output=True,
check=False,
text=True,
timeout=600,
)
return CommandResult(
command=command,
exit_code=completed.returncode,
stdout=completed.stdout[-6000:],
stderr=completed.stderr[-6000:],
)
def _git_status(app_dir: Path) -> dict[str, str]:
return {
"branch": _git_current_branch(app_dir),
"commit": _git_output(app_dir, ["git", "rev-parse", "--short", "HEAD"]),
"describe": _git_output(app_dir, ["git", "describe", "--tags", "--always", "--dirty"]),
}
def _git_current_branch(app_dir: Path) -> str:
branch = _git_output(app_dir, ["git", "branch", "--show-current"])
return branch or _git_output(app_dir, ["git", "rev-parse", "--abbrev-ref", "HEAD"])
def _git_output(app_dir: Path, command: list[str]) -> str:
try:
completed = subprocess.run(
command,
cwd=app_dir,
capture_output=True,
check=False,
text=True,
timeout=5,
)
except (FileNotFoundError, subprocess.TimeoutExpired):
return ""
if completed.returncode != 0:
return ""
return completed.stdout.strip()
def _version_key(value: str) -> str:
return value.strip().removeprefix("v")

View File

@@ -1,24 +1,29 @@
from __future__ import annotations from __future__ import annotations
import os
import json import json
import shlex import shlex
import shutil import shutil
import subprocess import subprocess
from datetime import datetime, timezone as datetime_timezone
from pathlib import Path from pathlib import Path
from urllib.parse import urlencode
from django.contrib import messages from django.contrib import messages
from django.contrib.admin.views.decorators import staff_member_required
from django.conf import settings from django.conf import settings
from django.http import FileResponse, Http404 from django.http import FileResponse, Http404
from django.db.models import Count, Q from django.db.models import Count, Q
from django.shortcuts import get_object_or_404, redirect, render from django.shortcuts import get_object_or_404, redirect, render
from django.urls import reverse
from django.utils import timezone from django.utils import timezone
from django.utils.http import url_has_allowed_host_and_scheme
from django.views.decorators.http import require_POST from django.views.decorators.http import require_POST
from pobsync import __version__ from pobsync import __version__
from pobsync.errors import PobsyncError from pobsync.errors import PobsyncError
from pobsync.config.defaults import DEFAULT_EXCLUDES, DEFAULT_RSYNC_ARGS from pobsync.config.defaults import DEFAULT_EXCLUDES, DEFAULT_RSYNC_ARGS
from .access import access_context, control_panel_admin_required, status_view_required
from .backup_runner import queue_backup_run from .backup_runner import queue_backup_run
from .config_checks import collect_effective_host_config_checks, collect_global_config_checks from .config_checks import collect_effective_host_config_checks, collect_global_config_checks
from .forms import ( from .forms import (
@@ -27,13 +32,24 @@ from .forms import (
HostConfigForm, HostConfigForm,
IncompleteCleanupForm, IncompleteCleanupForm,
ManualBackupForm, ManualBackupForm,
NotificationTargetForm,
RetentionApplyForm, RetentionApplyForm,
SshCredentialGenerateForm, SshCredentialGenerateForm,
ScheduleConfigForm, ScheduleConfigForm,
SshCredentialForm, SshCredentialForm,
) )
from .host_ops import ensure_host_directories from .host_ops import ensure_host_directories
from .models import BackupRun, GlobalConfig, HostConfig, PurgedSnapshot, ScheduleConfig, SnapshotRecord, SshCredential from .models import (
BackupRun,
GlobalConfig,
HostConfig,
NotificationDelivery,
NotificationTarget,
PurgedSnapshot,
ScheduleConfig,
SnapshotRecord,
SshCredential,
)
from .preflight import collect_backup_gate, effective_host_config_preview, run_remote_preflight from .preflight import collect_backup_gate, effective_host_config_preview, run_remote_preflight
from .retention import run_incomplete_cleanup, run_sql_retention_apply, run_sql_retention_plan from .retention import run_incomplete_cleanup, run_sql_retention_apply, run_sql_retention_plan
from .self_check import collect_self_checks, summarize_self_checks from .self_check import collect_self_checks, summarize_self_checks
@@ -41,11 +57,25 @@ from .scheduler import next_due_after
from .snapshot_discovery import discover_snapshots, inspect_snapshot_discovery from .snapshot_discovery import discover_snapshots, inspect_snapshot_discovery
from .ssh_keys import SshKeyError, delete_generated_key_files, generate_ssh_key, merge_known_hosts, scan_known_host from .ssh_keys import SshKeyError, delete_generated_key_files, generate_ssh_key, merge_known_hosts, scan_known_host
from .stats_summary import collect_dashboard_stats, collect_host_stats from .stats_summary import collect_dashboard_stats, collect_host_stats
from .updater import collect_update_status, run_git_fetch, run_git_pull, run_native_update
@staff_member_required @status_view_required
def dashboard(request): def dashboard(request):
global_config = GlobalConfig.objects.filter(name="default").first() return render(request, "pobsync_backend/dashboard.html", _dashboard_context(request))
@status_view_required
def dashboard_priority_live(request):
return render(request, "pobsync_backend/partials/dashboard_priority.html", _dashboard_context(request))
@status_view_required
def dashboard_hosts_live(request):
return render(request, "pobsync_backend/partials/dashboard_hosts.html", _dashboard_context(request))
def _host_cards_context(*, enabled: str = "") -> dict[str, object]:
hosts = list( hosts = list(
HostConfig.objects.select_related("schedule") HostConfig.objects.select_related("schedule")
.annotate( .annotate(
@@ -66,6 +96,11 @@ def dashboard(request):
) )
.order_by("host") .order_by("host")
) )
if enabled == "yes":
hosts = [host for host in hosts if host.enabled]
elif enabled == "no":
hosts = [host for host in hosts if not host.enabled]
for host_config in hosts: for host_config in hosts:
host_config.latest_snapshot = ( host_config.latest_snapshot = (
host_config.snapshots.select_related("base") host_config.snapshots.select_related("base")
@@ -74,13 +109,35 @@ def dashboard(request):
) )
host_config.next_run_at = _next_run_for_host(host_config) host_config.next_run_at = _next_run_for_host(host_config)
host_config.retention_warning = _retention_warning_for_host(host_config, _schedule_for_host(host_config)) host_config.retention_warning = _retention_warning_for_host(host_config, _schedule_for_host(host_config))
return {
"hosts": hosts,
"scheduler_timezone": timezone.get_current_timezone_name(),
"selected_enabled": enabled,
"counts": {
"hosts": len(hosts),
"enabled_hosts": sum(1 for host in hosts if host.enabled),
"disabled_hosts": sum(1 for host in hosts if not host.enabled),
},
}
def _dashboard_context(request) -> dict[str, object]:
global_config = GlobalConfig.objects.filter(name="default").first()
host_context = _host_cards_context()
hosts = host_context["hosts"]
action_items = _dashboard_action_items(hosts)
next_schedule_rows = _dashboard_next_schedule_rows()
recent_runs = BackupRun.objects.select_related("host", "snapshot").order_by("-created_at", "-id")[:6]
stats_summary = collect_dashboard_stats(hosts=hosts, global_config=global_config) stats_summary = collect_dashboard_stats(hosts=hosts, global_config=global_config)
context = { context = {
**access_context(request),
"hosts": hosts, "hosts": hosts,
"global_config": global_config, "global_config": global_config,
"stats_summary": stats_summary, "stats_summary": stats_summary,
"scheduler_timezone": timezone.get_current_timezone_name(), "scheduler_timezone": host_context["scheduler_timezone"],
"latest_runs": BackupRun.objects.select_related("host", "snapshot").order_by("-created_at")[:10], "action_items": action_items,
"next_schedule_rows": next_schedule_rows,
"recent_runs": recent_runs,
"counts": { "counts": {
"global_configs": GlobalConfig.objects.count(), "global_configs": GlobalConfig.objects.count(),
"hosts": HostConfig.objects.count(), "hosts": HostConfig.objects.count(),
@@ -101,10 +158,142 @@ def dashboard(request):
).count(), ).count(),
}, },
} }
return render(request, "pobsync_backend/dashboard.html", context) return context
@staff_member_required @status_view_required
def hosts_list(request):
enabled = request.GET.get("enabled", "").strip()
if enabled not in {"", "yes", "no"}:
enabled = ""
global_config = GlobalConfig.objects.filter(name="default").first()
context = _host_cards_context(enabled=enabled)
collect_dashboard_stats(hosts=context["hosts"], global_config=global_config)
return render(
request,
"pobsync_backend/hosts_list.html",
{
**access_context(request),
**context,
"global_config": global_config,
"show_host_controls": request.user.is_staff,
"total_count": HostConfig.objects.count(),
},
)
@control_panel_admin_required
@require_POST
def update_host_state(request, host: str):
host_config = get_object_or_404(HostConfig, host=host)
action = request.POST.get("action", "").strip()
next_url = request.POST.get("next") or reverse("hosts_list")
if not url_has_allowed_host_and_scheme(next_url, allowed_hosts={request.get_host()}):
next_url = reverse("hosts_list")
if action == "enable_host":
host_config.enabled = True
host_config.save(update_fields=["enabled", "updated_at"])
messages.success(request, f"Enabled host {host_config.host}.")
elif action == "disable_host":
host_config.enabled = False
host_config.save(update_fields=["enabled", "updated_at"])
messages.success(request, f"Disabled host {host_config.host}.")
elif action in {"enable_schedule", "disable_schedule", "enable_prune", "disable_prune"}:
try:
schedule = host_config.schedule
except ScheduleConfig.DoesNotExist:
messages.warning(request, f"{host_config.host} does not have a schedule yet.")
else:
if action == "enable_schedule":
schedule.enabled = True
message = f"Enabled backup schedule for {host_config.host}."
elif action == "disable_schedule":
schedule.enabled = False
message = f"Paused backup schedule for {host_config.host}."
elif action == "enable_prune":
schedule.prune = True
message = f"Enabled scheduled retention for {host_config.host}."
else:
schedule.prune = False
message = f"Paused scheduled retention for {host_config.host}."
schedule.save(update_fields=["enabled", "prune", "updated_at"])
messages.success(request, message)
else:
messages.error(request, "Unknown host state action.")
return redirect(next_url)
def _dashboard_action_items(hosts: list[HostConfig]) -> list[dict[str, object]]:
action_items: list[dict[str, object]] = []
for host_config in hosts:
if host_config.failed_run_count:
action_items.append(
{
"host": host_config,
"status": BackupRun.Status.FAILED,
"label": "Failed runs",
"message": f"{host_config.failed_run_count} failed run(s) need review.",
"url": _runs_list_url(host=host_config.host, status="failed", review="needed"),
}
)
if host_config.warning_run_count:
action_items.append(
{
"host": host_config,
"status": BackupRun.Status.WARNING,
"label": "Warnings",
"message": f"{host_config.warning_run_count} run(s) completed with warnings.",
"url": _runs_list_url(host=host_config.host, status="warning", review="needed"),
}
)
if host_config.retention_warning.get("has_warning"):
action_items.append(
{
"host": host_config,
"status": BackupRun.Status.WARNING,
"label": "Retention",
"message": _retention_warning_summary(host_config.retention_warning),
"url": reverse("host_detail", args=[host_config.host]),
}
)
return action_items
def _runs_list_url(**params: str) -> str:
return f"{reverse('runs_list')}?{urlencode(params)}"
def _dashboard_next_schedule_rows() -> list[dict[str, object]]:
rows = []
schedules = ScheduleConfig.objects.select_related("host").filter(enabled=True).order_by("host__host")
for schedule in schedules[:200]:
rows.append(
{
"schedule": schedule,
"next_run_at": _next_run_for_schedule(schedule, schedule.host),
}
)
rows.sort(key=lambda row: row["next_run_at"] or datetime.max.replace(tzinfo=datetime_timezone.utc))
return rows[:6]
def _retention_warning_summary(retention_warning) -> str:
parts = []
if retention_warning.get("prune_exceeded"):
parts.append(
f"Scheduled prune would delete {retention_warning.get('delete_count')} snapshot(s), "
f"above max {retention_warning.get('max_delete')}."
)
if retention_warning.get("incomplete_count"):
parts.append(f"{retention_warning.get('incomplete_count')} incomplete snapshot(s) need review.")
if retention_warning.get("error"):
parts.append(str(retention_warning.get("error")))
return " ".join(parts)
@status_view_required
def changelog(request): def changelog(request):
changelog_path = Path(settings.BASE_DIR) / "CHANGELOG.md" changelog_path = Path(settings.BASE_DIR) / "CHANGELOG.md"
try: try:
@@ -126,7 +315,7 @@ def changelog(request):
) )
@staff_member_required @control_panel_admin_required
def self_check(request): def self_check(request):
checks = collect_self_checks() checks = collect_self_checks()
return render( return render(
@@ -139,13 +328,106 @@ def self_check(request):
) )
@staff_member_required @control_panel_admin_required
def logs(request): def logs(request):
context = _log_context(request) context = _log_context(request)
return render(request, "pobsync_backend/logs.html", context) return render(request, "pobsync_backend/logs.html", context)
@staff_member_required @control_panel_admin_required
def updater(request):
action_result = None
check_release = request.method == "POST" and request.POST.get("action") == "check_release"
if request.method == "POST":
action = request.POST.get("action")
if action == "git_fetch":
action_result = run_git_fetch()
check_release = True
elif action == "git_pull":
action_result = run_git_pull()
check_release = True
elif action == "run_update":
action_result = run_native_update()
check_release = True
elif action != "check_release":
messages.error(request, "Unknown updater action.")
if action_result is not None:
if action_result.ok:
messages.success(request, "Updater action completed successfully.")
else:
messages.error(request, f"Updater action failed with exit code {action_result.exit_code}.")
return render(
request,
"pobsync_backend/updater.html",
{
"status": collect_update_status(check_release=check_release),
"action_result": action_result,
},
)
@control_panel_admin_required
def notification_targets(request):
targets = NotificationTarget.objects.order_by("name")
deliveries = NotificationDelivery.objects.select_related("target", "run", "run__host").order_by("-created_at")[:12]
return render(
request,
"pobsync_backend/notification_targets.html",
{
"targets": targets,
"deliveries": deliveries,
},
)
@control_panel_admin_required
def create_notification_target(request):
if request.method == "POST":
form = NotificationTargetForm(request.POST)
if form.is_valid():
target = form.save()
messages.success(request, f"Notification target {target.name} created.")
return redirect("notification_targets")
else:
form = NotificationTargetForm()
return render(
request,
"pobsync_backend/notification_target_form.html",
{
"form": form,
"target": None,
"title": "New notification target",
"submit_label": "Create target",
},
)
@control_panel_admin_required
def edit_notification_target(request, target_id: int):
target = get_object_or_404(NotificationTarget, id=target_id)
if request.method == "POST":
form = NotificationTargetForm(request.POST, instance=target)
if form.is_valid():
target = form.save()
messages.success(request, f"Notification target {target.name} updated.")
return redirect("notification_targets")
else:
form = NotificationTargetForm(instance=target)
return render(
request,
"pobsync_backend/notification_target_form.html",
{
"form": form,
"target": target,
"title": f"Edit notification target: {target.name}",
"submit_label": "Save target",
},
)
@status_view_required
def runs_list(request): def runs_list(request):
status = request.GET.get("status", "").strip() status = request.GET.get("status", "").strip()
run_type = request.GET.get("type", "").strip() run_type = request.GET.get("type", "").strip()
@@ -177,7 +459,7 @@ def runs_list(request):
return render(request, "pobsync_backend/runs_list.html", context) return render(request, "pobsync_backend/runs_list.html", context)
@staff_member_required @status_view_required
def snapshots_list(request): def snapshots_list(request):
kind = request.GET.get("kind", "").strip() kind = request.GET.get("kind", "").strip()
status = request.GET.get("status", "").strip() status = request.GET.get("status", "").strip()
@@ -203,7 +485,7 @@ def snapshots_list(request):
return render(request, "pobsync_backend/snapshots_list.html", context) return render(request, "pobsync_backend/snapshots_list.html", context)
@staff_member_required @status_view_required
def schedules_list(request): def schedules_list(request):
enabled = request.GET.get("enabled", "").strip() enabled = request.GET.get("enabled", "").strip()
prune = request.GET.get("prune", "").strip() prune = request.GET.get("prune", "").strip()
@@ -241,7 +523,7 @@ def schedules_list(request):
return render(request, "pobsync_backend/schedules_list.html", context) return render(request, "pobsync_backend/schedules_list.html", context)
@staff_member_required @status_view_required
def purged_snapshots(request): def purged_snapshots(request):
host = request.GET.get("host", "").strip() host = request.GET.get("host", "").strip()
action = request.GET.get("action", "").strip() action = request.GET.get("action", "").strip()
@@ -262,7 +544,7 @@ def purged_snapshots(request):
return render(request, "pobsync_backend/purged_snapshots.html", context) return render(request, "pobsync_backend/purged_snapshots.html", context)
@staff_member_required @control_panel_admin_required
def ssh_credentials(request): def ssh_credentials(request):
context = { context = {
"credentials": SshCredential.objects.order_by("name"), "credentials": SshCredential.objects.order_by("name"),
@@ -270,7 +552,7 @@ def ssh_credentials(request):
return render(request, "pobsync_backend/ssh_credentials.html", context) return render(request, "pobsync_backend/ssh_credentials.html", context)
@staff_member_required @control_panel_admin_required
def create_ssh_credential(request): def create_ssh_credential(request):
if request.method == "POST": if request.method == "POST":
form = SshCredentialForm(request.POST, request.FILES) form = SshCredentialForm(request.POST, request.FILES)
@@ -291,7 +573,7 @@ def create_ssh_credential(request):
) )
@staff_member_required @control_panel_admin_required
def generate_ssh_credential(request): def generate_ssh_credential(request):
if request.method == "POST": if request.method == "POST":
form = SshCredentialGenerateForm(request.POST) form = SshCredentialGenerateForm(request.POST)
@@ -327,7 +609,7 @@ def generate_ssh_credential(request):
) )
@staff_member_required @control_panel_admin_required
def edit_ssh_credential(request, credential_id: int): def edit_ssh_credential(request, credential_id: int):
credential = get_object_or_404(SshCredential, id=credential_id) credential = get_object_or_404(SshCredential, id=credential_id)
if request.method == "POST": if request.method == "POST":
@@ -349,7 +631,7 @@ def edit_ssh_credential(request, credential_id: int):
) )
@staff_member_required @control_panel_admin_required
@require_POST @require_POST
def delete_ssh_credential(request, credential_id: int): def delete_ssh_credential(request, credential_id: int):
credential = get_object_or_404(SshCredential, id=credential_id) credential = get_object_or_404(SshCredential, id=credential_id)
@@ -373,7 +655,7 @@ def delete_ssh_credential(request, credential_id: int):
return redirect("ssh_credentials") return redirect("ssh_credentials")
@staff_member_required @control_panel_admin_required
def edit_global_config(request): def edit_global_config(request):
global_config = GlobalConfig.objects.filter(name="default").first() global_config = GlobalConfig.objects.filter(name="default").first()
if request.method == "POST": if request.method == "POST":
@@ -399,7 +681,7 @@ def edit_global_config(request):
) )
@staff_member_required @control_panel_admin_required
def create_host_config(request): def create_host_config(request):
if request.method == "POST": if request.method == "POST":
form = CreateHostConfigForm(request.POST) form = CreateHostConfigForm(request.POST)
@@ -427,7 +709,7 @@ def create_host_config(request):
) )
@staff_member_required @status_view_required
def host_detail(request, host: str): def host_detail(request, host: str):
host_config = get_object_or_404(HostConfig, host=host) host_config = get_object_or_404(HostConfig, host=host)
global_config = GlobalConfig.objects.filter(name="default").first() global_config = GlobalConfig.objects.filter(name="default").first()
@@ -440,7 +722,9 @@ def host_detail(request, host: str):
has_global_config = global_config is not None has_global_config = global_config is not None
backup_gate = collect_backup_gate(host_config, global_config) backup_gate = collect_backup_gate(host_config, global_config)
stats_summary = collect_host_stats(host=host_config, limit=10) stats_summary = collect_host_stats(host=host_config, limit=10)
can_manage = request.user.is_staff
context = { context = {
**access_context(request),
"host": host_config, "host": host_config,
"schedule": schedule, "schedule": schedule,
"retention_warning": _retention_warning_for_host(host_config, schedule), "retention_warning": _retention_warning_for_host(host_config, schedule),
@@ -451,11 +735,11 @@ def host_detail(request, host: str):
"host_check_summary": summarize_self_checks(backup_gate.checks), "host_check_summary": summarize_self_checks(backup_gate.checks),
"backup_gate": backup_gate, "backup_gate": backup_gate,
"last_preflight": (host_config.config or {}).get("last_preflight") if isinstance(host_config.config, dict) else {}, "last_preflight": (host_config.config or {}).get("last_preflight") if isinstance(host_config.config, dict) else {},
"effective_config": effective_host_config_preview(host_config, global_config) if global_config else {}, "effective_config": effective_host_config_preview(host_config, global_config) if global_config and can_manage else {},
"stats_summary": stats_summary, "stats_summary": stats_summary,
"manual_backup_form": ManualBackupForm(initial=_default_manual_backup_initial(host_config)), "manual_backup_form": ManualBackupForm(initial=_default_manual_backup_initial(host_config)),
"can_queue_dry_run": host_config.enabled and has_global_config and backup_gate.can_queue_dry_run and active_run is None, "can_queue_dry_run": can_manage and host_config.enabled and has_global_config and backup_gate.can_queue_dry_run and active_run is None,
"can_queue_real_backup": host_config.enabled and has_global_config and backup_gate.can_queue_real and active_run is None, "can_queue_real_backup": can_manage and host_config.enabled and has_global_config and backup_gate.can_queue_real and active_run is None,
"has_global_config": has_global_config, "has_global_config": has_global_config,
"active_run": active_run, "active_run": active_run,
"latest_runs": host_config.runs.select_related("snapshot").order_by("-created_at")[:10], "latest_runs": host_config.runs.select_related("snapshot").order_by("-created_at")[:10],
@@ -475,7 +759,7 @@ def host_detail(request, host: str):
return render(request, "pobsync_backend/host_detail.html", context) return render(request, "pobsync_backend/host_detail.html", context)
@staff_member_required @control_panel_admin_required
@require_POST @require_POST
def prepare_host_directories(request, host: str): def prepare_host_directories(request, host: str):
host_config = get_object_or_404(HostConfig, host=host) host_config = get_object_or_404(HostConfig, host=host)
@@ -488,7 +772,7 @@ def prepare_host_directories(request, host: str):
return redirect("host_detail", host=host_config.host) return redirect("host_detail", host=host_config.host)
@staff_member_required @control_panel_admin_required
@require_POST @require_POST
def scan_host_known_key(request, host: str): def scan_host_known_key(request, host: str):
host_config = get_object_or_404(HostConfig, host=host) host_config = get_object_or_404(HostConfig, host=host)
@@ -510,7 +794,7 @@ def scan_host_known_key(request, host: str):
return redirect("host_detail", host=host_config.host) return redirect("host_detail", host=host_config.host)
@staff_member_required @control_panel_admin_required
@require_POST @require_POST
def run_host_preflight(request, host: str): def run_host_preflight(request, host: str):
host_config = get_object_or_404(HostConfig, host=host) host_config = get_object_or_404(HostConfig, host=host)
@@ -538,7 +822,7 @@ def run_host_preflight(request, host: str):
return redirect("host_detail", host=host_config.host) return redirect("host_detail", host=host_config.host)
@staff_member_required @control_panel_admin_required
@require_POST @require_POST
def queue_manual_backup(request, host: str): def queue_manual_backup(request, host: str):
host_config = get_object_or_404(HostConfig, host=host) host_config = get_object_or_404(HostConfig, host=host)
@@ -578,9 +862,22 @@ def queue_manual_backup(request, host: str):
return redirect("run_detail", run_id=run.id) return redirect("run_detail", run_id=run.id)
@staff_member_required @status_view_required
def run_detail(request, run_id: int): def run_detail(request, run_id: int):
run = get_object_or_404(BackupRun.objects.select_related("host", "snapshot"), id=run_id) run = get_object_or_404(BackupRun.objects.select_related("host", "snapshot"), id=run_id)
return render(request, "pobsync_backend/run_detail.html", _run_detail_context(run, request=request))
@status_view_required
def run_detail_live(request, run_id: int):
run = get_object_or_404(BackupRun.objects.select_related("host", "snapshot"), id=run_id)
context = _run_detail_context(run, request=request)
response = render(request, "pobsync_backend/partials/run_detail_live.html", context)
response["X-Pobsync-Refresh-Active"] = "true" if context["can_auto_refresh"] else "false"
return response
def _run_detail_context(run: BackupRun, *, request=None) -> dict[str, object]:
result = run.result if isinstance(run.result, dict) else {} result = run.result if isinstance(run.result, dict) else {}
run_stats = result.get("stats") if isinstance(result.get("stats"), dict) else {} run_stats = result.get("stats") if isinstance(result.get("stats"), dict) else {}
rsync_result = result.get("rsync") if isinstance(result.get("rsync"), dict) else {} rsync_result = result.get("rsync") if isinstance(result.get("rsync"), dict) else {}
@@ -590,14 +887,19 @@ def run_detail(request, run_id: int):
rsync_log_path = _run_rsync_log_path(run) rsync_log_path = _run_rsync_log_path(run)
rsync_log_tail = _run_rsync_log_tail(rsync_result, rsync_log_path) rsync_log_tail = _run_rsync_log_tail(rsync_result, rsync_log_path)
requested = result.get("requested") if isinstance(result.get("requested"), dict) else {} requested = result.get("requested") if isinstance(result.get("requested"), dict) else {}
context = { can_manage = bool(request and request.user.is_staff)
can_cancel = can_manage and run.status in {BackupRun.Status.QUEUED, BackupRun.Status.RUNNING}
return {
**(access_context(request) if request is not None else {}),
"run": run, "run": run,
"can_cancel": run.status in {BackupRun.Status.QUEUED, BackupRun.Status.RUNNING}, "can_cancel": can_cancel,
"can_auto_refresh": can_cancel,
"requested": requested, "requested": requested,
"execution": execution, "execution": execution,
"stats": run_stats if isinstance(run_stats, dict) else {}, "stats": run_stats if isinstance(run_stats, dict) else {},
"rsync": rsync_result, "rsync": rsync_result,
"rsync_command": _run_rsync_command(rsync_result), "rsync_command": _run_rsync_command(rsync_result),
"rsync_bwlimit_kbps": _run_rsync_bwlimit_kbps(rsync_result),
"failure": failure, "failure": failure,
"failure_summary": failure.get("message") or failure.get("summary") or "", "failure_summary": failure.get("message") or failure.get("summary") or "",
"prune_result": prune_result, "prune_result": prune_result,
@@ -606,6 +908,7 @@ def run_detail(request, run_id: int):
"rsync_log_path": str(rsync_log_path) if rsync_log_path is not None else "", "rsync_log_path": str(rsync_log_path) if rsync_log_path is not None else "",
"rsync_log_exists": bool(rsync_log_path and rsync_log_path.exists()), "rsync_log_exists": bool(rsync_log_path and rsync_log_path.exists()),
"rsync_log_tail": rsync_log_tail, "rsync_log_tail": rsync_log_tail,
"live_progress": _run_live_progress(run, rsync_log_path),
"dry_run_summary": _dry_run_summary( "dry_run_summary": _dry_run_summary(
result=result, result=result,
requested=requested, requested=requested,
@@ -616,10 +919,9 @@ def run_detail(request, run_id: int):
), ),
"result_json": _pretty_json(run.result), "result_json": _pretty_json(run.result),
} }
return render(request, "pobsync_backend/run_detail.html", context)
@staff_member_required @control_panel_admin_required
def run_rsync_log(request, run_id: int): def run_rsync_log(request, run_id: int):
run = get_object_or_404(BackupRun.objects.select_related("host", "snapshot"), id=run_id) run = get_object_or_404(BackupRun.objects.select_related("host", "snapshot"), id=run_id)
log_path = _run_rsync_log_path(run) log_path = _run_rsync_log_path(run)
@@ -628,7 +930,7 @@ def run_rsync_log(request, run_id: int):
return FileResponse(log_path.open("rb"), content_type="text/plain; charset=utf-8") return FileResponse(log_path.open("rb"), content_type="text/plain; charset=utf-8")
@staff_member_required @control_panel_admin_required
@require_POST @require_POST
def cancel_run(request, run_id: int): def cancel_run(request, run_id: int):
run = get_object_or_404(BackupRun.objects.select_related("host"), id=run_id) run = get_object_or_404(BackupRun.objects.select_related("host"), id=run_id)
@@ -652,7 +954,7 @@ def cancel_run(request, run_id: int):
return redirect("run_detail", run_id=run.id) return redirect("run_detail", run_id=run.id)
@staff_member_required @control_panel_admin_required
@require_POST @require_POST
def resolve_run_review(request, run_id: int): def resolve_run_review(request, run_id: int):
run = get_object_or_404(BackupRun.objects.select_related("host"), id=run_id) run = get_object_or_404(BackupRun.objects.select_related("host"), id=run_id)
@@ -661,16 +963,16 @@ def resolve_run_review(request, run_id: int):
return redirect("run_detail", run_id=run.id) return redirect("run_detail", run_id=run.id)
if run.reviewed_at: if run.reviewed_at:
messages.info(request, f"Run {run.id} was already marked reviewed.") messages.info(request, f"Run {run.id} was already marked reviewed.")
return redirect("run_detail", run_id=run.id) return _redirect_after_run_review(request, run)
run.reviewed_at = timezone.now() run.reviewed_at = timezone.now()
run.reviewed_by = request.user.get_username() run.reviewed_by = request.user.get_username()
run.save(update_fields=["reviewed_at", "reviewed_by"]) run.save(update_fields=["reviewed_at", "reviewed_by"])
messages.success(request, f"Run {run.id} marked reviewed.") messages.success(request, f"Run {run.id} marked reviewed.")
return redirect("run_detail", run_id=run.id) return _redirect_after_run_review(request, run)
@staff_member_required @control_panel_admin_required
@require_POST @require_POST
def resolve_host_incomplete_reviews(request, host: str): def resolve_host_incomplete_reviews(request, host: str):
host_config = get_object_or_404(HostConfig, host=host) host_config = get_object_or_404(HostConfig, host=host)
@@ -686,7 +988,7 @@ def resolve_host_incomplete_reviews(request, host: str):
return redirect("host_detail", host=host_config.host) return redirect("host_detail", host=host_config.host)
@staff_member_required @status_view_required
def snapshot_detail(request, snapshot_id: int): def snapshot_detail(request, snapshot_id: int):
snapshot = get_object_or_404( snapshot = get_object_or_404(
SnapshotRecord.objects.select_related("host", "base").prefetch_related("derived_snapshots", "backup_runs"), SnapshotRecord.objects.select_related("host", "base").prefetch_related("derived_snapshots", "backup_runs"),
@@ -704,7 +1006,7 @@ def snapshot_detail(request, snapshot_id: int):
return render(request, "pobsync_backend/snapshot_detail.html", context) return render(request, "pobsync_backend/snapshot_detail.html", context)
@staff_member_required @control_panel_admin_required
@require_POST @require_POST
def discover_host_snapshots(request, host: str): def discover_host_snapshots(request, host: str):
host_config = get_object_or_404(HostConfig, host=host) host_config = get_object_or_404(HostConfig, host=host)
@@ -725,7 +1027,7 @@ def discover_host_snapshots(request, host: str):
return redirect("host_detail", host=host_config.host) return redirect("host_detail", host=host_config.host)
@staff_member_required @control_panel_admin_required
def host_retention_plan(request, host: str): def host_retention_plan(request, host: str):
host_config = get_object_or_404(HostConfig, host=host) host_config = get_object_or_404(HostConfig, host=host)
kind = request.GET.get("kind", "scheduled") kind = request.GET.get("kind", "scheduled")
@@ -742,6 +1044,8 @@ def host_retention_plan(request, host: str):
scheduled_prune_limit = schedule.prune_max_delete if schedule and schedule.prune else None scheduled_prune_limit = schedule.prune_max_delete if schedule and schedule.prune else None
delete_count = len(plan["delete"]) delete_count = len(plan["delete"])
incomplete_count = len(plan["incomplete"]) incomplete_count = len(plan["incomplete"])
incomplete_reviewed_count = int(plan.get("incomplete_reviewed_count") or 0)
incomplete_unreviewed_count = int(plan.get("incomplete_unreviewed_count") or 0)
context = { context = {
"host": host_config, "host": host_config,
"kind": kind, "kind": kind,
@@ -750,6 +1054,8 @@ def host_retention_plan(request, host: str):
"schedule": schedule, "schedule": schedule,
"scheduled_prune_limit": scheduled_prune_limit, "scheduled_prune_limit": scheduled_prune_limit,
"scheduled_prune_exceeded": scheduled_prune_limit is not None and delete_count > scheduled_prune_limit, "scheduled_prune_exceeded": scheduled_prune_limit is not None and delete_count > scheduled_prune_limit,
"incomplete_reviewed_count": incomplete_reviewed_count,
"incomplete_unreviewed_count": incomplete_unreviewed_count,
"apply_form": RetentionApplyForm( "apply_form": RetentionApplyForm(
host_name=host_config.host, host_name=host_config.host,
expected_delete_count=delete_count, expected_delete_count=delete_count,
@@ -762,17 +1068,17 @@ def host_retention_plan(request, host: str):
), ),
"incomplete_cleanup_form": IncompleteCleanupForm( "incomplete_cleanup_form": IncompleteCleanupForm(
host_name=host_config.host, host_name=host_config.host,
expected_delete_count=incomplete_count, expected_delete_count=incomplete_reviewed_count,
initial={ initial={
"max_delete": incomplete_count, "max_delete": incomplete_reviewed_count,
"confirm_delete_count": incomplete_count, "confirm_delete_count": incomplete_reviewed_count,
}, },
), ),
} }
return render(request, "pobsync_backend/retention_plan.html", context) return render(request, "pobsync_backend/retention_plan.html", context)
@staff_member_required @control_panel_admin_required
@require_POST @require_POST
def apply_host_retention(request, host: str): def apply_host_retention(request, host: str):
host_config = get_object_or_404(HostConfig, host=host) host_config = get_object_or_404(HostConfig, host=host)
@@ -826,7 +1132,7 @@ def apply_host_retention(request, host: str):
return target return target
@staff_member_required @control_panel_admin_required
@require_POST @require_POST
def cleanup_host_incomplete_snapshots(request, host: str): def cleanup_host_incomplete_snapshots(request, host: str):
host_config = get_object_or_404(HostConfig, host=host) host_config = get_object_or_404(HostConfig, host=host)
@@ -836,7 +1142,7 @@ def cleanup_host_incomplete_snapshots(request, host: str):
messages.error(request, str(exc)) messages.error(request, str(exc))
return redirect("host_retention_plan", host=host_config.host) return redirect("host_retention_plan", host=host_config.host)
incomplete_count = len(plan.get("incomplete") or []) incomplete_count = int(plan.get("incomplete_reviewed_count") or 0)
form = IncompleteCleanupForm( form = IncompleteCleanupForm(
request.POST, request.POST,
host_name=host_config.host, host_name=host_config.host,
@@ -861,7 +1167,7 @@ def cleanup_host_incomplete_snapshots(request, host: str):
return redirect("host_retention_plan", host=host_config.host) return redirect("host_retention_plan", host=host_config.host)
@staff_member_required @control_panel_admin_required
def edit_host_config(request, host: str): def edit_host_config(request, host: str):
host_config = get_object_or_404(HostConfig, host=host) host_config = get_object_or_404(HostConfig, host=host)
global_config = GlobalConfig.objects.filter(name="default").first() global_config = GlobalConfig.objects.filter(name="default").first()
@@ -887,7 +1193,7 @@ def edit_host_config(request, host: str):
) )
@staff_member_required @control_panel_admin_required
def edit_host_schedule(request, host: str): def edit_host_schedule(request, host: str):
host_config = get_object_or_404(HostConfig, host=host) host_config = get_object_or_404(HostConfig, host=host)
schedule = _schedule_for_host(host_config) schedule = _schedule_for_host(host_config)
@@ -937,6 +1243,13 @@ def _next_run_for_schedule(schedule: ScheduleConfig | None, host_config: HostCon
return None return None
def _redirect_after_run_review(request, run: BackupRun):
next_url = request.POST.get("next", "").strip()
if next_url.startswith("/"):
return redirect(next_url)
return redirect("run_detail", run_id=run.id)
def _retention_warning_for_host(host_config: HostConfig, schedule: ScheduleConfig | None) -> dict[str, object]: def _retention_warning_for_host(host_config: HostConfig, schedule: ScheduleConfig | None) -> dict[str, object]:
incomplete_count = host_config.snapshots.filter( incomplete_count = host_config.snapshots.filter(
kind=SnapshotRecord.Kind.INCOMPLETE, kind=SnapshotRecord.Kind.INCOMPLETE,
@@ -1029,6 +1342,7 @@ def _default_manual_backup_initial(host_config: HostConfig) -> dict[str, object]
schedule = _schedule_for_host(host_config) schedule = _schedule_for_host(host_config)
return { return {
"dry_run": True, "dry_run": True,
"verbose_output": True,
"prune": bool(schedule.prune) if schedule else False, "prune": bool(schedule.prune) if schedule else False,
"prune_max_delete": schedule.prune_max_delete if schedule else 10, "prune_max_delete": schedule.prune_max_delete if schedule else 10,
"prune_protect_bases": bool(schedule.prune_protect_bases) if schedule else False, "prune_protect_bases": bool(schedule.prune_protect_bases) if schedule else False,
@@ -1136,6 +1450,23 @@ def _run_rsync_command(rsync_result: dict) -> list[str]:
return [str(part) for part in command] return [str(part) for part in command]
def _run_rsync_bwlimit_kbps(rsync_result: dict) -> int:
stored_limit = rsync_result.get("bwlimit_kbps")
if stored_limit is not None:
try:
return max(0, int(stored_limit))
except (TypeError, ValueError):
return 0
for part in _run_rsync_command(rsync_result):
if part.startswith("--bwlimit="):
try:
return max(0, int(part.split("=", 1)[1]))
except ValueError:
return 0
return 0
def _run_rsync_log_tail(rsync_result: dict, log_path: Path | None, *, max_lines: int = 30) -> list[str]: def _run_rsync_log_tail(rsync_result: dict, log_path: Path | None, *, max_lines: int = 30) -> list[str]:
log_tail = rsync_result.get("log_tail") log_tail = rsync_result.get("log_tail")
if isinstance(log_tail, list): if isinstance(log_tail, list):
@@ -1149,6 +1480,97 @@ def _run_rsync_log_tail(rsync_result: dict, log_path: Path | None, *, max_lines:
return [] return []
def _run_live_progress(run: BackupRun, log_path: Path | None) -> dict[str, object]:
if run.status not in {BackupRun.Status.QUEUED, BackupRun.Status.RUNNING}:
return {}
result = run.result if isinstance(run.result, dict) else {}
requested = result.get("requested") if isinstance(result.get("requested"), dict) else {}
if requested.get("dry_run") or result.get("dry_run"):
return {}
execution = result.get("execution") if isinstance(result.get("execution"), dict) else {}
rsync = result.get("rsync") if isinstance(result.get("rsync"), dict) else {}
progress: dict[str, object] = {
"phase": execution.get("phase") or ("queued" if run.status == BackupRun.Status.QUEUED else "running"),
"worker_pid": execution.get("worker_pid"),
"rsync_pid": rsync.get("pid"),
"rsync_pgid": rsync.get("pgid"),
}
log_stats = _live_log_stats(log_path)
if log_stats:
progress["log"] = log_stats
snapshot_path = _run_progress_snapshot_path(run, execution)
if snapshot_path is not None:
progress["snapshot"] = {
"path": str(snapshot_path),
**_scan_snapshot_progress(snapshot_path / "data" if (snapshot_path / "data").exists() else snapshot_path),
}
return progress
def _run_progress_snapshot_path(run: BackupRun, execution: dict) -> Path | None:
snapshot = execution.get("snapshot")
if isinstance(snapshot, str) and snapshot:
return Path(snapshot)
if run.snapshot_path:
return Path(run.snapshot_path)
return None
def _live_log_stats(log_path: Path | None) -> dict[str, object]:
if log_path is None:
return {}
try:
stat = log_path.stat()
except OSError:
return {"path": str(log_path), "exists": False}
modified_at = timezone.datetime.fromtimestamp(stat.st_mtime, tz=timezone.get_current_timezone())
return {
"path": str(log_path),
"exists": True,
"size_bytes": stat.st_size,
"modified_at": modified_at,
"seconds_since_modified": max(0, int((timezone.now() - modified_at).total_seconds())),
}
def _scan_snapshot_progress(data_path: Path, *, max_entries: int = 20000) -> dict[str, object]:
progress: dict[str, object] = {
"data_path": str(data_path),
"exists": data_path.exists(),
"files": 0,
"directories": 0,
"apparent_size_bytes": 0,
"scan_limited": False,
}
if not data_path.exists():
return progress
entries_seen = 0
for root, dirnames, filenames in os.walk(data_path):
progress["directories"] = int(progress["directories"]) + len(dirnames)
entries_seen += len(dirnames)
for filename in filenames:
file_path = Path(root) / filename
try:
file_stat = file_path.lstat()
except OSError:
continue
progress["files"] = int(progress["files"]) + 1
progress["apparent_size_bytes"] = int(progress["apparent_size_bytes"]) + int(file_stat.st_size)
entries_seen += 1
if entries_seen >= max_entries:
progress["scan_limited"] = True
return progress
if entries_seen >= max_entries:
progress["scan_limited"] = True
return progress
return progress
def _dry_run_summary( def _dry_run_summary(
*, *,
result: dict, result: dict,

View File

@@ -48,6 +48,7 @@ TEMPLATES = [
"django.template.context_processors.request", "django.template.context_processors.request",
"django.contrib.auth.context_processors.auth", "django.contrib.auth.context_processors.auth",
"django.contrib.messages.context_processors.messages", "django.contrib.messages.context_processors.messages",
"pobsync_backend.context_processors.pobsync_access",
], ],
}, },
}, },
@@ -55,6 +56,8 @@ TEMPLATES = [
WSGI_APPLICATION = "pobsync_server.wsgi.application" WSGI_APPLICATION = "pobsync_server.wsgi.application"
LOGIN_URL = "/admin/login/"
def _database_config() -> dict[str, object]: def _database_config() -> dict[str, object]:
engine = os.getenv("POBSYNC_DB_ENGINE", "sqlite").strip().lower() engine = os.getenv("POBSYNC_DB_ENGINE", "sqlite").strip().lower()
@@ -102,3 +105,7 @@ POBSYNC_BACKUP_ROOT = os.getenv("POBSYNC_BACKUP_ROOT", "/backups")
POBSYNC_ENV_FILE = os.getenv("POBSYNC_ENV_FILE", "/etc/pobsync/pobsync.env") POBSYNC_ENV_FILE = os.getenv("POBSYNC_ENV_FILE", "/etc/pobsync/pobsync.env")
POBSYNC_SERVICE_USER = os.getenv("POBSYNC_SERVICE_USER", "pobsync") POBSYNC_SERVICE_USER = os.getenv("POBSYNC_SERVICE_USER", "pobsync")
POBSYNC_SERVICE_GROUP = os.getenv("POBSYNC_SERVICE_GROUP", "pobsync") POBSYNC_SERVICE_GROUP = os.getenv("POBSYNC_SERVICE_GROUP", "pobsync")
POBSYNC_UPDATE_RELEASES_URL = os.getenv("POBSYNC_UPDATE_RELEASES_URL", "")
POBSYNC_UPDATE_RELEASES_TOKEN = os.getenv("POBSYNC_UPDATE_RELEASES_TOKEN", "")
POBSYNC_UPDATE_GIT_REMOTE = os.getenv("POBSYNC_UPDATE_GIT_REMOTE", "origin")
POBSYNC_UPDATE_COMMAND = os.getenv("POBSYNC_UPDATE_COMMAND", "sudo -n scripts/update-systemd")

View File

@@ -8,9 +8,15 @@ from pobsync_backend import api, views
urlpatterns = [ urlpatterns = [
path("", views.dashboard, name="dashboard"), path("", views.dashboard, name="dashboard"),
path("dashboard/priority-live/", views.dashboard_priority_live, name="dashboard_priority_live"),
path("dashboard/hosts-live/", views.dashboard_hosts_live, name="dashboard_hosts_live"),
path("changelog/", views.changelog, name="changelog"), path("changelog/", views.changelog, name="changelog"),
path("self-check/", views.self_check, name="self_check"), path("self-check/", views.self_check, name="self_check"),
path("logs/", views.logs, name="logs"), path("logs/", views.logs, name="logs"),
path("updater/", views.updater, name="updater"),
path("notifications/", views.notification_targets, name="notification_targets"),
path("notifications/new/", views.create_notification_target, name="create_notification_target"),
path("notifications/<int:target_id>/", views.edit_notification_target, name="edit_notification_target"),
path("purged-snapshots/", views.purged_snapshots, name="purged_snapshots"), path("purged-snapshots/", views.purged_snapshots, name="purged_snapshots"),
path("schedules/", views.schedules_list, name="schedules_list"), path("schedules/", views.schedules_list, name="schedules_list"),
path("config/global/", views.edit_global_config, name="edit_global_config"), path("config/global/", views.edit_global_config, name="edit_global_config"),
@@ -19,8 +25,10 @@ urlpatterns = [
path("ssh-credentials/generate/", views.generate_ssh_credential, name="generate_ssh_credential"), path("ssh-credentials/generate/", views.generate_ssh_credential, name="generate_ssh_credential"),
path("ssh-credentials/<int:credential_id>/", views.edit_ssh_credential, name="edit_ssh_credential"), path("ssh-credentials/<int:credential_id>/", views.edit_ssh_credential, name="edit_ssh_credential"),
path("ssh-credentials/<int:credential_id>/delete/", views.delete_ssh_credential, name="delete_ssh_credential"), path("ssh-credentials/<int:credential_id>/delete/", views.delete_ssh_credential, name="delete_ssh_credential"),
path("hosts/", views.hosts_list, name="hosts_list"),
path("hosts/new/", views.create_host_config, name="create_host_config"), path("hosts/new/", views.create_host_config, name="create_host_config"),
path("hosts/<str:host>/", views.host_detail, name="host_detail"), path("hosts/<str:host>/", views.host_detail, name="host_detail"),
path("hosts/<str:host>/state/", views.update_host_state, name="update_host_state"),
path("hosts/<str:host>/config/", views.edit_host_config, name="edit_host_config"), path("hosts/<str:host>/config/", views.edit_host_config, name="edit_host_config"),
path("hosts/<str:host>/prepare-directories/", views.prepare_host_directories, name="prepare_host_directories"), path("hosts/<str:host>/prepare-directories/", views.prepare_host_directories, name="prepare_host_directories"),
path("hosts/<str:host>/scan-known-key/", views.scan_host_known_key, name="scan_host_known_key"), path("hosts/<str:host>/scan-known-key/", views.scan_host_known_key, name="scan_host_known_key"),
@@ -37,6 +45,7 @@ urlpatterns = [
path("hosts/<str:host>/schedule/", views.edit_host_schedule, name="edit_host_schedule"), path("hosts/<str:host>/schedule/", views.edit_host_schedule, name="edit_host_schedule"),
path("runs/", views.runs_list, name="runs_list"), path("runs/", views.runs_list, name="runs_list"),
path("runs/<int:run_id>/", views.run_detail, name="run_detail"), path("runs/<int:run_id>/", views.run_detail, name="run_detail"),
path("runs/<int:run_id>/live/", views.run_detail_live, name="run_detail_live"),
path("runs/<int:run_id>/rsync-log/", views.run_rsync_log, name="run_rsync_log"), path("runs/<int:run_id>/rsync-log/", views.run_rsync_log, name="run_rsync_log"),
path("runs/<int:run_id>/cancel/", views.cancel_run, name="cancel_run"), path("runs/<int:run_id>/cancel/", views.cancel_run, name="cancel_run"),
path("runs/<int:run_id>/resolve-review/", views.resolve_run_review, name="resolve_run_review"), path("runs/<int:run_id>/resolve-review/", views.resolve_run_review, name="resolve_run_review"),