147 Commits

Author SHA1 Message Date
833edb2466 Merge pull request '(release) Prepare pobsync 1.1.0' (#44) from release/1.1.0 into master
Reviewed-on: #44
2026-05-21 15:24:55 +02:00
c7e9e69345 (release) Prepare pobsync 1.1.0
Bump the package and runtime version to 1.1.0, add release notes for
the UI-focused control panel update, and refresh version assertions for
the CLI entrypoint.

Refs 1.1
2026-05-21 15:23:50 +02:00
e79d871f36 Merge pull request 'issue-36-lightweight-live-refresh' (#43) from issue-36-lightweight-live-refresh into master
Reviewed-on: #43
2026-05-21 15:18:46 +02:00
ad45fbe46e (feature) Add live refresh for dashboard status panels
Split dashboard priority and host status sections into server-rendered
partials and wire them into the shared refresh hook so operational state
updates without a full page reload.

Refs #36
2026-05-21 15:17:11 +02:00
3cac7b61ac (feature) Add live refresh for run detail status
Add a server-rendered run detail partial and a small vanilla JavaScript
refresh hook so active backup runs update status, controls, timing, and
rsync log output without a full page reload.

Document the Django-template-first refresh pattern for future control
panel work.

Refs #36
2026-05-21 15:10:37 +02:00
1d6c21764b Merge pull request 'issue-37-clean-primary-navigation' (#42) from issue-37-clean-primary-navigation into master
Reviewed-on: #42
2026-05-21 15:02:02 +02:00
6f392bef65 (ui) Highlight current navigation section
Mark the active primary or system navigation link with aria-current and a
subtle visual state so staff users can see where they are in the control
panel without making Admin a primary app route.

Refs #37
2026-05-21 15:01:14 +02:00
6035c547ae (ui) Split primary and system navigation
Move operator routes into a primary navigation group and demote admin,
self-check, changelog, and status API links into a secondary system group
so the header reflects the control panel workflow more clearly.

Refs #37
2026-05-21 14:56:30 +02:00
a3a8fea071 Merge pull request 'issue-38-dashboard-responsive-layout' (#41) from issue-38-dashboard-responsive-layout into master
Reviewed-on: #41
2026-05-21 14:51:34 +02:00
0e2f48ab65 (ui) Remove dashboard panel overflow traps
Let dashboard-specific panels size naturally instead of inheriting generic
panel overflow behavior, and tighten activity and host card sizing so long
content wraps without creating internal scrollbars.

Refs #38
2026-05-21 14:50:05 +02:00
b55950e24a (ui) Add dashboard section responsive hooks
Give dashboard summary, trends, and host sections dedicated layout hooks
and tighten their responsive behavior so metrics and host cards remain
readable on narrower screens.

Refs #38
2026-05-21 14:46:23 +02:00
025cd0336c (ui) Harden dashboard responsive layout
Rework the dashboard priority grid to avoid cramped four-column layouts,
prevent stretched empty panels, and make long host and snapshot text wrap
safely across dashboard cards.

Refs #38
2026-05-21 14:43:24 +02:00
4c76ae9f52 Merge pull request 'Polish forms and action flows' (#35) from issue-25-forms-action-flows into master
Reviewed-on: #35
2026-05-21 14:28:20 +02:00
7a552715fe (ui) Clarify run action flows
Move run cancellation and review actions out of the page header into
dedicated action panels with clearer operator copy and consistent form
button styling.

Refs #25
2026-05-21 14:25:26 +02:00
0f0de5dc30 (ui) Standardize list filter actions
Give run, snapshot, schedule, purged snapshot, and log filters the same
responsive form layout with consistent Apply/Clear actions.

Refs #25
2026-05-21 14:22:11 +02:00
1604f0f6f4 (ui) Clarify destructive action flows
Make retention apply, incomplete cleanup, and SSH key deletion visibly
destructive with warning copy, danger styling, and consistent cancel actions
while keeping the existing confirmation requirements intact.

Refs #25
2026-05-21 14:17:07 +02:00
af548f11c4 (ui) Standardize primary form actions
Add shared form action styling and consistent Cancel links across config,
schedule, and SSH key forms so create/edit flows behave predictably.

Refs #25
2026-05-21 14:13:05 +02:00
c0eca3da55 Merge pull request 'issue-26-host-detail-control-page' (#34) from issue-26-host-detail-control-page into master
Reviewed-on: #34
2026-05-21 14:05:51 +02:00
212813e066 (ui) Make host diagnostics easier to scan
Replace the raw host check table with diagnostic cards and group effective
runtime config into operator-focused sections for backup target, connection,
and selection/retention details.

Refs #26
2026-05-21 14:03:43 +02:00
ab5291b8d3 (ui) Show host runs and snapshots as record cards
Replace the database-style Latest Runs and Snapshots tables on the host
detail page with scannable record cards and host-filtered View all links.

Refs #26
2026-05-21 13:59:16 +02:00
1929196287 (ui) Group host detail actions by context
Move host-level actions out of the page header and into the panels where
operators expect them: configuration, connection preflight, and snapshot
storage. This keeps the host control page calmer while preserving the same
actions.

Refs #26
2026-05-21 13:53:10 +02:00
9e75273fc5 (ui) Promote host detail operator controls
Add a first-screen host control workspace with status, backup actions,
schedule state, and current activity so the host detail page behaves as the
primary operator page instead of starting with raw configuration blocks.

Refs #26
2026-05-21 13:40:37 +02:00
5dd6ebb3db Merge pull request 'issue-27-dashboard-information-architecture' (#33) from issue-27-dashboard-information-architecture into master
Reviewed-on: #33
2026-05-21 13:29:18 +02:00
864a40e862 (ui) Surface storage pressure in dashboard priorities
Move backup root usage, runway, daily new data, and available capacity into
the top dashboard priority area so storage risk is visible before deeper trend
details.

Refs #27
2026-05-21 13:27:39 +02:00
9412feaa58 (ui) Rework dashboard around operator priorities
Move required actions, upcoming scheduled work, and recent run activity to the
top of the dashboard so the first screen answers what needs attention next.
Keep summary metrics, trends, and host cards as supporting drill-down content.

Refs #27
2026-05-21 13:21:09 +02:00
0fe2aa439f Merge pull request '(ui) Add review actions to filtered run lists' (#32) from issue-22-operational-review-actions into master
Reviewed-on: #32
2026-05-21 13:12:20 +02:00
fe4ae9d147 (ui) Add review actions to filtered run lists
Add inline Mark reviewed actions for failed and warning runs on the run list,
preserving active filters after review so Operational Status drill-downs can
be cleared without opening every run detail page.

Refs #22
2026-05-21 13:07:45 +02:00
0a3a3448d6 Merge pull request 'Make dashboard cards link to operational lists' (#31) from issue-23-dashboard-list-pages into master
Reviewed-on: #31
2026-05-21 12:49:15 +02:00
01b779c862 (ui) Add schedule overview for dashboard drill-down
Add a staff-only schedules page with filters for host, enabled state, and
prune state, including next run and last scheduler state.

Wire the dashboard Schedules metric to the new overview so all primary
dashboard count cards have useful destinations.

Refs #23
2026-05-21 12:39:57 +02:00
67d1af0baa (ui) Make dashboard operational status actionable
Turn the dashboard operational status rows into direct links to filtered run
lists, so failed, warning, running, and queued states can be investigated from
the first screen.

Also move the hosts anchor back to the actual Hosts section.

Refs #23
2026-05-21 12:00:06 +02:00
4e8e4f75fd (ui) Add dashboard-linked run and snapshot lists
Add staff-only list pages for backup runs and snapshots with practical
filters, then wire the dashboard summary cards and latest-runs panel to
those overviews.

This gives the dashboard real drill-down paths for run and snapshot counts
instead of leaving the data only partially visible on the first screen.

Refs #23
2026-05-21 11:52:35 +02:00
2be2d11b4a Merge pull request 'Create cohesive control panel redesign' (#30) from issue-28-control-panel-redesign into master
Reviewed-on: #30
2026-05-21 11:44:28 +02:00
b67ae7ff8b (ui) Extend page headers across utility views
Apply the shared page-header pattern to configuration, access,
operations, retention, log, and changelog pages so the control panel
uses one consistent title, context, and action structure.

Add representative view assertions for the new page context on utility
pages.

Refs #28
2026-05-21 11:42:01 +02:00
ad2cc5585e (ui) Add consistent page headers to key views
Introduce a shared page-header pattern with kicker, title, subtitle, and
actions, then apply it to the dashboard, host detail, run detail, snapshot
detail, and retention plan pages.

Scope the global app header styles to avoid leaking sticky navigation styles
onto page-level headers, and add view assertions for the new page context.

Refs #28
2026-05-21 11:37:25 +02:00
8aa3f1d1f5 (ui) Establish cohesive control panel styling
Refresh the shared base styling so the Django control panel has a calmer,
more polished production-tool feel across all pages. Update typography,
navigation, panels, metrics, host cards, tables, forms, buttons, messages,
focus states, and responsive behavior through reusable CSS variables and
component styles.

Refs #28
2026-05-21 11:31:49 +02:00
30cf93df27 Merge pull request 'Remove legacy-facing UI labels' (#29)
Reviewed-on: #29
2026-05-21 11:19:48 +02:00
01c4ccb316 (ui) Remove legacy-facing labels from operator pages
Replace refactor-era wording such as Source, Source root, SQL records,
database, runtime, and Django generation labels with operator-facing copy
around backup source, tracking records, changelog files, active config, and
pobsync-managed SSH keys.

Add view assertions so the old source/SQL labels do not quietly return.

Refs #24
2026-05-21 11:13:10 +02:00
00d4f2a70b Merge pull request 'release-hardening_1.0' (#21) from issue-8-release-hardening into master
Reviewed-on: #21
2026-05-21 03:56:24 +02:00
f8215a0c9a (release) Update 1.0 changelog for hardening work
Bring the 1.0.0 release notes up to date with the release-hardening work
completed after the initial metadata pass: worker heartbeat tracking,
incomplete snapshot cleanup, review resolution, SSH key management hardening,
purged snapshot audit history, and the in-app changelog page.

Refs #8
Refs #10
Refs #11
Refs #16
Refs #19
Refs #20
2026-05-21 03:51:27 +02:00
ea9e3e41e3 (release) Add purged snapshot audit overview
Record snapshot purge history whenever retention or incomplete cleanup removes
snapshot directories and SQL records. Store the purge reason, original kind,
path, action source, and triggering operator so manual, scheduled, CLI, and
incomplete cleanup actions remain auditable after the original snapshot record
is deleted.

Add a staff-only Purged Snapshots page with host/action filters and register
the audit model in Django admin.

Refs #16
Refs #8
2026-05-21 03:46:38 +02:00
5b5a5bc637 (release) Harden SSH key edit and delete flow
Make SSH credential management more explicit by adding an edit action in the
key overview and requiring name confirmation before deletion. Keep deletion
blocked while a key is still selected by hosts or global config, and cover
rename, delete confirmation, and in-use protection in view tests.

Refs #20
Refs #8
2026-05-21 03:38:55 +02:00
c2e5a534aa (release) Add review resolution for operational tasks
Add reviewed state for failed/warning runs and incomplete snapshot records,
then use it to clear dashboard and host “need review” tasks after an operator
has acknowledged them.

Expose Mark reviewed actions on run detail and host retention warnings, keep
reviewed records available for audit/debug, and exclude reviewed problem runs
from operational counts and latest issue summaries.

Refs #19
Refs #8
2026-05-21 03:34:41 +02:00
d0c23deb72 (release) Add explicit incomplete snapshot cleanup
Add a dedicated cleanup path for incomplete snapshots instead of letting
retention prune them implicitly. The retention plan now exposes a guarded
form that requires host and delete-count confirmation before removing
.incomplete snapshot directories and their SQL records.

Keep scheduled/manual retention behavior unchanged, add path safety checks,
and cover cleanup success, confirmation failures, max-delete limits, and
unexpected paths in tests.

Refs #10
2026-05-21 03:26:21 +02:00
4c8ed24561 (release) Track worker heartbeat for running jobs
Record worker pid, host, claim time, and heartbeat metadata on running
backup jobs so operators can see which worker owns a run.

Refresh the heartbeat while rsync is active and reconcile stale running
runs when the worker heartbeat stops. Add a worker option to tune or
disable stale-run reconciliation.

Refs #11
2026-05-21 03:16:38 +02:00
404b7f7500 (release) Add Django changelog page
Expose the repository CHANGELOG.md through a staff-only Django view and
link it from the main navigation.

Render a small safe subset of Markdown without adding a runtime dependency,
copy the changelog into the Docker image, and cover the page with view tests.
2026-05-21 03:10:31 +02:00
beca073ddc (release) Prepare 1.0.0 release metadata
Add the initial 1.0.0 changelog, bump the package/application version,
and expose the release version through `pobsync --version`.

Cover the version output in the console entrypoint tests.
2026-05-21 03:04:59 +02:00
362a9dde62 Merge pull request 'issue-7-config-cleanup-legacy-removal' (#18) from issue-7-config-cleanup-legacy-removal into master
Reviewed-on: #18
2026-05-21 02:57:38 +02:00
a73d34ac9f (refactor) Use operator-facing config errors
Replace remaining model-name based configuration errors with labels that
match the Django-first operating model.

Add coverage for missing global config and host configuration errors so
operator-facing messages stay readable.
2026-05-21 02:56:00 +02:00
1c8cbd96ca (refactor) Normalize maintainer command labels
Prefer --schedule-expression for scripted schedule updates while keeping
--cron as a compatibility alias.

Clean up management command help, errors, and output so operator-facing
text talks about hosts, global config, and Django backup configuration
instead of model names or old SQL-backed pobsync wording.
2026-05-21 02:52:42 +02:00
86873bd035 (refactor) Remove obsolete global config JSON storage
Drop the unused GlobalConfig.data field and remove the remaining YAML
config path helpers from PobsyncPaths.

Keep HostConfig.config as runtime state for preflight data, and relabel it
in the admin so it no longer reads as legacy compatibility storage.
2026-05-21 02:46:09 +02:00
2642f14e49 (refactor) Remove pobsync_home from global config
Drop the obsolete pobsync_home field from GlobalConfig and remove it from
runtime config generation, form saves, and configuration commands.

The runtime state root now comes exclusively from POBSYNC_HOME/settings,
which keeps the Django model focused on backup behavior instead of install
layout.
2026-05-21 02:41:02 +02:00
bb62382e18 (refactor) Remove YAML config import and export path
Drop the pre-Django YAML import/export management commands and remove the
file-based config loader fallback from the backup and retention engines.

Keep the runtime config bridge backed by Django models, and add tests that
ensure engine operations require an explicit Django config source.
2026-05-21 02:34:09 +02:00
c5865a5379 (refactor) Normalize runtime config labels
Hide the old pobsync_home field from the Django admin and replace legacy
operator-facing labels with runtime state root and backup root terminology.

Rename admin compatibility fieldsets, update self-check/config-check text,
and refresh management command help so Django/systemd stays the primary
mental model.
2026-05-21 02:24:55 +02:00
58d567f9bc (refactor) Retire configuration command aliases
Remove the short pobsync aliases for global config, host config, and
schedule changes so the public CLI no longer points operators toward the
old configuration workflow.

Keep operational aliases for backup, discovery, retention, worker, and
scheduler debugging, and document explicit Django management commands for
automation use.
2026-05-21 02:14:16 +02:00
2d9f453767 Merge pull request 'issue-6-restore-story' (#17) from issue-6-restore-story into master
Reviewed-on: #17
2026-05-21 02:06:22 +02:00
20a9f93378 (docs) Add targeted restore examples
Extend the restore guidance with directory and single-file dry-run
examples so operators can restore a focused path without copying an
entire snapshot.

Render the examples on snapshot detail pages using the selected
snapshot's data path and the host-specific staging destination.
2026-05-21 02:05:19 +02:00
b78f102e9d (docs) Add manual restore guidance for snapshots
Document the manual restore workflow in the README and surface snapshot-
specific restore commands on the snapshot detail page.

The guidance keeps restores intentionally manual for now: inspect the
snapshot data directory, run rsync with --dry-run, restore to staging
first, and treat hardlinked snapshot files as read-only.
2026-05-21 02:01:40 +02:00
8858e049ee Merge pull request '(ui) Add dashboard operational status summary' (#15) from issue-5-dashboard-1-0 into master
Reviewed-on: #15

This closes issue #5
2026-05-21 01:54:48 +02:00
a75b97c4c0 (ui) Add dashboard operational status summary
Make queued, running, warning, and failed run states more visible at the
top of the dashboard with contextual status summaries and highlighted
summary metrics.

Also show an all-clear message when configured hosts have no active or
problematic runs.
2026-05-21 01:51:13 +02:00
b4fc5a14b2 (ui) Clarify dashboard backup growth trends
Replace the dashboard trend metric grid with an operational summary that
explains storage usage, runway, average new data, link-dest savings, and
average duration in a more readable way.

Also add an empty state for fresh installs before completed backup stats
exist.
2026-05-21 01:46:49 +02:00
a0fd33fcb8 (ui) Improve dashboard host card scanability
Add per-host status chips for queued, running, warning, and failed runs so
the dashboard shows operational pressure without needing to open each host.

Restructure host cards into clearer backup activity and snapshot health
sections, with less visual clutter and better mobile wrapping.
2026-05-21 01:41:45 +02:00
ef1761385e (ui) Separate healthy and problematic dashboard runs
Split dashboard host cards into last successful backup and latest warning
or failed run so operators can quickly see whether a host is protected even
when recent activity produced an issue.

Also add queued and warning run counts to the dashboard summary metrics.
2026-05-21 01:34:38 +02:00
17215fd191 Merge pull request '(feature) Improve retention UX and prune safety' (#14) from issue-4-retention-ux-and-safety into master
Reviewed-on: #14
2026-05-21 01:27:19 +02:00
97753c3d3c (ui) Show retention apply details on run detail
Record planned delete counts, max-delete settings, base protection, and
ignored incomplete snapshots in retention apply results.

Surface those details on run detail pages so scheduled and manual prune
outcomes are understandable without reading the raw JSON payload.
2026-05-21 01:25:40 +02:00
994f7f66c4 (bugfix) Preserve scheduled backup warning status
Update the scheduler to reflect the actual scheduled BackupRun status after
a run completes, so prune warnings are shown as schedule warnings instead
of being reported as successful schedule executions.
2026-05-21 01:22:06 +02:00
f76b6cad14 (feature) Require delete count confirmation for retention apply
Make manual retention application more explicit by requiring operators to
confirm both the host name and the current number of planned deletions.

This reduces the risk of applying a stale or misunderstood retention plan
when the delete set changes between review and confirmation.
2026-05-21 01:19:08 +02:00
90e293facd (ui) Surface retention warnings on run detail
Show host-level retention warnings on run detail pages so successful or
warning runs still expose scheduled prune limit issues and incomplete
snapshots that need operator attention.
2026-05-21 01:13:44 +02:00
50eb7cf2f3 (ui) Make retention planning warnings explicit
Show keep/delete reasons in the retention plan, surface scheduled prune
limit warnings, and explain base snapshot protection before retention is
applied.

Also surface incomplete snapshots from the retention views without deleting
them automatically, so interrupted backups are visible on the dashboard,
host detail, and retention plan.
2026-05-21 01:10:45 +02:00
26265be440 Merge pull request '(feature) Add backup safety and preflight validation' (#13) from issue-3-backup-safety-preflight-validation into master
Reviewed-on: #13
2026-05-21 00:58:23 +02:00
5faef1492d (ui) Add readable dry-run summaries
Surface dry-run status, transfer estimates, file counts, warnings, and the full
rsync log link directly on the run detail page.

Keep raw rsync output and JSON available, but make the common review path easier
to scan before starting a real backup.
2026-05-21 00:55:19 +02:00
3045093dcf (feature) Add remote host connection preflight
Add an on-demand host preflight action that verifies SSH reachability,
remote rsync availability, and remote source root access.

Persist the latest preflight result on the host config, render it in Django,
and block real backups when the last remote preflight failed.
2026-05-21 00:50:05 +02:00
64a0ff8322 (feature) Add host backup preflight gates
Introduce a host preflight layer that separates dry-run blockers from real backup blockers.
Show the effective per-host backup configuration in Django before queueing a run.

Block real backup queueing when failed host checks remain, while still allowing dry-runs
when only local storage preparation is missing.
2026-05-21 00:41:45 +02:00
155ff63a73 Merge pull request '(feature) Improve run debugging and log filtering' (#12) from issue-2-run-detail-logging-polish into master
Reviewed-on: #12
2026-05-21 00:26:45 +02:00
a9e40df44b (feature) Add focused filtering to the service logs view
Extend the Django logs view with filters for service unit, severity, time
window, host, run id, and message text. Pass severity and time window directly
to journalctl, then apply host/run/message filtering to the returned pobsync
journal lines.

This makes failed or slow backups easier to investigate from the control panel
without needing shell access.
2026-05-21 00:24:07 +02:00
98695f9888 (ui) Surface run debugging details in Django
Restructure the run detail page into clearer sections for summary, failure
classification, requested options, rsync command, rsync log output, stats,
retention, and raw result data.

Show recent rsync log output inline with a link to the full log, and promote
failure and retention warning details out of the JSON payload so failed or slow
runs are easier to debug from the control panel.
2026-05-21 00:17:39 +02:00
d8c0ee5d1e Merge pull request '(ops) Complete native production install and update flow' (#9) from issue-1-production-install-update into master
Reviewed-on: #9
2026-05-20 01:50:23 +02:00
851f967f12 (ops) Expand native install self checks and recovery docs
Extend the runtime self check with native install diagnostics for the
environment file, service user, backup root ownership, and SQLite database
path. Export install metadata from the systemd units and pobsync-manage wrapper
so custom env files and service users are visible to Django checks.

Document restart, journal log inspection, and rollback steps in the README so
production updates have a clear recovery path.
2026-05-20 01:44:51 +02:00
c97c595253 (ops) Add terminal self-check command for native installs
Add check_pobsync_install so native deployments can run the same runtime
diagnostics from the terminal that are available in the Django Self Check view.

The command prints every check with status, returns a failing exit code when
install-critical checks fail, supports fail-on-warning for stricter automation,
and is documented in the installer output and README update flow.
2026-05-20 01:37:07 +02:00
f5acdf2fff (refactor) Remove obsolete source-copy deploy script
Delete the old scripts/deploy path that installed pobsync into /opt/pobsync/lib
with a standalone /opt/pobsync/bin wrapper.

Production deployment is now owned by the native systemd installer and updater,
so keeping the legacy deploy script would make the supported install story less
clear and easier to misuse.
2026-05-20 01:33:07 +02:00
e6ed7954de (ops) Add native update wrapper for production deploys
Add scripts/update-systemd as a safer routine deploy entrypoint for native
systemd installations. The wrapper keeps updates non-interactive, preserves the
existing environment file, skips OS package installation, and avoids superuser
creation prompts while still reusing the installer refresh flow.

Document the update path in the README and capture the maintenance expectation
in the development notes.
2026-05-20 01:30:02 +02:00
73e6bb7285 (ops) Install pobsync-manage for native management commands
Add a pobsync-manage wrapper that loads the native environment file before
running Django management commands, so production commands use the same
database and runtime settings as the systemd services.

Install the wrapper from the systemd installer, use it for migrations,
static collection, SSH key setup, and superuser creation, and document it
in the README for operational commands.
2026-05-20 01:27:08 +02:00
d451d01fe2 (docs) Move 1.0 roadmap tracking to Gitea
Create the 1.0 milestone and roadmap issues in Gitea, making Gitea the
leading tracker for release work.

Remove the repository roadmap document and README reference now that the
roadmap is tracked as milestone issues.
2026-05-20 01:17:35 +02:00
bfe17969e6 (docs) Add pobsync 1.0 roadmap
Document the remaining 1.0 work as ordered, reviewable chunks with
checkboxes for install/update flow, logging, preflight validation,
retention UX, dashboard polish, restore docs, config cleanup, and release
hardening.

Link the roadmap from the README and note how it can later be mirrored into
Gitea milestones and issues.
2026-05-20 01:07:17 +02:00
7cd87bc8a8 (ui) Remove global config count from dashboard summary
Drop the Global Configs metric from the dashboard because pobsync only uses
one default global configuration.

Keep global config management available through the existing dashboard
action and setup prompt.
2026-05-20 00:29:43 +02:00
0babc57f57 (feature) Link rsync logs from backup run detail
Record the final rsync log path for successful real backup runs, matching
the existing dry-run and failure result payloads.

Add a staff-only run log endpoint and surface the link on run detail pages,
including fallback log discovery for older runs based on snapshot_path.

Cover direct log links and inferred scheduled backup logs with view tests.
2026-05-20 00:09:59 +02:00
f41e59e695 (ui) Relax dashboard host card layout
Split each dashboard host card into a timeline area and a compact stats area
so snapshot, run, and schedule details have more room to scan.

Keep the same operational information visible while reducing cramped
column-like wrapping on desktop and mobile layouts.
2026-05-19 23:52:54 +02:00
f3dcb8a3b9 (ui) Replace dashboard host table with host cards
Restructure the dashboard Hosts section into a card per host so snapshot,
run, schedule, retention, and data metrics have room to breathe.

Add reusable host-card styling and keep the same links and operational data
available without forcing everything into a wide table.
2026-05-19 23:46:52 +02:00
c2d7342a47 (refactor) Remove unused schedule user field
Drop the legacy schedule user setting from the Django model, form, defaults,
and configure command.

Schedules are executed by the pobsync scheduler service under the configured
systemd service user, while remote SSH login users are configured separately
on global or host backup config.

Add a migration to remove the unused database column and update schedule
view tests around the simplified form.
2026-05-19 23:41:55 +02:00
5ca2733ea9 (bugfix) Prune snapshots with preserved read-only permissions
Make SQL retention delete the snapshot root when records point at the
snapshot data directory, matching how backup metadata is stored on disk.

Before removing a snapshot tree, temporarily add user write permission to
directories inside that snapshot so rsync-preserved source permissions do
not block cleanup.

Add a regression test for pruning snapshots whose data directory mirrors a
read-only remote root.
2026-05-19 23:29:36 +02:00
8bff241b12 (bugfix) Mark retention failures as backup warnings
Add a warning status for BackupRun records so successful snapshots are not
reported as failed when post-run SQL retention fails.

Keep the prune error in the run result, link the successful snapshot, and
let the management command complete with a warning instead of raising a
backup failure.

Include warning runs in backup trend summaries and add a regression test
for successful backups with failed retention cleanup.
2026-05-19 23:20:52 +02:00
1e04da9de8 (bugfix) Preserve existing schedule values in the edit form
Only apply default schedule initial values when creating a new schedule.

Avoid passing default initial data while editing an existing ScheduleConfig,
so the form renders the active cron-style expression, user, and retention
settings from the database.

Add a regression test that reopens an existing schedule and verifies the
stored values are shown instead of defaults.
2026-05-19 23:13:53 +02:00
39b6cf3469 (feature) Add scheduler timezone setup to the native installer
Prompt for the scheduler timezone during interactive installs and support
a --time-zone flag for scripted installs.

Detect a sensible default from POBSYNC_TIME_ZONE, timedatectl, or
/etc/timezone before falling back to UTC, and validate the value with
Python zoneinfo before writing it to pobsync.env.

Document that schedules are evaluated in POBSYNC_TIME_ZONE so production
installs can avoid UTC/local-time ambiguity.
2026-05-19 23:09:15 +02:00
124421b85c (bugfix) Show latest successful runs without requiring stats
Separate operational latest-run display from trend-stat collection so
successful backups without parsed stats still appear in dashboard host rows.

Keep trend summaries limited to runs with stats, but use all successful
real runs for the host latest-run indicator.

Render next scheduled run times with an explicit timezone label to avoid
ambiguity between UTC and local scheduler time.
2026-05-19 23:05:22 +02:00
86eee0f916 (feature) Show next scheduled run and backup run type in the UI
Add a scheduler helper that calculates the next due time for a cron-style
schedule expression and surface that value on the dashboard and host detail
pages.

Show the latest run type in host summaries and backup trend tables so
manual and scheduled backups are distinguishable in the Django UI.

Keep the calculation derived from existing ScheduleConfig data without
adding a migration.
2026-05-19 22:57:58 +02:00
9624fb469f (refactor) Clarify scheduler terminology in the Django UI
Rename the schedule form label from "Cron expression" to "Schedule
expression" and explain that cron-style timing is evaluated by the
pobsync scheduler service rather than host cron.

Update host detail and schedule form copy so operators can see that
schedules are SQL-backed and dispatched by pobsync itself.
2026-05-19 22:48:00 +02:00
e8169eae42 (feature) Add backup trend forecasting to the Django UI
Extend derived backup statistics with average daily new data and an
estimated days-until-full forecast based on recent successful real runs.

Show the new forecast metrics on the dashboard and add compact per-run
trend bars on the host detail page so new data and matched link-dest data
are easier to compare at a glance.

Keep the implementation migration-free by deriving everything from the
existing BackupRun result stats payload.
2026-05-19 22:39:46 +02:00
fc22842fc4 (feature) Summarize backup trends in the Django UI
Add a stats summary layer that aggregates recent successful real backup runs
into dashboard and host-level trend metrics.

Show backup-root usage, available space, average new data, average duration,
estimated runs until full, and link-dest savings on the dashboard. Add a host
trend table with recent run duration, file count, new data, matched data, and
snapshot links.

Keep the implementation based on existing run and snapshot stats JSON so the
UI gains useful trend visibility without introducing a schema migration yet.
2026-05-19 22:31:24 +02:00
6940dc55b7 (feature) Capture structured backup statistics
Parse rsync --stats output into structured run metrics for file counts,
transferred bytes, literal data, matched data, speedup, and estimated
link-dest savings.

Store collected stats on backup run results and successful snapshot metadata,
including snapshot data usage and backup-root capacity details for future
dashboard graphs and disk-full projections.

Render the collected metrics on run and snapshot detail pages, with tests
covering parsing, metadata persistence, and UI output.
2026-05-19 22:25:04 +02:00
728e5c740a (feature) Add optional verbose rsync output for manual backups
Expose a verbose rsync output option in the Django manual backup form and
store the selected value with the queued run request.

Propagate the option through the worker, direct management command, and
rsync command builder so real backups can emit itemized changes, file-list
progress, and stats when requested. Dry-runs continue to use verbose output
by default and report that consistently in requested options.

Cover the queue, worker, view, and rsync command behavior with focused
tests.
2026-05-19 22:13:33 +02:00
d52a9167d1 (bugfix) Reconcile failed dry-runs from rsync terminal logs
Classify rsync failures in run results so transport issues such as exit
255 and broken pipes show clearer diagnostic hints.

Teach the worker to reconcile running dry-runs when their log already
contains a terminal rsync error, and to fail stale dry-runs after their
timeout window. This prevents failed rsync processes from leaving runs
stuck in the running state indefinitely.
2026-05-19 21:10:08 +02:00
d67ba9cada (feature) Add managed rsync verbosity for dry-runs
Add default dry-run rsync output flags so long-running dry-runs expose
file-list, progress, stats, and itemized change information in their
run-specific log files.

Avoid duplicating user-supplied itemize or --info arguments so operators
can still tune rsync output from global or host configuration.
2026-05-19 20:57:29 +02:00
bbb0f652f3 (feature) Add cancellable backup runs and clearer dry-run logs
Add a cancel action for queued and running backup runs. Queued runs are
cancelled immediately, while running runs are marked for cancellation and
the worker terminates the active rsync process group.

Make dry-run log paths run-specific and add a defensive default dry-run
timeout so stuck dry-runs do not remain running indefinitely.

Remove rsync exit codes from run overview tables while keeping detailed
diagnostics available on the run detail payload.
2026-05-19 20:46:10 +02:00
088f43279e (feature) Add global and effective host config checks
Introduce reusable configuration checks for global settings and effective
host runtime configuration. The checks now surface risky backup settings
such as missing recursive rsync args, missing critical root excludes,
invalid SSH settings, missing credentials, and retention gaps.

Show these checks on the global config form, host edit form, and host
detail page so operators can validate the compounded host/global config
before starting real backup runs.
2026-05-19 20:24:29 +02:00
7e5d31d53b (bugfix) Guard dry-run logs and recursive rsync defaults
Clear the reused dry-run rsync log before each dry-run so run details
only show output from the current execution.

Populate new Django global configs with the existing safe rsync and
exclude defaults, including archive mode and standard pseudo-filesystem
exclusions.

Add a host check that fails when effective rsync args do not include
archive or recursive transfer, preventing real backups that only report
"skipping directory .".
2026-05-19 20:09:35 +02:00
8bd2a8ff1a (bugfix) Use service-level known_hosts for generated SSH keys
When a selected SSH credential has no pinned known_hosts entries, create
and use a pobsync service-level known_hosts file under POBSYNC_HOME/state.

Pass UserKnownHostsFile and StrictHostKeyChecking=accept-new to SSH so
unattended backups no longer depend on root's known_hosts or an
interactive shell session.

Keep pinned credential known_hosts behavior unchanged when entries are
configured explicitly.
2026-05-19 20:01:39 +02:00
d3ffca1843 (feature) Add host key scanning for SSH credentials
Add a host detail action that scans the target SSH host key with
ssh-keyscan and stores it on the selected SSH credential.

Merge scanned known_hosts entries without duplicates and let the
existing runtime config pass them through as UserKnownHostsFile for
unattended rsync over SSH.

Extend host checks to warn when the selected credential has no known_hosts
entries, making host key verification failures actionable from Django.
2026-05-19 19:55:40 +02:00
25d2a5b1a7 (bugfix) Surface rsync SSH failure details in run results
Include the selected SSH credential metadata and rsync log tail in
dry-run and failed backup results so Django shows the actual SSH or
rsync failure instead of only the exit code.

Warn in host checks when a host still uses database-stored private key
material, making it easier to spot old credentials after switching to
generated filesystem keys.
2026-05-19 19:49:33 +02:00
df3dcc47c9 (feature) Generate filesystem-backed SSH credentials
Add filesystem-backed SSH credentials for the native systemd deployment
path. Generated keys are stored below POBSYNC_HOME with 0600
permissions, while Django keeps the public key, fingerprint, path, and
selection metadata.

Add a Django SSH key generation view, delete action for unused generated
keys, and a management command used by the installer to ensure a default
backup key exists.

Update runtime config to use generated key paths directly as IdentityFile,
extend host checks to verify key readability, and keep legacy uploaded
keys available for compatibility.
2026-05-19 19:41:40 +02:00
ccacad3d37 (bugfix) Grant service user backup and journal access
Update the native installer so the pobsync service user gets journal
read access when the host exposes systemd-journal or adm groups.

Apply ownership and private directory modes to the configured backup
root, and reuse the existing environment backup root on reinstall so
production updates do not fall back to /backups.

Add a self-check for journal access and a host detail action that can
prepare missing backup directories for existing host configurations.
2026-05-19 19:25:05 +02:00
90f28410ce (feature) Add host doctor checks and Django log viewer
Add host-level checks for address, enabled state, SSH credential
selection, and backup directory readiness, and show them on the host
detail page.

Create host backup directories during host creation and prefill new
hosts from the default global config.

Add a staff-only logs view backed by journalctl with filtering by
pobsync unit, priority, and message text.

Improve runtime checks for gunicorn in virtualenv installs and ensure
the native installer grants the service user access to the backup root.
2026-05-19 19:11:57 +02:00
bb7907846e (bugfix) Restart systemd services during native updates
Change the native installer to enable services and then explicitly
restart web, worker, and scheduler so updated Django code is loaded
after each install or update.

Document that the installer refreshes app files and restarts systemd
services as part of the normal update flow.
2026-05-19 18:58:41 +02:00
96b91b2a69 (refactor) Quiet native installer output by default
Add step-based installer logging that reports pobsync actions as OK,
FAILED, or SKIPPED while suppressing noisy apt, pip, Django, and
systemd output during successful runs.

Show the captured output for the failed step when something breaks,
and add a --verbose flag to restore full command output for debugging.

Document the quieter installer behavior and verbose mode in the README
and development notes.
2026-05-19 18:52:31 +02:00
98d152da06 (bugfix) Add SSH private key file upload
Allow SSH credentials to be created from an uploaded private key file
as an alternative to pasting the key into a textarea.

Use multipart form handling in the credential views so server-side
keys can be imported without copy/paste wrapping or formatting damage.

Cover the upload path with a view test while keeping existing pasted
key validation behavior intact.
2026-05-19 18:48:17 +02:00
97797c574d (bugfix) Normalize pasted OpenSSH private keys
Canonicalize uploaded OpenSSH private keys before validation by
normalizing line endings, removing whitespace from the base64 body,
and re-wrapping it between the BEGIN and END markers.

Add SSH credential tests that generate a real ed25519 key, damage its
wrapping, and verify that validation succeeds after normalization.

Return a clearer validation error for PEM private keys, which are not
supported by the current credential flow.
2026-05-19 18:42:02 +02:00
c7cfb603b0 (bugfix) Improve SSH credential validation feedback
Normalize pasted private keys before validation and detect common SSH
credential mistakes, including public keys pasted into the private key
field and public keys that do not match the supplied private key.

Translate OpenSSH libcrypto parse failures into a clearer user-facing
message and disable browser spellcheck/autocomplete on SSH key fields.

Document the native update flow as git pull followed by the
non-interactive installer so deployments refresh cleanly.
2026-05-19 18:35:39 +02:00
38f946d1c4 (feature) Make the native installer interactive
Add an interactive setup flow to the systemd installer with defaults
for install paths, service identity, backup storage, bind address,
allowed hosts, CSRF origins, OS package installation, MariaDB support,
nginx setup, and first superuser creation.

Keep scripted installs supported through non-interactive mode, existing
overrides, environment variables, and explicit superuser flags.

Print a user-facing completion summary with the control panel URL,
Self Check reminder, first setup steps, and useful service log commands.
2026-05-19 18:22:18 +02:00
44d821c638 (docs) Reframe documentation around Django-first operations
Update the README to describe pobsync as a Django-first, SQL-backed
service with the control panel as the primary operational interface.

Move CLI examples out of the normal workflow and document them as
maintainer tooling for debugging, services, and migration tasks.
2026-05-19 18:17:43 +02:00
b1789d8621 (config) Install native runtime requirements and split docs
Teach the systemd installer to install required Debian/Ubuntu
packages by default, with an opt-out for users who manage system
dependencies themselves.

Add explicit MariaDB install-extra handling so native installs can
pull in both the Python extra and required client build packages.

Slim down the README to production setup and operations, and move
development, Docker, migration helper, and architecture notes into
docs/development.md.
2026-05-19 18:15:34 +02:00
372a857f15 (feature) Add full native installer and self-check page
Expand the systemd installer so it can perform a complete native
installation with sensible defaults: copy the checkout into the target
app directory, create runtime directories, write the environment file,
install dependencies, configure systemd units, and optionally configure
nginx.

Add a staff-only Django self-check page that verifies runtime settings,
required binaries, writable paths, database connectivity, global config
state, and systemd service status when available.

Document installer overrides and expose the self-check from the main
navigation.
2026-05-19 16:05:03 +02:00
b93e19a7c8 (refactor) Add native systemd production deployment
Make native systemd services the recommended production path for
pobsync while keeping Docker Compose available for development and
optional test installs.

Add web, worker, and scheduler systemd unit templates, a native
environment example, an optional nginx reverse proxy template, and an
installer that creates the venv, service user, env file, units, and
runs migrations/static collection.

Allow native deployments to configure POBSYNC_BACKUP_ROOT directly and
document the new production layout and update flow.
2026-05-19 15:59:07 +02:00
1297a839d4 (config) Harden Docker deployment for remote servers
Run the Django control panel with Gunicorn instead of the development
runserver and serve static files through WhiteNoise.

Add restart policies, healthchecks, .env-driven production settings, and
a sample .env file for single-server deployments. Update the Docker
entrypoint to collect static assets and document the remote server
deployment and update flow in the README.
2026-05-19 15:33:09 +02:00
c018011e83 (bugfix) Validate Django-managed SSH private keys
Validate uploaded SSH private keys with ssh-keygen before saving them so
invalid, malformed, or unsupported key material is rejected in the
control panel instead of failing later during rsync.

Auto-populate the public key when it is omitted, add an edit flow for
existing SSH credentials, and cover create, update, and invalid-key
paths with view tests.
2026-05-19 15:22:40 +02:00
e65537c6de (feature) Add Django-managed SSH credentials
Add SSH credentials as first-class Django data so backup keys can be
uploaded through the control panel instead of mounted into containers.

Credentials can be selected globally or overridden per host. At runtime
the selected key is materialized inside the container with restrictive
file permissions and injected into the rsync SSH command via IdentityFile.
Known hosts entries are handled the same way when configured.

Add control panel views for creating and listing SSH keys, expose the
fields in config forms and admin, document the workflow, and cover global
and host credential selection with tests.
2026-05-19 14:37:38 +02:00
91ce7ad4c5 (feature) Add host backup control actions
Turn the host detail page into a more useful operator surface for
starting greenfield backups from Django.

Add quick actions for dry-run and real backup runs, keep the advanced
manual options available, and show whether a host is ready, disabled, or
blocked by missing global config. Surface queued and running counts plus
a direct link to the active run.

Expose requested backup options on the run detail page and cover the new
control flow with view tests.
2026-05-19 14:25:28 +02:00
4fb33eca6c (feature) Add Django retention apply flow
Expose retention apply from the host retention plan page so planned
snapshot deletions can be executed from the Django UI.

The form requires explicit host confirmation, carries through the
selected retention kind and base-protection setting, and uses max_delete
as a deletion guard. The view delegates to the SQL retention apply
service and reports predictable pobsync errors back through Django
messages instead of surfacing a server error.

Add view coverage for confirmed deletion, invalid confirmation, and
POST-only enforcement.
2026-05-19 13:54:15 +02:00
83334803b9 (feature) show latest snapshot on the Django dashboard
Add latest snapshot context to each dashboard host row so imported legacy
snapshots are visible without opening every host page.

Link the latest snapshot directly to its detail page and show its kind
and status beside the host snapshot count.

Cover the dashboard latest-snapshot selection with a view test.
2026-05-19 13:44:28 +02:00
5c469f723a edit docker compose file 2026-05-19 13:33:57 +02:00
1d90454109 (feature) improve snapshot discovery visibility in Django
Add a discovery preflight that reports the configured backup root, host
root, and snapshot directory counts before importing anything.

Show discovery status on host detail pages so missing mounts or mismatched
host directories are visible from the UI.

Warn clearly when discovery scans zero snapshots, including whether the
host backup directory is missing or simply empty.
2026-05-19 13:21:31 +02:00
573177e118 (refactor) make Docker backup root static in Django setup
Remove backup_root from the normal Django global config form and display
the fixed container path /backups instead.

Always persist /backups from the setup form so Docker deployments do not
mix host paths with container paths.

Update tests and docs to clarify that the host backup directory is chosen
through the Docker mount, while Django always uses /backups internally.
2026-05-19 13:14:22 +02:00
3da877eb8a (feature) queue manual backups from the Django host page
Add a staff-only manual backup form to host detail pages with safe
dry-run defaults and optional retention settings.

Queue manual BackupRun records through the existing worker-backed runner
path instead of executing backups inside the web request.

Validate disabled hosts, missing global config, and invalid methods with
view tests covering the new UI flow.
2026-05-19 13:04:50 +02:00
fe8e65e12e (feature) add queued backup worker foundation
Move backup execution out of the management command into a reusable
backup runner service that can execute an existing BackupRun record.

Add queue primitives and a run_pobsync_worker command so manual backup
requests can be recorded as queued SQL state and processed outside the
web request path.

Add a worker Docker service and pobsync worker CLI alias, with tests for
queued run creation, worker execution, manual run typing, and command
mapping.
2026-05-19 13:00:12 +02:00
aea22597ba (bugfix) preserve saved global backup root in Django setup form
Fix the global config edit view so default initial values are only used
when creating a new config, preventing saved backup_root values from
being hidden by form defaults.

Keep pobsync_home as an internal runtime setting instead of exposing it
in the normal Django setup form.

Mount a host backup directory into Docker at /backups and document
POBSYNC_BACKUP_ROOT so backup_root behaves predictably in containers.
2026-05-19 12:48:32 +02:00
66e1f549b9 (feature) add Django detail views for backup runs and snapshots
Add staff-only run and snapshot detail pages so scheduler and command
output can be inspected from the Django UI.

Link dashboard and host detail tables to the new detail views, including
snapshot/base relationships and linked backup runs.

Render stored result and metadata JSON in readable form and cover the new
inspection views with tests.
2026-05-19 12:31:47 +02:00
6bcc15c174 (feature) add Django setup flow for initial pobsync configuration
Add staff-only UI routes for creating/editing the default GlobalConfig
and creating the first HostConfig from the dashboard.

Improve the empty dashboard state so a fresh database guides the user
towards the next useful setup action instead of only showing empty tables.

Cover the setup flow with view tests for empty state prompts, global
config creation, and host creation.
2026-05-19 12:25:45 +02:00
4dbde43465 (feature) Add host config editing view
Add a staff-only Django form for editing operational host settings while keeping
host identity stable. Support address, enablement, SSH/source overrides,
include/exclude lists, rsync extra args, and retention settings using the same
SQL-backed HostConfig model consumed by backup and scheduler flows.

Parse newline-separated list fields into JSON lists, preserve nullable
excludes_replace semantics, and cover rendering plus update behavior with view
tests.
2026-05-19 12:17:17 +02:00
6d7bf531ac (feature) Add schedule editing view for hosts
Add a staff-only Django form for creating and updating host schedules using the
SQL-backed ScheduleConfig model. Link the form from host detail pages, validate
cron expressions with the existing scheduler parser, and preserve scheduler/CLI
behavior by writing to the same source of truth.

Cover default rendering, schedule creation, updates, and invalid cron handling
with view tests.
2026-05-19 12:13:12 +02:00
123583a502 (feature) Add read-only retention plan view
Add a staff-only retention plan page for each host using the SQL-backed
retention service. Link it from the host detail page and show policy settings,
keep reasons, and snapshots that would be deleted for scheduled, manual, or all
snapshot kinds.

Keep the flow non-destructive for now, validate query parameters, and cover the
view with tests for rendering, base protection, and invalid kind handling.
2026-05-19 12:00:19 +02:00
3f3bdf2d45 (feature) Add snapshot discovery action to host view
Add a staff-only POST action on host detail pages to discover existing snapshots
for that host and record them into SQL. Show success or failure feedback through
Django messages, and keep the action non-destructive before adding heavier
backup or retention controls.

Cover the action with view tests for successful discovery, redirect behavior,
and method safety.
2026-05-19 11:56:45 +02:00
b0c6afad09 (feature) Add staff-only Django dashboard views
Add a small template-based UI for inspecting pobsync state through Django. The
dashboard shows host, schedule, snapshot, and backup run summaries, while host
detail pages show config, schedule, recent runs, and discovered snapshots.

Keep the views read-only and staff-protected, document the new dashboard URL,
and cover the routes with focused view tests.
2026-05-19 11:53:32 +02:00
2778a589ea (feature) Add staff-only service status API
Add /api/status/ for quick inspection of database backend, object counts, latest
backup run, and latest scheduler activity. Link it from the API index and reuse
schedule serialization between host summaries and status output.

Cover the endpoint with a focused API test and document the new status URL.
2026-05-19 11:46:22 +02:00
ccd89119da (feature) Add staff-only JSON inspection API
Expose lightweight Django JSON endpoints for hosts, snapshots, and backup runs
using the existing admin/staff authentication boundary. Include filters for
snapshot and run inspection, return resolved snapshot base metadata, and document
the new /api/ entrypoint.

Add endpoint tests for authentication, host summaries, snapshot lineage payloads,
and run filtering.
2026-05-19 11:43:50 +02:00
d158644567 Improve Django admin navigation for backup data
Add linked admin summaries for hosts, snapshots, and backup runs so the SQL-first
backup state is easier to inspect from the Django admin. Hosts now link to their
filtered snapshot and run lists, backup runs link back to their snapshot, and
snapshots show base/run relationships without requiring filesystem inspection.

Cover the new admin display helpers with focused tests.
2026-05-19 11:39:10 +02:00
e16c13a1e7 Move Docker web admin port to 8010
Publish the Django web container on host port 8010 while keeping the internal
runserver port at 8000. Update the Docker README URL so the admin location
matches the running compose setup.
2026-05-19 11:34:42 +02:00
797619acd9 Run post-backup pruning through SQL retention
Stop passing prune options into the legacy scheduled backup engine from the
Django backup command. Record the completed snapshot first, then apply retention
through the SQL-backed retention service so pruning sees the same SnapshotRecord
state as the admin and retention command.

Also record prune failures on BackupRun.result instead of leaving the run in an
ambiguous state.
2026-05-19 11:32:32 +02:00
254f915051 Plan Django retention from snapshot records 2026-05-19 11:24:48 +02:00
659377d894 Track snapshot base lineage in Django 2026-05-19 11:19:22 +02:00
5808800981 feat: link backup runs to snapshot records
Add a nullable SnapshotRecord foreign key to BackupRun and populate it
when run_pobsync_backup records a completed or failed snapshot. Keep the
existing snapshot_path for audit compatibility while making run-to-snapshot
navigation explicit in the database and admin.
2026-05-19 11:13:06 +02:00
109 changed files with 14433 additions and 739 deletions

7
.env.example Normal file
View File

@@ -0,0 +1,7 @@
POBSYNC_BACKUP_ROOT=/mnt/backups/pobsync
POBSYNC_DJANGO_ALLOWED_HOSTS=localhost,127.0.0.1,backup.example.com
POBSYNC_DJANGO_SECRET_KEY=change-me-to-a-long-random-secret
POBSYNC_DJANGO_DEBUG=0
POBSYNC_WEB_BIND=127.0.0.1
POBSYNC_GUNICORN_WORKERS=2
POBSYNC_GUNICORN_TIMEOUT=120

2
.gitignore vendored
View File

@@ -2,8 +2,10 @@ __pycache__/
*.py[cod] *.py[cod]
.venv/ .venv/
var/ var/
backups/
.pytest_cache/ .pytest_cache/
.mypy_cache/ .mypy_cache/
*.egg-info/ *.egg-info/
build/ build/
dist/ dist/
.env

59
CHANGELOG.md Normal file
View File

@@ -0,0 +1,59 @@
# Changelog
## 1.1.0 - 2026-05-21
UI-focused release for the Django control panel.
### Added
- Dedicated list pages for runs, snapshots, schedules, purged snapshots, and changelog navigation.
- Dashboard priority panels for required action, next scheduled work, recent activity, and storage pressure.
- Dashboard host cards with clearer backup activity, snapshot health, next run, and retention status.
- Lightweight live refresh for active run detail pages, including status, timing, controls, and rsync log output.
- Lightweight live refresh for dashboard priority and host status sections.
- Current-page navigation states for primary and system navigation.
- Responsive dashboard behavior for narrower screens.
### Changed
- Reworked the primary navigation around day-to-day operator workflows and moved admin/system links out of the main path.
- Simplified legacy-facing labels and removed source-of-truth wording that no longer applies to the Django-first model.
- Improved run and snapshot detail pages with clearer links between backup runs, snapshots, logs, and review actions.
- Improved dashboard spacing and card layouts to reduce cramped or overlapping text.
- Documented the Django-template-first partial refresh pattern for future UI work.
## 1.0.0 - 2026-05-21
Initial stable release of the Django-first pobsync control panel.
### Added
- Django control panel for hosts, global settings, schedules, SSH credentials, snapshots, runs, self-checks, and logs.
- Native systemd installer and updater for production backup servers.
- SQLite by default, with optional MariaDB support.
- Scheduler and worker services for queued manual backups and scheduled backups.
- Manual backup, dry-run, cancellation, verbose rsync logging, and run detail views.
- Snapshot discovery for existing backup directories and SQL-backed snapshot records.
- SQL retention planning and apply flow with base snapshot protection and incomplete snapshot visibility.
- Explicit cleanup flow for incomplete snapshots, separate from normal retention pruning.
- Purged snapshot audit overview with reason, action source, operator, host, kind, path, and timestamp.
- Dashboard and host pages with backup health, latest run/snapshot, next run, and storage/stat summaries.
- Review resolution for failed/warning runs and incomplete snapshot tasks so operational warnings can be acknowledged.
- Worker heartbeat metadata and stale running-run reconciliation for queued backup workers.
- SSH key generation, upload, edit, guarded delete, known_hosts management, and per-host key selection.
- In-app changelog page sourced from this changelog.
- Restore guidance on snapshot detail pages.
### Changed
- Django and the database are now the source of truth for configuration.
- Docker Compose is documented as development and disposable test tooling rather than the primary production path.
- The `pobsync` console entrypoint is now a maintainer layer around Django management commands.
- Scheduled pruning is evaluated by the pobsync scheduler service and recorded through Django, not host cron.
- Retention and incomplete cleanup now preserve audit history even after source snapshot records are removed.
### Removed
- Legacy YAML config import/export workflow.
- Public short aliases for configuration commands.
- Obsolete global config storage fields.

View File

@@ -10,7 +10,7 @@ RUN apt-get update \
WORKDIR /app WORKDIR /app
COPY pyproject.toml README.md ./ COPY pyproject.toml README.md CHANGELOG.md ./
COPY src ./src COPY src ./src
COPY manage.py ./ COPY manage.py ./
COPY scripts/docker-entrypoint ./scripts/docker-entrypoint COPY scripts/docker-entrypoint ./scripts/docker-entrypoint
@@ -24,4 +24,4 @@ RUN chmod +x ./scripts/docker-entrypoint
EXPOSE 8000 EXPOSE 8000
ENTRYPOINT ["./scripts/docker-entrypoint"] ENTRYPOINT ["./scripts/docker-entrypoint"]
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"] CMD ["gunicorn", "pobsync_server.wsgi:application", "--bind", "0.0.0.0:8000", "--workers", "2", "--timeout", "120"]

273
README.md
View File

@@ -1,155 +1,272 @@
# pobsync # pobsync
`pobsync` is a pull-based backup service. It runs on a central backup server and pulls data from remote machines via rsync over SSH. `pobsync` is a pull-based backup service. It runs on a central backup server and pulls data from remote machines via
rsync over SSH.
The refactor direction is SQL-first: The current refactor is Django-first and SQL-backed:
- Django is the management layer and source of truth. - The Django control panel is the primary interface for setup and operations.
- The database is the source of truth for hosts, schedules, runs, snapshots, credentials, and retention settings.
- SQLite is the default database; MariaDB is optional. - SQLite is the default database; MariaDB is optional.
- Backups still use the existing rsync snapshot engine internally. - Backups use the existing rsync snapshot engine internally.
- Scheduling is handled by a Django/Docker scheduler process, not host cron. - Scheduling is handled by a Django scheduler service, not host cron.
- Legacy YAML import/export exists only for migration and inspection. - SSH keys can be managed from Django and selected globally or per host.
## Requirements ## Recommended Production Install
On the backup server or in the container: The recommended production deployment is native systemd services on the backup server. Docker Compose remains available
for development and disposable test installs, but native systemd avoids Docker friction around SSH, filesystem mounts,
large backup storage, and host-level service logs.
- Python 3.11+ Recommended layout:
- rsync
- ssh
- SSH key-based access from the backup server to remotes
## Local Development
``` ```
python3 -m venv .venv /opt/pobsync/app # installed app checkout
. .venv/bin/activate /opt/pobsync/venv # Python virtualenv
python3 -m pip install -e . /etc/pobsync/pobsync.env # settings and secrets
mkdir -p var /var/lib/pobsync # SQLite database, state, runtime SSH key files, static files
python3 manage.py migrate /backups # backup storage, or set another absolute path
python3 manage.py createsuperuser
python3 manage.py runserver
``` ```
The admin is available at: From a checked-out copy of this repository, run:
- http://127.0.0.1:8000/admin/
## SQL-First Setup
Create global config:
``` ```
pobsync configure-global --backup-root /mnt/backups/pobsync sudo scripts/install-systemd
``` ```
Create a host config: When run from a terminal, the installer asks for the important paths and settings with sensible defaults already filled
in. It can also create the first Django superuser and prints the next steps when installation is complete.
The installer will, by default:
- install required Debian/Ubuntu OS packages with `apt-get`
- copy the checkout to `/opt/pobsync/app`
- create `/opt/pobsync/venv`
- write `/etc/pobsync/pobsync.env` if it does not exist
- install `pobsync-manage`, a Django management wrapper that loads `/etc/pobsync/pobsync.env`
- create `/var/lib/pobsync`, `/var/log/pobsync`, and the backup root
- install Python dependencies
- run migrations and collect static files
- generate a default SSH key for the service user if one does not exist yet
- install and start `pobsync-web`, `pobsync-worker`, and `pobsync-scheduler`
- guide you through the first login and setup steps
Common overrides:
``` ```
pobsync configure-host <host> --address <host-or-ip> sudo scripts/install-systemd \
--backup-root /mnt/backups/pobsync \
--time-zone Europe/Amsterdam \
--allowed-hosts backup.example.com,localhost,127.0.0.1 \
--csrf-trusted-origins https://backup.example.com
``` ```
Run a backup: Use `--no-install-os-packages` if you want to manage system packages yourself. Use `--force-env` only when you want the
installer to rewrite an existing `/etc/pobsync/pobsync.env`.
Use `--non-interactive` for scripted installs. Use `--verbose` when you want to see the underlying apt, pip, Django, and
systemd output.
Schedules are evaluated in `POBSYNC_TIME_ZONE`. The installer defaults this to the server timezone when it can detect
one, otherwise `UTC`; override it with `--time-zone Europe/Amsterdam` or by editing `/etc/pobsync/pobsync.env`.
For MariaDB support, add:
``` ```
pobsync backup <host> --prune sudo scripts/install-systemd --install-extras mariadb
``` ```
Create or update a schedule: ## Services
The installer creates:
- `pobsync-web.service`: Gunicorn Django control panel on `127.0.0.1:8010`
- `pobsync-worker.service`: queued backup worker
- `pobsync-scheduler.service`: SQL-backed schedule dispatcher
Check service state and logs:
``` ```
pobsync schedule <host> --cron "15 2 * * *" --prune systemctl status pobsync-web pobsync-worker pobsync-scheduler
journalctl -u pobsync-worker -f
``` ```
Run the scheduler: Restart after configuration changes:
``` ```
pobsync scheduler --loop --interval 60 sudo systemctl restart pobsync-web pobsync-worker pobsync-scheduler
``` ```
Plan or apply retention manually: ## Reverse Proxy
Use an existing reverse proxy by forwarding to:
``` ```
pobsync retention <host> http://127.0.0.1:8010
pobsync retention <host> --apply --yes --max-delete 10
``` ```
Discover snapshots already present on disk: To install a starter nginx site file:
``` ```
pobsync discover-snapshots --host <host> sudo scripts/install-systemd --with-nginx --server-name backup.example.com
``` ```
The `pobsync` executable is a thin wrapper around Django management commands. Direct Django access is also available: For HTTPS behind a reverse proxy, set:
``` ```
pobsync django check POBSYNC_DJANGO_ALLOWED_HOSTS=backup.example.com,localhost,127.0.0.1
python3 manage.py run_pobsync_backup <host> --prune POBSYNC_DJANGO_CSRF_TRUSTED_ORIGINS=https://backup.example.com
``` ```
## Migration Helpers ## Django UI
Import existing legacy YAML configs: After install, open the control panel through your reverse proxy or directly at:
``` ```
python3 manage.py import_pobsync_configs --prefix /opt/pobsync http://127.0.0.1:8010/
``` ```
Export SQL config to legacy runtime YAML for inspection or one-off compatibility: Create a superuser if needed:
``` ```
python3 manage.py export_pobsync_configs --prefix /opt/pobsync sudo -u pobsync pobsync-manage createsuperuser
``` ```
These commands are migration helpers, not the normal operating model. For other Django management commands on native installs, use `pobsync-manage` so the production environment file is
loaded before Django starts:
## Docker With SQLite
``` ```
docker compose up --build web sudo -u pobsync pobsync-manage showmigrations pobsync_backend
sudo -u pobsync pobsync-manage check
sudo -u pobsync pobsync-manage check_pobsync_install
``` ```
This starts Django on: The UI includes:
- http://127.0.0.1:8000/admin/ - dashboard and host detail pages
- global and per-host config forms
- schedule editing
- manual backup queueing
- snapshot discovery
- host checks for backup directories and SSH readiness
- host directory preparation for new or existing hosts
- SQL retention planning and apply flow
- Django-managed SSH keys
- `/self-check/` for runtime checks
- `/logs/` for filtered pobsync service logs
Run the scheduler alongside the web admin: ## Restoring Data
pobsync treats restores as an explicit manual operation. The control panel shows restore guidance on each snapshot
detail page, but it does not run restore commands for you yet. That is deliberate: restores should be inspected and
tested before data is copied back into a live system.
Each snapshot directory contains:
``` ```
docker compose up --build web scheduler <snapshot>/data/ # backed-up filesystem contents
<snapshot>/meta/ # metadata and rsync logs
``` ```
The container persists `/opt/pobsync` and the SQLite database in Docker volumes. Use the `data/` directory as the rsync source. Start with a dry run and restore to a staging path first:
## Docker With MariaDB
``` ```
docker compose --profile mariadb up --build web-mariadb rsync -aHAX --numeric-ids --info=progress2 --dry-run /backups/example.org/scheduled/<snapshot>/data/ /restore/example.org/
rsync -aHAX --numeric-ids --info=progress2 /backups/example.org/scheduled/<snapshot>/data/ /restore/example.org/
``` ```
With the scheduler: After validating the staged files, copy the specific files or directories back to the target machine. For a full-host
restore, use another dry run before writing to the remote root:
``` ```
docker compose --profile mariadb up --build web-mariadb scheduler-mariadb rsync -aHAX --numeric-ids --info=progress2 --dry-run /backups/example.org/scheduled/<snapshot>/data/ root@example.org:/
``` ```
SQLite remains the default because it is enough for a single backup server and keeps deployment simple. For most incidents, prefer a targeted restore instead of copying the whole snapshot. Keep paths relative to the
snapshot's `data/` directory:
## Current Architecture ```
rsync -aHAX --numeric-ids --info=progress2 --dry-run /backups/example.org/scheduled/<snapshot>/data/etc/nginx/ /restore/example.org/etc/nginx/
rsync -aHAX --numeric-ids --info=progress2 --dry-run /backups/example.org/scheduled/<snapshot>/data/home/example/site/public_html/index.php /restore/example.org/home/example/site/public_html/index.php
```
The public command surface is Django-first. The old YAML/cron CLI has been retired from the `pobsync` entrypoint. Snapshots may use hardlinks for files that are unchanged between backups. That saves disk space and is safe for normal
restore copies, but do not edit files inside snapshot directories. Treat snapshots as read-only and copy data out with
rsync.
The remaining internal engine code still contains reusable backup primitives: ## SSH Keys
- snapshot naming and metadata SSH keys can be managed from `/ssh-credentials/`. The recommended flow is to generate keys from Django or during the
- rsync command construction and execution installer. pobsync stores the private key on disk under the runtime state root (`POBSYNC_HOME`), keeps the public key
- retention planning and pruning visible in the UI, and lets you select a credential either as the global default or as a per-host override.
- host locking
Next refactor targets: Generated private keys are stored at:
- Record discovered snapshots into `SnapshotRecord`. ```
- Surface `SnapshotRecord` data through API/admin views instead of filesystem inspection. $POBSYNC_HOME/state/ssh-credentials/<id>/identity
- Move more snapshot lifecycle details into typed domain objects. ```
- Replace remaining dictionary-shaped config at engine boundaries.
- Remove legacy YAML import/export once production migration no longer needs it. The key file is written with `0600` permissions and injected into the rsync SSH command with `IdentityFile`. Copy the
public key shown in Django to the target host's `authorized_keys`.
Existing private keys can still be added manually, but generated filesystem keys are preferred for native systemd
production installs.
## Updates
From a fresh checkout or the existing app directory:
```
git pull
sudo scripts/update-systemd
```
The updater is a thin wrapper around the installer for normal production deploys. It preserves the existing
`/etc/pobsync/pobsync.env`, skips OS package installation, skips superuser creation, refreshes the installed app, updates
Python dependencies, runs migrations, collects static files, and restarts the systemd services so new Django code is
loaded.
Use the full installer again when you intentionally want to change install-time settings, install OS packages, enable
nginx, or rewrite the environment file:
```
sudo scripts/install-systemd --non-interactive
sudo scripts/install-systemd --force-env
```
Then check:
```
systemctl status pobsync-web pobsync-worker pobsync-scheduler
sudo -u pobsync pobsync-manage check
sudo -u pobsync pobsync-manage check_pobsync_install
```
Restart services manually after environment or reverse proxy changes:
```
sudo systemctl restart pobsync-web pobsync-worker pobsync-scheduler
```
Inspect service logs with:
```
journalctl -u pobsync-web -n 100 --no-pager
journalctl -u pobsync-worker -f
journalctl -u pobsync-scheduler -n 100 --no-pager
```
Rollback to a previous revision by checking out the known-good commit or tag, then running the updater again:
```
git switch master
git pull
git checkout <known-good-commit-or-tag>
sudo scripts/update-systemd
sudo -u pobsync pobsync-manage check_pobsync_install
```
## Development
Development, Docker, maintainer tooling, and architecture notes live in:
- [docs/development.md](docs/development.md)

24
deploy/bin/pobsync-manage Normal file
View File

@@ -0,0 +1,24 @@
#!/bin/sh
set -eu
APP_DIR="@POBSYNC_APP_DIR@"
VENV_DIR="@POBSYNC_VENV_DIR@"
ENV_FILE="@POBSYNC_ENV_FILE@"
SERVICE_USER="@POBSYNC_USER@"
SERVICE_GROUP="@POBSYNC_GROUP@"
if [ ! -f "$ENV_FILE" ]; then
echo "pobsync environment file not found: $ENV_FILE" >&2
exit 1
fi
set -a
# shellcheck disable=SC1090
. "$ENV_FILE"
set +a
export POBSYNC_ENV_FILE="$ENV_FILE"
export POBSYNC_SERVICE_USER="$SERVICE_USER"
export POBSYNC_SERVICE_GROUP="$SERVICE_GROUP"
cd "$APP_DIR"
exec "$VENV_DIR/bin/python" "$APP_DIR/manage.py" "$@"

15
deploy/nginx/pobsync.conf Normal file
View File

@@ -0,0 +1,15 @@
server {
listen 80;
server_name @POBSYNC_SERVER_NAME@;
client_max_body_size 16m;
location / {
proxy_pass http://127.0.0.1:8010;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}

View File

@@ -0,0 +1,18 @@
POBSYNC_DJANGO_DEBUG=0
POBSYNC_DJANGO_SECRET_KEY=change-me-to-a-long-random-secret
POBSYNC_DJANGO_ALLOWED_HOSTS=backup.example.com,localhost,127.0.0.1
POBSYNC_DJANGO_CSRF_TRUSTED_ORIGINS=https://backup.example.com
POBSYNC_HOME=/var/lib/pobsync
POBSYNC_BACKUP_ROOT=/backups
POBSYNC_SQLITE_PATH=/var/lib/pobsync/pobsync.sqlite3
POBSYNC_STATIC_ROOT=/var/lib/pobsync/static
POBSYNC_ENV_FILE=/etc/pobsync/pobsync.env
POBSYNC_SERVICE_USER=pobsync
POBSYNC_SERVICE_GROUP=pobsync
POBSYNC_WEB_BIND=127.0.0.1:8010
POBSYNC_GUNICORN_WORKERS=2
POBSYNC_GUNICORN_TIMEOUT=120
POBSYNC_WORKER_INTERVAL=15
POBSYNC_SCHEDULER_INTERVAL=60

View File

@@ -0,0 +1,20 @@
[Unit]
Description=pobsync schedule dispatcher
After=network-online.target pobsync-web.service
Wants=network-online.target
[Service]
Type=simple
User=@POBSYNC_USER@
Group=@POBSYNC_GROUP@
WorkingDirectory=@POBSYNC_APP_DIR@
EnvironmentFile=@POBSYNC_ENV_FILE@
Environment=POBSYNC_ENV_FILE=@POBSYNC_ENV_FILE@
Environment=POBSYNC_SERVICE_USER=@POBSYNC_USER@
Environment=POBSYNC_SERVICE_GROUP=@POBSYNC_GROUP@
ExecStart=/bin/sh -c 'exec @POBSYNC_VENV_DIR@/bin/python manage.py run_pobsync_scheduler --loop --interval "${POBSYNC_SCHEDULER_INTERVAL:-60}"'
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target

View File

@@ -0,0 +1,22 @@
[Unit]
Description=pobsync Django control panel
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=@POBSYNC_USER@
Group=@POBSYNC_GROUP@
WorkingDirectory=@POBSYNC_APP_DIR@
EnvironmentFile=@POBSYNC_ENV_FILE@
Environment=POBSYNC_ENV_FILE=@POBSYNC_ENV_FILE@
Environment=POBSYNC_SERVICE_USER=@POBSYNC_USER@
Environment=POBSYNC_SERVICE_GROUP=@POBSYNC_GROUP@
ExecStartPre=@POBSYNC_VENV_DIR@/bin/python manage.py migrate --noinput
ExecStartPre=@POBSYNC_VENV_DIR@/bin/python manage.py collectstatic --noinput --clear
ExecStart=/bin/sh -c 'exec @POBSYNC_VENV_DIR@/bin/gunicorn pobsync_server.wsgi:application --bind "${POBSYNC_WEB_BIND:-127.0.0.1:8010}" --workers "${POBSYNC_GUNICORN_WORKERS:-2}" --timeout "${POBSYNC_GUNICORN_TIMEOUT:-120}"'
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target

View File

@@ -0,0 +1,20 @@
[Unit]
Description=pobsync queued backup worker
After=network-online.target pobsync-web.service
Wants=network-online.target
[Service]
Type=simple
User=@POBSYNC_USER@
Group=@POBSYNC_GROUP@
WorkingDirectory=@POBSYNC_APP_DIR@
EnvironmentFile=@POBSYNC_ENV_FILE@
Environment=POBSYNC_ENV_FILE=@POBSYNC_ENV_FILE@
Environment=POBSYNC_SERVICE_USER=@POBSYNC_USER@
Environment=POBSYNC_SERVICE_GROUP=@POBSYNC_GROUP@
ExecStart=/bin/sh -c 'exec @POBSYNC_VENV_DIR@/bin/python manage.py run_pobsync_worker --loop --interval "${POBSYNC_WORKER_INTERVAL:-15}"'
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target

View File

@@ -1,40 +1,75 @@
services: services:
web: web:
build: . build: .
command: python manage.py runserver 0.0.0.0:8000 command: gunicorn pobsync_server.wsgi:application --bind 0.0.0.0:8000 --workers ${POBSYNC_GUNICORN_WORKERS:-2} --timeout ${POBSYNC_GUNICORN_TIMEOUT:-120}
restart: unless-stopped
environment: environment:
POBSYNC_DJANGO_DEBUG: "1" POBSYNC_DJANGO_DEBUG: "${POBSYNC_DJANGO_DEBUG:-0}"
POBSYNC_DJANGO_SECRET_KEY: "dev-only-change-me" POBSYNC_DJANGO_SECRET_KEY: "${POBSYNC_DJANGO_SECRET_KEY:-dev-only-change-me}"
POBSYNC_DJANGO_ALLOWED_HOSTS: "localhost,127.0.0.1,0.0.0.0" POBSYNC_DJANGO_ALLOWED_HOSTS: "${POBSYNC_DJANGO_ALLOWED_HOSTS:-localhost,127.0.0.1,0.0.0.0}"
POBSYNC_HOME: "/opt/pobsync" POBSYNC_HOME: "/opt/pobsync"
POBSYNC_SQLITE_PATH: "/var/lib/pobsync/pobsync.sqlite3" POBSYNC_SQLITE_PATH: "/var/lib/pobsync/pobsync.sqlite3"
ports: ports:
- "8000:8000" - "${POBSYNC_WEB_BIND:-0.0.0.0}:8010:8000"
volumes: volumes:
- pobsync_state:/opt/pobsync - pobsync_state:/opt/pobsync
- pobsync_db:/var/lib/pobsync - pobsync_db:/var/lib/pobsync
- ${POBSYNC_BACKUP_ROOT:-./backups}:/backups
healthcheck:
test: ["CMD", "python", "manage.py", "check"]
interval: 30s
timeout: 10s
retries: 3
scheduler: scheduler:
build: . build: .
command: python manage.py run_pobsync_scheduler --loop --interval 60 command: python manage.py run_pobsync_scheduler --loop --interval 60
restart: unless-stopped
environment: environment:
POBSYNC_DJANGO_DEBUG: "1" POBSYNC_DJANGO_DEBUG: "${POBSYNC_DJANGO_DEBUG:-0}"
POBSYNC_DJANGO_SECRET_KEY: "dev-only-change-me" POBSYNC_DJANGO_SECRET_KEY: "${POBSYNC_DJANGO_SECRET_KEY:-dev-only-change-me}"
POBSYNC_DJANGO_ALLOWED_HOSTS: "localhost,127.0.0.1,0.0.0.0" POBSYNC_DJANGO_ALLOWED_HOSTS: "${POBSYNC_DJANGO_ALLOWED_HOSTS:-localhost,127.0.0.1,0.0.0.0}"
POBSYNC_HOME: "/opt/pobsync" POBSYNC_HOME: "/opt/pobsync"
POBSYNC_SQLITE_PATH: "/var/lib/pobsync/pobsync.sqlite3" POBSYNC_SQLITE_PATH: "/var/lib/pobsync/pobsync.sqlite3"
volumes: volumes:
- pobsync_state:/opt/pobsync - pobsync_state:/opt/pobsync
- pobsync_db:/var/lib/pobsync - pobsync_db:/var/lib/pobsync
- ${POBSYNC_BACKUP_ROOT:-./backups}:/backups
healthcheck:
test: ["CMD", "python", "manage.py", "check"]
interval: 30s
timeout: 10s
retries: 3
worker:
build: .
command: python manage.py run_pobsync_worker --loop --interval 15
restart: unless-stopped
environment:
POBSYNC_DJANGO_DEBUG: "${POBSYNC_DJANGO_DEBUG:-0}"
POBSYNC_DJANGO_SECRET_KEY: "${POBSYNC_DJANGO_SECRET_KEY:-dev-only-change-me}"
POBSYNC_DJANGO_ALLOWED_HOSTS: "${POBSYNC_DJANGO_ALLOWED_HOSTS:-localhost,127.0.0.1,0.0.0.0}"
POBSYNC_HOME: "/opt/pobsync"
POBSYNC_SQLITE_PATH: "/var/lib/pobsync/pobsync.sqlite3"
volumes:
- pobsync_state:/opt/pobsync
- pobsync_db:/var/lib/pobsync
- ${POBSYNC_BACKUP_ROOT:-./backups}:/backups
healthcheck:
test: ["CMD", "python", "manage.py", "check"]
interval: 30s
timeout: 10s
retries: 3
web-mariadb: web-mariadb:
profiles: ["mariadb"] profiles: ["mariadb"]
build: . build: .
command: python manage.py runserver 0.0.0.0:8000 command: gunicorn pobsync_server.wsgi:application --bind 0.0.0.0:8000 --workers ${POBSYNC_GUNICORN_WORKERS:-2} --timeout ${POBSYNC_GUNICORN_TIMEOUT:-120}
restart: unless-stopped
environment: environment:
POBSYNC_DJANGO_DEBUG: "1" POBSYNC_DJANGO_DEBUG: "${POBSYNC_DJANGO_DEBUG:-0}"
POBSYNC_DJANGO_SECRET_KEY: "dev-only-change-me" POBSYNC_DJANGO_SECRET_KEY: "${POBSYNC_DJANGO_SECRET_KEY:-dev-only-change-me}"
POBSYNC_DJANGO_ALLOWED_HOSTS: "localhost,127.0.0.1,0.0.0.0" POBSYNC_DJANGO_ALLOWED_HOSTS: "${POBSYNC_DJANGO_ALLOWED_HOSTS:-localhost,127.0.0.1,0.0.0.0}"
POBSYNC_HOME: "/opt/pobsync" POBSYNC_HOME: "/opt/pobsync"
POBSYNC_DB_ENGINE: "mariadb" POBSYNC_DB_ENGINE: "mariadb"
POBSYNC_DB_HOST: "db" POBSYNC_DB_HOST: "db"
@@ -45,18 +80,25 @@ services:
db: db:
condition: service_healthy condition: service_healthy
ports: ports:
- "8000:8000" - "${POBSYNC_WEB_BIND:-0.0.0.0}:8010:8000"
volumes: volumes:
- pobsync_state:/opt/pobsync - pobsync_state:/opt/pobsync
- ${POBSYNC_BACKUP_ROOT:-./backups}:/backups
healthcheck:
test: ["CMD", "python", "manage.py", "check"]
interval: 30s
timeout: 10s
retries: 3
scheduler-mariadb: scheduler-mariadb:
profiles: ["mariadb"] profiles: ["mariadb"]
build: . build: .
command: python manage.py run_pobsync_scheduler --loop --interval 60 command: python manage.py run_pobsync_scheduler --loop --interval 60
restart: unless-stopped
environment: environment:
POBSYNC_DJANGO_DEBUG: "1" POBSYNC_DJANGO_DEBUG: "${POBSYNC_DJANGO_DEBUG:-0}"
POBSYNC_DJANGO_SECRET_KEY: "dev-only-change-me" POBSYNC_DJANGO_SECRET_KEY: "${POBSYNC_DJANGO_SECRET_KEY:-dev-only-change-me}"
POBSYNC_DJANGO_ALLOWED_HOSTS: "localhost,127.0.0.1,0.0.0.0" POBSYNC_DJANGO_ALLOWED_HOSTS: "${POBSYNC_DJANGO_ALLOWED_HOSTS:-localhost,127.0.0.1,0.0.0.0}"
POBSYNC_HOME: "/opt/pobsync" POBSYNC_HOME: "/opt/pobsync"
POBSYNC_DB_ENGINE: "mariadb" POBSYNC_DB_ENGINE: "mariadb"
POBSYNC_DB_HOST: "db" POBSYNC_DB_HOST: "db"
@@ -68,10 +110,44 @@ services:
condition: service_healthy condition: service_healthy
volumes: volumes:
- pobsync_state:/opt/pobsync - pobsync_state:/opt/pobsync
- ${POBSYNC_BACKUP_ROOT:-./backups}:/backups
healthcheck:
test: ["CMD", "python", "manage.py", "check"]
interval: 30s
timeout: 10s
retries: 3
worker-mariadb:
profiles: ["mariadb"]
build: .
command: python manage.py run_pobsync_worker --loop --interval 15
restart: unless-stopped
environment:
POBSYNC_DJANGO_DEBUG: "${POBSYNC_DJANGO_DEBUG:-0}"
POBSYNC_DJANGO_SECRET_KEY: "${POBSYNC_DJANGO_SECRET_KEY:-dev-only-change-me}"
POBSYNC_DJANGO_ALLOWED_HOSTS: "${POBSYNC_DJANGO_ALLOWED_HOSTS:-localhost,127.0.0.1,0.0.0.0}"
POBSYNC_HOME: "/opt/pobsync"
POBSYNC_DB_ENGINE: "mariadb"
POBSYNC_DB_HOST: "db"
POBSYNC_DB_NAME: "pobsync"
POBSYNC_DB_USER: "pobsync"
POBSYNC_DB_PASSWORD: "pobsync"
depends_on:
db:
condition: service_healthy
volumes:
- pobsync_state:/opt/pobsync
- ${POBSYNC_BACKUP_ROOT:-./backups}:/backups
healthcheck:
test: ["CMD", "python", "manage.py", "check"]
interval: 30s
timeout: 10s
retries: 3
db: db:
profiles: ["mariadb"] profiles: ["mariadb"]
image: mariadb:11 image: mariadb:11
restart: unless-stopped
environment: environment:
MARIADB_DATABASE: "pobsync" MARIADB_DATABASE: "pobsync"
MARIADB_USER: "pobsync" MARIADB_USER: "pobsync"

187
docs/development.md Normal file
View File

@@ -0,0 +1,187 @@
# Development Notes
This document contains development and optional Docker workflows. The recommended production path is the native
systemd installer documented in the README.
## Local Development
```
python3 -m venv .venv
. .venv/bin/activate
python3 -m pip install -e .
mkdir -p var
python3 manage.py migrate
python3 manage.py createsuperuser
python3 manage.py runserver
```
The admin is available at:
- http://127.0.0.1:8000/
- http://127.0.0.1:8000/admin/
Staff-only JSON endpoints are available at:
- http://127.0.0.1:8000/api/
- http://127.0.0.1:8000/api/status/
## Running Tests
The project test suite is currently run through the Docker image so the runtime dependencies match deployment:
```
docker compose build web scheduler worker
docker compose run --rm web python manage.py test pobsync_backend --verbosity 2
```
## Maintainer CLI
The Django UI is the normal operating surface. The `pobsync` entrypoint and direct `manage.py` commands are kept for
debugging, automated maintenance, and migrations. Prefer using the control panel for day-to-day host configuration,
schedule changes, manual backup queueing, snapshot discovery, retention planning, and SSH credential management.
Useful checks:
```
pobsync django check
python3 manage.py showmigrations pobsync_backend
```
The short `pobsync` aliases are limited to operational actions that are useful while debugging a running install.
Configuration aliases are intentionally not public commands; use the Django UI or explicit management commands instead.
## UI Refresh Pattern
The control panel stays Django-template-first. Pages that need live status should expose a small server-rendered partial
view and opt into refresh with `data-refresh-url` and `data-refresh-interval` on the container that should be replaced.
The shared script in `base.html` polls only those explicit regions, skips refreshes while the browser tab is hidden, and
lets the partial response turn polling off with the `X-Pobsync-Refresh-Active: false` header.
Use this for operational status surfaces such as running backup details. Avoid refreshing form-heavy sections while an
operator might be typing.
Worker and scheduler commands are normally run by systemd services:
```
pobsync worker --loop --interval 15
pobsync scheduler --loop --interval 60
```
One-off maintenance commands are still available when the UI is not the right tool:
```
pobsync backup <host> --dry-run
pobsync discover-snapshots --host <host>
pobsync retention <host>
```
For scripted configuration changes, call the Django management command explicitly so it is clear that this is an
automation/debugging path rather than the normal UI workflow:
```
pobsync django configure_pobsync_host <host> --address <host.example>
pobsync django configure_pobsync_schedule <host> --schedule-expression "15 2 * * *"
```
## Installer Development
The native installer is interactive by default when stdin is a terminal. It should keep every prompt backed by a command
line flag or environment variable so production installs remain scriptable.
Useful modes:
```
sudo scripts/install-systemd
sudo scripts/install-systemd --non-interactive
sudo scripts/install-systemd --verbose
sudo scripts/install-systemd --create-superuser --superuser-username admin
sudo scripts/update-systemd
```
The installer should print a short completion summary with the control panel URL, Self Check reminder, and service log
commands. Keep normal output user-facing: pobsync step names with OK, FAILED, or SKIPPED. Full apt, pip, Django, and
systemd output belongs behind `--verbose` or in the failed step output.
The updater is intentionally a small wrapper around the installer for routine production deploys. It should stay
non-interactive, preserve the existing environment file, skip OS package installation, skip superuser creation, and still
run the Django/runtime refresh steps needed after a code update.
## Docker With SQLite
Docker Compose is useful for local development and disposable test installs. Native systemd is preferred for production
backup servers.
```
docker compose up --build web
```
This starts Django on:
- http://127.0.0.1:8010/
- http://127.0.0.1:8010/admin/
- http://127.0.0.1:8010/api/
- http://127.0.0.1:8010/api/status/
Run the scheduler alongside the web admin:
```
docker compose up --build web scheduler worker
```
The web service runs Django through Gunicorn and serves static files with WhiteNoise. The container persists
`/opt/pobsync` and the SQLite database in Docker volumes.
Backup data is always available at `/backups` inside the containers. By default this uses `./backups` on the host.
Override the host-side mount with `POBSYNC_BACKUP_ROOT`:
```
POBSYNC_BACKUP_ROOT=/mnt/backups/pobsync docker compose up --build web scheduler worker
```
## Docker With MariaDB
```
docker compose --profile mariadb up --build web-mariadb
```
With the scheduler:
```
docker compose --profile mariadb up --build web-mariadb scheduler-mariadb worker-mariadb
```
SQLite remains the default because it is enough for a single backup server and keeps deployment simple.
For native systemd installs with MariaDB client support, run the installer with:
```
sudo scripts/install-systemd --install-extras mariadb
```
## Current Architecture
The public operating surface is Django-first. The CLI is now a maintainer layer around Django management commands and
the old YAML/cron workflow has been retired from the `pobsync` entrypoint.
Discovered snapshots are stored in `SnapshotRecord`, including the base snapshot metadata and a nullable SQL link to the
base record when it is known.
The Django retention command plans from `SnapshotRecord` instead of rediscovering snapshots from the filesystem.
Post-backup pruning from Django also uses the SQL retention service after the completed snapshot is recorded.
Staff-only JSON endpoints expose service status, hosts, snapshots, and backup runs for lightweight inspection.
Staff-only dashboard views expose the same operational state through Django templates.
Host pages include a safe snapshot discovery action that records existing snapshots into SQL.
Host pages also include a read-only SQL retention plan view before any destructive pruning action.
Schedules can be created or updated from host pages using the same SQL-backed scheduler model.
Host config can be edited from host pages while keeping host identity stable.
The remaining internal engine code still contains reusable backup primitives:
- snapshot naming and metadata
- rsync command construction and execution
- retention planning and pruning
- host locking
Next refactor targets:
- Move more snapshot lifecycle details into typed domain objects.
- Replace remaining dictionary-shaped config at engine boundaries.

View File

@@ -4,11 +4,13 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "pobsync" name = "pobsync"
version = "0.1.0" version = "1.1.0"
description = "Pull-based rsync backup tool with hardlinked snapshots" description = "Pull-based rsync backup tool with hardlinked snapshots"
requires-python = ">=3.11" requires-python = ">=3.11"
dependencies = [ dependencies = [
"Django>=5.2,<6.0", "Django>=5.2,<6.0",
"gunicorn>=23.0,<24.0",
"whitenoise>=6.9,<7.0",
"PyYAML>=6.0" "PyYAML>=6.0"
] ]

View File

@@ -1,97 +0,0 @@
#!/bin/sh
# Deploy pobsync runtime into /opt/pobsync without pip/venv.
# Copies python package sources into /opt/pobsync/lib and installs a stable entrypoint in /opt/pobsync/bin.
set -eu
PREFIX="/opt/pobsync"
usage() {
echo "Usage: $0 [--prefix /opt/pobsync]" >&2
exit 2
}
while [ $# -gt 0 ]; do
case "$1" in
--prefix)
[ $# -ge 2 ] || usage
PREFIX="$2"
shift 2
;;
-h|--help)
usage
;;
*)
echo "Unknown arg: $1" >&2
usage
;;
esac
done
# Determine repo root from this script location
SCRIPT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)"
REPO_ROOT="$(CDPATH= cd -- "${SCRIPT_DIR}/.." && pwd)"
SRC_PKG="${REPO_ROOT}/src/pobsync"
if [ ! -d "${SRC_PKG}" ]; then
echo "ERROR: expected python package at ${SRC_PKG}" >&2
exit 1
fi
BIN_DIR="${PREFIX}/bin"
LIB_DIR="${PREFIX}/lib"
DST_PKG="${LIB_DIR}/pobsync"
BUILD_FILE="${DST_PKG}/_build.txt"
mkdir -p "${BIN_DIR}" "${LIB_DIR}"
# Copy code into /opt/pobsync/lib/pobsync
# We use rsync if available (clean updates with --delete), otherwise fall back to cp -a.
if command -v rsync >/dev/null 2>&1; then
rsync -a --delete \
--exclude '__pycache__/' \
--exclude '*.pyc' \
--exclude '*.pyo' \
--exclude '*.pyd' \
"${SRC_PKG}/" "${DST_PKG}/"
else
# Fallback: wipe + copy
rm -rf "${DST_PKG}"
mkdir -p "${DST_PKG}"
cp -a "${SRC_PKG}/." "${DST_PKG}/"
fi
# Write build info (best-effort)
GIT_SHA="unknown"
if command -v git >/dev/null 2>&1 && [ -d "${REPO_ROOT}/.git" ]; then
GIT_SHA="$(cd "${REPO_ROOT}" && git rev-parse HEAD 2>/dev/null || echo unknown)"
fi
NOW_UTC="$(date -u +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || echo unknown)"
{
echo "deployed_at_utc=${NOW_UTC}"
echo "git_sha=${GIT_SHA}"
echo "repo_root=${REPO_ROOT}"
} > "${BUILD_FILE}"
# Install stable entrypoint that always runs code from /opt/pobsync/lib
WRAPPER="${BIN_DIR}/pobsync"
cat > "${WRAPPER}" <<EOF
#!/bin/sh
# managed-by=pobsync deploy
set -eu
PREFIX="${PREFIX}"
export PYTHONPATH="\${PREFIX}/lib"
export PYTHONUNBUFFERED=1
exec /usr/bin/python3 -m pobsync "\$@"
EOF
chmod 0755 "${WRAPPER}"
echo "OK"
echo "- deployed package to ${DST_PKG}"
echo "- wrote build info ${BUILD_FILE}"
echo "- installed entrypoint ${WRAPPER}"

View File

@@ -4,5 +4,6 @@ set -eu
mkdir -p "$(dirname "${POBSYNC_SQLITE_PATH:-/var/lib/pobsync/pobsync.sqlite3}")" mkdir -p "$(dirname "${POBSYNC_SQLITE_PATH:-/var/lib/pobsync/pobsync.sqlite3}")"
python manage.py migrate --noinput python manage.py migrate --noinput
python manage.py collectstatic --noinput --clear
exec "$@" exec "$@"

594
scripts/install-systemd Executable file
View File

@@ -0,0 +1,594 @@
#!/bin/sh
set -eu
SOURCE_DIR=${POBSYNC_SOURCE_DIR:-$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)}
APP_DIR=${POBSYNC_APP_DIR:-/opt/pobsync/app}
VENV_DIR=${POBSYNC_VENV_DIR:-/opt/pobsync/venv}
ENV_FILE=${POBSYNC_ENV_FILE:-/etc/pobsync/pobsync.env}
SERVICE_USER=${POBSYNC_SERVICE_USER:-pobsync}
SERVICE_GROUP=${POBSYNC_SERVICE_GROUP:-pobsync}
INSTALL_EXTRAS=${POBSYNC_INSTALL_EXTRAS:-}
SERVER_NAME=${POBSYNC_SERVER_NAME:-_}
ALLOWED_HOSTS=${POBSYNC_ALLOWED_HOSTS:-localhost,127.0.0.1}
CSRF_TRUSTED_ORIGINS=${POBSYNC_CSRF_TRUSTED_ORIGINS:-}
BACKUP_ROOT=${POBSYNC_BACKUP_ROOT:-/backups}
BACKUP_ROOT_EXPLICIT=0
if [ -n "${POBSYNC_BACKUP_ROOT:-}" ]; then
BACKUP_ROOT_EXPLICIT=1
fi
WEB_BIND=${POBSYNC_WEB_BIND:-127.0.0.1:8010}
TIME_ZONE=${POBSYNC_TIME_ZONE:-}
FORCE_ENV=0
INSTALL_OS_PACKAGES=1
WITH_NGINX=0
VERBOSE=0
INTERACTIVE=0
CREATE_SUPERUSER=ask
SUPERUSER_USERNAME=${POBSYNC_SUPERUSER_USERNAME:-}
SUPERUSER_EMAIL=${POBSYNC_SUPERUSER_EMAIL:-}
SUPERUSER_PASSWORD=${POBSYNC_SUPERUSER_PASSWORD:-}
if [ -t 0 ]; then
INTERACTIVE=1
fi
while [ "$#" -gt 0 ]; do
case "$1" in
--source-dir)
SOURCE_DIR=$2
shift 2
;;
--app-dir)
APP_DIR=$2
shift 2
;;
--venv-dir)
VENV_DIR=$2
shift 2
;;
--env-file)
ENV_FILE=$2
shift 2
;;
--service-user)
SERVICE_USER=$2
shift 2
;;
--service-group)
SERVICE_GROUP=$2
shift 2
;;
--backup-root)
BACKUP_ROOT=$2
BACKUP_ROOT_EXPLICIT=1
shift 2
;;
--allowed-hosts)
ALLOWED_HOSTS=$2
shift 2
;;
--csrf-trusted-origins)
CSRF_TRUSTED_ORIGINS=$2
shift 2
;;
--web-bind)
WEB_BIND=$2
shift 2
;;
--time-zone)
TIME_ZONE=$2
shift 2
;;
--force-env)
FORCE_ENV=1
shift
;;
--verbose)
VERBOSE=1
shift
;;
--interactive)
INTERACTIVE=1
shift
;;
--non-interactive)
INTERACTIVE=0
shift
;;
--no-install-os-packages)
INSTALL_OS_PACKAGES=0
shift
;;
--install-extras)
INSTALL_EXTRAS=$2
shift 2
;;
--with-nginx)
WITH_NGINX=1
shift
;;
--server-name)
SERVER_NAME=$2
shift 2
;;
--create-superuser)
CREATE_SUPERUSER=1
shift
;;
--no-create-superuser)
CREATE_SUPERUSER=0
shift
;;
--superuser-username)
SUPERUSER_USERNAME=$2
shift 2
;;
--superuser-email)
SUPERUSER_EMAIL=$2
shift 2
;;
--superuser-password)
SUPERUSER_PASSWORD=$2
shift 2
;;
*)
echo "Unknown argument: $1" >&2
exit 2
;;
esac
done
if [ "$(id -u)" -ne 0 ]; then
echo "Run this installer as root." >&2
exit 1
fi
if [ -f "$ENV_FILE" ] && [ "$FORCE_ENV" -ne 1 ] && [ "$BACKUP_ROOT_EXPLICIT" -ne 1 ]; then
set -a
# shellcheck disable=SC1090
. "$ENV_FILE"
set +a
if [ -n "${POBSYNC_BACKUP_ROOT:-}" ]; then
BACKUP_ROOT=$POBSYNC_BACKUP_ROOT
fi
fi
detect_time_zone() {
if [ -n "$TIME_ZONE" ]; then
printf '%s\n' "$TIME_ZONE"
return
fi
if [ -n "${POBSYNC_TIME_ZONE:-}" ]; then
printf '%s\n' "$POBSYNC_TIME_ZONE"
return
fi
if command -v timedatectl >/dev/null 2>&1; then
detected=$(timedatectl show -p Timezone --value 2>/dev/null || true)
if [ -n "$detected" ]; then
printf '%s\n' "$detected"
return
fi
fi
if [ -f /etc/timezone ]; then
detected=$(sed -n '1p' /etc/timezone | tr -d '[:space:]')
if [ -n "$detected" ]; then
printf '%s\n' "$detected"
return
fi
fi
printf 'UTC\n'
}
TIME_ZONE=$(detect_time_zone)
run_step() {
label=$1
shift
if [ "$VERBOSE" -eq 1 ]; then
echo "==> $label"
"$@"
echo "OK: $label"
return
fi
printf '%-48s' "$label"
log_file=$(mktemp)
if "$@" >"$log_file" 2>&1; then
rm -f "$log_file"
echo "OK"
return
fi
echo "FAILED"
echo
echo "Output from failed step '$label':" >&2
cat "$log_file" >&2
rm -f "$log_file"
exit 1
}
note_step() {
label=$1
status=$2
printf '%-48s%s\n' "$label" "$status"
}
prompt_value() {
prompt=$1
default=$2
if [ "$INTERACTIVE" -ne 1 ]; then
printf '%s\n' "$default"
return
fi
printf '%s [%s]: ' "$prompt" "$default" >&2
read -r answer
if [ -n "$answer" ]; then
printf '%s\n' "$answer"
else
printf '%s\n' "$default"
fi
}
prompt_yes_no() {
prompt=$1
default=$2
if [ "$INTERACTIVE" -ne 1 ]; then
printf '%s\n' "$default"
return
fi
if [ "$default" -eq 1 ]; then
suffix=Y/n
else
suffix=y/N
fi
while :; do
printf '%s [%s]: ' "$prompt" "$suffix" >&2
read -r answer
case "$answer" in
"")
printf '%s\n' "$default"
return
;;
y|Y|yes|YES|Yes)
printf '1\n'
return
;;
n|N|no|NO|No)
printf '0\n'
return
;;
esac
done
}
prompt_secret() {
prompt=$1
if [ "$INTERACTIVE" -ne 1 ]; then
printf '\n'
return
fi
printf '%s: ' "$prompt" >&2
stty -echo
read -r secret
stty echo
printf '\n' >&2
printf '%s\n' "$secret"
}
if [ "$INTERACTIVE" -eq 1 ]; then
echo "pobsync native installer"
echo
echo "Press Enter to accept defaults. Existing command-line flags are already applied as defaults."
echo
SOURCE_DIR=$(prompt_value "Source checkout" "$SOURCE_DIR")
APP_DIR=$(prompt_value "Install app directory" "$APP_DIR")
VENV_DIR=$(prompt_value "Python virtualenv directory" "$VENV_DIR")
ENV_FILE=$(prompt_value "Environment file" "$ENV_FILE")
SERVICE_USER=$(prompt_value "Service user" "$SERVICE_USER")
SERVICE_GROUP=$(prompt_value "Service group" "$SERVICE_GROUP")
BACKUP_ROOT=$(prompt_value "Backup storage path" "$BACKUP_ROOT")
WEB_BIND=$(prompt_value "Gunicorn bind address" "$WEB_BIND")
TIME_ZONE=$(prompt_value "Scheduler time zone" "$TIME_ZONE")
ALLOWED_HOSTS=$(prompt_value "Allowed hosts" "$ALLOWED_HOSTS")
CSRF_TRUSTED_ORIGINS=$(prompt_value "CSRF trusted origins, comma-separated or blank" "$CSRF_TRUSTED_ORIGINS")
INSTALL_OS_PACKAGES=$(prompt_yes_no "Install required OS packages with apt-get" "$INSTALL_OS_PACKAGES")
use_mariadb=0
if [ "$INSTALL_EXTRAS" = "mariadb" ] || [ "$INSTALL_EXTRAS" = "[mariadb]" ] || [ "$INSTALL_EXTRAS" = ".[mariadb]" ]; then
use_mariadb=1
fi
use_mariadb=$(prompt_yes_no "Install MariaDB Python/client support" "$use_mariadb")
if [ "$use_mariadb" -eq 1 ]; then
INSTALL_EXTRAS=mariadb
else
INSTALL_EXTRAS=
fi
WITH_NGINX=$(prompt_yes_no "Install starter nginx reverse proxy config" "$WITH_NGINX")
if [ "$WITH_NGINX" -eq 1 ]; then
SERVER_NAME=$(prompt_value "Nginx server_name" "$SERVER_NAME")
fi
if [ "$CREATE_SUPERUSER" = "ask" ]; then
CREATE_SUPERUSER=$(prompt_yes_no "Create first Django superuser after install" 1)
fi
if [ "$CREATE_SUPERUSER" -eq 1 ]; then
SUPERUSER_USERNAME=$(prompt_value "Superuser username" "${SUPERUSER_USERNAME:-admin}")
SUPERUSER_EMAIL=$(prompt_value "Superuser email, blank allowed" "$SUPERUSER_EMAIL")
if [ -z "$SUPERUSER_PASSWORD" ]; then
SUPERUSER_PASSWORD=$(prompt_secret "Superuser password, leave blank to run createsuperuser interactively later")
fi
fi
echo
fi
if [ "$CREATE_SUPERUSER" = "ask" ]; then
if [ -n "$SUPERUSER_USERNAME" ] && [ -n "$SUPERUSER_PASSWORD" ]; then
CREATE_SUPERUSER=1
else
CREATE_SUPERUSER=0
fi
fi
install_os_packages() {
if [ "$INSTALL_OS_PACKAGES" -ne 1 ]; then
note_step "Install OS packages" "SKIPPED"
return
fi
if command -v apt-get >/dev/null 2>&1; then
packages="python3 python3-venv python3-pip rsync openssh-client"
if [ "$WITH_NGINX" -eq 1 ]; then
packages="$packages nginx"
fi
if [ "$INSTALL_EXTRAS" = "mariadb" ] || [ "$INSTALL_EXTRAS" = "[mariadb]" ] || [ "$INSTALL_EXTRAS" = ".[mariadb]" ]; then
packages="$packages default-libmysqlclient-dev build-essential pkg-config"
fi
run_step "Install OS packages" sh -c "apt-get update && apt-get install -y --no-install-recommends $packages"
return
fi
echo "No supported package manager found; install python3, python3-venv, rsync, and openssh-client manually." >&2
}
install_os_packages
if ! command -v python3 >/dev/null 2>&1; then
echo "python3 is required." >&2
exit 1
fi
if ! env POBSYNC_INSTALL_TIME_ZONE="$TIME_ZONE" python3 -c "import os; from zoneinfo import ZoneInfo; ZoneInfo(os.environ['POBSYNC_INSTALL_TIME_ZONE'])" >/dev/null 2>&1; then
echo "Invalid time zone: $TIME_ZONE" >&2
echo "Use an IANA timezone such as UTC or Europe/Amsterdam." >&2
exit 1
fi
if ! command -v rsync >/dev/null 2>&1; then
echo "rsync is required." >&2
exit 1
fi
if ! command -v ssh >/dev/null 2>&1; then
echo "openssh-client is required." >&2
exit 1
fi
if [ ! -f "$SOURCE_DIR/manage.py" ]; then
echo "Source directory does not look like a pobsync checkout: $SOURCE_DIR" >&2
exit 1
fi
if ! getent group "$SERVICE_GROUP" >/dev/null 2>&1; then
run_step "Create service group" groupadd --system "$SERVICE_GROUP"
else
note_step "Create service group" "OK"
fi
if ! id "$SERVICE_USER" >/dev/null 2>&1; then
run_step "Create service user" useradd --system --home /var/lib/pobsync --shell /usr/sbin/nologin --gid "$SERVICE_GROUP" "$SERVICE_USER"
else
note_step "Create service user" "OK"
fi
grant_journal_access() {
for group in systemd-journal adm; do
if getent group "$group" >/dev/null 2>&1; then
usermod -a -G "$group" "$SERVICE_USER"
fi
done
}
run_step "Grant journal access" grant_journal_access
run_step "Prepare directories" mkdir -p /etc/pobsync /var/lib/pobsync /var/log/pobsync "$(dirname "$VENV_DIR")" "$APP_DIR" "$BACKUP_ROOT"
run_step "Set state directory permissions" chown "$SERVICE_USER:$SERVICE_GROUP" /var/lib/pobsync /var/log/pobsync "$BACKUP_ROOT"
run_step "Set private directory modes" chmod 0750 /var/lib/pobsync /var/log/pobsync "$BACKUP_ROOT"
if [ "$SOURCE_DIR" != "$APP_DIR" ]; then
run_step "Sync application files" rsync -a --delete \
--exclude .git \
--exclude .venv \
--exclude __pycache__ \
--exclude .pytest_cache \
--exclude .mypy_cache \
--exclude var \
"$SOURCE_DIR"/ "$APP_DIR"/
else
note_step "Sync application files" "SKIPPED"
fi
run_step "Create Python virtualenv" python3 -m venv "$VENV_DIR"
run_step "Upgrade pip" "$VENV_DIR/bin/python" -m pip install --upgrade pip
case "$INSTALL_EXTRAS" in
"")
pip_target=$APP_DIR
;;
mariadb)
pip_target="$APP_DIR[mariadb]"
;;
\[*\])
pip_target="$APP_DIR$INSTALL_EXTRAS"
;;
.\[*\])
pip_target="$APP_DIR${INSTALL_EXTRAS#.}"
;;
*)
echo "Unsupported install extras: $INSTALL_EXTRAS" >&2
exit 2
;;
esac
run_step "Install Python package" "$VENV_DIR/bin/python" -m pip install -e "$pip_target"
if [ ! -f "$ENV_FILE" ] || [ "$FORCE_ENV" -eq 1 ]; then
secret=$("$VENV_DIR/bin/python" -c "import secrets; print(secrets.token_urlsafe(48))")
cat > "$ENV_FILE" <<EOF
POBSYNC_DJANGO_DEBUG=0
POBSYNC_DJANGO_SECRET_KEY=$secret
POBSYNC_DJANGO_ALLOWED_HOSTS=$ALLOWED_HOSTS
POBSYNC_DJANGO_CSRF_TRUSTED_ORIGINS=$CSRF_TRUSTED_ORIGINS
POBSYNC_HOME=/var/lib/pobsync
POBSYNC_BACKUP_ROOT=$BACKUP_ROOT
POBSYNC_TIME_ZONE=$TIME_ZONE
POBSYNC_SQLITE_PATH=/var/lib/pobsync/pobsync.sqlite3
POBSYNC_STATIC_ROOT=/var/lib/pobsync/static
POBSYNC_ENV_FILE=$ENV_FILE
POBSYNC_SERVICE_USER=$SERVICE_USER
POBSYNC_SERVICE_GROUP=$SERVICE_GROUP
POBSYNC_WEB_BIND=$WEB_BIND
POBSYNC_GUNICORN_WORKERS=2
POBSYNC_GUNICORN_TIMEOUT=120
POBSYNC_WORKER_INTERVAL=15
POBSYNC_SCHEDULER_INTERVAL=60
EOF
chmod 0640 "$ENV_FILE"
chown "root:$SERVICE_GROUP" "$ENV_FILE"
note_step "Write environment file" "OK"
else
note_step "Write environment file" "SKIPPED"
echo "Keeping existing $ENV_FILE. Use --force-env to rewrite it."
fi
set -a
# shellcheck disable=SC1090
. "$ENV_FILE"
set +a
install_unit() {
src=$1
dest=$2
sed \
-e "s|@POBSYNC_APP_DIR@|$APP_DIR|g" \
-e "s|@POBSYNC_VENV_DIR@|$VENV_DIR|g" \
-e "s|@POBSYNC_ENV_FILE@|$ENV_FILE|g" \
-e "s|@POBSYNC_USER@|$SERVICE_USER|g" \
-e "s|@POBSYNC_GROUP@|$SERVICE_GROUP|g" \
"$src" > "$dest"
chmod 0644 "$dest"
}
install_units() {
install_unit "$APP_DIR/deploy/systemd/pobsync-web.service" /etc/systemd/system/pobsync-web.service
install_unit "$APP_DIR/deploy/systemd/pobsync-worker.service" /etc/systemd/system/pobsync-worker.service
install_unit "$APP_DIR/deploy/systemd/pobsync-scheduler.service" /etc/systemd/system/pobsync-scheduler.service
}
run_step "Install systemd units" install_units
install_manage_wrapper() {
sed \
-e "s|@POBSYNC_APP_DIR@|$APP_DIR|g" \
-e "s|@POBSYNC_VENV_DIR@|$VENV_DIR|g" \
-e "s|@POBSYNC_ENV_FILE@|$ENV_FILE|g" \
-e "s|@POBSYNC_USER@|$SERVICE_USER|g" \
-e "s|@POBSYNC_GROUP@|$SERVICE_GROUP|g" \
"$APP_DIR/deploy/bin/pobsync-manage" > /usr/local/bin/pobsync-manage
chmod 0755 /usr/local/bin/pobsync-manage
}
run_step "Install manage wrapper" install_manage_wrapper
run_step "Reload systemd" systemctl daemon-reload
run_step "Run database migrations" /usr/local/bin/pobsync-manage migrate --noinput
run_step "Ensure default SSH key" /usr/local/bin/pobsync-manage ensure_pobsync_ssh_key --name default --set-global-default
run_step "Collect static files" /usr/local/bin/pobsync-manage collectstatic --noinput --clear
run_step "Finalize state permissions" chown -R "$SERVICE_USER:$SERVICE_GROUP" /var/lib/pobsync /var/log/pobsync
superuser_exists=$("$VENV_DIR/bin/python" -c "import os; os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'pobsync_server.settings'); import django; django.setup(); from django.contrib.auth import get_user_model; print('yes' if get_user_model().objects.filter(is_superuser=True).exists() else 'no')")
if [ "$CREATE_SUPERUSER" -eq 1 ]; then
if [ "$superuser_exists" = "yes" ]; then
note_step "Create Django superuser" "SKIPPED"
elif [ -n "$SUPERUSER_USERNAME" ] && [ -n "$SUPERUSER_PASSWORD" ]; then
run_step "Create Django superuser" env \
DJANGO_SUPERUSER_USERNAME="$SUPERUSER_USERNAME" \
DJANGO_SUPERUSER_EMAIL="$SUPERUSER_EMAIL" \
DJANGO_SUPERUSER_PASSWORD="$SUPERUSER_PASSWORD" \
/usr/local/bin/pobsync-manage createsuperuser --noinput
run_step "Finalize superuser permissions" chown -R "$SERVICE_USER:$SERVICE_GROUP" /var/lib/pobsync /var/log/pobsync
else
note_step "Create Django superuser" "SKIPPED"
echo "No superuser password was provided; create one later with:"
echo " sudo -u $SERVICE_USER pobsync-manage createsuperuser"
fi
elif [ "$superuser_exists" != "yes" ]; then
note_step "Create Django superuser" "SKIPPED"
echo "No Django superuser exists yet. Create one with:"
echo " sudo -u $SERVICE_USER pobsync-manage createsuperuser"
else
note_step "Create Django superuser" "SKIPPED"
fi
run_step "Enable services" systemctl enable pobsync-web.service pobsync-worker.service pobsync-scheduler.service
run_step "Restart services" systemctl restart pobsync-web.service pobsync-worker.service pobsync-scheduler.service
if [ "$WITH_NGINX" -eq 1 ]; then
if ! command -v nginx >/dev/null 2>&1; then
note_step "Install nginx config" "SKIPPED"
echo "nginx is not installed; skipping nginx config." >&2
else
sed "s|@POBSYNC_SERVER_NAME@|$SERVER_NAME|g" "$APP_DIR/deploy/nginx/pobsync.conf" > /etc/nginx/sites-available/pobsync.conf
ln -sf /etc/nginx/sites-available/pobsync.conf /etc/nginx/sites-enabled/pobsync.conf
note_step "Install nginx config" "OK"
run_step "Validate nginx config" nginx -t
run_step "Reload nginx" systemctl reload nginx
fi
else
note_step "Install nginx config" "SKIPPED"
fi
if [ "$VERBOSE" -eq 1 ]; then
systemctl --no-pager --full status pobsync-web.service pobsync-worker.service pobsync-scheduler.service || true
fi
echo
echo "pobsync installation complete."
echo
echo "Open the Django control panel at:"
echo " http://$WEB_BIND/"
echo
echo "If pobsync is behind a reverse proxy, use your public hostname instead."
echo
echo "Recommended first steps:"
echo " 1. Log in to the Django control panel."
echo " 2. Open Self Check and resolve any warnings."
echo " 3. Configure global settings and backup storage."
echo " 4. Add an SSH key under SSH Keys."
echo " 5. Add a host and queue a dry-run backup."
echo
echo "Useful commands:"
echo " systemctl status pobsync-web pobsync-worker pobsync-scheduler"
echo " journalctl -u pobsync-worker -f"
echo " sudo -u $SERVICE_USER pobsync-manage check"
echo " sudo -u $SERVICE_USER pobsync-manage check_pobsync_install"

41
scripts/update-systemd Executable file
View File

@@ -0,0 +1,41 @@
#!/bin/sh
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
show_help() {
cat <<'EOF'
Usage: sudo scripts/update-systemd [options]
Refresh an existing native pobsync systemd install from the current checkout.
This is a thin, safer update wrapper around scripts/install-systemd. It keeps
the install non-interactive, preserves the existing environment file, skips
superuser creation, and skips OS package installation by default.
Common options are forwarded to install-systemd, for example:
--source-dir PATH
--app-dir PATH
--venv-dir PATH
--env-file PATH
--service-user USER
--service-group GROUP
--install-extras mariadb
--verbose
If OS packages need to be refreshed, run scripts/install-systemd directly.
EOF
}
case "${1:-}" in
-h|--help)
show_help
exit 0
;;
esac
exec "$SCRIPT_DIR/install-systemd" \
--non-interactive \
--no-install-os-packages \
--no-create-superuser \
"$@"

View File

@@ -1,3 +1,2 @@
__all__ = ["__version__"] __all__ = ["__version__"]
__version__ = "0.1.0" __version__ = "1.1.0"

View File

@@ -6,15 +6,15 @@ from typing import Sequence
from django.core.management import execute_from_command_line from django.core.management import execute_from_command_line
from pobsync import __version__
COMMAND_ALIASES = { COMMAND_ALIASES = {
"configure-global": "configure_pobsync_global",
"configure-host": "configure_pobsync_host",
"schedule": "configure_pobsync_schedule",
"backup": "run_pobsync_backup", "backup": "run_pobsync_backup",
"retention": "run_pobsync_retention", "retention": "run_pobsync_retention",
"discover-snapshots": "discover_pobsync_snapshots", "discover-snapshots": "discover_pobsync_snapshots",
"scheduler": "run_pobsync_scheduler", "scheduler": "run_pobsync_scheduler",
"worker": "run_pobsync_worker",
} }
@@ -28,11 +28,17 @@ Usage:
Commands: Commands:
{commands} {commands}
Configuration is managed from the Django control panel. Use
`pobsync django <management-command>` for automation or debugging.
""" """
def main(argv: Sequence[str] | None = None) -> int: def main(argv: Sequence[str] | None = None) -> int:
args = list(sys.argv[1:] if argv is None else argv) args = list(sys.argv[1:] if argv is None else argv)
if args and args[0] in {"--version", "version"}:
print(f"pobsync {__version__}")
return 0
if not args or args[0] in {"-h", "--help", "help"}: if not args or args[0] in {"-h", "--help", "help"}:
print(_usage()) print(_usage())
return 0 return 0

View File

@@ -2,12 +2,11 @@ from __future__ import annotations
from datetime import datetime, timezone from datetime import datetime, timezone
from pathlib import Path from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple from typing import Any, List
from ..config.source import ConfigSource, FileConfigSource from ..config.source import ConfigSource
from ..errors import ConfigError from ..errors import ConfigError
from ..paths import PobsyncPaths from ..retention import Snapshot, apply_base_protection, build_retention_plan
from ..retention import Snapshot, build_retention_plan
from ..snapshot_meta import iter_snapshot_dirs, read_snapshot_meta, resolve_host_root from ..snapshot_meta import iter_snapshot_dirs, read_snapshot_meta, resolve_host_root
from ..util import sanitize_host from ..util import sanitize_host
@@ -28,59 +27,6 @@ def _parse_snapshot_dt(dirname: str, meta: dict) -> datetime:
return datetime.fromtimestamp(0, tz=timezone.utc) return datetime.fromtimestamp(0, tz=timezone.utc)
def _apply_base_protection(
snapshots: List[Snapshot],
keep: set[str],
reasons: Dict[str, List[str]],
) -> Tuple[set[str], Dict[str, List[str]]]:
"""
Optional policy: if a kept snapshot has a base (kind+dirname), also keep that base snapshot.
This is NOT required for hardlink snapshots to remain readable, but can be useful
for performance (better base selection) or "chain" readability.
Adds reason: "base-of:<child_dirname>"
"""
# Index snapshots by (kind, dirname)
idx: Dict[Tuple[str, str], Snapshot] = {(s.kind, s.dirname): s for s in snapshots}
changed = True
while changed:
changed = False
# Iterate over a stable list of current keep items
for child_dirname in list(keep):
# Find the child snapshot (may exist in multiple kinds; check both)
child: Optional[Snapshot] = None
for k in ("scheduled", "manual"):
child = idx.get((k, child_dirname))
if child is not None:
break
if child is None:
continue
base = child.base
if not isinstance(base, dict):
continue
base_kind = base.get("kind")
base_dirname = base.get("dirname")
if not isinstance(base_kind, str) or not isinstance(base_dirname, str):
continue
base_snap = idx.get((base_kind, base_dirname))
if base_snap is None:
# Base might have been pruned already or never existed; ignore.
continue
if base_dirname not in keep:
keep.add(base_dirname)
reasons.setdefault(base_dirname, []).append(f"base-of:{child_dirname}")
changed = True
return keep, reasons
def run_retention_plan( def run_retention_plan(
prefix: Path, prefix: Path,
host: str, host: str,
@@ -93,10 +39,9 @@ def run_retention_plan(
if kind not in {"scheduled", "manual", "all"}: if kind not in {"scheduled", "manual", "all"}:
raise ConfigError("kind must be scheduled, manual, or all") raise ConfigError("kind must be scheduled, manual, or all")
paths = PobsyncPaths(home=prefix) if config_source is None:
raise ConfigError("A Django config source is required.")
source = config_source or FileConfigSource(prefix=paths.home) cfg = config_source.effective_config_for_host(host)
cfg = source.effective_config_for_host(host)
retention = cfg.get("retention") retention = cfg.get("retention")
if not isinstance(retention, dict): if not isinstance(retention, dict):
@@ -142,7 +87,7 @@ def run_retention_plan(
reasons = dict(plan.reasons) reasons = dict(plan.reasons)
if protect_bases: if protect_bases:
keep, reasons = _apply_base_protection(snapshots=snapshots, keep=keep, reasons=reasons) keep, reasons = apply_base_protection(snapshots=snapshots, keep=keep, reasons=reasons)
delete = [s for s in snapshots if s.dirname not in keep] delete = [s for s in snapshots if s.dirname not in keep]

View File

@@ -1,13 +1,14 @@
from __future__ import annotations from __future__ import annotations
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any, Callable
from ..config.source import ConfigSource, FileConfigSource from ..config.source import ConfigSource
from ..errors import ConfigError from ..errors import ConfigError
from ..lock import acquire_host_lock from ..lock import acquire_host_lock
from ..paths import PobsyncPaths from ..paths import PobsyncPaths
from ..rsync import build_rsync_command, build_ssh_command, run_rsync from ..rsync import build_rsync_command, build_ssh_command, run_rsync
from ..run_stats import collect_storage_stats, read_rsync_stats
from ..snapshot import ( from ..snapshot import (
HostBackupDirs, HostBackupDirs,
extract_ts_and_id_from_dirname, extract_ts_and_id_from_dirname,
@@ -21,6 +22,72 @@ from ..snapshot_meta import read_snapshot_meta
from ..util import ensure_dir, realpath_startswith, sanitize_host, write_yaml_atomic from ..util import ensure_dir, realpath_startswith, sanitize_host, write_yaml_atomic
DEFAULT_DRY_RUN_TIMEOUT_SECONDS = 900
def dry_run_log_path(host: str, run_id: int | None = None) -> Path:
host = sanitize_host(host)
run_dir = f"run-{run_id}" if run_id is not None else "adhoc"
return Path(f"/tmp/pobsync-dryrun/{host}/{run_dir}/rsync.log")
def _read_log_tail(log_path: Path, *, max_lines: int = 40) -> list[str]:
try:
lines = log_path.read_text(encoding="utf-8", errors="replace").splitlines()
except OSError:
return []
return lines[-max_lines:]
def classify_rsync_failure(exit_code: int | None, log_tail: list[str]) -> dict[str, str]:
joined_tail = "\n".join(log_tail).lower()
if exit_code == 255 and "broken pipe" in joined_tail:
return {
"category": "transport",
"message": "Rsync transport closed unexpectedly.",
"hint": "The SSH/rsync stream ended with a broken pipe. Check remote rsync availability, remote shell output, excludes, and connection stability.",
}
if exit_code == 255:
return {
"category": "transport",
"message": "Rsync transport failed.",
"hint": "Exit 255 usually comes from SSH or remote rsync startup. Check SSH access, known_hosts, remote rsync, and remote shell output.",
}
if exit_code == 124:
return {
"category": "timeout",
"message": "Rsync timed out.",
"hint": "Increase the rsync timeout or narrow the backup scope with source root, includes, or excludes.",
}
if "permission denied" in joined_tail:
return {
"category": "permissions",
"message": "Rsync hit a permission error.",
"hint": "Check the SSH user, key, and permissions on the remote source.",
}
return {
"category": "rsync",
"message": "Rsync failed.",
"hint": "Check the rsync log tail for the underlying error.",
}
def _collect_run_stats(
*,
log_path: Path,
backup_root: Path,
duration_seconds: int | None = None,
snapshot_dir: Path | None = None,
) -> dict[str, Any]:
stats: dict[str, Any] = {
"rsync": read_rsync_stats(log_path),
"storage": collect_storage_stats(backup_root=backup_root, snapshot_dir=snapshot_dir),
}
if duration_seconds is not None:
stats["duration_seconds"] = int(duration_seconds)
return stats
def _host_backup_dirs(backup_root: str, host: str) -> HostBackupDirs: def _host_backup_dirs(backup_root: str, host: str) -> HostBackupDirs:
return HostBackupDirs(root=Path(backup_root) / host) return HostBackupDirs(root=Path(backup_root) / host)
@@ -88,13 +155,17 @@ def run_scheduled(
prune_max_delete: int | None = None, prune_max_delete: int | None = None,
prune_protect_bases: bool = False, prune_protect_bases: bool = False,
config_source: ConfigSource | None = None, config_source: ConfigSource | None = None,
run_id: int | None = None,
cancel_check: Callable[[], bool] | None = None,
verbose_output: bool = False,
) -> dict[str, Any]: ) -> dict[str, Any]:
host = sanitize_host(host) host = sanitize_host(host)
paths = PobsyncPaths(home=prefix) paths = PobsyncPaths(home=prefix)
source = config_source or FileConfigSource(prefix=paths.home) if config_source is None:
cfg = source.effective_config_for_host(host) raise ConfigError("A Django config source is required.")
cfg = config_source.effective_config_for_host(host)
backup_root = cfg.get("backup_root") backup_root = cfg.get("backup_root")
if not isinstance(backup_root, str) or not backup_root.startswith("/"): if not isinstance(backup_root, str) or not backup_root.startswith("/"):
@@ -141,8 +212,10 @@ def run_scheduled(
# DRY RUN # DRY RUN
# ------------------------------------------------------------ # ------------------------------------------------------------
if dry_run: if dry_run:
dest = f"/tmp/pobsync-dryrun/{host}/" dryrun_log = dry_run_log_path(host, run_id=run_id)
dryrun_log = Path(f"/tmp/pobsync-dryrun/{host}/rsync.log") dest = str(dryrun_log.parent) + "/"
dryrun_log.unlink(missing_ok=True)
effective_timeout_seconds = timeout_seconds or DEFAULT_DRY_RUN_TIMEOUT_SECONDS
cmd = build_rsync_command( cmd = build_rsync_command(
rsync_binary=str(rsync_binary), rsync_binary=str(rsync_binary),
@@ -152,25 +225,44 @@ def run_scheduled(
dest=dest, dest=dest,
link_dest=link_dest, link_dest=link_dest,
dry_run=True, dry_run=True,
timeout_seconds=timeout_seconds, timeout_seconds=effective_timeout_seconds,
bwlimit_kbps=bwlimit_kbps, bwlimit_kbps=bwlimit_kbps,
extra_excludes=list(excludes), extra_excludes=list(excludes),
extra_includes=list(includes), extra_includes=list(includes),
) )
result = run_rsync(cmd, log_path=dryrun_log, timeout_seconds=timeout_seconds) result = run_rsync(
cmd,
log_path=dryrun_log,
timeout_seconds=effective_timeout_seconds,
cancel_check=cancel_check,
)
log_tail = _read_log_tail(dryrun_log)
stats = _collect_run_stats(
log_path=dryrun_log,
backup_root=Path(backup_root),
)
return { response = {
"ok": result.exit_code == 0, "ok": result.exit_code == 0,
"dry_run": True, "dry_run": True,
"host": host, "host": host,
"base": str(base_dir) if base_dir else None, "base": str(base_dir) if base_dir else None,
"log": str(dryrun_log), "log": str(dryrun_log),
"cancelled": result.cancelled,
"timeout_seconds": effective_timeout_seconds,
"verbose_output": True,
"ssh_credential": cfg.get("ssh_credential"),
"stats": stats,
"rsync": { "rsync": {
"exit_code": result.exit_code, "exit_code": result.exit_code,
"command": result.command, "command": result.command,
"log_tail": log_tail,
}, },
} }
if result.exit_code != 0:
response["failure"] = classify_rsync_failure(result.exit_code, log_tail)
return response
# ------------------------------------------------------------ # ------------------------------------------------------------
# REAL RUN # REAL RUN
@@ -209,6 +301,7 @@ def run_scheduled(
bwlimit_kbps=bwlimit_kbps, bwlimit_kbps=bwlimit_kbps,
extra_excludes=list(excludes), extra_excludes=list(excludes),
extra_includes=list(includes), extra_includes=list(includes),
verbose_output=bool(verbose_output),
) )
meta: dict[str, Any] = { meta: dict[str, Any] = {
@@ -217,26 +310,31 @@ def run_scheduled(
"host": host, "host": host,
"type": "scheduled", "type": "scheduled",
"label": None, "label": None,
"verbose_output": bool(verbose_output),
"status": "running", "status": "running",
"started_at": format_iso_z(ts), "started_at": format_iso_z(ts),
"ended_at": None, "ended_at": None,
"duration_seconds": None, "duration_seconds": None,
"base": _base_meta_from_path(base_dir, link_dest), "base": _base_meta_from_path(base_dir, link_dest),
"rsync": {"exit_code": None, "command": cmd, "stats": {}}, "rsync": {"exit_code": None, "command": cmd, "stats": {}},
# Keep existing fields for future expansion / compatibility with current structure.
"overrides": {"includes": [], "excludes": [], "base": None}, "overrides": {"includes": [], "excludes": [], "base": None},
} }
log_path.touch(exist_ok=True) log_path.touch(exist_ok=True)
write_yaml_atomic(meta_path, meta) write_yaml_atomic(meta_path, meta)
result = run_rsync(cmd, log_path=log_path, timeout_seconds=timeout_seconds) result = run_rsync(cmd, log_path=log_path, timeout_seconds=timeout_seconds, cancel_check=cancel_check)
end_ts = utc_now() end_ts = utc_now()
meta["ended_at"] = format_iso_z(end_ts) meta["ended_at"] = format_iso_z(end_ts)
meta["duration_seconds"] = int((end_ts - ts).total_seconds()) meta["duration_seconds"] = int((end_ts - ts).total_seconds())
meta["rsync"]["exit_code"] = result.exit_code meta["rsync"]["exit_code"] = result.exit_code
meta["status"] = "success" if result.exit_code == 0 else "failed" meta["status"] = "cancelled" if result.cancelled else ("success" if result.exit_code == 0 else "failed")
meta["stats"] = _collect_run_stats(
log_path=log_path,
backup_root=Path(backup_root),
duration_seconds=meta["duration_seconds"],
)
write_yaml_atomic(meta_path, meta) write_yaml_atomic(meta_path, meta)
if not log_path.exists(): if not log_path.exists():
@@ -252,17 +350,37 @@ def run_scheduled(
} }
if result.exit_code != 0: if result.exit_code != 0:
log_tail = _read_log_tail(log_path)
return { return {
"ok": False, "ok": False,
"dry_run": False, "dry_run": False,
"host": host, "host": host,
"snapshot": str(incomplete_dir), "snapshot": str(incomplete_dir),
"status": meta["status"], "status": meta["status"],
"rsync": {"exit_code": result.exit_code}, "cancelled": result.cancelled,
"log": str(log_path),
"verbose_output": bool(verbose_output),
"ssh_credential": cfg.get("ssh_credential"),
"stats": meta["stats"],
"rsync": {
"exit_code": result.exit_code,
"command": result.command,
"log_tail": log_tail,
},
"failure": classify_rsync_failure(result.exit_code, log_tail),
} }
final_dir = dirs.scheduled / snap_name final_dir = dirs.scheduled / snap_name
incomplete_dir.rename(final_dir) incomplete_dir.rename(final_dir)
final_log_path = final_dir / "meta" / "rsync.log"
final_meta_path = final_dir / "meta" / "meta.yaml"
meta["stats"] = _collect_run_stats(
log_path=final_log_path,
backup_root=Path(backup_root),
duration_seconds=meta["duration_seconds"],
snapshot_dir=final_dir / "data",
)
write_yaml_atomic(final_meta_path, meta)
prune_result = None prune_result = None
if prune: if prune:
@@ -285,6 +403,10 @@ def run_scheduled(
"host": host, "host": host,
"snapshot": str(final_dir), "snapshot": str(final_dir),
"base": str(base_dir) if base_dir else None, "base": str(base_dir) if base_dir else None,
"log": str(final_log_path),
"rsync": {"exit_code": result.exit_code}, "rsync": {"exit_code": result.exit_code},
"verbose_output": bool(verbose_output),
"duration_seconds": meta["duration_seconds"],
"stats": meta["stats"],
"prune": prune_result, "prune": prune_result,
} }

View File

@@ -1,53 +0,0 @@
from __future__ import annotations
from pathlib import Path
from typing import Any
import yaml
from ..errors import ConfigError, ValidationError
from ..validate import validate_dict
from .schemas import GLOBAL_SCHEMA, HOST_SCHEMA
def load_yaml_file(path: Path) -> dict[str, Any]:
if not path.exists():
raise ConfigError(f"Missing config file: {path}")
try:
raw = path.read_text(encoding="utf-8")
except OSError as e:
raise ConfigError(f"Cannot read config file: {path}: {e}") from e
try:
data = yaml.safe_load(raw)
except yaml.YAMLError as e:
raise ConfigError(f"Invalid YAML in {path}: {e}") from e
if data is None:
data = {}
if not isinstance(data, dict):
raise ConfigError(f"Config root must be a mapping in {path}")
return data
def load_global_config(path: Path) -> dict[str, Any]:
data = load_yaml_file(path)
try:
return validate_dict(data, GLOBAL_SCHEMA, path="global")
except ValidationError as e:
raise ConfigError(f"Invalid global config at {path}: {format_validation_error(e)}") from e
def load_host_config(path: Path) -> dict[str, Any]:
data = load_yaml_file(path)
try:
return validate_dict(data, HOST_SCHEMA, path="host")
except ValidationError as e:
raise ConfigError(f"Invalid host config at {path}: {format_validation_error(e)}") from e
def format_validation_error(err: ValidationError) -> str:
if err.path:
return f"{err.path}: {err}"
return str(err)

View File

@@ -83,7 +83,6 @@ OUTPUT_SCHEMA = Schema(
GLOBAL_SCHEMA = Schema( GLOBAL_SCHEMA = Schema(
fields={ fields={
"backup_root": FieldSpec(str, required=True), "backup_root": FieldSpec(str, required=True),
"pobsync_home": FieldSpec(str, required=False, default="/opt/pobsync"),
"ssh": FieldSpec(dict, required=False, schema=SSH_SCHEMA), "ssh": FieldSpec(dict, required=False, schema=SSH_SCHEMA),
"rsync": FieldSpec(dict, required=False, schema=RSYNC_SCHEMA), "rsync": FieldSpec(dict, required=False, schema=RSYNC_SCHEMA),
"defaults": FieldSpec(dict, required=False, schema=DEFAULTS_SCHEMA), "defaults": FieldSpec(dict, required=False, schema=DEFAULTS_SCHEMA),
@@ -95,7 +94,6 @@ GLOBAL_SCHEMA = Schema(
), ),
"logging": FieldSpec(dict, required=False, schema=LOGGING_SCHEMA), "logging": FieldSpec(dict, required=False, schema=LOGGING_SCHEMA),
"output": FieldSpec(dict, required=False, schema=OUTPUT_SCHEMA), "output": FieldSpec(dict, required=False, schema=OUTPUT_SCHEMA),
# Used by `init-host` as a convenience default
"retention_defaults": FieldSpec( "retention_defaults": FieldSpec(
dict, dict,
required=False, required=False,
@@ -131,4 +129,3 @@ HOST_SCHEMA = Schema(
}, },
allow_unknown=False, allow_unknown=False,
) )

View File

@@ -1,22 +1,8 @@
from __future__ import annotations from __future__ import annotations
from pathlib import Path
from typing import Any, Protocol from typing import Any, Protocol
from .load import load_global_config, load_host_config
from .merge import build_effective_config
class ConfigSource(Protocol): class ConfigSource(Protocol):
def effective_config_for_host(self, host: str) -> dict[str, Any]: def effective_config_for_host(self, host: str) -> dict[str, Any]:
"""Return the fully merged effective config for a host.""" """Return the fully merged effective config for a host."""
class FileConfigSource:
def __init__(self, prefix: Path) -> None:
self.prefix = prefix
def effective_config_for_host(self, host: str) -> dict[str, Any]:
global_cfg = load_global_config(self.prefix / "config" / "global.yaml")
host_cfg = load_host_config(self.prefix / "config" / "hosts" / f"{host}.yaml")
return build_effective_config(global_cfg, host_cfg)

View File

@@ -8,14 +8,6 @@ from pathlib import Path
class PobsyncPaths: class PobsyncPaths:
home: Path # usually /opt/pobsync home: Path # usually /opt/pobsync
@property
def config_dir(self) -> Path:
return self.home / "config"
@property
def hosts_dir(self) -> Path:
return self.config_dir / "hosts"
@property @property
def state_dir(self) -> Path: def state_dir(self) -> Path:
return self.home / "state" return self.home / "state"
@@ -28,11 +20,6 @@ class PobsyncPaths:
def logs_dir(self) -> Path: def logs_dir(self) -> Path:
return self.home / "logs" return self.home / "logs"
@property
def global_config_path(self) -> Path:
return self.config_dir / "global.yaml"
@property @property
def central_log_path(self) -> Path: def central_log_path(self) -> Path:
return self.logs_dir / "pobsync.log" return self.logs_dir / "pobsync.log"

View File

@@ -123,3 +123,50 @@ def build_retention_plan(
return RetentionResult(keep=keep, reasons=reasons) return RetentionResult(keep=keep, reasons=reasons)
def apply_base_protection(
*,
snapshots: Iterable[Snapshot],
keep: Set[str],
reasons: Dict[str, List[str]],
) -> Tuple[Set[str], Dict[str, List[str]]]:
"""
If a kept snapshot has a base (kind+dirname), also keep that base snapshot.
Hardlink snapshots remain readable without this, but keeping bases can make
future base selection and chain inspection easier.
"""
snapshot_list = list(snapshots)
index: Dict[Tuple[str, str], Snapshot] = {(snapshot.kind, snapshot.dirname): snapshot for snapshot in snapshot_list}
changed = True
while changed:
changed = False
for child_dirname in list(keep):
child = _find_snapshot_by_dirname(snapshot_list, child_dirname)
if child is None or not isinstance(child.base, dict):
continue
base_kind = child.base.get("kind")
base_dirname = child.base.get("dirname")
if not isinstance(base_kind, str) or not isinstance(base_dirname, str):
continue
base_snapshot = index.get((base_kind, base_dirname))
if base_snapshot is None or base_dirname in keep:
continue
keep.add(base_dirname)
reasons.setdefault(base_dirname, []).append(f"base-of:{child_dirname}")
changed = True
return keep, reasons
def _find_snapshot_by_dirname(snapshots: Iterable[Snapshot], dirname: str) -> Snapshot | None:
for kind in ("scheduled", "manual"):
for snapshot in snapshots:
if snapshot.kind == kind and snapshot.dirname == dirname:
return snapshot
return None

View File

@@ -1,16 +1,23 @@
from __future__ import annotations from __future__ import annotations
import os
import signal
import shlex import shlex
import subprocess import subprocess
import time
from dataclasses import dataclass from dataclasses import dataclass
from pathlib import Path from pathlib import Path
from typing import Sequence from typing import Callable, Sequence
DEFAULT_VERBOSE_OUTPUT_ARGS = ["--itemize-changes", "--info=flist2,progress2,stats2"]
@dataclass(frozen=True) @dataclass(frozen=True)
class RsyncResult: class RsyncResult:
exit_code: int exit_code: int
command: list[str] command: list[str]
cancelled: bool = False
def build_ssh_command(ssh_cfg: dict) -> list[str]: def build_ssh_command(ssh_cfg: dict) -> list[str]:
@@ -36,10 +43,14 @@ def build_rsync_command(
bwlimit_kbps: int, bwlimit_kbps: int,
extra_excludes: Sequence[str], extra_excludes: Sequence[str],
extra_includes: Sequence[str], extra_includes: Sequence[str],
verbose_output: bool = False,
) -> list[str]: ) -> list[str]:
cmd: list[str] = [rsync_binary] cmd: list[str] = [rsync_binary]
cmd.extend(list(rsync_args)) cmd.extend(list(rsync_args))
_append_stats_arg(cmd)
if dry_run or verbose_output:
_append_default_verbose_output_args(cmd)
# includes/excludes: keep it simple for now: # includes/excludes: keep it simple for now:
# - if includes are provided, user is responsible for correct rsync include logic. # - if includes are provided, user is responsible for correct rsync include logic.
@@ -66,7 +77,12 @@ def build_rsync_command(
def run_rsync(command: list[str], log_path: Path, timeout_seconds: int) -> RsyncResult: def run_rsync(
command: list[str],
log_path: Path,
timeout_seconds: int,
cancel_check: Callable[[], bool] | None = None,
) -> RsyncResult:
""" """
Run rsync and always write stdout/stderr to log_path. Run rsync and always write stdout/stderr to log_path.
@@ -77,17 +93,54 @@ def run_rsync(command: list[str], log_path: Path, timeout_seconds: int) -> Rsync
# Ensure the file exists early. # Ensure the file exists early.
log_path.touch(exist_ok=True) log_path.touch(exist_ok=True)
with log_path.open("ab") as f:
process = subprocess.Popen(command, stdout=f, stderr=subprocess.STDOUT, start_new_session=True)
started = time.monotonic()
while True:
exit_code = process.poll()
if exit_code is not None:
return RsyncResult(exit_code=exit_code, command=command)
if cancel_check is not None and cancel_check():
_terminate_process_group(process)
f.write(b"\n[pobsync] rsync cancelled\n")
return RsyncResult(exit_code=130, command=command, cancelled=True)
if timeout_seconds > 0 and time.monotonic() - started >= timeout_seconds:
_terminate_process_group(process)
f.write(b"\n[pobsync] rsync timed out\n")
return RsyncResult(exit_code=124, command=command)
time.sleep(1)
def _terminate_process_group(process: subprocess.Popen) -> None:
try: try:
with log_path.open("ab") as f: os.killpg(process.pid, signal.SIGTERM)
p = subprocess.run( process.wait(timeout=10)
command, except ProcessLookupError:
stdout=f, return
stderr=subprocess.STDOUT, except subprocess.TimeoutExpired:
timeout=timeout_seconds if timeout_seconds > 0 else None, os.killpg(process.pid, signal.SIGKILL)
) process.wait(timeout=10)
return RsyncResult(exit_code=p.returncode, command=command)
except subprocess.TimeoutExpired as e:
# Log timeout info and return a non-zero exit code. def _append_default_verbose_output_args(command: list[str]) -> None:
with log_path.open("ab") as f: if not _has_itemize_arg(command):
f.write(b"\n[pobsync] rsync timed out\n") command.append("--itemize-changes")
return RsyncResult(exit_code=124, command=command) if not any(arg.startswith("--info=") for arg in command):
command.append("--info=flist2,progress2,stats2")
def _append_stats_arg(command: list[str]) -> None:
if "--stats" not in command:
command.append("--stats")
def _has_itemize_arg(command: list[str]) -> bool:
for arg in command:
if arg == "--itemize-changes":
return True
if arg.startswith("-") and not arg.startswith("--") and "i" in arg:
return True
return False

238
src/pobsync/run_stats.py Normal file
View File

@@ -0,0 +1,238 @@
from __future__ import annotations
import os
import re
from pathlib import Path
from typing import Any
_COUNT_KEYS = {
"Number of files": "files_total",
"Number of created files": "files_created",
"Number of deleted files": "files_deleted",
"Number of regular files transferred": "files_transferred",
}
_BYTE_KEYS = {
"Total file size": "total_file_size_bytes",
"Total transferred file size": "total_transferred_file_size_bytes",
"Literal data": "literal_data_bytes",
"Matched data": "matched_data_bytes",
"File list size": "file_list_size_bytes",
"Total bytes sent": "bytes_sent",
"Total bytes received": "bytes_received",
}
_SIZE_UNITS = {
"": 1,
"b": 1,
"k": 1000,
"kb": 1000,
"m": 1000**2,
"mb": 1000**2,
"g": 1000**3,
"gb": 1000**3,
"t": 1000**4,
"tb": 1000**4,
"p": 1000**5,
"pb": 1000**5,
}
def parse_rsync_stats(text: str) -> dict[str, Any]:
stats: dict[str, Any] = {}
for raw_line in text.splitlines():
line = raw_line.strip()
if not line:
continue
_parse_colon_stat(line, stats)
_parse_sent_received(line, stats)
_parse_total_size_speedup(line, stats)
_add_derived_stats(stats)
return stats
def read_rsync_stats(log_path: Path) -> dict[str, Any]:
try:
text = log_path.read_text(encoding="utf-8", errors="replace")
except OSError:
return {}
return parse_rsync_stats(text)
def collect_storage_stats(*, backup_root: Path, snapshot_dir: Path | None = None) -> dict[str, Any]:
stats: dict[str, Any] = {
"backup_root": str(backup_root),
}
capacity = filesystem_capacity(backup_root)
if capacity:
stats["capacity"] = capacity
if snapshot_dir is not None:
snapshot_usage = tree_usage(snapshot_dir)
if snapshot_usage:
stats["snapshot"] = {
"path": str(snapshot_dir),
**snapshot_usage,
}
return stats
def filesystem_capacity(path: Path) -> dict[str, Any]:
try:
stat = path.stat()
statvfs = os.statvfs(path)
except OSError:
return {}
total = int(statvfs.f_frsize * statvfs.f_blocks)
available = int(statvfs.f_frsize * statvfs.f_bavail)
free = int(statvfs.f_frsize * statvfs.f_bfree)
used = max(total - free, 0)
return {
"path": str(path),
"total_bytes": total,
"used_bytes": used,
"free_bytes": free,
"available_bytes": available,
"used_percent": round((used / total) * 100, 2) if total else 0.0,
"device": getattr(stat, "st_dev", None),
}
def tree_usage(path: Path) -> dict[str, Any]:
apparent_size = 0
allocated_size = 0
files = 0
directories = 0
hardlinked_files = 0
hardlinked_apparent_size = 0
seen_allocated_inodes: set[tuple[int, int]] = set()
try:
root_stat = path.lstat()
except OSError:
return {}
if path.is_file():
files = 1
apparent_size = root_stat.st_size
allocated_size = int(getattr(root_stat, "st_blocks", 0) * 512)
if root_stat.st_nlink > 1:
hardlinked_files = 1
hardlinked_apparent_size = root_stat.st_size
else:
for current_root, dirnames, filenames in path.walk():
directories += len(dirnames)
for filename in filenames:
file_path = current_root / filename
try:
file_stat = file_path.lstat()
except OSError:
continue
files += 1
apparent_size += file_stat.st_size
inode_key = (file_stat.st_dev, file_stat.st_ino)
if inode_key not in seen_allocated_inodes:
allocated_size += int(getattr(file_stat, "st_blocks", 0) * 512)
seen_allocated_inodes.add(inode_key)
if file_stat.st_nlink > 1:
hardlinked_files += 1
hardlinked_apparent_size += file_stat.st_size
return {
"path": str(path),
"apparent_size_bytes": int(apparent_size),
"allocated_size_bytes": int(allocated_size),
"files": files,
"directories": directories,
"hardlinked_files": hardlinked_files,
"hardlinked_apparent_size_bytes": int(hardlinked_apparent_size),
"hardlink_apparent_ratio": round(hardlinked_apparent_size / apparent_size, 4) if apparent_size else 0.0,
}
def _parse_colon_stat(line: str, stats: dict[str, Any]) -> None:
if ":" not in line:
return
label, value = line.split(":", 1)
label = label.strip()
value = value.strip()
if label in _COUNT_KEYS:
parsed = _parse_int_prefix(value)
if parsed is not None:
stats[_COUNT_KEYS[label]] = parsed
elif label in _BYTE_KEYS:
parsed = _parse_byte_value(value)
if parsed is not None:
stats[_BYTE_KEYS[label]] = parsed
def _parse_sent_received(line: str, stats: dict[str, Any]) -> None:
match = re.search(
r"sent\s+(?P<sent>[\d,]+(?:\.\d+)?\s*[A-Za-z]*)\s+bytes\s+received\s+"
r"(?P<received>[\d,]+(?:\.\d+)?\s*[A-Za-z]*)\s+bytes\s+"
r"(?P<rate>[\d,]+(?:\.\d+)?\s*[A-Za-z]*)\s+bytes/sec",
line,
)
if not match:
return
sent = _parse_byte_value(match.group("sent"))
received = _parse_byte_value(match.group("received"))
rate = _parse_byte_value(match.group("rate"))
if sent is not None:
stats["bytes_sent"] = sent
if received is not None:
stats["bytes_received"] = received
if rate is not None:
stats["bytes_per_second"] = rate
def _parse_total_size_speedup(line: str, stats: dict[str, Any]) -> None:
match = re.search(
r"total size is\s+(?P<size>[\d,]+(?:\.\d+)?\s*[A-Za-z]*)\s+speedup is\s+(?P<speedup>[\d,]+(?:\.\d+)?)",
line,
)
if not match:
return
total_size = _parse_byte_value(match.group("size"))
if total_size is not None:
stats["total_file_size_bytes"] = total_size
stats["speedup"] = float(match.group("speedup").replace(",", ""))
def _add_derived_stats(stats: dict[str, Any]) -> None:
sent = stats.get("bytes_sent")
received = stats.get("bytes_received")
if isinstance(sent, int) and isinstance(received, int):
stats["bytes_sent_received"] = sent + received
literal = stats.get("literal_data_bytes")
matched = stats.get("matched_data_bytes")
if isinstance(literal, int) and isinstance(matched, int):
basis_total = literal + matched
stats["link_dest_estimated_savings_bytes"] = matched
stats["link_dest_estimated_savings_ratio"] = round(matched / basis_total, 4) if basis_total else 0.0
def _parse_int_prefix(value: str) -> int | None:
match = re.match(r"([\d,]+)", value)
if not match:
return None
return int(match.group(1).replace(",", ""))
def _parse_byte_value(value: str) -> int | None:
match = re.match(r"([\d,]+(?:\.\d+)?)\s*([A-Za-z]*)", value.strip())
if not match:
return None
number = float(match.group(1).replace(",", ""))
unit = match.group(2).lower()
multiplier = _SIZE_UNITS.get(unit)
if multiplier is None:
return int(number)
return int(number * multiplier)

View File

@@ -1,8 +1,32 @@
from __future__ import annotations from __future__ import annotations
from django.contrib import admin from django.contrib import admin
from django.db.models import Count
from django.urls import reverse
from django.utils.html import format_html
from django.utils.http import urlencode
from .models import BackupRun, GlobalConfig, HostConfig, ScheduleConfig, SnapshotRecord from .models import BackupRun, GlobalConfig, HostConfig, PurgedSnapshot, ScheduleConfig, SnapshotRecord, SshCredential
@admin.register(SshCredential)
class SshCredentialAdmin(admin.ModelAdmin):
list_display = ("name", "key_type", "generated", "has_public_key", "has_known_hosts", "updated_at")
readonly_fields = ("created_at", "updated_at", "fingerprint")
search_fields = ("name", "notes")
fieldsets = (
(None, {"fields": ("name", "key_type", "generated", "key_path", "fingerprint")}),
("Key material", {"fields": ("private_key", "public_key", "known_hosts", "notes")}),
("Timestamps", {"fields": ("created_at", "updated_at"), "classes": ("collapse",)}),
)
@admin.display(boolean=True, description="Public key")
def has_public_key(self, obj: SshCredential) -> bool:
return bool(obj.public_key.strip())
@admin.display(boolean=True, description="Known hosts")
def has_known_hosts(self, obj: SshCredential) -> bool:
return bool(obj.known_hosts.strip())
@admin.register(GlobalConfig) @admin.register(GlobalConfig)
@@ -10,8 +34,8 @@ class GlobalConfigAdmin(admin.ModelAdmin):
list_display = ("name", "backup_root", "ssh_user", "ssh_port", "updated_at") list_display = ("name", "backup_root", "ssh_user", "ssh_port", "updated_at")
readonly_fields = ("created_at", "updated_at") readonly_fields = ("created_at", "updated_at")
fieldsets = ( fieldsets = (
(None, {"fields": ("name", "backup_root", "pobsync_home")}), (None, {"fields": ("name", "backup_root")}),
("SSH", {"fields": ("ssh_user", "ssh_port", "ssh_options")}), ("SSH", {"fields": ("default_ssh_credential", "ssh_user", "ssh_port", "ssh_options")}),
( (
"Rsync", "Rsync",
{ {
@@ -26,45 +50,147 @@ class GlobalConfigAdmin(admin.ModelAdmin):
), ),
("Defaults", {"fields": ("default_source_root", "default_destination_subdir", "excludes_default")}), ("Defaults", {"fields": ("default_source_root", "default_destination_subdir", "excludes_default")}),
("Retention defaults", {"fields": ("retention_daily", "retention_weekly", "retention_monthly", "retention_yearly")}), ("Retention defaults", {"fields": ("retention_daily", "retention_weekly", "retention_monthly", "retention_yearly")}),
("Legacy JSON", {"fields": ("data",), "classes": ("collapse",)}),
("Timestamps", {"fields": ("created_at", "updated_at"), "classes": ("collapse",)}), ("Timestamps", {"fields": ("created_at", "updated_at"), "classes": ("collapse",)}),
) )
@admin.register(HostConfig) @admin.register(HostConfig)
class HostConfigAdmin(admin.ModelAdmin): class HostConfigAdmin(admin.ModelAdmin):
list_display = ("host", "address", "enabled", "updated_at") list_display = (
"host",
"address",
"enabled",
"schedule_state",
"snapshot_count_link",
"backup_run_count_link",
"latest_run_state",
"updated_at",
)
list_filter = ("enabled",) list_filter = ("enabled",)
search_fields = ("host", "address") search_fields = ("host", "address")
readonly_fields = ("created_at", "updated_at") readonly_fields = ("created_at", "updated_at")
fieldsets = ( fieldsets = (
(None, {"fields": ("host", "address", "enabled")}), (None, {"fields": ("host", "address", "enabled")}),
("SSH override", {"fields": ("ssh_user", "ssh_port")}), ("SSH override", {"fields": ("ssh_credential", "ssh_user", "ssh_port")}),
("Source", {"fields": ("source_root", "includes", "excludes_add", "excludes_replace")}), ("Source", {"fields": ("source_root", "includes", "excludes_add", "excludes_replace")}),
("Rsync override", {"fields": ("rsync_extra_args",)}), ("Rsync override", {"fields": ("rsync_extra_args",)}),
("Retention", {"fields": ("retention_daily", "retention_weekly", "retention_monthly", "retention_yearly")}), ("Retention", {"fields": ("retention_daily", "retention_weekly", "retention_monthly", "retention_yearly")}),
("Legacy JSON", {"fields": ("config",), "classes": ("collapse",)}), ("Runtime state", {"fields": ("config",), "classes": ("collapse",)}),
("Timestamps", {"fields": ("created_at", "updated_at"), "classes": ("collapse",)}), ("Timestamps", {"fields": ("created_at", "updated_at"), "classes": ("collapse",)}),
) )
def get_queryset(self, request):
return super().get_queryset(request).annotate(
snapshot_count=Count("snapshots", distinct=True),
backup_run_count=Count("runs", distinct=True),
)
@admin.display(description="Schedule")
def schedule_state(self, obj: HostConfig) -> str:
try:
schedule = obj.schedule
except ScheduleConfig.DoesNotExist:
return "none"
if not schedule.enabled:
return "disabled"
return schedule.cron_expr
@admin.display(description="Snapshots", ordering="snapshot_count")
def snapshot_count_link(self, obj: HostConfig) -> str:
count = getattr(obj, "snapshot_count", None)
if count is None:
count = obj.snapshots.count()
url = _admin_changelist_url("pobsync_backend", "snapshotrecord", {"host__id__exact": obj.pk})
return format_html('<a href="{}">{}</a>', url, count)
@admin.display(description="Runs", ordering="backup_run_count")
def backup_run_count_link(self, obj: HostConfig) -> str:
count = getattr(obj, "backup_run_count", None)
if count is None:
count = obj.runs.count()
url = _admin_changelist_url("pobsync_backend", "backuprun", {"host__id__exact": obj.pk})
return format_html('<a href="{}">{}</a>', url, count)
@admin.display(description="Latest run")
def latest_run_state(self, obj: HostConfig) -> str:
latest = obj.runs.order_by("-created_at").first()
if latest is None:
return "none"
return f"{latest.status} {latest.started_at:%Y-%m-%d %H:%M}" if latest.started_at else latest.status
@admin.register(BackupRun) @admin.register(BackupRun)
class BackupRunAdmin(admin.ModelAdmin): class BackupRunAdmin(admin.ModelAdmin):
list_display = ("host", "run_type", "status", "started_at", "ended_at", "snapshot_path") list_display = ("host", "run_type", "status", "started_at", "ended_at", "snapshot_link")
list_filter = ("run_type", "status", "started_at") list_filter = ("run_type", "status", "started_at")
search_fields = ("host__host", "snapshot_path") search_fields = ("host__host", "snapshot_path", "snapshot__dirname", "snapshot__path")
autocomplete_fields = ("snapshot",)
list_select_related = ("host", "snapshot")
date_hierarchy = "started_at"
@admin.display(description="Snapshot", ordering="snapshot__dirname")
def snapshot_link(self, obj: BackupRun) -> str:
if obj.snapshot is None:
return ""
url = reverse("admin:pobsync_backend_snapshotrecord_change", args=[obj.snapshot.pk])
return format_html('<a href="{}">{}</a>', url, obj.snapshot.dirname)
@admin.register(SnapshotRecord) @admin.register(SnapshotRecord)
class SnapshotRecordAdmin(admin.ModelAdmin): class SnapshotRecordAdmin(admin.ModelAdmin):
list_display = ("host", "kind", "dirname", "status", "started_at", "discovered_at") list_display = ("host", "kind", "dirname", "status", "base_link", "backup_run_count_link", "started_at", "discovered_at")
list_filter = ("kind", "status", "started_at", "discovered_at") list_filter = ("kind", "status", "base_kind", "started_at", "discovered_at")
search_fields = ("host__host", "dirname", "path") search_fields = (
"host__host",
"dirname",
"path",
"base__dirname",
"base_path",
"base_snapshot_id",
)
autocomplete_fields = ("base",)
list_select_related = ("host", "base")
readonly_fields = ("discovered_at",) readonly_fields = ("discovered_at",)
def get_queryset(self, request):
return super().get_queryset(request).annotate(backup_run_count=Count("backup_runs"))
@admin.display(description="Base", ordering="base__dirname")
def base_link(self, obj: SnapshotRecord) -> str:
if obj.base is not None:
url = reverse("admin:pobsync_backend_snapshotrecord_change", args=[obj.base.pk])
return format_html('<a href="{}">{}</a>', url, obj.base.dirname)
if obj.base_dirname:
return obj.base_dirname
return ""
@admin.display(description="Runs", ordering="backup_run_count")
def backup_run_count_link(self, obj: SnapshotRecord) -> str:
count = getattr(obj, "backup_run_count", None)
if count is None:
count = obj.backup_runs.count()
url = _admin_changelist_url("pobsync_backend", "backuprun", {"snapshot__id__exact": obj.pk})
return format_html('<a href="{}">{}</a>', url, count)
@admin.register(PurgedSnapshot)
class PurgedSnapshotAdmin(admin.ModelAdmin):
list_display = ("host_name", "kind", "dirname", "action", "reason", "triggered_by", "purged_at")
list_filter = ("action", "kind", "purged_at")
search_fields = ("host_name", "dirname", "path", "reason", "triggered_by")
list_select_related = ("host",)
readonly_fields = ("purged_at",)
date_hierarchy = "purged_at"
@admin.register(ScheduleConfig) @admin.register(ScheduleConfig)
class ScheduleConfigAdmin(admin.ModelAdmin): class ScheduleConfigAdmin(admin.ModelAdmin):
list_display = ("host", "cron_expr", "enabled", "prune", "last_status", "last_started_at", "updated_at") list_display = ("host", "cron_expr", "enabled", "prune", "last_status", "last_started_at", "updated_at")
list_filter = ("enabled", "prune", "last_status") list_filter = ("enabled", "prune", "last_status")
search_fields = ("host__host", "cron_expr") search_fields = ("host__host", "cron_expr")
list_select_related = ("host",)
def _admin_changelist_url(app_label: str, model_name: str, params: dict[str, object]) -> str:
base_url = reverse(f"admin:{app_label}_{model_name}_changelist")
return f"{base_url}?{urlencode(params)}"

194
src/pobsync_backend/api.py Normal file
View File

@@ -0,0 +1,194 @@
from __future__ import annotations
from typing import Any
from django.contrib.admin.views.decorators import staff_member_required
from django.db import connection
from django.db.models import Count
from django.http import JsonResponse
from django.utils import timezone
from .models import BackupRun, HostConfig, ScheduleConfig, SnapshotRecord
@staff_member_required
def api_index(request) -> JsonResponse:
return JsonResponse(
{
"ok": True,
"endpoints": {
"status": request.build_absolute_uri("/api/status/"),
"hosts": request.build_absolute_uri("/api/hosts/"),
"snapshots": request.build_absolute_uri("/api/snapshots/"),
"runs": request.build_absolute_uri("/api/runs/"),
},
}
)
@staff_member_required
def status(request) -> JsonResponse:
latest_run = BackupRun.objects.select_related("host", "snapshot").order_by("-created_at").first()
latest_schedule = ScheduleConfig.objects.select_related("host").order_by("-last_started_at", "-updated_at").first()
return JsonResponse(
{
"ok": True,
"generated_at": timezone.now().isoformat(),
"database": {
"vendor": connection.vendor,
"engine": connection.settings_dict["ENGINE"],
},
"counts": {
"hosts": HostConfig.objects.count(),
"enabled_hosts": HostConfig.objects.filter(enabled=True).count(),
"schedules": ScheduleConfig.objects.count(),
"enabled_schedules": ScheduleConfig.objects.filter(enabled=True).count(),
"snapshots": SnapshotRecord.objects.count(),
"runs": BackupRun.objects.count(),
"running_runs": BackupRun.objects.filter(status=BackupRun.Status.RUNNING).count(),
"failed_runs": BackupRun.objects.filter(status=BackupRun.Status.FAILED).count(),
},
"latest_run": None if latest_run is None else _run_payload(latest_run),
"latest_schedule": None if latest_schedule is None else _schedule_payload(latest_schedule),
}
)
@staff_member_required
def hosts(request) -> JsonResponse:
host_qs = (
HostConfig.objects.annotate(snapshot_count=Count("snapshots", distinct=True), run_count=Count("runs", distinct=True))
.select_related("schedule")
.order_by("host")
)
return JsonResponse({"ok": True, "hosts": [_host_payload(host) for host in host_qs]})
@staff_member_required
def snapshots(request) -> JsonResponse:
snapshot_qs = SnapshotRecord.objects.select_related("host", "base").order_by("host__host", "-started_at", "dirname")
host_filter = request.GET.get("host")
kind_filter = request.GET.get("kind")
if host_filter:
snapshot_qs = snapshot_qs.filter(host__host=host_filter)
if kind_filter:
snapshot_qs = snapshot_qs.filter(kind=kind_filter)
limit = _limit_from_request(request)
return JsonResponse({"ok": True, "snapshots": [_snapshot_payload(snapshot) for snapshot in snapshot_qs[:limit]]})
@staff_member_required
def runs(request) -> JsonResponse:
run_qs = BackupRun.objects.select_related("host", "snapshot").order_by("-created_at")
host_filter = request.GET.get("host")
status_filter = request.GET.get("status")
if host_filter:
run_qs = run_qs.filter(host__host=host_filter)
if status_filter:
run_qs = run_qs.filter(status=status_filter)
limit = _limit_from_request(request)
return JsonResponse({"ok": True, "runs": [_run_payload(run) for run in run_qs[:limit]]})
def _host_payload(host: HostConfig) -> dict[str, Any]:
try:
schedule = host.schedule
except ScheduleConfig.DoesNotExist:
schedule = None
return {
"host": host.host,
"address": host.address,
"enabled": host.enabled,
"snapshot_count": host.snapshot_count,
"run_count": host.run_count,
"retention": {
"daily": host.retention_daily,
"weekly": host.retention_weekly,
"monthly": host.retention_monthly,
"yearly": host.retention_yearly,
},
"schedule": None
if schedule is None
else _schedule_payload(schedule),
}
def _snapshot_payload(snapshot: SnapshotRecord) -> dict[str, Any]:
return {
"host": snapshot.host.host,
"kind": snapshot.kind,
"dirname": snapshot.dirname,
"path": snapshot.path,
"status": snapshot.status,
"started_at": _iso(snapshot.started_at),
"ended_at": _iso(snapshot.ended_at),
"discovered_at": _iso(snapshot.discovered_at),
"base": _base_payload(snapshot),
}
def _base_payload(snapshot: SnapshotRecord) -> dict[str, Any] | None:
if snapshot.base is not None:
return {
"kind": snapshot.base.kind,
"dirname": snapshot.base.dirname,
"path": snapshot.base.path,
"resolved": True,
}
if snapshot.base_kind and snapshot.base_dirname:
return {
"kind": snapshot.base_kind,
"dirname": snapshot.base_dirname,
"path": snapshot.base_path,
"snapshot_id": snapshot.base_snapshot_id,
"resolved": False,
}
return None
def _run_payload(run: BackupRun) -> dict[str, Any]:
return {
"id": run.pk,
"host": run.host.host,
"run_type": run.run_type,
"status": run.status,
"started_at": _iso(run.started_at),
"ended_at": _iso(run.ended_at),
"snapshot": None
if run.snapshot is None
else {
"kind": run.snapshot.kind,
"dirname": run.snapshot.dirname,
"path": run.snapshot.path,
},
"snapshot_path": run.snapshot_path,
"base_path": run.base_path,
"rsync_exit_code": run.rsync_exit_code,
}
def _schedule_payload(schedule: ScheduleConfig) -> dict[str, Any]:
return {
"host": schedule.host.host,
"cron_expr": schedule.cron_expr,
"enabled": schedule.enabled,
"prune": schedule.prune,
"last_due_key": schedule.last_due_key,
"last_status": schedule.last_status,
"last_started_at": _iso(schedule.last_started_at),
"last_finished_at": _iso(schedule.last_finished_at),
}
def _limit_from_request(request, *, default: int = 100, maximum: int = 500) -> int:
value = request.GET.get("limit", str(default))
try:
limit = int(value)
except ValueError:
return default
return max(1, min(limit, maximum))
def _iso(value) -> str | None:
return value.isoformat() if value is not None else None

View File

@@ -0,0 +1,354 @@
from __future__ import annotations
import os
import socket
from datetime import timedelta, timezone as datetime_timezone
from pathlib import Path
from django.db import transaction
from django.utils import timezone
from pobsync.commands.run_scheduled import DEFAULT_DRY_RUN_TIMEOUT_SECONDS, classify_rsync_failure, dry_run_log_path, run_scheduled
from pobsync_backend.config_source import DjangoConfigSource
from pobsync_backend.models import BackupRun, HostConfig
from pobsync_backend.retention import run_sql_retention_apply
from pobsync_backend.snapshot_discovery import infer_snapshot_kind, upsert_snapshot_record
def queue_backup_run(
*,
host: HostConfig,
run_type: str = BackupRun.RunType.MANUAL,
dry_run: bool = False,
verbose_output: bool = False,
prune: bool = False,
prune_max_delete: int = 10,
prune_protect_bases: bool = False,
) -> BackupRun:
return BackupRun.objects.create(
host=host,
run_type=run_type,
status=BackupRun.Status.QUEUED,
result={
"requested": {
"dry_run": bool(dry_run),
"verbose_output": bool(dry_run or verbose_output),
"prune": bool(prune),
"prune_max_delete": int(prune_max_delete),
"prune_protect_bases": bool(prune_protect_bases),
}
},
)
def execute_backup_run(
*,
run: BackupRun,
prefix: Path,
dry_run: bool = False,
verbose_output: bool = False,
prune: bool = False,
prune_max_delete: int = 10,
prune_protect_bases: bool = False,
) -> BackupRun:
run.status = BackupRun.Status.RUNNING
run.started_at = run.started_at or timezone.now()
run.result = _running_result(run=run, dry_run=bool(dry_run))
run.save(update_fields=["status", "started_at", "result"])
try:
result = run_scheduled(
prefix=prefix,
host=run.host.host,
dry_run=bool(dry_run),
prune=False,
config_source=DjangoConfigSource(),
run_id=run.id,
cancel_check=lambda: _run_cancel_requested(run.id),
verbose_output=bool(dry_run or verbose_output),
)
except Exception as exc:
run.refresh_from_db()
run.status = BackupRun.Status.CANCELLED if run.status == BackupRun.Status.CANCELLED else BackupRun.Status.FAILED
run.ended_at = timezone.now()
run.result = {
**(run.result if isinstance(run.result, dict) else {}),
"ok": False,
"error": str(exc),
"type": type(exc).__name__,
}
run.save(update_fields=["status", "ended_at", "result"])
raise
run.refresh_from_db()
if result.get("cancelled") or run.status == BackupRun.Status.CANCELLED:
run.status = BackupRun.Status.CANCELLED
else:
run.status = BackupRun.Status.SUCCESS if result.get("ok") else BackupRun.Status.FAILED
run.ended_at = timezone.now()
run.snapshot_path = str(result.get("snapshot") or "")
run.base_path = str(result.get("base") or "")
rsync = result.get("rsync") if isinstance(result.get("rsync"), dict) else {}
run.rsync_exit_code = rsync.get("exit_code")
run.result = result
snapshot_record = None
if run.snapshot_path:
snapshot_path = Path(run.snapshot_path)
try:
kind = infer_snapshot_kind(snapshot_path)
snapshot_record, _created = upsert_snapshot_record(host=run.host, kind=kind, snapshot_dir=snapshot_path)
except ValueError:
snapshot_record = None
if result.get("ok") and not result.get("dry_run") and prune:
try:
result["prune"] = run_sql_retention_apply(
prefix=prefix,
host=run.host.host,
kind="scheduled",
protect_bases=bool(prune_protect_bases),
yes=True,
max_delete=int(prune_max_delete),
action=run.run_type,
acquire_lock=False,
)
except Exception as exc:
result["prune"] = {"ok": False, "error": str(exc), "type": type(exc).__name__}
run.status = BackupRun.Status.WARNING
run.result = result
run.snapshot = snapshot_record
run.save(
update_fields=[
"status",
"ended_at",
"snapshot_path",
"snapshot",
"base_path",
"rsync_exit_code",
"result",
],
)
run.snapshot = snapshot_record
run.result = result
run.save(
update_fields=[
"status",
"ended_at",
"snapshot_path",
"snapshot",
"base_path",
"rsync_exit_code",
"result",
],
)
return run
def claim_next_queued_run() -> BackupRun | None:
with transaction.atomic():
run = (
BackupRun.objects.select_related("host")
.filter(status=BackupRun.Status.QUEUED, host__enabled=True)
.order_by("created_at", "id")
.first()
)
if run is None:
return None
run.status = BackupRun.Status.RUNNING
run.started_at = timezone.now()
run.save(update_fields=["status", "started_at"])
return run
def reconcile_running_runs(*, grace_seconds: int = 300, stale_worker_seconds: int = 24 * 60 * 60) -> int:
reconciled = 0
for run in BackupRun.objects.select_related("host").filter(status=BackupRun.Status.RUNNING).order_by("started_at", "id"):
if _reconcile_running_run(run=run, grace_seconds=grace_seconds, stale_worker_seconds=stale_worker_seconds):
reconciled += 1
return reconciled
def requested_options(run: BackupRun) -> dict[str, object]:
requested = run.result.get("requested") if isinstance(run.result, dict) else None
if not isinstance(requested, dict):
return {}
return requested
def _running_result(*, run: BackupRun, dry_run: bool) -> dict[str, object]:
result = dict(run.result) if isinstance(run.result, dict) else {}
execution = {
**_worker_execution_details(),
"started_at": (run.started_at or timezone.now()).isoformat(),
"heartbeat_at": timezone.now().isoformat(),
}
if dry_run:
execution["log"] = str(dry_run_log_path(run.host.host, run_id=run.id))
result["execution"] = execution
return result
def _run_cancel_requested(run_id: int) -> bool:
try:
run = BackupRun.objects.only("id", "status", "result").get(id=run_id)
except BackupRun.DoesNotExist:
return True
if run.status == BackupRun.Status.CANCELLED:
return True
if run.status == BackupRun.Status.RUNNING:
_refresh_run_heartbeat(run)
return False
def _reconcile_running_run(*, run: BackupRun, grace_seconds: int, stale_worker_seconds: int) -> bool:
result = run.result if isinstance(run.result, dict) else {}
requested = result.get("requested") if isinstance(result.get("requested"), dict) else {}
stale_worker = _running_worker_timed_out(run=run, stale_worker_seconds=stale_worker_seconds)
if not requested.get("dry_run"):
if stale_worker:
result.update(
{
"ok": False,
"host": run.host.host,
"failure": {
"category": "worker",
"message": "The worker heartbeat stopped before the run finished.",
"hint": "Check pobsync-worker.service logs before retrying the backup.",
},
}
)
run.status = BackupRun.Status.FAILED
run.ended_at = timezone.now()
run.result = result
run.save(update_fields=["status", "ended_at", "result"])
return True
return False
log_path = _execution_log_path(result)
log_tail = _read_log_tail(log_path) if log_path is not None else []
terminal_log = _terminal_rsync_log(log_tail)
timed_out = _running_dry_run_timed_out(run=run, grace_seconds=grace_seconds)
if not terminal_log and not timed_out and not stale_worker:
return False
exit_code = _exit_code_from_log(log_tail) or (124 if timed_out or stale_worker else 255)
failure = classify_rsync_failure(exit_code, log_tail)
if stale_worker and not terminal_log:
failure = {
"category": "worker",
"message": "The worker heartbeat stopped before the dry-run finished.",
"hint": "Check pobsync-worker.service logs before retrying the dry-run.",
}
result.update(
{
"ok": False,
"dry_run": True,
"host": run.host.host,
"base": result.get("base"),
"log": str(log_path) if log_path else "",
"failure": failure,
"rsync": {
**(result.get("rsync") if isinstance(result.get("rsync"), dict) else {}),
"exit_code": exit_code,
"log_tail": log_tail,
},
}
)
run.status = BackupRun.Status.FAILED
run.ended_at = timezone.now()
run.rsync_exit_code = exit_code
run.result = result
run.save(update_fields=["status", "ended_at", "rsync_exit_code", "result"])
return True
def _worker_execution_details() -> dict[str, object]:
return {
"worker_pid": os.getpid(),
"worker_host": socket.gethostname(),
"claimed_at": timezone.now().isoformat(),
}
def _refresh_run_heartbeat(run: BackupRun, *, interval_seconds: int = 30) -> None:
result = run.result if isinstance(run.result, dict) else {}
execution = result.get("execution") if isinstance(result.get("execution"), dict) else {}
heartbeat_at = _parse_iso_datetime(execution.get("heartbeat_at"))
if heartbeat_at is not None and timezone.now() < heartbeat_at + timedelta(seconds=interval_seconds):
return
result["execution"] = {
**execution,
"worker_pid": os.getpid(),
"worker_host": socket.gethostname(),
"heartbeat_at": timezone.now().isoformat(),
}
run.result = result
run.save(update_fields=["result"])
def _execution_log_path(result: dict[str, object]) -> Path | None:
execution = result.get("execution") if isinstance(result.get("execution"), dict) else {}
log = execution.get("log") or result.get("log")
if not isinstance(log, str) or not log:
return None
return Path(log)
def _read_log_tail(log_path: Path | None, *, max_lines: int = 40) -> list[str]:
if log_path is None:
return []
try:
return log_path.read_text(encoding="utf-8", errors="replace").splitlines()[-max_lines:]
except OSError:
return []
def _terminal_rsync_log(log_tail: list[str]) -> bool:
return any(line.startswith("rsync error:") for line in log_tail)
def _exit_code_from_log(log_tail: list[str]) -> int | None:
for line in reversed(log_tail):
if "code 255" in line:
return 255
if "code 124" in line:
return 124
if "code 12" in line:
return 12
return None
def _running_dry_run_timed_out(*, run: BackupRun, grace_seconds: int) -> bool:
if run.started_at is None:
return False
result = run.result if isinstance(run.result, dict) else {}
timeout_seconds = result.get("timeout_seconds")
if not isinstance(timeout_seconds, int) or timeout_seconds <= 0:
timeout_seconds = DEFAULT_DRY_RUN_TIMEOUT_SECONDS
return timezone.now() >= run.started_at + timedelta(seconds=timeout_seconds + grace_seconds)
def _running_worker_timed_out(*, run: BackupRun, stale_worker_seconds: int) -> bool:
if stale_worker_seconds <= 0:
return False
result = run.result if isinstance(run.result, dict) else {}
execution = result.get("execution") if isinstance(result.get("execution"), dict) else {}
heartbeat_at = _parse_iso_datetime(execution.get("heartbeat_at"))
if heartbeat_at is None:
heartbeat_at = run.started_at
if heartbeat_at is None:
return False
return timezone.now() >= heartbeat_at + timedelta(seconds=stale_worker_seconds)
def _parse_iso_datetime(value: object):
if not isinstance(value, str) or not value:
return None
try:
parsed = timezone.datetime.fromisoformat(value)
except ValueError:
return None
if timezone.is_naive(parsed):
return timezone.make_aware(parsed, timezone=datetime_timezone.utc)
return parsed

View File

@@ -0,0 +1,201 @@
from __future__ import annotations
import os
import shutil
from pathlib import Path
from django.conf import settings
from .models import GlobalConfig, HostConfig, SshCredential
from .self_check import SelfCheck
from .ssh_keys import identity_path
CRITICAL_ROOT_EXCLUDES = ("/proc/***", "/sys/***", "/dev/***", "/run/***", "/tmp/***")
def collect_global_config_checks(global_config: GlobalConfig) -> list[SelfCheck]:
checks = [
_absolute_path_check("Global backup root", global_config.backup_root),
_absolute_path_check("Runtime state root", settings.POBSYNC_HOME),
_runtime_backup_root_check(global_config),
_rsync_binary_check(global_config.rsync_binary),
_rsync_recursion_check(
"Global rsync recursion",
[*list(global_config.rsync_args or []), *list(global_config.rsync_extra_args or [])],
),
_source_root_check("Global source root", global_config.default_source_root),
_root_excludes_check(global_config.default_source_root, list(global_config.excludes_default or [])),
_retention_check(
"Global retention",
global_config.retention_daily,
global_config.retention_weekly,
global_config.retention_monthly,
global_config.retention_yearly,
),
_ssh_port_check("Global SSH port", global_config.ssh_port),
_credential_check("Global SSH credential", global_config.default_ssh_credential),
]
return checks
def collect_effective_host_config_checks(host: HostConfig, global_config: GlobalConfig) -> list[SelfCheck]:
source_root = host.source_root or global_config.default_source_root
ssh_user = host.ssh_user or global_config.ssh_user
ssh_port = host.ssh_port or global_config.ssh_port
credential = host.ssh_credential or global_config.default_ssh_credential
if host.excludes_replace is not None:
excludes = list(host.excludes_replace)
else:
excludes = [*list(global_config.excludes_default or []), *list(host.excludes_add or [])]
rsync_args = [
*list(global_config.rsync_args or []),
*list(global_config.rsync_extra_args or []),
*list(host.rsync_extra_args or []),
]
checks = [
_source_root_check("Host effective source root", source_root),
_ssh_user_check(ssh_user),
_ssh_port_check("Host effective SSH port", ssh_port),
_credential_check("Host effective SSH credential", credential),
_rsync_recursion_check("Host effective rsync recursion", rsync_args),
_root_excludes_check(source_root, excludes, host=host),
_includes_check(host),
_retention_check(
"Host retention",
host.retention_daily,
host.retention_weekly,
host.retention_monthly,
host.retention_yearly,
),
]
return checks
def has_recursive_rsync_arg(args: list[str]) -> bool:
for arg in args:
if arg in {"--archive", "--recursive"}:
return True
if arg.startswith("-") and not arg.startswith("--") and any(flag in arg for flag in ("a", "r")):
return True
return False
def _absolute_path_check(name: str, value: str) -> SelfCheck:
path = Path(value)
if not value:
return SelfCheck(name, "failed", "Path is empty.")
if not path.is_absolute():
return SelfCheck(name, "failed", f"{value} is not absolute.")
return SelfCheck(name, "ok", value)
def _runtime_backup_root_check(global_config: GlobalConfig) -> SelfCheck:
if global_config.backup_root == settings.POBSYNC_BACKUP_ROOT:
return SelfCheck("Runtime backup root", "ok", global_config.backup_root)
return SelfCheck(
"Runtime backup root",
"warning",
"Database backup root differs from the runtime backup root.",
f"database={global_config.backup_root} runtime={settings.POBSYNC_BACKUP_ROOT}",
)
def _rsync_binary_check(binary: str) -> SelfCheck:
if not binary:
return SelfCheck("Global rsync binary", "failed", "Rsync binary is empty.")
if Path(binary).is_absolute():
exists = Path(binary).exists()
message = binary if exists else f"{binary} does not exist."
return SelfCheck("Global rsync binary", "ok" if exists else "failed", message)
path = shutil.which(binary)
return SelfCheck("Global rsync binary", "ok" if path else "failed", path or f"{binary} was not found in PATH.")
def _rsync_recursion_check(name: str, args: list[str]) -> SelfCheck:
if has_recursive_rsync_arg(args):
return SelfCheck(name, "ok", "Rsync args include archive or recursive transfer.", " ".join(args))
return SelfCheck(
name,
"failed",
"Rsync args do not include archive or recursive transfer.",
"Add --archive or --recursive before running a real backup.",
)
def _source_root_check(name: str, source_root: str) -> SelfCheck:
if not source_root:
return SelfCheck(name, "failed", "Source root is empty.")
if not source_root.startswith("/"):
return SelfCheck(name, "failed", f"{source_root} is not absolute.")
return SelfCheck(name, "ok", source_root)
def _root_excludes_check(source_root: str, excludes: list[str], host: HostConfig | None = None) -> SelfCheck:
if source_root != "/":
return SelfCheck("Effective root excludes", "ok", "Source root is not /, critical OS excludes are less important.")
missing = [pattern for pattern in CRITICAL_ROOT_EXCLUDES if pattern not in excludes]
if missing:
detail = ", ".join(missing)
if host and host.excludes_replace is not None:
detail = f"excludes_replace is active; missing {detail}"
return SelfCheck(
"Effective root excludes",
"warning",
"Source root is / but some critical default excludes are missing.",
detail,
)
return SelfCheck("Effective root excludes", "ok", "Critical root excludes are present.")
def _includes_check(host: HostConfig) -> SelfCheck:
includes = list(host.includes or [])
if not includes:
return SelfCheck("Host includes", "ok", "No host include rules are configured.")
return SelfCheck(
"Host includes",
"warning",
"Includes are passed to rsync as raw --include rules.",
"Verify matching exclude rules if you intend to limit the backup scope.",
)
def _retention_check(name: str, daily: int, weekly: int, monthly: int, yearly: int) -> SelfCheck:
if any(value > 0 for value in (daily, weekly, monthly, yearly)):
return SelfCheck(name, "ok", f"d{daily} w{weekly} m{monthly} y{yearly}")
return SelfCheck(name, "warning", "All retention windows are zero.")
def _ssh_user_check(user: str) -> SelfCheck:
if user.strip():
return SelfCheck("Host effective SSH user", "ok", user.strip())
return SelfCheck("Host effective SSH user", "failed", "SSH user is empty.")
def _ssh_port_check(name: str, port: int | None) -> SelfCheck:
if port is None:
return SelfCheck(name, "failed", "SSH port is empty.")
if 1 <= int(port) <= 65535:
return SelfCheck(name, "ok", str(port))
return SelfCheck(name, "failed", f"{port} is outside the valid TCP port range.")
def _credential_check(name: str, credential: SshCredential | None) -> SelfCheck:
if credential is None:
return SelfCheck(name, "warning", "No SSH credential selected.")
if credential.key_path:
key_path = identity_path(credential)
if not key_path.exists():
return SelfCheck(name, "failed", f"{key_path} does not exist.")
if not os.access(key_path, os.R_OK):
return SelfCheck(name, "failed", f"{key_path} is not readable by this process.")
return SelfCheck(name, "ok", str(credential), str(key_path))
if credential.private_key:
return SelfCheck(
name,
"warning",
f"{credential} stores private key material in the database.",
"Generated filesystem keys are recommended for native installs.",
)
return SelfCheck(name, "failed", f"{credential} has no private key material or key path.")

View File

@@ -1,13 +1,10 @@
from __future__ import annotations from __future__ import annotations
from pathlib import Path
from typing import Any from typing import Any
from django.core.exceptions import ObjectDoesNotExist from django.core.exceptions import ObjectDoesNotExist
from pobsync.config.schemas import GLOBAL_SCHEMA, HOST_SCHEMA from pobsync.config.schemas import GLOBAL_SCHEMA, HOST_SCHEMA
from pobsync.paths import PobsyncPaths
from pobsync.util import write_yaml_atomic
from pobsync.validate import validate_dict from pobsync.validate import validate_dict
from .models import GlobalConfig, HostConfig from .models import GlobalConfig, HostConfig
@@ -17,10 +14,9 @@ class ConfigRepositoryError(RuntimeError):
pass pass
def _global_yaml_data(global_config: GlobalConfig) -> dict[str, Any]: def _global_runtime_data(global_config: GlobalConfig) -> dict[str, Any]:
data = { data = {
"backup_root": global_config.backup_root, "backup_root": global_config.backup_root,
"pobsync_home": global_config.pobsync_home,
"ssh": { "ssh": {
"user": global_config.ssh_user, "user": global_config.ssh_user,
"port": global_config.ssh_port, "port": global_config.ssh_port,
@@ -48,7 +44,7 @@ def _global_yaml_data(global_config: GlobalConfig) -> dict[str, Any]:
return validate_dict(data, GLOBAL_SCHEMA, path="global") return validate_dict(data, GLOBAL_SCHEMA, path="global")
def _host_yaml_data(host_config: HostConfig) -> dict[str, Any]: def _host_runtime_data(host_config: HostConfig) -> dict[str, Any]:
data: dict[str, Any] = { data: dict[str, Any] = {
"host": host_config.host, "host": host_config.host,
"address": host_config.address, "address": host_config.address,
@@ -77,50 +73,25 @@ def _host_yaml_data(host_config: HostConfig) -> dict[str, Any]:
return validate_dict(data, HOST_SCHEMA, path="host") return validate_dict(data, HOST_SCHEMA, path="host")
def global_config_object_data(global_config: GlobalConfig) -> dict[str, Any]:
return _global_runtime_data(global_config)
def host_config_object_data(host_config: HostConfig) -> dict[str, Any]:
return _host_runtime_data(host_config)
def global_config_data(name: str = "default") -> dict[str, Any]: def global_config_data(name: str = "default") -> dict[str, Any]:
try: try:
global_config = GlobalConfig.objects.get(name=name) global_config = GlobalConfig.objects.get(name=name)
except ObjectDoesNotExist as exc: except ObjectDoesNotExist as exc:
raise ConfigRepositoryError(f"Missing GlobalConfig {name!r}") from exc raise ConfigRepositoryError(f"Missing global config {name!r}") from exc
return _global_yaml_data(global_config) return _global_runtime_data(global_config)
def host_config_data(host: str) -> dict[str, Any]: def host_config_data(host: str) -> dict[str, Any]:
try: try:
host_config = HostConfig.objects.get(host=host, enabled=True) host_config = HostConfig.objects.get(host=host, enabled=True)
except ObjectDoesNotExist as exc: except ObjectDoesNotExist as exc:
raise ConfigRepositoryError(f"Missing enabled HostConfig {host!r}") from exc raise ConfigRepositoryError(f"Missing enabled host {host!r}") from exc
return _host_yaml_data(host_config) return _host_runtime_data(host_config)
def export_global_config(prefix: Path, name: str = "default") -> Path:
try:
global_config = GlobalConfig.objects.get(name=name)
except ObjectDoesNotExist as exc:
raise ConfigRepositoryError(f"Missing GlobalConfig {name!r}") from exc
paths = PobsyncPaths(home=prefix)
write_yaml_atomic(paths.global_config_path, _global_yaml_data(global_config))
return paths.global_config_path
def export_host_config(prefix: Path, host: str) -> Path:
try:
host_config = HostConfig.objects.get(host=host, enabled=True)
except ObjectDoesNotExist as exc:
raise ConfigRepositoryError(f"Missing enabled HostConfig {host!r}") from exc
paths = PobsyncPaths(home=prefix)
target = paths.hosts_dir / f"{host_config.host}.yaml"
write_yaml_atomic(target, _host_yaml_data(host_config))
return target
def export_runtime_configs(prefix: Path, host: str | None = None) -> list[Path]:
written = [export_global_config(prefix)]
hosts = HostConfig.objects.filter(enabled=True).order_by("host")
if host is not None:
hosts = hosts.filter(host=host)
for host_config in hosts:
written.append(export_host_config(prefix, host_config.host))
return written

View File

@@ -1,12 +1,93 @@
from __future__ import annotations from __future__ import annotations
import os
from pathlib import Path
from typing import Any from typing import Any
from django.conf import settings
from pobsync.config.merge import build_effective_config from pobsync.config.merge import build_effective_config
from pobsync.paths import PobsyncPaths
from .config_repository import global_config_data, host_config_data from .config_repository import global_config_data, host_config_data
from .models import GlobalConfig, HostConfig, SshCredential
from .ssh_keys import identity_path
class DjangoConfigSource: class DjangoConfigSource:
def effective_config_for_host(self, host: str) -> dict[str, Any]: def effective_config_for_host(self, host: str) -> dict[str, Any]:
return build_effective_config(global_config_data(), host_config_data(host)) config = build_effective_config(global_config_data(), host_config_data(host))
credential = _credential_for_host(host)
if credential is not None:
_attach_credential_options(config, credential)
return config
def _credential_for_host(host: str) -> SshCredential | None:
host_config = HostConfig.objects.select_related("ssh_credential").get(host=host, enabled=True)
if host_config.ssh_credential_id:
return host_config.ssh_credential
global_config = GlobalConfig.objects.select_related("default_ssh_credential").get(name="default")
return global_config.default_ssh_credential
def _attach_credential_options(config: dict[str, Any], credential: SshCredential) -> None:
ssh = config.setdefault("ssh", {})
options = list(ssh.get("options") or [])
paths = _materialize_credential(credential)
if not _has_ssh_option(options, "IdentityFile"):
options.append(f"-oIdentityFile={paths['identity_file']}")
if paths.get("known_hosts") and not _has_ssh_option(options, "UserKnownHostsFile"):
options.append(f"-oUserKnownHostsFile={paths['known_hosts']}")
if paths.get("accept_new_known_hosts"):
if not _has_ssh_option(options, "UserKnownHostsFile"):
options.append(f"-oUserKnownHostsFile={paths['accept_new_known_hosts']}")
if not _has_ssh_option(options, "StrictHostKeyChecking"):
options.append("-oStrictHostKeyChecking=accept-new")
ssh["options"] = options
config["ssh_credential"] = {
"id": credential.pk,
"name": credential.name,
"identity_file": paths["identity_file"],
"generated": credential.generated,
"storage": "filesystem" if credential.key_path else "database",
}
def _materialize_credential(credential: SshCredential) -> dict[str, str]:
paths = PobsyncPaths(home=Path(settings.POBSYNC_HOME))
credential_dir = paths.state_dir / "ssh-credentials" / str(credential.pk)
credential_dir.mkdir(mode=0o700, parents=True, exist_ok=True)
os.chmod(credential_dir, 0o700)
identity_file = identity_path(credential)
if credential.key_path:
os.chmod(identity_file, 0o600)
else:
identity_file.write_text(_with_trailing_newline(credential.private_key), encoding="utf-8")
os.chmod(identity_file, 0o600)
result = {"identity_file": str(identity_file)}
if credential.known_hosts.strip():
known_hosts = credential_dir / "known_hosts"
known_hosts.write_text(_with_trailing_newline(credential.known_hosts), encoding="utf-8")
os.chmod(known_hosts, 0o600)
result["known_hosts"] = str(known_hosts)
else:
known_hosts = paths.state_dir / "known_hosts"
known_hosts.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
known_hosts.touch(mode=0o600, exist_ok=True)
os.chmod(known_hosts, 0o600)
result["accept_new_known_hosts"] = str(known_hosts)
return result
def _has_ssh_option(options: list[str], name: str) -> bool:
prefix = f"-o{name}="
spaced = f"-o{name} "
return any(option == name or option.startswith(prefix) or option.startswith(spaced) for option in options)
def _with_trailing_newline(value: str) -> str:
return value if value.endswith("\n") else f"{value}\n"

View File

@@ -0,0 +1,409 @@
from __future__ import annotations
import os
import textwrap
import subprocess
from pathlib import Path
from tempfile import TemporaryDirectory
from django import forms
from django.conf import settings
from .models import GlobalConfig, HostConfig, ScheduleConfig, SshCredential
from .scheduler import parse_cron_expr
class NewlineListField(forms.CharField):
widget = forms.Textarea
def __init__(self, *args, **kwargs) -> None:
kwargs.setdefault("required", False)
super().__init__(*args, **kwargs)
def prepare_value(self, value):
if isinstance(value, list):
return "\n".join(str(item) for item in value)
return value
def to_python(self, value) -> list[str]:
if not value:
return []
if isinstance(value, list):
return [str(item).strip() for item in value if str(item).strip()]
return [line.strip() for line in str(value).splitlines() if line.strip()]
class NullableNewlineListField(NewlineListField):
def to_python(self, value) -> list[str] | None:
parsed = super().to_python(value)
return parsed or None
class HostConfigForm(forms.ModelForm):
includes = NewlineListField(help_text="One include path per line. Leave empty to include defaults.")
excludes_add = NewlineListField(help_text="One additional exclude pattern per line.")
excludes_replace = NullableNewlineListField(
help_text="Optional. When set, replaces global excludes; one pattern per line."
)
rsync_extra_args = NewlineListField(help_text="One extra rsync argument per line.")
class Meta:
model = HostConfig
fields = (
"address",
"enabled",
"ssh_credential",
"ssh_user",
"ssh_port",
"source_root",
"includes",
"excludes_add",
"excludes_replace",
"rsync_extra_args",
"retention_daily",
"retention_weekly",
"retention_monthly",
"retention_yearly",
)
help_texts = {
"ssh_credential": "Optional. Overrides the global SSH credential for this host.",
"ssh_user": "Leave empty to use the global SSH user.",
"ssh_port": "Leave empty to use the global SSH port.",
"source_root": "Leave empty to use the global default source root.",
}
class CreateHostConfigForm(HostConfigForm):
class Meta(HostConfigForm.Meta):
fields = ("host", *HostConfigForm.Meta.fields)
help_texts = {
**HostConfigForm.Meta.help_texts,
"host": "Stable internal host name used for backup paths.",
}
class GlobalConfigForm(forms.ModelForm):
ssh_options = NewlineListField(help_text="One SSH option per line.")
rsync_args = NewlineListField(help_text="One default rsync argument per line.")
rsync_extra_args = NewlineListField(help_text="One extra rsync argument per line.")
excludes_default = NewlineListField(help_text="One default exclude pattern per line.")
class Meta:
model = GlobalConfig
fields = (
"name",
"default_ssh_credential",
"ssh_user",
"ssh_port",
"ssh_options",
"rsync_binary",
"rsync_args",
"rsync_extra_args",
"rsync_timeout_seconds",
"rsync_bwlimit_kbps",
"default_source_root",
"default_destination_subdir",
"excludes_default",
"retention_daily",
"retention_weekly",
"retention_monthly",
"retention_yearly",
)
help_texts = {
"name": "Usually 'default'. The backup engine currently reads the default config.",
"default_ssh_credential": "Optional. Used by hosts without their own SSH credential.",
"default_source_root": "Used by hosts without a custom source root.",
"default_destination_subdir": "Optional subdirectory below each snapshot.",
}
def save(self, commit: bool = True):
instance = super().save(commit=False)
instance.backup_root = settings.POBSYNC_BACKUP_ROOT
if commit:
instance.save()
self.save_m2m()
return instance
class ManualBackupForm(forms.Form):
dry_run = forms.BooleanField(
label="Dry run",
required=False,
initial=True,
help_text="Queue rsync in dry-run mode without writing a snapshot.",
)
verbose_output = forms.BooleanField(
label="Verbose rsync output",
required=False,
help_text="Write itemized rsync changes, file-list progress, and stats to the run log. Dry-runs always use this.",
)
prune = forms.BooleanField(
label="Apply retention after success",
required=False,
help_text="Apply retention after a successful non-dry-run backup.",
)
prune_max_delete = forms.IntegerField(label="Retention max delete", min_value=0, initial=10)
prune_protect_bases = forms.BooleanField(
label="Protect base snapshots",
required=False,
help_text="Keep snapshots that are used as bases by other snapshots.",
)
class SshCredentialForm(forms.ModelForm):
private_key_file = forms.FileField(
required=False,
help_text="Optional. Upload the private key file directly to avoid copy/paste formatting problems.",
)
private_key = forms.CharField(
widget=forms.Textarea(attrs={"spellcheck": "false", "autocomplete": "off"}),
required=False,
help_text=(
"Paste the complete unencrypted OpenSSH private key, including BEGIN/END lines. "
"Leave empty when uploading a private key file."
),
)
public_key = forms.CharField(
widget=forms.Textarea(attrs={"spellcheck": "false", "autocomplete": "off"}),
required=False,
help_text="Optional. If set, pobsync verifies it matches the private key.",
)
known_hosts = forms.CharField(
widget=forms.Textarea(attrs={"spellcheck": "false", "autocomplete": "off"}),
required=False,
help_text="Optional known_hosts entries. When set, StrictHostKeyChecking can stay enabled.",
)
notes = forms.CharField(widget=forms.Textarea, required=False)
class Meta:
model = SshCredential
fields = ("name", "private_key", "public_key", "known_hosts", "notes")
def clean_private_key(self) -> str:
uploaded_file = self.files.get("private_key_file")
if uploaded_file:
try:
raw_private_key = uploaded_file.read().decode("utf-8")
except UnicodeDecodeError as exc:
raise forms.ValidationError("SSH private key files must be UTF-8 text files.") from exc
else:
raw_private_key = self.cleaned_data.get("private_key", "")
if not raw_private_key.strip():
if self.instance and self.instance.pk and self.instance.key_path:
return self.instance.private_key
raise forms.ValidationError("Paste a private key, upload a private key file, or generate a key in pobsync.")
private_key = normalize_private_key(raw_private_key)
public_key = validate_ssh_private_key(private_key)
self.derived_public_key = public_key
return f"{private_key}\n"
def clean(self):
cleaned_data = super().clean()
provided_public_key = normalize_public_key(cleaned_data.get("public_key", ""))
if provided_public_key:
cleaned_data["public_key"] = provided_public_key
elif self.instance and self.instance.pk and self.instance.key_path:
cleaned_data["public_key"] = self.instance.public_key
if cleaned_data.get("private_key") and provided_public_key and hasattr(self, "derived_public_key"):
if public_key_identity(provided_public_key) != public_key_identity(self.derived_public_key):
self.add_error(
"public_key",
forms.ValidationError("Public key does not match the supplied private key."),
)
elif cleaned_data.get("private_key") and not provided_public_key and hasattr(self, "derived_public_key"):
cleaned_data["public_key"] = self.derived_public_key
return cleaned_data
class SshCredentialGenerateForm(forms.Form):
name = forms.CharField(max_length=128)
key_type = forms.ChoiceField(
choices=(("ed25519", "ed25519"), ("rsa", "rsa")),
initial="ed25519",
help_text="ed25519 is recommended unless you need RSA for an older target.",
)
set_global_default = forms.BooleanField(
required=False,
initial=True,
help_text="Use this key as the global default when the default global config exists.",
)
known_hosts = forms.CharField(
widget=forms.Textarea(attrs={"spellcheck": "false", "autocomplete": "off"}),
required=False,
help_text="Optional known_hosts entries. This can also be filled later.",
)
notes = forms.CharField(widget=forms.Textarea, required=False)
def clean_name(self) -> str:
name = self.cleaned_data["name"].strip()
if SshCredential.objects.filter(name=name).exists():
raise forms.ValidationError("An SSH credential with this name already exists.")
return name
class RetentionApplyForm(forms.Form):
kind = forms.ChoiceField(choices=(("scheduled", "Scheduled"), ("manual", "Manual"), ("all", "All")))
protect_bases = forms.BooleanField(required=False)
max_delete = forms.IntegerField(min_value=0, initial=10)
confirm_delete_count = forms.IntegerField(min_value=0)
confirm_host = forms.CharField()
def __init__(self, *args, host_name: str, expected_delete_count: int | None = None, **kwargs) -> None:
self.host_name = host_name
self.expected_delete_count = expected_delete_count
super().__init__(*args, **kwargs)
self.fields["confirm_host"].help_text = f"Type {host_name} to confirm deletion."
if expected_delete_count is not None:
self.fields["confirm_delete_count"].help_text = (
f"Type {expected_delete_count} to confirm the current number of planned deletions."
)
def clean_confirm_host(self) -> str:
value = self.cleaned_data["confirm_host"].strip()
if value != self.host_name:
raise forms.ValidationError(f"Type {self.host_name} to confirm.")
return value
def clean_confirm_delete_count(self) -> int:
value = self.cleaned_data["confirm_delete_count"]
if self.expected_delete_count is not None and value != self.expected_delete_count:
raise forms.ValidationError(f"Type {self.expected_delete_count} to confirm the delete count.")
return value
class IncompleteCleanupForm(forms.Form):
max_delete = forms.IntegerField(min_value=0, initial=0)
confirm_delete_count = forms.IntegerField(min_value=0)
confirm_host = forms.CharField()
def __init__(self, *args, host_name: str, expected_delete_count: int, **kwargs) -> None:
self.host_name = host_name
self.expected_delete_count = expected_delete_count
super().__init__(*args, **kwargs)
self.fields["confirm_host"].help_text = f"Type {host_name} to confirm incomplete snapshot cleanup."
self.fields["confirm_delete_count"].help_text = (
f"Type {expected_delete_count} to confirm the current number of incomplete snapshots."
)
self.fields["max_delete"].help_text = (
f"Must be at least {expected_delete_count} for the incomplete snapshots shown here."
)
def clean_confirm_host(self) -> str:
value = self.cleaned_data["confirm_host"].strip()
if value != self.host_name:
raise forms.ValidationError(f"Type {self.host_name} to confirm.")
return value
def clean_confirm_delete_count(self) -> int:
value = self.cleaned_data["confirm_delete_count"]
if value != self.expected_delete_count:
raise forms.ValidationError(f"Type {self.expected_delete_count} to confirm the incomplete count.")
return value
class ScheduleConfigForm(forms.ModelForm):
cron_expr = forms.CharField(
label="Schedule expression",
help_text=(
'Five-field cron-style expression, for example "15 2 * * *". '
"This is evaluated by the pobsync scheduler service, not host cron."
),
)
prune_max_delete = forms.IntegerField(min_value=0)
class Meta:
model = ScheduleConfig
fields = (
"cron_expr",
"enabled",
"prune",
"prune_max_delete",
"prune_protect_bases",
)
def clean_cron_expr(self) -> str:
cron_expr = self.cleaned_data["cron_expr"].strip()
try:
parse_cron_expr(cron_expr)
except ValueError as exc:
raise forms.ValidationError(str(exc)) from exc
return cron_expr
def normalize_private_key(private_key: str) -> str:
normalized = private_key.replace("\r\n", "\n").replace("\r", "\n").strip().lstrip("\ufeff")
begin_marker = "-----BEGIN OPENSSH PRIVATE KEY-----"
end_marker = "-----END OPENSSH PRIVATE KEY-----"
if begin_marker in normalized and end_marker in normalized:
before_body, after_begin = normalized.split(begin_marker, 1)
body, after_end = after_begin.split(end_marker, 1)
if before_body.strip() or after_end.strip():
return normalized
compact_body = "".join(body.split())
wrapped_body = "\n".join(textwrap.wrap(compact_body, width=70))
return f"{begin_marker}\n{wrapped_body}\n{end_marker}"
return normalized
def normalize_public_key(public_key: str) -> str:
return " ".join(public_key.strip().split())
def public_key_identity(public_key: str) -> str:
parts = normalize_public_key(public_key).split()
if len(parts) >= 2:
return " ".join(parts[:2])
return normalize_public_key(public_key)
def validate_ssh_private_key(private_key: str) -> str:
if "BEGIN OPENSSH PRIVATE KEY" not in private_key:
stripped = private_key.strip()
if stripped.startswith(("ssh-ed25519 ", "ssh-rsa ", "ecdsa-sha2-", "sk-")):
raise forms.ValidationError("This looks like a public key. Paste the private key in this field.")
if "BEGIN RSA PRIVATE KEY" in stripped or "BEGIN EC PRIVATE KEY" in stripped:
raise forms.ValidationError(
"PEM private keys are not supported here yet. Convert it to an unencrypted OpenSSH key first."
)
raise forms.ValidationError("Invalid SSH private key: missing OpenSSH private key header.")
with TemporaryDirectory() as tmp:
key_path = Path(tmp) / "identity"
key_path.write_text(f"{private_key}\n", encoding="utf-8")
os.chmod(key_path, 0o600)
try:
result = subprocess.run(
["ssh-keygen", "-y", "-f", str(key_path)],
check=False,
stdin=subprocess.DEVNULL,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
timeout=5,
)
except FileNotFoundError as exc:
raise forms.ValidationError("ssh-keygen is not available in this container.") from exc
except subprocess.TimeoutExpired as exc:
raise forms.ValidationError("Could not validate SSH private key before timeout.") from exc
if result.returncode != 0:
message = result.stderr.strip() or "OpenSSH could not read this private key."
lower_message = message.lower()
if "passphrase" in lower_message:
message = "Encrypted SSH private keys are not supported for unattended backups."
elif "libcrypto" in lower_message:
message = (
"OpenSSH could not parse this key. It is usually incomplete, corrupted while copying, "
"or not an unencrypted OpenSSH private key."
)
raise forms.ValidationError(f"Invalid SSH private key: {message}")
public_key = result.stdout.strip()
if not public_key:
raise forms.ValidationError("Invalid SSH private key: no public key could be derived.")
return public_key

View File

@@ -0,0 +1,103 @@
from __future__ import annotations
import os
from pathlib import Path
from pobsync.snapshot_meta import resolve_host_root
from .config_checks import collect_effective_host_config_checks
from .models import GlobalConfig, HostConfig
from .self_check import SelfCheck
from .ssh_keys import identity_path
HOST_BACKUP_SUBDIRS = ("scheduled", "manual", ".incomplete")
def ensure_host_directories(host: HostConfig, global_config: GlobalConfig | None = None) -> Path:
global_config = global_config or GlobalConfig.objects.get(name="default")
host_root = resolve_host_root(global_config.backup_root, host.host)
for subdir in HOST_BACKUP_SUBDIRS:
(host_root / subdir).mkdir(parents=True, exist_ok=True)
return host_root
def collect_host_checks(host: HostConfig, global_config: GlobalConfig | None = None) -> list[SelfCheck]:
checks: list[SelfCheck] = []
try:
global_config = global_config or GlobalConfig.objects.get(name="default")
except GlobalConfig.DoesNotExist:
return [SelfCheck("Host global config", "failed", "Default global config does not exist.")]
checks.append(
SelfCheck(
"Host enabled",
"ok" if host.enabled else "warning",
"Host is enabled." if host.enabled else "Host is disabled.",
)
)
checks.append(
SelfCheck(
"Host address",
"ok" if host.address.strip() else "failed",
host.address.strip() or "Host address is empty.",
)
)
credential = host.ssh_credential or global_config.default_ssh_credential
if credential is None:
checks.append(SelfCheck("Host SSH credential", "warning", "No host or global SSH credential selected."))
else:
checks.append(SelfCheck("Host SSH credential", "ok", str(credential)))
if credential.key_path:
key_path = identity_path(credential)
checks.append(
_host_path_check("Host SSH key file", key_path, must_exist=True, must_be_writable=False, must_be_readable=True)
)
elif credential.private_key:
checks.append(
SelfCheck(
"Host SSH key storage",
"warning",
"Selected credential stores private key material in the database.",
"Generated filesystem keys are recommended for native systemd installs.",
)
)
if credential.known_hosts.strip():
checks.append(SelfCheck("Host known_hosts", "ok", "Selected credential has known_hosts entries."))
else:
checks.append(
SelfCheck(
"Host known_hosts",
"warning",
"Selected credential has no pinned known_hosts entries.",
"pobsync will use service-level StrictHostKeyChecking=accept-new on first connect.",
)
)
host_root = resolve_host_root(global_config.backup_root, host.host)
checks.append(_host_path_check("Host backup root", host_root, must_exist=True, must_be_writable=True))
for subdir in HOST_BACKUP_SUBDIRS:
checks.append(_host_path_check(f"Host directory: {subdir}", host_root / subdir, must_exist=True, must_be_writable=True))
checks.extend(collect_effective_host_config_checks(host, global_config))
return checks
def _host_path_check(
name: str,
path: Path,
*,
must_exist: bool,
must_be_writable: bool,
must_be_readable: bool = False,
) -> SelfCheck:
if must_exist and not path.exists():
return SelfCheck(name, "failed", f"{path} does not exist.")
target = path if path.exists() else path.parent
if not target.exists():
return SelfCheck(name, "failed", f"{target} does not exist.")
if must_be_writable and not os.access(target, os.W_OK):
return SelfCheck(name, "failed", f"{target} is not writable by this process.")
if must_be_readable and not os.access(target, os.R_OK):
return SelfCheck(name, "failed", f"{target} is not readable by this process.")
return SelfCheck(name, "ok", str(path))

View File

@@ -0,0 +1,44 @@
from __future__ import annotations
from django.core.management.base import BaseCommand, CommandError
from pobsync_backend.self_check import collect_self_checks, summarize_self_checks
class Command(BaseCommand):
help = "Run pobsync runtime self checks for native installs and updates."
def add_arguments(self, parser):
parser.add_argument(
"--fail-on-warning",
action="store_true",
help="Exit with an error when warnings are present.",
)
def handle(self, *args, **options):
checks = collect_self_checks()
summary = summarize_self_checks(checks)
for check in checks:
line = f"[{check.status.upper()}] {check.name}: {check.message}"
if check.detail:
line = f"{line} ({check.detail})"
if check.status == "failed":
self.stderr.write(line)
elif check.status == "warning":
self.stderr.write(line)
else:
self.stdout.write(line)
self.stdout.write(
"Summary: "
f"{summary['ok']} ok, "
f"{summary['warning']} warning(s), "
f"{summary['failed']} failed, "
f"{summary['skipped']} skipped"
)
if summary["failed"]:
raise CommandError("pobsync install self check failed.")
if options["fail_on_warning"] and summary["warning"]:
raise CommandError("pobsync install self check reported warnings.")

View File

@@ -1,9 +1,7 @@
from __future__ import annotations from __future__ import annotations
from pathlib import Path
from typing import Any from typing import Any
from django.conf import settings
from django.core.management.base import BaseCommand, CommandError from django.core.management.base import BaseCommand, CommandError
from pobsync.config.retention import parse_retention from pobsync.config.retention import parse_retention
@@ -13,12 +11,11 @@ from pobsync_backend.models import GlobalConfig
class Command(BaseCommand): class Command(BaseCommand):
help = "Create or update the SQL-backed global pobsync configuration." help = "Create or update the default global backup configuration."
def add_arguments(self, parser) -> None: def add_arguments(self, parser) -> None:
parser.add_argument("--name", default="default") parser.add_argument("--name", default="default")
parser.add_argument("--backup-root", required=True) parser.add_argument("--backup-root", required=True)
parser.add_argument("--pobsync-home", default=settings.POBSYNC_HOME)
parser.add_argument("--ssh-user", default="root") parser.add_argument("--ssh-user", default="root")
parser.add_argument("--ssh-port", type=int, default=22) parser.add_argument("--ssh-port", type=int, default=22)
parser.add_argument("--source-root", default="/") parser.add_argument("--source-root", default="/")
@@ -30,11 +27,9 @@ class Command(BaseCommand):
if not is_absolute_non_root(backup_root): if not is_absolute_non_root(backup_root):
raise CommandError("--backup-root must be an absolute path and must not be '/'") raise CommandError("--backup-root must be an absolute path and must not be '/'")
pobsync_home = str(Path(options["pobsync_home"]))
retention = parse_retention(options["retention"]) retention = parse_retention(options["retention"])
defaults = { defaults = {
"backup_root": backup_root, "backup_root": backup_root,
"pobsync_home": pobsync_home,
"ssh_user": options["ssh_user"], "ssh_user": options["ssh_user"],
"ssh_port": options["ssh_port"], "ssh_port": options["ssh_port"],
"ssh_options": ["-oBatchMode=yes", "-oStrictHostKeyChecking=accept-new"], "ssh_options": ["-oBatchMode=yes", "-oStrictHostKeyChecking=accept-new"],
@@ -53,8 +48,8 @@ class Command(BaseCommand):
} }
if GlobalConfig.objects.filter(name=options["name"]).exists() and not options["force"]: if GlobalConfig.objects.filter(name=options["name"]).exists() and not options["force"]:
raise CommandError(f"GlobalConfig {options['name']!r} already exists; use --force to update") raise CommandError(f"Global config {options['name']!r} already exists; use --force to update")
_obj, created = GlobalConfig.objects.update_or_create(name=options["name"], defaults=defaults) _obj, created = GlobalConfig.objects.update_or_create(name=options["name"], defaults=defaults)
action = "Created" if created else "Updated" action = "Created" if created else "Updated"
self.stdout.write(self.style.SUCCESS(f"{action} GlobalConfig {options['name']!r}.")) self.stdout.write(self.style.SUCCESS(f"{action} global config {options['name']!r}."))

View File

@@ -10,7 +10,7 @@ from pobsync_backend.models import GlobalConfig, HostConfig
class Command(BaseCommand): class Command(BaseCommand):
help = "Create or update a SQL-backed host pobsync configuration." help = "Create or update a host backup configuration."
def add_arguments(self, parser) -> None: def add_arguments(self, parser) -> None:
parser.add_argument("host") parser.add_argument("host")
@@ -29,7 +29,7 @@ class Command(BaseCommand):
def handle(self, *args: Any, **options: Any) -> None: def handle(self, *args: Any, **options: Any) -> None:
host = sanitize_host(options["host"]) host = sanitize_host(options["host"])
if HostConfig.objects.filter(host=host).exists() and not options["force"]: if HostConfig.objects.filter(host=host).exists() and not options["force"]:
raise CommandError(f"HostConfig {host!r} already exists; use --force to update") raise CommandError(f"Host {host!r} already exists; use --force to update")
retention = self._retention(options["retention"]) retention = self._retention(options["retention"])
defaults = { defaults = {
@@ -49,7 +49,7 @@ class Command(BaseCommand):
} }
_obj, created = HostConfig.objects.update_or_create(host=host, defaults=defaults) _obj, created = HostConfig.objects.update_or_create(host=host, defaults=defaults)
action = "Created" if created else "Updated" action = "Created" if created else "Updated"
self.stdout.write(self.style.SUCCESS(f"{action} HostConfig {host!r}.")) self.stdout.write(self.style.SUCCESS(f"{action} host {host!r}."))
def _retention(self, value: str | None) -> dict[str, int]: def _retention(self, value: str | None) -> dict[str, int]:
if value: if value:

View File

@@ -9,12 +9,16 @@ from pobsync_backend.scheduler import parse_cron_expr
class Command(BaseCommand): class Command(BaseCommand):
help = "Create, update, disable, or remove a SQL-backed pobsync schedule." help = "Create, update, disable, or remove a scheduler-managed host schedule."
def add_arguments(self, parser) -> None: def add_arguments(self, parser) -> None:
parser.add_argument("host") parser.add_argument("host")
parser.add_argument("--cron", help='Cron expression, e.g. "15 2 * * *"') parser.add_argument(
parser.add_argument("--user", default="root") "--schedule-expression",
"--cron",
dest="schedule_expression",
help='Five-field schedule expression, e.g. "15 2 * * *"',
)
parser.add_argument("--prune", action="store_true") parser.add_argument("--prune", action="store_true")
parser.add_argument("--prune-max-delete", type=int, default=10) parser.add_argument("--prune-max-delete", type=int, default=10)
parser.add_argument("--prune-protect-bases", action="store_true") parser.add_argument("--prune-protect-bases", action="store_true")
@@ -25,25 +29,25 @@ class Command(BaseCommand):
try: try:
host = HostConfig.objects.get(host=options["host"]) host = HostConfig.objects.get(host=options["host"])
except HostConfig.DoesNotExist as exc: except HostConfig.DoesNotExist as exc:
raise CommandError(f"Missing HostConfig {options['host']!r}") from exc raise CommandError(f"Missing host {options['host']!r}") from exc
if options["delete"]: if options["delete"]:
deleted, _details = ScheduleConfig.objects.filter(host=host).delete() deleted, _details = ScheduleConfig.objects.filter(host=host).delete()
self.stdout.write(self.style.SUCCESS(f"Deleted {deleted} schedule row(s) for {host.host!r}.")) self.stdout.write(self.style.SUCCESS(f"Deleted {deleted} schedule row(s) for {host.host!r}."))
return return
if not options["cron"]: schedule_expression = options["schedule_expression"]
raise CommandError("--cron is required unless --delete is used") if not schedule_expression:
raise CommandError("--schedule-expression is required unless --delete is used")
try: try:
parse_cron_expr(options["cron"]) parse_cron_expr(schedule_expression)
except ValueError as exc: except ValueError as exc:
raise CommandError(str(exc)) from exc raise CommandError(str(exc)) from exc
schedule, created = ScheduleConfig.objects.update_or_create( schedule, created = ScheduleConfig.objects.update_or_create(
host=host, host=host,
defaults={ defaults={
"cron_expr": options["cron"], "cron_expr": schedule_expression,
"user": options["user"],
"enabled": not options["disabled"], "enabled": not options["disabled"],
"prune": bool(options["prune"]), "prune": bool(options["prune"]),
"prune_max_delete": int(options["prune_max_delete"]), "prune_max_delete": int(options["prune_max_delete"]),

View File

@@ -20,14 +20,14 @@ class Command(BaseCommand):
try: try:
global_config = GlobalConfig.objects.get(name="default") global_config = GlobalConfig.objects.get(name="default")
except GlobalConfig.DoesNotExist as exc: except GlobalConfig.DoesNotExist as exc:
raise CommandError("Missing GlobalConfig 'default'") from exc raise CommandError("Missing default global config") from exc
host = None host = None
if options["host"]: if options["host"]:
try: try:
host = HostConfig.objects.get(host=options["host"], enabled=True) host = HostConfig.objects.get(host=options["host"], enabled=True)
except HostConfig.DoesNotExist as exc: except HostConfig.DoesNotExist as exc:
raise CommandError(f"Missing enabled HostConfig {options['host']!r}") from exc raise CommandError(f"Missing enabled host {options['host']!r}") from exc
kind = normalize_kind(options["kind"]) kind = normalize_kind(options["kind"])
kinds = ["scheduled", "manual", "incomplete"] if kind == "all" else [kind] kinds = ["scheduled", "manual", "incomplete"] if kind == "all" else [kind]

View File

@@ -0,0 +1,47 @@
from __future__ import annotations
from django.core.management.base import BaseCommand, CommandError
from pobsync_backend.models import GlobalConfig, SshCredential
from pobsync_backend.ssh_keys import SshKeyError, generate_ssh_key
class Command(BaseCommand):
help = "Ensure a filesystem-backed SSH key exists for pobsync backups."
def add_arguments(self, parser):
parser.add_argument("--name", default="default", help="Credential name to create or reuse.")
parser.add_argument("--key-type", default="ed25519", choices=("ed25519", "rsa"))
parser.add_argument(
"--set-global-default",
action="store_true",
help="Set this key as default on the default global config when it exists.",
)
def handle(self, *args, **options):
name = options["name"]
credential, created = SshCredential.objects.get_or_create(
name=name,
defaults={
"key_type": options["key_type"],
"notes": "Generated by pobsync installer.",
},
)
if not credential.key_path and not credential.private_key:
try:
generate_ssh_key(credential, key_type=options["key_type"])
except SshKeyError as exc:
raise CommandError(str(exc)) from exc
created = True
if options["set_global_default"]:
global_config = GlobalConfig.objects.filter(name="default").first()
if global_config is not None and global_config.default_ssh_credential_id is None:
global_config.default_ssh_credential = credential
global_config.save(update_fields=["default_ssh_credential", "updated_at"])
action = "created" if created else "exists"
self.stdout.write(self.style.SUCCESS(f"SSH credential {action}: {credential.name}"))
if credential.public_key:
self.stdout.write(credential.public_key)

View File

@@ -1,23 +0,0 @@
from __future__ import annotations
from pathlib import Path
from typing import Any
from django.conf import settings
from django.core.management.base import BaseCommand
from pobsync_backend.config_repository import export_runtime_configs
class Command(BaseCommand):
help = "Export Django database configs to pobsync runtime YAML files."
def add_arguments(self, parser) -> None:
parser.add_argument("--prefix", default=settings.POBSYNC_HOME, help="Pobsync home directory")
parser.add_argument("--host", default=None, help="Export only one enabled host")
def handle(self, *args: Any, **options: Any) -> None:
written = export_runtime_configs(prefix=Path(options["prefix"]), host=options["host"])
for path in written:
self.stdout.write(str(path))
self.stdout.write(self.style.SUCCESS(f"Exported {len(written)} config file(s)."))

View File

@@ -1,81 +0,0 @@
from __future__ import annotations
from pathlib import Path
from typing import Any
from django.conf import settings
from django.core.management.base import BaseCommand, CommandError
from pobsync.config.load import load_global_config, load_host_config
from pobsync.paths import PobsyncPaths
from pobsync_backend.models import GlobalConfig, HostConfig
class Command(BaseCommand):
help = "Import pobsync YAML configs into the Django database."
def add_arguments(self, parser) -> None:
parser.add_argument("--prefix", default=settings.POBSYNC_HOME, help="Pobsync home directory")
def handle(self, *args: Any, **options: Any) -> None:
paths = PobsyncPaths(home=Path(options["prefix"]))
if not paths.global_config_path.exists():
raise CommandError(f"Missing global config: {paths.global_config_path}")
global_cfg = load_global_config(paths.global_config_path)
global_ssh = global_cfg.get("ssh") or {}
global_rsync = global_cfg.get("rsync") or {}
global_defaults = global_cfg.get("defaults") or {}
retention_defaults = global_cfg.get("retention_defaults") or {}
GlobalConfig.objects.update_or_create(
name="default",
defaults={
"backup_root": global_cfg["backup_root"],
"pobsync_home": global_cfg.get("pobsync_home", str(paths.home)),
"ssh_user": global_ssh.get("user") or "root",
"ssh_port": global_ssh.get("port") or 22,
"ssh_options": global_ssh.get("options") or [],
"rsync_binary": global_rsync.get("binary") or "rsync",
"rsync_args": global_rsync.get("args") or [],
"rsync_extra_args": global_rsync.get("extra_args") or [],
"rsync_timeout_seconds": global_rsync.get("timeout_seconds") or 0,
"rsync_bwlimit_kbps": global_rsync.get("bwlimit_kbps") or 0,
"default_source_root": global_defaults.get("source_root") or "/",
"default_destination_subdir": global_defaults.get("destination_subdir") or "",
"excludes_default": global_cfg.get("excludes_default") or [],
"retention_daily": retention_defaults.get("daily", 14),
"retention_weekly": retention_defaults.get("weekly", 8),
"retention_monthly": retention_defaults.get("monthly", 12),
"retention_yearly": retention_defaults.get("yearly", 0),
"data": global_cfg,
},
)
count = 0
for host_path in sorted(paths.hosts_dir.glob("*.yaml")):
host_cfg = load_host_config(host_path)
host_ssh = host_cfg.get("ssh") or {}
host_rsync = host_cfg.get("rsync") or {}
host_retention = host_cfg.get("retention") or {}
HostConfig.objects.update_or_create(
host=host_cfg["host"],
defaults={
"address": host_cfg["address"],
"ssh_user": host_ssh.get("user") or "",
"ssh_port": host_ssh.get("port"),
"source_root": host_cfg.get("source_root") or "",
"includes": host_cfg.get("includes") or [],
"excludes_add": host_cfg.get("excludes_add") or [],
"excludes_replace": host_cfg.get("excludes_replace"),
"rsync_extra_args": host_rsync.get("extra_args") or [],
"retention_daily": host_retention.get("daily", 14),
"retention_weekly": host_retention.get("weekly", 8),
"retention_monthly": host_retention.get("monthly", 12),
"retention_yearly": host_retention.get("yearly", 0),
"config": host_cfg,
"enabled": True,
},
)
count += 1
self.stdout.write(self.style.SUCCESS(f"Imported global config and {count} host config(s)."))

View File

@@ -5,25 +5,24 @@ from typing import Any
from django.conf import settings from django.conf import settings
from django.core.management.base import BaseCommand, CommandError from django.core.management.base import BaseCommand, CommandError
from django.utils import timezone
from pobsync.commands.run_scheduled import run_scheduled
from pobsync.paths import PobsyncPaths from pobsync.paths import PobsyncPaths
from pobsync_backend.config_source import DjangoConfigSource from pobsync_backend.backup_runner import execute_backup_run
from pobsync_backend.models import BackupRun, HostConfig from pobsync_backend.models import BackupRun, HostConfig
from pobsync_backend.snapshot_discovery import infer_snapshot_kind, upsert_snapshot_record
class Command(BaseCommand): class Command(BaseCommand):
help = "Run a scheduled pobsync backup and record the result in Django." help = "Run a pobsync backup and record the result in Django."
def add_arguments(self, parser) -> None: def add_arguments(self, parser) -> None:
parser.add_argument("host", help="Host to back up") parser.add_argument("host", help="Host to back up")
parser.add_argument("--prefix", default=settings.POBSYNC_HOME, help="Pobsync home directory") parser.add_argument("--prefix", default=settings.POBSYNC_HOME, help="Runtime state root")
parser.add_argument("--dry-run", action="store_true", help="Run rsync --dry-run") parser.add_argument("--dry-run", action="store_true", help="Run rsync --dry-run")
parser.add_argument("--verbose-rsync", action="store_true", help="Write itemized rsync output to the run log")
parser.add_argument("--prune", action="store_true", help="Apply retention after a successful run") parser.add_argument("--prune", action="store_true", help="Apply retention after a successful run")
parser.add_argument("--prune-max-delete", type=int, default=10) parser.add_argument("--prune-max-delete", type=int, default=10)
parser.add_argument("--prune-protect-bases", action="store_true") parser.add_argument("--prune-protect-bases", action="store_true")
parser.add_argument("--manual", action="store_true", help="Record the run as manual instead of scheduled")
def handle(self, *args: Any, **options: Any) -> None: def handle(self, *args: Any, **options: Any) -> None:
host_name = options["host"] host_name = options["host"]
@@ -31,59 +30,38 @@ class Command(BaseCommand):
try: try:
host = HostConfig.objects.get(host=host_name, enabled=True) host = HostConfig.objects.get(host=host_name, enabled=True)
except HostConfig.DoesNotExist as exc: except HostConfig.DoesNotExist as exc:
raise CommandError(f"Missing enabled HostConfig {host_name!r}") from exc raise CommandError(f"Missing enabled host {host_name!r}") from exc
run = BackupRun.objects.create( run = BackupRun.objects.create(
host=host, host=host,
run_type=BackupRun.RunType.SCHEDULED, run_type=BackupRun.RunType.MANUAL if options["manual"] else BackupRun.RunType.SCHEDULED,
status=BackupRun.Status.RUNNING, status=BackupRun.Status.RUNNING,
started_at=timezone.now(), result={
"requested": {
"dry_run": bool(options["dry_run"]),
"verbose_output": bool(options["dry_run"] or options["verbose_rsync"]),
"prune": bool(options["prune"]),
"prune_max_delete": int(options["prune_max_delete"]),
"prune_protect_bases": bool(options["prune_protect_bases"]),
}
},
) )
execute_backup_run(
try: run=run,
result = run_scheduled( prefix=paths.home,
prefix=paths.home, dry_run=bool(options["dry_run"]),
host=host.host, verbose_output=bool(options["dry_run"] or options["verbose_rsync"]),
dry_run=bool(options["dry_run"]), prune=bool(options["prune"]),
prune=bool(options["prune"]), prune_max_delete=int(options["prune_max_delete"]),
prune_max_delete=int(options["prune_max_delete"]), prune_protect_bases=bool(options["prune_protect_bases"]),
prune_protect_bases=bool(options["prune_protect_bases"]),
config_source=DjangoConfigSource(),
)
except Exception as exc:
run.status = BackupRun.Status.FAILED
run.ended_at = timezone.now()
run.result = {"ok": False, "error": str(exc), "type": type(exc).__name__}
run.save(update_fields=["status", "ended_at", "result"])
raise
run.status = BackupRun.Status.SUCCESS if result.get("ok") else BackupRun.Status.FAILED
run.ended_at = timezone.now()
run.snapshot_path = str(result.get("snapshot") or "")
run.base_path = str(result.get("base") or "")
rsync = result.get("rsync") if isinstance(result.get("rsync"), dict) else {}
run.rsync_exit_code = rsync.get("exit_code")
run.result = result
run.save(
update_fields=[
"status",
"ended_at",
"snapshot_path",
"base_path",
"rsync_exit_code",
"result",
],
) )
if run.snapshot_path: run.refresh_from_db()
snapshot_path = Path(run.snapshot_path)
try:
kind = infer_snapshot_kind(snapshot_path)
upsert_snapshot_record(host=host, kind=kind, snapshot_dir=snapshot_path)
except ValueError:
pass
if result.get("ok"): if run.status == BackupRun.Status.SUCCESS:
self.stdout.write(self.style.SUCCESS(f"Backup completed for {host.host}.")) self.stdout.write(self.style.SUCCESS(f"Backup completed for {host.host}."))
return return
if run.status == BackupRun.Status.WARNING:
self.stdout.write(self.style.WARNING(f"Backup completed with warnings for {host.host}; run id={run.id}"))
return
raise CommandError(f"Backup failed for {host.host}; run id={run.id}") raise CommandError(f"Backup failed for {host.host}; run id={run.id}")

View File

@@ -7,18 +7,16 @@ from typing import Any
from django.conf import settings from django.conf import settings
from django.core.management.base import BaseCommand, CommandError from django.core.management.base import BaseCommand, CommandError
from pobsync.commands.retention_apply import run_retention_apply from pobsync.errors import ConfigError
from pobsync.commands.retention_plan import run_retention_plan from pobsync_backend.retention import run_sql_retention_apply, run_sql_retention_plan
from pobsync_backend.config_source import DjangoConfigSource
from pobsync_backend.models import HostConfig
class Command(BaseCommand): class Command(BaseCommand):
help = "Plan or apply retention using SQL-backed pobsync configuration." help = "Plan or apply retention using the Django backup configuration."
def add_arguments(self, parser) -> None: def add_arguments(self, parser) -> None:
parser.add_argument("host") parser.add_argument("host")
parser.add_argument("--prefix", default=settings.POBSYNC_HOME) parser.add_argument("--prefix", default=settings.POBSYNC_HOME, help="Runtime state root")
parser.add_argument("--kind", default="scheduled", choices=["scheduled", "manual", "all"]) parser.add_argument("--kind", default="scheduled", choices=["scheduled", "manual", "all"])
parser.add_argument("--protect-bases", action="store_true") parser.add_argument("--protect-bases", action="store_true")
parser.add_argument("--apply", action="store_true") parser.add_argument("--apply", action="store_true")
@@ -27,29 +25,26 @@ class Command(BaseCommand):
def handle(self, *args: Any, **options: Any) -> None: def handle(self, *args: Any, **options: Any) -> None:
host = options["host"] host = options["host"]
if not HostConfig.objects.filter(host=host, enabled=True).exists(): try:
raise CommandError(f"Missing enabled HostConfig {host!r}") if options["apply"]:
if not options["yes"]:
config_source = DjangoConfigSource() raise CommandError("--yes is required with --apply")
if options["apply"]: result = run_sql_retention_apply(
if not options["yes"]: prefix=Path(options["prefix"]),
raise CommandError("--yes is required with --apply") host=host,
result = run_retention_apply( kind=options["kind"],
prefix=Path(options["prefix"]), protect_bases=bool(options["protect_bases"]),
host=host, yes=True,
kind=options["kind"], max_delete=int(options["max_delete"]),
protect_bases=bool(options["protect_bases"]), action="cli",
yes=True, )
max_delete=int(options["max_delete"]), else:
config_source=config_source, result = run_sql_retention_plan(
) host=host,
else: kind=options["kind"],
result = run_retention_plan( protect_bases=bool(options["protect_bases"]),
prefix=Path(options["prefix"]), )
host=host, except ConfigError as exc:
kind=options["kind"], raise CommandError(str(exc)) from exc
protect_bases=bool(options["protect_bases"]),
config_source=config_source,
)
self.stdout.write(json.dumps(result, indent=2, sort_keys=False)) self.stdout.write(json.dumps(result, indent=2, sort_keys=False))

View File

@@ -10,7 +10,7 @@ from django.core.management.base import BaseCommand
from django.db import transaction from django.db import transaction
from django.utils import timezone from django.utils import timezone
from pobsync_backend.models import ScheduleConfig from pobsync_backend.models import BackupRun, ScheduleConfig
from pobsync_backend.scheduler import due_key, is_due from pobsync_backend.scheduler import due_key, is_due
@@ -18,7 +18,7 @@ class Command(BaseCommand):
help = "Run due pobsync schedules from the Django database." help = "Run due pobsync schedules from the Django database."
def add_arguments(self, parser) -> None: def add_arguments(self, parser) -> None:
parser.add_argument("--prefix", default=settings.POBSYNC_HOME, help="Pobsync home directory") parser.add_argument("--prefix", default=settings.POBSYNC_HOME, help="Runtime state root")
parser.add_argument("--once", action="store_true", help="Check once and exit") parser.add_argument("--once", action="store_true", help="Check once and exit")
parser.add_argument("--loop", action="store_true", help="Keep checking schedules") parser.add_argument("--loop", action="store_true", help="Keep checking schedules")
parser.add_argument("--interval", type=int, default=60, help="Loop interval in seconds") parser.add_argument("--interval", type=int, default=60, help="Loop interval in seconds")
@@ -52,12 +52,13 @@ class Command(BaseCommand):
if not is_due(schedule.cron_expr, now): if not is_due(schedule.cron_expr, now):
continue continue
schedule_started_at = timezone.now()
with transaction.atomic(): with transaction.atomic():
locked = ScheduleConfig.objects.select_for_update().get(pk=schedule.pk) locked = ScheduleConfig.objects.select_for_update().get(pk=schedule.pk)
if locked.last_due_key == current_due_key: if locked.last_due_key == current_due_key:
continue continue
locked.last_due_key = current_due_key locked.last_due_key = current_due_key
locked.last_started_at = timezone.now() locked.last_started_at = schedule_started_at
locked.last_status = "running" locked.last_status = "running"
locked.save(update_fields=["last_due_key", "last_started_at", "last_status", "updated_at"]) locked.save(update_fields=["last_due_key", "last_started_at", "last_status", "updated_at"])
@@ -72,6 +73,7 @@ class Command(BaseCommand):
prune_max_delete=schedule.prune_max_delete, prune_max_delete=schedule.prune_max_delete,
prune_protect_bases=schedule.prune_protect_bases, prune_protect_bases=schedule.prune_protect_bases,
) )
status = _latest_scheduled_run_status(host_id=schedule.host_id, started_at=schedule_started_at) or status
except Exception as exc: except Exception as exc:
status = "failed" status = "failed"
self.stderr.write(f"{schedule.host.host}: {type(exc).__name__}: {exc}") self.stderr.write(f"{schedule.host.host}: {type(exc).__name__}: {exc}")
@@ -83,3 +85,16 @@ class Command(BaseCommand):
ran += 1 ran += 1
return ran return ran
def _latest_scheduled_run_status(*, host_id: int, started_at) -> str | None:
run = (
BackupRun.objects.filter(
host_id=host_id,
run_type=BackupRun.RunType.SCHEDULED,
created_at__gte=started_at,
)
.order_by("-created_at", "-id")
.first()
)
return run.status if run is not None else None

View File

@@ -0,0 +1,60 @@
from __future__ import annotations
import time
from pathlib import Path
from typing import Any
from django.conf import settings
from django.core.management.base import BaseCommand
from pobsync.paths import PobsyncPaths
from pobsync_backend.backup_runner import claim_next_queued_run, execute_backup_run, reconcile_running_runs, requested_options
class Command(BaseCommand):
help = "Run queued pobsync backup jobs from the Django database."
def add_arguments(self, parser) -> None:
parser.add_argument("--prefix", default=settings.POBSYNC_HOME, help="Runtime state root")
parser.add_argument("--once", action="store_true", help="Process one queued run and exit")
parser.add_argument("--loop", action="store_true", help="Keep checking for queued runs")
parser.add_argument("--interval", type=int, default=15, help="Loop interval in seconds")
parser.add_argument(
"--stale-running-seconds",
type=int,
default=24 * 60 * 60,
help="Mark running runs failed after this many seconds without a worker heartbeat; use 0 to disable",
)
def handle(self, *args: Any, **options: Any) -> None:
if not options["once"] and not options["loop"]:
options["once"] = True
paths = PobsyncPaths(home=Path(options["prefix"]))
while True:
count = self._run_once(prefix=paths.home, stale_running_seconds=int(options["stale_running_seconds"]))
self.stdout.write(f"Ran {count} queued backup run(s).")
if options["once"]:
return
time.sleep(max(1, int(options["interval"])))
def _run_once(self, *, prefix: Path, stale_running_seconds: int = 24 * 60 * 60) -> int:
reconciled = reconcile_running_runs(stale_worker_seconds=stale_running_seconds)
run = claim_next_queued_run()
if run is None:
return reconciled
options = requested_options(run)
try:
execute_backup_run(
run=run,
prefix=prefix,
dry_run=bool(options.get("dry_run", False)),
verbose_output=bool(options.get("verbose_output", False)),
prune=bool(options.get("prune", False)),
prune_max_delete=int(options.get("prune_max_delete", 10)),
prune_protect_bases=bool(options.get("prune_protect_bases", False)),
)
except Exception as exc:
self.stderr.write(f"{run.host.host}: {type(exc).__name__}: {exc}")
return reconciled + 1

View File

@@ -0,0 +1,24 @@
from __future__ import annotations
import django.db.models.deletion
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("pobsync_backend", "0003_structured_config_fields"),
]
operations = [
migrations.AddField(
model_name="backuprun",
name="snapshot",
field=models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.SET_NULL,
related_name="backup_runs",
to="pobsync_backend.snapshotrecord",
),
),
]

View File

@@ -0,0 +1,44 @@
from __future__ import annotations
import django.db.models.deletion
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("pobsync_backend", "0004_backuprun_snapshot"),
]
operations = [
migrations.AddField(
model_name="snapshotrecord",
name="base",
field=models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.SET_NULL,
related_name="derived_snapshots",
to="pobsync_backend.snapshotrecord",
),
),
migrations.AddField(
model_name="snapshotrecord",
name="base_dirname",
field=models.CharField(blank=True, max_length=255),
),
migrations.AddField(
model_name="snapshotrecord",
name="base_snapshot_id",
field=models.CharField(blank=True, max_length=64),
),
migrations.AddField(
model_name="snapshotrecord",
name="base_kind",
field=models.CharField(blank=True, max_length=16),
),
migrations.AddField(
model_name="snapshotrecord",
name="base_path",
field=models.CharField(blank=True, max_length=1024),
),
]

View File

@@ -0,0 +1,51 @@
# Generated by Django 5.2.14 on 2026-05-19
import django.db.models.deletion
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("pobsync_backend", "0005_snapshotrecord_base"),
]
operations = [
migrations.CreateModel(
name="SshCredential",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
("name", models.CharField(max_length=128, unique=True)),
("private_key", models.TextField()),
("public_key", models.TextField(blank=True)),
("known_hosts", models.TextField(blank=True)),
("notes", models.TextField(blank=True)),
],
options={
"ordering": ["name"],
},
),
migrations.AddField(
model_name="globalconfig",
name="default_ssh_credential",
field=models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.SET_NULL,
related_name="global_configs",
to="pobsync_backend.sshcredential",
),
),
migrations.AddField(
model_name="hostconfig",
name="ssh_credential",
field=models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.SET_NULL,
related_name="hosts",
to="pobsync_backend.sshcredential",
),
),
]

View File

@@ -0,0 +1,35 @@
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("pobsync_backend", "0006_ssh_credentials"),
]
operations = [
migrations.AlterField(
model_name="sshcredential",
name="private_key",
field=models.TextField(blank=True, default=""),
),
migrations.AddField(
model_name="sshcredential",
name="key_path",
field=models.CharField(blank=True, max_length=1024),
),
migrations.AddField(
model_name="sshcredential",
name="key_type",
field=models.CharField(default="ed25519", max_length=32),
),
migrations.AddField(
model_name="sshcredential",
name="fingerprint",
field=models.CharField(blank=True, max_length=255),
),
migrations.AddField(
model_name="sshcredential",
name="generated",
field=models.BooleanField(default=False),
),
]

View File

@@ -0,0 +1,27 @@
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("pobsync_backend", "0007_filesystem_ssh_credentials"),
]
operations = [
migrations.AlterField(
model_name="backuprun",
name="status",
field=models.CharField(
choices=[
("queued", "Queued"),
("running", "Running"),
("success", "Success"),
("warning", "Warning"),
("failed", "Failed"),
("cancelled", "Cancelled"),
],
default="queued",
max_length=16,
),
),
]

View File

@@ -0,0 +1,14 @@
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
("pobsync_backend", "0008_alter_backuprun_status"),
]
operations = [
migrations.RemoveField(
model_name="scheduleconfig",
name="user",
),
]

View File

@@ -0,0 +1,14 @@
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
("pobsync_backend", "0009_remove_scheduleconfig_user"),
]
operations = [
migrations.RemoveField(
model_name="globalconfig",
name="pobsync_home",
),
]

View File

@@ -0,0 +1,14 @@
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
("pobsync_backend", "0010_remove_globalconfig_pobsync_home"),
]
operations = [
migrations.RemoveField(
model_name="globalconfig",
name="data",
),
]

View File

@@ -0,0 +1,30 @@
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("pobsync_backend", "0011_remove_globalconfig_data"),
]
operations = [
migrations.AddField(
model_name="backuprun",
name="reviewed_at",
field=models.DateTimeField(blank=True, null=True),
),
migrations.AddField(
model_name="backuprun",
name="reviewed_by",
field=models.CharField(blank=True, max_length=150),
),
migrations.AddField(
model_name="snapshotrecord",
name="reviewed_at",
field=models.DateTimeField(blank=True, null=True),
),
migrations.AddField(
model_name="snapshotrecord",
name="reviewed_by",
field=models.CharField(blank=True, max_length=150),
),
]

View File

@@ -0,0 +1,50 @@
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
("pobsync_backend", "0012_review_state"),
]
operations = [
migrations.CreateModel(
name="PurgedSnapshot",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("host_name", models.CharField(max_length=255)),
("kind", models.CharField(max_length=16)),
("dirname", models.CharField(max_length=255)),
("path", models.CharField(max_length=1024)),
("reason", models.CharField(blank=True, max_length=512)),
(
"action",
models.CharField(
choices=[
("manual", "Manual"),
("scheduled", "Scheduled"),
("cli", "CLI"),
("incomplete_cleanup", "Incomplete cleanup"),
],
max_length=32,
),
),
("triggered_by", models.CharField(blank=True, max_length=150)),
("metadata", models.JSONField(blank=True, default=dict)),
("purged_at", models.DateTimeField(auto_now_add=True)),
(
"host",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.SET_NULL,
related_name="purged_snapshots",
to="pobsync_backend.hostconfig",
),
),
],
options={
"ordering": ["-purged_at", "host_name", "dirname"],
},
),
]

View File

@@ -14,7 +14,13 @@ class TimestampedModel(models.Model):
class GlobalConfig(TimestampedModel): class GlobalConfig(TimestampedModel):
name = models.CharField(max_length=64, default="default", unique=True) name = models.CharField(max_length=64, default="default", unique=True)
backup_root = models.CharField(max_length=512) backup_root = models.CharField(max_length=512)
pobsync_home = models.CharField(max_length=512, default="/opt/pobsync") default_ssh_credential = models.ForeignKey(
"SshCredential",
on_delete=models.SET_NULL,
null=True,
blank=True,
related_name="global_configs",
)
ssh_user = models.CharField(max_length=64, default="root") ssh_user = models.CharField(max_length=64, default="root")
ssh_port = models.PositiveIntegerField(default=22) ssh_port = models.PositiveIntegerField(default=22)
ssh_options = models.JSONField(default=list, blank=True) ssh_options = models.JSONField(default=list, blank=True)
@@ -30,7 +36,6 @@ class GlobalConfig(TimestampedModel):
retention_weekly = models.PositiveIntegerField(default=8) retention_weekly = models.PositiveIntegerField(default=8)
retention_monthly = models.PositiveIntegerField(default=12) retention_monthly = models.PositiveIntegerField(default=12)
retention_yearly = models.PositiveIntegerField(default=0) retention_yearly = models.PositiveIntegerField(default=0)
data = models.JSONField(default=dict, blank=True)
class Meta: class Meta:
verbose_name = "global config" verbose_name = "global config"
@@ -44,6 +49,13 @@ class HostConfig(TimestampedModel):
host = models.CharField(max_length=255, unique=True) host = models.CharField(max_length=255, unique=True)
address = models.CharField(max_length=255) address = models.CharField(max_length=255)
enabled = models.BooleanField(default=True) enabled = models.BooleanField(default=True)
ssh_credential = models.ForeignKey(
"SshCredential",
on_delete=models.SET_NULL,
null=True,
blank=True,
related_name="hosts",
)
ssh_user = models.CharField(max_length=64, blank=True) ssh_user = models.CharField(max_length=64, blank=True)
ssh_port = models.PositiveIntegerField(null=True, blank=True) ssh_port = models.PositiveIntegerField(null=True, blank=True)
source_root = models.CharField(max_length=512, blank=True) source_root = models.CharField(max_length=512, blank=True)
@@ -64,6 +76,24 @@ class HostConfig(TimestampedModel):
return self.host return self.host
class SshCredential(TimestampedModel):
name = models.CharField(max_length=128, unique=True)
private_key = models.TextField(blank=True, default="")
public_key = models.TextField(blank=True)
key_path = models.CharField(max_length=1024, blank=True)
key_type = models.CharField(max_length=32, default="ed25519")
fingerprint = models.CharField(max_length=255, blank=True)
generated = models.BooleanField(default=False)
known_hosts = models.TextField(blank=True)
notes = models.TextField(blank=True)
class Meta:
ordering = ["name"]
def __str__(self) -> str:
return self.name
class BackupRun(models.Model): class BackupRun(models.Model):
class RunType(models.TextChoices): class RunType(models.TextChoices):
SCHEDULED = "scheduled", "Scheduled" SCHEDULED = "scheduled", "Scheduled"
@@ -73,6 +103,7 @@ class BackupRun(models.Model):
QUEUED = "queued", "Queued" QUEUED = "queued", "Queued"
RUNNING = "running", "Running" RUNNING = "running", "Running"
SUCCESS = "success", "Success" SUCCESS = "success", "Success"
WARNING = "warning", "Warning"
FAILED = "failed", "Failed" FAILED = "failed", "Failed"
CANCELLED = "cancelled", "Cancelled" CANCELLED = "cancelled", "Cancelled"
@@ -82,10 +113,19 @@ class BackupRun(models.Model):
started_at = models.DateTimeField(null=True, blank=True) started_at = models.DateTimeField(null=True, blank=True)
ended_at = models.DateTimeField(null=True, blank=True) ended_at = models.DateTimeField(null=True, blank=True)
snapshot_path = models.CharField(max_length=1024, blank=True) snapshot_path = models.CharField(max_length=1024, blank=True)
snapshot = models.ForeignKey(
"SnapshotRecord",
on_delete=models.SET_NULL,
null=True,
blank=True,
related_name="backup_runs",
)
base_path = models.CharField(max_length=1024, blank=True) base_path = models.CharField(max_length=1024, blank=True)
rsync_exit_code = models.IntegerField(null=True, blank=True) rsync_exit_code = models.IntegerField(null=True, blank=True)
result = models.JSONField(default=dict, blank=True) result = models.JSONField(default=dict, blank=True)
created_at = models.DateTimeField(auto_now_add=True) created_at = models.DateTimeField(auto_now_add=True)
reviewed_at = models.DateTimeField(null=True, blank=True)
reviewed_by = models.CharField(max_length=150, blank=True)
class Meta: class Meta:
ordering = ["-created_at"] ordering = ["-created_at"]
@@ -104,11 +144,24 @@ class SnapshotRecord(models.Model):
kind = models.CharField(max_length=16, choices=Kind.choices) kind = models.CharField(max_length=16, choices=Kind.choices)
dirname = models.CharField(max_length=255) dirname = models.CharField(max_length=255)
path = models.CharField(max_length=1024) path = models.CharField(max_length=1024)
base = models.ForeignKey(
"self",
on_delete=models.SET_NULL,
null=True,
blank=True,
related_name="derived_snapshots",
)
base_kind = models.CharField(max_length=16, blank=True)
base_dirname = models.CharField(max_length=255, blank=True)
base_path = models.CharField(max_length=1024, blank=True)
base_snapshot_id = models.CharField(max_length=64, blank=True)
status = models.CharField(max_length=32, blank=True) status = models.CharField(max_length=32, blank=True)
started_at = models.DateTimeField(null=True, blank=True) started_at = models.DateTimeField(null=True, blank=True)
ended_at = models.DateTimeField(null=True, blank=True) ended_at = models.DateTimeField(null=True, blank=True)
metadata = models.JSONField(default=dict, blank=True) metadata = models.JSONField(default=dict, blank=True)
discovered_at = models.DateTimeField(auto_now_add=True) discovered_at = models.DateTimeField(auto_now_add=True)
reviewed_at = models.DateTimeField(null=True, blank=True)
reviewed_by = models.CharField(max_length=150, blank=True)
class Meta: class Meta:
constraints = [ constraints = [
@@ -120,10 +173,34 @@ class SnapshotRecord(models.Model):
return f"{self.host}/{self.kind}/{self.dirname}" return f"{self.host}/{self.kind}/{self.dirname}"
class PurgedSnapshot(models.Model):
class Action(models.TextChoices):
MANUAL = "manual", "Manual"
SCHEDULED = "scheduled", "Scheduled"
CLI = "cli", "CLI"
INCOMPLETE_CLEANUP = "incomplete_cleanup", "Incomplete cleanup"
host = models.ForeignKey(HostConfig, on_delete=models.SET_NULL, null=True, blank=True, related_name="purged_snapshots")
host_name = models.CharField(max_length=255)
kind = models.CharField(max_length=16)
dirname = models.CharField(max_length=255)
path = models.CharField(max_length=1024)
reason = models.CharField(max_length=512, blank=True)
action = models.CharField(max_length=32, choices=Action.choices)
triggered_by = models.CharField(max_length=150, blank=True)
metadata = models.JSONField(default=dict, blank=True)
purged_at = models.DateTimeField(auto_now_add=True)
class Meta:
ordering = ["-purged_at", "host_name", "dirname"]
def __str__(self) -> str:
return f"{self.host_name}/{self.kind}/{self.dirname}"
class ScheduleConfig(TimestampedModel): class ScheduleConfig(TimestampedModel):
host = models.OneToOneField(HostConfig, on_delete=models.CASCADE, related_name="schedule") host = models.OneToOneField(HostConfig, on_delete=models.CASCADE, related_name="schedule")
cron_expr = models.CharField(max_length=128) cron_expr = models.CharField(max_length=128)
user = models.CharField(max_length=64, default="root")
enabled = models.BooleanField(default=True) enabled = models.BooleanField(default=True)
prune = models.BooleanField(default=False) prune = models.BooleanField(default=False)
prune_max_delete = models.PositiveIntegerField(default=10) prune_max_delete = models.PositiveIntegerField(default=10)

View File

@@ -0,0 +1,232 @@
from __future__ import annotations
import shlex
import subprocess
from dataclasses import dataclass
from typing import Any
from pobsync.config.merge import build_effective_config
from pobsync.rsync import build_ssh_command
from .config_repository import global_config_object_data, host_config_object_data
from .config_source import DjangoConfigSource
from .host_ops import collect_host_checks
from .models import GlobalConfig, HostConfig
from .self_check import SelfCheck
DRY_RUN_BLOCKING_CHECKS = {
"Host global config",
"Host address",
"Host SSH key file",
"Host effective source root",
"Host effective SSH user",
"Host effective SSH port",
"Host effective SSH credential",
"Host effective rsync recursion",
}
@dataclass(frozen=True)
class BackupGate:
state: str
message: str
checks: list[SelfCheck]
real_blockers: list[SelfCheck]
dry_run_blockers: list[SelfCheck]
warnings: list[SelfCheck]
@property
def can_queue_real(self) -> bool:
return not self.real_blockers
@property
def can_queue_dry_run(self) -> bool:
return not self.dry_run_blockers
def collect_backup_gate(host: HostConfig, global_config: GlobalConfig | None = None) -> BackupGate:
checks = collect_host_checks(host, global_config)
remote_preflight_check = _remote_preflight_self_check(host)
if remote_preflight_check is not None:
checks.append(remote_preflight_check)
real_blockers = [check for check in checks if check.status == "failed"]
dry_run_blockers = [check for check in real_blockers if check.name in DRY_RUN_BLOCKING_CHECKS]
warnings = [check for check in checks if check.status == "warning"]
if real_blockers:
state = "blocked"
message = "Real backups are blocked until failed host checks are resolved."
elif warnings:
state = "warning"
message = "Backups can run, but review the warnings first."
else:
state = "ready"
message = "This host is ready for backup runs."
return BackupGate(
state=state,
message=message,
checks=checks,
real_blockers=real_blockers,
dry_run_blockers=dry_run_blockers,
warnings=warnings,
)
def run_remote_preflight(host: HostConfig, *, timeout_seconds: int = 20) -> dict[str, Any]:
config = DjangoConfigSource().effective_config_for_host(host.host)
ssh_cfg = config.get("ssh", {}) or {}
rsync_cfg = config.get("rsync", {}) or {}
address = str(config.get("address") or host.address)
user = str(ssh_cfg.get("user") or "root")
source_root = str(config.get("source_root") or (config.get("defaults", {}) or {}).get("source_root") or "/")
rsync_binary = str(rsync_cfg.get("binary") or "rsync")
target = f"{user}@{address}"
ssh_cmd = build_ssh_command(ssh_cfg)
checks = [
_run_remote_check(
name="SSH reachability",
command=[*ssh_cmd, "-oBatchMode=yes", target, "true"],
timeout_seconds=timeout_seconds,
),
_run_remote_check(
name="Remote rsync",
command=[
*ssh_cmd,
"-oBatchMode=yes",
target,
"sh",
"-lc",
f"command -v {shlex.quote(rsync_binary)} >/dev/null",
],
timeout_seconds=timeout_seconds,
),
_run_remote_check(
name="Remote source root",
command=[
*ssh_cmd,
"-oBatchMode=yes",
target,
"sh",
"-lc",
f"test -e {shlex.quote(source_root)} && test -r {shlex.quote(source_root)}",
],
timeout_seconds=timeout_seconds,
),
]
result = {
"ok": all(check["ok"] for check in checks),
"checks": checks,
"target": target,
"source_root": source_root,
"rsync_binary": rsync_binary,
"timeout_seconds": timeout_seconds,
}
host.config = {**(host.config or {}), "last_preflight": result}
host.save(update_fields=["config", "updated_at"])
return result
def effective_host_config_preview(host: HostConfig, global_config: GlobalConfig) -> dict[str, Any]:
config = build_effective_config(global_config_object_data(global_config), host_config_object_data(host))
credential = host.ssh_credential or global_config.default_ssh_credential
ssh = config.get("ssh", {}) or {}
rsync = config.get("rsync", {}) or {}
retention = config.get("retention", {}) or {}
return {
"source_root": config.get("source_root", ""),
"destination_subdir": (config.get("defaults", {}) or {}).get("destination_subdir", ""),
"includes": list(config.get("includes") or []),
"excludes": list(config.get("excludes_effective") or []),
"ssh": {
"user": ssh.get("user", ""),
"port": ssh.get("port", ""),
"options": list(ssh.get("options") or []),
"credential": str(credential) if credential else "",
},
"rsync": {
"binary": rsync.get("binary", ""),
"args": list(rsync.get("args_effective") or []),
"timeout_seconds": rsync.get("timeout_seconds", 0),
"bwlimit_kbps": rsync.get("bwlimit_kbps", 0),
},
"retention": {
"daily": retention.get("daily", 0),
"weekly": retention.get("weekly", 0),
"monthly": retention.get("monthly", 0),
"yearly": retention.get("yearly", 0),
},
}
def _run_remote_check(*, name: str, command: list[str], timeout_seconds: int) -> dict[str, Any]:
try:
result = subprocess.run(
command,
check=False,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
timeout=timeout_seconds,
)
except subprocess.TimeoutExpired as exc:
return {
"name": name,
"ok": False,
"exit_code": 124,
"message": f"{name} timed out after {timeout_seconds}s.",
"detail": _clip_output((exc.stderr or exc.stdout or "").strip()),
}
except OSError as exc:
return {
"name": name,
"ok": False,
"exit_code": None,
"message": f"{name} could not start.",
"detail": str(exc),
}
return {
"name": name,
"ok": result.returncode == 0,
"exit_code": result.returncode,
"message": f"{name} passed." if result.returncode == 0 else f"{name} failed.",
"detail": _clip_output((result.stderr or result.stdout or "").strip()),
}
def _remote_preflight_self_check(host: HostConfig) -> SelfCheck | None:
preflight = (host.config or {}).get("last_preflight")
if not isinstance(preflight, dict):
return SelfCheck(
"Remote preflight",
"warning",
"No remote connection preflight has been run yet.",
"Run connection preflight before the first real backup.",
)
checks = preflight.get("checks")
if not isinstance(checks, list):
return SelfCheck("Remote preflight", "failed", "Stored remote preflight result is invalid.")
failed = [str(check.get("name", "unknown")) for check in checks if isinstance(check, dict) and not check.get("ok")]
if failed:
return SelfCheck(
"Remote preflight",
"failed",
"Remote connection preflight failed.",
", ".join(failed),
)
return SelfCheck(
"Remote preflight",
"ok",
"Remote connection preflight passed.",
f"{preflight.get('target', '')} {preflight.get('source_root', '')}".strip(),
)
def _clip_output(value: str, *, max_chars: int = 800) -> str:
if len(value) <= max_chars:
return value
return f"{value[:max_chars]}..."

View File

@@ -0,0 +1,352 @@
from __future__ import annotations
import shutil
import stat
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from pobsync.errors import ConfigError
from pobsync.lock import acquire_host_lock
from pobsync.paths import PobsyncPaths
from pobsync.retention import Snapshot, apply_base_protection, build_retention_plan
from pobsync.util import sanitize_host
from .models import HostConfig, PurgedSnapshot, SnapshotRecord
def run_sql_retention_plan(*, host: str, kind: str, protect_bases: bool) -> dict[str, Any]:
host = sanitize_host(host)
if kind not in {"scheduled", "manual", "all"}:
raise ConfigError("kind must be scheduled, manual, or all")
host_config = _enabled_host_config(host)
retention = _retention_for_host(host_config)
snapshots = _snapshots_for_retention(host_config=host_config, kind=kind)
incomplete_snapshots = _incomplete_snapshots_for_host(host_config)
plan = build_retention_plan(
snapshots=snapshots,
retention=retention,
now=datetime.now(timezone.utc),
)
keep = set(plan.keep)
reasons = dict(plan.reasons)
if protect_bases:
keep, reasons = apply_base_protection(snapshots=snapshots, keep=keep, reasons=reasons)
delete = [snapshot for snapshot in snapshots if snapshot.dirname not in keep]
keep_items = [snapshot for snapshot in snapshots if snapshot.dirname in keep]
return {
"ok": True,
"host": host,
"kind": kind,
"protect_bases": bool(protect_bases),
"retention": retention,
"source": "sql",
"keep": sorted(keep),
"keep_items": [_snapshot_to_item(snapshot, reasons=reasons.get(snapshot.dirname, [])) for snapshot in keep_items],
"delete": [_snapshot_to_item(snapshot, reasons=["outside retention policy"]) for snapshot in delete],
"incomplete": [
_snapshot_to_item(snapshot, reasons=["incomplete snapshot; excluded from retention cleanup"])
for snapshot in incomplete_snapshots
],
"reasons": reasons,
}
def run_sql_retention_apply(
*,
prefix: Path,
host: str,
kind: str,
protect_bases: bool,
yes: bool,
max_delete: int,
action: str = PurgedSnapshot.Action.MANUAL,
triggered_by: str = "",
acquire_lock: bool = True,
) -> dict[str, Any]:
host = sanitize_host(host)
if not yes:
raise ConfigError("Refusing to delete snapshots without --yes")
if max_delete < 0:
raise ConfigError("--max-delete must be >= 0")
paths = PobsyncPaths(home=prefix)
def _do_apply() -> dict[str, Any]:
plan = run_sql_retention_plan(host=host, kind=kind, protect_bases=bool(protect_bases))
delete_list = plan.get("delete") or []
incomplete_list = plan.get("incomplete") or []
if not isinstance(delete_list, list):
raise ConfigError("Invalid retention plan output: delete is not a list")
if not isinstance(incomplete_list, list):
raise ConfigError("Invalid retention plan output: incomplete is not a list")
if max_delete == 0 and len(delete_list) > 0:
raise ConfigError("Deletion blocked by --max-delete=0")
if len(delete_list) > max_delete:
raise ConfigError(f"Refusing to delete {len(delete_list)} snapshots (exceeds --max-delete={max_delete})")
actions: list[str] = []
deleted: list[dict[str, Any]] = []
for item in delete_list:
dirname = item.get("dirname") if isinstance(item, dict) else None
snap_kind = item.get("kind") if isinstance(item, dict) else None
snap_path = item.get("path") if isinstance(item, dict) else None
if not isinstance(dirname, str) or not isinstance(snap_kind, str) or not isinstance(snap_path, str):
continue
if snap_kind not in {"scheduled", "manual"}:
raise ConfigError(f"Refusing to delete unsupported snapshot kind: {snap_kind!r}")
path = _snapshot_delete_path(path=Path(snap_path), dirname=dirname)
reason = str(item.get("reason") or "outside retention policy")
if not path.exists():
actions.append(f"skip missing {snap_kind}/{dirname}")
continue
if not path.is_dir():
raise ConfigError(f"Refusing to delete non-directory path: {path}")
_remove_snapshot_tree(path)
_record_purged_snapshot(
host_config=_enabled_host_config(host),
kind=snap_kind,
dirname=dirname,
path=path,
reason=reason,
action=action,
triggered_by=triggered_by,
metadata={"source": "retention", "protect_bases": bool(protect_bases), "retention_kind": kind},
)
SnapshotRecord.objects.filter(host__host=host, kind=snap_kind, dirname=dirname).delete()
actions.append(f"deleted {snap_kind} {dirname}")
deleted.append({"dirname": dirname, "kind": snap_kind, "path": str(path), "reason": reason})
return {
"ok": True,
"host": host,
"kind": kind,
"protect_bases": bool(protect_bases),
"max_delete": max_delete,
"source": "sql",
"planned_delete_count": len(delete_list),
"incomplete_ignored_count": len(incomplete_list),
"deleted": deleted,
"actions": actions,
}
if acquire_lock:
with acquire_host_lock(paths.locks_dir, host, command="retention-apply"):
return _do_apply()
return _do_apply()
def run_incomplete_cleanup(
*,
prefix: Path,
host: str,
yes: bool,
max_delete: int,
triggered_by: str = "",
acquire_lock: bool = True,
) -> dict[str, Any]:
host = sanitize_host(host)
if not yes:
raise ConfigError("Refusing to delete incomplete snapshots without --yes")
if max_delete < 0:
raise ConfigError("--max-delete must be >= 0")
paths = PobsyncPaths(home=prefix)
def _do_cleanup() -> dict[str, Any]:
host_config = _enabled_host_config(host)
incomplete_list = [
_snapshot_to_item(snapshot, reasons=["manual incomplete cleanup"])
for snapshot in _incomplete_snapshots_for_host(host_config)
]
if max_delete == 0 and len(incomplete_list) > 0:
raise ConfigError("Incomplete cleanup blocked by --max-delete=0")
if len(incomplete_list) > max_delete:
raise ConfigError(
f"Refusing to delete {len(incomplete_list)} incomplete snapshots (exceeds --max-delete={max_delete})"
)
actions: list[str] = []
deleted: list[dict[str, Any]] = []
for item in incomplete_list:
dirname = item["dirname"]
snap_path = Path(item["path"])
path = _snapshot_delete_path(path=snap_path, dirname=dirname)
_validate_incomplete_delete_path(host=host, path=path, dirname=dirname)
if not path.exists():
actions.append(f"skip missing incomplete/{dirname}")
elif not path.is_dir():
raise ConfigError(f"Refusing to delete non-directory path: {path}")
else:
_remove_snapshot_tree(path)
actions.append(f"deleted incomplete {dirname}")
_record_purged_snapshot(
host_config=host_config,
kind=SnapshotRecord.Kind.INCOMPLETE,
dirname=dirname,
path=path,
reason="manual incomplete cleanup",
action=PurgedSnapshot.Action.INCOMPLETE_CLEANUP,
triggered_by=triggered_by,
metadata={"source": "incomplete_cleanup"},
)
SnapshotRecord.objects.filter(
host__host=host,
kind=SnapshotRecord.Kind.INCOMPLETE,
dirname=dirname,
).delete()
deleted.append({"dirname": dirname, "kind": SnapshotRecord.Kind.INCOMPLETE, "path": str(path)})
return {
"ok": True,
"host": host,
"kind": SnapshotRecord.Kind.INCOMPLETE,
"max_delete": max_delete,
"source": "sql",
"planned_delete_count": len(incomplete_list),
"deleted": deleted,
"actions": actions,
}
if acquire_lock:
with acquire_host_lock(paths.locks_dir, host, command="incomplete-cleanup"):
return _do_cleanup()
return _do_cleanup()
def _enabled_host_config(host: str) -> HostConfig:
try:
return HostConfig.objects.get(host=host, enabled=True)
except HostConfig.DoesNotExist as exc:
raise ConfigError(f"Missing enabled host {host!r}") from exc
def _retention_for_host(host_config: HostConfig) -> dict[str, int]:
return {
"daily": host_config.retention_daily,
"weekly": host_config.retention_weekly,
"monthly": host_config.retention_monthly,
"yearly": host_config.retention_yearly,
}
def _snapshots_for_retention(*, host_config: HostConfig, kind: str) -> list[Snapshot]:
kinds = ["scheduled", "manual"] if kind == "all" else [kind]
records = (
SnapshotRecord.objects.filter(host=host_config, kind__in=kinds)
.exclude(kind=SnapshotRecord.Kind.INCOMPLETE)
.select_related("base")
.order_by("-started_at", "dirname")
)
return [_snapshot_from_record(record) for record in records]
def _incomplete_snapshots_for_host(host_config: HostConfig) -> list[Snapshot]:
records = (
SnapshotRecord.objects.filter(host=host_config, kind=SnapshotRecord.Kind.INCOMPLETE)
.select_related("base")
.order_by("-started_at", "dirname")
)
return [_snapshot_from_record(record) for record in records]
def _snapshot_from_record(record: SnapshotRecord) -> Snapshot:
return Snapshot(
kind=record.kind,
dirname=record.dirname,
path=record.path,
dt=record.started_at or datetime.fromtimestamp(0, tz=timezone.utc),
status=record.status or None,
base=_base_meta_from_record(record),
)
def _base_meta_from_record(record: SnapshotRecord) -> dict[str, str] | None:
if record.base is not None:
return {
"kind": record.base.kind,
"dirname": record.base.dirname,
"path": record.base.path,
}
if record.base_kind and record.base_dirname:
return {
"kind": record.base_kind,
"dirname": record.base_dirname,
"path": record.base_path,
}
return None
def _snapshot_to_item(snapshot: Snapshot, *, reasons: list[str]) -> dict[str, Any]:
return {
"dirname": snapshot.dirname,
"kind": snapshot.kind,
"path": snapshot.path,
"dt": snapshot.dt.isoformat(),
"status": snapshot.status,
"reasons": reasons,
"reason": ", ".join(reasons),
}
def _snapshot_delete_path(*, path: Path, dirname: str) -> Path:
if path.name == "data" and path.parent.name == dirname:
return path.parent
return path
def _record_purged_snapshot(
*,
host_config: HostConfig,
kind: str,
dirname: str,
path: Path,
reason: str,
action: str,
triggered_by: str,
metadata: dict[str, Any],
) -> None:
PurgedSnapshot.objects.create(
host=host_config,
host_name=host_config.host,
kind=kind,
dirname=dirname,
path=str(path),
reason=reason,
action=action,
triggered_by=triggered_by,
metadata=metadata,
)
def _validate_incomplete_delete_path(*, host: str, path: Path, dirname: str) -> None:
path_parts = path.parts
if path.name != dirname or ".incomplete" not in path_parts or host not in path_parts:
raise ConfigError(f"Refusing to delete unexpected incomplete snapshot path: {path}")
incomplete_index = path_parts.index(".incomplete")
if incomplete_index == 0 or path_parts[incomplete_index - 1] != host:
raise ConfigError(f"Refusing to delete incomplete snapshot outside host backup root: {path}")
def _remove_snapshot_tree(path: Path) -> None:
_make_directories_user_writable(path)
shutil.rmtree(path)
def _make_directories_user_writable(path: Path) -> None:
for directory in [path, *[child for child in path.rglob("*") if child.is_dir() and not child.is_symlink()]]:
mode = directory.stat().st_mode
if mode & stat.S_IWUSR:
continue
directory.chmod(mode | stat.S_IWUSR)

View File

@@ -1,7 +1,7 @@
from __future__ import annotations from __future__ import annotations
from dataclasses import dataclass from dataclasses import dataclass
from datetime import datetime from datetime import datetime, timedelta
@dataclass(frozen=True) @dataclass(frozen=True)
@@ -36,6 +36,17 @@ def is_due(expr: str, moment: datetime) -> bool:
) )
def next_due_after(expr: str, moment: datetime, *, max_days: int = 366) -> datetime | None:
parse_cron_expr(expr)
candidate = moment.replace(second=0, microsecond=0) + timedelta(minutes=1)
deadline = candidate + timedelta(days=max_days)
while candidate <= deadline:
if is_due(expr, candidate):
return candidate
candidate += timedelta(minutes=1)
return None
def _field_matches(field: str, value: int, min_value: int, max_value: int, sunday_alias: bool = False) -> bool: def _field_matches(field: str, value: int, min_value: int, max_value: int, sunday_alias: bool = False) -> bool:
for part in field.split(","): for part in field.split(","):
if _part_matches(part.strip(), value, min_value, max_value, sunday_alias=sunday_alias): if _part_matches(part.strip(), value, min_value, max_value, sunday_alias=sunday_alias):

View File

@@ -0,0 +1,329 @@
from __future__ import annotations
import os
import pwd
import shutil
import subprocess
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Literal
from django.conf import settings
from django.db import connection
from .models import GlobalConfig
CheckStatus = Literal["ok", "warning", "failed", "skipped"]
@dataclass(frozen=True)
class SelfCheck:
name: str
status: CheckStatus
message: str
detail: str = ""
def collect_self_checks() -> list[SelfCheck]:
checks: list[SelfCheck] = []
checks.extend(_django_checks())
checks.extend(_install_checks())
checks.extend(_path_checks())
checks.extend(_binary_checks())
checks.extend(_database_checks())
checks.extend(_config_checks())
checks.extend(_systemd_checks())
return checks
def _native_runtime_available() -> bool:
return Path("/run/systemd/system").exists() and shutil.which("systemctl") is not None
def summarize_self_checks(checks: list[SelfCheck]) -> dict[str, int]:
return {
"ok": sum(1 for check in checks if check.status == "ok"),
"warning": sum(1 for check in checks if check.status == "warning"),
"failed": sum(1 for check in checks if check.status == "failed"),
"skipped": sum(1 for check in checks if check.status == "skipped"),
}
def _django_checks() -> list[SelfCheck]:
checks = [
SelfCheck(
"Django debug",
"warning" if settings.DEBUG else "ok",
"DEBUG is enabled." if settings.DEBUG else "DEBUG is disabled.",
),
SelfCheck(
"Django secret key",
"failed" if settings.SECRET_KEY == "dev-only-change-me" else "ok",
"Default development secret key is still active."
if settings.SECRET_KEY == "dev-only-change-me"
else "Secret key is configured.",
),
SelfCheck(
"Allowed hosts",
"ok" if settings.ALLOWED_HOSTS else "failed",
", ".join(settings.ALLOWED_HOSTS) if settings.ALLOWED_HOSTS else "No allowed hosts configured.",
),
]
return checks
def _path_checks() -> list[SelfCheck]:
checks = []
checks.append(
_path_check(
"State root",
Path(settings.POBSYNC_HOME),
must_be_absolute=True,
must_be_writable=True,
)
)
checks.append(
_path_check(
"Backup root",
Path(settings.POBSYNC_BACKUP_ROOT),
must_be_absolute=True,
must_exist=True,
must_be_writable=True,
)
)
checks.append(
_path_check(
"Static root",
Path(settings.STATIC_ROOT),
must_be_absolute=True,
must_exist=False,
must_be_writable=True,
)
)
db_settings = settings.DATABASES["default"]
if db_settings["ENGINE"] == "django.db.backends.sqlite3":
sqlite_path = Path(str(db_settings["NAME"]))
checks.append(
_path_check(
"SQLite directory",
sqlite_path.parent,
must_be_absolute=True,
must_exist=True,
must_be_writable=True,
)
)
checks.append(_sqlite_database_check(sqlite_path))
return checks
def _install_checks() -> list[SelfCheck]:
if not _native_runtime_available() and not Path(settings.POBSYNC_ENV_FILE).exists():
return [
SelfCheck(
"Environment file",
"skipped",
"Native environment file is not configured in this runtime.",
"This is expected inside Docker or local development.",
),
SelfCheck(
"Service user",
"skipped",
"Native service user check is not available in this runtime.",
"This is expected inside Docker or local development.",
),
SelfCheck(
"Backup root owner",
"skipped",
"Native backup root ownership check is not available in this runtime.",
"This is expected inside Docker or local development.",
),
]
checks = [_env_file_check(Path(settings.POBSYNC_ENV_FILE)), _service_user_check()]
checks.append(_backup_root_owner_check(Path(settings.POBSYNC_BACKUP_ROOT)))
return checks
def _env_file_check(path: Path) -> SelfCheck:
if not path.is_absolute():
return SelfCheck("Environment file", "failed", f"{path} is not absolute.")
if not path.exists():
return SelfCheck("Environment file", "failed", f"{path} does not exist.")
if not path.is_file():
return SelfCheck("Environment file", "failed", f"{path} is not a regular file.")
if not os.access(path, os.R_OK):
return SelfCheck("Environment file", "failed", f"{path} is not readable by this process.")
return SelfCheck("Environment file", "ok", str(path))
def _service_user_check() -> SelfCheck:
expected_user = settings.POBSYNC_SERVICE_USER
try:
current_user = pwd.getpwuid(os.geteuid()).pw_name
except KeyError:
return SelfCheck("Service user", "failed", f"Current uid {os.geteuid()} has no passwd entry.")
if current_user != expected_user:
return SelfCheck(
"Service user",
"warning",
f"Current process runs as {current_user}, expected {expected_user}.",
"Run terminal checks with sudo -u <service-user> pobsync-manage check_pobsync_install.",
)
return SelfCheck("Service user", "ok", current_user)
def _backup_root_owner_check(path: Path) -> SelfCheck:
if not path.exists():
return SelfCheck("Backup root owner", "failed", f"{path} does not exist.")
expected_user = settings.POBSYNC_SERVICE_USER
try:
owner = pwd.getpwuid(path.stat().st_uid).pw_name
except KeyError:
return SelfCheck("Backup root owner", "warning", f"{path} owner uid {path.stat().st_uid} has no passwd entry.")
if owner != expected_user:
return SelfCheck(
"Backup root owner",
"warning",
f"{path} is owned by {owner}, expected {expected_user}.",
)
return SelfCheck("Backup root owner", "ok", f"{path} owner={owner}")
def _sqlite_database_check(path: Path) -> SelfCheck:
if not path.is_absolute():
return SelfCheck("SQLite database", "failed", f"{path} is not absolute.")
if not path.exists():
return SelfCheck("SQLite database", "warning", f"{path} does not exist yet.")
if not path.is_file():
return SelfCheck("SQLite database", "failed", f"{path} is not a regular file.")
if not os.access(path, os.R_OK | os.W_OK):
return SelfCheck("SQLite database", "failed", f"{path} is not readable and writable by this process.")
return SelfCheck("SQLite database", "ok", str(path))
def _path_check(
name: str,
path: Path,
*,
must_be_absolute: bool,
must_exist: bool = False,
must_be_writable: bool,
) -> SelfCheck:
if must_be_absolute and not path.is_absolute():
return SelfCheck(name, "failed", f"{path} is not absolute.")
if must_exist and not path.exists():
return SelfCheck(name, "failed", f"{path} does not exist.")
target = path if path.exists() else path.parent
if not target.exists():
return SelfCheck(name, "failed", f"{target} does not exist.")
if must_be_writable and not os.access(target, os.W_OK):
return SelfCheck(name, "failed", f"{target} is not writable by this process.")
return SelfCheck(name, "ok", str(path))
def _binary_checks() -> list[SelfCheck]:
checks = []
for binary in ("rsync", "ssh", "ssh-keygen"):
path = shutil.which(binary)
checks.append(
SelfCheck(
f"Binary: {binary}",
"ok" if path else "failed",
path or f"{binary} was not found in PATH.",
)
)
gunicorn_path = shutil.which("gunicorn") or Path(sys.executable).parent / "gunicorn"
checks.append(
SelfCheck(
"Binary: gunicorn",
"ok" if Path(gunicorn_path).exists() else "failed",
str(gunicorn_path) if Path(gunicorn_path).exists() else "gunicorn was not found in PATH or next to Python.",
)
)
return checks
def _database_checks() -> list[SelfCheck]:
try:
with connection.cursor() as cursor:
cursor.execute("SELECT 1")
cursor.fetchone()
except Exception as exc:
return [SelfCheck("Database connection", "failed", f"{type(exc).__name__}: {exc}")]
return [SelfCheck("Database connection", "ok", settings.DATABASES["default"]["ENGINE"])]
def _config_checks() -> list[SelfCheck]:
try:
global_config = GlobalConfig.objects.get(name="default")
except GlobalConfig.DoesNotExist:
return [SelfCheck("Global config", "warning", "Default global config has not been created yet.")]
status: CheckStatus = "ok"
message = "Default global config exists."
if global_config.backup_root != settings.POBSYNC_BACKUP_ROOT:
status = "warning"
message = "Saved backup root differs from the active backup root."
return [
SelfCheck(
"Global config",
status,
message,
f"saved={global_config.backup_root} active={settings.POBSYNC_BACKUP_ROOT}",
)
]
def _systemd_checks() -> list[SelfCheck]:
if not _native_runtime_available():
return [
SelfCheck(
"Systemd services",
"skipped",
"systemd is not available in this runtime.",
"This is expected inside Docker.",
)
]
checks = []
for service in ("pobsync-web.service", "pobsync-worker.service", "pobsync-scheduler.service"):
result = subprocess.run(
["systemctl", "is-active", service],
check=False,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
timeout=5,
)
active_state = result.stdout.strip() or result.stderr.strip()
checks.append(
SelfCheck(
service,
"ok" if result.returncode == 0 else "failed",
active_state,
)
)
if shutil.which("journalctl") is not None:
result = subprocess.run(
["journalctl", "--no-pager", "-n", "1", "-u", "pobsync-web.service"],
check=False,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
timeout=5,
)
journal_error = result.stderr.strip()
journal_denied = "No journal files were opened" in journal_error or "permission" in journal_error.lower()
has_journal_access = result.returncode == 0 and not journal_denied
checks.append(
SelfCheck(
"Journal access",
"ok" if has_journal_access else "failed",
"pobsync can read service logs." if has_journal_access else "pobsync cannot read service logs.",
journal_error,
)
)
return checks

View File

@@ -55,6 +55,7 @@ def discover_snapshots(
created += 1 created += 1
else: else:
updated += 1 updated += 1
resolve_base_links(host=host_config)
return { return {
"ok": True, "ok": True,
@@ -64,10 +65,65 @@ def discover_snapshots(
} }
def inspect_snapshot_discovery(
*,
host: HostConfig,
global_config: GlobalConfig | None = None,
kinds: list[str] | None = None,
) -> dict[str, Any]:
try:
global_config = global_config or GlobalConfig.objects.get(name="default")
except GlobalConfig.DoesNotExist:
return {
"ok": False,
"reason": "missing_global_config",
"message": "Create the default global config before discovering snapshots.",
"backup_root": "",
"host_root": "",
"host_root_exists": False,
"kind_counts": {},
"total_candidates": 0,
}
kinds = kinds or ["scheduled", "manual", "incomplete"]
host_root = resolve_host_root(global_config.backup_root, host.host)
kind_counts = {kind: len(list(iter_snapshot_dirs(host_root, kind))) for kind in kinds}
total_candidates = sum(kind_counts.values())
host_root_exists = host_root.exists()
if not host_root_exists:
reason = "missing_host_root"
message = f"Host backup directory does not exist yet: {host_root}"
elif total_candidates == 0:
reason = "no_snapshots"
message = f"No snapshot directories found below {host_root}."
else:
reason = "ready"
message = f"Found {total_candidates} snapshot directories below {host_root}."
return {
"ok": True,
"reason": reason,
"message": message,
"backup_root": str(global_config.backup_root),
"host_root": str(host_root),
"host_root_exists": host_root_exists,
"kind_counts": kind_counts,
"total_candidates": total_candidates,
}
def upsert_snapshot_record(*, host: HostConfig, kind: str, snapshot_dir: Path) -> tuple[SnapshotRecord, bool]: def upsert_snapshot_record(*, host: HostConfig, kind: str, snapshot_dir: Path) -> tuple[SnapshotRecord, bool]:
meta = read_snapshot_meta(snapshot_dir) meta = read_snapshot_meta(snapshot_dir)
base_defaults = _base_defaults_from_meta(meta)
defaults = { defaults = {
"path": str(snapshot_dir), "path": str(snapshot_dir),
**base_defaults,
"base": _resolve_base_record(
host=host,
kind=base_defaults["base_kind"],
dirname=base_defaults["base_dirname"],
),
"status": str(meta.get("status") or ""), "status": str(meta.get("status") or ""),
"started_at": parse_snapshot_datetime(snapshot_dir.name, meta, "started_at"), "started_at": parse_snapshot_datetime(snapshot_dir.name, meta, "started_at"),
"ended_at": parse_snapshot_datetime(snapshot_dir.name, meta, "ended_at"), "ended_at": parse_snapshot_datetime(snapshot_dir.name, meta, "ended_at"),
@@ -81,6 +137,26 @@ def upsert_snapshot_record(*, host: HostConfig, kind: str, snapshot_dir: Path) -
) )
def resolve_base_links(*, host: HostConfig | None = None) -> int:
snapshot_qs = SnapshotRecord.objects.exclude(base_dirname="").filter(base__isnull=True)
if host is not None:
snapshot_qs = snapshot_qs.filter(host=host)
updated = 0
for snapshot in snapshot_qs.select_related("host"):
base = _resolve_base_record(
host=snapshot.host,
kind=snapshot.base_kind,
dirname=snapshot.base_dirname,
)
if base is None:
continue
snapshot.base = base
snapshot.save(update_fields=["base"])
updated += 1
return updated
def infer_snapshot_kind(snapshot_path: Path) -> str: def infer_snapshot_kind(snapshot_path: Path) -> str:
parent = snapshot_path.parent.name parent = snapshot_path.parent.name
if parent == "scheduled": if parent == "scheduled":
@@ -92,6 +168,29 @@ def infer_snapshot_kind(snapshot_path: Path) -> str:
raise ValueError(f"Cannot infer snapshot kind from path: {snapshot_path}") raise ValueError(f"Cannot infer snapshot kind from path: {snapshot_path}")
def _base_defaults_from_meta(meta: dict[str, Any]) -> dict[str, Any]:
base = meta.get("base")
if not isinstance(base, dict):
base = {}
return {
"base_kind": _base_value(base.get("kind")),
"base_dirname": _base_value(base.get("dirname")),
"base_path": _base_value(base.get("path")),
"base_snapshot_id": _base_value(base.get("id")),
}
def _base_value(value: Any) -> str:
return value if isinstance(value, str) else ""
def _resolve_base_record(*, host: HostConfig, kind: str, dirname: str) -> SnapshotRecord | None:
if not kind or not dirname:
return None
return SnapshotRecord.objects.filter(host=host, kind=kind, dirname=dirname).first()
def _parse_iso_z(value: str) -> datetime | None: def _parse_iso_z(value: str) -> datetime | None:
try: try:
if value.endswith("Z"): if value.endswith("Z"):

View File

@@ -0,0 +1,145 @@
from __future__ import annotations
import os
import shutil
import subprocess
from pathlib import Path
from django.conf import settings
from .models import SshCredential
class SshKeyError(RuntimeError):
pass
def credential_dir(credential: SshCredential) -> Path:
return Path(settings.POBSYNC_HOME) / "state" / "ssh-credentials" / str(credential.pk)
def identity_path(credential: SshCredential) -> Path:
if credential.key_path:
return Path(credential.key_path)
return credential_dir(credential) / "identity"
def generate_ssh_key(credential: SshCredential, *, key_type: str = "ed25519", force: bool = False) -> SshCredential:
if credential.pk is None:
raise SshKeyError("Credential must be saved before generating an SSH key.")
if shutil.which("ssh-keygen") is None:
raise SshKeyError("ssh-keygen is not available.")
key_dir = credential_dir(credential)
key_dir.mkdir(mode=0o700, parents=True, exist_ok=True)
os.chmod(key_dir, 0o700)
private_key = key_dir / "identity"
public_key_file = key_dir / "identity.pub"
if force:
private_key.unlink(missing_ok=True)
public_key_file.unlink(missing_ok=True)
elif private_key.exists() or public_key_file.exists():
raise SshKeyError(f"SSH key already exists for {credential.name}.")
result = subprocess.run(
[
"ssh-keygen",
"-t",
key_type,
"-N",
"",
"-C",
f"pobsync:{credential.name}",
"-f",
str(private_key),
],
check=False,
stdin=subprocess.DEVNULL,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
timeout=15,
)
if result.returncode != 0:
raise SshKeyError(result.stderr.strip() or "ssh-keygen failed.")
os.chmod(private_key, 0o600)
public_key = public_key_file.read_text(encoding="utf-8").strip()
fingerprint = fingerprint_for_key(private_key)
credential.private_key = ""
credential.public_key = public_key
credential.key_path = str(private_key)
credential.key_type = key_type
credential.fingerprint = fingerprint
credential.generated = True
credential.save(update_fields=["private_key", "public_key", "key_path", "key_type", "fingerprint", "generated", "updated_at"])
return credential
def fingerprint_for_key(private_key: Path) -> str:
result = subprocess.run(
["ssh-keygen", "-lf", str(private_key)],
check=False,
stdin=subprocess.DEVNULL,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
timeout=5,
)
if result.returncode != 0:
raise SshKeyError(result.stderr.strip() or "Could not fingerprint SSH key.")
return result.stdout.strip()
def delete_generated_key_files(credential: SshCredential) -> None:
path = identity_path(credential)
allowed_root = (Path(settings.POBSYNC_HOME) / "state" / "ssh-credentials").resolve()
try:
resolved = path.resolve()
except FileNotFoundError:
resolved = path
if allowed_root not in resolved.parents:
raise SshKeyError(f"Refusing to delete key outside {allowed_root}.")
path.unlink(missing_ok=True)
path.with_suffix(path.suffix + ".pub").unlink(missing_ok=True)
if path.name == "identity":
(path.parent / "identity.pub").unlink(missing_ok=True)
def scan_known_host(address: str, *, port: int = 22, timeout: int = 5) -> str:
if shutil.which("ssh-keyscan") is None:
raise SshKeyError("ssh-keyscan is not available.")
command = ["ssh-keyscan", "-T", str(timeout), "-p", str(port), address]
result = subprocess.run(
command,
check=False,
stdin=subprocess.DEVNULL,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
timeout=timeout + 2,
)
if result.returncode != 0 and not result.stdout.strip():
raise SshKeyError(result.stderr.strip() or f"Could not scan SSH host key for {address}.")
lines = [line.strip() for line in result.stdout.splitlines() if line.strip() and not line.startswith("#")]
if not lines:
raise SshKeyError(f"ssh-keyscan returned no host keys for {address}.")
return "\n".join(lines)
def merge_known_hosts(existing: str, scanned: str) -> str:
lines: list[str] = []
seen: set[str] = set()
for line in [*existing.splitlines(), *scanned.splitlines()]:
normalized = line.strip()
if not normalized or normalized in seen:
continue
seen.add(normalized)
lines.append(normalized)
return "\n".join(lines) + ("\n" if lines else "")

View File

@@ -0,0 +1,218 @@
from __future__ import annotations
from pathlib import Path
from typing import Any, Iterable
from django.utils import timezone
from pobsync.run_stats import filesystem_capacity
from .models import BackupRun, GlobalConfig, HostConfig, SnapshotRecord
def collect_dashboard_stats(*, hosts: Iterable[HostConfig], global_config: GlobalConfig | None) -> dict[str, Any]:
runs = list(
BackupRun.objects.select_related("host", "snapshot")
.filter(status__in=_COMPLETED_BACKUP_STATUSES)
.order_by("-started_at", "-created_at")[:100]
)
real_runs = [_run_summary(run) for run in runs if _is_real_run(run)]
real_runs = [run for run in real_runs if run["has_stats"]]
for host in hosts:
host.stats_summary = collect_host_stats(host=host)
literal_values = [_int_at(run, "rsync", "literal_data_bytes") for run in real_runs]
literal_values = [value for value in literal_values if value is not None]
matched_values = [_int_at(run, "rsync", "matched_data_bytes") for run in real_runs]
matched_values = [value for value in matched_values if value is not None]
duration_values = [_int_at(run, "duration_seconds") for run in real_runs]
duration_values = [value for value in duration_values if value is not None]
avg_literal = _average(literal_values)
total_literal = sum(literal_values)
total_matched = sum(matched_values)
savings_basis = total_literal + total_matched
capacity = _capacity_from_system(global_config) or _latest_capacity_from_runs(real_runs) or {}
available = _int_at(capacity, "available_bytes")
daily_literal = _average_daily_literal(real_runs)
link_dest_savings_ratio = round(total_matched / savings_basis, 4) if savings_basis else None
return {
"runs_sampled": len(real_runs),
"avg_duration_seconds": _average(duration_values),
"avg_daily_literal_data_bytes": daily_literal,
"avg_literal_data_bytes": avg_literal,
"total_literal_data_bytes": total_literal,
"total_matched_data_bytes": total_matched,
"link_dest_savings_ratio": link_dest_savings_ratio,
"link_dest_savings_percent": round(link_dest_savings_ratio * 100, 1) if link_dest_savings_ratio is not None else None,
"estimated_runs_until_full": int(available / avg_literal) if available and avg_literal else None,
"estimated_days_until_full": int(available / daily_literal) if available and daily_literal else None,
"capacity": capacity,
}
def collect_host_stats(*, host: HostConfig, limit: int = 8) -> dict[str, Any]:
runs = list(host.runs.select_related("snapshot").order_by("-started_at", "-created_at")[:50])
real_runs = [_run_summary(run) for run in runs if _is_real_run(run)]
completed_real_runs = [run for run in real_runs if run["status"] in _COMPLETED_BACKUP_STATUSES]
trend_runs = [run for run in completed_real_runs if run["has_stats"]][:limit]
latest_snapshot = host.snapshots.order_by("-started_at", "-discovered_at", "-id").first()
latest_snapshot_stats = _snapshot_summary(latest_snapshot) if latest_snapshot else {}
literal_values = [_int_at(run, "rsync", "literal_data_bytes") for run in trend_runs]
literal_values = [value for value in literal_values if value is not None]
matched_values = [_int_at(run, "rsync", "matched_data_bytes") for run in trend_runs]
matched_values = [value for value in matched_values if value is not None]
max_literal = max(literal_values) if literal_values else 0
max_matched = max(matched_values) if matched_values else 0
return {
"runs": [_with_bar_percentages(run, max_literal=max_literal, max_matched=max_matched) for run in trend_runs],
"latest_run": completed_real_runs[0] if completed_real_runs else {},
"latest_good_run": _first_run_with_status(real_runs, {BackupRun.Status.SUCCESS}),
"latest_problem_run": _first_run_with_status(real_runs, {BackupRun.Status.WARNING, BackupRun.Status.FAILED}),
"latest_snapshot": latest_snapshot_stats,
"avg_literal_data_bytes": _average(literal_values),
"avg_daily_literal_data_bytes": _average_daily_literal(trend_runs),
"total_literal_data_bytes": sum(literal_values),
"total_matched_data_bytes": sum(matched_values),
}
def _run_summary(run: BackupRun) -> dict[str, Any]:
result = run.result if isinstance(run.result, dict) else {}
stats = result.get("stats") if isinstance(result.get("stats"), dict) else {}
return {
"id": run.id,
"host": run.host.host,
"run_type": run.run_type,
"started_at": run.started_at,
"ended_at": run.ended_at,
"snapshot": run.snapshot,
"snapshot_path": run.snapshot_path,
"status": run.status,
"reviewed_at": run.reviewed_at,
"has_stats": bool(stats),
"duration_seconds": _int_at(stats, "duration_seconds"),
"rsync": stats.get("rsync") if isinstance(stats.get("rsync"), dict) else {},
"storage": stats.get("storage") if isinstance(stats.get("storage"), dict) else {},
}
def _snapshot_summary(snapshot: SnapshotRecord | None) -> dict[str, Any]:
if snapshot is None:
return {}
metadata = snapshot.metadata if isinstance(snapshot.metadata, dict) else {}
stats = metadata.get("stats") if isinstance(metadata.get("stats"), dict) else {}
storage = stats.get("storage") if isinstance(stats.get("storage"), dict) else {}
snapshot_storage = storage.get("snapshot") if isinstance(storage.get("snapshot"), dict) else {}
return {
"id": snapshot.id,
"dirname": snapshot.dirname,
"kind": snapshot.kind,
"status": snapshot.status,
"started_at": snapshot.started_at,
"apparent_size_bytes": _int_at(snapshot_storage, "apparent_size_bytes"),
"allocated_size_bytes": _int_at(snapshot_storage, "allocated_size_bytes"),
"hardlinked_files": _int_at(snapshot_storage, "hardlinked_files"),
}
def _is_real_run(run: BackupRun) -> bool:
result = run.result if isinstance(run.result, dict) else {}
if result.get("dry_run") is True:
return False
requested = result.get("requested") if isinstance(result.get("requested"), dict) else {}
return requested.get("dry_run") is not True
def _first_run_with_status(runs: list[dict[str, Any]], statuses: set[str]) -> dict[str, Any]:
for run in runs:
if run["status"] in statuses and run.get("reviewed_at") is None:
return run
return {}
def _capacity_from_system(global_config: GlobalConfig | None) -> dict[str, Any]:
if global_config is None or not global_config.backup_root:
return {}
return filesystem_capacity(Path(global_config.backup_root))
def _latest_capacity_from_runs(runs: list[dict[str, Any]]) -> dict[str, Any]:
for run in runs:
capacity = _dict_at(run, "storage", "capacity")
if capacity:
return capacity
return {}
def _average(values: list[int]) -> int | None:
if not values:
return None
return int(sum(values) / len(values))
def _average_daily_literal(runs: list[dict[str, Any]]) -> int | None:
values = [_int_at(run, "rsync", "literal_data_bytes") for run in runs]
values = [value for value in values if value is not None]
if not values:
return None
timestamps = [run["started_at"] for run in runs if run.get("started_at") is not None]
if len(timestamps) < 2:
return _average(values)
oldest = min(timestamps)
newest = max(timestamps)
if timezone.is_naive(oldest):
oldest = timezone.make_aware(oldest)
if timezone.is_naive(newest):
newest = timezone.make_aware(newest)
span_days = max((newest - oldest).total_seconds() / 86400, 1)
return int(sum(values) / span_days)
def _with_bar_percentages(run: dict[str, Any], *, max_literal: int, max_matched: int) -> dict[str, Any]:
run = dict(run)
literal = _int_at(run, "rsync", "literal_data_bytes") or 0
matched = _int_at(run, "rsync", "matched_data_bytes") or 0
run["literal_percent"] = _percentage(literal, max_literal)
run["matched_percent"] = _percentage(matched, max_matched)
return run
def _percentage(value: int, maximum: int) -> int:
if maximum <= 0 or value <= 0:
return 0
return max(1, min(100, int(value / maximum * 100)))
def _dict_at(data: dict[str, Any], *keys: str) -> dict[str, Any]:
value: Any = data
for key in keys:
if not isinstance(value, dict):
return {}
value = value.get(key)
return value if isinstance(value, dict) else {}
def _int_at(data: dict[str, Any], *keys: str) -> int | None:
value: Any = data
for key in keys:
if not isinstance(value, dict):
return None
value = value.get(key)
if isinstance(value, bool):
return None
if isinstance(value, int):
return value
if isinstance(value, float):
return int(value)
return None
_COMPLETED_BACKUP_STATUSES = [BackupRun.Status.SUCCESS, BackupRun.Status.WARNING]

View File

@@ -0,0 +1,948 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>{% block title %}pobsync{% endblock %}</title>
<style>
:root {
color-scheme: light;
--bg: #f2f5f8;
--bg-soft: #f8fafc;
--panel: #ffffff;
--panel-subtle: #fbfcfe;
--border: #d8e1eb;
--border-strong: #c7d2df;
--text: #121a24;
--muted: #65758a;
--muted-strong: #46566a;
--link: #075fae;
--link-strong: #064b89;
--success: #17633a;
--failed: #a73333;
--running: #8a5a00;
--queued: #075fae;
--shadow-sm: 0 1px 2px rgba(18, 26, 36, 0.05);
--shadow-md: 0 10px 28px rgba(18, 26, 36, 0.08);
--radius: 8px;
}
* { box-sizing: border-box; }
body {
margin: 0;
background: var(--bg);
color: var(--text);
font: 14px/1.5 system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
}
a { color: var(--link); font-weight: 560; text-decoration: none; }
a:hover { text-decoration: underline; }
a:focus-visible,
button:focus-visible,
input:focus-visible,
select:focus-visible,
textarea:focus-visible {
outline: 3px solid rgba(7, 95, 174, 0.24);
outline-offset: 2px;
}
body > header {
background: var(--panel);
border-bottom: 1px solid var(--border);
box-shadow: var(--shadow-sm);
padding: 0 24px;
position: sticky;
top: 0;
z-index: 20;
}
nav {
display: flex;
align-items: center;
gap: 6px;
max-width: 1180px;
margin: 0 auto;
min-height: 48px;
}
nav strong {
font-size: 16px;
margin-right: 10px;
}
nav a {
border-radius: 6px;
color: var(--muted-strong);
padding: 6px 8px;
}
nav a:hover {
background: var(--bg-soft);
color: var(--link-strong);
text-decoration: none;
}
nav strong a {
color: var(--link-strong);
font-weight: 750;
padding-left: 0;
}
nav strong a:hover { background: transparent; }
nav a[aria-current="page"] {
background: #eaf3fb;
color: var(--link-strong);
font-weight: 720;
}
.nav-primary,
.nav-secondary {
align-items: center;
display: flex;
flex-wrap: wrap;
gap: 4px;
}
.nav-secondary {
justify-content: flex-end;
}
.nav-secondary a {
font-size: 13px;
}
.nav-user {
margin-left: 6px;
white-space: nowrap;
}
nav .spacer { flex: 1; }
main {
max-width: 1180px;
margin: 0 auto;
padding: 28px 24px 42px;
}
h1 {
font-size: clamp(28px, 3vw, 36px);
letter-spacing: 0;
line-height: 1.15;
margin: 0 0 18px;
}
h2 {
font-size: 18px;
letter-spacing: 0;
line-height: 1.25;
margin: 0 0 14px;
}
h3 {
font-size: 15px;
letter-spacing: 0;
margin: 16px 0 8px;
}
p { margin: 0 0 12px; }
p:last-child { margin-bottom: 0; }
.page-header {
align-items: end;
display: flex;
gap: 18px;
justify-content: space-between;
margin-bottom: 20px;
}
.page-title {
display: grid;
gap: 5px;
min-width: 0;
}
.page-title h1 { margin-bottom: 0; overflow-wrap: anywhere; }
.page-kicker {
color: var(--muted);
font-size: 12px;
font-weight: 750;
letter-spacing: 0.05em;
text-transform: uppercase;
}
.page-subtitle {
color: var(--muted);
max-width: 760px;
}
.page-header .actions {
flex: 0 0 auto;
justify-content: flex-end;
margin-bottom: 0;
}
.grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(170px, 1fr));
gap: 12px;
margin-bottom: 20px;
}
.metric, .panel {
background: var(--panel);
border: 1px solid var(--border);
border-radius: var(--radius);
box-shadow: var(--shadow-sm);
}
.metric {
min-height: 88px;
padding: 14px 15px;
}
.metric .label {
color: var(--muted);
font-size: 11px;
font-weight: 700;
letter-spacing: 0.04em;
text-transform: uppercase;
}
.metric .value {
font-size: 27px;
font-weight: 760;
line-height: 1.15;
margin-top: 6px;
overflow-wrap: anywhere;
}
.metric.failed { border-color: #e8b4b4; background: #fff7f7; }
.metric.warning { border-color: #e7cf8a; background: #fffaf0; }
.metric.running { border-color: #e7cf8a; background: #fffaf0; }
.metric.queued { border-color: #b5cdea; background: #eef6ff; }
.metric-link {
color: inherit;
display: block;
text-decoration: none;
transition: border-color 0.15s ease, box-shadow 0.15s ease, transform 0.15s ease;
}
.metric-link:hover {
border-color: #9eb2c8;
box-shadow: var(--shadow);
transform: translateY(-1px);
}
.metric-link:focus-visible {
outline: 3px solid #93c5fd;
outline-offset: 2px;
}
.dashboard-summary-grid {
grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));
}
.dashboard-summary-grid .metric {
min-height: 78px;
}
.dashboard-summary-grid .metric .value {
font-size: 25px;
}
.dashboard-trends-panel,
.dashboard-hosts-panel {
overflow: visible;
}
.panel {
margin-bottom: 18px;
overflow: auto;
padding: 18px;
}
.panel > h2:first-child {
align-items: center;
display: flex;
gap: 10px;
justify-content: space-between;
}
.panel.highlight { border-left: 4px solid var(--border); }
.panel.highlight.failed { border-left-color: var(--failed); background: #fff7f7; }
.panel.highlight.warning { border-left-color: var(--running); background: #fffaf0; }
.panel.highlight.success { border-left-color: var(--success); background: #f5fbf7; }
table {
border-collapse: collapse;
min-width: 640px;
width: 100%;
}
th, td {
border-bottom: 1px solid var(--border);
padding: 10px 9px;
text-align: left;
vertical-align: top;
}
th {
background: var(--panel-subtle);
color: var(--muted);
font-size: 11px;
font-weight: 750;
letter-spacing: 0.04em;
text-transform: uppercase;
}
tbody tr:hover td { background: #f9fbfd; }
tr:last-child td { border-bottom: 0; }
.muted { color: var(--muted); }
.status {
display: inline-block;
border: 1px solid var(--border);
border-radius: 999px;
font-size: 12px;
font-weight: 700;
line-height: 1.35;
padding: 3px 8px;
white-space: nowrap;
}
.status.success { color: var(--success); border-color: #a7d8b9; background: #edf8f1; }
.status.ok { color: var(--success); border-color: #a7d8b9; background: #edf8f1; }
.status.failed { color: var(--failed); border-color: #e8b4b4; background: #fff0f0; }
.status.blocked { color: var(--failed); border-color: #e8b4b4; background: #fff0f0; }
.status.running { color: var(--running); border-color: #e7cf8a; background: #fff8df; }
.status.warning { color: var(--running); border-color: #e7cf8a; background: #fff8df; }
.status.queued { color: var(--queued); border-color: #b5cdea; background: #eef6ff; }
.status.skipped { color: var(--muted); background: #f7f9fb; }
.stack { display: grid; gap: 5px; }
.stack.spaced { margin-bottom: 14px; }
.two-col { display: grid; gap: 18px; grid-template-columns: minmax(0, 1fr) minmax(0, 1fr); }
.panel-grid {
display: grid;
gap: 18px;
grid-template-columns: repeat(auto-fit, minmax(280px, 1fr));
margin-bottom: 18px;
}
.panel-grid .panel { margin-bottom: 0; }
.actions {
align-items: center;
display: flex;
flex-wrap: wrap;
gap: 10px;
margin: 0 0 20px;
}
.actions.inline { margin: 12px 0 0; }
button, .button-link {
appearance: none;
background: var(--text);
border: 1px solid var(--text);
border-radius: 6px;
color: #fff;
cursor: pointer;
font: inherit;
font-weight: 700;
line-height: 1.25;
padding: 8px 13px;
}
button:hover, .button-link:hover {
background: #273343;
text-decoration: none;
}
button.secondary,
.button-link.secondary {
background: #fff;
border-color: var(--border-strong);
color: var(--text);
}
button.secondary:hover,
.button-link.secondary:hover { background: #eef3f8; }
button.danger,
.button-link.danger {
background: var(--failed);
border-color: var(--failed);
color: #fff;
}
button.danger:hover,
.button-link.danger:hover {
background: #842828;
border-color: #842828;
}
button.compact,
.button-link.compact {
font-size: 12px;
padding: 5px 8px;
}
button:disabled {
background: #d8dee6;
border-color: #d8dee6;
color: var(--muted);
cursor: not-allowed;
}
.inline-form { margin: 0; }
.dashboard-priority-grid {
align-items: start;
display: grid;
gap: 14px;
grid-template-columns: repeat(2, minmax(0, 1fr));
margin-bottom: 20px;
}
.priority-panel {
display: grid;
gap: 12px;
margin-bottom: 0;
min-width: 0;
overflow: visible;
}
.priority-panel > h2:first-child {
flex-wrap: wrap;
margin-bottom: 0;
}
.action-list,
.activity-list,
.schedule-list {
display: grid;
gap: 8px;
}
.record-list {
display: grid;
gap: 10px;
}
.record-card {
border: 1px solid var(--border);
border-radius: var(--radius);
display: grid;
gap: 10px;
padding: 12px;
}
.record-card-header {
align-items: start;
display: flex;
gap: 12px;
justify-content: space-between;
}
.record-title {
display: grid;
gap: 3px;
min-width: 0;
}
.record-title a {
font-weight: 750;
overflow-wrap: anywhere;
}
.record-facts {
display: grid;
gap: 8px 16px;
grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));
}
.record-fact {
display: grid;
gap: 2px;
min-width: 0;
}
.record-fact .label {
color: var(--muted);
font-size: 11px;
font-weight: 700;
letter-spacing: 0.04em;
text-transform: uppercase;
}
.record-fact strong,
.record-fact span {
overflow-wrap: anywhere;
}
.action-row,
.activity-row,
.schedule-row {
align-items: start;
border: 1px solid var(--border);
border-radius: 7px;
color: inherit;
display: grid;
gap: 9px;
padding: 10px;
text-decoration: none;
transition: border-color 0.15s ease, background 0.15s ease, box-shadow 0.15s ease;
}
.action-row,
.activity-row {
grid-template-columns: max-content minmax(0, 1fr);
}
.activity-row .status {
justify-self: start;
}
.schedule-row {
grid-template-columns: minmax(0, 1fr) max-content;
}
.action-row:hover,
.activity-row:hover,
.schedule-row:hover {
background: var(--panel-subtle);
border-color: var(--border-strong);
box-shadow: var(--shadow-sm);
}
.action-row.failed { border-color: #e8b4b4; background: #fff7f7; }
.action-row.warning { border-color: #e7cf8a; background: #fffaf0; }
.action-row span:not(.status),
.activity-row span:not(.status),
.schedule-row span {
display: grid;
gap: 2px;
min-width: 0;
}
.action-row strong,
.action-row .muted,
.activity-row strong,
.activity-row .muted,
.schedule-row strong,
.schedule-row .muted {
overflow-wrap: anywhere;
}
.schedule-time {
justify-items: end;
text-align: right;
}
.storage-priority {
display: grid;
gap: 12px;
}
.storage-priority .label {
color: var(--muted);
font-size: 12px;
font-weight: 650;
text-transform: uppercase;
}
.storage-priority .value {
font-size: 27px;
font-weight: 760;
line-height: 1.15;
margin-top: 4px;
}
.storage-priority-facts {
display: grid;
gap: 8px;
}
.storage-priority-facts > div {
align-items: baseline;
border-top: 1px solid var(--border);
display: flex;
gap: 10px;
justify-content: space-between;
padding-top: 8px;
}
.storage-priority-facts strong {
text-align: right;
overflow-wrap: anywhere;
}
.host-control-grid {
display: grid;
gap: 14px;
grid-template-columns: minmax(280px, 1.25fr) repeat(3, minmax(220px, 1fr));
margin-bottom: 20px;
}
.host-control-panel {
display: grid;
gap: 12px;
margin-bottom: 0;
}
.host-control-panel > h2:first-child { margin-bottom: 0; }
.host-control-primary {
display: grid;
gap: 8px;
}
.host-control-meta {
display: grid;
gap: 6px;
}
.host-control-meta > div {
align-items: baseline;
border-top: 1px solid var(--border);
display: flex;
gap: 10px;
justify-content: space-between;
padding-top: 7px;
}
.host-control-meta .label {
color: var(--muted);
font-size: 12px;
font-weight: 650;
text-transform: uppercase;
}
.host-control-meta strong {
text-align: right;
}
.status-summary {
align-items: center;
border: 1px solid var(--border);
border-radius: var(--radius);
display: flex;
flex-wrap: wrap;
gap: 8px;
padding: 11px 12px;
}
.status-summary.failed { border-color: #e8b4b4; background: #fff7f7; color: var(--failed); }
.status-summary.warning,
.status-summary.running { border-color: #e7cf8a; background: #fffaf0; color: var(--running); }
.status-summary.queued { border-color: #b5cdea; background: #eef6ff; color: var(--link); }
a.status-summary {
color: inherit;
text-decoration: none;
transition: border-color 0.15s ease, box-shadow 0.15s ease, transform 0.15s ease;
}
a.status-summary:hover {
border-color: var(--border-strong);
box-shadow: var(--shadow-sm);
transform: translateY(-1px);
}
.operator-state {
align-items: center;
display: flex;
flex-wrap: wrap;
gap: 8px;
margin-bottom: 14px;
}
.trend-bars {
display: grid;
gap: 5px;
min-width: 150px;
}
.trend-bar {
background: #edf2f7;
border-radius: 4px;
height: 8px;
overflow: hidden;
}
.trend-bar span {
background: var(--link);
display: block;
height: 100%;
min-width: 0;
}
.trend-bar.matched span { background: var(--success); }
.trend-legend {
color: var(--muted);
display: flex;
font-size: 12px;
gap: 10px;
}
.insight-grid {
display: grid;
gap: 18px 24px;
grid-template-columns: minmax(260px, 1.3fr) repeat(auto-fit, minmax(180px, 1fr));
}
.insight-main,
.insight-item {
display: grid;
gap: 4px;
min-width: 0;
}
.insight-main .label,
.insight-item .label {
color: var(--muted);
font-size: 12px;
font-weight: 650;
text-transform: uppercase;
}
.insight-main .value,
.insight-item .value {
font-size: 22px;
font-weight: 760;
overflow-wrap: anywhere;
}
.storage-meter {
background: #edf2f7;
border-radius: 999px;
height: 10px;
margin: 4px 0;
overflow: hidden;
}
.storage-meter span {
background: var(--link);
display: block;
height: 100%;
max-width: 100%;
}
.host-list {
display: grid;
gap: 12px;
}
.host-card {
border: 1px solid var(--border);
border-radius: var(--radius);
box-shadow: var(--shadow-sm);
min-width: 0;
padding: 16px;
}
.host-card:hover {
border-color: var(--border-strong);
box-shadow: var(--shadow-md);
}
.host-card-header {
align-items: start;
display: flex;
gap: 12px;
justify-content: space-between;
margin-bottom: 14px;
}
.host-card-title {
display: grid;
gap: 3px;
min-width: 0;
}
.host-card-title a {
font-size: 17px;
font-weight: 750;
overflow-wrap: anywhere;
}
.host-card-status {
display: flex;
flex: 0 1 auto;
flex-wrap: wrap;
gap: 6px;
justify-content: flex-end;
max-width: 50%;
}
.host-card-layout {
display: grid;
gap: 18px;
grid-template-columns: minmax(0, 1.7fr) minmax(240px, 0.9fr);
}
.host-card-section {
align-content: start;
display: grid;
gap: 10px;
min-width: 0;
}
.host-card-section-title {
color: var(--muted);
font-size: 12px;
font-weight: 650;
text-transform: uppercase;
}
.host-card-timeline {
display: grid;
gap: 16px 22px;
grid-template-columns: repeat(auto-fit, minmax(180px, 1fr));
}
.host-card-stats {
align-content: start;
display: grid;
background: var(--bg-soft);
border: 1px solid #e6edf4;
border-radius: var(--radius);
gap: 12px 18px;
grid-template-columns: repeat(2, minmax(0, 1fr));
padding: 12px;
}
.host-card-item {
display: grid;
gap: 3px;
min-width: 0;
}
.host-card-item .label {
color: var(--muted);
font-size: 12px;
font-weight: 650;
text-transform: uppercase;
}
.host-card-item .value {
overflow-wrap: anywhere;
}
.host-card-item .value a {
overflow-wrap: anywhere;
word-break: break-word;
}
.host-card-stat {
display: grid;
gap: 3px;
min-width: 0;
}
.host-card-stat .label {
color: var(--muted);
font-size: 11px;
font-weight: 650;
text-transform: uppercase;
}
.host-card-stat .value {
font-size: 16px;
font-weight: 750;
overflow-wrap: anywhere;
}
.host-card-stat.wide {
grid-column: 1 / -1;
}
.host-card-warning {
background: #fffaf0;
border: 1px solid #e7cf8a;
border-radius: 6px;
color: var(--running);
display: flex;
flex-wrap: wrap;
gap: 8px;
margin-top: 14px;
padding: 10px;
}
.host-card-warning > * {
min-width: 0;
}
.messages { display: grid; gap: 8px; margin-bottom: 18px; }
.message {
background: var(--panel);
border: 1px solid var(--border);
border-radius: var(--radius);
padding: 10px 12px;
}
.message.success { border-color: #a7d8b9; background: #edf8f1; color: var(--success); }
.message.error { border-color: #e8b4b4; background: #fff0f0; color: var(--failed); }
.message.warning { border-color: #e7cf8a; background: #fff8df; color: var(--running); }
.form-grid { display: grid; gap: 15px; max-width: 720px; }
.filter-form {
align-items: end;
display: grid;
gap: 15px;
grid-template-columns: repeat(auto-fit, minmax(180px, 1fr));
max-width: none;
}
.form-actions {
align-items: center;
border-top: 1px solid var(--border);
display: flex;
flex-wrap: wrap;
gap: 10px;
margin-top: 4px;
padding-top: 15px;
}
.form-actions .button-link.secondary { margin-left: auto; }
.filter-form .form-actions {
border-top: 0;
justify-content: flex-end;
margin-top: 0;
padding-top: 0;
}
.filter-form .form-actions .button-link.secondary { margin-left: 0; }
.field { display: grid; gap: 6px; }
.field label { font-weight: 700; }
.field input[type="text"], .field input[type="number"], .field select, .field textarea {
border: 1px solid var(--border);
border-radius: 6px;
font: inherit;
padding: 9px 10px;
width: 100%;
}
.field input[type="text"]:focus,
.field input[type="number"]:focus,
.field select:focus,
.field textarea:focus {
border-color: #8bb9e3;
}
.field textarea { min-height: 92px; resize: vertical; }
.field .helptext { color: var(--muted); font-size: 12px; }
.field input[type="checkbox"] { justify-self: start; }
pre {
background: #101820;
border-radius: var(--radius);
color: #edf4fb;
line-height: 1.5;
margin: 0;
overflow: auto;
padding: 12px;
white-space: pre-wrap;
word-break: break-word;
}
.errorlist {
color: var(--failed);
list-style: none;
margin: 0;
padding: 0;
}
@media (max-width: 800px) {
body > header { padding: 0 14px; position: static; }
main { padding: 18px 14px 32px; }
nav {
align-items: flex-start;
flex-wrap: wrap;
gap: 4px;
padding: 8px 0;
}
nav strong { flex-basis: 100%; margin-right: 0; }
.nav-secondary {
justify-content: flex-start;
}
.nav-user {
margin-left: 0;
}
nav .spacer { display: none; }
.page-header {
align-items: stretch;
display: grid;
}
.page-header .actions { justify-content: flex-start; }
.two-col,
.panel-grid { grid-template-columns: 1fr; }
.dashboard-priority-grid { grid-template-columns: 1fr; }
.host-control-grid { grid-template-columns: 1fr; }
.schedule-row { grid-template-columns: 1fr; }
.schedule-time { justify-items: start; text-align: left; }
.form-actions .button-link.secondary { margin-left: 0; }
.host-card-header { display: grid; }
.host-card-status { justify-content: flex-start; max-width: none; }
.host-card-layout { grid-template-columns: 1fr; }
.host-card-stats { grid-template-columns: repeat(auto-fit, minmax(140px, 1fr)); }
.insight-grid { grid-template-columns: 1fr; }
}
@media (max-width: 1100px) {
.dashboard-priority-grid {
grid-template-columns: 1fr;
}
.dashboard-summary-grid {
grid-template-columns: repeat(3, minmax(0, 1fr));
}
.host-card-layout {
grid-template-columns: 1fr;
}
.host-card-status {
justify-content: flex-start;
max-width: none;
}
.schedule-row {
grid-template-columns: 1fr;
}
.schedule-time {
justify-items: start;
text-align: left;
}
}
@media (max-width: 560px) {
.dashboard-summary-grid {
grid-template-columns: repeat(2, minmax(0, 1fr));
}
.metric {
min-height: 76px;
padding: 12px;
}
.metric .value {
font-size: 24px;
}
.host-card {
padding: 13px;
}
.host-card-stats {
grid-template-columns: 1fr;
}
}
</style>
</head>
<body>
<header>
<nav>
<strong><a href="{% url 'dashboard' %}">pobsync</a></strong>
<span class="nav-primary" aria-label="Primary navigation">
<a href="{% url 'dashboard' %}" {% if request.resolver_match.url_name == "dashboard" %}aria-current="page"{% endif %}>Dashboard</a>
<a href="{% url 'ssh_credentials' %}" {% if request.resolver_match.url_name == "ssh_credentials" or request.resolver_match.url_name == "create_ssh_credential" or request.resolver_match.url_name == "generate_ssh_credential" or request.resolver_match.url_name == "edit_ssh_credential" %}aria-current="page"{% endif %}>SSH Keys</a>
<a href="{% url 'logs' %}" {% if request.resolver_match.url_name == "logs" %}aria-current="page"{% endif %}>Logs</a>
<a href="{% url 'purged_snapshots' %}" {% if request.resolver_match.url_name == "purged_snapshots" %}aria-current="page"{% endif %}>Purged</a>
</span>
<span class="spacer"></span>
<span class="nav-secondary" aria-label="System navigation">
<a href="{% url 'self_check' %}" {% if request.resolver_match.url_name == "self_check" %}aria-current="page"{% endif %}>Self Check</a>
<a href="{% url 'changelog' %}" {% if request.resolver_match.url_name == "changelog" %}aria-current="page"{% endif %}>Changelog</a>
<a href="/api/status/">Status API</a>
<a href="{% url 'admin:index' %}">Admin</a>
</span>
<span class="muted nav-user">{{ request.user.username }}</span>
</nav>
</header>
<main>
{% if messages %}
<section class="messages" aria-label="Messages">
{% for message in messages %}
<div class="message {{ message.tags }}">{{ message }}</div>
{% endfor %}
</section>
{% endif %}
{% block content %}{% endblock %}
</main>
<script>
(() => {
const refreshRegion = async (region) => {
if (region.dataset.refreshActive !== "true" || document.hidden) return;
try {
const response = await fetch(region.dataset.refreshUrl, {
credentials: "same-origin",
headers: { "X-Requested-With": "XMLHttpRequest" },
});
if (!response.ok) return;
region.innerHTML = await response.text();
const refreshActive = response.headers.get("X-Pobsync-Refresh-Active");
if (refreshActive) region.dataset.refreshActive = refreshActive;
} catch (error) {
// Keep the current server-rendered content visible if a refresh fails.
}
};
document.querySelectorAll("[data-refresh-url]").forEach((region) => {
const interval = Number.parseInt(region.dataset.refreshInterval || "5000", 10);
window.setInterval(() => refreshRegion(region), Number.isFinite(interval) ? interval : 5000);
});
})();
</script>
</body>
</html>

View File

@@ -0,0 +1,46 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}Changelog - pobsync{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Release notes</div>
<h1>Changelog</h1>
<div class="page-subtitle">Installed release notes rendered from the repository changelog.</div>
</div>
<section class="actions" aria-label="Changelog actions">
<a class="button-link secondary" href="{% url 'dashboard' %}">Back to dashboard</a>
</section>
</header>
<section class="panel">
<div class="stack spaced">
<div><strong>Installed version:</strong> {{ app_version }}</div>
<div class="muted">Changelog file: {{ changelog_path }}</div>
{% if missing %}
<div class="status warning">missing</div>
{% endif %}
</div>
<div class="stack">
{% for block in changelog_blocks %}
{% if block.kind == "heading" %}
{% if block.level == 1 %}
<h2>{{ block.text }}</h2>
{% else %}
<h3>{{ block.text }}</h3>
{% endif %}
{% elif block.kind == "list" %}
<ul>
{% for item in block.items %}
<li>{{ item }}</li>
{% endfor %}
</ul>
{% else %}
<p>{{ block.text }}</p>
{% endif %}
{% endfor %}
</div>
</section>
{% endblock %}

View File

@@ -0,0 +1,106 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}pobsync dashboard{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Control panel</div>
<h1>Dashboard</h1>
<div class="page-subtitle">Backup health, required action, storage pressure, and recent activity in one place.</div>
</div>
<section class="actions" aria-label="Dashboard actions">
<a class="button-link" href="{% url 'create_host_config' %}">New host</a>
<a class="button-link secondary" href="{% url 'edit_global_config' %}">{% if global_config %}Edit global config{% else %}Create global config{% endif %}</a>
</section>
</header>
{% if not global_config or not counts.hosts %}
<section class="panel">
<h2>Setup</h2>
{% if not global_config %}
<p class="muted">No default global config exists yet. Create it first so discovery, backups, and retention know where pobsync stores snapshots.</p>
<div class="actions inline">
<a class="button-link" href="{% url 'edit_global_config' %}">Create global config</a>
</div>
{% elif not counts.hosts %}
<p class="muted">Global config is ready. Add the first host to make this dashboard useful.</p>
<div class="actions inline">
<a class="button-link" href="{% url 'create_host_config' %}">Add first host</a>
</div>
{% endif %}
</section>
{% endif %}
<div
data-refresh-url="{% url 'dashboard_priority_live' %}"
data-refresh-interval="10000"
data-refresh-active="true"
aria-live="polite"
>
{% include "pobsync_backend/partials/dashboard_priority.html" %}
</div>
<section class="grid dashboard-summary-grid" aria-label="Summary">
<a class="metric metric-link" href="#hosts"><div class="label">Hosts</div><div class="value">{{ counts.enabled_hosts }}/{{ counts.hosts }}</div></a>
<a class="metric metric-link" href="{% url 'schedules_list' %}"><div class="label">Schedules</div><div class="value">{{ counts.enabled_schedules }}/{{ counts.schedules }}</div></a>
<a class="metric metric-link" href="{% url 'snapshots_list' %}"><div class="label">Snapshots</div><div class="value">{{ counts.snapshots }}</div></a>
<a class="metric metric-link" href="{% url 'runs_list' %}"><div class="label">Runs</div><div class="value">{{ counts.runs }}</div></a>
<a class="metric metric-link {% if counts.warning_runs %}warning{% endif %}" href="{% url 'runs_list' %}?status=warning&amp;review=needed"><div class="label">Warnings</div><div class="value">{{ counts.warning_runs }}</div></a>
<a class="metric metric-link {% if counts.failed_runs %}failed{% endif %}" href="{% url 'runs_list' %}?status=failed&amp;review=needed"><div class="label">Failed</div><div class="value">{{ counts.failed_runs }}</div></a>
</section>
<section class="panel dashboard-trends-panel">
<h2>Backup Trends</h2>
{% if stats_summary.runs_sampled %}
<div class="insight-grid" aria-label="Backup trends">
<div class="insight-item">
<div class="label">Runway</div>
<div class="value">
{% if stats_summary.estimated_days_until_full %}
{{ stats_summary.estimated_days_until_full }} days
{% elif stats_summary.estimated_runs_until_full %}
{{ stats_summary.estimated_runs_until_full }} runs
{% else %}
unknown
{% endif %}
</div>
<div class="muted">Estimated from average new data per day.</div>
</div>
<div class="insight-item">
<div class="label">New Data</div>
<div class="value">{{ stats_summary.avg_daily_literal_data_bytes|filesizeformat }}/day</div>
<div class="muted">{{ stats_summary.avg_literal_data_bytes|filesizeformat }} per backup on average.</div>
</div>
<div class="insight-item">
<div class="label">Link-Dest Savings</div>
<div class="value">
{% if stats_summary.link_dest_savings_percent is not None %}
{{ stats_summary.link_dest_savings_percent|floatformat:1 }}%
{% else %}
unknown
{% endif %}
</div>
<div class="muted">{{ stats_summary.total_matched_data_bytes|filesizeformat }} reused across sampled runs.</div>
</div>
<div class="insight-item">
<div class="label">Average Duration</div>
<div class="value">{{ stats_summary.avg_duration_seconds|default:"unknown" }}{% if stats_summary.avg_duration_seconds is not None %}s{% endif %}</div>
<div class="muted">Based on {{ stats_summary.runs_sampled }} completed backup run{{ stats_summary.runs_sampled|pluralize }} with stats.</div>
</div>
</div>
{% else %}
<p class="muted">No completed backup runs with stats yet. This section will show disk usage, growth estimates, and link-dest savings after the first real backup finishes.</p>
{% endif %}
</section>
<div
data-refresh-url="{% url 'dashboard_hosts_live' %}"
data-refresh-interval="15000"
data-refresh-active="true"
aria-live="polite"
>
{% include "pobsync_backend/partials/dashboard_hosts.html" %}
</div>
{% endblock %}

View File

@@ -0,0 +1,74 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}Global Config{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Configuration</div>
<h1>{% if global_config %}Global Config{% else %}Create Global Config{% endif %}</h1>
<div class="page-subtitle">Defaults used by hosts unless a host overrides them explicitly.</div>
</div>
<section class="actions" aria-label="Global config actions">
<a class="button-link" href="{% url 'dashboard' %}">Back to dashboard</a>
</section>
</header>
<section class="panel">
<h2>{% if global_config %}Edit Global Config{% else %}Create Global Config{% endif %}</h2>
<div class="stack spaced">
<div><strong>Backup root:</strong> {{ backup_root }}</div>
<div class="muted">This path is managed by the service environment and is saved with the config.</div>
</div>
<form method="post" class="form-grid">
{% csrf_token %}
{{ form.non_field_errors }}
{% for field in form %}
<div class="field">
{{ field.errors }}
<label for="{{ field.id_for_label }}">{{ field.label }}</label>
{{ field }}
{% if field.help_text %}<div class="helptext">{{ field.help_text }}</div>{% endif %}
</div>
{% endfor %}
<div class="form-actions">
<button type="submit">Save global config</button>
<a class="button-link secondary" href="{% url 'dashboard' %}">Cancel</a>
</div>
</form>
</section>
{% if config_checks %}
<section class="panel">
<h2>Config Check</h2>
<section class="grid" aria-label="Global config check summary">
<div class="metric"><div class="label">OK</div><div class="value">{{ config_check_summary.ok }}</div></div>
<div class="metric"><div class="label">Warnings</div><div class="value">{{ config_check_summary.warning }}</div></div>
<div class="metric"><div class="label">Failed</div><div class="value">{{ config_check_summary.failed }}</div></div>
<div class="metric"><div class="label">Skipped</div><div class="value">{{ config_check_summary.skipped }}</div></div>
</section>
<table>
<thead>
<tr>
<th>Status</th>
<th>Check</th>
<th>Message</th>
<th>Detail</th>
</tr>
</thead>
<tbody>
{% for check in config_checks %}
<tr>
<td><span class="status {{ check.status }}">{{ check.status }}</span></td>
<td>{{ check.name }}</td>
<td>{{ check.message }}</td>
<td class="muted">{{ check.detail }}</td>
</tr>
{% endfor %}
</tbody>
</table>
</section>
{% endif %}
{% endblock %}

View File

@@ -0,0 +1,527 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}{{ host.host }} | pobsync{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Host</div>
<h1>{{ host.host }}</h1>
<div class="page-subtitle">{{ host.address }} · {{ host.enabled|yesno:"enabled,disabled" }}</div>
</div>
</header>
{% if retention_warning.has_warning %}
<section class="panel highlight warning">
<h2>Retention Warnings</h2>
<div class="stack">
{% if retention_warning.prune_exceeded %}
<div>
Scheduled pruning would delete {{ retention_warning.delete_count }} snapshot(s), above max delete
{{ retention_warning.max_delete }}. Scheduled pruning will refuse this plan until the limit or retention
selection is adjusted.
</div>
{% endif %}
{% if retention_warning.incomplete_count %}
<div>
{{ retention_warning.incomplete_count }} incomplete snapshot(s) exist. Retention does not delete incomplete
snapshots automatically; inspect them before cleanup.
</div>
<form method="post" action="{% url 'resolve_host_incomplete_reviews' host.host %}">
{% csrf_token %}
<button type="submit" class="secondary">Mark incomplete reviewed</button>
</form>
{% endif %}
{% if retention_warning.error %}
<div>{{ retention_warning.error }}</div>
{% endif %}
</div>
</section>
{% endif %}
<section class="host-control-grid" aria-label="Host control workspace">
<article class="panel host-control-panel">
<h2>Host Status</h2>
<div class="host-control-primary">
<div>
{% if host.enabled %}
<span class="status ok">enabled</span>
{% else %}
<span class="status failed">disabled</span>
{% endif %}
<span class="muted">{{ host.address }}</span>
</div>
{% if active_run %}
<a class="status-summary {{ active_run.status }}" href="{% url 'run_detail' active_run.id %}">
<span class="status {{ active_run.status }}">{{ active_run.status }}</span>
<strong>Run {{ active_run.id }} is active.</strong>
</a>
{% elif counts.failed_runs %}
<a class="status-summary failed" href="{% url 'runs_list' %}?host={{ host.host }}&amp;status=failed&amp;review=needed">
<span class="status failed">failed</span>
<strong>{{ counts.failed_runs }} failed run{{ counts.failed_runs|pluralize }} need review.</strong>
</a>
{% elif retention_warning.has_warning %}
<span class="status-summary warning">
<span class="status warning">warning</span>
<strong>Retention needs attention.</strong>
</span>
{% else %}
<span class="status-summary success">
<span class="status ok">ok</span>
<strong>No active blockers for this host.</strong>
</span>
{% endif %}
</div>
<div class="host-control-meta">
<div><span class="label">Snapshots</span><strong>{{ counts.snapshots }}</strong></div>
<div><span class="label">Runs</span><strong>{{ counts.runs }}</strong></div>
<div><span class="label">Incomplete</span><strong>{{ counts.incomplete_snapshots }}</strong></div>
</div>
</article>
<article class="panel host-control-panel">
<h2>Backup Control</h2>
<div class="operator-state">
{% if active_run %}
<span class="status {{ active_run.status }}">{{ active_run.status }}</span>
<a href="{% url 'run_detail' active_run.id %}">Run {{ active_run.id }}</a>
{% elif has_global_config and host.enabled %}
<span class="status {{ backup_gate.state }}">{{ backup_gate.state }}</span>
<span class="muted">{{ backup_gate.message }}</span>
{% elif not host.enabled %}
<span class="status failed">disabled</span>
{% elif not has_global_config %}
<span class="status failed">missing global config</span>
{% endif %}
</div>
<section class="actions inline" aria-label="Quick backup actions">
<form class="inline-form" method="post" action="{% url 'queue_manual_backup' host.host %}">
{% csrf_token %}
<input type="hidden" name="dry_run" value="on">
<input type="hidden" name="verbose_output" value="on">
<input type="hidden" name="prune_max_delete" value="10">
<button type="submit" class="secondary" {% if not can_queue_dry_run %}disabled{% endif %}>Queue dry-run</button>
</form>
<form class="inline-form" method="post" action="{% url 'queue_manual_backup' host.host %}">
{% csrf_token %}
<input type="hidden" name="prune_max_delete" value="10">
<button type="submit" {% if not can_queue_real_backup %}disabled{% endif %}>Queue backup</button>
</form>
</section>
{% if active_run %}
<p class="muted">Wait for the active run to finish, or cancel it from the run detail page.</p>
{% elif not can_queue_dry_run or not can_queue_real_backup %}
{% if not has_global_config %}
<p class="muted">Create the default global config before queueing backups.</p>
{% elif not host.enabled %}
<p class="muted">Enable this host before queueing backups.</p>
{% elif backup_gate.real_blockers %}
<p class="muted">Real backups are blocked by failed preflight checks. Dry-runs may still be available when storage-only checks fail.</p>
{% endif %}
{% endif %}
</article>
<article class="panel host-control-panel">
<h2>Schedule <a class="button-link secondary compact" href="{% url 'edit_host_schedule' host.host %}">Edit schedule</a></h2>
{% if schedule %}
<div class="host-control-meta">
<div><span class="label">Schedule expression</span><strong>{{ schedule.cron_expr }}</strong></div>
<div><span class="label">Next run</span><strong>{% if next_run_at %}{{ next_run_at|date:"Y-m-d H:i T" }}{% else %}none{% endif %}</strong></div>
<div><span class="label">Timezone</span><strong>{{ scheduler_timezone }}</strong></div>
<div><span class="label">Prune</span><strong>{{ schedule.prune|yesno:"yes,no" }}</strong></div>
<div><span class="label">Last status</span><strong>{{ schedule.last_status|default:"none" }}</strong></div>
</div>
<p class="muted">Evaluated by the pobsync scheduler service.</p>
{% else %}
<p class="muted">No schedule configured.</p>
<a class="button-link secondary compact" href="{% url 'edit_host_schedule' host.host %}">Add schedule</a>
{% endif %}
</article>
<article class="panel host-control-panel">
<h2>Current Activity</h2>
{% if latest_runs %}
{% with run=latest_runs.0 %}
<a class="activity-row" href="{% url 'run_detail' run.id %}">
<span class="status {{ run.status }}">{{ run.status }}</span>
<span>
<strong>Run {{ run.id }}</strong>
<span class="muted">{{ run.run_type }} · {{ run.started_at|default:run.created_at }}</span>
</span>
</a>
{% endwith %}
{% else %}
<p class="muted">No backup runs recorded for this host.</p>
{% endif %}
{% if stats_summary.latest_run.duration_seconds is not None %}
<div class="host-control-meta">
<div><span class="label">Latest duration</span><strong>{{ stats_summary.latest_run.duration_seconds }}s</strong></div>
<div><span class="label">New data</span><strong>{{ stats_summary.latest_run.rsync.literal_data_bytes|filesizeformat }}</strong></div>
</div>
{% endif %}
</article>
</section>
<section class="grid" aria-label="Host summary">
<div class="metric"><div class="label">Snapshots</div><div class="value">{{ counts.snapshots }}</div></div>
<div class="metric"><div class="label">Runs</div><div class="value">{{ counts.runs }}</div></div>
<div class="metric"><div class="label">Queued</div><div class="value">{{ counts.queued_runs }}</div></div>
<div class="metric"><div class="label">Running</div><div class="value">{{ counts.running_runs }}</div></div>
<div class="metric"><div class="label">Failed Runs</div><div class="value">{{ counts.failed_runs }}</div></div>
<div class="metric"><div class="label">Incomplete</div><div class="value">{{ counts.incomplete_snapshots }}</div></div>
</section>
{% if stats_summary.runs %}
<section class="panel">
<h2>Backup Trends</h2>
<section class="grid" aria-label="Host backup trend summary">
<div class="metric"><div class="label">Avg New Data</div><div class="value">{{ stats_summary.avg_literal_data_bytes|filesizeformat }}</div></div>
<div class="metric"><div class="label">Avg Daily New</div><div class="value">{{ stats_summary.avg_daily_literal_data_bytes|filesizeformat }}</div></div>
<div class="metric"><div class="label">Total New Data</div><div class="value">{{ stats_summary.total_literal_data_bytes|filesizeformat }}</div></div>
<div class="metric"><div class="label">Matched Data</div><div class="value">{{ stats_summary.total_matched_data_bytes|filesizeformat }}</div></div>
<div class="metric"><div class="label">Latest Duration</div><div class="value">{{ stats_summary.latest_run.duration_seconds|default:"" }}{% if stats_summary.latest_run.duration_seconds is not None %}s{% endif %}</div></div>
</section>
<table>
<thead>
<tr>
<th>Run</th>
<th>Type</th>
<th>Started</th>
<th>Duration</th>
<th>Files</th>
<th>New Data</th>
<th>Matched</th>
<th>Trend</th>
<th>Snapshot</th>
</tr>
</thead>
<tbody>
{% for run in stats_summary.runs %}
<tr>
<td><a href="{% url 'run_detail' run.id %}">Run {{ run.id }}</a></td>
<td>{{ run.run_type }}</td>
<td>{{ run.started_at|default:"" }}</td>
<td>{{ run.duration_seconds|default:"" }}{% if run.duration_seconds is not None %}s{% endif %}</td>
<td>{{ run.rsync.files_total|default:"" }}</td>
<td>{{ run.rsync.literal_data_bytes|filesizeformat }}</td>
<td>{{ run.rsync.matched_data_bytes|filesizeformat }}</td>
<td>
<div class="trend-bars" aria-label="Run data trend">
<div class="trend-bar" title="New data"><span style="width: {{ run.literal_percent }}%"></span></div>
<div class="trend-bar matched" title="Matched data"><span style="width: {{ run.matched_percent }}%"></span></div>
<div class="trend-legend"><span>new</span><span>matched</span></div>
</div>
</td>
<td>{% if run.snapshot %}<a href="{% url 'snapshot_detail' run.snapshot.id %}">{{ run.snapshot.dirname }}</a>{% else %}<span class="muted">{{ run.snapshot_path }}</span>{% endif %}</td>
</tr>
{% endfor %}
</tbody>
</table>
</section>
{% endif %}
<section class="panel">
<h2>Host Check</h2>
<section class="grid" aria-label="Host check summary">
<div class="metric"><div class="label">OK</div><div class="value">{{ host_check_summary.ok }}</div></div>
<div class="metric"><div class="label">Warnings</div><div class="value">{{ host_check_summary.warning }}</div></div>
<div class="metric"><div class="label">Failed</div><div class="value">{{ host_check_summary.failed }}</div></div>
<div class="metric"><div class="label">Skipped</div><div class="value">{{ host_check_summary.skipped }}</div></div>
</section>
<div class="record-list">
{% for check in host_checks %}
<article class="record-card">
<div class="record-card-header">
<div class="record-title">
<strong>{{ check.name }}</strong>
<span class="muted">{{ check.message }}</span>
</div>
<span class="status {{ check.status }}">{{ check.status }}</span>
</div>
{% if check.detail %}
<div class="record-fact">
<span class="label">Detail</span>
<span class="muted">{{ check.detail }}</span>
</div>
{% endif %}
</article>
{% endfor %}
</div>
</section>
<div class="panel-grid">
<section class="panel">
<h2>Configuration</h2>
<div class="host-control-meta">
<div><span class="label">Address</span><strong>{{ host.address }}</strong></div>
<div><span class="label">SSH key</span><strong>{{ host.ssh_credential|default:"global default" }}</strong></div>
<div><span class="label">SSH</span><strong>{{ host.ssh_user|default:"global" }}{% if host.ssh_port %}:{{ host.ssh_port }}{% endif %}</strong></div>
<div><span class="label">Backup source</span><strong>{{ host.source_root|default:"global default" }}</strong></div>
<div><span class="label">Retention</span><strong>d{{ host.retention_daily }} w{{ host.retention_weekly }} m{{ host.retention_monthly }} y{{ host.retention_yearly }}</strong></div>
</div>
<div class="actions inline">
<a class="button-link secondary compact" href="{% url 'edit_host_config' host.host %}">Edit config</a>
<a class="button-link secondary compact" href="{% url 'host_retention_plan' host.host %}">Plan retention</a>
</div>
</section>
<section class="panel">
<h2>Connection Preflight &amp; SSH</h2>
{% if last_preflight %}
<div class="host-control-meta">
<div>
<span class="label">Preflight</span>
<strong>
<span class="status {% if last_preflight.ok %}ok{% else %}failed{% endif %}">
{% if last_preflight.ok %}ok{% else %}failed{% endif %}
</span>
</strong>
</div>
<div><span class="label">Target</span><strong>{{ last_preflight.target }}</strong></div>
<div><span class="label">Backup source</span><strong>{{ last_preflight.source_root }}</strong></div>
<div><span class="label">Remote rsync</span><strong>{{ last_preflight.rsync_binary }}</strong></div>
</div>
{% else %}
<p class="muted">No connection preflight recorded yet.</p>
{% endif %}
<div class="actions inline">
<form method="post" action="{% url 'run_host_preflight' host.host %}">
{% csrf_token %}
<button type="submit" class="secondary compact">Run connection preflight</button>
</form>
<form method="post" action="{% url 'scan_host_known_key' host.host %}">
{% csrf_token %}
<button type="submit" class="secondary compact">Scan SSH host key</button>
</form>
</div>
{% if last_preflight.checks %}
<div class="activity-list">
{% for check in last_preflight.checks %}
<div class="activity-row">
<span class="status {% if check.ok %}ok{% else %}failed{% endif %}">
{% if check.ok %}ok{% else %}failed{% endif %}
</span>
<span>
<strong>{{ check.name }}</strong>
<span class="muted">{{ check.message }}{% if check.detail %} · {{ check.detail }}{% endif %}</span>
</span>
</div>
{% endfor %}
</div>
{% endif %}
</section>
<section class="panel">
<h2>Snapshot Storage</h2>
<div class="host-control-meta">
<div><span class="label">Backup root</span><strong>{{ discovery.backup_root|default:"" }}</strong></div>
<div><span class="label">Host root</span><strong>{{ discovery.host_root|default:"" }}</strong></div>
<div><span class="label">Status</span><strong>{{ discovery.message }}</strong></div>
{% if discovery.kind_counts %}
<div>
<span class="label">On disk</span>
<strong>
scheduled {{ discovery.kind_counts.scheduled|default:0 }},
manual {{ discovery.kind_counts.manual|default:0 }},
incomplete {{ discovery.kind_counts.incomplete|default:0 }}
</strong>
</div>
{% endif %}
</div>
<div class="actions inline">
<form method="post" action="{% url 'discover_host_snapshots' host.host %}">
{% csrf_token %}
<button type="submit" class="secondary compact">Discover snapshots</button>
</form>
<form method="post" action="{% url 'prepare_host_directories' host.host %}">
{% csrf_token %}
<button type="submit" class="secondary compact">Prepare directories</button>
</form>
</div>
</section>
</div>
{% if effective_config %}
<section class="panel">
<h2>Effective Config</h2>
<p class="muted">Runtime settings after global defaults and host overrides are combined.</p>
<div class="record-list">
<article class="record-card">
<div class="record-card-header">
<div class="record-title">
<strong>Backup target</strong>
<span class="muted">Source and destination used by rsync.</span>
</div>
</div>
<div class="record-facts">
<div class="record-fact"><span class="label">Backup source:</span><strong>{{ effective_config.source_root }}</strong></div>
<div class="record-fact"><span class="label">Destination subdir:</span><strong>{{ effective_config.destination_subdir|default:"none" }}</strong></div>
</div>
</article>
<article class="record-card">
<div class="record-card-header">
<div class="record-title">
<strong>Connection</strong>
<span class="muted">SSH and rsync execution settings.</span>
</div>
</div>
<div class="record-facts">
<div class="record-fact"><span class="label">SSH:</span><strong>{{ effective_config.ssh.user }}@{{ host.address }}:{{ effective_config.ssh.port }}</strong></div>
<div class="record-fact"><span class="label">SSH key:</span><strong>{{ effective_config.ssh.credential|default:"none selected" }}</strong></div>
<div class="record-fact"><span class="label">SSH options:</span><span>{{ effective_config.ssh.options|join:" " }}</span></div>
<div class="record-fact"><span class="label">Rsync binary:</span><strong>{{ effective_config.rsync.binary }}</strong></div>
<div class="record-fact"><span class="label">Rsync args:</span><span>{{ effective_config.rsync.args|join:" " }}</span></div>
<div class="record-fact"><span class="label">Timeout:</span><strong>{{ effective_config.rsync.timeout_seconds }}s</strong></div>
<div class="record-fact"><span class="label">Bandwidth limit:</span><strong>{{ effective_config.rsync.bwlimit_kbps }} KB/s</strong></div>
</div>
</article>
<article class="record-card">
<div class="record-card-header">
<div class="record-title">
<strong>Selection &amp; retention</strong>
<span class="muted">Include/exclude rules and retention counts.</span>
</div>
</div>
<div class="record-facts">
<div class="record-fact">
<span class="label">Retention:</span>
<strong>
d{{ effective_config.retention.daily }}
w{{ effective_config.retention.weekly }}
m{{ effective_config.retention.monthly }}
y{{ effective_config.retention.yearly }}
</strong>
</div>
<div class="record-fact"><span class="label">Includes:</span><strong>{{ effective_config.includes|length }}</strong></div>
<div class="record-fact"><span class="label">Excludes:</span><strong>{{ effective_config.excludes|length }}</strong></div>
</div>
<div class="two-col">
<div class="stack">
{% if effective_config.includes %}
<pre>{{ effective_config.includes|join:"&#10;" }}</pre>
{% else %}
<div class="muted">No include rules configured.</div>
{% endif %}
</div>
<div class="stack">
{% if effective_config.excludes %}
<pre>{{ effective_config.excludes|join:"&#10;" }}</pre>
{% else %}
<div class="muted">No exclude rules configured.</div>
{% endif %}
</div>
</div>
</article>
</div>
</section>
{% endif %}
<section class="panel">
<h2>Backup Options</h2>
<p class="muted">Use this when the quick actions above need a custom label, include/exclude override, or prune limit.</p>
<form method="post" action="{% url 'queue_manual_backup' host.host %}" class="form-grid">
{% csrf_token %}
{{ manual_backup_form.non_field_errors }}
{% for field in manual_backup_form %}
<div class="field">
{{ field.errors }}
<label for="{{ field.id_for_label }}">{{ field.label }}</label>
{{ field }}
{% if field.help_text %}<div class="helptext">{{ field.help_text }}</div>{% endif %}
</div>
{% endfor %}
<div class="form-actions">
<button type="submit" {% if not can_queue_dry_run and not can_queue_real_backup %}disabled{% endif %}>Queue with options</button>
</div>
</form>
</section>
<section class="panel">
<h2>Latest Runs <a class="button-link secondary compact" href="{% url 'runs_list' %}?host={{ host.host }}">View all</a></h2>
<div class="record-list">
{% for run in latest_runs %}
<article class="record-card">
<div class="record-card-header">
<div class="record-title">
<a href="{% url 'run_detail' run.id %}">Run {{ run.id }}</a>
<span class="muted">{{ run.run_type }}{% if run.result.duration_seconds %} · {{ run.result.duration_seconds }}s{% endif %}</span>
</div>
<span class="status {{ run.status }}">{{ run.status }}</span>
</div>
<div class="record-facts">
<div class="record-fact">
<span class="label">Started</span>
<strong>{{ run.started_at|default:run.created_at }}</strong>
</div>
<div class="record-fact">
<span class="label">Ended</span>
<strong>{{ run.ended_at|default:"running or queued" }}</strong>
</div>
<div class="record-fact">
<span class="label">Snapshot</span>
{% if run.snapshot %}
<strong><a href="{% url 'snapshot_detail' run.snapshot.id %}">{{ run.snapshot.dirname }}</a></strong>
{% elif run.snapshot_path %}
<span class="muted">{{ run.snapshot_path }}</span>
{% else %}
<span class="muted">none</span>
{% endif %}
</div>
<div class="record-fact">
<span class="label">Base</span>
<span class="muted">{{ run.base_path|default:"none" }}</span>
</div>
</div>
</article>
{% empty %}
<p class="muted">No backup runs recorded for this host.</p>
{% endfor %}
</div>
</section>
<section class="panel">
<h2>Snapshots <a class="button-link secondary compact" href="{% url 'snapshots_list' %}?host={{ host.host }}">View all</a></h2>
<div class="record-list">
{% for snapshot in snapshots %}
<article class="record-card">
<div class="record-card-header">
<div class="record-title">
<a href="{% url 'snapshot_detail' snapshot.id %}">{{ snapshot.dirname }}</a>
<span class="muted">{{ snapshot.kind }}</span>
</div>
<span class="status {{ snapshot.status }}">{{ snapshot.status }}</span>
</div>
<div class="record-facts">
<div class="record-fact">
<span class="label">Started</span>
<strong>{{ snapshot.started_at|default:"unknown" }}</strong>
</div>
<div class="record-fact">
<span class="label">Ended</span>
<strong>{{ snapshot.ended_at|default:"unknown" }}</strong>
</div>
<div class="record-fact">
<span class="label">Base</span>
{% if snapshot.base %}
<strong><a href="{% url 'snapshot_detail' snapshot.base.id %}">{{ snapshot.base.dirname }}</a></strong>
{% elif snapshot.base_dirname %}
<span class="muted">{{ snapshot.base_dirname }}</span>
{% else %}
<span class="muted">none</span>
{% endif %}
</div>
<div class="record-fact">
<span class="label">Path</span>
<span class="muted">{{ snapshot.path|default:"not recorded" }}</span>
</div>
</div>
</article>
{% empty %}
<p class="muted">No snapshots discovered for this host.</p>
{% endfor %}
</div>
</section>
{% endblock %}

View File

@@ -0,0 +1,78 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}{% if host %}Config | {{ host.host }}{% else %}New Host{% endif %}{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Configuration</div>
<h1>{% if host %}Config: {{ host.host }}{% else %}New Host{% endif %}</h1>
<div class="page-subtitle">Host-specific backup, retention, SSH, include, and exclude settings.</div>
</div>
<section class="actions" aria-label="Config actions">
{% if host %}
<a class="button-link" href="{% url 'host_detail' host.host %}">Back to host</a>
{% else %}
<a class="button-link" href="{% url 'dashboard' %}">Back to dashboard</a>
{% endif %}
</section>
</header>
<section class="panel">
<h2>{% if host %}Edit Host Config{% else %}Create Host Config{% endif %}</h2>
<form method="post" class="form-grid">
{% csrf_token %}
{{ form.non_field_errors }}
{% for field in form %}
<div class="field">
{{ field.errors }}
<label for="{{ field.id_for_label }}">{{ field.label }}</label>
{{ field }}
{% if field.help_text %}<div class="helptext">{{ field.help_text }}</div>{% endif %}
</div>
{% endfor %}
<div class="form-actions">
<button type="submit">{% if host %}Save config{% else %}Create host{% endif %}</button>
{% if host %}
<a class="button-link secondary" href="{% url 'host_detail' host.host %}">Cancel</a>
{% else %}
<a class="button-link secondary" href="{% url 'dashboard' %}">Cancel</a>
{% endif %}
</div>
</form>
</section>
{% if config_checks %}
<section class="panel">
<h2>Effective Config Check</h2>
<section class="grid" aria-label="Host config check summary">
<div class="metric"><div class="label">OK</div><div class="value">{{ config_check_summary.ok }}</div></div>
<div class="metric"><div class="label">Warnings</div><div class="value">{{ config_check_summary.warning }}</div></div>
<div class="metric"><div class="label">Failed</div><div class="value">{{ config_check_summary.failed }}</div></div>
<div class="metric"><div class="label">Skipped</div><div class="value">{{ config_check_summary.skipped }}</div></div>
</section>
<table>
<thead>
<tr>
<th>Status</th>
<th>Check</th>
<th>Message</th>
<th>Detail</th>
</tr>
</thead>
<tbody>
{% for check in config_checks %}
<tr>
<td><span class="status {{ check.status }}">{{ check.status }}</span></td>
<td>{{ check.name }}</td>
<td>{{ check.message }}</td>
<td class="muted">{{ check.detail }}</td>
</tr>
{% endfor %}
</tbody>
</table>
</section>
{% endif %}
{% endblock %}

View File

@@ -0,0 +1,74 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}Logs | pobsync{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Operations</div>
<h1>Logs</h1>
<div class="page-subtitle">Filter pobsync service logs by unit, priority, host, run, or message content.</div>
</div>
<section class="actions" aria-label="Log actions">
<a class="button-link secondary" href="{% url 'dashboard' %}">Back to dashboard</a>
</section>
</header>
<section class="panel">
<h2>Filter</h2>
<form method="get" class="filter-form">
<div class="field">
<label for="unit">Unit</label>
<select id="unit" name="unit">
<option value="">All pobsync units</option>
{% for unit in units %}
<option value="{{ unit }}" {% if selected_unit == unit %}selected{% endif %}>{{ unit }}</option>
{% endfor %}
</select>
</div>
<div class="field">
<label for="priority">Priority</label>
<select id="priority" name="priority">
{% for value, label in priorities.items %}
<option value="{{ value }}" {% if selected_priority == value %}selected{% endif %}>{{ label }}</option>
{% endfor %}
</select>
</div>
<div class="field">
<label for="window">Time window</label>
<select id="window" name="window">
{% for value, label in time_windows.items %}
<option value="{{ value }}" {% if selected_window == value %}selected{% endif %}>{{ label }}</option>
{% endfor %}
</select>
</div>
<div class="field">
<label for="host">Host contains</label>
<input id="host" name="host" value="{{ host_filter }}" placeholder="web-01.example.test">
</div>
<div class="field">
<label for="run">Run</label>
<input id="run" name="run" value="{{ run_filter }}" inputmode="numeric" placeholder="12">
</div>
<div class="field">
<label for="q">Message contains</label>
<input id="q" name="q" value="{{ query }}">
</div>
<div class="form-actions">
<button type="submit">Filter logs</button>
<a class="button-link secondary" href="{% url 'logs' %}">Clear</a>
</div>
</form>
</section>
<section class="panel">
<h2>Messages</h2>
{% if error %}
<p class="status failed">{{ error }}</p>
{% else %}
<pre>{% for line in lines %}{{ line }}
{% empty %}No log messages matched the current filter.
{% endfor %}</pre>
{% endif %}
</section>
{% endblock %}

View File

@@ -0,0 +1,123 @@
<section class="panel dashboard-hosts-panel" id="hosts">
<h2>Hosts</h2>
<div class="host-list">
{% for host in hosts %}
<article class="host-card">
<div class="host-card-header">
<div class="host-card-title">
<a href="{% url 'host_detail' host.host %}">{{ host.host }}</a>
<span class="muted">{{ host.address }}</span>
</div>
<div class="host-card-status">
<span class="status {% if host.enabled %}ok{% else %}skipped{% endif %}">{{ host.enabled|yesno:"enabled,disabled" }}</span>
{% if host.queued_run_count %}
<span class="status queued">queued {{ host.queued_run_count }}</span>
{% endif %}
{% if host.running_run_count %}
<span class="status running">running {{ host.running_run_count }}</span>
{% endif %}
{% if host.warning_run_count %}
<span class="status warning">warning {{ host.warning_run_count }}</span>
{% endif %}
{% if host.failed_run_count %}
<span class="status failed">failed {{ host.failed_run_count }}</span>
{% endif %}
</div>
</div>
<div class="host-card-layout">
<div class="host-card-section">
<div class="host-card-section-title">Backup activity</div>
<div class="host-card-timeline">
<div class="host-card-item">
<div class="label">Latest Snapshot</div>
<div class="value">
{% if host.latest_snapshot %}
<a href="{% url 'snapshot_detail' host.latest_snapshot.id %}">{{ host.latest_snapshot.dirname }}</a>
<div class="muted">{{ host.latest_snapshot.kind }} {{ host.latest_snapshot.status }}</div>
{% else %}
<span class="muted">none</span>
{% endif %}
</div>
</div>
<div class="host-card-item">
<div class="label">Last Good Backup</div>
<div class="value">
{% if host.stats_summary.latest_good_run.id %}
<a href="{% url 'run_detail' host.stats_summary.latest_good_run.id %}">Run {{ host.stats_summary.latest_good_run.id }}</a>
<div class="muted">{{ host.stats_summary.latest_good_run.run_type }} {{ host.stats_summary.latest_good_run.duration_seconds|default:"" }}{% if host.stats_summary.latest_good_run.duration_seconds is not None %}s{% endif %}</div>
{% else %}
<span class="muted">none</span>
{% endif %}
</div>
</div>
<div class="host-card-item">
<div class="label">Latest Issue</div>
<div class="value">
{% if host.stats_summary.latest_problem_run.id %}
<a href="{% url 'run_detail' host.stats_summary.latest_problem_run.id %}">Run {{ host.stats_summary.latest_problem_run.id }}</a>
<div><span class="status {{ host.stats_summary.latest_problem_run.status }}">{{ host.stats_summary.latest_problem_run.status }}</span></div>
<div class="muted">{{ host.stats_summary.latest_problem_run.run_type }} {{ host.stats_summary.latest_problem_run.duration_seconds|default:"" }}{% if host.stats_summary.latest_problem_run.duration_seconds is not None %}s{% endif %}</div>
{% else %}
<span class="muted">none</span>
{% endif %}
</div>
</div>
<div class="host-card-item">
<div class="label">Next Run</div>
<div class="value">
{% if host.next_run_at %}
{{ host.next_run_at|date:"Y-m-d H:i T" }}
<div class="muted">{{ scheduler_timezone }}</div>
{% else %}
<span class="muted">none</span>
{% endif %}
</div>
</div>
</div>
</div>
<div class="host-card-section">
<div class="host-card-section-title">Snapshot health</div>
<div class="host-card-stats">
<div class="host-card-stat">
<div class="label">Snapshots</div>
<div class="value">{{ host.snapshot_count }}</div>
</div>
<div class="host-card-stat">
<div class="label">Runs</div>
<div class="value">{{ host.run_count }}</div>
</div>
<div class="host-card-stat">
<div class="label">New Data</div>
<div class="value">{{ host.stats_summary.latest_run.rsync.literal_data_bytes|filesizeformat }}</div>
</div>
<div class="host-card-stat">
<div class="label">Retention</div>
<div class="value">d{{ host.retention_daily }} w{{ host.retention_weekly }} m{{ host.retention_monthly }} y{{ host.retention_yearly }}</div>
</div>
</div>
</div>
</div>
{% if host.retention_warning.has_warning %}
<div class="host-card-warning">
<span class="status warning">retention</span>
{% if host.retention_warning.prune_exceeded %}
Scheduled prune would delete {{ host.retention_warning.delete_count }} snapshot(s), above max {{ host.retention_warning.max_delete }}.
{% endif %}
{% if host.retention_warning.incomplete_count %}
{{ host.retention_warning.incomplete_count }} incomplete snapshot(s) need review.
<form method="post" action="{% url 'resolve_host_incomplete_reviews' host.host %}">
{% csrf_token %}
<button type="submit" class="secondary">Mark reviewed</button>
</form>
{% endif %}
{% if host.retention_warning.error %}
{{ host.retention_warning.error }}
{% endif %}
</div>
{% endif %}
</article>
{% empty %}
<p class="muted">No hosts configured yet.</p>
{% endfor %}
</div>
</section>

View File

@@ -0,0 +1,130 @@
<section class="dashboard-priority-grid" aria-label="Operator priorities">
<article class="panel priority-panel dashboard-panel-required">
<h2>Required Action</h2>
{% if action_items %}
<div class="action-list">
{% for item in action_items %}
<a class="action-row {{ item.status }}" href="{{ item.url }}">
<span class="status {{ item.status }}">{{ item.label }}</span>
<span>
<strong>{{ item.host.host }}</strong>
<span class="muted">{{ item.message }}</span>
</span>
</a>
{% endfor %}
</div>
{% elif counts.hosts %}
<p><span class="status ok">ok</span> No queued, running, unreviewed warning/failed runs, or retention warnings.</p>
{% else %}
<p class="muted">Add a host to start tracking backup status here.</p>
{% endif %}
{% if counts.running_runs or counts.queued_runs %}
<div class="operator-state">
{% if counts.running_runs %}
<a class="status-summary running" href="{% url 'runs_list' %}?status=running">
<span class="status running">running</span>
<strong>{{ counts.running_runs }} backup run{{ counts.running_runs|pluralize }} in progress.</strong>
</a>
{% endif %}
{% if counts.queued_runs %}
<a class="status-summary queued" href="{% url 'runs_list' %}?status=queued">
<span class="status queued">queued</span>
<strong>{{ counts.queued_runs }} backup run{{ counts.queued_runs|pluralize }} waiting.</strong>
</a>
{% endif %}
</div>
{% endif %}
</article>
<article class="panel priority-panel dashboard-panel-schedules">
<h2>Next Scheduled Work <a class="button-link secondary compact" href="{% url 'schedules_list' %}">View all</a></h2>
{% if next_schedule_rows %}
<div class="schedule-list">
{% for row in next_schedule_rows %}
<a class="schedule-row" href="{% url 'host_detail' row.schedule.host.host %}">
<span>
<strong>{{ row.schedule.host.host }}</strong>
<span class="muted">{{ row.schedule.cron_expr }}</span>
</span>
<span class="schedule-time">
{% if row.next_run_at %}
{{ row.next_run_at|date:"Y-m-d H:i T" }}
<span class="muted">{{ scheduler_timezone }}</span>
{% else %}
<span class="muted">not due</span>
{% endif %}
</span>
</a>
{% endfor %}
</div>
{% else %}
<p class="muted">No enabled schedules yet.</p>
{% endif %}
</article>
<article class="panel priority-panel dashboard-panel-activity">
<h2>Recent Activity <a class="button-link secondary compact" href="{% url 'runs_list' %}">View all</a></h2>
{% if recent_runs %}
<div class="activity-list">
{% for run in recent_runs %}
<a class="activity-row" href="{% url 'run_detail' run.id %}">
<span class="status {{ run.status }}">{{ run.status }}</span>
<span>
<strong>Run {{ run.id }}</strong>
<span class="muted">{{ run.host.host }} · {{ run.run_type }} · {{ run.started_at|default:run.created_at }}</span>
</span>
</a>
{% endfor %}
</div>
{% else %}
<p class="muted">No backup runs recorded yet.</p>
{% endif %}
</article>
<article class="panel priority-panel dashboard-panel-storage">
<h2>Storage Pressure</h2>
{% if stats_summary.runs_sampled %}
<div class="storage-priority">
<div>
<div class="label">Backup root used</div>
<div class="value">
{% if stats_summary.capacity.used_percent is not None %}
{{ stats_summary.capacity.used_percent|floatformat:1 }}%
{% else %}
unknown
{% endif %}
</div>
{% if stats_summary.capacity.used_percent is not None %}
<div class="storage-meter" aria-label="Backup root storage usage">
<span style="width: {{ stats_summary.capacity.used_percent|floatformat:0 }}%"></span>
</div>
{% endif %}
</div>
<div class="storage-priority-facts">
<div>
<span class="label">Runway</span>
<strong>
{% if stats_summary.estimated_days_until_full %}
{{ stats_summary.estimated_days_until_full }} days
{% elif stats_summary.estimated_runs_until_full %}
{{ stats_summary.estimated_runs_until_full }} runs
{% else %}
unknown
{% endif %}
</strong>
</div>
<div>
<span class="label">New data</span>
<strong>{{ stats_summary.avg_daily_literal_data_bytes|filesizeformat }}/day</strong>
</div>
<div>
<span class="label">Available</span>
<strong>{{ stats_summary.capacity.available_bytes|filesizeformat }}</strong>
</div>
</div>
</div>
{% else %}
<p class="muted">Storage pressure appears after the first completed backup with stats.</p>
{% endif %}
</article>
</section>

View File

@@ -0,0 +1,150 @@
<section class="grid" aria-label="Run summary">
<div class="metric"><div class="label">Host</div><div class="value">{{ run.host.host }}</div></div>
<div class="metric"><div class="label">Status</div><div class="value"><span class="status {{ run.status }}">{{ run.status }}</span></div></div>
<div class="metric"><div class="label">Type</div><div class="value">{{ run.run_type }}</div></div>
<div class="metric"><div class="label">Rsync</div><div class="value">{{ run.rsync_exit_code|default:"" }}</div></div>
</section>
{% if can_cancel %}
<section class="panel highlight warning">
<h2>Run Control</h2>
<p>
Cancelling a queued run stops it immediately. Cancelling a running run asks the worker to stop
and records the cancellation request on this run.
</p>
<form method="post" action="{% url 'cancel_run' run.id %}">
{% csrf_token %}
<div class="form-actions">
<button type="submit" class="danger">Cancel run</button>
</div>
</form>
</section>
{% endif %}
{% if failure %}
<section class="panel highlight failed">
<h2>Failure</h2>
<div class="stack">
<div><strong>Category:</strong> {{ failure.category|default:"unknown" }}</div>
<div><strong>Summary:</strong> {{ failure_summary }}</div>
<div><strong>Hint:</strong> {{ failure.hint|default:"" }}</div>
</div>
</section>
{% endif %}
{% if run.status == "failed" or run.status == "warning" %}
{% if not run.reviewed_at %}
<section class="panel highlight warning">
<h2>Review Required</h2>
<p>Mark this run as reviewed after you have checked the failure or warning and no longer need it in the action queue.</p>
<form method="post" action="{% url 'resolve_run_review' run.id %}">
{% csrf_token %}
<div class="form-actions">
<button type="submit" class="secondary">Mark reviewed</button>
</div>
</form>
</section>
{% endif %}
{% endif %}
{% if run.reviewed_at %}
<section class="panel highlight success">
<h2>Review</h2>
<div class="stack">
<div><strong>Reviewed:</strong> {{ run.reviewed_at }}</div>
<div><strong>Reviewed by:</strong> {{ run.reviewed_by|default:"unknown" }}</div>
</div>
</section>
{% endif %}
{% if dry_run_summary %}
<section class="panel highlight {{ dry_run_summary.highlight_class }}">
<h2>Dry Run Summary</h2>
<section class="grid" aria-label="Dry run summary">
<div class="metric"><div class="label">Status</div><div class="value">{{ dry_run_summary.status }}</div></div>
<div class="metric"><div class="label">Files Seen</div><div class="value">{{ dry_run_summary.files_seen|default:"unknown" }}</div></div>
<div class="metric"><div class="label">Would Transfer</div><div class="value">{{ dry_run_summary.files_would_transfer|default:"unknown" }}</div></div>
<div class="metric"><div class="label">Transfer Estimate</div><div class="value">{{ dry_run_summary.transfer_estimate_bytes|filesizeformat }}</div></div>
<div class="metric"><div class="label">Total Size</div><div class="value">{{ dry_run_summary.total_file_size_bytes|filesizeformat }}</div></div>
<div class="metric"><div class="label">Link-Dest Saving</div><div class="value">{{ dry_run_summary.link_dest_estimated_savings_bytes|filesizeformat }}</div></div>
</section>
<div class="stack">
{% if dry_run_summary.duration_seconds is not None %}
<div><strong>Duration:</strong> {{ dry_run_summary.duration_seconds }}s</div>
{% endif %}
<div>
<strong>Log:</strong>
{% if dry_run_summary.log_available %}
<a href="{% url 'run_rsync_log' run.id %}">Open full rsync log</a>
{% elif rsync_log_path %}
<span class="muted">{{ rsync_log_path }} (missing)</span>
{% else %}
<span class="muted">not recorded yet</span>
{% endif %}
</div>
{% if dry_run_summary.warnings %}
<div><strong>Warnings:</strong></div>
<ul>
{% for warning in dry_run_summary.warnings %}
<li>{{ warning }}</li>
{% endfor %}
</ul>
{% else %}
<div><strong>Warnings:</strong> none recorded</div>
{% endif %}
</div>
</section>
{% endif %}
<div class="two-col">
<section class="panel">
<h2>Timing</h2>
<div class="stack">
<div><strong>Created:</strong> {{ run.created_at }}</div>
<div><strong>Started:</strong> {{ run.started_at|default:"" }}</div>
<div><strong>Ended:</strong> {{ run.ended_at|default:"" }}</div>
{% if execution %}
<div><strong>Worker:</strong> {{ execution.worker_host|default:"unknown" }}{% if execution.worker_pid %} pid {{ execution.worker_pid }}{% endif %}</div>
<div><strong>Worker heartbeat:</strong> {{ execution.heartbeat_at|default:"" }}</div>
{% endif %}
</div>
</section>
<section class="panel">
<h2>Snapshot</h2>
<div class="stack">
<div><strong>Snapshot:</strong> {% if run.snapshot %}<a href="{% url 'snapshot_detail' run.snapshot.id %}">{{ run.snapshot.dirname }}</a>{% else %}{{ run.snapshot_path|default:"" }}{% endif %}</div>
<div><strong>Base:</strong> {{ run.base_path|default:"" }}</div>
<div>
<strong>Rsync log:</strong>
{% if rsync_log_exists %}
<a href="{% url 'run_rsync_log' run.id %}">{{ rsync_log_path }}</a>
{% elif rsync_log_path %}
<span class="muted">{{ rsync_log_path }} (missing)</span>
{% else %}
<span class="muted">none</span>
{% endif %}
</div>
</div>
</section>
</div>
<section class="panel">
<h2>Rsync Log</h2>
<div class="stack spaced">
{% if rsync_log_exists %}
<div><a href="{% url 'run_rsync_log' run.id %}">Open full rsync log</a></div>
<div class="muted">{{ rsync_log_path }}</div>
{% elif rsync_log_path %}
<div class="muted">{{ rsync_log_path }} (missing)</div>
{% else %}
<div class="muted">No rsync log path recorded yet.</div>
{% endif %}
</div>
{% if rsync_log_tail %}
<pre>{% for line in rsync_log_tail %}{{ line }}{% if not forloop.last %}
{% endif %}{% endfor %}</pre>
{% else %}
<p class="muted">No recent rsync log output recorded yet.</p>
{% endif %}
</section>

View File

@@ -0,0 +1,79 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}Purged Snapshots | pobsync{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Retention</div>
<h1>Purged Snapshots</h1>
<div class="page-subtitle">Audit trail for snapshots removed by retention or manual purge actions.</div>
</div>
<section class="actions" aria-label="Purged snapshot actions">
<a class="button-link secondary" href="{% url 'dashboard' %}">Back to dashboard</a>
</section>
</header>
<section class="panel">
<h2>Filters</h2>
<form method="get" class="filter-form">
<div class="field">
<label for="host">Host</label>
<select id="host" name="host">
<option value="">All hosts</option>
{% for host in hosts %}
<option value="{{ host.host }}" {% if selected_host == host.host %}selected{% endif %}>{{ host.host }}</option>
{% endfor %}
</select>
</div>
<div class="field">
<label for="action">Action</label>
<select id="action" name="action">
<option value="">All actions</option>
{% for value, label in actions %}
<option value="{{ value }}" {% if selected_action == value %}selected{% endif %}>{{ label }}</option>
{% endfor %}
</select>
</div>
<div class="form-actions">
<button type="submit">Apply filters</button>
<a class="button-link secondary" href="{% url 'purged_snapshots' %}">Clear</a>
</div>
</form>
</section>
<section class="panel">
<h2>Purged Snapshot History</h2>
<p class="muted">Showing up to 200 of {{ total_count }} purged snapshot record(s).</p>
<table>
<thead>
<tr>
<th>Purged</th>
<th>Host</th>
<th>Kind</th>
<th>Dirname</th>
<th>Action</th>
<th>Reason</th>
<th>Triggered by</th>
<th>Path</th>
</tr>
</thead>
<tbody>
{% for snapshot in purged_snapshots %}
<tr>
<td>{{ snapshot.purged_at }}</td>
<td>{% if snapshot.host %}<a href="{% url 'host_detail' snapshot.host.host %}">{{ snapshot.host_name }}</a>{% else %}{{ snapshot.host_name }}{% endif %}</td>
<td>{{ snapshot.kind }}</td>
<td>{{ snapshot.dirname }}</td>
<td><span class="status skipped">{{ snapshot.get_action_display }}</span></td>
<td>{{ snapshot.reason|default:"" }}</td>
<td>{{ snapshot.triggered_by|default:"" }}</td>
<td class="muted">{{ snapshot.path }}</td>
</tr>
{% empty %}
<tr><td colspan="8" class="muted">No purged snapshots recorded yet.</td></tr>
{% endfor %}
</tbody>
</table>
</section>
{% endblock %}

View File

@@ -0,0 +1,244 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}Retention plan | {{ host.host }}{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Retention</div>
<h1>{{ host.host }}</h1>
<div class="page-subtitle">Preview which snapshots stay, which would be deleted, and whether incomplete cleanup is needed.</div>
</div>
<section class="actions" aria-label="Retention filters">
<a class="button-link" href="{% url 'host_detail' host.host %}">Back to host</a>
<a class="button-link" href="{% url 'host_retention_plan' host.host %}?kind=scheduled">Scheduled</a>
<a class="button-link" href="{% url 'host_retention_plan' host.host %}?kind=manual">Manual</a>
<a class="button-link" href="{% url 'host_retention_plan' host.host %}?kind=all">All</a>
<a class="button-link" href="{% url 'host_retention_plan' host.host %}?kind={{ kind }}&protect_bases=1">Protect bases</a>
</section>
</header>
<section class="grid" aria-label="Retention plan summary">
<div class="metric"><div class="label">Kind</div><div class="value">{{ plan.kind }}</div></div>
<div class="metric"><div class="label">Keep</div><div class="value">{{ plan.keep|length }}</div></div>
<div class="metric"><div class="label">Would Delete</div><div class="value">{{ plan.delete|length }}</div></div>
<div class="metric"><div class="label">Scheduled Limit</div><div class="value">{{ scheduled_prune_limit|default:"none" }}</div></div>
<div class="metric"><div class="label">Incomplete</div><div class="value">{{ plan.incomplete|length }}</div></div>
</section>
{% if scheduled_prune_exceeded %}
<section class="panel highlight warning">
<h2>Scheduled Prune Limit</h2>
<p>
This plan would delete {{ plan.delete|length }} snapshot(s), which exceeds the scheduled prune limit of
{{ scheduled_prune_limit }}. Scheduled pruning will refuse to apply this plan until the limit or retention
selection is adjusted.
</p>
</section>
{% endif %}
{% if plan.incomplete %}
<section class="panel highlight warning">
<h2>Incomplete Snapshots</h2>
<p>
{{ plan.incomplete|length }} incomplete snapshot(s) exist for this host. Retention does not delete incomplete
snapshots automatically because they can indicate an interrupted backup that should be inspected first.
</p>
<p>
After inspection, use the dedicated cleanup form below to delete only incomplete snapshot directories and their
tracking records. Successful scheduled and manual snapshots are not touched by this cleanup.
</p>
</section>
{% endif %}
<section class="panel">
<h2>Policy</h2>
<div class="stack">
<div><strong>Daily:</strong> {{ plan.retention.daily }}</div>
<div><strong>Weekly:</strong> {{ plan.retention.weekly }}</div>
<div><strong>Monthly:</strong> {{ plan.retention.monthly }}</div>
<div><strong>Yearly:</strong> {{ plan.retention.yearly }}</div>
<div><strong>Protect bases:</strong> {{ protect_bases|yesno:"yes,no" }}</div>
<div class="muted">
{% if protect_bases %}
Base snapshots referenced by kept snapshots are also kept and marked with a base-of reason.
{% else %}
Base snapshots are only kept when they match the regular retention policy.
{% endif %}
</div>
{% if schedule %}
<div><strong>Schedule pruning:</strong> {{ schedule.prune|yesno:"enabled,disabled" }}</div>
<div><strong>Schedule max delete:</strong> {{ schedule.prune_max_delete }}</div>
{% endif %}
</div>
</section>
<section class="panel">
<h2>Would Delete</h2>
<table>
<thead>
<tr>
<th>Kind</th>
<th>Dirname</th>
<th>Started</th>
<th>Status</th>
<th>Reason</th>
<th>Path</th>
</tr>
</thead>
<tbody>
{% for snapshot in plan.delete %}
<tr>
<td>{{ snapshot.kind }}</td>
<td>{{ snapshot.dirname }}</td>
<td>{{ snapshot.dt }}</td>
<td>{{ snapshot.status|default:"" }}</td>
<td>{{ snapshot.reason }}</td>
<td class="muted">{{ snapshot.path }}</td>
</tr>
{% empty %}
<tr><td colspan="6" class="muted">Retention would not delete snapshots for this selection.</td></tr>
{% endfor %}
</tbody>
</table>
</section>
{% if plan.delete %}
<section class="panel highlight warning">
<h2>Apply Retention</h2>
<p class="muted">
This permanently deletes the snapshot directories listed in Would Delete. Confirm the host and delete count
before applying the plan.
</p>
<form method="post" action="{% url 'apply_host_retention' host.host %}" class="form-grid">
{% csrf_token %}
{{ apply_form.non_field_errors }}
{{ apply_form.kind.as_hidden }}
<div class="field">
{{ apply_form.max_delete.errors }}
<label for="{{ apply_form.max_delete.id_for_label }}">Max delete</label>
{{ apply_form.max_delete }}
<div class="helptext">Must be at least {{ plan.delete|length }} for the snapshots shown in Would Delete.</div>
</div>
<div class="field">
{{ apply_form.protect_bases.errors }}
<label for="{{ apply_form.protect_bases.id_for_label }}">Protect bases</label>
{{ apply_form.protect_bases }}
</div>
<div class="field">
{{ apply_form.confirm_host.errors }}
<label for="{{ apply_form.confirm_host.id_for_label }}">Confirm host</label>
{{ apply_form.confirm_host }}
<div class="helptext">{{ apply_form.confirm_host.help_text }}</div>
</div>
<div class="field">
{{ apply_form.confirm_delete_count.errors }}
<label for="{{ apply_form.confirm_delete_count.id_for_label }}">Confirm delete count</label>
{{ apply_form.confirm_delete_count }}
<div class="helptext">{{ apply_form.confirm_delete_count.help_text }}</div>
</div>
<div class="form-actions">
<button type="submit" class="danger">Apply retention</button>
<a class="button-link secondary" href="{% url 'host_retention_plan' host.host %}?kind={{ kind }}">Cancel</a>
</div>
</form>
</section>
{% endif %}
<section class="panel">
<h2>Keep Reasons</h2>
<table>
<thead>
<tr>
<th>Kind</th>
<th>Dirname</th>
<th>Started</th>
<th>Status</th>
<th>Reasons</th>
</tr>
</thead>
<tbody>
{% for snapshot in plan.keep_items %}
<tr>
<td>{{ snapshot.kind }}</td>
<td>{{ snapshot.dirname }}</td>
<td>{{ snapshot.dt }}</td>
<td>{{ snapshot.status|default:"" }}</td>
<td>{{ snapshot.reason }}</td>
</tr>
{% empty %}
<tr><td colspan="5" class="muted">No snapshots matched this retention selection.</td></tr>
{% endfor %}
</tbody>
</table>
</section>
{% if plan.incomplete %}
<section class="panel">
<h2>Incomplete Snapshots</h2>
<table>
<thead>
<tr>
<th>Dirname</th>
<th>Started</th>
<th>Status</th>
<th>Reason</th>
<th>Path</th>
</tr>
</thead>
<tbody>
{% for snapshot in plan.incomplete %}
<tr>
<td>{{ snapshot.dirname }}</td>
<td>{{ snapshot.dt }}</td>
<td>{{ snapshot.status|default:"" }}</td>
<td>{{ snapshot.reason }}</td>
<td class="muted">{{ snapshot.path }}</td>
</tr>
{% endfor %}
</tbody>
</table>
<h3>Cleanup Incomplete Snapshots</h3>
<p class="muted">
This deletes only incomplete snapshot directories and their tracking records. Successful manual and scheduled
snapshots are not touched.
</p>
<form method="post" action="{% url 'cleanup_host_incomplete_snapshots' host.host %}" class="form-grid">
{% csrf_token %}
{{ incomplete_cleanup_form.non_field_errors }}
<div class="field">
{{ incomplete_cleanup_form.max_delete.errors }}
<label for="{{ incomplete_cleanup_form.max_delete.id_for_label }}">Max delete</label>
{{ incomplete_cleanup_form.max_delete }}
<div class="helptext">{{ incomplete_cleanup_form.max_delete.help_text }}</div>
</div>
<div class="field">
{{ incomplete_cleanup_form.confirm_host.errors }}
<label for="{{ incomplete_cleanup_form.confirm_host.id_for_label }}">Confirm host</label>
{{ incomplete_cleanup_form.confirm_host }}
<div class="helptext">{{ incomplete_cleanup_form.confirm_host.help_text }}</div>
</div>
<div class="field">
{{ incomplete_cleanup_form.confirm_delete_count.errors }}
<label for="{{ incomplete_cleanup_form.confirm_delete_count.id_for_label }}">Confirm incomplete count</label>
{{ incomplete_cleanup_form.confirm_delete_count }}
<div class="helptext">{{ incomplete_cleanup_form.confirm_delete_count.help_text }}</div>
</div>
<div class="form-actions">
<button type="submit" class="danger">Delete incomplete snapshots</button>
<a class="button-link secondary" href="{% url 'host_retention_plan' host.host %}?kind={{ kind }}">Cancel</a>
</div>
</form>
</section>
{% endif %}
{% endblock %}

View File

@@ -0,0 +1,123 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}Run {{ run.id }} | {{ run.host.host }}{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Backup run</div>
<h1>Run {{ run.id }}</h1>
<div class="page-subtitle">{{ run.host.host }} · {{ run.run_type }} · {{ run.status }}</div>
</div>
<section class="actions" aria-label="Run actions">
<a class="button-link" href="{% url 'host_detail' run.host.host %}">Back to host</a>
</section>
</header>
<div
data-refresh-url="{% url 'run_detail_live' run.id %}"
data-refresh-interval="5000"
data-refresh-active="{{ can_auto_refresh|yesno:'true,false' }}"
aria-live="polite"
>
{% include "pobsync_backend/partials/run_detail_live.html" %}
</div>
{% if requested %}
<section class="panel">
<h2>Requested Options</h2>
<div class="stack">
<div><strong>Dry run:</strong> {{ requested.dry_run|yesno:"yes,no" }}</div>
<div><strong>Verbose rsync output:</strong> {{ requested.verbose_output|yesno:"yes,no" }}</div>
<div><strong>Apply retention:</strong> {{ requested.prune|yesno:"yes,no" }}</div>
<div><strong>Retention max delete:</strong> {{ requested.prune_max_delete }}</div>
<div><strong>Protect bases:</strong> {{ requested.prune_protect_bases|yesno:"yes,no" }}</div>
</div>
</section>
{% endif %}
<section class="panel">
<h2>Rsync Command</h2>
{% if rsync_command %}
<pre>{% for part in rsync_command %}{{ part }}{% if not forloop.last %}
{% endif %}{% endfor %}</pre>
{% else %}
<p class="muted">No rsync command recorded yet.</p>
{% endif %}
</section>
{% if stats %}
<section class="panel">
<h2>Stats</h2>
<div class="two-col">
<div class="stack">
<div><strong>Duration:</strong> {{ stats.duration_seconds|default:"" }}{% if stats.duration_seconds is not None %}s{% endif %}</div>
<div><strong>Files seen:</strong> {{ stats.rsync.files_total|default:"" }}</div>
<div><strong>Files transferred:</strong> {{ stats.rsync.files_transferred|default:"" }}</div>
<div><strong>Files created:</strong> {{ stats.rsync.files_created|default:"" }}</div>
<div><strong>Files deleted:</strong> {{ stats.rsync.files_deleted|default:"" }}</div>
</div>
<div class="stack">
<div><strong>Total file size:</strong> {{ stats.rsync.total_file_size_bytes|filesizeformat }}</div>
<div><strong>Transferred file size:</strong> {{ stats.rsync.total_transferred_file_size_bytes|filesizeformat }}</div>
<div><strong>Literal data:</strong> {{ stats.rsync.literal_data_bytes|filesizeformat }}</div>
<div><strong>Matched data:</strong> {{ stats.rsync.matched_data_bytes|filesizeformat }}</div>
<div><strong>Estimated link-dest saving:</strong> {{ stats.rsync.link_dest_estimated_savings_bytes|filesizeformat }}</div>
</div>
</div>
</section>
{% endif %}
{% if has_prune_result %}
<section class="panel highlight {% if prune_result.ok %}success{% else %}warning{% endif %}">
<h2>Retention</h2>
<div class="stack">
<div><strong>Status:</strong> {% if prune_result.ok %}ok{% else %}warning{% endif %}</div>
{% if prune_result.kind %}<div><strong>Kind:</strong> {{ prune_result.kind }}</div>{% endif %}
{% if prune_result.planned_delete_count is not None %}<div><strong>Planned deletions:</strong> {{ prune_result.planned_delete_count }}</div>{% endif %}
{% if prune_result.deleted %}<div><strong>Deleted:</strong> {{ prune_result.deleted|length }}</div>{% endif %}
{% if prune_result.max_delete is not None %}<div><strong>Max delete:</strong> {{ prune_result.max_delete }}</div>{% endif %}
{% if prune_result.protect_bases is not None %}<div><strong>Protect bases:</strong> {{ prune_result.protect_bases|yesno:"yes,no" }}</div>{% endif %}
{% if prune_result.incomplete_ignored_count %}<div><strong>Incomplete ignored:</strong> {{ prune_result.incomplete_ignored_count }}</div>{% endif %}
{% if prune_result.actions %}
<div><strong>Actions:</strong></div>
<ul>
{% for action in prune_result.actions %}
<li>{{ action }}</li>
{% endfor %}
</ul>
{% endif %}
{% if prune_result.error %}<div><strong>Error:</strong> {{ prune_result.error }}</div>{% endif %}
{% if prune_result.type %}<div><strong>Type:</strong> {{ prune_result.type }}</div>{% endif %}
</div>
</section>
{% endif %}
{% if retention_warning.has_warning %}
<section class="panel highlight warning">
<h2>Retention Warnings</h2>
<div class="stack">
{% if retention_warning.prune_exceeded %}
<div>
Scheduled pruning for this host would delete {{ retention_warning.delete_count }} snapshot(s), above max
delete {{ retention_warning.max_delete }}.
</div>
{% endif %}
{% if retention_warning.incomplete_count %}
<div>
{{ retention_warning.incomplete_count }} incomplete snapshot(s) exist for this host and are excluded from
retention cleanup.
</div>
{% endif %}
{% if retention_warning.error %}
<div>{{ retention_warning.error }}</div>
{% endif %}
</div>
</section>
{% endif %}
<section class="panel">
<h2>Raw Result</h2>
<pre>{{ result_json }}</pre>
</section>
{% endblock %}

View File

@@ -0,0 +1,121 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}Runs | pobsync{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Activity</div>
<h1>Runs</h1>
<div class="page-subtitle">Review queued, running, completed, warning, failed, and cancelled backup runs.</div>
</div>
<section class="actions" aria-label="Run list actions">
<a class="button-link secondary" href="{% url 'dashboard' %}">Back to dashboard</a>
</section>
</header>
<section class="panel">
<h2>Filters</h2>
<form method="get" class="filter-form">
<div class="field">
<label for="status">Status</label>
<select id="status" name="status">
<option value="">All statuses</option>
{% for value, label in statuses %}
<option value="{{ value }}" {% if selected_status == value %}selected{% endif %}>{{ label }}</option>
{% endfor %}
</select>
</div>
<div class="field">
<label for="type">Type</label>
<select id="type" name="type">
<option value="">All types</option>
{% for value, label in run_types %}
<option value="{{ value }}" {% if selected_type == value %}selected{% endif %}>{{ label }}</option>
{% endfor %}
</select>
</div>
<div class="field">
<label for="host">Host</label>
<select id="host" name="host">
<option value="">All hosts</option>
{% for host in hosts %}
<option value="{{ host.host }}" {% if selected_host == host.host %}selected{% endif %}>{{ host.host }}</option>
{% endfor %}
</select>
</div>
<div class="field">
<label for="review">Review</label>
<select id="review" name="review">
<option value="">All review states</option>
<option value="needed" {% if selected_review == "needed" %}selected{% endif %}>Needs review</option>
<option value="reviewed" {% if selected_review == "reviewed" %}selected{% endif %}>Reviewed</option>
</select>
</div>
<div class="form-actions">
<button type="submit">Apply filters</button>
<a class="button-link secondary" href="{% url 'runs_list' %}">Clear</a>
</div>
</form>
</section>
<section class="panel">
<h2>Backup Runs</h2>
<p class="muted">Showing up to 200 of {{ total_count }} run{{ total_count|pluralize }}.</p>
<table>
<thead>
<tr>
<th>Run</th>
<th>Host</th>
<th>Status</th>
<th>Type</th>
<th>Created</th>
<th>Started</th>
<th>Ended</th>
<th>Snapshot</th>
<th>Review</th>
</tr>
</thead>
<tbody>
{% for run in runs %}
<tr>
<td><a href="{% url 'run_detail' run.id %}">Run {{ run.id }}</a></td>
<td><a href="{% url 'host_detail' run.host.host %}">{{ run.host.host }}</a></td>
<td><span class="status {{ run.status }}">{{ run.status }}</span></td>
<td>{{ run.run_type }}</td>
<td>{{ run.created_at }}</td>
<td>{{ run.started_at|default:"" }}</td>
<td>{{ run.ended_at|default:"" }}</td>
<td>
{% if run.snapshot %}
<a href="{% url 'snapshot_detail' run.snapshot.id %}">{{ run.snapshot.dirname }}</a>
{% elif run.snapshot_path %}
<span class="muted">{{ run.snapshot_path }}</span>
{% else %}
<span class="muted">none</span>
{% endif %}
</td>
<td>
{% if run.reviewed_at %}
reviewed
{% elif run.status == "failed" or run.status == "warning" %}
<div class="stack">
<span class="status warning">needed</span>
<form class="inline-form" method="post" action="{% url 'resolve_run_review' run.id %}">
{% csrf_token %}
<input type="hidden" name="next" value="{{ request.get_full_path }}">
<button type="submit" class="secondary compact">Mark reviewed</button>
</form>
</div>
{% else %}
<span class="muted">none</span>
{% endif %}
</td>
</tr>
{% empty %}
<tr><td colspan="9" class="muted">No runs matched the current filter.</td></tr>
{% endfor %}
</tbody>
</table>
</section>
{% endblock %}

View File

@@ -0,0 +1,39 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}Schedule | {{ host.host }}{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Schedule</div>
<h1>{{ host.host }}</h1>
<div class="page-subtitle">Automatic backup timing and scheduled prune behavior for this host.</div>
</div>
<section class="actions" aria-label="Schedule actions">
<a class="button-link" href="{% url 'host_detail' host.host %}">Back to host</a>
</section>
</header>
<section class="panel">
<h2>{% if schedule %}Edit Schedule{% else %}Create Schedule{% endif %}</h2>
<p class="muted">Schedules use cron-style timing syntax, but they are evaluated by the pobsync scheduler service.</p>
<form method="post" class="form-grid">
{% csrf_token %}
{{ form.non_field_errors }}
{% for field in form %}
<div class="field">
{{ field.errors }}
<label for="{{ field.id_for_label }}">{{ field.label }}</label>
{{ field }}
{% if field.help_text %}<div class="helptext">{{ field.help_text }}</div>{% endif %}
</div>
{% endfor %}
<div class="form-actions">
<button type="submit">Save schedule</button>
<a class="button-link secondary" href="{% url 'host_detail' host.host %}">Cancel</a>
</div>
</form>
</section>
{% endblock %}

View File

@@ -0,0 +1,101 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}Schedules | pobsync{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Scheduler</div>
<h1>Schedules</h1>
<div class="page-subtitle">Review configured backup schedules, next run times, prune settings, and recent scheduler state.</div>
</div>
<section class="actions" aria-label="Schedule list actions">
<a class="button-link secondary" href="{% url 'dashboard' %}">Back to dashboard</a>
</section>
</header>
<section class="panel">
<h2>Filters</h2>
<form method="get" class="filter-form">
<div class="field">
<label for="host">Host</label>
<select id="host" name="host">
<option value="">All hosts</option>
{% for host in hosts %}
<option value="{{ host.host }}" {% if selected_host == host.host %}selected{% endif %}>{{ host.host }}</option>
{% endfor %}
</select>
</div>
<div class="field">
<label for="enabled">Enabled</label>
<select id="enabled" name="enabled">
<option value="">All schedules</option>
<option value="yes" {% if selected_enabled == "yes" %}selected{% endif %}>Enabled</option>
<option value="no" {% if selected_enabled == "no" %}selected{% endif %}>Disabled</option>
</select>
</div>
<div class="field">
<label for="prune">Prune</label>
<select id="prune" name="prune">
<option value="">All prune states</option>
<option value="yes" {% if selected_prune == "yes" %}selected{% endif %}>Prune enabled</option>
<option value="no" {% if selected_prune == "no" %}selected{% endif %}>Prune disabled</option>
</select>
</div>
<div class="form-actions">
<button type="submit">Apply filters</button>
<a class="button-link secondary" href="{% url 'schedules_list' %}">Clear</a>
</div>
</form>
</section>
<section class="panel">
<h2>Configured Schedules</h2>
<p class="muted">Showing up to 200 of {{ total_count }} schedule{{ total_count|pluralize }}. Times use {{ scheduler_timezone }}.</p>
<table>
<thead>
<tr>
<th>Host</th>
<th>Expression</th>
<th>Enabled</th>
<th>Next Run</th>
<th>Prune</th>
<th>Last Status</th>
<th>Last Started</th>
<th>Last Finished</th>
<th>Actions</th>
</tr>
</thead>
<tbody>
{% for row in schedule_rows %}
{% with schedule=row.schedule %}
<tr>
<td><a href="{% url 'host_detail' schedule.host.host %}">{{ schedule.host.host }}</a></td>
<td><code>{{ schedule.cron_expr }}</code></td>
<td><span class="status {% if schedule.enabled %}ok{% else %}skipped{% endif %}">{{ schedule.enabled|yesno:"enabled,disabled" }}</span></td>
<td>
{% if row.next_run_at %}
{{ row.next_run_at|date:"Y-m-d H:i T" }}
{% else %}
<span class="muted">none</span>
{% endif %}
</td>
<td>
<span class="status {% if schedule.prune %}ok{% else %}skipped{% endif %}">{{ schedule.prune|yesno:"enabled,disabled" }}</span>
{% if schedule.prune %}
<div class="muted">max {{ schedule.prune_max_delete }}{% if schedule.prune_protect_bases %}, protects bases{% endif %}</div>
{% endif %}
</td>
<td>{% if schedule.last_status %}<span class="status {{ schedule.last_status }}">{{ schedule.last_status }}</span>{% else %}<span class="muted">none</span>{% endif %}</td>
<td>{{ schedule.last_started_at|default:"" }}</td>
<td>{{ schedule.last_finished_at|default:"" }}</td>
<td><a class="button-link secondary" href="{% url 'edit_host_schedule' schedule.host.host %}">Edit</a></td>
</tr>
{% endwith %}
{% empty %}
<tr><td colspan="9" class="muted">No schedules matched the current filter.</td></tr>
{% endfor %}
</tbody>
</table>
</section>
{% endblock %}

View File

@@ -0,0 +1,47 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}Self Check | pobsync{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Operations</div>
<h1>Self Check</h1>
<div class="page-subtitle">Runtime, filesystem, service, and configuration checks for this pobsync installation.</div>
</div>
<section class="actions" aria-label="Self check actions">
<a class="button-link secondary" href="{% url 'dashboard' %}">Back to dashboard</a>
</section>
</header>
<section class="grid" aria-label="Self check summary">
<div class="metric"><div class="label">OK</div><div class="value">{{ summary.ok }}</div></div>
<div class="metric"><div class="label">Warnings</div><div class="value">{{ summary.warning }}</div></div>
<div class="metric"><div class="label">Failed</div><div class="value">{{ summary.failed }}</div></div>
<div class="metric"><div class="label">Skipped</div><div class="value">{{ summary.skipped }}</div></div>
</section>
<section class="panel">
<h2>Checks</h2>
<table>
<thead>
<tr>
<th>Status</th>
<th>Check</th>
<th>Message</th>
<th>Detail</th>
</tr>
</thead>
<tbody>
{% for check in checks %}
<tr>
<td><span class="status {{ check.status }}">{{ check.status }}</span></td>
<td>{{ check.name }}</td>
<td>{{ check.message }}</td>
<td class="muted">{{ check.detail }}</td>
</tr>
{% endfor %}
</tbody>
</table>
</section>
{% endblock %}

View File

@@ -0,0 +1,166 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}Snapshot {{ snapshot.dirname }} | {{ snapshot.host.host }}{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Snapshot</div>
<h1>{{ snapshot.dirname }}</h1>
<div class="page-subtitle">{{ snapshot.host.host }} · {{ snapshot.kind }} · {{ snapshot.status }}</div>
</div>
<section class="actions" aria-label="Snapshot actions">
<a class="button-link" href="{% url 'host_detail' snapshot.host.host %}">Back to host</a>
</section>
</header>
<section class="grid" aria-label="Snapshot summary">
<div class="metric"><div class="label">Host</div><div class="value">{{ snapshot.host.host }}</div></div>
<div class="metric"><div class="label">Kind</div><div class="value">{{ snapshot.kind }}</div></div>
<div class="metric"><div class="label">Status</div><div class="value">{{ snapshot.status|default:"" }}</div></div>
<div class="metric"><div class="label">Runs</div><div class="value">{{ backup_runs|length }}</div></div>
</section>
<div class="two-col">
<section class="panel">
<h2>Snapshot</h2>
<div class="stack">
<div><strong>Path:</strong> {{ snapshot.path }}</div>
<div><strong>Started:</strong> {{ snapshot.started_at|default:"" }}</div>
<div><strong>Ended:</strong> {{ snapshot.ended_at|default:"" }}</div>
<div><strong>Discovered:</strong> {{ snapshot.discovered_at }}</div>
</div>
</section>
<section class="panel">
<h2>Base</h2>
<div class="stack">
<div><strong>Record:</strong> {% if snapshot.base %}<a href="{% url 'snapshot_detail' snapshot.base.id %}">{{ snapshot.base.dirname }}</a>{% endif %}</div>
<div><strong>Kind:</strong> {{ snapshot.base_kind }}</div>
<div><strong>Dirname:</strong> {{ snapshot.base_dirname }}</div>
<div><strong>Path:</strong> {{ snapshot.base_path }}</div>
</div>
</section>
</div>
{% if stats %}
<section class="panel">
<h2>Stats</h2>
<div class="two-col">
<div class="stack">
<div><strong>Duration:</strong> {{ stats.duration_seconds|default:"" }}{% if stats.duration_seconds is not None %}s{% endif %}</div>
<div><strong>Files seen:</strong> {{ stats.rsync.files_total|default:"" }}</div>
<div><strong>Files transferred:</strong> {{ stats.rsync.files_transferred|default:"" }}</div>
<div><strong>Total file size:</strong> {{ stats.rsync.total_file_size_bytes|filesizeformat }}</div>
<div><strong>Estimated link-dest saving:</strong> {{ stats.rsync.link_dest_estimated_savings_bytes|filesizeformat }}</div>
</div>
<div class="stack">
<div><strong>Snapshot apparent size:</strong> {{ stats.storage.snapshot.apparent_size_bytes|filesizeformat }}</div>
<div><strong>Snapshot allocated size:</strong> {{ stats.storage.snapshot.allocated_size_bytes|filesizeformat }}</div>
<div><strong>Hardlinked files:</strong> {{ stats.storage.snapshot.hardlinked_files|default:"" }}</div>
<div><strong>Backup root used:</strong> {{ stats.storage.capacity.used_percent|default:"" }}%</div>
<div><strong>Backup root available:</strong> {{ stats.storage.capacity.available_bytes|filesizeformat }}</div>
</div>
</div>
</section>
{% endif %}
<section class="panel">
<h2>Restore Guidance</h2>
<div class="stack spaced">
<div><strong>Snapshot data path:</strong> {{ restore.source_path }}</div>
<div><strong>Example staging destination:</strong> {{ restore.destination_path }}</div>
<div class="muted">
Restore from the snapshot's <code>data/</code> directory. Start with a dry run, restore to a staging path first,
and only then copy data back to a live host or service path.
</div>
</div>
<div class="stack spaced">
<div><strong>Inspect the snapshot:</strong></div>
<pre>{{ restore.inspect_command }}</pre>
</div>
<div class="stack spaced">
<div><strong>Dry-run restore to staging:</strong></div>
<pre>{{ restore.dry_run_command }}</pre>
</div>
<div class="stack spaced">
<div><strong>Restore to staging:</strong></div>
<pre>{{ restore.local_command }}</pre>
</div>
<div class="stack spaced">
<div><strong>Dry-run a directory restore:</strong></div>
<pre>{{ restore.partial_dry_run_command }}</pre>
<div class="muted">Replace <code>{{ restore.example_relative_path }}</code> with the path you want to restore.</div>
</div>
<div class="stack spaced">
<div><strong>Dry-run a single file restore:</strong></div>
<pre>{{ restore.file_dry_run_command }}</pre>
<div class="muted">Replace <code>{{ restore.example_file_relative_path }}</code> with the file you want to restore.</div>
</div>
<div class="stack spaced">
<div><strong>Dry-run restore back to the original host:</strong></div>
<pre>{{ restore.remote_dry_run_command }}</pre>
</div>
<p class="muted">
Snapshots can contain hardlinks to files shared with earlier snapshots. Treat snapshot directories as read-only:
copy data out with rsync instead of editing files in place.
</p>
</section>
<section class="panel">
<h2>Backup Runs</h2>
<table>
<thead>
<tr>
<th>Run</th>
<th>Status</th>
<th>Started</th>
<th>Ended</th>
</tr>
</thead>
<tbody>
{% for run in backup_runs %}
<tr>
<td><a href="{% url 'run_detail' run.id %}">Run {{ run.id }}</a></td>
<td><span class="status {{ run.status }}">{{ run.status }}</span></td>
<td>{{ run.started_at|default:"" }}</td>
<td>{{ run.ended_at|default:"" }}</td>
</tr>
{% empty %}
<tr><td colspan="4" class="muted">No backup runs linked to this snapshot.</td></tr>
{% endfor %}
</tbody>
</table>
</section>
<section class="panel">
<h2>Derived Snapshots</h2>
<table>
<thead>
<tr>
<th>Kind</th>
<th>Status</th>
<th>Started</th>
<th>Dirname</th>
</tr>
</thead>
<tbody>
{% for child in derived_snapshots %}
<tr>
<td>{{ child.kind }}</td>
<td>{{ child.status }}</td>
<td>{{ child.started_at|default:"" }}</td>
<td><a href="{% url 'snapshot_detail' child.id %}">{{ child.dirname }}</a></td>
</tr>
{% empty %}
<tr><td colspan="4" class="muted">No derived snapshots linked to this snapshot.</td></tr>
{% endfor %}
</tbody>
</table>
</section>
<section class="panel">
<h2>Metadata</h2>
<pre>{{ metadata_json }}</pre>
</section>
{% endblock %}

View File

@@ -0,0 +1,96 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}Snapshots | pobsync{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Snapshots</div>
<h1>Snapshots</h1>
<div class="page-subtitle">Browse discovered scheduled, manual, and incomplete snapshots across all hosts.</div>
</div>
<section class="actions" aria-label="Snapshot list actions">
<a class="button-link secondary" href="{% url 'dashboard' %}">Back to dashboard</a>
</section>
</header>
<section class="panel">
<h2>Filters</h2>
<form method="get" class="filter-form">
<div class="field">
<label for="host">Host</label>
<select id="host" name="host">
<option value="">All hosts</option>
{% for host in hosts %}
<option value="{{ host.host }}" {% if selected_host == host.host %}selected{% endif %}>{{ host.host }}</option>
{% endfor %}
</select>
</div>
<div class="field">
<label for="kind">Kind</label>
<select id="kind" name="kind">
<option value="">All kinds</option>
{% for value, label in kinds %}
<option value="{{ value }}" {% if selected_kind == value %}selected{% endif %}>{{ label }}</option>
{% endfor %}
</select>
</div>
<div class="field">
<label for="status">Status</label>
<select id="status" name="status">
<option value="">All statuses</option>
{% for value in statuses %}
<option value="{{ value }}" {% if selected_status == value %}selected{% endif %}>{{ value }}</option>
{% endfor %}
</select>
</div>
<div class="form-actions">
<button type="submit">Apply filters</button>
<a class="button-link secondary" href="{% url 'snapshots_list' %}">Clear</a>
</div>
</form>
</section>
<section class="panel">
<h2>Snapshot Records</h2>
<p class="muted">Showing up to 200 of {{ total_count }} snapshot{{ total_count|pluralize }}.</p>
<table>
<thead>
<tr>
<th>Snapshot</th>
<th>Host</th>
<th>Kind</th>
<th>Status</th>
<th>Started</th>
<th>Ended</th>
<th>Base</th>
<th>Path</th>
</tr>
</thead>
<tbody>
{% for snapshot in snapshots %}
<tr>
<td><a href="{% url 'snapshot_detail' snapshot.id %}">{{ snapshot.dirname }}</a></td>
<td><a href="{% url 'host_detail' snapshot.host.host %}">{{ snapshot.host.host }}</a></td>
<td>{{ snapshot.kind }}</td>
<td>{% if snapshot.status %}<span class="status {{ snapshot.status }}">{{ snapshot.status }}</span>{% else %}<span class="muted">unknown</span>{% endif %}</td>
<td>{{ snapshot.started_at|default:"" }}</td>
<td>{{ snapshot.ended_at|default:"" }}</td>
<td>
{% if snapshot.base %}
<a href="{% url 'snapshot_detail' snapshot.base.id %}">{{ snapshot.base.dirname }}</a>
{% elif snapshot.base_dirname %}
<span class="muted">{{ snapshot.base_dirname }}</span>
{% else %}
<span class="muted">none</span>
{% endif %}
</td>
<td class="muted">{{ snapshot.path }}</td>
</tr>
{% empty %}
<tr><td colspan="8" class="muted">No snapshots matched the current filter.</td></tr>
{% endfor %}
</tbody>
</table>
</section>
{% endblock %}

View File

@@ -0,0 +1,75 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}{% if credential %}SSH Key | {{ credential.name }}{% else %}New SSH Key{% endif %} | pobsync{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Access</div>
<h1>{% if credential %}SSH Key: {{ credential.name }}{% else %}New SSH Key{% endif %}</h1>
<div class="page-subtitle">{% if credential %}Review key metadata, known hosts, and deletion safety for this credential.{% else %}Register an existing private key for use by pobsync backups.{% endif %}</div>
</div>
<section class="actions" aria-label="SSH key form actions">
<a class="button-link" href="{% url 'ssh_credentials' %}">Back to SSH keys</a>
</section>
</header>
<section class="panel">
<h2>{% if credential %}Edit SSH Credential{% else %}Create SSH Credential{% endif %}</h2>
{% if credential and credential.public_key %}
<div class="field">
<label>Public key</label>
<pre><code>{{ credential.public_key }}</code></pre>
</div>
{% endif %}
{% if credential and credential.key_path %}
<p class="muted">Private key path: <code>{{ credential.key_path }}</code></p>
{% endif %}
{% if credential and credential.fingerprint %}
<p class="muted">Fingerprint: <code>{{ credential.fingerprint }}</code></p>
{% endif %}
<form method="post" enctype="multipart/form-data" class="form-grid">
{% csrf_token %}
{{ form.non_field_errors }}
{% for field in form %}
<div class="field">
{{ field.errors }}
<label for="{{ field.id_for_label }}">{{ field.label }}</label>
{{ field }}
{% if field.help_text %}<div class="helptext">{{ field.help_text }}</div>{% endif %}
</div>
{% endfor %}
<div class="form-actions">
<button type="submit">{% if credential %}Save SSH key{% else %}Create SSH key{% endif %}</button>
<a class="button-link secondary" href="{% url 'ssh_credentials' %}">Cancel</a>
</div>
</form>
</section>
{% if credential %}
<section class="panel">
<h2>Delete SSH Key</h2>
{% if credential.hosts.exists or credential.global_configs.exists %}
<p class="muted">
This SSH key is still selected by {{ credential.hosts.count }} host(s) or
{{ credential.global_configs.count }} global config(s). Select another key there before deleting it.
</p>
{% else %}
<p class="muted">Type <strong>{{ credential.name }}</strong> to confirm deletion.</p>
{% endif %}
<form method="post" action="{% url 'delete_ssh_credential' credential.id %}">
{% csrf_token %}
<div class="field">
<label for="confirm_name">Confirm key name</label>
<input id="confirm_name" name="confirm_name" type="text" {% if credential.hosts.exists or credential.global_configs.exists %}disabled{% endif %}>
</div>
<div class="form-actions">
<button type="submit" class="danger" {% if credential.hosts.exists or credential.global_configs.exists %}disabled{% endif %}>Delete SSH key</button>
<a class="button-link secondary" href="{% url 'ssh_credentials' %}">Cancel</a>
</div>
</form>
</section>
{% endif %}
{% endblock %}

View File

@@ -0,0 +1,38 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}Generate SSH Key | pobsync{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Access</div>
<h1>Generate SSH Key</h1>
<div class="page-subtitle">Create a pobsync-managed SSH key pair for one or more backup targets.</div>
</div>
<section class="actions" aria-label="SSH key form actions">
<a class="button-link" href="{% url 'ssh_credentials' %}">Back to SSH keys</a>
</section>
</header>
<section class="panel">
<h2>Create Key Pair</h2>
<form method="post" class="form-grid">
{% csrf_token %}
{{ form.non_field_errors }}
{% for field in form %}
<div class="field">
{{ field.errors }}
<label for="{{ field.id_for_label }}">{{ field.label }}</label>
{{ field }}
{% if field.help_text %}<div class="helptext">{{ field.help_text }}</div>{% endif %}
</div>
{% endfor %}
<div class="form-actions">
<button type="submit">Generate SSH key</button>
<a class="button-link secondary" href="{% url 'ssh_credentials' %}">Cancel</a>
</div>
</form>
</section>
{% endblock %}

View File

@@ -0,0 +1,52 @@
{% extends "pobsync_backend/base.html" %}
{% block title %}SSH Keys | pobsync{% endblock %}
{% block content %}
<header class="page-header">
<div class="page-title">
<div class="page-kicker">Access</div>
<h1>SSH Keys</h1>
<div class="page-subtitle">Manage the key pairs pobsync uses to reach backup targets.</div>
</div>
<section class="actions" aria-label="SSH key actions">
<a class="button-link" href="{% url 'generate_ssh_credential' %}">Generate SSH key</a>
<a class="button-link secondary" href="{% url 'create_ssh_credential' %}">Add existing key</a>
<a class="button-link secondary" href="{% url 'dashboard' %}">Back to dashboard</a>
</section>
</header>
<section class="panel">
<h2>Credentials</h2>
<table>
<thead>
<tr>
<th>Name</th>
<th>Type</th>
<th>Public key</th>
<th>Fingerprint</th>
<th>Known hosts</th>
<th>Hosts</th>
<th>Updated</th>
<th>Actions</th>
</tr>
</thead>
<tbody>
{% for credential in credentials %}
<tr>
<td><a href="{% url 'edit_ssh_credential' credential.id %}">{{ credential.name }}</a></td>
<td>{{ credential.key_type }}</td>
<td>{% if credential.public_key %}<code>{{ credential.public_key|truncatechars:44 }}</code>{% else %}no{% endif %}</td>
<td>{% if credential.fingerprint %}<code>{{ credential.fingerprint }}</code>{% else %}<span class="muted">unknown</span>{% endif %}</td>
<td>{{ credential.known_hosts|yesno:"yes,no" }}</td>
<td>{{ credential.hosts.count }}</td>
<td>{{ credential.updated_at }}</td>
<td><a class="button-link secondary" href="{% url 'edit_ssh_credential' credential.id %}">Edit</a></td>
</tr>
{% empty %}
<tr><td colspan="8" class="muted">No SSH credentials configured yet.</td></tr>
{% endfor %}
</tbody>
</table>
</section>
{% endblock %}

View File

@@ -0,0 +1,112 @@
from __future__ import annotations
from datetime import datetime, timezone
from django.contrib.admin.sites import AdminSite
from django.test import TestCase
from pobsync_backend.admin import BackupRunAdmin, GlobalConfigAdmin, HostConfigAdmin, SnapshotRecordAdmin
from pobsync_backend.models import BackupRun, GlobalConfig, HostConfig, ScheduleConfig, SnapshotRecord
class AdminDisplayTests(TestCase):
def test_admin_hides_old_global_state_fields_and_labels_host_runtime_state(self) -> None:
site = AdminSite()
global_admin = GlobalConfigAdmin(GlobalConfig, site)
host_admin = HostConfigAdmin(HostConfig, site)
global_fieldsets = list(global_admin.fieldsets)
host_fieldsets = list(host_admin.fieldsets)
global_fields = [field for _name, options in global_fieldsets for field in options["fields"]]
fieldset_names = [name for name, _options in [*global_fieldsets, *host_fieldsets]]
self.assertNotIn("pobsync_home", global_fields)
self.assertNotIn("data", global_fields)
self.assertIn("Runtime state", fieldset_names)
self.assertNotIn("Compatibility data", fieldset_names)
self.assertNotIn("Legacy JSON", fieldset_names)
def test_host_admin_links_to_related_snapshots_and_runs(self) -> None:
site = AdminSite()
admin = HostConfigAdmin(HostConfig, site)
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
snapshot = SnapshotRecord.objects.create(
host=host,
kind="scheduled",
dirname="20260519-021500Z__ABCDEFGH",
path="/backups/web-01/scheduled/20260519-021500Z__ABCDEFGH",
)
BackupRun.objects.create(host=host, status=BackupRun.Status.SUCCESS, snapshot=snapshot)
snapshot_link = str(admin.snapshot_count_link(host))
run_link = str(admin.backup_run_count_link(host))
self.assertIn("/admin/pobsync_backend/snapshotrecord/", snapshot_link)
self.assertIn(f"host__id__exact={host.pk}", snapshot_link)
self.assertIn(">1<", snapshot_link)
self.assertIn("/admin/pobsync_backend/backuprun/", run_link)
self.assertIn(f"host__id__exact={host.pk}", run_link)
self.assertIn(">1<", run_link)
def test_host_admin_summarizes_schedule_and_latest_run(self) -> None:
site = AdminSite()
admin = HostConfigAdmin(HostConfig, site)
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
self.assertEqual(admin.schedule_state(host), "none")
ScheduleConfig.objects.create(host=host, cron_expr="15 2 * * *", enabled=True)
host.refresh_from_db()
BackupRun.objects.create(
host=host,
status=BackupRun.Status.SUCCESS,
started_at=datetime(2026, 5, 19, 2, 15, tzinfo=timezone.utc),
)
self.assertEqual(admin.schedule_state(host), "15 2 * * *")
self.assertEqual(admin.latest_run_state(host), "success 2026-05-19 02:15")
def test_snapshot_admin_links_to_base_and_backup_runs(self) -> None:
site = AdminSite()
admin = SnapshotRecordAdmin(SnapshotRecord, site)
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
base = SnapshotRecord.objects.create(
host=host,
kind="scheduled",
dirname="20260518-021500Z__BASESNAP",
path="/backups/web-01/scheduled/20260518-021500Z__BASESNAP",
)
child = SnapshotRecord.objects.create(
host=host,
kind="scheduled",
dirname="20260519-021500Z__CHILDSNP",
path="/backups/web-01/scheduled/20260519-021500Z__CHILDSNP",
base=base,
)
BackupRun.objects.create(host=host, status=BackupRun.Status.SUCCESS, snapshot=child)
base_link = str(admin.base_link(child))
run_link = str(admin.backup_run_count_link(child))
self.assertIn(f"/admin/pobsync_backend/snapshotrecord/{base.pk}/change/", base_link)
self.assertIn(base.dirname, base_link)
self.assertIn("/admin/pobsync_backend/backuprun/", run_link)
self.assertIn(f"snapshot__id__exact={child.pk}", run_link)
self.assertIn(">1<", run_link)
def test_backup_run_admin_links_to_snapshot(self) -> None:
site = AdminSite()
admin = BackupRunAdmin(BackupRun, site)
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
snapshot = SnapshotRecord.objects.create(
host=host,
kind="scheduled",
dirname="20260519-021500Z__ABCDEFGH",
path="/backups/web-01/scheduled/20260519-021500Z__ABCDEFGH",
)
run = BackupRun.objects.create(host=host, status=BackupRun.Status.SUCCESS, snapshot=snapshot)
snapshot_link = str(admin.snapshot_link(run))
self.assertIn(f"/admin/pobsync_backend/snapshotrecord/{snapshot.pk}/change/", snapshot_link)
self.assertIn(snapshot.dirname, snapshot_link)

View File

@@ -0,0 +1,141 @@
from __future__ import annotations
from datetime import datetime, timezone
from django.contrib.auth import get_user_model
from django.test import TestCase
from django.utils import timezone as django_timezone
from pobsync_backend.models import BackupRun, HostConfig, ScheduleConfig, SnapshotRecord
class ApiTests(TestCase):
def setUp(self) -> None:
user_model = get_user_model()
self.staff_user = user_model.objects.create_user(
username="admin",
password="secret",
is_staff=True,
is_superuser=True,
)
def test_api_requires_staff_login(self) -> None:
response = self.client.get("/api/hosts/")
self.assertEqual(response.status_code, 302)
self.assertIn("/admin/login/", response["Location"])
def test_hosts_endpoint_returns_counts_and_schedule(self) -> None:
self.client.force_login(self.staff_user)
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
ScheduleConfig.objects.create(host=host, cron_expr="15 2 * * *", enabled=True, prune=True)
snapshot = self._snapshot(host, "20260519-021500Z__ABCDEFGH")
BackupRun.objects.create(host=host, status=BackupRun.Status.SUCCESS, snapshot=snapshot)
response = self.client.get("/api/hosts/")
self.assertEqual(response.status_code, 200)
payload = response.json()
self.assertTrue(payload["ok"])
self.assertEqual(payload["hosts"][0]["host"], "web-01")
self.assertEqual(payload["hosts"][0]["snapshot_count"], 1)
self.assertEqual(payload["hosts"][0]["run_count"], 1)
self.assertEqual(payload["hosts"][0]["schedule"]["cron_expr"], "15 2 * * *")
self.assertTrue(payload["hosts"][0]["schedule"]["prune"])
def test_snapshots_endpoint_filters_and_returns_base_payload(self) -> None:
self.client.force_login(self.staff_user)
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
other_host = HostConfig.objects.create(host="db-01", address="db-01.example.test")
base = self._snapshot(host, "20260518-021500Z__BASESNAP")
self._snapshot(other_host, "20260519-021500Z__OTHERSNP")
child = self._snapshot(host, "20260519-021500Z__CHILDSNP", base=base)
response = self.client.get("/api/snapshots/", {"host": host.host, "kind": "scheduled"})
self.assertEqual(response.status_code, 200)
snapshots = response.json()["snapshots"]
self.assertEqual([snapshot["dirname"] for snapshot in snapshots], [child.dirname, base.dirname])
self.assertEqual(snapshots[0]["base"]["dirname"], base.dirname)
self.assertTrue(snapshots[0]["base"]["resolved"])
def test_runs_endpoint_filters_by_status_and_limit(self) -> None:
self.client.force_login(self.staff_user)
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
snapshot = self._snapshot(host, "20260519-021500Z__ABCDEFGH")
BackupRun.objects.create(host=host, status=BackupRun.Status.FAILED)
BackupRun.objects.create(host=host, status=BackupRun.Status.SUCCESS, snapshot=snapshot)
response = self.client.get("/api/runs/", {"host": host.host, "status": "success", "limit": "1"})
self.assertEqual(response.status_code, 200)
runs = response.json()["runs"]
self.assertEqual(len(runs), 1)
self.assertEqual(runs[0]["status"], "success")
self.assertEqual(runs[0]["snapshot"]["dirname"], snapshot.dirname)
def test_api_index_lists_endpoints(self) -> None:
self.client.force_login(self.staff_user)
response = self.client.get("/api/")
self.assertEqual(response.status_code, 200)
endpoints = response.json()["endpoints"]
self.assertEqual(endpoints["status"], "http://testserver/api/status/")
self.assertEqual(endpoints["hosts"], "http://testserver/api/hosts/")
self.assertEqual(endpoints["snapshots"], "http://testserver/api/snapshots/")
self.assertEqual(endpoints["runs"], "http://testserver/api/runs/")
def test_status_endpoint_returns_counts_and_latest_activity(self) -> None:
self.client.force_login(self.staff_user)
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
schedule = ScheduleConfig.objects.create(
host=host,
cron_expr="15 2 * * *",
enabled=True,
prune=True,
last_due_key="202605190215",
last_status="success",
last_started_at=django_timezone.now(),
)
snapshot = self._snapshot(host, "20260519-021500Z__ABCDEFGH")
BackupRun.objects.create(
host=host,
status=BackupRun.Status.SUCCESS,
snapshot=snapshot,
started_at=datetime(2026, 5, 19, 2, 15, tzinfo=timezone.utc),
)
response = self.client.get("/api/status/")
self.assertEqual(response.status_code, 200)
payload = response.json()
self.assertTrue(payload["ok"])
self.assertEqual(payload["database"]["vendor"], "sqlite")
self.assertEqual(payload["counts"]["hosts"], 1)
self.assertEqual(payload["counts"]["enabled_hosts"], 1)
self.assertEqual(payload["counts"]["enabled_schedules"], 1)
self.assertEqual(payload["counts"]["snapshots"], 1)
self.assertEqual(payload["counts"]["runs"], 1)
self.assertEqual(payload["latest_run"]["host"], host.host)
self.assertEqual(payload["latest_run"]["snapshot"]["dirname"], snapshot.dirname)
self.assertEqual(payload["latest_schedule"]["host"], host.host)
self.assertEqual(payload["latest_schedule"]["last_due_key"], schedule.last_due_key)
def _snapshot(
self,
host: HostConfig,
dirname: str,
*,
base: SnapshotRecord | None = None,
) -> SnapshotRecord:
started_at = datetime.strptime(dirname.split("__", 1)[0], "%Y%m%d-%H%M%SZ").replace(tzinfo=timezone.utc)
return SnapshotRecord.objects.create(
host=host,
kind=SnapshotRecord.Kind.SCHEDULED,
dirname=dirname,
path=f"/backups/{host.host}/scheduled/{dirname}",
status="success",
started_at=started_at,
base=base,
)

View File

@@ -0,0 +1,240 @@
from __future__ import annotations
from datetime import timedelta
from pathlib import Path
from tempfile import TemporaryDirectory
from unittest.mock import patch
from django.test import TestCase
from django.utils import timezone
from pobsync.util import write_yaml_atomic
from pobsync_backend.backup_runner import queue_backup_run, reconcile_running_runs
from pobsync_backend.management.commands.run_pobsync_worker import Command
from pobsync_backend.models import BackupRun, GlobalConfig, HostConfig, SnapshotRecord
class BackupWorkerTests(TestCase):
def test_queue_backup_run_records_requested_options(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
run = queue_backup_run(
host=host,
dry_run=True,
prune=True,
prune_max_delete=3,
prune_protect_bases=True,
)
self.assertEqual(run.status, BackupRun.Status.QUEUED)
self.assertEqual(run.run_type, BackupRun.RunType.MANUAL)
self.assertEqual(
run.result["requested"],
{
"dry_run": True,
"verbose_output": True,
"prune": True,
"prune_max_delete": 3,
"prune_protect_bases": True,
},
)
def test_queue_backup_run_can_request_verbose_output(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
run = queue_backup_run(host=host, verbose_output=True)
self.assertTrue(run.result["requested"]["verbose_output"])
def test_worker_executes_next_queued_run(self) -> None:
with TemporaryDirectory() as tmp:
backup_root = Path(tmp) / "backups"
GlobalConfig.objects.create(name="default", backup_root=str(backup_root))
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
snapshot_dir = backup_root / host.host / "scheduled" / "20260519-021500Z__ABCDEFGH"
meta_dir = snapshot_dir / "meta"
meta_dir.mkdir(parents=True)
write_yaml_atomic(meta_dir / "meta.yaml", {"status": "success", "started_at": "2026-05-19T02:15:00Z"})
run = queue_backup_run(host=host, verbose_output=True)
with patch("pobsync_backend.backup_runner.run_scheduled") as run_scheduled:
def fake_run_scheduled(**kwargs):
run.refresh_from_db()
self.assertIn("execution", run.result)
self.assertIn("worker_pid", run.result["execution"])
self.assertIn("worker_host", run.result["execution"])
self.assertIn("heartbeat_at", run.result["execution"])
return {
"ok": True,
"dry_run": False,
"host": host.host,
"snapshot": str(snapshot_dir),
"base": None,
"rsync": {"exit_code": 0},
}
run_scheduled.side_effect = fake_run_scheduled
count = Command()._run_once(prefix=Path(tmp) / "home")
run_scheduled.assert_called_once()
self.assertEqual(count, 1)
self.assertEqual(run_scheduled.call_args.kwargs["run_id"], run.id)
self.assertTrue(run_scheduled.call_args.kwargs["verbose_output"])
run.refresh_from_db()
self.assertEqual(run.status, BackupRun.Status.SUCCESS)
self.assertEqual(SnapshotRecord.objects.count(), 1)
self.assertEqual(run.snapshot, SnapshotRecord.objects.get())
def test_worker_refreshes_heartbeat_while_run_is_active(self) -> None:
with TemporaryDirectory() as tmp:
GlobalConfig.objects.create(name="default", backup_root=str(Path(tmp) / "backups"))
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
run = queue_backup_run(host=host)
with patch("pobsync_backend.backup_runner.run_scheduled") as run_scheduled:
def fake_run_scheduled(**kwargs):
run.refresh_from_db()
old_heartbeat = timezone.now() - timedelta(seconds=120)
run.result["execution"]["heartbeat_at"] = old_heartbeat.isoformat()
run.save(update_fields=["result"])
self.assertFalse(kwargs["cancel_check"]())
run.refresh_from_db()
self.assertGreater(
timezone.datetime.fromisoformat(run.result["execution"]["heartbeat_at"]),
old_heartbeat,
)
return {
"ok": True,
"dry_run": False,
"host": host.host,
"snapshot": "",
"base": None,
"rsync": {"exit_code": 0},
}
run_scheduled.side_effect = fake_run_scheduled
Command()._run_once(prefix=Path(tmp) / "home")
def test_worker_reconciles_stale_real_run_after_heartbeat_timeout(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
run = queue_backup_run(host=host)
run.status = BackupRun.Status.RUNNING
run.started_at = timezone.now() - timedelta(seconds=120)
run.result["execution"] = {
"worker_pid": 123,
"worker_host": "backup",
"heartbeat_at": (timezone.now() - timedelta(seconds=90)).isoformat(),
}
run.save(update_fields=["status", "started_at", "result"])
reconciled = reconcile_running_runs(stale_worker_seconds=30)
self.assertEqual(reconciled, 1)
run.refresh_from_db()
self.assertEqual(run.status, BackupRun.Status.FAILED)
self.assertEqual(run.result["failure"]["category"], "worker")
self.assertIn("heartbeat stopped", run.result["failure"]["message"])
def test_worker_records_dry_run_log_path_while_running(self) -> None:
with TemporaryDirectory() as tmp:
GlobalConfig.objects.create(name="default", backup_root=str(Path(tmp) / "backups"))
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
run = queue_backup_run(host=host, dry_run=True)
with patch("pobsync_backend.backup_runner.run_scheduled") as run_scheduled:
def fake_run_scheduled(**kwargs):
run.refresh_from_db()
self.assertEqual(
run.result["execution"]["log"],
f"/tmp/pobsync-dryrun/{host.host}/run-{run.id}/rsync.log",
)
return {
"ok": True,
"dry_run": True,
"host": host.host,
"base": None,
"log": run.result["execution"]["log"],
"rsync": {"exit_code": 0},
}
run_scheduled.side_effect = fake_run_scheduled
count = Command()._run_once(prefix=Path(tmp) / "home")
self.assertEqual(count, 1)
run.refresh_from_db()
self.assertEqual(run.status, BackupRun.Status.SUCCESS)
self.assertEqual(run.result["log"], f"/tmp/pobsync-dryrun/{host.host}/run-{run.id}/rsync.log")
def test_worker_preserves_cancelled_status_from_running_run(self) -> None:
with TemporaryDirectory() as tmp:
GlobalConfig.objects.create(name="default", backup_root=str(Path(tmp) / "backups"))
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
run = queue_backup_run(host=host, dry_run=True)
with patch("pobsync_backend.backup_runner.run_scheduled") as run_scheduled:
def fake_run_scheduled(**kwargs):
BackupRun.objects.filter(id=run.id).update(status=BackupRun.Status.CANCELLED)
self.assertTrue(kwargs["cancel_check"]())
return {
"ok": False,
"dry_run": True,
"cancelled": True,
"host": host.host,
"base": None,
"log": f"/tmp/pobsync-dryrun/{host.host}/run-{run.id}/rsync.log",
"rsync": {"exit_code": 130},
}
run_scheduled.side_effect = fake_run_scheduled
count = Command()._run_once(prefix=Path(tmp) / "home")
self.assertEqual(count, 1)
run.refresh_from_db()
self.assertEqual(run.status, BackupRun.Status.CANCELLED)
self.assertEqual(run.rsync_exit_code, 130)
def test_worker_returns_zero_without_queued_runs(self) -> None:
count = Command()._run_once(prefix=Path("/opt/pobsync"))
self.assertEqual(count, 0)
def test_worker_reconciles_running_dry_run_with_terminal_broken_pipe_log(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
run = queue_backup_run(host=host, dry_run=True)
log_path = Path("/tmp/pobsync-dryrun/web-01/run-test-broken-pipe.log")
log_path.parent.mkdir(parents=True, exist_ok=True)
log_path.write_text(
"rsync error: unexplained error (code 255) at rsync.c(716) [generator=3.4.1]\n"
"rsync: [generator] write error: Broken pipe (32)\n",
encoding="utf-8",
)
run.status = BackupRun.Status.RUNNING
run.started_at = timezone.now()
run.result["execution"] = {"log": str(log_path)}
run.save(update_fields=["status", "started_at", "result"])
reconciled = reconcile_running_runs()
self.assertEqual(reconciled, 1)
run.refresh_from_db()
self.assertEqual(run.status, BackupRun.Status.FAILED)
self.assertEqual(run.rsync_exit_code, 255)
self.assertEqual(run.result["failure"]["category"], "transport")
self.assertIn("Broken pipe", "\n".join(run.result["rsync"]["log_tail"]))
def test_worker_reconciles_stale_running_dry_run_after_timeout(self) -> None:
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
run = queue_backup_run(host=host, dry_run=True)
run.status = BackupRun.Status.RUNNING
run.started_at = timezone.now() - timedelta(seconds=1300)
run.result["timeout_seconds"] = 900
run.save(update_fields=["status", "started_at", "result"])
reconciled = reconcile_running_runs(grace_seconds=300)
self.assertEqual(reconciled, 1)
run.refresh_from_db()
self.assertEqual(run.status, BackupRun.Status.FAILED)
self.assertEqual(run.rsync_exit_code, 124)
self.assertEqual(run.result["failure"]["category"], "timeout")

View File

@@ -1,71 +1,63 @@
from __future__ import annotations from __future__ import annotations
import tempfile
from pathlib import Path
from django.test import TestCase from django.test import TestCase
from pobsync.config.load import load_global_config, load_host_config from pobsync_backend.config_repository import ConfigRepositoryError, global_config_data, host_config_data
from pobsync_backend.config_repository import export_runtime_configs
from pobsync_backend.models import GlobalConfig, HostConfig from pobsync_backend.models import GlobalConfig, HostConfig
class ConfigRepositoryTests(TestCase): class ConfigRepositoryTests(TestCase):
def test_exports_database_configs_to_engine_yaml(self) -> None: def test_builds_runtime_config_from_database_fields(self) -> None:
with tempfile.TemporaryDirectory() as tmp: GlobalConfig.objects.create(
prefix = Path(tmp) name="default",
GlobalConfig.objects.create( backup_root="/backups",
name="default", ssh_user="backup",
backup_root="/backups", ssh_port=2222,
pobsync_home=str(prefix), rsync_args=["--archive"],
ssh_user="backup", excludes_default=["/proc/***"],
ssh_port=2222, retention_daily=7,
rsync_args=["--archive"], retention_weekly=4,
excludes_default=["/proc/***"], retention_monthly=3,
retention_daily=7, retention_yearly=1,
retention_weekly=4, )
retention_monthly=3, HostConfig.objects.create(
retention_yearly=1, host="web-01",
data={ address="web-01.example.test",
"backup_root": "/ignored", ssh_user="root",
"pobsync_home": "/ignored", includes=[],
"ssh": {"user": "ignored", "port": 22, "options": []}, excludes_add=["/tmp/***"],
"unknown": "must-not-leak", retention_daily=7,
"retention_defaults": {"daily": 99, "weekly": 99, "monthly": 99, "yearly": 99}, retention_weekly=4,
}, retention_monthly=3,
) retention_yearly=1,
HostConfig.objects.create( config={
host="web-01", "host": "ignored",
address="web-01.example.test", "address": "ignored",
ssh_user="root", "retention": {"daily": 99, "weekly": 99, "monthly": 99, "yearly": 99},
includes=[], "excludes_add": ["/ignored/***"],
excludes_add=["/tmp/***"], "unknown": "must-not-leak",
retention_daily=7, },
retention_weekly=4, )
retention_monthly=3,
retention_yearly=1,
config={
"host": "ignored",
"address": "ignored",
"retention": {"daily": 99, "weekly": 99, "monthly": 99, "yearly": 99},
"excludes_add": ["/ignored/***"],
"unknown": "must-not-leak",
},
)
written = export_runtime_configs(prefix=prefix, host="web-01") global_cfg = global_config_data()
host_cfg = host_config_data("web-01")
self.assertEqual(len(written), 2) self.assertEqual(global_cfg["backup_root"], "/backups")
global_cfg = load_global_config(prefix / "config" / "global.yaml") self.assertEqual(global_cfg["ssh"]["user"], "backup")
host_cfg = load_host_config(prefix / "config" / "hosts" / "web-01.yaml") self.assertEqual(global_cfg["ssh"]["port"], 2222)
self.assertEqual(global_cfg["backup_root"], "/backups") self.assertEqual(global_cfg["retention_defaults"]["daily"], 7)
self.assertEqual(global_cfg["pobsync_home"], str(prefix)) self.assertEqual(host_cfg["host"], "web-01")
self.assertEqual(global_cfg["ssh"]["user"], "backup") self.assertEqual(host_cfg["address"], "web-01.example.test")
self.assertEqual(global_cfg["ssh"]["port"], 2222) self.assertEqual(host_cfg["retention"]["daily"], 7)
self.assertEqual(global_cfg["retention_defaults"]["daily"], 7) self.assertEqual(host_cfg["excludes_add"], ["/tmp/***"])
self.assertEqual(host_cfg["host"], "web-01") self.assertNotIn("unknown", global_cfg)
self.assertEqual(host_cfg["address"], "web-01.example.test") self.assertNotIn("unknown", host_cfg)
self.assertEqual(host_cfg["retention"]["daily"], 7)
self.assertEqual(host_cfg["excludes_add"], ["/tmp/***"]) def test_missing_config_errors_use_operator_labels(self) -> None:
self.assertNotIn("unknown", global_cfg) with self.assertRaisesMessage(ConfigRepositoryError, "Missing global config 'default'"):
self.assertNotIn("unknown", host_cfg) global_config_data()
GlobalConfig.objects.create(name="default", backup_root="/backups")
with self.assertRaisesMessage(ConfigRepositoryError, "Missing enabled host 'web-01'"):
host_config_data("web-01")

View File

@@ -16,7 +16,6 @@ class ConfigureCommandsTests(TestCase):
call_command( call_command(
"configure_pobsync_global", "configure_pobsync_global",
backup_root="/backups", backup_root="/backups",
pobsync_home="/opt/pobsync",
retention="daily=3,weekly=2,monthly=1,yearly=0", retention="daily=3,weekly=2,monthly=1,yearly=0",
stdout=out, stdout=out,
) )
@@ -24,7 +23,7 @@ class ConfigureCommandsTests(TestCase):
config = GlobalConfig.objects.get(name="default") config = GlobalConfig.objects.get(name="default")
self.assertEqual(config.backup_root, "/backups") self.assertEqual(config.backup_root, "/backups")
self.assertEqual(config.retention_daily, 3) self.assertEqual(config.retention_daily, 3)
self.assertIn("Created GlobalConfig", out.getvalue()) self.assertIn("Created global config", out.getvalue())
def test_configure_host_uses_global_retention_defaults(self) -> None: def test_configure_host_uses_global_retention_defaults(self) -> None:
GlobalConfig.objects.create( GlobalConfig.objects.create(
@@ -62,7 +61,7 @@ class ConfigureCommandsTests(TestCase):
call_command( call_command(
"configure_pobsync_schedule", "configure_pobsync_schedule",
host.host, host.host,
cron="15 2 * * *", schedule_expression="15 2 * * *",
prune=True, prune=True,
stdout=out, stdout=out,
) )

View File

@@ -9,6 +9,14 @@ from pobsync.cli import main
class ConsoleEntrypointTests(SimpleTestCase): class ConsoleEntrypointTests(SimpleTestCase):
def test_version_prints_package_version(self) -> None:
stdout = StringIO()
with patch("sys.stdout", stdout):
exit_code = main(["--version"])
self.assertEqual(exit_code, 0)
self.assertEqual(stdout.getvalue().strip(), "pobsync 1.1.0")
def test_maps_backup_alias_to_django_command(self) -> None: def test_maps_backup_alias_to_django_command(self) -> None:
with patch("pobsync.cli.execute_from_command_line") as execute: with patch("pobsync.cli.execute_from_command_line") as execute:
exit_code = main(["backup", "web-01", "--dry-run"]) exit_code = main(["backup", "web-01", "--dry-run"])
@@ -31,18 +39,25 @@ class ConsoleEntrypointTests(SimpleTestCase):
self.assertEqual(exit_code, 0) self.assertEqual(exit_code, 0)
execute.assert_called_once_with(["pobsync", "check"]) execute.assert_called_once_with(["pobsync", "check"])
def test_maps_schedule_alias_to_django_command(self) -> None:
with patch("pobsync.cli.execute_from_command_line") as execute:
exit_code = main(["schedule", "web-01", "--cron", "15 2 * * *"])
self.assertEqual(exit_code, 0)
execute.assert_called_once_with(
["pobsync", "configure_pobsync_schedule", "web-01", "--cron", "15 2 * * *"]
)
def test_maps_discover_snapshots_alias_to_django_command(self) -> None: def test_maps_discover_snapshots_alias_to_django_command(self) -> None:
with patch("pobsync.cli.execute_from_command_line") as execute: with patch("pobsync.cli.execute_from_command_line") as execute:
exit_code = main(["discover-snapshots", "--host", "web-01"]) exit_code = main(["discover-snapshots", "--host", "web-01"])
self.assertEqual(exit_code, 0) self.assertEqual(exit_code, 0)
execute.assert_called_once_with(["pobsync", "discover_pobsync_snapshots", "--host", "web-01"]) execute.assert_called_once_with(["pobsync", "discover_pobsync_snapshots", "--host", "web-01"])
def test_maps_worker_alias_to_django_command(self) -> None:
with patch("pobsync.cli.execute_from_command_line") as execute:
exit_code = main(["worker", "--once"])
self.assertEqual(exit_code, 0)
execute.assert_called_once_with(["pobsync", "run_pobsync_worker", "--once"])
def test_configuration_aliases_are_not_public_commands(self) -> None:
stderr = StringIO()
with patch("sys.stderr", stderr):
exit_code = main(["schedule", "web-01", "--cron", "15 2 * * *"])
self.assertEqual(exit_code, 2)
self.assertIn("Unknown pobsync command", stderr.getvalue())
self.assertIn("pobsync django <management-command>", stderr.getvalue())

View File

@@ -1,9 +1,13 @@
from __future__ import annotations from __future__ import annotations
from django.test import TestCase import stat
from pathlib import Path
from tempfile import TemporaryDirectory
from django.test import TestCase, override_settings
from pobsync_backend.config_source import DjangoConfigSource from pobsync_backend.config_source import DjangoConfigSource
from pobsync_backend.models import GlobalConfig, HostConfig from pobsync_backend.models import GlobalConfig, HostConfig, SshCredential
class DjangoConfigSourceTests(TestCase): class DjangoConfigSourceTests(TestCase):
@@ -11,7 +15,6 @@ class DjangoConfigSourceTests(TestCase):
GlobalConfig.objects.create( GlobalConfig.objects.create(
name="default", name="default",
backup_root="/backups", backup_root="/backups",
pobsync_home="/opt/pobsync",
rsync_args=["--archive"], rsync_args=["--archive"],
rsync_extra_args=["--numeric-ids"], rsync_extra_args=["--numeric-ids"],
excludes_default=["/proc/***"], excludes_default=["/proc/***"],
@@ -19,21 +22,6 @@ class DjangoConfigSourceTests(TestCase):
retention_weekly=4, retention_weekly=4,
retention_monthly=3, retention_monthly=3,
retention_yearly=1, retention_yearly=1,
data={
"backup_root": "/ignored",
"pobsync_home": "/ignored",
"ssh": {"user": "root", "port": 22, "options": []},
"rsync": {
"binary": "rsync",
"args": ["--archive"],
"timeout_seconds": 0,
"bwlimit_kbps": 0,
"extra_args": ["--numeric-ids"],
},
"defaults": {"source_root": "/", "destination_subdir": ""},
"excludes_default": ["/proc/***"],
"retention_defaults": {"daily": 7, "weekly": 4, "monthly": 3, "yearly": 1},
},
) )
HostConfig.objects.create( HostConfig.objects.create(
host="web-01", host="web-01",
@@ -58,3 +46,95 @@ class DjangoConfigSourceTests(TestCase):
self.assertEqual(cfg["address"], "web-01.example.test") self.assertEqual(cfg["address"], "web-01.example.test")
self.assertEqual(cfg["excludes_effective"], ["/proc/***", "/tmp/***"]) self.assertEqual(cfg["excludes_effective"], ["/proc/***", "/tmp/***"])
self.assertEqual(cfg["rsync"]["args_effective"], ["--archive", "--numeric-ids", "--delete"]) self.assertEqual(cfg["rsync"]["args_effective"], ["--archive", "--numeric-ids", "--delete"])
def test_materializes_global_ssh_credential_for_runtime_config(self) -> None:
credential = SshCredential.objects.create(
name="backup-key",
private_key="PRIVATE KEY",
known_hosts="web-01.example.test ssh-ed25519 AAAATEST",
)
GlobalConfig.objects.create(
name="default",
backup_root="/backups",
default_ssh_credential=credential,
ssh_options=["-oBatchMode=yes"],
)
HostConfig.objects.create(host="web-01", address="web-01.example.test")
with TemporaryDirectory() as tmp, override_settings(POBSYNC_HOME=str(Path(tmp) / "home")):
cfg = DjangoConfigSource().effective_config_for_host("web-01")
identity_file = Path(tmp) / "home" / "state" / "ssh-credentials" / str(credential.pk) / "identity"
known_hosts = identity_file.parent / "known_hosts"
self.assertEqual(identity_file.read_text(encoding="utf-8"), "PRIVATE KEY\n")
self.assertEqual(known_hosts.read_text(encoding="utf-8"), "web-01.example.test ssh-ed25519 AAAATEST\n")
self.assertEqual(stat.S_IMODE(identity_file.stat().st_mode), 0o600)
self.assertIn("-oBatchMode=yes", cfg["ssh"]["options"])
self.assertIn(f"-oIdentityFile={identity_file}", cfg["ssh"]["options"])
self.assertIn(f"-oUserKnownHostsFile={known_hosts}", cfg["ssh"]["options"])
self.assertNotIn("-oStrictHostKeyChecking=accept-new", cfg["ssh"]["options"])
self.assertEqual(cfg["ssh_credential"]["storage"], "database")
def test_host_ssh_credential_overrides_global_credential(self) -> None:
global_credential = SshCredential.objects.create(name="global-key", private_key="GLOBAL")
host_credential = SshCredential.objects.create(name="host-key", private_key="HOST")
GlobalConfig.objects.create(
name="default",
backup_root="/backups",
default_ssh_credential=global_credential,
)
HostConfig.objects.create(
host="web-01",
address="web-01.example.test",
ssh_credential=host_credential,
)
with TemporaryDirectory() as tmp, override_settings(POBSYNC_HOME=str(Path(tmp) / "home")):
cfg = DjangoConfigSource().effective_config_for_host("web-01")
host_identity_file = Path(tmp) / "home" / "state" / "ssh-credentials" / str(host_credential.pk) / "identity"
global_identity_file = Path(tmp) / "home" / "state" / "ssh-credentials" / str(global_credential.pk) / "identity"
self.assertEqual(host_identity_file.read_text(encoding="utf-8"), "HOST\n")
self.assertFalse(global_identity_file.exists())
self.assertIn(f"-oIdentityFile={host_identity_file}", cfg["ssh"]["options"])
def test_filesystem_ssh_credential_uses_existing_key_path(self) -> None:
with TemporaryDirectory() as tmp:
identity_file = Path(tmp) / "identity"
identity_file.write_text("PRIVATE KEY\n", encoding="utf-8")
identity_file.chmod(0o600)
credential = SshCredential.objects.create(name="backup-key", key_path=str(identity_file), public_key="PUBLIC")
GlobalConfig.objects.create(
name="default",
backup_root="/backups",
default_ssh_credential=credential,
)
HostConfig.objects.create(host="web-01", address="web-01.example.test")
cfg = DjangoConfigSource().effective_config_for_host("web-01")
self.assertIn(f"-oIdentityFile={identity_file}", cfg["ssh"]["options"])
self.assertEqual(cfg["ssh_credential"]["storage"], "filesystem")
def test_missing_known_hosts_uses_service_accept_new_file(self) -> None:
with TemporaryDirectory() as tmp:
identity_file = Path(tmp) / "identity"
identity_file.write_text("PRIVATE KEY\n", encoding="utf-8")
identity_file.chmod(0o600)
credential = SshCredential.objects.create(name="backup-key", key_path=str(identity_file), public_key="PUBLIC")
GlobalConfig.objects.create(
name="default",
backup_root="/backups",
default_ssh_credential=credential,
)
HostConfig.objects.create(host="web-01", address="web-01.example.test")
with override_settings(POBSYNC_HOME=str(Path(tmp) / "home")):
cfg = DjangoConfigSource().effective_config_for_host("web-01")
service_known_hosts = Path(tmp) / "home" / "state" / "known_hosts"
self.assertTrue(service_known_hosts.exists())
self.assertIn(f"-oUserKnownHostsFile={service_known_hosts}", cfg["ssh"]["options"])
self.assertIn("-oStrictHostKeyChecking=accept-new", cfg["ssh"]["options"])

View File

@@ -7,6 +7,7 @@ from tempfile import TemporaryDirectory
from django.test import SimpleTestCase from django.test import SimpleTestCase
from pobsync.commands.retention_plan import run_retention_plan from pobsync.commands.retention_plan import run_retention_plan
from pobsync.errors import ConfigError
from pobsync.util import write_yaml_atomic from pobsync.util import write_yaml_atomic
@@ -24,6 +25,15 @@ class FakeConfigSource:
class RetentionConfigSourceTests(SimpleTestCase): class RetentionConfigSourceTests(SimpleTestCase):
def test_retention_plan_requires_explicit_config_source(self) -> None:
with self.assertRaisesMessage(ConfigError, "A Django config source is required."):
run_retention_plan(
prefix=Path("/missing-prefix"),
host="web-01",
kind="scheduled",
protect_bases=False,
)
def test_retention_plan_uses_injected_config_source(self) -> None: def test_retention_plan_uses_injected_config_source(self) -> None:
with TemporaryDirectory() as tmp: with TemporaryDirectory() as tmp:
root = Path(tmp) / "backups" root = Path(tmp) / "backups"

View File

@@ -9,6 +9,7 @@ from django.core.management import call_command
from django.core.management.base import CommandError from django.core.management.base import CommandError
from django.test import TestCase from django.test import TestCase
from pobsync.errors import ConfigError
from pobsync.util import write_yaml_atomic from pobsync.util import write_yaml_atomic
from pobsync_backend.models import BackupRun, GlobalConfig, HostConfig, SnapshotRecord from pobsync_backend.models import BackupRun, GlobalConfig, HostConfig, SnapshotRecord
@@ -31,7 +32,7 @@ class RunBackupRecordsSnapshotTests(TestCase):
}, },
) )
with patch("pobsync_backend.management.commands.run_pobsync_backup.run_scheduled") as run_scheduled: with patch("pobsync_backend.backup_runner.run_scheduled") as run_scheduled:
run_scheduled.return_value = { run_scheduled.return_value = {
"ok": True, "ok": True,
"dry_run": False, "dry_run": False,
@@ -43,12 +44,109 @@ class RunBackupRecordsSnapshotTests(TestCase):
call_command("run_pobsync_backup", host.host, prefix=str(Path(tmp) / "home"), stdout=StringIO()) call_command("run_pobsync_backup", host.host, prefix=str(Path(tmp) / "home"), stdout=StringIO())
self.assertEqual(BackupRun.objects.count(), 1) self.assertEqual(BackupRun.objects.count(), 1)
run = BackupRun.objects.get()
self.assertEqual(SnapshotRecord.objects.count(), 1) self.assertEqual(SnapshotRecord.objects.count(), 1)
record = SnapshotRecord.objects.get() record = SnapshotRecord.objects.get()
self.assertEqual(run.snapshot, record)
self.assertEqual(record.host, host) self.assertEqual(record.host, host)
self.assertEqual(record.kind, "scheduled") self.assertEqual(record.kind, "scheduled")
self.assertEqual(record.status, "success") self.assertEqual(record.status, "success")
def test_prune_uses_sql_retention_after_snapshot_record_is_created(self) -> None:
with TemporaryDirectory() as tmp:
backup_root = Path(tmp) / "backups"
GlobalConfig.objects.create(name="default", backup_root=str(backup_root))
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
snapshot_dir = backup_root / host.host / "scheduled" / "20260519-021500Z__ABCDEFGH"
meta_dir = snapshot_dir / "meta"
meta_dir.mkdir(parents=True)
write_yaml_atomic(meta_dir / "meta.yaml", {"status": "success", "started_at": "2026-05-19T02:15:00Z"})
with (
patch("pobsync_backend.backup_runner.run_scheduled") as run_scheduled,
patch(
"pobsync_backend.backup_runner.run_sql_retention_apply"
) as retention_apply,
):
run_scheduled.return_value = {
"ok": True,
"dry_run": False,
"host": host.host,
"snapshot": str(snapshot_dir),
"base": None,
"rsync": {"exit_code": 0},
}
retention_apply.return_value = {"ok": True, "source": "sql", "deleted": []}
call_command(
"run_pobsync_backup",
host.host,
prefix=str(Path(tmp) / "home"),
prune=True,
prune_max_delete=3,
prune_protect_bases=True,
stdout=StringIO(),
)
run_scheduled.assert_called_once()
self.assertFalse(run_scheduled.call_args.kwargs["prune"])
retention_apply.assert_called_once_with(
prefix=Path(tmp) / "home",
host=host.host,
kind="scheduled",
protect_bases=True,
yes=True,
max_delete=3,
action=BackupRun.RunType.SCHEDULED,
acquire_lock=False,
)
run = BackupRun.objects.get()
self.assertEqual(run.status, BackupRun.Status.SUCCESS)
self.assertEqual(run.result["prune"], {"ok": True, "source": "sql", "deleted": []})
def test_prune_failure_marks_backup_run_as_warning(self) -> None:
with TemporaryDirectory() as tmp:
backup_root = Path(tmp) / "backups"
GlobalConfig.objects.create(name="default", backup_root=str(backup_root))
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
snapshot_dir = backup_root / host.host / "scheduled" / "20260519-021500Z__ABCDEFGH"
meta_dir = snapshot_dir / "meta"
meta_dir.mkdir(parents=True)
write_yaml_atomic(meta_dir / "meta.yaml", {"status": "success", "started_at": "2026-05-19T02:15:00Z"})
with (
patch("pobsync_backend.backup_runner.run_scheduled") as run_scheduled,
patch(
"pobsync_backend.backup_runner.run_sql_retention_apply"
) as retention_apply,
):
run_scheduled.return_value = {
"ok": True,
"dry_run": False,
"host": host.host,
"snapshot": str(snapshot_dir),
"base": None,
"rsync": {"exit_code": 0},
}
retention_apply.side_effect = ConfigError("Deletion blocked by --max-delete=0")
output = StringIO()
call_command(
"run_pobsync_backup",
host.host,
prefix=str(Path(tmp) / "home"),
prune=True,
prune_max_delete=0,
stdout=output,
)
run = BackupRun.objects.get()
self.assertEqual(run.status, BackupRun.Status.WARNING)
self.assertIsNotNone(run.snapshot)
self.assertIn("completed with warnings", output.getvalue())
self.assertEqual(run.result["prune"]["ok"], False)
self.assertEqual(run.result["prune"]["type"], "ConfigError")
self.assertEqual(run.result["prune"]["error"], "Deletion blocked by --max-delete=0")
def test_failed_backup_upserts_incomplete_snapshot_record(self) -> None: def test_failed_backup_upserts_incomplete_snapshot_record(self) -> None:
with TemporaryDirectory() as tmp: with TemporaryDirectory() as tmp:
backup_root = Path(tmp) / "backups" backup_root = Path(tmp) / "backups"
@@ -59,7 +157,7 @@ class RunBackupRecordsSnapshotTests(TestCase):
meta_dir.mkdir(parents=True) meta_dir.mkdir(parents=True)
write_yaml_atomic(meta_dir / "meta.yaml", {"status": "failed", "started_at": "2026-05-19T02:15:00Z"}) write_yaml_atomic(meta_dir / "meta.yaml", {"status": "failed", "started_at": "2026-05-19T02:15:00Z"})
with patch("pobsync_backend.management.commands.run_pobsync_backup.run_scheduled") as run_scheduled: with patch("pobsync_backend.backup_runner.run_scheduled") as run_scheduled:
run_scheduled.return_value = { run_scheduled.return_value = {
"ok": False, "ok": False,
"dry_run": False, "dry_run": False,
@@ -74,6 +172,7 @@ class RunBackupRecordsSnapshotTests(TestCase):
run = BackupRun.objects.get() run = BackupRun.objects.get()
self.assertEqual(run.status, BackupRun.Status.FAILED) self.assertEqual(run.status, BackupRun.Status.FAILED)
record = SnapshotRecord.objects.get() record = SnapshotRecord.objects.get()
self.assertEqual(run.snapshot, record)
self.assertEqual(record.kind, "incomplete") self.assertEqual(record.kind, "incomplete")
self.assertEqual(record.status, "failed") self.assertEqual(record.status, "failed")
@@ -82,7 +181,7 @@ class RunBackupRecordsSnapshotTests(TestCase):
GlobalConfig.objects.create(name="default", backup_root=str(Path(tmp) / "backups")) GlobalConfig.objects.create(name="default", backup_root=str(Path(tmp) / "backups"))
host = HostConfig.objects.create(host="web-01", address="web-01.example.test") host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
with patch("pobsync_backend.management.commands.run_pobsync_backup.run_scheduled") as run_scheduled: with patch("pobsync_backend.backup_runner.run_scheduled") as run_scheduled:
run_scheduled.return_value = { run_scheduled.return_value = {
"ok": True, "ok": True,
"dry_run": True, "dry_run": True,
@@ -99,4 +198,30 @@ class RunBackupRecordsSnapshotTests(TestCase):
) )
self.assertEqual(BackupRun.objects.count(), 1) self.assertEqual(BackupRun.objects.count(), 1)
self.assertIsNone(BackupRun.objects.get().snapshot)
self.assertEqual(SnapshotRecord.objects.count(), 0) self.assertEqual(SnapshotRecord.objects.count(), 0)
def test_manual_flag_records_manual_run_type(self) -> None:
with TemporaryDirectory() as tmp:
GlobalConfig.objects.create(name="default", backup_root=str(Path(tmp) / "backups"))
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
with patch("pobsync_backend.backup_runner.run_scheduled") as run_scheduled:
run_scheduled.return_value = {
"ok": True,
"dry_run": True,
"host": host.host,
"base": None,
"rsync": {"exit_code": 0},
}
call_command(
"run_pobsync_backup",
host.host,
prefix=str(Path(tmp) / "home"),
dry_run=True,
manual=True,
stdout=StringIO(),
)
run = BackupRun.objects.get()
self.assertEqual(run.run_type, BackupRun.RunType.MANUAL)

View File

@@ -7,6 +7,7 @@ from unittest.mock import patch
from django.test import SimpleTestCase from django.test import SimpleTestCase
from pobsync.commands.run_scheduled import run_scheduled from pobsync.commands.run_scheduled import run_scheduled
from pobsync.errors import ConfigError
from pobsync.rsync import RsyncResult from pobsync.rsync import RsyncResult
@@ -34,6 +35,10 @@ class FakeConfigSource:
class RunScheduledConfigSourceTests(SimpleTestCase): class RunScheduledConfigSourceTests(SimpleTestCase):
def test_requires_explicit_config_source(self) -> None:
with self.assertRaisesMessage(ConfigError, "A Django config source is required."):
run_scheduled(prefix=Path("/missing-prefix"), host="web-01", dry_run=True)
def test_dry_run_uses_injected_config_source(self) -> None: def test_dry_run_uses_injected_config_source(self) -> None:
with patch("pobsync.commands.run_scheduled.run_rsync") as run_rsync: with patch("pobsync.commands.run_scheduled.run_rsync") as run_rsync:
run_rsync.return_value = RsyncResult(exit_code=0, command=["rsync", "--archive"]) run_rsync.return_value = RsyncResult(exit_code=0, command=["rsync", "--archive"])
@@ -49,6 +54,228 @@ class RunScheduledConfigSourceTests(SimpleTestCase):
self.assertEqual(result["host"], "web-01") self.assertEqual(result["host"], "web-01")
run_rsync.assert_called_once() run_rsync.assert_called_once()
def test_failed_dry_run_includes_log_tail(self) -> None:
def fake_run_rsync(command, log_path, timeout_seconds, cancel_check=None):
log_path.parent.mkdir(parents=True, exist_ok=True)
log_path.write_text("Permission denied (publickey).\nrsync error\n", encoding="utf-8")
return RsyncResult(exit_code=12, command=command)
with patch("pobsync.commands.run_scheduled.run_rsync", side_effect=fake_run_rsync):
result = run_scheduled(
prefix=Path("/missing-prefix"),
host="web-01",
dry_run=True,
config_source=FakeConfigSource(),
)
self.assertFalse(result["ok"])
self.assertEqual(result["rsync"]["exit_code"], 12)
self.assertEqual(result["rsync"]["log_tail"], ["Permission denied (publickey).", "rsync error"])
self.assertEqual(result["failure"]["category"], "permissions")
def test_failed_dry_run_classifies_broken_pipe(self) -> None:
def fake_run_rsync(command, log_path, timeout_seconds, cancel_check=None):
log_path.parent.mkdir(parents=True, exist_ok=True)
log_path.write_text(
"rsync error: unexplained error (code 255) at rsync.c(716) [generator=3.4.1]\n"
"rsync: [generator] write error: Broken pipe (32)\n",
encoding="utf-8",
)
return RsyncResult(exit_code=255, command=command)
with patch("pobsync.commands.run_scheduled.run_rsync", side_effect=fake_run_rsync):
result = run_scheduled(
prefix=Path("/missing-prefix"),
host="web-01",
dry_run=True,
config_source=FakeConfigSource(),
)
self.assertFalse(result["ok"])
self.assertEqual(result["rsync"]["exit_code"], 255)
self.assertEqual(result["failure"]["category"], "transport")
self.assertIn("broken pipe", result["failure"]["hint"].lower())
def test_dry_run_clears_previous_log_before_running(self) -> None:
def fake_run_rsync(command, log_path, timeout_seconds, cancel_check=None):
self.assertFalse(log_path.exists())
log_path.parent.mkdir(parents=True, exist_ok=True)
log_path.write_text("current run only\n", encoding="utf-8")
return RsyncResult(exit_code=0, command=command)
old_log = Path("/tmp/pobsync-dryrun/web-01/adhoc/rsync.log")
old_log.parent.mkdir(parents=True, exist_ok=True)
old_log.write_text("old failure\n", encoding="utf-8")
with patch("pobsync.commands.run_scheduled.run_rsync", side_effect=fake_run_rsync):
result = run_scheduled(
prefix=Path("/missing-prefix"),
host="web-01",
dry_run=True,
config_source=FakeConfigSource(),
)
self.assertTrue(result["ok"])
self.assertEqual(result["rsync"]["log_tail"], ["current run only"])
def test_dry_run_uses_run_specific_log_path_and_default_timeout(self) -> None:
def fake_run_rsync(command, log_path, timeout_seconds, cancel_check=None):
self.assertEqual(log_path, Path("/tmp/pobsync-dryrun/web-01/run-42/rsync.log"))
self.assertEqual(timeout_seconds, 900)
self.assertIn("--itemize-changes", command)
self.assertIn("--info=flist2,progress2,stats2", command)
self.assertIn("--stats", command)
log_path.parent.mkdir(parents=True, exist_ok=True)
log_path.write_text(
"Number of files: 42\n"
"Number of regular files transferred: 3\n"
"Total file size: 1,000 bytes\n"
"Literal data: 100 bytes\n"
"Matched data: 900 bytes\n",
encoding="utf-8",
)
return RsyncResult(exit_code=0, command=command)
with patch("pobsync.commands.run_scheduled.run_rsync", side_effect=fake_run_rsync):
result = run_scheduled(
prefix=Path("/missing-prefix"),
host="web-01",
dry_run=True,
config_source=FakeConfigSource(),
run_id=42,
)
self.assertTrue(result["ok"])
self.assertEqual(result["log"], "/tmp/pobsync-dryrun/web-01/run-42/rsync.log")
self.assertEqual(result["timeout_seconds"], 900)
self.assertEqual(result["stats"]["rsync"]["files_total"], 42)
self.assertEqual(result["stats"]["rsync"]["link_dest_estimated_savings_ratio"], 0.9)
def test_dry_run_does_not_duplicate_custom_output_args(self) -> None:
config_source = FakeConfigSource()
def effective_config_for_host(host: str) -> dict:
config = FakeConfigSource.effective_config_for_host(config_source, host)
config["rsync"]["args_effective"] = ["--archive", "--itemize-changes", "--info=name1,stats2"]
return config
config_source.effective_config_for_host = effective_config_for_host
with patch("pobsync.commands.run_scheduled.run_rsync") as run_rsync:
run_rsync.return_value = RsyncResult(exit_code=0, command=["rsync"])
run_scheduled(
prefix=Path("/missing-prefix"),
host="web-01",
dry_run=True,
config_source=config_source,
)
command = run_rsync.call_args.args[0]
self.assertEqual(command.count("--itemize-changes"), 1)
self.assertNotIn("--info=flist2,progress2,stats2", command)
self.assertIn("--info=name1,stats2", command)
def test_real_run_can_request_verbose_output_args(self) -> None:
with TemporaryDirectory() as tmp:
prefix = Path(tmp) / "home"
with patch("pobsync.commands.run_scheduled.run_rsync") as run_rsync:
run_rsync.return_value = RsyncResult(exit_code=0, command=["rsync", "--archive"])
result = run_scheduled(
prefix=prefix,
host="web-01",
dry_run=False,
verbose_output=True,
config_source=FakeConfigSource(backup_root=str(Path(tmp) / "backups")),
)
command = run_rsync.call_args.args[0]
self.assertTrue(result["ok"])
self.assertIn("--stats", command)
self.assertIn("--itemize-changes", command)
self.assertIn("--info=flist2,progress2,stats2", command)
self.assertTrue(result["verbose_output"])
def test_real_run_keeps_default_output_quiet(self) -> None:
with TemporaryDirectory() as tmp:
prefix = Path(tmp) / "home"
with patch("pobsync.commands.run_scheduled.run_rsync") as run_rsync:
run_rsync.return_value = RsyncResult(exit_code=0, command=["rsync", "--archive"])
result = run_scheduled(
prefix=prefix,
host="web-01",
dry_run=False,
config_source=FakeConfigSource(backup_root=str(Path(tmp) / "backups")),
)
command = run_rsync.call_args.args[0]
self.assertTrue(result["ok"])
self.assertIn("--stats", command)
self.assertNotIn("--itemize-changes", command)
self.assertNotIn("--info=flist2,progress2,stats2", command)
self.assertFalse(result["verbose_output"])
def test_successful_real_run_records_stats_in_result_and_metadata(self) -> None:
def fake_run_rsync(command, log_path, timeout_seconds, cancel_check=None):
log_path.parent.mkdir(parents=True, exist_ok=True)
log_path.write_text(
"Number of files: 10\n"
"Number of regular files transferred: 2\n"
"Total file size: 2,000 bytes\n"
"Total transferred file size: 500 bytes\n"
"Literal data: 500 bytes\n"
"Matched data: 1,500 bytes\n",
encoding="utf-8",
)
data_dir = log_path.parent.parent / "data"
data_dir.mkdir(parents=True, exist_ok=True)
(data_dir / "payload.txt").write_text("payload", encoding="utf-8")
return RsyncResult(exit_code=0, command=command)
with TemporaryDirectory() as tmp:
backup_root = Path(tmp) / "backups"
with patch("pobsync.commands.run_scheduled.run_rsync", side_effect=fake_run_rsync):
result = run_scheduled(
prefix=Path(tmp) / "home",
host="web-01",
dry_run=False,
config_source=FakeConfigSource(backup_root=str(backup_root)),
)
meta_path = Path(result["snapshot"]) / "meta" / "meta.yaml"
meta_text = meta_path.read_text(encoding="utf-8")
self.assertTrue(result["ok"])
self.assertEqual(result["log"], str(Path(result["snapshot"]) / "meta" / "rsync.log"))
self.assertEqual(result["stats"]["rsync"]["files_total"], 10)
self.assertEqual(result["stats"]["rsync"]["files_transferred"], 2)
self.assertEqual(result["stats"]["rsync"]["link_dest_estimated_savings_bytes"], 1500)
self.assertIn("snapshot", result["stats"]["storage"])
self.assertIn("capacity", result["stats"]["storage"])
self.assertIn("stats:", meta_text)
self.assertIn("files_total: 10", meta_text)
def test_dry_run_reports_cancelled_rsync(self) -> None:
def fake_run_rsync(command, log_path, timeout_seconds, cancel_check=None):
self.assertTrue(cancel_check())
log_path.parent.mkdir(parents=True, exist_ok=True)
log_path.write_text("[pobsync] rsync cancelled\n", encoding="utf-8")
return RsyncResult(exit_code=130, command=command, cancelled=True)
with patch("pobsync.commands.run_scheduled.run_rsync", side_effect=fake_run_rsync):
result = run_scheduled(
prefix=Path("/missing-prefix"),
host="web-01",
dry_run=True,
config_source=FakeConfigSource(),
cancel_check=lambda: True,
)
self.assertFalse(result["ok"])
self.assertTrue(result["cancelled"])
self.assertEqual(result["rsync"]["exit_code"], 130)
def test_successful_real_run_applies_prune_when_requested(self) -> None: def test_successful_real_run_applies_prune_when_requested(self) -> None:
with TemporaryDirectory() as tmp: with TemporaryDirectory() as tmp:
prefix = Path(tmp) / "home" prefix = Path(tmp) / "home"

View File

@@ -0,0 +1,60 @@
from __future__ import annotations
import os
from pathlib import Path
from tempfile import TemporaryDirectory
from django.test import SimpleTestCase
from pobsync.run_stats import parse_rsync_stats, tree_usage
class RunStatsTests(SimpleTestCase):
def test_parse_rsync_stats_extracts_counts_bytes_and_savings(self) -> None:
stats = parse_rsync_stats(
"""
Number of files: 1,234 (reg: 1,200, dir: 34)
Number of created files: 12 (reg: 10, dir: 2)
Number of deleted files: 3
Number of regular files transferred: 8
Total file size: 1.50M bytes
Total transferred file size: 24.00K bytes
Literal data: 24.00K bytes
Matched data: 976.00K bytes
File list size: 8.00K
Total bytes sent: 10.00K
Total bytes received: 2.00K
sent 10.00K bytes received 2.00K bytes 1.20K bytes/sec
total size is 1.50M speedup is 125.00
"""
)
self.assertEqual(stats["files_total"], 1234)
self.assertEqual(stats["files_created"], 12)
self.assertEqual(stats["files_deleted"], 3)
self.assertEqual(stats["files_transferred"], 8)
self.assertEqual(stats["total_file_size_bytes"], 1_500_000)
self.assertEqual(stats["total_transferred_file_size_bytes"], 24_000)
self.assertEqual(stats["literal_data_bytes"], 24_000)
self.assertEqual(stats["matched_data_bytes"], 976_000)
self.assertEqual(stats["bytes_sent_received"], 12_000)
self.assertEqual(stats["bytes_per_second"], 1_200)
self.assertEqual(stats["speedup"], 125.0)
self.assertEqual(stats["link_dest_estimated_savings_bytes"], 976_000)
self.assertEqual(stats["link_dest_estimated_savings_ratio"], 0.976)
def test_tree_usage_reports_hardlinked_files(self) -> None:
with TemporaryDirectory() as tmp:
root = Path(tmp)
source = root / "source"
linked = root / "linked"
source.write_bytes(b"abc")
os.link(source, linked)
stats = tree_usage(root)
self.assertEqual(stats["files"], 2)
self.assertEqual(stats["apparent_size_bytes"], 6)
self.assertEqual(stats["hardlinked_files"], 2)
self.assertEqual(stats["hardlinked_apparent_size_bytes"], 6)
self.assertEqual(stats["hardlink_apparent_ratio"], 1.0)

Some files were not shown because too many files have changed in this diff Show More