Compare commits
32 Commits
5faef1492d
...
v1.0.0
| Author | SHA1 | Date | |
|---|---|---|---|
| 00d4f2a70b | |||
| f8215a0c9a | |||
| ea9e3e41e3 | |||
| 5b5a5bc637 | |||
| c2e5a534aa | |||
| d0c23deb72 | |||
| 4c8ed24561 | |||
| 404b7f7500 | |||
| beca073ddc | |||
| 362a9dde62 | |||
| a73d34ac9f | |||
| 1c8cbd96ca | |||
| 86873bd035 | |||
| 2642f14e49 | |||
| bb62382e18 | |||
| c5865a5379 | |||
| 58d567f9bc | |||
| 2d9f453767 | |||
| 20a9f93378 | |||
| b78f102e9d | |||
| 8858e049ee | |||
| a75b97c4c0 | |||
| b4fc5a14b2 | |||
| a0fd33fcb8 | |||
| ef1761385e | |||
| 17215fd191 | |||
| 97753c3d3c | |||
| 994f7f66c4 | |||
| f76b6cad14 | |||
| 90e293facd | |||
| 50eb7cf2f3 | |||
| 26265be440 |
37
CHANGELOG.md
Normal file
37
CHANGELOG.md
Normal file
@@ -0,0 +1,37 @@
|
||||
# Changelog
|
||||
|
||||
## 1.0.0 - 2026-05-21
|
||||
|
||||
Initial stable release of the Django-first pobsync control panel.
|
||||
|
||||
### Added
|
||||
|
||||
- Django control panel for hosts, global settings, schedules, SSH credentials, snapshots, runs, self-checks, and logs.
|
||||
- Native systemd installer and updater for production backup servers.
|
||||
- SQLite by default, with optional MariaDB support.
|
||||
- Scheduler and worker services for queued manual backups and scheduled backups.
|
||||
- Manual backup, dry-run, cancellation, verbose rsync logging, and run detail views.
|
||||
- Snapshot discovery for existing backup directories and SQL-backed snapshot records.
|
||||
- SQL retention planning and apply flow with base snapshot protection and incomplete snapshot visibility.
|
||||
- Explicit cleanup flow for incomplete snapshots, separate from normal retention pruning.
|
||||
- Purged snapshot audit overview with reason, action source, operator, host, kind, path, and timestamp.
|
||||
- Dashboard and host pages with backup health, latest run/snapshot, next run, and storage/stat summaries.
|
||||
- Review resolution for failed/warning runs and incomplete snapshot tasks so operational warnings can be acknowledged.
|
||||
- Worker heartbeat metadata and stale running-run reconciliation for queued backup workers.
|
||||
- SSH key generation, upload, edit, guarded delete, known_hosts management, and per-host key selection.
|
||||
- In-app changelog page sourced from this changelog.
|
||||
- Restore guidance on snapshot detail pages.
|
||||
|
||||
### Changed
|
||||
|
||||
- Django and the database are now the source of truth for configuration.
|
||||
- Docker Compose is documented as development and disposable test tooling rather than the primary production path.
|
||||
- The `pobsync` console entrypoint is now a maintainer layer around Django management commands.
|
||||
- Scheduled pruning is evaluated by the pobsync scheduler service and recorded through Django, not host cron.
|
||||
- Retention and incomplete cleanup now preserve audit history even after source snapshot records are removed.
|
||||
|
||||
### Removed
|
||||
|
||||
- Legacy YAML config import/export workflow.
|
||||
- Public short aliases for configuration commands.
|
||||
- Obsolete global config storage fields.
|
||||
@@ -10,7 +10,7 @@ RUN apt-get update \
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY pyproject.toml README.md ./
|
||||
COPY pyproject.toml README.md CHANGELOG.md ./
|
||||
COPY src ./src
|
||||
COPY manage.py ./
|
||||
COPY scripts/docker-entrypoint ./scripts/docker-entrypoint
|
||||
|
||||
43
README.md
43
README.md
@@ -154,11 +154,50 @@ The UI includes:
|
||||
- `/self-check/` for runtime checks
|
||||
- `/logs/` for filtered pobsync service logs
|
||||
|
||||
## Restoring Data
|
||||
|
||||
pobsync 1.0 treats restores as an explicit manual operation. The control panel shows restore guidance on each snapshot
|
||||
detail page, but it does not run restore commands for you yet. That is deliberate: restores should be inspected and
|
||||
tested before data is copied back into a live system.
|
||||
|
||||
Each snapshot directory contains:
|
||||
|
||||
```
|
||||
<snapshot>/data/ # backed-up filesystem contents
|
||||
<snapshot>/meta/ # metadata and rsync logs
|
||||
```
|
||||
|
||||
Use the `data/` directory as the rsync source. Start with a dry run and restore to a staging path first:
|
||||
|
||||
```
|
||||
rsync -aHAX --numeric-ids --info=progress2 --dry-run /backups/example.org/scheduled/<snapshot>/data/ /restore/example.org/
|
||||
rsync -aHAX --numeric-ids --info=progress2 /backups/example.org/scheduled/<snapshot>/data/ /restore/example.org/
|
||||
```
|
||||
|
||||
After validating the staged files, copy the specific files or directories back to the target machine. For a full-host
|
||||
restore, use another dry run before writing to the remote root:
|
||||
|
||||
```
|
||||
rsync -aHAX --numeric-ids --info=progress2 --dry-run /backups/example.org/scheduled/<snapshot>/data/ root@example.org:/
|
||||
```
|
||||
|
||||
For most incidents, prefer a targeted restore instead of copying the whole snapshot. Keep paths relative to the
|
||||
snapshot's `data/` directory:
|
||||
|
||||
```
|
||||
rsync -aHAX --numeric-ids --info=progress2 --dry-run /backups/example.org/scheduled/<snapshot>/data/etc/nginx/ /restore/example.org/etc/nginx/
|
||||
rsync -aHAX --numeric-ids --info=progress2 --dry-run /backups/example.org/scheduled/<snapshot>/data/home/example/site/public_html/index.php /restore/example.org/home/example/site/public_html/index.php
|
||||
```
|
||||
|
||||
Snapshots may use hardlinks for files that are unchanged between backups. That saves disk space and is safe for normal
|
||||
restore copies, but do not edit files inside snapshot directories. Treat snapshots as read-only and copy data out with
|
||||
rsync.
|
||||
|
||||
## SSH Keys
|
||||
|
||||
SSH keys can be managed from `/ssh-credentials/`. The recommended flow is to generate keys from Django or during the
|
||||
installer. pobsync stores the private key on disk under `POBSYNC_HOME`, keeps the public key visible in the UI, and lets
|
||||
you select a credential either as the global default or as a per-host override.
|
||||
installer. pobsync stores the private key on disk under the runtime state root (`POBSYNC_HOME`), keeps the public key
|
||||
visible in the UI, and lets you select a credential either as the global default or as a per-host override.
|
||||
|
||||
Generated private keys are stored at:
|
||||
|
||||
|
||||
@@ -47,6 +47,9 @@ pobsync django check
|
||||
python3 manage.py showmigrations pobsync_backend
|
||||
```
|
||||
|
||||
The short `pobsync` aliases are limited to operational actions that are useful while debugging a running install.
|
||||
Configuration aliases are intentionally not public commands; use the Django UI or explicit management commands instead.
|
||||
|
||||
Worker and scheduler commands are normally run by systemd services:
|
||||
|
||||
```
|
||||
@@ -62,6 +65,14 @@ pobsync discover-snapshots --host <host>
|
||||
pobsync retention <host>
|
||||
```
|
||||
|
||||
For scripted configuration changes, call the Django management command explicitly so it is clear that this is an
|
||||
automation/debugging path rather than the normal UI workflow:
|
||||
|
||||
```
|
||||
pobsync django configure_pobsync_host <host> --address <host.example>
|
||||
pobsync django configure_pobsync_schedule <host> --schedule-expression "15 2 * * *"
|
||||
```
|
||||
|
||||
## Installer Development
|
||||
|
||||
The native installer is interactive by default when stdin is a terminal. It should keep every prompt backed by a command
|
||||
@@ -85,23 +96,6 @@ The updater is intentionally a small wrapper around the installer for routine pr
|
||||
non-interactive, preserve the existing environment file, skip OS package installation, skip superuser creation, and still
|
||||
run the Django/runtime refresh steps needed after a code update.
|
||||
|
||||
## Migration Helpers
|
||||
|
||||
Import existing legacy YAML configs:
|
||||
|
||||
```
|
||||
python3 manage.py import_pobsync_configs --prefix /opt/pobsync
|
||||
```
|
||||
|
||||
Export SQL config to legacy runtime YAML for inspection or one-off compatibility:
|
||||
|
||||
```
|
||||
python3 manage.py export_pobsync_configs --prefix /opt/pobsync
|
||||
```
|
||||
|
||||
These commands are migration helpers, not the normal operating model. After import, review and continue operating from
|
||||
the Django control panel.
|
||||
|
||||
## Docker With SQLite
|
||||
|
||||
Docker Compose is useful for local development and disposable test installs. Native systemd is preferred for production
|
||||
@@ -181,4 +175,3 @@ Next refactor targets:
|
||||
|
||||
- Move more snapshot lifecycle details into typed domain objects.
|
||||
- Replace remaining dictionary-shaped config at engine boundaries.
|
||||
- Remove legacy YAML import/export once production migration no longer needs it.
|
||||
|
||||
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "pobsync"
|
||||
version = "0.1.0"
|
||||
version = "1.0.0"
|
||||
description = "Pull-based rsync backup tool with hardlinked snapshots"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
|
||||
@@ -1,3 +1,2 @@
|
||||
__all__ = ["__version__"]
|
||||
__version__ = "0.1.0"
|
||||
|
||||
__version__ = "1.0.0"
|
||||
|
||||
@@ -6,11 +6,10 @@ from typing import Sequence
|
||||
|
||||
from django.core.management import execute_from_command_line
|
||||
|
||||
from pobsync import __version__
|
||||
|
||||
|
||||
COMMAND_ALIASES = {
|
||||
"configure-global": "configure_pobsync_global",
|
||||
"configure-host": "configure_pobsync_host",
|
||||
"schedule": "configure_pobsync_schedule",
|
||||
"backup": "run_pobsync_backup",
|
||||
"retention": "run_pobsync_retention",
|
||||
"discover-snapshots": "discover_pobsync_snapshots",
|
||||
@@ -29,11 +28,17 @@ Usage:
|
||||
|
||||
Commands:
|
||||
{commands}
|
||||
|
||||
Configuration is managed from the Django control panel. Use
|
||||
`pobsync django <management-command>` for automation or debugging.
|
||||
"""
|
||||
|
||||
|
||||
def main(argv: Sequence[str] | None = None) -> int:
|
||||
args = list(sys.argv[1:] if argv is None else argv)
|
||||
if args and args[0] in {"--version", "version"}:
|
||||
print(f"pobsync {__version__}")
|
||||
return 0
|
||||
if not args or args[0] in {"-h", "--help", "help"}:
|
||||
print(_usage())
|
||||
return 0
|
||||
|
||||
@@ -4,9 +4,8 @@ from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, List
|
||||
|
||||
from ..config.source import ConfigSource, FileConfigSource
|
||||
from ..config.source import ConfigSource
|
||||
from ..errors import ConfigError
|
||||
from ..paths import PobsyncPaths
|
||||
from ..retention import Snapshot, apply_base_protection, build_retention_plan
|
||||
from ..snapshot_meta import iter_snapshot_dirs, read_snapshot_meta, resolve_host_root
|
||||
from ..util import sanitize_host
|
||||
@@ -40,10 +39,9 @@ def run_retention_plan(
|
||||
if kind not in {"scheduled", "manual", "all"}:
|
||||
raise ConfigError("kind must be scheduled, manual, or all")
|
||||
|
||||
paths = PobsyncPaths(home=prefix)
|
||||
|
||||
source = config_source or FileConfigSource(prefix=paths.home)
|
||||
cfg = source.effective_config_for_host(host)
|
||||
if config_source is None:
|
||||
raise ConfigError("A Django config source is required.")
|
||||
cfg = config_source.effective_config_for_host(host)
|
||||
|
||||
retention = cfg.get("retention")
|
||||
if not isinstance(retention, dict):
|
||||
|
||||
@@ -3,7 +3,7 @@ from __future__ import annotations
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable
|
||||
|
||||
from ..config.source import ConfigSource, FileConfigSource
|
||||
from ..config.source import ConfigSource
|
||||
from ..errors import ConfigError
|
||||
from ..lock import acquire_host_lock
|
||||
from ..paths import PobsyncPaths
|
||||
@@ -163,8 +163,9 @@ def run_scheduled(
|
||||
host = sanitize_host(host)
|
||||
paths = PobsyncPaths(home=prefix)
|
||||
|
||||
source = config_source or FileConfigSource(prefix=paths.home)
|
||||
cfg = source.effective_config_for_host(host)
|
||||
if config_source is None:
|
||||
raise ConfigError("A Django config source is required.")
|
||||
cfg = config_source.effective_config_for_host(host)
|
||||
|
||||
backup_root = cfg.get("backup_root")
|
||||
if not isinstance(backup_root, str) or not backup_root.startswith("/"):
|
||||
@@ -316,7 +317,6 @@ def run_scheduled(
|
||||
"duration_seconds": None,
|
||||
"base": _base_meta_from_path(base_dir, link_dest),
|
||||
"rsync": {"exit_code": None, "command": cmd, "stats": {}},
|
||||
# Keep existing fields for future expansion / compatibility with current structure.
|
||||
"overrides": {"includes": [], "excludes": [], "base": None},
|
||||
}
|
||||
|
||||
|
||||
@@ -1,53 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
from ..errors import ConfigError, ValidationError
|
||||
from ..validate import validate_dict
|
||||
from .schemas import GLOBAL_SCHEMA, HOST_SCHEMA
|
||||
|
||||
|
||||
def load_yaml_file(path: Path) -> dict[str, Any]:
|
||||
if not path.exists():
|
||||
raise ConfigError(f"Missing config file: {path}")
|
||||
try:
|
||||
raw = path.read_text(encoding="utf-8")
|
||||
except OSError as e:
|
||||
raise ConfigError(f"Cannot read config file: {path}: {e}") from e
|
||||
|
||||
try:
|
||||
data = yaml.safe_load(raw)
|
||||
except yaml.YAMLError as e:
|
||||
raise ConfigError(f"Invalid YAML in {path}: {e}") from e
|
||||
|
||||
if data is None:
|
||||
data = {}
|
||||
if not isinstance(data, dict):
|
||||
raise ConfigError(f"Config root must be a mapping in {path}")
|
||||
return data
|
||||
|
||||
|
||||
def load_global_config(path: Path) -> dict[str, Any]:
|
||||
data = load_yaml_file(path)
|
||||
try:
|
||||
return validate_dict(data, GLOBAL_SCHEMA, path="global")
|
||||
except ValidationError as e:
|
||||
raise ConfigError(f"Invalid global config at {path}: {format_validation_error(e)}") from e
|
||||
|
||||
|
||||
def load_host_config(path: Path) -> dict[str, Any]:
|
||||
data = load_yaml_file(path)
|
||||
try:
|
||||
return validate_dict(data, HOST_SCHEMA, path="host")
|
||||
except ValidationError as e:
|
||||
raise ConfigError(f"Invalid host config at {path}: {format_validation_error(e)}") from e
|
||||
|
||||
|
||||
def format_validation_error(err: ValidationError) -> str:
|
||||
if err.path:
|
||||
return f"{err.path}: {err}"
|
||||
return str(err)
|
||||
|
||||
@@ -83,7 +83,6 @@ OUTPUT_SCHEMA = Schema(
|
||||
GLOBAL_SCHEMA = Schema(
|
||||
fields={
|
||||
"backup_root": FieldSpec(str, required=True),
|
||||
"pobsync_home": FieldSpec(str, required=False, default="/opt/pobsync"),
|
||||
"ssh": FieldSpec(dict, required=False, schema=SSH_SCHEMA),
|
||||
"rsync": FieldSpec(dict, required=False, schema=RSYNC_SCHEMA),
|
||||
"defaults": FieldSpec(dict, required=False, schema=DEFAULTS_SCHEMA),
|
||||
@@ -95,7 +94,6 @@ GLOBAL_SCHEMA = Schema(
|
||||
),
|
||||
"logging": FieldSpec(dict, required=False, schema=LOGGING_SCHEMA),
|
||||
"output": FieldSpec(dict, required=False, schema=OUTPUT_SCHEMA),
|
||||
# Used by `init-host` as a convenience default
|
||||
"retention_defaults": FieldSpec(
|
||||
dict,
|
||||
required=False,
|
||||
@@ -131,4 +129,3 @@ HOST_SCHEMA = Schema(
|
||||
},
|
||||
allow_unknown=False,
|
||||
)
|
||||
|
||||
|
||||
@@ -1,22 +1,8 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any, Protocol
|
||||
|
||||
from .load import load_global_config, load_host_config
|
||||
from .merge import build_effective_config
|
||||
|
||||
|
||||
class ConfigSource(Protocol):
|
||||
def effective_config_for_host(self, host: str) -> dict[str, Any]:
|
||||
"""Return the fully merged effective config for a host."""
|
||||
|
||||
|
||||
class FileConfigSource:
|
||||
def __init__(self, prefix: Path) -> None:
|
||||
self.prefix = prefix
|
||||
|
||||
def effective_config_for_host(self, host: str) -> dict[str, Any]:
|
||||
global_cfg = load_global_config(self.prefix / "config" / "global.yaml")
|
||||
host_cfg = load_host_config(self.prefix / "config" / "hosts" / f"{host}.yaml")
|
||||
return build_effective_config(global_cfg, host_cfg)
|
||||
|
||||
@@ -8,14 +8,6 @@ from pathlib import Path
|
||||
class PobsyncPaths:
|
||||
home: Path # usually /opt/pobsync
|
||||
|
||||
@property
|
||||
def config_dir(self) -> Path:
|
||||
return self.home / "config"
|
||||
|
||||
@property
|
||||
def hosts_dir(self) -> Path:
|
||||
return self.config_dir / "hosts"
|
||||
|
||||
@property
|
||||
def state_dir(self) -> Path:
|
||||
return self.home / "state"
|
||||
@@ -28,11 +20,6 @@ class PobsyncPaths:
|
||||
def logs_dir(self) -> Path:
|
||||
return self.home / "logs"
|
||||
|
||||
@property
|
||||
def global_config_path(self) -> Path:
|
||||
return self.config_dir / "global.yaml"
|
||||
|
||||
@property
|
||||
def central_log_path(self) -> Path:
|
||||
return self.logs_dir / "pobsync.log"
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ from django.urls import reverse
|
||||
from django.utils.html import format_html
|
||||
from django.utils.http import urlencode
|
||||
|
||||
from .models import BackupRun, GlobalConfig, HostConfig, ScheduleConfig, SnapshotRecord, SshCredential
|
||||
from .models import BackupRun, GlobalConfig, HostConfig, PurgedSnapshot, ScheduleConfig, SnapshotRecord, SshCredential
|
||||
|
||||
|
||||
@admin.register(SshCredential)
|
||||
@@ -34,7 +34,7 @@ class GlobalConfigAdmin(admin.ModelAdmin):
|
||||
list_display = ("name", "backup_root", "ssh_user", "ssh_port", "updated_at")
|
||||
readonly_fields = ("created_at", "updated_at")
|
||||
fieldsets = (
|
||||
(None, {"fields": ("name", "backup_root", "pobsync_home")}),
|
||||
(None, {"fields": ("name", "backup_root")}),
|
||||
("SSH", {"fields": ("default_ssh_credential", "ssh_user", "ssh_port", "ssh_options")}),
|
||||
(
|
||||
"Rsync",
|
||||
@@ -50,7 +50,6 @@ class GlobalConfigAdmin(admin.ModelAdmin):
|
||||
),
|
||||
("Defaults", {"fields": ("default_source_root", "default_destination_subdir", "excludes_default")}),
|
||||
("Retention defaults", {"fields": ("retention_daily", "retention_weekly", "retention_monthly", "retention_yearly")}),
|
||||
("Legacy JSON", {"fields": ("data",), "classes": ("collapse",)}),
|
||||
("Timestamps", {"fields": ("created_at", "updated_at"), "classes": ("collapse",)}),
|
||||
)
|
||||
|
||||
@@ -76,7 +75,7 @@ class HostConfigAdmin(admin.ModelAdmin):
|
||||
("Source", {"fields": ("source_root", "includes", "excludes_add", "excludes_replace")}),
|
||||
("Rsync override", {"fields": ("rsync_extra_args",)}),
|
||||
("Retention", {"fields": ("retention_daily", "retention_weekly", "retention_monthly", "retention_yearly")}),
|
||||
("Legacy JSON", {"fields": ("config",), "classes": ("collapse",)}),
|
||||
("Runtime state", {"fields": ("config",), "classes": ("collapse",)}),
|
||||
("Timestamps", {"fields": ("created_at", "updated_at"), "classes": ("collapse",)}),
|
||||
)
|
||||
|
||||
@@ -174,6 +173,16 @@ class SnapshotRecordAdmin(admin.ModelAdmin):
|
||||
return format_html('<a href="{}">{}</a>', url, count)
|
||||
|
||||
|
||||
@admin.register(PurgedSnapshot)
|
||||
class PurgedSnapshotAdmin(admin.ModelAdmin):
|
||||
list_display = ("host_name", "kind", "dirname", "action", "reason", "triggered_by", "purged_at")
|
||||
list_filter = ("action", "kind", "purged_at")
|
||||
search_fields = ("host_name", "dirname", "path", "reason", "triggered_by")
|
||||
list_select_related = ("host",)
|
||||
readonly_fields = ("purged_at",)
|
||||
date_hierarchy = "purged_at"
|
||||
|
||||
|
||||
@admin.register(ScheduleConfig)
|
||||
class ScheduleConfigAdmin(admin.ModelAdmin):
|
||||
list_display = ("host", "cron_expr", "enabled", "prune", "last_status", "last_started_at", "updated_at")
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import timedelta
|
||||
import os
|
||||
import socket
|
||||
from datetime import timedelta, timezone as datetime_timezone
|
||||
from pathlib import Path
|
||||
|
||||
from django.db import transaction
|
||||
@@ -107,6 +109,7 @@ def execute_backup_run(
|
||||
protect_bases=bool(prune_protect_bases),
|
||||
yes=True,
|
||||
max_delete=int(prune_max_delete),
|
||||
action=run.run_type,
|
||||
acquire_lock=False,
|
||||
)
|
||||
except Exception as exc:
|
||||
@@ -158,10 +161,10 @@ def claim_next_queued_run() -> BackupRun | None:
|
||||
return run
|
||||
|
||||
|
||||
def reconcile_running_runs(*, grace_seconds: int = 300) -> int:
|
||||
def reconcile_running_runs(*, grace_seconds: int = 300, stale_worker_seconds: int = 24 * 60 * 60) -> int:
|
||||
reconciled = 0
|
||||
for run in BackupRun.objects.select_related("host").filter(status=BackupRun.Status.RUNNING).order_by("started_at", "id"):
|
||||
if _reconcile_running_run(run=run, grace_seconds=grace_seconds):
|
||||
if _reconcile_running_run(run=run, grace_seconds=grace_seconds, stale_worker_seconds=stale_worker_seconds):
|
||||
reconciled += 1
|
||||
return reconciled
|
||||
|
||||
@@ -176,7 +179,9 @@ def requested_options(run: BackupRun) -> dict[str, object]:
|
||||
def _running_result(*, run: BackupRun, dry_run: bool) -> dict[str, object]:
|
||||
result = dict(run.result) if isinstance(run.result, dict) else {}
|
||||
execution = {
|
||||
**_worker_execution_details(),
|
||||
"started_at": (run.started_at or timezone.now()).isoformat(),
|
||||
"heartbeat_at": timezone.now().isoformat(),
|
||||
}
|
||||
if dry_run:
|
||||
execution["log"] = str(dry_run_log_path(run.host.host, run_id=run.id))
|
||||
@@ -185,24 +190,56 @@ def _running_result(*, run: BackupRun, dry_run: bool) -> dict[str, object]:
|
||||
|
||||
|
||||
def _run_cancel_requested(run_id: int) -> bool:
|
||||
return BackupRun.objects.filter(id=run_id, status=BackupRun.Status.CANCELLED).exists()
|
||||
try:
|
||||
run = BackupRun.objects.only("id", "status", "result").get(id=run_id)
|
||||
except BackupRun.DoesNotExist:
|
||||
return True
|
||||
if run.status == BackupRun.Status.CANCELLED:
|
||||
return True
|
||||
if run.status == BackupRun.Status.RUNNING:
|
||||
_refresh_run_heartbeat(run)
|
||||
return False
|
||||
|
||||
|
||||
def _reconcile_running_run(*, run: BackupRun, grace_seconds: int) -> bool:
|
||||
def _reconcile_running_run(*, run: BackupRun, grace_seconds: int, stale_worker_seconds: int) -> bool:
|
||||
result = run.result if isinstance(run.result, dict) else {}
|
||||
requested = result.get("requested") if isinstance(result.get("requested"), dict) else {}
|
||||
stale_worker = _running_worker_timed_out(run=run, stale_worker_seconds=stale_worker_seconds)
|
||||
if not requested.get("dry_run"):
|
||||
if stale_worker:
|
||||
result.update(
|
||||
{
|
||||
"ok": False,
|
||||
"host": run.host.host,
|
||||
"failure": {
|
||||
"category": "worker",
|
||||
"message": "The worker heartbeat stopped before the run finished.",
|
||||
"hint": "Check pobsync-worker.service logs before retrying the backup.",
|
||||
},
|
||||
}
|
||||
)
|
||||
run.status = BackupRun.Status.FAILED
|
||||
run.ended_at = timezone.now()
|
||||
run.result = result
|
||||
run.save(update_fields=["status", "ended_at", "result"])
|
||||
return True
|
||||
return False
|
||||
|
||||
log_path = _execution_log_path(result)
|
||||
log_tail = _read_log_tail(log_path) if log_path is not None else []
|
||||
terminal_log = _terminal_rsync_log(log_tail)
|
||||
timed_out = _running_dry_run_timed_out(run=run, grace_seconds=grace_seconds)
|
||||
if not terminal_log and not timed_out:
|
||||
if not terminal_log and not timed_out and not stale_worker:
|
||||
return False
|
||||
|
||||
exit_code = _exit_code_from_log(log_tail) or (124 if timed_out else 255)
|
||||
exit_code = _exit_code_from_log(log_tail) or (124 if timed_out or stale_worker else 255)
|
||||
failure = classify_rsync_failure(exit_code, log_tail)
|
||||
if stale_worker and not terminal_log:
|
||||
failure = {
|
||||
"category": "worker",
|
||||
"message": "The worker heartbeat stopped before the dry-run finished.",
|
||||
"hint": "Check pobsync-worker.service logs before retrying the dry-run.",
|
||||
}
|
||||
result.update(
|
||||
{
|
||||
"ok": False,
|
||||
@@ -226,6 +263,30 @@ def _reconcile_running_run(*, run: BackupRun, grace_seconds: int) -> bool:
|
||||
return True
|
||||
|
||||
|
||||
def _worker_execution_details() -> dict[str, object]:
|
||||
return {
|
||||
"worker_pid": os.getpid(),
|
||||
"worker_host": socket.gethostname(),
|
||||
"claimed_at": timezone.now().isoformat(),
|
||||
}
|
||||
|
||||
|
||||
def _refresh_run_heartbeat(run: BackupRun, *, interval_seconds: int = 30) -> None:
|
||||
result = run.result if isinstance(run.result, dict) else {}
|
||||
execution = result.get("execution") if isinstance(result.get("execution"), dict) else {}
|
||||
heartbeat_at = _parse_iso_datetime(execution.get("heartbeat_at"))
|
||||
if heartbeat_at is not None and timezone.now() < heartbeat_at + timedelta(seconds=interval_seconds):
|
||||
return
|
||||
result["execution"] = {
|
||||
**execution,
|
||||
"worker_pid": os.getpid(),
|
||||
"worker_host": socket.gethostname(),
|
||||
"heartbeat_at": timezone.now().isoformat(),
|
||||
}
|
||||
run.result = result
|
||||
run.save(update_fields=["result"])
|
||||
|
||||
|
||||
def _execution_log_path(result: dict[str, object]) -> Path | None:
|
||||
execution = result.get("execution") if isinstance(result.get("execution"), dict) else {}
|
||||
log = execution.get("log") or result.get("log")
|
||||
@@ -266,3 +327,28 @@ def _running_dry_run_timed_out(*, run: BackupRun, grace_seconds: int) -> bool:
|
||||
if not isinstance(timeout_seconds, int) or timeout_seconds <= 0:
|
||||
timeout_seconds = DEFAULT_DRY_RUN_TIMEOUT_SECONDS
|
||||
return timezone.now() >= run.started_at + timedelta(seconds=timeout_seconds + grace_seconds)
|
||||
|
||||
|
||||
def _running_worker_timed_out(*, run: BackupRun, stale_worker_seconds: int) -> bool:
|
||||
if stale_worker_seconds <= 0:
|
||||
return False
|
||||
result = run.result if isinstance(run.result, dict) else {}
|
||||
execution = result.get("execution") if isinstance(result.get("execution"), dict) else {}
|
||||
heartbeat_at = _parse_iso_datetime(execution.get("heartbeat_at"))
|
||||
if heartbeat_at is None:
|
||||
heartbeat_at = run.started_at
|
||||
if heartbeat_at is None:
|
||||
return False
|
||||
return timezone.now() >= heartbeat_at + timedelta(seconds=stale_worker_seconds)
|
||||
|
||||
|
||||
def _parse_iso_datetime(value: object):
|
||||
if not isinstance(value, str) or not value:
|
||||
return None
|
||||
try:
|
||||
parsed = timezone.datetime.fromisoformat(value)
|
||||
except ValueError:
|
||||
return None
|
||||
if timezone.is_naive(parsed):
|
||||
return timezone.make_aware(parsed, timezone=datetime_timezone.utc)
|
||||
return parsed
|
||||
|
||||
@@ -17,7 +17,7 @@ CRITICAL_ROOT_EXCLUDES = ("/proc/***", "/sys/***", "/dev/***", "/run/***", "/tmp
|
||||
def collect_global_config_checks(global_config: GlobalConfig) -> list[SelfCheck]:
|
||||
checks = [
|
||||
_absolute_path_check("Global backup root", global_config.backup_root),
|
||||
_absolute_path_check("Global pobsync home", global_config.pobsync_home),
|
||||
_absolute_path_check("Runtime state root", settings.POBSYNC_HOME),
|
||||
_runtime_backup_root_check(global_config),
|
||||
_rsync_binary_check(global_config.rsync_binary),
|
||||
_rsync_recursion_check(
|
||||
@@ -97,7 +97,7 @@ def _runtime_backup_root_check(global_config: GlobalConfig) -> SelfCheck:
|
||||
return SelfCheck(
|
||||
"Runtime backup root",
|
||||
"warning",
|
||||
"Database backup root differs from runtime POBSYNC_BACKUP_ROOT.",
|
||||
"Database backup root differs from the runtime backup root.",
|
||||
f"database={global_config.backup_root} runtime={settings.POBSYNC_BACKUP_ROOT}",
|
||||
)
|
||||
|
||||
|
||||
@@ -1,13 +1,10 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from django.core.exceptions import ObjectDoesNotExist
|
||||
|
||||
from pobsync.config.schemas import GLOBAL_SCHEMA, HOST_SCHEMA
|
||||
from pobsync.paths import PobsyncPaths
|
||||
from pobsync.util import write_yaml_atomic
|
||||
from pobsync.validate import validate_dict
|
||||
|
||||
from .models import GlobalConfig, HostConfig
|
||||
@@ -17,10 +14,9 @@ class ConfigRepositoryError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def _global_yaml_data(global_config: GlobalConfig) -> dict[str, Any]:
|
||||
def _global_runtime_data(global_config: GlobalConfig) -> dict[str, Any]:
|
||||
data = {
|
||||
"backup_root": global_config.backup_root,
|
||||
"pobsync_home": global_config.pobsync_home,
|
||||
"ssh": {
|
||||
"user": global_config.ssh_user,
|
||||
"port": global_config.ssh_port,
|
||||
@@ -48,7 +44,7 @@ def _global_yaml_data(global_config: GlobalConfig) -> dict[str, Any]:
|
||||
return validate_dict(data, GLOBAL_SCHEMA, path="global")
|
||||
|
||||
|
||||
def _host_yaml_data(host_config: HostConfig) -> dict[str, Any]:
|
||||
def _host_runtime_data(host_config: HostConfig) -> dict[str, Any]:
|
||||
data: dict[str, Any] = {
|
||||
"host": host_config.host,
|
||||
"address": host_config.address,
|
||||
@@ -78,57 +74,24 @@ def _host_yaml_data(host_config: HostConfig) -> dict[str, Any]:
|
||||
|
||||
|
||||
def global_config_object_data(global_config: GlobalConfig) -> dict[str, Any]:
|
||||
return _global_yaml_data(global_config)
|
||||
return _global_runtime_data(global_config)
|
||||
|
||||
|
||||
def host_config_object_data(host_config: HostConfig) -> dict[str, Any]:
|
||||
return _host_yaml_data(host_config)
|
||||
return _host_runtime_data(host_config)
|
||||
|
||||
|
||||
def global_config_data(name: str = "default") -> dict[str, Any]:
|
||||
try:
|
||||
global_config = GlobalConfig.objects.get(name=name)
|
||||
except ObjectDoesNotExist as exc:
|
||||
raise ConfigRepositoryError(f"Missing GlobalConfig {name!r}") from exc
|
||||
return _global_yaml_data(global_config)
|
||||
raise ConfigRepositoryError(f"Missing global config {name!r}") from exc
|
||||
return _global_runtime_data(global_config)
|
||||
|
||||
|
||||
def host_config_data(host: str) -> dict[str, Any]:
|
||||
try:
|
||||
host_config = HostConfig.objects.get(host=host, enabled=True)
|
||||
except ObjectDoesNotExist as exc:
|
||||
raise ConfigRepositoryError(f"Missing enabled HostConfig {host!r}") from exc
|
||||
return _host_yaml_data(host_config)
|
||||
|
||||
|
||||
def export_global_config(prefix: Path, name: str = "default") -> Path:
|
||||
try:
|
||||
global_config = GlobalConfig.objects.get(name=name)
|
||||
except ObjectDoesNotExist as exc:
|
||||
raise ConfigRepositoryError(f"Missing GlobalConfig {name!r}") from exc
|
||||
|
||||
paths = PobsyncPaths(home=prefix)
|
||||
write_yaml_atomic(paths.global_config_path, _global_yaml_data(global_config))
|
||||
return paths.global_config_path
|
||||
|
||||
|
||||
def export_host_config(prefix: Path, host: str) -> Path:
|
||||
try:
|
||||
host_config = HostConfig.objects.get(host=host, enabled=True)
|
||||
except ObjectDoesNotExist as exc:
|
||||
raise ConfigRepositoryError(f"Missing enabled HostConfig {host!r}") from exc
|
||||
|
||||
paths = PobsyncPaths(home=prefix)
|
||||
target = paths.hosts_dir / f"{host_config.host}.yaml"
|
||||
write_yaml_atomic(target, _host_yaml_data(host_config))
|
||||
return target
|
||||
|
||||
|
||||
def export_runtime_configs(prefix: Path, host: str | None = None) -> list[Path]:
|
||||
written = [export_global_config(prefix)]
|
||||
hosts = HostConfig.objects.filter(enabled=True).order_by("host")
|
||||
if host is not None:
|
||||
hosts = hosts.filter(host=host)
|
||||
for host_config in hosts:
|
||||
written.append(export_host_config(prefix, host_config.host))
|
||||
return written
|
||||
raise ConfigRepositoryError(f"Missing enabled host {host!r}") from exc
|
||||
return _host_runtime_data(host_config)
|
||||
|
||||
@@ -119,7 +119,6 @@ class GlobalConfigForm(forms.ModelForm):
|
||||
def save(self, commit: bool = True):
|
||||
instance = super().save(commit=False)
|
||||
instance.backup_root = settings.POBSYNC_BACKUP_ROOT
|
||||
instance.pobsync_home = settings.POBSYNC_HOME
|
||||
if commit:
|
||||
instance.save()
|
||||
self.save_m2m()
|
||||
@@ -249,12 +248,18 @@ class RetentionApplyForm(forms.Form):
|
||||
kind = forms.ChoiceField(choices=(("scheduled", "Scheduled"), ("manual", "Manual"), ("all", "All")))
|
||||
protect_bases = forms.BooleanField(required=False)
|
||||
max_delete = forms.IntegerField(min_value=0, initial=10)
|
||||
confirm_delete_count = forms.IntegerField(min_value=0)
|
||||
confirm_host = forms.CharField()
|
||||
|
||||
def __init__(self, *args, host_name: str, **kwargs) -> None:
|
||||
def __init__(self, *args, host_name: str, expected_delete_count: int | None = None, **kwargs) -> None:
|
||||
self.host_name = host_name
|
||||
self.expected_delete_count = expected_delete_count
|
||||
super().__init__(*args, **kwargs)
|
||||
self.fields["confirm_host"].help_text = f"Type {host_name} to confirm deletion."
|
||||
if expected_delete_count is not None:
|
||||
self.fields["confirm_delete_count"].help_text = (
|
||||
f"Type {expected_delete_count} to confirm the current number of planned deletions."
|
||||
)
|
||||
|
||||
def clean_confirm_host(self) -> str:
|
||||
value = self.cleaned_data["confirm_host"].strip()
|
||||
@@ -262,6 +267,42 @@ class RetentionApplyForm(forms.Form):
|
||||
raise forms.ValidationError(f"Type {self.host_name} to confirm.")
|
||||
return value
|
||||
|
||||
def clean_confirm_delete_count(self) -> int:
|
||||
value = self.cleaned_data["confirm_delete_count"]
|
||||
if self.expected_delete_count is not None and value != self.expected_delete_count:
|
||||
raise forms.ValidationError(f"Type {self.expected_delete_count} to confirm the delete count.")
|
||||
return value
|
||||
|
||||
|
||||
class IncompleteCleanupForm(forms.Form):
|
||||
max_delete = forms.IntegerField(min_value=0, initial=0)
|
||||
confirm_delete_count = forms.IntegerField(min_value=0)
|
||||
confirm_host = forms.CharField()
|
||||
|
||||
def __init__(self, *args, host_name: str, expected_delete_count: int, **kwargs) -> None:
|
||||
self.host_name = host_name
|
||||
self.expected_delete_count = expected_delete_count
|
||||
super().__init__(*args, **kwargs)
|
||||
self.fields["confirm_host"].help_text = f"Type {host_name} to confirm incomplete snapshot cleanup."
|
||||
self.fields["confirm_delete_count"].help_text = (
|
||||
f"Type {expected_delete_count} to confirm the current number of incomplete snapshots."
|
||||
)
|
||||
self.fields["max_delete"].help_text = (
|
||||
f"Must be at least {expected_delete_count} for the incomplete snapshots shown here."
|
||||
)
|
||||
|
||||
def clean_confirm_host(self) -> str:
|
||||
value = self.cleaned_data["confirm_host"].strip()
|
||||
if value != self.host_name:
|
||||
raise forms.ValidationError(f"Type {self.host_name} to confirm.")
|
||||
return value
|
||||
|
||||
def clean_confirm_delete_count(self) -> int:
|
||||
value = self.cleaned_data["confirm_delete_count"]
|
||||
if value != self.expected_delete_count:
|
||||
raise forms.ValidationError(f"Type {self.expected_delete_count} to confirm the incomplete count.")
|
||||
return value
|
||||
|
||||
|
||||
class ScheduleConfigForm(forms.ModelForm):
|
||||
cron_expr = forms.CharField(
|
||||
|
||||
@@ -1,9 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from django.conf import settings
|
||||
from django.core.management.base import BaseCommand, CommandError
|
||||
|
||||
from pobsync.config.retention import parse_retention
|
||||
@@ -13,12 +11,11 @@ from pobsync_backend.models import GlobalConfig
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = "Create or update the SQL-backed global pobsync configuration."
|
||||
help = "Create or update the default global backup configuration."
|
||||
|
||||
def add_arguments(self, parser) -> None:
|
||||
parser.add_argument("--name", default="default")
|
||||
parser.add_argument("--backup-root", required=True)
|
||||
parser.add_argument("--pobsync-home", default=settings.POBSYNC_HOME)
|
||||
parser.add_argument("--ssh-user", default="root")
|
||||
parser.add_argument("--ssh-port", type=int, default=22)
|
||||
parser.add_argument("--source-root", default="/")
|
||||
@@ -30,11 +27,9 @@ class Command(BaseCommand):
|
||||
if not is_absolute_non_root(backup_root):
|
||||
raise CommandError("--backup-root must be an absolute path and must not be '/'")
|
||||
|
||||
pobsync_home = str(Path(options["pobsync_home"]))
|
||||
retention = parse_retention(options["retention"])
|
||||
defaults = {
|
||||
"backup_root": backup_root,
|
||||
"pobsync_home": pobsync_home,
|
||||
"ssh_user": options["ssh_user"],
|
||||
"ssh_port": options["ssh_port"],
|
||||
"ssh_options": ["-oBatchMode=yes", "-oStrictHostKeyChecking=accept-new"],
|
||||
@@ -53,8 +48,8 @@ class Command(BaseCommand):
|
||||
}
|
||||
|
||||
if GlobalConfig.objects.filter(name=options["name"]).exists() and not options["force"]:
|
||||
raise CommandError(f"GlobalConfig {options['name']!r} already exists; use --force to update")
|
||||
raise CommandError(f"Global config {options['name']!r} already exists; use --force to update")
|
||||
|
||||
_obj, created = GlobalConfig.objects.update_or_create(name=options["name"], defaults=defaults)
|
||||
action = "Created" if created else "Updated"
|
||||
self.stdout.write(self.style.SUCCESS(f"{action} GlobalConfig {options['name']!r}."))
|
||||
self.stdout.write(self.style.SUCCESS(f"{action} global config {options['name']!r}."))
|
||||
|
||||
@@ -10,7 +10,7 @@ from pobsync_backend.models import GlobalConfig, HostConfig
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = "Create or update a SQL-backed host pobsync configuration."
|
||||
help = "Create or update a host backup configuration."
|
||||
|
||||
def add_arguments(self, parser) -> None:
|
||||
parser.add_argument("host")
|
||||
@@ -29,7 +29,7 @@ class Command(BaseCommand):
|
||||
def handle(self, *args: Any, **options: Any) -> None:
|
||||
host = sanitize_host(options["host"])
|
||||
if HostConfig.objects.filter(host=host).exists() and not options["force"]:
|
||||
raise CommandError(f"HostConfig {host!r} already exists; use --force to update")
|
||||
raise CommandError(f"Host {host!r} already exists; use --force to update")
|
||||
|
||||
retention = self._retention(options["retention"])
|
||||
defaults = {
|
||||
@@ -49,7 +49,7 @@ class Command(BaseCommand):
|
||||
}
|
||||
_obj, created = HostConfig.objects.update_or_create(host=host, defaults=defaults)
|
||||
action = "Created" if created else "Updated"
|
||||
self.stdout.write(self.style.SUCCESS(f"{action} HostConfig {host!r}."))
|
||||
self.stdout.write(self.style.SUCCESS(f"{action} host {host!r}."))
|
||||
|
||||
def _retention(self, value: str | None) -> dict[str, int]:
|
||||
if value:
|
||||
|
||||
@@ -9,11 +9,16 @@ from pobsync_backend.scheduler import parse_cron_expr
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = "Create, update, disable, or remove a SQL-backed pobsync schedule."
|
||||
help = "Create, update, disable, or remove a scheduler-managed host schedule."
|
||||
|
||||
def add_arguments(self, parser) -> None:
|
||||
parser.add_argument("host")
|
||||
parser.add_argument("--cron", help='Cron expression, e.g. "15 2 * * *"')
|
||||
parser.add_argument(
|
||||
"--schedule-expression",
|
||||
"--cron",
|
||||
dest="schedule_expression",
|
||||
help='Five-field schedule expression, e.g. "15 2 * * *"',
|
||||
)
|
||||
parser.add_argument("--prune", action="store_true")
|
||||
parser.add_argument("--prune-max-delete", type=int, default=10)
|
||||
parser.add_argument("--prune-protect-bases", action="store_true")
|
||||
@@ -24,24 +29,25 @@ class Command(BaseCommand):
|
||||
try:
|
||||
host = HostConfig.objects.get(host=options["host"])
|
||||
except HostConfig.DoesNotExist as exc:
|
||||
raise CommandError(f"Missing HostConfig {options['host']!r}") from exc
|
||||
raise CommandError(f"Missing host {options['host']!r}") from exc
|
||||
|
||||
if options["delete"]:
|
||||
deleted, _details = ScheduleConfig.objects.filter(host=host).delete()
|
||||
self.stdout.write(self.style.SUCCESS(f"Deleted {deleted} schedule row(s) for {host.host!r}."))
|
||||
return
|
||||
|
||||
if not options["cron"]:
|
||||
raise CommandError("--cron is required unless --delete is used")
|
||||
schedule_expression = options["schedule_expression"]
|
||||
if not schedule_expression:
|
||||
raise CommandError("--schedule-expression is required unless --delete is used")
|
||||
try:
|
||||
parse_cron_expr(options["cron"])
|
||||
parse_cron_expr(schedule_expression)
|
||||
except ValueError as exc:
|
||||
raise CommandError(str(exc)) from exc
|
||||
|
||||
schedule, created = ScheduleConfig.objects.update_or_create(
|
||||
host=host,
|
||||
defaults={
|
||||
"cron_expr": options["cron"],
|
||||
"cron_expr": schedule_expression,
|
||||
"enabled": not options["disabled"],
|
||||
"prune": bool(options["prune"]),
|
||||
"prune_max_delete": int(options["prune_max_delete"]),
|
||||
|
||||
@@ -20,14 +20,14 @@ class Command(BaseCommand):
|
||||
try:
|
||||
global_config = GlobalConfig.objects.get(name="default")
|
||||
except GlobalConfig.DoesNotExist as exc:
|
||||
raise CommandError("Missing GlobalConfig 'default'") from exc
|
||||
raise CommandError("Missing default global config") from exc
|
||||
|
||||
host = None
|
||||
if options["host"]:
|
||||
try:
|
||||
host = HostConfig.objects.get(host=options["host"], enabled=True)
|
||||
except HostConfig.DoesNotExist as exc:
|
||||
raise CommandError(f"Missing enabled HostConfig {options['host']!r}") from exc
|
||||
raise CommandError(f"Missing enabled host {options['host']!r}") from exc
|
||||
|
||||
kind = normalize_kind(options["kind"])
|
||||
kinds = ["scheduled", "manual", "incomplete"] if kind == "all" else [kind]
|
||||
|
||||
@@ -1,23 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from django.conf import settings
|
||||
from django.core.management.base import BaseCommand
|
||||
|
||||
from pobsync_backend.config_repository import export_runtime_configs
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = "Export Django database configs to pobsync runtime YAML files."
|
||||
|
||||
def add_arguments(self, parser) -> None:
|
||||
parser.add_argument("--prefix", default=settings.POBSYNC_HOME, help="Pobsync home directory")
|
||||
parser.add_argument("--host", default=None, help="Export only one enabled host")
|
||||
|
||||
def handle(self, *args: Any, **options: Any) -> None:
|
||||
written = export_runtime_configs(prefix=Path(options["prefix"]), host=options["host"])
|
||||
for path in written:
|
||||
self.stdout.write(str(path))
|
||||
self.stdout.write(self.style.SUCCESS(f"Exported {len(written)} config file(s)."))
|
||||
@@ -1,81 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from django.conf import settings
|
||||
from django.core.management.base import BaseCommand, CommandError
|
||||
|
||||
from pobsync.config.load import load_global_config, load_host_config
|
||||
from pobsync.paths import PobsyncPaths
|
||||
from pobsync_backend.models import GlobalConfig, HostConfig
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = "Import pobsync YAML configs into the Django database."
|
||||
|
||||
def add_arguments(self, parser) -> None:
|
||||
parser.add_argument("--prefix", default=settings.POBSYNC_HOME, help="Pobsync home directory")
|
||||
|
||||
def handle(self, *args: Any, **options: Any) -> None:
|
||||
paths = PobsyncPaths(home=Path(options["prefix"]))
|
||||
if not paths.global_config_path.exists():
|
||||
raise CommandError(f"Missing global config: {paths.global_config_path}")
|
||||
|
||||
global_cfg = load_global_config(paths.global_config_path)
|
||||
global_ssh = global_cfg.get("ssh") or {}
|
||||
global_rsync = global_cfg.get("rsync") or {}
|
||||
global_defaults = global_cfg.get("defaults") or {}
|
||||
retention_defaults = global_cfg.get("retention_defaults") or {}
|
||||
GlobalConfig.objects.update_or_create(
|
||||
name="default",
|
||||
defaults={
|
||||
"backup_root": global_cfg["backup_root"],
|
||||
"pobsync_home": global_cfg.get("pobsync_home", str(paths.home)),
|
||||
"ssh_user": global_ssh.get("user") or "root",
|
||||
"ssh_port": global_ssh.get("port") or 22,
|
||||
"ssh_options": global_ssh.get("options") or [],
|
||||
"rsync_binary": global_rsync.get("binary") or "rsync",
|
||||
"rsync_args": global_rsync.get("args") or [],
|
||||
"rsync_extra_args": global_rsync.get("extra_args") or [],
|
||||
"rsync_timeout_seconds": global_rsync.get("timeout_seconds") or 0,
|
||||
"rsync_bwlimit_kbps": global_rsync.get("bwlimit_kbps") or 0,
|
||||
"default_source_root": global_defaults.get("source_root") or "/",
|
||||
"default_destination_subdir": global_defaults.get("destination_subdir") or "",
|
||||
"excludes_default": global_cfg.get("excludes_default") or [],
|
||||
"retention_daily": retention_defaults.get("daily", 14),
|
||||
"retention_weekly": retention_defaults.get("weekly", 8),
|
||||
"retention_monthly": retention_defaults.get("monthly", 12),
|
||||
"retention_yearly": retention_defaults.get("yearly", 0),
|
||||
"data": global_cfg,
|
||||
},
|
||||
)
|
||||
|
||||
count = 0
|
||||
for host_path in sorted(paths.hosts_dir.glob("*.yaml")):
|
||||
host_cfg = load_host_config(host_path)
|
||||
host_ssh = host_cfg.get("ssh") or {}
|
||||
host_rsync = host_cfg.get("rsync") or {}
|
||||
host_retention = host_cfg.get("retention") or {}
|
||||
HostConfig.objects.update_or_create(
|
||||
host=host_cfg["host"],
|
||||
defaults={
|
||||
"address": host_cfg["address"],
|
||||
"ssh_user": host_ssh.get("user") or "",
|
||||
"ssh_port": host_ssh.get("port"),
|
||||
"source_root": host_cfg.get("source_root") or "",
|
||||
"includes": host_cfg.get("includes") or [],
|
||||
"excludes_add": host_cfg.get("excludes_add") or [],
|
||||
"excludes_replace": host_cfg.get("excludes_replace"),
|
||||
"rsync_extra_args": host_rsync.get("extra_args") or [],
|
||||
"retention_daily": host_retention.get("daily", 14),
|
||||
"retention_weekly": host_retention.get("weekly", 8),
|
||||
"retention_monthly": host_retention.get("monthly", 12),
|
||||
"retention_yearly": host_retention.get("yearly", 0),
|
||||
"config": host_cfg,
|
||||
"enabled": True,
|
||||
},
|
||||
)
|
||||
count += 1
|
||||
|
||||
self.stdout.write(self.style.SUCCESS(f"Imported global config and {count} host config(s)."))
|
||||
@@ -16,7 +16,7 @@ class Command(BaseCommand):
|
||||
|
||||
def add_arguments(self, parser) -> None:
|
||||
parser.add_argument("host", help="Host to back up")
|
||||
parser.add_argument("--prefix", default=settings.POBSYNC_HOME, help="Pobsync home directory")
|
||||
parser.add_argument("--prefix", default=settings.POBSYNC_HOME, help="Runtime state root")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Run rsync --dry-run")
|
||||
parser.add_argument("--verbose-rsync", action="store_true", help="Write itemized rsync output to the run log")
|
||||
parser.add_argument("--prune", action="store_true", help="Apply retention after a successful run")
|
||||
@@ -30,7 +30,7 @@ class Command(BaseCommand):
|
||||
try:
|
||||
host = HostConfig.objects.get(host=host_name, enabled=True)
|
||||
except HostConfig.DoesNotExist as exc:
|
||||
raise CommandError(f"Missing enabled HostConfig {host_name!r}") from exc
|
||||
raise CommandError(f"Missing enabled host {host_name!r}") from exc
|
||||
|
||||
run = BackupRun.objects.create(
|
||||
host=host,
|
||||
|
||||
@@ -12,11 +12,11 @@ from pobsync_backend.retention import run_sql_retention_apply, run_sql_retention
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = "Plan or apply retention using SQL-backed pobsync configuration."
|
||||
help = "Plan or apply retention using the Django backup configuration."
|
||||
|
||||
def add_arguments(self, parser) -> None:
|
||||
parser.add_argument("host")
|
||||
parser.add_argument("--prefix", default=settings.POBSYNC_HOME)
|
||||
parser.add_argument("--prefix", default=settings.POBSYNC_HOME, help="Runtime state root")
|
||||
parser.add_argument("--kind", default="scheduled", choices=["scheduled", "manual", "all"])
|
||||
parser.add_argument("--protect-bases", action="store_true")
|
||||
parser.add_argument("--apply", action="store_true")
|
||||
@@ -36,6 +36,7 @@ class Command(BaseCommand):
|
||||
protect_bases=bool(options["protect_bases"]),
|
||||
yes=True,
|
||||
max_delete=int(options["max_delete"]),
|
||||
action="cli",
|
||||
)
|
||||
else:
|
||||
result = run_sql_retention_plan(
|
||||
|
||||
@@ -10,7 +10,7 @@ from django.core.management.base import BaseCommand
|
||||
from django.db import transaction
|
||||
from django.utils import timezone
|
||||
|
||||
from pobsync_backend.models import ScheduleConfig
|
||||
from pobsync_backend.models import BackupRun, ScheduleConfig
|
||||
from pobsync_backend.scheduler import due_key, is_due
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ class Command(BaseCommand):
|
||||
help = "Run due pobsync schedules from the Django database."
|
||||
|
||||
def add_arguments(self, parser) -> None:
|
||||
parser.add_argument("--prefix", default=settings.POBSYNC_HOME, help="Pobsync home directory")
|
||||
parser.add_argument("--prefix", default=settings.POBSYNC_HOME, help="Runtime state root")
|
||||
parser.add_argument("--once", action="store_true", help="Check once and exit")
|
||||
parser.add_argument("--loop", action="store_true", help="Keep checking schedules")
|
||||
parser.add_argument("--interval", type=int, default=60, help="Loop interval in seconds")
|
||||
@@ -52,12 +52,13 @@ class Command(BaseCommand):
|
||||
if not is_due(schedule.cron_expr, now):
|
||||
continue
|
||||
|
||||
schedule_started_at = timezone.now()
|
||||
with transaction.atomic():
|
||||
locked = ScheduleConfig.objects.select_for_update().get(pk=schedule.pk)
|
||||
if locked.last_due_key == current_due_key:
|
||||
continue
|
||||
locked.last_due_key = current_due_key
|
||||
locked.last_started_at = timezone.now()
|
||||
locked.last_started_at = schedule_started_at
|
||||
locked.last_status = "running"
|
||||
locked.save(update_fields=["last_due_key", "last_started_at", "last_status", "updated_at"])
|
||||
|
||||
@@ -72,6 +73,7 @@ class Command(BaseCommand):
|
||||
prune_max_delete=schedule.prune_max_delete,
|
||||
prune_protect_bases=schedule.prune_protect_bases,
|
||||
)
|
||||
status = _latest_scheduled_run_status(host_id=schedule.host_id, started_at=schedule_started_at) or status
|
||||
except Exception as exc:
|
||||
status = "failed"
|
||||
self.stderr.write(f"{schedule.host.host}: {type(exc).__name__}: {exc}")
|
||||
@@ -83,3 +85,16 @@ class Command(BaseCommand):
|
||||
ran += 1
|
||||
|
||||
return ran
|
||||
|
||||
|
||||
def _latest_scheduled_run_status(*, host_id: int, started_at) -> str | None:
|
||||
run = (
|
||||
BackupRun.objects.filter(
|
||||
host_id=host_id,
|
||||
run_type=BackupRun.RunType.SCHEDULED,
|
||||
created_at__gte=started_at,
|
||||
)
|
||||
.order_by("-created_at", "-id")
|
||||
.first()
|
||||
)
|
||||
return run.status if run is not None else None
|
||||
|
||||
@@ -15,10 +15,16 @@ class Command(BaseCommand):
|
||||
help = "Run queued pobsync backup jobs from the Django database."
|
||||
|
||||
def add_arguments(self, parser) -> None:
|
||||
parser.add_argument("--prefix", default=settings.POBSYNC_HOME, help="Pobsync home directory")
|
||||
parser.add_argument("--prefix", default=settings.POBSYNC_HOME, help="Runtime state root")
|
||||
parser.add_argument("--once", action="store_true", help="Process one queued run and exit")
|
||||
parser.add_argument("--loop", action="store_true", help="Keep checking for queued runs")
|
||||
parser.add_argument("--interval", type=int, default=15, help="Loop interval in seconds")
|
||||
parser.add_argument(
|
||||
"--stale-running-seconds",
|
||||
type=int,
|
||||
default=24 * 60 * 60,
|
||||
help="Mark running runs failed after this many seconds without a worker heartbeat; use 0 to disable",
|
||||
)
|
||||
|
||||
def handle(self, *args: Any, **options: Any) -> None:
|
||||
if not options["once"] and not options["loop"]:
|
||||
@@ -26,14 +32,14 @@ class Command(BaseCommand):
|
||||
|
||||
paths = PobsyncPaths(home=Path(options["prefix"]))
|
||||
while True:
|
||||
count = self._run_once(prefix=paths.home)
|
||||
count = self._run_once(prefix=paths.home, stale_running_seconds=int(options["stale_running_seconds"]))
|
||||
self.stdout.write(f"Ran {count} queued backup run(s).")
|
||||
if options["once"]:
|
||||
return
|
||||
time.sleep(max(1, int(options["interval"])))
|
||||
|
||||
def _run_once(self, *, prefix: Path) -> int:
|
||||
reconciled = reconcile_running_runs()
|
||||
def _run_once(self, *, prefix: Path, stale_running_seconds: int = 24 * 60 * 60) -> int:
|
||||
reconciled = reconcile_running_runs(stale_worker_seconds=stale_running_seconds)
|
||||
run = claim_next_queued_run()
|
||||
if run is None:
|
||||
return reconciled
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
from django.db import migrations
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
dependencies = [
|
||||
("pobsync_backend", "0009_remove_scheduleconfig_user"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.RemoveField(
|
||||
model_name="globalconfig",
|
||||
name="pobsync_home",
|
||||
),
|
||||
]
|
||||
@@ -0,0 +1,14 @@
|
||||
from django.db import migrations
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
dependencies = [
|
||||
("pobsync_backend", "0010_remove_globalconfig_pobsync_home"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.RemoveField(
|
||||
model_name="globalconfig",
|
||||
name="data",
|
||||
),
|
||||
]
|
||||
30
src/pobsync_backend/migrations/0012_review_state.py
Normal file
30
src/pobsync_backend/migrations/0012_review_state.py
Normal file
@@ -0,0 +1,30 @@
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
dependencies = [
|
||||
("pobsync_backend", "0011_remove_globalconfig_data"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="backuprun",
|
||||
name="reviewed_at",
|
||||
field=models.DateTimeField(blank=True, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="backuprun",
|
||||
name="reviewed_by",
|
||||
field=models.CharField(blank=True, max_length=150),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="snapshotrecord",
|
||||
name="reviewed_at",
|
||||
field=models.DateTimeField(blank=True, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="snapshotrecord",
|
||||
name="reviewed_by",
|
||||
field=models.CharField(blank=True, max_length=150),
|
||||
),
|
||||
]
|
||||
50
src/pobsync_backend/migrations/0013_purgedsnapshot.py
Normal file
50
src/pobsync_backend/migrations/0013_purgedsnapshot.py
Normal file
@@ -0,0 +1,50 @@
|
||||
from django.db import migrations, models
|
||||
import django.db.models.deletion
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
dependencies = [
|
||||
("pobsync_backend", "0012_review_state"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.CreateModel(
|
||||
name="PurgedSnapshot",
|
||||
fields=[
|
||||
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
|
||||
("host_name", models.CharField(max_length=255)),
|
||||
("kind", models.CharField(max_length=16)),
|
||||
("dirname", models.CharField(max_length=255)),
|
||||
("path", models.CharField(max_length=1024)),
|
||||
("reason", models.CharField(blank=True, max_length=512)),
|
||||
(
|
||||
"action",
|
||||
models.CharField(
|
||||
choices=[
|
||||
("manual", "Manual"),
|
||||
("scheduled", "Scheduled"),
|
||||
("cli", "CLI"),
|
||||
("incomplete_cleanup", "Incomplete cleanup"),
|
||||
],
|
||||
max_length=32,
|
||||
),
|
||||
),
|
||||
("triggered_by", models.CharField(blank=True, max_length=150)),
|
||||
("metadata", models.JSONField(blank=True, default=dict)),
|
||||
("purged_at", models.DateTimeField(auto_now_add=True)),
|
||||
(
|
||||
"host",
|
||||
models.ForeignKey(
|
||||
blank=True,
|
||||
null=True,
|
||||
on_delete=django.db.models.deletion.SET_NULL,
|
||||
related_name="purged_snapshots",
|
||||
to="pobsync_backend.hostconfig",
|
||||
),
|
||||
),
|
||||
],
|
||||
options={
|
||||
"ordering": ["-purged_at", "host_name", "dirname"],
|
||||
},
|
||||
),
|
||||
]
|
||||
@@ -14,7 +14,6 @@ class TimestampedModel(models.Model):
|
||||
class GlobalConfig(TimestampedModel):
|
||||
name = models.CharField(max_length=64, default="default", unique=True)
|
||||
backup_root = models.CharField(max_length=512)
|
||||
pobsync_home = models.CharField(max_length=512, default="/opt/pobsync")
|
||||
default_ssh_credential = models.ForeignKey(
|
||||
"SshCredential",
|
||||
on_delete=models.SET_NULL,
|
||||
@@ -37,7 +36,6 @@ class GlobalConfig(TimestampedModel):
|
||||
retention_weekly = models.PositiveIntegerField(default=8)
|
||||
retention_monthly = models.PositiveIntegerField(default=12)
|
||||
retention_yearly = models.PositiveIntegerField(default=0)
|
||||
data = models.JSONField(default=dict, blank=True)
|
||||
|
||||
class Meta:
|
||||
verbose_name = "global config"
|
||||
@@ -126,6 +124,8 @@ class BackupRun(models.Model):
|
||||
rsync_exit_code = models.IntegerField(null=True, blank=True)
|
||||
result = models.JSONField(default=dict, blank=True)
|
||||
created_at = models.DateTimeField(auto_now_add=True)
|
||||
reviewed_at = models.DateTimeField(null=True, blank=True)
|
||||
reviewed_by = models.CharField(max_length=150, blank=True)
|
||||
|
||||
class Meta:
|
||||
ordering = ["-created_at"]
|
||||
@@ -160,6 +160,8 @@ class SnapshotRecord(models.Model):
|
||||
ended_at = models.DateTimeField(null=True, blank=True)
|
||||
metadata = models.JSONField(default=dict, blank=True)
|
||||
discovered_at = models.DateTimeField(auto_now_add=True)
|
||||
reviewed_at = models.DateTimeField(null=True, blank=True)
|
||||
reviewed_by = models.CharField(max_length=150, blank=True)
|
||||
|
||||
class Meta:
|
||||
constraints = [
|
||||
@@ -171,6 +173,31 @@ class SnapshotRecord(models.Model):
|
||||
return f"{self.host}/{self.kind}/{self.dirname}"
|
||||
|
||||
|
||||
class PurgedSnapshot(models.Model):
|
||||
class Action(models.TextChoices):
|
||||
MANUAL = "manual", "Manual"
|
||||
SCHEDULED = "scheduled", "Scheduled"
|
||||
CLI = "cli", "CLI"
|
||||
INCOMPLETE_CLEANUP = "incomplete_cleanup", "Incomplete cleanup"
|
||||
|
||||
host = models.ForeignKey(HostConfig, on_delete=models.SET_NULL, null=True, blank=True, related_name="purged_snapshots")
|
||||
host_name = models.CharField(max_length=255)
|
||||
kind = models.CharField(max_length=16)
|
||||
dirname = models.CharField(max_length=255)
|
||||
path = models.CharField(max_length=1024)
|
||||
reason = models.CharField(max_length=512, blank=True)
|
||||
action = models.CharField(max_length=32, choices=Action.choices)
|
||||
triggered_by = models.CharField(max_length=150, blank=True)
|
||||
metadata = models.JSONField(default=dict, blank=True)
|
||||
purged_at = models.DateTimeField(auto_now_add=True)
|
||||
|
||||
class Meta:
|
||||
ordering = ["-purged_at", "host_name", "dirname"]
|
||||
|
||||
def __str__(self) -> str:
|
||||
return f"{self.host_name}/{self.kind}/{self.dirname}"
|
||||
|
||||
|
||||
class ScheduleConfig(TimestampedModel):
|
||||
host = models.OneToOneField(HostConfig, on_delete=models.CASCADE, related_name="schedule")
|
||||
cron_expr = models.CharField(max_length=128)
|
||||
|
||||
@@ -12,7 +12,7 @@ from pobsync.paths import PobsyncPaths
|
||||
from pobsync.retention import Snapshot, apply_base_protection, build_retention_plan
|
||||
from pobsync.util import sanitize_host
|
||||
|
||||
from .models import HostConfig, SnapshotRecord
|
||||
from .models import HostConfig, PurgedSnapshot, SnapshotRecord
|
||||
|
||||
|
||||
def run_sql_retention_plan(*, host: str, kind: str, protect_bases: bool) -> dict[str, Any]:
|
||||
@@ -23,6 +23,7 @@ def run_sql_retention_plan(*, host: str, kind: str, protect_bases: bool) -> dict
|
||||
host_config = _enabled_host_config(host)
|
||||
retention = _retention_for_host(host_config)
|
||||
snapshots = _snapshots_for_retention(host_config=host_config, kind=kind)
|
||||
incomplete_snapshots = _incomplete_snapshots_for_host(host_config)
|
||||
|
||||
plan = build_retention_plan(
|
||||
snapshots=snapshots,
|
||||
@@ -36,6 +37,7 @@ def run_sql_retention_plan(*, host: str, kind: str, protect_bases: bool) -> dict
|
||||
keep, reasons = apply_base_protection(snapshots=snapshots, keep=keep, reasons=reasons)
|
||||
|
||||
delete = [snapshot for snapshot in snapshots if snapshot.dirname not in keep]
|
||||
keep_items = [snapshot for snapshot in snapshots if snapshot.dirname in keep]
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
@@ -45,7 +47,12 @@ def run_sql_retention_plan(*, host: str, kind: str, protect_bases: bool) -> dict
|
||||
"retention": retention,
|
||||
"source": "sql",
|
||||
"keep": sorted(keep),
|
||||
"delete": [_snapshot_to_delete_item(snapshot) for snapshot in delete],
|
||||
"keep_items": [_snapshot_to_item(snapshot, reasons=reasons.get(snapshot.dirname, [])) for snapshot in keep_items],
|
||||
"delete": [_snapshot_to_item(snapshot, reasons=["outside retention policy"]) for snapshot in delete],
|
||||
"incomplete": [
|
||||
_snapshot_to_item(snapshot, reasons=["incomplete snapshot; excluded from retention cleanup"])
|
||||
for snapshot in incomplete_snapshots
|
||||
],
|
||||
"reasons": reasons,
|
||||
}
|
||||
|
||||
@@ -58,6 +65,8 @@ def run_sql_retention_apply(
|
||||
protect_bases: bool,
|
||||
yes: bool,
|
||||
max_delete: int,
|
||||
action: str = PurgedSnapshot.Action.MANUAL,
|
||||
triggered_by: str = "",
|
||||
acquire_lock: bool = True,
|
||||
) -> dict[str, Any]:
|
||||
host = sanitize_host(host)
|
||||
@@ -71,8 +80,11 @@ def run_sql_retention_apply(
|
||||
def _do_apply() -> dict[str, Any]:
|
||||
plan = run_sql_retention_plan(host=host, kind=kind, protect_bases=bool(protect_bases))
|
||||
delete_list = plan.get("delete") or []
|
||||
incomplete_list = plan.get("incomplete") or []
|
||||
if not isinstance(delete_list, list):
|
||||
raise ConfigError("Invalid retention plan output: delete is not a list")
|
||||
if not isinstance(incomplete_list, list):
|
||||
raise ConfigError("Invalid retention plan output: incomplete is not a list")
|
||||
if max_delete == 0 and len(delete_list) > 0:
|
||||
raise ConfigError("Deletion blocked by --max-delete=0")
|
||||
if len(delete_list) > max_delete:
|
||||
@@ -91,6 +103,7 @@ def run_sql_retention_apply(
|
||||
raise ConfigError(f"Refusing to delete unsupported snapshot kind: {snap_kind!r}")
|
||||
|
||||
path = _snapshot_delete_path(path=Path(snap_path), dirname=dirname)
|
||||
reason = str(item.get("reason") or "outside retention policy")
|
||||
if not path.exists():
|
||||
actions.append(f"skip missing {snap_kind}/{dirname}")
|
||||
continue
|
||||
@@ -98,9 +111,19 @@ def run_sql_retention_apply(
|
||||
raise ConfigError(f"Refusing to delete non-directory path: {path}")
|
||||
|
||||
_remove_snapshot_tree(path)
|
||||
_record_purged_snapshot(
|
||||
host_config=_enabled_host_config(host),
|
||||
kind=snap_kind,
|
||||
dirname=dirname,
|
||||
path=path,
|
||||
reason=reason,
|
||||
action=action,
|
||||
triggered_by=triggered_by,
|
||||
metadata={"source": "retention", "protect_bases": bool(protect_bases), "retention_kind": kind},
|
||||
)
|
||||
SnapshotRecord.objects.filter(host__host=host, kind=snap_kind, dirname=dirname).delete()
|
||||
actions.append(f"deleted {snap_kind} {dirname}")
|
||||
deleted.append({"dirname": dirname, "kind": snap_kind, "path": str(path)})
|
||||
deleted.append({"dirname": dirname, "kind": snap_kind, "path": str(path), "reason": reason})
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
@@ -109,6 +132,8 @@ def run_sql_retention_apply(
|
||||
"protect_bases": bool(protect_bases),
|
||||
"max_delete": max_delete,
|
||||
"source": "sql",
|
||||
"planned_delete_count": len(delete_list),
|
||||
"incomplete_ignored_count": len(incomplete_list),
|
||||
"deleted": deleted,
|
||||
"actions": actions,
|
||||
}
|
||||
@@ -119,11 +144,92 @@ def run_sql_retention_apply(
|
||||
return _do_apply()
|
||||
|
||||
|
||||
def run_incomplete_cleanup(
|
||||
*,
|
||||
prefix: Path,
|
||||
host: str,
|
||||
yes: bool,
|
||||
max_delete: int,
|
||||
triggered_by: str = "",
|
||||
acquire_lock: bool = True,
|
||||
) -> dict[str, Any]:
|
||||
host = sanitize_host(host)
|
||||
if not yes:
|
||||
raise ConfigError("Refusing to delete incomplete snapshots without --yes")
|
||||
if max_delete < 0:
|
||||
raise ConfigError("--max-delete must be >= 0")
|
||||
|
||||
paths = PobsyncPaths(home=prefix)
|
||||
|
||||
def _do_cleanup() -> dict[str, Any]:
|
||||
host_config = _enabled_host_config(host)
|
||||
incomplete_list = [
|
||||
_snapshot_to_item(snapshot, reasons=["manual incomplete cleanup"])
|
||||
for snapshot in _incomplete_snapshots_for_host(host_config)
|
||||
]
|
||||
if max_delete == 0 and len(incomplete_list) > 0:
|
||||
raise ConfigError("Incomplete cleanup blocked by --max-delete=0")
|
||||
if len(incomplete_list) > max_delete:
|
||||
raise ConfigError(
|
||||
f"Refusing to delete {len(incomplete_list)} incomplete snapshots (exceeds --max-delete={max_delete})"
|
||||
)
|
||||
|
||||
actions: list[str] = []
|
||||
deleted: list[dict[str, Any]] = []
|
||||
|
||||
for item in incomplete_list:
|
||||
dirname = item["dirname"]
|
||||
snap_path = Path(item["path"])
|
||||
path = _snapshot_delete_path(path=snap_path, dirname=dirname)
|
||||
_validate_incomplete_delete_path(host=host, path=path, dirname=dirname)
|
||||
|
||||
if not path.exists():
|
||||
actions.append(f"skip missing incomplete/{dirname}")
|
||||
elif not path.is_dir():
|
||||
raise ConfigError(f"Refusing to delete non-directory path: {path}")
|
||||
else:
|
||||
_remove_snapshot_tree(path)
|
||||
actions.append(f"deleted incomplete {dirname}")
|
||||
|
||||
_record_purged_snapshot(
|
||||
host_config=host_config,
|
||||
kind=SnapshotRecord.Kind.INCOMPLETE,
|
||||
dirname=dirname,
|
||||
path=path,
|
||||
reason="manual incomplete cleanup",
|
||||
action=PurgedSnapshot.Action.INCOMPLETE_CLEANUP,
|
||||
triggered_by=triggered_by,
|
||||
metadata={"source": "incomplete_cleanup"},
|
||||
)
|
||||
SnapshotRecord.objects.filter(
|
||||
host__host=host,
|
||||
kind=SnapshotRecord.Kind.INCOMPLETE,
|
||||
dirname=dirname,
|
||||
).delete()
|
||||
deleted.append({"dirname": dirname, "kind": SnapshotRecord.Kind.INCOMPLETE, "path": str(path)})
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"host": host,
|
||||
"kind": SnapshotRecord.Kind.INCOMPLETE,
|
||||
"max_delete": max_delete,
|
||||
"source": "sql",
|
||||
"planned_delete_count": len(incomplete_list),
|
||||
"deleted": deleted,
|
||||
"actions": actions,
|
||||
}
|
||||
|
||||
if acquire_lock:
|
||||
with acquire_host_lock(paths.locks_dir, host, command="incomplete-cleanup"):
|
||||
return _do_cleanup()
|
||||
return _do_cleanup()
|
||||
|
||||
|
||||
def _enabled_host_config(host: str) -> HostConfig:
|
||||
try:
|
||||
return HostConfig.objects.get(host=host, enabled=True)
|
||||
except HostConfig.DoesNotExist as exc:
|
||||
raise ConfigError(f"Missing enabled HostConfig {host!r}") from exc
|
||||
raise ConfigError(f"Missing enabled host {host!r}") from exc
|
||||
|
||||
|
||||
def _retention_for_host(host_config: HostConfig) -> dict[str, int]:
|
||||
@@ -146,6 +252,15 @@ def _snapshots_for_retention(*, host_config: HostConfig, kind: str) -> list[Snap
|
||||
return [_snapshot_from_record(record) for record in records]
|
||||
|
||||
|
||||
def _incomplete_snapshots_for_host(host_config: HostConfig) -> list[Snapshot]:
|
||||
records = (
|
||||
SnapshotRecord.objects.filter(host=host_config, kind=SnapshotRecord.Kind.INCOMPLETE)
|
||||
.select_related("base")
|
||||
.order_by("-started_at", "dirname")
|
||||
)
|
||||
return [_snapshot_from_record(record) for record in records]
|
||||
|
||||
|
||||
def _snapshot_from_record(record: SnapshotRecord) -> Snapshot:
|
||||
return Snapshot(
|
||||
kind=record.kind,
|
||||
@@ -173,13 +288,15 @@ def _base_meta_from_record(record: SnapshotRecord) -> dict[str, str] | None:
|
||||
return None
|
||||
|
||||
|
||||
def _snapshot_to_delete_item(snapshot: Snapshot) -> dict[str, Any]:
|
||||
def _snapshot_to_item(snapshot: Snapshot, *, reasons: list[str]) -> dict[str, Any]:
|
||||
return {
|
||||
"dirname": snapshot.dirname,
|
||||
"kind": snapshot.kind,
|
||||
"path": snapshot.path,
|
||||
"dt": snapshot.dt.isoformat(),
|
||||
"status": snapshot.status,
|
||||
"reasons": reasons,
|
||||
"reason": ", ".join(reasons),
|
||||
}
|
||||
|
||||
|
||||
@@ -189,6 +306,39 @@ def _snapshot_delete_path(*, path: Path, dirname: str) -> Path:
|
||||
return path
|
||||
|
||||
|
||||
def _record_purged_snapshot(
|
||||
*,
|
||||
host_config: HostConfig,
|
||||
kind: str,
|
||||
dirname: str,
|
||||
path: Path,
|
||||
reason: str,
|
||||
action: str,
|
||||
triggered_by: str,
|
||||
metadata: dict[str, Any],
|
||||
) -> None:
|
||||
PurgedSnapshot.objects.create(
|
||||
host=host_config,
|
||||
host_name=host_config.host,
|
||||
kind=kind,
|
||||
dirname=dirname,
|
||||
path=str(path),
|
||||
reason=reason,
|
||||
action=action,
|
||||
triggered_by=triggered_by,
|
||||
metadata=metadata,
|
||||
)
|
||||
|
||||
|
||||
def _validate_incomplete_delete_path(*, host: str, path: Path, dirname: str) -> None:
|
||||
path_parts = path.parts
|
||||
if path.name != dirname or ".incomplete" not in path_parts or host not in path_parts:
|
||||
raise ConfigError(f"Refusing to delete unexpected incomplete snapshot path: {path}")
|
||||
incomplete_index = path_parts.index(".incomplete")
|
||||
if incomplete_index == 0 or path_parts[incomplete_index - 1] != host:
|
||||
raise ConfigError(f"Refusing to delete incomplete snapshot outside host backup root: {path}")
|
||||
|
||||
|
||||
def _remove_snapshot_tree(path: Path) -> None:
|
||||
_make_directories_user_writable(path)
|
||||
shutil.rmtree(path)
|
||||
|
||||
@@ -76,10 +76,17 @@ def _django_checks() -> list[SelfCheck]:
|
||||
|
||||
def _path_checks() -> list[SelfCheck]:
|
||||
checks = []
|
||||
checks.append(_path_check("POBSYNC_HOME", Path(settings.POBSYNC_HOME), must_be_absolute=True, must_be_writable=True))
|
||||
checks.append(
|
||||
_path_check(
|
||||
"POBSYNC_BACKUP_ROOT",
|
||||
"State root",
|
||||
Path(settings.POBSYNC_HOME),
|
||||
must_be_absolute=True,
|
||||
must_be_writable=True,
|
||||
)
|
||||
)
|
||||
checks.append(
|
||||
_path_check(
|
||||
"Backup root",
|
||||
Path(settings.POBSYNC_BACKUP_ROOT),
|
||||
must_be_absolute=True,
|
||||
must_exist=True,
|
||||
@@ -259,7 +266,7 @@ def _config_checks() -> list[SelfCheck]:
|
||||
message = "Default global config exists."
|
||||
if global_config.backup_root != settings.POBSYNC_BACKUP_ROOT:
|
||||
status = "warning"
|
||||
message = "Global config backup root differs from runtime POBSYNC_BACKUP_ROOT."
|
||||
message = "Global config backup root differs from the runtime backup root."
|
||||
return [
|
||||
SelfCheck(
|
||||
"Global config",
|
||||
|
||||
@@ -37,6 +37,8 @@ def collect_dashboard_stats(*, hosts: Iterable[HostConfig], global_config: Globa
|
||||
available = _int_at(capacity, "available_bytes")
|
||||
daily_literal = _average_daily_literal(real_runs)
|
||||
|
||||
link_dest_savings_ratio = round(total_matched / savings_basis, 4) if savings_basis else None
|
||||
|
||||
return {
|
||||
"runs_sampled": len(real_runs),
|
||||
"avg_duration_seconds": _average(duration_values),
|
||||
@@ -44,7 +46,8 @@ def collect_dashboard_stats(*, hosts: Iterable[HostConfig], global_config: Globa
|
||||
"avg_literal_data_bytes": avg_literal,
|
||||
"total_literal_data_bytes": total_literal,
|
||||
"total_matched_data_bytes": total_matched,
|
||||
"link_dest_savings_ratio": round(total_matched / savings_basis, 4) if savings_basis else None,
|
||||
"link_dest_savings_ratio": link_dest_savings_ratio,
|
||||
"link_dest_savings_percent": round(link_dest_savings_ratio * 100, 1) if link_dest_savings_ratio is not None else None,
|
||||
"estimated_runs_until_full": int(available / avg_literal) if available and avg_literal else None,
|
||||
"estimated_days_until_full": int(available / daily_literal) if available and daily_literal else None,
|
||||
"capacity": capacity,
|
||||
@@ -52,9 +55,10 @@ def collect_dashboard_stats(*, hosts: Iterable[HostConfig], global_config: Globa
|
||||
|
||||
|
||||
def collect_host_stats(*, host: HostConfig, limit: int = 8) -> dict[str, Any]:
|
||||
runs = list(host.runs.select_related("snapshot").filter(status__in=_COMPLETED_BACKUP_STATUSES).order_by("-started_at", "-created_at")[:50])
|
||||
runs = list(host.runs.select_related("snapshot").order_by("-started_at", "-created_at")[:50])
|
||||
real_runs = [_run_summary(run) for run in runs if _is_real_run(run)]
|
||||
trend_runs = [run for run in real_runs if run["has_stats"]][:limit]
|
||||
completed_real_runs = [run for run in real_runs if run["status"] in _COMPLETED_BACKUP_STATUSES]
|
||||
trend_runs = [run for run in completed_real_runs if run["has_stats"]][:limit]
|
||||
latest_snapshot = host.snapshots.order_by("-started_at", "-discovered_at", "-id").first()
|
||||
latest_snapshot_stats = _snapshot_summary(latest_snapshot) if latest_snapshot else {}
|
||||
|
||||
@@ -67,7 +71,9 @@ def collect_host_stats(*, host: HostConfig, limit: int = 8) -> dict[str, Any]:
|
||||
|
||||
return {
|
||||
"runs": [_with_bar_percentages(run, max_literal=max_literal, max_matched=max_matched) for run in trend_runs],
|
||||
"latest_run": real_runs[0] if real_runs else {},
|
||||
"latest_run": completed_real_runs[0] if completed_real_runs else {},
|
||||
"latest_good_run": _first_run_with_status(real_runs, {BackupRun.Status.SUCCESS}),
|
||||
"latest_problem_run": _first_run_with_status(real_runs, {BackupRun.Status.WARNING, BackupRun.Status.FAILED}),
|
||||
"latest_snapshot": latest_snapshot_stats,
|
||||
"avg_literal_data_bytes": _average(literal_values),
|
||||
"avg_daily_literal_data_bytes": _average_daily_literal(trend_runs),
|
||||
@@ -87,6 +93,8 @@ def _run_summary(run: BackupRun) -> dict[str, Any]:
|
||||
"ended_at": run.ended_at,
|
||||
"snapshot": run.snapshot,
|
||||
"snapshot_path": run.snapshot_path,
|
||||
"status": run.status,
|
||||
"reviewed_at": run.reviewed_at,
|
||||
"has_stats": bool(stats),
|
||||
"duration_seconds": _int_at(stats, "duration_seconds"),
|
||||
"rsync": stats.get("rsync") if isinstance(stats.get("rsync"), dict) else {},
|
||||
@@ -121,6 +129,13 @@ def _is_real_run(run: BackupRun) -> bool:
|
||||
return requested.get("dry_run") is not True
|
||||
|
||||
|
||||
def _first_run_with_status(runs: list[dict[str, Any]], statuses: set[str]) -> dict[str, Any]:
|
||||
for run in runs:
|
||||
if run["status"] in statuses and run.get("reviewed_at") is None:
|
||||
return run
|
||||
return {}
|
||||
|
||||
|
||||
def _capacity_from_system(global_config: GlobalConfig | None) -> dict[str, Any]:
|
||||
if global_config is None or not global_config.backup_root:
|
||||
return {}
|
||||
|
||||
@@ -61,6 +61,10 @@
|
||||
.metric { padding: 14px; }
|
||||
.metric .label { color: var(--muted); font-size: 12px; text-transform: uppercase; }
|
||||
.metric .value { font-size: 26px; font-weight: 650; margin-top: 4px; }
|
||||
.metric.failed { border-color: #e8b4b4; background: #fff7f7; }
|
||||
.metric.warning { border-color: #e7cf8a; background: #fffaf0; }
|
||||
.metric.running { border-color: #e7cf8a; background: #fffaf0; }
|
||||
.metric.queued { border-color: #b5cdea; background: #eef6ff; }
|
||||
.panel { padding: 16px; margin-bottom: 18px; overflow: auto; }
|
||||
.panel.highlight { border-left: 4px solid var(--border); }
|
||||
.panel.highlight.failed { border-left-color: var(--failed); background: #fff7f7; }
|
||||
@@ -118,6 +122,23 @@
|
||||
cursor: not-allowed;
|
||||
}
|
||||
.inline-form { margin: 0; }
|
||||
.status-overview {
|
||||
display: grid;
|
||||
gap: 8px;
|
||||
}
|
||||
.status-summary {
|
||||
align-items: center;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 6px;
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 8px;
|
||||
padding: 10px;
|
||||
}
|
||||
.status-summary.failed { border-color: #e8b4b4; background: #fff7f7; color: var(--failed); }
|
||||
.status-summary.warning,
|
||||
.status-summary.running { border-color: #e7cf8a; background: #fffaf0; color: var(--running); }
|
||||
.status-summary.queued { border-color: #b5cdea; background: #eef6ff; color: var(--link); }
|
||||
.operator-state {
|
||||
align-items: center;
|
||||
display: flex;
|
||||
@@ -149,6 +170,43 @@
|
||||
font-size: 12px;
|
||||
gap: 10px;
|
||||
}
|
||||
.insight-grid {
|
||||
display: grid;
|
||||
gap: 18px 24px;
|
||||
grid-template-columns: minmax(260px, 1.3fr) repeat(auto-fit, minmax(180px, 1fr));
|
||||
}
|
||||
.insight-main,
|
||||
.insight-item {
|
||||
display: grid;
|
||||
gap: 4px;
|
||||
min-width: 0;
|
||||
}
|
||||
.insight-main .label,
|
||||
.insight-item .label {
|
||||
color: var(--muted);
|
||||
font-size: 12px;
|
||||
font-weight: 650;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
.insight-main .value,
|
||||
.insight-item .value {
|
||||
font-size: 22px;
|
||||
font-weight: 650;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
.storage-meter {
|
||||
background: #edf2f7;
|
||||
border-radius: 999px;
|
||||
height: 10px;
|
||||
margin: 4px 0;
|
||||
overflow: hidden;
|
||||
}
|
||||
.storage-meter span {
|
||||
background: var(--link);
|
||||
display: block;
|
||||
height: 100%;
|
||||
max-width: 100%;
|
||||
}
|
||||
.host-list {
|
||||
display: grid;
|
||||
gap: 12px;
|
||||
@@ -176,23 +234,42 @@
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
.host-card-status {
|
||||
flex: 0 0 auto;
|
||||
display: flex;
|
||||
flex: 0 1 auto;
|
||||
flex-wrap: wrap;
|
||||
gap: 6px;
|
||||
justify-content: flex-end;
|
||||
max-width: 50%;
|
||||
}
|
||||
.host-card-layout {
|
||||
display: grid;
|
||||
gap: 18px;
|
||||
grid-template-columns: minmax(0, 1fr) minmax(240px, 320px);
|
||||
gap: 24px;
|
||||
grid-template-columns: minmax(0, 2fr) minmax(260px, 1fr);
|
||||
}
|
||||
.host-card-section {
|
||||
align-content: start;
|
||||
display: grid;
|
||||
gap: 10px;
|
||||
min-width: 0;
|
||||
}
|
||||
.host-card-section-title {
|
||||
color: var(--muted);
|
||||
font-size: 12px;
|
||||
font-weight: 650;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
.host-card-timeline {
|
||||
display: grid;
|
||||
gap: 12px;
|
||||
gap: 16px 22px;
|
||||
grid-template-columns: repeat(auto-fit, minmax(210px, 1fr));
|
||||
}
|
||||
.host-card-stats {
|
||||
align-content: start;
|
||||
display: grid;
|
||||
gap: 10px;
|
||||
border-top: 1px solid #e6edf4;
|
||||
gap: 12px 18px;
|
||||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||
padding-top: 12px;
|
||||
}
|
||||
.host-card-item {
|
||||
display: grid;
|
||||
@@ -209,13 +286,9 @@
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
.host-card-stat {
|
||||
background: #f8fafc;
|
||||
border: 1px solid #e6edf4;
|
||||
border-radius: 6px;
|
||||
display: grid;
|
||||
gap: 3px;
|
||||
min-width: 0;
|
||||
padding: 10px;
|
||||
}
|
||||
.host-card-stat .label {
|
||||
color: var(--muted);
|
||||
@@ -231,6 +304,17 @@
|
||||
.host-card-stat.wide {
|
||||
grid-column: 1 / -1;
|
||||
}
|
||||
.host-card-warning {
|
||||
background: #fffaf0;
|
||||
border: 1px solid #e7cf8a;
|
||||
border-radius: 6px;
|
||||
color: var(--running);
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 8px;
|
||||
margin-top: 14px;
|
||||
padding: 10px;
|
||||
}
|
||||
.messages { display: grid; gap: 8px; margin-bottom: 18px; }
|
||||
.message {
|
||||
background: var(--panel);
|
||||
@@ -275,7 +359,11 @@
|
||||
main { padding: 16px; }
|
||||
nav { padding: 0; }
|
||||
.two-col { grid-template-columns: 1fr; }
|
||||
.host-card-header { display: grid; }
|
||||
.host-card-status { justify-content: flex-start; max-width: none; }
|
||||
.host-card-layout { grid-template-columns: 1fr; }
|
||||
.host-card-stats { grid-template-columns: repeat(auto-fit, minmax(140px, 1fr)); }
|
||||
.insight-grid { grid-template-columns: 1fr; }
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
@@ -287,6 +375,8 @@
|
||||
<a href="{% url 'ssh_credentials' %}">SSH Keys</a>
|
||||
<a href="{% url 'self_check' %}">Self Check</a>
|
||||
<a href="{% url 'logs' %}">Logs</a>
|
||||
<a href="{% url 'purged_snapshots' %}">Purged</a>
|
||||
<a href="{% url 'changelog' %}">Changelog</a>
|
||||
<a href="/api/status/">Status API</a>
|
||||
<span class="spacer"></span>
|
||||
<span class="muted">{{ request.user.username }}</span>
|
||||
|
||||
41
src/pobsync_backend/templates/pobsync_backend/changelog.html
Normal file
41
src/pobsync_backend/templates/pobsync_backend/changelog.html
Normal file
@@ -0,0 +1,41 @@
|
||||
{% extends "pobsync_backend/base.html" %}
|
||||
|
||||
{% block title %}Changelog - pobsync{% endblock %}
|
||||
|
||||
{% block content %}
|
||||
<h1>Changelog</h1>
|
||||
|
||||
<section class="actions" aria-label="Changelog actions">
|
||||
<a class="button-link secondary" href="{% url 'dashboard' %}">Back to dashboard</a>
|
||||
</section>
|
||||
|
||||
<section class="panel">
|
||||
<div class="stack spaced">
|
||||
<div><strong>Installed version:</strong> {{ app_version }}</div>
|
||||
<div class="muted">Source: {{ changelog_path }}</div>
|
||||
{% if missing %}
|
||||
<div class="status warning">missing</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
|
||||
<div class="stack">
|
||||
{% for block in changelog_blocks %}
|
||||
{% if block.kind == "heading" %}
|
||||
{% if block.level == 1 %}
|
||||
<h2>{{ block.text }}</h2>
|
||||
{% else %}
|
||||
<h3>{{ block.text }}</h3>
|
||||
{% endif %}
|
||||
{% elif block.kind == "list" %}
|
||||
<ul>
|
||||
{% for item in block.items %}
|
||||
<li>{{ item }}</li>
|
||||
{% endfor %}
|
||||
</ul>
|
||||
{% else %}
|
||||
<p>{{ block.text }}</p>
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
</div>
|
||||
</section>
|
||||
{% endblock %}
|
||||
@@ -32,22 +32,109 @@
|
||||
<div class="metric"><div class="label">Schedules</div><div class="value">{{ counts.enabled_schedules }}/{{ counts.schedules }}</div></div>
|
||||
<div class="metric"><div class="label">Snapshots</div><div class="value">{{ counts.snapshots }}</div></div>
|
||||
<div class="metric"><div class="label">Runs</div><div class="value">{{ counts.runs }}</div></div>
|
||||
<div class="metric"><div class="label">Running</div><div class="value">{{ counts.running_runs }}</div></div>
|
||||
<div class="metric"><div class="label">Failed</div><div class="value">{{ counts.failed_runs }}</div></div>
|
||||
<div class="metric {% if counts.queued_runs %}queued{% endif %}"><div class="label">Queued</div><div class="value">{{ counts.queued_runs }}</div></div>
|
||||
<div class="metric {% if counts.running_runs %}running{% endif %}"><div class="label">Running</div><div class="value">{{ counts.running_runs }}</div></div>
|
||||
<div class="metric {% if counts.warning_runs %}warning{% endif %}"><div class="label">Warnings</div><div class="value">{{ counts.warning_runs }}</div></div>
|
||||
<div class="metric {% if counts.failed_runs %}failed{% endif %}"><div class="label">Failed</div><div class="value">{{ counts.failed_runs }}</div></div>
|
||||
</section>
|
||||
|
||||
{% if stats_summary.runs_sampled %}
|
||||
<section class="grid" aria-label="Backup trends">
|
||||
<div class="metric"><div class="label">Backup Root Used</div><div class="value">{{ stats_summary.capacity.used_percent|default:"" }}{% if stats_summary.capacity.used_percent is not None %}%{% endif %}</div></div>
|
||||
<div class="metric"><div class="label">Available</div><div class="value">{{ stats_summary.capacity.available_bytes|filesizeformat }}</div></div>
|
||||
<div class="metric"><div class="label">Avg New Data</div><div class="value">{{ stats_summary.avg_literal_data_bytes|filesizeformat }}</div></div>
|
||||
<div class="metric"><div class="label">Avg Daily New</div><div class="value">{{ stats_summary.avg_daily_literal_data_bytes|filesizeformat }}</div></div>
|
||||
<div class="metric"><div class="label">Avg Duration</div><div class="value">{{ stats_summary.avg_duration_seconds|default:"" }}{% if stats_summary.avg_duration_seconds is not None %}s{% endif %}</div></div>
|
||||
<div class="metric"><div class="label">Link-Dest Savings</div><div class="value">{{ stats_summary.link_dest_savings_ratio|default:"" }}</div></div>
|
||||
<div class="metric"><div class="label">Runs Until Full</div><div class="value">{{ stats_summary.estimated_runs_until_full|default:"" }}</div></div>
|
||||
<div class="metric"><div class="label">Days Until Full</div><div class="value">{{ stats_summary.estimated_days_until_full|default:"" }}</div></div>
|
||||
</section>
|
||||
{% endif %}
|
||||
<section class="panel">
|
||||
<h2>Operational Status</h2>
|
||||
{% if counts.failed_runs or counts.warning_runs or counts.running_runs or counts.queued_runs %}
|
||||
<div class="status-overview">
|
||||
{% if counts.failed_runs %}
|
||||
<div class="status-summary failed">
|
||||
<span class="status failed">failed</span>
|
||||
<strong>{{ counts.failed_runs }} failed run{{ counts.failed_runs|pluralize }} need{{ counts.failed_runs|pluralize:"s," }} review.</strong>
|
||||
</div>
|
||||
{% endif %}
|
||||
{% if counts.warning_runs %}
|
||||
<div class="status-summary warning">
|
||||
<span class="status warning">warning</span>
|
||||
<strong>{{ counts.warning_runs }} run{{ counts.warning_runs|pluralize }} completed with warnings.</strong>
|
||||
</div>
|
||||
{% endif %}
|
||||
{% if counts.running_runs %}
|
||||
<div class="status-summary running">
|
||||
<span class="status running">running</span>
|
||||
<strong>{{ counts.running_runs }} backup run{{ counts.running_runs|pluralize }} in progress.</strong>
|
||||
</div>
|
||||
{% endif %}
|
||||
{% if counts.queued_runs %}
|
||||
<div class="status-summary queued">
|
||||
<span class="status queued">queued</span>
|
||||
<strong>{{ counts.queued_runs }} backup run{{ counts.queued_runs|pluralize }} waiting for the worker.</strong>
|
||||
</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% elif counts.hosts %}
|
||||
<p><span class="status ok">ok</span> No queued, running, or unreviewed warning/failed runs.</p>
|
||||
{% else %}
|
||||
<p class="muted">Add a host to start tracking backup status here.</p>
|
||||
{% endif %}
|
||||
</section>
|
||||
|
||||
<section class="panel">
|
||||
<h2>Backup Trends</h2>
|
||||
{% if stats_summary.runs_sampled %}
|
||||
<div class="insight-grid" aria-label="Backup trends">
|
||||
<div class="insight-main">
|
||||
<div class="label">Storage Used</div>
|
||||
<div class="value">
|
||||
{% if stats_summary.capacity.used_percent is not None %}
|
||||
{{ stats_summary.capacity.used_percent|floatformat:1 }}%
|
||||
{% else %}
|
||||
unknown
|
||||
{% endif %}
|
||||
</div>
|
||||
{% if stats_summary.capacity.used_percent is not None %}
|
||||
<div class="storage-meter" aria-label="Backup root storage usage">
|
||||
<span style="width: {{ stats_summary.capacity.used_percent|floatformat:0 }}%"></span>
|
||||
</div>
|
||||
{% endif %}
|
||||
<div class="muted">
|
||||
{{ stats_summary.capacity.available_bytes|filesizeformat }} available from the backup root.
|
||||
</div>
|
||||
</div>
|
||||
<div class="insight-item">
|
||||
<div class="label">Runway</div>
|
||||
<div class="value">
|
||||
{% if stats_summary.estimated_days_until_full %}
|
||||
{{ stats_summary.estimated_days_until_full }} days
|
||||
{% elif stats_summary.estimated_runs_until_full %}
|
||||
{{ stats_summary.estimated_runs_until_full }} runs
|
||||
{% else %}
|
||||
unknown
|
||||
{% endif %}
|
||||
</div>
|
||||
<div class="muted">Estimated from average new data per day.</div>
|
||||
</div>
|
||||
<div class="insight-item">
|
||||
<div class="label">New Data</div>
|
||||
<div class="value">{{ stats_summary.avg_daily_literal_data_bytes|filesizeformat }}/day</div>
|
||||
<div class="muted">{{ stats_summary.avg_literal_data_bytes|filesizeformat }} per backup on average.</div>
|
||||
</div>
|
||||
<div class="insight-item">
|
||||
<div class="label">Link-Dest Savings</div>
|
||||
<div class="value">
|
||||
{% if stats_summary.link_dest_savings_percent is not None %}
|
||||
{{ stats_summary.link_dest_savings_percent|floatformat:1 }}%
|
||||
{% else %}
|
||||
unknown
|
||||
{% endif %}
|
||||
</div>
|
||||
<div class="muted">{{ stats_summary.total_matched_data_bytes|filesizeformat }} reused across sampled runs.</div>
|
||||
</div>
|
||||
<div class="insight-item">
|
||||
<div class="label">Average Duration</div>
|
||||
<div class="value">{{ stats_summary.avg_duration_seconds|default:"unknown" }}{% if stats_summary.avg_duration_seconds is not None %}s{% endif %}</div>
|
||||
<div class="muted">Based on {{ stats_summary.runs_sampled }} completed backup run{{ stats_summary.runs_sampled|pluralize }} with stats.</div>
|
||||
</div>
|
||||
</div>
|
||||
{% else %}
|
||||
<p class="muted">No completed backup runs with stats yet. This section will show disk usage, growth estimates, and link-dest savings after the first real backup finishes.</p>
|
||||
{% endif %}
|
||||
</section>
|
||||
|
||||
<section class="panel">
|
||||
<h2>Hosts</h2>
|
||||
@@ -61,63 +148,111 @@
|
||||
</div>
|
||||
<div class="host-card-status">
|
||||
<span class="status {% if host.enabled %}ok{% else %}skipped{% endif %}">{{ host.enabled|yesno:"enabled,disabled" }}</span>
|
||||
{% if host.queued_run_count %}
|
||||
<span class="status queued">queued {{ host.queued_run_count }}</span>
|
||||
{% endif %}
|
||||
{% if host.running_run_count %}
|
||||
<span class="status running">running {{ host.running_run_count }}</span>
|
||||
{% endif %}
|
||||
{% if host.warning_run_count %}
|
||||
<span class="status warning">warning {{ host.warning_run_count }}</span>
|
||||
{% endif %}
|
||||
{% if host.failed_run_count %}
|
||||
<span class="status failed">failed {{ host.failed_run_count }}</span>
|
||||
{% endif %}
|
||||
</div>
|
||||
</div>
|
||||
<div class="host-card-layout">
|
||||
<div class="host-card-timeline">
|
||||
<div class="host-card-item">
|
||||
<div class="label">Latest Snapshot</div>
|
||||
<div class="value">
|
||||
{% if host.latest_snapshot %}
|
||||
<a href="{% url 'snapshot_detail' host.latest_snapshot.id %}">{{ host.latest_snapshot.dirname }}</a>
|
||||
<div class="muted">{{ host.latest_snapshot.kind }} {{ host.latest_snapshot.status }}</div>
|
||||
{% else %}
|
||||
<span class="muted">none</span>
|
||||
{% endif %}
|
||||
<div class="host-card-section">
|
||||
<div class="host-card-section-title">Backup activity</div>
|
||||
<div class="host-card-timeline">
|
||||
<div class="host-card-item">
|
||||
<div class="label">Latest Snapshot</div>
|
||||
<div class="value">
|
||||
{% if host.latest_snapshot %}
|
||||
<a href="{% url 'snapshot_detail' host.latest_snapshot.id %}">{{ host.latest_snapshot.dirname }}</a>
|
||||
<div class="muted">{{ host.latest_snapshot.kind }} {{ host.latest_snapshot.status }}</div>
|
||||
{% else %}
|
||||
<span class="muted">none</span>
|
||||
{% endif %}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="host-card-item">
|
||||
<div class="label">Latest Run</div>
|
||||
<div class="value">
|
||||
{% if host.stats_summary.latest_run.id %}
|
||||
<a href="{% url 'run_detail' host.stats_summary.latest_run.id %}">Run {{ host.stats_summary.latest_run.id }}</a>
|
||||
<div class="muted">{{ host.stats_summary.latest_run.run_type }} {{ host.stats_summary.latest_run.duration_seconds|default:"" }}{% if host.stats_summary.latest_run.duration_seconds is not None %}s{% endif %}</div>
|
||||
{% else %}
|
||||
<span class="muted">none</span>
|
||||
{% endif %}
|
||||
<div class="host-card-item">
|
||||
<div class="label">Last Good Backup</div>
|
||||
<div class="value">
|
||||
{% if host.stats_summary.latest_good_run.id %}
|
||||
<a href="{% url 'run_detail' host.stats_summary.latest_good_run.id %}">Run {{ host.stats_summary.latest_good_run.id }}</a>
|
||||
<div class="muted">{{ host.stats_summary.latest_good_run.run_type }} {{ host.stats_summary.latest_good_run.duration_seconds|default:"" }}{% if host.stats_summary.latest_good_run.duration_seconds is not None %}s{% endif %}</div>
|
||||
{% else %}
|
||||
<span class="muted">none</span>
|
||||
{% endif %}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="host-card-item">
|
||||
<div class="label">Next Run</div>
|
||||
<div class="value">
|
||||
{% if host.next_run_at %}
|
||||
{{ host.next_run_at|date:"Y-m-d H:i T" }}
|
||||
<div class="muted">{{ scheduler_timezone }}</div>
|
||||
{% else %}
|
||||
<span class="muted">none</span>
|
||||
{% endif %}
|
||||
<div class="host-card-item">
|
||||
<div class="label">Latest Issue</div>
|
||||
<div class="value">
|
||||
{% if host.stats_summary.latest_problem_run.id %}
|
||||
<a href="{% url 'run_detail' host.stats_summary.latest_problem_run.id %}">Run {{ host.stats_summary.latest_problem_run.id }}</a>
|
||||
<div><span class="status {{ host.stats_summary.latest_problem_run.status }}">{{ host.stats_summary.latest_problem_run.status }}</span></div>
|
||||
<div class="muted">{{ host.stats_summary.latest_problem_run.run_type }} {{ host.stats_summary.latest_problem_run.duration_seconds|default:"" }}{% if host.stats_summary.latest_problem_run.duration_seconds is not None %}s{% endif %}</div>
|
||||
{% else %}
|
||||
<span class="muted">none</span>
|
||||
{% endif %}
|
||||
</div>
|
||||
</div>
|
||||
<div class="host-card-item">
|
||||
<div class="label">Next Run</div>
|
||||
<div class="value">
|
||||
{% if host.next_run_at %}
|
||||
{{ host.next_run_at|date:"Y-m-d H:i T" }}
|
||||
<div class="muted">{{ scheduler_timezone }}</div>
|
||||
{% else %}
|
||||
<span class="muted">none</span>
|
||||
{% endif %}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="host-card-stats">
|
||||
<div class="host-card-stat">
|
||||
<div class="label">Snapshots</div>
|
||||
<div class="value">{{ host.snapshot_count }}</div>
|
||||
</div>
|
||||
<div class="host-card-stat">
|
||||
<div class="label">Runs</div>
|
||||
<div class="value">{{ host.run_count }}</div>
|
||||
</div>
|
||||
<div class="host-card-stat">
|
||||
<div class="label">New Data</div>
|
||||
<div class="value">{{ host.stats_summary.latest_run.rsync.literal_data_bytes|filesizeformat }}</div>
|
||||
</div>
|
||||
<div class="host-card-stat">
|
||||
<div class="label">Retention</div>
|
||||
<div class="value">d{{ host.retention_daily }} w{{ host.retention_weekly }} m{{ host.retention_monthly }} y{{ host.retention_yearly }}</div>
|
||||
<div class="host-card-section">
|
||||
<div class="host-card-section-title">Snapshot health</div>
|
||||
<div class="host-card-stats">
|
||||
<div class="host-card-stat">
|
||||
<div class="label">Snapshots</div>
|
||||
<div class="value">{{ host.snapshot_count }}</div>
|
||||
</div>
|
||||
<div class="host-card-stat">
|
||||
<div class="label">Runs</div>
|
||||
<div class="value">{{ host.run_count }}</div>
|
||||
</div>
|
||||
<div class="host-card-stat">
|
||||
<div class="label">New Data</div>
|
||||
<div class="value">{{ host.stats_summary.latest_run.rsync.literal_data_bytes|filesizeformat }}</div>
|
||||
</div>
|
||||
<div class="host-card-stat">
|
||||
<div class="label">Retention</div>
|
||||
<div class="value">d{{ host.retention_daily }} w{{ host.retention_weekly }} m{{ host.retention_monthly }} y{{ host.retention_yearly }}</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
{% if host.retention_warning.has_warning %}
|
||||
<div class="host-card-warning">
|
||||
<span class="status warning">retention</span>
|
||||
{% if host.retention_warning.prune_exceeded %}
|
||||
Scheduled prune would delete {{ host.retention_warning.delete_count }} snapshot(s), above max {{ host.retention_warning.max_delete }}.
|
||||
{% endif %}
|
||||
{% if host.retention_warning.incomplete_count %}
|
||||
{{ host.retention_warning.incomplete_count }} incomplete snapshot(s) need review.
|
||||
<form method="post" action="{% url 'resolve_host_incomplete_reviews' host.host %}">
|
||||
{% csrf_token %}
|
||||
<button type="submit" class="secondary">Mark reviewed</button>
|
||||
</form>
|
||||
{% endif %}
|
||||
{% if host.retention_warning.error %}
|
||||
{{ host.retention_warning.error }}
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endif %}
|
||||
</article>
|
||||
{% empty %}
|
||||
<p class="muted">No hosts configured yet.</p>
|
||||
|
||||
@@ -68,6 +68,34 @@
|
||||
</section>
|
||||
</div>
|
||||
|
||||
{% if retention_warning.has_warning %}
|
||||
<section class="panel highlight warning">
|
||||
<h2>Retention Warnings</h2>
|
||||
<div class="stack">
|
||||
{% if retention_warning.prune_exceeded %}
|
||||
<div>
|
||||
Scheduled pruning would delete {{ retention_warning.delete_count }} snapshot(s), above max delete
|
||||
{{ retention_warning.max_delete }}. Scheduled pruning will refuse this plan until the limit or retention
|
||||
selection is adjusted.
|
||||
</div>
|
||||
{% endif %}
|
||||
{% if retention_warning.incomplete_count %}
|
||||
<div>
|
||||
{{ retention_warning.incomplete_count }} incomplete snapshot(s) exist. Retention does not delete incomplete
|
||||
snapshots automatically; inspect them before cleanup.
|
||||
</div>
|
||||
<form method="post" action="{% url 'resolve_host_incomplete_reviews' host.host %}">
|
||||
{% csrf_token %}
|
||||
<button type="submit" class="secondary">Mark incomplete reviewed</button>
|
||||
</form>
|
||||
{% endif %}
|
||||
{% if retention_warning.error %}
|
||||
<div>{{ retention_warning.error }}</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
</section>
|
||||
{% endif %}
|
||||
|
||||
{% if effective_config %}
|
||||
<section class="panel">
|
||||
<h2>Effective Config</h2>
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
{% extends "pobsync_backend/base.html" %}
|
||||
|
||||
{% block title %}Purged Snapshots | pobsync{% endblock %}
|
||||
|
||||
{% block content %}
|
||||
<h1>Purged Snapshots</h1>
|
||||
|
||||
<section class="actions" aria-label="Purged snapshot actions">
|
||||
<a class="button-link secondary" href="{% url 'dashboard' %}">Back to dashboard</a>
|
||||
</section>
|
||||
|
||||
<section class="panel">
|
||||
<h2>Filters</h2>
|
||||
<form method="get" class="form-grid">
|
||||
<div class="field">
|
||||
<label for="host">Host</label>
|
||||
<select id="host" name="host">
|
||||
<option value="">All hosts</option>
|
||||
{% for host in hosts %}
|
||||
<option value="{{ host.host }}" {% if selected_host == host.host %}selected{% endif %}>{{ host.host }}</option>
|
||||
{% endfor %}
|
||||
</select>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="action">Action</label>
|
||||
<select id="action" name="action">
|
||||
<option value="">All actions</option>
|
||||
{% for value, label in actions %}
|
||||
<option value="{{ value }}" {% if selected_action == value %}selected{% endif %}>{{ label }}</option>
|
||||
{% endfor %}
|
||||
</select>
|
||||
</div>
|
||||
<div class="actions">
|
||||
<button type="submit">Apply filters</button>
|
||||
<a class="button-link secondary" href="{% url 'purged_snapshots' %}">Clear</a>
|
||||
</div>
|
||||
</form>
|
||||
</section>
|
||||
|
||||
<section class="panel">
|
||||
<h2>Purged Snapshot History</h2>
|
||||
<p class="muted">Showing up to 200 of {{ total_count }} purged snapshot record(s).</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Purged</th>
|
||||
<th>Host</th>
|
||||
<th>Kind</th>
|
||||
<th>Dirname</th>
|
||||
<th>Action</th>
|
||||
<th>Reason</th>
|
||||
<th>Triggered by</th>
|
||||
<th>Path</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for snapshot in purged_snapshots %}
|
||||
<tr>
|
||||
<td>{{ snapshot.purged_at }}</td>
|
||||
<td>{% if snapshot.host %}<a href="{% url 'host_detail' snapshot.host.host %}">{{ snapshot.host_name }}</a>{% else %}{{ snapshot.host_name }}{% endif %}</td>
|
||||
<td>{{ snapshot.kind }}</td>
|
||||
<td>{{ snapshot.dirname }}</td>
|
||||
<td><span class="status skipped">{{ snapshot.get_action_display }}</span></td>
|
||||
<td>{{ snapshot.reason|default:"" }}</td>
|
||||
<td>{{ snapshot.triggered_by|default:"" }}</td>
|
||||
<td class="muted">{{ snapshot.path }}</td>
|
||||
</tr>
|
||||
{% empty %}
|
||||
<tr><td colspan="8" class="muted">No purged snapshots recorded yet.</td></tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</section>
|
||||
{% endblock %}
|
||||
@@ -18,8 +18,35 @@
|
||||
<div class="metric"><div class="label">Kind</div><div class="value">{{ plan.kind }}</div></div>
|
||||
<div class="metric"><div class="label">Keep</div><div class="value">{{ plan.keep|length }}</div></div>
|
||||
<div class="metric"><div class="label">Would Delete</div><div class="value">{{ plan.delete|length }}</div></div>
|
||||
<div class="metric"><div class="label">Scheduled Limit</div><div class="value">{{ scheduled_prune_limit|default:"none" }}</div></div>
|
||||
<div class="metric"><div class="label">Incomplete</div><div class="value">{{ plan.incomplete|length }}</div></div>
|
||||
</section>
|
||||
|
||||
{% if scheduled_prune_exceeded %}
|
||||
<section class="panel highlight warning">
|
||||
<h2>Scheduled Prune Limit</h2>
|
||||
<p>
|
||||
This plan would delete {{ plan.delete|length }} snapshot(s), which exceeds the scheduled prune limit of
|
||||
{{ scheduled_prune_limit }}. Scheduled pruning will refuse to apply this plan until the limit or retention
|
||||
selection is adjusted.
|
||||
</p>
|
||||
</section>
|
||||
{% endif %}
|
||||
|
||||
{% if plan.incomplete %}
|
||||
<section class="panel highlight warning">
|
||||
<h2>Incomplete Snapshots</h2>
|
||||
<p>
|
||||
{{ plan.incomplete|length }} incomplete snapshot(s) exist for this host. Retention does not delete incomplete
|
||||
snapshots automatically because they can indicate an interrupted backup that should be inspected first.
|
||||
</p>
|
||||
<p>
|
||||
After inspection, use the dedicated cleanup form below to delete only incomplete snapshot directories and their
|
||||
SQL records. Successful scheduled and manual snapshots are not touched by this cleanup.
|
||||
</p>
|
||||
</section>
|
||||
{% endif %}
|
||||
|
||||
<section class="panel">
|
||||
<h2>Policy</h2>
|
||||
<div class="stack">
|
||||
@@ -28,6 +55,17 @@
|
||||
<div><strong>Monthly:</strong> {{ plan.retention.monthly }}</div>
|
||||
<div><strong>Yearly:</strong> {{ plan.retention.yearly }}</div>
|
||||
<div><strong>Protect bases:</strong> {{ protect_bases|yesno:"yes,no" }}</div>
|
||||
<div class="muted">
|
||||
{% if protect_bases %}
|
||||
Base snapshots referenced by kept snapshots are also kept and marked with a base-of reason.
|
||||
{% else %}
|
||||
Base snapshots are only kept when they match the regular retention policy.
|
||||
{% endif %}
|
||||
</div>
|
||||
{% if schedule %}
|
||||
<div><strong>Schedule pruning:</strong> {{ schedule.prune|yesno:"enabled,disabled" }}</div>
|
||||
<div><strong>Schedule max delete:</strong> {{ schedule.prune_max_delete }}</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -40,6 +78,7 @@
|
||||
<th>Dirname</th>
|
||||
<th>Started</th>
|
||||
<th>Status</th>
|
||||
<th>Reason</th>
|
||||
<th>Path</th>
|
||||
</tr>
|
||||
</thead>
|
||||
@@ -50,10 +89,11 @@
|
||||
<td>{{ snapshot.dirname }}</td>
|
||||
<td>{{ snapshot.dt }}</td>
|
||||
<td>{{ snapshot.status|default:"" }}</td>
|
||||
<td>{{ snapshot.reason }}</td>
|
||||
<td class="muted">{{ snapshot.path }}</td>
|
||||
</tr>
|
||||
{% empty %}
|
||||
<tr><td colspan="5" class="muted">Retention would not delete snapshots for this selection.</td></tr>
|
||||
<tr><td colspan="6" class="muted">Retention would not delete snapshots for this selection.</td></tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
@@ -71,7 +111,7 @@
|
||||
{{ apply_form.max_delete.errors }}
|
||||
<label for="{{ apply_form.max_delete.id_for_label }}">Max delete</label>
|
||||
{{ apply_form.max_delete }}
|
||||
<div class="helptext">Must be at least the number of snapshots shown in Would Delete.</div>
|
||||
<div class="helptext">Must be at least {{ plan.delete|length }} for the snapshots shown in Would Delete.</div>
|
||||
</div>
|
||||
|
||||
<div class="field">
|
||||
@@ -87,6 +127,13 @@
|
||||
<div class="helptext">{{ apply_form.confirm_host.help_text }}</div>
|
||||
</div>
|
||||
|
||||
<div class="field">
|
||||
{{ apply_form.confirm_delete_count.errors }}
|
||||
<label for="{{ apply_form.confirm_delete_count.id_for_label }}">Confirm delete count</label>
|
||||
{{ apply_form.confirm_delete_count }}
|
||||
<div class="helptext">{{ apply_form.confirm_delete_count.help_text }}</div>
|
||||
</div>
|
||||
|
||||
<div class="actions">
|
||||
<button type="submit">Apply retention</button>
|
||||
</div>
|
||||
@@ -99,20 +146,85 @@
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Kind</th>
|
||||
<th>Dirname</th>
|
||||
<th>Started</th>
|
||||
<th>Status</th>
|
||||
<th>Reasons</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for dirname, reasons in plan.reasons.items %}
|
||||
{% for snapshot in plan.keep_items %}
|
||||
<tr>
|
||||
<td>{{ dirname }}</td>
|
||||
<td>{{ reasons|join:", " }}</td>
|
||||
<td>{{ snapshot.kind }}</td>
|
||||
<td>{{ snapshot.dirname }}</td>
|
||||
<td>{{ snapshot.dt }}</td>
|
||||
<td>{{ snapshot.status|default:"" }}</td>
|
||||
<td>{{ snapshot.reason }}</td>
|
||||
</tr>
|
||||
{% empty %}
|
||||
<tr><td colspan="2" class="muted">No snapshots matched this retention selection.</td></tr>
|
||||
<tr><td colspan="5" class="muted">No snapshots matched this retention selection.</td></tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</section>
|
||||
|
||||
{% if plan.incomplete %}
|
||||
<section class="panel">
|
||||
<h2>Incomplete Snapshots</h2>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Dirname</th>
|
||||
<th>Started</th>
|
||||
<th>Status</th>
|
||||
<th>Reason</th>
|
||||
<th>Path</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for snapshot in plan.incomplete %}
|
||||
<tr>
|
||||
<td>{{ snapshot.dirname }}</td>
|
||||
<td>{{ snapshot.dt }}</td>
|
||||
<td>{{ snapshot.status|default:"" }}</td>
|
||||
<td>{{ snapshot.reason }}</td>
|
||||
<td class="muted">{{ snapshot.path }}</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h3>Cleanup Incomplete Snapshots</h3>
|
||||
<form method="post" action="{% url 'cleanup_host_incomplete_snapshots' host.host %}" class="form-grid">
|
||||
{% csrf_token %}
|
||||
{{ incomplete_cleanup_form.non_field_errors }}
|
||||
|
||||
<div class="field">
|
||||
{{ incomplete_cleanup_form.max_delete.errors }}
|
||||
<label for="{{ incomplete_cleanup_form.max_delete.id_for_label }}">Max delete</label>
|
||||
{{ incomplete_cleanup_form.max_delete }}
|
||||
<div class="helptext">{{ incomplete_cleanup_form.max_delete.help_text }}</div>
|
||||
</div>
|
||||
|
||||
<div class="field">
|
||||
{{ incomplete_cleanup_form.confirm_host.errors }}
|
||||
<label for="{{ incomplete_cleanup_form.confirm_host.id_for_label }}">Confirm host</label>
|
||||
{{ incomplete_cleanup_form.confirm_host }}
|
||||
<div class="helptext">{{ incomplete_cleanup_form.confirm_host.help_text }}</div>
|
||||
</div>
|
||||
|
||||
<div class="field">
|
||||
{{ incomplete_cleanup_form.confirm_delete_count.errors }}
|
||||
<label for="{{ incomplete_cleanup_form.confirm_delete_count.id_for_label }}">Confirm incomplete count</label>
|
||||
{{ incomplete_cleanup_form.confirm_delete_count }}
|
||||
<div class="helptext">{{ incomplete_cleanup_form.confirm_delete_count.help_text }}</div>
|
||||
</div>
|
||||
|
||||
<div class="actions">
|
||||
<button type="submit">Delete incomplete snapshots</button>
|
||||
</div>
|
||||
</form>
|
||||
</section>
|
||||
{% endif %}
|
||||
{% endblock %}
|
||||
|
||||
@@ -13,6 +13,14 @@
|
||||
<button type="submit" class="secondary">Cancel run</button>
|
||||
</form>
|
||||
{% endif %}
|
||||
{% if run.status == "failed" or run.status == "warning" %}
|
||||
{% if not run.reviewed_at %}
|
||||
<form method="post" action="{% url 'resolve_run_review' run.id %}">
|
||||
{% csrf_token %}
|
||||
<button type="submit" class="secondary">Mark reviewed</button>
|
||||
</form>
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
</section>
|
||||
|
||||
<section class="grid" aria-label="Run summary">
|
||||
@@ -33,6 +41,16 @@
|
||||
</section>
|
||||
{% endif %}
|
||||
|
||||
{% if run.reviewed_at %}
|
||||
<section class="panel highlight success">
|
||||
<h2>Review</h2>
|
||||
<div class="stack">
|
||||
<div><strong>Reviewed:</strong> {{ run.reviewed_at }}</div>
|
||||
<div><strong>Reviewed by:</strong> {{ run.reviewed_by|default:"unknown" }}</div>
|
||||
</div>
|
||||
</section>
|
||||
{% endif %}
|
||||
|
||||
{% if dry_run_summary %}
|
||||
<section class="panel highlight {{ dry_run_summary.highlight_class }}">
|
||||
<h2>Dry Run Summary</h2>
|
||||
@@ -79,6 +97,10 @@
|
||||
<div><strong>Created:</strong> {{ run.created_at }}</div>
|
||||
<div><strong>Started:</strong> {{ run.started_at|default:"" }}</div>
|
||||
<div><strong>Ended:</strong> {{ run.ended_at|default:"" }}</div>
|
||||
{% if execution %}
|
||||
<div><strong>Worker:</strong> {{ execution.worker_host|default:"unknown" }}{% if execution.worker_pid %} pid {{ execution.worker_pid }}{% endif %}</div>
|
||||
<div><strong>Worker heartbeat:</strong> {{ execution.heartbeat_at|default:"" }}</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -172,13 +194,49 @@
|
||||
<div class="stack">
|
||||
<div><strong>Status:</strong> {% if prune_result.ok %}ok{% else %}warning{% endif %}</div>
|
||||
{% if prune_result.source %}<div><strong>Source:</strong> {{ prune_result.source }}</div>{% endif %}
|
||||
{% if prune_result.kind %}<div><strong>Kind:</strong> {{ prune_result.kind }}</div>{% endif %}
|
||||
{% if prune_result.planned_delete_count is not None %}<div><strong>Planned deletions:</strong> {{ prune_result.planned_delete_count }}</div>{% endif %}
|
||||
{% if prune_result.deleted %}<div><strong>Deleted:</strong> {{ prune_result.deleted|length }}</div>{% endif %}
|
||||
{% if prune_result.max_delete is not None %}<div><strong>Max delete:</strong> {{ prune_result.max_delete }}</div>{% endif %}
|
||||
{% if prune_result.protect_bases is not None %}<div><strong>Protect bases:</strong> {{ prune_result.protect_bases|yesno:"yes,no" }}</div>{% endif %}
|
||||
{% if prune_result.incomplete_ignored_count %}<div><strong>Incomplete ignored:</strong> {{ prune_result.incomplete_ignored_count }}</div>{% endif %}
|
||||
{% if prune_result.actions %}
|
||||
<div><strong>Actions:</strong></div>
|
||||
<ul>
|
||||
{% for action in prune_result.actions %}
|
||||
<li>{{ action }}</li>
|
||||
{% endfor %}
|
||||
</ul>
|
||||
{% endif %}
|
||||
{% if prune_result.error %}<div><strong>Error:</strong> {{ prune_result.error }}</div>{% endif %}
|
||||
{% if prune_result.type %}<div><strong>Type:</strong> {{ prune_result.type }}</div>{% endif %}
|
||||
</div>
|
||||
</section>
|
||||
{% endif %}
|
||||
|
||||
{% if retention_warning.has_warning %}
|
||||
<section class="panel highlight warning">
|
||||
<h2>Retention Warnings</h2>
|
||||
<div class="stack">
|
||||
{% if retention_warning.prune_exceeded %}
|
||||
<div>
|
||||
Scheduled pruning for this host would delete {{ retention_warning.delete_count }} snapshot(s), above max
|
||||
delete {{ retention_warning.max_delete }}.
|
||||
</div>
|
||||
{% endif %}
|
||||
{% if retention_warning.incomplete_count %}
|
||||
<div>
|
||||
{{ retention_warning.incomplete_count }} incomplete snapshot(s) exist for this host and are excluded from
|
||||
retention cleanup.
|
||||
</div>
|
||||
{% endif %}
|
||||
{% if retention_warning.error %}
|
||||
<div>{{ retention_warning.error }}</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
</section>
|
||||
{% endif %}
|
||||
|
||||
<section class="panel">
|
||||
<h2>Raw Result</h2>
|
||||
<pre>{{ result_json }}</pre>
|
||||
|
||||
@@ -60,6 +60,48 @@
|
||||
</section>
|
||||
{% endif %}
|
||||
|
||||
<section class="panel">
|
||||
<h2>Restore Guidance</h2>
|
||||
<div class="stack spaced">
|
||||
<div><strong>Snapshot data source:</strong> {{ restore.source_path }}</div>
|
||||
<div><strong>Example staging destination:</strong> {{ restore.destination_path }}</div>
|
||||
<div class="muted">
|
||||
Restore from the snapshot's <code>data/</code> directory. Start with a dry run, restore to a staging path first,
|
||||
and only then copy data back to a live host or service path.
|
||||
</div>
|
||||
</div>
|
||||
<div class="stack spaced">
|
||||
<div><strong>Inspect the snapshot:</strong></div>
|
||||
<pre>{{ restore.inspect_command }}</pre>
|
||||
</div>
|
||||
<div class="stack spaced">
|
||||
<div><strong>Dry-run restore to staging:</strong></div>
|
||||
<pre>{{ restore.dry_run_command }}</pre>
|
||||
</div>
|
||||
<div class="stack spaced">
|
||||
<div><strong>Restore to staging:</strong></div>
|
||||
<pre>{{ restore.local_command }}</pre>
|
||||
</div>
|
||||
<div class="stack spaced">
|
||||
<div><strong>Dry-run a directory restore:</strong></div>
|
||||
<pre>{{ restore.partial_dry_run_command }}</pre>
|
||||
<div class="muted">Replace <code>{{ restore.example_relative_path }}</code> with the path you want to restore.</div>
|
||||
</div>
|
||||
<div class="stack spaced">
|
||||
<div><strong>Dry-run a single file restore:</strong></div>
|
||||
<pre>{{ restore.file_dry_run_command }}</pre>
|
||||
<div class="muted">Replace <code>{{ restore.example_file_relative_path }}</code> with the file you want to restore.</div>
|
||||
</div>
|
||||
<div class="stack spaced">
|
||||
<div><strong>Dry-run restore back to the source host:</strong></div>
|
||||
<pre>{{ restore.remote_dry_run_command }}</pre>
|
||||
</div>
|
||||
<p class="muted">
|
||||
Snapshots can contain hardlinks to files shared with earlier snapshots. Treat snapshot directories as read-only:
|
||||
copy data out with rsync instead of editing files in place.
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="panel">
|
||||
<h2>Backup Runs</h2>
|
||||
<table>
|
||||
|
||||
@@ -45,9 +45,21 @@
|
||||
{% if credential %}
|
||||
<section class="panel">
|
||||
<h2>Delete SSH Key</h2>
|
||||
{% if credential.hosts.exists or credential.global_configs.exists %}
|
||||
<p class="muted">
|
||||
This SSH key is still selected by {{ credential.hosts.count }} host(s) or
|
||||
{{ credential.global_configs.count }} global config(s). Select another key there before deleting it.
|
||||
</p>
|
||||
{% else %}
|
||||
<p class="muted">Type <strong>{{ credential.name }}</strong> to confirm deletion.</p>
|
||||
{% endif %}
|
||||
<form method="post" action="{% url 'delete_ssh_credential' credential.id %}">
|
||||
{% csrf_token %}
|
||||
<button type="submit" class="danger">Delete SSH key</button>
|
||||
<div class="field">
|
||||
<label for="confirm_name">Confirm key name</label>
|
||||
<input id="confirm_name" name="confirm_name" type="text" {% if credential.hosts.exists or credential.global_configs.exists %}disabled{% endif %}>
|
||||
</div>
|
||||
<button type="submit" class="danger" {% if credential.hosts.exists or credential.global_configs.exists %}disabled{% endif %}>Delete SSH key</button>
|
||||
</form>
|
||||
</section>
|
||||
{% endif %}
|
||||
|
||||
@@ -23,6 +23,7 @@
|
||||
<th>Known hosts</th>
|
||||
<th>Hosts</th>
|
||||
<th>Updated</th>
|
||||
<th>Actions</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
@@ -35,9 +36,10 @@
|
||||
<td>{{ credential.known_hosts|yesno:"yes,no" }}</td>
|
||||
<td>{{ credential.hosts.count }}</td>
|
||||
<td>{{ credential.updated_at }}</td>
|
||||
<td><a class="button-link secondary" href="{% url 'edit_ssh_credential' credential.id %}">Edit</a></td>
|
||||
</tr>
|
||||
{% empty %}
|
||||
<tr><td colspan="7" class="muted">No SSH credentials configured yet.</td></tr>
|
||||
<tr><td colspan="8" class="muted">No SSH credentials configured yet.</td></tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
@@ -5,11 +5,27 @@ from datetime import datetime, timezone
|
||||
from django.contrib.admin.sites import AdminSite
|
||||
from django.test import TestCase
|
||||
|
||||
from pobsync_backend.admin import BackupRunAdmin, HostConfigAdmin, SnapshotRecordAdmin
|
||||
from pobsync_backend.models import BackupRun, HostConfig, ScheduleConfig, SnapshotRecord
|
||||
from pobsync_backend.admin import BackupRunAdmin, GlobalConfigAdmin, HostConfigAdmin, SnapshotRecordAdmin
|
||||
from pobsync_backend.models import BackupRun, GlobalConfig, HostConfig, ScheduleConfig, SnapshotRecord
|
||||
|
||||
|
||||
class AdminDisplayTests(TestCase):
|
||||
def test_admin_hides_old_global_state_fields_and_labels_host_runtime_state(self) -> None:
|
||||
site = AdminSite()
|
||||
global_admin = GlobalConfigAdmin(GlobalConfig, site)
|
||||
host_admin = HostConfigAdmin(HostConfig, site)
|
||||
|
||||
global_fieldsets = list(global_admin.fieldsets)
|
||||
host_fieldsets = list(host_admin.fieldsets)
|
||||
global_fields = [field for _name, options in global_fieldsets for field in options["fields"]]
|
||||
fieldset_names = [name for name, _options in [*global_fieldsets, *host_fieldsets]]
|
||||
|
||||
self.assertNotIn("pobsync_home", global_fields)
|
||||
self.assertNotIn("data", global_fields)
|
||||
self.assertIn("Runtime state", fieldset_names)
|
||||
self.assertNotIn("Compatibility data", fieldset_names)
|
||||
self.assertNotIn("Legacy JSON", fieldset_names)
|
||||
|
||||
def test_host_admin_links_to_related_snapshots_and_runs(self) -> None:
|
||||
site = AdminSite()
|
||||
admin = HostConfigAdmin(HostConfig, site)
|
||||
|
||||
@@ -61,6 +61,9 @@ class BackupWorkerTests(TestCase):
|
||||
def fake_run_scheduled(**kwargs):
|
||||
run.refresh_from_db()
|
||||
self.assertIn("execution", run.result)
|
||||
self.assertIn("worker_pid", run.result["execution"])
|
||||
self.assertIn("worker_host", run.result["execution"])
|
||||
self.assertIn("heartbeat_at", run.result["execution"])
|
||||
return {
|
||||
"ok": True,
|
||||
"dry_run": False,
|
||||
@@ -82,6 +85,57 @@ class BackupWorkerTests(TestCase):
|
||||
self.assertEqual(SnapshotRecord.objects.count(), 1)
|
||||
self.assertEqual(run.snapshot, SnapshotRecord.objects.get())
|
||||
|
||||
def test_worker_refreshes_heartbeat_while_run_is_active(self) -> None:
|
||||
with TemporaryDirectory() as tmp:
|
||||
GlobalConfig.objects.create(name="default", backup_root=str(Path(tmp) / "backups"))
|
||||
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
run = queue_backup_run(host=host)
|
||||
|
||||
with patch("pobsync_backend.backup_runner.run_scheduled") as run_scheduled:
|
||||
def fake_run_scheduled(**kwargs):
|
||||
run.refresh_from_db()
|
||||
old_heartbeat = timezone.now() - timedelta(seconds=120)
|
||||
run.result["execution"]["heartbeat_at"] = old_heartbeat.isoformat()
|
||||
run.save(update_fields=["result"])
|
||||
|
||||
self.assertFalse(kwargs["cancel_check"]())
|
||||
run.refresh_from_db()
|
||||
self.assertGreater(
|
||||
timezone.datetime.fromisoformat(run.result["execution"]["heartbeat_at"]),
|
||||
old_heartbeat,
|
||||
)
|
||||
return {
|
||||
"ok": True,
|
||||
"dry_run": False,
|
||||
"host": host.host,
|
||||
"snapshot": "",
|
||||
"base": None,
|
||||
"rsync": {"exit_code": 0},
|
||||
}
|
||||
|
||||
run_scheduled.side_effect = fake_run_scheduled
|
||||
Command()._run_once(prefix=Path(tmp) / "home")
|
||||
|
||||
def test_worker_reconciles_stale_real_run_after_heartbeat_timeout(self) -> None:
|
||||
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
run = queue_backup_run(host=host)
|
||||
run.status = BackupRun.Status.RUNNING
|
||||
run.started_at = timezone.now() - timedelta(seconds=120)
|
||||
run.result["execution"] = {
|
||||
"worker_pid": 123,
|
||||
"worker_host": "backup",
|
||||
"heartbeat_at": (timezone.now() - timedelta(seconds=90)).isoformat(),
|
||||
}
|
||||
run.save(update_fields=["status", "started_at", "result"])
|
||||
|
||||
reconciled = reconcile_running_runs(stale_worker_seconds=30)
|
||||
|
||||
self.assertEqual(reconciled, 1)
|
||||
run.refresh_from_db()
|
||||
self.assertEqual(run.status, BackupRun.Status.FAILED)
|
||||
self.assertEqual(run.result["failure"]["category"], "worker")
|
||||
self.assertIn("heartbeat stopped", run.result["failure"]["message"])
|
||||
|
||||
def test_worker_records_dry_run_log_path_while_running(self) -> None:
|
||||
with TemporaryDirectory() as tmp:
|
||||
GlobalConfig.objects.create(name="default", backup_root=str(Path(tmp) / "backups"))
|
||||
|
||||
@@ -1,71 +1,63 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from django.test import TestCase
|
||||
|
||||
from pobsync.config.load import load_global_config, load_host_config
|
||||
from pobsync_backend.config_repository import export_runtime_configs
|
||||
from pobsync_backend.config_repository import ConfigRepositoryError, global_config_data, host_config_data
|
||||
from pobsync_backend.models import GlobalConfig, HostConfig
|
||||
|
||||
|
||||
class ConfigRepositoryTests(TestCase):
|
||||
def test_exports_database_configs_to_engine_yaml(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
prefix = Path(tmp)
|
||||
GlobalConfig.objects.create(
|
||||
name="default",
|
||||
backup_root="/backups",
|
||||
pobsync_home=str(prefix),
|
||||
ssh_user="backup",
|
||||
ssh_port=2222,
|
||||
rsync_args=["--archive"],
|
||||
excludes_default=["/proc/***"],
|
||||
retention_daily=7,
|
||||
retention_weekly=4,
|
||||
retention_monthly=3,
|
||||
retention_yearly=1,
|
||||
data={
|
||||
"backup_root": "/ignored",
|
||||
"pobsync_home": "/ignored",
|
||||
"ssh": {"user": "ignored", "port": 22, "options": []},
|
||||
"unknown": "must-not-leak",
|
||||
"retention_defaults": {"daily": 99, "weekly": 99, "monthly": 99, "yearly": 99},
|
||||
},
|
||||
)
|
||||
HostConfig.objects.create(
|
||||
host="web-01",
|
||||
address="web-01.example.test",
|
||||
ssh_user="root",
|
||||
includes=[],
|
||||
excludes_add=["/tmp/***"],
|
||||
retention_daily=7,
|
||||
retention_weekly=4,
|
||||
retention_monthly=3,
|
||||
retention_yearly=1,
|
||||
config={
|
||||
"host": "ignored",
|
||||
"address": "ignored",
|
||||
"retention": {"daily": 99, "weekly": 99, "monthly": 99, "yearly": 99},
|
||||
"excludes_add": ["/ignored/***"],
|
||||
"unknown": "must-not-leak",
|
||||
},
|
||||
)
|
||||
def test_builds_runtime_config_from_database_fields(self) -> None:
|
||||
GlobalConfig.objects.create(
|
||||
name="default",
|
||||
backup_root="/backups",
|
||||
ssh_user="backup",
|
||||
ssh_port=2222,
|
||||
rsync_args=["--archive"],
|
||||
excludes_default=["/proc/***"],
|
||||
retention_daily=7,
|
||||
retention_weekly=4,
|
||||
retention_monthly=3,
|
||||
retention_yearly=1,
|
||||
)
|
||||
HostConfig.objects.create(
|
||||
host="web-01",
|
||||
address="web-01.example.test",
|
||||
ssh_user="root",
|
||||
includes=[],
|
||||
excludes_add=["/tmp/***"],
|
||||
retention_daily=7,
|
||||
retention_weekly=4,
|
||||
retention_monthly=3,
|
||||
retention_yearly=1,
|
||||
config={
|
||||
"host": "ignored",
|
||||
"address": "ignored",
|
||||
"retention": {"daily": 99, "weekly": 99, "monthly": 99, "yearly": 99},
|
||||
"excludes_add": ["/ignored/***"],
|
||||
"unknown": "must-not-leak",
|
||||
},
|
||||
)
|
||||
|
||||
written = export_runtime_configs(prefix=prefix, host="web-01")
|
||||
global_cfg = global_config_data()
|
||||
host_cfg = host_config_data("web-01")
|
||||
|
||||
self.assertEqual(len(written), 2)
|
||||
global_cfg = load_global_config(prefix / "config" / "global.yaml")
|
||||
host_cfg = load_host_config(prefix / "config" / "hosts" / "web-01.yaml")
|
||||
self.assertEqual(global_cfg["backup_root"], "/backups")
|
||||
self.assertEqual(global_cfg["pobsync_home"], str(prefix))
|
||||
self.assertEqual(global_cfg["ssh"]["user"], "backup")
|
||||
self.assertEqual(global_cfg["ssh"]["port"], 2222)
|
||||
self.assertEqual(global_cfg["retention_defaults"]["daily"], 7)
|
||||
self.assertEqual(host_cfg["host"], "web-01")
|
||||
self.assertEqual(host_cfg["address"], "web-01.example.test")
|
||||
self.assertEqual(host_cfg["retention"]["daily"], 7)
|
||||
self.assertEqual(host_cfg["excludes_add"], ["/tmp/***"])
|
||||
self.assertNotIn("unknown", global_cfg)
|
||||
self.assertNotIn("unknown", host_cfg)
|
||||
self.assertEqual(global_cfg["backup_root"], "/backups")
|
||||
self.assertEqual(global_cfg["ssh"]["user"], "backup")
|
||||
self.assertEqual(global_cfg["ssh"]["port"], 2222)
|
||||
self.assertEqual(global_cfg["retention_defaults"]["daily"], 7)
|
||||
self.assertEqual(host_cfg["host"], "web-01")
|
||||
self.assertEqual(host_cfg["address"], "web-01.example.test")
|
||||
self.assertEqual(host_cfg["retention"]["daily"], 7)
|
||||
self.assertEqual(host_cfg["excludes_add"], ["/tmp/***"])
|
||||
self.assertNotIn("unknown", global_cfg)
|
||||
self.assertNotIn("unknown", host_cfg)
|
||||
|
||||
def test_missing_config_errors_use_operator_labels(self) -> None:
|
||||
with self.assertRaisesMessage(ConfigRepositoryError, "Missing global config 'default'"):
|
||||
global_config_data()
|
||||
|
||||
GlobalConfig.objects.create(name="default", backup_root="/backups")
|
||||
|
||||
with self.assertRaisesMessage(ConfigRepositoryError, "Missing enabled host 'web-01'"):
|
||||
host_config_data("web-01")
|
||||
|
||||
@@ -16,7 +16,6 @@ class ConfigureCommandsTests(TestCase):
|
||||
call_command(
|
||||
"configure_pobsync_global",
|
||||
backup_root="/backups",
|
||||
pobsync_home="/opt/pobsync",
|
||||
retention="daily=3,weekly=2,monthly=1,yearly=0",
|
||||
stdout=out,
|
||||
)
|
||||
@@ -24,7 +23,7 @@ class ConfigureCommandsTests(TestCase):
|
||||
config = GlobalConfig.objects.get(name="default")
|
||||
self.assertEqual(config.backup_root, "/backups")
|
||||
self.assertEqual(config.retention_daily, 3)
|
||||
self.assertIn("Created GlobalConfig", out.getvalue())
|
||||
self.assertIn("Created global config", out.getvalue())
|
||||
|
||||
def test_configure_host_uses_global_retention_defaults(self) -> None:
|
||||
GlobalConfig.objects.create(
|
||||
@@ -62,7 +61,7 @@ class ConfigureCommandsTests(TestCase):
|
||||
call_command(
|
||||
"configure_pobsync_schedule",
|
||||
host.host,
|
||||
cron="15 2 * * *",
|
||||
schedule_expression="15 2 * * *",
|
||||
prune=True,
|
||||
stdout=out,
|
||||
)
|
||||
|
||||
@@ -9,6 +9,14 @@ from pobsync.cli import main
|
||||
|
||||
|
||||
class ConsoleEntrypointTests(SimpleTestCase):
|
||||
def test_version_prints_package_version(self) -> None:
|
||||
stdout = StringIO()
|
||||
with patch("sys.stdout", stdout):
|
||||
exit_code = main(["--version"])
|
||||
|
||||
self.assertEqual(exit_code, 0)
|
||||
self.assertEqual(stdout.getvalue().strip(), "pobsync 1.0.0")
|
||||
|
||||
def test_maps_backup_alias_to_django_command(self) -> None:
|
||||
with patch("pobsync.cli.execute_from_command_line") as execute:
|
||||
exit_code = main(["backup", "web-01", "--dry-run"])
|
||||
@@ -31,15 +39,6 @@ class ConsoleEntrypointTests(SimpleTestCase):
|
||||
self.assertEqual(exit_code, 0)
|
||||
execute.assert_called_once_with(["pobsync", "check"])
|
||||
|
||||
def test_maps_schedule_alias_to_django_command(self) -> None:
|
||||
with patch("pobsync.cli.execute_from_command_line") as execute:
|
||||
exit_code = main(["schedule", "web-01", "--cron", "15 2 * * *"])
|
||||
|
||||
self.assertEqual(exit_code, 0)
|
||||
execute.assert_called_once_with(
|
||||
["pobsync", "configure_pobsync_schedule", "web-01", "--cron", "15 2 * * *"]
|
||||
)
|
||||
|
||||
def test_maps_discover_snapshots_alias_to_django_command(self) -> None:
|
||||
with patch("pobsync.cli.execute_from_command_line") as execute:
|
||||
exit_code = main(["discover-snapshots", "--host", "web-01"])
|
||||
@@ -53,3 +52,12 @@ class ConsoleEntrypointTests(SimpleTestCase):
|
||||
|
||||
self.assertEqual(exit_code, 0)
|
||||
execute.assert_called_once_with(["pobsync", "run_pobsync_worker", "--once"])
|
||||
|
||||
def test_configuration_aliases_are_not_public_commands(self) -> None:
|
||||
stderr = StringIO()
|
||||
with patch("sys.stderr", stderr):
|
||||
exit_code = main(["schedule", "web-01", "--cron", "15 2 * * *"])
|
||||
|
||||
self.assertEqual(exit_code, 2)
|
||||
self.assertIn("Unknown pobsync command", stderr.getvalue())
|
||||
self.assertIn("pobsync django <management-command>", stderr.getvalue())
|
||||
|
||||
@@ -15,7 +15,6 @@ class DjangoConfigSourceTests(TestCase):
|
||||
GlobalConfig.objects.create(
|
||||
name="default",
|
||||
backup_root="/backups",
|
||||
pobsync_home="/opt/pobsync",
|
||||
rsync_args=["--archive"],
|
||||
rsync_extra_args=["--numeric-ids"],
|
||||
excludes_default=["/proc/***"],
|
||||
@@ -23,21 +22,6 @@ class DjangoConfigSourceTests(TestCase):
|
||||
retention_weekly=4,
|
||||
retention_monthly=3,
|
||||
retention_yearly=1,
|
||||
data={
|
||||
"backup_root": "/ignored",
|
||||
"pobsync_home": "/ignored",
|
||||
"ssh": {"user": "root", "port": 22, "options": []},
|
||||
"rsync": {
|
||||
"binary": "rsync",
|
||||
"args": ["--archive"],
|
||||
"timeout_seconds": 0,
|
||||
"bwlimit_kbps": 0,
|
||||
"extra_args": ["--numeric-ids"],
|
||||
},
|
||||
"defaults": {"source_root": "/", "destination_subdir": ""},
|
||||
"excludes_default": ["/proc/***"],
|
||||
"retention_defaults": {"daily": 7, "weekly": 4, "monthly": 3, "yearly": 1},
|
||||
},
|
||||
)
|
||||
HostConfig.objects.create(
|
||||
host="web-01",
|
||||
@@ -72,7 +56,6 @@ class DjangoConfigSourceTests(TestCase):
|
||||
GlobalConfig.objects.create(
|
||||
name="default",
|
||||
backup_root="/backups",
|
||||
pobsync_home="/opt/pobsync",
|
||||
default_ssh_credential=credential,
|
||||
ssh_options=["-oBatchMode=yes"],
|
||||
)
|
||||
@@ -99,7 +82,6 @@ class DjangoConfigSourceTests(TestCase):
|
||||
GlobalConfig.objects.create(
|
||||
name="default",
|
||||
backup_root="/backups",
|
||||
pobsync_home="/opt/pobsync",
|
||||
default_ssh_credential=global_credential,
|
||||
)
|
||||
HostConfig.objects.create(
|
||||
@@ -127,7 +109,6 @@ class DjangoConfigSourceTests(TestCase):
|
||||
GlobalConfig.objects.create(
|
||||
name="default",
|
||||
backup_root="/backups",
|
||||
pobsync_home="/opt/pobsync",
|
||||
default_ssh_credential=credential,
|
||||
)
|
||||
HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
@@ -146,7 +127,6 @@ class DjangoConfigSourceTests(TestCase):
|
||||
GlobalConfig.objects.create(
|
||||
name="default",
|
||||
backup_root="/backups",
|
||||
pobsync_home="/opt/pobsync",
|
||||
default_ssh_credential=credential,
|
||||
)
|
||||
HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
|
||||
@@ -7,6 +7,7 @@ from tempfile import TemporaryDirectory
|
||||
from django.test import SimpleTestCase
|
||||
|
||||
from pobsync.commands.retention_plan import run_retention_plan
|
||||
from pobsync.errors import ConfigError
|
||||
from pobsync.util import write_yaml_atomic
|
||||
|
||||
|
||||
@@ -24,6 +25,15 @@ class FakeConfigSource:
|
||||
|
||||
|
||||
class RetentionConfigSourceTests(SimpleTestCase):
|
||||
def test_retention_plan_requires_explicit_config_source(self) -> None:
|
||||
with self.assertRaisesMessage(ConfigError, "A Django config source is required."):
|
||||
run_retention_plan(
|
||||
prefix=Path("/missing-prefix"),
|
||||
host="web-01",
|
||||
kind="scheduled",
|
||||
protect_bases=False,
|
||||
)
|
||||
|
||||
def test_retention_plan_uses_injected_config_source(self) -> None:
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = Path(tmp) / "backups"
|
||||
|
||||
@@ -96,6 +96,7 @@ class RunBackupRecordsSnapshotTests(TestCase):
|
||||
protect_bases=True,
|
||||
yes=True,
|
||||
max_delete=3,
|
||||
action=BackupRun.RunType.SCHEDULED,
|
||||
acquire_lock=False,
|
||||
)
|
||||
run = BackupRun.objects.get()
|
||||
|
||||
@@ -7,6 +7,7 @@ from unittest.mock import patch
|
||||
from django.test import SimpleTestCase
|
||||
|
||||
from pobsync.commands.run_scheduled import run_scheduled
|
||||
from pobsync.errors import ConfigError
|
||||
from pobsync.rsync import RsyncResult
|
||||
|
||||
|
||||
@@ -34,6 +35,10 @@ class FakeConfigSource:
|
||||
|
||||
|
||||
class RunScheduledConfigSourceTests(SimpleTestCase):
|
||||
def test_requires_explicit_config_source(self) -> None:
|
||||
with self.assertRaisesMessage(ConfigError, "A Django config source is required."):
|
||||
run_scheduled(prefix=Path("/missing-prefix"), host="web-01", dry_run=True)
|
||||
|
||||
def test_dry_run_uses_injected_config_source(self) -> None:
|
||||
with patch("pobsync.commands.run_scheduled.run_rsync") as run_rsync:
|
||||
run_rsync.return_value = RsyncResult(exit_code=0, command=["rsync", "--archive"])
|
||||
|
||||
@@ -8,7 +8,7 @@ from zoneinfo import ZoneInfo
|
||||
from django.test import SimpleTestCase, TestCase
|
||||
|
||||
from pobsync_backend.management.commands.run_pobsync_scheduler import Command
|
||||
from pobsync_backend.models import HostConfig, ScheduleConfig
|
||||
from pobsync_backend.models import BackupRun, HostConfig, ScheduleConfig
|
||||
from pobsync_backend.scheduler import due_key, is_due, next_due_after
|
||||
|
||||
|
||||
@@ -64,3 +64,30 @@ class SchedulerCommandTests(TestCase):
|
||||
self.assertEqual(call.call_count, 1)
|
||||
schedule = ScheduleConfig.objects.get(host=host)
|
||||
self.assertEqual(schedule.last_status, "success")
|
||||
|
||||
def test_run_due_records_warning_status_from_scheduled_backup_run(self) -> None:
|
||||
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
ScheduleConfig.objects.create(host=host, cron_expr="* * * * *", prune=True, prune_max_delete=1)
|
||||
|
||||
def create_warning_run(*args, **kwargs) -> None:
|
||||
BackupRun.objects.create(
|
||||
host=host,
|
||||
run_type=BackupRun.RunType.SCHEDULED,
|
||||
status=BackupRun.Status.WARNING,
|
||||
result={
|
||||
"ok": True,
|
||||
"prune": {
|
||||
"ok": False,
|
||||
"type": "ConfigError",
|
||||
"error": "Refusing to delete 2 snapshots (exceeds --max-delete=1)",
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
command = Command()
|
||||
with patch("pobsync_backend.management.commands.run_pobsync_scheduler.call_command", side_effect=create_warning_run):
|
||||
count = command._run_due(prefix=Path("/opt/pobsync"), dry_run=False)
|
||||
|
||||
self.assertEqual(count, 1)
|
||||
schedule = ScheduleConfig.objects.get(host=host)
|
||||
self.assertEqual(schedule.last_status, "warning")
|
||||
|
||||
@@ -11,8 +11,8 @@ from django.core.management import call_command
|
||||
from django.test import TestCase
|
||||
|
||||
from pobsync.errors import ConfigError
|
||||
from pobsync_backend.models import HostConfig, SnapshotRecord
|
||||
from pobsync_backend.retention import run_sql_retention_apply, run_sql_retention_plan
|
||||
from pobsync_backend.models import HostConfig, PurgedSnapshot, SnapshotRecord
|
||||
from pobsync_backend.retention import run_incomplete_cleanup, run_sql_retention_apply, run_sql_retention_plan
|
||||
|
||||
|
||||
class SqlRetentionTests(TestCase):
|
||||
@@ -32,7 +32,10 @@ class SqlRetentionTests(TestCase):
|
||||
|
||||
self.assertEqual(plan["source"], "sql")
|
||||
self.assertEqual(plan["keep"], [new.dirname])
|
||||
self.assertEqual([item["dirname"] for item in plan["keep_items"]], [new.dirname])
|
||||
self.assertEqual([item["dirname"] for item in plan["delete"]], [old.dirname])
|
||||
self.assertEqual(plan["delete"][0]["reason"], "outside retention policy")
|
||||
self.assertEqual(plan["incomplete"], [])
|
||||
|
||||
def test_plan_can_protect_base_snapshot_from_sql_relation(self) -> None:
|
||||
host = HostConfig.objects.create(
|
||||
@@ -84,7 +87,26 @@ class SqlRetentionTests(TestCase):
|
||||
self.assertTrue(new_dir.exists())
|
||||
self.assertTrue(SnapshotRecord.objects.filter(pk=new.pk).exists())
|
||||
self.assertFalse(SnapshotRecord.objects.filter(pk=old.pk).exists())
|
||||
self.assertEqual(result["deleted"], [{"dirname": old.dirname, "kind": "scheduled", "path": str(old_dir)}])
|
||||
self.assertEqual(
|
||||
result["deleted"],
|
||||
[
|
||||
{
|
||||
"dirname": old.dirname,
|
||||
"kind": "scheduled",
|
||||
"path": str(old_dir),
|
||||
"reason": "outside retention policy",
|
||||
}
|
||||
],
|
||||
)
|
||||
self.assertEqual(result["planned_delete_count"], 1)
|
||||
self.assertEqual(result["max_delete"], 1)
|
||||
self.assertEqual(result["incomplete_ignored_count"], 0)
|
||||
purged = PurgedSnapshot.objects.get(dirname=old.dirname)
|
||||
self.assertEqual(purged.host_name, host.host)
|
||||
self.assertEqual(purged.kind, "scheduled")
|
||||
self.assertEqual(purged.path, str(old_dir))
|
||||
self.assertEqual(purged.reason, "outside retention policy")
|
||||
self.assertEqual(purged.action, PurgedSnapshot.Action.MANUAL)
|
||||
|
||||
def test_apply_deletes_snapshot_with_readonly_data_directory(self) -> None:
|
||||
with TemporaryDirectory() as tmp:
|
||||
@@ -120,7 +142,17 @@ class SqlRetentionTests(TestCase):
|
||||
|
||||
self.assertFalse(old_dir.exists())
|
||||
self.assertFalse(SnapshotRecord.objects.filter(pk=old.pk).exists())
|
||||
self.assertEqual(result["deleted"], [{"dirname": old.dirname, "kind": "scheduled", "path": str(old_dir)}])
|
||||
self.assertEqual(
|
||||
result["deleted"],
|
||||
[
|
||||
{
|
||||
"dirname": old.dirname,
|
||||
"kind": "scheduled",
|
||||
"path": str(old_dir),
|
||||
"reason": "outside retention policy",
|
||||
}
|
||||
],
|
||||
)
|
||||
|
||||
def test_apply_respects_max_delete(self) -> None:
|
||||
host = HostConfig.objects.create(
|
||||
@@ -146,6 +178,81 @@ class SqlRetentionTests(TestCase):
|
||||
acquire_lock=False,
|
||||
)
|
||||
|
||||
def test_incomplete_cleanup_deletes_directory_and_record(self) -> None:
|
||||
with TemporaryDirectory() as tmp:
|
||||
prefix = Path(tmp) / "home"
|
||||
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
incomplete_dir = Path(tmp) / "backups" / host.host / ".incomplete" / "20260519-031500Z__BROKEN01"
|
||||
incomplete_dir.mkdir(parents=True)
|
||||
incomplete_dir.joinpath("partial-file").write_text("interrupted\n")
|
||||
record = SnapshotRecord.objects.create(
|
||||
host=host,
|
||||
kind=SnapshotRecord.Kind.INCOMPLETE,
|
||||
dirname=incomplete_dir.name,
|
||||
path=str(incomplete_dir),
|
||||
status="failed",
|
||||
started_at=datetime(2026, 5, 19, 3, 15, tzinfo=timezone.utc),
|
||||
)
|
||||
|
||||
result = run_incomplete_cleanup(
|
||||
prefix=prefix,
|
||||
host=host.host,
|
||||
yes=True,
|
||||
max_delete=1,
|
||||
acquire_lock=False,
|
||||
)
|
||||
|
||||
self.assertFalse(incomplete_dir.exists())
|
||||
self.assertFalse(SnapshotRecord.objects.filter(pk=record.pk).exists())
|
||||
self.assertEqual(
|
||||
result["deleted"],
|
||||
[{"dirname": incomplete_dir.name, "kind": SnapshotRecord.Kind.INCOMPLETE, "path": str(incomplete_dir)}],
|
||||
)
|
||||
self.assertEqual(result["planned_delete_count"], 1)
|
||||
purged = PurgedSnapshot.objects.get(dirname=incomplete_dir.name)
|
||||
self.assertEqual(purged.action, PurgedSnapshot.Action.INCOMPLETE_CLEANUP)
|
||||
self.assertEqual(purged.reason, "manual incomplete cleanup")
|
||||
|
||||
def test_incomplete_cleanup_respects_max_delete(self) -> None:
|
||||
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
SnapshotRecord.objects.create(
|
||||
host=host,
|
||||
kind=SnapshotRecord.Kind.INCOMPLETE,
|
||||
dirname="20260519-031500Z__BROKEN01",
|
||||
path=f"/backups/{host.host}/.incomplete/20260519-031500Z__BROKEN01",
|
||||
status="failed",
|
||||
started_at=datetime(2026, 5, 19, 3, 15, tzinfo=timezone.utc),
|
||||
)
|
||||
|
||||
with self.assertRaisesRegex(ConfigError, "blocked by --max-delete=0"):
|
||||
run_incomplete_cleanup(
|
||||
prefix=Path("/tmp/pobsync-test"),
|
||||
host=host.host,
|
||||
yes=True,
|
||||
max_delete=0,
|
||||
acquire_lock=False,
|
||||
)
|
||||
|
||||
def test_incomplete_cleanup_rejects_unexpected_path(self) -> None:
|
||||
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
SnapshotRecord.objects.create(
|
||||
host=host,
|
||||
kind=SnapshotRecord.Kind.INCOMPLETE,
|
||||
dirname="20260519-031500Z__BROKEN01",
|
||||
path=f"/backups/{host.host}/scheduled/20260519-031500Z__BROKEN01",
|
||||
status="failed",
|
||||
started_at=datetime(2026, 5, 19, 3, 15, tzinfo=timezone.utc),
|
||||
)
|
||||
|
||||
with self.assertRaisesRegex(ConfigError, "unexpected incomplete snapshot path"):
|
||||
run_incomplete_cleanup(
|
||||
prefix=Path("/tmp/pobsync-test"),
|
||||
host=host.host,
|
||||
yes=True,
|
||||
max_delete=1,
|
||||
acquire_lock=False,
|
||||
)
|
||||
|
||||
def test_management_command_plans_from_sql(self) -> None:
|
||||
host = HostConfig.objects.create(
|
||||
host="web-01",
|
||||
|
||||
@@ -12,7 +12,15 @@ from django.test import TestCase, override_settings
|
||||
from django.urls import reverse
|
||||
|
||||
from pobsync.util import write_yaml_atomic
|
||||
from pobsync_backend.models import BackupRun, GlobalConfig, HostConfig, ScheduleConfig, SnapshotRecord, SshCredential
|
||||
from pobsync_backend.models import (
|
||||
BackupRun,
|
||||
GlobalConfig,
|
||||
HostConfig,
|
||||
PurgedSnapshot,
|
||||
ScheduleConfig,
|
||||
SnapshotRecord,
|
||||
SshCredential,
|
||||
)
|
||||
|
||||
|
||||
class ViewTests(TestCase):
|
||||
@@ -31,6 +39,30 @@ class ViewTests(TestCase):
|
||||
self.assertEqual(response.status_code, 302)
|
||||
self.assertIn("/admin/login/", response["Location"])
|
||||
|
||||
def test_changelog_requires_staff_login(self) -> None:
|
||||
response = self.client.get(reverse("changelog"))
|
||||
|
||||
self.assertEqual(response.status_code, 302)
|
||||
self.assertIn("/admin/login/", response["Location"])
|
||||
|
||||
def test_changelog_renders_repository_changelog(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
with TemporaryDirectory() as tmp:
|
||||
changelog = Path(tmp) / "CHANGELOG.md"
|
||||
changelog.write_text(
|
||||
"# Changelog\n\n## 1.0.0 - 2026-05-21\n\n- Django control panel\n- Native systemd installer\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
with override_settings(BASE_DIR=Path(tmp)):
|
||||
response = self.client.get(reverse("changelog"))
|
||||
|
||||
self.assertEqual(response.status_code, 200)
|
||||
self.assertContains(response, "Installed version:")
|
||||
self.assertContains(response, "1.0.0 - 2026-05-21")
|
||||
self.assertContains(response, "Django control panel")
|
||||
self.assertContains(response, "Native systemd installer")
|
||||
|
||||
def test_dashboard_renders_hosts_and_latest_runs(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
@@ -42,6 +74,27 @@ class ViewTests(TestCase):
|
||||
snapshot=snapshot,
|
||||
started_at=datetime(2026, 5, 19, 2, 15, tzinfo=timezone.utc),
|
||||
)
|
||||
warning_run = BackupRun.objects.create(
|
||||
host=host,
|
||||
run_type=BackupRun.RunType.SCHEDULED,
|
||||
status=BackupRun.Status.WARNING,
|
||||
started_at=datetime(2026, 5, 19, 3, 15, tzinfo=timezone.utc),
|
||||
result={
|
||||
"ok": True,
|
||||
"prune": {
|
||||
"ok": False,
|
||||
"error": "Retention warning",
|
||||
},
|
||||
},
|
||||
)
|
||||
BackupRun.objects.create(host=host, status=BackupRun.Status.QUEUED)
|
||||
BackupRun.objects.create(host=host, status=BackupRun.Status.RUNNING)
|
||||
BackupRun.objects.create(
|
||||
host=host,
|
||||
run_type=BackupRun.RunType.MANUAL,
|
||||
status=BackupRun.Status.FAILED,
|
||||
started_at=datetime(2026, 5, 19, 1, 15, tzinfo=timezone.utc),
|
||||
)
|
||||
|
||||
response = self.client.get(reverse("dashboard"))
|
||||
|
||||
@@ -50,8 +103,24 @@ class ViewTests(TestCase):
|
||||
self.assertContains(response, "web-01")
|
||||
self.assertContains(response, "20260519-021500Z__ABCDEFGH")
|
||||
self.assertContains(response, "success")
|
||||
self.assertContains(response, "Last Good Backup")
|
||||
self.assertContains(response, "Latest Issue")
|
||||
self.assertContains(response, f"Run {run.id}")
|
||||
self.assertContains(response, f"Run {warning_run.id}")
|
||||
self.assertContains(response, "warning")
|
||||
self.assertContains(response, "manual")
|
||||
self.assertContains(response, "scheduled")
|
||||
self.assertContains(response, "Backup activity")
|
||||
self.assertContains(response, "Snapshot health")
|
||||
self.assertContains(response, "queued 1")
|
||||
self.assertContains(response, "running 1")
|
||||
self.assertContains(response, "warning 1")
|
||||
self.assertContains(response, "failed 1")
|
||||
self.assertContains(response, "Operational Status")
|
||||
self.assertContains(response, "1 failed run needs review.")
|
||||
self.assertContains(response, "1 run completed with warnings.")
|
||||
self.assertContains(response, "1 backup run in progress.")
|
||||
self.assertContains(response, "1 backup run waiting for the worker.")
|
||||
|
||||
def test_dashboard_renders_backup_trend_summary(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
@@ -88,10 +157,15 @@ class ViewTests(TestCase):
|
||||
response = self.client.get(reverse("dashboard"))
|
||||
|
||||
self.assertEqual(response.status_code, 200)
|
||||
self.assertContains(response, "Backup Root Used")
|
||||
self.assertContains(response, "Runs Until Full")
|
||||
self.assertContains(response, "Avg Daily New")
|
||||
self.assertContains(response, "Days Until Full")
|
||||
self.assertContains(response, "Backup Trends")
|
||||
self.assertContains(response, "Storage Used")
|
||||
self.assertContains(response, "Runway")
|
||||
self.assertContains(response, "New Data")
|
||||
self.assertContains(response, "Link-Dest Savings")
|
||||
self.assertContains(response, "80.0%")
|
||||
self.assertContains(response, "10 days")
|
||||
self.assertContains(response, "Warnings")
|
||||
self.assertContains(response, "Queued")
|
||||
self.assertContains(response, "Next Run")
|
||||
self.assertContains(response, "UTC")
|
||||
self.assertContains(response, "10")
|
||||
@@ -99,6 +173,82 @@ class ViewTests(TestCase):
|
||||
self.assertContains(response, "manual")
|
||||
self.assertContains(response, "1000")
|
||||
|
||||
def test_dashboard_explains_missing_backup_trends(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
GlobalConfig.objects.create(name="default", backup_root="/opt/pobsync/backups")
|
||||
HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
|
||||
response = self.client.get(reverse("dashboard"))
|
||||
|
||||
self.assertEqual(response.status_code, 200)
|
||||
self.assertContains(response, "Backup Trends")
|
||||
self.assertContains(response, "No completed backup runs with stats yet.")
|
||||
self.assertContains(response, "growth estimates")
|
||||
|
||||
def test_dashboard_shows_all_clear_operational_status(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
GlobalConfig.objects.create(name="default", backup_root="/opt/pobsync/backups")
|
||||
HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
|
||||
response = self.client.get(reverse("dashboard"))
|
||||
|
||||
self.assertEqual(response.status_code, 200)
|
||||
self.assertContains(response, "Operational Status")
|
||||
self.assertContains(response, "No queued, running, or unreviewed warning/failed runs.")
|
||||
|
||||
def test_dashboard_surfaces_retention_warnings(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
host = HostConfig.objects.create(
|
||||
host="web-01",
|
||||
address="web-01.example.test",
|
||||
retention_daily=0,
|
||||
retention_weekly=0,
|
||||
retention_monthly=0,
|
||||
retention_yearly=0,
|
||||
)
|
||||
ScheduleConfig.objects.create(host=host, cron_expr="15 2 * * *", enabled=True, prune=True, prune_max_delete=1)
|
||||
self._snapshot(host, "20260517-021500Z__OLDSNP1")
|
||||
self._snapshot(host, "20260518-021500Z__OLDSNP2")
|
||||
self._snapshot(host, "20260519-021500Z__NEWSNAP")
|
||||
SnapshotRecord.objects.create(
|
||||
host=host,
|
||||
kind=SnapshotRecord.Kind.INCOMPLETE,
|
||||
dirname="20260519-031500Z__BROKEN01",
|
||||
path=f"/backups/{host.host}/.incomplete/20260519-031500Z__BROKEN01",
|
||||
status="failed",
|
||||
started_at=datetime(2026, 5, 19, 3, 15, tzinfo=timezone.utc),
|
||||
)
|
||||
|
||||
response = self.client.get(reverse("dashboard"))
|
||||
|
||||
self.assertEqual(response.status_code, 200)
|
||||
self.assertContains(response, "Scheduled prune would delete 2 snapshot(s), above max 1.")
|
||||
self.assertContains(response, "1 incomplete snapshot(s) need review.")
|
||||
self.assertContains(response, "Mark reviewed")
|
||||
|
||||
def test_dashboard_ignores_reviewed_problem_runs(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
BackupRun.objects.create(
|
||||
host=host,
|
||||
status=BackupRun.Status.FAILED,
|
||||
reviewed_at=datetime(2026, 5, 19, 4, 15, tzinfo=timezone.utc),
|
||||
reviewed_by="admin",
|
||||
)
|
||||
BackupRun.objects.create(
|
||||
host=host,
|
||||
status=BackupRun.Status.WARNING,
|
||||
reviewed_at=datetime(2026, 5, 19, 4, 20, tzinfo=timezone.utc),
|
||||
reviewed_by="admin",
|
||||
)
|
||||
|
||||
response = self.client.get(reverse("dashboard"))
|
||||
|
||||
self.assertEqual(response.status_code, 200)
|
||||
self.assertContains(response, "No queued, running, or unreviewed warning/failed runs.")
|
||||
self.assertNotContains(response, "failed 1")
|
||||
self.assertNotContains(response, "warning 1")
|
||||
|
||||
def test_dashboard_links_latest_snapshot_for_each_host(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
@@ -143,7 +293,7 @@ class ViewTests(TestCase):
|
||||
self.assertContains(response, "Self Check")
|
||||
self.assertContains(response, "Django debug")
|
||||
self.assertContains(response, "Database connection")
|
||||
self.assertContains(response, "POBSYNC_HOME")
|
||||
self.assertContains(response, "State root")
|
||||
|
||||
def test_logs_view_renders_filtered_journal_messages(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
@@ -186,6 +336,61 @@ class ViewTests(TestCase):
|
||||
self.assertIn("--since", command)
|
||||
self.assertIn("6 hours ago", command)
|
||||
|
||||
def test_purged_snapshots_view_renders_history(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
PurgedSnapshot.objects.create(
|
||||
host=host,
|
||||
host_name=host.host,
|
||||
kind=SnapshotRecord.Kind.SCHEDULED,
|
||||
dirname="20260518-021500Z__OLDSNAP",
|
||||
path=f"/backups/{host.host}/scheduled/20260518-021500Z__OLDSNAP",
|
||||
reason="outside retention policy",
|
||||
action=PurgedSnapshot.Action.SCHEDULED,
|
||||
triggered_by="pobsync-scheduler",
|
||||
)
|
||||
|
||||
response = self.client.get(reverse("purged_snapshots"))
|
||||
|
||||
self.assertEqual(response.status_code, 200)
|
||||
self.assertContains(response, "Purged Snapshots")
|
||||
self.assertContains(response, "20260518-021500Z__OLDSNAP")
|
||||
self.assertContains(response, "outside retention policy")
|
||||
self.assertContains(response, "Scheduled")
|
||||
self.assertContains(response, "pobsync-scheduler")
|
||||
|
||||
def test_purged_snapshots_view_filters_by_host_and_action(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
web = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
db = HostConfig.objects.create(host="db-01", address="db-01.example.test")
|
||||
PurgedSnapshot.objects.create(
|
||||
host=web,
|
||||
host_name=web.host,
|
||||
kind=SnapshotRecord.Kind.SCHEDULED,
|
||||
dirname="20260518-021500Z__WEBOLD",
|
||||
path=f"/backups/{web.host}/scheduled/20260518-021500Z__WEBOLD",
|
||||
reason="outside retention policy",
|
||||
action=PurgedSnapshot.Action.MANUAL,
|
||||
)
|
||||
PurgedSnapshot.objects.create(
|
||||
host=db,
|
||||
host_name=db.host,
|
||||
kind=SnapshotRecord.Kind.INCOMPLETE,
|
||||
dirname="20260518-021500Z__DBBROKEN",
|
||||
path=f"/backups/{db.host}/.incomplete/20260518-021500Z__DBBROKEN",
|
||||
reason="manual incomplete cleanup",
|
||||
action=PurgedSnapshot.Action.INCOMPLETE_CLEANUP,
|
||||
)
|
||||
|
||||
response = self.client.get(
|
||||
reverse("purged_snapshots"),
|
||||
{"host": db.host, "action": PurgedSnapshot.Action.INCOMPLETE_CLEANUP},
|
||||
)
|
||||
|
||||
self.assertEqual(response.status_code, 200)
|
||||
self.assertContains(response, "20260518-021500Z__DBBROKEN")
|
||||
self.assertNotContains(response, "20260518-021500Z__WEBOLD")
|
||||
|
||||
def test_ssh_credentials_view_creates_key(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
|
||||
@@ -264,13 +469,46 @@ class ViewTests(TestCase):
|
||||
generate_ssh_key(credential)
|
||||
key_path = Path(credential.key_path)
|
||||
|
||||
response = self.client.post(reverse("delete_ssh_credential", args=[credential.id]), follow=True)
|
||||
response = self.client.post(
|
||||
reverse("delete_ssh_credential", args=[credential.id]),
|
||||
{"confirm_name": credential.name},
|
||||
follow=True,
|
||||
)
|
||||
|
||||
self.assertRedirects(response, reverse("ssh_credentials"))
|
||||
self.assertContains(response, "SSH key deleted: generated-key.")
|
||||
self.assertFalse(SshCredential.objects.exists())
|
||||
self.assertFalse(key_path.exists())
|
||||
|
||||
def test_ssh_credentials_view_requires_delete_confirmation(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
credential = SshCredential.objects.create(name="backup-key")
|
||||
|
||||
response = self.client.post(
|
||||
reverse("delete_ssh_credential", args=[credential.id]),
|
||||
{"confirm_name": "wrong"},
|
||||
follow=True,
|
||||
)
|
||||
|
||||
self.assertRedirects(response, reverse("edit_ssh_credential", args=[credential.id]))
|
||||
self.assertContains(response, "Type backup-key to confirm SSH key deletion.")
|
||||
self.assertTrue(SshCredential.objects.filter(pk=credential.pk).exists())
|
||||
|
||||
def test_ssh_credentials_view_blocks_delete_when_key_is_in_use(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
credential = SshCredential.objects.create(name="backup-key")
|
||||
HostConfig.objects.create(host="web-01", address="web-01.example.test", ssh_credential=credential)
|
||||
|
||||
response = self.client.post(
|
||||
reverse("delete_ssh_credential", args=[credential.id]),
|
||||
{"confirm_name": credential.name},
|
||||
follow=True,
|
||||
)
|
||||
|
||||
self.assertRedirects(response, reverse("edit_ssh_credential", args=[credential.id]))
|
||||
self.assertContains(response, "SSH key backup-key is still in use and cannot be deleted.")
|
||||
self.assertTrue(SshCredential.objects.filter(pk=credential.pk).exists())
|
||||
|
||||
def test_ssh_credentials_view_rejects_invalid_key(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
|
||||
@@ -334,7 +572,7 @@ class ViewTests(TestCase):
|
||||
response = self.client.post(
|
||||
reverse("edit_ssh_credential", args=[credential.id]),
|
||||
{
|
||||
"name": "backup-key",
|
||||
"name": "renamed-backup-key",
|
||||
"private_key": "UPDATED KEY",
|
||||
"public_key": "",
|
||||
"known_hosts": "",
|
||||
@@ -345,6 +583,7 @@ class ViewTests(TestCase):
|
||||
|
||||
self.assertRedirects(response, reverse("ssh_credentials"))
|
||||
credential.refresh_from_db()
|
||||
self.assertEqual(credential.name, "renamed-backup-key")
|
||||
self.assertEqual(credential.private_key, "UPDATED KEY\n")
|
||||
self.assertEqual(credential.public_key, "UPDATED PUBLIC KEY")
|
||||
self.assertEqual(credential.notes, "rotated")
|
||||
@@ -381,7 +620,6 @@ class ViewTests(TestCase):
|
||||
self.assertContains(response, "Global config saved for default.")
|
||||
config = GlobalConfig.objects.get(name="default")
|
||||
self.assertEqual(config.backup_root, "/backups")
|
||||
self.assertEqual(config.pobsync_home, "/opt/pobsync")
|
||||
self.assertEqual(config.default_ssh_credential, credential)
|
||||
self.assertEqual(config.ssh_user, "backup")
|
||||
self.assertEqual(config.ssh_port, 2222)
|
||||
@@ -408,7 +646,6 @@ class ViewTests(TestCase):
|
||||
GlobalConfig.objects.create(
|
||||
name="default",
|
||||
backup_root="/mnt/pobsync/backups",
|
||||
pobsync_home="/custom/legacy/home",
|
||||
)
|
||||
|
||||
response = self.client.get(reverse("edit_global_config"))
|
||||
@@ -418,8 +655,10 @@ class ViewTests(TestCase):
|
||||
self.assertContains(response, "/backups")
|
||||
self.assertContains(response, "Config Check")
|
||||
self.assertContains(response, "Runtime backup root")
|
||||
self.assertContains(response, "Runtime state root")
|
||||
self.assertNotContains(response, "/opt/pobsync/backups")
|
||||
self.assertNotContains(response, "Pobsync home")
|
||||
self.assertNotContains(response, "Global pobsync home")
|
||||
|
||||
def test_global_config_form_renders_config_check_for_non_recursive_rsync(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
@@ -436,7 +675,6 @@ class ViewTests(TestCase):
|
||||
GlobalConfig.objects.create(
|
||||
name="default",
|
||||
backup_root="/mnt/pobsync/backups",
|
||||
pobsync_home="/custom/legacy/home",
|
||||
)
|
||||
|
||||
response = self.client.post(
|
||||
@@ -465,7 +703,6 @@ class ViewTests(TestCase):
|
||||
self.assertRedirects(response, reverse("dashboard"))
|
||||
config = GlobalConfig.objects.get(name="default")
|
||||
self.assertEqual(config.backup_root, "/backups")
|
||||
self.assertEqual(config.pobsync_home, "/opt/pobsync")
|
||||
|
||||
def test_create_host_config_form_creates_host(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
@@ -1168,9 +1405,15 @@ class ViewTests(TestCase):
|
||||
"hint": "Check network connectivity.",
|
||||
},
|
||||
"prune": {
|
||||
"ok": False,
|
||||
"type": "ConfigError",
|
||||
"error": "Deletion blocked by --max-delete=0",
|
||||
"ok": True,
|
||||
"source": "sql",
|
||||
"kind": "scheduled",
|
||||
"planned_delete_count": 1,
|
||||
"max_delete": 1,
|
||||
"protect_bases": True,
|
||||
"incomplete_ignored_count": 1,
|
||||
"deleted": [{"dirname": "20260518-021500Z__OLD"}],
|
||||
"actions": ["deleted scheduled 20260518-021500Z__OLD"],
|
||||
},
|
||||
},
|
||||
)
|
||||
@@ -1182,7 +1425,70 @@ class ViewTests(TestCase):
|
||||
self.assertContains(response, "transport")
|
||||
self.assertContains(response, "Check network connectivity.")
|
||||
self.assertContains(response, "Retention")
|
||||
self.assertContains(response, "Deletion blocked by --max-delete=0")
|
||||
self.assertContains(response, "Planned deletions")
|
||||
self.assertContains(response, "Max delete")
|
||||
self.assertContains(response, "Protect bases")
|
||||
self.assertContains(response, "Incomplete ignored")
|
||||
self.assertContains(response, "deleted scheduled 20260518-021500Z__OLD")
|
||||
|
||||
def test_run_review_action_marks_problem_run_reviewed(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
run = BackupRun.objects.create(host=host, status=BackupRun.Status.FAILED, result={"ok": False})
|
||||
|
||||
response = self.client.post(reverse("resolve_run_review", args=[run.id]), follow=True)
|
||||
|
||||
run.refresh_from_db()
|
||||
self.assertIsNotNone(run.reviewed_at)
|
||||
self.assertEqual(run.reviewed_by, self.staff_user.username)
|
||||
self.assertRedirects(response, reverse("run_detail", args=[run.id]))
|
||||
self.assertContains(response, f"Run {run.id} marked reviewed.")
|
||||
self.assertContains(response, "Review")
|
||||
self.assertContains(response, self.staff_user.username)
|
||||
self.assertNotContains(response, "Mark reviewed")
|
||||
|
||||
def test_run_review_action_ignores_successful_run(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
run = BackupRun.objects.create(host=host, status=BackupRun.Status.SUCCESS, result={"ok": True})
|
||||
|
||||
response = self.client.post(reverse("resolve_run_review", args=[run.id]), follow=True)
|
||||
|
||||
run.refresh_from_db()
|
||||
self.assertIsNone(run.reviewed_at)
|
||||
self.assertRedirects(response, reverse("run_detail", args=[run.id]))
|
||||
self.assertContains(response, f"Run {run.id} does not need review.")
|
||||
|
||||
def test_run_detail_surfaces_host_retention_warnings(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
host = HostConfig.objects.create(
|
||||
host="web-01",
|
||||
address="web-01.example.test",
|
||||
retention_daily=0,
|
||||
retention_weekly=0,
|
||||
retention_monthly=0,
|
||||
retention_yearly=0,
|
||||
)
|
||||
ScheduleConfig.objects.create(host=host, cron_expr="15 2 * * *", enabled=True, prune=True, prune_max_delete=1)
|
||||
self._snapshot(host, "20260517-021500Z__OLDSNP1")
|
||||
self._snapshot(host, "20260518-021500Z__OLDSNP2")
|
||||
self._snapshot(host, "20260519-021500Z__NEWSNAP")
|
||||
SnapshotRecord.objects.create(
|
||||
host=host,
|
||||
kind=SnapshotRecord.Kind.INCOMPLETE,
|
||||
dirname="20260519-031500Z__BROKEN01",
|
||||
path=f"/backups/{host.host}/.incomplete/20260519-031500Z__BROKEN01",
|
||||
status="failed",
|
||||
started_at=datetime(2026, 5, 19, 3, 15, tzinfo=timezone.utc),
|
||||
)
|
||||
run = BackupRun.objects.create(host=host, status=BackupRun.Status.SUCCESS, result={"ok": True})
|
||||
|
||||
response = self.client.get(reverse("run_detail", args=[run.id]))
|
||||
|
||||
self.assertEqual(response.status_code, 200)
|
||||
self.assertContains(response, "Retention Warnings")
|
||||
self.assertContains(response, "Scheduled pruning for this host would delete 2 snapshot(s)")
|
||||
self.assertContains(response, "1 incomplete snapshot(s) exist for this host")
|
||||
|
||||
def test_run_detail_infers_rsync_log_from_snapshot_path(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
@@ -1216,6 +1522,29 @@ class ViewTests(TestCase):
|
||||
self.assertContains(response, "Cancel run")
|
||||
self.assertContains(response, reverse("cancel_run", args=[run.id]))
|
||||
|
||||
def test_run_detail_renders_worker_execution_metadata(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
run = BackupRun.objects.create(
|
||||
host=host,
|
||||
status=BackupRun.Status.RUNNING,
|
||||
result={
|
||||
"execution": {
|
||||
"worker_host": "backup-01",
|
||||
"worker_pid": 4242,
|
||||
"heartbeat_at": "2026-05-21T10:30:00+00:00",
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
response = self.client.get(reverse("run_detail", args=[run.id]))
|
||||
|
||||
self.assertEqual(response.status_code, 200)
|
||||
self.assertContains(response, "Worker:")
|
||||
self.assertContains(response, "backup-01")
|
||||
self.assertContains(response, "pid 4242")
|
||||
self.assertContains(response, "Worker heartbeat:")
|
||||
|
||||
def test_cancel_run_marks_queued_run_cancelled(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
@@ -1286,6 +1615,19 @@ class ViewTests(TestCase):
|
||||
self.assertContains(response, "Stats")
|
||||
self.assertContains(response, "Files seen:</strong> 100")
|
||||
self.assertContains(response, "Hardlinked files:</strong> 9")
|
||||
self.assertContains(response, "Restore Guidance")
|
||||
self.assertContains(response, f"{base.path}/data")
|
||||
self.assertContains(response, f"/restore/{host.host}")
|
||||
self.assertContains(response, "rsync -aHAX --numeric-ids --info=progress2 --dry-run")
|
||||
self.assertContains(response, f"{base.path}/data/")
|
||||
self.assertContains(response, "root@web-01.example.test:/")
|
||||
self.assertContains(response, "Dry-run a directory restore")
|
||||
self.assertContains(response, f"{base.path}/data/etc/nginx/")
|
||||
self.assertContains(response, f"/restore/{host.host}/etc/nginx/")
|
||||
self.assertContains(response, "Dry-run a single file restore")
|
||||
self.assertContains(response, f"{base.path}/data/home/example/site/public_html/index.php")
|
||||
self.assertContains(response, f"/restore/{host.host}/home/example/site/public_html/index.php")
|
||||
self.assertContains(response, "Treat snapshot directories as read-only")
|
||||
self.assertContains(response, child.dirname)
|
||||
self.assertContains(response, f"Run {run.id}")
|
||||
self.assertContains(response, reverse("run_detail", args=[run.id]))
|
||||
@@ -1351,6 +1693,32 @@ class ViewTests(TestCase):
|
||||
self.assertContains(response, new_snapshot.dirname)
|
||||
self.assertContains(response, "newest")
|
||||
self.assertContains(response, "Would Delete")
|
||||
self.assertContains(response, "outside retention policy")
|
||||
self.assertContains(response, "Confirm delete count")
|
||||
self.assertContains(response, "Type 1 to confirm the current number of planned deletions.")
|
||||
|
||||
def test_retention_plan_warns_when_scheduled_prune_limit_is_exceeded(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
host = HostConfig.objects.create(
|
||||
host="web-01",
|
||||
address="web-01.example.test",
|
||||
retention_daily=0,
|
||||
retention_weekly=0,
|
||||
retention_monthly=0,
|
||||
retention_yearly=0,
|
||||
)
|
||||
ScheduleConfig.objects.create(host=host, cron_expr="15 2 * * *", enabled=True, prune=True, prune_max_delete=1)
|
||||
self._snapshot(host, "20260517-021500Z__OLDSNP1")
|
||||
self._snapshot(host, "20260518-021500Z__OLDSNP2")
|
||||
self._snapshot(host, "20260519-021500Z__NEWSNAP")
|
||||
|
||||
response = self.client.get(reverse("host_retention_plan", args=[host.host]))
|
||||
|
||||
self.assertEqual(response.status_code, 200)
|
||||
self.assertContains(response, "Scheduled Prune Limit")
|
||||
self.assertContains(response, "would delete 2 snapshot(s)")
|
||||
self.assertContains(response, "scheduled prune limit of")
|
||||
self.assertContains(response, "Schedule max delete:</strong> 1")
|
||||
|
||||
def test_retention_plan_can_enable_base_protection(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
@@ -1371,8 +1739,160 @@ class ViewTests(TestCase):
|
||||
|
||||
self.assertEqual(response.status_code, 200)
|
||||
self.assertContains(response, "Protect bases:</strong> yes")
|
||||
self.assertContains(response, "Base snapshots referenced by kept snapshots")
|
||||
self.assertContains(response, f"base-of:{child.dirname}")
|
||||
|
||||
def test_retention_plan_surfaces_incomplete_snapshots_without_deleting_them(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
host = HostConfig.objects.create(
|
||||
host="web-01",
|
||||
address="web-01.example.test",
|
||||
retention_daily=0,
|
||||
retention_weekly=0,
|
||||
retention_monthly=0,
|
||||
retention_yearly=0,
|
||||
)
|
||||
self._snapshot(host, "20260518-021500Z__OLDSNAP")
|
||||
self._snapshot(host, "20260519-021500Z__NEWSNAP")
|
||||
SnapshotRecord.objects.create(
|
||||
host=host,
|
||||
kind=SnapshotRecord.Kind.INCOMPLETE,
|
||||
dirname="20260519-031500Z__BROKEN01",
|
||||
path=f"/backups/{host.host}/.incomplete/20260519-031500Z__BROKEN01",
|
||||
status="failed",
|
||||
started_at=datetime(2026, 5, 19, 3, 15, tzinfo=timezone.utc),
|
||||
)
|
||||
|
||||
response = self.client.get(reverse("host_retention_plan", args=[host.host]))
|
||||
|
||||
self.assertEqual(response.status_code, 200)
|
||||
self.assertContains(response, "Incomplete Snapshots")
|
||||
self.assertContains(response, "20260519-031500Z__BROKEN01")
|
||||
self.assertContains(response, "excluded from retention cleanup")
|
||||
self.assertContains(response, "Delete incomplete snapshots")
|
||||
self.assertContains(response, "Type 1 to confirm the current number of incomplete snapshots.")
|
||||
|
||||
def test_incomplete_cleanup_deletes_incomplete_snapshot_after_confirmation(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
with TemporaryDirectory() as tmp:
|
||||
home = Path(tmp) / "home"
|
||||
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
incomplete_dir = Path(tmp) / "backups" / host.host / ".incomplete" / "20260519-031500Z__BROKEN01"
|
||||
incomplete_dir.mkdir(parents=True)
|
||||
incomplete_dir.joinpath("partial-file").write_text("interrupted\n")
|
||||
record = SnapshotRecord.objects.create(
|
||||
host=host,
|
||||
kind=SnapshotRecord.Kind.INCOMPLETE,
|
||||
dirname=incomplete_dir.name,
|
||||
path=str(incomplete_dir),
|
||||
status="failed",
|
||||
started_at=datetime(2026, 5, 19, 3, 15, tzinfo=timezone.utc),
|
||||
)
|
||||
|
||||
with override_settings(POBSYNC_HOME=str(home)):
|
||||
response = self.client.post(
|
||||
reverse("cleanup_host_incomplete_snapshots", args=[host.host]),
|
||||
{
|
||||
"max_delete": "1",
|
||||
"confirm_host": host.host,
|
||||
"confirm_delete_count": "1",
|
||||
},
|
||||
follow=True,
|
||||
)
|
||||
|
||||
self.assertFalse(incomplete_dir.exists())
|
||||
|
||||
self.assertRedirects(response, reverse("host_retention_plan", args=[host.host]))
|
||||
self.assertContains(response, "Deleted 1 incomplete snapshot(s) for web-01.")
|
||||
self.assertFalse(SnapshotRecord.objects.filter(pk=record.pk).exists())
|
||||
|
||||
def test_incomplete_cleanup_rejects_bad_confirmation(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
SnapshotRecord.objects.create(
|
||||
host=host,
|
||||
kind=SnapshotRecord.Kind.INCOMPLETE,
|
||||
dirname="20260519-031500Z__BROKEN01",
|
||||
path=f"/backups/{host.host}/.incomplete/20260519-031500Z__BROKEN01",
|
||||
status="failed",
|
||||
started_at=datetime(2026, 5, 19, 3, 15, tzinfo=timezone.utc),
|
||||
)
|
||||
|
||||
response = self.client.post(
|
||||
reverse("cleanup_host_incomplete_snapshots", args=[host.host]),
|
||||
{
|
||||
"max_delete": "1",
|
||||
"confirm_host": "wrong",
|
||||
"confirm_delete_count": "1",
|
||||
},
|
||||
follow=True,
|
||||
)
|
||||
|
||||
self.assertRedirects(response, reverse("host_retention_plan", args=[host.host]))
|
||||
self.assertContains(response, "Incomplete cleanup confirmation is invalid.")
|
||||
self.assertEqual(SnapshotRecord.objects.filter(kind=SnapshotRecord.Kind.INCOMPLETE).count(), 1)
|
||||
|
||||
def test_host_detail_surfaces_retention_warnings(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
host = HostConfig.objects.create(
|
||||
host="web-01",
|
||||
address="web-01.example.test",
|
||||
retention_daily=0,
|
||||
retention_weekly=0,
|
||||
retention_monthly=0,
|
||||
retention_yearly=0,
|
||||
)
|
||||
ScheduleConfig.objects.create(host=host, cron_expr="15 2 * * *", enabled=True, prune=True, prune_max_delete=1)
|
||||
self._snapshot(host, "20260517-021500Z__OLDSNP1")
|
||||
self._snapshot(host, "20260518-021500Z__OLDSNP2")
|
||||
self._snapshot(host, "20260519-021500Z__NEWSNAP")
|
||||
|
||||
response = self.client.get(reverse("host_detail", args=[host.host]))
|
||||
|
||||
self.assertEqual(response.status_code, 200)
|
||||
self.assertContains(response, "Retention Warnings")
|
||||
self.assertContains(response, "Scheduled pruning would delete 2 snapshot(s), above max delete")
|
||||
|
||||
def test_host_detail_can_mark_incomplete_snapshots_reviewed(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
incomplete = SnapshotRecord.objects.create(
|
||||
host=host,
|
||||
kind=SnapshotRecord.Kind.INCOMPLETE,
|
||||
dirname="20260519-031500Z__BROKEN01",
|
||||
path=f"/backups/{host.host}/.incomplete/20260519-031500Z__BROKEN01",
|
||||
status="failed",
|
||||
started_at=datetime(2026, 5, 19, 3, 15, tzinfo=timezone.utc),
|
||||
)
|
||||
|
||||
response = self.client.post(reverse("resolve_host_incomplete_reviews", args=[host.host]), follow=True)
|
||||
|
||||
incomplete.refresh_from_db()
|
||||
self.assertIsNotNone(incomplete.reviewed_at)
|
||||
self.assertEqual(incomplete.reviewed_by, self.staff_user.username)
|
||||
self.assertRedirects(response, reverse("host_detail", args=[host.host]))
|
||||
self.assertContains(response, "Marked 1 incomplete snapshot(s) reviewed for web-01.")
|
||||
self.assertNotContains(response, "Retention Warnings")
|
||||
|
||||
def test_host_detail_does_not_warn_for_reviewed_incomplete_snapshots(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
SnapshotRecord.objects.create(
|
||||
host=host,
|
||||
kind=SnapshotRecord.Kind.INCOMPLETE,
|
||||
dirname="20260519-031500Z__BROKEN01",
|
||||
path=f"/backups/{host.host}/.incomplete/20260519-031500Z__BROKEN01",
|
||||
status="failed",
|
||||
started_at=datetime(2026, 5, 19, 3, 15, tzinfo=timezone.utc),
|
||||
reviewed_at=datetime(2026, 5, 19, 4, 15, tzinfo=timezone.utc),
|
||||
reviewed_by="admin",
|
||||
)
|
||||
|
||||
response = self.client.get(reverse("host_detail", args=[host.host]))
|
||||
|
||||
self.assertEqual(response.status_code, 200)
|
||||
self.assertNotContains(response, "Retention Warnings")
|
||||
|
||||
def test_retention_plan_rejects_invalid_kind(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
@@ -1412,6 +1932,7 @@ class ViewTests(TestCase):
|
||||
"kind": "scheduled",
|
||||
"max_delete": "1",
|
||||
"confirm_host": host.host,
|
||||
"confirm_delete_count": "1",
|
||||
},
|
||||
follow=True,
|
||||
)
|
||||
@@ -1423,6 +1944,10 @@ class ViewTests(TestCase):
|
||||
self.assertContains(response, "Retention deleted 1 snapshot(s) for web-01.")
|
||||
self.assertFalse(SnapshotRecord.objects.filter(pk=old_snapshot.pk).exists())
|
||||
self.assertTrue(SnapshotRecord.objects.filter(pk=new_snapshot.pk).exists())
|
||||
purged = PurgedSnapshot.objects.get(dirname=old_snapshot.dirname)
|
||||
self.assertEqual(purged.action, PurgedSnapshot.Action.MANUAL)
|
||||
self.assertEqual(purged.triggered_by, self.staff_user.username)
|
||||
self.assertEqual(purged.reason, "outside retention policy")
|
||||
|
||||
def test_retention_apply_rejects_bad_confirmation(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
@@ -1435,6 +1960,7 @@ class ViewTests(TestCase):
|
||||
"kind": "scheduled",
|
||||
"max_delete": "1",
|
||||
"confirm_host": "wrong",
|
||||
"confirm_delete_count": "1",
|
||||
},
|
||||
follow=True,
|
||||
)
|
||||
@@ -1443,6 +1969,34 @@ class ViewTests(TestCase):
|
||||
self.assertContains(response, "Retention apply confirmation is invalid.")
|
||||
self.assertEqual(SnapshotRecord.objects.count(), 1)
|
||||
|
||||
def test_retention_apply_rejects_mismatched_delete_count_confirmation(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
host = HostConfig.objects.create(
|
||||
host="web-01",
|
||||
address="web-01.example.test",
|
||||
retention_daily=0,
|
||||
retention_weekly=0,
|
||||
retention_monthly=0,
|
||||
retention_yearly=0,
|
||||
)
|
||||
self._snapshot(host, "20260518-021500Z__OLDSNAP")
|
||||
self._snapshot(host, "20260519-021500Z__NEWSNAP")
|
||||
|
||||
response = self.client.post(
|
||||
reverse("apply_host_retention", args=[host.host]),
|
||||
{
|
||||
"kind": "scheduled",
|
||||
"max_delete": "1",
|
||||
"confirm_host": host.host,
|
||||
"confirm_delete_count": "0",
|
||||
},
|
||||
follow=True,
|
||||
)
|
||||
|
||||
self.assertRedirects(response, reverse("host_retention_plan", args=[host.host]))
|
||||
self.assertContains(response, "Retention apply confirmation is invalid.")
|
||||
self.assertEqual(SnapshotRecord.objects.count(), 2)
|
||||
|
||||
def test_retention_apply_requires_post(self) -> None:
|
||||
self.client.force_login(self.staff_user)
|
||||
host = HostConfig.objects.create(host="web-01", address="web-01.example.test")
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shlex
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
@@ -9,11 +10,12 @@ from django.contrib import messages
|
||||
from django.contrib.admin.views.decorators import staff_member_required
|
||||
from django.conf import settings
|
||||
from django.http import FileResponse, Http404
|
||||
from django.db.models import Count
|
||||
from django.db.models import Count, Q
|
||||
from django.shortcuts import get_object_or_404, redirect, render
|
||||
from django.utils import timezone
|
||||
from django.views.decorators.http import require_POST
|
||||
|
||||
from pobsync import __version__
|
||||
from pobsync.errors import PobsyncError
|
||||
from pobsync.config.defaults import DEFAULT_EXCLUDES, DEFAULT_RSYNC_ARGS
|
||||
|
||||
@@ -23,6 +25,7 @@ from .forms import (
|
||||
CreateHostConfigForm,
|
||||
GlobalConfigForm,
|
||||
HostConfigForm,
|
||||
IncompleteCleanupForm,
|
||||
ManualBackupForm,
|
||||
RetentionApplyForm,
|
||||
SshCredentialGenerateForm,
|
||||
@@ -30,9 +33,9 @@ from .forms import (
|
||||
SshCredentialForm,
|
||||
)
|
||||
from .host_ops import ensure_host_directories
|
||||
from .models import BackupRun, GlobalConfig, HostConfig, ScheduleConfig, SnapshotRecord, SshCredential
|
||||
from .models import BackupRun, GlobalConfig, HostConfig, PurgedSnapshot, ScheduleConfig, SnapshotRecord, SshCredential
|
||||
from .preflight import collect_backup_gate, effective_host_config_preview, run_remote_preflight
|
||||
from .retention import run_sql_retention_apply, run_sql_retention_plan
|
||||
from .retention import run_incomplete_cleanup, run_sql_retention_apply, run_sql_retention_plan
|
||||
from .self_check import collect_self_checks, summarize_self_checks
|
||||
from .scheduler import next_due_after
|
||||
from .snapshot_discovery import discover_snapshots, inspect_snapshot_discovery
|
||||
@@ -45,7 +48,22 @@ def dashboard(request):
|
||||
global_config = GlobalConfig.objects.filter(name="default").first()
|
||||
hosts = list(
|
||||
HostConfig.objects.select_related("schedule")
|
||||
.annotate(snapshot_count=Count("snapshots", distinct=True), run_count=Count("runs", distinct=True))
|
||||
.annotate(
|
||||
snapshot_count=Count("snapshots", distinct=True),
|
||||
run_count=Count("runs", distinct=True),
|
||||
queued_run_count=Count("runs", filter=Q(runs__status=BackupRun.Status.QUEUED), distinct=True),
|
||||
running_run_count=Count("runs", filter=Q(runs__status=BackupRun.Status.RUNNING), distinct=True),
|
||||
warning_run_count=Count(
|
||||
"runs",
|
||||
filter=Q(runs__status=BackupRun.Status.WARNING, runs__reviewed_at__isnull=True),
|
||||
distinct=True,
|
||||
),
|
||||
failed_run_count=Count(
|
||||
"runs",
|
||||
filter=Q(runs__status=BackupRun.Status.FAILED, runs__reviewed_at__isnull=True),
|
||||
distinct=True,
|
||||
),
|
||||
)
|
||||
.order_by("host")
|
||||
)
|
||||
for host_config in hosts:
|
||||
@@ -55,6 +73,7 @@ def dashboard(request):
|
||||
.first()
|
||||
)
|
||||
host_config.next_run_at = _next_run_for_host(host_config)
|
||||
host_config.retention_warning = _retention_warning_for_host(host_config, _schedule_for_host(host_config))
|
||||
stats_summary = collect_dashboard_stats(hosts=hosts, global_config=global_config)
|
||||
context = {
|
||||
"hosts": hosts,
|
||||
@@ -70,13 +89,43 @@ def dashboard(request):
|
||||
"enabled_schedules": ScheduleConfig.objects.filter(enabled=True).count(),
|
||||
"snapshots": SnapshotRecord.objects.count(),
|
||||
"runs": BackupRun.objects.count(),
|
||||
"queued_runs": BackupRun.objects.filter(status=BackupRun.Status.QUEUED).count(),
|
||||
"running_runs": BackupRun.objects.filter(status=BackupRun.Status.RUNNING).count(),
|
||||
"failed_runs": BackupRun.objects.filter(status=BackupRun.Status.FAILED).count(),
|
||||
"warning_runs": BackupRun.objects.filter(
|
||||
status=BackupRun.Status.WARNING,
|
||||
reviewed_at__isnull=True,
|
||||
).count(),
|
||||
"failed_runs": BackupRun.objects.filter(
|
||||
status=BackupRun.Status.FAILED,
|
||||
reviewed_at__isnull=True,
|
||||
).count(),
|
||||
},
|
||||
}
|
||||
return render(request, "pobsync_backend/dashboard.html", context)
|
||||
|
||||
|
||||
@staff_member_required
|
||||
def changelog(request):
|
||||
changelog_path = Path(settings.BASE_DIR) / "CHANGELOG.md"
|
||||
try:
|
||||
changelog_text = changelog_path.read_text(encoding="utf-8")
|
||||
missing = False
|
||||
except FileNotFoundError:
|
||||
changelog_text = "CHANGELOG.md was not found in this installation."
|
||||
missing = True
|
||||
|
||||
return render(
|
||||
request,
|
||||
"pobsync_backend/changelog.html",
|
||||
{
|
||||
"app_version": __version__,
|
||||
"changelog_blocks": _parse_changelog(changelog_text),
|
||||
"changelog_path": changelog_path,
|
||||
"missing": missing,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@staff_member_required
|
||||
def self_check(request):
|
||||
checks = collect_self_checks()
|
||||
@@ -96,6 +145,27 @@ def logs(request):
|
||||
return render(request, "pobsync_backend/logs.html", context)
|
||||
|
||||
|
||||
@staff_member_required
|
||||
def purged_snapshots(request):
|
||||
host = request.GET.get("host", "").strip()
|
||||
action = request.GET.get("action", "").strip()
|
||||
purged = PurgedSnapshot.objects.select_related("host").order_by("-purged_at", "host_name", "dirname")
|
||||
if host:
|
||||
purged = purged.filter(host_name=host)
|
||||
if action:
|
||||
purged = purged.filter(action=action)
|
||||
|
||||
context = {
|
||||
"purged_snapshots": purged[:200],
|
||||
"hosts": HostConfig.objects.order_by("host"),
|
||||
"actions": PurgedSnapshot.Action.choices,
|
||||
"selected_host": host,
|
||||
"selected_action": action,
|
||||
"total_count": purged.count(),
|
||||
}
|
||||
return render(request, "pobsync_backend/purged_snapshots.html", context)
|
||||
|
||||
|
||||
@staff_member_required
|
||||
def ssh_credentials(request):
|
||||
context = {
|
||||
@@ -190,6 +260,9 @@ def delete_ssh_credential(request, credential_id: int):
|
||||
if credential.hosts.exists() or credential.global_configs.exists():
|
||||
messages.error(request, f"SSH key {credential.name} is still in use and cannot be deleted.")
|
||||
return redirect("edit_ssh_credential", credential_id=credential.id)
|
||||
if request.POST.get("confirm_name", "").strip() != credential.name:
|
||||
messages.error(request, f"Type {credential.name} to confirm SSH key deletion.")
|
||||
return redirect("edit_ssh_credential", credential_id=credential.id)
|
||||
|
||||
name = credential.name
|
||||
try:
|
||||
@@ -274,6 +347,7 @@ def host_detail(request, host: str):
|
||||
context = {
|
||||
"host": host_config,
|
||||
"schedule": schedule,
|
||||
"retention_warning": _retention_warning_for_host(host_config, schedule),
|
||||
"next_run_at": _next_run_for_schedule(schedule, host_config),
|
||||
"scheduler_timezone": timezone.get_current_timezone_name(),
|
||||
"discovery": inspect_snapshot_discovery(host=host_config),
|
||||
@@ -295,7 +369,10 @@ def host_detail(request, host: str):
|
||||
"runs": host_config.runs.count(),
|
||||
"queued_runs": queued_runs.count(),
|
||||
"running_runs": running_runs.count(),
|
||||
"failed_runs": host_config.runs.filter(status=BackupRun.Status.FAILED).count(),
|
||||
"failed_runs": host_config.runs.filter(
|
||||
status=BackupRun.Status.FAILED,
|
||||
reviewed_at__isnull=True,
|
||||
).count(),
|
||||
"incomplete_snapshots": host_config.snapshots.filter(kind=SnapshotRecord.Kind.INCOMPLETE).count(),
|
||||
},
|
||||
}
|
||||
@@ -413,6 +490,7 @@ def run_detail(request, run_id: int):
|
||||
rsync_result = result.get("rsync") if isinstance(result.get("rsync"), dict) else {}
|
||||
failure = result.get("failure") if isinstance(result.get("failure"), dict) else {}
|
||||
prune_result = result.get("prune") if isinstance(result.get("prune"), dict) else {}
|
||||
execution = result.get("execution") if isinstance(result.get("execution"), dict) else {}
|
||||
rsync_log_path = _run_rsync_log_path(run)
|
||||
rsync_log_tail = _run_rsync_log_tail(rsync_result, rsync_log_path)
|
||||
requested = result.get("requested") if isinstance(result.get("requested"), dict) else {}
|
||||
@@ -420,6 +498,7 @@ def run_detail(request, run_id: int):
|
||||
"run": run,
|
||||
"can_cancel": run.status in {BackupRun.Status.QUEUED, BackupRun.Status.RUNNING},
|
||||
"requested": requested,
|
||||
"execution": execution,
|
||||
"stats": run_stats if isinstance(run_stats, dict) else {},
|
||||
"rsync": rsync_result,
|
||||
"rsync_command": _run_rsync_command(rsync_result),
|
||||
@@ -427,6 +506,7 @@ def run_detail(request, run_id: int):
|
||||
"failure_summary": failure.get("message") or failure.get("summary") or "",
|
||||
"prune_result": prune_result,
|
||||
"has_prune_result": bool(prune_result),
|
||||
"retention_warning": _retention_warning_for_host(run.host, _schedule_for_host(run.host)),
|
||||
"rsync_log_path": str(rsync_log_path) if rsync_log_path is not None else "",
|
||||
"rsync_log_exists": bool(rsync_log_path and rsync_log_path.exists()),
|
||||
"rsync_log_tail": rsync_log_tail,
|
||||
@@ -476,18 +556,54 @@ def cancel_run(request, run_id: int):
|
||||
return redirect("run_detail", run_id=run.id)
|
||||
|
||||
|
||||
@staff_member_required
|
||||
@require_POST
|
||||
def resolve_run_review(request, run_id: int):
|
||||
run = get_object_or_404(BackupRun.objects.select_related("host"), id=run_id)
|
||||
if run.status not in {BackupRun.Status.FAILED, BackupRun.Status.WARNING}:
|
||||
messages.warning(request, f"Run {run.id} does not need review.")
|
||||
return redirect("run_detail", run_id=run.id)
|
||||
if run.reviewed_at:
|
||||
messages.info(request, f"Run {run.id} was already marked reviewed.")
|
||||
return redirect("run_detail", run_id=run.id)
|
||||
|
||||
run.reviewed_at = timezone.now()
|
||||
run.reviewed_by = request.user.get_username()
|
||||
run.save(update_fields=["reviewed_at", "reviewed_by"])
|
||||
messages.success(request, f"Run {run.id} marked reviewed.")
|
||||
return redirect("run_detail", run_id=run.id)
|
||||
|
||||
|
||||
@staff_member_required
|
||||
@require_POST
|
||||
def resolve_host_incomplete_reviews(request, host: str):
|
||||
host_config = get_object_or_404(HostConfig, host=host)
|
||||
reviewed_count = host_config.snapshots.filter(
|
||||
kind=SnapshotRecord.Kind.INCOMPLETE,
|
||||
reviewed_at__isnull=True,
|
||||
).update(reviewed_at=timezone.now(), reviewed_by=request.user.get_username())
|
||||
|
||||
if reviewed_count:
|
||||
messages.success(request, f"Marked {reviewed_count} incomplete snapshot(s) reviewed for {host_config.host}.")
|
||||
else:
|
||||
messages.info(request, f"No incomplete snapshots needed review for {host_config.host}.")
|
||||
return redirect("host_detail", host=host_config.host)
|
||||
|
||||
|
||||
@staff_member_required
|
||||
def snapshot_detail(request, snapshot_id: int):
|
||||
snapshot = get_object_or_404(
|
||||
SnapshotRecord.objects.select_related("host", "base").prefetch_related("derived_snapshots", "backup_runs"),
|
||||
id=snapshot_id,
|
||||
)
|
||||
restore = _snapshot_restore_guidance(snapshot)
|
||||
context = {
|
||||
"snapshot": snapshot,
|
||||
"stats": snapshot.metadata.get("stats") if isinstance(snapshot.metadata, dict) else {},
|
||||
"metadata_json": _pretty_json(snapshot.metadata),
|
||||
"backup_runs": snapshot.backup_runs.select_related("host").order_by("-created_at"),
|
||||
"derived_snapshots": snapshot.derived_snapshots.select_related("host").order_by("-started_at", "dirname"),
|
||||
"restore": restore,
|
||||
}
|
||||
return render(request, "pobsync_backend/snapshot_detail.html", context)
|
||||
|
||||
@@ -526,17 +642,34 @@ def host_retention_plan(request, host: str):
|
||||
except PobsyncError as exc:
|
||||
messages.error(request, str(exc))
|
||||
return redirect("host_detail", host=host_config.host)
|
||||
schedule = _schedule_for_host(host_config)
|
||||
scheduled_prune_limit = schedule.prune_max_delete if schedule and schedule.prune else None
|
||||
delete_count = len(plan["delete"])
|
||||
incomplete_count = len(plan["incomplete"])
|
||||
context = {
|
||||
"host": host_config,
|
||||
"kind": kind,
|
||||
"protect_bases": protect_bases,
|
||||
"plan": plan,
|
||||
"schedule": schedule,
|
||||
"scheduled_prune_limit": scheduled_prune_limit,
|
||||
"scheduled_prune_exceeded": scheduled_prune_limit is not None and delete_count > scheduled_prune_limit,
|
||||
"apply_form": RetentionApplyForm(
|
||||
host_name=host_config.host,
|
||||
expected_delete_count=delete_count,
|
||||
initial={
|
||||
"kind": kind,
|
||||
"protect_bases": protect_bases,
|
||||
"max_delete": len(plan["delete"]),
|
||||
"max_delete": delete_count,
|
||||
"confirm_delete_count": delete_count,
|
||||
},
|
||||
),
|
||||
"incomplete_cleanup_form": IncompleteCleanupForm(
|
||||
host_name=host_config.host,
|
||||
expected_delete_count=incomplete_count,
|
||||
initial={
|
||||
"max_delete": incomplete_count,
|
||||
"confirm_delete_count": incomplete_count,
|
||||
},
|
||||
),
|
||||
}
|
||||
@@ -547,7 +680,26 @@ def host_retention_plan(request, host: str):
|
||||
@require_POST
|
||||
def apply_host_retention(request, host: str):
|
||||
host_config = get_object_or_404(HostConfig, host=host)
|
||||
form = RetentionApplyForm(request.POST, host_name=host_config.host)
|
||||
raw_kind = request.POST.get("kind", "scheduled")
|
||||
raw_protect_bases = request.POST.get("protect_bases") in {"1", "true", "on", "yes"}
|
||||
expected_delete_count = None
|
||||
if raw_kind in {"scheduled", "manual", "all"}:
|
||||
try:
|
||||
plan = run_sql_retention_plan(
|
||||
host=host_config.host,
|
||||
kind=raw_kind,
|
||||
protect_bases=raw_protect_bases,
|
||||
)
|
||||
except PobsyncError as exc:
|
||||
messages.error(request, str(exc))
|
||||
return redirect("host_retention_plan", host=host_config.host)
|
||||
expected_delete_count = len(plan.get("delete") or [])
|
||||
|
||||
form = RetentionApplyForm(
|
||||
request.POST,
|
||||
host_name=host_config.host,
|
||||
expected_delete_count=expected_delete_count,
|
||||
)
|
||||
if not form.is_valid():
|
||||
messages.error(request, "Retention apply confirmation is invalid.")
|
||||
return redirect("host_retention_plan", host=host_config.host)
|
||||
@@ -562,6 +714,8 @@ def apply_host_retention(request, host: str):
|
||||
protect_bases=protect_bases,
|
||||
yes=True,
|
||||
max_delete=form.cleaned_data["max_delete"],
|
||||
action=PurgedSnapshot.Action.MANUAL,
|
||||
triggered_by=request.user.get_username(),
|
||||
)
|
||||
except PobsyncError as exc:
|
||||
messages.error(request, str(exc))
|
||||
@@ -576,6 +730,41 @@ def apply_host_retention(request, host: str):
|
||||
return target
|
||||
|
||||
|
||||
@staff_member_required
|
||||
@require_POST
|
||||
def cleanup_host_incomplete_snapshots(request, host: str):
|
||||
host_config = get_object_or_404(HostConfig, host=host)
|
||||
try:
|
||||
plan = run_sql_retention_plan(host=host_config.host, kind="all", protect_bases=True)
|
||||
except PobsyncError as exc:
|
||||
messages.error(request, str(exc))
|
||||
return redirect("host_retention_plan", host=host_config.host)
|
||||
|
||||
incomplete_count = len(plan.get("incomplete") or [])
|
||||
form = IncompleteCleanupForm(
|
||||
request.POST,
|
||||
host_name=host_config.host,
|
||||
expected_delete_count=incomplete_count,
|
||||
)
|
||||
if not form.is_valid():
|
||||
messages.error(request, "Incomplete cleanup confirmation is invalid.")
|
||||
return redirect("host_retention_plan", host=host_config.host)
|
||||
|
||||
try:
|
||||
result = run_incomplete_cleanup(
|
||||
prefix=Path(settings.POBSYNC_HOME),
|
||||
host=host_config.host,
|
||||
yes=True,
|
||||
max_delete=form.cleaned_data["max_delete"],
|
||||
triggered_by=request.user.get_username(),
|
||||
)
|
||||
except PobsyncError as exc:
|
||||
messages.error(request, str(exc))
|
||||
else:
|
||||
messages.success(request, f"Deleted {len(result['deleted'])} incomplete snapshot(s) for {host_config.host}.")
|
||||
return redirect("host_retention_plan", host=host_config.host)
|
||||
|
||||
|
||||
@staff_member_required
|
||||
def edit_host_config(request, host: str):
|
||||
host_config = get_object_or_404(HostConfig, host=host)
|
||||
@@ -652,6 +841,46 @@ def _next_run_for_schedule(schedule: ScheduleConfig | None, host_config: HostCon
|
||||
return None
|
||||
|
||||
|
||||
def _retention_warning_for_host(host_config: HostConfig, schedule: ScheduleConfig | None) -> dict[str, object]:
|
||||
incomplete_count = host_config.snapshots.filter(
|
||||
kind=SnapshotRecord.Kind.INCOMPLETE,
|
||||
reviewed_at__isnull=True,
|
||||
).count()
|
||||
warning: dict[str, object] = {
|
||||
"has_warning": incomplete_count > 0,
|
||||
"incomplete_count": incomplete_count,
|
||||
}
|
||||
if schedule is None or not schedule.prune or not host_config.enabled:
|
||||
return warning
|
||||
try:
|
||||
plan = run_sql_retention_plan(
|
||||
host=host_config.host,
|
||||
kind="scheduled",
|
||||
protect_bases=bool(schedule.prune_protect_bases),
|
||||
)
|
||||
except PobsyncError as exc:
|
||||
warning.update(
|
||||
{
|
||||
"has_warning": True,
|
||||
"error": str(exc),
|
||||
}
|
||||
)
|
||||
return warning
|
||||
|
||||
delete_count = len(plan.get("delete") or [])
|
||||
warning.update(
|
||||
{
|
||||
"delete_count": delete_count,
|
||||
"max_delete": schedule.prune_max_delete,
|
||||
"protect_bases": bool(schedule.prune_protect_bases),
|
||||
"prune_exceeded": delete_count > schedule.prune_max_delete,
|
||||
}
|
||||
)
|
||||
if warning["prune_exceeded"]:
|
||||
warning["has_warning"] = True
|
||||
return warning
|
||||
|
||||
|
||||
def _default_schedule_initial() -> dict[str, object]:
|
||||
return {
|
||||
"cron_expr": "15 2 * * *",
|
||||
@@ -714,6 +943,81 @@ def _pretty_json(value: object) -> str:
|
||||
return json.dumps(value or {}, indent=2, sort_keys=True)
|
||||
|
||||
|
||||
def _parse_changelog(text: str) -> list[dict[str, object]]:
|
||||
blocks: list[dict[str, object]] = []
|
||||
paragraph: list[str] = []
|
||||
list_items: list[str] = []
|
||||
|
||||
def flush_paragraph() -> None:
|
||||
if paragraph:
|
||||
blocks.append({"kind": "paragraph", "text": " ".join(paragraph)})
|
||||
paragraph.clear()
|
||||
|
||||
def flush_list() -> None:
|
||||
if list_items:
|
||||
blocks.append({"kind": "list", "items": list(list_items)})
|
||||
list_items.clear()
|
||||
|
||||
for raw_line in text.splitlines():
|
||||
line = raw_line.strip()
|
||||
if not line:
|
||||
flush_paragraph()
|
||||
flush_list()
|
||||
continue
|
||||
|
||||
if line.startswith("#"):
|
||||
flush_paragraph()
|
||||
flush_list()
|
||||
marker, _space, heading = line.partition(" ")
|
||||
level = min(max(len(marker), 1), 3)
|
||||
blocks.append({"kind": "heading", "level": level, "text": heading.strip() or line.lstrip("#").strip()})
|
||||
continue
|
||||
|
||||
if line.startswith("- "):
|
||||
flush_paragraph()
|
||||
list_items.append(line[2:].strip())
|
||||
continue
|
||||
|
||||
flush_list()
|
||||
paragraph.append(line)
|
||||
|
||||
flush_paragraph()
|
||||
flush_list()
|
||||
return blocks
|
||||
|
||||
|
||||
def _snapshot_restore_guidance(snapshot: SnapshotRecord) -> dict[str, str]:
|
||||
source_path = Path(snapshot.path) / "data"
|
||||
destination_path = Path("/restore") / snapshot.host.host
|
||||
example_relative_path = Path("etc") / "nginx"
|
||||
example_file_relative_path = Path("home") / "example" / "site" / "public_html" / "index.php"
|
||||
quoted_source = _quote_path_with_trailing_slash(source_path)
|
||||
quoted_destination = _quote_path_with_trailing_slash(destination_path)
|
||||
quoted_partial_source = _quote_path_with_trailing_slash(source_path / example_relative_path)
|
||||
quoted_partial_destination = _quote_path_with_trailing_slash(destination_path / example_relative_path)
|
||||
quoted_file_source = shlex.quote(str(source_path / example_file_relative_path))
|
||||
quoted_file_destination = shlex.quote(str(destination_path / example_file_relative_path))
|
||||
quoted_remote_destination = shlex.quote(f"root@{snapshot.host.address or snapshot.host.host}:/")
|
||||
common_args = "rsync -aHAX --numeric-ids --info=progress2"
|
||||
|
||||
return {
|
||||
"source_path": str(source_path),
|
||||
"destination_path": str(destination_path),
|
||||
"example_relative_path": str(example_relative_path),
|
||||
"example_file_relative_path": str(example_file_relative_path),
|
||||
"inspect_command": f"ls -la {quoted_source}",
|
||||
"dry_run_command": f"{common_args} --dry-run {quoted_source} {quoted_destination}",
|
||||
"local_command": f"{common_args} {quoted_source} {quoted_destination}",
|
||||
"partial_dry_run_command": f"{common_args} --dry-run {quoted_partial_source} {quoted_partial_destination}",
|
||||
"file_dry_run_command": f"{common_args} --dry-run {quoted_file_source} {quoted_file_destination}",
|
||||
"remote_dry_run_command": f"{common_args} --dry-run {quoted_source} {quoted_remote_destination}",
|
||||
}
|
||||
|
||||
|
||||
def _quote_path_with_trailing_slash(path: Path) -> str:
|
||||
return shlex.quote(str(path).rstrip("/") + "/")
|
||||
|
||||
|
||||
def _run_rsync_log_path(run: BackupRun) -> Path | None:
|
||||
if isinstance(run.result, dict):
|
||||
log = run.result.get("log")
|
||||
|
||||
@@ -8,8 +8,10 @@ from pobsync_backend import api, views
|
||||
|
||||
urlpatterns = [
|
||||
path("", views.dashboard, name="dashboard"),
|
||||
path("changelog/", views.changelog, name="changelog"),
|
||||
path("self-check/", views.self_check, name="self_check"),
|
||||
path("logs/", views.logs, name="logs"),
|
||||
path("purged-snapshots/", views.purged_snapshots, name="purged_snapshots"),
|
||||
path("config/global/", views.edit_global_config, name="edit_global_config"),
|
||||
path("ssh-credentials/", views.ssh_credentials, name="ssh_credentials"),
|
||||
path("ssh-credentials/new/", views.create_ssh_credential, name="create_ssh_credential"),
|
||||
@@ -26,10 +28,21 @@ urlpatterns = [
|
||||
path("hosts/<str:host>/discover-snapshots/", views.discover_host_snapshots, name="discover_host_snapshots"),
|
||||
path("hosts/<str:host>/retention-apply/", views.apply_host_retention, name="apply_host_retention"),
|
||||
path("hosts/<str:host>/retention-plan/", views.host_retention_plan, name="host_retention_plan"),
|
||||
path(
|
||||
"hosts/<str:host>/incomplete-cleanup/",
|
||||
views.cleanup_host_incomplete_snapshots,
|
||||
name="cleanup_host_incomplete_snapshots",
|
||||
),
|
||||
path("hosts/<str:host>/schedule/", views.edit_host_schedule, name="edit_host_schedule"),
|
||||
path("runs/<int:run_id>/", views.run_detail, name="run_detail"),
|
||||
path("runs/<int:run_id>/rsync-log/", views.run_rsync_log, name="run_rsync_log"),
|
||||
path("runs/<int:run_id>/cancel/", views.cancel_run, name="cancel_run"),
|
||||
path("runs/<int:run_id>/resolve-review/", views.resolve_run_review, name="resolve_run_review"),
|
||||
path(
|
||||
"hosts/<str:host>/resolve-incomplete-reviews/",
|
||||
views.resolve_host_incomplete_reviews,
|
||||
name="resolve_host_incomplete_reviews",
|
||||
),
|
||||
path("snapshots/<int:snapshot_id>/", views.snapshot_detail, name="snapshot_detail"),
|
||||
path("api/", api.api_index),
|
||||
path("api/status/", api.status),
|
||||
|
||||
Reference in New Issue
Block a user