add Phase 2A container and recovery readiness gates

This commit is contained in:
makearmy 2026-07-12 18:59:45 -04:00
parent da03be61a9
commit a9fec0d83d
15 changed files with 629 additions and 30 deletions

View file

@ -33,7 +33,7 @@ Before execution, record site-specific values in the maintenance ticket:
URLs can be supplied without editing the scripts:
```bash
export PUBLIC_HEALTH_URLS='https://example.invalid/health https://example.invalid/'
export PUBLIC_HEALTH_URLS='https://db.lasereverything.net/'
```
Health URLs must be non-secret public HTTPS URLs. The scripts reject user-info credentials, query strings, fragments, whitespace/control characters, and non-HTTPS schemes. Never place tokens, signed parameters, credentials, or private endpoints in `PUBLIC_HEALTH_URLS`; the scripts do not read environment files to discover URLs and do not log redirected effective URLs.
@ -51,6 +51,9 @@ export LE_PHASE2A_RECOVERY_READY=YES
export LE_PHASE2A_BACKUPS_READY=YES
```
Both attestations remain unset in `maintenance-worksheet.md`. Do not set them
until independent recovery access and the proportional backup plan are complete.
## Invocation and enforced records
Invoke every script as a separate process, never with `source` or `.`. Explicitly capture its return status:
@ -108,6 +111,28 @@ addition, removal, state or invocation change, checksum failure, or other failed
unit is a hard stop. The scripts do not reset them. `nxserver.service` is not an
expected critical unit; its unused `nomachine` package was removed normally.
## Docker disposition and authoritative baseline
The checksummed `container-disposition/` baseline contains exactly 88 containers:
87 running, retired `directus` intentionally stopped, and `sonarr` as the sole
unhealthy running container. Sonarr is a reviewed unrelated pre-existing fault;
its empty configuration, absent port 8989 listener, and failing health check do
not require repair for Phase 2A. Any identity/state drift or any additional
unhealthy container invalidates preflight.
All 87 running containers are authoritative for post-reboot validation. Healthy
containers must return healthy; containers without health checks must return
running. Sonarr may remain unhealthy or improve, but must remain present and
running. Directus must remain stopped and unchanged. `castopod` and
`castopod-redis` use restart policy `no`; failure to return is recorded as a
validation failure requiring explicit operator attention, never an automatic
start or restart.
The protected Directus archive at `/srv/directus-archive/20260710-175408`
verified successfully against its actual `SHA256SUMS` with return code 0.
Preflight repeats that check silently with `sha256sum --status -c SHA256SUMS` and
hard-stops on failure without printing archive contents.
## Snapper readiness history and maintenance gate
On 2026-07-12 the ordinary empty `/.snapshots` directory was replaced by Btrfs
@ -132,13 +157,18 @@ The older preview remains at:
It occupies approximately 2.4 GiB and must not be deleted yet. It is historical,
not the approved baseline. Cleanup remains a later explicit action.
GRUB/grub-btrfs findings and the unexecuted standalone generation procedure are
in `grub-btrfs-readiness.md`. Snapshot coverage, external subvolumes/mounts, and
the minimum application-consistent database/configuration backup set are in
`backup-readiness-plan.md`.
## Hard stop conditions
Stop rather than improvise if any of the following occurs:
- no independent recovery access or root path;
- insufficient space or an unhealthy `/boot`;
- degraded storage, failed backup, unresolved failed units, or unhealthy production service;
- degraded storage, failed backup, undispositioned failed units, or an unhealthy production service/container other than the exact reviewed Sonarr disposition;
- unexpected package count/version, downgrade, repository error, signature error, keyring error, mirror inconsistency, or partial transaction;
- SSH access is unstable before the OpenSSH restart hook;
- DKMS/initramfs/kernel/bootloader generation fails;