le-app/docs/le-app-database-migration/phase2a-host-maintenance/README.md

# Phase 2A scheduled host-maintenance review package

Status: **review-only and uncommitted**. Operator provisionally prefers Path A, a scheduled complete Arch host upgrade. This is not execution approval.

Proposed package command:

```bash
pacman -Syu --needed rootlesskit slirp4netns openai-codex
```

This package deliberately separates:

1. read-only preflight: `preflight-read-only.sh`;
2. package execution: `execute-maintenance.sh` — generated, never run during preparation;
3. reboot: `reboot-procedure.md` — separate approval and manual command;
4. post-reboot validation: `post-reboot-validate.sh`;
5. later contained rootless-Docker provisioning — explicitly outside Phase 2A.

The scripts do not enable lingering, configure rootless Docker, modify systemd or nftables, authenticate Codex, copy data, or change production configuration. Package hooks may restart services during the approved upgrade; the execution script records this but does not add its own service restarts.

## Required operator inputs

Before execution, record site-specific values in the maintenance ticket:

- independent console/recovery method and a tested root login path;
- public health URLs, including the canonical application utility endpoint;
- production-critical unit names beyond the conservative defaults in the scripts;
- expected running Docker container names and health states;
- bootloader type and known-good previous kernel entry;
- backup identifiers and restoration instructions;
- maintenance window, user notification, and rollback decision owner.

URLs can be supplied without editing the scripts:

```bash
export PUBLIC_HEALTH_URLS='https://db.lasereverything.net/'
```

Health URLs must be non-secret public HTTPS URLs. The scripts reject user-info credentials, query strings, fragments, whitespace/control characters, and non-HTTPS schemes. Never place tokens, signed parameters, credentials, or private endpoints in `PUBLIC_HEALTH_URLS`; the scripts do not read environment files to discover URLs and do not log redirected effective URLs.

Unit names can be extended similarly:

```bash
export EXTRA_CRITICAL_UNITS='example.service another.service'
```

The operator must separately verify independent recovery and restorable backups, then set the non-secret attestations:

```bash
export LE_PHASE2A_RECOVERY_READY=YES
export LE_PHASE2A_BACKUPS_READY=YES
```

Both attestations remain unset in `maintenance-worksheet.md`. Do not set them
until independent recovery access and the proportional backup plan are complete.

## Invocation and enforced records

Invoke every script as a separate process, never with `source` or `.`. Explicitly capture its return status:

```bash
sudo --preserve-env=PUBLIC_HEALTH_URLS,EXTRA_CRITICAL_UNITS,LE_PHASE2A_RECOVERY_READY,LE_PHASE2A_BACKUPS_READY \
  bash docs/le-app-database-migration/phase2a-host-maintenance/preflight-read-only.sh
preflight_rc=$?
printf 'preflight_rc=%s\n' "$preflight_rc"
```

The preflight creates an atomic unique root-owned mode `0700` directory below `/var/log/le-phase2a-preflight`, prints its path, and returns nonzero on `HARD_STOP`. Its machine-readable `result.manifest` records hostname, boot ID, kernel, timestamp, result, failures, and inventory hashes. Maintenance accepts only a checksum-valid `PASS` record from the same host and boot that is no more than **1,800 seconds (30 minutes)** old:

```bash
sudo bash docs/le-app-database-migration/phase2a-host-maintenance/execute-maintenance.sh \
  --preflight /var/log/le-phase2a-preflight/preflight.YYYYMMDDTHHMMSSZ.XXXXXXXX
maintenance_rc=$?
printf 'maintenance_rc=%s\n' "$maintenance_rc"
```

Maintenance and post-reboot validation create their own atomic unique root-only records below `/var/log/le-phase2a-maintenance` and `/var/log/le-phase2a-post-reboot`. Post-reboot invocation requires the exact maintenance record:

```bash
sudo bash docs/le-app-database-migration/phase2a-host-maintenance/post-reboot-validate.sh \
  /var/log/le-phase2a-maintenance/maintenance.YYYYMMDDTHHMMSSZ.XXXXXXXX
validation_rc=$?
printf 'validation_rc=%s\n' "$validation_rc"
```

These records can contain IP addresses, usernames, unit/container names, package state, and limited relevant journal metadata. They never intentionally capture process/container environments, environment-file contents, private keys, credentials, secret files, or command history. Treat every record as operationally sensitive: keep it root-only, do not commit it, and disclose it only through the approved maintenance channel.

## Transaction comparison limitation

The maintenance script refreshes an isolated sync database, normalizes package name, old version, new version, action, count, and downgrade status, and requires exact equality with the approved retained preview. Any detectable mismatch stops execution.

The final live `pacman -Syu` must refresh the live sync databases as part of the complete transaction. Pacman cannot expose that post-refresh final interactive transaction for wrapper comparison without either staging a live `pacman -Sy`-only state or replacing the transaction with an unsafe custom workflow. The script therefore requires two exact operator phrases before launch and then requires the operator to review Pacman's final package table at its native interactive `Y/n` prompt. Answer `n` on any difference. This limitation is explicit and is not represented as machine-enforced equality.

## Approved package-preview evidence

The refreshed post-NoMachine preview is:

```text
/tmp/le-phase2-package-decision-post-nomachine-20260712.1vyjKk
```

Its durable normalized baseline and inventories are stored in
`approved-transaction/`. The authoritative transaction is 220 packages: 215
upgrades, 5 new packages, 1,985.98 MiB download, and +760.30 MiB installed-size
delta. The prior 221-package transaction is superseded.

Exactly 234 failed `drkonqi-coredump-processor@…` units are reviewed as
nonproduction desktop crash-processing failures. Their checksummed complete
state and invocation-ID fingerprint is under `failed-unit-disposition/`. Any
addition, removal, state or invocation change, checksum failure, or other failed
unit is a hard stop. The scripts do not reset them. `nxserver.service` is not an
expected critical unit; its unused `nomachine` package was removed normally.

## Docker disposition and authoritative baseline

The checksummed `container-disposition/` baseline contains exactly 88 containers:
87 running, retired `directus` intentionally stopped, and `sonarr` as the sole
unhealthy running container. Sonarr is a reviewed unrelated pre-existing fault;
its empty configuration, absent port 8989 listener, and failing health check do
not require repair for Phase 2A. Any identity/state drift or any additional
unhealthy container invalidates preflight.

All 87 running containers are authoritative for post-reboot validation. Healthy
containers must return healthy; containers without health checks must return
running. Sonarr may remain unhealthy or improve, but must remain present and
running. Directus must remain stopped and unchanged. `castopod` and
`castopod-redis` use restart policy `no`; failure to return is recorded as a
validation failure requiring explicit operator attention, never an automatic
start or restart.

The protected Directus archive at `/srv/directus-archive/20260710-175408`
verified successfully against its actual `SHA256SUMS` with return code 0.
Preflight repeats that check silently with `sha256sum --status -c SHA256SUMS` and
hard-stops on failure without printing archive contents.

## Snapper readiness history and maintenance gate

On 2026-07-12 the ordinary empty `/.snapshots` directory was replaced by Btrfs
subvolume ID 300, owned `root:root` mode `0750`. Manual snapshot 1 succeeded and
exact snap-pac testing created and verified snapshots 2 and 3. Btrfs quotas were
intentionally left disabled.

Preflight verifies that `/.snapshots` remains a Btrfs subvolume. After both
operator approvals and immediately before Pacman, maintenance creates a uniquely
described explicit Snapper snapshot, captures and validates its numeric ID,
verifies its numbered directory and Btrfs snapshot subvolume, and records the ID
in the result. It verifies the snapshot again after Pacman, independently of
snap-pac's return code. Unavailable quota accounting is a warning, not evidence
of snapshot failure. No script enables quotas or runs `snapper setup-quota`.

The older preview remains at:

```text
/tmp/le-phase2-package-decision-20260712.Mn7mIx
```

It occupies approximately 2.4 GiB and must not be deleted yet. It is historical,
not the approved baseline. Cleanup remains a later explicit action.

GRUB/grub-btrfs findings and the unexecuted standalone generation procedure are
in `grub-btrfs-readiness.md`. Snapshot coverage, external subvolumes/mounts, and
the minimum application-consistent database/configuration backup set are in
`backup-readiness-plan.md`.

## Hard stop conditions

Stop rather than improvise if any of the following occurs:

- no independent recovery access or root path;
- insufficient space or an unhealthy `/boot`;
- degraded storage, failed backup, undispositioned failed units, or an unhealthy production service/container other than the exact reviewed Sonarr disposition;
- unexpected package count/version, downgrade, repository error, signature error, keyring error, mirror inconsistency, or partial transaction;
- SSH access is unstable before the OpenSSH restart hook;
- DKMS/initramfs/kernel/bootloader generation fails;
- package database inconsistency or unexplained `.pacnew` affecting boot, SSH, PAM, networking, PHP, database, mail, proxy, or firewall configuration;
- production containers or public endpoints fail after the transaction;
- the operator cannot identify a known-good previous kernel and recovery route.

Do not proceed from package maintenance into rootless-Docker provisioning. That later work remains separately approval-gated by `host-runtime-plan.md`.