le-app/docs/le-app-database-migration/phase2a-host-maintenance/README.md

# Phase 2A scheduled host-maintenance review package

Status: **review-only and uncommitted**. Operator provisionally prefers Path A, a scheduled complete Arch host upgrade. This is not execution approval.

Proposed package command:

```bash
pacman -Syu --needed rootlesskit slirp4netns openai-codex
```

This package deliberately separates:

1. read-only preflight: `preflight-read-only.sh`;
2. package execution: `execute-maintenance.sh` — generated, never run during preparation;
3. reboot: `reboot-procedure.md` — separate approval and manual command;
4. post-reboot validation: `post-reboot-validate.sh`;
5. later contained rootless-Docker provisioning — explicitly outside Phase 2A.

The scripts do not enable lingering, configure rootless Docker, modify systemd or nftables, authenticate Codex, copy data, or change production configuration. Package hooks may restart services during the approved upgrade; the execution script records this but does not add its own service restarts.

## Required operator inputs

Before execution, record site-specific values in the maintenance ticket:

- independent console/recovery method and a tested root login path;
- public health URLs, including the canonical application utility endpoint;
- production-critical unit names beyond the conservative defaults in the scripts;
- expected running Docker container names and health states;
- bootloader type and known-good previous kernel entry;
- backup identifiers and restoration instructions;
- maintenance window, user notification, and rollback decision owner.

URLs can be supplied without editing the scripts:

```bash
export PUBLIC_HEALTH_URLS='https://example.invalid/health https://example.invalid/'
```

Health URLs must be non-secret public HTTPS URLs. The scripts reject user-info credentials, query strings, fragments, whitespace/control characters, and non-HTTPS schemes. Never place tokens, signed parameters, credentials, or private endpoints in `PUBLIC_HEALTH_URLS`; the scripts do not read environment files to discover URLs and do not log redirected effective URLs.

Unit names can be extended similarly:

```bash
export EXTRA_CRITICAL_UNITS='example.service another.service'
```

The operator must separately verify independent recovery and restorable backups, then set the non-secret attestations:

```bash
export LE_PHASE2A_RECOVERY_READY=YES
export LE_PHASE2A_BACKUPS_READY=YES
```

## Invocation and enforced records

Invoke every script as a separate process, never with `source` or `.`. Explicitly capture its return status:

```bash
sudo --preserve-env=PUBLIC_HEALTH_URLS,EXTRA_CRITICAL_UNITS,LE_PHASE2A_RECOVERY_READY,LE_PHASE2A_BACKUPS_READY \
  bash docs/le-app-database-migration/phase2a-host-maintenance/preflight-read-only.sh
preflight_rc=$?
printf 'preflight_rc=%s\n' "$preflight_rc"
```

The preflight creates an atomic unique root-owned mode `0700` directory below `/var/log/le-phase2a-preflight`, prints its path, and returns nonzero on `HARD_STOP`. Its machine-readable `result.manifest` records hostname, boot ID, kernel, timestamp, result, failures, and inventory hashes. Maintenance accepts only a checksum-valid `PASS` record from the same host and boot that is no more than **1,800 seconds (30 minutes)** old:

```bash
sudo bash docs/le-app-database-migration/phase2a-host-maintenance/execute-maintenance.sh \
  --preflight /var/log/le-phase2a-preflight/preflight.YYYYMMDDTHHMMSSZ.XXXXXXXX
maintenance_rc=$?
printf 'maintenance_rc=%s\n' "$maintenance_rc"
```

Maintenance and post-reboot validation create their own atomic unique root-only records below `/var/log/le-phase2a-maintenance` and `/var/log/le-phase2a-post-reboot`. Post-reboot invocation requires the exact maintenance record:

```bash
sudo bash docs/le-app-database-migration/phase2a-host-maintenance/post-reboot-validate.sh \
  /var/log/le-phase2a-maintenance/maintenance.YYYYMMDDTHHMMSSZ.XXXXXXXX
validation_rc=$?
printf 'validation_rc=%s\n' "$validation_rc"
```

These records can contain IP addresses, usernames, unit/container names, package state, and limited relevant journal metadata. They never intentionally capture process/container environments, environment-file contents, private keys, credentials, secret files, or command history. Treat every record as operationally sensitive: keep it root-only, do not commit it, and disclose it only through the approved maintenance channel.

## Transaction comparison limitation

The maintenance script refreshes an isolated sync database, normalizes package name, old version, new version, action, count, and downgrade status, and requires exact equality with the approved retained preview. Any detectable mismatch stops execution.

The final live `pacman -Syu` must refresh the live sync databases as part of the complete transaction. Pacman cannot expose that post-refresh final interactive transaction for wrapper comparison without either staging a live `pacman -Sy`-only state or replacing the transaction with an unsafe custom workflow. The script therefore requires two exact operator phrases before launch and then requires the operator to review Pacman's final package table at its native interactive `Y/n` prompt. Answer `n` on any difference. This limitation is explicit and is not represented as machine-enforced equality.

## Approved package-preview evidence

The refreshed post-NoMachine preview is:

```text
/tmp/le-phase2-package-decision-post-nomachine-20260712.1vyjKk
```

Its durable normalized baseline and inventories are stored in
`approved-transaction/`. The authoritative transaction is 220 packages: 215
upgrades, 5 new packages, 1,985.98 MiB download, and +760.30 MiB installed-size
delta. The prior 221-package transaction is superseded.

Exactly 234 failed `drkonqi-coredump-processor@…` units are reviewed as
nonproduction desktop crash-processing failures. Their checksummed complete
state and invocation-ID fingerprint is under `failed-unit-disposition/`. Any
addition, removal, state or invocation change, checksum failure, or other failed
unit is a hard stop. The scripts do not reset them. `nxserver.service` is not an
expected critical unit; its unused `nomachine` package was removed normally.

## Snapper readiness history and maintenance gate

On 2026-07-12 the ordinary empty `/.snapshots` directory was replaced by Btrfs
subvolume ID 300, owned `root:root` mode `0750`. Manual snapshot 1 succeeded and
exact snap-pac testing created and verified snapshots 2 and 3. Btrfs quotas were
intentionally left disabled.

Preflight verifies that `/.snapshots` remains a Btrfs subvolume. After both
operator approvals and immediately before Pacman, maintenance creates a uniquely
described explicit Snapper snapshot, captures and validates its numeric ID,
verifies its numbered directory and Btrfs snapshot subvolume, and records the ID
in the result. It verifies the snapshot again after Pacman, independently of
snap-pac's return code. Unavailable quota accounting is a warning, not evidence
of snapshot failure. No script enables quotas or runs `snapper setup-quota`.

The older preview remains at:

```text
/tmp/le-phase2-package-decision-20260712.Mn7mIx
```

It occupies approximately 2.4 GiB and must not be deleted yet. It is historical,
not the approved baseline. Cleanup remains a later explicit action.

## Hard stop conditions

Stop rather than improvise if any of the following occurs:

- no independent recovery access or root path;
- insufficient space or an unhealthy `/boot`;
- degraded storage, failed backup, unresolved failed units, or unhealthy production service;
- unexpected package count/version, downgrade, repository error, signature error, keyring error, mirror inconsistency, or partial transaction;
- SSH access is unstable before the OpenSSH restart hook;
- DKMS/initramfs/kernel/bootloader generation fails;
- package database inconsistency or unexplained `.pacnew` affecting boot, SSH, PAM, networking, PHP, database, mail, proxy, or firewall configuration;
- production containers or public endpoints fail after the transaction;
- the operator cannot identify a known-good previous kernel and recovery route.

Do not proceed from package maintenance into rootless-Docker provisioning. That later work remains separately approval-gated by `host-runtime-plan.md`.