le-app/docs/le-app-database-migration/phase2a-host-maintenance

Phase 2A scheduled host-maintenance review package

Status: review-only and uncommitted. Operator provisionally prefers Path A, a scheduled complete Arch host upgrade. This is not execution approval.

Proposed package command:

pacman -Syu --needed rootlesskit slirp4netns openai-codex

This package deliberately separates:

1. read-only preflight: preflight-read-only.sh;
2. package execution: execute-maintenance.sh — generated, never run during preparation;
3. reboot: reboot-procedure.md — separate approval and manual command;
4. post-reboot validation: post-reboot-validate.sh;
5. later contained rootless-Docker provisioning — explicitly outside Phase 2A.

The scripts do not enable lingering, configure rootless Docker, modify systemd or nftables, authenticate Codex, copy data, or change production configuration. Package hooks may restart services during the approved upgrade; the execution script records this but does not add its own service restarts.

Required operator inputs

Before execution, record site-specific values in the maintenance ticket:

• independent console/recovery method and a tested root login path;
• public health URLs, including the canonical application utility endpoint;
• production-critical unit names beyond the conservative defaults in the scripts;
• expected running Docker container names and health states;
• bootloader type and known-good previous kernel entry;
• backup identifiers and restoration instructions;
• maintenance window, user notification, and rollback decision owner.

URLs can be supplied without editing the scripts:

export PUBLIC_HEALTH_URLS='https://example.invalid/health https://example.invalid/'

Health URLs must be non-secret public HTTPS URLs. The scripts reject user-info credentials, query strings, fragments, whitespace/control characters, and non-HTTPS schemes. Never place tokens, signed parameters, credentials, or private endpoints in PUBLIC_HEALTH_URLS; the scripts do not read environment files to discover URLs and do not log redirected effective URLs.

Unit names can be extended similarly:

export EXTRA_CRITICAL_UNITS='example.service another.service'

The operator must separately verify independent recovery and restorable backups, then set the non-secret attestations:

export LE_PHASE2A_RECOVERY_READY=YES
export LE_PHASE2A_BACKUPS_READY=YES

Invocation and enforced records

Invoke every script as a separate process, never with source or .. Explicitly capture its return status:

sudo --preserve-env=PUBLIC_HEALTH_URLS,EXTRA_CRITICAL_UNITS,LE_PHASE2A_RECOVERY_READY,LE_PHASE2A_BACKUPS_READY \
  bash docs/le-app-database-migration/phase2a-host-maintenance/preflight-read-only.sh
preflight_rc=$?
printf 'preflight_rc=%s\n' "$preflight_rc"

The preflight creates an atomic unique root-owned mode 0700 directory below /var/log/le-phase2a-preflight, prints its path, and returns nonzero on HARD_STOP. Its machine-readable result.manifest records hostname, boot ID, kernel, timestamp, result, failures, and inventory hashes. Maintenance accepts only a checksum-valid PASS record from the same host and boot that is no more than 1,800 seconds (30 minutes) old:

sudo bash docs/le-app-database-migration/phase2a-host-maintenance/execute-maintenance.sh \
  --preflight /var/log/le-phase2a-preflight/preflight.YYYYMMDDTHHMMSSZ.XXXXXXXX
maintenance_rc=$?
printf 'maintenance_rc=%s\n' "$maintenance_rc"

Maintenance and post-reboot validation create their own atomic unique root-only records below /var/log/le-phase2a-maintenance and /var/log/le-phase2a-post-reboot. Post-reboot invocation requires the exact maintenance record:

sudo bash docs/le-app-database-migration/phase2a-host-maintenance/post-reboot-validate.sh \
  /var/log/le-phase2a-maintenance/maintenance.YYYYMMDDTHHMMSSZ.XXXXXXXX
validation_rc=$?
printf 'validation_rc=%s\n' "$validation_rc"

These records can contain IP addresses, usernames, unit/container names, package state, and limited relevant journal metadata. They never intentionally capture process/container environments, environment-file contents, private keys, credentials, secret files, or command history. Treat every record as operationally sensitive: keep it root-only, do not commit it, and disclose it only through the approved maintenance channel.

Transaction comparison limitation

The maintenance script refreshes an isolated sync database, normalizes package name, old version, new version, action, count, and downgrade status, and requires exact equality with the approved retained preview. Any detectable mismatch stops execution.

The final live pacman -Syu must refresh the live sync databases as part of the complete transaction. Pacman cannot expose that post-refresh final interactive transaction for wrapper comparison without either staging a live pacman -Sy-only state or replacing the transaction with an unsafe custom workflow. The script therefore requires two exact operator phrases before launch and then requires the operator to review Pacman's final package table at its native interactive Y/n prompt. Answer n on any difference. This limitation is explicit and is not represented as machine-enforced equality.

Approved package-preview evidence

The refreshed post-NoMachine preview is:

/tmp/le-phase2-package-decision-post-nomachine-20260712.1vyjKk

Its durable normalized baseline and inventories are stored in approved-transaction/. The authoritative transaction is 220 packages: 215 upgrades, 5 new packages, 1,985.98 MiB download, and +760.30 MiB installed-size delta. The prior 221-package transaction is superseded.

Exactly 234 failed drkonqi-coredump-processor@… units are reviewed as nonproduction desktop crash-processing failures. Their checksummed complete state and invocation-ID fingerprint is under failed-unit-disposition/. Any addition, removal, state or invocation change, checksum failure, or other failed unit is a hard stop. The scripts do not reset them. nxserver.service is not an expected critical unit; its unused nomachine package was removed normally.

Snapper readiness history and maintenance gate

On 2026-07-12 the ordinary empty /.snapshots directory was replaced by Btrfs subvolume ID 300, owned root:root mode 0750. Manual snapshot 1 succeeded and exact snap-pac testing created and verified snapshots 2 and 3. Btrfs quotas were intentionally left disabled.

Preflight verifies that /.snapshots remains a Btrfs subvolume. After both operator approvals and immediately before Pacman, maintenance creates a uniquely described explicit Snapper snapshot, captures and validates its numeric ID, verifies its numbered directory and Btrfs snapshot subvolume, and records the ID in the result. It verifies the snapshot again after Pacman, independently of snap-pac's return code. Unavailable quota accounting is a warning, not evidence of snapshot failure. No script enables quotas or runs snapper setup-quota.

The older preview remains at:

/tmp/le-phase2-package-decision-20260712.Mn7mIx

It occupies approximately 2.4 GiB and must not be deleted yet. It is historical, not the approved baseline. Cleanup remains a later explicit action.

Hard stop conditions

Stop rather than improvise if any of the following occurs:

• no independent recovery access or root path;
• insufficient space or an unhealthy /boot;
• degraded storage, failed backup, unresolved failed units, or unhealthy production service;
• unexpected package count/version, downgrade, repository error, signature error, keyring error, mirror inconsistency, or partial transaction;
• SSH access is unstable before the OpenSSH restart hook;
• DKMS/initramfs/kernel/bootloader generation fails;
• package database inconsistency or unexplained .pacnew affecting boot, SSH, PAM, networking, PHP, database, mail, proxy, or firewall configuration;
• production containers or public endpoints fail after the transaction;
• the operator cannot identify a known-good previous kernel and recovery route.

Do not proceed from package maintenance into rootless-Docker provisioning. That later work remains separately approval-gated by host-runtime-plan.md.