add Phase 2A host maintenance procedure

docs/le-app-database-migration/phase2a-host-maintenance/rollback-recovery.md

@@ -0,0 +1,59 @@
+# Rollback and recovery
+
+Rollback is emergency recovery, not a routine alternative to a complete Arch upgrade. Stop and coordinate through the operator whenever possible.
+
+## New kernel fails
+
+1. Use the independent console or bootloader menu.
+2. Select the previously verified kernel or fallback initramfs entry recorded before reboot.
+3. Do not repeatedly boot the failed entry or regenerate boot artifacts blindly.
+4. After the previous kernel boots, record `uname -a`, boot journal, failed units, DKMS state, `/boot` contents, and bootloader entries.
+5. Keep production stopped or degraded only as required for safety; obtain explicit approval before rebuilding initramfs, changing boot defaults, or removing a kernel.
+
+## Package recovery
+
+The first recovery source is an exact, validated package archive still present in `/var/cache/pacman/pkg`. Arch Linux Archive is the fallback source when the exact archive is unavailable.
+
+Emergency package restoration must:
+
+- use the exact before-inventory and transaction log;
+- restore a coherent dependency set, not one guessed library;
+- verify package signatures and checksums;
+- preserve full command output;
+- be approved as emergency recovery;
+- be followed by `pacman -Dk`, package/file consistency checks, initramfs/DKMS/bootloader validation, and production health tests.
+
+Do not improvise a broad downgrade, use `--overwrite` casually, disable signature checking, delete the package database lock without proving no Pacman process exists, or mix repository dates. Arch does not support partial upgrades; a downgrade can require an internally consistent dated repository snapshot.
+
+## SSH recovery
+
+- Expect the OpenSSH hook to restart a running `sshd.service` during the transaction.
+- Keep the active session open and confirm a second session before reboot when the maintenance plan allows.
+- If SSH fails, use the independent console. Inspect `sshd -t`, service status/journal, host keys, PAM, `sshd_config`, and any new `.pacnew` without replacing configuration automatically.
+- Do not weaken authentication, expose a new port, copy keys, or disable the firewall as a shortcut.
+
+## Docker and service recovery
+
+- Compare against `running-containers-before.txt` and `containers-before.txt`.
+- Inspect Docker daemon and container logs, mounts, networks, health checks, and storage availability before restarting anything.
+- Restart only an identified failed service with operator approval; do not restart all services or recreate containers/volumes indiscriminately.
+- Never delete Docker data, volumes, databases, images, or production configuration as recovery.
+- Database recovery follows the service-specific backup/runbook and requires explicit approval before any mutation.
+
+## Mandatory stop conditions
+
+Stop rather than improvise on:
+
+- package signature/keyring/database errors;
+- interrupted Pacman transaction or uncertain package state;
+- failed kernel, initramfs, DKMS, or bootloader generation;
+- missing previous-kernel boot option or loss of console access;
+- SSH/PAM/network/firewall configuration ambiguity;
+- new unexplained `.pacnew`/`.pacsave` for critical configuration;
+- storage, filesystem, database, Docker-volume, or backup inconsistency;
+- failed production database, proxy, mail, Forgejo, Docker, or public endpoint;
+- any proposed action that would modify production data/configuration, systemd, nftables, credentials, or package history beyond the approved recovery scope.
+
+Rootless-Docker provisioning and contained Codex setup are never part of recovery. Resume them only after the host is healthy and a later phase is explicitly approved.
+
+All recovery evidence remains in the root-only unique preflight, maintenance, and post-reboot record directories. It may contain operationally sensitive metadata and must not be committed or copied into a user-readable workspace.