diff --git a/docs/le-app-database-migration/host-runtime-plan.md b/docs/le-app-database-migration/host-runtime-plan.md index f01bd29..b33516f 100644 --- a/docs/le-app-database-migration/host-runtime-plan.md +++ b/docs/le-app-database-migration/host-runtime-plan.md @@ -52,11 +52,13 @@ pacman -Syu --needed rootlesskit slirp4netns openai-codex was a 221-package full host upgrade affecting both installed kernels and OpenSSH. It is a host-maintenance operation requiring a fresh exact transaction preview, reboot planning, service-health checks, `.pacnew` review, and separate operator approval. A partial Arch upgrade is not acceptable. -The unresolved package paths are: +The package paths are: 1. schedule and approve the complete refreshed host upgrade; or 2. investigate clean-chroot, checksum-pinned packages compatible with the current system, without partially upgrading repository packages. +The operator provisionally selected Path A on 2026-07-12 as the preferred Phase 2 package path because it is the shortest safe route consistent with Arch's complete-upgrade model. This records planning direction only: the transaction, maintenance window, package execution, and reboot remain separately approval-gated. The uncommitted review package is under `docs/le-app-database-migration/phase2a-host-maintenance/`. + Do not install the unmodified AUR `docker-rootless-extras` package. Its install hook reloads global sysctls, which exceeds this project's boundary. Do not curl an installer into a shell. Install only a reviewed Moby `contrib/dockerd-rootless.sh` matching the Docker engine actually installed after the chosen package path. At the current checkpoint Docker is `29.6.1`; the candidate is: @@ -311,7 +313,7 @@ Never recursively copy or `rsync` the nonproduction workspace into production. C ## Unresolved decisions checklist -- Phase 2 package path: scheduled full host maintenance or clean-chroot/current-system-compatible investigation. +- Phase 2 package path: Path A is provisionally preferred; package execution and reboot are not approved. - Approved DNS resolver IPv4 addresses; `192.168.10.1` is currently observed but not approved. - Complete current host, public/NAT, LAN, management, VPN, WireGuard, Docker bridge, and other container-network inventory. - Codex authentication: device authentication or dedicated API key. diff --git a/docs/le-app-database-migration/implementation-plan.md b/docs/le-app-database-migration/implementation-plan.md index 7e113e6..93e3583 100644 --- a/docs/le-app-database-migration/implementation-plan.md +++ b/docs/le-app-database-migration/implementation-plan.md @@ -10,7 +10,7 @@ Status: revised pre-implementation plan. Work stops at the Milestone 0 host-runt 2. **Host runtime gate — requires human approval/provisioning** - Preserve the completed inert `le_app_codex` Phase 1 identity/workspace; do not rerun bootstrap artifacts. - - Select the Phase 2 package path separately; never perform a partial Arch upgrade or install the unmodified AUR extras package. + - Preserve the provisional Path A selection: prepare a scheduled complete Arch host upgrade, but require separate approval for the final transaction and reboot; never perform a partial Arch upgrade or install the unmodified AUR extras package. - Use a version-matched, checksum-pinned Moby launcher and a genuine per-user Docker service managed through root-controlled `user@1200.service`; never enable lingering or run `dockerd` from a system service with `User=le_app_codex`. - Apply containment and the proposed `user-1200.slice` limits to the complete user manager; accept only `unix:///run/user/1200/docker.sock`. - Create an IPv4-only root-owned namespace with default-deny egress and explicit host/LAN/WireGuard/metadata/private/loopback/bridge/SMTP/IPv6 denial after DNS and complete address inventories are approved. diff --git a/docs/le-app-database-migration/phase2a-host-maintenance/MANIFEST.sha256 b/docs/le-app-database-migration/phase2a-host-maintenance/MANIFEST.sha256 new file mode 100644 index 0000000..486cbbd --- /dev/null +++ b/docs/le-app-database-migration/phase2a-host-maintenance/MANIFEST.sha256 @@ -0,0 +1,6 @@ +756dfad7eddf89406383804f891266bd814c6ca974a5a250fddc72b67c055ddb README.md +9d9db1819ef473954ea9f5009dc985f3b8b090406246bb22a10ffa09caf3e692 preflight-read-only.sh +ab7220b2042b6a256c26189b37e66637cc7cf741420265ecefd94ab6fa6c06b7 execute-maintenance.sh +5b88c68da055d3578e8b19fdf7623c3de204d5ef0af4b17e113938d8bd0a2c5e reboot-procedure.md +b6a67a01d4c4f786906d98603ff3075deff29fb6bc4483a642f18a187e4f552a post-reboot-validate.sh +0805397870c66a523f693cfdba33f7b08c94793b62ddadf720ff84c22796ea13 rollback-recovery.md diff --git a/docs/le-app-database-migration/phase2a-host-maintenance/README.md b/docs/le-app-database-migration/phase2a-host-maintenance/README.md new file mode 100644 index 0000000..c6b4cf5 --- /dev/null +++ b/docs/le-app-database-migration/phase2a-host-maintenance/README.md @@ -0,0 +1,121 @@ +# Phase 2A scheduled host-maintenance review package + +Status: **review-only and uncommitted**. Operator provisionally prefers Path A, a scheduled complete Arch host upgrade. This is not execution approval. + +Proposed package command: + +```bash +pacman -Syu --needed rootlesskit slirp4netns openai-codex +``` + +This package deliberately separates: + +1. read-only preflight: `preflight-read-only.sh`; +2. package execution: `execute-maintenance.sh` — generated, never run during preparation; +3. reboot: `reboot-procedure.md` — separate approval and manual command; +4. post-reboot validation: `post-reboot-validate.sh`; +5. later contained rootless-Docker provisioning — explicitly outside Phase 2A. + +The scripts do not enable lingering, configure rootless Docker, modify systemd or nftables, authenticate Codex, copy data, or change production configuration. Package hooks may restart services during the approved upgrade; the execution script records this but does not add its own service restarts. + +## Required operator inputs + +Before execution, record site-specific values in the maintenance ticket: + +- independent console/recovery method and a tested root login path; +- public health URLs, including the canonical application utility endpoint; +- production-critical unit names beyond the conservative defaults in the scripts; +- expected running Docker container names and health states; +- bootloader type and known-good previous kernel entry; +- backup identifiers and restoration instructions; +- maintenance window, user notification, and rollback decision owner. + +URLs can be supplied without editing the scripts: + +```bash +export PUBLIC_HEALTH_URLS='https://example.invalid/health https://example.invalid/' +``` + +Health URLs must be non-secret public HTTPS URLs. The scripts reject user-info credentials, query strings, fragments, whitespace/control characters, and non-HTTPS schemes. Never place tokens, signed parameters, credentials, or private endpoints in `PUBLIC_HEALTH_URLS`; the scripts do not read environment files to discover URLs and do not log redirected effective URLs. + +Unit names can be extended similarly: + +```bash +export EXTRA_CRITICAL_UNITS='example.service another.service' +``` + +The operator must separately verify independent recovery and restorable backups, then set the non-secret attestations: + +```bash +export LE_PHASE2A_RECOVERY_READY=YES +export LE_PHASE2A_BACKUPS_READY=YES +``` + +## Invocation and enforced records + +Invoke every script as a separate process, never with `source` or `.`. Explicitly capture its return status: + +```bash +sudo --preserve-env=PUBLIC_HEALTH_URLS,EXTRA_CRITICAL_UNITS,LE_PHASE2A_RECOVERY_READY,LE_PHASE2A_BACKUPS_READY \ + bash docs/le-app-database-migration/phase2a-host-maintenance/preflight-read-only.sh +preflight_rc=$? +printf 'preflight_rc=%s\n' "$preflight_rc" +``` + +The preflight creates an atomic unique root-owned mode `0700` directory below `/var/log/le-phase2a-preflight`, prints its path, and returns nonzero on `HARD_STOP`. Its machine-readable `result.manifest` records hostname, boot ID, kernel, timestamp, result, failures, and inventory hashes. Maintenance accepts only a checksum-valid `PASS` record from the same host and boot that is no more than **1,800 seconds (30 minutes)** old: + +```bash +sudo bash docs/le-app-database-migration/phase2a-host-maintenance/execute-maintenance.sh \ + --preflight /var/log/le-phase2a-preflight/preflight.YYYYMMDDTHHMMSSZ.XXXXXXXX +maintenance_rc=$? +printf 'maintenance_rc=%s\n' "$maintenance_rc" +``` + +Maintenance and post-reboot validation create their own atomic unique root-only records below `/var/log/le-phase2a-maintenance` and `/var/log/le-phase2a-post-reboot`. Post-reboot invocation requires the exact maintenance record: + +```bash +sudo bash docs/le-app-database-migration/phase2a-host-maintenance/post-reboot-validate.sh \ + /var/log/le-phase2a-maintenance/maintenance.YYYYMMDDTHHMMSSZ.XXXXXXXX +validation_rc=$? +printf 'validation_rc=%s\n' "$validation_rc" +``` + +These records can contain IP addresses, usernames, unit/container names, package state, and limited relevant journal metadata. They never intentionally capture process/container environments, environment-file contents, private keys, credentials, secret files, or command history. Treat every record as operationally sensitive: keep it root-only, do not commit it, and disclose it only through the approved maintenance channel. + +## Transaction comparison limitation + +The maintenance script refreshes an isolated sync database, normalizes package name, old version, new version, action, count, and downgrade status, and requires exact equality with the approved retained preview. Any detectable mismatch stops execution. + +The final live `pacman -Syu` must refresh the live sync databases as part of the complete transaction. Pacman cannot expose that post-refresh final interactive transaction for wrapper comparison without either staging a live `pacman -Sy`-only state or replacing the transaction with an unsafe custom workflow. The script therefore requires two exact operator phrases before launch and then requires the operator to review Pacman's final package table at its native interactive `Y/n` prompt. Answer `n` on any difference. This limitation is explicit and is not represented as machine-enforced equality. + +## Package-preview evidence + +The retained preview is: + +```text +/tmp/le-phase2-package-decision-20260712.Mn7mIx +``` + +Expected report checksum: + +```text +195d74afe121556294318d0ed49e3fb5504088dbb31a4c0786b8adea44f1369d phase2-package-decision-report.md +``` + +The directory occupies approximately 2.4 GiB. Do not delete it yet. It becomes cleanup-eligible only after the operator accepts the maintenance record, the upgrade/reboot succeeds, and retained evidence has been copied to durable approved storage if required. Cleanup is a later explicit command and is not included in any script. + +## Hard stop conditions + +Stop rather than improvise if any of the following occurs: + +- no independent recovery access or root path; +- insufficient space or an unhealthy `/boot`; +- degraded storage, failed backup, unresolved failed units, or unhealthy production service; +- unexpected package count/version, downgrade, repository error, signature error, keyring error, mirror inconsistency, or partial transaction; +- SSH access is unstable before the OpenSSH restart hook; +- DKMS/initramfs/kernel/bootloader generation fails; +- package database inconsistency or unexplained `.pacnew` affecting boot, SSH, PAM, networking, PHP, database, mail, proxy, or firewall configuration; +- production containers or public endpoints fail after the transaction; +- the operator cannot identify a known-good previous kernel and recovery route. + +Do not proceed from package maintenance into rootless-Docker provisioning. That later work remains separately approval-gated by `host-runtime-plan.md`. diff --git a/docs/le-app-database-migration/phase2a-host-maintenance/execute-maintenance.sh b/docs/le-app-database-migration/phase2a-host-maintenance/execute-maintenance.sh new file mode 100644 index 0000000..6e89195 --- /dev/null +++ b/docs/le-app-database-migration/phase2a-host-maintenance/execute-maintenance.sh @@ -0,0 +1,316 @@ +#!/usr/bin/env bash +# GENERATED FOR REVIEW. Run only as a separate process after explicit approval. +# No shell-wide error options and no parent-shell termination commands. + +fail() { + printf 'HARD_STOP: %s\n' "$1" >&2 + return "${2:-1}" +} + +manifest_value() { + key=$1 + file=$2 + value=$(awk -F= -v k="$key" '$1==k {sub(/^[^=]*=/, ""); print; found++} END {if(found!=1) exit 1}' "$file") + rc=$? + if [ "$rc" -ne 0 ]; then return "$rc"; fi + printf '%s\n' "$value" + return 0 +} + +validate_preflight() { + preflight=$1 + if [ -z "$preflight" ] || [ ! -d "$preflight" ] || [ -L "$preflight" ]; then + fail 'preflight path is missing, not a directory, or a symlink' 30 + return $? + fi + owner=$(stat -c %u "$preflight" 2>/dev/null) + mode=$(stat -c %a "$preflight" 2>/dev/null) + if [ "$owner" != 0 ] || [ "$mode" != 700 ]; then + fail 'preflight record must be root-owned mode 0700' 31 + return $? + fi + manifest=$preflight/result.manifest + if [ ! -f "$manifest" ] || [ -L "$manifest" ] || [ ! -f "$preflight/result.manifest.sha256" ]; then + fail 'preflight result manifest is missing or a symlink' 32 + return $? + fi + (cd "$preflight" && sha256sum -c result.manifest.sha256 > /dev/null 2>&1) + rc=$? + if [ "$rc" -ne 0 ]; then fail 'preflight result manifest checksum failed' 32; return $?; fi + format=$(manifest_value format "$manifest") || { fail 'malformed preflight format' 33; return $?; } + recorded_host=$(manifest_value hostname "$manifest") || { fail 'malformed preflight hostname' 33; return $?; } + recorded_boot=$(manifest_value boot_id "$manifest") || { fail 'malformed preflight boot ID' 33; return $?; } + recorded_kernel=$(manifest_value running_kernel "$manifest") || { fail 'malformed preflight kernel' 33; return $?; } + recorded_time=$(manifest_value timestamp_epoch "$manifest") || { fail 'malformed preflight timestamp' 33; return $?; } + recorded_overall=$(manifest_value overall "$manifest") || { fail 'malformed preflight result' 33; return $?; } + recorded_failure_count=$(manifest_value failure_count "$manifest") || { fail 'malformed preflight failure count' 33; return $?; } + recorded_failed_checks_hash=$(manifest_value failed_checks_sha256 "$manifest") || { fail 'malformed failed-check hash' 33; return $?; } + recorded_hash=$(manifest_value inventory_hashes_sha256 "$manifest") || { fail 'malformed inventory hash' 33; return $?; } + [ "$format" = le-phase2a-preflight-v1 ] || { fail 'unsupported preflight format' 34; return $?; } + [ "$recorded_overall" = PASS ] || { fail 'preflight result is not PASS' 35; return $?; } + [ "$recorded_failure_count" = 0 ] || { fail 'preflight records mandatory failures' 35; return $?; } + if [ ! -f "$preflight/failed-mandatory-checks.txt" ] || [ -L "$preflight/failed-mandatory-checks.txt" ]; then + fail 'failed-check list is missing or a symlink' 35 + return $? + fi + actual_failed_checks_hash=$(sha256sum "$preflight/failed-mandatory-checks.txt" 2>/dev/null | awk '{print $1}') + [ "$actual_failed_checks_hash" = "$recorded_failed_checks_hash" ] || { fail 'failed-check list checksum mismatch' 35; return $?; } + [ "$recorded_host" = "$(hostname)" ] || { fail 'preflight belongs to another host' 36; return $?; } + [ "$recorded_boot" = "$(cat /proc/sys/kernel/random/boot_id)" ] || { fail 'preflight belongs to another boot' 37; return $?; } + [ "$recorded_kernel" = "$(uname -r)" ] || { fail 'running kernel changed since preflight' 37; return $?; } + case "$recorded_time" in *[!0-9]*|'') fail 'preflight timestamp is invalid' 38; return $? ;; esac + now=$(date +%s) + age=$((now - recorded_time)) + if [ "$age" -lt 0 ] || [ "$age" -gt 1800 ]; then + fail "preflight is stale; age=${age}s limit=1800s" 39 + return $? + fi + actual_hash=$(sha256sum "$preflight/inventory-hashes.sha256" 2>/dev/null | awk '{print $1}') + [ "$actual_hash" = "$recorded_hash" ] || { fail 'preflight inventory-hash manifest checksum mismatch' 40; return $?; } + (cd "$preflight" && sha256sum -c inventory-hashes.sha256 > /dev/null 2>&1) + rc=$? + if [ "$rc" -ne 0 ]; then + fail 'one or more preflight inventory files failed checksum validation' 41 + return $? + fi + validated_preflight=$preflight + return 0 +} + +validate_health_url_file() { + file=$1 + [ -s "$file" ] || { fail 'validated preflight contains no health URLs' 42; return $?; } + while IFS= read -r url; do + case "$url" in https://*) ;; *) fail 'unsupported health URL scheme' 42; return $? ;; esac + authority=${url#https://}; authority=${authority%%/*} + case "$url" in *'?'*|*'#'*|*$'\n'*|*$'\r'*|*$'\t'*|*' '*) fail 'unsafe health URL content' 42; return $? ;; esac + case "$authority" in ''|*@*) fail 'health URL user-info or empty authority rejected' 42; return $? ;; esac + done < "$file" + return 0 +} + +rerun_volatile_checks() { + systemctl --failed --no-legend --plain --no-pager > "$record/volatile-failed-units.txt" 2>&1 + rc=$? + [ "$rc" -eq 0 ] || { fail 'cannot capture current failed units' "$rc"; return $?; } + [ ! -s "$record/volatile-failed-units.txt" ] || { fail 'one or more failed units are present' 45; return $?; } + pacman -Dk > "$record/volatile-pacman-Dk.txt" 2>&1 + rc=$? + [ "$rc" -eq 0 ] || { fail 'package database consistency failed immediately before execution' "$rc"; return $?; } + df -hT / /boot /var/cache/pacman/pkg /tmp > "$record/volatile-disk-space.txt" 2>&1 + rc=$? + [ "$rc" -eq 0 ] || { fail 'cannot verify current disk space' "$rc"; return $?; } + while IFS= read -r container; do + [ -n "$container" ] || continue + running=$(docker inspect -f '{{.State.Running}}' "$container" 2>/dev/null) + health=$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}no-healthcheck{{end}}' "$container" 2>/dev/null) + printf '%s|running=%s|health=%s\n' "$container" "${running:-missing}" "${health:-unknown}" \ + >> "$record/volatile-container-health.txt" + if [ "$running" != true ] || [ "$health" = unhealthy ]; then + fail "previously running container is unavailable: $container" 43 + return $? + fi + done < "$validated_preflight/running-containers-before.txt" + : > "$record/volatile-public-health.txt" + while IFS= read -r url; do + metrics=$(curl --fail --silent --show-error --max-redirs 0 \ + --connect-timeout 10 --max-time 30 --output /dev/null \ + --write-out 'http=%{http_code}|remote=%{remote_ip}|time=%{time_total}' \ + "$url" 2>> "$record/volatile-public-health-errors.txt") + rc=$? + printf '%s|%s\n' "$url" "$metrics" >> "$record/volatile-public-health.txt" + if [ "$rc" -ne 0 ]; then fail "public health failed immediately before execution: $url" 44; return $?; fi + done < "$validated_preflight/health-urls.sanitized" + return 0 +} + +normalize_transaction() { + source_file=$1 + output_file=$2 + downgrade_file=$3 + : > "$output_file" + : > "$downgrade_file" + while IFS='|' read -r name new_version repository location download_bytes; do + [ -n "$name" ] || continue + old_line=$(pacman -Q "$name" 2>/dev/null) + if [ -n "$old_line" ]; then + old_version=${old_line#* } + comparison=$(vercmp "$new_version" "$old_version") + if [ "$comparison" -lt 0 ]; then action=downgrade + elif [ "$comparison" -eq 0 ]; then action=same + else action=upgrade + fi + else + old_version=- + action=new + fi + printf '%s|%s|%s|%s\n' "$name" "$old_version" "$new_version" "$action" >> "$output_file" + if [ "$action" = downgrade ]; then + printf '%s|%s|%s|%s\n' "$name" "$old_version" "$new_version" "$action" >> "$downgrade_file" + fi + done < "$source_file" + sort -o "$output_file" "$output_file" + return 0 +} + +main() { + if [ "$(id -u)" -ne 0 ]; then fail 'run as a separate root process' 10; return $?; fi + if [ "$1" != --preflight ] || [ -z "${2:-}" ] || [ -n "${3:-}" ]; then + fail 'usage: execute-maintenance.sh --preflight /var/log/le-phase2a-preflight/preflight.XXXXXXXX' 11 + return $? + fi + validate_preflight "$2" || return $? + validate_health_url_file "$validated_preflight/health-urls.sanitized" || return $? + + umask 077 + install -d -o root -g root -m 0700 /var/log/le-phase2a-maintenance + rc=$? + [ "$rc" -eq 0 ] || { fail 'cannot create root-only maintenance record parent' "$rc"; return $?; } + record=$(mktemp -d "/var/log/le-phase2a-maintenance/maintenance.$(date -u +%Y%m%dT%H%M%SZ).XXXXXXXX") + rc=$? + if [ "$rc" -ne 0 ] || [ -z "$record" ] || [ ! -d "$record" ]; then + fail 'cannot atomically create unique maintenance record' 12 + return $? + fi + chown root:root "$record"; chmod 0700 "$record" + maintenance_start=$(date -u +%FT%TZ) + printf '%s\n' "$record" > "$record/RECORD_PATH.txt" + cp -a "$validated_preflight" "$record/preflight-record" + rc=$? + [ "$rc" -eq 0 ] || { fail 'cannot copy validated preflight record' "$rc"; return $?; } + + report=/tmp/le-phase2-package-decision-20260712.Mn7mIx/phase2-package-decision-report.md + report_manifest=/tmp/le-phase2-package-decision-20260712.Mn7mIx/SHA256SUMS.main + printf '%s %s\n' 195d74afe121556294318d0ed49e3fb5504088dbb31a4c0786b8adea44f1369d "$report" | sha256sum -c - \ + > "$record/approved-report-checksum.txt" 2>&1 + rc=$? + [ "$rc" -eq 0 ] || { fail 'approved report checksum failed' "$rc"; return $?; } + cp "$report" "$report_manifest" "$record/" + rc=$? + [ "$rc" -eq 0 ] || { fail 'cannot copy approved report and checksum manifest' "$rc"; return $?; } + + uname -a > "$record/running-kernel-before.txt" 2>&1 + rc=$?; [ "$rc" -eq 0 ] || { fail 'cannot capture running kernel' "$rc"; return $?; } + cat /proc/sys/kernel/random/boot_id > "$record/boot-id-before.txt" 2>&1 + rc=$?; [ "$rc" -eq 0 ] || { fail 'cannot capture boot ID' "$rc"; return $?; } + pacman -Q > "$record/packages-before.txt" 2>&1 + rc=$?; [ "$rc" -eq 0 ] || { fail 'cannot capture installed package inventory' "$rc"; return $?; } + cp "$validated_preflight/bootloader-inventory.txt" "$validated_preflight/kernel-initramfs-before.txt" \ + "$validated_preflight/running-containers-before.txt" "$validated_preflight/containers-all.psv" \ + "$validated_preflight/critical-units-baseline.psv" "$validated_preflight/database-baseline.psv" "$record/" + rc=$?; [ "$rc" -eq 0 ] || { fail 'cannot copy essential rollback baseline evidence' "$rc"; return $?; } + systemctl --failed --no-legend --plain --no-pager > "$record/failed-units-before.txt" 2>&1 + rc=$?; [ "$rc" -eq 0 ] || { fail 'cannot capture failed-unit baseline' "$rc"; return $?; } + [ ! -s "$record/failed-units-before.txt" ] || { fail 'failed units appeared before transaction preview' 45; return $?; } + + rerun_volatile_checks || return $? + + preview_root=$(mktemp -d /tmp/le-phase2a-execution-preview.XXXXXXXX) + rc=$? + if [ "$rc" -ne 0 ] || [ -z "$preview_root" ] || [ ! -d "$preview_root" ]; then + fail 'cannot create isolated transaction preview directory' 13 + return $? + fi + chmod 0700 "$preview_root"; mkdir -m 0700 "$preview_root/db" "$preview_root/cache" + ln -s /var/lib/pacman/local "$preview_root/db/local" + + fakeroot -- pacman --config /etc/pacman.conf --dbpath "$preview_root/db" \ + --cachedir "$preview_root/cache" --logfile "$preview_root/pacman-preview.log" -Sy \ + > "$record/isolated-sync-refresh.log" 2>&1 + rc=$? + [ "$rc" -eq 0 ] || { fail 'isolated repository refresh failed; live databases untouched' "$rc"; return $?; } + fakeroot -- pacman --config /etc/pacman.conf --dbpath "$preview_root/db" \ + --cachedir "$preview_root/cache" --logfile "$preview_root/pacman-preview.log" \ + -Su --needed --print-format '%n|%v|%r|%l|%s' rootlesskit slirp4netns openai-codex \ + > "$record/latest-transaction.psv" 2> "$record/latest-transaction.err" + rc=$? + [ "$rc" -eq 0 ] || { fail 'latest complete transaction resolution failed' "$rc"; return $?; } + normalize_transaction "$record/latest-transaction.psv" "$record/latest-normalized-actions.psv" "$record/latest-downgrades.psv" + [ ! -s "$record/latest-downgrades.psv" ] || { fail 'latest transaction contains a downgrade' 14; return $?; } + + awk -F'|' '{print $1"|"$2"|"$3"|"$5}' \ + /tmp/le-phase2-package-decision-20260712.Mn7mIx/inventory.psv | sort \ + > "$record/approved-normalized-actions.psv" + rc=$? + [ "$rc" -eq 0 ] || { fail 'cannot normalize approved preview' "$rc"; return $?; } + diff -u "$record/approved-normalized-actions.psv" "$record/latest-normalized-actions.psv" \ + > "$record/approved-vs-latest.diff" 2>&1 + diff_rc=$? + if [ "$diff_rc" -ne 0 ]; then + fail 'latest package names, versions, or actions differ from the approved preview' 15 + return $? + fi + latest_count=$(wc -l < "$record/latest-normalized-actions.psv") + approved_count=$(wc -l < "$record/approved-normalized-actions.psv") + [ "$latest_count" = "$approved_count" ] || { fail 'transaction package counts differ' 15; return $?; } + for required in rootlesskit slirp4netns openai-codex; do + grep -q "^${required}|" "$record/latest-normalized-actions.psv" + rc=$?; [ "$rc" -eq 0 ] || { fail "latest transaction omits $required" 16; return $?; } + done + + cp "$record/latest-transaction.psv" "$record/latest-normalized-actions.psv" \ + "$record/approved-normalized-actions.psv" "$record/approved-vs-latest.diff" "$record/transaction-for-approval.psv" + rc=$?; [ "$rc" -eq 0 ] || { fail 'cannot preserve transaction preview evidence' "$rc"; return $?; } + sha256sum "$record/latest-transaction.psv" "$record/latest-normalized-actions.psv" \ + "$record/approved-normalized-actions.psv" > "$record/transaction-preview-hashes.sha256" + + printf '\nMachine comparison passed: packages=%s, no downgrades, exact match to approved preview.\n' "$latest_count" + cat "$record/latest-normalized-actions.psv" + printf '\nType exactly: APPROVE MATCHED COMPLETE HOST UPGRADE\n> ' + IFS= read -r approval_one + if [ "$approval_one" != 'APPROVE MATCHED COMPLETE HOST UPGRADE' ]; then + fail 'first exact operator approval not granted' 20 + return $? + fi + + rerun_volatile_checks || return $? + printf '\nLIMITATION: the final live -Syu refresh may change after this isolated comparison.\n' + printf 'Pacman will display its final interactive transaction and ask Y/n. The wrapper cannot machine-compare that post-refresh transaction without an unsafe live -Sy-only staging step.\n' + printf 'Type exactly: ACKNOWLEDGE FINAL PACMAN TRANSACTION REQUIRES MANUAL REVIEW\n> ' + IFS= read -r approval_two + if [ "$approval_two" != 'ACKNOWLEDGE FINAL PACMAN TRANSACTION REQUIRES MANUAL REVIEW' ]; then + fail 'second exact operator acknowledgement not granted' 21 + return $? + fi + + printf 'At Pacman Y/n, answer no if any package, version, action, count, downgrade status, signature, or repository differs.\n' + script -q -e -f -c 'pacman -Syu --needed rootlesskit slirp4netns openai-codex' "$record/pacman-complete.log" + rc=$? + if [ "$rc" -ne 0 ]; then fail "Pacman returned $rc; stop without ad hoc repair" "$rc"; return $?; fi + + pacman -Q > "$record/packages-after.txt" 2>&1 + rc=$?; [ "$rc" -eq 0 ] || { fail 'cannot capture post-transaction package inventory' "$rc"; return $?; } + diff -u "$record/packages-before.txt" "$record/packages-after.txt" > "$record/package-inventory.diff" 2>&1 + diff_rc=$?; [ "$diff_rc" -le 1 ] || { fail 'package inventory comparison failed' "$diff_rc"; return $?; } + find /etc -xdev -type f \( -name '*.pacnew' -o -name '*.pacsave' \) -printf '%p\n' | sort \ + > "$record/pacnew-pacsave-after.txt" 2>&1 + rc=$?; [ "$rc" -eq 0 ] || { fail 'cannot inventory pacnew/pacsave files' "$rc"; return $?; } + pacdiff -o > "$record/pacdiff-outstanding.txt" 2>&1 + pacdiff_rc=$? + if [ "$pacdiff_rc" -ne 0 ] && [ "$pacdiff_rc" -ne 1 ]; then + printf 'pacdiff-observation|rc=%s\n' "$pacdiff_rc" > "$record/warnings-after.txt" + fi + grep -E -i 'hook|restart|reload|dkms|dracut|initramfs|mkinitcpio|grub|pacnew|pacsave|warning|error|failed' \ + "$record/pacman-complete.log" > "$record/hook-restart-summary.txt" 2>&1 + systemctl --failed --no-legend --plain --no-pager > "$record/failed-units-after.txt" 2>&1 + journal_units='sshd.service accounts-daemon.service docker.service' + while IFS='|' read -r unit _; do journal_units="$journal_units $unit"; done < "$record/critical-units-baseline.psv" + journalctl --since "$maintenance_start" --no-pager $(for unit in $journal_units; do printf -- '-u %q ' "$unit"; done) \ + > "$record/relevant-service-journal.txt" 2>&1 + dkms status > "$record/dkms-after.txt" 2>&1 + find /boot /usr/lib/modules -maxdepth 2 -type f \ + \( -name 'vmlinuz*' -o -name 'initramfs*' -o -name 'initrd*' -o -name pkgbase \) \ + -printf '%p size=%s mtime=%TY-%Tm-%TdT%TH:%TM:%TS\n' | sort > "$record/kernel-boot-artifacts-after.txt" 2>&1 + docker ps -a --format '{{.Names}}|{{.Image}}|{{.Status}}|{{.Ports}}' | sort > "$record/containers-after.txt" 2>&1 + find "$record" -maxdepth 1 -type f ! -name SHA256SUMS -print0 | sort -z | xargs -0 sha256sum \ + > "$record/SHA256SUMS" 2>&1 + chmod -R go-rwx "$record" + + printf 'PACKAGE TRANSACTION COMPLETED; REBOOT NOT PERFORMED.\n' + printf 'MAINTENANCE_RECORD=%s\n' "$record" + printf 'Review all warnings, hooks, services, kernel artifacts, and pacnew/pacsave files.\n' + printf 'Reboot and rootless-Docker provisioning remain separate approval gates.\n' + return 0 +} + +main "$@" diff --git a/docs/le-app-database-migration/phase2a-host-maintenance/post-reboot-validate.sh b/docs/le-app-database-migration/phase2a-host-maintenance/post-reboot-validate.sh new file mode 100644 index 0000000..cd089e5 --- /dev/null +++ b/docs/le-app-database-migration/phase2a-host-maintenance/post-reboot-validate.sh @@ -0,0 +1,222 @@ +#!/usr/bin/env bash +# Read-only validation plus root-only evidence recording. Run as a separate process. + +add_failure() { + printf '%s\n' "$1" >> "$post_record/failed-checks.txt" + failures=$((failures + 1)) + return 0 +} + +check() { + label=$1; outfile=$2; shift 2 + "$@" > "$post_record/$outfile" 2>&1 + rc=$? + if [ "$rc" -ne 0 ]; then add_failure "$label|rc=$rc|file=$outfile"; fi + return 0 +} + +check_shell() { + label=$1; outfile=$2; command_text=$3 + /usr/bin/bash -o pipefail -c "$command_text" > "$post_record/$outfile" 2>&1 + rc=$? + if [ "$rc" -ne 0 ]; then add_failure "$label|rc=$rc|file=$outfile"; fi + return 0 +} + +validate_health_url_file() { + file=$1 + if [ ! -s "$file" ]; then add_failure 'health-urls|missing'; return 0; fi + while IFS= read -r url; do + case "$url" in https://*) ;; *) add_failure 'health-url|unsupported-scheme'; continue ;; esac + authority=${url#https://}; authority=${authority%%/*} + case "$url" in *'?'*|*'#'*|*$'\n'*|*$'\r'*|*$'\t'*|*' '*) add_failure 'health-url|unsafe-content'; continue ;; esac + case "$authority" in ''|*@*) add_failure 'health-url|userinfo-or-empty-authority'; continue ;; esac + done < "$file" + return 0 +} + +compare_containers() { + : > "$post_record/container-comparison.psv" + while IFS= read -r container; do + [ -n "$container" ] || continue + running=$(docker inspect -f '{{.State.Running}}' "$container" 2>/dev/null) + health=$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}no-healthcheck{{end}}' "$container" 2>/dev/null) + printf '%s|running=%s|health=%s\n' "$container" "${running:-missing}" "${health:-unknown}" \ + >> "$post_record/container-comparison.psv" + if [ "$running" != true ] || [ "$health" = unhealthy ]; then + add_failure "container-regression|name=$container|running=${running:-missing}|health=${health:-unknown}" + fi + done < "$baseline/running-containers-before.txt" + return 0 +} + +compare_units() { + : > "$post_record/unit-comparison.psv" + while IFS='|' read -r unit old_load old_active old_sub; do + [ -n "$unit" ] || continue + new_load=$(systemctl show "$unit" -p LoadState --value 2>/dev/null) + new_active=$(systemctl show "$unit" -p ActiveState --value 2>/dev/null) + new_sub=$(systemctl show "$unit" -p SubState --value 2>/dev/null) + printf '%s|before=%s,%s,%s|after=%s,%s,%s\n' \ + "$unit" "$old_load" "$old_active" "$old_sub" "${new_load:-missing}" "${new_active:-missing}" "${new_sub:-missing}" \ + >> "$post_record/unit-comparison.psv" + if [ "$old_load" = loaded ] && [ "$new_load" != loaded ]; then + add_failure "critical-unit-missing|unit=$unit" + elif [ "$old_active" = active ] && [ "$new_active" != active ]; then + add_failure "critical-unit-not-active|unit=$unit|state=${new_active:-missing}" + fi + done < "$baseline/critical-units-baseline.psv" + return 0 +} + +validate_databases() { + : > "$post_record/database-validation.txt" + while IFS='|' read -r model engine identity state detail; do + [ -n "$model" ] || continue + if [ "$model" = unit ]; then + active=$(systemctl show "$identity" -p ActiveState --value 2>/dev/null) + printf '%s|%s|unit=%s|active=%s\n' "$model" "$engine" "$identity" "${active:-missing}" \ + >> "$post_record/database-validation.txt" + if [ "$active" != active ]; then add_failure "database-unit-not-active|unit=$identity"; fi + if [ "$engine" = mysql ]; then + mysqladmin ping >> "$post_record/database-validation.txt" 2>&1 + rc=$?; [ "$rc" -eq 0 ] || add_failure "mysqladmin-ping-failed|unit=$identity|rc=$rc" + elif [ "$engine" = postgresql ]; then + pg_isready >> "$post_record/database-validation.txt" 2>&1 + rc=$?; [ "$rc" -eq 0 ] || add_failure "pg-isready-failed|unit=$identity|rc=$rc" + fi + elif [ "$model" = container ]; then + running=$(docker inspect -f '{{.State.Running}}' "$identity" 2>/dev/null) + health=$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}no-healthcheck{{end}}' "$identity" 2>/dev/null) + printf '%s|%s|container=%s|running=%s|health=%s\n' \ + "$model" "$engine" "$identity" "${running:-missing}" "${health:-unknown}" \ + >> "$post_record/database-validation.txt" + if [ "$running" != true ] || [ "$health" = unhealthy ]; then + add_failure "database-container-failed|container=$identity|running=${running:-missing}|health=${health:-unknown}" + fi + fi + done < "$baseline/database-baseline.psv" + return 0 +} + +check_package_files() { + pacman -Qkk > "$post_record/pacman-Qkk.txt" 2>&1 + qkk_rc=$? + printf 'pacman-Qkk-return=%s\n' "$qkk_rc" > "$post_record/pacman-Qkk-classification.txt" + grep -E -i 'missing|could not read|permission denied|unreadable' "$post_record/pacman-Qkk.txt" \ + > "$post_record/pacman-Qkk-missing-or-unreadable.txt" 2>/dev/null + missing_rc=$? + grep -E -i 'altered|modified' "$post_record/pacman-Qkk.txt" \ + > "$post_record/pacman-Qkk-modified.txt" 2>/dev/null + if [ "$missing_rc" -eq 0 ] && [ -s "$post_record/pacman-Qkk-missing-or-unreadable.txt" ]; then + add_failure 'packaged-files-missing-or-unreadable' + fi + printf 'Modified configuration/package files require manual classification; no repair performed.\n' \ + >> "$post_record/pacman-Qkk-classification.txt" + return 0 +} + +main() { + failures=0 + maintenance=${1:-} + if [ "$(id -u)" -ne 0 ]; then + printf 'HARD_STOP: run as a separate root process\n' >&2 + return 10 + fi + if [ -z "$maintenance" ] || [ ! -d "$maintenance" ] || [ -L "$maintenance" ]; then + printf 'HARD_STOP: supply the exact maintenance record path\n' >&2 + return 11 + fi + owner=$(stat -c %u "$maintenance" 2>/dev/null); mode=$(stat -c %a "$maintenance" 2>/dev/null) + if [ "$owner" != 0 ] || [ "$mode" != 700 ]; then + printf 'HARD_STOP: maintenance record must be root-owned mode 0700\n' >&2 + return 12 + fi + baseline=$maintenance/preflight-record + required='running-containers-before.txt critical-units-baseline.psv database-baseline.psv health-urls.sanitized inventory-hashes.sha256 result.manifest result.manifest.sha256' + for file in $required; do + if [ ! -f "$baseline/$file" ] || [ -L "$baseline/$file" ]; then + printf 'HARD_STOP: missing baseline file %s\n' "$file" >&2 + return 13 + fi + done + (cd "$baseline" && sha256sum -c inventory-hashes.sha256 > /dev/null 2>&1) + rc=$? + if [ "$rc" -ne 0 ]; then + printf 'HARD_STOP: copied preflight baseline checksum failure\n' >&2 + return 14 + fi + (cd "$baseline" && sha256sum -c result.manifest.sha256 > /dev/null 2>&1) + rc=$? + if [ "$rc" -ne 0 ]; then + printf 'HARD_STOP: copied preflight result manifest checksum failure\n' >&2 + return 14 + fi + + umask 077 + install -d -o root -g root -m 0700 /var/log/le-phase2a-post-reboot + rc=$?; [ "$rc" -eq 0 ] || { printf 'HARD_STOP: cannot create post-reboot record parent\n' >&2; return 15; } + post_record=$(mktemp -d "/var/log/le-phase2a-post-reboot/validation.$(date -u +%Y%m%dT%H%M%SZ).XXXXXXXX") + rc=$? + if [ "$rc" -ne 0 ] || [ -z "$post_record" ] || [ ! -d "$post_record" ]; then + printf 'HARD_STOP: cannot atomically create unique post-reboot record\n' >&2 + return 16 + fi + chown root:root "$post_record"; chmod 0700 "$post_record" + : > "$post_record/failed-checks.txt" + printf 'maintenance_record=%s\n' "$maintenance" > "$post_record/source.txt" + + check 'running-kernel' running-kernel.txt uname -a + check_shell 'installed-kernels' installed-kernels.txt 'pacman -Q linux linux-headers linux-zen linux-zen-headers' + check 'failed-units' failed-units.txt systemctl --failed --no-legend --plain --no-pager + if [ -s "$post_record/failed-units.txt" ]; then add_failure 'failed-units-present'; fi + check_shell 'ssh-state' ssh-state.txt \ + 'systemctl show sshd.service -p LoadState -p ActiveState -p SubState -p ExecMainStatus; journalctl -b -u sshd.service --no-pager' + check_shell 'docker-daemon' docker-daemon.txt \ + "systemctl show docker.service -p LoadState -p ActiveState -p SubState -p ExecMainStatus; docker info" + check_shell 'all-containers' containers-after.psv \ + "docker ps -a --format '{{.Names}}|{{.Image}}|{{.Status}}|{{.Ports}}' | sort" + compare_containers + compare_units + validate_databases + + check_shell 'mounts-and-capacity' mounts-capacity.txt 'findmnt --real; df -hT / /boot /var/cache/pacman/pkg /tmp' + check_shell 'network-routes-firewall-ports' network-after.txt \ + 'ip -brief address; ip route show table all; ip -6 route show table all; nft list ruleset; ss -lntup' + wg show > "$post_record/wireguard-after.txt" 2>&1 + wg_rc=$?; if [ "$wg_rc" -ne 0 ]; then printf 'wireguard-observation|rc=%s\n' "$wg_rc" > "$post_record/warnings.txt"; fi + + validate_health_url_file "$baseline/health-urls.sanitized" + : > "$post_record/public-health.txt" + while IFS= read -r url; do + metrics=$(curl --fail --silent --show-error --max-redirs 0 \ + --connect-timeout 10 --max-time 30 --output /dev/null \ + --write-out 'http=%{http_code}|remote=%{remote_ip}|time=%{time_total}' \ + "$url" 2>> "$post_record/public-health-errors.txt") + rc=$? + printf '%s|%s\n' "$url" "$metrics" >> "$post_record/public-health.txt" + if [ "$rc" -ne 0 ]; then add_failure "public-health-failed|url=$url|rc=$rc"; fi + done < "$baseline/health-urls.sanitized" + + pacman -Dk > "$post_record/pacman-Dk.txt" 2>&1 + dk_rc=$? + if [ "$dk_rc" -ne 0 ]; then add_failure "pacman-Dk-failed|rc=$dk_rc"; fi + check_package_files + check_shell 'target-packages' target-packages.txt 'pacman -Q rootlesskit slirp4netns openai-codex' + check_shell 'pacnew-pacsave-inventory' pacnew-pacsave.txt \ + "find /etc -xdev -type f \( -name '*.pacnew' -o -name '*.pacsave' \) -printf '%p\\n' | sort; pacdiff -o 2>&1" + + find "$post_record" -maxdepth 1 -type f ! -name SHA256SUMS -print0 | sort -z | xargs -0 sha256sum \ + > "$post_record/SHA256SUMS" 2>&1 + chmod -R go-rwx "$post_record" + printf 'POST_REBOOT_RECORD=%s\n' "$post_record" + if [ "$failures" -ne 0 ]; then + printf 'POST_REBOOT_RESULT=HARD_STOP failures=%s\n' "$failures" + return 20 + fi + printf 'POST_REBOOT_RESULT=PASS failures=0\n' + printf 'Rootless-Docker provisioning remains separately approval-gated.\n' + return 0 +} + +main "$@" diff --git a/docs/le-app-database-migration/phase2a-host-maintenance/preflight-read-only.sh b/docs/le-app-database-migration/phase2a-host-maintenance/preflight-read-only.sh new file mode 100644 index 0000000..b6f79b0 --- /dev/null +++ b/docs/le-app-database-migration/phase2a-host-maintenance/preflight-read-only.sh @@ -0,0 +1,325 @@ +#!/usr/bin/env bash +# Read-only host inspection plus root-only evidence recording. +# Run as a separate process; this script changes no shell options and uses return only. + +add_failure() { + printf '%s\n' "$1" >> "$record/failed-mandatory-checks.txt" + overall=HARD_STOP + return 0 +} + +mandatory() { + label=$1 + outfile=$2 + shift 2 + "$@" > "$record/$outfile" 2>&1 + rc=$? + if [ "$rc" -ne 0 ]; then + add_failure "$label|rc=$rc|file=$outfile" + fi + return 0 +} + +mandatory_shell() { + label=$1 + outfile=$2 + command_text=$3 + /usr/bin/bash -o pipefail -c "$command_text" > "$record/$outfile" 2>&1 + rc=$? + if [ "$rc" -ne 0 ]; then + add_failure "$label|rc=$rc|file=$outfile" + fi + return 0 +} + +optional_shell() { + label=$1 + outfile=$2 + command_text=$3 + /usr/bin/bash -o pipefail -c "$command_text" > "$record/$outfile" 2>&1 + rc=$? + if [ "$rc" -ne 0 ]; then + printf '%s|rc=%s|file=%s\n' "$label" "$rc" "$outfile" >> "$record/warnings.txt" + fi + return 0 +} + +validate_health_urls() { + : > "$record/health-urls.sanitized" + if [ -z "${PUBLIC_HEALTH_URLS:-}" ]; then + add_failure 'public-health-urls|unset' + return 0 + fi + case "$PUBLIC_HEALTH_URLS" in + *$'\n'*|*$'\r'*|*$'\t'*) + add_failure 'public-health-urls|control-character-rejected' + return 0 + ;; + esac + for url in $PUBLIC_HEALTH_URLS; do + case "$url" in + https://*) ;; + *) add_failure 'public-health-url|unsupported-scheme-or-malformed'; continue ;; + esac + authority=${url#https://} + authority=${authority%%/*} + case "$url" in + *'?'*|*'#'*|*$'\n'*|*$'\r'*|*$'\t'*|*' '*) + add_failure 'public-health-url|query-fragment-or-control-character-rejected' + continue + ;; + esac + case "$authority" in + ''|*@*) add_failure 'public-health-url|empty-authority-or-userinfo-rejected'; continue ;; + esac + printf '%s\n' "$url" >> "$record/health-urls.sanitized" + done + if [ ! -s "$record/health-urls.sanitized" ]; then + add_failure 'public-health-urls|no-valid-non-secret-https-url' + fi + return 0 +} + +check_public_health() { + : > "$record/public-health.txt" + while IFS= read -r url; do + [ -n "$url" ] || continue + metrics=$(curl --fail --silent --show-error --max-redirs 0 \ + --connect-timeout 10 --max-time 30 --output /dev/null \ + --write-out 'http=%{http_code}|remote=%{remote_ip}|time=%{time_total}' \ + "$url" 2>> "$record/public-health-errors.txt") + rc=$? + printf '%s|%s\n' "$url" "$metrics" >> "$record/public-health.txt" + if [ "$rc" -ne 0 ]; then + add_failure "public-health|rc=$rc|url=$url" + fi + done < "$record/health-urls.sanitized" + return 0 +} + +capture_bootloader() { + found=0 + : > "$record/bootloader-inventory.txt" + if [ -d /boot/loader/entries ]; then + printf '%s\n' 'type=systemd-boot' >> "$record/bootloader-inventory.txt" + bootctl status >> "$record/bootloader-inventory.txt" 2>&1 + bootctl_rc=$? + find /boot/loader/entries -maxdepth 1 -type f -print -exec sed -n '1,160p' {} \; \ + >> "$record/bootloader-inventory.txt" 2>&1 + find_rc=$? + if [ "$bootctl_rc" -eq 0 ] && [ "$find_rc" -eq 0 ]; then found=1; fi + fi + if [ -f /boot/grub/grub.cfg ]; then + printf '%s\n' 'type=grub' >> "$record/bootloader-inventory.txt" + grub-install --version >> "$record/bootloader-inventory.txt" 2>&1 + grub_rc=$? + grep -E '^menuentry |^submenu |linux.*vmlinuz|initrd' /boot/grub/grub.cfg \ + >> "$record/bootloader-inventory.txt" 2>&1 + grep_rc=$? + if [ "$grub_rc" -eq 0 ] && [ "$grep_rc" -eq 0 ]; then found=1; fi + fi + if [ "$found" -ne 1 ]; then + add_failure 'bootloader-inventory|no-verified-supported-bootloader' + fi + return 0 +} + +capture_unit_baseline() { + systemctl list-units --type=service --all --no-legend --no-pager \ + > "$record/all-service-units.txt" 2>&1 + rc=$? + if [ "$rc" -ne 0 ]; then + add_failure "service-unit-inventory|rc=$rc" + return 0 + fi + : > "$record/critical-units-baseline.psv" + awk '{print $1}' "$record/all-service-units.txt" | while IFS= read -r unit; do + case "$unit" in + docker.service|nginx.service|php-fpm.service|mariadb.service|mysqld.service|postgresql.service|postfix.service|dovecot.service|rspamd.service|forgejo.service|sshd.service) + ;; + *) continue ;; + esac + load=$(systemctl show "$unit" -p LoadState --value 2>/dev/null) + active=$(systemctl show "$unit" -p ActiveState --value 2>/dev/null) + sub=$(systemctl show "$unit" -p SubState --value 2>/dev/null) + printf '%s|%s|%s|%s\n' "$unit" "$load" "$active" "$sub" + done > "$record/critical-units-baseline.psv" + for unit in ${EXTRA_CRITICAL_UNITS:-}; do + load=$(systemctl show "$unit" -p LoadState --value 2>/dev/null) + if [ "$load" = loaded ]; then + active=$(systemctl show "$unit" -p ActiveState --value 2>/dev/null) + sub=$(systemctl show "$unit" -p SubState --value 2>/dev/null) + printf '%s|%s|%s|%s\n' "$unit" "$load" "$active" "$sub" \ + >> "$record/critical-units-baseline.psv" + else + add_failure "extra-critical-unit|not-loaded|unit=$unit" + fi + done + sort -u -o "$record/critical-units-baseline.psv" "$record/critical-units-baseline.psv" + return 0 +} + +capture_database_baseline() { + : > "$record/database-baseline.psv" + while IFS='|' read -r unit load active sub; do + [ "$active" = active ] || continue + case "$unit" in + mariadb.service|mysqld.service) printf 'unit|mysql|%s|%s|%s\n' "$unit" "$active" "$sub" ;; + postgresql.service) printf 'unit|postgresql|%s|%s|%s\n' "$unit" "$active" "$sub" ;; + esac + done < "$record/critical-units-baseline.psv" >> "$record/database-baseline.psv" + while IFS='|' read -r name image status ports; do + case "$status" in Up\ *) ;; *) continue ;; esac + lowered=$(printf '%s %s' "$name" "$image" | tr '[:upper:]' '[:lower:]') + case "$lowered" in + *maria*|*mysql*) printf 'container|mysql|%s|%s|%s\n' "$name" "$status" "$ports" ;; + *postgres*) printf 'container|postgresql|%s|%s|%s\n' "$name" "$status" "$ports" ;; + esac + done < "$record/containers-all.psv" >> "$record/database-baseline.psv" + return 0 +} + +validate_database_baseline() { + : > "$record/database-readiness-before.txt" + while IFS='|' read -r model engine identity state detail; do + [ "$model" = unit ] || continue + if [ "$engine" = mysql ]; then + mysqladmin ping >> "$record/database-readiness-before.txt" 2>&1 + rc=$?; [ "$rc" -eq 0 ] || add_failure "mysqladmin-ping|unit=$identity|rc=$rc" + elif [ "$engine" = postgresql ]; then + pg_isready >> "$record/database-readiness-before.txt" 2>&1 + rc=$?; [ "$rc" -eq 0 ] || add_failure "pg-isready|unit=$identity|rc=$rc" + fi + done < "$record/database-baseline.psv" + while IFS='|' read -r model engine identity state detail; do + [ "$model" = container ] || continue + running=$(docker inspect -f '{{.State.Running}}' "$identity" 2>/dev/null) + health=$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}no-healthcheck{{end}}' "$identity" 2>/dev/null) + printf '%s|%s|running=%s|health=%s\n' "$engine" "$identity" "${running:-missing}" "${health:-unknown}" \ + >> "$record/database-readiness-before.txt" + if [ "$running" != true ] || [ "$health" = unhealthy ]; then + add_failure "database-container|name=$identity|running=${running:-missing}|health=${health:-unknown}" + fi + done < "$record/database-baseline.psv" + return 0 +} + +main() { + if [ "$(id -u)" -ne 0 ]; then + printf 'HARD_STOP: run as a separate root process so records are root-only\n' >&2 + return 10 + fi + umask 077 + install -d -o root -g root -m 0700 /var/log/le-phase2a-preflight + rc=$? + if [ "$rc" -ne 0 ]; then + printf 'HARD_STOP: cannot create root-only preflight record parent\n' >&2 + return 11 + fi + record=$(mktemp -d "/var/log/le-phase2a-preflight/preflight.$(date -u +%Y%m%dT%H%M%SZ).XXXXXXXX") + rc=$? + if [ "$rc" -ne 0 ] || [ -z "$record" ] || [ ! -d "$record" ]; then + printf 'HARD_STOP: cannot atomically create unique preflight record\n' >&2 + return 12 + fi + chown root:root "$record" + chmod 0700 "$record" + + overall=PASS + : > "$record/failed-mandatory-checks.txt" + : > "$record/warnings.txt" + timestamp_epoch=$(date +%s) + timestamp_utc=$(date -u +%FT%TZ) + hostname_value=$(hostname) + boot_id=$(cat /proc/sys/kernel/random/boot_id 2>/dev/null) + running_kernel=$(uname -r) + [ -n "$hostname_value" ] || add_failure 'hostname|empty' + [ -n "$boot_id" ] || add_failure 'boot-id|unavailable' + [ -n "$running_kernel" ] || add_failure 'running-kernel|unavailable' + [ "${LE_PHASE2A_RECOVERY_READY:-}" = YES ] || add_failure 'recovery-access-attestation|missing' + [ "${LE_PHASE2A_BACKUPS_READY:-}" = YES ] || add_failure 'backup-readiness-attestation|missing' + + mandatory 'uptime' uptime.txt uptime + mandatory 'logged-in-sessions' logged-in-sessions.txt who -a + mandatory 'filesystem-capacity' filesystem-capacity.txt df -hT / /boot /var/cache/pacman/pkg /tmp + mandatory 'filesystem-inodes' filesystem-inodes.txt df -hi / /boot /var/cache/pacman/pkg /tmp + mandatory 'installed-package-inventory' packages-before.txt pacman -Q + mandatory 'failed-unit-inventory' failed-units-before.txt systemctl --failed --no-legend --plain --no-pager + if [ -s "$record/failed-units-before.txt" ]; then add_failure 'failed-units|present'; fi + mandatory_shell 'kernel-initramfs-inventory' kernel-initramfs-before.txt \ + "find /usr/lib/modules /boot -maxdepth 2 -type f \\ + \( -name vmlinuz -o -name 'vmlinuz-*' -o -name 'initramfs*' -o -name 'initrd*' -o -name pkgbase \) \\ + -printf '%p size=%s mtime=%TY-%Tm-%TdT%TH:%TM:%TS\\n' | sort; test \"\$(find /boot -maxdepth 1 -type f \\ + \( -name 'vmlinuz-*' -o -name 'initramfs*' -o -name 'initrd*' \) | wc -l)\" -gt 0" + capture_bootloader + optional_shell 'dkms-status' dkms-before.txt 'command -v dkms >/dev/null && dkms status' + mandatory_shell 'docker-container-inventory' containers-all.psv \ + "docker ps -a --format '{{.Names}}|{{.Image}}|{{.Status}}|{{.Ports}}' | sort" + mandatory_shell 'running-container-inventory' running-containers-before.txt \ + "docker ps --format '{{.Names}}' | sort" + capture_unit_baseline + capture_database_baseline + validate_database_baseline + + validate_health_urls + check_public_health + mandatory_shell 'network-firewall-listener-inventory' network-firewall-before.txt \ + 'findmnt --real; ip -brief address; ip route show table all; ip -6 route show table all; nft list ruleset; ss -lntup' + optional_shell 'wireguard-observation' wireguard-before.txt 'wg show' + mandatory 'package-database-consistency' pacman-Dk-before.txt pacman -Dk + optional_shell 'packaged-file-consistency-observation' pacman-Qkk-before.txt 'pacman -Qkk' + grep -E -i 'missing|could not read|permission denied|unreadable' "$record/pacman-Qkk-before.txt" \ + > "$record/pacman-Qkk-missing-or-unreadable-before.txt" 2>/dev/null + if [ -s "$record/pacman-Qkk-missing-or-unreadable-before.txt" ]; then + add_failure 'packaged-files-missing-or-unreadable-before-maintenance' + fi + mandatory_shell 'pacnew-pacsave-inventory' pacnew-pacsave-before.txt \ + "find /etc -xdev -type f \( -name '*.pacnew' -o -name '*.pacsave' \) -printf '%p\\n' | sort" + mandatory_shell 'migration-git-state' migration-git-state.txt \ + 'git -C /srv/codex-work/lasereverything.net.db/.worktrees/payload-migration status --short --branch; git -C /srv/codex-work/lasereverything.net.db/.worktrees/payload-migration rev-parse HEAD; git -C /srv/codex-work/lasereverything.net.db/.worktrees/payload-migration rev-parse origin/migration/le-app-database' + mandatory_shell 'preview-report-integrity' preview-report-checksum.txt \ + "printf '%s %s\\n' 195d74afe121556294318d0ed49e3fb5504088dbb31a4c0786b8adea44f1369d /tmp/le-phase2-package-decision-20260712.Mn7mIx/phase2-package-decision-report.md | sha256sum -c -" + + inventory_files='packages-before.txt failed-units-before.txt kernel-initramfs-before.txt bootloader-inventory.txt containers-all.psv running-containers-before.txt critical-units-baseline.psv database-baseline.psv database-readiness-before.txt health-urls.sanitized public-health.txt network-firewall-before.txt pacman-Dk-before.txt pacman-Qkk-before.txt pacman-Qkk-missing-or-unreadable-before.txt pacnew-pacsave-before.txt migration-git-state.txt preview-report-checksum.txt' + ( + cd "$record" || return 1 + sha256sum $inventory_files > inventory-hashes.sha256 + sha256sum -c inventory-hashes.sha256 > inventory-hash-verification.txt + ) + hash_rc=$? + if [ "$hash_rc" -ne 0 ]; then + add_failure "inventory-hashes|rc=$hash_rc" + fi + hashes_hash=$(sha256sum "$record/inventory-hashes.sha256" | awk '{print $1}') + + failure_count=$(wc -l < "$record/failed-mandatory-checks.txt") + failed_checks_hash=$(sha256sum "$record/failed-mandatory-checks.txt" | awk '{print $1}') + { + printf 'format=le-phase2a-preflight-v1\n' + printf 'hostname=%s\n' "$hostname_value" + printf 'boot_id=%s\n' "$boot_id" + printf 'running_kernel=%s\n' "$running_kernel" + printf 'timestamp_epoch=%s\n' "$timestamp_epoch" + printf 'timestamp_utc=%s\n' "$timestamp_utc" + printf 'overall=%s\n' "$overall" + printf 'failure_count=%s\n' "$failure_count" + printf 'failed_checks_file=failed-mandatory-checks.txt\n' + printf 'failed_checks_sha256=%s\n' "$failed_checks_hash" + printf 'inventory_hashes_sha256=%s\n' "$hashes_hash" + } > "$record/result.manifest" + (cd "$record" && sha256sum result.manifest > result.manifest.sha256) + rc=$? + if [ "$rc" -ne 0 ]; then + overall=HARD_STOP + printf '%s\n' "result-manifest-checksum|rc=$rc" >> "$record/failed-mandatory-checks.txt" + fi + chmod -R go-rwx "$record" + printf 'PREFLIGHT_RECORD=%s\n' "$record" + printf 'PREFLIGHT_RESULT=%s failures=%s\n' "$overall" "$failure_count" + if [ "$overall" != PASS ]; then + return 20 + fi + return 0 +} + +main "$@" diff --git a/docs/le-app-database-migration/phase2a-host-maintenance/reboot-procedure.md b/docs/le-app-database-migration/phase2a-host-maintenance/reboot-procedure.md new file mode 100644 index 0000000..3d0fe9c --- /dev/null +++ b/docs/le-app-database-migration/phase2a-host-maintenance/reboot-procedure.md @@ -0,0 +1,65 @@ +# Separate reboot procedure + +This procedure begins only after successful package execution and review of its unique maintenance record. It never reboots automatically. Maintenance and validation records are atomic unique root-owned mode `0700` directories. They can contain sensitive operational metadata and must not be committed. + +## Pre-reboot verification + +Set the exact record path printed by `execute-maintenance.sh`: + +```bash +record=/var/log/le-phase2a-maintenance/maintenance.YYYYMMDDTHHMMSSZ.XXXXXXXX +test -d "$record" || { printf 'STOP: invalid record path\n' >&2; return 1; } +``` + +Record the running kernel and review installed kernel packages: + +```bash +uname -a | tee "$record/running-kernel-immediately-before-reboot.txt" +pacman -Q linux linux-headers linux-zen linux-zen-headers | + tee "$record/installed-kernels-immediately-before-reboot.txt" +``` + +Verify both new module trees, kernel images, and nonempty initramfs files: + +```bash +find /usr/lib/modules /boot -maxdepth 2 -type f \ + \( -name vmlinuz -o -name 'vmlinuz-*' -o -name 'initramfs*' -o -name 'initrd*' -o -name pkgbase \) \ + -printf '%p size=%s mtime=%TY-%Tm-%TdT%TH:%TM:%TS\n' | + sort | tee "$record/verified-kernel-initramfs-files.txt" + +find /boot -maxdepth 1 -type f \ + \( -name 'vmlinuz-*' -o -name 'initramfs*' -o -name 'initrd*' \) -size 0 -print +``` + +Any zero-length artifact is a hard stop. Review DKMS and package hook output: + +```bash +dkms status | tee "$record/dkms-immediately-before-reboot.txt" +systemctl --failed --no-pager | tee "$record/failed-units-immediately-before-reboot.txt" +sed -n '1,240p' "$record/hook-restart-summary.txt" +``` + +Confirm bootloader state without changing it: + +```bash +bootctl status 2>&1 | tee "$record/bootctl-status-before-reboot.txt" +find /boot/loader/entries -maxdepth 1 -type f -print -exec sed -n '1,160p' {} \; 2>/dev/null | + tee "$record/systemd-boot-entries-before-reboot.txt" + +grub-install --version 2>&1 | tee "$record/grub-version-before-reboot.txt" +test -f /boot/grub/grub.cfg && + grep -E "^menuentry |^submenu |linux.*vmlinuz|initrd" /boot/grub/grub.cfg | + tee "$record/grub-entries-before-reboot.txt" +``` + +The operator must identify and record both the intended new-kernel entry and a known-good previous-kernel/fallback entry. Stop if boot entries or initramfs references are missing, DKMS failed, storage is degraded, backups are unavailable, SSH is unstable, or production is unhealthy. + +## Separately approved reboot command + +Only after explicit approval for this exact reboot: + +```bash +sudo systemctl reboot +``` + +Do not embed this command in the maintenance script, a timer, `at`, a shell trap, or a command chain. Keep independent console access open and proceed to `post-reboot-validate.sh` after the host returns. diff --git a/docs/le-app-database-migration/phase2a-host-maintenance/rollback-recovery.md b/docs/le-app-database-migration/phase2a-host-maintenance/rollback-recovery.md new file mode 100644 index 0000000..a802118 --- /dev/null +++ b/docs/le-app-database-migration/phase2a-host-maintenance/rollback-recovery.md @@ -0,0 +1,59 @@ +# Rollback and recovery + +Rollback is emergency recovery, not a routine alternative to a complete Arch upgrade. Stop and coordinate through the operator whenever possible. + +## New kernel fails + +1. Use the independent console or bootloader menu. +2. Select the previously verified kernel or fallback initramfs entry recorded before reboot. +3. Do not repeatedly boot the failed entry or regenerate boot artifacts blindly. +4. After the previous kernel boots, record `uname -a`, boot journal, failed units, DKMS state, `/boot` contents, and bootloader entries. +5. Keep production stopped or degraded only as required for safety; obtain explicit approval before rebuilding initramfs, changing boot defaults, or removing a kernel. + +## Package recovery + +The first recovery source is an exact, validated package archive still present in `/var/cache/pacman/pkg`. Arch Linux Archive is the fallback source when the exact archive is unavailable. + +Emergency package restoration must: + +- use the exact before-inventory and transaction log; +- restore a coherent dependency set, not one guessed library; +- verify package signatures and checksums; +- preserve full command output; +- be approved as emergency recovery; +- be followed by `pacman -Dk`, package/file consistency checks, initramfs/DKMS/bootloader validation, and production health tests. + +Do not improvise a broad downgrade, use `--overwrite` casually, disable signature checking, delete the package database lock without proving no Pacman process exists, or mix repository dates. Arch does not support partial upgrades; a downgrade can require an internally consistent dated repository snapshot. + +## SSH recovery + +- Expect the OpenSSH hook to restart a running `sshd.service` during the transaction. +- Keep the active session open and confirm a second session before reboot when the maintenance plan allows. +- If SSH fails, use the independent console. Inspect `sshd -t`, service status/journal, host keys, PAM, `sshd_config`, and any new `.pacnew` without replacing configuration automatically. +- Do not weaken authentication, expose a new port, copy keys, or disable the firewall as a shortcut. + +## Docker and service recovery + +- Compare against `running-containers-before.txt` and `containers-before.txt`. +- Inspect Docker daemon and container logs, mounts, networks, health checks, and storage availability before restarting anything. +- Restart only an identified failed service with operator approval; do not restart all services or recreate containers/volumes indiscriminately. +- Never delete Docker data, volumes, databases, images, or production configuration as recovery. +- Database recovery follows the service-specific backup/runbook and requires explicit approval before any mutation. + +## Mandatory stop conditions + +Stop rather than improvise on: + +- package signature/keyring/database errors; +- interrupted Pacman transaction or uncertain package state; +- failed kernel, initramfs, DKMS, or bootloader generation; +- missing previous-kernel boot option or loss of console access; +- SSH/PAM/network/firewall configuration ambiguity; +- new unexplained `.pacnew`/`.pacsave` for critical configuration; +- storage, filesystem, database, Docker-volume, or backup inconsistency; +- failed production database, proxy, mail, Forgejo, Docker, or public endpoint; +- any proposed action that would modify production data/configuration, systemd, nftables, credentials, or package history beyond the approved recovery scope. + +Rootless-Docker provisioning and contained Codex setup are never part of recovery. Resume them only after the host is healthy and a later phase is explicitly approved. + +All recovery evidence remains in the root-only unique preflight, maintenance, and post-reboot record directories. It may contain operationally sensitive metadata and must not be committed or copied into a user-readable workspace.