makearmy/le-app
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

add Phase 2A host maintenance procedure

Browse source
This commit is contained in:
makearmy 2026-07-12 16:10:24 -04:00
parent 13ec935e87
commit 2cf249d6f2
9 changed files with 1119 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

6
docs/le-app-database-migration/phase2a-host-maintenance/MANIFEST.sha256 Normal file
View file

@ -0,0 +1,6 @@
756dfad7eddf89406383804f891266bd814c6ca974a5a250fddc72b67c055ddb README.md
9d9db1819ef473954ea9f5009dc985f3b8b090406246bb22a10ffa09caf3e692 preflight-read-only.sh
ab7220b2042b6a256c26189b37e66637cc7cf741420265ecefd94ab6fa6c06b7 execute-maintenance.sh
5b88c68da055d3578e8b19fdf7623c3de204d5ef0af4b17e113938d8bd0a2c5e reboot-procedure.md
b6a67a01d4c4f786906d98603ff3075deff29fb6bc4483a642f18a187e4f552a post-reboot-validate.sh
0805397870c66a523f693cfdba33f7b08c94793b62ddadf720ff84c22796ea13 rollback-recovery.md

121
docs/le-app-database-migration/phase2a-host-maintenance/README.md Normal file
View file

@ -0,0 +1,121 @@
# Phase 2A scheduled host-maintenance review package
Status: **review-only and uncommitted**. Operator provisionally prefers Path A, a scheduled complete Arch host upgrade. This is not execution approval.
Proposed package command:
```bash
pacman -Syu --needed rootlesskit slirp4netns openai-codex
```
This package deliberately separates:
1. read-only preflight: `preflight-read-only.sh`;
2. package execution: `execute-maintenance.sh` — generated, never run during preparation;
3. reboot: `reboot-procedure.md` — separate approval and manual command;
4. post-reboot validation: `post-reboot-validate.sh`;
5. later contained rootless-Docker provisioning — explicitly outside Phase 2A.
The scripts do not enable lingering, configure rootless Docker, modify systemd or nftables, authenticate Codex, copy data, or change production configuration. Package hooks may restart services during the approved upgrade; the execution script records this but does not add its own service restarts.
## Required operator inputs
Before execution, record site-specific values in the maintenance ticket:
- independent console/recovery method and a tested root login path;
- public health URLs, including the canonical application utility endpoint;
- production-critical unit names beyond the conservative defaults in the scripts;
- expected running Docker container names and health states;
- bootloader type and known-good previous kernel entry;
- backup identifiers and restoration instructions;
- maintenance window, user notification, and rollback decision owner.
URLs can be supplied without editing the scripts:
```bash
export PUBLIC_HEALTH_URLS='https://example.invalid/health https://example.invalid/'
```
Health URLs must be non-secret public HTTPS URLs. The scripts reject user-info credentials, query strings, fragments, whitespace/control characters, and non-HTTPS schemes. Never place tokens, signed parameters, credentials, or private endpoints in `PUBLIC_HEALTH_URLS`; the scripts do not read environment files to discover URLs and do not log redirected effective URLs.
Unit names can be extended similarly:
```bash
export EXTRA_CRITICAL_UNITS='example.service another.service'
```
The operator must separately verify independent recovery and restorable backups, then set the non-secret attestations:
```bash
export LE_PHASE2A_RECOVERY_READY=YES
export LE_PHASE2A_BACKUPS_READY=YES
```
## Invocation and enforced records
Invoke every script as a separate process, never with `source` or `.`. Explicitly capture its return status:
```bash
sudo --preserve-env=PUBLIC_HEALTH_URLS,EXTRA_CRITICAL_UNITS,LE_PHASE2A_RECOVERY_READY,LE_PHASE2A_BACKUPS_READY \
bash docs/le-app-database-migration/phase2a-host-maintenance/preflight-read-only.sh
preflight_rc=$?
printf 'preflight_rc=%s\n' "$preflight_rc"
```
The preflight creates an atomic unique root-owned mode `0700` directory below `/var/log/le-phase2a-preflight`, prints its path, and returns nonzero on `HARD_STOP`. Its machine-readable `result.manifest` records hostname, boot ID, kernel, timestamp, result, failures, and inventory hashes. Maintenance accepts only a checksum-valid `PASS` record from the same host and boot that is no more than **1,800 seconds (30 minutes)** old:
```bash
sudo bash docs/le-app-database-migration/phase2a-host-maintenance/execute-maintenance.sh \
--preflight /var/log/le-phase2a-preflight/preflight.YYYYMMDDTHHMMSSZ.XXXXXXXX
maintenance_rc=$?
printf 'maintenance_rc=%s\n' "$maintenance_rc"
```
Maintenance and post-reboot validation create their own atomic unique root-only records below `/var/log/le-phase2a-maintenance` and `/var/log/le-phase2a-post-reboot`. Post-reboot invocation requires the exact maintenance record:
```bash
sudo bash docs/le-app-database-migration/phase2a-host-maintenance/post-reboot-validate.sh \
/var/log/le-phase2a-maintenance/maintenance.YYYYMMDDTHHMMSSZ.XXXXXXXX
validation_rc=$?
printf 'validation_rc=%s\n' "$validation_rc"
```
These records can contain IP addresses, usernames, unit/container names, package state, and limited relevant journal metadata. They never intentionally capture process/container environments, environment-file contents, private keys, credentials, secret files, or command history. Treat every record as operationally sensitive: keep it root-only, do not commit it, and disclose it only through the approved maintenance channel.
## Transaction comparison limitation
The maintenance script refreshes an isolated sync database, normalizes package name, old version, new version, action, count, and downgrade status, and requires exact equality with the approved retained preview. Any detectable mismatch stops execution.
The final live `pacman -Syu` must refresh the live sync databases as part of the complete transaction. Pacman cannot expose that post-refresh final interactive transaction for wrapper comparison without either staging a live `pacman -Sy`-only state or replacing the transaction with an unsafe custom workflow. The script therefore requires two exact operator phrases before launch and then requires the operator to review Pacman's final package table at its native interactive `Y/n` prompt. Answer `n` on any difference. This limitation is explicit and is not represented as machine-enforced equality.
## Package-preview evidence
The retained preview is:
```text
/tmp/le-phase2-package-decision-20260712.Mn7mIx
```
Expected report checksum:
```text
195d74afe121556294318d0ed49e3fb5504088dbb31a4c0786b8adea44f1369d phase2-package-decision-report.md
```
The directory occupies approximately 2.4 GiB. Do not delete it yet. It becomes cleanup-eligible only after the operator accepts the maintenance record, the upgrade/reboot succeeds, and retained evidence has been copied to durable approved storage if required. Cleanup is a later explicit command and is not included in any script.
## Hard stop conditions
Stop rather than improvise if any of the following occurs:
- no independent recovery access or root path;
- insufficient space or an unhealthy `/boot`;
- degraded storage, failed backup, unresolved failed units, or unhealthy production service;
- unexpected package count/version, downgrade, repository error, signature error, keyring error, mirror inconsistency, or partial transaction;
- SSH access is unstable before the OpenSSH restart hook;
- DKMS/initramfs/kernel/bootloader generation fails;
- package database inconsistency or unexplained `.pacnew` affecting boot, SSH, PAM, networking, PHP, database, mail, proxy, or firewall configuration;
- production containers or public endpoints fail after the transaction;
- the operator cannot identify a known-good previous kernel and recovery route.
Do not proceed from package maintenance into rootless-Docker provisioning. That later work remains separately approval-gated by `host-runtime-plan.md`.

316
docs/le-app-database-migration/phase2a-host-maintenance/execute-maintenance.sh Normal file
View file

@ -0,0 +1,316 @@
#!/usr/bin/env bash
# GENERATED FOR REVIEW. Run only as a separate process after explicit approval.
# No shell-wide error options and no parent-shell termination commands.
fail() {
printf 'HARD_STOP: %s\n' "$1" >&2
return "${2:-1}"
}
manifest_value() {
key=$1
file=$2
value=$(awk -F= -v k="$key" '$1==k {sub(/^[^=]*=/, ""); print; found++} END {if(found!=1) exit 1}' "$file")
rc=$?
if [ "$rc" -ne 0 ]; then return "$rc"; fi
printf '%s\n' "$value"
return 0
}
validate_preflight() {
preflight=$1
if [ -z "$preflight" ] || [ ! -d "$preflight" ] || [ -L "$preflight" ]; then
fail 'preflight path is missing, not a directory, or a symlink' 30
return $?
fi
owner=$(stat -c %u "$preflight" 2>/dev/null)
mode=$(stat -c %a "$preflight" 2>/dev/null)
if [ "$owner" != 0 ] || [ "$mode" != 700 ]; then
fail 'preflight record must be root-owned mode 0700' 31
return $?
fi
manifest=$preflight/result.manifest
if [ ! -f "$manifest" ] || [ -L "$manifest" ] || [ ! -f "$preflight/result.manifest.sha256" ]; then
fail 'preflight result manifest is missing or a symlink' 32
return $?
fi
(cd "$preflight" && sha256sum -c result.manifest.sha256 > /dev/null 2>&1)
rc=$?
if [ "$rc" -ne 0 ]; then fail 'preflight result manifest checksum failed' 32; return $?; fi
format=$(manifest_value format "$manifest") || { fail 'malformed preflight format' 33; return $?; }
recorded_host=$(manifest_value hostname "$manifest") || { fail 'malformed preflight hostname' 33; return $?; }
recorded_boot=$(manifest_value boot_id "$manifest") || { fail 'malformed preflight boot ID' 33; return $?; }
recorded_kernel=$(manifest_value running_kernel "$manifest") || { fail 'malformed preflight kernel' 33; return $?; }
recorded_time=$(manifest_value timestamp_epoch "$manifest") || { fail 'malformed preflight timestamp' 33; return $?; }
recorded_overall=$(manifest_value overall "$manifest") || { fail 'malformed preflight result' 33; return $?; }
recorded_failure_count=$(manifest_value failure_count "$manifest") || { fail 'malformed preflight failure count' 33; return $?; }
recorded_failed_checks_hash=$(manifest_value failed_checks_sha256 "$manifest") || { fail 'malformed failed-check hash' 33; return $?; }
recorded_hash=$(manifest_value inventory_hashes_sha256 "$manifest") || { fail 'malformed inventory hash' 33; return $?; }
[ "$format" = le-phase2a-preflight-v1 ] || { fail 'unsupported preflight format' 34; return $?; }
[ "$recorded_overall" = PASS ] || { fail 'preflight result is not PASS' 35; return $?; }
[ "$recorded_failure_count" = 0 ] || { fail 'preflight records mandatory failures' 35; return $?; }
if [ ! -f "$preflight/failed-mandatory-checks.txt" ] || [ -L "$preflight/failed-mandatory-checks.txt" ]; then
fail 'failed-check list is missing or a symlink' 35
return $?
fi
actual_failed_checks_hash=$(sha256sum "$preflight/failed-mandatory-checks.txt" 2>/dev/null | awk '{print $1}')
[ "$actual_failed_checks_hash" = "$recorded_failed_checks_hash" ] || { fail 'failed-check list checksum mismatch' 35; return $?; }
[ "$recorded_host" = "$(hostname)" ] || { fail 'preflight belongs to another host' 36; return $?; }
[ "$recorded_boot" = "$(cat /proc/sys/kernel/random/boot_id)" ] || { fail 'preflight belongs to another boot' 37; return $?; }
[ "$recorded_kernel" = "$(uname -r)" ] || { fail 'running kernel changed since preflight' 37; return $?; }
case "$recorded_time" in *[!0-9]*|'') fail 'preflight timestamp is invalid' 38; return $? ;; esac
now=$(date +%s)
age=$((now - recorded_time))
if [ "$age" -lt 0 ] || [ "$age" -gt 1800 ]; then
fail "preflight is stale; age=${age}s limit=1800s" 39
return $?
fi
actual_hash=$(sha256sum "$preflight/inventory-hashes.sha256" 2>/dev/null | awk '{print $1}')
[ "$actual_hash" = "$recorded_hash" ] || { fail 'preflight inventory-hash manifest checksum mismatch' 40; return $?; }
(cd "$preflight" && sha256sum -c inventory-hashes.sha256 > /dev/null 2>&1)
rc=$?
if [ "$rc" -ne 0 ]; then
fail 'one or more preflight inventory files failed checksum validation' 41
return $?
fi
validated_preflight=$preflight
return 0
}
validate_health_url_file() {
file=$1
[ -s "$file" ] || { fail 'validated preflight contains no health URLs' 42; return $?; }
while IFS= read -r url; do
case "$url" in https://*) ;; *) fail 'unsupported health URL scheme' 42; return $? ;; esac
authority=${url#https://}; authority=${authority%%/*}
case "$url" in *'?'*|*'#'*|*$'\n'*|*$'\r'*|*$'\t'*|*' '*) fail 'unsafe health URL content' 42; return $? ;; esac
case "$authority" in ''|*@*) fail 'health URL user-info or empty authority rejected' 42; return $? ;; esac
done < "$file"
return 0
}
rerun_volatile_checks() {
systemctl --failed --no-legend --plain --no-pager > "$record/volatile-failed-units.txt" 2>&1
rc=$?
[ "$rc" -eq 0 ] || { fail 'cannot capture current failed units' "$rc"; return $?; }
[ ! -s "$record/volatile-failed-units.txt" ] || { fail 'one or more failed units are present' 45; return $?; }
pacman -Dk > "$record/volatile-pacman-Dk.txt" 2>&1
rc=$?
[ "$rc" -eq 0 ] || { fail 'package database consistency failed immediately before execution' "$rc"; return $?; }
df -hT / /boot /var/cache/pacman/pkg /tmp > "$record/volatile-disk-space.txt" 2>&1
rc=$?
[ "$rc" -eq 0 ] || { fail 'cannot verify current disk space' "$rc"; return $?; }
while IFS= read -r container; do
[ -n "$container" ] || continue
running=$(docker inspect -f '{{.State.Running}}' "$container" 2>/dev/null)
health=$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}no-healthcheck{{end}}' "$container" 2>/dev/null)
printf '%s|running=%s|health=%s\n' "$container" "${running:-missing}" "${health:-unknown}" \
>> "$record/volatile-container-health.txt"
if [ "$running" != true ] || [ "$health" = unhealthy ]; then
fail "previously running container is unavailable: $container" 43
return $?
fi
done < "$validated_preflight/running-containers-before.txt"
: > "$record/volatile-public-health.txt"
while IFS= read -r url; do
metrics=$(curl --fail --silent --show-error --max-redirs 0 \
--connect-timeout 10 --max-time 30 --output /dev/null \
--write-out 'http=%{http_code}|remote=%{remote_ip}|time=%{time_total}' \
"$url" 2>> "$record/volatile-public-health-errors.txt")
rc=$?
printf '%s|%s\n' "$url" "$metrics" >> "$record/volatile-public-health.txt"
if [ "$rc" -ne 0 ]; then fail "public health failed immediately before execution: $url" 44; return $?; fi
done < "$validated_preflight/health-urls.sanitized"
return 0
}
normalize_transaction() {
source_file=$1
output_file=$2
downgrade_file=$3
: > "$output_file"
: > "$downgrade_file"
while IFS='|' read -r name new_version repository location download_bytes; do
[ -n "$name" ] || continue
old_line=$(pacman -Q "$name" 2>/dev/null)
if [ -n "$old_line" ]; then
old_version=${old_line#* }
comparison=$(vercmp "$new_version" "$old_version")
if [ "$comparison" -lt 0 ]; then action=downgrade
elif [ "$comparison" -eq 0 ]; then action=same
else action=upgrade
fi
else
old_version=-
action=new
fi
printf '%s|%s|%s|%s\n' "$name" "$old_version" "$new_version" "$action" >> "$output_file"
if [ "$action" = downgrade ]; then
printf '%s|%s|%s|%s\n' "$name" "$old_version" "$new_version" "$action" >> "$downgrade_file"
fi
done < "$source_file"
sort -o "$output_file" "$output_file"
return 0
}
main() {
if [ "$(id -u)" -ne 0 ]; then fail 'run as a separate root process' 10; return $?; fi
if [ "$1" != --preflight ] || [ -z "${2:-}" ] || [ -n "${3:-}" ]; then
fail 'usage: execute-maintenance.sh --preflight /var/log/le-phase2a-preflight/preflight.XXXXXXXX' 11
return $?
fi
validate_preflight "$2" || return $?
validate_health_url_file "$validated_preflight/health-urls.sanitized" || return $?
umask 077
install -d -o root -g root -m 0700 /var/log/le-phase2a-maintenance
rc=$?
[ "$rc" -eq 0 ] || { fail 'cannot create root-only maintenance record parent' "$rc"; return $?; }
record=$(mktemp -d "/var/log/le-phase2a-maintenance/maintenance.$(date -u +%Y%m%dT%H%M%SZ).XXXXXXXX")
rc=$?
if [ "$rc" -ne 0 ] || [ -z "$record" ] || [ ! -d "$record" ]; then
fail 'cannot atomically create unique maintenance record' 12
return $?
fi
chown root:root "$record"; chmod 0700 "$record"
maintenance_start=$(date -u +%FT%TZ)
printf '%s\n' "$record" > "$record/RECORD_PATH.txt"
cp -a "$validated_preflight" "$record/preflight-record"
rc=$?
[ "$rc" -eq 0 ] || { fail 'cannot copy validated preflight record' "$rc"; return $?; }
report=/tmp/le-phase2-package-decision-20260712.Mn7mIx/phase2-package-decision-report.md
report_manifest=/tmp/le-phase2-package-decision-20260712.Mn7mIx/SHA256SUMS.main
printf '%s %s\n' 195d74afe121556294318d0ed49e3fb5504088dbb31a4c0786b8adea44f1369d "$report" | sha256sum -c - \
> "$record/approved-report-checksum.txt" 2>&1
rc=$?
[ "$rc" -eq 0 ] || { fail 'approved report checksum failed' "$rc"; return $?; }
cp "$report" "$report_manifest" "$record/"
rc=$?
[ "$rc" -eq 0 ] || { fail 'cannot copy approved report and checksum manifest' "$rc"; return $?; }
uname -a > "$record/running-kernel-before.txt" 2>&1
rc=$?; [ "$rc" -eq 0 ] || { fail 'cannot capture running kernel' "$rc"; return $?; }
cat /proc/sys/kernel/random/boot_id > "$record/boot-id-before.txt" 2>&1
rc=$?; [ "$rc" -eq 0 ] || { fail 'cannot capture boot ID' "$rc"; return $?; }
pacman -Q > "$record/packages-before.txt" 2>&1
rc=$?; [ "$rc" -eq 0 ] || { fail 'cannot capture installed package inventory' "$rc"; return $?; }
cp "$validated_preflight/bootloader-inventory.txt" "$validated_preflight/kernel-initramfs-before.txt" \
"$validated_preflight/running-containers-before.txt" "$validated_preflight/containers-all.psv" \
"$validated_preflight/critical-units-baseline.psv" "$validated_preflight/database-baseline.psv" "$record/"
rc=$?; [ "$rc" -eq 0 ] || { fail 'cannot copy essential rollback baseline evidence' "$rc"; return $?; }
systemctl --failed --no-legend --plain --no-pager > "$record/failed-units-before.txt" 2>&1
rc=$?; [ "$rc" -eq 0 ] || { fail 'cannot capture failed-unit baseline' "$rc"; return $?; }
[ ! -s "$record/failed-units-before.txt" ] || { fail 'failed units appeared before transaction preview' 45; return $?; }
rerun_volatile_checks || return $?
preview_root=$(mktemp -d /tmp/le-phase2a-execution-preview.XXXXXXXX)
rc=$?
if [ "$rc" -ne 0 ] || [ -z "$preview_root" ] || [ ! -d "$preview_root" ]; then
fail 'cannot create isolated transaction preview directory' 13
return $?
fi
chmod 0700 "$preview_root"; mkdir -m 0700 "$preview_root/db" "$preview_root/cache"
ln -s /var/lib/pacman/local "$preview_root/db/local"
fakeroot -- pacman --config /etc/pacman.conf --dbpath "$preview_root/db" \
--cachedir "$preview_root/cache" --logfile "$preview_root/pacman-preview.log" -Sy \
> "$record/isolated-sync-refresh.log" 2>&1
rc=$?
[ "$rc" -eq 0 ] || { fail 'isolated repository refresh failed; live databases untouched' "$rc"; return $?; }
fakeroot -- pacman --config /etc/pacman.conf --dbpath "$preview_root/db" \
--cachedir "$preview_root/cache" --logfile "$preview_root/pacman-preview.log" \
-Su --needed --print-format '%n|%v|%r|%l|%s' rootlesskit slirp4netns openai-codex \
> "$record/latest-transaction.psv" 2> "$record/latest-transaction.err"
rc=$?
[ "$rc" -eq 0 ] || { fail 'latest complete transaction resolution failed' "$rc"; return $?; }
normalize_transaction "$record/latest-transaction.psv" "$record/latest-normalized-actions.psv" "$record/latest-downgrades.psv"
[ ! -s "$record/latest-downgrades.psv" ] || { fail 'latest transaction contains a downgrade' 14; return $?; }
awk -F'|' '{print $1"|"$2"|"$3"|"$5}' \
/tmp/le-phase2-package-decision-20260712.Mn7mIx/inventory.psv | sort \
> "$record/approved-normalized-actions.psv"
rc=$?
[ "$rc" -eq 0 ] || { fail 'cannot normalize approved preview' "$rc"; return $?; }
diff -u "$record/approved-normalized-actions.psv" "$record/latest-normalized-actions.psv" \
> "$record/approved-vs-latest.diff" 2>&1
diff_rc=$?
if [ "$diff_rc" -ne 0 ]; then
fail 'latest package names, versions, or actions differ from the approved preview' 15
return $?
fi
latest_count=$(wc -l < "$record/latest-normalized-actions.psv")
approved_count=$(wc -l < "$record/approved-normalized-actions.psv")
[ "$latest_count" = "$approved_count" ] || { fail 'transaction package counts differ' 15; return $?; }
for required in rootlesskit slirp4netns openai-codex; do
grep -q "^${required}|" "$record/latest-normalized-actions.psv"
rc=$?; [ "$rc" -eq 0 ] || { fail "latest transaction omits $required" 16; return $?; }
done
cp "$record/latest-transaction.psv" "$record/latest-normalized-actions.psv" \
"$record/approved-normalized-actions.psv" "$record/approved-vs-latest.diff" "$record/transaction-for-approval.psv"
rc=$?; [ "$rc" -eq 0 ] || { fail 'cannot preserve transaction preview evidence' "$rc"; return $?; }
sha256sum "$record/latest-transaction.psv" "$record/latest-normalized-actions.psv" \
"$record/approved-normalized-actions.psv" > "$record/transaction-preview-hashes.sha256"
printf '\nMachine comparison passed: packages=%s, no downgrades, exact match to approved preview.\n' "$latest_count"
cat "$record/latest-normalized-actions.psv"
printf '\nType exactly: APPROVE MATCHED COMPLETE HOST UPGRADE\n> '
IFS= read -r approval_one
if [ "$approval_one" != 'APPROVE MATCHED COMPLETE HOST UPGRADE' ]; then
fail 'first exact operator approval not granted' 20
return $?
fi
rerun_volatile_checks || return $?
printf '\nLIMITATION: the final live -Syu refresh may change after this isolated comparison.\n'
printf 'Pacman will display its final interactive transaction and ask Y/n. The wrapper cannot machine-compare that post-refresh transaction without an unsafe live -Sy-only staging step.\n'
printf 'Type exactly: ACKNOWLEDGE FINAL PACMAN TRANSACTION REQUIRES MANUAL REVIEW\n> '
IFS= read -r approval_two
if [ "$approval_two" != 'ACKNOWLEDGE FINAL PACMAN TRANSACTION REQUIRES MANUAL REVIEW' ]; then
fail 'second exact operator acknowledgement not granted' 21
return $?
fi
printf 'At Pacman Y/n, answer no if any package, version, action, count, downgrade status, signature, or repository differs.\n'
script -q -e -f -c 'pacman -Syu --needed rootlesskit slirp4netns openai-codex' "$record/pacman-complete.log"
rc=$?
if [ "$rc" -ne 0 ]; then fail "Pacman returned $rc; stop without ad hoc repair" "$rc"; return $?; fi
pacman -Q > "$record/packages-after.txt" 2>&1
rc=$?; [ "$rc" -eq 0 ] || { fail 'cannot capture post-transaction package inventory' "$rc"; return $?; }
diff -u "$record/packages-before.txt" "$record/packages-after.txt" > "$record/package-inventory.diff" 2>&1
diff_rc=$?; [ "$diff_rc" -le 1 ] || { fail 'package inventory comparison failed' "$diff_rc"; return $?; }
find /etc -xdev -type f \( -name '*.pacnew' -o -name '*.pacsave' \) -printf '%p\n' | sort \
> "$record/pacnew-pacsave-after.txt" 2>&1
rc=$?; [ "$rc" -eq 0 ] || { fail 'cannot inventory pacnew/pacsave files' "$rc"; return $?; }
pacdiff -o > "$record/pacdiff-outstanding.txt" 2>&1
pacdiff_rc=$?
if [ "$pacdiff_rc" -ne 0 ] && [ "$pacdiff_rc" -ne 1 ]; then
printf 'pacdiff-observation|rc=%s\n' "$pacdiff_rc" > "$record/warnings-after.txt"
fi
grep -E -i 'hook|restart|reload|dkms|dracut|initramfs|mkinitcpio|grub|pacnew|pacsave|warning|error|failed' \
"$record/pacman-complete.log" > "$record/hook-restart-summary.txt" 2>&1
systemctl --failed --no-legend --plain --no-pager > "$record/failed-units-after.txt" 2>&1
journal_units='sshd.service accounts-daemon.service docker.service'
while IFS='|' read -r unit _; do journal_units="$journal_units $unit"; done < "$record/critical-units-baseline.psv"
journalctl --since "$maintenance_start" --no-pager $(for unit in $journal_units; do printf -- '-u %q ' "$unit"; done) \
> "$record/relevant-service-journal.txt" 2>&1
dkms status > "$record/dkms-after.txt" 2>&1
find /boot /usr/lib/modules -maxdepth 2 -type f \
\( -name 'vmlinuz*' -o -name 'initramfs*' -o -name 'initrd*' -o -name pkgbase \) \
-printf '%p size=%s mtime=%TY-%Tm-%TdT%TH:%TM:%TS\n' | sort > "$record/kernel-boot-artifacts-after.txt" 2>&1
docker ps -a --format '{{.Names}}|{{.Image}}|{{.Status}}|{{.Ports}}' | sort > "$record/containers-after.txt" 2>&1
find "$record" -maxdepth 1 -type f ! -name SHA256SUMS -print0 | sort -z | xargs -0 sha256sum \
> "$record/SHA256SUMS" 2>&1
chmod -R go-rwx "$record"
printf 'PACKAGE TRANSACTION COMPLETED; REBOOT NOT PERFORMED.\n'
printf 'MAINTENANCE_RECORD=%s\n' "$record"
printf 'Review all warnings, hooks, services, kernel artifacts, and pacnew/pacsave files.\n'
printf 'Reboot and rootless-Docker provisioning remain separate approval gates.\n'
return 0
}
main "$@"

222
docs/le-app-database-migration/phase2a-host-maintenance/post-reboot-validate.sh Normal file
View file

@ -0,0 +1,222 @@
#!/usr/bin/env bash
# Read-only validation plus root-only evidence recording. Run as a separate process.
add_failure() {
printf '%s\n' "$1" >> "$post_record/failed-checks.txt"
failures=$((failures + 1))
return 0
}
check() {
label=$1; outfile=$2; shift 2
"$@" > "$post_record/$outfile" 2>&1
rc=$?
if [ "$rc" -ne 0 ]; then add_failure "$label|rc=$rc|file=$outfile"; fi
return 0
}
check_shell() {
label=$1; outfile=$2; command_text=$3
/usr/bin/bash -o pipefail -c "$command_text" > "$post_record/$outfile" 2>&1
rc=$?
if [ "$rc" -ne 0 ]; then add_failure "$label|rc=$rc|file=$outfile"; fi
return 0
}
validate_health_url_file() {
file=$1
if [ ! -s "$file" ]; then add_failure 'health-urls|missing'; return 0; fi
while IFS= read -r url; do
case "$url" in https://*) ;; *) add_failure 'health-url|unsupported-scheme'; continue ;; esac
authority=${url#https://}; authority=${authority%%/*}
case "$url" in *'?'*|*'#'*|*$'\n'*|*$'\r'*|*$'\t'*|*' '*) add_failure 'health-url|unsafe-content'; continue ;; esac
case "$authority" in ''|*@*) add_failure 'health-url|userinfo-or-empty-authority'; continue ;; esac
done < "$file"
return 0
}
compare_containers() {
: > "$post_record/container-comparison.psv"
while IFS= read -r container; do
[ -n "$container" ] || continue
running=$(docker inspect -f '{{.State.Running}}' "$container" 2>/dev/null)
health=$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}no-healthcheck{{end}}' "$container" 2>/dev/null)
printf '%s|running=%s|health=%s\n' "$container" "${running:-missing}" "${health:-unknown}" \
>> "$post_record/container-comparison.psv"
if [ "$running" != true ] || [ "$health" = unhealthy ]; then
add_failure "container-regression|name=$container|running=${running:-missing}|health=${health:-unknown}"
fi
done < "$baseline/running-containers-before.txt"
return 0
}
compare_units() {
: > "$post_record/unit-comparison.psv"
while IFS='|' read -r unit old_load old_active old_sub; do
[ -n "$unit" ] || continue
new_load=$(systemctl show "$unit" -p LoadState --value 2>/dev/null)
new_active=$(systemctl show "$unit" -p ActiveState --value 2>/dev/null)
new_sub=$(systemctl show "$unit" -p SubState --value 2>/dev/null)
printf '%s|before=%s,%s,%s|after=%s,%s,%s\n' \
"$unit" "$old_load" "$old_active" "$old_sub" "${new_load:-missing}" "${new_active:-missing}" "${new_sub:-missing}" \
>> "$post_record/unit-comparison.psv"
if [ "$old_load" = loaded ] && [ "$new_load" != loaded ]; then
add_failure "critical-unit-missing|unit=$unit"
elif [ "$old_active" = active ] && [ "$new_active" != active ]; then
add_failure "critical-unit-not-active|unit=$unit|state=${new_active:-missing}"
fi
done < "$baseline/critical-units-baseline.psv"
return 0
}
validate_databases() {
: > "$post_record/database-validation.txt"
while IFS='|' read -r model engine identity state detail; do
[ -n "$model" ] || continue
if [ "$model" = unit ]; then
active=$(systemctl show "$identity" -p ActiveState --value 2>/dev/null)
printf '%s|%s|unit=%s|active=%s\n' "$model" "$engine" "$identity" "${active:-missing}" \
>> "$post_record/database-validation.txt"
if [ "$active" != active ]; then add_failure "database-unit-not-active|unit=$identity"; fi
if [ "$engine" = mysql ]; then
mysqladmin ping >> "$post_record/database-validation.txt" 2>&1
rc=$?; [ "$rc" -eq 0 ] || add_failure "mysqladmin-ping-failed|unit=$identity|rc=$rc"
elif [ "$engine" = postgresql ]; then
pg_isready >> "$post_record/database-validation.txt" 2>&1
rc=$?; [ "$rc" -eq 0 ] || add_failure "pg-isready-failed|unit=$identity|rc=$rc"
fi
elif [ "$model" = container ]; then
running=$(docker inspect -f '{{.State.Running}}' "$identity" 2>/dev/null)
health=$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}no-healthcheck{{end}}' "$identity" 2>/dev/null)
printf '%s|%s|container=%s|running=%s|health=%s\n' \
"$model" "$engine" "$identity" "${running:-missing}" "${health:-unknown}" \
>> "$post_record/database-validation.txt"
if [ "$running" != true ] || [ "$health" = unhealthy ]; then
add_failure "database-container-failed|container=$identity|running=${running:-missing}|health=${health:-unknown}"
fi
fi
done < "$baseline/database-baseline.psv"
return 0
}
check_package_files() {
pacman -Qkk > "$post_record/pacman-Qkk.txt" 2>&1
qkk_rc=$?
printf 'pacman-Qkk-return=%s\n' "$qkk_rc" > "$post_record/pacman-Qkk-classification.txt"
grep -E -i 'missing|could not read|permission denied|unreadable' "$post_record/pacman-Qkk.txt" \
> "$post_record/pacman-Qkk-missing-or-unreadable.txt" 2>/dev/null
missing_rc=$?
grep -E -i 'altered|modified' "$post_record/pacman-Qkk.txt" \
> "$post_record/pacman-Qkk-modified.txt" 2>/dev/null
if [ "$missing_rc" -eq 0 ] && [ -s "$post_record/pacman-Qkk-missing-or-unreadable.txt" ]; then
add_failure 'packaged-files-missing-or-unreadable'
fi
printf 'Modified configuration/package files require manual classification; no repair performed.\n' \
>> "$post_record/pacman-Qkk-classification.txt"
return 0
}
main() {
failures=0
maintenance=${1:-}
if [ "$(id -u)" -ne 0 ]; then
printf 'HARD_STOP: run as a separate root process\n' >&2
return 10
fi
if [ -z "$maintenance" ] || [ ! -d "$maintenance" ] || [ -L "$maintenance" ]; then
printf 'HARD_STOP: supply the exact maintenance record path\n' >&2
return 11
fi
owner=$(stat -c %u "$maintenance" 2>/dev/null); mode=$(stat -c %a "$maintenance" 2>/dev/null)
if [ "$owner" != 0 ] || [ "$mode" != 700 ]; then
printf 'HARD_STOP: maintenance record must be root-owned mode 0700\n' >&2
return 12
fi
baseline=$maintenance/preflight-record
required='running-containers-before.txt critical-units-baseline.psv database-baseline.psv health-urls.sanitized inventory-hashes.sha256 result.manifest result.manifest.sha256'
for file in $required; do
if [ ! -f "$baseline/$file" ] || [ -L "$baseline/$file" ]; then
printf 'HARD_STOP: missing baseline file %s\n' "$file" >&2
return 13
fi
done
(cd "$baseline" && sha256sum -c inventory-hashes.sha256 > /dev/null 2>&1)
rc=$?
if [ "$rc" -ne 0 ]; then
printf 'HARD_STOP: copied preflight baseline checksum failure\n' >&2
return 14
fi
(cd "$baseline" && sha256sum -c result.manifest.sha256 > /dev/null 2>&1)
rc=$?
if [ "$rc" -ne 0 ]; then
printf 'HARD_STOP: copied preflight result manifest checksum failure\n' >&2
return 14
fi
umask 077
install -d -o root -g root -m 0700 /var/log/le-phase2a-post-reboot
rc=$?; [ "$rc" -eq 0 ] || { printf 'HARD_STOP: cannot create post-reboot record parent\n' >&2; return 15; }
post_record=$(mktemp -d "/var/log/le-phase2a-post-reboot/validation.$(date -u +%Y%m%dT%H%M%SZ).XXXXXXXX")
rc=$?
if [ "$rc" -ne 0 ] || [ -z "$post_record" ] || [ ! -d "$post_record" ]; then
printf 'HARD_STOP: cannot atomically create unique post-reboot record\n' >&2
return 16
fi
chown root:root "$post_record"; chmod 0700 "$post_record"
: > "$post_record/failed-checks.txt"
printf 'maintenance_record=%s\n' "$maintenance" > "$post_record/source.txt"
check 'running-kernel' running-kernel.txt uname -a
check_shell 'installed-kernels' installed-kernels.txt 'pacman -Q linux linux-headers linux-zen linux-zen-headers'
check 'failed-units' failed-units.txt systemctl --failed --no-legend --plain --no-pager
if [ -s "$post_record/failed-units.txt" ]; then add_failure 'failed-units-present'; fi
check_shell 'ssh-state' ssh-state.txt \
'systemctl show sshd.service -p LoadState -p ActiveState -p SubState -p ExecMainStatus; journalctl -b -u sshd.service --no-pager'
check_shell 'docker-daemon' docker-daemon.txt \
"systemctl show docker.service -p LoadState -p ActiveState -p SubState -p ExecMainStatus; docker info"
check_shell 'all-containers' containers-after.psv \
"docker ps -a --format '{{.Names}}|{{.Image}}|{{.Status}}|{{.Ports}}' | sort"
compare_containers
compare_units
validate_databases
check_shell 'mounts-and-capacity' mounts-capacity.txt 'findmnt --real; df -hT / /boot /var/cache/pacman/pkg /tmp'
check_shell 'network-routes-firewall-ports' network-after.txt \
'ip -brief address; ip route show table all; ip -6 route show table all; nft list ruleset; ss -lntup'
wg show > "$post_record/wireguard-after.txt" 2>&1
wg_rc=$?; if [ "$wg_rc" -ne 0 ]; then printf 'wireguard-observation|rc=%s\n' "$wg_rc" > "$post_record/warnings.txt"; fi
validate_health_url_file "$baseline/health-urls.sanitized"
: > "$post_record/public-health.txt"
while IFS= read -r url; do
metrics=$(curl --fail --silent --show-error --max-redirs 0 \
--connect-timeout 10 --max-time 30 --output /dev/null \
--write-out 'http=%{http_code}|remote=%{remote_ip}|time=%{time_total}' \
"$url" 2>> "$post_record/public-health-errors.txt")
rc=$?
printf '%s|%s\n' "$url" "$metrics" >> "$post_record/public-health.txt"
if [ "$rc" -ne 0 ]; then add_failure "public-health-failed|url=$url|rc=$rc"; fi
done < "$baseline/health-urls.sanitized"
pacman -Dk > "$post_record/pacman-Dk.txt" 2>&1
dk_rc=$?
if [ "$dk_rc" -ne 0 ]; then add_failure "pacman-Dk-failed|rc=$dk_rc"; fi
check_package_files
check_shell 'target-packages' target-packages.txt 'pacman -Q rootlesskit slirp4netns openai-codex'
check_shell 'pacnew-pacsave-inventory' pacnew-pacsave.txt \
"find /etc -xdev -type f \( -name '*.pacnew' -o -name '*.pacsave' \) -printf '%p\\n' | sort; pacdiff -o 2>&1"
find "$post_record" -maxdepth 1 -type f ! -name SHA256SUMS -print0 | sort -z | xargs -0 sha256sum \
> "$post_record/SHA256SUMS" 2>&1
chmod -R go-rwx "$post_record"
printf 'POST_REBOOT_RECORD=%s\n' "$post_record"
if [ "$failures" -ne 0 ]; then
printf 'POST_REBOOT_RESULT=HARD_STOP failures=%s\n' "$failures"
return 20
fi
printf 'POST_REBOOT_RESULT=PASS failures=0\n'
printf 'Rootless-Docker provisioning remains separately approval-gated.\n'
return 0
}
main "$@"

325
docs/le-app-database-migration/phase2a-host-maintenance/preflight-read-only.sh Normal file
View file

@ -0,0 +1,325 @@
#!/usr/bin/env bash
# Read-only host inspection plus root-only evidence recording.
# Run as a separate process; this script changes no shell options and uses return only.
add_failure() {
printf '%s\n' "$1" >> "$record/failed-mandatory-checks.txt"
overall=HARD_STOP
return 0
}
mandatory() {
label=$1
outfile=$2
shift 2
"$@" > "$record/$outfile" 2>&1
rc=$?
if [ "$rc" -ne 0 ]; then
add_failure "$label|rc=$rc|file=$outfile"
fi
return 0
}
mandatory_shell() {
label=$1
outfile=$2
command_text=$3
/usr/bin/bash -o pipefail -c "$command_text" > "$record/$outfile" 2>&1
rc=$?
if [ "$rc" -ne 0 ]; then
add_failure "$label|rc=$rc|file=$outfile"
fi
return 0
}
optional_shell() {
label=$1
outfile=$2
command_text=$3
/usr/bin/bash -o pipefail -c "$command_text" > "$record/$outfile" 2>&1
rc=$?
if [ "$rc" -ne 0 ]; then
printf '%s|rc=%s|file=%s\n' "$label" "$rc" "$outfile" >> "$record/warnings.txt"
fi
return 0
}
validate_health_urls() {
: > "$record/health-urls.sanitized"
if [ -z "${PUBLIC_HEALTH_URLS:-}" ]; then
add_failure 'public-health-urls|unset'
return 0
fi
case "$PUBLIC_HEALTH_URLS" in
*$'\n'*|*$'\r'*|*$'\t'*)
add_failure 'public-health-urls|control-character-rejected'
return 0
;;
esac
for url in $PUBLIC_HEALTH_URLS; do
case "$url" in
https://*) ;;
*) add_failure 'public-health-url|unsupported-scheme-or-malformed'; continue ;;
esac
authority=${url#https://}
authority=${authority%%/*}
case "$url" in
*'?'*|*'#'*|*$'\n'*|*$'\r'*|*$'\t'*|*' '*)
add_failure 'public-health-url|query-fragment-or-control-character-rejected'
continue
;;
esac
case "$authority" in
''|*@*) add_failure 'public-health-url|empty-authority-or-userinfo-rejected'; continue ;;
esac
printf '%s\n' "$url" >> "$record/health-urls.sanitized"
done
if [ ! -s "$record/health-urls.sanitized" ]; then
add_failure 'public-health-urls|no-valid-non-secret-https-url'
fi
return 0
}
check_public_health() {
: > "$record/public-health.txt"
while IFS= read -r url; do
[ -n "$url" ] || continue
metrics=$(curl --fail --silent --show-error --max-redirs 0 \
--connect-timeout 10 --max-time 30 --output /dev/null \
--write-out 'http=%{http_code}|remote=%{remote_ip}|time=%{time_total}' \
"$url" 2>> "$record/public-health-errors.txt")
rc=$?
printf '%s|%s\n' "$url" "$metrics" >> "$record/public-health.txt"
if [ "$rc" -ne 0 ]; then
add_failure "public-health|rc=$rc|url=$url"
fi
done < "$record/health-urls.sanitized"
return 0
}
capture_bootloader() {
found=0
: > "$record/bootloader-inventory.txt"
if [ -d /boot/loader/entries ]; then
printf '%s\n' 'type=systemd-boot' >> "$record/bootloader-inventory.txt"
bootctl status >> "$record/bootloader-inventory.txt" 2>&1
bootctl_rc=$?
find /boot/loader/entries -maxdepth 1 -type f -print -exec sed -n '1,160p' {} \; \
>> "$record/bootloader-inventory.txt" 2>&1
find_rc=$?
if [ "$bootctl_rc" -eq 0 ] && [ "$find_rc" -eq 0 ]; then found=1; fi
fi
if [ -f /boot/grub/grub.cfg ]; then
printf '%s\n' 'type=grub' >> "$record/bootloader-inventory.txt"
grub-install --version >> "$record/bootloader-inventory.txt" 2>&1
grub_rc=$?
grep -E '^menuentry |^submenu |linux.*vmlinuz|initrd' /boot/grub/grub.cfg \
>> "$record/bootloader-inventory.txt" 2>&1
grep_rc=$?
if [ "$grub_rc" -eq 0 ] && [ "$grep_rc" -eq 0 ]; then found=1; fi
fi
if [ "$found" -ne 1 ]; then
add_failure 'bootloader-inventory|no-verified-supported-bootloader'
fi
return 0
}
capture_unit_baseline() {
systemctl list-units --type=service --all --no-legend --no-pager \
> "$record/all-service-units.txt" 2>&1
rc=$?
if [ "$rc" -ne 0 ]; then
add_failure "service-unit-inventory|rc=$rc"
return 0
fi
: > "$record/critical-units-baseline.psv"
awk '{print $1}' "$record/all-service-units.txt" | while IFS= read -r unit; do
case "$unit" in
docker.service|nginx.service|php-fpm.service|mariadb.service|mysqld.service|postgresql.service|postfix.service|dovecot.service|rspamd.service|forgejo.service|sshd.service)
;;
*) continue ;;
esac
load=$(systemctl show "$unit" -p LoadState --value 2>/dev/null)
active=$(systemctl show "$unit" -p ActiveState --value 2>/dev/null)
sub=$(systemctl show "$unit" -p SubState --value 2>/dev/null)
printf '%s|%s|%s|%s\n' "$unit" "$load" "$active" "$sub"
done > "$record/critical-units-baseline.psv"
for unit in ${EXTRA_CRITICAL_UNITS:-}; do
load=$(systemctl show "$unit" -p LoadState --value 2>/dev/null)
if [ "$load" = loaded ]; then
active=$(systemctl show "$unit" -p ActiveState --value 2>/dev/null)
sub=$(systemctl show "$unit" -p SubState --value 2>/dev/null)
printf '%s|%s|%s|%s\n' "$unit" "$load" "$active" "$sub" \
>> "$record/critical-units-baseline.psv"
else
add_failure "extra-critical-unit|not-loaded|unit=$unit"
fi
done
sort -u -o "$record/critical-units-baseline.psv" "$record/critical-units-baseline.psv"
return 0
}
capture_database_baseline() {
: > "$record/database-baseline.psv"
while IFS='|' read -r unit load active sub; do
[ "$active" = active ] || continue
case "$unit" in
mariadb.service|mysqld.service) printf 'unit|mysql|%s|%s|%s\n' "$unit" "$active" "$sub" ;;
postgresql.service) printf 'unit|postgresql|%s|%s|%s\n' "$unit" "$active" "$sub" ;;
esac
done < "$record/critical-units-baseline.psv" >> "$record/database-baseline.psv"
while IFS='|' read -r name image status ports; do
case "$status" in Up\ *) ;; *) continue ;; esac
lowered=$(printf '%s %s' "$name" "$image" | tr '[:upper:]' '[:lower:]')
case "$lowered" in
*maria*|*mysql*) printf 'container|mysql|%s|%s|%s\n' "$name" "$status" "$ports" ;;
*postgres*) printf 'container|postgresql|%s|%s|%s\n' "$name" "$status" "$ports" ;;
esac
done < "$record/containers-all.psv" >> "$record/database-baseline.psv"
return 0
}
validate_database_baseline() {
: > "$record/database-readiness-before.txt"
while IFS='|' read -r model engine identity state detail; do
[ "$model" = unit ] || continue
if [ "$engine" = mysql ]; then
mysqladmin ping >> "$record/database-readiness-before.txt" 2>&1
rc=$?; [ "$rc" -eq 0 ] || add_failure "mysqladmin-ping|unit=$identity|rc=$rc"
elif [ "$engine" = postgresql ]; then
pg_isready >> "$record/database-readiness-before.txt" 2>&1
rc=$?; [ "$rc" -eq 0 ] || add_failure "pg-isready|unit=$identity|rc=$rc"
fi
done < "$record/database-baseline.psv"
while IFS='|' read -r model engine identity state detail; do
[ "$model" = container ] || continue
running=$(docker inspect -f '{{.State.Running}}' "$identity" 2>/dev/null)
health=$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}no-healthcheck{{end}}' "$identity" 2>/dev/null)
printf '%s|%s|running=%s|health=%s\n' "$engine" "$identity" "${running:-missing}" "${health:-unknown}" \
>> "$record/database-readiness-before.txt"
if [ "$running" != true ] || [ "$health" = unhealthy ]; then
add_failure "database-container|name=$identity|running=${running:-missing}|health=${health:-unknown}"
fi
done < "$record/database-baseline.psv"
return 0
}
main() {
if [ "$(id -u)" -ne 0 ]; then
printf 'HARD_STOP: run as a separate root process so records are root-only\n' >&2
return 10
fi
umask 077
install -d -o root -g root -m 0700 /var/log/le-phase2a-preflight
rc=$?
if [ "$rc" -ne 0 ]; then
printf 'HARD_STOP: cannot create root-only preflight record parent\n' >&2
return 11
fi
record=$(mktemp -d "/var/log/le-phase2a-preflight/preflight.$(date -u +%Y%m%dT%H%M%SZ).XXXXXXXX")
rc=$?
if [ "$rc" -ne 0 ] || [ -z "$record" ] || [ ! -d "$record" ]; then
printf 'HARD_STOP: cannot atomically create unique preflight record\n' >&2
return 12
fi
chown root:root "$record"
chmod 0700 "$record"
overall=PASS
: > "$record/failed-mandatory-checks.txt"
: > "$record/warnings.txt"
timestamp_epoch=$(date +%s)
timestamp_utc=$(date -u +%FT%TZ)
hostname_value=$(hostname)
boot_id=$(cat /proc/sys/kernel/random/boot_id 2>/dev/null)
running_kernel=$(uname -r)
[ -n "$hostname_value" ] || add_failure 'hostname|empty'
[ -n "$boot_id" ] || add_failure 'boot-id|unavailable'
[ -n "$running_kernel" ] || add_failure 'running-kernel|unavailable'
[ "${LE_PHASE2A_RECOVERY_READY:-}" = YES ] || add_failure 'recovery-access-attestation|missing'
[ "${LE_PHASE2A_BACKUPS_READY:-}" = YES ] || add_failure 'backup-readiness-attestation|missing'
mandatory 'uptime' uptime.txt uptime
mandatory 'logged-in-sessions' logged-in-sessions.txt who -a
mandatory 'filesystem-capacity' filesystem-capacity.txt df -hT / /boot /var/cache/pacman/pkg /tmp
mandatory 'filesystem-inodes' filesystem-inodes.txt df -hi / /boot /var/cache/pacman/pkg /tmp
mandatory 'installed-package-inventory' packages-before.txt pacman -Q
mandatory 'failed-unit-inventory' failed-units-before.txt systemctl --failed --no-legend --plain --no-pager
if [ -s "$record/failed-units-before.txt" ]; then add_failure 'failed-units|present'; fi
mandatory_shell 'kernel-initramfs-inventory' kernel-initramfs-before.txt \
"find /usr/lib/modules /boot -maxdepth 2 -type f \\
\( -name vmlinuz -o -name 'vmlinuz-*' -o -name 'initramfs*' -o -name 'initrd*' -o -name pkgbase \) \\
-printf '%p size=%s mtime=%TY-%Tm-%TdT%TH:%TM:%TS\\n' | sort; test \"\$(find /boot -maxdepth 1 -type f \\
\( -name 'vmlinuz-*' -o -name 'initramfs*' -o -name 'initrd*' \) | wc -l)\" -gt 0"
capture_bootloader
optional_shell 'dkms-status' dkms-before.txt 'command -v dkms >/dev/null && dkms status'
mandatory_shell 'docker-container-inventory' containers-all.psv \
"docker ps -a --format '{{.Names}}|{{.Image}}|{{.Status}}|{{.Ports}}' | sort"
mandatory_shell 'running-container-inventory' running-containers-before.txt \
"docker ps --format '{{.Names}}' | sort"
capture_unit_baseline
capture_database_baseline
validate_database_baseline
validate_health_urls
check_public_health
mandatory_shell 'network-firewall-listener-inventory' network-firewall-before.txt \
'findmnt --real; ip -brief address; ip route show table all; ip -6 route show table all; nft list ruleset; ss -lntup'
optional_shell 'wireguard-observation' wireguard-before.txt 'wg show'
mandatory 'package-database-consistency' pacman-Dk-before.txt pacman -Dk
optional_shell 'packaged-file-consistency-observation' pacman-Qkk-before.txt 'pacman -Qkk'
grep -E -i 'missing|could not read|permission denied|unreadable' "$record/pacman-Qkk-before.txt" \
> "$record/pacman-Qkk-missing-or-unreadable-before.txt" 2>/dev/null
if [ -s "$record/pacman-Qkk-missing-or-unreadable-before.txt" ]; then
add_failure 'packaged-files-missing-or-unreadable-before-maintenance'
fi
mandatory_shell 'pacnew-pacsave-inventory' pacnew-pacsave-before.txt \
"find /etc -xdev -type f \( -name '*.pacnew' -o -name '*.pacsave' \) -printf '%p\\n' | sort"
mandatory_shell 'migration-git-state' migration-git-state.txt \
'git -C /srv/codex-work/lasereverything.net.db/.worktrees/payload-migration status --short --branch; git -C /srv/codex-work/lasereverything.net.db/.worktrees/payload-migration rev-parse HEAD; git -C /srv/codex-work/lasereverything.net.db/.worktrees/payload-migration rev-parse origin/migration/le-app-database'
mandatory_shell 'preview-report-integrity' preview-report-checksum.txt \
"printf '%s %s\\n' 195d74afe121556294318d0ed49e3fb5504088dbb31a4c0786b8adea44f1369d /tmp/le-phase2-package-decision-20260712.Mn7mIx/phase2-package-decision-report.md | sha256sum -c -"
inventory_files='packages-before.txt failed-units-before.txt kernel-initramfs-before.txt bootloader-inventory.txt containers-all.psv running-containers-before.txt critical-units-baseline.psv database-baseline.psv database-readiness-before.txt health-urls.sanitized public-health.txt network-firewall-before.txt pacman-Dk-before.txt pacman-Qkk-before.txt pacman-Qkk-missing-or-unreadable-before.txt pacnew-pacsave-before.txt migration-git-state.txt preview-report-checksum.txt'
(
cd "$record" || return 1
sha256sum $inventory_files > inventory-hashes.sha256
sha256sum -c inventory-hashes.sha256 > inventory-hash-verification.txt
)
hash_rc=$?
if [ "$hash_rc" -ne 0 ]; then
add_failure "inventory-hashes|rc=$hash_rc"
fi
hashes_hash=$(sha256sum "$record/inventory-hashes.sha256" | awk '{print $1}')
failure_count=$(wc -l < "$record/failed-mandatory-checks.txt")
failed_checks_hash=$(sha256sum "$record/failed-mandatory-checks.txt" | awk '{print $1}')
{
printf 'format=le-phase2a-preflight-v1\n'
printf 'hostname=%s\n' "$hostname_value"
printf 'boot_id=%s\n' "$boot_id"
printf 'running_kernel=%s\n' "$running_kernel"
printf 'timestamp_epoch=%s\n' "$timestamp_epoch"
printf 'timestamp_utc=%s\n' "$timestamp_utc"
printf 'overall=%s\n' "$overall"
printf 'failure_count=%s\n' "$failure_count"
printf 'failed_checks_file=failed-mandatory-checks.txt\n'
printf 'failed_checks_sha256=%s\n' "$failed_checks_hash"
printf 'inventory_hashes_sha256=%s\n' "$hashes_hash"
} > "$record/result.manifest"
(cd "$record" && sha256sum result.manifest > result.manifest.sha256)
rc=$?
if [ "$rc" -ne 0 ]; then
overall=HARD_STOP
printf '%s\n' "result-manifest-checksum|rc=$rc" >> "$record/failed-mandatory-checks.txt"
fi
chmod -R go-rwx "$record"
printf 'PREFLIGHT_RECORD=%s\n' "$record"
printf 'PREFLIGHT_RESULT=%s failures=%s\n' "$overall" "$failure_count"
if [ "$overall" != PASS ]; then
return 20
fi
return 0
}
main "$@"

65
docs/le-app-database-migration/phase2a-host-maintenance/reboot-procedure.md Normal file
View file

@ -0,0 +1,65 @@
# Separate reboot procedure
This procedure begins only after successful package execution and review of its unique maintenance record. It never reboots automatically. Maintenance and validation records are atomic unique root-owned mode `0700` directories. They can contain sensitive operational metadata and must not be committed.
## Pre-reboot verification
Set the exact record path printed by `execute-maintenance.sh`:
```bash
record=/var/log/le-phase2a-maintenance/maintenance.YYYYMMDDTHHMMSSZ.XXXXXXXX
test -d "$record" || { printf 'STOP: invalid record path\n' >&2; return 1; }
```
Record the running kernel and review installed kernel packages:
```bash
uname -a | tee "$record/running-kernel-immediately-before-reboot.txt"
pacman -Q linux linux-headers linux-zen linux-zen-headers |
tee "$record/installed-kernels-immediately-before-reboot.txt"
```
Verify both new module trees, kernel images, and nonempty initramfs files:
```bash
find /usr/lib/modules /boot -maxdepth 2 -type f \
\( -name vmlinuz -o -name 'vmlinuz-*' -o -name 'initramfs*' -o -name 'initrd*' -o -name pkgbase \) \
-printf '%p size=%s mtime=%TY-%Tm-%TdT%TH:%TM:%TS\n' |
sort | tee "$record/verified-kernel-initramfs-files.txt"
find /boot -maxdepth 1 -type f \
\( -name 'vmlinuz-*' -o -name 'initramfs*' -o -name 'initrd*' \) -size 0 -print
```
Any zero-length artifact is a hard stop. Review DKMS and package hook output:
```bash
dkms status | tee "$record/dkms-immediately-before-reboot.txt"
systemctl --failed --no-pager | tee "$record/failed-units-immediately-before-reboot.txt"
sed -n '1,240p' "$record/hook-restart-summary.txt"
```
Confirm bootloader state without changing it:
```bash
bootctl status 2>&1 | tee "$record/bootctl-status-before-reboot.txt"
find /boot/loader/entries -maxdepth 1 -type f -print -exec sed -n '1,160p' {} \; 2>/dev/null |
tee "$record/systemd-boot-entries-before-reboot.txt"
grub-install --version 2>&1 | tee "$record/grub-version-before-reboot.txt"
test -f /boot/grub/grub.cfg &&
grep -E "^menuentry |^submenu |linux.*vmlinuz|initrd" /boot/grub/grub.cfg |
tee "$record/grub-entries-before-reboot.txt"
```
The operator must identify and record both the intended new-kernel entry and a known-good previous-kernel/fallback entry. Stop if boot entries or initramfs references are missing, DKMS failed, storage is degraded, backups are unavailable, SSH is unstable, or production is unhealthy.
## Separately approved reboot command
Only after explicit approval for this exact reboot:
```bash
sudo systemctl reboot
```
Do not embed this command in the maintenance script, a timer, `at`, a shell trap, or a command chain. Keep independent console access open and proceed to `post-reboot-validate.sh` after the host returns.

59
docs/le-app-database-migration/phase2a-host-maintenance/rollback-recovery.md Normal file
View file

@ -0,0 +1,59 @@
# Rollback and recovery
Rollback is emergency recovery, not a routine alternative to a complete Arch upgrade. Stop and coordinate through the operator whenever possible.
## New kernel fails
1. Use the independent console or bootloader menu.
2. Select the previously verified kernel or fallback initramfs entry recorded before reboot.
3. Do not repeatedly boot the failed entry or regenerate boot artifacts blindly.
4. After the previous kernel boots, record `uname -a`, boot journal, failed units, DKMS state, `/boot` contents, and bootloader entries.
5. Keep production stopped or degraded only as required for safety; obtain explicit approval before rebuilding initramfs, changing boot defaults, or removing a kernel.
## Package recovery
The first recovery source is an exact, validated package archive still present in `/var/cache/pacman/pkg`. Arch Linux Archive is the fallback source when the exact archive is unavailable.
Emergency package restoration must:
- use the exact before-inventory and transaction log;
- restore a coherent dependency set, not one guessed library;
- verify package signatures and checksums;
- preserve full command output;
- be approved as emergency recovery;
- be followed by `pacman -Dk`, package/file consistency checks, initramfs/DKMS/bootloader validation, and production health tests.
Do not improvise a broad downgrade, use `--overwrite` casually, disable signature checking, delete the package database lock without proving no Pacman process exists, or mix repository dates. Arch does not support partial upgrades; a downgrade can require an internally consistent dated repository snapshot.
## SSH recovery
- Expect the OpenSSH hook to restart a running `sshd.service` during the transaction.
- Keep the active session open and confirm a second session before reboot when the maintenance plan allows.
- If SSH fails, use the independent console. Inspect `sshd -t`, service status/journal, host keys, PAM, `sshd_config`, and any new `.pacnew` without replacing configuration automatically.
- Do not weaken authentication, expose a new port, copy keys, or disable the firewall as a shortcut.
## Docker and service recovery
- Compare against `running-containers-before.txt` and `containers-before.txt`.
- Inspect Docker daemon and container logs, mounts, networks, health checks, and storage availability before restarting anything.
- Restart only an identified failed service with operator approval; do not restart all services or recreate containers/volumes indiscriminately.
- Never delete Docker data, volumes, databases, images, or production configuration as recovery.
- Database recovery follows the service-specific backup/runbook and requires explicit approval before any mutation.
## Mandatory stop conditions
Stop rather than improvise on:
- package signature/keyring/database errors;
- interrupted Pacman transaction or uncertain package state;
- failed kernel, initramfs, DKMS, or bootloader generation;
- missing previous-kernel boot option or loss of console access;
- SSH/PAM/network/firewall configuration ambiguity;
- new unexplained `.pacnew`/`.pacsave` for critical configuration;
- storage, filesystem, database, Docker-volume, or backup inconsistency;
- failed production database, proxy, mail, Forgejo, Docker, or public endpoint;
- any proposed action that would modify production data/configuration, systemd, nftables, credentials, or package history beyond the approved recovery scope.
Rootless-Docker provisioning and contained Codex setup are never part of recovery. Resume them only after the host is healthy and a later phase is explicitly approved.
All recovery evidence remains in the root-only unique preflight, maintenance, and post-reboot record directories. It may contain operationally sensitive metadata and must not be committed or copied into a user-readable workspace.