diff --git a/docs/le-app-database-migration/phase2b-stage2-verification/MANIFEST.sha256 b/docs/le-app-database-migration/phase2b-stage2-verification/MANIFEST.sha256 new file mode 100644 index 0000000..0f261b2 --- /dev/null +++ b/docs/le-app-database-migration/phase2b-stage2-verification/MANIFEST.sha256 @@ -0,0 +1,4 @@ +0a1cffbe98e0cdd7e70b02c92a6e14ca51b615f3888a405627abb1bd054bafef README.md +e41030a9850097ad91899ec14b2ce1a85f218fddf9b38aa317663450b29b6470 ROLLBACK.md +19b8de84c9e0e6025d66835a2eb834a5ee6c1eb6599e8650ce287f0cf069950e tests/40-verify-stage2.sh +10d4e45f0508e7dd6024b52a58e2848cb792f72a9e3cc070bf4c316df1ee4880 tests/41-stage2-static-tests.sh diff --git a/docs/le-app-database-migration/phase2b-stage2-verification/README.md b/docs/le-app-database-migration/phase2b-stage2-verification/README.md new file mode 100644 index 0000000..cc02260 --- /dev/null +++ b/docs/le-app-database-migration/phase2b-stage2-verification/README.md @@ -0,0 +1,37 @@ +# Phase 2B Stage 2 — installed-state verification and daemon reload + +Status: prepared for review only, 2026-07-12. Do not run until these uncommitted files have been reviewed and committed. Stage 2 does not authorize runtime activation. + +## Scope + +`tests/40-verify-stage2.sh` is the only Stage 2 operator entry point. It runs as root and: + +1. requires a clean committed worktree and verifies this Stage 2 manifest, the unchanged Stage 1 artifact manifest, and the Stage 1 inventory checksums; +2. proves that the Stage 1 artifact directory is unchanged since the commit recorded in the installed checksummed state; +3. strictly resolves `/var/lib/le-app-codex-phase2b/stage1/current`, requires a `COMPLETE` state with exactly 16 files and no pending records, and binds its manifest digest to the recorded source commit; +4. verifies all 16 destinations against the committed sources and state records by SHA-256, byte size, UID, GID, mode, regular-file type, and single-link count; +5. confirms the runtime is inert, then runs the sole Stage 2 mutation: `systemctl daemon-reload`; +6. validates the installed system units and drop-ins with `systemd-analyze verify`, `systemctl cat`, and exact or membership-checked `systemctl show` properties; parses the installed Docker unit with an explicit user-unit search path without starting a user manager; +7. runs root-capable `nft --check -f /etc/nftables.d/50-le-app-codex.nft` and never loads the policy; +8. explicitly rechecks the namespace helper and rootless launcher hashes; and +9. repeats every inert-state assertion after the reload and static validation. + +The inert-state gates require the approval marker, namespace, namespace path, project veth, rootless Docker socket, lingering, UID-1200 processes, and both Phase 2B nftables tables to be absent. They require `le-app-codex-netns.service` and `user@1200.service` to be inactive and the namespace service to remain static rather than enabled. + +## Explicit exclusions + +Stage 2 does not create the approval marker; start, restart, or enable any unit; use `systemctl --user`; enable lingering; invoke the namespace helper; create a namespace or veth; load nftables; start Docker or Codex; authenticate Codex; copy a repository, migration source, data, or secrets; contact production services; or modify production. + +The script makes no cleanup attempt after a failure. It preserves evidence and exits nonzero. See `ROLLBACK.md` for the Stage 2 no-op rollback and the separately approval-gated Stage 1 removal path. + +## Review and future execution + +Before committing, run only the non-operational `tests/41-stage2-static-tests.sh`. It checks syntax, manifests, immutable Stage 1 bytes, prohibited command literals, exact 16-file specifications, and the existing temporary-filesystem Stage 1 state tests. It never runs the Stage 2 operator script. + +After review, commit the Stage 2 package. From a clean checkout of that commit, the future operator invocation is: + +```console +sudo /bin/bash docs/le-app-database-migration/phase2b-stage2-verification/tests/40-verify-stage2.sh +``` + +Require the final summary to report `daemon_reload=1`, `installed_files=16`, and `failures=0`. Any other result is a hard stop; do not continue to namespace or runtime activation. diff --git a/docs/le-app-database-migration/phase2b-stage2-verification/ROLLBACK.md b/docs/le-app-database-migration/phase2b-stage2-verification/ROLLBACK.md new file mode 100644 index 0000000..5b37205 --- /dev/null +++ b/docs/le-app-database-migration/phase2b-stage2-verification/ROLLBACK.md @@ -0,0 +1,9 @@ +# Phase 2B Stage 2 rollback and failure handling + +Stage 2 installs and removes no files. Its only mutation is `systemctl daemon-reload`, which refreshes the system manager's view of the already-installed Stage 1 unit files and drop-ins. A successful or failed Stage 2 run does not create a runtime resource that needs teardown. + +If Stage 2 fails before `daemon-reload`, it has changed nothing. Preserve the output, correct only the reviewed repository or installed-state discrepancy, and rerun the entire verifier from a clean committed worktree. + +If Stage 2 fails after `daemon-reload`, do not start, enable, or clean up any service. Preserve the output and require the postconditions to remain inert: no approval marker; inactive and non-enabled `le-app-codex-netns.service`; inactive `user@1200.service`; no linger, UID-1200 processes, Docker socket, namespace, project veth, or Phase 2B nftables tables. Repair only through a separately reviewed change, then rerun the complete Stage 2 verifier. Another `daemon-reload` after repaired unit bytes is sufficient to refresh the manager; no service action is authorized. + +Full Stage 1 removal is outside Stage 2 and requires separate approval. If approved while every inert-state gate still passes, use only the unchanged Stage 1 `tests/31-rollback-stage1.sh` from the clean artifact worktree whose manifest digest matches the installed state. After that rollback succeeds, run `systemctl daemon-reload` to discard the removed unit definitions, then verify the Stage 1 files and state are absent and all runtime-negative assertions still pass. Never use Stage 1 rollback after an approval marker, namespace, firewall table, user manager, Docker daemon, or Codex process has been activated. diff --git a/docs/le-app-database-migration/phase2b-stage2-verification/tests/40-verify-stage2.sh b/docs/le-app-database-migration/phase2b-stage2-verification/tests/40-verify-stage2.sh new file mode 100644 index 0000000..304d42d --- /dev/null +++ b/docs/le-app-database-migration/phase2b-stage2-verification/tests/40-verify-stage2.sh @@ -0,0 +1,379 @@ +#!/usr/bin/env bash + +set -o pipefail +export LC_ALL=C +export PATH=/usr/bin:/bin +export SYSTEMD_COLORS=0 +export SYSTEMD_LOG_COLOR=0 +export SYSTEMD_PAGER=cat + +failures=0 +verified_files=0 +daemon_reload=0 +stage2_root=$(CDPATH= cd -- "$(dirname -- "$0")/.." 2>/dev/null && pwd) +migration_root=$(CDPATH= cd -- "$stage2_root/.." 2>/dev/null && pwd) +stage1_root=$migration_root/phase2b-contained-runtime +repo_root=$(CDPATH= cd -- "$migration_root/../.." 2>/dev/null && pwd) +payload_root=$stage1_root/payload +state_parent=/var/lib/le-app-codex-phase2b +state_transaction=$state_parent/stage1 +installed_policy=/etc/nftables.d/50-le-app-codex.nft +approval_marker=/etc/le-app-codex-runtime/OPERATOR-INPUTS-APPROVED +context_hash=1b645e55bd6af77d420732b143423e7a27b12fbb5a71497e5d89c52481b6a535 +context_source=$payload_root/srv/le-app-codex/home/.docker/contexts/meta/$context_hash/meta.json +context_destination=/srv/le-app-codex/home/.docker/contexts/meta/$context_hash/meta.json + +pass() { printf 'PASS: %s\n' "$1"; } +fail() { printf 'FAIL: %s\n' "$1" >&2; failures=$((failures + 1)); } + +verify_committed_manifest() { + local package_root=$1 label=$2 package_relative manifest line expected path actual committed rc tracked relative + local record_count=0 tree_count=0 + declare -A listed=() + package_relative=${package_root#"$repo_root"/} + manifest=$package_root/MANIFEST.sha256 + if [ -f "$manifest" ] && [ ! -L "$manifest" ]; then pass "$label manifest is a regular file"; else fail "$label manifest missing or unsafe"; return; fi + tracked=$(/usr/bin/git -c safe.directory="$repo_root" -C "$repo_root" ls-files --error-unmatch -- "$package_relative/MANIFEST.sha256" 2>/dev/null) + if [ "$tracked" = "$package_relative/MANIFEST.sha256" ]; then pass "$label manifest is tracked"; else fail "$label manifest is not tracked"; fi + while IFS= read -r line || [ -n "$line" ]; do + if [[ "$line" =~ ^([0-9a-f]{64})\ \ (.+)$ ]]; then + expected=${BASH_REMATCH[1]} + path=${BASH_REMATCH[2]} + else + fail "$label malformed manifest record" + continue + fi + if [[ "$path" = /* ]] || [[ "$path" = *'|'* ]] || [[ "$path" = *'/../'* ]] || [[ "$path" = ../* ]] || [[ "$path" = */.. ]] || [ "$path" = MANIFEST.sha256 ] || [ -n "${listed[$path]:-}" ]; then + fail "$label unsafe or duplicate manifest path $path" + continue + fi + listed["$path"]=1 + record_count=$((record_count + 1)) + actual=$(/usr/bin/sha256sum "$package_root/$path" 2>/dev/null | /usr/bin/cut -d' ' -f1) + [ "$actual" = "$expected" ] && pass "$label working hash $path" || fail "$label working hash mismatch $path" + relative=$package_relative/$path + tracked=$(/usr/bin/git -c safe.directory="$repo_root" -C "$repo_root" ls-files --error-unmatch -- "$relative" 2>/dev/null) + if [ "$tracked" = "$relative" ]; then + committed=$(/usr/bin/git -c safe.directory="$repo_root" -C "$repo_root" show "HEAD:$relative" 2>/dev/null | /usr/bin/sha256sum | /usr/bin/cut -d' ' -f1) + rc=$? + if [ "$rc" -eq 0 ] && [ "$committed" = "$expected" ]; then pass "$label committed hash $path"; else fail "$label committed hash mismatch $path"; fi + else + fail "$label untracked manifest path $path" + fi + done < "$manifest" + while IFS= read -r tracked; do + relative=${tracked#"$package_relative"/} + [ "$relative" = MANIFEST.sha256 ] && continue + tree_count=$((tree_count + 1)) + [ -n "${listed[$relative]:-}" ] || fail "$label manifest omits tracked path $relative" + done < <(/usr/bin/git -c safe.directory="$repo_root" -C "$repo_root" ls-tree -r --name-only HEAD -- "$package_relative" 2>/dev/null) + if [ "$record_count" -gt 0 ] && [ "$record_count" -eq "$tree_count" ]; then pass "$label manifest covers $record_count committed files"; else fail "$label manifest/tree count mismatch records=$record_count tree=$tree_count"; fi +} + +verify_state_tree() { + local entry name generation generation_name child child_name child_count checksum_record expected_checksum checksum_lines + local current_count=0 generations_count=0 generation_count=0 + if [ -d "$state_parent" ] && [ ! -L "$state_parent" ] && [ "$(/usr/bin/stat -c '%u:%g:%a' "$state_parent" 2>/dev/null)" = 0:0:700 ]; then pass 'Stage 1 state parent metadata'; else fail 'Stage 1 state parent metadata'; return; fi + while IFS= read -r -d '' entry; do + name=${entry##*/} + if [ "$name" = stage1 ]; then + [ -d "$entry" ] && [ ! -L "$entry" ] || fail 'Stage 1 state transaction is not a real directory' + else + fail "unexpected state-parent entry $entry" + fi + done < <(/usr/bin/find "$state_parent" -mindepth 1 -maxdepth 1 -print0 2>/dev/null) + while IFS= read -r -d '' entry; do + name=${entry##*/} + case "$name" in + current) stage1_state_verify_file "$entry" 0 0 0600 && current_count=$((current_count + 1)) || fail 'unsafe Stage 1 current pointer' ;; + generations) stage1_state_path_is_real_directory "$entry" && [ "$(/usr/bin/stat -c '%u:%g:%a' "$entry" 2>/dev/null)" = 0:0:700 ] && generations_count=$((generations_count + 1)) || fail 'unsafe Stage 1 generations directory' ;; + *) fail "unexpected Stage 1 state entry $entry" ;; + esac + done < <(/usr/bin/find "$state_transaction" -mindepth 1 -maxdepth 1 -print0 2>/dev/null) + [ "$current_count" -eq 1 ] || fail 'Stage 1 state requires exactly one current pointer' + [ "$generations_count" -eq 1 ] || fail 'Stage 1 state requires exactly one generations directory' + while IFS= read -r -d '' generation; do + generation_name=${generation##*/} + generation_count=$((generation_count + 1)) + [[ "$generation_name" =~ ^gen-[0-9]{8}T[0-9]{6}Z-[0-9]+-[0-9]+$ ]] || fail "invalid Stage 1 generation name $generation_name" + stage1_state_path_is_real_directory "$generation" && [ "$(/usr/bin/stat -c '%u:%g:%a' "$generation" 2>/dev/null)" = 0:0:700 ] || fail "unsafe Stage 1 generation $generation" + child_count=0 + while IFS= read -r -d '' child; do + child_name=${child##*/} + if [ "$child_name" != manifest ] && [ "$child_name" != manifest.sha256 ]; then fail "unexpected generation content $child"; fi + stage1_state_verify_file "$child" 0 0 0600 || fail "unsafe generation file $child" + child_count=$((child_count + 1)) + done < <(/usr/bin/find "$generation" -mindepth 1 -maxdepth 1 -print0 2>/dev/null) + [ "$child_count" -eq 2 ] || fail "generation must contain exactly two files $generation" + checksum_lines=$(/usr/bin/awk 'END { print NR+0 }' "$generation/manifest.sha256" 2>/dev/null) + checksum_record=$(<"$generation/manifest.sha256") + expected_checksum="$(/usr/bin/sha256sum "$generation/manifest" 2>/dev/null | /usr/bin/cut -d' ' -f1) manifest" + if [ "$checksum_lines" -eq 1 ] && [ "$checksum_record" = "$expected_checksum" ] && (cd "$generation" && /usr/bin/sha256sum -c manifest.sha256 >/dev/null 2>&1); then pass "checksummed state generation $generation_name"; else fail "generation checksum mismatch $generation"; fi + done < <(/usr/bin/find "$state_transaction/generations" -mindepth 1 -maxdepth 1 -print0 2>/dev/null) + [ "$generation_count" -gt 0 ] || fail 'Stage 1 state has no generations' +} + +verify_directory_record() { + local path=$1 spec expected_uid expected_gid expected_mode expected_kind actual + spec=${stage1_allowed_directory_specs[$path]} + IFS='|' read -r expected_uid expected_gid expected_mode expected_kind <<< "$spec" + actual=$(/usr/bin/stat -c '%u:%g:%a:%F' "$path" 2>/dev/null) + if [ ! -L "$path" ] && [ "$actual" = "$expected_uid:$expected_gid:${expected_mode#0}:directory" ]; then pass "recorded directory $path"; else fail "recorded directory mismatch $path"; fi +} + +verify_installed_file() { + local path=$1 label=${2:-$1} spec expected_hash expected_size expected_uid expected_gid expected_mode actual_hash actual_size actual_stat links + spec=${stage1_allowed_file_specs[$path]:-} + if [ -z "$spec" ]; then fail "$label has no committed Stage 1 specification"; return; fi + IFS='|' read -r expected_hash expected_size expected_uid expected_gid expected_mode <<< "$spec" + actual_hash=$(/usr/bin/sha256sum "$path" 2>/dev/null | /usr/bin/cut -d' ' -f1) + actual_size=$(/usr/bin/stat -c '%s' "$path" 2>/dev/null) + actual_stat=$(/usr/bin/stat -c '%u:%g:%a:%F' "$path" 2>/dev/null) + links=$(/usr/bin/stat -c '%h' "$path" 2>/dev/null) + if [ ! -L "$path" ] && [ "$actual_hash" = "$expected_hash" ] && [ "$actual_size" = "$expected_size" ] && [ "$actual_stat" = "$expected_uid:$expected_gid:${expected_mode#0}:regular file" ] && [ "$links" = 1 ]; then pass "$label hash/size/metadata"; else fail "$label hash, size, metadata, type, or link-count mismatch"; fi +} + +verify_inert_state() { + local phase=$1 active sub enabled enabled_rc processes process_rc namespaces namespace_rc links link_rc tables table_rc + if [ ! -e "$approval_marker" ] && [ ! -L "$approval_marker" ]; then pass "$phase approval marker absent"; else fail "$phase approval marker present"; fi + active=$(/usr/bin/systemctl show le-app-codex-netns.service --property=ActiveState --value 2>/dev/null) + sub=$(/usr/bin/systemctl show le-app-codex-netns.service --property=SubState --value 2>/dev/null) + if [ "$active" = inactive ] && [ "$sub" = dead ]; then pass "$phase namespace service inactive"; else fail "$phase namespace service state ${active:-unknown}/${sub:-unknown}"; fi + enabled=$(/usr/bin/systemctl is-enabled le-app-codex-netns.service 2>/dev/null) + enabled_rc=$? + if { [ "$enabled_rc" -eq 0 ] || [ "$enabled_rc" -eq 1 ]; } && [ "$enabled" = static ]; then pass "$phase namespace service static, not enabled"; else fail "$phase namespace service enablement ${enabled:-unknown} rc=$enabled_rc"; fi + active=$(/usr/bin/systemctl show user@1200.service --property=ActiveState --value 2>/dev/null) + sub=$(/usr/bin/systemctl show user@1200.service --property=SubState --value 2>/dev/null) + if [ "$active" = inactive ] && [ "$sub" = dead ]; then pass "$phase user@1200.service inactive"; else fail "$phase user@1200.service state ${active:-unknown}/${sub:-unknown}"; fi + if [ ! -e /var/lib/systemd/linger/le_app_codex ] && [ ! -L /var/lib/systemd/linger/le_app_codex ] && [ ! -e /var/lib/systemd/linger/1200 ] && [ ! -L /var/lib/systemd/linger/1200 ]; then pass "$phase lingering absent"; else fail "$phase lingering present"; fi + processes=$(/usr/bin/pgrep -u 1200 2>/dev/null) + process_rc=$? + if [ "$process_rc" -eq 1 ] && [ -z "$processes" ]; then pass "$phase UID 1200 has no processes"; elif [ "$process_rc" -eq 0 ]; then fail "$phase UID 1200 processes found: $processes"; else fail "$phase UID 1200 process query failed rc=$process_rc"; fi + if [ ! -e /run/user/1200/docker.sock ] && [ ! -L /run/user/1200/docker.sock ]; then pass "$phase rootless Docker socket absent"; else fail "$phase rootless Docker socket present"; fi + namespaces=$(/usr/bin/ip netns list 2>/dev/null) + namespace_rc=$? + if [ "$namespace_rc" -ne 0 ]; then fail "$phase namespace query failed rc=$namespace_rc"; elif printf '%s\n' "$namespaces" | /usr/bin/awk '$1 == "le-app-codex" { found=1 } END { exit !found }'; then fail "$phase le-app-codex namespace present"; elif [ -e /run/netns/le-app-codex ] || [ -L /run/netns/le-app-codex ]; then fail "$phase namespace path present"; else pass "$phase namespace absent"; fi + links=$(/usr/bin/ip -o link show 2>/dev/null) + link_rc=$? + if [ "$link_rc" -ne 0 ]; then fail "$phase link query failed rc=$link_rc"; elif printf '%s\n' "$links" | /usr/bin/awk -F': ' '$2 ~ /^lecodex-host(@|:|$)/ { found=1 } END { exit !found }'; then fail "$phase project veth present"; else pass "$phase project veth absent"; fi + tables=$(/usr/bin/nft list tables 2>/dev/null) + table_rc=$? + if [ "$table_rc" -ne 0 ]; then + fail "$phase nftables query failed rc=$table_rc" + else + if printf '%s\n' "$tables" | /usr/bin/awk '$1 == "table" && $2 == "inet" && $3 == "le_app_codex" { found=1 } END { exit !found }'; then fail "$phase inet Phase 2B table present"; else pass "$phase inet Phase 2B table absent"; fi + if printf '%s\n' "$tables" | /usr/bin/awk '$1 == "table" && $2 == "ip" && $3 == "le_app_codex_nat" { found=1 } END { exit !found }'; then fail "$phase NAT Phase 2B table present"; else pass "$phase NAT Phase 2B table absent"; fi + fi +} + +verify_analyze() { + local label=$1 output rc + shift + output=$("$@" 2>&1) + rc=$? + [ -z "$output" ] || printf '%s\n' "$output" + if [ "$rc" -eq 0 ]; then pass "$label"; else fail "$label rc=$rc"; fi +} + +verify_systemctl_cat() { + local unit=$1 output rc expected + shift + output=$(/usr/bin/systemctl cat --no-pager "$unit" 2>&1) + rc=$? + printf '%s\n' "$output" + if [ "$rc" -eq 0 ]; then pass "systemctl cat $unit"; else fail "systemctl cat $unit rc=$rc"; fi + for expected in "$@"; do + if printf '%s\n' "$output" | /usr/bin/grep -Fqx "# $expected"; then pass "$unit includes $expected"; else fail "$unit omits $expected"; fi + done +} + +capture_systemctl_show() { + local unit=$1 + shift + systemctl_show_output=$(/usr/bin/systemctl show --no-pager "$unit" "$@" 2>&1) + systemctl_show_rc=$? + printf '%s\n' "$systemctl_show_output" + if [ "$systemctl_show_rc" -eq 0 ]; then pass "systemctl show $unit"; else fail "systemctl show $unit rc=$systemctl_show_rc"; fi +} + +expect_show_exact() { + local label=$1 output=$2 property=$3 expected=$4 line count=0 actual= + while IFS= read -r line; do + if [[ "$line" = "$property="* ]]; then actual=${line#*=}; count=$((count + 1)); fi + done <<< "$output" + if [ "$count" -eq 1 ] && [ "$actual" = "$expected" ]; then pass "$label $property=$expected"; else fail "$label $property expected $expected, got ${actual:-missing}"; fi +} + +expect_show_word() { + local label=$1 output=$2 property=$3 expected=$4 line count=0 actual= word found=0 + while IFS= read -r line; do + if [[ "$line" = "$property="* ]]; then actual=${line#*=}; count=$((count + 1)); fi + done <<< "$output" + for word in $actual; do [ "$word" = "$expected" ] && found=1; done + if [ "$count" -eq 1 ] && [ "$found" -eq 1 ]; then pass "$label $property contains $expected"; else fail "$label $property omits $expected"; fi +} + +if [ "$(/usr/bin/id -u)" = 0 ]; then pass 'running as root'; else fail 'must run as root'; fi +if [ -n "$repo_root" ] && [ -e "$repo_root/.git" ] && [ ! -L "$repo_root/.git" ] && [ -d "$stage1_root" ] && [ -d "$stage2_root" ]; then pass 'repository and artifact roots resolved'; else fail 'repository or artifact root resolution'; fi +repo_head=$(/usr/bin/git -c safe.directory="$repo_root" -C "$repo_root" rev-parse HEAD 2>/dev/null) +repo_status=$(/usr/bin/git -c safe.directory="$repo_root" -C "$repo_root" status --porcelain --untracked-files=all 2>/dev/null) +repo_status_rc=$? +if [[ "$repo_head" =~ ^[0-9a-f]{40}$ ]]; then pass "committed repository HEAD $repo_head"; else fail 'repository HEAD unavailable'; fi +if [ "$repo_status_rc" -eq 0 ] && [ -z "$repo_status" ]; then pass 'repository worktree clean'; else fail 'repository worktree dirty or unreadable'; fi + +verify_committed_manifest "$stage2_root" 'Stage 2' +verify_committed_manifest "$stage1_root" 'Stage 1' +if (cd "$stage1_root/inventory" && /usr/bin/sha256sum -c SHA256SUMS); then pass 'Stage 1 inventory checksums'; else fail 'Stage 1 inventory checksums'; fi + +source "$stage1_root/tests/stage1-state-lib.sh" + +sources=( + "$payload_root/usr/local/libexec/dockerd-rootless-29.6.1.sh" + "$payload_root/usr/local/libexec/le-app-codex-netns" + "$payload_root/etc/le-app-codex-runtime/resolv.conf" + "$payload_root/etc/nftables.d/50-le-app-codex.nft" + "$payload_root/etc/systemd/system/le-app-codex-netns.service" + "$payload_root/etc/systemd/system/user-1200.slice.d/50-le-app-codex-resources.conf" + "$payload_root/etc/systemd/system/user@1200.service.d/50-le-app-codex-containment.conf" + "$payload_root/srv/le-app-codex/home/.config/systemd/user/docker.service" + "$payload_root/srv/le-app-codex/home/.config/docker/daemon.json" + "$payload_root/srv/le-app-codex/home/.docker/config.json" + "$context_source" + "$stage1_root/tests/00-read-only-preflight.sh" + "$stage1_root/tests/00-root-preinstall-validate.sh" + "$stage1_root/tests/10-inert-containment.sh" + "$stage1_root/tests/20-rootless-docker.sh" + "$stage1_root/tests/run-inert-without-user-manager.sh" +) +destinations=( + /usr/local/libexec/dockerd-rootless-29.6.1.sh + /usr/local/libexec/le-app-codex-netns + /etc/le-app-codex-runtime/resolv.conf + /etc/nftables.d/50-le-app-codex.nft + /etc/systemd/system/le-app-codex-netns.service + /etc/systemd/system/user-1200.slice.d/50-le-app-codex-resources.conf + /etc/systemd/system/user@1200.service.d/50-le-app-codex-containment.conf + /srv/le-app-codex/home/.config/systemd/user/docker.service + /srv/le-app-codex/home/.config/docker/daemon.json + /srv/le-app-codex/home/.docker/config.json + "$context_destination" + /srv/le-app-codex/phase2b-tests/00-read-only-preflight.sh + /srv/le-app-codex/phase2b-tests/00-root-preinstall-validate.sh + /srv/le-app-codex/phase2b-tests/10-inert-containment.sh + /srv/le-app-codex/phase2b-tests/20-rootless-docker.sh + /srv/le-app-codex/phase2b-tests/run-inert-without-user-manager.sh +) +uids=(0 0 0 0 0 0 0 1200 1200 1200 1200 1200 1200 1200 1200 1200) +gids=(0 0 0 0 0 0 0 1200 1200 1200 1200 1200 1200 1200 1200 1200) +modes=(0755 0755 0644 0644 0644 0644 0644 0640 0640 0640 0640 0750 0750 0750 0750 0750) +allowed_directories=( + /srv/le-app-codex/home/.docker/contexts/meta/$context_hash /srv/le-app-codex/home/.docker/contexts/meta /srv/le-app-codex/home/.docker/contexts /srv/le-app-codex/home/.docker + /srv/le-app-codex/home/.config/systemd/user /srv/le-app-codex/home/.config/systemd /srv/le-app-codex/home/.config/docker /srv/le-app-codex/home/.config /srv/le-app-codex/phase2b-tests + /etc/systemd/system/user@1200.service.d /etc/systemd/system/user-1200.slice.d /etc/le-app-codex-runtime /etc/nftables.d /usr/local/libexec "$state_parent" + /srv/le-app-codex/home /srv/le-app-codex /usr/local /etc/systemd/system /etc +) +directory_uids=(1200 1200 1200 1200 1200 1200 1200 1200 1200 0 0 0 0 0 0 1200 0 0 0 0) +directory_gids=(1200 1200 1200 1200 1200 1200 1200 1200 1200 0 0 0 0 0 0 1200 1200 0 0 0) +directory_modes=(0750 0750 0750 0750 0750 0750 0750 0750 0750 0755 0755 0755 0755 0755 0700 0700 0750 0755 0755 0755) +directory_kinds=(project project project project project project project project project project project project shared shared state project-boundary project-boundary shared-base shared-base shared-base) +declare -A stage1_allowed_file_specs=() stage1_allowed_directory_specs=() stage1_never_created_directories=() +for index in "${!destinations[@]}"; do + source_hash=$(/usr/bin/sha256sum "${sources[$index]}" 2>/dev/null | /usr/bin/cut -d' ' -f1) + source_size=$(/usr/bin/stat -c '%s' "${sources[$index]}" 2>/dev/null) + if [[ "$source_hash" =~ ^[0-9a-f]{64}$ ]] && [[ "$source_size" =~ ^[0-9]+$ ]]; then + stage1_allowed_file_specs["${destinations[$index]}"]="$source_hash|$source_size|${uids[$index]}|${gids[$index]}|${modes[$index]}" + else + fail "committed Stage 1 source unreadable ${sources[$index]}" + fi +done +for index in "${!allowed_directories[@]}"; do stage1_allowed_directory_specs["${allowed_directories[$index]}"]="${directory_uids[$index]}|${directory_gids[$index]}|${directory_modes[$index]}|${directory_kinds[$index]}"; done +for directory in /srv/le-app-codex/home /srv/le-app-codex /usr/local /etc/systemd/system /etc; do stage1_never_created_directories["$directory"]=1; done +if [ "${#sources[@]}" -eq 16 ] && [ "${#destinations[@]}" -eq 16 ] && [ "${#stage1_allowed_file_specs[@]}" -eq 16 ]; then pass 'exact 16-file Stage 1 specification'; else fail 'Stage 1 specification cardinality'; fi + +current_stage1_digest=$(/usr/bin/sha256sum "$stage1_root/MANIFEST.sha256" 2>/dev/null | /usr/bin/cut -d' ' -f1) +stage1_state_resolve_current "$state_transaction" 0 0 0600 || fail 'authoritative Stage 1 state generation' +if [ "$failures" -eq 0 ]; then verify_state_tree; fi +if [ "$failures" -eq 0 ]; then stage1_state_validate_records "$stage1_current_manifest" "$current_stage1_digest" "$(/usr/bin/hostname)" 16 || fail 'authoritative Stage 1 state records'; fi +completed_directory_records=$((${#stage1_validated_created_directories[@]} + ${#stage1_validated_existing_directories[@]})) +if [ "$failures" -eq 0 ] && [ "$stage1_validated_status" = COMPLETE ] && [ "${#stage1_validated_file_paths[@]}" -eq 16 ] && [ "${#stage1_validated_pending_paths[@]}" -eq 0 ] && [ "${#stage1_validated_pending_directories[@]}" -eq 0 ] && [ "$completed_directory_records" -eq "${#stage1_allowed_directory_specs[@]}" ] && [ "$completed_directory_records" -eq 20 ]; then pass 'Stage 1 state COMPLETE with exact file/directory records and nothing pending'; else fail 'Stage 1 state is not exact COMPLETE'; fi + +if [ "$failures" -eq 0 ]; then + if /usr/bin/git -c safe.directory="$repo_root" -C "$repo_root" cat-file -e "$stage1_validated_source_commit^{commit}" 2>/dev/null; then pass "recorded source commit exists $stage1_validated_source_commit"; else fail 'recorded source commit missing'; fi + if /usr/bin/git -c safe.directory="$repo_root" -C "$repo_root" merge-base --is-ancestor "$stage1_validated_source_commit" HEAD 2>/dev/null; then pass 'recorded source commit is an ancestor of HEAD'; else fail 'recorded source commit is not an ancestor of HEAD'; fi + stage1_relative=${stage1_root#"$repo_root"/} + historical_digest=$(/usr/bin/git -c safe.directory="$repo_root" -C "$repo_root" show "$stage1_validated_source_commit:$stage1_relative/MANIFEST.sha256" 2>/dev/null | /usr/bin/sha256sum | /usr/bin/cut -d' ' -f1) + historical_rc=$? + if [ "$historical_rc" -eq 0 ] && [ "$historical_digest" = "$stage1_validated_artifact_digest" ]; then pass 'recorded artifact digest matches recorded commit'; else fail 'recorded artifact digest/source commit mismatch'; fi + if /usr/bin/git -c safe.directory="$repo_root" -C "$repo_root" diff --quiet "$stage1_validated_source_commit" HEAD -- "$stage1_relative"; then pass 'Stage 1 artifact tree unchanged since installation commit'; else fail 'Stage 1 artifact tree changed since installation commit'; fi +fi + +if [ "$failures" -eq 0 ]; then + for destination in "${destinations[@]}"; do verify_installed_file "$destination"; done + for directory in "${stage1_validated_created_directories[@]}" "${stage1_validated_existing_directories[@]}"; do [ -z "$directory" ] || verify_directory_record "$directory"; done + [ "$failures" -ne 0 ] || verified_files=16 +fi + +if [ "$failures" -eq 0 ]; then verify_inert_state 'pre-reload'; fi +if [ "$failures" -ne 0 ]; then + printf 'Phase 2B Stage 2 daemon_reload=0 installed_files=%s failures=%s\n' "$verified_files" "$failures" + exit 1 +fi + +if /usr/bin/systemctl daemon-reload; then daemon_reload=1; pass 'systemctl daemon-reload'; else fail 'systemctl daemon-reload'; fi + +if [ "$failures" -eq 0 ]; then + verify_analyze 'installed system units and drop-ins' /usr/bin/systemd-analyze verify --recursive-errors=no le-app-codex-netns.service user@1200.service user-1200.slice + user_unit_path=/srv/le-app-codex/home/.config/systemd/user:/etc/systemd/user:/run/systemd/user:/usr/local/lib/systemd/user:/usr/lib/systemd/user + verify_analyze 'installed Docker unit with explicit user-unit path' /usr/bin/env SYSTEMD_UNIT_PATH="$user_unit_path" /usr/bin/systemd-analyze verify --recursive-errors=no docker.service + + verify_systemctl_cat le-app-codex-netns.service /etc/systemd/system/le-app-codex-netns.service + verify_systemctl_cat user@1200.service /etc/systemd/system/user@1200.service.d/50-le-app-codex-containment.conf + verify_systemctl_cat user-1200.slice /etc/systemd/system/user-1200.slice.d/50-le-app-codex-resources.conf + + capture_systemctl_show le-app-codex-netns.service --property=LoadState --property=ActiveState --property=SubState --property=UnitFileState --property=FragmentPath --property=Type --property=RemainAfterExit + netns_show=$systemctl_show_output + expect_show_exact le-app-codex-netns.service "$netns_show" LoadState loaded + expect_show_exact le-app-codex-netns.service "$netns_show" ActiveState inactive + expect_show_exact le-app-codex-netns.service "$netns_show" SubState dead + expect_show_exact le-app-codex-netns.service "$netns_show" UnitFileState static + expect_show_exact le-app-codex-netns.service "$netns_show" FragmentPath /etc/systemd/system/le-app-codex-netns.service + expect_show_exact le-app-codex-netns.service "$netns_show" Type oneshot + expect_show_exact le-app-codex-netns.service "$netns_show" RemainAfterExit yes + + capture_systemctl_show user@1200.service --property=LoadState --property=ActiveState --property=SubState --property=DropInPaths --property=Requires --property=After --property=NetworkNamespacePath --property=ProtectSystem --property=PrivateTmp --property=ProtectHome --property=ProtectProc --property=ProcSubset --property=DevicePolicy + user_show=$systemctl_show_output + expect_show_exact user@1200.service "$user_show" LoadState loaded + expect_show_exact user@1200.service "$user_show" ActiveState inactive + expect_show_exact user@1200.service "$user_show" SubState dead + expect_show_word user@1200.service "$user_show" DropInPaths /etc/systemd/system/user@1200.service.d/50-le-app-codex-containment.conf + expect_show_word user@1200.service "$user_show" Requires le-app-codex-netns.service + expect_show_word user@1200.service "$user_show" After le-app-codex-netns.service + expect_show_exact user@1200.service "$user_show" NetworkNamespacePath /run/netns/le-app-codex + expect_show_exact user@1200.service "$user_show" ProtectSystem strict + expect_show_exact user@1200.service "$user_show" PrivateTmp yes + expect_show_exact user@1200.service "$user_show" ProtectHome yes + expect_show_exact user@1200.service "$user_show" ProtectProc invisible + expect_show_exact user@1200.service "$user_show" ProcSubset all + expect_show_exact user@1200.service "$user_show" DevicePolicy closed + + capture_systemctl_show user-1200.slice --property=LoadState --property=ActiveState --property=DropInPaths --property=CPUQuotaPerSecUSec --property=MemoryHigh --property=MemoryMax --property=MemorySwapMax --property=TasksMax + slice_show=$systemctl_show_output + expect_show_exact user-1200.slice "$slice_show" LoadState loaded + expect_show_exact user-1200.slice "$slice_show" ActiveState inactive + expect_show_word user-1200.slice "$slice_show" DropInPaths /etc/systemd/system/user-1200.slice.d/50-le-app-codex-resources.conf + expect_show_exact user-1200.slice "$slice_show" CPUQuotaPerSecUSec 6s + expect_show_exact user-1200.slice "$slice_show" MemoryHigh 51539607552 + expect_show_exact user-1200.slice "$slice_show" MemoryMax 68719476736 + expect_show_exact user-1200.slice "$slice_show" MemorySwapMax 17179869184 + expect_show_exact user-1200.slice "$slice_show" TasksMax 4096 +fi + +if [ "$failures" -eq 0 ]; then + if /usr/bin/nft --check -f "$installed_policy"; then pass 'installed nftables policy syntax checked without load'; else fail 'installed nftables policy syntax check'; fi + verify_installed_file /usr/local/libexec/le-app-codex-netns 'installed namespace helper' + verify_installed_file /usr/local/libexec/dockerd-rootless-29.6.1.sh 'installed rootless launcher' +fi + +verify_inert_state 'post-reload' +printf 'Phase 2B Stage 2 daemon_reload=%s installed_files=%s failures=%s\n' "$daemon_reload" "$verified_files" "$failures" +[ "$daemon_reload" -eq 1 ] && [ "$verified_files" -eq 16 ] && [ "$failures" -eq 0 ] diff --git a/docs/le-app-database-migration/phase2b-stage2-verification/tests/41-stage2-static-tests.sh b/docs/le-app-database-migration/phase2b-stage2-verification/tests/41-stage2-static-tests.sh new file mode 100644 index 0000000..8ea429e --- /dev/null +++ b/docs/le-app-database-migration/phase2b-stage2-verification/tests/41-stage2-static-tests.sh @@ -0,0 +1,92 @@ +#!/usr/bin/env bash + +set -o pipefail +export LC_ALL=C + +failures=0 +stage2_root=$(CDPATH= cd -- "$(dirname -- "$0")/.." 2>/dev/null && pwd) +migration_root=$(CDPATH= cd -- "$stage2_root/.." 2>/dev/null && pwd) +stage1_root=$migration_root/phase2b-contained-runtime +operator=$stage2_root/tests/40-verify-stage2.sh + +pass() { printf 'PASS: %s\n' "$1"; } +fail() { printf 'FAIL: %s\n' "$1" >&2; failures=$((failures + 1)); } + +array_line_count() { + local name=$1 + /usr/bin/awk -v opening="$name=(" ' + $0 == opening { inside=1; next } + inside && $0 == ")" { exit } + inside { count++ } + END { print count+0 } + ' "$operator" +} + +emit_array_lines() { + local file=$1 name=$2 + /usr/bin/awk -v opening="$name=(" ' + $0 == opening { inside=1; next } + inside && $0 == ")" { exit } + inside { sub(/^[[:space:]]+/, ""); print } + ' "$file" +} + +verify_working_manifest_coverage() { + local line path file relative records=0 files=0 + declare -A listed=() + while IFS= read -r line || [ -n "$line" ]; do + if [[ "$line" =~ ^[0-9a-f]{64}\ \ (.+)$ ]]; then path=${BASH_REMATCH[1]}; else fail 'Stage 2 malformed manifest record'; continue; fi + if [ "$path" = MANIFEST.sha256 ] || [ -n "${listed[$path]:-}" ]; then fail "Stage 2 unsafe or duplicate manifest record $path"; else listed["$path"]=1; records=$((records + 1)); fi + done < "$stage2_root/MANIFEST.sha256" + while IFS= read -r -d '' file; do + relative=${file#"$stage2_root"/} + [ "$relative" = MANIFEST.sha256 ] && continue + files=$((files + 1)) + [ -n "${listed[$relative]:-}" ] || fail "Stage 2 manifest omits $relative" + done < <(/usr/bin/find "$stage2_root" -type f -print0) + if [ "$records" -eq "$files" ] && [ "$records" -gt 0 ]; then pass "Stage 2 manifest covers $records working files"; else fail "Stage 2 manifest coverage records=$records files=$files"; fi +} + +if /bin/bash -n "$operator" && /bin/bash -n "$stage2_root/tests/41-stage2-static-tests.sh"; then pass 'Stage 2 Bash syntax'; else fail 'Stage 2 Bash syntax'; fi +if (cd "$stage1_root" && /usr/bin/sha256sum -c MANIFEST.sha256); then pass 'unchanged Stage 1 artifact manifest'; else fail 'Stage 1 artifact manifest'; fi +if (cd "$stage1_root/inventory" && /usr/bin/sha256sum -c SHA256SUMS); then pass 'unchanged Stage 1 inventory'; else fail 'Stage 1 inventory'; fi +if (cd "$stage2_root" && /usr/bin/sha256sum -c MANIFEST.sha256); then pass 'Stage 2 artifact manifest'; else fail 'Stage 2 artifact manifest'; fi +verify_working_manifest_coverage + +repo_root=$(/usr/bin/git -C "$stage2_root" rev-parse --show-toplevel 2>/dev/null) +stage1_relative=${stage1_root#"$repo_root"/} +stage1_status=$(/usr/bin/git -C "$repo_root" status --porcelain --untracked-files=all -- "$stage1_relative" 2>/dev/null) +if [ -z "$stage1_status" ] && /usr/bin/git -C "$repo_root" diff --quiet HEAD -- "$stage1_relative"; then pass 'Stage 1 artifact directory byte-for-byte unchanged'; else fail 'Stage 1 artifact directory changed'; fi + +source_count=$(array_line_count sources) +destination_count=$(array_line_count destinations) +if [ "$source_count" -eq 16 ] && [ "$destination_count" -eq 16 ]; then pass 'operator has exact 16 source and destination records'; else fail "operator source/destination counts $source_count/$destination_count"; fi +stage1_rollback=$stage1_root/tests/31-rollback-stage1.sh +if /usr/bin/diff -u <(emit_array_lines "$stage1_rollback" sources) <(emit_array_lines "$operator" sources | /usr/bin/sed 's/\$stage1_root/\$artifact_root/g') >/dev/null && \ + /usr/bin/diff -u <(emit_array_lines "$stage1_rollback" destinations) <(emit_array_lines "$operator" destinations) >/dev/null; then pass 'operator 16-file map exactly matches Stage 1'; else fail 'operator 16-file map differs from Stage 1'; fi +if /usr/bin/grep -Fqx 'uids=(0 0 0 0 0 0 0 1200 1200 1200 1200 1200 1200 1200 1200 1200)' "$operator" && \ + /usr/bin/grep -Fqx 'gids=(0 0 0 0 0 0 0 1200 1200 1200 1200 1200 1200 1200 1200 1200)' "$operator" && \ + /usr/bin/grep -Fqx 'modes=(0755 0755 0644 0644 0644 0644 0644 0640 0640 0640 0640 0750 0750 0750 0750 0750)' "$operator"; then pass 'operator ownership and mode vectors'; else fail 'operator ownership or mode vectors'; fi + +reload_count=$(/usr/bin/grep -Fc '/usr/bin/systemctl daemon-reload' "$operator") +nft_check_count=$(/usr/bin/grep -Fc '/usr/bin/nft --check -f "$installed_policy"' "$operator") +dockerd_path_count=$(/usr/bin/grep -Fc '/usr/local/libexec/dockerd-rootless-29.6.1.sh' "$operator") +if [ "$reload_count" -eq 1 ]; then pass 'exactly one daemon-reload command'; else fail "daemon-reload command count $reload_count"; fi +if [ "$nft_check_count" -eq 1 ]; then pass 'exactly one nft --check command against installed policy'; else fail "nft --check command count $nft_check_count"; fi +if [ "$dockerd_path_count" -eq 3 ]; then pass 'rootless launcher appears only in source, destination, and hash recheck'; else fail "unexpected rootless launcher literal count $dockerd_path_count"; fi + +if /usr/bin/grep -Eq '/usr/bin/systemctl[[:space:]]+(start|stop|restart|try-restart|reload-or-restart|enable|disable|mask|unmask|preset|reenable)|/usr/bin/systemctl[[:space:]]+--user|/usr/bin/loginctl|/usr/bin/ip[[:space:]]+netns[[:space:]]+(add|delete|exec)|/usr/bin/ip[[:space:]]+link[[:space:]]+(add|delete|set)|/usr/bin/nft[[:space:]]+-f|/usr/bin/(cp|install|ln|mkdir|mv|rm)[[:space:]]' "$operator"; then + fail 'operator contains a prohibited mutation command literal' +else + pass 'operator contains no prohibited mutation command literal' +fi +if /usr/bin/grep -Eq 'OPERATOR-INPUTS-APPROVED.*(touch|install|printf)|le-app-codex-netns[[:space:]]+(up|down)|/usr/bin/docker[[:space:]]+(run|start)|/usr/bin/codex[[:space:]]+(login|auth)' "$operator"; then + fail 'operator contains a prohibited activation literal' +else + pass 'operator contains no prohibited activation literal' +fi + +if /bin/bash "$stage1_root/tests/32-stage1-state-tests.sh"; then pass 'existing non-mutating Stage 1 state tests'; else fail 'Stage 1 state tests'; fi + +printf 'Phase 2B Stage 2 static tests failures=%s\n' "$failures" +[ "$failures" -eq 0 ]