makearmy/le-app
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

add Phase 2B Stage 2 verification

Browse source
This commit is contained in:
makearmy 2026-07-12 23:54:05 -04:00
parent df6b53e639
commit 909f0a8200
5 changed files with 521 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

4
docs/le-app-database-migration/phase2b-stage2-verification/MANIFEST.sha256 Normal file
View file

@ -0,0 +1,4 @@
0a1cffbe98e0cdd7e70b02c92a6e14ca51b615f3888a405627abb1bd054bafef README.md
e41030a9850097ad91899ec14b2ce1a85f218fddf9b38aa317663450b29b6470 ROLLBACK.md
19b8de84c9e0e6025d66835a2eb834a5ee6c1eb6599e8650ce287f0cf069950e tests/40-verify-stage2.sh
10d4e45f0508e7dd6024b52a58e2848cb792f72a9e3cc070bf4c316df1ee4880 tests/41-stage2-static-tests.sh

37
docs/le-app-database-migration/phase2b-stage2-verification/README.md Normal file
View file

@ -0,0 +1,37 @@
# Phase 2B Stage 2 — installed-state verification and daemon reload
Status: prepared for review only, 2026-07-12. Do not run until these uncommitted files have been reviewed and committed. Stage 2 does not authorize runtime activation.
## Scope
`tests/40-verify-stage2.sh` is the only Stage 2 operator entry point. It runs as root and:
1. requires a clean committed worktree and verifies this Stage 2 manifest, the unchanged Stage 1 artifact manifest, and the Stage 1 inventory checksums;
2. proves that the Stage 1 artifact directory is unchanged since the commit recorded in the installed checksummed state;
3. strictly resolves `/var/lib/le-app-codex-phase2b/stage1/current`, requires a `COMPLETE` state with exactly 16 files and no pending records, and binds its manifest digest to the recorded source commit;
4. verifies all 16 destinations against the committed sources and state records by SHA-256, byte size, UID, GID, mode, regular-file type, and single-link count;
5. confirms the runtime is inert, then runs the sole Stage 2 mutation: `systemctl daemon-reload`;
6. validates the installed system units and drop-ins with `systemd-analyze verify`, `systemctl cat`, and exact or membership-checked `systemctl show` properties; parses the installed Docker unit with an explicit user-unit search path without starting a user manager;
7. runs root-capable `nft --check -f /etc/nftables.d/50-le-app-codex.nft` and never loads the policy;
8. explicitly rechecks the namespace helper and rootless launcher hashes; and
9. repeats every inert-state assertion after the reload and static validation.
The inert-state gates require the approval marker, namespace, namespace path, project veth, rootless Docker socket, lingering, UID-1200 processes, and both Phase 2B nftables tables to be absent. They require `le-app-codex-netns.service` and `user@1200.service` to be inactive and the namespace service to remain static rather than enabled.
## Explicit exclusions
Stage 2 does not create the approval marker; start, restart, or enable any unit; use `systemctl --user`; enable lingering; invoke the namespace helper; create a namespace or veth; load nftables; start Docker or Codex; authenticate Codex; copy a repository, migration source, data, or secrets; contact production services; or modify production.
The script makes no cleanup attempt after a failure. It preserves evidence and exits nonzero. See `ROLLBACK.md` for the Stage 2 no-op rollback and the separately approval-gated Stage 1 removal path.
## Review and future execution
Before committing, run only the non-operational `tests/41-stage2-static-tests.sh`. It checks syntax, manifests, immutable Stage 1 bytes, prohibited command literals, exact 16-file specifications, and the existing temporary-filesystem Stage 1 state tests. It never runs the Stage 2 operator script.
After review, commit the Stage 2 package. From a clean checkout of that commit, the future operator invocation is:
```console
sudo /bin/bash docs/le-app-database-migration/phase2b-stage2-verification/tests/40-verify-stage2.sh
```
Require the final summary to report `daemon_reload=1`, `installed_files=16`, and `failures=0`. Any other result is a hard stop; do not continue to namespace or runtime activation.

9
docs/le-app-database-migration/phase2b-stage2-verification/ROLLBACK.md Normal file
View file

@ -0,0 +1,9 @@
# Phase 2B Stage 2 rollback and failure handling
Stage 2 installs and removes no files. Its only mutation is `systemctl daemon-reload`, which refreshes the system manager's view of the already-installed Stage 1 unit files and drop-ins. A successful or failed Stage 2 run does not create a runtime resource that needs teardown.
If Stage 2 fails before `daemon-reload`, it has changed nothing. Preserve the output, correct only the reviewed repository or installed-state discrepancy, and rerun the entire verifier from a clean committed worktree.
If Stage 2 fails after `daemon-reload`, do not start, enable, or clean up any service. Preserve the output and require the postconditions to remain inert: no approval marker; inactive and non-enabled `le-app-codex-netns.service`; inactive `user@1200.service`; no linger, UID-1200 processes, Docker socket, namespace, project veth, or Phase 2B nftables tables. Repair only through a separately reviewed change, then rerun the complete Stage 2 verifier. Another `daemon-reload` after repaired unit bytes is sufficient to refresh the manager; no service action is authorized.
Full Stage 1 removal is outside Stage 2 and requires separate approval. If approved while every inert-state gate still passes, use only the unchanged Stage 1 `tests/31-rollback-stage1.sh` from the clean artifact worktree whose manifest digest matches the installed state. After that rollback succeeds, run `systemctl daemon-reload` to discard the removed unit definitions, then verify the Stage 1 files and state are absent and all runtime-negative assertions still pass. Never use Stage 1 rollback after an approval marker, namespace, firewall table, user manager, Docker daemon, or Codex process has been activated.

379
docs/le-app-database-migration/phase2b-stage2-verification/tests/40-verify-stage2.sh Normal file
View file

@ -0,0 +1,379 @@
#!/usr/bin/env bash
set -o pipefail
export LC_ALL=C
export PATH=/usr/bin:/bin
export SYSTEMD_COLORS=0
export SYSTEMD_LOG_COLOR=0
export SYSTEMD_PAGER=cat
failures=0
verified_files=0
daemon_reload=0
stage2_root=$(CDPATH= cd -- "$(dirname -- "$0")/.." 2>/dev/null && pwd)
migration_root=$(CDPATH= cd -- "$stage2_root/.." 2>/dev/null && pwd)
stage1_root=$migration_root/phase2b-contained-runtime
repo_root=$(CDPATH= cd -- "$migration_root/../.." 2>/dev/null && pwd)
payload_root=$stage1_root/payload
state_parent=/var/lib/le-app-codex-phase2b
state_transaction=$state_parent/stage1
installed_policy=/etc/nftables.d/50-le-app-codex.nft
approval_marker=/etc/le-app-codex-runtime/OPERATOR-INPUTS-APPROVED
context_hash=1b645e55bd6af77d420732b143423e7a27b12fbb5a71497e5d89c52481b6a535
context_source=$payload_root/srv/le-app-codex/home/.docker/contexts/meta/$context_hash/meta.json
context_destination=/srv/le-app-codex/home/.docker/contexts/meta/$context_hash/meta.json
pass() { printf 'PASS: %s\n' "$1"; }
fail() { printf 'FAIL: %s\n' "$1" >&2; failures=$((failures + 1)); }
verify_committed_manifest() {
local package_root=$1 label=$2 package_relative manifest line expected path actual committed rc tracked relative
local record_count=0 tree_count=0
declare -A listed=()
package_relative=${package_root#"$repo_root"/}
manifest=$package_root/MANIFEST.sha256
if [ -f "$manifest" ] && [ ! -L "$manifest" ]; then pass "$label manifest is a regular file"; else fail "$label manifest missing or unsafe"; return; fi
tracked=$(/usr/bin/git -c safe.directory="$repo_root" -C "$repo_root" ls-files --error-unmatch -- "$package_relative/MANIFEST.sha256" 2>/dev/null)
if [ "$tracked" = "$package_relative/MANIFEST.sha256" ]; then pass "$label manifest is tracked"; else fail "$label manifest is not tracked"; fi
while IFS= read -r line || [ -n "$line" ]; do
if [[ "$line" =~ ^([0-9a-f]{64})\ \ (.+)$ ]]; then
expected=${BASH_REMATCH[1]}
path=${BASH_REMATCH[2]}
else
fail "$label malformed manifest record"
continue
fi
if [[ "$path" = /* ]] || [[ "$path" = *'|'* ]] || [[ "$path" = *'/../'* ]] || [[ "$path" = ../* ]] || [[ "$path" = */.. ]] || [ "$path" = MANIFEST.sha256 ] || [ -n "${listed[$path]:-}" ]; then
fail "$label unsafe or duplicate manifest path $path"
continue
fi
listed["$path"]=1
record_count=$((record_count + 1))
actual=$(/usr/bin/sha256sum "$package_root/$path" 2>/dev/null | /usr/bin/cut -d' ' -f1)
[ "$actual" = "$expected" ] && pass "$label working hash $path" || fail "$label working hash mismatch $path"
relative=$package_relative/$path
tracked=$(/usr/bin/git -c safe.directory="$repo_root" -C "$repo_root" ls-files --error-unmatch -- "$relative" 2>/dev/null)
if [ "$tracked" = "$relative" ]; then
committed=$(/usr/bin/git -c safe.directory="$repo_root" -C "$repo_root" show "HEAD:$relative" 2>/dev/null | /usr/bin/sha256sum | /usr/bin/cut -d' ' -f1)
rc=$?
if [ "$rc" -eq 0 ] && [ "$committed" = "$expected" ]; then pass "$label committed hash $path"; else fail "$label committed hash mismatch $path"; fi
else
fail "$label untracked manifest path $path"
fi
done < "$manifest"
while IFS= read -r tracked; do
relative=${tracked#"$package_relative"/}
[ "$relative" = MANIFEST.sha256 ] && continue
tree_count=$((tree_count + 1))
[ -n "${listed[$relative]:-}" ] || fail "$label manifest omits tracked path $relative"
done < <(/usr/bin/git -c safe.directory="$repo_root" -C "$repo_root" ls-tree -r --name-only HEAD -- "$package_relative" 2>/dev/null)
if [ "$record_count" -gt 0 ] && [ "$record_count" -eq "$tree_count" ]; then pass "$label manifest covers $record_count committed files"; else fail "$label manifest/tree count mismatch records=$record_count tree=$tree_count"; fi
}
verify_state_tree() {
local entry name generation generation_name child child_name child_count checksum_record expected_checksum checksum_lines
local current_count=0 generations_count=0 generation_count=0
if [ -d "$state_parent" ] && [ ! -L "$state_parent" ] && [ "$(/usr/bin/stat -c '%u:%g:%a' "$state_parent" 2>/dev/null)" = 0:0:700 ]; then pass 'Stage 1 state parent metadata'; else fail 'Stage 1 state parent metadata'; return; fi
while IFS= read -r -d '' entry; do
name=${entry##*/}
if [ "$name" = stage1 ]; then
[ -d "$entry" ] && [ ! -L "$entry" ] || fail 'Stage 1 state transaction is not a real directory'
else
fail "unexpected state-parent entry $entry"
fi
done < <(/usr/bin/find "$state_parent" -mindepth 1 -maxdepth 1 -print0 2>/dev/null)
while IFS= read -r -d '' entry; do
name=${entry##*/}
case "$name" in
current) stage1_state_verify_file "$entry" 0 0 0600 && current_count=$((current_count + 1)) || fail 'unsafe Stage 1 current pointer' ;;
generations) stage1_state_path_is_real_directory "$entry" && [ "$(/usr/bin/stat -c '%u:%g:%a' "$entry" 2>/dev/null)" = 0:0:700 ] && generations_count=$((generations_count + 1)) || fail 'unsafe Stage 1 generations directory' ;;
*) fail "unexpected Stage 1 state entry $entry" ;;
esac
done < <(/usr/bin/find "$state_transaction" -mindepth 1 -maxdepth 1 -print0 2>/dev/null)
[ "$current_count" -eq 1 ] || fail 'Stage 1 state requires exactly one current pointer'
[ "$generations_count" -eq 1 ] || fail 'Stage 1 state requires exactly one generations directory'
while IFS= read -r -d '' generation; do
generation_name=${generation##*/}
generation_count=$((generation_count + 1))
[[ "$generation_name" =~ ^gen-[0-9]{8}T[0-9]{6}Z-[0-9]+-[0-9]+$ ]] || fail "invalid Stage 1 generation name $generation_name"
stage1_state_path_is_real_directory "$generation" && [ "$(/usr/bin/stat -c '%u:%g:%a' "$generation" 2>/dev/null)" = 0:0:700 ] || fail "unsafe Stage 1 generation $generation"
child_count=0
while IFS= read -r -d '' child; do
child_name=${child##*/}
if [ "$child_name" != manifest ] && [ "$child_name" != manifest.sha256 ]; then fail "unexpected generation content $child"; fi
stage1_state_verify_file "$child" 0 0 0600 || fail "unsafe generation file $child"
child_count=$((child_count + 1))
done < <(/usr/bin/find "$generation" -mindepth 1 -maxdepth 1 -print0 2>/dev/null)
[ "$child_count" -eq 2 ] || fail "generation must contain exactly two files $generation"
checksum_lines=$(/usr/bin/awk 'END { print NR+0 }' "$generation/manifest.sha256" 2>/dev/null)
checksum_record=$(<"$generation/manifest.sha256")
expected_checksum="$(/usr/bin/sha256sum "$generation/manifest" 2>/dev/null | /usr/bin/cut -d' ' -f1) manifest"
if [ "$checksum_lines" -eq 1 ] && [ "$checksum_record" = "$expected_checksum" ] && (cd "$generation" && /usr/bin/sha256sum -c manifest.sha256 >/dev/null 2>&1); then pass "checksummed state generation $generation_name"; else fail "generation checksum mismatch $generation"; fi
done < <(/usr/bin/find "$state_transaction/generations" -mindepth 1 -maxdepth 1 -print0 2>/dev/null)
[ "$generation_count" -gt 0 ] || fail 'Stage 1 state has no generations'
}
verify_directory_record() {
local path=$1 spec expected_uid expected_gid expected_mode expected_kind actual
spec=${stage1_allowed_directory_specs[$path]}
IFS='|' read -r expected_uid expected_gid expected_mode expected_kind <<< "$spec"
actual=$(/usr/bin/stat -c '%u:%g:%a:%F' "$path" 2>/dev/null)
if [ ! -L "$path" ] && [ "$actual" = "$expected_uid:$expected_gid:${expected_mode#0}:directory" ]; then pass "recorded directory $path"; else fail "recorded directory mismatch $path"; fi
}
verify_installed_file() {
local path=$1 label=${2:-$1} spec expected_hash expected_size expected_uid expected_gid expected_mode actual_hash actual_size actual_stat links
spec=${stage1_allowed_file_specs[$path]:-}
if [ -z "$spec" ]; then fail "$label has no committed Stage 1 specification"; return; fi
IFS='|' read -r expected_hash expected_size expected_uid expected_gid expected_mode <<< "$spec"
actual_hash=$(/usr/bin/sha256sum "$path" 2>/dev/null | /usr/bin/cut -d' ' -f1)
actual_size=$(/usr/bin/stat -c '%s' "$path" 2>/dev/null)
actual_stat=$(/usr/bin/stat -c '%u:%g:%a:%F' "$path" 2>/dev/null)
links=$(/usr/bin/stat -c '%h' "$path" 2>/dev/null)
if [ ! -L "$path" ] && [ "$actual_hash" = "$expected_hash" ] && [ "$actual_size" = "$expected_size" ] && [ "$actual_stat" = "$expected_uid:$expected_gid:${expected_mode#0}:regular file" ] && [ "$links" = 1 ]; then pass "$label hash/size/metadata"; else fail "$label hash, size, metadata, type, or link-count mismatch"; fi
}
verify_inert_state() {
local phase=$1 active sub enabled enabled_rc processes process_rc namespaces namespace_rc links link_rc tables table_rc
if [ ! -e "$approval_marker" ] && [ ! -L "$approval_marker" ]; then pass "$phase approval marker absent"; else fail "$phase approval marker present"; fi
active=$(/usr/bin/systemctl show le-app-codex-netns.service --property=ActiveState --value 2>/dev/null)
sub=$(/usr/bin/systemctl show le-app-codex-netns.service --property=SubState --value 2>/dev/null)
if [ "$active" = inactive ] && [ "$sub" = dead ]; then pass "$phase namespace service inactive"; else fail "$phase namespace service state ${active:-unknown}/${sub:-unknown}"; fi
enabled=$(/usr/bin/systemctl is-enabled le-app-codex-netns.service 2>/dev/null)
enabled_rc=$?
if { [ "$enabled_rc" -eq 0 ] || [ "$enabled_rc" -eq 1 ]; } && [ "$enabled" = static ]; then pass "$phase namespace service static, not enabled"; else fail "$phase namespace service enablement ${enabled:-unknown} rc=$enabled_rc"; fi
active=$(/usr/bin/systemctl show user@1200.service --property=ActiveState --value 2>/dev/null)
sub=$(/usr/bin/systemctl show user@1200.service --property=SubState --value 2>/dev/null)
if [ "$active" = inactive ] && [ "$sub" = dead ]; then pass "$phase user@1200.service inactive"; else fail "$phase user@1200.service state ${active:-unknown}/${sub:-unknown}"; fi
if [ ! -e /var/lib/systemd/linger/le_app_codex ] && [ ! -L /var/lib/systemd/linger/le_app_codex ] && [ ! -e /var/lib/systemd/linger/1200 ] && [ ! -L /var/lib/systemd/linger/1200 ]; then pass "$phase lingering absent"; else fail "$phase lingering present"; fi
processes=$(/usr/bin/pgrep -u 1200 2>/dev/null)
process_rc=$?
if [ "$process_rc" -eq 1 ] && [ -z "$processes" ]; then pass "$phase UID 1200 has no processes"; elif [ "$process_rc" -eq 0 ]; then fail "$phase UID 1200 processes found: $processes"; else fail "$phase UID 1200 process query failed rc=$process_rc"; fi
if [ ! -e /run/user/1200/docker.sock ] && [ ! -L /run/user/1200/docker.sock ]; then pass "$phase rootless Docker socket absent"; else fail "$phase rootless Docker socket present"; fi
namespaces=$(/usr/bin/ip netns list 2>/dev/null)
namespace_rc=$?
if [ "$namespace_rc" -ne 0 ]; then fail "$phase namespace query failed rc=$namespace_rc"; elif printf '%s\n' "$namespaces" | /usr/bin/awk '$1 == "le-app-codex" { found=1 } END { exit !found }'; then fail "$phase le-app-codex namespace present"; elif [ -e /run/netns/le-app-codex ] || [ -L /run/netns/le-app-codex ]; then fail "$phase namespace path present"; else pass "$phase namespace absent"; fi
links=$(/usr/bin/ip -o link show 2>/dev/null)
link_rc=$?
if [ "$link_rc" -ne 0 ]; then fail "$phase link query failed rc=$link_rc"; elif printf '%s\n' "$links" | /usr/bin/awk -F': ' '$2 ~ /^lecodex-host(@|:|$)/ { found=1 } END { exit !found }'; then fail "$phase project veth present"; else pass "$phase project veth absent"; fi
tables=$(/usr/bin/nft list tables 2>/dev/null)
table_rc=$?
if [ "$table_rc" -ne 0 ]; then
fail "$phase nftables query failed rc=$table_rc"
else
if printf '%s\n' "$tables" | /usr/bin/awk '$1 == "table" && $2 == "inet" && $3 == "le_app_codex" { found=1 } END { exit !found }'; then fail "$phase inet Phase 2B table present"; else pass "$phase inet Phase 2B table absent"; fi
if printf '%s\n' "$tables" | /usr/bin/awk '$1 == "table" && $2 == "ip" && $3 == "le_app_codex_nat" { found=1 } END { exit !found }'; then fail "$phase NAT Phase 2B table present"; else pass "$phase NAT Phase 2B table absent"; fi
fi
}
verify_analyze() {
local label=$1 output rc
shift
output=$("$@" 2>&1)
rc=$?
[ -z "$output" ] || printf '%s\n' "$output"
if [ "$rc" -eq 0 ]; then pass "$label"; else fail "$label rc=$rc"; fi
}
verify_systemctl_cat() {
local unit=$1 output rc expected
shift
output=$(/usr/bin/systemctl cat --no-pager "$unit" 2>&1)
rc=$?
printf '%s\n' "$output"
if [ "$rc" -eq 0 ]; then pass "systemctl cat $unit"; else fail "systemctl cat $unit rc=$rc"; fi
for expected in "$@"; do
if printf '%s\n' "$output" | /usr/bin/grep -Fqx "# $expected"; then pass "$unit includes $expected"; else fail "$unit omits $expected"; fi
done
}
capture_systemctl_show() {
local unit=$1
shift
systemctl_show_output=$(/usr/bin/systemctl show --no-pager "$unit" "$@" 2>&1)
systemctl_show_rc=$?
printf '%s\n' "$systemctl_show_output"
if [ "$systemctl_show_rc" -eq 0 ]; then pass "systemctl show $unit"; else fail "systemctl show $unit rc=$systemctl_show_rc"; fi
}
expect_show_exact() {
local label=$1 output=$2 property=$3 expected=$4 line count=0 actual=
while IFS= read -r line; do
if [[ "$line" = "$property="* ]]; then actual=${line#*=}; count=$((count + 1)); fi
done <<< "$output"
if [ "$count" -eq 1 ] && [ "$actual" = "$expected" ]; then pass "$label $property=$expected"; else fail "$label $property expected $expected, got ${actual:-missing}"; fi
}
expect_show_word() {
local label=$1 output=$2 property=$3 expected=$4 line count=0 actual= word found=0
while IFS= read -r line; do
if [[ "$line" = "$property="* ]]; then actual=${line#*=}; count=$((count + 1)); fi
done <<< "$output"
for word in $actual; do [ "$word" = "$expected" ] && found=1; done
if [ "$count" -eq 1 ] && [ "$found" -eq 1 ]; then pass "$label $property contains $expected"; else fail "$label $property omits $expected"; fi
}
if [ "$(/usr/bin/id -u)" = 0 ]; then pass 'running as root'; else fail 'must run as root'; fi
if [ -n "$repo_root" ] && [ -e "$repo_root/.git" ] && [ ! -L "$repo_root/.git" ] && [ -d "$stage1_root" ] && [ -d "$stage2_root" ]; then pass 'repository and artifact roots resolved'; else fail 'repository or artifact root resolution'; fi
repo_head=$(/usr/bin/git -c safe.directory="$repo_root" -C "$repo_root" rev-parse HEAD 2>/dev/null)
repo_status=$(/usr/bin/git -c safe.directory="$repo_root" -C "$repo_root" status --porcelain --untracked-files=all 2>/dev/null)
repo_status_rc=$?
if [[ "$repo_head" =~ ^[0-9a-f]{40}$ ]]; then pass "committed repository HEAD $repo_head"; else fail 'repository HEAD unavailable'; fi
if [ "$repo_status_rc" -eq 0 ] && [ -z "$repo_status" ]; then pass 'repository worktree clean'; else fail 'repository worktree dirty or unreadable'; fi
verify_committed_manifest "$stage2_root" 'Stage 2'
verify_committed_manifest "$stage1_root" 'Stage 1'
if (cd "$stage1_root/inventory" && /usr/bin/sha256sum -c SHA256SUMS); then pass 'Stage 1 inventory checksums'; else fail 'Stage 1 inventory checksums'; fi
source "$stage1_root/tests/stage1-state-lib.sh"
sources=(
"$payload_root/usr/local/libexec/dockerd-rootless-29.6.1.sh"
"$payload_root/usr/local/libexec/le-app-codex-netns"
"$payload_root/etc/le-app-codex-runtime/resolv.conf"
"$payload_root/etc/nftables.d/50-le-app-codex.nft"
"$payload_root/etc/systemd/system/le-app-codex-netns.service"
"$payload_root/etc/systemd/system/user-1200.slice.d/50-le-app-codex-resources.conf"
"$payload_root/etc/systemd/system/user@1200.service.d/50-le-app-codex-containment.conf"
"$payload_root/srv/le-app-codex/home/.config/systemd/user/docker.service"
"$payload_root/srv/le-app-codex/home/.config/docker/daemon.json"
"$payload_root/srv/le-app-codex/home/.docker/config.json"
"$context_source"
"$stage1_root/tests/00-read-only-preflight.sh"
"$stage1_root/tests/00-root-preinstall-validate.sh"
"$stage1_root/tests/10-inert-containment.sh"
"$stage1_root/tests/20-rootless-docker.sh"
"$stage1_root/tests/run-inert-without-user-manager.sh"
)
destinations=(
/usr/local/libexec/dockerd-rootless-29.6.1.sh
/usr/local/libexec/le-app-codex-netns
/etc/le-app-codex-runtime/resolv.conf
/etc/nftables.d/50-le-app-codex.nft
/etc/systemd/system/le-app-codex-netns.service
/etc/systemd/system/user-1200.slice.d/50-le-app-codex-resources.conf
/etc/systemd/system/user@1200.service.d/50-le-app-codex-containment.conf
/srv/le-app-codex/home/.config/systemd/user/docker.service
/srv/le-app-codex/home/.config/docker/daemon.json
/srv/le-app-codex/home/.docker/config.json
"$context_destination"
/srv/le-app-codex/phase2b-tests/00-read-only-preflight.sh
/srv/le-app-codex/phase2b-tests/00-root-preinstall-validate.sh
/srv/le-app-codex/phase2b-tests/10-inert-containment.sh
/srv/le-app-codex/phase2b-tests/20-rootless-docker.sh
/srv/le-app-codex/phase2b-tests/run-inert-without-user-manager.sh
)
uids=(0 0 0 0 0 0 0 1200 1200 1200 1200 1200 1200 1200 1200 1200)
gids=(0 0 0 0 0 0 0 1200 1200 1200 1200 1200 1200 1200 1200 1200)
modes=(0755 0755 0644 0644 0644 0644 0644 0640 0640 0640 0640 0750 0750 0750 0750 0750)
allowed_directories=(
/srv/le-app-codex/home/.docker/contexts/meta/$context_hash /srv/le-app-codex/home/.docker/contexts/meta /srv/le-app-codex/home/.docker/contexts /srv/le-app-codex/home/.docker
/srv/le-app-codex/home/.config/systemd/user /srv/le-app-codex/home/.config/systemd /srv/le-app-codex/home/.config/docker /srv/le-app-codex/home/.config /srv/le-app-codex/phase2b-tests
/etc/systemd/system/user@1200.service.d /etc/systemd/system/user-1200.slice.d /etc/le-app-codex-runtime /etc/nftables.d /usr/local/libexec "$state_parent"
/srv/le-app-codex/home /srv/le-app-codex /usr/local /etc/systemd/system /etc
)
directory_uids=(1200 1200 1200 1200 1200 1200 1200 1200 1200 0 0 0 0 0 0 1200 0 0 0 0)
directory_gids=(1200 1200 1200 1200 1200 1200 1200 1200 1200 0 0 0 0 0 0 1200 1200 0 0 0)
directory_modes=(0750 0750 0750 0750 0750 0750 0750 0750 0750 0755 0755 0755 0755 0755 0700 0700 0750 0755 0755 0755)
directory_kinds=(project project project project project project project project project project project project shared shared state project-boundary project-boundary shared-base shared-base shared-base)
declare -A stage1_allowed_file_specs=() stage1_allowed_directory_specs=() stage1_never_created_directories=()
for index in "${!destinations[@]}"; do
source_hash=$(/usr/bin/sha256sum "${sources[$index]}" 2>/dev/null | /usr/bin/cut -d' ' -f1)
source_size=$(/usr/bin/stat -c '%s' "${sources[$index]}" 2>/dev/null)
if [[ "$source_hash" =~ ^[0-9a-f]{64}$ ]] && [[ "$source_size" =~ ^[0-9]+$ ]]; then
stage1_allowed_file_specs["${destinations[$index]}"]="$source_hash|$source_size|${uids[$index]}|${gids[$index]}|${modes[$index]}"
else
fail "committed Stage 1 source unreadable ${sources[$index]}"
fi
done
for index in "${!allowed_directories[@]}"; do stage1_allowed_directory_specs["${allowed_directories[$index]}"]="${directory_uids[$index]}|${directory_gids[$index]}|${directory_modes[$index]}|${directory_kinds[$index]}"; done
for directory in /srv/le-app-codex/home /srv/le-app-codex /usr/local /etc/systemd/system /etc; do stage1_never_created_directories["$directory"]=1; done
if [ "${#sources[@]}" -eq 16 ] && [ "${#destinations[@]}" -eq 16 ] && [ "${#stage1_allowed_file_specs[@]}" -eq 16 ]; then pass 'exact 16-file Stage 1 specification'; else fail 'Stage 1 specification cardinality'; fi
current_stage1_digest=$(/usr/bin/sha256sum "$stage1_root/MANIFEST.sha256" 2>/dev/null | /usr/bin/cut -d' ' -f1)
stage1_state_resolve_current "$state_transaction" 0 0 0600 || fail 'authoritative Stage 1 state generation'
if [ "$failures" -eq 0 ]; then verify_state_tree; fi
if [ "$failures" -eq 0 ]; then stage1_state_validate_records "$stage1_current_manifest" "$current_stage1_digest" "$(/usr/bin/hostname)" 16 || fail 'authoritative Stage 1 state records'; fi
completed_directory_records=$((${#stage1_validated_created_directories[@]} + ${#stage1_validated_existing_directories[@]}))
if [ "$failures" -eq 0 ] && [ "$stage1_validated_status" = COMPLETE ] && [ "${#stage1_validated_file_paths[@]}" -eq 16 ] && [ "${#stage1_validated_pending_paths[@]}" -eq 0 ] && [ "${#stage1_validated_pending_directories[@]}" -eq 0 ] && [ "$completed_directory_records" -eq "${#stage1_allowed_directory_specs[@]}" ] && [ "$completed_directory_records" -eq 20 ]; then pass 'Stage 1 state COMPLETE with exact file/directory records and nothing pending'; else fail 'Stage 1 state is not exact COMPLETE'; fi
if [ "$failures" -eq 0 ]; then
if /usr/bin/git -c safe.directory="$repo_root" -C "$repo_root" cat-file -e "$stage1_validated_source_commit^{commit}" 2>/dev/null; then pass "recorded source commit exists $stage1_validated_source_commit"; else fail 'recorded source commit missing'; fi
if /usr/bin/git -c safe.directory="$repo_root" -C "$repo_root" merge-base --is-ancestor "$stage1_validated_source_commit" HEAD 2>/dev/null; then pass 'recorded source commit is an ancestor of HEAD'; else fail 'recorded source commit is not an ancestor of HEAD'; fi
stage1_relative=${stage1_root#"$repo_root"/}
historical_digest=$(/usr/bin/git -c safe.directory="$repo_root" -C "$repo_root" show "$stage1_validated_source_commit:$stage1_relative/MANIFEST.sha256" 2>/dev/null | /usr/bin/sha256sum | /usr/bin/cut -d' ' -f1)
historical_rc=$?
if [ "$historical_rc" -eq 0 ] && [ "$historical_digest" = "$stage1_validated_artifact_digest" ]; then pass 'recorded artifact digest matches recorded commit'; else fail 'recorded artifact digest/source commit mismatch'; fi
if /usr/bin/git -c safe.directory="$repo_root" -C "$repo_root" diff --quiet "$stage1_validated_source_commit" HEAD -- "$stage1_relative"; then pass 'Stage 1 artifact tree unchanged since installation commit'; else fail 'Stage 1 artifact tree changed since installation commit'; fi
fi
if [ "$failures" -eq 0 ]; then
for destination in "${destinations[@]}"; do verify_installed_file "$destination"; done
for directory in "${stage1_validated_created_directories[@]}" "${stage1_validated_existing_directories[@]}"; do [ -z "$directory" ] || verify_directory_record "$directory"; done
[ "$failures" -ne 0 ] || verified_files=16
fi
if [ "$failures" -eq 0 ]; then verify_inert_state 'pre-reload'; fi
if [ "$failures" -ne 0 ]; then
printf 'Phase 2B Stage 2 daemon_reload=0 installed_files=%s failures=%s\n' "$verified_files" "$failures"
exit 1
fi
if /usr/bin/systemctl daemon-reload; then daemon_reload=1; pass 'systemctl daemon-reload'; else fail 'systemctl daemon-reload'; fi
if [ "$failures" -eq 0 ]; then
verify_analyze 'installed system units and drop-ins' /usr/bin/systemd-analyze verify --recursive-errors=no le-app-codex-netns.service user@1200.service user-1200.slice
user_unit_path=/srv/le-app-codex/home/.config/systemd/user:/etc/systemd/user:/run/systemd/user:/usr/local/lib/systemd/user:/usr/lib/systemd/user
verify_analyze 'installed Docker unit with explicit user-unit path' /usr/bin/env SYSTEMD_UNIT_PATH="$user_unit_path" /usr/bin/systemd-analyze verify --recursive-errors=no docker.service
verify_systemctl_cat le-app-codex-netns.service /etc/systemd/system/le-app-codex-netns.service
verify_systemctl_cat user@1200.service /etc/systemd/system/user@1200.service.d/50-le-app-codex-containment.conf
verify_systemctl_cat user-1200.slice /etc/systemd/system/user-1200.slice.d/50-le-app-codex-resources.conf
capture_systemctl_show le-app-codex-netns.service --property=LoadState --property=ActiveState --property=SubState --property=UnitFileState --property=FragmentPath --property=Type --property=RemainAfterExit
netns_show=$systemctl_show_output
expect_show_exact le-app-codex-netns.service "$netns_show" LoadState loaded
expect_show_exact le-app-codex-netns.service "$netns_show" ActiveState inactive
expect_show_exact le-app-codex-netns.service "$netns_show" SubState dead
expect_show_exact le-app-codex-netns.service "$netns_show" UnitFileState static
expect_show_exact le-app-codex-netns.service "$netns_show" FragmentPath /etc/systemd/system/le-app-codex-netns.service
expect_show_exact le-app-codex-netns.service "$netns_show" Type oneshot
expect_show_exact le-app-codex-netns.service "$netns_show" RemainAfterExit yes
capture_systemctl_show user@1200.service --property=LoadState --property=ActiveState --property=SubState --property=DropInPaths --property=Requires --property=After --property=NetworkNamespacePath --property=ProtectSystem --property=PrivateTmp --property=ProtectHome --property=ProtectProc --property=ProcSubset --property=DevicePolicy
user_show=$systemctl_show_output
expect_show_exact user@1200.service "$user_show" LoadState loaded
expect_show_exact user@1200.service "$user_show" ActiveState inactive
expect_show_exact user@1200.service "$user_show" SubState dead
expect_show_word user@1200.service "$user_show" DropInPaths /etc/systemd/system/user@1200.service.d/50-le-app-codex-containment.conf
expect_show_word user@1200.service "$user_show" Requires le-app-codex-netns.service
expect_show_word user@1200.service "$user_show" After le-app-codex-netns.service
expect_show_exact user@1200.service "$user_show" NetworkNamespacePath /run/netns/le-app-codex
expect_show_exact user@1200.service "$user_show" ProtectSystem strict
expect_show_exact user@1200.service "$user_show" PrivateTmp yes
expect_show_exact user@1200.service "$user_show" ProtectHome yes
expect_show_exact user@1200.service "$user_show" ProtectProc invisible
expect_show_exact user@1200.service "$user_show" ProcSubset all
expect_show_exact user@1200.service "$user_show" DevicePolicy closed
capture_systemctl_show user-1200.slice --property=LoadState --property=ActiveState --property=DropInPaths --property=CPUQuotaPerSecUSec --property=MemoryHigh --property=MemoryMax --property=MemorySwapMax --property=TasksMax
slice_show=$systemctl_show_output
expect_show_exact user-1200.slice "$slice_show" LoadState loaded
expect_show_exact user-1200.slice "$slice_show" ActiveState inactive
expect_show_word user-1200.slice "$slice_show" DropInPaths /etc/systemd/system/user-1200.slice.d/50-le-app-codex-resources.conf
expect_show_exact user-1200.slice "$slice_show" CPUQuotaPerSecUSec 6s
expect_show_exact user-1200.slice "$slice_show" MemoryHigh 51539607552
expect_show_exact user-1200.slice "$slice_show" MemoryMax 68719476736
expect_show_exact user-1200.slice "$slice_show" MemorySwapMax 17179869184
expect_show_exact user-1200.slice "$slice_show" TasksMax 4096
fi
if [ "$failures" -eq 0 ]; then
if /usr/bin/nft --check -f "$installed_policy"; then pass 'installed nftables policy syntax checked without load'; else fail 'installed nftables policy syntax check'; fi
verify_installed_file /usr/local/libexec/le-app-codex-netns 'installed namespace helper'
verify_installed_file /usr/local/libexec/dockerd-rootless-29.6.1.sh 'installed rootless launcher'
fi
verify_inert_state 'post-reload'
printf 'Phase 2B Stage 2 daemon_reload=%s installed_files=%s failures=%s\n' "$daemon_reload" "$verified_files" "$failures"
[ "$daemon_reload" -eq 1 ] && [ "$verified_files" -eq 16 ] && [ "$failures" -eq 0 ]

92
docs/le-app-database-migration/phase2b-stage2-verification/tests/41-stage2-static-tests.sh Normal file
View file

@ -0,0 +1,92 @@
#!/usr/bin/env bash
set -o pipefail
export LC_ALL=C
failures=0
stage2_root=$(CDPATH= cd -- "$(dirname -- "$0")/.." 2>/dev/null && pwd)
migration_root=$(CDPATH= cd -- "$stage2_root/.." 2>/dev/null && pwd)
stage1_root=$migration_root/phase2b-contained-runtime
operator=$stage2_root/tests/40-verify-stage2.sh
pass() { printf 'PASS: %s\n' "$1"; }
fail() { printf 'FAIL: %s\n' "$1" >&2; failures=$((failures + 1)); }
array_line_count() {
local name=$1
/usr/bin/awk -v opening="$name=(" '
$0 == opening { inside=1; next }
inside && $0 == ")" { exit }
inside { count++ }
END { print count+0 }
' "$operator"
}
emit_array_lines() {
local file=$1 name=$2
/usr/bin/awk -v opening="$name=(" '
$0 == opening { inside=1; next }
inside && $0 == ")" { exit }
inside { sub(/^[[:space:]]+/, ""); print }
' "$file"
}
verify_working_manifest_coverage() {
local line path file relative records=0 files=0
declare -A listed=()
while IFS= read -r line || [ -n "$line" ]; do
if [[ "$line" =~ ^[0-9a-f]{64}\ \ (.+)$ ]]; then path=${BASH_REMATCH[1]}; else fail 'Stage 2 malformed manifest record'; continue; fi
if [ "$path" = MANIFEST.sha256 ] || [ -n "${listed[$path]:-}" ]; then fail "Stage 2 unsafe or duplicate manifest record $path"; else listed["$path"]=1; records=$((records + 1)); fi
done < "$stage2_root/MANIFEST.sha256"
while IFS= read -r -d '' file; do
relative=${file#"$stage2_root"/}
[ "$relative" = MANIFEST.sha256 ] && continue
files=$((files + 1))
[ -n "${listed[$relative]:-}" ] || fail "Stage 2 manifest omits $relative"
done < <(/usr/bin/find "$stage2_root" -type f -print0)
if [ "$records" -eq "$files" ] && [ "$records" -gt 0 ]; then pass "Stage 2 manifest covers $records working files"; else fail "Stage 2 manifest coverage records=$records files=$files"; fi
}
if /bin/bash -n "$operator" && /bin/bash -n "$stage2_root/tests/41-stage2-static-tests.sh"; then pass 'Stage 2 Bash syntax'; else fail 'Stage 2 Bash syntax'; fi
if (cd "$stage1_root" && /usr/bin/sha256sum -c MANIFEST.sha256); then pass 'unchanged Stage 1 artifact manifest'; else fail 'Stage 1 artifact manifest'; fi
if (cd "$stage1_root/inventory" && /usr/bin/sha256sum -c SHA256SUMS); then pass 'unchanged Stage 1 inventory'; else fail 'Stage 1 inventory'; fi
if (cd "$stage2_root" && /usr/bin/sha256sum -c MANIFEST.sha256); then pass 'Stage 2 artifact manifest'; else fail 'Stage 2 artifact manifest'; fi
verify_working_manifest_coverage
repo_root=$(/usr/bin/git -C "$stage2_root" rev-parse --show-toplevel 2>/dev/null)
stage1_relative=${stage1_root#"$repo_root"/}
stage1_status=$(/usr/bin/git -C "$repo_root" status --porcelain --untracked-files=all -- "$stage1_relative" 2>/dev/null)
if [ -z "$stage1_status" ] && /usr/bin/git -C "$repo_root" diff --quiet HEAD -- "$stage1_relative"; then pass 'Stage 1 artifact directory byte-for-byte unchanged'; else fail 'Stage 1 artifact directory changed'; fi
source_count=$(array_line_count sources)
destination_count=$(array_line_count destinations)
if [ "$source_count" -eq 16 ] && [ "$destination_count" -eq 16 ]; then pass 'operator has exact 16 source and destination records'; else fail "operator source/destination counts $source_count/$destination_count"; fi
stage1_rollback=$stage1_root/tests/31-rollback-stage1.sh
if /usr/bin/diff -u <(emit_array_lines "$stage1_rollback" sources) <(emit_array_lines "$operator" sources | /usr/bin/sed 's/\$stage1_root/\$artifact_root/g') >/dev/null && \
/usr/bin/diff -u <(emit_array_lines "$stage1_rollback" destinations) <(emit_array_lines "$operator" destinations) >/dev/null; then pass 'operator 16-file map exactly matches Stage 1'; else fail 'operator 16-file map differs from Stage 1'; fi
if /usr/bin/grep -Fqx 'uids=(0 0 0 0 0 0 0 1200 1200 1200 1200 1200 1200 1200 1200 1200)' "$operator" && \
/usr/bin/grep -Fqx 'gids=(0 0 0 0 0 0 0 1200 1200 1200 1200 1200 1200 1200 1200 1200)' "$operator" && \
/usr/bin/grep -Fqx 'modes=(0755 0755 0644 0644 0644 0644 0644 0640 0640 0640 0640 0750 0750 0750 0750 0750)' "$operator"; then pass 'operator ownership and mode vectors'; else fail 'operator ownership or mode vectors'; fi
reload_count=$(/usr/bin/grep -Fc '/usr/bin/systemctl daemon-reload' "$operator")
nft_check_count=$(/usr/bin/grep -Fc '/usr/bin/nft --check -f "$installed_policy"' "$operator")
dockerd_path_count=$(/usr/bin/grep -Fc '/usr/local/libexec/dockerd-rootless-29.6.1.sh' "$operator")
if [ "$reload_count" -eq 1 ]; then pass 'exactly one daemon-reload command'; else fail "daemon-reload command count $reload_count"; fi
if [ "$nft_check_count" -eq 1 ]; then pass 'exactly one nft --check command against installed policy'; else fail "nft --check command count $nft_check_count"; fi
if [ "$dockerd_path_count" -eq 3 ]; then pass 'rootless launcher appears only in source, destination, and hash recheck'; else fail "unexpected rootless launcher literal count $dockerd_path_count"; fi
if /usr/bin/grep -Eq '/usr/bin/systemctl[[:space:]]+(start|stop|restart|try-restart|reload-or-restart|enable|disable|mask|unmask|preset|reenable)|/usr/bin/systemctl[[:space:]]+--user|/usr/bin/loginctl|/usr/bin/ip[[:space:]]+netns[[:space:]]+(add|delete|exec)|/usr/bin/ip[[:space:]]+link[[:space:]]+(add|delete|set)|/usr/bin/nft[[:space:]]+-f|/usr/bin/(cp|install|ln|mkdir|mv|rm)[[:space:]]' "$operator"; then
fail 'operator contains a prohibited mutation command literal'
else
pass 'operator contains no prohibited mutation command literal'
fi
if /usr/bin/grep -Eq 'OPERATOR-INPUTS-APPROVED.*(touch|install|printf)|le-app-codex-netns[[:space:]]+(up|down)|/usr/bin/docker[[:space:]]+(run|start)|/usr/bin/codex[[:space:]]+(login|auth)' "$operator"; then
fail 'operator contains a prohibited activation literal'
else
pass 'operator contains no prohibited activation literal'
fi
if /bin/bash "$stage1_root/tests/32-stage1-state-tests.sh"; then pass 'existing non-mutating Stage 1 state tests'; else fail 'Stage 1 state tests'; fi
printf 'Phase 2B Stage 2 static tests failures=%s\n' "$failures"
[ "$failures" -eq 0 ]