# Milestone 0 host runtime and security plan Status: planning-only. Commands in this document are proposed for administrator review. None have been executed. ## Permissions-only access audit The audit inspected only existence, filesystem type, owner/mode, effective read/write/traverse permission, Docker context endpoints, and socket metadata. It did not read, print, copy, or inspect secret values, production databases, container metadata, or archive contents. | Boundary | Effective access for `sol6_vi` | Finding | |---|---|---| | `/var/www` | read/traverse, not write | Production tree names and readable files are exposed to the account | | `/var/www/lasereverything.net.db` and `app/` | read/traverse, not write | Production application and deployment configuration are readable | | production `.env.local` and BGBye `.env` | read, not write; mode `0644` | **Unsafe for a `sol6_vi`-owned rootless daemon**; values were not read | | `/var/lib/docker` | no read/write/traverse | Production Docker data is denied by host permissions | | `/var/lib/mysql` | no read/write/traverse | Production database storage is denied | | `/var/run/docker.sock` and `/run/docker.sock` | no read/write | Production daemon is denied; only the `default` context points to it | | `/srv/directus-archive` | no read/write/traverse | Root-only archive boundary works; timestamped child is inaccessible | | `/srv/codex-migration` | read/traverse, not write | Scrubbed migration source is available read-only | | `/srv/codex-secrets` | read/traverse at directory level, not write | Migration env is readable as intended; access must be narrowed for the new account to the one nonproduction file | | workspace | read/write | Current development tree is owned by `sol6_vi` | Conclusion: rootless Docker owned by `sol6_vi` would protect the production daemon but not production files already readable to that user. Any controller of that daemon could mount the readable production environment files. The production secret file modes also deserve independent remediation. ## Runtime account model comparison | Model | Advantages | Risks / cost | Decision | |---|---|---|---| | `sol6_vi` owns rootless Docker | Minimal provisioning; current workspace already writable | Can bind-mount world-readable production secrets and application files; mixes general Codex work with container authority | Rejected on this host | | dedicated `le_app_codex` | Daemon owner has an explicit filesystem allow-list; project storage and cleanup are attributable; production paths can be proven inaccessible | Requires account, ACL, subordinate IDs, separate Git credentials/session, storage, and user service | **Recommended** | Future autonomous migration work should run in a Codex session whose OS identity is `le_app_codex`. Do not grant `sol6_vi` generic access to the development daemon socket as a convenience; that weakens accountability. If orchestration must be initiated from another account, use a narrowly audited command wrapper or SSH into `le_app_codex`, not a shared Docker group and not TCP exposure of the daemon. ## Exact proposed identity and filesystem model Proposed fixed identity: - user and primary group: `le_app_codex`; - UID/GID: `1200` (verified unused at audit time; recheck immediately before creation); - shell/home: `/bin/bash`, `/srv/le-app-codex/home`; - no supplementary production groups, no `docker` group, no sudo/wheel membership; - subordinate UID and GID range: `165536:65536`, non-overlapping with the observed `sol6_ii:100000:65536` allocation; - dedicated project root: Btrfs subvolume `/srv/le-app-codex`, owner `root:le_app_codex`, mode `0750`; - sole checkout: `/srv/le-app-codex/lasereverything.net.db`; - authoritative scrubbed input: `/srv/le-app-codex/database-source-readonly`, root-owned and read-only to the development group; - nonproduction credentials only: `/srv/le-app-codex/nonproduction-secrets`; - daemon data root: `/srv/le-app-codex/container-runtime/docker`; - runtime socket: `/run/user/1200/docker.sock`; - Docker client/config: `/srv/le-app-codex/home/.docker` and `/srv/le-app-codex/home/.config/docker`; - rootless Docker context: `le-app-codex`; - default nonproduction Compose project: `le-app-testing`, enforced by wrapper preflight. Verified candidate availability at this planning checkpoint: - neither account nor group name `le_app_codex` exists; - UID `1200` is unused in `/etc/passwd` and GID `1200` is unused in `/etc/group`; - no passwd primary UID/GID or group GID collision exists at `1200`; - `/etc/subuid` and `/etc/subgid` contain one observed allocation, `100000–165535`, owned by the historical `sol6_ii` entry; - candidate range `165536–231071` does not overlap any entry in either subordinate-ID file; - `/srv/le-app-codex` does not exist. Therefore the current exact recommendation is UID/GID `1200` and subordinate range `165536:65536`, subject to rerunning the documented collision checks immediately before provisioning. If any check produces output or a non-free result, stop and select new IDs; do not force reuse. ## Canonical naming replacements | Superseded proposal | Canonical replacement | |---|---| | account/group `le_payload_dev` | `le_app_codex` | | nonproduction root `/srv/le-payload-dev` | `/srv/le-app-codex` | | checkout `/srv/le-payload-dev/lasereverything.net.db` | `/srv/le-app-codex/lasereverything.net.db` | | source directory `source` | `database-source-readonly` | | secrets directory `secrets` | `nonproduction-secrets` | | runtime directory `runtime` | `container-runtime` | | disposable-data directory `data` | `testing-data` | | artifact directory `artifacts` | `build-artifacts` | | Docker context `le-payload-dev` | `le-app-codex` | | Compose project `le_payload_dev` | `le-app-testing` | | branch `migration/payload` | `migration/le-app-database` | | documentation `docs/payload-migration` | `docs/le-app-database-migration` | `payload` remains valid only for the actual Payload CMS service, application directory, packages, configuration, and Payload-specific migrations. Historical inventory names such as `le-payload-pg` and `.worktrees/payload-migration` remain unchanged because they identify real transitional resources. Canonical nonproduction tree: ```text /srv/le-app-codex/ ├── home/ # dedicated account home and user configuration ├── lasereverything.net.db/ # sole persistent nonproduction Git checkout ├── database-source-readonly/ # root-owned scrubbed authoritative input ├── nonproduction-secrets/ # testing/development credentials only ├── container-runtime/ # rootless Docker images, layers and configuration ├── testing-data/ # disposable databases, media, models and caches └── build-artifacts/ # reports, logs and browser/test evidence ``` Ownership and permissions: | Path | Owner:group | Mode/policy | |---|---|---| | `/srv/le-app-codex` | `root:le_app_codex` | `0750`; prevents unrelated users and prevents project-root restructuring by the development account | | `home` | `le_app_codex:le_app_codex` | `0700`; account, Docker client, user systemd, NSS/browser configuration | | `lasereverything.net.db` | `le_app_codex:le_app_codex` | directories `0750`, normal Git files subject to repository modes; sole checkout | | `database-source-readonly` | `root:le_app_codex` | directories `0550`, files `0440`; account and daemon may read but cannot alter; no writable symlink target | | `nonproduction-secrets` | `root:le_app_codex` | directory `0750`, provisioned files `0640`; nonproduction values only; optional `generated/` is `0700 le_app_codex` for ephemeral fixture secrets | | `container-runtime` | `le_app_codex:le_app_codex` | `0700`; rootless Docker images, layers and configuration only | | `testing-data` | `le_app_codex:le_app_codex` | `0700`; child ownership may use rootless subordinate IDs | | `build-artifacts` | `le_app_codex:le_app_codex` | `0700`; TTL cleanup; snapshot artifacts treated as sensitive | Only the dedicated account receives: - read/write access to its workspace and runtime/storage subtree; - group read/traverse access to canonical root-owned `database-source-readonly`; - group read access to specifically provisioned files under canonical `nonproduction-secrets`; - a separate nonproduction Git credential/deploy key with access limited to the application repository. It receives no ACL or group membership for `/var/www`, production database paths, production sockets/volumes, or `/srv/directus-archive`. ## Exact privileged prerequisites and commands These commands are a reviewable proposal for an administrator on Garuda/Arch Linux. Recheck UID/GID and subordinate ranges first. Do not paste secrets into the shell history. ```bash # 1. Recheck names, IDs, and ranges across all four account databases. getent passwd le_app_codex getent group le_app_codex getent passwd 1200 getent group 1200 awk -F: '$3 == 1200 || $4 == 1200 { print; found=1 } END { exit found ? 1 : 0 }' /etc/passwd awk -F: '$3 == 1200 { print; found=1 } END { exit found ? 1 : 0 }' /etc/group awk -F: 'BEGIN { a=165536; b=231071 } { x=$2; y=$2+$3-1; if (a<=y && x<=b) { print FILENAME ":" $0; found=1 } } END { exit found ? 1 : 0 }' /etc/subuid /etc/subgid # 2. Install rootless prerequisites. docker-rootless-extras is an Arch AUR # package and must be built/reviewed under the administrator's normal AUR policy. sudo pacman -S --needed rootlesskit slirp4netns fuse-overlayfs shadow # Then install the reviewed docker-rootless-extras AUR package by the host's # approved AUR procedure; do not curl a remote install script into a shell. # 3. Create the isolated primary group and project subvolume. sudo groupadd --gid 1200 le_app_codex sudo btrfs subvolume create /srv/le-app-codex sudo chown root:le_app_codex /srv/le-app-codex sudo chmod 0750 /srv/le-app-codex # 4. Create the account with its canonical home inside the project root. sudo useradd --uid 1200 --gid le_app_codex --create-home \ --home-dir /srv/le-app-codex/home --shell /bin/bash le_app_codex sudo usermod --add-subuids 165536-231071 --add-subgids 165536-231071 le_app_codex # 5. Create the canonical children; no sibling workspace is created. sudo mkdir -p \ /srv/le-app-codex/lasereverything.net.db \ /srv/le-app-codex/database-source-readonly \ /srv/le-app-codex/nonproduction-secrets/generated \ /srv/le-app-codex/container-runtime/docker \ /srv/le-app-codex/testing-data/{postgres,media,legacy-uploads,bgbye-models,tmp} \ /srv/le-app-codex/build-artifacts/{browser,migration-reports} # Root owns the boundary and authoritative source; development owns mutable areas. sudo chown root:le_app_codex /srv/le-app-codex sudo chmod 0750 /srv/le-app-codex sudo chown le_app_codex:le_app_codex /srv/le-app-codex/home sudo chmod 0700 /srv/le-app-codex/home sudo chown -R le_app_codex:le_app_codex \ /srv/le-app-codex/lasereverything.net.db \ /srv/le-app-codex/container-runtime /srv/le-app-codex/testing-data /srv/le-app-codex/build-artifacts \ /srv/le-app-codex/nonproduction-secrets/generated sudo chmod 0700 /srv/le-app-codex/container-runtime /srv/le-app-codex/testing-data \ /srv/le-app-codex/build-artifacts /srv/le-app-codex/nonproduction-secrets/generated sudo chown root:le_app_codex /srv/le-app-codex/database-source-readonly /srv/le-app-codex/nonproduction-secrets sudo chmod 0550 /srv/le-app-codex/database-source-readonly sudo chmod 0750 /srv/le-app-codex/nonproduction-secrets # 6. Apply a 250 GiB ceiling to the project subvolume. sudo btrfs quota enable /srv sudo btrfs qgroup limit 250G /srv/le-app-codex # 7. Transitional source access is used only by the administrator-run copy and # verification procedure below. The final account needs no ACL on /srv/codex-*. # 8. Remediate production secret readability separately. # Root-owned Compose can still read mode 0600 files. Confirm the production # operator before applying because this changes production file permissions. sudo chown root:root /var/www/lasereverything.net.db/app/.env.local sudo chmod 0600 /var/www/lasereverything.net.db/app/.env.local sudo chown root:root /var/www/lasereverything.net.db/bgbye/.env sudo chmod 0600 /var/www/lasereverything.net.db/bgbye/.env # 9. As le_app_codex, configure the rootless daemon with an explicit data root. sudo -iu le_app_codex mkdir -p ~/.config/docker # Administrator writes ~/.config/docker/daemon.json as documented below, # then the dedicated user enables the packaged rootless user service. sudo -iu le_app_codex systemctl --user enable --now docker.socket # 10. Optional only if reviewed stacks must survive logout. sudo loginctl enable-linger le_app_codex ``` Proposed `/srv/le-app-codex/home/.config/docker/daemon.json`: ```json { "data-root": "/srv/le-app-codex/container-runtime/docker", "live-restore": false, "log-driver": "local", "log-opts": { "max-size": "20m", "max-file": "5" } } ``` The exact Arch rootless unit name (`docker.service` versus `docker.socket`) must be verified from the reviewed package before enabling. Cgroup v2 is present; after provisioning, verify the rootless/security options and delegated controllers before relying on CPU, memory, or PID limits. The nonstandard home is compatible with rootless Docker and user-level systemd because it is the account home recorded in `/etc/passwd`, resides on local Btrfs, and is owned by the account. Required environment contract: - `HOME=/srv/le-app-codex/home` comes from the login/account record; - `XDG_RUNTIME_DIR=/run/user/1200` is created by logind/systemd and must not be redirected into `/srv` or `/tmp`; - `XDG_DATA_HOME=/srv/le-app-codex/home/.local/share`; - Docker daemon data root is `/srv/le-app-codex/container-runtime/docker`; - Docker client configuration is `/srv/le-app-codex/home/.docker`; - user systemd configuration is `/srv/le-app-codex/home/.config/systemd/user`; - the `le-app-codex` context points only to `unix:///run/user/1200/docker.sock`; - wrappers export `COMPOSE_PROJECT_NAME=le-app-testing`. Proposed `/srv/le-app-codex/home/.config/environment.d/10-le-app-codex.conf`: ```text HOME=/srv/le-app-codex/home XDG_DATA_HOME=/srv/le-app-codex/home/.local/share DOCKER_CONFIG=/srv/le-app-codex/home/.docker COMPOSE_PROJECT_NAME=le-app-testing ``` `XDG_RUNTIME_DIR` is deliberately absent from this file; logind must create `/run/user/1200` with the correct lifetime and ownership. `DOCKER_HOST` is also omitted so the named context remains the explicit authority and scripts can reject context drift. After the user service is installed, create the context as the dedicated account: ```bash sudo -iu le_app_codex docker context create le-app-codex \ --description 'Laser Everything Codex nonproduction rootless runtime' \ --docker host=unix:///run/user/1200/docker.sock sudo -iu le_app_codex docker context use le-app-codex ``` ## Required positive and negative verification Run as `le_app_codex`, printing only permissions/status: ```bash test ! -r /var/www/lasereverything.net.db/app/.env.local test ! -x /var/lib/docker test ! -x /var/lib/mysql test ! -r /var/run/docker.sock test ! -x /srv/directus-archive test -r /srv/le-app-codex/database-source-readonly/README.txt test ! -w /srv/le-app-codex/database-source-readonly/README.txt test -r /srv/le-app-codex/nonproduction-secrets/migration.env test ! -w /srv/le-app-codex/nonproduction-secrets/migration.env DOCKER_HOST=unix:///run/user/1200/docker.sock docker info --format '{{json .SecurityOptions}}' ``` The final line must show rootless security. The development context must resolve only to `/run/user/1200/docker.sock`; wrappers reject `default`, `/var/run/docker.sock`, TCP endpoints, SSH contexts, and any unknown endpoint. ## Production path and socket denial preflight Every stack/job wrapper will fail before Compose evaluation if: - effective UID is not the approved dedicated UID; - Docker endpoint is not exactly the approved rootless Unix socket; - Compose project name lacks the fixed nonproduction prefix; - active Docker context is not exactly `le-app-codex`; - any resolved bind source is `/var/www`, `/var/lib/docker`, `/var/lib/mysql`, `/srv/directus-archive`, `/var/run`, `/run/docker.sock`, or a descendant; - the Git checkout is not exactly `/srv/le-app-codex/lasereverything.net.db` in nonproduction or `/var/www/lasereverything.net.db` in production; - any non-finite application service receives `.migration.env`, `.migration-source`, a MariaDB source URL, or migration credentials; - destination database host/port is not the approved nonproduction target/service; - production-named secret scopes, SMTP hosts, OAuth variables, Ko-fi secrets, or webhook credentials appear in fixture/snapshot/rehearsal rendered Compose; - reset/purge targets a bind mount, external volume, unknown project label, or any path outside `/srv/le-app-codex`. Tests render each Compose mode and inspect mounts, secrets, environment variable **names/classifications**, networks, contexts, labels, and volume ownership without printing values. ## Authoritative migration-source protection - The authoritative MariaDB remains read-only at the database privilege and filesystem layers. - `.migration-source` and `.migration.env` are mounted read-only only into finite `inventory`, `import`, and `validation` jobs. - Frontend, Payload, BGBye, proxy, Mailpit, fixture seed, package installation, build, lint, unit, integration, and ordinary browser-test services never receive those mounts or variables. - Legacy Directus restores the scrubbed SQL into a separate project-scoped writable MariaDB volume and copies uploads into `/srv/le-app-codex/testing-data/legacy-uploads` or a project volume. - Legacy Directus never mounts the source database storage or uploads, even read-only; the finite clone job is the only bridge. - Reset/purge operates only on labeled nonproduction destinations. It does not follow source symlinks and refuses all external/bind volumes. - `/srv/directus-archive` remains inaccessible and is never mentioned as a mount source in executable Compose configuration. The complete root-only archive remains permanently outside the development boundary at `/srv/directus-archive/20260710-175408`. It is never moved, mounted, linked, or copied into `/srv/le-app-codex`. ## Nonproduction storage plan The host currently has approximately 650 GiB available on the Btrfs filesystem containing `/srv` and `/home`. Proposed aggregate project quota: 250 GiB. | Path | Planning allowance | Ownership | Cleanup / backup | |---|---:|---|---| | `container-runtime/docker` | 80 GiB soft budget | `le_app_codex` | prune only project/unused development images after digest recording; no backup | | `testing-data/postgres` | 25 GiB | `le_app_codex` / rootless mapped IDs | disposable; reset by labeled volume; no backup | | `testing-data/media` | 15 GiB | fixed container UID mapped by rootless daemon | fixture disposable; snapshot copied data disposable; no authoritative backup role | | `testing-data/legacy-uploads` | 10 GiB | rootless mapped service UID | copied from scrubbed source; disposable; never source of truth | | `testing-data/bgbye-models` | 35 GiB | BGBye service/rootless mapping | checksummed cache; LRU/manual cleanup; redownloadable | | `testing-data/tmp` | 10 GiB | service-specific mapped UID | tmpfs or age-based cleanup; no backup | | `build-artifacts/browser` | 5 GiB | `le_app_codex` | TTL 7 days; snapshot artifacts treated as sensitive and never committed | | `build-artifacts/migration-reports` | 10 GiB | `le_app_codex` | detailed reports TTL 30 days; committed summaries structural only | | checkout/home metadata | 10 GiB | `le_app_codex` | Git remote is source backup; no personal exports committed | The listed soft budgets total 200 GiB. The 250 GiB hard qgroup ceiling reserves 50 GiB emergency headroom. Startup refuses when host free space is below 100 GiB, project usage exceeds 200 GiB, or required free space for a rehearsal cannot be reserved. The production filesystem is not used for spillover. ## Resources intentionally outside `/srv/le-app-codex` | Resource | Permanent location | Why it remains outside | |---|---|---| | Complete untouched Directus archive | `/srv/directus-archive/20260710-175408` | Root-only disaster-recovery evidence; must not enter the Codex trust boundary | | Production checkout/deployment root | `/var/www/lasereverything.net.db` | Canonical production policy and existing shared-server operations boundary | | Production Docker image/layer storage | Host runtime-managed location | Operational daemon state, declared by the canonical production deployment but not Git content | | Production PostgreSQL/media storage | Approved production volume/storage paths | Persistent production data requires independent ownership, capacity, and backup controls | | Shared public reverse-proxy configuration/certificates | Existing TITANSERVER ingress paths | Server-wide infrastructure serves multiple applications and is never container-controlled | | Existing self-hosted SMTP infrastructure | Existing mail-server deployment | Production email remains independent; Mailpit is nonproduction-only | Transitional `/srv/codex-*` paths remain temporarily outside during the rollback period only. They are not permanent alternate project roots and become cleanup candidates solely through the explicit process below. ## Production ingress boundary Production will use an **internal application-gateway container behind TITANSERVER's existing shared host reverse proxy**. - The server-wide proxy remains host-managed, terminates public TLS, owns ports 80/443, ACME, and cross-application routing. - The repository-controlled gateway binds only to a dedicated host loopback port or private production network and routes frontend versus Payload internal services. - The host proxy forwards approved public hostnames to that one gateway and overwrites—not trusts from clients—the forwarding chain and scheme headers. - The application repository supplies a versioned host-proxy include/template and contract tests, but an operator explicitly reviews/renders/includes it. Containers never mount or edit host proxy configuration or certificates. - The gateway is not published directly to the internet. Payload admin exposure can be separately restricted at the host proxy. - Existing `forms.lasereverything.net` behavior remains under the shared ingress until cutover authorization. Local development uses its own gateway/TLS because it does not share production ingress. This design preserves server-wide ownership while keeping application routing rules testable and versioned. ## Transitional resource inventory Metadata-only audit results at the planning checkpoint: | Resource | Current state | |---|---| | `/srv/codex-work/lasereverything.net.db` | Primary Git checkout, branch `temporary/utilities-only`, commit `305ea41`; owns the current local Git database | | nested `.worktrees/payload-migration` | Migration worktree, branch `migration/le-app-database`; planning commit begins at `ed1c84e` | | `.migration-source` | Symlink to `/srv/codex-migration/current` | | `.migration.env` | Symlink to `/srv/codex-secrets/le-payload-migration.env` | | `/srv/codex-migration/current` | Symlink to `source-20260710-175408` | | `/srv/codex-migration/source-20260710-175408` | Scrubbed source snapshot, mode `0750`, approximately 150 MB aggregate at audit time | | `/srv/codex-secrets/le-payload-migration.env` | Current readable nonproduction migration environment, mode `0640`; values not inspected | | `/srv/codex-secrets/le-payload-migration-admin.env` | Root-only transitional administration environment, mode `0600`; values not inspected and not copied into the development environment | | `/srv/codex-work` | Approximately 2.0 GB aggregate, including repository history/worktrees and application files | | recorded rootful MariaDB | container `le-directus-source`, volume `le_directus_source_data_20260710-175408` | | recorded rootful PostgreSQL | container `le-payload-pg`, volume `le_payload_pg_data_20260710-175408` | | retired Directus | container `directus`; not a disposable migration resource and not removed by consolidation | The current user cannot query the rootful production Docker daemon. Before consolidation, an administrator must produce a metadata-only inventory without printing container environments, labels that contain secrets, database contents, or mount-file contents: ```bash sudo docker ps -a --filter name=^/le-directus-source$ \ --filter name=^/le-payload-pg$ --format '{{.Names}} {{.Image}} {{.Status}}' sudo docker inspect le-directus-source le-payload-pg \ --format '{{.Name}} image={{.Image}} mounts={{range .Mounts}}{{.Type}}:{{.Name}}:{{.Destination}};{{end}}' sudo docker volume inspect \ le_directus_source_data_20260710-175408 \ le_payload_pg_data_20260710-175408 \ --format '{{.Name}} driver={{.Driver}} mountpoint={{.Mountpoint}} labels={{json .Labels}}' ``` If names, image IDs, volumes, or mounts differ from the recorded manifest, stop and revise the plan. Do not inspect `.Config.Env` or print secret values. ## Reversible consolidation procedure No step deletes or mutates a transitional resource. Consolidation is copy/recreate/verify, followed by a rollback observation period. ### Phase A — manifests and Git preservation 1. Push `migration/le-app-database`, including planning commits, to `origin` and verify the remote commit. 2. Record structural manifests for every transitional path, symlink, owner/mode, aggregate size, Git branch/commit/remote, source checksum manifest, and the administrator container/volume inventory above. 3. Confirm the full root-only archive remains independently present and inaccessible; do not traverse or copy it. 4. Record the exact existing rootful container/volume names and leave them stopped/running exactly as found unless a later approved step says otherwise. ### Phase B — canonical checkout After account/runtime approval, create one clean checkout as the dedicated user: ```bash sudo -iu le_app_codex git clone --branch migration/le-app-database --single-branch \ ssh://FORGE/makearmy/le-app.git \ /srv/le-app-codex/lasereverything.net.db sudo -iu le_app_codex git -C /srv/le-app-codex/lasereverything.net.db \ fetch --prune origin sudo -iu le_app_codex git -C /srv/le-app-codex/lasereverything.net.db \ status --short --branch sudo -iu le_app_codex git -C /srv/le-app-codex/lasereverything.net.db \ rev-parse HEAD ``` The HEAD must equal the reviewed remote `migration/le-app-database` commit, the worktree must be clean, and no nested permanent worktree may exist. Provision a dedicated repository-scoped Git credential for `le_app_codex`; do not copy `sol6_vi`'s private key. ### Phase C — scrubbed source copy The administrator copies from the resolved source directory, never from the root-only archive: ```bash source_old=/srv/codex-migration/source-20260710-175408 source_new=/srv/le-app-codex/database-source-readonly sudo rsync -aHAX --numeric-ids --delete-delay "$source_old/" "$source_new/" # Verify supplied manifests without emitting filenames or record contents. sudo sh -c "cd '$source_new' && sha256sum --check --status CHECKSUMS.sha256" sudo sh -c "cd '$source_new' && sha256sum --check --status uploads.sha256" # Freeze the canonical input against development writes. sudo chown -R root:le_app_codex "$source_new" sudo find "$source_new" -type d -exec chmod 0550 {} + sudo find "$source_new" -type f -exec chmod 0440 {} + ``` Before using `--delete-delay`, assert `source_new` resolves exactly to `/srv/le-app-codex/database-source-readonly` and is not a symlink. The source checksum files, row-count manifests, schema, dump, extensions, and all uploads must validate. The old copy remains untouched. ### Phase D — nonproduction secrets - Preserve both old credential files in place during rollback. - Do not copy `le-payload-migration-admin.env`. - Generate new fixture/snapshot/rootless database, Payload, Mailpit, and local gateway credentials directly into root-owned `/srv/le-app-codex/nonproduction-secrets` with mode `0640`. - If the MariaDB read-only credential from the scrubbed environment is temporarily required to compare the old rootful source, transfer only the minimum required variable set through an administrator-reviewed, non-logging process; replace it once the rootless source clone is verified. - No production SMTP, OAuth, Ko-fi, Directus token, webhook, or server credential enters the new directory. ### Phase E — rootless database recreation 1. Restore the scrubbed SQL from canonical read-only `source` into a new project-scoped rootless MariaDB volume. 2. Create a distinct import account with `SELECT` only; application/migration jobs never receive the MariaDB administrative credential. 3. Mount/copy uploads only through finite jobs; the MariaDB and legacy Directus services never receive writable access to canonical `source`. 4. Create fresh disposable rootless PostgreSQL and Payload media volumes. 5. Verify source table counts/checksum summaries against supplied manifests and verify PostgreSQL health/reset isolation. 6. Record new rootless image digests, volume names, Compose project labels, and service health without recording secrets. ### Phase F — rollback observation period Keep `/srv/codex-work`, `/srv/codex-migration`, `/srv/codex-secrets`, rootful `le-directus-source`, rootful `le-payload-pg`, and their recorded volumes intact for at least 30 days **and** through at least two complete successful rootless migration rehearsals, whichever is later. During this period: - new work occurs only in the canonical checkout; - old resources are read-only/frozen where possible; - checksum, Git, service, and validation comparisons remain reproducible; - rollback means stop the rootless stack and resume analysis from the untouched transitional resources, not copy changes backward. ## Later cleanup eligibility — explicit approval required Only after the rollback criteria and a fresh inventory may the following become eligible for removal: - `/srv/codex-work/lasereverything.net.db/.worktrees/payload-migration` after Git confirms it is clean, pushed, and removable with `git worktree remove`; - `/srv/codex-work/lasereverything.net.db/.migration-source` and `.migration.env` symlinks; - `/srv/codex-work/lasereverything.net.db`, then `/srv/codex-work` only if no other entries exist; - `/srv/codex-migration/current`, `source-20260710-175408`, then `/srv/codex-migration` after canonical checksums pass and rollback expires; - `/srv/codex-secrets/le-payload-migration.env` and `le-payload-migration-admin.env`, then `/srv/codex-secrets`, after credentials are revoked/obsolete and rollback expires; - rootful containers `le-directus-source` and `le-payload-pg`; - rootful volumes `le_directus_source_data_20260710-175408` and `le_payload_pg_data_20260710-175408` after verifying their exact attachment and backup status. Every deletion requires a separate explicit approval showing the resolved path/object, before/after checksum or Git proof, attachment status, and rollback consequence. Never include the retired `directus` container, production volumes, `/var/www`, or `/srv/directus-archive/20260710-175408` in this cleanup set. ## Canonical production promotion Promotion path: ```text /srv/le-app-codex/lasereverything.net.db -> reviewed Git commit + signed/recorded release manifest -> CI-built immutable image digests tested in fixture/snapshot/rehearsal/staging -> clean checkout or verified release artifact at /var/www/lasereverything.net.db -> production deployment invoked only from /var/www/lasereverything.net.db ``` Required workflow: 1. The canonical nonproduction checkout is clean and all accepted commits are pushed. 2. CI builds images once, records digests/SBOM/provenance, and runs the accepted test gates. 3. Staging deploys those exact digests from the same Compose core and an approved staging overlay. 4. A reviewed release manifest pins Git commit, image digests, migration set, configuration schema, health contract, and rollback compatibility. 5. On TITANSERVER, an operator updates a clean Git checkout at `/var/www/lasereverything.net.db` to the reviewed commit (or atomically unpacks a verified release artifact into that exact root), verifies a clean tree, and verifies the release manifest. 6. From that directory only, deployment scripts render/validate production Compose and the host-ingress include, run the gated schema job, pull pinned digests, and roll services with health checks. The production checkout contains all application source/configuration needed to reproduce the deployment contract. Physical image layers, declared production PostgreSQL/media volumes, and shared host-proxy configuration may remain outside it, but their names/locations/interfaces are declared in the production overlay and release manifest rooted there. Forbidden promotion inputs include `/srv/le-app-codex/container-runtime`, `database-source-readonly`, `nonproduction-secrets`, `testing-data`, `build-artifacts`, local CA material, Mailpit state, test traces, rootless Docker state, disposable databases/volumes, caches, and development `.env` files. No `rsync` or recursive copy from `/srv/le-app-codex` to `/var/www` is permitted. ## Local HTTPS and CA trust The local gateway uses a repository-pinned proxy image and an ignored, generated development CA to serve: - `app.le.localhost`; - `cms.le.localhost`; - `mail.le.localhost`; - optional `legacy.le.localhost`. All bind to loopback port 8443, avoiding privileged ports and host DNS changes (`*.localhost` resolves locally). Certificates and private keys live under `/srv/le-app-codex/nonproduction-secrets/local-ca`, mode `0700/0600`, excluded from Git and build contexts. Automated Playwright tests may use a dedicated browser profile with its own trust database. Manual Chromium trust can be limited to `/srv/le-app-codex/home/.pki/nssdb` using the already installed `certutil`; system-wide CA trust is unnecessary. Importing the CA into even that user/browser profile is a host/user-state change and requires explicit approval. Removal deletes the certificate from that profile and the ignored CA directory. ## Mailpit and outbound protection Mailpit exists only in fixture, snapshot, rehearsal, and default staging modes: - internal Compose networks only; UI reaches the browser through the local gateway; - no public host bind and no production ingress route; - SMTP relay/forwarding and webhook forwarding disabled; - no production SMTP variables mounted; - tests query Mailpit's internal HTTP API and follow local HTTPS verification/recovery links; - registration, verification, recovery, supporter claim, and notification paths are covered. Defense in depth: 1. Mode-specific environment schema rejects production SMTP hostnames/sender domains, ports 25/465/587 as external endpoints, production credential variable scopes, OAuth secrets, Ko-fi production tokens, and webhook credentials. 2. Runtime services attach only to an `internal: true` application network in nonproduction. Mail is addressed to the internal `mailpit:1025` service. 3. Snapshot-derived recipient addresses are rejected or rewritten at the transport boundary to deterministic reserved-domain aliases. Only synthetic fixture accounts may be addressed directly. 4. No production credential exists in rendered Compose, container environment, secret mounts, or image history. 5. A finite `egress-test` runs from the exact frontend and Payload runtime network namespaces: Mailpit delivery must succeed; connection attempts to controlled external test endpoints on SMTP ports must fail; DNS/HTTP behavior is recorded separately. Firewall-level isolation will not be claimed until these tests pass under the selected rootless runtime. If Docker `internal` networking still permits an undesired host/external route, the implementation stops and proposes an approved rootless egress control rather than asserting safety. Tests use an administrator-approved controlled endpoint, never a real third-party SMTP server. Production remains unchanged: application services use the existing self-hosted SMTP infrastructure and real recipients through production-only secrets and network policy. Mailpit never replaces or modifies it. ## BGBye CPU/GPU profiles Milestone 0 starts with a deterministic CPU-compatible BGBye image/profile: - same FastAPI paths, methods endpoint shape, request fields, status codes, content types, health/readiness schema, and error contract as GPU; - one pinned CPU-capable segmentation model for real tiny-image processing in routine tests; - image upload/proxy limits, invalid method/file handling, alpha PNG output, concurrency serialization, temporary-file lifecycle, status polling, and FFmpeg contract can be exercised; - model artifacts are pinned/checksummed and readiness fails until loaded. CUDA-only validation is deferred to an opt-in GPU profile/staging gate: - NVIDIA runtime/device visibility and driver compatibility; - every production model/backend, CUDA placement, GPU lock/concurrency, memory cleanup/OOM behavior, performance envelope, and longer video processing; - production model-cache checksum and offline startup behavior. CPU and GPU images share server contract tests. Capability differences appear in explicit health/method metadata rather than changing endpoints. A production GPU image is released only after model preload/readiness, representative image/video output, cleanup, restart, and resource tests pass on GPU staging. Rootless GPU provisioning does not block the first stack. ## Rootless runtime rollback/removal Rollback must be run only after confirming the selected context/socket is the dedicated one: ```bash # As le_app_codex: stop project resources, then the user daemon. DOCKER_HOST=unix:///run/user/1200/docker.sock docker compose \ --project-name le-app-testing down --remove-orphans systemctl --user disable --now docker.socket docker.service # As administrator: disable optional lingering. sudo loginctl disable-linger le_app_codex # Verify production daemon endpoint and data were never referenced by project metadata/logs. # Use the planning preflight/audit script; do not connect to or mutate production Docker. # Remove only the dedicated project subvolume after an explicit artifact review. sudo btrfs subvolume delete /srv/le-app-codex # Remove subordinate allocations and account after the subvolume/runtime is gone. sudo usermod --del-subuids 165536-231071 --del-subgids 165536-231071 le_app_codex sudo userdel --remove le_app_codex sudo groupdel le_app_codex 2>/dev/null || true ``` The dedicated Docker context lives only in the dedicated user's home and disappears with that home. If another operator created a client context pointing to the user socket, remove that context explicitly from that operator's client configuration. Production secret permission hardening is not rolled back merely because development is removed; it is an independent security correction. Any temporary transitional ACLs, if an administrator added them contrary to the final copy-based design, must be inventoried and removed before account deletion. Removal does not touch `/var/run/docker.sock`, production contexts, `/var/lib/docker`, `/var/lib/mysql`, `/var/www` application content, production volumes/networks, transitional source during its rollback retention, or `/srv/directus-archive`. Final proof compares before/after permission metadata and confirms: production socket remains denied, production service state is unchanged, no project mount referenced forbidden paths, the dedicated UID/subordinate mappings are gone, and only `/srv/le-app-codex` resources were deleted.