le-app/docs/payload-migration/environment-parity.md

Milestone 0: reproducible application environment

Status: proposal for review. It describes future files and services; it does not authorize production changes or privileged host configuration.

Complete baseline runtime inventory

The inventory is based on commit 4884b5d, its checked-in Compose/Dockerfiles, application routes, Directus deployment exports, and the frozen structural migration manifests.

Component	Responsibility and dependencies	Persistence / lifecycle
frontend	Existing Next.js UI, middleware, server-rendered public pages, authenticated portal, compatibility APIs, file browser, SVGnest iframe, upload validation, Ko-fi webhook and claim endpoints, and BGBye proxy	Stateless image; Next cache on tmpfs; reads curated files; uses secure HTTP-only auth cookies
payload	Dedicated Payload admin, auth, content/media API, access control, hooks, jobs, and migrations	PostgreSQL plus Payload media volume; independent image and release lifecycle
postgres	Payload relational database	Named volume in nonproduction; managed/dedicated storage may replace it in production
bgbye	FastAPI image/video background removal with CUDA models, FFmpeg video processing, status polling, and ten-minute temporary-file cleanup task	Model caches are persistent; video/frame/output workspace is ephemeral; GPU is optional only if a functional CPU test profile is implemented
proxy	Single browser entrypoint, local TLS, host routing, forwarded headers, redirects, body limits, timeouts, and security headers	Configuration is versioned; certificates are mode-specific and never baked into application images
mailpit	Captures registration, recovery, verification, supporter-claim, and notification mail in nonproduction	Disposable by default; optional named volume for debugging; never deployed as the production relay
migration job image	Structural inventory, schema application, source import, file copy, sequence repair, checksums, and reports	Reads authorized source mount read-only; writes disposable PostgreSQL/media and non-personal summary reports
validation job image	Read-only source/destination comparison: counts, IDs, relationships, timestamps, ownership, permissions, files, checksums, and API contracts	No source writes; generated detailed reports are local and excluded from Git/build contexts
e2e job image	Browser and API tests against proxy-visible hostnames	Ephemeral browser artifacts; traces/screenshots are excluded when they may contain personal data
optional legacy-directus	Local reference for Studio presentation, permissions, flows, relations, files, and legacy API shapes	Uses a fresh writable MariaDB clone and copied uploads; never connects to the authoritative read-only source
optional legacy-mariadb	Writable database solely for the legacy reference service	Disposable named volume populated from the scrubbed dump
fixture-seed job	Loads deterministic synthetic users, content, ownership cases, and representative tiny files	Runs only against disposable fixture PostgreSQL/media
curated file library	Existing app/files manuals, specifications, software/reference files, and file-server routes	Separate read-only volume in runtime; approximately 461 MB / 282 files at baseline
Payload media	Migrated Directus uploads and future CMS uploads	Dedicated persistent volume/object-storage contract; not mixed with curated files
retained unreferenced uploads	2,411 files pending classification	Read-only retention input; never deleted by application reset/import jobs
external Ko-fi	Sends signed webhook events and is the source for historical membership backfill	Nonproduction uses fixtures; no real webhook secret or outbound calls
production SMTP	Sends account and notification mail	Replaced by Mailpit in every nonproduction mode; credentials only in production secret scope
reverse-proxy/TLS provider	Public TLS and routing	Production-specific infrastructure around the same proxy contract

Existing application processes and integrations

• Authentication currently uses ma_at, ma_rt, ma_ra, and ma_v cookies, middleware redirects, periodic token validation, username-to-email lookup, recent-auth confirmation, registration, login, logout, and password change. Payload must add recovery and verification endpoints and mail templates because the baseline UI does not yet provide a complete recovery journey.
• Public/catalog consumers cover laser sources, four settings families, materials/coatings/hazards, buying guides, and projects.
• Authenticated consumers cover profile/avatar/password, projects and attachments, rigs, settings submission/deletion, claims, memberships/supporter badges, and account linking.
• Upload paths include avatars, project hero/attachments, settings photos/screenshots, Payload media, curated file downloads, and background-removal inputs/outputs.
• app/scripts/backfill-kofi.mjs is a remaining Directus consumer and becomes an idempotent Payload membership import job.
• /api/webhooks/kofi, supporter claim start/verify/unlink, and badge derivation become Payload-backed compatibility routes. Nonproduction uses signed fixture events only.
• Directus's two event flows (unique_key_claims, claim_auto_assign) must be reconstructed as transactional Payload hooks/jobs after behavior is resolved in the isolated legacy reference.
• BGBye has an in-process startup cleanup loop every ten minutes. There is no verified host cron/systemd job in the repository. Future scheduled Payload work must be an explicit worker/job service, not an undocumented host cron.
• SVGnest is vendored static client code embedded through the frontend. The separate dashboard/ is a static legacy/operations dashboard in the monorepo but is not in the baseline Compose runtime; Milestone 0 must either add an explicit static service/profile or formally classify it as non-required. Default proposal: keep it as an optional dashboard profile, not part of the public application critical path.
• External links such as image hosting, paste, Forgejo, and community/support sites are navigation dependencies, not services required for the core stack to start. Smoke tests verify links are rendered but do not require those external systems.

Common Compose architecture

The repository will use a project-name-scoped Compose model:

infra/compose/compose.yaml              # core networks, services, health contracts
infra/compose/compose.dev.yaml          # bind mounts/watch/debug ports
infra/compose/compose.snapshot.yaml     # authorized read-only snapshot mounts
infra/compose/compose.staging.yaml      # staging hosts/resources/secrets/storage
infra/compose/compose.production.yaml   # production hosts/resources/external infrastructure
infra/compose/profiles/*.yaml           # legacy Directus, GPU/CPU, dashboard, test tooling

Core service definitions and Dockerfiles are shared. Mode selection composes files and profiles rather than copying service definitions.

Mode	Services/data	Intended use
fixture	proxy, frontend, Payload, PostgreSQL, Mailpit, BGBye, fixture seed; optional dashboard	Daily development, CI, API tests, deterministic browser tests
snapshot	Same application images plus read-only scrubbed source/uploads and import/validation jobs	Realistic edge cases and file/API compatibility checks
rehearsal	Snapshot topology with destination reset, committed schema migration, full import, validation, then smoke/e2e gates	Repeatable cutover rehearsal; destructive only to project-scoped destination volumes
staging	Exact promoted image digests, staging PostgreSQL/media/secrets, capture or allow-listed staging SMTP	Production-shape acceptance and release candidate verification
production	Exact accepted digests, production PostgreSQL/media/SMTP/TLS/resources	Live service; no source snapshot, legacy Directus, fixture seed, Mailpit, or migration credentials mounted

Compose health dependencies use explicit checks rather than startup order:

• proxy: configuration valid and HTTPS route responds;
• frontend: /api/health/live and /api/health/ready (Payload/BGBye dependency status separated);
• Payload: liveness plus database-backed readiness;
• PostgreSQL: pg_isready;
• BGBye: existing /health, expanded to distinguish model readiness;
• Mailpit: HTTP health endpoint;
• jobs: finite exit status and machine-readable summary.

No application service receives .migration.env. Only explicitly invoked inventory/import/validation jobs receive migration variables. Package installation, linting, builds, unit tests, fixture mode, frontend, Payload, proxy, Mailpit, and BGBye do not load it.

Two nonproduction data modes

Fixture mode

• A small, deterministic, non-personal dataset committed as factories/fixtures.
• Stable IDs exercise numeric IDs, UUIDs, ownership, public/private content, junctions, nulls, Unicode, timestamp boundaries, and authorization failures.
• Tiny generated fixtures cover JPEG, PNG, WebP/AVIF where supported, SVG rejection/handling, PDF, ZIP, plain text, and an executable-signature rejection case.
• Synthetic mailbox addresses use reserved domains and never identify real users.
• Resets remove only Compose-project-scoped PostgreSQL, media, mail, and temporary volumes.

Snapshot mode

• Uses the credential-scrubbed MariaDB endpoint and authorized source/upload mounts read-only.
• Uses the same frontend, Payload, proxy, BGBye, and job images as fixture mode.
• Destination PostgreSQL and copied destination media are disposable and writable.
• Detailed validation artifacts stay local and ignored; committed reports contain structural summaries only.
• Outbound SMTP is forced to Mailpit and outbound network policy denies real SMTP/OAuth/Ko-fi delivery.

Optional legacy Directus reference

The legacy-reference profile is local-only and off by default:

1. Verify the archived Directus image by immutable image ID/digest before use; never pull latest as a substitute.
2. Create a new project-scoped MariaDB volume and restore the scrubbed dump into it.
3. Copy uploads into a new writable project-scoped volume; never mount .migration-source/uploads writable.
4. Generate a new local KEY, SECRET, administrator address/password, and any test policy tokens. None may match production values.
5. Disable SMTP, OAuth, external webhooks, telemetry where supported, public registration, and outbound network access.
6. Bind only to the internal Compose network and route through the local proxy/admin hostname; do not publish its container port publicly.
7. Label the UI and responses clearly as scrubbed, disposable, local reference data.

This service is evidence, not an authoritative source. The frozen read-only MariaDB snapshot remains the migration authority. The archived image currently identified in the manifest is sha256:dd5537…0801ce2a; the complete value remains in the local manifest and must be supplied through an authorized local image import, not copied into a public registry without approval.

Reverse proxy and browser parity

Proposed local endpoints use wildcard loopback names without editing DNS:

• https://app.le.localhost:8443 — frontend and compatibility APIs;
• https://cms.le.localhost:8443 — Payload admin/API;
• https://mail.le.localhost:8443 — Mailpit UI;
• optional https://legacy.le.localhost:8443 — isolated Directus reference.

The versioned proxy configuration must:

• set Host, X-Forwarded-Host, X-Forwarded-Proto=https, X-Forwarded-Port, X-Real-IP, and X-Forwarded-For consistently;
• support upgrade headers where an application endpoint needs them;
• reproduce legacy redirects and canonical-host behavior;
• set explicit per-route request limits: at least the verified historical 100 MB proxy ceiling, with lower application limits retained for avatars/settings/projects/BGBye;
• use 300-second upstream timeouts only where file/background processing requires them, with shorter defaults elsewhere;
• preserve streaming/download headers and range behavior for representative files;
• run application containers with fixed non-root UIDs and preflight media/file permissions;
• derive cookie Secure, HttpOnly, SameSite, domain, path, expiry, refresh, and recent-auth behavior from one shared auth contract.

Local HTTPS is required to test secure cookies. A repository-controlled development CA/certificate generator may create ignored local certificates, but trusting that CA in the host browser is a host change and requires user approval. Port 8443 avoids privileged rootless port binding; cookie semantics do not depend on port. Production uses the same routing contract with its real TLS termination and standard port 443.

Safe email

• Mailpit is the only SMTP host supplied to fixture, snapshot, rehearsal, and default staging modes.
• Nonproduction secrets/config validation fails closed if an SMTP hostname, sender domain, or port matches a production allow/deny signature.
• Egress policy blocks TCP 25/465/587 from application containers in snapshot/rehearsal modes except to the internal Mailpit service.
• Tests retrieve messages through Mailpit's API and follow recovery/verification links through the local HTTPS proxy.
• Snapshot users are never sent messages automatically. Tests create synthetic accounts or redirect recipients at the mail transport boundary.
• Production SMTP credentials are provided only by the production secret mechanism and are never stored in Compose YAML, .env examples, images, test traces, or migration reports.

Rootless Codex-controlled runtime

Required target: a dedicated rootless Docker daemon owned by the separate le_payload_dev account. Future Codex migration sessions must execute as that account (preferred) or through a narrowly scoped account transition; the daemon socket is not shared broadly. Rootless Podman is an acceptable fallback only after Compose/profile behavior is proven; Docker is preferred because the repository and production already use Compose.

Codex must be able to build images, create/remove only its project containers/networks/volumes, inspect their logs, reset disposable PostgreSQL, run finite jobs, and execute integration/browser tests. It must not access /var/run/docker.sock, the production daemon/context, production containers/volumes/networks, /var/www, or /srv/directus-archive.

Current host observation:

• sol6_vi is not in the Docker group but can read the production application's world-readable environment files under /var/www;
• Docker and Docker Compose clients exist;
• no Podman client is installed;
• the default Docker context does not provide a usable daemon/security result to this user;
• no enabled user-level rootless Docker or Podman service was detected.

Therefore a sol6_vi-owned daemon is rejected: it could bind-mount those readable production secrets. Milestone 0 requires the isolated account and host changes described in host-runtime-plan.md, and work must stop before performing them. An administrator must approve/provision:

1. dedicated le_payload_dev identity and restricted filesystem ACLs;
2. rootless prerequisites (newuidmap/newgidmap, subordinate UID/GID ranges, rootless networking and overlay helpers as required by the OS);
3. rootless Docker daemon installation for le_payload_dev using its dedicated data root and named context;
4. optional user-service lingering if stacks must survive logout;
5. deterministic CPU BGBye first; optional GPU access only after core acceptance;
6. user/browser-profile-scoped development CA trust;
7. capacity limits for images, caches, snapshots, media, PostgreSQL, and test artifacts.

No production socket forwarding, Docker-group membership, broad sudo rule, daemon TCP exposure, or bind mount into production paths is acceptable.

One-command workflows

The repository-root commands will be thin, reviewed wrappers around Compose and finite job containers:

./scripts/stack-up fixture
./scripts/stack-up snapshot
./scripts/stack-down
./scripts/reset
./scripts/schema
./scripts/import
./scripts/validate
./scripts/smoke-test

Each wrapper validates the selected rootless context, project name, mode, required mounts, and forbidden production paths before doing work. reset requires an explicit nonproduction project label and refuses unknown database hosts. stack-down does not delete volumes by default. A separate explicit disposable purge command may remove only labeled project volumes.

From a clean checkout, fixture mode requires only the rootless runtime, ignored local secrets generated from examples, and cached/built images. Snapshot mode additionally requires the authorized read-only source path and migration environment. There are no undocumented restore, chmod, hostname, secret, or database steps.

Build context and secret isolation

Every context (app, payload, bgbye, proxy, jobs/tests, and repository root where used) excludes:

• .git, .worktrees, .migration-source, .migration.env;
• *.sql, *.sql.gz, database dumps and snapshot manifests containing data;
• source and destination uploaded user files;
• migration-generated detailed reports/logs;
• browser traces/screenshots/videos from snapshot mode;
• local secrets, certificates/private keys, dependency trees, and build outputs.

Build verification inspects image histories and filesystems for forbidden names and secret fingerprints. Compose configuration tests assert that migration mounts/variables appear only on authorized finite jobs and never in production output.

Proposed final repository layout

/
├── apps/
│   ├── web/                         # current Next.js frontend + compatibility APIs
│   └── payload/                     # dedicated Payload admin/API
├── services/
│   ├── bgbye/                       # FastAPI GPU/CPU worker
│   ├── proxy/                       # versioned local/production routing contract
│   └── dashboard/                   # optional static dashboard profile
├── packages/
│   ├── contracts/                   # API DTOs, auth/cookie, errors, health contracts
│   ├── config/                      # validated environment schemas, no secret values
│   ├── fixtures/                    # deterministic synthetic factories and tiny files
│   └── test-helpers/                # API/browser/mail/file helpers
├── migration/
│   ├── inventory/                   # structural inventory tooling
│   ├── mappings/                    # reviewed field/relationship mappings
│   ├── schema/                      # committed Payload/Drizzle migrations
│   ├── import/                      # idempotent source/file import
│   ├── validation/                  # counts, relationships, ownership, checksums
│   └── legacy-reference/            # isolated restore/bootstrap tooling
├── infra/
│   ├── compose/                     # core and mode overlays/profiles
│   ├── proxy/                       # routes, headers, limits, local TLS config
│   ├── health/                      # healthcheck commands/contracts
│   └── env/                         # non-secret examples and mode schemas
├── scripts/                         # one-command wrappers and safety guards
├── tests/
│   ├── unit/
│   ├── integration/
│   ├── contracts/
│   ├── e2e/
│   └── fixtures/files/
├── docs/payload-migration/
├── directus/                        # retained legacy evidence during migration
├── .nvmrc / .node-version
└── Dockerfiles remain beside each build context

The current app, bgbye, and dashboard directories will be moved only in a dedicated mechanical commit after build-context and path references are covered by tests. Milestone 0 may initially implement the target logical boundaries without a risky all-at-once directory move.

The repository tree above exists at exactly one persistent nonproduction checkout, /srv/le-payload-dev/lasereverything.net.db, and at the canonical production root, /var/www/lasereverything.net.db. Runtime data is never placed inside either Git checkout unless it is a deliberately versioned non-secret fixture. The complete canonical host directory policy and transitional consolidation are defined in host-runtime-plan.md.

End-to-end test matrix

Area	Fixture mode	Snapshot/rehearsal mode	Core assertions
Registration	Full	Synthetic account only	validation, duplicate handling, role/status, captured verification mail
Login/logout	Full	Transitional-hash synthetic/test account; no personal output	username/email login, invalid credentials, refresh/expiry, cookie clearing
Password recovery	Full	Synthetic recipient only	enumeration resistance, token expiry/single use, Mailpit link, new login
Cookies/sessions	Full browser matrix	Full	Secure/HttpOnly/SameSite/path/domain, proxy scheme, redirects, refresh, recent-auth
Account/profile	Full	Structural/authorized test account	self-read/update, sensitive re-auth, avatar, cross-user denial
Projects	CRUD/files	Counts/contracts/representative records	ownership, public reads, attachment links, delete boundaries
Rigs	CRUD	Counts/contracts	owner presets and update/delete isolation
Machine settings	Four families	Full counts/relationships/API parity	create/read/update/delete ownership, images, submission_id compatibility
Claims/memberships	Full state fixtures	Flow/API parity	unique key, auto-assignment, pending transitions, supporter visibility
Upload/download	Representative file matrix	Realistic files/checksums	signatures, size limits, names, MIME, ranges, traversal prevention, permissions
Admin	Full	Read/controlled test writes	admin login, collections, roles, file management, forbidden member access
Background removal	Tiny image; deterministic allowed method	Representative image only	proxy body limit, model readiness, output PNG, invalid method/file, cleanup
Public utilities	Full	Full	toolkit, SVGnest, file browser, legacy redirects, external-link rendering
Compatibility APIs	Contract fixtures	Response-shape differential tests	Directus-like filters/fields/sort/pagination/errors where still consumed
Email	Mailpit	Mailpit only	no external delivery, correct recipient rewrite, recovery/verification/support flows
Authorization	Full adversarial matrix	Selected real relationship topology	anonymous/member/owner/admin/bot boundaries and field-level restrictions
Health/restart	Full	Full	dependency readiness, graceful restart, persistent DB/media, ephemeral temp data

Snapshot test logs report identifiers only as one-way test-local aliases or aggregates; they do not emit emails, usernames, hashes, record bodies, or user-provided content.

Promotion to staging and production

1. CI under Node 24.18.0 builds each application image once and records its digest and source commit.
2. Fixture integration/e2e tests run those digests through the common core Compose model.
3. Authorized snapshot rehearsal runs the same digests with read-only source mounts and disposable destinations.
4. Staging references the same digests and core services plus staging overlay values; it does not rebuild.
5. A release manifest records digests, committed database migration set, config schema version, validation summaries, and rollback compatibility.
6. Production references that release manifest and production overlay. Schema migration is a distinct gated job; application rollout follows readiness checks.
7. Rollback changes image digests and only rolls schema back when the committed migration explicitly supports it. Source archives and retired Directus remain outside this deployment model until separately authorized for retirement.