le-app/docs/le-app-database-migration/environment-parity.md

# Milestone 0: reproducible application environment

Status: proposal for review. It describes future files and services; it does not authorize production changes or privileged host configuration.

## Complete baseline runtime inventory

The inventory is based on commit `4884b5d`, its checked-in Compose/Dockerfiles, application routes, Directus deployment exports, and the frozen structural migration manifests.

| Component | Responsibility and dependencies | Persistence / lifecycle |
|---|---|---|
| `frontend` | Existing Next.js UI, middleware, server-rendered public pages, authenticated portal, compatibility APIs, file browser, SVGnest iframe, upload validation, Ko-fi webhook and claim endpoints, and BGBye proxy | Stateless image; Next cache on tmpfs; reads curated files; uses secure HTTP-only auth cookies |
| `payload` | Dedicated Payload admin, auth, content/media API, access control, hooks, jobs, and migrations | PostgreSQL plus Payload media volume; independent image and release lifecycle |
| `postgres` | Payload relational database | Named volume in nonproduction; managed/dedicated storage may replace it in production |
| `bgbye` | FastAPI image/video background removal with CUDA models, FFmpeg video processing, status polling, and ten-minute temporary-file cleanup task | Model caches are persistent; video/frame/output workspace is ephemeral; GPU is optional only if a functional CPU test profile is implemented |
| `proxy` | Single browser entrypoint, local TLS, host routing, forwarded headers, redirects, body limits, timeouts, and security headers | Configuration is versioned; certificates are mode-specific and never baked into application images |
| `mailpit` | Captures registration, recovery, verification, supporter-claim, and notification mail in nonproduction | Disposable by default; optional named volume for debugging; never deployed as the production relay |
| `migration` job image | Structural inventory, schema application, source import, file copy, sequence repair, checksums, and reports | Reads authorized source mount read-only; writes disposable PostgreSQL/media and non-personal summary reports |
| `validation` job image | Read-only source/destination comparison: counts, IDs, relationships, timestamps, ownership, permissions, files, checksums, and API contracts | No source writes; generated detailed reports are local and excluded from Git/build contexts |
| `e2e` job image | Browser and API tests against proxy-visible hostnames | Ephemeral browser artifacts; traces/screenshots are excluded when they may contain personal data |
| optional `legacy-directus` | Local reference for Studio presentation, permissions, flows, relations, files, and legacy API shapes | Uses a fresh writable MariaDB clone and copied uploads; never connects to the authoritative read-only source |
| optional `legacy-mariadb` | Writable database solely for the legacy reference service | Disposable named volume populated from the scrubbed dump |
| `fixture-seed` job | Loads deterministic synthetic users, content, ownership cases, and representative tiny files | Runs only against disposable fixture PostgreSQL/media |
| curated file library | Existing `app/files` manuals, specifications, software/reference files, and file-server routes | Separate read-only volume in runtime; approximately 461 MB / 282 files at baseline |
| Payload media | Migrated Directus uploads and future CMS uploads | Dedicated persistent volume/object-storage contract; not mixed with curated files |
| retained unreferenced uploads | 2,411 files pending classification | Read-only retention input; never deleted by application reset/import jobs |
| external Ko-fi | Sends signed webhook events and is the source for historical membership backfill | Nonproduction uses fixtures; no real webhook secret or outbound calls |
| production SMTP | Sends account and notification mail | Replaced by Mailpit in every nonproduction mode; credentials only in production secret scope |
| reverse-proxy/TLS provider | Public TLS and routing | Production-specific infrastructure around the same proxy contract |

### Existing application processes and integrations

- Authentication currently uses `ma_at`, `ma_rt`, `ma_ra`, and `ma_v` cookies, middleware redirects, periodic token validation, username-to-email lookup, recent-auth confirmation, registration, login, logout, and password change. Payload must add recovery and verification endpoints and mail templates because the baseline UI does not yet provide a complete recovery journey.
- Public/catalog consumers cover laser sources, four settings families, materials/coatings/hazards, buying guides, and projects.
- Authenticated consumers cover profile/avatar/password, projects and attachments, rigs, settings submission/deletion, claims, memberships/supporter badges, and account linking.
- Upload paths include avatars, project hero/attachments, settings photos/screenshots, Payload media, curated file downloads, and background-removal inputs/outputs.
- `app/scripts/backfill-kofi.mjs` is a remaining Directus consumer and becomes an idempotent Payload membership import job.
- `/api/webhooks/kofi`, supporter claim start/verify/unlink, and badge derivation become Payload-backed compatibility routes. Nonproduction uses signed fixture events only.
- Directus's two event flows (`unique_key_claims`, `claim_auto_assign`) must be reconstructed as transactional Payload hooks/jobs after behavior is resolved in the isolated legacy reference.
- BGBye has an in-process startup cleanup loop every ten minutes. There is no verified host cron/systemd job in the repository. Future scheduled Payload work must be an explicit worker/job service, not an undocumented host cron.
- SVGnest is vendored static client code embedded through the frontend. The separate `dashboard/` is a static legacy/operations dashboard in the monorepo but is not in the baseline Compose runtime; Milestone 0 must either add an explicit static service/profile or formally classify it as non-required. Default proposal: keep it as an optional `dashboard` profile, not part of the public application critical path.
- External links such as image hosting, paste, Forgejo, and community/support sites are navigation dependencies, not services required for the core stack to start. Smoke tests verify links are rendered but do not require those external systems.

## Common Compose architecture

The repository will use a project-name-scoped Compose model:

```text
infra/compose/compose.yaml              # core networks, services, health contracts
infra/compose/compose.dev.yaml          # bind mounts/watch/debug ports
infra/compose/compose.snapshot.yaml     # authorized read-only snapshot mounts
infra/compose/compose.staging.yaml      # staging hosts/resources/secrets/storage
infra/compose/compose.production.yaml   # production hosts/resources/external infrastructure
infra/compose/profiles/*.yaml           # legacy Directus, GPU/CPU, dashboard, test tooling
```

Core service definitions and Dockerfiles are shared. Mode selection composes files and profiles rather than copying service definitions.

| Mode | Services/data | Intended use |
|---|---|---|
| `fixture` | proxy, frontend, Payload, PostgreSQL, Mailpit, BGBye, fixture seed; optional dashboard | Daily development, CI, API tests, deterministic browser tests |
| `snapshot` | Same application images plus read-only scrubbed source/uploads and import/validation jobs | Realistic edge cases and file/API compatibility checks |
| `rehearsal` | Snapshot topology with destination reset, committed schema migration, full import, validation, then smoke/e2e gates | Repeatable cutover rehearsal; destructive only to project-scoped destination volumes |
| `staging` | Exact promoted image digests, staging PostgreSQL/media/secrets, capture or allow-listed staging SMTP | Production-shape acceptance and release candidate verification |
| `production` | Exact accepted digests, production PostgreSQL/media/SMTP/TLS/resources | Live service; no source snapshot, legacy Directus, fixture seed, Mailpit, or migration credentials mounted |

Compose health dependencies use explicit checks rather than startup order:

- proxy: configuration valid and HTTPS route responds;
- frontend: `/api/health/live` and `/api/health/ready` (Payload/BGBye dependency status separated);
- Payload: liveness plus database-backed readiness;
- PostgreSQL: `pg_isready`;
- BGBye: existing `/health`, expanded to distinguish model readiness;
- Mailpit: HTTP health endpoint;
- jobs: finite exit status and machine-readable summary.

No application service receives `.migration.env`. Only explicitly invoked inventory/import/validation jobs receive migration variables. Package installation, linting, builds, unit tests, fixture mode, frontend, Payload, proxy, Mailpit, and BGBye do not load it.

## Two nonproduction data modes

### Fixture mode

- A small, deterministic, non-personal dataset committed as factories/fixtures.
- Stable IDs exercise numeric IDs, UUIDs, ownership, public/private content, junctions, nulls, Unicode, timestamp boundaries, and authorization failures.
- Tiny generated fixtures cover JPEG, PNG, WebP/AVIF where supported, SVG rejection/handling, PDF, ZIP, plain text, and an executable-signature rejection case.
- Synthetic mailbox addresses use reserved domains and never identify real users.
- Resets remove only Compose-project-scoped PostgreSQL, media, mail, and temporary volumes.

### Snapshot mode

- Uses the credential-scrubbed MariaDB endpoint and authorized source/upload mounts read-only.
- Uses the same frontend, Payload, proxy, BGBye, and job images as fixture mode.
- Destination PostgreSQL and copied destination media are disposable and writable.
- Detailed validation artifacts stay local and ignored; committed reports contain structural summaries only.
- Outbound SMTP is forced to Mailpit and outbound network policy denies real SMTP/OAuth/Ko-fi delivery.

## Optional legacy Directus reference

The `legacy-reference` profile is local-only and off by default:

1. Verify the archived Directus image by immutable image ID/digest before use; never pull `latest` as a substitute.
2. Create a new project-scoped MariaDB volume and restore the scrubbed dump into it.
3. Copy uploads into a new writable project-scoped volume; never mount `.migration-source/uploads` writable.
4. Generate a new local `KEY`, `SECRET`, administrator address/password, and any test policy tokens. None may match production values.
5. Disable SMTP, OAuth, external webhooks, telemetry where supported, public registration, and outbound network access.
6. Bind only to the internal Compose network and route through the local proxy/admin hostname; do not publish its container port publicly.
7. Label the UI and responses clearly as scrubbed, disposable, local reference data.

This service is evidence, not an authoritative source. The frozen read-only MariaDB snapshot remains the migration authority. The archived image currently identified in the manifest is `sha256:dd5537…0801ce2a`; the complete value remains in the local manifest and must be supplied through an authorized local image import, not copied into a public registry without approval.

## Reverse proxy and browser parity

Proposed local endpoints use wildcard loopback names without editing DNS:

- `https://app.le.localhost:8443` — frontend and compatibility APIs;
- `https://cms.le.localhost:8443` — Payload admin/API;
- `https://mail.le.localhost:8443` — Mailpit UI;
- optional `https://legacy.le.localhost:8443` — isolated Directus reference.

The versioned proxy configuration must:

- set `Host`, `X-Forwarded-Host`, `X-Forwarded-Proto=https`, `X-Forwarded-Port`, `X-Real-IP`, and `X-Forwarded-For` consistently;
- support upgrade headers where an application endpoint needs them;
- reproduce legacy redirects and canonical-host behavior;
- set explicit per-route request limits: at least the verified historical 100 MB proxy ceiling, with lower application limits retained for avatars/settings/projects/BGBye;
- use 300-second upstream timeouts only where file/background processing requires them, with shorter defaults elsewhere;
- preserve streaming/download headers and range behavior for representative files;
- run application containers with fixed non-root UIDs and preflight media/file permissions;
- derive cookie `Secure`, `HttpOnly`, `SameSite`, domain, path, expiry, refresh, and recent-auth behavior from one shared auth contract.

Local HTTPS is required to test secure cookies. A repository-controlled development CA/certificate generator may create ignored local certificates, but trusting that CA in the host browser is a host change and requires user approval. Port 8443 avoids privileged rootless port binding; cookie semantics do not depend on port. Production uses the same routing contract with its real TLS termination and standard port 443.

## Safe email

- Mailpit is the only SMTP host supplied to fixture, snapshot, rehearsal, and default staging modes.
- Nonproduction secrets/config validation fails closed if an SMTP hostname, sender domain, or port matches a production allow/deny signature.
- Egress policy blocks TCP 25/465/587 from application containers in snapshot/rehearsal modes except to the internal Mailpit service.
- Tests retrieve messages through Mailpit's API and follow recovery/verification links through the local HTTPS proxy.
- Snapshot users are never sent messages automatically. Tests create synthetic accounts or redirect recipients at the mail transport boundary.
- Production SMTP credentials are provided only by the production secret mechanism and are never stored in Compose YAML, `.env` examples, images, test traces, or migration reports.

## Rootless Codex-controlled runtime

Required target: a dedicated rootless Docker daemon owned by the separate `le_app_codex` account. Future Codex migration sessions must execute as that account (preferred) or through a narrowly scoped account transition; the daemon socket is not shared broadly. Rootless Podman is an acceptable fallback only after Compose/profile behavior is proven; Docker is preferred because the repository and production already use Compose.

Codex must be able to build images, create/remove only its project containers/networks/volumes, inspect their logs, reset disposable PostgreSQL, run finite jobs, and execute integration/browser tests. It must not access `/var/run/docker.sock`, the production daemon/context, production containers/volumes/networks, `/var/www`, or `/srv/directus-archive`.

Current host observation:

- `sol6_vi` is not in the Docker group but can read the production application's world-readable environment files under `/var/www`;
- Docker and Docker Compose clients exist;
- no Podman client is installed;
- the default Docker context does not provide a usable daemon/security result to this user;
- no enabled user-level rootless Docker or Podman service was detected.

Therefore a `sol6_vi`-owned daemon is rejected: it could bind-mount those readable production secrets. Milestone 0 requires the isolated account and host changes described in `host-runtime-plan.md`, and work must stop before performing them. An administrator must approve/provision:

1. dedicated `le_app_codex` identity and restricted filesystem ACLs;
2. rootless prerequisites (`newuidmap`/`newgidmap`, subordinate UID/GID ranges, rootless networking and overlay helpers as required by the OS);
3. rootless Docker daemon installation for `le_app_codex` using its dedicated data root and named context;
4. optional user-service lingering if stacks must survive logout;
5. deterministic CPU BGBye first; optional GPU access only after core acceptance;
6. user/browser-profile-scoped development CA trust;
7. capacity limits for images, caches, snapshots, media, PostgreSQL, and test artifacts.

No production socket forwarding, Docker-group membership, broad sudo rule, daemon TCP exposure, or bind mount into production paths is acceptable.

## One-command workflows

The repository-root commands will be thin, reviewed wrappers around Compose and finite job containers:

```text
./scripts/stack-up fixture
./scripts/stack-up snapshot
./scripts/stack-down
./scripts/reset
./scripts/schema
./scripts/import
./scripts/validate
./scripts/smoke-test
```

Each wrapper validates the selected rootless context, project name, mode, required mounts, and forbidden production paths before doing work. `reset` requires an explicit nonproduction project label and refuses unknown database hosts. `stack-down` does not delete volumes by default. A separate explicit disposable purge command may remove only labeled project volumes.

From a clean checkout, fixture mode requires only the rootless runtime, ignored local secrets generated from examples, and cached/built images. Snapshot mode additionally requires the authorized read-only source path and migration environment. There are no undocumented restore, chmod, hostname, secret, or database steps.

## Build context and secret isolation

Every context (`app`, `payload`, `bgbye`, proxy, jobs/tests, and repository root where used) excludes:

- `.git`, `.worktrees`, `.migration-source`, `.migration.env`;
- `*.sql`, `*.sql.gz`, database dumps and snapshot manifests containing data;
- source and destination uploaded user files;
- migration-generated detailed reports/logs;
- browser traces/screenshots/videos from snapshot mode;
- local secrets, certificates/private keys, dependency trees, and build outputs.

Build verification inspects image histories and filesystems for forbidden names and secret fingerprints. Compose configuration tests assert that migration mounts/variables appear only on authorized finite jobs and never in production output.

## Proposed final repository layout

```text
/
├── apps/
│   ├── web/                         # current Next.js frontend + compatibility APIs
│   └── payload/                     # dedicated Payload admin/API
├── services/
│   ├── bgbye/                       # FastAPI GPU/CPU worker
│   ├── proxy/                       # versioned local/production routing contract
│   └── dashboard/                   # optional static dashboard profile
├── packages/
│   ├── contracts/                   # API DTOs, auth/cookie, errors, health contracts
│   ├── config/                      # validated environment schemas, no secret values
│   ├── fixtures/                    # deterministic synthetic factories and tiny files
│   └── test-helpers/                # API/browser/mail/file helpers
├── migration/
│   ├── inventory/                   # structural inventory tooling
│   ├── mappings/                    # reviewed field/relationship mappings
│   ├── schema/                      # committed Payload/Drizzle migrations
│   ├── import/                      # idempotent source/file import
│   ├── validation/                  # counts, relationships, ownership, checksums
│   └── legacy-reference/            # isolated restore/bootstrap tooling
├── infra/
│   ├── compose/                     # core and mode overlays/profiles
│   ├── proxy/                       # routes, headers, limits, local TLS config
│   ├── health/                      # healthcheck commands/contracts
│   └── env/                         # non-secret examples and mode schemas
├── scripts/                         # one-command wrappers and safety guards
├── tests/
│   ├── unit/
│   ├── integration/
│   ├── contracts/
│   ├── e2e/
│   └── fixtures/files/
├── docs/le-app-database-migration/
├── directus/                        # retained legacy evidence during migration
├── .nvmrc / .node-version
└── Dockerfiles remain beside each build context
```

The current `app`, `bgbye`, and `dashboard` directories will be moved only in a dedicated mechanical commit after build-context and path references are covered by tests. Milestone 0 may initially implement the target logical boundaries without a risky all-at-once directory move.

The repository tree above exists at exactly one persistent nonproduction checkout, `/srv/le-app-codex/lasereverything.net.db`, and at the canonical production root, `/var/www/lasereverything.net.db`. Runtime data is never placed inside either Git checkout unless it is a deliberately versioned non-secret fixture. The complete canonical host directory policy and transitional consolidation are defined in `host-runtime-plan.md`.

## End-to-end test matrix

| Area | Fixture mode | Snapshot/rehearsal mode | Core assertions |
|---|---|---|---|
| Registration | Full | Synthetic account only | validation, duplicate handling, role/status, captured verification mail |
| Login/logout | Full | Transitional-hash synthetic/test account; no personal output | username/email login, invalid credentials, refresh/expiry, cookie clearing |
| Password recovery | Full | Synthetic recipient only | enumeration resistance, token expiry/single use, Mailpit link, new login |
| Cookies/sessions | Full browser matrix | Full | Secure/HttpOnly/SameSite/path/domain, proxy scheme, redirects, refresh, recent-auth |
| Account/profile | Full | Structural/authorized test account | self-read/update, sensitive re-auth, avatar, cross-user denial |
| Projects | CRUD/files | Counts/contracts/representative records | ownership, public reads, attachment links, delete boundaries |
| Rigs | CRUD | Counts/contracts | owner presets and update/delete isolation |
| Machine settings | Four families | Full counts/relationships/API parity | create/read/update/delete ownership, images, `submission_id` compatibility |
| Claims/memberships | Full state fixtures | Flow/API parity | unique key, auto-assignment, pending transitions, supporter visibility |
| Upload/download | Representative file matrix | Realistic files/checksums | signatures, size limits, names, MIME, ranges, traversal prevention, permissions |
| Admin | Full | Read/controlled test writes | admin login, collections, roles, file management, forbidden member access |
| Background removal | Tiny image; deterministic allowed method | Representative image only | proxy body limit, model readiness, output PNG, invalid method/file, cleanup |
| Public utilities | Full | Full | toolkit, SVGnest, file browser, legacy redirects, external-link rendering |
| Compatibility APIs | Contract fixtures | Response-shape differential tests | Directus-like filters/fields/sort/pagination/errors where still consumed |
| Email | Mailpit | Mailpit only | no external delivery, correct recipient rewrite, recovery/verification/support flows |
| Authorization | Full adversarial matrix | Selected real relationship topology | anonymous/member/owner/admin/bot boundaries and field-level restrictions |
| Health/restart | Full | Full | dependency readiness, graceful restart, persistent DB/media, ephemeral temp data |

Snapshot test logs report identifiers only as one-way test-local aliases or aggregates; they do not emit emails, usernames, hashes, record bodies, or user-provided content.

## Promotion to staging and production

1. CI under Node 24.18.0 builds each application image once and records its digest and source commit.
2. Fixture integration/e2e tests run those digests through the common core Compose model.
3. Authorized snapshot rehearsal runs the same digests with read-only source mounts and disposable destinations.
4. Staging references the same digests and core services plus staging overlay values; it does not rebuild.
5. A release manifest records digests, committed database migration set, config schema version, validation summaries, and rollback compatibility.
6. Production references that release manifest and production overlay. Schema migration is a distinct gated job; application rollout follows readiness checks.
7. Rollback changes image digests and only rolls schema back when the committed migration explicitly supports it. Source archives and retired Directus remain outside this deployment model until separately authorized for retirement.