diff --git a/.dockerignore b/.dockerignore deleted file mode 100644 index 570a252..0000000 --- a/.dockerignore +++ /dev/null @@ -1,16 +0,0 @@ -.git -.worktrees -.migration-source -.migration.env -**/*.sql -**/*.sql.gz -**/node_modules -**/.next -payload/migration/generated -**/uploads -**/media -**/migration/generated -**/test-results -**/playwright-report -**/blob-report -**/.auth diff --git a/.gitignore b/.gitignore index 5ec752e..28ab8db 100644 --- a/.gitignore +++ b/.gitignore @@ -53,10 +53,3 @@ app/components/account.zip # Accidental root lockfile; applications have their own lockfiles /package-lock.json - -# Local migration inputs and isolated worktrees -.worktrees/ -.migration-source -.migration-source/ -.migration.env -payload/migration/generated/ diff --git a/.node-version b/.node-version deleted file mode 100644 index ca5c350..0000000 --- a/.node-version +++ /dev/null @@ -1 +0,0 @@ -24.18.0 diff --git a/.nvmrc b/.nvmrc deleted file mode 100644 index ca5c350..0000000 --- a/.nvmrc +++ /dev/null @@ -1 +0,0 @@ -24.18.0 diff --git a/app/.dockerignore b/app/.dockerignore index 5ade1b1..6c349af 100644 --- a/app/.dockerignore +++ b/app/.dockerignore @@ -17,14 +17,3 @@ debug *.bak *.zip .DS_Store - -.migration-source -.migration.env -.worktrees -*.sql -*.sql.gz -uploads -media -migration/generated -test-results -playwright-report diff --git a/bgbye/.dockerignore b/bgbye/.dockerignore deleted file mode 100644 index a7b0a1e..0000000 --- a/bgbye/.dockerignore +++ /dev/null @@ -1,21 +0,0 @@ -.git -.gitignore -.worktrees -.migration-source -.migration.env -node_modules -.next -.cache -.u2net -.ormbg -.models -temp_videos -uploads -media -migration -test-results -playwright-report -*.sql -*.sql.gz -*.log -*.bak diff --git a/docs/le-app-database-migration/architecture-proposal.md b/docs/le-app-database-migration/architecture-proposal.md deleted file mode 100644 index e48cf76..0000000 --- a/docs/le-app-database-migration/architecture-proposal.md +++ /dev/null @@ -1,152 +0,0 @@ -# Laser Everything database transition architecture - -Status: pre-implementation decision record. No production changes are authorized by this document. - -## Objective and Milestone 0 - -The target is a unified, reproducible application environment, not merely a database conversion. Unified means one repository, one versioned orchestration model, shared service contracts, consistent application images, and documented promotion from development through production. Services remain separate processes and containers. - -**Milestone 0: complete reproducible application environment** must be accepted and implemented before the large Payload/data-migration phase. It delivers: - -- complete service and persistence inventory for the `4884b5d` application; -- common Compose core plus mode-specific overlays/profiles; -- rootless, Codex-controlled nonproduction containers and volumes; -- production-like local HTTPS routing and cookie behavior; -- fixture and scrubbed-snapshot data modes using identical application images; -- safe captured email and outbound-network safeguards; -- optional isolated legacy Directus reference environment; -- deterministic one-command lifecycle, migration, validation, smoke, and browser-test workflows; -- image-by-digest promotion to staging and production. - -The detailed runtime inventory, orchestration design, repository tree, and test matrix are in [`environment-parity.md`](./environment-parity.md). The access audit, recommended account boundary, exact host prerequisites, ingress integration, storage, networking proof, and rollback plan are in [`host-runtime-plan.md`](./host-runtime-plan.md). The gated delivery sequence is in [`implementation-plan.md`](./implementation-plan.md). - -Milestone 0 host rule: nonproduction container isolation is not treated as a substitute for host filesystem permissions. The chosen daemon owner must be unable to read production paths and secrets, because a container controller can bind-mount anything readable to that daemon account. - -## Verified baseline - -The migration branch initially started from `main` at `0a7ee5f`. A tree and history comparison against the final pre-utilities production commit `4884b5d` established: - -- `main` is an ancestor of `4884b5d`. -- `4884b5d` contains exactly two later commits: `92ce8bc` and `4884b5d`. -- Those commits repair production registration/authentication behavior and add the verified Directus deployment contract, policies, flows, exports, and schema snapshot. -- The application was not replaced or reduced in those commits. - -Conclusion: `main` is a complete application tree, but `4884b5d` is the correct and more complete pre-utilities production baseline. The dedicated `migration/le-app-database` branch has therefore been fast-forwarded to `4884b5d`. The temporary utilities branch is not an ancestor of the migration work. - -## Service boundary - -The default architecture is three independent services: - -```text -browser - | - v -existing Next.js application (frontend + compatibility API) - | | - | server-to-server HTTP | background-removal calls - v v -dedicated Payload service existing bgbye service - | - v -dedicated PostgreSQL database -``` - -Payload owns content administration, authentication records, media metadata, access policy, hooks, and its generated APIs. The existing application remains the public frontend and initially retains its public API routes. Those routes become a compatibility layer backed by Payload, permitting endpoint-by-endpoint cutover and response-shape comparison. - -Deeply embedding Payload into the existing frontend would reduce one network boundary and allow direct Local API calls. It would also force the frontend onto Payload's exact Next.js dependency range, combine release and failure domains, increase the blast radius of admin changes, and make rollback harder. That integration is deferred pending explicit approval. - -The complete topology adds a reverse proxy, PostgreSQL, captured mail, migration/test jobs, and optional local-only Directus reference services around this accepted boundary. Payload remains independently deployable and is never merged into the frontend process by Milestone 0. - -## Runtime and dependency matrix - -| Component | Pinned version | Basis | -|---|---:|---| -| Node.js | 24.18.0 LTS (Krypton) | Development, Docker, CI, and production target | -| npm | 11.16.0 | Version shipped with the pinned Node line | -| Payload | 3.86.0 | Exact package version | -| `@payloadcms/next` | 3.86.0 | Must exactly match Payload | -| `@payloadcms/db-postgres` | 3.86.0 | Must exactly match Payload | -| Next.js (Payload service only) | 15.4.11 | Explicitly accepted by `@payloadcms/next@3.86.0` peer range | -| React / React DOM | 19.1.0 | Accepted by Next.js 15.4.11 and conservative relative to the existing app | - -The existing frontend's Next.js 15.3.2 is outside Payload 3.86.0's accepted ranges. This is another reason to keep Payload in a dedicated workspace. Package metadata was checked with exact-version `npm view`; the official Payload installation documentation lists 15.4.11 as an accepted line. Node 26 on the host is not a target runtime and must not be used to create authoritative lockfiles or release builds. - -References: - -- https://payloadcms.com/docs/getting-started/installation -- https://payloadcms.com/docs/database/postgres -- https://payloadcms.com/docs/database/migrations -- https://nodejs.org/en/blog/release/v24.18.0 - -## Schema lifecycle - -Schema push is permitted only while prototyping against the disposable database. Before the first authoritative rehearsal: - -1. Disable push in the PostgreSQL adapter. -2. Commit a generated initial Payload migration. -3. Keep four separate commands with single responsibilities: - - `migration:reset`: destructively empties only the disposable destination. - - `migration:schema`: applies committed Payload/Drizzle migrations. - - `migration:import`: imports source records and files. - - `migration:validate`: performs read-only cross-system checks. -4. Define `migration:rehearse` only as deterministic composition of those commands. - -Database credentials are loaded by database-specific scripts only. Installation, builds, linting, and type checking do not read `.migration.env`. - -## Collection strategy - -Payload will expose first-class collections for: - -- users and media; -- buying-guide categories, subcategories, and entries; -- hazards and material taxonomy; -- materials, coatings, colors, opacity, status, and their junctions; -- laser sources, software, scan lenses, focus lenses, configurations, apertures, expanders, and diameters; -- fiber, UV, CO2 galvo, and CO2 gantry settings; -- projects and project files; -- user rigs, rig types, preferences, memberships, and claims. - -Directus junction tables become Payload relationships where doing so preserves ordering and ownership semantics. A junction remains an explicit collection when it carries its own meaningful fields or permissions. - -## ID preservation decision - -Source identifiers will be preserved as the actual Payload document IDs, not merely copied into an auxiliary legacy field. - -- UUID primary keys remain UUID/string IDs. -- Integer and medium-integer primary keys remain numeric custom IDs. -- Collections whose Directus primary key is named `submission_id` use that numeric value as Payload's custom `id`; the compatibility layer returns it as `submission_id` to existing consumers. -- Payload's PostgreSQL adapter will enable `allowIDOnCreate` for controlled migration imports. -- Every relationship is imported using the preserved target ID, avoiding translation tables and making verification direct. -- New numeric IDs must begin above the migrated maximum. Sequence values are explicitly advanced after import. -- The import rejects duplicate, null, lossy, or out-of-range identifiers before inserting records. - -This mixed per-collection custom-ID approach is preferable to globally converting every integer to UUID, which would break public URLs, foreign keys, and API assumptions. Payload officially supports explicit custom ID fields, and the PostgreSQL adapter supports IDs supplied during controlled create operations. - -References: - -- https://payloadcms.com/docs/fields/overview#custom-id-fields -- https://payloadcms.com/docs/database/postgres - -## File strategy - -All 535 database-referenced originals and all 2,411 currently unreferenced physical files remain retained. Referenced files become Payload media documents while preserving the Directus file UUID, original filename, MIME type, size, timestamps, ownership where valid, and checksum. Derived variants are classified separately from originals. - -Unreferenced files are not imported as editorial media until classified, but they remain in a migration-retention manifest with path, size, type, and checksum. No file is deleted by reset or migration tooling. - -The repository's separate curated file library (`app/files`, 282 files and approximately 461 MB at the baseline) is not Directus upload storage. It remains a distinct, read-only application asset volume in normal operation. Payload media and retained migration uploads use separate volumes and permissions. - -## Production promotion rule - -CI builds the frontend, Payload, BGBye, and proxy images once from reviewed commits under their pinned runtimes. Integration tests run those exact digests. Staging and production consume the same core Compose definitions and immutable image digests; they do not rebuild from source or use separately maintained service definitions. Environment overlays may change secrets, hostnames, resource policies, external storage/database endpoints, replica counts, and production TLS integration, but not application image contents or service contracts. - -The permanent production checkout and deployment root is `/var/www/lasereverything.net.db`. It contains the complete versioned stack: frontend, Payload, BGBye, Compose definitions, migrations, deployment scripts, internal gateway contract, health checks, and validation tooling. No production checkout or application deployment may run from `/srv`, `/opt`, a Codex workspace, a temporary worktree, or another project root. Operational Docker layers, declared database/media volumes, and server-wide proxy files may live elsewhere, but the deployment rooted at the canonical production checkout declares and controls their contracts. - -The sole persistent nonproduction project root is `/srv/le-app-codex`, with the checkout at `/srv/le-app-codex/lasereverything.net.db`. The dedicated account design does not retain the current nested `.worktrees/payload-migration` layout. Temporary worktrees may be created under the canonical nonproduction root for a specific documented task and must be removed when that task ends. - -Production database schema changes use committed Payload migrations as an explicit release job before application rollout. Imports are migration-only jobs and are not part of ordinary application startup. Promotion requires recorded image digests, migration status, validation results, rollback compatibility, and health checks. - -Promotion never copies the nonproduction checkout or filesystem into production. A reviewed commit/release manifest is fetched into a clean checkout or unpacked as a verified release artifact at `/var/www/lasereverything.net.db`; production then pulls the already-tested immutable image digests. Development caches, local secrets, scrubbed snapshots, reports, test artifacts, rootless state, and disposable volumes are never promotion inputs. - -## Directus metadata disposition - -See `metadata-disposition.md` for the explicit include, translate, archive-only, and exclude decisions. Exclusion means “not imported into Payload runtime tables,” never “destroyed from the frozen snapshot or archive.” diff --git a/docs/le-app-database-migration/environment-parity.md b/docs/le-app-database-migration/environment-parity.md deleted file mode 100644 index 68ad642..0000000 --- a/docs/le-app-database-migration/environment-parity.md +++ /dev/null @@ -1,276 +0,0 @@ -# Milestone 0: reproducible application environment - -Status: proposal for review. It describes future files and services; it does not authorize production changes or privileged host configuration. - -## Complete baseline runtime inventory - -The inventory is based on commit `4884b5d`, its checked-in Compose/Dockerfiles, application routes, Directus deployment exports, and the frozen structural migration manifests. - -| Component | Responsibility and dependencies | Persistence / lifecycle | -|---|---|---| -| `frontend` | Existing Next.js UI, middleware, server-rendered public pages, authenticated portal, compatibility APIs, file browser, SVGnest iframe, upload validation, Ko-fi webhook and claim endpoints, and BGBye proxy | Stateless image; Next cache on tmpfs; reads curated files; uses secure HTTP-only auth cookies | -| `payload` | Dedicated Payload admin, auth, content/media API, access control, hooks, jobs, and migrations | PostgreSQL plus Payload media volume; independent image and release lifecycle | -| `postgres` | Payload relational database | Named volume in nonproduction; managed/dedicated storage may replace it in production | -| `bgbye` | FastAPI image/video background removal with CUDA models, FFmpeg video processing, status polling, and ten-minute temporary-file cleanup task | Model caches are persistent; video/frame/output workspace is ephemeral; GPU is optional only if a functional CPU test profile is implemented | -| `proxy` | Single browser entrypoint, local TLS, host routing, forwarded headers, redirects, body limits, timeouts, and security headers | Configuration is versioned; certificates are mode-specific and never baked into application images | -| `mailpit` | Captures registration, recovery, verification, supporter-claim, and notification mail in nonproduction | Disposable by default; optional named volume for debugging; never deployed as the production relay | -| `migration` job image | Structural inventory, schema application, source import, file copy, sequence repair, checksums, and reports | Reads authorized source mount read-only; writes disposable PostgreSQL/media and non-personal summary reports | -| `validation` job image | Read-only source/destination comparison: counts, IDs, relationships, timestamps, ownership, permissions, files, checksums, and API contracts | No source writes; generated detailed reports are local and excluded from Git/build contexts | -| `e2e` job image | Browser and API tests against proxy-visible hostnames | Ephemeral browser artifacts; traces/screenshots are excluded when they may contain personal data | -| optional `legacy-directus` | Local reference for Studio presentation, permissions, flows, relations, files, and legacy API shapes | Uses a fresh writable MariaDB clone and copied uploads; never connects to the authoritative read-only source | -| optional `legacy-mariadb` | Writable database solely for the legacy reference service | Disposable named volume populated from the scrubbed dump | -| `fixture-seed` job | Loads deterministic synthetic users, content, ownership cases, and representative tiny files | Runs only against disposable fixture PostgreSQL/media | -| curated file library | Existing `app/files` manuals, specifications, software/reference files, and file-server routes | Separate read-only volume in runtime; approximately 461 MB / 282 files at baseline | -| Payload media | Migrated Directus uploads and future CMS uploads | Dedicated persistent volume/object-storage contract; not mixed with curated files | -| retained unreferenced uploads | 2,411 files pending classification | Read-only retention input; never deleted by application reset/import jobs | -| external Ko-fi | Sends signed webhook events and is the source for historical membership backfill | Nonproduction uses fixtures; no real webhook secret or outbound calls | -| production SMTP | Sends account and notification mail | Replaced by Mailpit in every nonproduction mode; credentials only in production secret scope | -| reverse-proxy/TLS provider | Public TLS and routing | Production-specific infrastructure around the same proxy contract | - -### Existing application processes and integrations - -- Authentication currently uses `ma_at`, `ma_rt`, `ma_ra`, and `ma_v` cookies, middleware redirects, periodic token validation, username-to-email lookup, recent-auth confirmation, registration, login, logout, and password change. Payload must add recovery and verification endpoints and mail templates because the baseline UI does not yet provide a complete recovery journey. -- Public/catalog consumers cover laser sources, four settings families, materials/coatings/hazards, buying guides, and projects. -- Authenticated consumers cover profile/avatar/password, projects and attachments, rigs, settings submission/deletion, claims, memberships/supporter badges, and account linking. -- Upload paths include avatars, project hero/attachments, settings photos/screenshots, Payload media, curated file downloads, and background-removal inputs/outputs. -- `app/scripts/backfill-kofi.mjs` is a remaining Directus consumer and becomes an idempotent Payload membership import job. -- `/api/webhooks/kofi`, supporter claim start/verify/unlink, and badge derivation become Payload-backed compatibility routes. Nonproduction uses signed fixture events only. -- Directus's two event flows (`unique_key_claims`, `claim_auto_assign`) must be reconstructed as transactional Payload hooks/jobs after behavior is resolved in the isolated legacy reference. -- BGBye has an in-process startup cleanup loop every ten minutes. There is no verified host cron/systemd job in the repository. Future scheduled Payload work must be an explicit worker/job service, not an undocumented host cron. -- SVGnest is vendored static client code embedded through the frontend. The separate `dashboard/` is a static legacy/operations dashboard in the monorepo but is not in the baseline Compose runtime; Milestone 0 must either add an explicit static service/profile or formally classify it as non-required. Default proposal: keep it as an optional `dashboard` profile, not part of the public application critical path. -- External links such as image hosting, paste, Forgejo, and community/support sites are navigation dependencies, not services required for the core stack to start. Smoke tests verify links are rendered but do not require those external systems. - -## Common Compose architecture - -The repository will use a project-name-scoped Compose model: - -```text -infra/compose/compose.yaml # core networks, services, health contracts -infra/compose/compose.dev.yaml # bind mounts/watch/debug ports -infra/compose/compose.snapshot.yaml # authorized read-only snapshot mounts -infra/compose/compose.staging.yaml # staging hosts/resources/secrets/storage -infra/compose/compose.production.yaml # production hosts/resources/external infrastructure -infra/compose/profiles/*.yaml # legacy Directus, GPU/CPU, dashboard, test tooling -``` - -Core service definitions and Dockerfiles are shared. Mode selection composes files and profiles rather than copying service definitions. - -| Mode | Services/data | Intended use | -|---|---|---| -| `fixture` | proxy, frontend, Payload, PostgreSQL, Mailpit, BGBye, fixture seed; optional dashboard | Daily development, CI, API tests, deterministic browser tests | -| `snapshot` | Same application images plus read-only scrubbed source/uploads and import/validation jobs | Realistic edge cases and file/API compatibility checks | -| `rehearsal` | Snapshot topology with destination reset, committed schema migration, full import, validation, then smoke/e2e gates | Repeatable cutover rehearsal; destructive only to project-scoped destination volumes | -| `staging` | Exact promoted image digests, staging PostgreSQL/media/secrets, capture or allow-listed staging SMTP | Production-shape acceptance and release candidate verification | -| `production` | Exact accepted digests, production PostgreSQL/media/SMTP/TLS/resources | Live service; no source snapshot, legacy Directus, fixture seed, Mailpit, or migration credentials mounted | - -Compose health dependencies use explicit checks rather than startup order: - -- proxy: configuration valid and HTTPS route responds; -- frontend: `/api/health/live` and `/api/health/ready` (Payload/BGBye dependency status separated); -- Payload: liveness plus database-backed readiness; -- PostgreSQL: `pg_isready`; -- BGBye: existing `/health`, expanded to distinguish model readiness; -- Mailpit: HTTP health endpoint; -- jobs: finite exit status and machine-readable summary. - -No application service receives `.migration.env`. Only explicitly invoked inventory/import/validation jobs receive migration variables. Package installation, linting, builds, unit tests, fixture mode, frontend, Payload, proxy, Mailpit, and BGBye do not load it. - -## Two nonproduction data modes - -### Fixture mode - -- A small, deterministic, non-personal dataset committed as factories/fixtures. -- Stable IDs exercise numeric IDs, UUIDs, ownership, public/private content, junctions, nulls, Unicode, timestamp boundaries, and authorization failures. -- Tiny generated fixtures cover JPEG, PNG, WebP/AVIF where supported, SVG rejection/handling, PDF, ZIP, plain text, and an executable-signature rejection case. -- Synthetic mailbox addresses use reserved domains and never identify real users. -- Resets remove only Compose-project-scoped PostgreSQL, media, mail, and temporary volumes. - -### Snapshot mode - -- Uses the credential-scrubbed MariaDB endpoint and authorized source/upload mounts read-only. -- Uses the same frontend, Payload, proxy, BGBye, and job images as fixture mode. -- Destination PostgreSQL and copied destination media are disposable and writable. -- Detailed validation artifacts stay local and ignored; committed reports contain structural summaries only. -- Outbound SMTP is forced to Mailpit and outbound network policy denies real SMTP/OAuth/Ko-fi delivery. - -## Optional legacy Directus reference - -The `legacy-reference` profile is local-only and off by default: - -1. Verify the archived Directus image by immutable image ID/digest before use; never pull `latest` as a substitute. -2. Create a new project-scoped MariaDB volume and restore the scrubbed dump into it. -3. Copy uploads into a new writable project-scoped volume; never mount `.migration-source/uploads` writable. -4. Generate a new local `KEY`, `SECRET`, administrator address/password, and any test policy tokens. None may match production values. -5. Disable SMTP, OAuth, external webhooks, telemetry where supported, public registration, and outbound network access. -6. Bind only to the internal Compose network and route through the local proxy/admin hostname; do not publish its container port publicly. -7. Label the UI and responses clearly as scrubbed, disposable, local reference data. - -This service is evidence, not an authoritative source. The frozen read-only MariaDB snapshot remains the migration authority. The archived image currently identified in the manifest is `sha256:dd5537…0801ce2a`; the complete value remains in the local manifest and must be supplied through an authorized local image import, not copied into a public registry without approval. - -## Reverse proxy and browser parity - -Proposed local endpoints use wildcard loopback names without editing DNS: - -- `https://app.le.localhost:8443` — frontend and compatibility APIs; -- `https://cms.le.localhost:8443` — Payload admin/API; -- `https://mail.le.localhost:8443` — Mailpit UI; -- optional `https://legacy.le.localhost:8443` — isolated Directus reference. - -The versioned proxy configuration must: - -- set `Host`, `X-Forwarded-Host`, `X-Forwarded-Proto=https`, `X-Forwarded-Port`, `X-Real-IP`, and `X-Forwarded-For` consistently; -- support upgrade headers where an application endpoint needs them; -- reproduce legacy redirects and canonical-host behavior; -- set explicit per-route request limits: at least the verified historical 100 MB proxy ceiling, with lower application limits retained for avatars/settings/projects/BGBye; -- use 300-second upstream timeouts only where file/background processing requires them, with shorter defaults elsewhere; -- preserve streaming/download headers and range behavior for representative files; -- run application containers with fixed non-root UIDs and preflight media/file permissions; -- derive cookie `Secure`, `HttpOnly`, `SameSite`, domain, path, expiry, refresh, and recent-auth behavior from one shared auth contract. - -Local HTTPS is required to test secure cookies. A repository-controlled development CA/certificate generator may create ignored local certificates, but trusting that CA in the host browser is a host change and requires user approval. Port 8443 avoids privileged rootless port binding; cookie semantics do not depend on port. Production uses the same routing contract with its real TLS termination and standard port 443. - -## Safe email - -- Mailpit is the only SMTP host supplied to fixture, snapshot, rehearsal, and default staging modes. -- Nonproduction secrets/config validation fails closed if an SMTP hostname, sender domain, or port matches a production allow/deny signature. -- Egress policy blocks TCP 25/465/587 from application containers in snapshot/rehearsal modes except to the internal Mailpit service. -- Tests retrieve messages through Mailpit's API and follow recovery/verification links through the local HTTPS proxy. -- Snapshot users are never sent messages automatically. Tests create synthetic accounts or redirect recipients at the mail transport boundary. -- Production SMTP credentials are provided only by the production secret mechanism and are never stored in Compose YAML, `.env` examples, images, test traces, or migration reports. - -## Rootless Codex-controlled runtime - -Required target: a dedicated rootless Docker daemon owned by the separate `le_app_codex` account. Future Codex migration sessions must execute as that account (preferred) or through a narrowly scoped account transition; the daemon socket is not shared broadly. Rootless Podman is an acceptable fallback only after Compose/profile behavior is proven; Docker is preferred because the repository and production already use Compose. - -Codex must be able to build images, create/remove only its project containers/networks/volumes, inspect their logs, reset disposable PostgreSQL, run finite jobs, and execute integration/browser tests. It must not access `/var/run/docker.sock`, the production daemon/context, production containers/volumes/networks, `/var/www`, or `/srv/directus-archive`. - -Current host observation: - -- `sol6_vi` is not in the Docker group but can read the production application's world-readable environment files under `/var/www`; -- Docker and Docker Compose clients exist; -- no Podman client is installed; -- the default Docker context does not provide a usable daemon/security result to this user; -- no enabled user-level rootless Docker or Podman service was detected. - -Therefore a `sol6_vi`-owned daemon is rejected: it could bind-mount those readable production secrets. Milestone 0 requires the isolated account and host changes described in `host-runtime-plan.md`, and work must stop before performing them. An administrator must approve/provision: - -1. dedicated `le_app_codex` identity and restricted filesystem ACLs; -2. rootless prerequisites (`newuidmap`/`newgidmap`, subordinate UID/GID ranges, rootless networking and overlay helpers as required by the OS); -3. rootless Docker daemon installation for `le_app_codex` using its dedicated data root and named context; -4. optional user-service lingering if stacks must survive logout; -5. deterministic CPU BGBye first; optional GPU access only after core acceptance; -6. user/browser-profile-scoped development CA trust; -7. capacity limits for images, caches, snapshots, media, PostgreSQL, and test artifacts. - -No production socket forwarding, Docker-group membership, broad sudo rule, daemon TCP exposure, or bind mount into production paths is acceptable. - -## One-command workflows - -The repository-root commands will be thin, reviewed wrappers around Compose and finite job containers: - -```text -./scripts/stack-up fixture -./scripts/stack-up snapshot -./scripts/stack-down -./scripts/reset -./scripts/schema -./scripts/import -./scripts/validate -./scripts/smoke-test -``` - -Each wrapper validates the selected rootless context, project name, mode, required mounts, and forbidden production paths before doing work. `reset` requires an explicit nonproduction project label and refuses unknown database hosts. `stack-down` does not delete volumes by default. A separate explicit disposable purge command may remove only labeled project volumes. - -From a clean checkout, fixture mode requires only the rootless runtime, ignored local secrets generated from examples, and cached/built images. Snapshot mode additionally requires the authorized read-only source path and migration environment. There are no undocumented restore, chmod, hostname, secret, or database steps. - -## Build context and secret isolation - -Every context (`app`, `payload`, `bgbye`, proxy, jobs/tests, and repository root where used) excludes: - -- `.git`, `.worktrees`, `.migration-source`, `.migration.env`; -- `*.sql`, `*.sql.gz`, database dumps and snapshot manifests containing data; -- source and destination uploaded user files; -- migration-generated detailed reports/logs; -- browser traces/screenshots/videos from snapshot mode; -- local secrets, certificates/private keys, dependency trees, and build outputs. - -Build verification inspects image histories and filesystems for forbidden names and secret fingerprints. Compose configuration tests assert that migration mounts/variables appear only on authorized finite jobs and never in production output. - -## Proposed final repository layout - -```text -/ -├── apps/ -│ ├── web/ # current Next.js frontend + compatibility APIs -│ └── payload/ # dedicated Payload admin/API -├── services/ -│ ├── bgbye/ # FastAPI GPU/CPU worker -│ ├── proxy/ # versioned local/production routing contract -│ └── dashboard/ # optional static dashboard profile -├── packages/ -│ ├── contracts/ # API DTOs, auth/cookie, errors, health contracts -│ ├── config/ # validated environment schemas, no secret values -│ ├── fixtures/ # deterministic synthetic factories and tiny files -│ └── test-helpers/ # API/browser/mail/file helpers -├── migration/ -│ ├── inventory/ # structural inventory tooling -│ ├── mappings/ # reviewed field/relationship mappings -│ ├── schema/ # committed Payload/Drizzle migrations -│ ├── import/ # idempotent source/file import -│ ├── validation/ # counts, relationships, ownership, checksums -│ └── legacy-reference/ # isolated restore/bootstrap tooling -├── infra/ -│ ├── compose/ # core and mode overlays/profiles -│ ├── proxy/ # routes, headers, limits, local TLS config -│ ├── health/ # healthcheck commands/contracts -│ └── env/ # non-secret examples and mode schemas -├── scripts/ # one-command wrappers and safety guards -├── tests/ -│ ├── unit/ -│ ├── integration/ -│ ├── contracts/ -│ ├── e2e/ -│ └── fixtures/files/ -├── docs/le-app-database-migration/ -├── directus/ # retained legacy evidence during migration -├── .nvmrc / .node-version -└── Dockerfiles remain beside each build context -``` - -The current `app`, `bgbye`, and `dashboard` directories will be moved only in a dedicated mechanical commit after build-context and path references are covered by tests. Milestone 0 may initially implement the target logical boundaries without a risky all-at-once directory move. - -The repository tree above exists at exactly one persistent nonproduction checkout, `/srv/le-app-codex/lasereverything.net.db`, and at the canonical production root, `/var/www/lasereverything.net.db`. Runtime data is never placed inside either Git checkout unless it is a deliberately versioned non-secret fixture. The complete canonical host directory policy and transitional consolidation are defined in `host-runtime-plan.md`. - -## End-to-end test matrix - -| Area | Fixture mode | Snapshot/rehearsal mode | Core assertions | -|---|---|---|---| -| Registration | Full | Synthetic account only | validation, duplicate handling, role/status, captured verification mail | -| Login/logout | Full | Transitional-hash synthetic/test account; no personal output | username/email login, invalid credentials, refresh/expiry, cookie clearing | -| Password recovery | Full | Synthetic recipient only | enumeration resistance, token expiry/single use, Mailpit link, new login | -| Cookies/sessions | Full browser matrix | Full | Secure/HttpOnly/SameSite/path/domain, proxy scheme, redirects, refresh, recent-auth | -| Account/profile | Full | Structural/authorized test account | self-read/update, sensitive re-auth, avatar, cross-user denial | -| Projects | CRUD/files | Counts/contracts/representative records | ownership, public reads, attachment links, delete boundaries | -| Rigs | CRUD | Counts/contracts | owner presets and update/delete isolation | -| Machine settings | Four families | Full counts/relationships/API parity | create/read/update/delete ownership, images, `submission_id` compatibility | -| Claims/memberships | Full state fixtures | Flow/API parity | unique key, auto-assignment, pending transitions, supporter visibility | -| Upload/download | Representative file matrix | Realistic files/checksums | signatures, size limits, names, MIME, ranges, traversal prevention, permissions | -| Admin | Full | Read/controlled test writes | admin login, collections, roles, file management, forbidden member access | -| Background removal | Tiny image; deterministic allowed method | Representative image only | proxy body limit, model readiness, output PNG, invalid method/file, cleanup | -| Public utilities | Full | Full | toolkit, SVGnest, file browser, legacy redirects, external-link rendering | -| Compatibility APIs | Contract fixtures | Response-shape differential tests | Directus-like filters/fields/sort/pagination/errors where still consumed | -| Email | Mailpit | Mailpit only | no external delivery, correct recipient rewrite, recovery/verification/support flows | -| Authorization | Full adversarial matrix | Selected real relationship topology | anonymous/member/owner/admin/bot boundaries and field-level restrictions | -| Health/restart | Full | Full | dependency readiness, graceful restart, persistent DB/media, ephemeral temp data | - -Snapshot test logs report identifiers only as one-way test-local aliases or aggregates; they do not emit emails, usernames, hashes, record bodies, or user-provided content. - -## Promotion to staging and production - -1. CI under Node 24.18.0 builds each application image once and records its digest and source commit. -2. Fixture integration/e2e tests run those digests through the common core Compose model. -3. Authorized snapshot rehearsal runs the same digests with read-only source mounts and disposable destinations. -4. Staging references the same digests and core services plus staging overlay values; it does not rebuild. -5. A release manifest records digests, committed database migration set, config schema version, validation summaries, and rollback compatibility. -6. Production references that release manifest and production overlay. Schema migration is a distinct gated job; application rollout follows readiness checks. -7. Rollback changes image digests and only rolls schema back when the committed migration explicitly supports it. Source archives and retired Directus remain outside this deployment model until separately authorized for retirement. diff --git a/docs/le-app-database-migration/host-runtime-plan.md b/docs/le-app-database-migration/host-runtime-plan.md deleted file mode 100644 index 2482f48..0000000 --- a/docs/le-app-database-migration/host-runtime-plan.md +++ /dev/null @@ -1,567 +0,0 @@ -# Milestone 0 host runtime and security plan - -Status: planning-only. Commands in this document are proposed for administrator review. None have been executed. - -## Permissions-only access audit - -The audit inspected only existence, filesystem type, owner/mode, effective read/write/traverse permission, Docker context endpoints, and socket metadata. It did not read, print, copy, or inspect secret values, production databases, container metadata, or archive contents. - -| Boundary | Effective access for `sol6_vi` | Finding | -|---|---|---| -| `/var/www` | read/traverse, not write | Production tree names and readable files are exposed to the account | -| `/var/www/lasereverything.net.db` and `app/` | read/traverse, not write | Production application and deployment configuration are readable | -| production `.env.local` and BGBye `.env` | read, not write; mode `0644` | **Unsafe for a `sol6_vi`-owned rootless daemon**; values were not read | -| `/var/lib/docker` | no read/write/traverse | Production Docker data is denied by host permissions | -| `/var/lib/mysql` | no read/write/traverse | Production database storage is denied | -| `/var/run/docker.sock` and `/run/docker.sock` | no read/write | Production daemon is denied; only the `default` context points to it | -| `/srv/directus-archive` | no read/write/traverse | Root-only archive boundary works; timestamped child is inaccessible | -| `/srv/codex-migration` | read/traverse, not write | Scrubbed migration source is available read-only | -| `/srv/codex-secrets` | read/traverse at directory level, not write | Migration env is readable as intended; access must be narrowed for the new account to the one nonproduction file | -| workspace | read/write | Current development tree is owned by `sol6_vi` | - -Conclusion: rootless Docker owned by `sol6_vi` would protect the production daemon but not production files already readable to that user. Any controller of that daemon could mount the readable production environment files. The production secret file modes also deserve independent remediation. - -## Runtime account model comparison - -| Model | Advantages | Risks / cost | Decision | -|---|---|---|---| -| `sol6_vi` owns rootless Docker | Minimal provisioning; current workspace already writable | Can bind-mount world-readable production secrets and application files; mixes general Codex work with container authority | Rejected on this host | -| dedicated `le_app_codex` | Daemon owner has an explicit filesystem allow-list; project storage and cleanup are attributable; production paths can be proven inaccessible | Requires account, ACL, subordinate IDs, separate Git credentials/session, storage, and user service | **Recommended** | - -Future autonomous migration work should run in a Codex session whose OS identity is `le_app_codex`. Do not grant `sol6_vi` generic access to the development daemon socket as a convenience; that weakens accountability. If orchestration must be initiated from another account, use a narrowly audited command wrapper or SSH into `le_app_codex`, not a shared Docker group and not TCP exposure of the daemon. - -## Exact proposed identity and filesystem model - -Proposed fixed identity: - -- user and primary group: `le_app_codex`; -- UID/GID: `1200` (verified unused at audit time; recheck immediately before creation); -- shell/home: `/bin/bash`, `/srv/le-app-codex/home`; -- no supplementary production groups, no `docker` group, no sudo/wheel membership; -- subordinate UID and GID range: `165536:65536`, non-overlapping with the observed `sol6_ii:100000:65536` allocation; -- dedicated project root: Btrfs subvolume `/srv/le-app-codex`, owner `root:le_app_codex`, mode `0750`; -- sole checkout: `/srv/le-app-codex/lasereverything.net.db`; -- authoritative scrubbed input: `/srv/le-app-codex/database-source-readonly`, root-owned and read-only to the development group; -- nonproduction credentials only: `/srv/le-app-codex/nonproduction-secrets`; -- daemon data root: `/srv/le-app-codex/container-runtime/docker`; -- runtime socket: `/run/user/1200/docker.sock`; -- Docker client/config: `/srv/le-app-codex/home/.docker` and `/srv/le-app-codex/home/.config/docker`; -- rootless Docker context: `le-app-codex`; -- default nonproduction Compose project: `le-app-testing`, enforced by wrapper preflight. - -Verified candidate availability at this planning checkpoint: - -- neither account nor group name `le_app_codex` exists; -- UID `1200` is unused in `/etc/passwd` and GID `1200` is unused in `/etc/group`; -- no passwd primary UID/GID or group GID collision exists at `1200`; -- `/etc/subuid` and `/etc/subgid` contain one observed allocation, `100000–165535`, owned by the historical `sol6_ii` entry; -- candidate range `165536–231071` does not overlap any entry in either subordinate-ID file; -- `/srv/le-app-codex` does not exist. - -Therefore the current exact recommendation is UID/GID `1200` and subordinate range `165536:65536`, subject to rerunning the documented collision checks immediately before provisioning. If any check produces output or a non-free result, stop and select new IDs; do not force reuse. - -## Canonical naming replacements - -| Superseded proposal | Canonical replacement | -|---|---| -| account/group `le_payload_dev` | `le_app_codex` | -| nonproduction root `/srv/le-payload-dev` | `/srv/le-app-codex` | -| checkout `/srv/le-payload-dev/lasereverything.net.db` | `/srv/le-app-codex/lasereverything.net.db` | -| source directory `source` | `database-source-readonly` | -| secrets directory `secrets` | `nonproduction-secrets` | -| runtime directory `runtime` | `container-runtime` | -| disposable-data directory `data` | `testing-data` | -| artifact directory `artifacts` | `build-artifacts` | -| Docker context `le-payload-dev` | `le-app-codex` | -| Compose project `le_payload_dev` | `le-app-testing` | -| branch `migration/payload` | `migration/le-app-database` | -| documentation `docs/payload-migration` | `docs/le-app-database-migration` | - -`payload` remains valid only for the actual Payload CMS service, application directory, packages, configuration, and Payload-specific migrations. Historical inventory names such as `le-payload-pg` and `.worktrees/payload-migration` remain unchanged because they identify real transitional resources. - -Canonical nonproduction tree: - -```text -/srv/le-app-codex/ -├── home/ # dedicated account home and user configuration -├── lasereverything.net.db/ # sole persistent nonproduction Git checkout -├── database-source-readonly/ # root-owned scrubbed authoritative input -├── nonproduction-secrets/ # testing/development credentials only -├── container-runtime/ # rootless Docker images, layers and configuration -├── testing-data/ # disposable databases, media, models and caches -└── build-artifacts/ # reports, logs and browser/test evidence -``` - -Ownership and permissions: - -| Path | Owner:group | Mode/policy | -|---|---|---| -| `/srv/le-app-codex` | `root:le_app_codex` | `0750`; prevents unrelated users and prevents project-root restructuring by the development account | -| `home` | `le_app_codex:le_app_codex` | `0700`; account, Docker client, user systemd, NSS/browser configuration | -| `lasereverything.net.db` | `le_app_codex:le_app_codex` | directories `0750`, normal Git files subject to repository modes; sole checkout | -| `database-source-readonly` | `root:le_app_codex` | directories `0550`, files `0440`; account and daemon may read but cannot alter; no writable symlink target | -| `nonproduction-secrets` | `root:le_app_codex` | directory `0750`, provisioned files `0640`; nonproduction values only; optional `generated/` is `0700 le_app_codex` for ephemeral fixture secrets | -| `container-runtime` | `le_app_codex:le_app_codex` | `0700`; rootless Docker images, layers and configuration only | -| `testing-data` | `le_app_codex:le_app_codex` | `0700`; child ownership may use rootless subordinate IDs | -| `build-artifacts` | `le_app_codex:le_app_codex` | `0700`; TTL cleanup; snapshot artifacts treated as sensitive | - -Only the dedicated account receives: - -- read/write access to its workspace and runtime/storage subtree; -- group read/traverse access to canonical root-owned `database-source-readonly`; -- group read access to specifically provisioned files under canonical `nonproduction-secrets`; -- a separate nonproduction Git credential/deploy key with access limited to the application repository. - -It receives no ACL or group membership for `/var/www`, production database paths, production sockets/volumes, or `/srv/directus-archive`. - -## Exact privileged prerequisites and commands - -These commands are a reviewable proposal for an administrator on Garuda/Arch Linux. Recheck UID/GID and subordinate ranges first. Do not paste secrets into the shell history. - -```bash -# 1. Recheck names, IDs, and ranges across all four account databases. -getent passwd le_app_codex -getent group le_app_codex -getent passwd 1200 -getent group 1200 -awk -F: '$3 == 1200 || $4 == 1200 { print; found=1 } END { exit found ? 1 : 0 }' /etc/passwd -awk -F: '$3 == 1200 { print; found=1 } END { exit found ? 1 : 0 }' /etc/group -awk -F: 'BEGIN { a=165536; b=231071 } - { x=$2; y=$2+$3-1; if (a<=y && x<=b) { print FILENAME ":" $0; found=1 } } - END { exit found ? 1 : 0 }' /etc/subuid /etc/subgid - -# 2. Install rootless prerequisites. docker-rootless-extras is an Arch AUR -# package and must be built/reviewed under the administrator's normal AUR policy. -sudo pacman -S --needed rootlesskit slirp4netns fuse-overlayfs shadow -# Then install the reviewed docker-rootless-extras AUR package by the host's -# approved AUR procedure; do not curl a remote install script into a shell. - -# 3. Create the isolated primary group and project subvolume. -sudo groupadd --gid 1200 le_app_codex -sudo btrfs subvolume create /srv/le-app-codex -sudo chown root:le_app_codex /srv/le-app-codex -sudo chmod 0750 /srv/le-app-codex - -# 4. Create the account with its canonical home inside the project root. -sudo useradd --uid 1200 --gid le_app_codex --create-home \ - --home-dir /srv/le-app-codex/home --shell /bin/bash le_app_codex -sudo usermod --add-subuids 165536-231071 --add-subgids 165536-231071 le_app_codex - -# 5. Create the canonical children; no sibling workspace is created. -sudo mkdir -p \ - /srv/le-app-codex/lasereverything.net.db \ - /srv/le-app-codex/database-source-readonly \ - /srv/le-app-codex/nonproduction-secrets/generated \ - /srv/le-app-codex/container-runtime/docker \ - /srv/le-app-codex/testing-data/{postgres,media,legacy-uploads,bgbye-models,tmp} \ - /srv/le-app-codex/build-artifacts/{browser,migration-reports} - -# Root owns the boundary and authoritative source; development owns mutable areas. -sudo chown root:le_app_codex /srv/le-app-codex -sudo chmod 0750 /srv/le-app-codex -sudo chown le_app_codex:le_app_codex /srv/le-app-codex/home -sudo chmod 0700 /srv/le-app-codex/home -sudo chown -R le_app_codex:le_app_codex \ - /srv/le-app-codex/lasereverything.net.db \ - /srv/le-app-codex/container-runtime /srv/le-app-codex/testing-data /srv/le-app-codex/build-artifacts \ - /srv/le-app-codex/nonproduction-secrets/generated -sudo chmod 0700 /srv/le-app-codex/container-runtime /srv/le-app-codex/testing-data \ - /srv/le-app-codex/build-artifacts /srv/le-app-codex/nonproduction-secrets/generated -sudo chown root:le_app_codex /srv/le-app-codex/database-source-readonly /srv/le-app-codex/nonproduction-secrets -sudo chmod 0550 /srv/le-app-codex/database-source-readonly -sudo chmod 0750 /srv/le-app-codex/nonproduction-secrets - -# 6. Apply a 250 GiB ceiling to the project subvolume. -sudo btrfs quota enable /srv -sudo btrfs qgroup limit 250G /srv/le-app-codex - -# 7. Transitional source access is used only by the administrator-run copy and -# verification procedure below. The final account needs no ACL on /srv/codex-*. - -# 8. Remediate production secret readability separately. -# Root-owned Compose can still read mode 0600 files. Confirm the production -# operator before applying because this changes production file permissions. -sudo chown root:root /var/www/lasereverything.net.db/app/.env.local -sudo chmod 0600 /var/www/lasereverything.net.db/app/.env.local -sudo chown root:root /var/www/lasereverything.net.db/bgbye/.env -sudo chmod 0600 /var/www/lasereverything.net.db/bgbye/.env - -# 9. As le_app_codex, configure the rootless daemon with an explicit data root. -sudo -iu le_app_codex mkdir -p ~/.config/docker -# Administrator writes ~/.config/docker/daemon.json as documented below, -# then the dedicated user enables the packaged rootless user service. -sudo -iu le_app_codex systemctl --user enable --now docker.socket - -# 10. Optional only if reviewed stacks must survive logout. -sudo loginctl enable-linger le_app_codex -``` - -Proposed `/srv/le-app-codex/home/.config/docker/daemon.json`: - -```json -{ - "data-root": "/srv/le-app-codex/container-runtime/docker", - "live-restore": false, - "log-driver": "local", - "log-opts": { "max-size": "20m", "max-file": "5" } -} -``` - -The exact Arch rootless unit name (`docker.service` versus `docker.socket`) must be verified from the reviewed package before enabling. Cgroup v2 is present; after provisioning, verify the rootless/security options and delegated controllers before relying on CPU, memory, or PID limits. - -The nonstandard home is compatible with rootless Docker and user-level systemd because it is the account home recorded in `/etc/passwd`, resides on local Btrfs, and is owned by the account. Required environment contract: - -- `HOME=/srv/le-app-codex/home` comes from the login/account record; -- `XDG_RUNTIME_DIR=/run/user/1200` is created by logind/systemd and must not be redirected into `/srv` or `/tmp`; -- `XDG_DATA_HOME=/srv/le-app-codex/home/.local/share`; -- Docker daemon data root is `/srv/le-app-codex/container-runtime/docker`; -- Docker client configuration is `/srv/le-app-codex/home/.docker`; -- user systemd configuration is `/srv/le-app-codex/home/.config/systemd/user`; -- the `le-app-codex` context points only to `unix:///run/user/1200/docker.sock`; -- wrappers export `COMPOSE_PROJECT_NAME=le-app-testing`. - -Proposed `/srv/le-app-codex/home/.config/environment.d/10-le-app-codex.conf`: - -```text -HOME=/srv/le-app-codex/home -XDG_DATA_HOME=/srv/le-app-codex/home/.local/share -DOCKER_CONFIG=/srv/le-app-codex/home/.docker -COMPOSE_PROJECT_NAME=le-app-testing -``` - -`XDG_RUNTIME_DIR` is deliberately absent from this file; logind must create `/run/user/1200` with the correct lifetime and ownership. `DOCKER_HOST` is also omitted so the named context remains the explicit authority and scripts can reject context drift. - -After the user service is installed, create the context as the dedicated account: - -```bash -sudo -iu le_app_codex docker context create le-app-codex \ - --description 'Laser Everything Codex nonproduction rootless runtime' \ - --docker host=unix:///run/user/1200/docker.sock -sudo -iu le_app_codex docker context use le-app-codex -``` - -## Required positive and negative verification - -Run as `le_app_codex`, printing only permissions/status: - -```bash -test ! -r /var/www/lasereverything.net.db/app/.env.local -test ! -x /var/lib/docker -test ! -x /var/lib/mysql -test ! -r /var/run/docker.sock -test ! -x /srv/directus-archive -test -r /srv/le-app-codex/database-source-readonly/README.txt -test ! -w /srv/le-app-codex/database-source-readonly/README.txt -test -r /srv/le-app-codex/nonproduction-secrets/migration.env -test ! -w /srv/le-app-codex/nonproduction-secrets/migration.env -DOCKER_HOST=unix:///run/user/1200/docker.sock docker info --format '{{json .SecurityOptions}}' -``` - -The final line must show rootless security. The development context must resolve only to `/run/user/1200/docker.sock`; wrappers reject `default`, `/var/run/docker.sock`, TCP endpoints, SSH contexts, and any unknown endpoint. - -## Production path and socket denial preflight - -Every stack/job wrapper will fail before Compose evaluation if: - -- effective UID is not the approved dedicated UID; -- Docker endpoint is not exactly the approved rootless Unix socket; -- Compose project name lacks the fixed nonproduction prefix; -- active Docker context is not exactly `le-app-codex`; -- any resolved bind source is `/var/www`, `/var/lib/docker`, `/var/lib/mysql`, `/srv/directus-archive`, `/var/run`, `/run/docker.sock`, or a descendant; -- the Git checkout is not exactly `/srv/le-app-codex/lasereverything.net.db` in nonproduction or `/var/www/lasereverything.net.db` in production; -- any non-finite application service receives `.migration.env`, `.migration-source`, a MariaDB source URL, or migration credentials; -- destination database host/port is not the approved nonproduction target/service; -- production-named secret scopes, SMTP hosts, OAuth variables, Ko-fi secrets, or webhook credentials appear in fixture/snapshot/rehearsal rendered Compose; -- reset/purge targets a bind mount, external volume, unknown project label, or any path outside `/srv/le-app-codex`. - -Tests render each Compose mode and inspect mounts, secrets, environment variable **names/classifications**, networks, contexts, labels, and volume ownership without printing values. - -## Authoritative migration-source protection - -- The authoritative MariaDB remains read-only at the database privilege and filesystem layers. -- `.migration-source` and `.migration.env` are mounted read-only only into finite `inventory`, `import`, and `validation` jobs. -- Frontend, Payload, BGBye, proxy, Mailpit, fixture seed, package installation, build, lint, unit, integration, and ordinary browser-test services never receive those mounts or variables. -- Legacy Directus restores the scrubbed SQL into a separate project-scoped writable MariaDB volume and copies uploads into `/srv/le-app-codex/testing-data/legacy-uploads` or a project volume. -- Legacy Directus never mounts the source database storage or uploads, even read-only; the finite clone job is the only bridge. -- Reset/purge operates only on labeled nonproduction destinations. It does not follow source symlinks and refuses all external/bind volumes. -- `/srv/directus-archive` remains inaccessible and is never mentioned as a mount source in executable Compose configuration. - -The complete root-only archive remains permanently outside the development boundary at `/srv/directus-archive/20260710-175408`. It is never moved, mounted, linked, or copied into `/srv/le-app-codex`. - -## Nonproduction storage plan - -The host currently has approximately 650 GiB available on the Btrfs filesystem containing `/srv` and `/home`. Proposed aggregate project quota: 250 GiB. - -| Path | Planning allowance | Ownership | Cleanup / backup | -|---|---:|---|---| -| `container-runtime/docker` | 80 GiB soft budget | `le_app_codex` | prune only project/unused development images after digest recording; no backup | -| `testing-data/postgres` | 25 GiB | `le_app_codex` / rootless mapped IDs | disposable; reset by labeled volume; no backup | -| `testing-data/media` | 15 GiB | fixed container UID mapped by rootless daemon | fixture disposable; snapshot copied data disposable; no authoritative backup role | -| `testing-data/legacy-uploads` | 10 GiB | rootless mapped service UID | copied from scrubbed source; disposable; never source of truth | -| `testing-data/bgbye-models` | 35 GiB | BGBye service/rootless mapping | checksummed cache; LRU/manual cleanup; redownloadable | -| `testing-data/tmp` | 10 GiB | service-specific mapped UID | tmpfs or age-based cleanup; no backup | -| `build-artifacts/browser` | 5 GiB | `le_app_codex` | TTL 7 days; snapshot artifacts treated as sensitive and never committed | -| `build-artifacts/migration-reports` | 10 GiB | `le_app_codex` | detailed reports TTL 30 days; committed summaries structural only | -| checkout/home metadata | 10 GiB | `le_app_codex` | Git remote is source backup; no personal exports committed | - -The listed soft budgets total 200 GiB. The 250 GiB hard qgroup ceiling reserves 50 GiB emergency headroom. Startup refuses when host free space is below 100 GiB, project usage exceeds 200 GiB, or required free space for a rehearsal cannot be reserved. The production filesystem is not used for spillover. - -## Resources intentionally outside `/srv/le-app-codex` - -| Resource | Permanent location | Why it remains outside | -|---|---|---| -| Complete untouched Directus archive | `/srv/directus-archive/20260710-175408` | Root-only disaster-recovery evidence; must not enter the Codex trust boundary | -| Production checkout/deployment root | `/var/www/lasereverything.net.db` | Canonical production policy and existing shared-server operations boundary | -| Production Docker image/layer storage | Host runtime-managed location | Operational daemon state, declared by the canonical production deployment but not Git content | -| Production PostgreSQL/media storage | Approved production volume/storage paths | Persistent production data requires independent ownership, capacity, and backup controls | -| Shared public reverse-proxy configuration/certificates | Existing TITANSERVER ingress paths | Server-wide infrastructure serves multiple applications and is never container-controlled | -| Existing self-hosted SMTP infrastructure | Existing mail-server deployment | Production email remains independent; Mailpit is nonproduction-only | - -Transitional `/srv/codex-*` paths remain temporarily outside during the rollback period only. They are not permanent alternate project roots and become cleanup candidates solely through the explicit process below. - -## Production ingress boundary - -Production will use an **internal application-gateway container behind TITANSERVER's existing shared host reverse proxy**. - -- The server-wide proxy remains host-managed, terminates public TLS, owns ports 80/443, ACME, and cross-application routing. -- The repository-controlled gateway binds only to a dedicated host loopback port or private production network and routes frontend versus Payload internal services. -- The host proxy forwards approved public hostnames to that one gateway and overwrites—not trusts from clients—the forwarding chain and scheme headers. -- The application repository supplies a versioned host-proxy include/template and contract tests, but an operator explicitly reviews/renders/includes it. Containers never mount or edit host proxy configuration or certificates. -- The gateway is not published directly to the internet. Payload admin exposure can be separately restricted at the host proxy. -- Existing `forms.lasereverything.net` behavior remains under the shared ingress until cutover authorization. - -Local development uses its own gateway/TLS because it does not share production ingress. This design preserves server-wide ownership while keeping application routing rules testable and versioned. - -## Transitional resource inventory - -Metadata-only audit results at the planning checkpoint: - -| Resource | Current state | -|---|---| -| `/srv/codex-work/lasereverything.net.db` | Primary Git checkout, branch `temporary/utilities-only`, commit `305ea41`; owns the current local Git database | -| nested `.worktrees/payload-migration` | Migration worktree, branch `migration/le-app-database`; planning commit begins at `ed1c84e` | -| `.migration-source` | Symlink to `/srv/codex-migration/current` | -| `.migration.env` | Symlink to `/srv/codex-secrets/le-payload-migration.env` | -| `/srv/codex-migration/current` | Symlink to `source-20260710-175408` | -| `/srv/codex-migration/source-20260710-175408` | Scrubbed source snapshot, mode `0750`, approximately 150 MB aggregate at audit time | -| `/srv/codex-secrets/le-payload-migration.env` | Current readable nonproduction migration environment, mode `0640`; values not inspected | -| `/srv/codex-secrets/le-payload-migration-admin.env` | Root-only transitional administration environment, mode `0600`; values not inspected and not copied into the development environment | -| `/srv/codex-work` | Approximately 2.0 GB aggregate, including repository history/worktrees and application files | -| recorded rootful MariaDB | container `le-directus-source`, volume `le_directus_source_data_20260710-175408` | -| recorded rootful PostgreSQL | container `le-payload-pg`, volume `le_payload_pg_data_20260710-175408` | -| retired Directus | container `directus`; not a disposable migration resource and not removed by consolidation | - -The current user cannot query the rootful production Docker daemon. Before consolidation, an administrator must produce a metadata-only inventory without printing container environments, labels that contain secrets, database contents, or mount-file contents: - -```bash -sudo docker ps -a --filter name=^/le-directus-source$ \ - --filter name=^/le-payload-pg$ --format '{{.Names}} {{.Image}} {{.Status}}' -sudo docker inspect le-directus-source le-payload-pg \ - --format '{{.Name}} image={{.Image}} mounts={{range .Mounts}}{{.Type}}:{{.Name}}:{{.Destination}};{{end}}' -sudo docker volume inspect \ - le_directus_source_data_20260710-175408 \ - le_payload_pg_data_20260710-175408 \ - --format '{{.Name}} driver={{.Driver}} mountpoint={{.Mountpoint}} labels={{json .Labels}}' -``` - -If names, image IDs, volumes, or mounts differ from the recorded manifest, stop and revise the plan. Do not inspect `.Config.Env` or print secret values. - -## Reversible consolidation procedure - -No step deletes or mutates a transitional resource. Consolidation is copy/recreate/verify, followed by a rollback observation period. - -### Phase A — manifests and Git preservation - -1. Push `migration/le-app-database`, including planning commits, to `origin` and verify the remote commit. -2. Record structural manifests for every transitional path, symlink, owner/mode, aggregate size, Git branch/commit/remote, source checksum manifest, and the administrator container/volume inventory above. -3. Confirm the full root-only archive remains independently present and inaccessible; do not traverse or copy it. -4. Record the exact existing rootful container/volume names and leave them stopped/running exactly as found unless a later approved step says otherwise. - -### Phase B — canonical checkout - -After account/runtime approval, create one clean checkout as the dedicated user: - -```bash -sudo -iu le_app_codex git clone --branch migration/le-app-database --single-branch \ - ssh://FORGE/makearmy/le-app.git \ - /srv/le-app-codex/lasereverything.net.db -sudo -iu le_app_codex git -C /srv/le-app-codex/lasereverything.net.db \ - fetch --prune origin -sudo -iu le_app_codex git -C /srv/le-app-codex/lasereverything.net.db \ - status --short --branch -sudo -iu le_app_codex git -C /srv/le-app-codex/lasereverything.net.db \ - rev-parse HEAD -``` - -The HEAD must equal the reviewed remote `migration/le-app-database` commit, the worktree must be clean, and no nested permanent worktree may exist. Provision a dedicated repository-scoped Git credential for `le_app_codex`; do not copy `sol6_vi`'s private key. - -### Phase C — scrubbed source copy - -The administrator copies from the resolved source directory, never from the root-only archive: - -```bash -source_old=/srv/codex-migration/source-20260710-175408 -source_new=/srv/le-app-codex/database-source-readonly - -sudo rsync -aHAX --numeric-ids --delete-delay "$source_old/" "$source_new/" - -# Verify supplied manifests without emitting filenames or record contents. -sudo sh -c "cd '$source_new' && sha256sum --check --status CHECKSUMS.sha256" -sudo sh -c "cd '$source_new' && sha256sum --check --status uploads.sha256" - -# Freeze the canonical input against development writes. -sudo chown -R root:le_app_codex "$source_new" -sudo find "$source_new" -type d -exec chmod 0550 {} + -sudo find "$source_new" -type f -exec chmod 0440 {} + -``` - -Before using `--delete-delay`, assert `source_new` resolves exactly to `/srv/le-app-codex/database-source-readonly` and is not a symlink. The source checksum files, row-count manifests, schema, dump, extensions, and all uploads must validate. The old copy remains untouched. - -### Phase D — nonproduction secrets - -- Preserve both old credential files in place during rollback. -- Do not copy `le-payload-migration-admin.env`. -- Generate new fixture/snapshot/rootless database, Payload, Mailpit, and local gateway credentials directly into root-owned `/srv/le-app-codex/nonproduction-secrets` with mode `0640`. -- If the MariaDB read-only credential from the scrubbed environment is temporarily required to compare the old rootful source, transfer only the minimum required variable set through an administrator-reviewed, non-logging process; replace it once the rootless source clone is verified. -- No production SMTP, OAuth, Ko-fi, Directus token, webhook, or server credential enters the new directory. - -### Phase E — rootless database recreation - -1. Restore the scrubbed SQL from canonical read-only `source` into a new project-scoped rootless MariaDB volume. -2. Create a distinct import account with `SELECT` only; application/migration jobs never receive the MariaDB administrative credential. -3. Mount/copy uploads only through finite jobs; the MariaDB and legacy Directus services never receive writable access to canonical `source`. -4. Create fresh disposable rootless PostgreSQL and Payload media volumes. -5. Verify source table counts/checksum summaries against supplied manifests and verify PostgreSQL health/reset isolation. -6. Record new rootless image digests, volume names, Compose project labels, and service health without recording secrets. - -### Phase F — rollback observation period - -Keep `/srv/codex-work`, `/srv/codex-migration`, `/srv/codex-secrets`, rootful `le-directus-source`, rootful `le-payload-pg`, and their recorded volumes intact for at least 30 days **and** through at least two complete successful rootless migration rehearsals, whichever is later. During this period: - -- new work occurs only in the canonical checkout; -- old resources are read-only/frozen where possible; -- checksum, Git, service, and validation comparisons remain reproducible; -- rollback means stop the rootless stack and resume analysis from the untouched transitional resources, not copy changes backward. - -## Later cleanup eligibility — explicit approval required - -Only after the rollback criteria and a fresh inventory may the following become eligible for removal: - -- `/srv/codex-work/lasereverything.net.db/.worktrees/payload-migration` after Git confirms it is clean, pushed, and removable with `git worktree remove`; -- `/srv/codex-work/lasereverything.net.db/.migration-source` and `.migration.env` symlinks; -- `/srv/codex-work/lasereverything.net.db`, then `/srv/codex-work` only if no other entries exist; -- `/srv/codex-migration/current`, `source-20260710-175408`, then `/srv/codex-migration` after canonical checksums pass and rollback expires; -- `/srv/codex-secrets/le-payload-migration.env` and `le-payload-migration-admin.env`, then `/srv/codex-secrets`, after credentials are revoked/obsolete and rollback expires; -- rootful containers `le-directus-source` and `le-payload-pg`; -- rootful volumes `le_directus_source_data_20260710-175408` and `le_payload_pg_data_20260710-175408` after verifying their exact attachment and backup status. - -Every deletion requires a separate explicit approval showing the resolved path/object, before/after checksum or Git proof, attachment status, and rollback consequence. Never include the retired `directus` container, production volumes, `/var/www`, or `/srv/directus-archive/20260710-175408` in this cleanup set. - -## Canonical production promotion - -Promotion path: - -```text -/srv/le-app-codex/lasereverything.net.db - -> reviewed Git commit + signed/recorded release manifest - -> CI-built immutable image digests tested in fixture/snapshot/rehearsal/staging - -> clean checkout or verified release artifact at /var/www/lasereverything.net.db - -> production deployment invoked only from /var/www/lasereverything.net.db -``` - -Required workflow: - -1. The canonical nonproduction checkout is clean and all accepted commits are pushed. -2. CI builds images once, records digests/SBOM/provenance, and runs the accepted test gates. -3. Staging deploys those exact digests from the same Compose core and an approved staging overlay. -4. A reviewed release manifest pins Git commit, image digests, migration set, configuration schema, health contract, and rollback compatibility. -5. On TITANSERVER, an operator updates a clean Git checkout at `/var/www/lasereverything.net.db` to the reviewed commit (or atomically unpacks a verified release artifact into that exact root), verifies a clean tree, and verifies the release manifest. -6. From that directory only, deployment scripts render/validate production Compose and the host-ingress include, run the gated schema job, pull pinned digests, and roll services with health checks. - -The production checkout contains all application source/configuration needed to reproduce the deployment contract. Physical image layers, declared production PostgreSQL/media volumes, and shared host-proxy configuration may remain outside it, but their names/locations/interfaces are declared in the production overlay and release manifest rooted there. - -Forbidden promotion inputs include `/srv/le-app-codex/container-runtime`, `database-source-readonly`, `nonproduction-secrets`, `testing-data`, `build-artifacts`, local CA material, Mailpit state, test traces, rootless Docker state, disposable databases/volumes, caches, and development `.env` files. No `rsync` or recursive copy from `/srv/le-app-codex` to `/var/www` is permitted. - -## Local HTTPS and CA trust - -The local gateway uses a repository-pinned proxy image and an ignored, generated development CA to serve: - -- `app.le.localhost`; -- `cms.le.localhost`; -- `mail.le.localhost`; -- optional `legacy.le.localhost`. - -All bind to loopback port 8443, avoiding privileged ports and host DNS changes (`*.localhost` resolves locally). Certificates and private keys live under `/srv/le-app-codex/nonproduction-secrets/local-ca`, mode `0700/0600`, excluded from Git and build contexts. - -Automated Playwright tests may use a dedicated browser profile with its own trust database. Manual Chromium trust can be limited to `/srv/le-app-codex/home/.pki/nssdb` using the already installed `certutil`; system-wide CA trust is unnecessary. Importing the CA into even that user/browser profile is a host/user-state change and requires explicit approval. Removal deletes the certificate from that profile and the ignored CA directory. - -## Mailpit and outbound protection - -Mailpit exists only in fixture, snapshot, rehearsal, and default staging modes: - -- internal Compose networks only; UI reaches the browser through the local gateway; -- no public host bind and no production ingress route; -- SMTP relay/forwarding and webhook forwarding disabled; -- no production SMTP variables mounted; -- tests query Mailpit's internal HTTP API and follow local HTTPS verification/recovery links; -- registration, verification, recovery, supporter claim, and notification paths are covered. - -Defense in depth: - -1. Mode-specific environment schema rejects production SMTP hostnames/sender domains, ports 25/465/587 as external endpoints, production credential variable scopes, OAuth secrets, Ko-fi production tokens, and webhook credentials. -2. Runtime services attach only to an `internal: true` application network in nonproduction. Mail is addressed to the internal `mailpit:1025` service. -3. Snapshot-derived recipient addresses are rejected or rewritten at the transport boundary to deterministic reserved-domain aliases. Only synthetic fixture accounts may be addressed directly. -4. No production credential exists in rendered Compose, container environment, secret mounts, or image history. -5. A finite `egress-test` runs from the exact frontend and Payload runtime network namespaces: Mailpit delivery must succeed; connection attempts to controlled external test endpoints on SMTP ports must fail; DNS/HTTP behavior is recorded separately. - -Firewall-level isolation will not be claimed until these tests pass under the selected rootless runtime. If Docker `internal` networking still permits an undesired host/external route, the implementation stops and proposes an approved rootless egress control rather than asserting safety. Tests use an administrator-approved controlled endpoint, never a real third-party SMTP server. - -Production remains unchanged: application services use the existing self-hosted SMTP infrastructure and real recipients through production-only secrets and network policy. Mailpit never replaces or modifies it. - -## BGBye CPU/GPU profiles - -Milestone 0 starts with a deterministic CPU-compatible BGBye image/profile: - -- same FastAPI paths, methods endpoint shape, request fields, status codes, content types, health/readiness schema, and error contract as GPU; -- one pinned CPU-capable segmentation model for real tiny-image processing in routine tests; -- image upload/proxy limits, invalid method/file handling, alpha PNG output, concurrency serialization, temporary-file lifecycle, status polling, and FFmpeg contract can be exercised; -- model artifacts are pinned/checksummed and readiness fails until loaded. - -CUDA-only validation is deferred to an opt-in GPU profile/staging gate: - -- NVIDIA runtime/device visibility and driver compatibility; -- every production model/backend, CUDA placement, GPU lock/concurrency, memory cleanup/OOM behavior, performance envelope, and longer video processing; -- production model-cache checksum and offline startup behavior. - -CPU and GPU images share server contract tests. Capability differences appear in explicit health/method metadata rather than changing endpoints. A production GPU image is released only after model preload/readiness, representative image/video output, cleanup, restart, and resource tests pass on GPU staging. Rootless GPU provisioning does not block the first stack. - -## Rootless runtime rollback/removal - -Rollback must be run only after confirming the selected context/socket is the dedicated one: - -```bash -# As le_app_codex: stop project resources, then the user daemon. -DOCKER_HOST=unix:///run/user/1200/docker.sock docker compose \ - --project-name le-app-testing down --remove-orphans -systemctl --user disable --now docker.socket docker.service - -# As administrator: disable optional lingering. -sudo loginctl disable-linger le_app_codex - -# Verify production daemon endpoint and data were never referenced by project metadata/logs. -# Use the planning preflight/audit script; do not connect to or mutate production Docker. - -# Remove only the dedicated project subvolume after an explicit artifact review. -sudo btrfs subvolume delete /srv/le-app-codex - -# Remove subordinate allocations and account after the subvolume/runtime is gone. -sudo usermod --del-subuids 165536-231071 --del-subgids 165536-231071 le_app_codex -sudo userdel --remove le_app_codex -sudo groupdel le_app_codex 2>/dev/null || true -``` - -The dedicated Docker context lives only in the dedicated user's home and disappears with that home. If another operator created a client context pointing to the user socket, remove that context explicitly from that operator's client configuration. - -Production secret permission hardening is not rolled back merely because development is removed; it is an independent security correction. Any temporary transitional ACLs, if an administrator added them contrary to the final copy-based design, must be inventoried and removed before account deletion. Removal does not touch `/var/run/docker.sock`, production contexts, `/var/lib/docker`, `/var/lib/mysql`, `/var/www` application content, production volumes/networks, transitional source during its rollback retention, or `/srv/directus-archive`. - -Final proof compares before/after permission metadata and confirms: production socket remains denied, production service state is unchanged, no project mount referenced forbidden paths, the dedicated UID/subordinate mappings are gone, and only `/srv/le-app-codex` resources were deleted. diff --git a/docs/le-app-database-migration/implementation-plan.md b/docs/le-app-database-migration/implementation-plan.md deleted file mode 100644 index 67c90af..0000000 --- a/docs/le-app-database-migration/implementation-plan.md +++ /dev/null @@ -1,90 +0,0 @@ -# Laser Everything database transition implementation plan - -Status: revised pre-implementation plan. Work stops at the Milestone 0 host-runtime approval gate. - -## Milestone 0 — complete reproducible application environment - -1. **Accepted architecture record** - - Preserve baseline `4884b5d`, dedicated Payload service, PostgreSQL, exact dependency matrix, mixed ID preservation, and committed-migration workflow. - - Review `architecture-proposal.md`, `environment-parity.md`, and host prerequisites. - -2. **Host runtime gate — requires human approval/provisioning** - - Provision the dedicated `le_app_codex` account and a rootless Docker context owned only by it. - - Restrict production environment files currently readable to `sol6_vi`; verify the development account cannot traverse production paths. - - Use deterministic CPU BGBye initially; defer optional rootless GPU provisioning. - - Approve browser-profile-scoped local CA trust for HTTPS testing. - - Confirm nonproduction storage/capacity locations. - - Verify forbidden production paths and sockets are inaccessible. - - Execute every positive and negative preflight in `host-runtime-plan.md` before application work. - -3. **Reversible canonical-path consolidation — separately approval-gated** - - Inventory the transitional Git roots, symlinks, credential filenames, rootful disposable containers, and volumes without reading secret values. - - Create the sole checkout at `/srv/le-app-codex/lasereverything.net.db` and verify `migration/le-app-database` commit/remote state. - - Copy and checksum the scrubbed source into root-owned read-only `/srv/le-app-codex/database-source-readonly`. - - Create new nonproduction-only secrets and rootless MariaDB/PostgreSQL resources; do not reuse production credentials. - - Preserve all old resources through the rollback period. Delete nothing without a later explicit approval. - -4. **Repository/environment structure** - - Add common Compose core, mode overlays/profiles, environment schemas/examples, proxy configuration, health contracts, and safe defaults. - - Complete Docker/build-context exclusions for every context. - - Add one-command wrappers with context/path/project safety checks. - - Add automated external-SMTP denial and forbidden-mount/context tests; make snapshot startup fail closed. - -5. **Fixture stack** - - Build exact pinned images under Node 24.18.0. - - Bring up proxy, frontend, Payload shell, PostgreSQL, Mailpit, deterministic CPU BGBye profile, and fixtures. - - Prove HTTPS cookies, forwarded headers, uploads, mail capture, health, logs, reset, and teardown. - -6. **Snapshot/rehearsal stack** - - Mount only authorized scrubbed source and uploads read-only on finite jobs. - - Add disposable destination media/PostgreSQL and detailed-report exclusions. - - Add isolated legacy Directus reference profile using a writable clone and local-only credentials. - - Prove no production SMTP/OAuth/webhook/token configuration or outbound delivery. - - Keep authoritative source and `.migration.env` mounted only on finite inventory/import/validation jobs. - -7. **Milestone 0 acceptance** - - Clean-checkout fixture startup and smoke test require no undocumented steps. - - Snapshot startup passes mount/secret/egress safety assertions. - - Same image digests run in fixture and snapshot modes. - - Rootless user can control only its nonproduction project resources. - - Production release dry-run proves a clean reviewed commit and immutable digests deploy only from `/var/www/lasereverything.net.db`. - -## Milestone 1 — structural inventory and mapping - -- Generate structural-only collections, fields, relationships, indexes, policies, permissions, flows, operations, row counts, and application-consumer inventory. -- Commit reviewed field-by-field MariaDB-to-PostgreSQL mapping without record samples. -- Classify all files and Directus metadata; retain unresolved files. -- Record only genuine product decisions. - -## Milestone 2 — Payload schema and security model - -- Implement collections, custom IDs, indexes, relationships, media, users, roles, access functions, hooks, and jobs. -- Prototype with schema push only on disposable development PostgreSQL. -- Replace push with committed deterministic migrations before authoritative rehearsal. -- Implement captured email recovery/verification and shared cookie/session contract. - -## Milestone 3 — deterministic migration pipeline - -- Implement separate reset, schema, import, and validation jobs. -- Import in dependency order with preserved IDs/timestamps/ownership and repaired sequences. -- Preserve password hashes for compatibility research without logging them. -- Copy referenced media and retain/classify all unreferenced files. -- Make imports restartable/idempotent or fail safely with explicit checkpoints. - -## Milestone 4 — application compatibility and end-to-end behavior - -- Move each Directus consumer behind shared Payload/compatibility clients. -- Rebuild Ko-fi, claims, memberships, flows, and backfill behavior. -- Implement API contract comparisons and the full fixture/snapshot test matrix. -- Validate password-hash transitional login and forced rehash strategy. - -## Milestone 5 — full rehearsals and staging candidate - -- Run repeated clean destination rehearsals. -- Verify counts, IDs, relationships, timestamps, ownership, permissions, files, checksums, API behavior, and browser workflows. -- Produce aggregate/non-personal validation summaries and unresolved product decisions. -- Build immutable release images and promote their digests to staging. - -## Production boundary - -Stop for explicit authorization before production deployment, reverse-proxy/DNS changes, production secrets or SMTP use, email to real users, final write freeze/cutover, Directus restart/retirement, production database mutation, or archive destruction. diff --git a/docs/le-app-database-migration/metadata-disposition.md b/docs/le-app-database-migration/metadata-disposition.md deleted file mode 100644 index f313cf2..0000000 --- a/docs/le-app-database-migration/metadata-disposition.md +++ /dev/null @@ -1,28 +0,0 @@ -# Directus metadata disposition - -This inventory is structural and contains no example user records or user-provided content. - -| Directus area | Disposition | Reason | -|---|---|---| -| `directus_users` | Translate into Payload users | Preserve user IDs, supported profile fields, status, role mapping, timestamps, and password hashes pending compatibility findings | -| `directus_files`, `directus_folders` | Translate into media and folder metadata | Required to retain referenced files, hierarchy, checksums, timestamps, and ownership | -| `directus_collections`, `directus_fields`, `directus_relations` | Design input; do not import as runtime records | Re-expressed as committed Payload TypeScript collection definitions and migrations | -| `directus_roles`, `directus_policies`, `directus_access`, `directus_permissions` | Design input and verification fixtures | Re-expressed as Payload access functions and role fields; Directus policy rows have no runtime meaning in Payload | -| `directus_flows`, `directus_operations` | Translate behavior into hooks/jobs/endpoints | Flow graphs are implementation evidence, not portable executable definitions | -| `directus_settings` | Selective manual mapping | Only settings with a Payload or application equivalent are relevant | -| `directus_presets` | Archive-only pending editor review | UI bookmarks, filters, layouts, and per-user preferences are Directus Studio-specific | -| `directus_dashboards`, `directus_panels` | Archive-only | Directus Insights configuration is not portable and both source tables are empty | -| `directus_comments`, `directus_notifications` | Exclude from runtime | Empty in the scrubbed source; Directus collaboration internals | -| `directus_activity`, `directus_revisions` | Exclude from working migration | Intentionally scrubbed volatile audit history; untouched root-only archive remains authoritative | -| `directus_sessions` | Exclude | Intentionally scrubbed and cryptographically incompatible; all users receive new sessions | -| Static tokens and user `token` values | Exclude | Revoked credential material must never be migrated | -| TFA secrets, `auth_data` | Exclude | Intentionally scrubbed authentication secrets; users re-enroll if required | -| OAuth clients, codes, consents, tokens | Exclude | Empty/scrubbed live authorization material; providers must be configured afresh | -| `directus_shares` | Exclude | Scrubbed bearer-like public access grants must not survive cutover | -| `directus_migrations` | Exclude | Records Directus's schema history, not Payload's migration history | -| `directus_extensions` | Inspect source extension directory; do not import rows | Extension behavior must be reimplemented and reviewed; the metadata table is empty | -| deployment project/run tables | Archive-only | Directus deployment subsystem state has no Payload runtime equivalent | -| `directus_versions` | Exclude from initial runtime | Empty; Payload versioning will begin under its own schema if enabled | -| translations | Archive-only pending localization decision | Empty in source; no initial runtime effect | - -No exclusion authorizes deletion of source data, frozen files, manifests, or the root-only archive. diff --git a/payload/.dockerignore b/payload/.dockerignore deleted file mode 100644 index 47a4626..0000000 --- a/payload/.dockerignore +++ /dev/null @@ -1,14 +0,0 @@ -.migration-source -.migration.env -.worktrees -node_modules -.next -migration/generated -*.sql -*.sql.gz -uploads -media -test-results -playwright-report -blob-report -.auth