146 lines
10 KiB
Markdown
146 lines
10 KiB
Markdown
# Payload migration architecture proposal
|
|
|
|
Status: pre-implementation decision record. No production changes are authorized by this document.
|
|
|
|
## Objective and Milestone 0
|
|
|
|
The target is a unified, reproducible application environment, not merely a database conversion. Unified means one repository, one versioned orchestration model, shared service contracts, consistent application images, and documented promotion from development through production. Services remain separate processes and containers.
|
|
|
|
**Milestone 0: complete reproducible application environment** must be accepted and implemented before the large Payload/data-migration phase. It delivers:
|
|
|
|
- complete service and persistence inventory for the `4884b5d` application;
|
|
- common Compose core plus mode-specific overlays/profiles;
|
|
- rootless, Codex-controlled nonproduction containers and volumes;
|
|
- production-like local HTTPS routing and cookie behavior;
|
|
- fixture and scrubbed-snapshot data modes using identical application images;
|
|
- safe captured email and outbound-network safeguards;
|
|
- optional isolated legacy Directus reference environment;
|
|
- deterministic one-command lifecycle, migration, validation, smoke, and browser-test workflows;
|
|
- image-by-digest promotion to staging and production.
|
|
|
|
The detailed runtime inventory, orchestration design, repository tree, and test matrix are in [`environment-parity.md`](./environment-parity.md). The access audit, recommended account boundary, exact host prerequisites, ingress integration, storage, networking proof, and rollback plan are in [`host-runtime-plan.md`](./host-runtime-plan.md). The gated delivery sequence is in [`implementation-plan.md`](./implementation-plan.md).
|
|
|
|
Milestone 0 host rule: nonproduction container isolation is not treated as a substitute for host filesystem permissions. The chosen daemon owner must be unable to read production paths and secrets, because a container controller can bind-mount anything readable to that daemon account.
|
|
|
|
## Verified baseline
|
|
|
|
The migration branch initially started from `main` at `0a7ee5f`. A tree and history comparison against the final pre-utilities production commit `4884b5d` established:
|
|
|
|
- `main` is an ancestor of `4884b5d`.
|
|
- `4884b5d` contains exactly two later commits: `92ce8bc` and `4884b5d`.
|
|
- Those commits repair production registration/authentication behavior and add the verified Directus deployment contract, policies, flows, exports, and schema snapshot.
|
|
- The application was not replaced or reduced in those commits.
|
|
|
|
Conclusion: `main` is a complete application tree, but `4884b5d` is the correct and more complete pre-utilities production baseline. The dedicated `migration/payload` branch has therefore been fast-forwarded to `4884b5d`. The temporary utilities branch is not an ancestor of the migration work.
|
|
|
|
## Service boundary
|
|
|
|
The default architecture is three independent services:
|
|
|
|
```text
|
|
browser
|
|
|
|
|
v
|
|
existing Next.js application (frontend + compatibility API)
|
|
| |
|
|
| server-to-server HTTP | background-removal calls
|
|
v v
|
|
dedicated Payload service existing bgbye service
|
|
|
|
|
v
|
|
dedicated PostgreSQL database
|
|
```
|
|
|
|
Payload owns content administration, authentication records, media metadata, access policy, hooks, and its generated APIs. The existing application remains the public frontend and initially retains its public API routes. Those routes become a compatibility layer backed by Payload, permitting endpoint-by-endpoint cutover and response-shape comparison.
|
|
|
|
Deeply embedding Payload into the existing frontend would reduce one network boundary and allow direct Local API calls. It would also force the frontend onto Payload's exact Next.js dependency range, combine release and failure domains, increase the blast radius of admin changes, and make rollback harder. That integration is deferred pending explicit approval.
|
|
|
|
The complete topology adds a reverse proxy, PostgreSQL, captured mail, migration/test jobs, and optional local-only Directus reference services around this accepted boundary. Payload remains independently deployable and is never merged into the frontend process by Milestone 0.
|
|
|
|
## Runtime and dependency matrix
|
|
|
|
| Component | Pinned version | Basis |
|
|
|---|---:|---|
|
|
| Node.js | 24.18.0 LTS (Krypton) | Development, Docker, CI, and production target |
|
|
| npm | 11.16.0 | Version shipped with the pinned Node line |
|
|
| Payload | 3.86.0 | Exact package version |
|
|
| `@payloadcms/next` | 3.86.0 | Must exactly match Payload |
|
|
| `@payloadcms/db-postgres` | 3.86.0 | Must exactly match Payload |
|
|
| Next.js (Payload service only) | 15.4.11 | Explicitly accepted by `@payloadcms/next@3.86.0` peer range |
|
|
| React / React DOM | 19.1.0 | Accepted by Next.js 15.4.11 and conservative relative to the existing app |
|
|
|
|
The existing frontend's Next.js 15.3.2 is outside Payload 3.86.0's accepted ranges. This is another reason to keep Payload in a dedicated workspace. Package metadata was checked with exact-version `npm view`; the official Payload installation documentation lists 15.4.11 as an accepted line. Node 26 on the host is not a target runtime and must not be used to create authoritative lockfiles or release builds.
|
|
|
|
References:
|
|
|
|
- https://payloadcms.com/docs/getting-started/installation
|
|
- https://payloadcms.com/docs/database/postgres
|
|
- https://payloadcms.com/docs/database/migrations
|
|
- https://nodejs.org/en/blog/release/v24.18.0
|
|
|
|
## Schema lifecycle
|
|
|
|
Schema push is permitted only while prototyping against the disposable database. Before the first authoritative rehearsal:
|
|
|
|
1. Disable push in the PostgreSQL adapter.
|
|
2. Commit a generated initial Payload migration.
|
|
3. Keep four separate commands with single responsibilities:
|
|
- `migration:reset`: destructively empties only the disposable destination.
|
|
- `migration:schema`: applies committed Payload/Drizzle migrations.
|
|
- `migration:import`: imports source records and files.
|
|
- `migration:validate`: performs read-only cross-system checks.
|
|
4. Define `migration:rehearse` only as deterministic composition of those commands.
|
|
|
|
Database credentials are loaded by database-specific scripts only. Installation, builds, linting, and type checking do not read `.migration.env`.
|
|
|
|
## Collection strategy
|
|
|
|
Payload will expose first-class collections for:
|
|
|
|
- users and media;
|
|
- buying-guide categories, subcategories, and entries;
|
|
- hazards and material taxonomy;
|
|
- materials, coatings, colors, opacity, status, and their junctions;
|
|
- laser sources, software, scan lenses, focus lenses, configurations, apertures, expanders, and diameters;
|
|
- fiber, UV, CO2 galvo, and CO2 gantry settings;
|
|
- projects and project files;
|
|
- user rigs, rig types, preferences, memberships, and claims.
|
|
|
|
Directus junction tables become Payload relationships where doing so preserves ordering and ownership semantics. A junction remains an explicit collection when it carries its own meaningful fields or permissions.
|
|
|
|
## ID preservation decision
|
|
|
|
Source identifiers will be preserved as the actual Payload document IDs, not merely copied into an auxiliary legacy field.
|
|
|
|
- UUID primary keys remain UUID/string IDs.
|
|
- Integer and medium-integer primary keys remain numeric custom IDs.
|
|
- Collections whose Directus primary key is named `submission_id` use that numeric value as Payload's custom `id`; the compatibility layer returns it as `submission_id` to existing consumers.
|
|
- Payload's PostgreSQL adapter will enable `allowIDOnCreate` for controlled migration imports.
|
|
- Every relationship is imported using the preserved target ID, avoiding translation tables and making verification direct.
|
|
- New numeric IDs must begin above the migrated maximum. Sequence values are explicitly advanced after import.
|
|
- The import rejects duplicate, null, lossy, or out-of-range identifiers before inserting records.
|
|
|
|
This mixed per-collection custom-ID approach is preferable to globally converting every integer to UUID, which would break public URLs, foreign keys, and API assumptions. Payload officially supports explicit custom ID fields, and the PostgreSQL adapter supports IDs supplied during controlled create operations.
|
|
|
|
References:
|
|
|
|
- https://payloadcms.com/docs/fields/overview#custom-id-fields
|
|
- https://payloadcms.com/docs/database/postgres
|
|
|
|
## File strategy
|
|
|
|
All 535 database-referenced originals and all 2,411 currently unreferenced physical files remain retained. Referenced files become Payload media documents while preserving the Directus file UUID, original filename, MIME type, size, timestamps, ownership where valid, and checksum. Derived variants are classified separately from originals.
|
|
|
|
Unreferenced files are not imported as editorial media until classified, but they remain in a migration-retention manifest with path, size, type, and checksum. No file is deleted by reset or migration tooling.
|
|
|
|
The repository's separate curated file library (`app/files`, 282 files and approximately 461 MB at the baseline) is not Directus upload storage. It remains a distinct, read-only application asset volume in normal operation. Payload media and retained migration uploads use separate volumes and permissions.
|
|
|
|
## Production promotion rule
|
|
|
|
CI builds the frontend, Payload, BGBye, and proxy images once from reviewed commits under their pinned runtimes. Integration tests run those exact digests. Staging and production consume the same core Compose definitions and immutable image digests; they do not rebuild from source or use separately maintained service definitions. Environment overlays may change secrets, hostnames, resource policies, external storage/database endpoints, replica counts, and production TLS integration, but not application image contents or service contracts.
|
|
|
|
Production database schema changes use committed Payload migrations as an explicit release job before application rollout. Imports are migration-only jobs and are not part of ordinary application startup. Promotion requires recorded image digests, migration status, validation results, rollback compatibility, and health checks.
|
|
|
|
## Directus metadata disposition
|
|
|
|
See `metadata-disposition.md` for the explicit include, translate, archive-only, and exclude decisions. Exclusion means “not imported into Payload runtime tables,” never “destroyed from the frozen snapshot or archive.”
|