> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://cla-demo.postmansolutions.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://cla-demo.postmansolutions.com/_mcp/server. # 1040 Fluxx Platform Interop---Requirements-&-Sizing-(v2 Workflow) **Purpose:** Translate the new event-driven 1040/Palantir workflow described in 1040\_FLUXX\_INTEROP\_V2.md into a concrete, research-backed requirements list, gap analysis, and sized backlog (Epics → Stories, T-shirt sized Small/Medium/Large) across the `fluxx-*` microservices. **Method:** Every finding below is based on direct inspection of the repos in this workspace (`fluxx-classifier`, `fluxx-document-splitter`, `fluxx-extractor`, `fluxx-file-intake`, `fluxx-postal`, `fluxx-registration`, `fluxx-sql-server`, `fluxx-statuscheck`) — READMEs, `docs/`, `config.json` / `pubsub.config.json` topic wiring, and the actual TypeScript source (handlers, orchestrators, repositories, migrations). Where the code disagrees with the informal description in the requirements doc, the code's actual behavior is called out explicitly. --- ## 1. Naming Clarification (Assumption) There is no repo literally named `fluxx-fileupload`. The upload endpoint described in requirement #1 ("Palantir calls the fluxx-fileupload endpoint with a PDF") maps to the existing **`fluxx-file-intake`** service's `POST /v1/upload/:applicationId` endpoint (`uploadController.ts` → `uploadService.handleMultiFileUpload` / `handleChildFileUpload`). This document proceeds on that assumption — flag if a distinct, new "fileupload" surface is actually intended. Also note: `fluxx-statuscheck` is the only other `fluxx-*` repo not directly named in the requirements, but it consumes almost every topic in the platform for observability and will need updates to stay in sync with the new flow (see Epic 8). --- ## 2. Current State — What The Code Actually Does Today The informal description at the top of `1040_FLUXX_INTEROP_V2.md` (upload → extractor calls Databricks → polls job → Fluxx calls Palantir) undersells how much has already been built. The real current-state, confirmed in source, is: | Stage | Service | Behavior (verified in code) | | ------------------------------------------- | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 1. Upload | `fluxx-file-intake` | `POST /v1/upload/:applicationId` accepts file(s), also supports **child file uploads** (`parentFileId` + `clientApplicationClassifierId` query params) — this already exists to support split-children re-entering the pipeline. Publishes `validated-file` → (internal copy) → `accepted-file`. | | 2. Classification | `fluxx-classifier` | **Already calls a real classifier** (not just a blob-exists placeholder as its README states). `AcceptedFileHandler` → `ClassificationOrchestrator.runClassifications()` → resolves a classifier implementation (e.g. `tax1040Classifier.ts`) → `startClassification()` calls the configured Databricks endpoint (`classificationUrl`) and gets a `runId`. **It then polls** via `ClassificationPollingOrchestrator` (DB-backed job/lock model, exponential backoff, `pollingUrl`) until terminal status, then publishes `classified-file` (or `problem`). | | 3. Splitting | `fluxx-document-splitter` | **Already fully implemented** (not a placeholder as its README states). `ClassifiedFileHandler` downloads the classification JSON + source PDF, validates page ranges, splits the PDF (`pdfSplitter.ts`), **uploads each split file back through `fluxx-file-intake`'s child-upload endpoint** (`FileIntakeService.uploadChildFile`), publishes one `split-file` event per output file, and a `split-file-result` manifest event. | | 4. Extraction trigger | `fluxx-extractor` | Subscribes to **`accepted-file`** today (`extractor-accepted-file` subscription; see `pubsub.config.json`) — i.e., it currently reacts to the *original* uploaded file, not to `split-file`. `ExtractionTriggerHandler` looks up extractor config from `fluxx-registration`, generates a SAS URL for the blob, and calls `ExtractorService.runDocumentExtractions()`. | | 5. Extraction execution | `fluxx-extractor` | `DatabricksAIExtractor.startExtraction()` calls the Databricks orchestrator (`extractionUrl`), gets a `runId`, and a separate **`extractionJobPollingWorker.ts`** polls `getRunStatus()` until terminal. | | 6. Delivery to Palantir | `fluxx-extractor` | On success, `DatabricksAIExtractor.processExtractionResult()` downloads the result blob(s) and **calls `PalantirApiClient.ingestExtractedFields()` directly** — i.e., **the extractor talks to Palantir directly today; `fluxx-postal` is not in this loop at all.** | | Delivery infra (built, unused by this flow) | `fluxx-postal` | Fully built to consume `to-send` / `to-retry` topics, look up `deliveryUrl` from `fluxx-registration` (`RegistrationService.getDeliveryRoute`), and POST the message body via `PostalService.postMessage`. **Nothing in the platform publishes to `to-send` today** — the topic reference doc (`fluxx-file-intake/docs/topics-subscriptions-reference.md`) literally lists its publisher as `TBD`. Postal also has **no database of its own** — every delivery does a live HTTP call to `fluxx-registration` for the route, and `deliveryMethod` is **hard-coded to `'http'`** in `clientApplicationRepository.ts` (not a real stored preference). | | Registration store | `fluxx-registration` + `fluxx-sql-server` | Central DB (`autoextract`, schema-per-domain e.g. `[registration].[ClientApplicationExtractor]`) already stores `extractionUrl`, `pollingUrl`, `agent`, `ontologyActionUrl`, `extractorTypeId` per client-application-extractor, and an analogous classifier table. Palantir is already a seeded client application with a `DBRICKS` extractor row (`V1.22.0__createPalintrSeedRegistration.sql`). | ### Key architectural gaps this exposes (beyond the literal 7 bullet points) 1. **Two separate poll-and-lock engines exist** (classifier's `ClassificationPollingOrchestrator`, extractor's `extractionJobPollingWorker`) that both need to be replaced/augmented with event-driven completion — this is a bigger lift than "stop polling," it's swapping the completion trigger for two independently-built subsystems. 2. **Extractor talks to Palantir directly**, hard-coded to a single client (Palantir-specific `PalantirApiClient`, `ontologyActionUrl`). Moving delivery to `fluxx-postal` means **removing this direct dependency from `fluxx-extractor`** and generalizing delivery. 3. **`to-send` has no publisher today.** Wiring the extractor's completion event into `to-send` (or a new topic postal also subscribes to) is the actual "last mile" connecting the new flow to the already-built postal delivery pipeline. 4. **`deliveryMethod`/content-shape (payload vs. SAS URL vs. blob URL) does not exist as a real, configurable concept anywhere in the schema.** This is wholly new (requirement #7's "one of three things, whichever is registered"). 5. **`fluxx-extractor` subscribes to `accepted-file`, not `split-file`,** even though `fluxx-document-splitter`'s own `config.json` already declares an `extractor-split-file` subscription on `split-file` — i.e., the SB topology partially anticipates this move already, but the extractor app code and its own `pubsub.config.json` haven't been updated to match. 6. **No component currently owns "what to do when there's no classifier/splitter path"** — i.e., a backward-compatibility/parallel-run story is needed if both the old (`accepted-file` → extractor direct) and new (`accepted-file` → classifier → splitter → `split-file` → extractor) paths must coexist during rollout. --- ## 3. Target State (New Workflow) ```mermaid sequenceDiagram participant Palantir participant FileIntake as fluxx-file-intake participant Classifier as fluxx-classifier participant Databricks participant Splitter as fluxx-document-splitter participant Extractor as fluxx-extractor participant Postal as fluxx-postal participant Registration as fluxx-registration Palantir->>Palantir: generate fileId (GUID) Palantir->>FileIntake: POST /v1/upload (PDF + fileId) FileIntake-->>Classifier: accepted-file (fileId) Classifier->>Databricks: call classify endpoint (fileId, fire-and-forget) Databricks-->>Databricks: classify, write result to blob Databricks-->>Classifier: publish classification-complete (fileId, topic) Classifier-->>Splitter: classified-file (fileId) Splitter->>FileIntake: upload each split child (new child fileId, parentFileId=original fileId) Splitter-->>Extractor: split-file (child fileId, one per child) Extractor->>Databricks: call extraction endpoint (child fileId, fire-and-forget) Databricks-->>Databricks: extract, write result to blob Databricks-->>Extractor: publish extraction-complete (fileId, topic) Extractor->>Registration: resolve delivery preference (payload/sasUrl/blobUrl) Extractor-->>Postal: to-send (fileId, parentFileId) Postal->>Registration: resolve delivery route (own DB, cached from registration) Postal->>Palantir: POST registered endpoint (payload | sasUrl | blobUrl, fileId) ``` Key differences from current state, in requirement order: 1. Upload endpoint unchanged (`fluxx-file-intake`), **except that the `FileID` is now generated by Palantir and supplied on upload** rather than generated server-side (see §3a). 2. Classifier calls Databricks classify endpoint — **already true today**; the new part is *how it learns of completion*, and that the call must now include the Palantir-issued `fileId`. 3. **No polling** — Databricks publishes to a topic on completion; classifier subscribes instead of polling. 4. Splitter behavior **already matches** the requirement (splits + publishes one event per file) — needs verification/regression testing against classifier's new completion path, not new build. 5. Extractor is triggered by `split-file` (not `accepted-file` as today) and calls the existing Databricks extraction endpoint, **including the relevant `fileId` in the call** (see §3a for which `fileId` — parent vs. split child). 6. **No polling** — Databricks publishes extraction completion to a topic; extractor subscribes instead of polling. 7. `fluxx-postal` — not the extractor — contacts the registered Palantir endpoint, with the payload shape chosen from 3 registered options. This requires **postal's own DB table for registrations/delivery preference** and a producer that actually publishes to `to-send`. --- ## 3a. FileID Ownership Change: Palantir-Generated GUID **New design direction:** Palantir generates the `FileID` (a GUID/UUID) client-side at upload time and submits it along with the PDF. This becomes the canonical `FileID` for that document across the entire platform, and must be explicitly included in both the classification call and the extraction call made to Databricks — making `fileId` (not just the internal `jobId`/`runId`) a first-class cross-system correlation key. **What changes vs. today (confirmed in source):** | Area | Today | New Design | | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | FileID origin | `fluxx-file-intake`'s `UploadService.handleMultiFileUpload`/`handleChildFileUpload` generates the ID server-side via `FileUtils.generateFileId()` (`crypto.randomUUID()`), *before* the caller ever sees it. The caller cannot supply one today. | Palantir generates the GUID and passes it in the upload request; `fluxx-file-intake` must accept and use the caller-supplied ID instead of generating its own for this client. | | Format compatibility | `FileUtils.isValidGuid()` already validates a standard 8-4-4-4-12 GUID — the same format `crypto.randomUUID()` produces. | No format change needed — a Palantir-issued GUID passes the existing validation as-is. | | Propagation through existing events | `fileId` is already threaded, unchanged, through `accepted-file` → `classified-file` → `split-file` → `split-file-result` message contracts. | **No message-contract changes needed for these existing topics** — they already carry `fileId`; only its origin changes. | | Classification call to Databricks | `Tax1040Classifier.startClassification()` is currently a **mock** — the commented-out real call only sends `{ jobId, fileUrl }`. No `fileId` field exists in the classification request today. | The real (non-mock) classification call must be built to include `fileId` explicitly in the request body, so Databricks can echo it back on completion. | | Extraction call to Databricks | `DatabricksApiClient.triggerJobRun()` / `DatabricksRunNowRequest` sends `{ name, client_id, created_by, metadata: { output_path }, files: [{ file_path, agent }] }` — **no `fileId` field exists in this request today**, confirmed in `databricks.apiClient.ts`. | Add `fileId` to `DatabricksRunNowRequest` (and the corresponding completion/status response) so it is passed on every extraction call and can be used to correlate the eventual completion event (Epic B). | | Split-child files | When `fluxx-document-splitter` uploads each split PDF back through `fluxx-file-intake`'s child-upload endpoint, **each split child gets its own new, server-generated `fileId`** (tracked via `parentFileId` back to the original). | **Open question (Q6 below):** does extraction on a split child pass the child's own `fileId`, or the original Palantir-issued parent `fileId`? See Q6 for the recommended default. | --- ## 4. Open Questions / Assumptions Requiring Decision Before Sizing Is Final These materially affect sizing and should be confirmed with the Databricks/infra team and product owner before work starts. Each is flagged inline in the epics below as well. | # | Question | Why it matters | Recommended default (used for sizing) | | -- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Q1 | Can Databricks publish directly to an Azure Service Bus topic on job completion, or does it only support an HTTP callback/webhook? | Determines whether classifier/extractor need a **new inbound Service Bus subscription** (cheaper, consistent with existing patterns) or a **new inbound HTTP webhook route** (more work: auth, idempotency, exposing a public endpoint). | Assume Service Bus topic publish from Databricks is available (mirrors the existing `malware-scan` Event Grid → Service Bus pattern already in the platform). Webhook fallback sized separately as a swap-in alternative. | | Q2 | Who generates the SAS URL for delivery option 2? | Postal has **no blob storage client today**. Extractor already has `BlobStorageService.generateSignedUrl`. | Extractor generates the SAS URL (and the plain blob URL) at extraction-completion time and includes both, plus the raw payload, in the message it hands to `to-send`; postal picks the field it needs based on registered preference. Avoids giving postal blob credentials. | | Q3 | Does "standalone DB table for postal registrations" mean postal gets its **own copy/cache** of registration+delivery-preference data (denormalized, refreshed via `updated-registration` events which already exist and are unused), or does it **replace** the live HTTP call to `fluxx-registration` entirely? | Changes whether postal needs to subscribe to `updated-registration` (already published, currently has zero consumers) and reconcile a cache, vs. simple pass-through storage. | Postal owns a local read-optimized table populated by subscribing to the existing (currently unused) `updated-registration` topic, with a one-time backfill from `fluxx-registration`, and falls back to the live HTTP call on cache miss. | | Q4 | Should the old `accepted-file` → extractor-direct path be removed immediately, or run in parallel during rollout (feature flag by client application)? | Affects whether extractor needs to support two trigger topics simultaneously for a transition period. | Assume a feature-flagged parallel run per client application (Palantir cut over first), removed once validated. | | Q5 | Which three delivery content values map to requirement #7 exactly — `payload`, `sasUrl`, `blobUrl` — and is this a single enum column or three booleans? | Schema design in `fluxx-registration`/`fluxx-sql-server` and postal. | Single enum column `DeliveryContentType` (`Payload` \| `SasUrl` \| `BlobUrl`) on the client application (or a new delivery-preferences table), defaulting to `Payload` for backward compatibility. | | Q6 | When a split child is extracted, does the extraction call to Databricks (and the resulting delivery to Palantir) use the **split child's own `fileId`** or the **original Palantir-issued parent `fileId`**? Also: should file-intake reject/collide-check a duplicate Palantir-supplied `fileId`, or treat resubmission as idempotent? | Determines correlation-key design in Epic B (extraction) and downstream delivery in Epic C, plus validation rules in Epic G. | Use the split child's own `fileId` for the extraction call and for `to-send`/Palantir delivery (since extraction and delivery are per-split-document), while carrying the original Palantir `fileId` alongside as `parentFileId`/`rootFileId` for lineage — Palantir can correlate results back to its original submission via that field. File-intake treats a duplicate Palantir-supplied `fileId` as a `409 Conflict` (reject), not silent idempotent replay, unless product says otherwise. | --- ## 5. Requirements List (Traceable to Source Bullets) | Req # | Requirement | Primary Service(s) | Status Today | | ----- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | R1 | Palantir uploads a PDF to the file intake endpoint | `fluxx-file-intake` | ✅ Exists | | R2 | Classifier calls a configured Databricks endpoint to classify the document | `fluxx-classifier` | ✅ Exists (via `ClassificationOrchestrator` + classifier resolver) | | R3 | No polling — Databricks classifies, writes to blob, publishes to a topic; classifier reacts to that | `fluxx-classifier` | ❌ Currently polls (`ClassificationPollingOrchestrator`) | | R4 | Splitter picks up the published classification message, splits the PDF, publishes one event per split file | `fluxx-document-splitter` | ✅ Exists, but trigger source (message shape from classifier) must remain compatible with R3's rework | | R5 | Extractor picks up the published split-file message and calls the existing Databricks extraction endpoint | `fluxx-extractor` | ⚠️ Partially — extraction call exists, but trigger is `accepted-file`, not `split-file` | | R6 | No polling — Databricks extracts, writes to blob, publishes to a topic; extractor reacts to that | `fluxx-extractor` | ❌ Currently polls (`extractionJobPollingWorker`) | | R7 | Postal picks up the published extraction message and contacts the registered Palantir endpoint with payload, sasUrl, or blobUrl (per registration) | `fluxx-postal`, `fluxx-registration` | ❌ Postal not in the loop at all today (extractor calls Palantir directly); only one hard-coded delivery mode exists | | R8 | Postal needs a standalone DB table for postal registrations | `fluxx-postal`, `fluxx-sql-server` | ❌ Does not exist | | R9 | End-to-end workflow (upload → Palantir notified) must work with all of the above changes | All | ❌ Net-new integration path | | R10 | Palantir generates the `FileID` (GUID) at upload time and passes it with the file; this becomes the canonical `FileID` used everywhere across the platform, and is passed to Databricks on both classification and extraction calls | `fluxx-file-intake`, `fluxx-classifier`, `fluxx-extractor` | ❌ Net new — file-intake currently always generates `fileId` server-side, and neither the classification nor extraction request bodies sent to Databricks include `fileId` today | --- ## 6. Epics & Sized Stories Sizing scale used throughout: * **S (Small):** \~1–3 days, single service, low risk, no schema change or an additive/backward-compatible schema change. * **M (Medium):** \~3–5 days (up to \~1 week), may span 2 services or require a schema migration, moderate coordination/testing. * **L (Large):** \~2+ weeks, cross-service, new infrastructure/data model, higher risk, may depend on external teams (Databricks/infra) or require a rollout/feature-flag strategy. ### Epic A — Classifier: Event-Driven Classification Completion (Remove Polling) Replaces `ClassificationPollingOrchestrator` with a reactive completion path. | Story | Size | Details | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | A1. Spike: confirm Databricks completion notification mechanism (Service Bus publish vs. webhook) and payload contract (jobId/runId/fileId correlation, blob location) | S | Resolves Q1. Output: a short design note + topic/webhook contract to unblock A2+. | | A2. Define and register new inbound topic/subscription (e.g. `classifier-job-complete`) in `config.json`/`pubsub.config.json`, and add app-config keys | S | Mirrors existing topic-registration patterns already in the repo. | | A3. Build the real (non-mock) `Tax1040Classifier.startClassification()` call and include `fileId` explicitly in the outbound request body sent to the Databricks classify endpoint | M | Currently a hard-coded mock (`// TODO: Implement the actual call...`) that only sends `{ jobId, fileUrl }` — no real HTTP call exists yet, and `fileId` is not part of the request at all today. | | A4. New `ClassificationCompleteHandler` (or webhook controller) that correlates the incoming completion event to the existing job by **`fileId` (primary) and `runId`/`jobId` (secondary)** in `jobRepository`, replacing the terminal-status branch currently in `pollJobStatus()` | M | Reuses `updateJobSucceeded`/`updateJobFailed` logic already in `ClassificationOrchestrator`; just changes the trigger and adds `fileId` as a correlation key. | | A5. Remove/retire `ClassificationPollingOrchestrator`, its `setTimeout` loop, `getJobsToPoll` lock-based DB queries, and related polling config (`MAX_POLLING_ATTEMPTS`, `POLL_INTERVAL_MS`, etc.), behind a feature flag until A4 is proven in an environment | M | Must keep DB columns for now (see Epic F) for audit/history; only the *loop* goes away. | | A6. Update/extend contract + unit tests for the new handler and the real classify call; remove or repurpose polling-specific tests | M | Existing `test/contract`, `test/unit` structure reused. | | A7. Update `fluxx-classifier` README/docs to reflect real (event-driven) architecture | S | Also corrects the current README, which already understates existing behavior. | **Epic A total: \~1 L equivalent (6 stories, 1 spike)** --- ### Epic B — Extractor: Trigger from `split-file` + Event-Driven Extraction Completion (Remove Polling) | Story | Size | Details | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | B1. Spike: confirm Databricks extraction completion notification mechanism (same as A1, likely shared design) | S | Can be combined with A1 if same team/contract. | | B2. Change extractor's inbound subscription from `accepted-file` to `split-file`; update `pubsub.config.json`/`config.json`, `ExtractionTriggerMessageBody` type to match the `split-file` event contract (`clientApplicationId`, `internalUrl`, `fileLocationUrl`, `fileId`) | M | `split-file` already exists and is published by the splitter today; this is primarily a rewire + type/shape mapping in `ExtractionTriggerHandler`. | | B3. Feature-flag / dual-subscription support so `accepted-file`-triggered extraction can run in parallel with `split-file`-triggered extraction during rollout (Q4) | M | Needed only if a phased cutover per client application is required; otherwise can be dropped in favor of a hard cutover for Palantir only. | | B4. Add `fileId` to `DatabricksRunNowRequest` and thread it through `DatabricksAIExtractor.startExtraction()` — no `fileId` field exists in the extraction request body today (confirmed: only `name`, `client_id`, `created_by`, `metadata.output_path`, `files[]`) | M | Resolves Q6 for which `fileId` is sent (split child's own `fileId`, with parent/root `fileId` carried as auxiliary metadata). | | B5. New inbound topic/subscription + handler for extraction completion (mirrors A2/A4), correlating to `jobRepository`/`jobStatusRepository` by **`fileId` (primary) and `runId` (secondary)** | M | Reuses `updateJobStarted`/status update plumbing already in `ExtractorService`. | | B6. Remove/retire `extractionJobPollingWorker.ts` and related polling config, behind a feature flag | M | Same caution as A5 — keep schema, remove the loop. | | B7. Update/extend tests (unit + contract) for new trigger, `fileId`-bearing extraction request, and completion handlers | M | | | B8. Update `fluxx-extractor` README to reflect the new trigger source and event-driven completion | S | | **Epic B total: \~1 L equivalent (8 stories)** --- ### Epic C — Decouple Palantir Delivery From Extractor → Route via Postal This is the biggest behavioral change: today `DatabricksAIExtractor.processExtractionResult()` calls `PalantirApiClient.ingestExtractedFields()` directly. That needs to become "publish a delivery-ready message"; `fluxx-postal` becomes responsible for the actual outbound call. | Story | Size | Details | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---- | ------------------------------------------------------------------------------------ | | C1. Define the `to-send` (or new topic) message contract to carry: `clientApplicationId`, `fileId`, `jobId`, extracted `payload` (JSON), `sasUrl`, `blobUrl`, and enough metadata for postal to pick the right shape | M | This is the concrete fix for the "no publisher for `to-send`" gap found in research. | | C2. In `fluxx-extractor`, replace the direct `PalantirApiClient.ingestExtractedFields()` call in the (now event-driven, per Epic B) completion handler with: download/locate result blob → generate SAS URL (`BlobStorageService.generateSignedUrl`, already exists) → build the C1 message → publish to `to-send` | M | Removes Palantir-specific coupling from the extractor entirely. | | C3. Remove `PalantirApiClient` and its Palantir-specific config (`ontologyRid`, ontology action URL calls) from `fluxx-extractor` once C2 is validated; note `ontologyActionUrl` remains a *registration* concept, just consumed by postal instead (see Epic E) | S | Cleans up now-dead code/config in the extractor. | | C4. In `fluxx-postal`, extend `DeliveryService`/`PostalService` to build the outbound HTTP body according to the registered `DeliveryContentType` (payload passthrough vs. `{ sasUrl }` vs. `{ blobUrl }`) instead of forwarding the message body verbatim | M | Builds on Epic D/E's schema work. | | C5. Contract/integration test: end-to-end simulated flow from a fake "extraction complete" event through `to-send` to a mock Palantir receiver, for all 3 delivery content types | M | High value regression test given this is the new "last mile." | | C6. Update `fluxx-postal` and `fluxx-extractor` docs/README to describe the new delivery ownership | S | | **Epic C total: \~1 L equivalent (6 stories)** --- ### Epic D — Postal: Standalone Database for Registrations/Delivery Preferences Directly implements requirement R8. | Story | Size | Details | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | D1. Design schema for a postal-owned table (e.g. `[postal].[ClientApplicationDeliveryRoute]`): `clientApplicationId`, `deliveryUrl`, `deliveryContentType`, `isActive`, audit columns, sync metadata (`lastSyncedAt`, `sourceVersion`) | S | Schema-only design story; feeds D2. | | D2. `fluxx-sql-server` migration(s) to create the new schema/table (new `V1.x.x` script), consistent with existing Flyway conventions | S | Straightforward additive migration, no cross-service coordination needed to create it. | | D3. Add `DbConnector`/repository layer to `fluxx-postal` (net-new — postal currently has no DB dependency at all): connection config, `postalRegistrationRepository.ts`, DI wiring in `index.ts` | M | Postal has none of the existing DB plumbing that other services have (`framework-nodejs-sql` usage, migrations task wiring, docker-compose `db` service) — this is new infrastructure for this repo, not just a table. | | D4. Subscribe `fluxx-postal` to the existing (currently unconsumed) `updated-registration` topic to keep the local table in sync; add subscription to `config.json`/`pubsub.config.json` | M | Resolves Q3. Reuses an existing, already-defined but unused topic. | | D5. One-time backfill job/script to populate postal's table from `fluxx-registration`'s `/v1/clientApplication/delivery-route` (bulk) endpoint (already exists: `getDeliveryRoutes()`) | S | Can reuse the existing bulk endpoint as-is. | | D6. Update `DeliveryService.selectDeliverer` to read from the new local repository first, falling back to the live `RegistrationService` HTTP call on cache miss (keeps current behavior as a safety net) | M | Directly modifies the existing, well-isolated `deliveryService.ts`. | | D7. Docker-compose / dev-container updates for postal to bring up its own DB dependency locally (mirrors the pattern already used by `fluxx-registration`, `fluxx-file-intake`, etc.) | S | Copy-paste-adapt of an existing pattern already used by 4+ sibling repos. | | D8. Unit + contract tests for the new repository, sync handler, and fallback behavior | M | | **Epic D total: \~1 L equivalent (8 stories, includes net-new infra for the repo)** --- ### Epic E — Registration: Delivery Content-Type Preference & Cleanup | Story | Size | Details | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---- | -------------------------------------------------------------------------------------------------------------------------------- | | E1. `fluxx-sql-server` migration: add `DeliveryContentType` column (enum-like `VARCHAR`/lookup table: `Payload`/`SasUrl`/`BlobUrl`) to `[registration].[ClientApplication]` (or a new delivery-preferences table if 1:many is anticipated), default `Payload` | S | Additive, backward-compatible. | | E2. Update `clientApplicationRepository.ts` (`getDeliveryRoute`, `getDeliveryRoutes`) to return the **real** stored `deliveryContentType` instead of the hard-coded `deliveryMethod: 'http'` string found in research | S | Small, isolated fix — the hook already exists (`ClientDeliveryRouteDto.deliveryMethod`), it just needs to stop being hard-coded. | | E3. Add create/update API + schema validation for setting `deliveryContentType` on a client application (admin/onboarding flow) | M | Extends existing `clientApplicationController.ts` create/update paths. | | E4. Confirm/relocate `ontologyActionUrl` semantics now that delivery is postal-owned — decide whether it moves from `[registration].[ClientApplicationExtractor]` to a more general delivery-route concept, or stays extractor-specific and is passed through in the C1 message | M | Directly affects Epic C's message contract; should be resolved alongside C1. | | E5. Publish an `updated-registration` event whenever delivery preference changes (verify this already happens broadly — topic exists — and specifically covers the new field) | S | Topic/publish mechanism already exists; just needs to cover the new field. | | E6. Update `fluxx-registration` OpenAPI docs/README for the new field | S | | **Epic E total: \~1 M-to-L equivalent (6 mostly-small stories)** --- ### Epic F — Shared Data/Schema Cleanup & Migration Hygiene (`fluxx-sql-server`) | Story | Size | Details | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---- | ------------------------------------------------------------------------------------------------------------------ | | F1. Consolidated migration review: confirm no breaking changes are needed to existing `ExtractorJob`/`ClassifierJob`/`SplitterJob` tables — new event-driven flow reuses `runId`-based correlation already in place | S | Research/validation story, likely no code change. | | F2. Add new migration(s) for Epic D (postal schema) and Epic E (delivery content type) — bundle release ordering so `fluxx-sql-server` ships ahead of the services that depend on the new columns/tables | S | Coordination/sequencing story. | | F3. Decide fate of now-partially-unused polling columns (`pollingUrl`, `pollAttemptCount`, `nextPollAt`, lock columns) — keep for manual/backfill retry tooling vs. deprecate; document decision, no immediate schema change recommended | S | Avoid premature deletion; these remain useful for ops/manual re-trigger even after auto-polling is removed. | | F4. Add seed/test data for a second (non-Palantir) client application with each of the 3 `deliveryContentType` values, to support cross-service integration testing | S | Mirrors existing seed pattern (`V1.5.0__docai_client_app_seed.sql`, `V1.22.0__createPalintrSeedRegistration.sql`). | **Epic F total: \~1 S-to-M equivalent (4 small stories)** --- ### Epic G — File Intake: Confirm Fit for Palantir Entry Point (Low Risk) | Story | Size | Details | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | G1. Confirm `POST /v1/upload/:applicationId` meets Palantir's per-form single-PDF upload pattern as-is (auth, file-count limits, response shape) — likely **no code change**, verification only | S | Research indicates existing endpoint already fits; treat as a verification spike, not a build story. | | G2. If Palantir isn't yet a registered/onboarded client application for file-intake (distinct from its extractor registration), complete onboarding via `fluxx-registration` | S | Administrative/config story, not necessarily code. | | G3. Confirm `validated-file` → `accepted-file` path requires no changes to support the higher/parallel volume implied by per-form uploads (vs. previously batching) | S | Verification story; flag if load/perf testing is warranted. | | G4. Accept a caller-supplied `fileId` on `POST /v1/upload/:applicationId` (new optional field/part in the multipart request), validate it with the existing `FileUtils.isValidGuid()`, and use it in `fileService.insert()` in place of `FileUtils.generateFileId()` when present | M | Core code change for R10. Today `handleMultiFileUpload`/`handleChildFileUpload` always call `FileUtils.generateFileId()` unconditionally — this becomes conditional on caller input. | | G5. Enforce uniqueness/collision handling for a caller-supplied `fileId`: reject with `409 Conflict` (per Q6 default) if the ID already exists in the `File` table before insert | S | `FileID` is already the table's primary key; this adds an explicit pre-check + clear error response instead of relying on a raw DB constraint violation. | | G6. Preserve backward compatibility: client applications that do **not** supply a `fileId` continue to get one server-generated via `FileUtils.generateFileId()`, unchanged | S | Ensures non-Palantir clients (and existing split-child uploads via `uploadChildFile`, which should keep generating their own `fileId` per Q6) are unaffected. | | G7. Update `fluxx-file-intake` OpenAPI schema/docs and README to document the new optional caller-supplied `fileId` field | S | | **Epic G total: \~1 M equivalent (7 stories) — was the lowest-risk epic; the FileID change adds real (but still modest) build work** --- ### Epic H — Status Check, Observability, Testing & Rollout | Story | Size | Details | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | H1. Add `fluxx-statuscheck` subscriptions for any new topics introduced (classifier/extractor completion topics, if Q1 resolves to Service Bus) so file status tracking stays complete end-to-end | M | Statuscheck already subscribes broadly (`status-accepted-file`, `status-classified-file`, `status-classifier-job-status`, `status-problem`); this extends the same pattern. | | H2. Add `to-send`/delivery outcome visibility to status tracking (currently no consumer of postal's delivery success/failure) | M | New for the platform — today nothing surfaces "was Palantir actually notified" outside postal's own logs. | | H3. End-to-end integration test environment/script: upload → classify → split → extract → deliver, using local Docker Compose emulators already present in every repo | L | Cross-repo test harness; highest-value regression protection for this whole initiative given 4+ services change trigger/completion behavior simultaneously. | | H4. Feature-flag / config-driven rollout plan per client application (start with Palantir only), with rollback path to the current poll-based flow | M | Directly supports Q4's phased-cutover assumption; coordinate with Epics A–C's feature flags. | | H5. Runbook/update to `physical_architecture.md` and topic reference docs (`topics-subscriptions-reference.md`) across affected repos to reflect final, shipped topology | S | Docs currently describe some topics (`to-send`, `outbox`) inconsistently with what's implemented; needs a cleanup pass once the above ships. | **Epic H total: \~1 L equivalent (5 stories, includes one explicit L for the E2E harness)** --- ## 7. Summary Rollup | Epic | Focus | Rough Size | | ---- | ----------------------------------------------------------------- | ---------- | | A | Classifier — event-driven completion | L | | B | Extractor — split-file trigger + event-driven completion | L | | C | Decouple Palantir delivery → route via Postal | L | | D | Postal — standalone DB for registrations/delivery preference | L | | E | Registration — delivery content-type preference | M–L | | F | SQL Server — schema/migration hygiene | S–M | | G | File Intake — entry-point confirmation + Palantir-supplied FileID | M | | H | Statuscheck, observability, E2E testing, rollout | L | **Overall:** 5 Large epics, 1 Medium epic, 1 Medium/Small epic, 1 Small-to-Medium epic — consistent with this being a cross-cutting architectural shift (poll → event-driven; extractor-direct → postal-mediated delivery; server-generated → Palantir-generated `FileID`) touching 7 of the 8 `fluxx-*` services plus the shared SQL schema. The three biggest unknowns that could swing sizing up further are **Q1** (Databricks' actual completion-notification capability — webhook support would add new inbound HTTP surface + auth work to both classifier and extractor), **Q4** (whether a parallel/flagged rollout is required per client application, which roughly doubles the trigger-handling work in Epics A and B), and **Q6** (whether split-child extraction/delivery correlates on the child's own `fileId` or the original Palantir `fileId`, which affects the message contracts touched in Epics B and C). ## 8. Suggested Sequencing 1. **Epic G (G4–G6, FileID acceptance)** — land Palantir-supplied `FileID` support in `fluxx-file-intake` first; every other epic's correlation-key design (A3/A4, B4/B5, C1) depends on `fileId` being reliably caller-supplied and validated at the point of entry. 2. **Epic F (partial) + Epic E** — schema groundwork (delivery content type, seed data) so downstream services have something to build against. 3. **Epic D** — stand up postal's DB and sync mechanism (can proceed in parallel with #2; no hard dependency). 4. **Epic C** — wire extractor → `to-send` → postal → Palantir using the *existing* poll-based completion as the trigger for now (de-risks the biggest behavioral change — delivery ownership — independently of the polling removal). 5. **Epic A and Epic B (completion-handling + Databricks request stories, including `fileId` propagation)** — once Q1 and Q6 are resolved, swap each service's completion trigger from polling to event-driven and add `fileId` to the classify/extract request payloads, reusing the delivery path already proven in step 4. 6. **Epic B (split-file trigger rewire)** — switch extractor's inbound trigger from `accepted-file` to `split-file`, feature-flagged per Q4. 7. **Epic G (G1–G3, remaining verification/onboarding stories)** — confirm/onboard file-intake fit beyond the FileID work already landed in step 1 (can happen any time, independent of the above). 8. **Epic H** — observability and E2E testing threaded throughout, with the full harness (H3) finalized once steps 1–6 are functionally complete, then rollout (H4) client-application-by-client-application starting with Palantir.