> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://cla-demo.postmansolutions.com/design/research-and-spikes/document-splitter-service-architecture-design/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://cla-demo.postmansolutions.com/_mcp/server. # Document Splitter Service:-Architecture Design ## 1. Background & Problem Statement Once the classifier service has identified the document types present in an uploaded file, the platform must logically separate those document segments so they can be routed to the correct extractor independently. Today this responsibility sits **inside `ClassifyFileTool.ts`** in Paperboy — classification and splitting are tightly coupled in a service. This creates several challenges: * The split logic cannot be scaled or replaced independently * Failures in splitting silently swallow errors and leave the parent file in a partial state | Service | Repository | Responsibility | | --------------------------- | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | | **fluxx-classifier** | `fluxx-classifier` (new repo) | Consume a PDF, identify document types and page boundaries, return a `ClassificationResult` | | **fluxx-document-splitter** | `fluxx-document-splitter` (new repo) | Consume a `FileClassifiedEvent` + original PDF reference, produce discrete split documents stored in blob storage, return a `FileSplitEvent` | ## 2. Current State (As-Is) in Paperboy ### 2.1 Where Splitting Lives Today | File | Responsibility | | ---------------------------------------------------- | ------------------------------------------------------------------ | | `backend/src/api/v1/agent/tools/ClassifyFileTool.ts` | **Classifies AND splits** — does both in one `execute()` call | | `backend/src/utils/pdfProcessor.ts` | `splitPDFByPageRanges()` — low-level PDF page copy using `pdf-lib` | | `backend/src/api/v1/files/model.ts` | Persists split child file records (`parent_file_id` FK) | | `local-deploy/db/migrations/V3__create_filedata.sql` | `FileData` table with `parent_file_id`, `form_index` columns | ### 2.2 Current Split Data Flow ``` ClassifyFileTool.execute() │ ├─ 1. Download PDF from Azure Blob Storage ├─ 2. Extract text via pdf-parse ├─ 3. Chunk text (10 pages/chunk) ├─ 4. Call Azure OpenAI per chunk → classification JSON ├─ 5. Merge chunk results → master classification JSON ├─ 6. Upload classification JSON to Blob (classifications/) ├─ 7. Update FileData.status = 'classified' │ └─ IF is_multi_form = true OR forms.length > 1: ├─ 8. Build page ranges from forms[] ├─ 9. Call splitPDFByPageRanges() → Buffer[] ├─ 10. Upload each split PDF to Blob (files/) └─ 11. Insert child FileData rows (parent_file_id = parent.id) ``` ### 2.3 Classification JSON Schema (Current Output) ```json { "is_multi_form": true, "num_pages": 12, "classified_in_chunks": true, "num_chunks": 2, "forms": [ { "form_type": "Form 1040", "page_start": 1, "page_end": 4, "confidence": 0.92 }, { "form_type": "Schedule C", "page_start": 5, "page_end": 8, "confidence": 0.88 }, { "form_type": "Extra form data", "page_ranges": [ { "start": 9, "end": 10 }, { "start": 11, "end": 12 } ], "confidence": 0.5 } ] } ``` ### 2.4 FileData Table (Split Records) ``` FileData id (GUID, PK) parent_file_id (GUID, FK → FileData.id) ← NULL for originals (uploaded file) form_index (INT) ← 0-based position in split form_type (NVARCHAR) ← e.g. "Form 1040" blob_path (NVARCHAR) ← files/{splitFileName}.pdf classification_json_path (NVARCHAR) ← shared with parent status (NVARCHAR) ← 'classified' after split ``` --- ## 3. Challenges with the Current Architecture | # | Problem | Impact | | -- | ---------------------------------------- | ----------------------------------------------------------------------------- | | P1 | Classify + Split are coupled in one tool | Cannot split without re-classifying; cannot replace split logic independently | --- ## 4. Recommended Architecture ### 4.1 Design Principles 1. **Single Responsibility** — The Document Splitter receives a classification result and emits addressable split documents. It does not classify. 2. **Contract-Driven** — Input is a well-defined `FileClassifiedEvent` message. Output is a `FileSplitEvent` message and a split manifest persisted to blob storage. 3. **Fault Isolated** — Splitting failures must not corrupt the parent file record; errors are surfaced explicitly via a `ProblemEvent` published to the `problem` topic. 4. **Independently Scalable** — The splitter is callable independently of classification, enabling re-split on demand. ### 4.2 Proposed Service Boundary ``` ┌─────────────────────────────────────────────────────────────────┐ │ Agentic Loop (Orchestrator) │ │ │ │ fluxx-classifier ──► output ClassificationResult JSON in Blob │ │ │ │ │ │ published a message │ │ │ │ │ ▼ (if multiple form type) │ │ fluxx-document-splitter ──► output FileSplitEvent │ │ │ │ │ │ published a message │ │ │ │ │ ▼ │ │ ExtractFileTool (operates on child FileData rows) │ └─────────────────────────────────────────────────────────────────┘ ``` Classification and Document Splitter become **independent** with a well-defined handoff. ### 4.3 New Service: `fluxx-document-splitter` **Location:** `https://github.com/CLADevOps/fluxx-document-splitter` **Consumed message (`FileClassifiedEvent`) — published by classifier:** ```typescript { clientApplicationID: string; // application/tenant identifier internalURL: string; // blob URL of the original PDF classificationResult: string; // blob URL of the classification JSON fileId: string; // identifier of the file record } ``` **Processing steps (`ClassifiedFileHandler`):** 1. Download and parse the classification JSON from blob using `classificationResult` URL 2. If `is_multi_form === false` or `forms.length <= 1`, exit gracefully — no split needed 3. Download the original PDF from blob using `internalURL` 4. Build a timestamp prefix (`YYYYMMDDHHmmss`) and derive `baseName` / `ext` from the filename 5. For each form in `forms[]`: * **Non-contiguous `page_ranges`**: create one split PDF per segment, named `{timestamp}_{baseName}_{formType}_Part{n}.pdf` — uploaded to `{clientApplicationID}/files_splitted/` * **Contiguous `page_start` / `page_end`**: create one split PDF named `{timestamp}_{baseName}_{formType}.pdf` — uploaded to `{clientApplicationID}/files_splitted/` * Forms with neither field are logged as a warning and skipped 6. Build and upload a split manifest JSON to `{clientApplicationID}/splits/{timestamp}_{baseName}_splitResult.json` 7. Publish a `FileSplitEvent` to the `split-file` topic 8. On any error: publish a `ProblemEvent` to the `problem` topic and re-throw > **Note:** The current implementation does **not** write to any database. There are no `FileData` row inserts or status updates. All output is blob-resident (split PDFs + manifest) and message-resident (`FileSplitEvent`). **Published message (`FileSplitEvent`) — consumed by downstream (e.g. extractor):** ```typescript { clientApplicationID: string; totalForms: number; splitFiles: Array<{ fileName: string; formType: string; formIndex: number; // 0-based index in forms[] pageStart: number; pageEnd: number; blobPath: string; // full Azure Blob URL of the split PDF }>; timestamp: string; // ISO-8601 } ``` **Split manifest (blob artifact):** ```json { "clientApplicationID": "F156A0E0-...", "originalFile": "006ADDEC-63ED-496E-A645-F83CB23450DB.pdf", "timestamp": "2026-04-17T12:00:00.000Z", "totalForms": 3, "splitFiles": [ { "fileName": "20260417120000_006ADDEC-63ED-496E-A645-F83CB23450DB_Form_1040.pdf", "formType": "Form 1040", "formIndex": 0, "pageStart": 1, "pageEnd": 4, "blobPath": "https://..." } ] } ``` **Error event (`ProblemEvent`) — published on failure:** ```typescript { timestamp: string; // ISO-8601 problem: { type: string; // RFC-7807 problem type URI title: string; detail: string; // error message instance: string; // clientApplicationID or 'unknown-client' status: number; // 404 for blob not found, 500 otherwise }; } ``` ### 4.4 Updated `fluxx-classifier` fluxx-classifier is **simplified to classification only**: * Removes all split logic * Sets `FileData.status = 'classified'` on success * Returns the classification result; the agentic loop decides whether to call `SplitDocumentTool` next This makes the agentic loop's workflow explicit: ``` classify_file → (if multi_form) → split_document → extract_file ``` ### 4.5 Service Bus Topology The service uses Azure Service Bus for all inter-service communication. Topic and subscription names are sourced from Azure App Configuration at runtime. | Direction | Topic | Subscription | Event type | | --------------------- | ----------------- | ----------------------------------- | --------------------- | | **Inbound** | `classified-file` | `document-splitter-classified-file` | `FileClassifiedEvent` | | **Outbound** | `split-file` | `extractor-split-file` (downstream) | `FileSplitEvent` | | **Outbound (errors)** | `problem` | `status-problem` (downstream) | `ProblemEvent` | **Retry behaviour:** * `MaxDeliveryCount: 3` on all subscriptions * Success path: message is `completed` after the handler returns * Failure path: message is `abandoned`, making it eligible for retry up to the max delivery count ### 4.6 Blob Storage Layout All split artifacts are written to the **`files`** container under paths scoped to `clientApplicationID`: ``` files/ ← Azure container name {clientApplicationID}/ files_splitted/ {timestamp}_{fileName}_{FormType}.pdf ← contiguous split {timestamp}_{fileName}_{FormType}_Part1.pdf ← non-contiguous segment 1 {timestamp}_{fileName}_{FormType}_Part2.pdf ← non-contiguous segment 2 splits/ {timestamp}_{fileName}_splitResult.json ← split manifest classifications/ {timestamp}_{fileName}_classificationResult.json ← written by fluxx-classifier ``` > `{timestamp}` format: `YYYYMMDDHHmmss`\ > `{FormType}` is sanitized to `[a-zA-Z0-9_-]` characters only ### 4.7 End-to-End Data Flow (To-Be) ``` 1. User: "Classify and process the Johnson tax return" │ ▼ 2. AgenticLoop → classify_file(fileName="johnson_tax_return.pdf") │ ▼ 3. ClassifyFileTool (fluxx-classifier) ├─ Download PDF from Blob ├─ Chunk text → LLM per chunk → merge → ClassificationResult ├─ Upload classification JSON → {clientApplicationID}/classifications/ ├─ Update FileData.status = 'classified' └─ Publish FileClassifiedEvent: { clientApplicationID, internalURL, classificationResult, fileId } │ ▼ (message consumed if is_multi_form=true) 4. fluxx-document-splitter → ClassifiedFileHandler.handleClassifiedFile() │ ▼ 5. Processing ├─ Download & parse ClassificationResult from blob (classificationResult URL) ├─ is_multi_form === false OR forms.length ≤ 1 → exit gracefully (no split) ├─ Download original PDF (internalURL) ├─ For each form[i]: │ ├─ page_ranges[] present → splitPdfByPageRanges() per segment → upload _Part{n}.pdf │ ├─ page_start/page_end present → splitPdfByPageRanges() → upload {formType}.pdf │ └─ neither → warn and skip ├─ Build split manifest → upload to {clientApplicationID}/splits/ ├─ Publish FileSplitEvent to split-file topic └─ On error → publish ProblemEvent to problem topic + rethrow → abandoned for retry │ ▼ 6. To Be Determined (TBD) ``` --- ## 5. Component Diagram ``` WORK IN PROGRESS (Dexter is creating the diagram, this include the Classifier and Document Splitter) This section will be updated soon. ``` --- ## 7. Open Questions | # | Question | Recommended Action | | -- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Q1 | **Page boundary accuracy**: The current character-offset chunking (`charsPerPage = totalChars / numPages`) does not respect actual page boundaries in the text stream. Are classification boundaries accurate enough for production? | Spike a page-aware text extraction approach using `pdf-lib` page iteration before committing to chunked strategy | | Q2 | **Re-split on classification change**: If a user re-classifies a file after it has already been split, what happens to the existing child records? Current code creates duplicates. | Define a `re-split` policy: either hard-delete + recreate children, or version-stamp splits | | Q3 | **Non-contiguous `page_ranges`** (e.g., "Extra form data" spanning pages 9-10 and 11-12): Should these be collapsed into one split file or remain as separate records? | Currently the code splits them as `{timestamp}_{filename}_Extra_form_data_Part1.pdf`, `{timestamp}_{filename}_Extra_form_data_Part2.pdf` | | Q4 | **Image-only PDFs**: `pdf-parse` returns empty text for scanned PDFs; the classifier returns `Unidentified Form Data` for the whole document; splitting produces one child record. Is that the intended behavior? | Determine if Azure Document Intelligence OCR should pre-process image PDFs before classification | | Q5 | **Confidence threshold**: Classification results include a `confidence` score (0.0–1.0) per form. There is currently no minimum threshold; a 10% confidence classification still triggers a split. | Define a minimum confidence threshold (suggested: 0.6) below which forms are grouped into "Extra form data" | | Q6 | **Max split count**: No limit on how many child files can be produced from one parent. A 200-page file with 200 single-page forms would create 200 blob uploads synchronously. | Should define maximum page that can be splitted or classified | | Q7 | **No database writes**: The current implementation does not write `FileData` rows or update parent file status. How will downstream consumers (e.g. the extractor) discover the split file records if no DB rows are created? | Decide whether the extractor should consume `FileSplitEvent` directly (event-driven, no DB lookup) or whether a DB write step needs to be added to this service | --- ## 8. Risks | # | Risk | Likelihood | Impact | Mitigation | | -- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ | ------ | -------------------------------------------------------------------------------------------------------------------------------- | | R2 | LLM maybe misidentifies page boundaries due to character-offset text chunking (need to validate further) | Medium | High | Validate split output page counts against expected counts; alert if delta > 1 page | | R4 | Blob storage costs increase linearly with split count (original + N children stored in `files_splitted/`) | Low (dev) / High (prod volume) | Medium | Evaluate whether split children should be stored permanently or generated on-demand and cached with a TTL | | R5 | No database integration means split state is only recoverable by re-reading the manifest from blob storage. If the manifest upload fails after split PDFs are already uploaded, blobs are orphaned with no way to track them. | Medium | Medium | Add an atomic write step or store the manifest before uploading splits; alternatively introduce a DB write to track split status | --- ## 9. References Repository PaperBoy [https://github.com/MWChapelCLA/Paperboy](https://github.com/MWChapelCLA/Paperboy) | Resource | Path / Repo | | ------------------------------------------------------- | ---------------------------------------------------- | | Current classification + split logic (to be refactored) | `backend/src/api/v1/agent/tools/ClassifyFileTool.ts` | | PDF splitting utility | `backend/src/utils/pdfProcessor.ts` | | Blob storage helper | `backend/src/utils/blobHelper.ts` | | Tool registry pattern | `backend/src/api/v1/agent/toolRegistry.ts` | | Agentic loop | `backend/src/api/v1/agent/agenticLoop.ts` | | Extract tool (downstream consumer) | `backend/src/api/v1/agent/tools/ExtractFileTool.ts` | | Architecture docs | `docs/ARCHITECTURE.md` | --- ## Docs - [Current Implementation and To Be](https://cla-demo.postmansolutions.com/design/research-and-spikes/document-splitter-service-architecture-design/current-implementation-and-to-be.md) - [Paperboy Current Process](https://cla-demo.postmansolutions.com/design/research-and-spikes/document-splitter-service-architecture-design/paperboy-current-process.md)