Document Splitter Service:-Architecture Design

1. Background & Problem Statement

Once the classifier service has identified the document types present in an uploaded file, the platform must logically separate those document segments so they can be routed to the correct extractor independently. Today this responsibility sits inside ClassifyFileTool.ts in Paperboy — classification and splitting are tightly coupled in a service. This creates several challenges:

The split logic cannot be scaled or replaced independently
Failures in splitting silently swallow errors and leave the parent file in a partial state

Service	Repository	Responsibility
fluxx-classifier	`fluxx-classifier` (new repo)	Consume a PDF, identify document types and page boundaries, return a `ClassificationResult`
fluxx-document-splitter	`fluxx-document-splitter` (new repo)	Consume a `FileClassifiedEvent` + original PDF reference, produce discrete split documents stored in blob storage, return a `FileSplitEvent`

2. Current State (As-Is) in Paperboy

2.1 Where Splitting Lives Today

File	Responsibility
`backend/src/api/v1/agent/tools/ClassifyFileTool.ts`	Classifies AND splits — does both in one `execute()` call
`backend/src/utils/pdfProcessor.ts`	`splitPDFByPageRanges()` — low-level PDF page copy using `pdf-lib`
`backend/src/api/v1/files/model.ts`	Persists split child file records (`parent_file_id` FK)
`local-deploy/db/migrations/V3__create_filedata.sql`	`FileData` table with `parent_file_id`, `form_index` columns

2.2 Current Split Data Flow

  ClassifyFileTool.execute()
         │
         ├─ 1. Download PDF from Azure Blob Storage
         ├─ 2. Extract text via pdf-parse
         ├─ 3. Chunk text (10 pages/chunk)
         ├─ 4. Call Azure OpenAI per chunk → classification JSON
         ├─ 5. Merge chunk results → master classification JSON
         ├─ 6. Upload classification JSON to Blob (classifications/)
         ├─ 7. Update FileData.status = 'classified'
         │
         └─ IF is_multi_form = true OR forms.length > 1:
              ├─ 8. Build page ranges from forms[]
              ├─ 9. Call splitPDFByPageRanges() → Buffer[]
              ├─ 10. Upload each split PDF to Blob (files/)
              └─ 11. Insert child FileData rows (parent_file_id = parent.id)

2.3 Classification JSON Schema (Current Output)

{
  "is_multi_form": true,
  "num_pages": 12,
  "classified_in_chunks": true,
  "num_chunks": 2,
  "forms": [
    {
      "form_type": "Form 1040",
      "page_start": 1,
      "page_end": 4,
      "confidence": 0.92
    },
    {
      "form_type": "Schedule C",
      "page_start": 5,
      "page_end": 8,
      "confidence": 0.88
    },
    {
      "form_type": "Extra form data",
      "page_ranges": [
        { "start": 9, "end": 10 },
        { "start": 11, "end": 12 }
      ],
      "confidence": 0.5
    }
  ]
}

2.4 FileData Table (Split Records)

FileData
  id                       (GUID, PK)
  parent_file_id           (GUID, FK → FileData.id)    ← NULL for originals (uploaded file)
  form_index               (INT)                       ← 0-based position in split
  form_type                (NVARCHAR)                  ← e.g. "Form 1040"
  blob_path                (NVARCHAR)                  ← files/{splitFileName}.pdf
  classification_json_path (NVARCHAR)                  ← shared with parent
  status                   (NVARCHAR)                  ← 'classified' after split

3. Challenges with the Current Architecture

#	Problem	Impact
P1	Classify + Split are coupled in one tool	Cannot split without re-classifying; cannot replace split logic independently

4. Recommended Architecture

4.1 Design Principles

Single Responsibility — The Document Splitter receives a classification result and emits addressable split documents. It does not classify.
Contract-Driven — Input is a well-defined FileClassifiedEvent message. Output is a FileSplitEvent message and a split manifest persisted to blob storage.
Fault Isolated — Splitting failures must not corrupt the parent file record; errors are surfaced explicitly via a ProblemEvent published to the problem topic.
Independently Scalable — The splitter is callable independently of classification, enabling re-split on demand.

4.2 Proposed Service Boundary

┌─────────────────────────────────────────────────────────────────┐
│                     Agentic Loop (Orchestrator)                 │
│                                                                 │
│   fluxx-classifier ──► output ClassificationResult JSON in Blob │
│         │                                                       │
│         │  published a message                                  │
│         │                                                       │
│         ▼  (if multiple form type)                              │
│   fluxx-document-splitter ──► output FileSplitEvent             │
│         │                                                       │
│         │  published a message                                  │
│         │                                                       │
│         ▼                                                       │
│   ExtractFileTool (operates on child FileData rows)             │
└─────────────────────────────────────────────────────────────────┘

Classification and Document Splitter become independent with a well-defined handoff.

4.3 New Service: `fluxx-document-splitter`

Location: https://github.com/CLADevOps/fluxx-document-splitter

Consumed message (FileClassifiedEvent) — published by classifier:

{
  clientApplicationID: string;  // application/tenant identifier
  internalURL: string;           // blob URL of the original PDF
  classificationResult: string;  // blob URL of the classification JSON
  fileId: string;                // identifier of the file record
}

Processing steps (ClassifiedFileHandler):

Download and parse the classification JSON from blob using classificationResult URL
If is_multi_form === false or forms.length <= 1, exit gracefully — no split needed
Download the original PDF from blob using internalURL
Build a timestamp prefix (YYYYMMDDHHmmss) and derive baseName / ext from the filename
For each form in forms[]:
- Non-contiguous page_ranges: create one split PDF per segment, named {timestamp}_{baseName}_{formType}_Part{n}.pdf — uploaded to {clientApplicationID}/files_splitted/
- Contiguous page_start / page_end: create one split PDF named {timestamp}_{baseName}_{formType}.pdf — uploaded to {clientApplicationID}/files_splitted/
- Forms with neither field are logged as a warning and skipped
Build and upload a split manifest JSON to {clientApplicationID}/splits/{timestamp}_{baseName}_splitResult.json
Publish a FileSplitEvent to the split-file topic
On any error: publish a ProblemEvent to the problem topic and re-throw

Note: The current implementation does not write to any database. There are no FileData row inserts or status updates. All output is blob-resident (split PDFs + manifest) and message-resident (FileSplitEvent).

Published message (FileSplitEvent) — consumed by downstream (e.g. extractor):

{
  clientApplicationID: string;
  totalForms: number;
  splitFiles: Array<{
    fileName: string;
    formType: string;
    formIndex: number;   // 0-based index in forms[]
    pageStart: number;
    pageEnd: number;
    blobPath: string;    // full Azure Blob URL of the split PDF
  }>;
  timestamp: string;     // ISO-8601
}

Split manifest (blob artifact):

{
  "clientApplicationID": "F156A0E0-...",
  "originalFile": "006ADDEC-63ED-496E-A645-F83CB23450DB.pdf",
  "timestamp": "2026-04-17T12:00:00.000Z",
  "totalForms": 3,
  "splitFiles": [
    {
      "fileName": "20260417120000_006ADDEC-63ED-496E-A645-F83CB23450DB_Form_1040.pdf",
      "formType": "Form 1040",
      "formIndex": 0,
      "pageStart": 1,
      "pageEnd": 4,
      "blobPath": "https://..."
    }
  ]
}

Error event (ProblemEvent) — published on failure:

{
  timestamp: string;          // ISO-8601
  problem: {
    type: string;             // RFC-7807 problem type URI
    title: string;
    detail: string;           // error message
    instance: string;         // clientApplicationID or 'unknown-client'
    status: number;           // 404 for blob not found, 500 otherwise
  };
}

4.4 Updated `fluxx-classifier`

fluxx-classifier is simplified to classification only:

Removes all split logic
Sets FileData.status = 'classified' on success
Returns the classification result; the agentic loop decides whether to call SplitDocumentTool next

This makes the agentic loop’s workflow explicit:

classify_file → (if multi_form) → split_document → extract_file

4.5 Service Bus Topology

The service uses Azure Service Bus for all inter-service communication. Topic and subscription names are sourced from Azure App Configuration at runtime.

Direction	Topic	Subscription	Event type
Inbound	`classified-file`	`document-splitter-classified-file`	`FileClassifiedEvent`
Outbound	`split-file`	`extractor-split-file` (downstream)	`FileSplitEvent`
Outbound (errors)	`problem`	`status-problem` (downstream)	`ProblemEvent`

Retry behaviour:

MaxDeliveryCount: 3 on all subscriptions
Success path: message is completed after the handler returns
Failure path: message is abandoned, making it eligible for retry up to the max delivery count

4.6 Blob Storage Layout

All split artifacts are written to the files container under paths scoped to clientApplicationID:

files/                                           ← Azure container name
  {clientApplicationID}/
    files_splitted/
      {timestamp}_{fileName}_{FormType}.pdf              ← contiguous split
      {timestamp}_{fileName}_{FormType}_Part1.pdf        ← non-contiguous segment 1
      {timestamp}_{fileName}_{FormType}_Part2.pdf        ← non-contiguous segment 2
    splits/
      {timestamp}_{fileName}_splitResult.json            ← split manifest
    classifications/
      {timestamp}_{fileName}_classificationResult.json               ← written by fluxx-classifier

{timestamp} format: YYYYMMDDHHmmss
{FormType} is sanitized to [a-zA-Z0-9_-] characters only

4.7 End-to-End Data Flow (To-Be)

1. User: "Classify and process the Johnson tax return"
         │
         ▼
2. AgenticLoop → classify_file(fileName="johnson_tax_return.pdf")
         │
         ▼
3. ClassifyFileTool (fluxx-classifier)
   ├─ Download PDF from Blob
   ├─ Chunk text → LLM per chunk → merge → ClassificationResult
   ├─ Upload classification JSON → {clientApplicationID}/classifications/
   ├─ Update FileData.status = 'classified'
   └─ Publish FileClassifiedEvent: { clientApplicationID, internalURL, classificationResult, fileId }
         │
         ▼  (message consumed if is_multi_form=true)
4. fluxx-document-splitter → ClassifiedFileHandler.handleClassifiedFile()
         │
         ▼
5. Processing
   ├─ Download & parse ClassificationResult from blob (classificationResult URL)
   ├─ is_multi_form === false OR forms.length ≤ 1 → exit gracefully (no split)
   ├─ Download original PDF (internalURL)
   ├─ For each form[i]:
   │    ├─ page_ranges[] present → splitPdfByPageRanges() per segment → upload _Part{n}.pdf
   │    ├─ page_start/page_end present → splitPdfByPageRanges() → upload {formType}.pdf
   │    └─ neither → warn and skip
   ├─ Build split manifest → upload to {clientApplicationID}/splits/
   ├─ Publish FileSplitEvent to split-file topic
   └─ On error → publish ProblemEvent to problem topic + rethrow → abandoned for retry
         │
         ▼
6. To Be Determined (TBD)

5. Component Diagram

WORK IN PROGRESS (Dexter is creating the diagram, this include the Classifier and Document Splitter)
This section will be updated soon.

7. Open Questions

#	Question	Recommended Action
Q1	Page boundary accuracy: The current character-offset chunking (`charsPerPage = totalChars / numPages`) does not respect actual page boundaries in the text stream. Are classification boundaries accurate enough for production?	Spike a page-aware text extraction approach using `pdf-lib` page iteration before committing to chunked strategy
Q2	Re-split on classification change: If a user re-classifies a file after it has already been split, what happens to the existing child records? Current code creates duplicates.	Define a `re-split` policy: either hard-delete + recreate children, or version-stamp splits
Q3	Non-contiguous `page_ranges` (e.g., “Extra form data” spanning pages 9-10 and 11-12): Should these be collapsed into one split file or remain as separate records?	Currently the code splits them as `{timestamp}_{filename}_Extra_form_data_Part1.pdf`, `{timestamp}_{filename}_Extra_form_data_Part2.pdf`
Q4	Image-only PDFs: `pdf-parse` returns empty text for scanned PDFs; the classifier returns `Unidentified Form Data` for the whole document; splitting produces one child record. Is that the intended behavior?	Determine if Azure Document Intelligence OCR should pre-process image PDFs before classification
Q5	Confidence threshold: Classification results include a `confidence` score (0.0–1.0) per form. There is currently no minimum threshold; a 10% confidence classification still triggers a split.	Define a minimum confidence threshold (suggested: 0.6) below which forms are grouped into “Extra form data”
Q6	Max split count: No limit on how many child files can be produced from one parent. A 200-page file with 200 single-page forms would create 200 blob uploads synchronously.	Should define maximum page that can be splitted or classified
Q7	No database writes: The current implementation does not write `FileData` rows or update parent file status. How will downstream consumers (e.g. the extractor) discover the split file records if no DB rows are created?	Decide whether the extractor should consume `FileSplitEvent` directly (event-driven, no DB lookup) or whether a DB write step needs to be added to this service

8. Risks

#	Risk	Likelihood	Impact	Mitigation
R2	LLM maybe misidentifies page boundaries due to character-offset text chunking (need to validate further)	Medium	High	Validate split output page counts against expected counts; alert if delta > 1 page
R4	Blob storage costs increase linearly with split count (original + N children stored in `files_splitted/`)	Low (dev) / High (prod volume)	Medium	Evaluate whether split children should be stored permanently or generated on-demand and cached with a TTL
R5	No database integration means split state is only recoverable by re-reading the manifest from blob storage. If the manifest upload fails after split PDFs are already uploaded, blobs are orphaned with no way to track them.	Medium	Medium	Add an atomic write step or store the manifest before uploading splits; alternatively introduce a DB write to track split status

9. References Repository PaperBoy

https://github.com/MWChapelCLA/Paperboy

Resource	Path / Repo
Current classification + split logic (to be refactored)	`backend/src/api/v1/agent/tools/ClassifyFileTool.ts`
PDF splitting utility	`backend/src/utils/pdfProcessor.ts`
Blob storage helper	`backend/src/utils/blobHelper.ts`
Tool registry pattern	`backend/src/api/v1/agent/toolRegistry.ts`
Agentic loop	`backend/src/api/v1/agent/agenticLoop.ts`
Extract tool (downstream consumer)	`backend/src/api/v1/agent/tools/ExtractFileTool.ts`
Architecture docs	`docs/ARCHITECTURE.md`