High Level Design-(HLD)---fluxx hqx classifier

Service Name: fluxx-hqx-classifier

Version: 1.0.0

Date: March 27, 2026

Status: Draft

Overview

Purpose

The fluxx-hqx-classifier service provides an asynchronous PDF classification system specifically designed for HQ Exchange financial and accounting documents. The service processes uploaded PDF documents by splitting them into pages, converting pages to images, and using LLM (Large Language Model) to classify documents into predefined HQ Exchange document categories. The service batches images into groups of 20 for efficient processing and returns a merged, collated JSON output identifying the document type.

Business Context

HQ Exchange is a platform for financial document exchange and processing. This classifier service enables automated categorization of uploaded financial documents (such as trial balances, bank statements, general ledgers, etc.) to streamline document routing, processing workflows, and compliance requirements within the HQ Exchange ecosystem.

Key Features

HQ Exchange Specialization: Optimized for financial and accounting document classification
Predefined Document Taxonomy: Classifies into 28+ predefined HQ Exchange document types
Asynchronous PDF Processing: Non-blocking PDF upload and processing
Page-to-Image Conversion: Converts PDF pages to images for LLM processing
Batch Processing: Groups images into batches of 20 for optimal LLM performance
Multi-Document Support: Handles documents containing multiple document types
Status Tracking: Real-time status monitoring of classification jobs
Structured JSON Output: Returns document type identification with confidence scores

Supported Document Types

The service classifies documents into the following HQ Exchange categories:

Primary Financial Documents:

Trial Balance
General Ledger
General Ledger Detail
Bank Statements
Bank Reconciliations
Cash

Accounts & Reconciliations:

Accounts Payable
Accounts Receivable
Confirmations

Asset Management:

Fixed Assets
Fixed Asset Additions
Fixed Asset Disposals
Prepaid Expenses
Inventory

Liabilities & Agreements:

Debt Agreements
Accrued Expenses
Accrued Payroll
Lease Agreements
Leases

Investment & Financial Instruments:

Investments
Investment Statements
Check Register

Operational & Administrative:

Legal Invoices
Board Minutes
Personnel and Policy Manuals
Revenue
Related Party Transactions

Other:

Other Information Not Previously Requested by CLA

System Architecture

High-Level Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                        API Gateway / LB                          │
└─────────────────────────────────────────────────────────────────┘
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                    fluxx-hqx-classifier API                      │
│  ┌────────────────────────────────────────────────────────────┐ │
│  │  Middleware (CORS, Auth, CSRF, Correlation ID)            │ │
│  └────────────────────────────────────────────────────────────┘ │
│  ┌────────────────────────────────────────────────────────────┐ │
│  │  Route Handlers                                            │ │
│  │  - POST /api/v1/Classifications                            │ │
│  │  - GET /api/v1/Classifications/{jobId}                     │ │
│  └────────────────────────────────────────────────────────────┘ │
│  ┌────────────────────────────────────────────────────────────┐ │
│  │  Service Layer                                             │ │
│  │  - ClassificationService                                   │ │
│  │  - JobStatusService                                        │ │
│  └────────────────────────────────────────────────────────────┘ │
│  ┌────────────────────────────────────────────────────────────┐ │
│  │  Processing Layer                                          │ │
│  │  - PDFSplitterService                                      │ │
│  │  - ImageConverterService                                   │ │
│  │  - BatchProcessorService                                   │ │
│  │  - HQXLLMInvocationService                                 │ │
│  │  - ResultCollatorService                                   │ │
│  └────────────────────────────────────────────────────────────┘ │
│  ┌────────────────────────────────────────────────────────────┐ │
│  │  Repository Layer                                          │ │
│  │  - JobRepository                                           │ │
│  │  - ClassificationResultRepository                          │ │
│  └────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  External Services & Storage                                     │
│  ┌──────────────┐ ┌──────────────┐ ┌──────────────┐            │
│  │ Azure Blob   │ │ Azure Service│ │   SQL DB     │            │
│  │ Storage      │ │     Bus      │ │  (Job Store) │            │
│  └──────────────┘ └──────────────┘ └──────────────┘            │
│  ┌──────────────┐ ┌──────────────┐                             │
│  │  LLM API     │ │ App Insights │                             │
│  │ (Azure       │ │              │                             │
│  │  OpenAI)     │ │              │                             │
│  └──────────────┘ └──────────────┘                             │
└─────────────────────────────────────────────────────────────────┘

Component Overview

| Component | Responsibility |

|-----------|---------------|

| API Layer | Handles HTTP requests, validation, and response formatting |

| Service Layer | Orchestrates business logic and HQX-specific workflow coordination |

| Processing Layer | Executes document processing, conversion, and HQX-specialized LLM invocation |

| Repository Layer | Manages data persistence and retrieval |

| Message Queue | Enables asynchronous job processing |

| Blob Storage | Stores uploaded PDFs, intermediate images, and results |

| Database | Stores job metadata, status, and classification results |

| LLM Service | External API for HQ Exchange document classification |

API Endpoints

1. POST /api/v1/Classifications

Purpose: Initiates asynchronous HQ Exchange document classification job

Request:

POST /api/v1/Classifications
Content-Type: multipart/form-data
  
{
  "file": <PDF binary>,
  "metadata": {
    "clientId": "12345",
    "engagementId": "ENG-2026-001",
    "uploadedBy": "user@company.com"
  }
}

Request Parameters:

|-----------|------|----------|-------------|

Response (201 Created):

{
  "jobId": "550e8400-e29b-41d4-a716-446655440000",
  "status": "in-progress",
  "createdAt": "2026-03-27T12:00:00Z",
  "statusUrl": "/api/v1/Classifications/550e8400-e29b-41d4-a716-446655440000"
}

Response Headers:

Location: /api/v1/Classifications/{jobId}

Error Responses:

400 Bad Request - Invalid file format or missing parameters
401 Unauthorized - Missing or invalid authentication
413 Payload Too Large - PDF exceeds maximum file size (50MB)
422 Unprocessable Entity - Invalid PDF or corrupted file
500 Internal Server Error - Unexpected server error

2. GET /api/v1/Classifications/{jobId}

Purpose: Retrieves HQ Exchange classification job status and results

Request:

GET /api/v1/Classifications/550e8400-e29b-41d4-a716-446655440000

Path Parameters:

|-----------|------|----------|-------------|

Response - In Progress (200 OK):

{
  "jobId": "550e8400-e29b-41d4-a716-446655440000",
  "status": "in-progress",
  "progress": {
    "totalPages": 45,
    "processedPages": 30,
    "percentComplete": 67
  },
  "createdAt": "2026-03-27T12:00:00Z",
  "updatedAt": "2026-03-27T12:05:00Z"
}

Response - Success (200 OK):

{
  "jobId": "550e8400-e29b-41d4-a716-446655440000",
  "status": "success",
  "createdAt": "2026-03-27T12:00:00Z",
  "completedAt": "2026-03-27T12:10:00Z",
  "result": {
    "is_multi_form": true,
    "num_pages": 45,
    "forms": [
      {
        "page_start": 1,
        "page_end": 15,
        "form_type": "Trial Balance",
        "confidence": 0.96
      },
      {
        "page_start": 16,
        "page_end": 28,
        "form_type": "General Ledger Detail",
        "confidence": 0.94
      },
      {
        "page_start": 29,
        "page_end": 35,
        "form_type": "Bank Reconciliations",
        "confidence": 0.92
      },
      {
        "page_start": 36,
        "page_end": 40,
        "form_type": "Accounts Payable",
        "confidence": 0.89
      },
      {
        "form_type": "Other Information Not Previously Requested by CLA",
        "confidence": 0.55,
        "page_ranges": [
          {
            "start": 41,
            "end": 45
          }
        ]
      }
    ],
    "classified_in_chunks": true,
    "num_chunks": 3,
    "primary_document_type": "Trial Balance",
    "document_category": "Financial Statements"
  }
}

Response - Failed (200 OK):

{
  "jobId": "550e8400-e29b-41d4-a716-446655440000",
  "status": "failed",
  "createdAt": "2026-03-27T12:00:00Z",
  "failedAt": "2026-03-27T12:02:00Z",
  "error": {
    "code": "PDF_PROCESSING_ERROR",
    "message": "Failed to convert page 15 to image",
    "details": "Corrupted PDF page detected"
  }
}

Error Responses:

404 Not Found - Job ID does not exist
401 Unauthorized - Missing or invalid authentication
500 Internal Server Error - Unexpected server error

Data Flow

Classification Job Flow

1. Client Uploads HQ Exchange PDF Document
   ↓
2. API validates request and creates job record
   ↓
3. PDF saved to Blob Storage (hqx-uploaded-pdfs container)
   ↓
4. Job message posted to Service Bus queue (hqx-classification-jobs)
   ↓
5. Background worker picks up message
   ↓
6. PDF Split into individual pages
   ↓
7. Each page converted to image (PNG/JPEG)
   ↓
8. Images batched into groups of 20
   ↓
9. Each batch sent to LLM with HQX-specific classification prompt
   ↓
10. LLM classifies pages into HQ Exchange document types
   ↓
11. Results merged and collated by document type
   ↓
12. Primary document type determined (most pages)
   ↓
13. JSON result saved to Blob Storage
   ↓
14. Job status updated to "success"
   ↓
15. Client polls GET /api/v1/Classifications/{jobId} endpoint to retrieve results

Async Processing Pattern

The service implements an async request-reply pattern:

Client Request: POST /api/v1/Classifications returns immediately with jobId
Background Processing: Worker processes job asynchronously using HQX-specific classification
Status Polling: Client polls GET /api/v1/Classifications/{jobId} until complete
Result Retrieval: Final classification results with HQ Exchange document types

HQX-Specific Classification Logic

The LLM is provided with:

Predefined taxonomy of 28 HQ Exchange document types
Financial document context for accurate classification
Category groupings to improve classification accuracy
Examples and patterns for each document type

Batching Strategy

Batch Size: 20 images per LLM request
Rationale: Balances LLM token limits with processing efficiency
Parallel Processing: Multiple batches can be processed concurrently
Error Handling: Failed batches are retried independently
HQX Optimization: Batch images from similar page ranges for better context

External Dependencies

Azure Blob Storage

Purpose: Store uploaded PDFs, intermediate images, and classification results

Containers:

hqx-uploaded-pdfs - Original HQ Exchange PDF files
hqx-page-images - Converted page images
hqx-classification-results - Final JSON outputs

Configuration:

Managed Identity authentication
Lifecycle policies for automatic cleanup (90 days for compliance)
Encryption at rest enabled
Immutable storage for audit trail

Azure Service Bus

Purpose: Queue job messages for asynchronous processing

Queue: hqx-classification-jobs

Message Format:

{
  "jobId": "550e8400-e29b-41d4-a716-446655440000",
  "pdfBlobPath": "hqx-uploaded-pdfs/550e8400.pdf",
  "metadata": {
    "clientId": "12345",
    "engagementId": "ENG-2026-001"
  },
  "createdAt": "2026-03-27T12:00:00Z"
}

Configuration:

Message TTL: 48 hours (extended for complex documents)
Max delivery count: 3
Dead letter queue enabled

SQL Database

Purpose: Store job metadata, status, and classification results

Tables:

HQXClassificationJobs - Job tracking and status
HQXJobProgress - Detailed progress information
HQXDocumentTypeMapping - Reference data for document types

Configuration:

Managed Identity authentication
Connection pooling enabled
Automatic retry policies
Audit logging enabled

LLM API (Azure OpenAI)

Purpose: Perform HQ Exchange-specific document classification

Model: GPT-4 Vision or GPT-4 Turbo with Vision

HQX-Specific System Prompt:

You are an expert in HQ Exchange financial document classification. You specialize in identifying and categorizing financial and accounting documents used in audit engagements.
  
Classify the provided document pages into one of these HQ Exchange document types:
  
Financial Statements:
- Trial Balance
- General Ledger  
- General Ledger Detail
  
Banking:
- Bank Statements
- Bank Reconciliations
- Cash
- Check Register
  
Accounts:
- Accounts Payable
- Accounts Receivable
- Confirmations
  
Asset Management:
- Fixed Assets
- Fixed Asset Additions
- Fixed Asset Disposals
- Prepaid Expenses
- Inventory
  
Liabilities:
- Debt Agreements
- Accrued Expenses
- Accrued Payroll
- Lease Agreements
- Leases
  
Investments:
- Investments
- Investment Statements
  
Operational:
- Legal Invoices
- Board Minutes
- Revenue
- Related Party Transactions
- Personnel and Policy Manuals
  
Other:
- Other Information Not Previously Requested by CLA
  
Return a JSON object with a 'forms' array where each entry has:
- page_start: starting page number
- page_end: ending page number
- form_type: one of the document types listed above (exact match)
- confidence: score between 0 and 1
  
For non-contiguous pages, use page_ranges array instead.

Configuration:

Managed Identity authentication
Circuit breaker pattern for fault tolerance
Retry policy with exponential backoff
Request timeout: 90 seconds (extended for complex financial documents)
Temperature: 0.1 (low for consistent classification)

Azure Application Insights

Purpose: Logging, monitoring, and distributed tracing

Telemetry:

Request/response logging
Custom metrics (jobs processed, success rate, document type distribution)
Performance counters
Exception tracking
Document type classification accuracy tracking

Technology Stack

Core Framework

Runtime: Node.js 20 LTS
Language: TypeScript 5.x
Web Framework: Fastify (via framework-nodejs-fastify)
Framework: framework-nodejs-fastify with full feature integration

Architecture Organization

The service follows feature-based organization as mandated by API_DEVELOPMENT_STANDARDS.md:

Feature Structure: Code organized by business feature (classifications) rather than technical layer
Version Management: Features own their versions (v1/, v2/), enabling independent version evolution
Modular Design: Classification feature is self-contained with its own routes, services, repositories, and models
Shared Components: Common utilities, configuration, and middleware in shared/ directory
HQX Specialization: HQX-specific services and configurations isolated within the feature

This structure enables:

Clean separation of concerns
Independent feature deployment
Clear API versioning boundaries
Easy integration with HQ Exchange ecosystem

Key Libraries

| Library | Purpose |

|---------|---------|

| framework-nodejs-fastify | Core web framework integration |

| framework-nodejs-appconfig | Configuration management |

| framework-nodejs-logging | Structured logging |

| framework-nodejs-storage | Azure Blob Storage client |

| framework-nodejs-sql | SQL Database connectivity |

| framework-nodejs-pubsub | Azure Service Bus integration |

| framework-nodejs-service-invocation | HTTP client for LLM API calls |

| pdf-lib or pdf2pic | PDF processing and page extraction |

| sharp | Image processing and optimization |

| @azure/identity | Azure authentication |

Development Tools

Testing: Vitest
Linting: ESLint + Prettier
API Spec: OpenAPI 3.0.3
API Testing: Postman collections

Non-Functional Requirements

Performance

| Metric | Target |

|--------|--------|

| API Response Time (POST /api/v1/Classifications) | < 500ms (job creation) |

| API Response Time (GET /api/v1/Classifications/{jobId}) | < 200ms |

| PDF Processing Throughput | 15-25 pages/minute per worker (optimized for financial docs) |

| Concurrent Job Processing | 15+ jobs simultaneously |

| Maximum PDF Size | 50 MB |

| Maximum Page Count | 500 pages per PDF |

| Classification Accuracy | > 95% for standard HQX document types |

Scalability

Horizontal Scaling: Multiple worker instances via Kubernetes
Queue-Based: Service Bus enables distributed processing
Stateless Workers: Workers can be scaled independently
Database Connection Pooling: Efficient resource utilization
Auto-scaling: Based on queue depth and processing time

Reliability

Availability Target: 99.9% (8.76 hours downtime per year max)
Error Recovery: Automatic retry for transient failures
Circuit Breaker: Prevents cascade failures to LLM API
Dead Letter Queue: Failed messages captured for analysis
Job Timeout: Maximum processing time of 3 hours (for large financial documents)
Data Durability: 99.999999999% (11 9’s) via Azure Storage

Security

Authentication: OAuth 2.0 / Azure AD
Authorization: RBAC with role-based endpoint access
CSRF Protection: Enabled for state-changing operations
Data Encryption:

- In transit: TLS 1.3

- At rest: Azure Storage encryption with customer-managed keys

Secrets Management: Azure Key Vault for API keys
Input Validation: File type, size, and content validation
Audit Logging: All operations logged for compliance

Security Considerations

Authentication & Authorization

All endpoints require valid OAuth 2.0 Bearer token
Role-based access control:

- hqx.classifier.write - POST /api/v1/Classifications permission

- hqx.classifier.read - GET /api/v1/Classifications permission

Data Protection

PII Handling: Financial documents may contain sensitive client data
Data Retention: 90-day retention for audit compliance
Access Logging: All data access logged to Application Insights with user context
Encryption: All data encrypted at rest and in transit
Data Residency: Configurable regional storage for compliance

API Security

Rate Limiting: 50 requests per minute per user (financial documents are typically larger)
CORS: Restricted to HQ Exchange approved origins
CSRF Tokens: Required for POST requests
File Upload Validation:

- File type whitelist (PDF only)

- Magic number validation

- Maximum file size enforcement (50MB)

- PDF structure validation

- Malware scanning integration

Network Security

Private Endpoints: Azure services connected via private endpoints
NSG Rules: Network security groups restrict access
TLS Enforcement: HTTPS only, TLS 1.3 minimum
API Gateway: All traffic routed through Azure API Management

Compliance

SOC 2 Type II: Compliance maintained
GDPR: Data protection compliance
HIPAA: If applicable for healthcare-related financial documents
Audit Trail: Complete audit trail for all classification activities

Deployment Architecture

Kubernetes Deployment

Architecture:
  - AKS Cluster (3+ nodes, dedicated node pool for workers)
  - Ingress Controller (NGINX with WAF)
  - Horizontal Pod Autoscaler (based on queue depth)
  - Persistent Volume Claims for temporary storage
  - Pod Priority Classes for critical workloads

Environment Configuration

|-------------|---------|----------|-----------------|

| Development | Local/dev testing | 1 | 1 |

| Staging | Pre-production validation | 2 | 2 |

Resource Requirements

API Pods:

CPU: 250m (request) / 1000m (limit)
Memory: 512Mi (request) / 1Gi (limit)

Worker Pods:

CPU: 1000m (request) / 3000m (limit)
Memory: 2Gi (request) / 4Gi (limit)

Health Checks

Liveness Probe: GET /health/live (10s interval)
Readiness Probe: GET /health/ready (5s interval)
Startup Probe: Initial 45s delay (account for LLM model loading)

Monitoring & Observability

Key Metrics

Application Metrics:

Total HQX classification jobs submitted
Jobs completed successfully by document type
Jobs failed
Average processing time per document type
LLM API call success rate
Queue depth (Service Bus)
Document type distribution

Performance Metrics:

API endpoint response times (P50, P95, P99)
Background worker processing rate
Database query performance
Blob storage operation latency
LLM API response times

Business Metrics:

Documents classified per day by type
Most common document types
Average document size by type
Peak processing hours
Client usage by engagementId
Classification accuracy by document type

HQX-Specific Metrics:

Trial Balance processing time
General Ledger processing volume
Bank Statement classification accuracy
Multi-document file frequency

Alerts

Job failure rate > 5% (stricter for financial documents)
Average processing time > 45 minutes
Queue backlog > 50 messages
API error rate > 3%
Database connection failures
LLM API circuit breaker open
Classification confidence < 0.7 (low confidence indicator)
Unusual document type distribution (potential data quality issue)

Logging Strategy

Structured JSON logs to Application Insights
Log Levels:

- INFO: Job lifecycle events, document type classifications

- WARN: Low confidence classifications, retries

- ERROR: Processing failures and exceptions

Correlation IDs: Track requests across services
PII Redaction: Client names and sensitive financial data not logged
Audit Trail: Complete classification history for compliance

Distributed Tracing

OpenTelemetry integration
Trace job flow from API request through completion
Track LLM API calls and performance by document type
Identify bottlenecks specific to HQX document processing
End-to-end latency tracking

Dashboards

Operational Dashboard:

Real-time job status
Queue depth and processing rate
Error rates and types
System health indicators

Business Dashboard:

Document type distribution
Processing volume trends
Client usage patterns
Classification accuracy metrics

Appendix

Glossary

| Term | Definition |

|------|------------|

| HQ Exchange | Platform for financial document exchange and audit workflow management |

| Classification Job | An asynchronous task to process and classify an HQX financial document |

| Document Type | One of 28 predefined HQ Exchange financial/accounting document categories |

| Batch | Group of 20 page images sent to LLM for processing |

| Primary Document Type | The document type that appears on the most pages in a multi-document file |

| Job Status | Current state of classification job (in-progress, success, failed) |

HQ Exchange Document Type Reference

Complete List of Supported Document Types:

Financial Statements

- Trial Balance

- General Ledger

- General Ledger Detail

Banking

- Bank Statements

- Bank Reconciliations

- Cash

- Check Register

Accounts

- Accounts Payable

- Accounts Receivable

- Confirmations

Asset Management

- Fixed Assets

- Fixed Asset Additions

- Fixed Asset Disposals

- Prepaid Expenses

- Inventory

Liabilities

- Debt Agreements

- Accrued Expenses

- Accrued Payroll

- Lease Agreements

- Leases

Investments

- Investments

- Investment Statements

Operational

- Legal Invoices

- Board Minutes

- Revenue

- Related Party Transactions

- Personnel and Policy Manuals

Other

- Other Information Not Previously Requested by CLA

Future Enhancements

Real-time Notifications: WebSocket support for live status updates
Webhook Support: Callback URLs for job completion with document type
Multi-format Support: Excel spreadsheets, Word documents (common in HQX)
OCR Integration: Enhanced text extraction for scanned financial documents
Confidence Threshold Configuration: Allow clients to set minimum confidence levels
Bulk Classification: Upload multiple documents in single request
Priority Queues: Expedited processing for time-sensitive audit deadlines
Result Caching: Cache classification results for identical documents
Custom Document Types: Allow clients to define custom HQX document categories
AI Model Fine-tuning: Train on historical HQX classification data for improved accuracy

Document Owner: HQ Exchange Architecture Team

Reviewers: Security Team, Engineering Team, Product Team, Compliance Team

Next Review Date: June 27, 2026

Table of Contents

Overview

Purpose

Business Context

Key Features

Supported Document Types

System Architecture

High-Level Architecture Diagram

Component Overview

API Endpoints

1. POST /api/v1/Classifications

2. GET /api/v1/Classifications/{jobId}

Data Flow

Classification Job Flow

Async Processing Pattern

HQX-Specific Classification Logic

Batching Strategy

External Dependencies

Azure Blob Storage

Azure Service Bus

SQL Database

LLM API (Azure OpenAI)

Azure Application Insights

Technology Stack

Core Framework

Architecture Organization

Key Libraries

Development Tools

Non-Functional Requirements

Performance

Scalability

Reliability

Security

Security Considerations

Authentication & Authorization

Data Protection

API Security

Network Security

Compliance

Deployment Architecture

Kubernetes Deployment

Environment Configuration

Resource Requirements

Health Checks

Monitoring & Observability

Key Metrics

Alerts

Logging Strategy

Distributed Tracing

Dashboards

Appendix

Glossary

HQ Exchange Document Type Reference

Future Enhancements