Skip to navigation
Account Payable to Fluxx HLD

Documents AI Intake and Extract Service for Fluxx

Overview

This document outlines the migration strategy from Azure Durable Functions orchestrators to standalone service classes. The goal is to transform workflow-based orchestration into sequential service operations that focus solely on SQL database updates for Document and Invoice entities.

Migration Strategy

Phase 1: Document AI Invoice File Intake Service

  • Source: invoiceIntakeOrchestrator + related activities

  • Target: DocAiInvoiceFileIntakeService class

  • Called By: fluxx-file-intake microservice

  • Service Name: docai_invoice_file_intake_service

Phase 2: Document AI Invoice Extraction Service

  • Source: invoiceIngestOrchestrator + related activities  

  • Target: DocAiInvoiceExtractionService class

  • Called By: fluxx-extraction-service microservice

  • Service Name: docai_invoice_extraction_service

Service Design Details

1. Document AI Invoice File Intake Service

Purpose: Handle document intake, validation, and duplicate detection for the Fluxx file intake microservice.

Called By: fluxx-file-intake microservice  

Service Name: docai_invoice_file_intake_service

Class Structure

class DocAiInvoiceFileIntakeService {
    // Core Methods
    public async processFileIntake(input: FileIntakeInput): Promise<FileIntakeResult>
    // Private Methods
    private async calculateClientIdHash(accountId: string): Promise<string>
    private async initializeDocument(pdfPath: string, clientId: string, workflowId: string): Promise<[string, string]>
    private async checkDuplicateDocument(data: ProcessData): Promise<ProcessData>
    private async validateInvoice(data: ProcessData): Promise<ProcessData>
    private async handleSalesTax(data: ProcessData): Promise<ProcessData>
    private async updateDocumentRecord(documentId: string, updates: DocumentUpdates): Promise<void>
}

Functions to Migrate

✅ Already Implemented Functions:

| Function | Source File | Line | Status |

|----------|------------|------|--------|

| calculateClientIdHashFunction | invoiceIngestActivities.ts | 115 | ✅ Complete |

| initializeDocument | invoiceIngestActivities.ts | 135 | ✅ Complete |

| checkDuplicateDocument | invoiceIntakeActivities.ts | 130 | ✅ Complete |

| validateInvoice | invoiceIntakeActivities.ts | 228 | ✅ Complete |

| handleSalesTax | invoiceIntakeActivities.ts | Referenced | ✅ Complete |

✅ Helper Functions (Already Implemented):

| Function | Source File | Purpose |

|----------|------------|---------|

| calculateHash | invoiceIngestActivities.ts | Hash calculation |

| flattenDict | invoiceIntakeActivities.ts | Object flattening |

| getRandomId | invoiceIntakeActivities.ts | ID generation |

| updateItem | Various activities | Database updates |

❌ Functions Needing Implementation:

| Function | Purpose | Priority |

|----------|---------|----------|

| SQL Document table operations | Direct SQL updates | High |

| Error handling without workflow context | Simplified error management | High |

| Service class wrapper | Orchestrate sequential calls | High |

| Configuration database client | Client config retrieval | Medium |

Sequential Process Flow

Input: accountId, pdfPath, workflowId Calculate Client ID Hash Initialize Document Record Check for Duplicates Handle Sales Tax Processing Validate Invoice Data Update Document SQL Table Return Success/Failure Result

Input/Output Interfaces

interface FileIntakeInput {
    accountId: string;
    workflowId: string;
    pdfPath: string;
    pdfPrefix: string;
    clientConfig?: any;
}
  
interface FileIntakeResult {
    success: boolean;
    documentId: string;
    invoiceId: string;
    clientId: string;
    error?: string;
}

2. Document AI Invoice Extraction Service

Purpose: Handle PDF processing, data extraction, and entity mapping for the Fluxx extraction microservice.

Called By: fluxx-extraction-service microservice  

Service Name: docai_invoice_extraction_service

Class Structure

class DocAiInvoiceExtractionService {
    // Core Methods
    public async processInvoiceExtraction(input: ExtractionInput): Promise<ExtractionResult>
    // Private Methods
    private async generateInvoiceId(data: ProcessData): Promise<ProcessData>
    private async generateImages(data: ProcessData): Promise<ProcessData>
    private async getBoundingRegions(data: ProcessData): Promise<ProcessData>
    private async extractInvoice(data: ProcessData): Promise<ProcessData>
    private async mapInvoice(data: ProcessData): Promise<ProcessData>
    private async transformInvoice(data: ProcessData): Promise<ProcessData>
    private async postProcessing(data: ProcessData): Promise<ProcessData>
    private async updateDocumentRecord(documentId: string, updates: DocumentUpdates): Promise<void>
    private async updateInvoiceRecord(invoiceId: string, extractedData: InvoiceData): Promise<void>
}

Functions to Migrate

✅ Already Implemented Functions:

| Function | Source File | Line | Status |

|----------|------------|------|--------|

| generateInvoiceId | invoiceIngestActivities.ts | 234 | ✅ Complete |

| generateImages | invoiceIngestActivities.ts | Referenced | ✅ Complete |

| getBoundingRegions | invoiceIngestActivities.ts | Referenced | ✅ Complete |

| extractInvoice | invoiceIngestActivities.ts | Referenced | ✅ Complete |

| mapInvoice | invoiceIngestActivities.ts | Referenced | ✅ Complete |

| transformInvoice | invoiceIngestActivities.ts | Referenced | ✅ Complete |

| postProcessing | invoiceIngestActivities.ts | Referenced | ✅ Complete |

✅ Helper Functions (Already Implemented):

| Function | Source File | Purpose |

|----------|------------|---------|

| getBlobContent | invoiceIngestActivities.ts | Azure Blob operations |

| calculateHash | invoiceIngestActivities.ts | Content hashing |

| storeDocument | invoiceIngestActivities.ts | Document storage |

| getClient | invoiceIngestActivities.ts | Client config retrieval |

❌ Functions Needing Implementation:

| Function | Purpose | Priority |

|----------|---------|----------|

| SQL Document table operations | Direct SQL updates | High |

| SQL Invoice (extracted_entities) operations | Invoice data persistence | High |

| Azure Document Intelligence client | OCR and form recognition | High |

| OpenAI API client | AI-based extraction | High |

| Service class wrapper | Orchestrate sequential calls | High |

| Image processing utilities | PDF to image conversion | Medium |

Sequential Process Flow

Input: ProcessData with PDF Generate Invoice ID from Content Hash Convert PDF to Images Extract Bounding Regions viaDocument Intelligence Perform AI-based Invoice Extraction Map AI Results to Invoice Structure Transform and Combine Extractions Post-process Freight/Tariff Data Update Document SQL Table Update Invoice SQL Table Return Extraction Results

Input/Output Interfaces

interface ExtractionInput {
    documentId: string;
    invoiceId: string;
    clientId: string;
    pdfPath: string;
    pdfPrefix: string;
    clientConfig: any;
    imageUrls?: string[];
    imagePrefixes?: string[];
}
  
interface ExtractionResult {
    success: boolean;
    invoiceId: string;
    extractedData: InvoiceData;
    documentUpdates: DocumentUpdates;
    error?: string;
}

Key Architectural Changes

1. Removal of Workflow Management

  • ❌ Remove: Stepper updates, stage transitions, external event waiting

  • ❌ Remove: Workflow status tracking, substatus management

  • ❌ Remove: Orchestration context and Durable Functions dependencies

  • ✅ Keep: Core business logic and data processing

2. Database Strategy Changes

  • From: Cosmos DB document operations with workflow tracking

  • To: Direct SQL Server operations on Document and Invoice tables

  • Focus: Only update data entities, not workflow state

3. Error Handling Simplification

  • From: Complex workflow failure handling with retry and human intervention

  • To: Simple success/failure responses with error details

  • Benefit: Easier testing and debugging

4. Execution Model

  • From: Distributed orchestration with activity functions

  • To: Sequential method calls within service classes

  • Benefit: Reduced complexity and improved performance

Implementation Status - COMPLETED ✅

Phase 1: File Intake Service ✅

  1. ✅ Analyzed existing functions - Completed

  2. ✅ Created service class structure - /src/file-intake-service/

  3. ✅ Implemented SQL database operations - Mock implementation with proper schema

  4. ✅ Migrated and adapted existing activity functions - Workflow logic removed

  5. ❌ Create unit tests - TODO

  6. ❌ Integration testing - TODO

Actual SQL Schema Implemented:

[Documents] (
    [DocumentId] BIGINT IDENTITY(1,1) PRIMARY KEY,
    [BlobPath] VARCHAR(800) NOT NULL,
    [ClientId] BIGINT NOT NULL,
    [DateUploaded] DATETIMEOFFSET DEFAULT SYSDATETIMEOFFSET(),
    [DocumentHash] VARCHAR(48) UNIQUE NOT NULL,
    [Name] VARCHAR(200) NOT NULL,
    FOREIGN KEY ([ClientId]) REFERENCES [dbo].[Clients]([ClientId])
);

Phase 2: Invoice Extraction Service ✅

  1. ✅ Analyzed existing functions - Completed

  2. ✅ Created service class structure - /src/extraction-service/

  3. ✅ Implemented SQL database operations - Mock implementation with proper schema

  4. ✅ Migrated and adapted existing activity functions - Workflow logic removed

  5. ❌ Create unit tests - TODO

  6. ❌ Integration testing - TODO

Actual SQL Schema Implemented:

[ExtractedEntities] (
    [ExtractedEntityId] BIGINT IDENTITY(1,1) PRIMARY KEY,
    [CreatedOn] DATETIMEOFFSET DEFAULT SYSDATETIMEOFFSET(),
    [DateExported] DATETIMEOFFSET,
    [DateFiled] DATETIMEOFFSET,
    [DocumentId] BIGINT NOT NULL,
    [ExtractedEntitiesPayload] NVARCHAR(MAX) NOT NULL,
    [ModifiedOn] DATETIMEOFFSET DEFAULT SYSDATETIMEOFFSET(),
    [Stage] VARCHAR(50) NOT NULL,
    [Status] VARCHAR(50) NOT NULL,
    [Substatus] VARCHAR(50),
    FOREIGN KEY ([DocumentId]) REFERENCES [dbo].[Documents]([DocumentId])
);

Phase 3: Integration and Testing 🚧

  1. ✅ Created service integration layer - Example usage provided

  2. ✅ Client Configuration Integration - Both services updated to support actual client configuration format

  3. ❌ Performance testing - TODO

  4. ❌ Validation testing - TODO

  5. ✅ Documentation updates - This design document updated

Client Configuration Integration - COMPLETED ✅

Both services have been updated to accept and use the actual client configuration format:

Key Configuration Elements Integrated:

  • OpenAI Configuration: Prompts, models, and roles from ingest_config.openai_config

  • Freight Patterns: Post-processing patterns from ingest_config.freight_patterns

  • Unique Identifier Fields: Validation fields from ingest_config.unique_identifier_fields

  • Stage Configurations: Workflow stage settings and ordinal positions

Service Updates:

  • File Intake Service: Uses client config for validation rules and unique identifiers

  • Extraction Service: Uses OpenAI config for AI extraction and freight patterns for post-processing

  • Interface Updates: ClientConfig interface matches actual production format with comprehensive typing

Benefits of Configuration Integration:

  • Client-specific AI prompts and models

  • Customizable freight processing patterns

  • Flexible validation rules per client

  • Stage-based workflow management

Microservice Architecture - IMPLEMENTED ✅

Document AI Invoice File Intake Service (/src/docai-invoice-file-intake-service/)

  • Responsibility: Document intake and validation for Fluxx file intake microservice

  • Database Access: Documents table (read/write)

  • Called By: fluxx-file-intake microservice

  • Service Name: docai_invoice_file_intake_service

  • Input: Account ID, PDF path, workflow ID

  • Output: Document ID, Client ID, Document Hash

  • Status: ✅ Implemented with mock database

Document AI Invoice Extraction Service (/src/docai-invoice-extraction-service/)

  • Responsibility: PDF processing and data extraction for Fluxx extraction microservice

  • Database Access: Documents table (read-only), ExtractedEntities table (read/write)

  • Called By: fluxx-extraction-service microservice  

  • Service Name: docai_invoice_extraction_service

  • Input: Document ID from Document AI Invoice File Intake Service

  • Output: Extracted invoice data in ExtractedEntities table

  • Status: ✅ Implemented with mock database

Dependencies and Prerequisites

Database Requirements

  • ❌ SQL Server connection string - For Document table operations

  • ❌ SQL Server connection string - For Invoice (extracted_entities) table

  • ❌ Database schema definitions - Table structures and relationships

Azure Service Requirements

  • ❌ Azure Blob Storage client configuration - For PDF and image operations

  • ❌ Azure Document Intelligence API keys - For OCR and form recognition

  • ❌ OpenAI API configuration - For AI-based extraction

  • ❌ Client configuration database access - For client-specific settings

Development Tools

  • ✅ TypeScript/Node.js environment - Already configured

  • ❌ SQL Server driver/ORM - For database operations

  • ❌ Testing framework setup - Unit and integration tests

  • ❌ Logging and monitoring - For service observability

Migration Benefits

Performance Improvements

  • Reduced Latency: No orchestration overhead

  • Better Resource Utilization: Sequential execution vs. distributed coordination

  • Simplified Scaling: Stateless service classes vs. stateful orchestrators

Maintenance Benefits

  • Easier Debugging: Linear execution flow

  • Simpler Testing: Unit test individual methods vs. orchestration flows

  • Reduced Dependencies: No Durable Functions runtime requirements

Operational Benefits

  • Direct SQL Operations: Better data consistency and transaction control

  • Simplified Deployment: Standard service deployment vs. function app deployment

  • Better Monitoring: Standard application monitoring vs. orchestration tracking

Risk Mitigation

Data Integrity Risks

  • Mitigation: Implement proper SQL transactions

  • Validation: Comprehensive testing of database operations

Performance Risks

  • Mitigation: Performance testing and benchmarking

  • Monitoring: Implement proper service metrics

Integration Risks

  • Mitigation: Gradual migration with parallel testing

  • Rollback: Maintain existing orchestrators during transition

Next Steps

  1. Create InvoiceFileIntakeService class - Start with Phase 1

  2. Implement SQL database operations - Define connection and schema

  3. Migrate activity functions - Remove workflow dependencies

  4. Create unit tests - Validate sequential execution

  5. Integration testing - End-to-end validation

  6. Repeat for InvoiceExtractionService - Phase 2 implementation