> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://cla-demo.postmansolutions.com/design/account-payable-to-fluxx-hld/documents-ai-intake-and-extract-service-for-fluxx/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://cla-demo.postmansolutions.com/_mcp/server. # Documents AI Intake and Extract Service for Fluxx ## Overview This document outlines the migration strategy from Azure Durable Functions orchestrators to standalone service classes. The goal is to transform workflow-based orchestration into sequential service operations that focus solely on SQL database updates for Document and Invoice entities. ## Migration Strategy ### Phase 1: Document AI Invoice File Intake Service * **Source**: `invoiceIntakeOrchestrator` + related activities * **Target**: `DocAiInvoiceFileIntakeService` class * **Called By**: `fluxx-file-intake` microservice * **Service Name**: `docai_invoice_file_intake_service` ### Phase 2: Document AI Invoice Extraction Service * **Source**: `invoiceIngestOrchestrator` + related activities * **Target**: `DocAiInvoiceExtractionService` class * **Called By**: `fluxx-extraction-service` microservice * **Service Name**: `docai_invoice_extraction_service` --- ## Service Design Details ### 1. Document AI Invoice File Intake Service **Purpose**: Handle document intake, validation, and duplicate detection for the Fluxx file intake microservice. **Called By**: `fluxx-file-intake` microservice **Service Name**: `docai_invoice_file_intake_service` #### **Class Structure** ```typescript class DocAiInvoiceFileIntakeService {     // Core Methods     public async processFileIntake(input: FileIntakeInput): Promise     // Private Methods     private async calculateClientIdHash(accountId: string): Promise     private async initializeDocument(pdfPath: string, clientId: string, workflowId: string): Promise<[string, string]>     private async checkDuplicateDocument(data: ProcessData): Promise     private async validateInvoice(data: ProcessData): Promise     private async handleSalesTax(data: ProcessData): Promise     private async updateDocumentRecord(documentId: string, updates: DocumentUpdates): Promise } ``` #### **Functions to Migrate** **✅ Already Implemented Functions:** \| Function | Source File | Line | Status | \|----------|------------|------|--------| \| `calculateClientIdHashFunction` | `invoiceIngestActivities.ts` | 115 | ✅ Complete | \| `initializeDocument` | `invoiceIngestActivities.ts` | 135 | ✅ Complete | \| `checkDuplicateDocument` | `invoiceIntakeActivities.ts` | 130 | ✅ Complete | \| `validateInvoice` | `invoiceIntakeActivities.ts` | 228 | ✅ Complete | \| `handleSalesTax` | `invoiceIntakeActivities.ts` | Referenced | ✅ Complete | **✅ Helper Functions (Already Implemented):** \| Function | Source File | Purpose | \|----------|------------|---------| \| `calculateHash` | `invoiceIngestActivities.ts` | Hash calculation | \| `flattenDict` | `invoiceIntakeActivities.ts` | Object flattening | \| `getRandomId` | `invoiceIntakeActivities.ts` | ID generation | \| `updateItem` | Various activities | Database updates | **❌ Functions Needing Implementation:** \| Function | Purpose | Priority | \|----------|---------|----------| \| SQL Document table operations | Direct SQL updates | High | \| Error handling without workflow context | Simplified error management | High | \| Service class wrapper | Orchestrate sequential calls | High | \| Configuration database client | Client config retrieval | Medium | #### **Sequential Process Flow** ```mermaid graph TD     A[Input: accountId, pdfPath, workflowId] --> B[Calculate Client ID Hash]     B --> C[Initialize Document Record]     C --> D[Check for Duplicates]     D --> E[Handle Sales Tax Processing]     E --> F[Validate Invoice Data]     F --> G[Update Document SQL Table]     G --> H[Return Success/Failure Result] ``` #### **Input/Output Interfaces** ```typescript interface FileIntakeInput {     accountId: string;     workflowId: string;     pdfPath: string;     pdfPrefix: string;     clientConfig?: any; } interface FileIntakeResult {     success: boolean;     documentId: string;     invoiceId: string;     clientId: string;     error?: string; } ``` --- ### 2. Document AI Invoice Extraction Service **Purpose**: Handle PDF processing, data extraction, and entity mapping for the Fluxx extraction microservice. **Called By**: `fluxx-extraction-service` microservice **Service Name**: `docai_invoice_extraction_service` #### **Class Structure** ```typescript class DocAiInvoiceExtractionService {     // Core Methods     public async processInvoiceExtraction(input: ExtractionInput): Promise     // Private Methods     private async generateInvoiceId(data: ProcessData): Promise     private async generateImages(data: ProcessData): Promise     private async getBoundingRegions(data: ProcessData): Promise     private async extractInvoice(data: ProcessData): Promise     private async mapInvoice(data: ProcessData): Promise     private async transformInvoice(data: ProcessData): Promise     private async postProcessing(data: ProcessData): Promise     private async updateDocumentRecord(documentId: string, updates: DocumentUpdates): Promise     private async updateInvoiceRecord(invoiceId: string, extractedData: InvoiceData): Promise } ``` #### **Functions to Migrate** **✅ Already Implemented Functions:** \| Function | Source File | Line | Status | \|----------|------------|------|--------| \| `generateInvoiceId` | `invoiceIngestActivities.ts` | 234 | ✅ Complete | \| `generateImages` | `invoiceIngestActivities.ts` | Referenced | ✅ Complete | \| `getBoundingRegions` | `invoiceIngestActivities.ts` | Referenced | ✅ Complete | \| `extractInvoice` | `invoiceIngestActivities.ts` | Referenced | ✅ Complete | \| `mapInvoice` | `invoiceIngestActivities.ts` | Referenced | ✅ Complete | \| `transformInvoice` | `invoiceIngestActivities.ts` | Referenced | ✅ Complete | \| `postProcessing` | `invoiceIngestActivities.ts` | Referenced | ✅ Complete | **✅ Helper Functions (Already Implemented):** \| Function | Source File | Purpose | \|----------|------------|---------| \| `getBlobContent` | `invoiceIngestActivities.ts` | Azure Blob operations | \| `calculateHash` | `invoiceIngestActivities.ts` | Content hashing | \| `storeDocument` | `invoiceIngestActivities.ts` | Document storage | \| `getClient` | `invoiceIngestActivities.ts` | Client config retrieval | **❌ Functions Needing Implementation:** \| Function | Purpose | Priority | \|----------|---------|----------| \| SQL Document table operations | Direct SQL updates | High | \| SQL Invoice (extracted\_entities) operations | Invoice data persistence | High | \| Azure Document Intelligence client | OCR and form recognition | High | \| OpenAI API client | AI-based extraction | High | \| Service class wrapper | Orchestrate sequential calls | High | \| Image processing utilities | PDF to image conversion | Medium | #### **Sequential Process Flow** ```mermaid graph TD     A[Input: ProcessData with PDF] --> B[Generate Invoice ID from Content Hash]     B --> C[Convert PDF to Images]     C --> D[Extract Bounding Regions via Document Intelligence]     D --> E[Perform AI-based Invoice Extraction]     E --> F[Map AI Results to Invoice Structure]     F --> G[Transform and Combine Extractions]     G --> H[Post-process Freight/Tariff Data]     H --> I[Update Document SQL Table]     I --> J[Update Invoice SQL Table]     J --> K[Return Extraction Results] ``` #### **Input/Output Interfaces** ```typescript interface ExtractionInput {     documentId: string;     invoiceId: string;     clientId: string;     pdfPath: string;     pdfPrefix: string;     clientConfig: any;     imageUrls?: string[];     imagePrefixes?: string[]; } interface ExtractionResult {     success: boolean;     invoiceId: string;     extractedData: InvoiceData;     documentUpdates: DocumentUpdates;     error?: string; } ``` --- ## Key Architectural Changes ### **1. Removal of Workflow Management** * ❌ **Remove**: Stepper updates, stage transitions, external event waiting * ❌ **Remove**: Workflow status tracking, substatus management * ❌ **Remove**: Orchestration context and Durable Functions dependencies * ✅ **Keep**: Core business logic and data processing ### **2. Database Strategy Changes** * **From**: Cosmos DB document operations with workflow tracking * **To**: Direct SQL Server operations on Document and Invoice tables * **Focus**: Only update data entities, not workflow state ### **3. Error Handling Simplification** * **From**: Complex workflow failure handling with retry and human intervention * **To**: Simple success/failure responses with error details * **Benefit**: Easier testing and debugging ### **4. Execution Model** * **From**: Distributed orchestration with activity functions * **To**: Sequential method calls within service classes * **Benefit**: Reduced complexity and improved performance --- ## Implementation Status - COMPLETED ✅ ### **Phase 1: File Intake Service** ✅ 1. ✅ **Analyzed existing functions** - Completed 2. ✅ **Created service class structure** - `/src/file-intake-service/` 3. ✅ **Implemented SQL database operations** - Mock implementation with proper schema 4. ✅ **Migrated and adapted existing activity functions** - Workflow logic removed 5. ❌ **Create unit tests** - TODO 6. ❌ **Integration testing** - TODO **Actual SQL Schema Implemented:** ```sql [Documents] (     [DocumentId] BIGINT IDENTITY(1,1) PRIMARY KEY,     [BlobPath] VARCHAR(800) NOT NULL,     [ClientId] BIGINT NOT NULL,     [DateUploaded] DATETIMEOFFSET DEFAULT SYSDATETIMEOFFSET(),     [DocumentHash] VARCHAR(48) UNIQUE NOT NULL,     [Name] VARCHAR(200) NOT NULL,     FOREIGN KEY ([ClientId]) REFERENCES [dbo].[Clients]([ClientId]) ); ``` ### **Phase 2: Invoice Extraction Service** ✅ 1. ✅ **Analyzed existing functions** - Completed 2. ✅ **Created service class structure** - `/src/extraction-service/` 3. ✅ **Implemented SQL database operations** - Mock implementation with proper schema 4. ✅ **Migrated and adapted existing activity functions** - Workflow logic removed 5. ❌ **Create unit tests** - TODO 6. ❌ **Integration testing** - TODO **Actual SQL Schema Implemented:** ```sql [ExtractedEntities] (     [ExtractedEntityId] BIGINT IDENTITY(1,1) PRIMARY KEY,     [CreatedOn] DATETIMEOFFSET DEFAULT SYSDATETIMEOFFSET(),     [DateExported] DATETIMEOFFSET,     [DateFiled] DATETIMEOFFSET,     [DocumentId] BIGINT NOT NULL,     [ExtractedEntitiesPayload] NVARCHAR(MAX) NOT NULL,     [ModifiedOn] DATETIMEOFFSET DEFAULT SYSDATETIMEOFFSET(),     [Stage] VARCHAR(50) NOT NULL,     [Status] VARCHAR(50) NOT NULL,     [Substatus] VARCHAR(50),     FOREIGN KEY ([DocumentId]) REFERENCES [dbo].[Documents]([DocumentId]) ); ``` ### **Phase 3: Integration and Testing** 🚧 1. ✅ **Created service integration layer** - Example usage provided 2. ✅ **Client Configuration Integration** - Both services updated to support actual client configuration format 3. ❌ **Performance testing** - TODO 4. ❌ **Validation testing** - TODO 5. ✅ **Documentation updates** - This design document updated ### **Client Configuration Integration - COMPLETED ✅** Both services have been updated to accept and use the actual client configuration format: **Key Configuration Elements Integrated:** * **OpenAI Configuration**: Prompts, models, and roles from `ingest_config.openai_config` * **Freight Patterns**: Post-processing patterns from `ingest_config.freight_patterns` * **Unique Identifier Fields**: Validation fields from `ingest_config.unique_identifier_fields` * **Stage Configurations**: Workflow stage settings and ordinal positions **Service Updates:** * File Intake Service: Uses client config for validation rules and unique identifiers * Extraction Service: Uses OpenAI config for AI extraction and freight patterns for post-processing * Interface Updates: ClientConfig interface matches actual production format with comprehensive typing **Benefits of Configuration Integration:** * Client-specific AI prompts and models * Customizable freight processing patterns * Flexible validation rules per client * Stage-based workflow management ## Microservice Architecture - IMPLEMENTED ✅ ### **Document AI Invoice File Intake Service** (`/src/docai-invoice-file-intake-service/`) * **Responsibility**: Document intake and validation for Fluxx file intake microservice * **Database Access**: Documents table (read/write) * **Called By**: `fluxx-file-intake` microservice * **Service Name**: `docai_invoice_file_intake_service` * **Input**: Account ID, PDF path, workflow ID * **Output**: Document ID, Client ID, Document Hash * **Status**: ✅ Implemented with mock database ### **Document AI Invoice Extraction Service** (`/src/docai-invoice-extraction-service/`) * **Responsibility**: PDF processing and data extraction for Fluxx extraction microservice * **Database Access**: Documents table (read-only), ExtractedEntities table (read/write) * **Called By**: `fluxx-extraction-service` microservice * **Service Name**: `docai_invoice_extraction_service` * **Input**: Document ID from Document AI Invoice File Intake Service * **Output**: Extracted invoice data in ExtractedEntities table * **Status**: ✅ Implemented with mock database --- ## Dependencies and Prerequisites ### **Database Requirements** * ❌ **SQL Server connection string** - For Document table operations * ❌ **SQL Server connection string** - For Invoice (extracted\_entities) table * ❌ **Database schema definitions** - Table structures and relationships ### **Azure Service Requirements** * ❌ **Azure Blob Storage client configuration** - For PDF and image operations * ❌ **Azure Document Intelligence API keys** - For OCR and form recognition * ❌ **OpenAI API configuration** - For AI-based extraction * ❌ **Client configuration database access** - For client-specific settings ### **Development Tools** * ✅ **TypeScript/Node.js environment** - Already configured * ❌ **SQL Server driver/ORM** - For database operations * ❌ **Testing framework setup** - Unit and integration tests * ❌ **Logging and monitoring** - For service observability --- ## Migration Benefits ### **Performance Improvements** * **Reduced Latency**: No orchestration overhead * **Better Resource Utilization**: Sequential execution vs. distributed coordination * **Simplified Scaling**: Stateless service classes vs. stateful orchestrators ### **Maintenance Benefits** * **Easier Debugging**: Linear execution flow * **Simpler Testing**: Unit test individual methods vs. orchestration flows * **Reduced Dependencies**: No Durable Functions runtime requirements ### **Operational Benefits** * **Direct SQL Operations**: Better data consistency and transaction control * **Simplified Deployment**: Standard service deployment vs. function app deployment * **Better Monitoring**: Standard application monitoring vs. orchestration tracking --- ## Risk Mitigation ### **Data Integrity Risks** * **Mitigation**: Implement proper SQL transactions * **Validation**: Comprehensive testing of database operations ### **Performance Risks** * **Mitigation**: Performance testing and benchmarking * **Monitoring**: Implement proper service metrics ### **Integration Risks** * **Mitigation**: Gradual migration with parallel testing * **Rollback**: Maintain existing orchestrators during transition --- ## Next Steps 1. **Create `InvoiceFileIntakeService` class** - Start with Phase 1 2. **Implement SQL database operations** - Define connection and schema 3. **Migrate activity functions** - Remove workflow dependencies 4. **Create unit tests** - Validate sequential execution 5. **Integration testing** - End-to-end validation 6. **Repeat for `InvoiceExtractionService`** - Phase 2 implementation ## Docs - [Extractor Code](https://cla-demo.postmansolutions.com/design/account-payable-to-fluxx-hld/documents-ai-intake-and-extract-service-for-fluxx/extractor-code.md) - [User Stories for Document AI Microservices Implementation](https://cla-demo.postmansolutions.com/design/account-payable-to-fluxx-hld/documents-ai-intake-and-extract-service-for-fluxx/user-stories-for-document-ai-microservices-implementation.md)