Account Payable to Fluxx HLD

Architecture Overview

The migration transforms the current Python/Temporal monolithic architecture into three distinct Node.js repositories deployed on Azure Kubernetes Service, leveraging Azure Durable Functions for workflow orchestration and Azure Managed Identity for security.

Authentication Architecture with Azure Managed Identity

Managed Identity Flow

Repository Structure & Service Distribution

Repository 1: web-accountspayable (Frontend)

Purpose: User interface layer - remains unchanged from current React implementation

• Technology: React, TypeScript, NGINX
• Deployment: AKS pod with ingress controller
• Authentication: MSAL.js integration with Azure AD
• Responsibilities:
  • Invoice review and approval interfaces
  • Real-time workflow status dashboard
  • User management and configuration
  • Document viewer and annotation tools

Repository 2: web-api-accountspayable (Backend API)

Purpose: Business logic API layer using Fastify framework

• Technology: Fastify, TypeScript, Node.js
• Deployment: AKS pods with horizontal auto-scaling
• Authentication: Azure Managed Identity + JWT validation
• Core Modules:
  • Invoice Management Module: CRUD operations for invoice documents
  • Workflow Control Module: Durable Functions integration and workflow triggers
  • User Management Module: Authentication, authorization, and role management
  • Client Configuration Module: Multi-tenant settings and business rules
  • Integration Module: Fluxx microservices and ERP system connectors
  • Export Module: CSV, IIF, and custom format generation
  • Audit Module: User action logging and compliance tracking

Repository 3: db-accountspayable (Database Layer)

Purpose: Data access layer as reusable Fastify plugins

• Technology: Fastify plugins, TypeScript
• Deployment: Shared library consumed by API services
• Authentication: Azure Managed Identity for all database connections
• Core Modules:
  • Cosmos DB Plugin: Document storage and retrieval with partitioning
  • PostgreSQL Plugin: Relational data access with connection pooling
  • Redis Plugin: Caching and session management
  • Migration Module: Database schema versioning and updates
  • Connection Management: Pool management and retry logic

Workflow Migration: Temporal → Azure Durable Functions

Current State Analysis

The existing Temporal workflows consist of 9 sequential stages with complex state management, error handling, and human intervention points. Each stage has specific retry policies, timeout configurations, and failure handling mechanisms.

Target State Design

Main Orchestrator Function

• Name: InvoiceProcessingOrchestrator
• Trigger: HTTP trigger from Fastify API
• Responsibilities:
  • Coordinate all workflow stages
  • Maintain workflow state across long-running processes
  • Handle timeouts and escalations
  • Provide status updates to frontend

Activity Functions Mapping

Entity Functions

• WorkflowStateEntity: Maintains persistent workflow state
• ConfigurationEntity: Manages client-specific configurations
• ApprovalEntity: Handles approval routing and escalation

Node.js Migration Strategy

Technology Stack Replacement

Migration Work Breakdown

Phase 1: Foundation (4-6 weeks)

• Setup three repository structures
• Configure Azure infrastructure with Bicep
• Implement Managed Identity authentication
• Create base Fastify application structure
• Setup CI/CD pipelines for all repositories

Phase 2: Data Layer (3-4 weeks)

• Migrate database schemas to Azure
• Implement Fastify database plugins
• Create data access patterns with Managed Identity
• Migrate existing data with validation
• Performance testing and optimization

Phase 3: API Layer (6-8 weeks)

• Port all REST endpoints from Python to Fastify
• Implement JSON schema validation
• Integrate with Fluxx microservices
• Create authentication middleware
• Comprehensive API testing

Phase 4: Workflow Migration (8-10 weeks)

• Design and implement Azure Durable Functions
• Create activity functions for each workflow stage
• Implement state management and error handling
• Integration with Fluxx services
• End-to-end workflow testing

Phase 5: Integration & Testing (4-6 weeks)

• Frontend integration updates
• Performance optimization
• Security testing and compliance
• User acceptance testing
• Production deployment preparation

Fluxx Integration Architecture

Integration Points

• Document Classification: Replace internal document intelligence with Fluxx API
• Data Extraction: Leverage Fluxx ML models for invoice parsing
• Validation Services: Use Fluxx validation algorithms
• AI Coding: Utilize Fluxx AI for GL account coding

Service Communication

• RESTful API calls with Azure Managed Identity authentication
• Circuit breaker patterns for resilience
• Async processing with retry mechanisms
• Comprehensive logging and monitoring

User Stories

Epic 1: Infrastructure Foundation

Story 1.1: Repository Setup and Structure

As a DevOps Engineer
I want to create three separate repositories with proper CI/CD pipelines
So that each service can be developed and deployed independently

Acceptance Criteria:

• Three repositories created with appropriate naming conventions
• Azure DevOps pipelines configured for each repository
• Docker containers and Kubernetes manifests created
• Environment-specific configuration management
• Security scanning and code quality gates

Story 1.2: Azure Infrastructure Provisioning

As a Cloud Architect
I want to provision Azure infrastructure using Infrastructure as Code
So that the environment is consistent and reproducible

Acceptance Criteria:

• AKS cluster with appropriate node pools
• Azure Cosmos DB and PostgreSQL with Managed Identity
• Azure Functions app for Durable Functions
• Azure Cache for Redis with Managed Identity
• Networking, security groups, and ingress configuration

Epic 2: Database Layer Migration

Story 2.1: Database Schema Migration

As a Database Engineer
I want to migrate existing database schemas to Azure services
So that data structure and relationships are preserved

Acceptance Criteria:

• PostgreSQL schema migrated with indexes and constraints
• Cosmos DB collections created with proper partitioning
• Data migration scripts with validation
• Performance benchmarking completed
• Backup and recovery procedures established

Story 2.2: Database Access Layer Development

As a Backend Developer
I want to create Fastify plugins for database access
So that data operations are consistent across services

Acceptance Criteria:

• Cosmos DB plugin with Managed Identity authentication
• PostgreSQL plugin with connection pooling
• Redis plugin for caching operations
• Error handling and retry logic implemented
• Transaction support where needed

Epic 3: API Layer Migration

Story 3.1: Fastify Framework Setup

As a Backend Developer
I want to establish Fastify application foundation
So that APIs are high-performance and well-structured

Acceptance Criteria:

• Fastify server with TypeScript configuration
• Plugin architecture for modularity
• JSON schema validation with comprehensive schemas
• Swagger documentation auto-generation
• Logging, monitoring, and health checks

Story 3.2: Authentication and Authorization

As a Security Engineer
I want to implement Azure Managed Identity authentication
So that API access is secure and credential-free

Acceptance Criteria:

• JWT validation middleware implementation
• Role-based access control (RBAC) system
• Managed Identity configuration for Azure services
• Session management with Redis
• Security headers and CORS configuration

Story 3.3: Invoice Management API Migration

As a Backend Developer
I want to port invoice CRUD operations to Fastify
So that frontend functionality remains unchanged

Acceptance Criteria:

• All invoice endpoints migrated with identical functionality
• JSON schema validation for all payloads
• Database operations using new plugin architecture
• Error handling matching existing behavior
• Performance benchmarks meet or exceed current system

Story 3.4: Fluxx Integration Implementation

As a Integration Developer
I want to integrate with Fluxx microservices
So that document processing leverages existing capabilities

Acceptance Criteria:

• HTTP client with circuit breaker and retry logic
• Managed Identity authentication to Fluxx services
• Error handling and fallback mechanisms
• Comprehensive logging and monitoring
• Performance optimization for concurrent requests

Epic 4: Workflow Migration to Azure Durable Functions

Story 4.1: Durable Functions Project Setup

As a Backend Developer
I want to create Azure Durable Functions project structure
So that workflow orchestration is reliable and maintainable

Acceptance Criteria:

• Functions v4 programming model with TypeScript
• Local development environment with emulation
• Managed Identity configuration for all dependencies
• Extension bundles for optimal performance
• Monitoring and Application Insights integration

Story 4.2: Main Orchestrator Implementation

As a Workflow Developer
I want to create the main workflow orchestrator function
So that invoice processing follows defined business rules

Acceptance Criteria:

• Orchestrator function handling all 9 workflow stages
• State persistence across long-running processes
• Error handling with appropriate retry policies
• Support for manual interventions and approvals
• Real-time status updates to frontend

Story 4.3: Activity Functions Development

As a Workflow Developer
I want to implement activity functions for each workflow stage
So that business logic is properly encapsulated and testable

Acceptance Criteria:

• Ingest activity integrating with Fluxx services
• Intake activity with data validation logic
• Matching activity with ERP integration
• Coding activity with business rules engine
• Manual review and approval activities
• Export and filing activities with ERP integration

Story 4.4: State Management and Error Handling

As a Workflow Developer
I want to implement robust state management and error handling
So that workflows are resilient and recoverable

Acceptance Criteria:

• Entity functions for persistent state management
• Comprehensive error handling with proper logging
• Retry policies matching business requirements
• Timeout handling and escalation procedures
• Workflow restart and recovery capabilities

Epic 5: Integration and Testing

Story 5.1: End-to-End Integration Testing

As a QA Engineer
I want to validate complete workflow functionality
So that the migrated system works as expected

Acceptance Criteria:

• Complete invoice processing workflows tested
• Integration with Fluxx services validated
• ERP system integration verified
• Performance benchmarks achieved
• Error scenarios properly handled

Story 5.2: Frontend Integration Updates

As a Frontend Developer
I want to update React application to work with new backend
So that user experience remains consistent

Acceptance Criteria:

• API endpoints updated to new Fastify backend
• Authentication flow updated for Azure AD
• Real-time workflow status integration
• Error handling for new API responses
• No breaking changes to existing UI functionality

Story 5.3: Performance Optimization and Monitoring

As a Site Reliability Engineer
I want to optimize performance and implement monitoring
So that the system performs well in production

Acceptance Criteria:

• Application Insights integration across all services
• Performance metrics and alerting configured
• Auto-scaling policies implemented
• Resource optimization completed
• Capacity planning documentation created

Epic 6: Production Deployment

Story 6.1: Production Environment Preparation

As a DevOps Engineer
I want to prepare production environment for deployment
So that the system can handle production workloads

Acceptance Criteria:

• Production infrastructure provisioned and configured
• Security reviews and penetration testing completed
• Disaster recovery and backup procedures implemented
• Monitoring and alerting systems configured
• Documentation and runbooks completed

Story 6.2: Data Migration and Cutover

As a Data Engineer
I want to migrate production data safely
So that business operations can continue without interruption

Acceptance Criteria:

• Production data migrated with validation
• Rollback procedures tested and documented
• Parallel running period with data synchronization
• User acceptance testing in production environment
• Go-live support and monitoring

I’ll revise Epic 7 to focus on direct API calls to the Fluxx status server for uploads, classification, and extraction statuses within each epic stage, removing the Service Bus messaging approach.

Epic 7: Fluxx Status Server Integration

Story 7.1: Fluxx Status Server Client Implementation

As a Backend Developer
I want to create a query to the Fluxx Status Server
So that APAutoExtract can query upload, classification, and extraction statuses directly

Acceptance Criteria:

• HTTP client library with Azure Managed Identity authentication to Fluxx status server
• API methods for querying upload status by document ID
• API methods for querying classification status and progress
• API methods for querying extraction status and results
• Retry logic with exponential backoff for transient failures
• Circuit breaker pattern for handling Fluxx status server outages

Story 7.2: Upload Status Integration in Ingest Stage

As a Workflow Developer
I want to integrate upload status polling in the ingest activity
So that the workflow waits for successful document upload before proceeding

Acceptance Criteria:

• Ingest activity polls Fluxx status server for upload completion
• Configurable polling intervals (e.g., every 5 seconds) until upload completes
• Timeout handling if upload takes longer than expected (e.g., 10 minutes)
• Status caching in Redis to reduce API calls for concurrent workflows
• Error handling for upload failures with appropriate workflow termination
• Progress updates sent to frontend for real-time user feedback

Story 7.3: Classification Status Integration in Intake Stage

As a Workflow Developer
I want to monitor classification progress during the intake activity
So that the workflow can proceed once document classification is complete

Acceptance Criteria:

• Intake activity polls classification status endpoint
• Progress percentage tracking and display in workflow status
• Classification result retrieval and validation
• Handling of classification confidence scores and thresholds
• Error handling for classification failures with retry options
• Integration with manual review workflow if classification confidence is low

Story 7.4: Extraction Status Integration in Data Processing

As a Workflow Developer
I want to track data extraction progress and retrieve results
So that extracted invoice data is available for matching and validation

Acceptance Criteria:

• Data extraction status polling with progress tracking
• Retrieval of extracted structured data from Fluxx services
• Data validation and transformation for internal data model
• Handling of extraction quality scores and confidence levels
• Error handling for extraction failures with fallback to manual data entry
• Integration with data validation workflows

Story 7.5: Status Caching and Performance Optimization

As a Performance Engineer
I want to implement intelligent status caching
So that Fluxx status server load is minimized while maintaining responsiveness

Acceptance Criteria:

• Redis caching layer for status responses with appropriate TTL
• Smart polling intervals based on processing stage and estimated completion time
• Bulk status queries for multiple documents when possible
• Connection pooling for Fluxx status server HTTP connections
• Performance metrics tracking for status query response times
• Rate limiting to respect Fluxx status server API limits

Story 7.6: Real-time Status Dashboard Updates

As a Frontend Developer
I want to display real-time Fluxx processing status in the UI
So that users can monitor document processing progress across all stages

Acceptance Criteria:

• WebSocket integration for pushing status updates to frontend
• Progress bars showing upload, classification, and extraction progress
• Status timeline view showing progression through all Fluxx processing stages
• Error state display with detailed error messages from Fluxx services
• Retry and cancellation options for failed or stuck processes
• Processing queue view showing pending documents in Fluxx services

Story 7.7: Error Handling and Fallback Mechanisms

As a Site Reliability Engineer
I want to implement robust error handling for Fluxx status integration
So that workflow failures are gracefully handled and recoverable

Acceptance Criteria:

• Comprehensive error classification for different Fluxx status server failures
• Automatic retry mechanisms with configurable retry policies
• Fallback to manual processing when Fluxx services are unavailable
• Error escalation and notification procedures for critical failures
• Workflow pause and resume capabilities during Fluxx service outages
• Comprehensive error logging and alerting for troubleshooting

Story 7.8: Status History and Audit Trail

As a Compliance Officer
I want to maintain detailed status history and audit trails
So that document processing can be tracked for compliance and troubleshooting

Acceptance Criteria:

• PostgreSQL storage of complete status history for each document
• Audit trail showing all status queries and responses
• Processing timeline with timestamps for each stage completion
• Integration with Application Insights for monitoring and analytics
• Status report generation for compliance and operational review
• Data retention policies for status history and audit data

Story 7.9: Configuration and Monitoring

As a DevOps Engineer
I want to configure monitoring and alerting for Fluxx status integration
So that issues can be detected and resolved proactively

Acceptance Criteria:

• Application Insights integration for status query telemetry
• Custom metrics for response times, error rates, and processing delays
• Alerting rules for Fluxx status server availability issues
• Dashboard showing Fluxx processing health and performance metrics
• Configuration management for polling intervals and timeout values
• Capacity planning metrics for scaling decisions

Story 7.10: Integration Testing and Validation

As a QA Engineer
I want to validate Fluxx status server integration
So that status tracking works reliably across all processing stages

Acceptance Criteria:

• End-to-end testing of status tracking through complete workflows
• Status accuracy validation comparing Fluxx responses with actual processing
• Error scenario testing including network failures and service timeouts
• Performance testing under high-volume concurrent status queries
• Load testing to validate status server integration under stress
• Monitoring validation for all failure and performance scenarios

Integration Points in Each Workflow Stage

Ingest Stage (WorkflowInvoiceIngest)

• Upload Status Check: Poll Fluxx upload service for document upload completion
• Upload Validation: Verify document is successfully stored and accessible
• Error Handling: Retry upload if initial attempt fails

Intake Stage (WorkflowInvoiceIntake)

• Classification Status: Monitor document classification progress
• Classification Results: Retrieve classification confidence and category
• Quality Validation: Verify classification meets confidence thresholds

Matching Stage (WorkflowInvoiceMatching)

• Extraction Status: Track data extraction progress and completion
• Data Retrieval: Fetch extracted structured data from Fluxx services
• Data Validation: Verify extraction quality and completeness

All Subsequent Stages

• Status Monitoring: Continue monitoring for any additional Fluxx processing
• Error Recovery: Handle any delayed failures or status updates
• Final Validation: Confirm all Fluxx processing is complete before workflow closure

I’ll create Epic 8 for configuring Fluxx application-level settings to work with our extractors, classifiers, and post-processing events.

Epic 8: Fluxx Application Configuration & Registration

Story 8.1: Fluxx Configuration Management API

As a System Administrator
I want to create APIs to manage Fluxx application-level configurations
So that extractors, classifiers, and post-processors can be configured for optimal performance

Acceptance Criteria:

• REST API endpoints for CRUD operations on Fluxx configurations
• Configuration schema validation for extractor parameters
• Configuration versioning and rollback capabilities
• Multi-tenant configuration support with client isolation
• Configuration audit trail with change tracking
• Integration with Azure Key Vault for sensitive configuration values

Story 8.2: Document Extractor Configuration

As a Configuration Manager
I want to configure Fluxx document extractors for invoice processing
So that data extraction is optimized for our specific document types and formats

Acceptance Criteria:

• Configuration interface for OCR engine parameters (confidence thresholds, language settings)
• Field extraction rules configuration (vendor name patterns, date formats, amount parsing)
• Template-based extraction configuration for known invoice formats
• Custom field mapping configuration for client-specific requirements
• Extraction quality thresholds and fallback mechanisms
• Performance tuning parameters (batch sizes, parallel processing limits)

Story 8.3: Document Classifier Configuration

As a ML Engineer
I want to configure Fluxx document classifiers for accurate document categorization
So that invoices are properly classified and routed through appropriate processing workflows

Acceptance Criteria:

• Machine learning model configuration for document classification
• Training data management and model retraining parameters
• Classification confidence thresholds and uncertainty handling
• Custom classification categories for client-specific document types
• A/B testing configuration for model performance comparison
• Classification feedback loop configuration for continuous improvement

Story 8.4: Post-Processing Event Configuration

As a Business Analyst
I want to configure post-processing events and business rules
So that extracted data undergoes appropriate validation and transformation

Acceptance Criteria:

• Business rule engine configuration for data validation
• Custom validation rules for client-specific requirements
• Data transformation pipelines configuration
• Quality assurance checkpoints and manual review triggers
• Integration points configuration for downstream systems
• Event-driven processing configuration for real-time updates

This Epic 8 ensures that Fluxx services are properly configured and optimized for the specific needs of the APAutoExtract system, providing flexibility for different client requirements while maintaining performance and reliability.

This comprehensive migration strategy ensures a systematic approach to transforming the existing Python/Temporal system into a modern, cloud-native microservices architecture leveraging Azure’s managed services for security, scalability, and reliability.