Stage Mapper for web api documents ai

Supported Stages

The system will support the following stages:

• extraction - Initial data extraction from documents

• matching - Match invoice data with purchase orders

• manual - Manual review and approval

• filing - File the invoice in the appropriate system

• coding - Apply accounting codes

• export_quickbooks - Export to QuickBooks

• closed - Final stage (terminal)

Client Configuration Structure

Each client’s configuration will include a stages object defining available stages and their resolutions:

{  
  "stages": {  
    "extraction": {  
      "resolutions": {  
        "failed": {  
          "stage_name": "closed"  
        },  
        "continue": {  
          "stage_name": "matching"  
        }  
      }  
    },  
    "matching": {  
      "resolutions": {  
        "failed": {  
          "stage_name": "closed"  
        },  
        "continue": {  
          "stage_name": "filing"  
        }  
      }  
    },  
    "manual": {  
      "resolutions": {  
        "failed": {  
          "stage_name": "closed"
        },  
        "continue": {  
          "stage_name": "filing"  
        }  
      }  
    },  
    "filing": {  
      "resolutions": {  
        "failed": {  
          "stage_name": "closed"  
        },  
        "continue": {  
          "stage_name": "closed"  
        }  
      }  
    },  
    "coding": {  
      "resolutions": {  
        "failed": {  
          "stage_name": "closed"  
        },  
        "continue": {  
          "stage_name": "manual"  
        }  
      }  
    },  
    "export_quickbooks": {  
      "resolutions": {  
        "failed": {  
          "stage_name": "closed"  
        },  
        "continue": {  
          "stage_name": "closed"  
        }  
      }  
    }  
  }  
}  

Technical Requirements

1. StageMapper Utility Class

Location: src/helpers/stageMapper.ts

Key Methods:

• getNextStage(currentStage, resolution, clientConfig) - Determine next stage based on current stage and resolution

• transitionStage(entityId, currentStage, resolution, clientConfig) - Update entity stage in database

• validateStageConfig(clientConfig) - Validate client stage configuration

• getAvailableResolutions(stage, clientConfig) - Get available resolutions for a stage

2. TypeScript Types

Location: src/types/stageMapper.types.ts

Required Interfaces:

interface StageResolution {  
  stage_name: string;  
  label?: string;  
}  
  
interface StageResolutions {  
  failed?: StageResolution;  
  continue?: StageResolution;  
  [key: string]: StageResolution | undefined;  
}  
  
interface StageConfig {  
  resolutions: StageResolutions;  
}  
  
interface StagesConfig {  
  extraction?: StageConfig;  
  matching?: StageConfig;  
  manual?: StageConfig;  
  filing?: StageConfig;  
  coding?: StageConfig;  
  export_quickbooks?: StageConfig;  
  [key: string]: StageConfig | undefined;  
}  

3. Data Model Integration

• Update EntitiesModel to support stage transitions

• Ensure stage field in ExtractedEntities table is updated correctly

• Create snapshot before stage transitions (using existing snapshot functionality)

4. Client Type Updates

Location: src/api/v1/client/clients.type.ts

Update ClientConfig interface to include optional stages configuration.

5. Invoice Extraction Integration

Location: src/api/v1/invoiceExtraction/invoiceExtraction.service.ts

Integrate StageMapper into the invoice extraction workflow:

• On Extraction Success: Use StageMapper to transition from extraction stage with continue resolution

• On Extraction Failure: Use StageMapper to transition from extraction stage with failed resolution

• Pass client configuration to StageMapper for stage determination

• Ensure proper error handling and logging for stage transitions

• Maintain backward compatibility if stage configuration is not present

Key Integration Points:

// On successful extraction  
await StageMapper.transitionStage(  
  extractedEntityId,  
  'extraction',  
  'continue',  
  clientConfig  
);  
  
// On extraction failure  
await StageMapper.transitionStage(  
  extractedEntityId,  
  'extraction',  
  'failed',  
  clientConfig  
);  

Acceptance Criteria

Functional Requirements

1. ✅ StageMapper can determine the next stage based on current stage and resolution

2. ✅ StageMapper can update entity stage in the database

3. ✅ StageMapper validates client stage configuration

4. ✅ StageMapper handles missing stage configurations gracefully (use defaults or throw errors)

5. ✅ StageMapper supports both “failed” and “continue” resolution types

6. ✅ StageMapper creates snapshots before stage transitions

7. ✅ Invoice extraction service calls StageMapper on successful extraction

8. ✅ Invoice extraction service calls StageMapper on failed extraction

9. ✅ Stage transitions do not break existing extraction workflow

10. ✅ System works correctly when stage configuration is missing (fallback behavior)

Non-Functional Requirements

1. ✅ Code follows SOLID principles

2. ✅ Unit tests achieve 85%+ code coverage

3. ✅ All functions have proper TypeScript types (no any)

4. ✅ Error handling with descriptive messages

5. ✅ Logging for stage transitions (including entity ID, old stage, new stage, resolution)

6. ✅ No sensitive data or PII in logs

7. ✅ Stage transitions complete within 100ms (excluding database latency)

8. ✅ No performance degradation to existing extraction workflow

Edge Cases

1. ✅ Handle undefined stage in configuration

Test Scenarios

Unit Tests (src/helpers/tests/stageMapper.test.ts)

1. getNextStage()

   - ✅ Should return correct next stage for valid configuration

   - ✅ Should return null for missing stage configuration

   - ✅ Should return null for missing resolution

   - ✅ Should handle custom resolution types

   - ✅ Should handle nested stage configurations

   - ✅ Should handle all supported stages (extraction, matching, manual, filing, coding, export_quickbooks)

2. transitionStage()

   - ✅ Should update entity stage successfully

   - ✅ Should create snapshot before transition

   - ✅ Should handle invalid entity ID

   - ✅ Should handle database errors gracefully

   - ✅ Should sanitize stage values for SQL (max 50 chars)

   - ✅ Should log stage transitions with all relevant data

   - ✅ Should update ModifiedOn timestamp

   - ✅ Should rollback on snapshot creation failure

3. validateStageConfig()

   - ✅ Should validate correct configuration

   - ✅ Should reject missing resolutions object

   - ✅ Should reject invalid stage names

   - ✅ Should handle empty configuration

   - ✅ Should validate resolution structure (stage_name required)

   - ✅ Should allow optional label field

4. getAvailableResolutions()

   - ✅ Should return all resolutions for a stage

   - ✅ Should return empty array for missing stage

   - ✅ Should include custom resolutions beyond failed/continue

   - ✅ Should return resolution labels when present

Integration Tests (src/api/v1/invoiceExtraction/tests/invoiceExtraction.service.test.ts)

1. Extraction Success Flow

   - ✅ Should call StageMapper with ‘continue’ resolution on successful extraction

   - ✅ Should transition from ‘extraction’ to configured next stage

   - ✅ Should use client-specific stage configuration

   - ✅ Should create snapshot before stage transition

   - ✅ Should complete extraction even if stage transition fails (log error)

   - ✅ Should update entity with correct stage value

2. Extraction Failure Flow

   - ✅ Should call StageMapper with ‘failed’ resolution on extraction failure

   - ✅ Should transition to failed stage (typically ‘closed’)

   - ✅ Should create snapshot before stage transition

   - ✅ Should log extraction error and stage transition

   - ✅ Should handle missing client stage configuration gracefully

3. Configuration Scenarios

   - ✅ Should work with default stage configuration (no client config)

   - ✅ Should work with client-specific stage configuration

   - ✅ Should handle missing ‘extraction’ stage in config (use default or skip transition)

   - ✅ Should handle malformed stage configuration without breaking extraction

4. Backward Compatibility

   - ✅ Should work when StageMapper is not available (graceful degradation)

   - ✅ Should maintain existing behavior if stage config is undefined

   - ✅ Should not break existing extraction tests

End-to-End Test Scenarios

1. Happy Path

   - ✅ Upload document → Extraction succeeds → Stage transitions to ‘matching’

   - ✅ Entity stage updated in database

   - ✅ Snapshot created with ‘extraction’ stage

Definition of Done

• StageMapper utility class created in src/helpers/stageMapper.ts

• TypeScript types defined in src/types/stageMapper.types.ts

• Unit tests with 85%+ coverage in src/helpers/tests/stageMapper.test.ts

• Integration tests for invoice extraction updated

• Invoice extraction service integrated with StageMapper

• Stage transitions on extraction success implemented

• Stage transitions on extraction failure implemented

• All unit tests passing (including new StageMapper tests)

• All integration tests passing (including extraction workflow tests)

• ESLint and Prettier checks passing

• No TypeScript errors or warnings

• Client types updated to include stages configuration

• Performance benchmarks meet requirements (<100ms stage transitions)

• Error handling tested and verified

• Logging verified (no PII, proper log levels)

• Code reviewed and approved

• Documentation complete (inline comments + this story)

• Backward compatibility verified (works without stage config)

1. Load Testing

   - ✅ Stage transitions complete within 100ms (excluding DB latency)

   - ✅ No performance degradation with 100 concurrent extractions

   - ✅ Database connection pooling handles stage transition load

2. Stress Testing

   - ✅ Handle rapid stage transitions for same entity (race conditions)

   - ✅ Handle database connection failures gracefully

   - ✅ Recover from transient errorstion

   - Should reject missing resolutions

   - Should reject invalid stage names

   - Should handle empty configuration

4. getAvailableResolutions()

   - Should return all resolutions for a stage

   - Should return empty array for missing stage

   - Should include custom resolutions

Implementation Notes

Dependencies

• EntitiesModel - for database operations

• sanitizeStageStatus() - from invoiceExtraction.utils.ts for SQL safety

• Client configuration from database or default config

Error Handling

• Use custom error types for stage mapping errors

• Log all stage transitions for audit trail

• Never expose sensitive data in error messages

Performance Considerations

• Stage lookups should be fast (O(1) object access)

• Database updates should be atomic

• Snapshot creation should not block stage transition

Security Considerations

• Validate all client configuration inputs

• Sanitize stage names before SQL queries

• Ensure proper authorization before stage transitions

Future Enhancements

1. Support for conditional stage transitions (based on entity data)

2. Stage transition webhooks/notifications

3. Stage transition rollback functionality

4. Visual workflow builder for stage configuration

5. Audit log for all stage transitions

6. Support for parallel stages

7. Time-based automatic stage transitions

Related Stories

• INVOICE_EXTRACTION_STORIES.md - Current extraction workflow

• EXTRACTED_ENTITIES_IMAGE_BOUNDING_STORIES.md - Entity data structure

Definition of Done

• StageMapper utility class created in src/helpers/stageMapper.ts

• TypeScript types defined in src/types/stageMapper.types.ts

• Unit tests with 85%+ coverage in src/helpers/tests/stageMapper.test.ts

• All tests passing

• ESLint and Prettier checks passing

• Client types updated to include stages configuration

• Code reviewed and approved

• Documentation complete

Questions/Risks

1. Q: Should stage transitions be synchronous or asynchronous?

   A: Start with synchronous, consider async for external integrations

2. Q: What happens if a stage transition fails mid-process?

   A: Use database transactions to ensure atomicity

3. Q: Should we support custom resolution types beyond “failed” and “continue”?

   A: Yes, use [key: string] to allow flexibility

4. Q: How do we handle stage configuration updates for existing entities?

   A: Entities use the configuration at time of transition; config changes only affect new transitions