QE Strategy---AutoExtract Platform

Introduction

The Auto Extract Platform project aims to centralize and scale the platform, involving three main initiatives: Tax Auto Extract Worker, Accounts Payable Extract Worker, and SALT Auto Extract Worker.

Key Components and Functional Areas - Background

1. Classification Service: This service is expected to classify incoming documents from different external services based on their type. For example, SALT Auto Extract services are expected to push State and Local Tax-related documents like Form 540 from California or invoice documents from the AP-Accounts Payable Extract service. 
2. Document Splitter Service: This service is expected to split documents that contain multiple different forms into individual documents for each form. 
3. Message Broker/Service Bus: All communication with external services on the status of the workflows/processes will happen through messages published to a message broker/service bus. 
4. Extraction Services/Workers: There will be individual dedicated extraction services/workers that leverage custom LLM models available on Microsoft Document Intelligence Studio. The main workers/services in scope are Tax Auto Extract Worker, AP-Accounts Payable Extract Worker, and SALT Auto Extract Worker, each calling their respective LLMs for data extraction. 
5. Post Processor Service: Post processors can be stood up to handle any of the extraction results before writing the extraction document to a database with a standardized format. Long-term storage for analytics could be added during the post-process depending on data retention policies and security concerns with GRC. 
6. Logging, Error Handling, State Management/Tracking, Recovery, Resilience, Performance: These aspects will be tested through APIs, focusing on the performance of individual services and the workflow orchestration.

QA Strategy

The QA strategy will be implemented using an Agile Scrum methodology, focusing on continuous integration, iterative testing, and progressive quality assurance throughout the development lifecycle.

Functional Testing Areas

Classification Service Testing

• Validate document type classification accuracy across different external services
• Test classification logic for various document types (e.g., SALT tax documents, Form 540, invoice documents)

Document Splitter Service Testing

• Validate capabilities to split multi-form documents into individual documents
• Ensure accurate separation of different forms within complex documents
• Verify individual form extraction integrity

Extraction Services/Workers Testing - More under the AI Model functional Testing

• Validate individual extraction worker functionality
• Test specific LLM model performance for each dedicated worker:
  • Tax Auto Extract Worker
  • Accounts Payable Extract Worker
  • SALT Auto Extract Worker

API Testing

• Request platform to expose key business logic through APIs
• Conduct comprehensive API functionality verification
• Test API endpoints for each service
• Validate request/response patterns
• Ensure proper error handling and status codes
• Develop reusable, automated API tests to exercise core platform functions
• Conduct API regression tests and expand test coverage to include negative tests (e.g., invalid API requests, timeouts)
• Manual API Testing: Use Postman for quick exploratory testing and request validation.  
  • Local API Automation: Use Postman for automating API tests locally.  
  • Pipeline API Testing: Push for Postman integration in CI/CD for automated API validation.  
  • End-to-End (E2E) API Testing: Develop custom standalone scripts in TypeScript for complex API flows.

Manual Testing

• For complex, edge-case scenarios or where automation is not feasible, conduct manual testing. This includes exploratory testing, usability checks and validating new or evolving features

UI Testing

• If UI is present:

  • Adopt the approved Playwright framework for UI automation.  
  • If NO UI:
    • Focus on API and backend automation, using Postman for API tests and TypeScript scripts for broader E2E validation.

Automation Testing

• Develop reusable test cases and automate tests in parallel with development (Shift-Left approach)
• Integrate automated tests into the CI/CD pipeline to ensure they run with every deployment, with automated smoke and regression tests to verify build stability and functionality

Test Case Design

• Identify test scenarios: based on requirements and user stories.
• Define test cases: detailed steps to execute the test scenario.
• Specify test data: data required for executing the test case.
• Define expected results: the expected outcome of the test case.

Test Evidence

• Add comment on the test case including screenshot. This would be an additional effort for the QA team, but this will serve as proof in case other teams would check what was previously implemented in the specific user story.

AI Model Functional Testing

Core Evaluation Metrics

• F1 Score Assessment
  • Calculate precision and recall for each extraction worker
  • Establish performance thresholds
• Groundedness Evaluation
  • Verify responses rooted in source document content
  • Assess direct attribution of extracted information
• Prompt Variability Testing
  • Create comprehensive prompt library
  • Test model performance across different scenarios

Validation Approach

• Manually curate diverse ground truth dataset
• Initial manual validation
• Gradual automation of testing processes
• Continuous model performance monitoring

Non-Functional Testing Areas

Performance Testing

• Individual service performance benchmarking (API performance tests with jmeter/k6)
• Load testing for extraction services
• Response time and throughput measurements

Reliability and Resilience Testing (Stretch goal?)

• Use chaos engineering techniques
• Simulate various failure scenarios
• Conduct chaos-mesh experiments (stretch goal)
• Verify recovery and retry of failed document processing
• Test state management, error handling, and graceful termination
• Ensure platform can handle unexpected issues and recover gracefully

Security Testing (Stretch goal?)

AI Model Performance Testing (Stretch goal?)

Agile Testing Phases

In-Sprint Testing

1. Sprint Planning
   • Identify specific testing requirements
   • Define clear acceptance criteria
   • Create test cases aligned with user stories
2. Development Phase Testing
   • Perform continuous unit testing
   • Enable immediate defect identification and resolution
3. In-Sprint Acceptance Testing
   • Validate completed user stories
   • Verify acceptance criteria are met
   • Conduct thorough functionality checks
   • Ensure alignment with sprint goals

Regression and Continuous Testing

1. Regression Test Suite
   • Develop comprehensive regression test cases
   • Expand test coverage incrementally
   • Automate regression test execution
2. Continuous Integration Testing
   • Implement automated test runs on each code commit
   • Provide quick feedback on potential integration issues
   • Prevent regression of existing functionality

Environment Management

• Ensure access to stable and reliable QA testing environments
• Implement test reports for visibility
• Maintain real-time test dashboards
• Provide early feedback on testability and potential gaps in requirements

Recommended Environment Landscape

1. Development Environment - Initial code development and unit testing
2. Integration Environment - Validate integration between different services and components
3. UAT/Staging Environment - Pre-production validation
4. Production Environment

Knowledge Transfer and Collaboration

• Ensure thorough documentation of testing processes, test cases, findings and knowledge transfer
• Facilitate knowledge sharing between Data Science and Enterprise Application Development teams
• Conduct regular collaborative sessions
• Document and share testing insights and learnings
• Standardize testing tools and frameworks across the team for consistency and ease of maintenance

Tools & Technologies

• Primary API Testing Tool: Postman
• E2E Test Scripting: TypeScript
• UI Automation (If applicable): Playwright