Skip to navigation
Quality Engineering

Test Plan for E2E Automation

Objective

The objective of this Test Plan is to validate the functionality, reliability, and performance of the tax document ingestion and processing platform through a comprehensive backend-focused E2E automation framework. The framework will integrate with and test backend APIs, Azure Service Bus (SB) flows, BLOB Storage operations, and SQL DB persistence across a phased rollout of supported AI extraction tools.

The focus on backend automation - excluding the Document AI UI- stems from practical priorities:

  • Increased Test Stability in complex microservices environment
  • Faster feedback in CI/CD
  • Reduced flakiness due to browser or UI changes
  • Reliability and repeatability in headless environments (e.g., AKS)

Scope

In Scope

  • Fluxx Platform, Document AI, SaaS deployment
  • Upload, Scanning, and processing of various document types via Blob and/or API
  • Service Bus message flow validations and schema conformity
  • Extraction result comparison with Golden Samples (GS) for:
    • JSON Structure/Schema
    • Key-value data
    • Confidence score (on JSON) validation with threshold checks
  • SQL DB and blob storage persistence and metadata validation
  • TBD: Post-processing flows (External Blobs, Data Lake) integration with schema (JSON data) compliance
  • Performance and throughput analysis (files per minute, extraction latency)
  • E-mail file intake - TBD

Out of Scope

  • Classifier Service
    • File splitting
    • File conversion
  • Post-Processing
  • Document AI UI interactions
  • Model accuracy tuning or training validations (owned by Data Science team)
  • UI-triggered-only flows (unless backend alternatives are available)

Product Scope & Use Case Coverage

PhaseInput TypeKey FlowOutput/Validation
1Individual File Processing in Fluxx (Invoice files)Upload -> Scan -> Extract -> StoreExtracted JSON, SB, messages, SQL read/write, blob storage
2Document AI as a client flow - mock UIUpload -> Scan -> Extract -> StoreExtracted JSON, SB, messages, SQL read/write, blob storage
2Document AI as a client flow - email uploadFile Intake -> Scan -> Extract -> StoreExtracted JSON, SB, messages, SQL read/write, blob storage, Complete Invoice process

End-to-End Test Flow

#FunctionActionsVerification Approach
1Registration ServiceCreate Registration Endpoint POSTVerify records created
2Upload fileFile-Intake POSTTriggers Blob Created msg: Submitted-File topic, Status Check service: created, file metadata
3File ValidationScan, file type checkStatus check service: Validated or failed validation
4Save to internal file storageAZ blob storageVerify internal storage, status check: copied or failed copying
5File-Intake Servicepublishes SB msgVerify SB msg: accepted-file topic
6ExtractorPolling: isReady = True or SB msg?
7Extraction StoredAZ SQL storageVerify extracted data stored, Status Check: extracted
8Postal ServiceSends POST or SB msgSB Msg: To-Send topic
9STatus checkverify completedStatus check service: completed
10Documents AIextractor sends message to Fluxxverify Files match
11Compare DataData compared to Golden SampleProvide accuracy rating of extraction to Golden Sample

Steps in Flow

  1. Registration Service
    1. Create Registration Endpoint POST
    2. Creates records in SQL DB
  2. Upload file to blob (email, direct)
    1. Triggers Blob Created Event - Submitted-File
    2. Status Check service: created
  3. File Validation
    1. Scan, file type, quarantine if bad
    2. Status Check service: validated OR
    3. Status Check service: failed validation
  4. Save to Internal file storage - AZ blob
    1. Verify file present in internal storage bucket
    2. Status Check service: copied OR
    3. Status Check service: failed copying
  5. File Intake file service
    1. Publishes new file related data event to SB - accepted-file topic
  6. Classify - OOS for MVP
    1. Convert
    2. Splitter
    3. Process
  7. Document Extractor(s)
    1. External processing, polling for isReady: True
  8. Raw Extract Data Storage - Az Blob
    1. Status Check service: extracted
  9. Post-Processing - OOS
  10. Postal Service
  11. To-Send topic created, sends http POST or SB message
  12. Completion
  13. Status Check Service: completed
  14. UI support for current status - GET /status/files/<id>

Functional Requirement Verifications

#FunctionVerification Approach
1Blob UpdateSB message, File presence, checksum, metadata validation
2Virus ScanSB message, allow/block logic (stretch goal), logging
6ExtractionPubish or polling, message (extraction start, complete), Extraction Results: Verify extracted JSON & compared to GS
7Confidence ScoresThreshold comparison vs GS with warning for drop
8NoSQL PersistenceFileId, metadata (all file and location info & process status updates etc., insert/update validation
9Access/Data Lake uploadSchema of payload to Acc. Tax & Data Lake, API Status codes
10Re-upload Same Doc ???Validate Overwrite, extraction re-trigger and SQL metadata updates (TBD)
12Documents AI - Invoice processingTBD

Non-Functional Requirement Verifications (TBD)

  • Performance: Time to extract per file batch (10, 50, 100). Files from the primary test dataset used for functional testing
  • Throughput: Parallel upload handling, Blob lag
  • Reliability: Retry logic, SB message persistence and reprocessing validation
  • Observability: Timestamps for SB/Blob checkpoints
  • Scalability: Simulate high file count across doc types (TBD)

Additional Information

Integration with Azure Service Bus

  • Each processing module emits completion events via Azure Service Bus.
  • Test Framework subscribes to relevant SB topics/queues to:
    • Detect when file processing is complete
    • Trigger result verification against GS
    • Capture delays and issues

Test Data

  • 5 PDF documents from APAI team will be uploaded and extracted
  • ClientId 66 in DocsAI is used
    • Client created during runtime with DocsAI Extractor details