📄 1099B-NonGovt-Extractor-Service Design

📘 Project Overview

Repository Name: poc-autoextract-platform-consolidated-1099-Extractor-Service

🧭 Overview

The `Consolidated-1099-Extractor-Service is a cloud-native Node.js microservice deployed on Azure Kubernetes Service (AKS). It automates the extraction of structured data from Consolidated 199 tax documents using Azure Document Intelligence and OpenAI. The service is triggered by messages from Azure Service Bus and publishes results to a downstream topic once processing is complete. Follow the Framework V2 format for all components

🔄 Workflow Summary

1. Trigger: Listens to the classified-file topic on Azure Service Bus.

    {
      "eventType": "file.classified",
      "fileId": "uuid",
      "parentUrl": "https://...",
      "childUrl": "https://...",
      "clientApplicationId": "uuid",
      "classifications": ["1099Gov"]
    }

2. Download: Retrieves the document from Azure Blob Storage.
3. Analyze: Uses Azure Document Intelligence and OpenAI to extract structured data.
4. Merge: Combines outputs from both services into a unified JSON format.
5. Upload: Stores the result back in Blob Storage.
6. Notify: Publishes a message to the extracted-result topic with a result URL.

    {
      "eventType": "result.extracted",
      "fileId": "uuid",
      "clientApplicationId": "uuid",
      "resultUrl": "https://.../results/<fileId>",
      "timestamp": "ISO8601"
    }

🧩 Component List

⚙️ Infrastructure

• Platform: Azure Kubernetes Service (AKS)
• Automation: https://spacelift.io/ (manages AKS, Azure Service Bus, Key Vault, and Helm deployments)
• Secrets: Azure Key Vault + CSI Driver
• Observability: Azure Monitor, Application Insights

📁 File Structure

1099B-NonGovt-Extractor-Service/ ├── src/ │ ├── index.js │ ├── config/ │ │ └── env.js │ ├── messaging/ │ │ ├── subscriber.js │ │ └── publisher.js │ ├── services/ │ │ ├── blobService.js │ │ ├── pdfParser.js │ │ ├── documentAI.js │ │ ├── openAIExtractor.js │ │ └── resultMerger.js │ └── utils/ │ └── logger.js ├── Dockerfile ├── helm/ │ └── chart.yaml ├── spacelift/ │ ├── stacks/ │ │ └── aks-stack.yaml │ └── modules/ │ └── azure-kubernetes/ │ └── main.tf ├── package.json └── README.md

🧩 Module Responsibilities

index.js

• Entry point
• Initializes environment and starts the subscriber

config/env.js

• Loads and validates environment variables

messaging/subscriber.js

• Subscribes to classified-file topic
• Parses incoming payload:

messaging/publisher.js

• Publishes to extracted-result topic:

services/blobService.js

• Downloads and uploads blobs from Azure Blob Storage

services/pdfParser.js

• Parses PDF using pdf-lib or pdf2json
• Extracts tables and text

services/documentAI.js

• Calls Azure Document Intelligence with custom model

services/openAIExtractor.js

• Sends prompt to Azure OpenAI
• Validates and refines LLM output

services/resultMerger.js

• Merges Document AI and LLM results
• Formats final JSON output

utils/logger.js

• Centralized logging with context and correlation IDs

🧪 Testing Strategy

• Unit Tests: Jest for all service modules
• Integration Tests: Mock Azure SDKs and Service Bus
• E2E Tests: Simulate full message flow in AKS dev cluster

🚀 Deployment

• CI/CD: GitHub Actions → Spacelift → AKS
• Helm: Used for Kubernetes manifests
• Spacelift:
  • Stack: aks-stack.yaml
  • Modules: azure-kubernetes/main.tf
  • Triggers: GitHub PRs and merges to main
  • Secrets: Managed via Spacelift + Azure Key Vault