📄 1099B-NonGovt-Extractor-Service Design

📘 Project Overview

Repository Name: poc-autoextract-platform-1099B-NonGovt-Extractor-Service

🧭 Overview

The 1099B-NonGovt-Extractor-Service is a cloud-native Node.js microservice deployed on Azure Kubernetes Service (AKS). It automates the extraction of structured data from 1099-B Non-Government tax documents using Azure Document Intelligence and OpenAI. The service is triggered by messages from Azure Service Bus and publishes results to a downstream topic once processing is complete. Follow the Framework V2 format for all components

🔄 Workflow Summary

1. Trigger: Listens to the classified-file topic on Azure Service Bus.

    {
      "eventType": "file.classified",
      "fileId": "uuid",
      "parentUrl": "https://...",
      "childUrl": "https://...",
      "clientApplicationId": "uuid",
      "classifications": ["1099NonGov"]
    }

2. Download: Retrieves the document from Azure Blob Storage.
3. Analyze: Uses Azure Document Intelligence and OpenAI to extract structured data.
4. Merge: Combines outputs from both services into a unified JSON format.
5. Upload: Stores the result back in Blob Storage.
6. Notify: Publishes a message to the extracted-result topic with a result URL.

    {
      "eventType": "result.extracted",
      "fileId": "uuid",
      "clientApplicationId": "uuid",
      "resultUrl": "https://.../results/<fileId>",
      "timestamp": "ISO8601"
    }

🧩 Component List

⚙️ Infrastructure

• Platform: Azure Kubernetes Service (AKS)
• Automation: https://spacelift.io/ (manages AKS, Azure Service Bus, Key Vault, and Helm deployments)
• Secrets: Azure Key Vault + CSI Driver
• Observability: Azure Monitor, Application Insights

📁 File Structure

1099B-NonGovt-Extractor-Service/ ├── src/ │ ├── index.js │ ├── config/ │ │ └── env.js │ ├── messaging/ │ │ ├── subscriber.js │ │ └── publisher.js │ ├── services/ │ │ ├── blobService.js │ │ ├── pdfParser.js │ │ ├── documentAI.js │ │ ├── openAIExtractor.js │ │ └── resultMerger.js │ └── utils/ │ └── logger.js ├── Dockerfile ├── helm/ │ └── chart.yaml ├── spacelift/ │ ├── stacks/ │ │ └── aks-stack.yaml │ └── modules/ │ └── azure-kubernetes/ │ └── main.tf ├── package.json └── README.md

🧩 Module Responsibilities

index.js

• Entry point
• Initializes environment and starts the subscriber

config/env.js

• Loads and validates environment variables

messaging/subscriber.js

• Subscribes to classified-file topic
• Parses incoming payload:

messaging/publisher.js

• Publishes to extracted-result topic:

services/blobService.js

• Downloads and uploads blobs from Azure Blob Storage

services/pdfParser.js

• Parses PDF using pdf-lib or pdf2json
• Extracts tables and text

services/documentAI.js

• Calls Azure Document Intelligence with custom model

services/openAIExtractor.js

• Sends prompt to Azure OpenAI
• Validates and refines LLM output

services/resultMerger.js

• Merges Document AI and LLM results
• Formats final JSON output

utils/logger.js

• Centralized logging with context and correlation IDs

🧪 Testing Strategy

• Unit Tests: Jest for all service modules
• Integration Tests: Mock Azure SDKs and Service Bus
• E2E Tests: Simulate full message flow in AKS dev cluster

🚀 Deployment

• CI/CD: GitHub Actions → Spacelift → AKS
• Helm: Used for Kubernetes manifests
• Spacelift:
  • Stack: aks-stack.yaml
  • Modules: azure-kubernetes/main.tf
  • Triggers: GitHub PRs and merges to main
  • Secrets: Managed via Spacelift + Azure Key Vault