> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://cla-demo.postmansolutions.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://cla-demo.postmansolutions.com/_mcp/server.

# Consolidated 1099 Extractor Service

# 📄 1099B-NonGovt-Extractor-Service Design

## 📘 Project Overview

Repository Name: `poc-autoextract-platform-consolidated-1099-Extractor-Service`

## 🧭 Overview

The \`Consolidated-1099-Extractor-Service is a cloud-native Node.js microservice deployed on Azure Kubernetes Service (AKS). It automates the extraction of structured data from Consolidated 199 tax documents using Azure Document Intelligence and OpenAI. The service is triggered by messages from Azure Service Bus and publishes results to a downstream topic once processing is complete. Follow the Framework V2 format for all components

### 🔄 Workflow Summary

1. **Trigger**: Listens to the `classified-file` topic on Azure Service Bus.

```
    {
      "eventType": "file.classified",
      "fileId": "uuid",
      "parentUrl": "https://...",
      "childUrl": "https://...",
      "clientApplicationId": "uuid",
      "classifications": ["1099Gov"]
    }
```

2. **Download**: Retrieves the document from Azure Blob Storage.
3. **Analyze**: Uses Azure Document Intelligence and OpenAI to extract structured data.
4. **Merge**: Combines outputs from both services into a unified JSON format.
5. **Upload**: Stores the result back in Blob Storage.
6. **Notify**: Publishes a message to the `extracted-result` topic with a result URL.

```
    {
      "eventType": "result.extracted",
      "fileId": "uuid",
      "clientApplicationId": "uuid",
      "resultUrl": "https://.../results/<fileId>",
      "timestamp": "ISO8601"
    }
```

---

## 🧩 Component List

| Component                              | Description                                                                                            |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| **Azure Service Bus Subscriber**       | Listens to the `classified-file` topic and initiates the extraction pipeline.                          |
| **Blob Downloader**                    | Downloads the original document from Azure Blob Storage.                                               |
| **PDF Parser**                         | Parses the PDF using a Node.js library (e.g., `pdf-lib` or `pdf2json`) to extract raw text and tables. |
| **Azure Document Intelligence Client** | Sends the document to Azure Form Recognizer for structured analysis.                                   |
| **OpenAI Client**                      | Sends extracted text to Azure OpenAI for LLM-based interpretation and validation.                      |
| **Result Merger**                      | Merges outputs from Document Intelligence and OpenAI into a single JSON structure.                     |
| **Blob Uploader**                      | Uploads the final result JSON to a designated Blob Storage container.                                  |
| **Azure Service Bus Publisher**        | Sends a message to the `extracted-result` topic with metadata and result URL.                          |
| **AKS Deployment**                     | Hosts the Node.js service in a scalable containerized environment.                                     |
| **Spacelift Automation**               | Manages infrastructure provisioning and deployment pipelines for AKS, Service Bus, and Key Vault.      |
| **Azure Key Vault + CSI Driver**       | Securely injects secrets (API keys, connection strings) into the AKS pods.                             |

---

## ⚙️ Infrastructure

* **Platform**: Azure Kubernetes Service (AKS)
* **Automation**: [https://spacelift.io/](https://spacelift.io/) (manages AKS, Azure Service Bus, Key Vault, and Helm deployments)
* **Secrets**: Azure Key Vault + CSI Driver
* **Observability**: Azure Monitor, Application Insights

---

## 📁 File Structure

1099B-NonGovt-Extractor-Service/
├── src/
│   ├── index.js
│   ├── config/
│   │   └── env.js
│   ├── messaging/
│   │   ├── subscriber.js
│   │   └── publisher.js
│   ├── services/
│   │   ├── blobService.js
│   │   ├── pdfParser.js
│   │   ├── documentAI.js
│   │   ├── openAIExtractor.js
│   │   └── resultMerger.js
│   └── utils/
│       └── logger.js
├── Dockerfile
├── helm/
│   └── chart.yaml
├── spacelift/
│   ├── stacks/
│   │   └── aks-stack.yaml
│   └── modules/
│       └── azure-kubernetes/
│           └── main.tf
├── package.json
└── README.md

---

## 🧩 Module Responsibilities

### `index.js`

* Entry point
* Initializes environment and starts the subscriber

### `config/env.js`

* Loads and validates environment variables

### `messaging/subscriber.js`

* Subscribes to `classified-file` topic
* Parses incoming payload:

### `messaging/publisher.js`

* Publishes to `extracted-result` topic:

### `services/blobService.js`

* Downloads and uploads blobs from Azure Blob Storage

### `services/pdfParser.js`

* Parses PDF using `pdf-lib` or `pdf2json`
* Extracts tables and text

### `services/documentAI.js`

* Calls Azure Document Intelligence with custom model

### `services/openAIExtractor.js`

* Sends prompt to Azure OpenAI
* Validates and refines LLM output

### `services/resultMerger.js`

* Merges Document AI and LLM results
* Formats final JSON output

### `utils/logger.js`

* Centralized logging with context and correlation IDs

---

## 🧪 Testing Strategy

* **Unit Tests**: Jest for all service modules
* **Integration Tests**: Mock Azure SDKs and Service Bus
* **E2E Tests**: Simulate full message flow in AKS dev cluster

---

## 🚀 Deployment

* **CI/CD**: GitHub Actions → Spacelift → AKS
* **Helm**: Used for Kubernetes manifests
* **Spacelift**:
  * Stack: `aks-stack.yaml`
  * Modules: `azure-kubernetes/main.tf`
  * Triggers: GitHub PRs and merges to `main`
  * Secrets: Managed via Spacelift + Azure Key Vault