> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://cla-demo.postmansolutions.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://cla-demo.postmansolutions.com/_mcp/server. # Extractor Service Evaluation This document evaluates the current development progress of the Extractor Service as part of the service's spike/evaluation user story: [#729007](https://https://dev.azure.com/CLADevOps/Development/_workitems/edit/729007) --- ## Endpoints Overview The Extractor Service provides REST API endpoints for job tracking and status monitoring. OpenAPI documentation is available at `/docs` (Swagger UI). #### **Extractor Endpoints** | Method | Path | Purpose | Status | Documented | | ------ | -------------------------------- | ------------------------------------- | ----------- | ---------- | | GET | `/v1/extractor` | Simple health check endpoint | Implemented | Yes | | GET | `/v1/extractor/job/:jobId` | Get extraction job details by job ID | Implemented | Yes | | GET | `/v1/extractor/job/file/:fileId` | Get extraction job details by file ID | Implemented | Yes | **Implementation Files:** * Routes: `src/routes/v1/extractor.ts` * Controller: `src/controllers/extractorController.ts` * Repository: `src/repositories/jobRepository.ts`, `src/repositories/jobStatusRepository.ts` * Schemas: `src/routes/schemas.ts` #### **OpenAPI Documentation Endpoints** | Method | Path | Purpose | Status | Documented | | ------ | --------------- | ------------------------ | ----------- | ------------------ | | GET | `/docs` | Swagger UI documentation | Implemented | Yes | | GET | `/openapi.yaml` | OpenAPI YAML spec | Implemented | Yes | | GET | `/health` | Health check endpoint | Implemented | Framework-provided | #### **Service Bus Subscriptions (Message Handlers)** | Subscription | Topic | Purpose | Status | Documented | | --------------------- | --------------------- | --------------------------------------------- | ----------- | ---------------- | | `acceptedFileHandler` | `accepted-file-topic` | Process accepted files and trigger extraction | Implemented | In code comments | **Implementation Files:** * Handler: `src/subscriptions/acceptedFileHandler.ts` * Message Receiver: `src/subscriptions/messageReceiver.ts` * Topic Service: `src/services/topicService.ts` --- ## Architecture Components ### Core Services | Component | Purpose | Status | Files | | ------------------------------ | ------------------------------------------------- | ----------- | -------------------------------------------- | | **ExtractorService** | Orchestrates document extraction workflow | Implemented | `src/services/extractorService.ts` | | **ExtractionJobPollingWorker** | Polls long-running extraction jobs for completion | Implemented | `src/services/extractionJobPollingWorker.ts` | | **BlobStorageService** | Manages Azure Blob Storage operations | Implemented | `src/services/storage/blobStorage.ts` | | **TopicService** | Manages Service Bus topic subscriptions | Implemented | `src/services/topicService.ts` | | **KeyvaultClient** | Retrieves secrets from Azure Key Vault | Implemented | `src/services/keyvaultClient.ts` | ### Extractors | Extractor | Purpose | Status | Files | | ----------------------- | -------------------------------------------------- | ----------- | ------------------------------------------------ | | **OpenAIExtractor** | Extracts data using OpenAI API | Implemented | `src/services/extractors/openAIExtractor.ts` | | **DocumentAIExtractor** | Extracts data using Document AI service | Implemented | `src/services/extractors/documentAIExtractor.ts` | | **ExtractorFactory** | Selects appropriate extractor based on URL pattern | Implemented | `src/services/extractors/extractorFactory.ts` | ### API Clients | Client | Purpose | Status | Files | | ----------------------------- | ---------------------------------------------- | ----------- | ----------------------------------------- | | **RegistrationServiceClient** | Communicates with Registration Service | Implemented | `src/apiClient/registration.apiClient.ts` | | **ExtractionClient** | Communicates with external extraction services | Implemented | `src/apiClient/extraction.apiClient.ts` | ### Repositories | Repository | Purpose | Status | Files | | ----------------------- | --------------------------------------------------- | ----------- | ----------------------------------------- | | **JobRepository** | Manages extraction job records with polling support | Implemented | `src/repositories/jobRepository.ts` | | **JobStatusRepository** | Manages job status history | Implemented | `src/repositories/jobStatusRepository.ts` | --- ## Code Potential enhancements and optimizations have been included in the tables below. ## Recommended Changes | No. | Severity | Description | Impact | References | | --- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | | 1 | **Medium** | Missing POST endpoint for manual extraction job submission. Currently only supports Service Bus triggered extractions. | Users cannot manually trigger extractions via API; completely dependent on message queue integration | Missing route in `src/routes/v1/extractor.ts` | | 2 | **Medium** | ExtractorFactory uses basic string matching for extractor selection. No proper extractor type registration or metadata-driven selection. | Fragile extractor selection logic prone to false matches; difficult to extend with new extractors | `src/services/extractors/extractorFactory.ts` lines 6-7 | | 3 | **Medium** | Polling worker configuration is hardcoded in startup. No dynamic configuration or management API for polling parameters. | Cannot adjust polling behavior without redeployment; no runtime control over worker behavior | `src/index.ts` polling worker initialization | | 4 | **Medium** | Only the first extractor is processed per file (MVP limitation). Multi-extractor support is commented out but not implemented. | Cannot run multiple extractors on the same file; limits flexibility for complex extraction scenarios | `src/services/extractorService.ts` line 25 | | 5 | **Medium** | Missing DELETE or PATCH endpoints for job management. No way to cancel running jobs or update job status manually. | Cannot manually intervene when jobs get stuck; limited operational control | Missing routes | | 6 | **Low** | Blob URL signed token expiration is hardcoded to BLOB\_EXPIRY\_IN\_MINUTES constant. No per-request configuration. | Inflexible for different extraction durations; may expire during long-running extractions | `src/subscriptions/acceptedFileHandler.ts` line 47 | | 7 | **Low** | Missing pagination on job history endpoint. Could return large result sets for jobs with many status updates. | Performance risk for long-running jobs; API response size issues | `src/repositories/jobStatusRepository.ts` getHistoryByJobId | | 8 | **Low** | GET /v1/extractor endpoint returns simple string "Extract GET request received". No meaningful health check data. | Limited diagnostic value; cannot determine actual service health | `src/routes/v1/extractor.ts` line 13 | | 9 | **Low** | No API versioning strategy beyond URL path. Cannot deprecate endpoints gracefully. | Limited API evolution capability; breaking changes are difficult | Routes structure | --- ## Possible Enhancements | No. | Priority | Description | | --- | -------- | ---------------------------------------------------------------------------------------------------------------- | | 1 | Medium | Implement polling logic with exponential backoff instead of polling interval | | 2 | Medium | Add Application Insights integration for custom metrics and distributed tracing | | 3 | Low | Add pagination on job history endpoint to handle large result sets for jos with many status updates | | 4 | Low | Add polling worker-specific logging with Pod/Woker IDs for indicating which pod processes each polling attempts. | --- ## Documentation Review ### Code Documentation | Document Type | Status | Quality | Notes | | ------------------ | ------ | -------------------------------------------- | -------------------------- | | README.md | ✅ Good | Comprehensive setup and running instructions | Well-structured | | OpenAPI Spec | ✅ Good | All endpoints documented with schemas | Minor naming inconsistency | | Configuration Docs | ✅ Good | Required and optional env vars documented | Clear examples | ### Recommended Documentation | No. | Document | Priority | Description | | --- | --------------------------- | -------- | ------------------------------------------------------------- | | 1 | Architecture Diagram | High | Visual representation of service components and data flow | | 2 | Sequence Diagrams | High | Show extraction workflow, polling cycle, and message handling | | 3 | Extractor Development Guide | Low | How to add new extractor types | --- ## Summary ### Current State Assessment The Extractor Service has achieved **\~70% completion** of core functionality. The foundation is solid with: * Working message-driven extraction workflow * Database persistence and job tracking * Multiple extractor implementations * Good unit test coverage * Clean architecture with dependency injection However, several **gaps** may prevent production readiness: 1. No manual API for job submission (only message-driven) 2. MVP limitations (single extractor per file, no multi-processing) 3. Proper extractor type factory pattern/system 4. Metrics and observability instrumentation ### Conclusion The Extractor Service demonstrates solid engineering practices and has successfully implemented the core extraction workflow. The codebase is well-structured, testable, and follows TypeScript best practices. However, several production requirements remain incomplete, particularly operational control. ---