Skip to navigation
Research and SpikesServices Evaluation

Extractor Service Evaluation

This document evaluates the current development progress of the Extractor Service as part of the service’s spike/evaluation user story: #729007

Endpoints Overview

The Extractor Service provides REST API endpoints for job tracking and status monitoring. OpenAPI documentation is available at /docs (Swagger UI).

Extractor Endpoints

MethodPathPurposeStatusDocumented
GET/v1/extractorSimple health check endpointImplementedYes
GET/v1/extractor/job/:jobIdGet extraction job details by job IDImplementedYes
GET/v1/extractor/job/file/:fileIdGet extraction job details by file IDImplementedYes

Implementation Files:

  • Routes: src/routes/v1/extractor.ts
  • Controller: src/controllers/extractorController.ts
  • Repository: src/repositories/jobRepository.ts, src/repositories/jobStatusRepository.ts
  • Schemas: src/routes/schemas.ts

OpenAPI Documentation Endpoints

MethodPathPurposeStatusDocumented
GET/docsSwagger UI documentationImplementedYes
GET/openapi.yamlOpenAPI YAML specImplementedYes
GET/healthHealth check endpointImplementedFramework-provided

Service Bus Subscriptions (Message Handlers)

SubscriptionTopicPurposeStatusDocumented
acceptedFileHandleraccepted-file-topicProcess accepted files and trigger extractionImplementedIn code comments

Implementation Files:

  • Handler: src/subscriptions/acceptedFileHandler.ts
  • Message Receiver: src/subscriptions/messageReceiver.ts
  • Topic Service: src/services/topicService.ts

Architecture Components

Core Services

ComponentPurposeStatusFiles
ExtractorServiceOrchestrates document extraction workflowImplementedsrc/services/extractorService.ts
ExtractionJobPollingWorkerPolls long-running extraction jobs for completionImplementedsrc/services/extractionJobPollingWorker.ts
BlobStorageServiceManages Azure Blob Storage operationsImplementedsrc/services/storage/blobStorage.ts
TopicServiceManages Service Bus topic subscriptionsImplementedsrc/services/topicService.ts
KeyvaultClientRetrieves secrets from Azure Key VaultImplementedsrc/services/keyvaultClient.ts

Extractors

ExtractorPurposeStatusFiles
OpenAIExtractorExtracts data using OpenAI APIImplementedsrc/services/extractors/openAIExtractor.ts
DocumentAIExtractorExtracts data using Document AI serviceImplementedsrc/services/extractors/documentAIExtractor.ts
ExtractorFactorySelects appropriate extractor based on URL patternImplementedsrc/services/extractors/extractorFactory.ts

API Clients

ClientPurposeStatusFiles
RegistrationServiceClientCommunicates with Registration ServiceImplementedsrc/apiClient/registration.apiClient.ts
ExtractionClientCommunicates with external extraction servicesImplementedsrc/apiClient/extraction.apiClient.ts

Repositories

RepositoryPurposeStatusFiles
JobRepositoryManages extraction job records with polling supportImplementedsrc/repositories/jobRepository.ts
JobStatusRepositoryManages job status historyImplementedsrc/repositories/jobStatusRepository.ts

Code

Potential enhancements and optimizations have been included in the tables below.

No.SeverityDescriptionImpactReferences
1MediumMissing POST endpoint for manual extraction job submission. Currently only supports Service Bus triggered extractions.Users cannot manually trigger extractions via API; completely dependent on message queue integrationMissing route in src/routes/v1/extractor.ts
2MediumExtractorFactory uses basic string matching for extractor selection. No proper extractor type registration or metadata-driven selection.Fragile extractor selection logic prone to false matches; difficult to extend with new extractorssrc/services/extractors/extractorFactory.ts lines 6-7
3MediumPolling worker configuration is hardcoded in startup. No dynamic configuration or management API for polling parameters.Cannot adjust polling behavior without redeployment; no runtime control over worker behaviorsrc/index.ts polling worker initialization
4MediumOnly the first extractor is processed per file (MVP limitation). Multi-extractor support is commented out but not implemented.Cannot run multiple extractors on the same file; limits flexibility for complex extraction scenariossrc/services/extractorService.ts line 25
5MediumMissing DELETE or PATCH endpoints for job management. No way to cancel running jobs or update job status manually.Cannot manually intervene when jobs get stuck; limited operational controlMissing routes
6LowBlob URL signed token expiration is hardcoded to BLOB_EXPIRY_IN_MINUTES constant. No per-request configuration.Inflexible for different extraction durations; may expire during long-running extractionssrc/subscriptions/acceptedFileHandler.ts line 47
7LowMissing pagination on job history endpoint. Could return large result sets for jobs with many status updates.Performance risk for long-running jobs; API response size issuessrc/repositories/jobStatusRepository.ts getHistoryByJobId
8LowGET /v1/extractor endpoint returns simple string “Extract GET request received”. No meaningful health check data.Limited diagnostic value; cannot determine actual service healthsrc/routes/v1/extractor.ts line 13
9LowNo API versioning strategy beyond URL path. Cannot deprecate endpoints gracefully.Limited API evolution capability; breaking changes are difficultRoutes structure

Possible Enhancements

No.PriorityDescription
1MediumImplement polling logic with exponential backoff instead of polling interval
2MediumAdd Application Insights integration for custom metrics and distributed tracing
3LowAdd pagination on job history endpoint to handle large result sets for jos with many status updates
4LowAdd polling worker-specific logging with Pod/Woker IDs for indicating which pod processes each polling attempts.

Documentation Review

Code Documentation

Document TypeStatusQualityNotes
README.md✅ GoodComprehensive setup and running instructionsWell-structured
OpenAPI Spec✅ GoodAll endpoints documented with schemasMinor naming inconsistency
Configuration Docs✅ GoodRequired and optional env vars documentedClear examples
No.DocumentPriorityDescription
1Architecture DiagramHighVisual representation of service components and data flow
2Sequence DiagramsHighShow extraction workflow, polling cycle, and message handling
3Extractor Development GuideLowHow to add new extractor types

Summary

Current State Assessment

The Extractor Service has achieved ~70% completion of core functionality. The foundation is solid with:

  • Working message-driven extraction workflow
  • Database persistence and job tracking
  • Multiple extractor implementations
  • Good unit test coverage
  • Clean architecture with dependency injection

However, several gaps may prevent production readiness:

  1. No manual API for job submission (only message-driven)
  2. MVP limitations (single extractor per file, no multi-processing)
  3. Proper extractor type factory pattern/system
  4. Metrics and observability instrumentation

Conclusion

The Extractor Service demonstrates solid engineering practices and has successfully implemented the core extraction workflow. The codebase is well-structured, testable, and follows TypeScript best practices. However, several production requirements remain incomplete, particularly operational control.