Skip to navigation
Research and SpikesJob Tracking

Technical Breakdown

1. Database Schema (Flyway Migrations - fluxx-sql-server repo)

A. Job Table

  • Create [extractor].[Job] table
  • Include:
    • JobID, FileId, ClientApplicationExtractorID
    • Status, StatusMessage, ExtractorResponse
    • PollAttemptCount, LastPollAttemptAt
    • IsActive, CreatedAt, UpdatedAt, CompletedAt
  • Add FK constraints to [file].[File] and [registration].[ClientApplicationExtractor]

B. JobStatus Table

  • Create [extractor].[JobStatus] table
  • Include:
    • JobStatusID, JobID, Status, Message
    • AttemptNumber, Timestamp
  • Add FK constraint to [extractor].[Job]

2. Registration Service

A. Endpoint

  • Create GET /registration/:clientAppId/extractors
    • Returns list of active extractors for a client application

B. Schema and Response DTO

  • Use ClientApplicationExtractorID and ExtractionUrl
  • Ensure IsActive = true filter is enforced

C. Swagger/OpenAPI

  • Document new route and schema in OpenAPI spec

3. Extractor Service - Infrastructure Setup

A. SQL Access

  • Add SQL connector for [extractor] schema
  • Enable read/write access to Job and JobStatus tables

B. Key Vault Integration

  • Configure secret retrieval:
    • Secret name = ClientApplicationExtractorID
    • Value = API key for ExtractionUrl

C. Blob Storage Access

  • Add write capability to blob container with HNS enabled
  • Use folder path:
/results/{ClientApplicationID}/{FileId}/{JobId}/result.json

4. Extractor Service - Job Lifecycle Logic

A. FileAccepted Handler

  • Consume TOPIC_ACCEPTED_FILE
  • Parse message and retrieve:
    • ClientApplicationID, FileID
  • Call Registration API to get extractors
  • Select first extractor

B. Job Initialization

  • Generate JobID
  • Retrieve API key from Key Vault
  • Create Job record in DB with Status = Queued
  • Publish JobQueued message to Service Bus

C. Submit Job

  • Retrieve file from blob storage
  • Send API request to ExtractionUrl with jobId and fileUrl
  • Update Job status to Started
  • Insert status record into JobStatus

5. Polling and Result Handling

A. Polling Logic

  • If polling is required:
    • Periodically call {pollingUrl}/{jobId}
    • Store status in JobStatus with attempt count and timestamp
    • Update main Job status fields and poll counters

B. Finalization

  • On job completion (success or failure):
    • Store result JSON in blob storage under expected path
    • Update Job record with final status and CompletedAt
    • Publish JobSucceeded or JobFailed message to Service Bus

6. Status Messaging (Extractor Service)

  • Publish messages to TOPIC_JOB_STATUS for:
    • JobQueued
    • JobStarted
    • JobSucceeded
    • JobFailed
  • Ensure message schema is aligned with Status service expectations

7. Internal API for Status Querying (Extractor Service)

  • Create endpoints:
    • GET /extractor/job/:jobId
    • GET /extractor/job/file/:fileId
  • Response includes:
    • Current status
    • Result blob path
    • History of status changes (from JobStatus)