Topics,-Messages,-Publishers and Subscribers

Overview

Our autoextract platform is designed to make it easy for our R&D teams to create and use new models for extracting important data from raw documents like PDFs and PNGs. The main goal is to simplify the process of trying out and using new models for different types of data extraction. The beauty of this platform is its modular setup. You can plug in new models without having to redo all the work needed to provide supporting systems such as file-intake to each new model. Our platform provides a file intake microservice, classifiers, an extractor-API, and post-processors for the extracted data. We aim put each new model in contact with documents for extraction and to capture the extracted data in a shared but flexible way to reduce duplication of efforts when new models are created.

To ensure each step completes at its own pace while allowing the system to scale each step independently, our platform leverages publish/subscribe topics and messages. These tools are used to share event details and status updates between our services as files move through the system. Using a publish/subscribe model, different components of the platform can seamlessly exchange information, keeping the file intake microservice, classifiers, extractor-API, and post-processors coordinated but loosely coupled. This approach boosts the system’s modularity and offers real-time visibility into the processing workflow, allowing for timely interventions and optimizations.

Topics Overview

Topic	Purpose	Publishers	Subscriptions	Consumers
validated-file	This topic contains events for files that passed intake validation, so downstream processing can continue with trusted inputs.	fluxx-file-intake	copier-validated-file, status-validated-file	fluxx-file-intake, fluxx-statuscheck
accepted-file	This topic contains events for files accepted into processing context, so extraction, classification, and status workflows can begin.	fluxx-file-intake	extractor-accepted-file, status-accepted-file, classifier-accepted-file	fluxx-extractor, fluxx-statuscheck, fluxx-classifier
invalidated-file	This topic contains events for files or batches marked for reprocessing, so failed or stale processing attempts can be retried.	web-api-documents-ai	extractor-invalidated-file, status-invalidated-file (planned)	fluxx-extractor, fluxx-statuscheck
updated-registration	This topic contains registration change events for Client and ClientApplication records, so dependent services can react to configuration updates.	fluxx-registration
problem	This topic contains standardized problem details for validation and processing failures, so errors can be tracked and handled consistently across services.	any service	status-problem-file	fluxx-statuscheck
job-status	This topic contains extraction job lifecycle updates, so progress and outcomes remain visible for operations and audit.	fluxx-extractor
extracted-result	This topic contains final extraction outcomes and result references, so downstream consumers can use extracted outputs.	fluxx-extractor	status-extracted-file	fluxx-statuscheck
classified-file	This topic contains file classification outcomes, so downstream processes can act on classification decisions.	fluxx-classifier	document-splitter-classified-file, status-classified-file	fluxx-document-splitter, fluxx-statuscheck
split-file	This topic contains one event per split output file, so each resulting file can be processed independently.	fluxx-document-splitter
classifier-job-status	This topic contains classification job lifecycle updates, so progress and outcomes remain visible for operations and audit.	fluxx-classifier	status-split-file-result	fluxx-statuscheck
malware-scan	This topic contains malware scan outcomes for uploaded files, so clean files continue processing and malicious files are blocked.	Event Grid / SB	malware-scan-events	fluxx-file-intake
split-file-result	This topic contains successful document-splitting result events, so downstream consumers can continue processing completed outputs.	fluxx-document-splitter
outbox			postal-outbox	fluxx-postal
outbox-retries			postal-outbox-retries	fluxx-postal
to-retry	This topic contains retry scheduling payloads for outbound delivery, so failed sends can be retried with backoff behavior.	fluxx-postal	postal-to-retry	fluxx-postal
to-send	This topic contains outbound payloads ready for first delivery attempts, so initial sends can be executed.	TBD	postal-to-send	fluxx-postal

Topic Message Contracts

validated-file

Message Format

{
	"clientApplicationId": "string",
	"fileId": "string",
	"timestamp": "string"
}

Message Body (Example)

{
	"clientApplicationId": "123e4567-e89b-12d3-a456-426614174000",
	"fileId": "43434343-4343-4343-4343-434343434343",
	"timestamp": "2026-05-04T14:44:53.000Z"
}

accepted-file

Message Format

{
	"applicationId": "string",
	"clientApplicationId": "string",
	"fileHashValue": "string",
	"fileId": "string",
	"fileLocationUrl": "string",
	"internalUrl": "string",
	"timestamp": "string"
}

Message Body (Example)

{
	"applicationId": "333e4567-e89b-12d3-a456-426614174111",
	"clientApplicationId": "123e4567-e89b-12d3-a456-426614174000",
	"fileHashValue": "sha256:8c6976e5b5410415bde908bd4dee15dfb16c9f6f1b8f7d3f6a5f5a8d0f1e2c3b",
	"fileId": "43434343-4343-4343-4343-434343434343",
	"fileLocationUrl": "http://azure.blobs.com/internal-999-999-99999-999/files/43434343-4343-4343-4343-434343434343.pdf",
	"internalUrl": "http://azure.blobs.com/internal-999-999-99999-999/43434343-4343-4343-4343-34343434343_435683745876923457.pdf",
	"timestamp": "2026-05-04T14:44:53.000Z"
}

invalidated-file

Message Format

{
	"applicationId": "string",
	"clientApplicationId": "string",
	"fileHashValue": "string",
	"fileLocationUrl": "string",
	"fileId": "string",
	"internalUrl": "string",
	"timestamp": "Date"
}

Message Body (Example)

{
	"applicationId": "333e4567-e89b-12d3-a456-426614174111",
	"clientApplicationId": "123e4567-e89b-12d3-a456-426614174000",
	"fileHashValue": "sha256:8c6976e5b5410415bde908bd4dee15dfb16c9f6f1b8f7d3f6a5f5a8d0f1e2c3b",
	"fileLocationUrl": "http://azure.blobs.com/internal-999-999-99999-999/files/43434343-4343-4343-4343-434343434343.pdf",
	"fileId": "43434343-4343-4343-4343-434343434343",
	"internalUrl": "http://azure.blobs.com/internal-999-999-99999-999/43434343-4343-4343-4343-34343434343_435683745876923457.pdf",
	"timestamp": "2026-05-04T14:44:53.000Z"
}

updated-registration

Message Format

{
	"changedObject": "string",
	"changeType": "ChangeType",
	"timestamp": "string",
	"clientId": "string",
	"changedFields": "Record<string, unknown> | null"
}

Message Body (Example)

{
	"changedObject": "Client",
	"changeType": "update",
	"timestamp": "2026-05-04T14:44:53.000Z",
	"clientId": "43434343-4343-4343-4343-434343434343",
	"changedFields": {
		"name": "Test Client",
		"isActive": true
	}
}

problem

Message Format

{
	"problem": {
		"detail": "string",
		"instance": "string",
		"status": "number",
		"title": "string",
		"type": "string"
	},
	"timestamp": "string"
}

Message Body (Example)

{
	"problem": {
		"detail": "Validation failed for the provided file metadata.",
		"instance": "/platform/autoextract/problems/123454386874601802",
		"status": 400,
		"title": "Validation Error",
		"type": "https://example.com/probs/validation-error"
	},
	"timestamp": "2026-05-04T14:44:53.000Z"
}

job-status

Message Format

{
	"clientApplicationExtractorId": "string",
	"errorMessage": "string",
	"extractedEntityId": "number",
	"fileId": "string",
	"jobId": "string",
	"message": "string",
	"pollAttemptCount": "string",
	"resultSummary": "string",
	"status": "string",
	"timestamp": "string"
}

Message Body (Example)

{
	"clientApplicationExtractorId": "5CCEC11F-441B-4ADB-9888-97E7521A95A9",
	"errorMessage": "",
	"extractedEntityId": 123456,
	"fileId": "EFAA7BBD-4953-42B9-BA2E-DE0E20360BB6",
	"jobId": "A486B861-D1DC-4A99-BC43-2EA4B1A0988D",
	"message": "Job completed successfully.",
	"pollAttemptCount": "3",
	"resultSummary": "Extraction produced 24 mapped fields.",
	"status": "Succeeded",
	"timestamp": "2026-05-04T14:44:53.000Z"
}

extracted-result

Message Format

{
	"extractedEntityId": "number?",
	"extractionStatus": "ExtractorJobStatus",
	"jobId": "string",
	"resultUrl": "string?",
	"timestamp": "string"
}

Message Body (Example)

{
	"extractedEntityId": 123456,
	"extractionStatus": "Succeeded",
	"jobId": "9f2c1e5b-4f2d-4b1a-9a6c-8b2e1f3a4c5d",
	"resultUrl": "https://example.com/results/9f2c1e5b-4f2d-4b1a-9a6c-8b2e1f3a4c5d.json",
	"timestamp": "2026-05-04T14:44:53.000Z"
}

classified-file

Message Format

{
	"clientApplicationId": "string",
	"classifiedOutputUrl": "string",
	"fileId": "string",
	"internalUrl": "string",
	"timestamp": "Date",
        "applicationId" : "string",
}

Message Body (Example)

{
	"clientApplicationId": "123e4567-e89b-12d3-a456-426614174000",
	"classifiedOutputUrl": "http://cla.com/platform/autoextract/classifier/results/43434343-4343-4343-4343-434343435555.json",
	"fileId": "43434343-4343-4343-4343-434343435555",
	"internalUrl": "http://azure.blobs.com/internal-999-999-99999-999/43434343-4343-4343-4343-434343435555.pdf",
	"timestamp": "2026-05-04T14:44:53.000Z",
        "applicationId" : "123e4567-e89b-12d3-a456-434343435555",
}

split-file

Message Format

{
	"clientApplicationId": "string",
	"internalUrl": "string",
	"fileLocationUrl": "string",
	"timestamp": "Date"
}

Message Body (Example)

{
	"clientApplicationId": "123e4567-e89b-12d3-a456-426614174000",
	"internalUrl": "http://azure.blobs.com/internal-999-999-99999-999/43434343-4343-4343-4343-434343435555.pdf",
	"fileLocationUrl": "http://azure.blobs.com/internal-999-999-99999-999/files/43434343-4343-4343-4343-434343435555.pdf",
	"timestamp": "2026-05-04T14:44:53.000Z"
}

classifier-job-status

Message Format

{
	"status": "Started | Processing | Failed | Succeeded",
	"jobId": "string",
	"fileId": "string",
	"clientApplicationClassifierId": "string",
	"timestamp": "Date",
	"classificationId": "string?",
	"errorMessage": "string?",
	"resultSummary": "string?",
	"pollAttemptCount": "number?",
	"message": "string?"
}

Message Body (Example)

{
	"status": "Succeeded",
	"jobId": "9f2c1e5b-4f2d-4b1a-9a6c-8b2e1f3a4c5d",
	"fileId": "43434343-4343-4343-4343-434343435555",
	"clientApplicationClassifierId": "5CCEC11F-441B-4ADB-9888-97E7521A95A9",
	"resultSummary": "Classification completed with confidence score 0.98.",
	"timestamp": "2026-05-04T14:44:53.000Z"
}

malware-scan

Message Format

{
	"id": "string",
	"subject": "string",
	"data": {
		"correlationId": "string",
		"blobUri": "string",
		"scannedObjectUri": "string",
		"eTag": "string",
		"scanFinishedTimeUtc": "string",
		"scanResultType": "string",
		"scanResultDetails": "string | null"
	},
	"eventType": "string",
	"dataVersion": "string",
	"metadataVersion": "string",
	"eventTime": "string",
	"topic": "string"
}

Message Body (Example)

{
	"id": "0f4d1b6f-2c7a-4c28-8f95-4f8b6d90d5ef",
	"subject": "/blobServices/default/containers/uploads/blobs/43434343-4343-4343-4343-434343435555.pdf",
	"data": {
		"correlationId": "8b1e3f8f-2c2a-4b47-a5cc-8f5aa35bd4e4",
		"blobUri": "https://storage.example.com/uploads/43434343-4343-4343-4343-434343435555.pdf",
		"scannedObjectUri": "https://storage.example.com/uploads/43434343-4343-4343-4343-434343435555.pdf",
		"eTag": "0x8DB3A8C5E51A12F",
		"scanFinishedTimeUtc": "2026-05-04T14:44:53.000Z",
		"scanResultType": "No threats found",
		"scanResultDetails": null
	},
	"eventType": "Microsoft.Security.MalwareScanningResult",
	"dataVersion": "1.0",
	"metadataVersion": "1",
	"eventTime": "2026-05-04T14:44:53.000Z",
	"topic": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-example/providers/Microsoft.Storage/storageAccounts/storageexample"
}

split-file-result

Message Format

{
	"clientApplicationId": "string",
	"fileId": "string",
	"parentFileBlobUrl": "string",
	"classificationUrl": "string",
	"timestamp": "Date",
	"totalSplitFiles": 1,
	"splitFiles": [
		{
			"fileName": "string",
			"formType": "string",
			"formIndex": "number",
			"pageStart": "number",
			"pageEnd": "number",
			"blobUrl": "string"
		}
	]
}

outbox

Message Format

{}

Message Body (Example)

{}

outbox-retries

Message Format

{}

Message Body (Example)

{}

to-retry

Message Format

{
	"clientApplicationId": "string",
	"body": {},
	"retryScheduling": {
		"timestamp": "Date",
		"deltaMillis": "number",
		"scalingDirection": "rampup | backoff",
		"previousTimestamp": "Date"
	}
}

Message Body (Example)

{
	"clientApplicationId": "5CCEC11F-441B-4ADB-9888-97E7521A95A9",
	"body": {},
	"retryScheduling": {
		"timestamp": "2026-05-04T14:44:53.000Z",
		"deltaMillis": 3000,
		"scalingDirection": "rampup",
		"previousTimestamp": "2026-05-04T14:44:50.000Z"
	}
}

to-send

Message Format

{
	"clientApplicationId": "string",
	"body": {},
	"retryScheduling": {
		"timestamp": "Date",
		"deltaMillis": "number",
		"scalingDirection": "rampup | backoff",
		"previousTimestamp": "Date"
	}
}

Message Body (Example)

{
	"clientApplicationId": "5CCEC11F-441B-4ADB-9888-97E7521A95A9",
	"body": {},
	"retryScheduling": {
		"timestamp": "2026-05-04T14:44:53.000Z",
		"deltaMillis": 3000,
		"scalingDirection": "rampup",
		"previousTimestamp": "2026-05-04T14:44:50.000Z"
	}
}

Message Body (Example)

{
	"clientApplicationId": "123e4567-e89b-12d3-a456-426614174000",
	"fileId": "43434343-4343-4343-4343-434343435555",
	"parentFileBlobUrl": "http://azure.blobs.com/internal-999-999-99999-999/43434343-4343-4343-4343-434343435555.pdf",
	"classificationUrl": "http://cla.com/platform/autoextract/classifier/results/43434343-4343-4343-4343-434343435555.json",
	"timestamp": "2026-05-04T14:44:53.000Z",
	"totalSplitFiles": 1,
	"splitFiles": [
		{
			"fileName": "20260504_144453_43434343-4343-4343-4343-434343435555_split_1.pdf",
			"formType": "A19",
			"formIndex": 0,
			"pageStart": 1,
			"pageEnd": 3,
			"blobUrl": "http://azure.blobs.com/internal-999-999-99999-999/split/20260504_144453_43434343-4343-4343-4343-434343435555_split_1.pdf"
		}
	]
}

Subscriptions Catalog

Subscription-first view for operations, ownership, and troubleshooting.

Subscriptions Matrix

Jump to Subscription Details.

Subscription	Source Topic	Consumer Service	Purpose	Contract
copier-validated-file	validated-file	fluxx-file-intake	Consume validated files for copier workflow	validated-file
status-validated-file	validated-file	fluxx-statuscheck	Track validated file status	validated-file
extractor-accepted-file	accepted-file	fluxx-extractor	Start extraction from accepted inputs	accepted-file
status-accepted-file	accepted-file	fluxx-statuscheck	Track accepted file status	accepted-file
classifier-accepted-file	accepted-file	fluxx-classifier	Start classification from accepted inputs	accepted-file
extractor-invalidated-file	invalidated-file	fluxx-extractor	Reprocess invalidated files	invalidated-file
status-invalidated-file (planned)	invalidated-file	fluxx-statuscheck	Track invalidated file status	invalidated-file
status-problem-file	problem	fluxx-statuscheck	Consume standardized problem events	problem
document-splitter-classified-file	classified-file	fluxx-document-splitter	Split classified source files	classified-file
extraction-split-file	split-file	fluxx-extractor	Extract data from split output files	split-file
malware-scan-events	malware-scan	fluxx-file-intake	Consume malware scanning outcomes	malware-scan
postal-outbox	outbox	fluxx-postal	Process outbound ready items	outbox
postal-outbox-retries	outbox-retries	fluxx-postal	Process outbound retry items	outbox-retries
postal-to-retry	to-retry	fluxx-postal	Schedule and execute retries	to-retry
postal-to-send	to-send	fluxx-postal	Execute initial outbound sends	to-send

Subscription Details

Back to Subscriptions Matrix.

copier-validated-file

Source Topic: validated-file
Consumer: fluxx-file-intake
Trigger Condition: New validated-file event.
Expected Message Fields: See validated-file.

status-validated-file

Source Topic: validated-file
Consumer: fluxx-statuscheck
Trigger Condition: New validated-file event.
Expected Message Fields: See validated-file.

extractor-accepted-file

Source Topic: accepted-file
Consumer: fluxx-extractor
Trigger Condition: New accepted-file event.
Expected Message Fields: See accepted-file.

status-accepted-file

Source Topic: accepted-file
Consumer: fluxx-statuscheck
Trigger Condition: New accepted-file event.
Expected Message Fields: See accepted-file.

classifier-accepted-file

Source Topic: accepted-file
Consumer: fluxx-classifier
Trigger Condition: New accepted-file event.
Expected Message Fields: See accepted-file.

extractor-invalidated-file

Source Topic: invalidated-file
Consumer: fluxx-extractor
Trigger Condition: New invalidated-file event.
Expected Message Fields: See invalidated-file.

status-invalidated-file (planned)

Source Topic: invalidated-file
Consumer: fluxx-statuscheck
Trigger Condition: New invalidated-file event.
Expected Message Fields: See invalidated-file.

status-problem-file

Source Topic: problem
Consumer: fluxx-statuscheck
Trigger Condition: New problem event.
Expected Message Fields: See problem.

document-splitter-classified-file

Source Topic: classified-file
Consumer: fluxx-document-splitter
Trigger Condition: New classified-file event.
Expected Message Fields: See classified-file.

extraction-split-file

Source Topic: split-file
Consumer: fluxx-extractor
Trigger Condition: New split-file event.
Expected Message Fields: See split-file.

malware-scan-events

Source Topic: malware-scan
Consumer: fluxx-file-intake
Trigger Condition: New malware-scan event.
Expected Message Fields: See malware-scan.

postal-outbox

Source Topic: outbox
Consumer: fluxx-postal
Trigger Condition: New outbox event.
Expected Message Fields: See outbox.

postal-outbox-retries

Source Topic: outbox-retries
Consumer: fluxx-postal
Trigger Condition: New outbox-retries event.
Expected Message Fields: See outbox-retries.

postal-to-retry

Source Topic: to-retry
Consumer: fluxx-postal
Trigger Condition: New to-retry event.
Expected Message Fields: See to-retry.

postal-to-send

Source Topic: to-send
Consumer: fluxx-postal
Trigger Condition: New to-send event.
Expected Message Fields: See to-send.

Overview

Quick Navigation

Topics Overview

Topic Message Contracts

validated-file

accepted-file

invalidated-file

updated-registration

problem

job-status

extracted-result

classified-file

split-file

classifier-job-status

malware-scan

split-file-result

outbox

outbox-retries

to-retry

to-send

Subscriptions Catalog

Subscriptions Matrix

Subscription Details

copier-validated-file

status-validated-file

extractor-accepted-file

status-accepted-file

classifier-accepted-file

extractor-invalidated-file

status-invalidated-file (planned)

status-problem-file

document-splitter-classified-file

extraction-split-file

malware-scan-events

postal-outbox

postal-outbox-retries

postal-to-retry

postal-to-send