File Intake Service

Overview

The File Intake Microservice is the entry point for all files into the AutoExtract Platform. It accepts files via a REST API upload endpoint, stores them in Azure Blob Storage, and records their metadata in SQL Server.

Once a file is stored, Microsoft Defender for Cloud automatically scans the blob for malware and publishes the result to an Azure Service Bus topic. The service subscribes to that topic and routes the file accordingly:

• Clean files - a validated-file event is published, which triggers the file to be copied into the internal accepted-file storage and an accepted-file event is emitted for downstream services.
• Malicious or unscannable files - the blob is soft-deleted (or left as-is if unreachable) and a problem event is published to inform the rest of the system that the file was rejected and why.

The service also subscribes to the validated-file topic directly, which is responsible for performing the copy from the uploads area into the accepted-file container and publishing the final accepted-file event.

---

Flow Chart

---

Detailed Workflow

1. File Upload (REST API)

• Clients submit files via POST /v1/upload/:applicationId as a multipart request.
• The service validates each file against the client’s registered configuration:
  • Allowed file extensions - fetched from the Registration Service for the given application.
  • Maximum file size - also sourced from the Registration Service (default: 500 MB).
• Valid files are uploaded to the uploads Azure Blob Storage container. Each file is stored under the client’s folder path and renamed to {UUID}.{extension}.
• A SHA-256 hash is computed and stored with the file metadata in SQL Server.
• If any files fail validation a problem event is published to Service Bus and the failure is included in the API response.
• The endpoint supports uploading multiple files in a single request. The response indicates per-file success or failure.

Child File Uploads

A child file upload is used when a parent document has been split into sub-files (e.g. for per-page classification). The same endpoint accepts optional query parameters:

• parentFileId - links the new file record to an existing parent file in the database.
• clientApplicationClassifierId - determines the storage subfolder path.

Child files go to the splits container rather than uploads. File-type validation is not applied to child uploads.

---

2. Malware Scanning (Microsoft Defender for Cloud)

The service does not call a scanner API directly. Microsoft Defender for Cloud is configured to automatically scan every blob written to the uploads container and publish the result as an event on the malware-scan Service Bus topic.

The service subscribes to this topic and handles four possible result states:

The scan result is always persisted to the database first, before any routing decision is made.

For scan errors, Microsoft Defender includes a specific error code (e.g. SAM259206 = file exceeds 50 GB limit). These codes are mapped to human-readable messages in the problem event.

---

3. File Acceptance (validated-file to accepted-file)

When a clean validated-file event arrives on the Service Bus topic, the service:

1. Looks up the file record and client configuration in the database.
2. Resolves the destination folder by calling the Registration Service.
3. Copies the blob from the uploads container to the accepted-file container.
4. Publishes an accepted-file event to Service Bus (consumed by the classifier and status services).
5. Updates the InternalUrl field in the database to point to the new accepted-file blob location.

---

4. Problem Events

Any rejection (validation failure, malicious file, scan error) results in a problem event published to the problem Service Bus topic. The event follows RFC 9457 (Problem Details for HTTP APIs) format and includes:

• type - a URI identifying the category of problem (e.g. malware-detected, bad-request, malware-scan-error).
• title - a short human-readable title.
• detail - a message describing what happened and for which file.
• instance - a URI uniquely identifying this occurrence.
• status - HTTP-equivalent status code.

---

Service Bus Topics

Subscriptions (inbound)

Publications (outbound)

---

REST API Endpoints

---

Azure Blob Storage Containers

---

Sequence Diagram: File Intake Upload Endpoint Calling Registration

A full technical sequence diagram has been created to document the API interactions between the File Intake upload endpoint and the Registration service. This diagram provides a step-by-step view of how a file upload triggers validation, registration lookups, and system responses.

You can view the complete diagram here: View File Intake - Registration Sequence Diagram

---

Why This Approach?

• Single entry point: All files enter the system through one service, giving a consistent place to enforce validation and security policy.
• Event-driven malware scanning: Microsoft Defender for Cloud handles scanning asynchronously. The service reacts to results rather than polling or calling a scanner API, keeping the upload path fast.
• Fail-safe by default: Any unknown scan result, scan error, or file that cannot be scanned is treated as a rejection. Files only advance if the scan explicitly confirms no threat.
• Soft-delete on malicious files: Azure Blob soft-delete retains malicious blobs for a configurable retention period, allowing forensic review before permanent deletion.
• File persistence: Files remain in their respective containers, providing a history and reference point for troubleshooting and reprocessing.
• Per-client configuration: Allowed file types and size limits are driven by the client’s registration record, so policy is enforced without code changes.