> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://cla-demo.postmansolutions.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://cla-demo.postmansolutions.com/_mcp/server. # File Intake Service ## Overview The File Intake Microservice is the entry point for all files into the AutoExtract Platform. It accepts files via a REST API upload endpoint, stores them in Azure Blob Storage, and records their metadata in SQL Server. Once a file is stored, **Microsoft Defender for Cloud** automatically scans the blob for malware and publishes the result to an Azure Service Bus topic. The service subscribes to that topic and routes the file accordingly: * **Clean files** - a `validated-file` event is published, which triggers the file to be copied into the internal accepted-file storage and an `accepted-file` event is emitted for downstream services. * **Malicious or unscannable files** - the blob is soft-deleted (or left as-is if unreachable) and a `problem` event is published to inform the rest of the system that the file was rejected and why. The service also subscribes to the `validated-file` topic directly, which is responsible for performing the copy from the uploads area into the accepted-file container and publishing the final `accepted-file` event. --- ## Flow Chart ```mermaid flowchart TD %% Nodes client([Client / UI / Documents AI]) uploadEndpoint[POST /v1/upload/:applicationId] uploadsBucket[(uploads container\nAzure Blob Storage)] defender[Microsoft Defender for Cloud\nauto-scans blob] malwareScanEvent([Event: malware-scan-result]) isMalware{Scan Result?} handleClean[Publish validated-file event] handleMalicious[Soft-delete blob\nPublish problem event] handleError[Publish problem event\nmalware-scan-error] validatedFileEvent([Event: validated-file]) copier[Copy file to accepted-file container] acceptedBucket[(accepted-file container\nAzure Blob Storage)] updateDb[Update DB: InternalUrl] acceptedFileEvent([Event: accepted-file]) problemEvent([Event: problem]) done((downstream\nservices)) %% Arrows client -- multipart file upload --> uploadEndpoint uploadEndpoint -- validate + store --> uploadsBucket uploadsBucket -- blob written triggers --> defender defender -- result --> malwareScanEvent malwareScanEvent --> isMalware isMalware -- No threats found --> handleClean isMalware -- Malicious --> handleMalicious isMalware -- Error / Not Scanned --> handleError handleClean --> validatedFileEvent handleMalicious --> problemEvent handleError --> problemEvent validatedFileEvent --> copier copier -- copies to --> acceptedBucket copier --> updateDb copier --> acceptedFileEvent acceptedFileEvent --> done ``` --- ## Detailed Workflow ### 1. File Upload (REST API) * Clients submit files via `POST /v1/upload/:applicationId` as a multipart request. * The service validates each file against the client's registered configuration: * **Allowed file extensions** - fetched from the Registration Service for the given application. * **Maximum file size** - also sourced from the Registration Service (default: 500 MB). * Valid files are uploaded to the **`uploads`** Azure Blob Storage container. Each file is stored under the client's folder path and renamed to `{UUID}.{extension}`. * A SHA-256 hash is computed and stored with the file metadata in SQL Server. * If any files fail validation a `problem` event is published to Service Bus and the failure is included in the API response. * The endpoint supports uploading multiple files in a single request. The response indicates per-file success or failure. #### Child File Uploads A child file upload is used when a parent document has been split into sub-files (e.g. for per-page classification). The same endpoint accepts optional query parameters: * `parentFileId` - links the new file record to an existing parent file in the database. * `clientApplicationClassifierId` - determines the storage subfolder path. Child files go to the **`splits`** container rather than `uploads`. File-type validation is not applied to child uploads. --- ### 2. Malware Scanning (Microsoft Defender for Cloud) The service does **not** call a scanner API directly. Microsoft Defender for Cloud is configured to automatically scan every blob written to the `uploads` container and publish the result as an event on the `malware-scan` Service Bus topic. The service subscribes to this topic and handles four possible result states: | Result | Action | | ---------------- | ------------------------------------------------------------------------------------------------------- | | No threats found | Persist scan result to DB. Publish validated-file event to continue processing. | | Malicious | Persist scan result to DB. Soft-delete the blob from storage. Publish problem event (malware-detected). | | Error | Persist scan result to DB. Publish problem event (malware-scan-error) with error detail. | | Not Scanned | Persist scan result to DB. Publish problem event (malware-scan-error) with error detail. | The scan result is **always persisted to the database first**, before any routing decision is made. For scan errors, Microsoft Defender includes a specific error code (e.g. SAM259206 = file exceeds 50 GB limit). These codes are mapped to human-readable messages in the problem event. --- ### 3. File Acceptance (validated-file to accepted-file) When a clean `validated-file` event arrives on the Service Bus topic, the service: 1. Looks up the file record and client configuration in the database. 2. Resolves the destination folder by calling the Registration Service. 3. Copies the blob from the `uploads` container to the **`accepted-file`** container. 4. Publishes an `accepted-file` event to Service Bus (consumed by the classifier and status services). 5. Updates the InternalUrl field in the database to point to the new accepted-file blob location. --- ### 4. Problem Events Any rejection (validation failure, malicious file, scan error) results in a `problem` event published to the `problem` Service Bus topic. The event follows RFC 9457 (Problem Details for HTTP APIs) format and includes: * **type** - a URI identifying the category of problem (e.g. malware-detected, bad-request, malware-scan-error). * **title** - a short human-readable title. * **detail** - a message describing what happened and for which file. * **instance** - a URI uniquely identifying this occurrence. * **status** - HTTP-equivalent status code. --- ## Service Bus Topics **Subscriptions (inbound)** | Topic | Purpose | | ------------------- | --------------------------------------------------------------------------------- | | validated-file | Receives notification that a clean file is ready to be copied to accepted storage | | malware-scan-result | Receives malware scan outcomes from Microsoft Defender for Cloud | **Publications (outbound)** | Topic | Purpose | | -------------- | -------------------------------------------------------------------------- | | validated-file | Published when a clean scan result re-triggers the validated-file pipeline | | accepted-file | Published after a file is successfully copied to accepted storage | | problem | Published when a file is rejected for any reason | --- ## REST API Endpoints | Method | Path | Purpose | | ------ | ------------------------- | ------------------------------------------------------------ | | POST | /v1/upload/:applicationId | Upload one or more files for a given application | | GET | /v1/file/metadata/:fileId | Retrieve metadata and possible duplicate file IDs for a file | | GET | /openapi.json | OpenAPI specification (JSON) | | GET | /openapi.yaml | OpenAPI specification (YAML) | --- ## Azure Blob Storage Containers | Container | Contents | | ------------- | --------------------------------------------------------------------------- | | uploads | Raw files as received from clients, scanned by Defender | | splits | Child/split files produced from a parent document | | validated | Reserved for future use | | accepted-file | Files that have passed malware scan and are ready for downstream extraction | --- ## Sequence Diagram: File Intake Upload Endpoint Calling Registration A full technical sequence diagram has been created to document the API interactions between the **File Intake** upload endpoint and the **Registration** service. This diagram provides a step-by-step view of how a file upload triggers validation, registration lookups, and system responses. You can view the complete diagram here: [**View File Intake - Registration Sequence Diagram**](https://lucid.app/lucidchart/a0b6a8fb-f2e7-408b-a159-1083cbcc3eb7/edit?viewport_loc=-952%2C1021%2C5012%2C3004%2C0_0\&invitationId=inv_88061227-19ac-4c6b-82b2-e1e9af444b0d) --- ## Why This Approach? * **Single entry point**: All files enter the system through one service, giving a consistent place to enforce validation and security policy. * **Event-driven malware scanning**: Microsoft Defender for Cloud handles scanning asynchronously. The service reacts to results rather than polling or calling a scanner API, keeping the upload path fast. * **Fail-safe by default**: Any unknown scan result, scan error, or file that cannot be scanned is treated as a rejection. Files only advance if the scan explicitly confirms no threat. * **Soft-delete on malicious files**: Azure Blob soft-delete retains malicious blobs for a configurable retention period, allowing forensic review before permanent deletion. * **File persistence**: Files remain in their respective containers, providing a history and reference point for troubleshooting and reprocessing. * **Per-client configuration**: Allowed file types and size limits are driven by the client's registration record, so policy is enforced without code changes.