> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://cla-demo.postmansolutions.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://cla-demo.postmansolutions.com/_mcp/server. # Design Document Status: **LIVING?** Authors: Autobots Contributors: Autobots Requirements Source: ***\[Ask around about existing requirements and designs]*** ## Objective The Platform will allow us to create automatic document identification and data extraction applications with minimal configuration and coding. Each new application may share the same core platform instance or not based on its requirements and the clients' requirements. The models used to comprehend the files and documents will be configurable for each application. In addition, each application will be able to select one or more post-processors for the raw extracted data so that storage formats can be standardized for example. ## Background For clarity it is probably a good idea to say up-front that this document is a work in progress and that many sections are incomplete or not even started. As we define the requirements related to those sections or as decision change we intend to updat this document to help us keep track of what is needed, decided, etc. ***\[Ask around about context and foundation docs and links for existing apps and pain points]*** ## Overview The existing document scanner solutions are each implementated separately using different tools and resources. The result is a significant amount of duplicated effort and functionality in those applications. We see this as an opportunity to build a single copy of those duplicate implementations and to define an interface for plugging in the desired model that understands what is of interest per category of documents. The goal is to make it possible for each newly developed model to be able to be plugged in, tested and deployed in much less time than is currently required for the reimplementation of the entire scanning application. We would like to remove as much of the duplicated work needed to build user interfaces, database storage solutions, application APIs and other shared functionality for each new extraction model in order to further decrease the effort of supporting new types of documents and extractions. The direct consumers of the platform will be the existing extractor applications and new ones like them in the future. When designing the system we should be sure to remember that the user will be using one of the applications to interact with the platform to send new documents for extraction, to repeat extraction steps and to see what files are currently in the system. At this time the platform does not support any kind of interaction with the ERP database(s). It deals with the incoming files and performs the extractions. Once extraction is complete the applications that talk to the platform can use that extracted data any way they like. The platform is finished with the files and the data once the extracted data is stored in the platform's storage for the extracted and normalized data. ### Functional Requirements | Requirement ID | Requirement Description | Category | Notes | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | FR-001 | The system shall restrict upload, review, and classification correction actions to authorized users. | Access Control | Need to have followup conversation with DS team about ingesting corrections | | FR-002 | The system shall support role-based access control (RBAC) to manage permissions for different user types (e.g., Admin, Reviewer, Viewer). | Access Control | Is this up to the calling applications? What different actions do we feel 'users' can take within the platform | | FR-003 | The system shall support REST API integration for data ingestion and export. | API / Integration | | | FR-037 | The API endpoints shall all be versioned | API / Integration | We will want to do URL versioning for each individual service | | FR-038 | Any messages or topic queue's shall be versioned | | | | FR-004 | The system shall identify and classify document types and route the document to the appropriate extractor model. | Classification | | | FR-005 | The system shall identify if a document needs to be split and allow for documents to be split through a document splitter service. | Classification | | | FR-006 | The system shall assign a document id to each split document - with a linkage to the original document id assigned as the parent document. | Classification | | | FR-007 | The system shall allow for classification overriding based on the different extractors available to allow calling application to route directly to specific models. | Classification | | | FR-008 | The system shall allow for multiple classification models to be able to be supported / hosted. Classifications have a limit to the number of classes & models - to get more accurate classification models the system needs to support multiple classifiers in parallel and when the most appropriate classifier classifies a document it routes to the appropriate extraction model. There is an unknown upper limit of the number of classifications the system will need to make. | Classification | Further conversations need | | FR-009 | The system shall ensure all data processing complies with data protection laws (e.g., GDPR, CCPA) and internal policies. | Compliance | To get a review from Governance and Risk | | FR-010 | The processed documents will be moved into a processed folder after they have been extracted. | Data Management | Do we want to utilize less highly available storage? What about retention policy | | FR-011 | The system shall securely handle and store sensitive financial data to prevent unauthorized access and breaches. | Data Security | | | FR-012 | The system shall allow users to upload documents (PDF, Word, and image-based files) via a secure upload interface. | Document Upload | Document type uploads should be limited on upload as a configurable setting | | FR-013 | The system shall support batch uploading of multiple files in a single operation. | Document Upload | | | FR-014 | The system shall run virus check and basic data validations. | Document Upload | We should come up with a list of validations | | FR-015 | The system shall automatically ingest documents received via email or third-party integrations. | Document Upload | Might not be part of the platform - if the platform is only exposing external azure buckets then the calling applications should be responsible for moving documents to the external bucket | | FR-016 | The system shall allow for uploads from n+1 number clients and segregate data from each client. | Document Upload | n+1 means unknown upper bounds and multiple | | FR-017 | The system shall allow for uploads from n+1 number of applications and segregate data accordingly from each application. | Document Upload | application or source or service or function | | FR-018 | The system will create a mechanism for identifying document uploads to distinguish unique documents from duplicated documents already uploaded. | Document Upload | | | FR-019 | The system shall log all errors and exceptions encountered during extraction and conversion for audit and troubleshooting. | Exception Logging | | | FR-020 | The system shall allow for n+1 number of extractor models, these models shall be information accessible to consumers of the platform. | Extraction | | | FR-021 | The system shall serialize data into a json object upon return from the extractor. | Extraction | | | FR-022 | The system shall include all meta data of the extraction results with the end payload. | Extraction | All metadata that needs to be included needs to be tracked | | FR-023 | The system shall capture and store document metadata, including file name, upload date, and source, linked to the document record. | Metadata Management | | | FR-024 | The system shall create a unique correlation id for each document upload to distinguish it and track it throughout its extraction journey. | Observability | | | FR-025 | The system shall log all status information about a request including but not limited to: user, client, application, origination source, process step, datetime, parent documentId, documentId, etc. | Observability | The full list needs to be determined | | FR-026 | The system shall allow for n+1 levels of parent/child document relationships. | Observability | | | FR-027 | Documents that have an error shall be placed in an error folder with the corresponding name of the file and the correlation id of the process run. | Observability | This is about errors that cannot be retried - true failures | | FR-028 | The system shall maintain a full audit trail of data processing activities for compliance and review with datetime. | Observability | Transaction logging | | FR-030 | The system shall allow for consuming applications to view details of registered extraction models for easy access to view payload expectations, version, and deployment type - see FR-033 as this is part of the guidance to onboard an extractor. | Observability | | | FR-031 | The system shall allow for endpoints to be available for updating output schema specs of registered extraction models - this is for teams managing extractor deployments to update the information available in FR-030. | Observability | | | FR-032 | The system shall allow for users to view the status of their document via a REST endpoint utilizing their correlationId. | Observability | | | FR-033 | The team shall create guidance on extraction model architecture to allow them to be implemented into the framework. | On-Boarding | | | FR-034 | The system shall allow for restrictions of client and/or application level platform artifacts. | Access Control | | | FR-035 | Data will be written to a long term storage location after processing - see FR-009 for restrictions. | Data Management | | | FR-036 | The system shall require the extractors to be versioned. | Extraction | | ## Infrastructure What we will need: * Containers on Azure to run the platform itself * We can start with a pretty minimal container * It should be running some Linux on Intel 64-bit *(or whatever our current "best" standard is - seems best if it matches the dev containers)* * The container needs to support some recent version of node (the latest one seen in the framework and used in our POCs is v22.11.0 as of 3/12/25) * Appropriate K8 configuration for containers to allow scaling to handle load spikes * One or more storage locations for documents to scan. They might exist currently for SOME clients and apps, but we should consider a set for "just platform" because it is significantly more challenging to use other teams' resources and even more challenging to use resources that do not belong to CLA but to a specific client * Some database storage to keep the scan results in * For NoSQL needs we will use Cosmos DB (managed Azure MongoDB-like) * For relational (SQL) needs we will use managed MS SQLServer * For caching and temporary storage, we will use Redis What we probably do not need: * API Rate Limiting: Per Tom Ruhland the product owner did not want that at this time (02/28/25) ***\[We will need to understand the requirements better to give more details on storage limits, processors and RAM and all that kind of stuff]*** ***\[We plan to store the final extraction results in NoSQL database because it is not the same shape for each extractor AND it is like a tree-shaped graph for some]*** ## Detailed Design ***\[This is the meat and potatoes section of the design document. For larger projects, it should span many pages with multiple sub-sections. Begin with the high-level overviews and then detail out in the sub sections. This becomes the decision log of the design implementations for each module, plage, API, and sub section within the technology stack. What libraries were chosen and why, what API’s were implemented and why, what design patterns were chosen to do specific UOW, ect.]*** ### Overview The AutoExtract Platform efficiently manages the ingestion, classification, extraction, and post-processing of documents. It is built with several microservices, each dedicated to a specific part of the workflow. The platform is highly extensible, allowing for the addition of new classifiers, extractors, and post-processors as needed. Our microservices communicate important system events through message bus topics, ensuring that all interested components are informed. This architecture enables seamless cooperation among the microservices through choreographed messaging, enhancing the overall efficiency and scalability of the system. ### Key Services #### **File Intake Service** * **Role:** Handles incoming files from various sources, validates them, and stores them in designated locations trusted for processing. * **Triggers:** A new file is uploaded or detected in the monitored storage location. * **Actions:** * Validates file format, size, and metadata. * Accepts or rejects files based on predefined business rules. * Publishes a readiness message for further processing. * **Contributions:** * **Centralized File Management:** Ensures consistency and validation before processing. * **Streamlined Intake Process:** Publishes a single message per file, simplifying further processing. * **Predictable File Persistence:** Places files in trusted locations for historical reference and troubleshooting. * **Error Handling:** * Rejects unsupported or corrupt files. * Logs errors and maintains a separate storage for rejected files. #### **Classifier Service** * **Role:** Identifies file contents and splits multi-document files into individual documents. * **Triggers:** Receives notification of an accepted file. * **Actions:** * Uses configurable machine learning models for classification. * Splits multi-document files into individual documents when needed. * Assigns metadata and routing labels for downstream services. * **Contributions:** * **Configurable Classification:** Allows customization per client application. * **Decoupled Deployment:** Models can be updated independently, minimizing system impact. * **Focused Processing:** Dedicated solely to classification and segmentation. * **Error Handling:** * Falls back to default classification if model processing fails. * Publishes a problem message to the `"problem"` topic following the standard **problem reporting procedure**. #### **Extractor Service** * **Role:** Uses extraction models to collect meaningful information from document/image files. * **Triggers:** Receives classified files for data extraction. * **Actions:** * Runs OCR and entity recognition to extract structured data. * Provides a **consistent interface** for integration with Power BI, Microsoft Doc Intelligence, and REST-based AI models. * Stores raw extracted data in NoSQL storage. * **Contributions:** * **Consistent Interface:** Standardizes interaction with diverse extraction technologies. * **Configurable Extraction:** Allows customization per client application. * **Focused Processing:** Dedicated exclusively to raw data extraction. * **Error Handling:** * Flags low-confidence extractions for manual review. * If an extraction error occurs, publishes a problem message to the `"problem"` topic following the standard **problem reporting procedure**. #### **Postprocessor Service** * **Role:** Refines extracted data, ensuring its usability for final storage. * **Triggers:** Detects new raw extracted data. * **Actions:** * Cleanses and normalizes extracted data. * Applies business logic to polish output for client applications. * Stores final processed data in NoSQL storage. * **Contributions:** * **Configurable Postprocessing:** Each client can define normalization requirements. * **Focused Processing:** Dedicated solely to refining raw extracted data. * **Completion Identification:** Ensures proper tracking of finished processing. * **Error Handling:** * If an irrecoverable issue arises, publishes a problem message to the `"problem"` topic following the standard **problem reporting procedure**. #### **Status-Check Service** * **Role:** Tracks file status throughout the processing pipeline, maintaining historical records. * **Triggers:** Listens to all system-wide events. * **Actions:** * Maintains audit trails and timestamps for every file. * Tracks files actively being processed and those recently rejected. * Provides status updates on request. * **Contributions:** * **Status Checking:** Helps track progress for client applications. * **Troubleshooting:** Assists in workflow diagnostics and debugging. * **Listing Files:** Maintains real-time visibility into processed and rejected files. * **Error Handling:** * Detects missing event sequences and triggers alerts. * Provides historical file activity queries. ### Event-Driven Workflow The AutoExtract Platform follows a fully **event-driven architecture**, ensuring that microservices remain **loosely coupled** and react only to messages relevant to their function. This approach enhances scalability, fault tolerance, and deployment flexibility. #### **Principles of Event-Based Communication** * **Decoupled Services:** Each service operates independently and does not rely on direct communication with other services. * **Message-Driven Interaction:** Services respond only to relevant events, subscribing to messages instead of making direct calls to other components. * **Resilient Processing:** If a service temporarily goes offline (e.g., during deployment or maintenance), it resumes processing when back online, seamlessly picking up pending messages. * **Azure Service Bus Backbone:** Asynchronous messaging ensures reliable communication across services, preventing bottlenecks and enabling parallel execution. * **No Hard Dependencies:** A service does not need to know where the message originated—only that an event occurred that it cares about. #### **Platform Workflow: Standard Processing Flow** 1. **File Intake Service** detects a new `"submitted-file"` event and processes the external file. * If valid, it copies the file into the accepted file location, ensuring only "safe" and "proper" files enter the system. * Once the copy is complete, it signals readiness for further processing by publishing a `"accepted-file"` event. * If rejected (e.g., wrong type, virus detected), the file moves into a designated quarantine area, and a **problem message** is published to the `"problem"` topic. 2. **Classifier Service** listens for `"accepted-file"` events and begins classification. * Assigns metadata and routing tags. * If classification fails, it follows the **problem reporting procedure**, publishing a problem message to the `"problem"` topic. 3. **Extractor Service** listens for `"classified-file"` events and extracts structured data. * Submits the file to the model-based extractor system. * Waits for extraction completion. * If successful, stores extracted data in NoSQL and publishes an `"extracted-result"` event. * If an extraction error occurs, it follows the **problem reporting procedure**, publishing a problem message to the `"problem"` topic. 4. **Postprocessor Service** listens for `"extracted-result"` events and refines the data. * Cleanses, normalizes, and applies business rules. * Confirms readiness for final storage and publishes a `"postprocessed-result"` event. * If an irrecoverable issue arises, it reports the problem via the **problem reporting procedure**, publishing to the `"problem"` topic. 5. **Status-Check Service** continuously tracks the progress of files. * Updates processing history. * Tracks files from intake through completion. * Provides real-time visibility into system operations. This workflow ensures each service operates independently, reacting dynamically to system events while maintaining reliability and resilience. The **event-driven architecture** enables flexible processing without direct dependencies, ensuring robust performance even during service downtimes. The platform follows a standard **problem reporting procedure**, publishing problem messages to the `"problem"` topic, adhering to the main recommendation of the **problem+json** format as documented elsewhere. For request-response messaging, errors are returned as problem+json responses. In pub-sub messaging, all errors are published to the `"problem"` topic. The platform consistently adheres to its implementation of problem+json. ## UX Design (if applicable) The Platform does not currently have any UI planned ## Alternatives Considered ***\[each of these should describe a choice made and a reason - it would be best if we could give a better reason than "enterprise standards" but that is not always possible]*** ***\[please add more if we notice other decisions we made or that we need to make]*** * Azure Cloud: We chose Azure because it was required at the time * TypeScript: Our part of CLA wants to be a TypeScript shop. * Vitest: Goes well with TypeScript, has decent support for mocking, assertions and coverage estimation. We had Jest but this works better with ESM JavaScript on Node. * Pino: Personal preference but it is also being used inside the framework packages already * Application Insights: So far the best we can tell, ops and infra teams are recommending this choice (the main alternative would be Azure Monitor Logs which is not well supported by Pino OR the framework) * Docker: The most popular and well supported containerization tool and it plays well with K8s * Kubernetes (AKS): \[Another one not decided by the team but by ops or infra or Aaron] * Azure message bus: \[Aaron told me this one was decided] * Temporal (really NONE for now): It is significantly nicer to use than Camunda and is used with a code-like interface instead of a GUI * For database to store final results we are going to use CosmosDB because the data looks like tree-like documents. Aaron selected the SQL-like interface not the Mongo one and I don't know the specifical of why. * For database to store raw extractions before postprocessing we can use Redis because these are tree-like too but kept for a short time and only fetched by ID * Azure blob storage (because we use Azure it would be weird to use S3) * Azure SQL Server for our relational needs (intake configs and status details so far) * APIM (Because this is the Azure API "helper" and is provided by the team that is creating our "Landing Zone"). We have been asked about our APIM strategy/plan/design a few times. As of early July we do not know with certainty that this is compatible with the APIM we will GET but it is our hope that it is pretty close to this: Our system will rely on Azure API Management (APIM) to validate JWTs issued by Microsoft Entra ID. We assume APIM will be configured to enforce token validation and forward trusted claims to our services. Our services will implement RBAC based on those claims. Final implementation details depend on landing zone configuration, which is not yet available. * Postman for API testing and API doc viewing? ### Some choices for specific standards and encodings * **Timestamps:** ISO 8601 UTC date with Z abbreviation and millisecond precision (very standard and good precision) * **Problems:** Following problem+json standards, see this wiki page: [HTTP Status Codes and Problem+JSON](https://dev.azure.com/CLADevOps/Development/_wiki/wikis/AutoExtract/8093/HTTP-Status-Codes-and-Problem-JSON) * **IDs:** Where possible we will use UUID v4 for ID values because they are easy to generate and reveal nothing about the system * **Hashing:** For file-hashes used to detect duplicates we will be using MD5 for now since it is fast, well supported and good enough for most purposes as a fail-hash algorithm. We WILL NOT use it where we are concerned about security. There are more modern hashing algorithms that are better for that. ## Dependencies ***\[What external teams / tools is this project/solution dependent on? How are they leveraged, what is the risk of the dependency? Is it a 3rd party dependency or another team within the firm?]*** * The model creators to create the models AND to help define how best to interact with them * The infrastructure administrators to create and own the low-level infrastructure like cloud servers, networks and all that * The ops and monitoring team to help us know what to provide and how best to help them monitor The Platform apps * The DB team to advise about database stuff, to own the database and possibly to create the schema ## Migrations ***\[What needs to be done to get data into this system? Is it a greenfield application? Is it replacing something that already exists and we need to massage some data into our new system?]*** ## Technical Debt ***\[List all technical debt as they come up. Link to User Stories are acceptable as updates to the US can reflect more real time than documenting here.]*** ## Security ***\[Security review documents, pen testing results,]*** ## Reliability ***\[Uptime requirements, health checks, fault isolation, recovery and redundancy plans.]*** ## Data Integrity ***\[Are there data validation services being leveraged? How do we handle data / row locking on updates? Simultaneous user edits? Are there risks of bad data being entered?]*** ## Privacy ***\[GRC results, PII stored, steps taken to mitigate exposure of data.]*** ## Scalability ***\[Can this application be scaled? Performance testing concerns? NFR’s linked to the load limits. How would the team approach scaling aspects of the application? Approaches to this is highly dependent on the architecture of the application.]*** ## Latency ***\[Response times, load times, all detailed on the acceptable limits for major areas of the application and any exceptions to the high-level limits.]*** ## Accessibility ***\[The AutoExtract Platform is a backend solution with no current UI. We are not building anything like that as far as I know but we should verify before we finish this section]*** ## Test Plan ***\[Link to E2E test plans. Detail the depth of different tests: where is the automation coverage, where is manual testing done, integrations testing. Data integrity tests, performance tests, ect.]*** ## Monitoring ***\[What is monitored? Where is it monitored? How do we modify logging levels?]*** ## Alerting ***\[How are alerts communicated to the team? What mechanism and where is the distribution list? What communication channels are set up for it?]*** ## SLO/SLA ***\[Links to established SLO/SLA. Internal and external.]*** ## Rollback Strategy ***\[What happens if the application is unusable or low adoption? What happens if the launch fails?]*** ## Launch plan ***\[How will the application be rolled out? Is there a canary deploy and a incremental roll after that? Is an MVP being pushed to test specific areas? Blue/Green deploy? Stabilization period?]***