> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://cla-demo.postmansolutions.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://cla-demo.postmansolutions.com/_mcp/server. # Blob Storage Accounts # Blob Container Process Flow – Improvement Proposal ## Overview The current File Intake flow is functionally correct and aligns with an event-driven architecture. However, there are opportunities to simplify the process, reduce unnecessary coupling, and improve performance and maintainability. This document outlines proposed improvements focusing on: 1. Removal of self-subscription (`validated-file` loop) 2. Eliminating unnecessary blob copy operations 3. Improving container lifecycle design 4. Introducing idempotency safeguards --- ## 1. Remove Self-Subscription to `validated-file` Topic ### Current Behavior * File Intake Service publishes a message to the `validated-file` topic after validation. * The same service subscribes to this topic. * Upon receiving the event, it copies the blob from `uploads` → `validated`. ### Problem * This creates a **self-referential loop**: \- The same service is both the producer and consumer of the same event. * Adds **unnecessary complexity and latency**: \- Extra event hop before processing continues * Makes debugging and tracing harder due to indirect flow ### Proposed Change Move the blob promotion logic directly into the malware scan result handler (or validation completion handler): ```text Upload → Validate → (Malware Scan) → IF clean:     Copy (or promote) blob     Publish accepted-file ``` ### Benefits * Eliminates unnecessary topic subscription * Reduces end-to-end processing latency * Simplifies flow and improves readability * Clearer ownership of responsibilities per stage --- 2. Avoid Blob Copy – Use Logical Promotion Instead --- ### Current Behavior * Blob is physically copied from: * `uploads` container → `validated` container ### Problems * **Duplicate storage usage** * Additional **I/O overhead** on every file * Increased failure points (partial copy, retry issues) * Synchronization risks between containers ### Proposed Change Replace physical copy with a **logical promotion model**: * Keep the file in a single container * Track state using: * Database field (`status`) * Blob metadata/tags (optional) Example states: Plain Text uploaded → validated → accepted Show more lines ### Alternative (if separation is required) If container separation is necessary for compliance or isolation: * Retain containers but minimize copy operations (e.g., move/rename pattern if supported) ### Benefits * Eliminates unnecessary data duplication * Improves performance and reduces cost * Simplifies recovery and retry logic * Reduces storage management complexity --- 3. Improve Container Lifecycle Design --- ### Current Structure * `uploads` * `validated` ### Gaps Identified * No explicit representation of malware scanning state * Lifecycle stages are not clearly separated * Harder to audit file journey ### Proposed Design Introduce a clearer lifecycle model: Plain Text uploads → initial ingestion quarantine → pending malware scan validated → clean files accepted → finalized / ready for downstream Show more lines ### Notes * `quarantine` stage helps isolate files before scan completion * `accepted` stage can be logical or explicit depending on downstream requirements ### Benefits * Improves traceability and auditability * Enhances security posture (clear separation of infected vs clean) * Makes lifecycle transitions explicit and maintainable --- 4. Introduce Idempotency in Event Processing --- ### Current Risk * Event-driven pipeline relies on: * Service Bus topics * Multiple asynchronous handlers * Retry scenarios (e.g., transient failures) may cause: * Duplicate blob operations * Duplicate event publishing * Data inconsistencies ### Proposed Safeguards #### a. Database-Level Idempotency * Track processing state per file: Plain Text isValidated isAccepted processedAt timestamps Show more lines #### b. Event Deduplication * Use unique identifiers: Plain Text fileId + eventType Show more lines * Skip processing if already handled #### c. Blob Operation Guards * Check if target blob already exists before copying * Avoid overwriting or duplicating blobs #### d. Retry-Safe Logic * Ensure operations are safe to execute multiple times without side effects ### Benefits * Prevents duplicate processing and data corruption * Improves reliability under retry conditions * Makes the system resilient to transient failures --- ## Summary of Improvements | Area | Improvement | Impact | | ---------------- | ------------------------------- | ---------------------------------------- | | Event Flow | Remove self-subscription | Simpler, faster pipeline | | Storage Usage | Avoid blob copy | Reduced cost + better performance | | Lifecycle Design | Add structured container stages | Better clarity and security | | Reliability | Add idempotency | Prevents duplication and inconsistencies | --- ## Conclusion These improvements aim to: * Simplify the event-driven workflow * Reduce unnecessary operations and coupling * Improve reliability and observability * Better align the system with scalable, maintainable Azure storage patterns The proposed changes can be implemented incrementally, starting with removal of the self-subscription pattern and introducing idempotency safeguards. \============================================================================================== # Azure Blob Storage Cleanup Plan ## Identification and Removal of Unused Containers with Risk Assessment ### Objective To identify unused Azure Blob containers and safely remove them to reduce storage costs, improve governance, and eliminate unnecessary data while minimizing operational and compliance risks. --- ### Scope This plan applies to all Azure Blob Storage accounts across production, non-production, and sandbox environments. --- ### Identification of Unused Containers Unused containers will be identified using the following criteria: * Containers with no blobs (empty containers) * Containers with no read/write activity over a defined period (e.g., 60–90 days) * Containers with no recent data modifications or ingestion * Containers not associated with active applications, services, or integrations * Containers not used for logging, monitoring, backup, or compliance retention --- ### Cleanup and Removal Plan #### Phase 1: Initial Identification * Extract and inventory all containers across storage accounts * Classify containers into: \- Active \- Inactive (candidate for removal) \- Critical (protected or compliance-related) --- #### Phase 2: Remove Empty Containers * Identify containers with zero blobs * Validate that no dependencies exist * Obtain required approvals * Proceed with deletion of empty containers --- #### Phase 3: Dependency Validation * For non-empty containers, verify no dependencies from: \- Applications (App Services, Functions, APIs) \- Automation jobs or scripts \- Logging or monitoring pipelines \- External integrations * Confirm status with application and data owners --- #### Phase 4: Data Review and Retention Check * Review data classification and sensitivity * Confirm no regulatory or business retention requirements * Archive data if required instead of deleting --- #### Phase 5: Safe Deletion Preparation * Enable soft delete for containers * Take backups if necessary * Document containers approved for deletion --- #### Phase 6: Controlled Deletion * Delete approved containers in phases (avoid bulk deletion) * Track and log all deletion activities * Monitor systems after each batch deletion --- #### Phase 7: Post-Deletion Monitoring * Monitor applications for failures or missing dependencies * Validate no impact to business operations * Retain logs for audit and tracking purposes --- ### Risk Assessment #### Technical Risks * **Hidden dependencies may cause failures** *Mitigation:* Perform dependency validation and stakeholder confirmation * **Accidental data loss** *Mitigation:* Enable soft delete and perform backups before deletion --- #### Operational Risks * **Application disruption due to missing storage** *Mitigation:* Use phased deletion and monitor after each step * **Misclassification of active containers as unused** *Mitigation:* Apply multiple validation checks and approval workflows --- #### Security and Compliance Risks * **Violation of data retention requirements** *Mitigation:* Validate against compliance policies before deletion * **Loss of audit or log data** *Mitigation:* Archive required data instead of deleting --- ### Governance and Controls * Enforce container tagging (Owner, Environment, Retention) * Conduct periodic cleanup reviews (e.g., quarterly) * Implement lifecycle management policies for automated cleanup * Maintain audit logs for all deletion activities --- ### Conclusion This plan ensures unused Azure Blob containers are identified and removed in a controlled and risk-aware manner. By validating dependencies, confirming retention requirements, and executing phased deletion with safeguards, the organization minimizes operational disruption while optimizing storage cost and efficiency.