Blob Container Process Flow – Improvement Proposal

Overview

The current File Intake flow is functionally correct and aligns with an event-driven architecture. However, there are opportunities to simplify the process, reduce unnecessary coupling, and improve performance and maintainability.

This document outlines proposed improvements focusing on:

1. Removal of self-subscription (validated-file loop)

2. Eliminating unnecessary blob copy operations

3. Improving container lifecycle design

4. Introducing idempotency safeguards

1. Remove Self-Subscription to validated-file Topic

Current Behavior

• File Intake Service publishes a message to the validated-file topic after validation.

• The same service subscribes to this topic.

• Upon receiving the event, it copies the blob from uploads → validated.

Problem

• This creates a self-referential loop:

  - The same service is both the producer and consumer of the same event.

• Adds unnecessary complexity and latency:

  - Extra event hop before processing continues

• Makes debugging and tracing harder due to indirect flow

Proposed Change

Move the blob promotion logic directly into the malware scan result handler (or validation completion handler):

Upload → Validate → (Malware Scan) →
IF clean:
    Copy (or promote) blob
    Publish accepted-file

Benefits

• Eliminates unnecessary topic subscription
• Reduces end-to-end processing latency
• Simplifies flow and improves readability
• Clearer ownership of responsibilities per stage

2. Avoid Blob Copy – Use Logical Promotion Instead

Current Behavior

• Blob is physically copied from:
  • uploads container → validated container

Problems

• Duplicate storage usage
• Additional I/O overhead on every file
• Increased failure points (partial copy, retry issues)
• Synchronization risks between containers

Proposed Change

Replace physical copy with a logical promotion model:

• Keep the file in a single container
• Track state using:
  • Database field (status)
  • Blob metadata/tags (optional) Example states:

Plain Text

uploaded → validated → accepted

Show more lines

Alternative (if separation is required)

If container separation is necessary for compliance or isolation:

• Retain containers but minimize copy operations (e.g., move/rename pattern if supported)

Benefits

• Eliminates unnecessary data duplication
• Improves performance and reduces cost
• Simplifies recovery and retry logic
• Reduces storage management complexity

3. Improve Container Lifecycle Design

Current Structure

• uploads
• validated

Gaps Identified

• No explicit representation of malware scanning state
• Lifecycle stages are not clearly separated
• Harder to audit file journey

Proposed Design

Introduce a clearer lifecycle model:

Plain Text

uploads → initial ingestion

quarantine → pending malware scan

validated → clean files

accepted → finalized / ready for downstream

Show more lines

Notes

• quarantine stage helps isolate files before scan completion
• accepted stage can be logical or explicit depending on downstream requirements

Benefits

• Improves traceability and auditability
• Enhances security posture (clear separation of infected vs clean)
• Makes lifecycle transitions explicit and maintainable

4. Introduce Idempotency in Event Processing

Current Risk

• Event-driven pipeline relies on:
  • Service Bus topics
  • Multiple asynchronous handlers
• Retry scenarios (e.g., transient failures) may cause:
  • Duplicate blob operations
  • Duplicate event publishing
  • Data inconsistencies

Proposed Safeguards

a. Database-Level Idempotency

• Track processing state per file:

Plain Text

isValidated

isAccepted

processedAt timestamps

Show more lines

b. Event Deduplication

• Use unique identifiers:

Plain Text

fileId + eventType

Show more lines

• Skip processing if already handled

c. Blob Operation Guards

• Check if target blob already exists before copying
• Avoid overwriting or duplicating blobs

d. Retry-Safe Logic

• Ensure operations are safe to execute multiple times without side effects

Benefits

• Prevents duplicate processing and data corruption
• Improves reliability under retry conditions
• Makes the system resilient to transient failures

Summary of Improvements

Conclusion

These improvements aim to:

• Simplify the event-driven workflow
• Reduce unnecessary operations and coupling
• Improve reliability and observability
• Better align the system with scalable, maintainable Azure storage patterns The proposed changes can be implemented incrementally, starting with removal of the self-subscription pattern and introducing idempotency safeguards.

==============================================================================================

Azure Blob Storage Cleanup Plan 

Identification and Removal of Unused Containers with Risk Assessment 

Objective 

To identify unused Azure Blob containers and safely remove them to reduce storage costs, improve governance, and eliminate unnecessary data while minimizing operational and compliance risks. 

Scope 

This plan applies to all Azure Blob Storage accounts across production, non-production, and sandbox environments. 

Identification of Unused Containers 

Unused containers will be identified using the following criteria: 

• Containers with no blobs (empty containers) 

• Containers with no read/write activity over a defined period (e.g., 60–90 days) 

• Containers with no recent data modifications or ingestion 

• Containers not associated with active applications, services, or integrations 

• Containers not used for logging, monitoring, backup, or compliance retention 

Cleanup and Removal Plan 

Phase 1: Initial Identification 

• Extract and inventory all containers across storage accounts 

• Classify containers into: 

  - Active 

  - Inactive (candidate for removal) 

  - Critical (protected or compliance-related) 

Phase 2: Remove Empty Containers 

• Identify containers with zero blobs 

• Validate that no dependencies exist 

• Obtain required approvals 

• Proceed with deletion of empty containers 

Phase 3: Dependency Validation 

• For non-empty containers, verify no dependencies from: 

  - Applications (App Services, Functions, APIs) 

  - Automation jobs or scripts 

  - Logging or monitoring pipelines 

  - External integrations 

• Confirm status with application and data owners 

Phase 4: Data Review and Retention Check 

• Review data classification and sensitivity 

• Confirm no regulatory or business retention requirements 

• Archive data if required instead of deleting 

Phase 5: Safe Deletion Preparation 

• Enable soft delete for containers 

• Take backups if necessary 

• Document containers approved for deletion 

Phase 6: Controlled Deletion 

• Delete approved containers in phases (avoid bulk deletion) 

• Track and log all deletion activities 

• Monitor systems after each batch deletion 

Phase 7: Post-Deletion Monitoring 

• Monitor applications for failures or missing dependencies 

• Validate no impact to business operations 

• Retain logs for audit and tracking purposes 

Risk Assessment 

Technical Risks 

• Hidden dependencies may cause failures 

  Mitigation: Perform dependency validation and stakeholder confirmation 

• Accidental data loss 

  Mitigation: Enable soft delete and perform backups before deletion 

Operational Risks 

• Application disruption due to missing storage 

  Mitigation: Use phased deletion and monitor after each step 

• Misclassification of active containers as unused 

  Mitigation: Apply multiple validation checks and approval workflows 

Security and Compliance Risks 

• Violation of data retention requirements 

  Mitigation: Validate against compliance policies before deletion 

• Loss of audit or log data 

  Mitigation: Archive required data instead of deleting 

Governance and Controls 

• Enforce container tagging (Owner, Environment, Retention) 

• Conduct periodic cleanup reviews (e.g., quarterly) 

• Implement lifecycle management policies for automated cleanup 

• Maintain audit logs for all deletion activities 

Conclusion 

This plan ensures unused Azure Blob containers are identified and removed in a controlled and risk-aware manner. By validating dependencies, confirming retention requirements, and executing phased deletion with safeguards, the organization minimizes operational disruption while optimizing storage cost and efficiency.