Service Bus Message Error Classification Strategy

Spike: Error Classification Design (User Story 762531)

1. Summary

This document defines a standardized error classification strategy for Azure Service Bus message processing across Fluxx microservices. The strategy distinguishes between transient failures (temporary, retriable) and permanent failures (non-retriable) to optimize system reliability and operational efficiency.

Key Outcomes

Clear taxonomy of message processing errors
Decision rules for retry vs dead-letter actions
Improved system reliability and reduced operational noise
Ensure the approach is consistent, configurable, observable, debuggable, and testable.

2. Problem Statement

Current State

Currently in fluxx-extractor (and likely other services):

All message processing errors are treated the same way
Failed messages are always abandoned for retry, regardless of error type
No distinction between transient (network timeout) vs permanent (invalid data) failures
Validation errors retry indefinitely until max delivery count
Dead-letter queues contain mix of retriable and non-retriable messages
Limited error metadata makes troubleshooting difficult

Impact

Wasted Service Bus resources on non-retriable errors
Delayed identification of systematic issues (e.g., missing configuration)
Poor operational visibility into failure reasons
Inconsistent behavior across microservices

Design Goals

Primary Goals

Clear Error Taxonomy - Establish standardized categories for all message processing failures
Intelligent Retry Logic - Apply conditional retry based on error classification
Operational Correctness - Dead-letter permanent failures immediately, retry transient failures
Cross-Service Consistency - Same approach in all Fluxx microservices
Enhanced Observability - Rich metadata and structured logging for debugging

3. Error Classification Framework

3.1 Classification Principles

Transient Errors

Errors that are temporary and likely to succeed on retry without any code or data changes.

Characteristics:

External dependency temporarily unavailable
Network connectivity issues
Resource contention that resolves over time
Rate limiting that expires

Strategy: Retry with exponential backoff

Permanent Errors

Errors that will never succeed regardless of retries without intervention.

Characteristics:

Invalid input data
Missing configuration
Business rule violations
Resources that don’t exist

Strategy: Immediate dead-letter for manual investigation

4. Error Taxonomy

4.1 Transient Error Categories

TRANSIENT_SERVICE

External service temporarily unavailable

Attribute	Value
Retryable	Yes
Max Retries	10 attempts
Retry Delay	Service Bus default (lock duration)
HTTP Status Codes	500, 502, 503, 504
Common Causes	- Service deployment in progress - Temporary outage - Circuit breaker open - Service restarting
Resolution	Automatic - wait for service recovery
Example	Registration API returns 503 during deployment

Rationale: Services typically recover within minutes. Retry allows automatic recovery without manual intervention.

TRANSIENT_NETWORK

Network connectivity issues

Attribute	Value
Retryable	Yes
Max Retries	10 attempts
Error Codes	ECONNREFUSED, ETIMEDOUT, ENOTFOUND, ENETUNREACH, ECONNRESET
Common Causes	- DNS resolution failure - Network partition - Firewall blocking - Service not listening
Resolution	Automatic - network typically recovers
Example	Connection refused when calling external API

Rationale: Network issues are often transient. DNS, routing, and firewall issues typically resolve quickly.

TRANSIENT_THROTTLING

Rate limiting by external service

Attribute	Value
Retryable	Yes
Max Retries	10 attempts
Retry Strategy	Honor `Retry-After` header if present
HTTP Status Codes	429
Common Causes	- API quota exceeded - Too many requests per second - Burst limit exceeded
Resolution	Automatic - wait for quota reset
Example	Document AI API returns 429 with `Retry-After: 60`

Rationale: Rate limits typically reset automatically. Respecting Retry-After prevents additional throttling.

Future Enhancement: Implement exponential backoff or circuit breaker pattern.

TRANSIENT_RESOURCE

Resource contention that resolves over time

Attribute	Value
Retryable	Yes
Max Retries	10 attempts
Common Causes	- Database deadlock - Lock timeout - Service Bus message lock lost - Connection pool exhausted
Resolution	Automatic - contention resolves as resources free up
Example	Database deadlock during concurrent job updates

Rationale: Resource contention is temporary by nature. Deadlocks resolve automatically, and pools refill.

4.2 Permanent Error Categories

PERMANENT_VALIDATION

Invalid message data or structure

Attribute	Value
Retryable	No
Action	Immediate dead-letter
HTTP Status Codes	400
Common Causes	- Missing required fields - Invalid data format - Type mismatch - Schema validation failure
Resolution	Manual - fix data source, update message, resubmit
Example	Message missing `applicationId` field

Rationale: Invalid data will never become valid through retry. Immediate dead-lettering allows quick identification and fix.

Dead-Letter Metadata:

{
  "deadLetterReason": "PERMANENT_VALIDATION",
  "deadLetterErrorDescription": "[Attempt 1] Missing required field: applicationId",
  "originalMessage": { /* message body */ }
}

PERMANENT_CONFIGURATION

Missing or invalid configuration

Attribute	Value
Retryable	No
Action	Immediate dead-letter
Common Causes	- No extractors configured - Missing environment variables - Invalid configuration values
Resolution	Manual - update configuration, resubmit message
Example	No extractors configured for `applicationId`

Rationale: Configuration issues require deployment or database changes. Retry wastes resources and delays detection.

Operational Response:

Alert: Page on-call immediately (high severity)
Investigate: Check configuration in database/environment
Fix: Add missing configuration
Reprocess: Resubmit dead-lettered messages

PERMANENT_NOT_FOUND

Resource doesn’t exist

Attribute	Value
Retryable	No
Action	Immediate dead-letter
HTTP Status Codes	404
Common Causes	- Application ID doesn’t exist - File deleted from blob storage - Extractor removed - Stale reference
Resolution	Manual - create resource or discard obsolete message
Example	Registration API returns 404 for `applicationId`

Rationale: Resources don’t spontaneously appear. Retry wastes resources checking for non-existent entities.

Decision Tree:

404 Error
    ↓
Is resource expected to exist?
    ↓
YES → Create resource → Resubmit
    ↓
NO → Discard message (obsolete data)

PERMANENT_AUTHORIZATION

Authentication or permission failure

Attribute	Value
Retryable	No
Action	Immediate dead-letter
HTTP Status Codes	401, 403
Common Causes	- Invalid API key - Expired token - Insufficient permissions - Certificate expired
Resolution	Manual - update credentials, grant permissions
Example	External API returns 401 with invalid API key

Rationale: Credentials don’t fix themselves. Immediate dead-lettering highlights security/access issues.

Security Consideration: Don’t log credentials in error metadata.

PERMANENT_BUSINESS_RULE

Business logic prevents processing

Attribute	Value
Retryable	No
Action	Immediate dead-letter
HTTP Status Codes	409
Common Causes	- Duplicate entry - Invalid state transition - Constraint violation - Business invariant broken
Resolution	Manual - investigate business logic, fix state, decide on reprocess
Example	Attempting to reprocess already-completed job

Rationale: Business rules are deterministic. Retry won’t change the outcome without state changes.

PERMANENT_MALFORMED

Message structure invalid

Attribute	Value
Retryable	No
Action	Immediate dead-letter
Common Causes	- Missing message body - Unparseable JSON - Corrupted message - Wrong message type
Resolution	Manual - investigate producer, fix serialization, discard corrupt messages
Example	Service Bus message with null body

Rationale: Malformed messages indicate producer bugs. Retry wastes lock time on unparseable data.

5. HTTP Status Code Mapping

5.1 Complete Status Code Classification

Status Code	Category	Retryable	Rationale
2xx Success	-	N/A	No error
400 Bad Request	`PERMANENT_VALIDATION`		Client error - data invalid
401 Unauthorized	`PERMANENT_AUTHORIZATION`		Invalid credentials
403 Forbidden	`PERMANENT_AUTHORIZATION`		Insufficient permissions
404 Not Found	`PERMANENT_NOT_FOUND`		Resource doesn’t exist
405 Method Not Allowed	`PERMANENT_VALIDATION`		Wrong HTTP method
408 Request Timeout	`TRANSIENT_TIMEOUT`		Network/server slow
409 Conflict	`PERMANENT_BUSINESS_RULE`		Business constraint
410 Gone	`PERMANENT_NOT_FOUND`		Resource permanently deleted
422 Unprocessable Entity	`PERMANENT_VALIDATION`		Semantic validation failure
429 Too Many Requests	`TRANSIENT_THROTTLING`		Rate limited
500 Internal Server Error	`TRANSIENT_SERVICE`		Server error
502 Bad Gateway	`TRANSIENT_SERVICE`		Upstream issue
503 Service Unavailable	`TRANSIENT_SERVICE`		Temporarily down
504 Gateway Timeout	`TRANSIENT_SERVICE`		Upstream timeout

5.2 Edge Cases

408 Request Timeout

Classification: TRANSIENT_TIMEOUT Retryable

Rationale: Despite being 4xx, this indicates network/server performance issue rather than client error.

429 Too Many Requests

Classification: TRANSIENT_THROTTLING Retryable

Special Handling: Check for Retry-After header and respect the delay.

409 Conflict

Classification: PERMANENT_BUSINESS_RULE Not retryable

Rationale: Conflicts require state changes or business logic investigation. Automatic retry won’t resolve.

6. Network Error Code Mapping

6.1 Node.js Error Codes

Error Code	Category	Description	Typical Cause
ECONNREFUSED	`TRANSIENT_NETWORK`	Connection refused	Service not listening on port
ETIMEDOUT	`TRANSIENT_NETWORK`	Connection timeout	Network slow or service unresponsive
ENOTFOUND	`TRANSIENT_NETWORK`	DNS lookup failed	DNS issue or hostname typo
ENETUNREACH	`TRANSIENT_NETWORK`	Network unreachable	Routing issue or network down
ECONNRESET	`TRANSIENT_NETWORK`	Connection reset	Server dropped connection
ECONNABORTED	`TRANSIENT_NETWORK`	Connection aborted	Client/server terminated connection
EHOSTUNREACH	`TRANSIENT_NETWORK`	Host unreachable	Firewall or routing issue
EPIPE	`TRANSIENT_NETWORK`	Broken pipe	Connection closed unexpectedly
EAI_AGAIN	`TRANSIENT_NETWORK`	DNS temporary failure	DNS server issue

Note: All network errors are considered transient unless they persist beyond max retries.

7. Database Error Classification

7.1 SQL Error Patterns

Error Pattern	Category	Example Message
Deadlock	`TRANSIENT_RESOURCE`	”Transaction deadlock detected”
Timeout	`TRANSIENT_RESOURCE`	”Lock wait timeout exceeded”
Connection	`TRANSIENT_NETWORK`	”Connection refused”, “Cannot connect”
Constraint	`PERMANENT_VALIDATION`	”Duplicate key”, “Foreign key constraint”
Syntax	`PERMANENT_VALIDATION`	”Syntax error in SQL”
Permission	`PERMANENT_AUTHORIZATION`	”Access denied for user”

8. Service Bus Error Classification

8.1 Service Bus Specific Errors

Error	Category	Description	Action
MessageLockLostError	`TRANSIENT_RESOURCE`	Lock expired	Message auto-released, will retry
SessionLockLostError	`TRANSIENT_RESOURCE`	Session lock expired	Session will be reacquired
MessagingEntityNotFoundError	`PERMANENT_CONFIGURATION`	Queue/topic doesn’t exist	Fix configuration
QuotaExceededException	`TRANSIENT_RESOURCE`	Service Bus quota exceeded	Wait for quota reset
MessageSizeExceededException	`PERMANENT_VALIDATION`	Message too large	Reduce message size or split

9. Decision Matrix

9.1 Retry Decision Table

This comprehensive table shows exactly how the system decides whether to retry or dead-letter a message based on error characteristics.

Error Type	Is MessageProcessingError?	Is Retryable?	Delivery Count	HTTP Status	Error Code	Action	Rationale
Permanent Validation	Yes	No	Any	400, 422	-	DEAD_LETTER	Invalid data will never become valid
Permanent Configuration	Yes	No	Any	-	-	DEAD_LETTER	Requires manual config update
Permanent Not Found	Yes	No	Any	404, 410	-	DEAD_LETTER	Resource doesn’t exist
Permanent Authorization	Yes	No	Any	401, 403	-	DEAD_LETTER	Invalid credentials/permissions
Permanent Business Rule	Yes	No	Any	409	-	DEAD_LETTER	Business constraint violation
Permanent Malformed	Yes	No	Any	-	-	DEAD_LETTER	Message structure invalid
Transient Service	Yes	Yes	< 10	500, 502, 503, 504	-	RETRY	Service may recover
Transient Service	Yes	Yes	≥ 10	500, 502, 503, 504	-	DEAD_LETTER	Exceeded max attempts
Transient Network	Yes	Yes	< 10	-	ECONNREFUSED, ETIMEDOUT	RETRY	Network may recover
Transient Network	Yes	Yes	≥ 10	-	ECONNREFUSED, ETIMEDOUT	DEAD_LETTER	Exceeded max attempts
Transient Throttling	Yes	Yes	< 10	429	-	RETRY	Rate limit will reset
Transient Throttling	Yes	Yes	≥ 10	429	-	DEAD_LETTER	Exceeded max attempts
Transient Resource	Yes	Yes	< 10	-	Deadlock	RETRY	Contention will resolve
Transient Resource	Yes	Yes	≥ 10	-	Deadlock	DEAD_LETTER	Exceeded max attempts
Unclassified HTTP 5xx	No	N/A	< 10	500, 502, 503, 504	-	RETRY	Treat as transient
Unclassified HTTP 5xx	No	N/A	≥ 10	500, 502, 503, 504	-	DEAD_LETTER	Exceeded max attempts
Unclassified HTTP 4xx	No	N/A	Any	400, 401, 403, 404	-	DEAD_LETTER	Treat as permanent
Unclassified Network	No	N/A	< 10	-	ECONNREFUSED, etc.	RETRY	Treat as transient
Unclassified Network	No	N/A	≥ 10	-	ECONNREFUSED, etc.	DEAD_LETTER	Exceeded max attempts
Unclassified Unknown	No	N/A	< 10	-	-	RETRY	Conservative approach
Unclassified Unknown	No	N/A	≥ 10	-	-	DEAD_LETTER	Exceeded max attempts

9.2 Quick Reference Decision Tree

Use PERMANENT errors when retry will NEVER succeed

without manual intervention (code, config, or data change)

PERMANENT_VALIDATION

Missing required fields
Invalid data format (email, date, etc.)
Wrong data type (string vs number)
Value out of acceptable range
Schema validation failure
HTTP: 400 (Bad Request), 422 (Unprocessable Entity)

PERMANENT_CONFIGURATION

No extractors configured
Missing environment variable
Invalid connection string
Feature flag disabled
Service misconfigured

PERMANENT_NOT_FOUND

Application doesn’t exist
File deleted from storage
Extractor ID invalid
Stale reference to removed entity
HTTP: 404 (Not Found), 410 (Gone)

PERMANENT_AUTHORIZATION

Invalid API key
Expired access token
Insufficient permissions
Certificate expired
Service account disabled
HTTP: 401 (Unauthorized), 403 (Forbidden)

PERMANENT_BUSINESS_RULE

Duplicate operation attempt
Invalid state transition
Workflow already completed
Business constraint violation
HTTP: 409 (Conflict)

PERMANENT_MALFORMED

Message body is null/undefined
Unparseable JSON
Corrupted message data
Wrong message type on queue
Encoding issues

10. Configuration Parameters

10.1 Recommended Settings

Parameter	Value	Rationale
maxDeliveryCount	10	Balances retry opportunity vs resource waste
lockDuration	5 minutes	Sufficient for typical processing, prevents excessive locking
defaultMessageTTL	7 days	Allows reasonable time for manual DLQ processing
enableDeadLetteringOnExpiration	true	Preserve expired messages for investigation
treatUnknownAsTransient	false	Conservative approach - dead-letter for investigation

10.2 Service Bus Subscription Configuration

subscription:
  name: "extractor-accepted-file"
  maxDeliveryCount: 10
  lockDuration: "PT5M"  # ISO 8601: 5 minutes
  defaultMessageTimeToLive: "P7D"  # ISO 8601: 7 days
  deadLetteringOnMessageExpiration: true
  enableBatchedOperations: true
  maxSizeInMegabytes: 1024

11. Observability Strategy

11.1 Structured Logging Schema

{
  "timestamp": "ISO 8601",
  "level": "error|warn|info",
  "msg": "Human-readable message",
  "messageId": "Service Bus message ID",
  "source": "Topic name",
  "subscription": "Subscription name",
  "deliveryCount": 1,
  "category": "ERROR_CATEGORY",
  "retryable": true|false,
  "action": "RETRY|DEAD_LETTER",
  "error": "Error message",
  "metadata": {
    "applicationId": "...",
    "fileId": "...",
    "status": 500,
    "url": "..."
  }
}

11.2 Log Levels by Category

Category	Log Level	Rationale
`TRANSIENT_*`	`warn`	Expected transient failures
`PERMANENT_*`	`error`	Unexpected issues requiring investigation
`UNCLASSIFIED`	`error`	Unknown errors need investigation
Successful retry	`info`	Audit trail

12. Open Questions

#	Question	Options	Recommendation
1	Should we retry unclassified errors?	A) Yes (conservative) B) No (strict)	A - Be conservative until we understand error patterns
2	Max delivery count?	A) 5 B) 10 C) 20	B - 10 attempts balances reliability vs resource usage
3	Should `PERMANENT_NOT_FOUND` always dead-letter?	A) Yes B) Retry 1-2 times (eventual consistency)	A - 404s rarely resolve themselves in our architecture
4	Log sensitive data in metadata?	A) Yes (full context) B) No (sanitize)	B - Sanitize PII, credentials
5	Implement exponential backoff for throttling?	A) Phase 1 B) Phase 2	B - Keep simple for initial release

Spike: Error Classification Design (User Story 762531)

1. Summary

Key Outcomes

2. Problem Statement

Current State

Impact

Design Goals

Primary Goals

3. Error Classification Framework

3.1 Classification Principles

Transient Errors

Permanent Errors

4. Error Taxonomy

4.1 Transient Error Categories

TRANSIENT_SERVICE

TRANSIENT_NETWORK

TRANSIENT_THROTTLING

TRANSIENT_RESOURCE

4.2 Permanent Error Categories

PERMANENT_VALIDATION

PERMANENT_CONFIGURATION

PERMANENT_NOT_FOUND

PERMANENT_AUTHORIZATION

PERMANENT_BUSINESS_RULE

PERMANENT_MALFORMED

5. HTTP Status Code Mapping

5.1 Complete Status Code Classification

5.2 Edge Cases

408 Request Timeout

429 Too Many Requests

409 Conflict

6. Network Error Code Mapping

6.1 Node.js Error Codes

7. Database Error Classification

7.1 SQL Error Patterns

8. Service Bus Error Classification

8.1 Service Bus Specific Errors

9. Decision Matrix

9.1 Retry Decision Table

9.2 Quick Reference Decision Tree

PERMANENT_VALIDATION

PERMANENT_CONFIGURATION

PERMANENT_NOT_FOUND

PERMANENT_AUTHORIZATION

PERMANENT_BUSINESS_RULE

PERMANENT_MALFORMED

10. Configuration Parameters

10.1 Recommended Settings

10.2 Service Bus Subscription Configuration

11. Observability Strategy

11.1 Structured Logging Schema

11.2 Log Levels by Category

12. Open Questions

References

1. Azure Service Bus Documentation

2. Industry Standards