> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://cla-demo.postmansolutions.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://cla-demo.postmansolutions.com/_mcp/server.

# Service Bus Message Error Classification Strategy

## **Spike: Error Classification Design (User Story 762531)**

## **1. Summary**

This document defines a **standardized error classification strategy** for Azure Service Bus message processing across Fluxx microservices. The strategy distinguishes between **transient failures** (temporary, retriable) and **permanent failures** (non-retriable) to optimize system reliability and operational efficiency.

### **Key Outcomes**

* Clear taxonomy of message processing errors
* Decision rules for retry vs dead-letter actions
* Improved system reliability and reduced operational noise
* Ensure the approach is consistent, configurable, observable, debuggable, and testable.

---

## **2. Problem Statement**

### **Current State**

Currently in `fluxx-extractor` (and likely other services):

* All message processing errors are treated the same way
* Failed messages are **always abandoned** for retry, regardless of error type
* No distinction between transient (network timeout) vs permanent (invalid data) failures
* Validation errors retry indefinitely until max delivery count
* Dead-letter queues contain mix of retriable and non-retriable messages
* Limited error metadata makes troubleshooting difficult

### **Impact**

* Wasted Service Bus resources on non-retriable errors
* Delayed identification of systematic issues (e.g., missing configuration)
* Poor operational visibility into failure reasons
* Inconsistent behavior across microservices

---

## Design Goals

### **Primary Goals**

1. **Clear Error Taxonomy** - Establish standardized categories for all message processing failures
2. **Intelligent Retry Logic** - Apply conditional retry based on error classification
3. **Operational Correctness** - Dead-letter permanent failures immediately, retry transient failures
4. **Cross-Service Consistency** - Same approach in all Fluxx microservices
5. **Enhanced Observability** - Rich metadata and structured logging for debugging

## **3. Error Classification Framework**

### **3.1 Classification Principles**

#### **Transient Errors**

Errors that are **temporary** and likely to **succeed on retry** without any code or data changes.

**Characteristics:**

* External dependency temporarily unavailable
* Network connectivity issues
* Resource contention that resolves over time
* Rate limiting that expires

**Strategy:** Retry with exponential backoff

#### **Permanent Errors**

Errors that will **never succeed** regardless of retries without intervention.

**Characteristics:**

* Invalid input data
* Missing configuration
* Business rule violations
* Resources that don't exist

**Strategy:** Immediate dead-letter for manual investigation

---

## **4. Error Taxonomy**

### **4.1 Transient Error Categories**

#### **TRANSIENT\_SERVICE**

External service temporarily unavailable

| Attribute             | Value                                                                                           |
| --------------------- | ----------------------------------------------------------------------------------------------- |
| **Retryable**         | Yes                                                                                             |
| **Max Retries**       | 10 attempts                                                                                     |
| **Retry Delay**       | Service Bus default (lock duration)                                                             |
| **HTTP Status Codes** | 500, 502, 503, 504                                                                              |
| **Common Causes**     | - Service deployment in progress - Temporary outage - Circuit breaker open - Service restarting |
| **Resolution**        | Automatic - wait for service recovery                                                           |
| **Example**           | Registration API returns 503 during deployment                                                  |

**Rationale:** Services typically recover within minutes. Retry allows automatic recovery without manual intervention.

---

#### **TRANSIENT\_NETWORK**

Network connectivity issues

| Attribute         | Value                                                                                    |
| ----------------- | ---------------------------------------------------------------------------------------- |
| **Retryable**     | Yes                                                                                      |
| **Max Retries**   | 10 attempts                                                                              |
| **Error Codes**   | ECONNREFUSED, ETIMEDOUT, ENOTFOUND, ENETUNREACH, ECONNRESET                              |
| **Common Causes** | - DNS resolution failure - Network partition - Firewall blocking - Service not listening |
| **Resolution**    | Automatic - network typically recovers                                                   |
| **Example**       | Connection refused when calling external API                                             |

**Rationale:** Network issues are often transient. DNS, routing, and firewall issues typically resolve quickly.

---

#### **TRANSIENT\_THROTTLING**

Rate limiting by external service

| Attribute             | Value                                                                      |
| --------------------- | -------------------------------------------------------------------------- |
| **Retryable**         | Yes                                                                        |
| **Max Retries**       | 10 attempts                                                                |
| **Retry Strategy**    | Honor `Retry-After` header if present                                      |
| **HTTP Status Codes** | 429                                                                        |
| **Common Causes**     | - API quota exceeded - Too many requests per second - Burst limit exceeded |
| **Resolution**        | Automatic - wait for quota reset                                           |
| **Example**           | Document AI API returns 429 with `Retry-After: 60`                         |

**Rationale:** Rate limits typically reset automatically. Respecting `Retry-After` prevents additional throttling.

**Future Enhancement:** Implement exponential backoff or circuit breaker pattern.

---

#### **TRANSIENT\_RESOURCE**

Resource contention that resolves over time

| Attribute         | Value                                                                                          |
| ----------------- | ---------------------------------------------------------------------------------------------- |
| **Retryable**     | Yes                                                                                            |
| **Max Retries**   | 10 attempts                                                                                    |
| **Common Causes** | - Database deadlock - Lock timeout - Service Bus message lock lost - Connection pool exhausted |
| **Resolution**    | Automatic - contention resolves as resources free up                                           |
| **Example**       | Database deadlock during concurrent job updates                                                |

**Rationale:** Resource contention is temporary by nature. Deadlocks resolve automatically, and pools refill.

---

### **4.2 Permanent Error Categories**

#### **PERMANENT\_VALIDATION**

Invalid message data or structure

| Attribute             | Value                                                                                       |
| --------------------- | ------------------------------------------------------------------------------------------- |
| **Retryable**         | No                                                                                          |
| **Action**            | Immediate dead-letter                                                                       |
| **HTTP Status Codes** | 400                                                                                         |
| **Common Causes**     | - Missing required fields - Invalid data format - Type mismatch - Schema validation failure |
| **Resolution**        | Manual - fix data source, update message, resubmit                                          |
| **Example**           | Message missing `applicationId` field                                                       |

**Rationale:** Invalid data will never become valid through retry. Immediate dead-lettering allows quick identification and fix.

**Dead-Letter Metadata:**

```json
{
  "deadLetterReason": "PERMANENT_VALIDATION",
  "deadLetterErrorDescription": "[Attempt 1] Missing required field: applicationId",
  "originalMessage": { /* message body */ }
}
```

---

#### **PERMANENT\_CONFIGURATION**

Missing or invalid configuration

| Attribute         | Value                                                                                     |
| ----------------- | ----------------------------------------------------------------------------------------- |
| **Retryable**     | No                                                                                        |
| **Action**        | Immediate dead-letter                                                                     |
| **Common Causes** | - No extractors configured - Missing environment variables - Invalid configuration values |
| **Resolution**    | Manual - update configuration, resubmit message                                           |
| **Example**       | No extractors configured for `applicationId`                                              |

**Rationale:** Configuration issues require deployment or database changes. Retry wastes resources and delays detection.

**Operational Response:**

1. **Alert:** Page on-call immediately (high severity)
2. **Investigate:** Check configuration in database/environment
3. **Fix:** Add missing configuration
4. **Reprocess:** Resubmit dead-lettered messages

---

#### **PERMANENT\_NOT\_FOUND**

Resource doesn't exist

| Attribute             | Value                                                                                                 |
| --------------------- | ----------------------------------------------------------------------------------------------------- |
| **Retryable**         | No                                                                                                    |
| **Action**            | Immediate dead-letter                                                                                 |
| **HTTP Status Codes** | 404                                                                                                   |
| **Common Causes**     | - Application ID doesn't exist - File deleted from blob storage - Extractor removed - Stale reference |
| **Resolution**        | Manual - create resource or discard obsolete message                                                  |
| **Example**           | Registration API returns 404 for `applicationId`                                                      |

**Rationale:** Resources don't spontaneously appear. Retry wastes resources checking for non-existent entities.

**Decision Tree:**

```
404 Error
    ↓
Is resource expected to exist?
    ↓
YES → Create resource → Resubmit
    ↓
NO → Discard message (obsolete data)
```

---

#### **PERMANENT\_AUTHORIZATION**

Authentication or permission failure

| Attribute             | Value                                                                              |
| --------------------- | ---------------------------------------------------------------------------------- |
| **Retryable**         | No                                                                                 |
| **Action**            | Immediate dead-letter                                                              |
| **HTTP Status Codes** | 401, 403                                                                           |
| **Common Causes**     | - Invalid API key - Expired token - Insufficient permissions - Certificate expired |
| **Resolution**        | Manual - update credentials, grant permissions                                     |
| **Example**           | External API returns 401 with invalid API key                                      |

**Rationale:** Credentials don't fix themselves. Immediate dead-lettering highlights security/access issues.

**Security Consideration:** Don't log credentials in error metadata.

---

#### **PERMANENT\_BUSINESS\_RULE**

Business logic prevents processing

| Attribute             | Value                                                                                           |
| --------------------- | ----------------------------------------------------------------------------------------------- |
| **Retryable**         | No                                                                                              |
| **Action**            | Immediate dead-letter                                                                           |
| **HTTP Status Codes** | 409                                                                                             |
| **Common Causes**     | - Duplicate entry - Invalid state transition - Constraint violation - Business invariant broken |
| **Resolution**        | Manual - investigate business logic, fix state, decide on reprocess                             |
| **Example**           | Attempting to reprocess already-completed job                                                   |

**Rationale:** Business rules are deterministic. Retry won't change the outcome without state changes.

---

#### **PERMANENT\_MALFORMED**

Message structure invalid

| Attribute         | Value                                                                              |
| ----------------- | ---------------------------------------------------------------------------------- |
| **Retryable**     | No                                                                                 |
| **Action**        | Immediate dead-letter                                                              |
| **Common Causes** | - Missing message body - Unparseable JSON - Corrupted message - Wrong message type |
| **Resolution**    | Manual - investigate producer, fix serialization, discard corrupt messages         |
| **Example**       | Service Bus message with null body                                                 |

**Rationale:** Malformed messages indicate producer bugs. Retry wastes lock time on unparseable data.

---

## **5. HTTP Status Code Mapping**

### **5.1 Complete Status Code Classification**

| Status Code                   | Category                  | Retryable | Rationale                    |
| ----------------------------- | ------------------------- | --------- | ---------------------------- |
| **2xx Success**               | -                         | N/A       | No error                     |
| **400 Bad Request**           | `PERMANENT_VALIDATION`    |           | Client error - data invalid  |
| **401 Unauthorized**          | `PERMANENT_AUTHORIZATION` |           | Invalid credentials          |
| **403 Forbidden**             | `PERMANENT_AUTHORIZATION` |           | Insufficient permissions     |
| **404 Not Found**             | `PERMANENT_NOT_FOUND`     |           | Resource doesn't exist       |
| **405 Method Not Allowed**    | `PERMANENT_VALIDATION`    |           | Wrong HTTP method            |
| **408 Request Timeout**       | `TRANSIENT_TIMEOUT`       |           | Network/server slow          |
| **409 Conflict**              | `PERMANENT_BUSINESS_RULE` |           | Business constraint          |
| **410 Gone**                  | `PERMANENT_NOT_FOUND`     |           | Resource permanently deleted |
| **422 Unprocessable Entity**  | `PERMANENT_VALIDATION`    |           | Semantic validation failure  |
| **429 Too Many Requests**     | `TRANSIENT_THROTTLING`    |           | Rate limited                 |
| **500 Internal Server Error** | `TRANSIENT_SERVICE`       |           | Server error                 |
| **502 Bad Gateway**           | `TRANSIENT_SERVICE`       |           | Upstream issue               |
| **503 Service Unavailable**   | `TRANSIENT_SERVICE`       |           | Temporarily down             |
| **504 Gateway Timeout**       | `TRANSIENT_SERVICE`       |           | Upstream timeout             |

### **5.2 Edge Cases**

#### **408 Request Timeout**

**Classification:** `TRANSIENT_TIMEOUT` Retryable

**Rationale:** Despite being 4xx, this indicates network/server performance issue rather than client error.

#### **429 Too Many Requests**

**Classification:** `TRANSIENT_THROTTLING` Retryable

**Special Handling:** Check for `Retry-After` header and respect the delay.

#### **409 Conflict**

**Classification:** `PERMANENT_BUSINESS_RULE` Not retryable

**Rationale:** Conflicts require state changes or business logic investigation. Automatic retry won't resolve.

---

## **6. Network Error Code Mapping**

### **6.1 Node.js Error Codes**

| Error Code       | Category            | Retryable | Description           | Typical Cause                        |
| ---------------- | ------------------- | --------- | --------------------- | ------------------------------------ |
| **ECONNREFUSED** | `TRANSIENT_NETWORK` |           | Connection refused    | Service not listening on port        |
| **ETIMEDOUT**    | `TRANSIENT_NETWORK` |           | Connection timeout    | Network slow or service unresponsive |
| **ENOTFOUND**    | `TRANSIENT_NETWORK` |           | DNS lookup failed     | DNS issue or hostname typo           |
| **ENETUNREACH**  | `TRANSIENT_NETWORK` |           | Network unreachable   | Routing issue or network down        |
| **ECONNRESET**   | `TRANSIENT_NETWORK` |           | Connection reset      | Server dropped connection            |
| **ECONNABORTED** | `TRANSIENT_NETWORK` |           | Connection aborted    | Client/server terminated connection  |
| **EHOSTUNREACH** | `TRANSIENT_NETWORK` |           | Host unreachable      | Firewall or routing issue            |
| **EPIPE**        | `TRANSIENT_NETWORK` |           | Broken pipe           | Connection closed unexpectedly       |
| **EAI\_AGAIN**   | `TRANSIENT_NETWORK` |           | DNS temporary failure | DNS server issue                     |

**Note:** All network errors are considered transient unless they persist beyond max retries.

---

## **7. Database Error Classification**

### **7.1 SQL Error Patterns**

| Error Pattern  | Category                  | Retryable | Example Message                           |
| -------------- | ------------------------- | --------- | ----------------------------------------- |
| **Deadlock**   | `TRANSIENT_RESOURCE`      |           | "Transaction deadlock detected"           |
| **Timeout**    | `TRANSIENT_RESOURCE`      |           | "Lock wait timeout exceeded"              |
| **Connection** | `TRANSIENT_NETWORK`       |           | "Connection refused", "Cannot connect"    |
| **Constraint** | `PERMANENT_VALIDATION`    |           | "Duplicate key", "Foreign key constraint" |
| **Syntax**     | `PERMANENT_VALIDATION`    |           | "Syntax error in SQL"                     |
| **Permission** | `PERMANENT_AUTHORIZATION` |           | "Access denied for user"                  |

---

## **8. Service Bus Error Classification**

### **8.1 Service Bus Specific Errors**

| Error                            | Category                  | Retryable | Description                | Action                            |
| -------------------------------- | ------------------------- | --------- | -------------------------- | --------------------------------- |
| **MessageLockLostError**         | `TRANSIENT_RESOURCE`      |           | Lock expired               | Message auto-released, will retry |
| **SessionLockLostError**         | `TRANSIENT_RESOURCE`      |           | Session lock expired       | Session will be reacquired        |
| **MessagingEntityNotFoundError** | `PERMANENT_CONFIGURATION` |           | Queue/topic doesn't exist  | Fix configuration                 |
| **QuotaExceededException**       | `TRANSIENT_RESOURCE`      |           | Service Bus quota exceeded | Wait for quota reset              |
| **MessageSizeExceededException** | `PERMANENT_VALIDATION`    |           | Message too large          | Reduce message size or split      |

---

## **9. Decision Matrix**

### **9.1 Retry Decision Table**

This comprehensive table shows exactly how the system decides whether to retry or dead-letter a message based on error characteristics.

| Error Type                  | Is MessageProcessingError? | Is Retryable? | Delivery Count | HTTP Status        | Error Code              | Action           | Rationale                            |
| --------------------------- | -------------------------- | ------------- | -------------- | ------------------ | ----------------------- | ---------------- | ------------------------------------ |
| **Permanent Validation**    | Yes                        | No            | Any            | 400, 422           | -                       | **DEAD\_LETTER** | Invalid data will never become valid |
| **Permanent Configuration** | Yes                        | No            | Any            | -                  | -                       | **DEAD\_LETTER** | Requires manual config update        |
| **Permanent Not Found**     | Yes                        | No            | Any            | 404, 410           | -                       | **DEAD\_LETTER** | Resource doesn't exist               |
| **Permanent Authorization** | Yes                        | No            | Any            | 401, 403           | -                       | **DEAD\_LETTER** | Invalid credentials/permissions      |
| **Permanent Business Rule** | Yes                        | No            | Any            | 409                | -                       | **DEAD\_LETTER** | Business constraint violation        |
| **Permanent Malformed**     | Yes                        | No            | Any            | -                  | -                       | **DEAD\_LETTER** | Message structure invalid            |
| **Transient Service**       | Yes                        | Yes           | \< 10          | 500, 502, 503, 504 | -                       | **RETRY**        | Service may recover                  |
| **Transient Service**       | Yes                        | Yes           | ≥ 10           | 500, 502, 503, 504 | -                       | **DEAD\_LETTER** | Exceeded max attempts                |
| **Transient Network**       | Yes                        | Yes           | \< 10          | -                  | ECONNREFUSED, ETIMEDOUT | **RETRY**        | Network may recover                  |
| **Transient Network**       | Yes                        | Yes           | ≥ 10           | -                  | ECONNREFUSED, ETIMEDOUT | **DEAD\_LETTER** | Exceeded max attempts                |
| **Transient Throttling**    | Yes                        | Yes           | \< 10          | 429                | -                       | **RETRY**        | Rate limit will reset                |
| **Transient Throttling**    | Yes                        | Yes           | ≥ 10           | 429                | -                       | **DEAD\_LETTER** | Exceeded max attempts                |
| **Transient Resource**      | Yes                        | Yes           | \< 10          | -                  | Deadlock                | **RETRY**        | Contention will resolve              |
| **Transient Resource**      | Yes                        | Yes           | ≥ 10           | -                  | Deadlock                | **DEAD\_LETTER** | Exceeded max attempts                |
| **Unclassified HTTP 5xx**   | No                         | N/A           | \< 10          | 500, 502, 503, 504 | -                       | **RETRY**        | Treat as transient                   |
| **Unclassified HTTP 5xx**   | No                         | N/A           | ≥ 10           | 500, 502, 503, 504 | -                       | **DEAD\_LETTER** | Exceeded max attempts                |
| **Unclassified HTTP 4xx**   | No                         | N/A           | Any            | 400, 401, 403, 404 | -                       | **DEAD\_LETTER** | Treat as permanent                   |
| **Unclassified Network**    | No                         | N/A           | \< 10          | -                  | ECONNREFUSED, etc.      | **RETRY**        | Treat as transient                   |
| **Unclassified Network**    | No                         | N/A           | ≥ 10           | -                  | ECONNREFUSED, etc.      | **DEAD\_LETTER** | Exceeded max attempts                |
| **Unclassified Unknown**    | No                         | N/A           | \< 10          | -                  | -                       | **RETRY**        | Conservative approach                |
| **Unclassified Unknown**    | No                         | N/A           | ≥ 10           | -                  | -                       | **DEAD\_LETTER** | Exceeded max attempts                |

### **9.2 Quick Reference Decision Tree**

> **Use PERMANENT errors when retry will NEVER succeed**

> without manual intervention (code, config, or data change)

### PERMANENT\_VALIDATION

* Missing required fields

* Invalid data format (email, date, etc.)

* Wrong data type (string vs number)

* Value out of acceptable range

* Schema validation failure

* HTTP: 400 (Bad Request), 422 (Unprocessable Entity)

### PERMANENT\_CONFIGURATION

* No extractors configured

* Missing environment variable

* Invalid connection string

* Feature flag disabled

* Service misconfigured

### PERMANENT\_NOT\_FOUND

* Application doesn't exist

* File deleted from storage

* Extractor ID invalid

* Stale reference to removed entity

* HTTP: 404 (Not Found), 410 (Gone)

### PERMANENT\_AUTHORIZATION

* Invalid API key

* Expired access token

* Insufficient permissions

* Certificate expired

* Service account disabled

* HTTP: 401 (Unauthorized), 403 (Forbidden)

### PERMANENT\_BUSINESS\_RULE

* Duplicate operation attempt

* Invalid state transition

* Workflow already completed

* Business constraint violation

* HTTP: 409 (Conflict)

### PERMANENT\_MALFORMED

* Message body is null/undefined

* Unparseable JSON

* Corrupted message data

* Wrong message type on queue

* Encoding issues

---

## **10. Configuration Parameters**

### **10.1 Recommended Settings**

| Parameter                           | Value     | Rationale                                                     |
| ----------------------------------- | --------- | ------------------------------------------------------------- |
| **maxDeliveryCount**                | 10        | Balances retry opportunity vs resource waste                  |
| **lockDuration**                    | 5 minutes | Sufficient for typical processing, prevents excessive locking |
| **defaultMessageTTL**               | 7 days    | Allows reasonable time for manual DLQ processing              |
| **enableDeadLetteringOnExpiration** | true      | Preserve expired messages for investigation                   |
| **treatUnknownAsTransient**         | false     | Conservative approach - dead-letter for investigation         |

### **10.2 Service Bus Subscription Configuration**

```yaml
subscription:
  name: "extractor-accepted-file"
  maxDeliveryCount: 10
  lockDuration: "PT5M"  # ISO 8601: 5 minutes
  defaultMessageTimeToLive: "P7D"  # ISO 8601: 7 days
  deadLetteringOnMessageExpiration: true
  enableBatchedOperations: true
  maxSizeInMegabytes: 1024
```

---

## **11. Observability Strategy**

### **11.1 Structured Logging Schema**

```json
{
  "timestamp": "ISO 8601",
  "level": "error|warn|info",
  "msg": "Human-readable message",
  "messageId": "Service Bus message ID",
  "source": "Topic name",
  "subscription": "Subscription name",
  "deliveryCount": 1,
  "category": "ERROR_CATEGORY",
  "retryable": true|false,
  "action": "RETRY|DEAD_LETTER",
  "error": "Error message",
  "metadata": {
    "applicationId": "...",
    "fileId": "...",
    "status": 500,
    "url": "..."
  }
}
```

### **11.2 Log Levels by Category**

| Category         | Log Level | Rationale                                 |
| ---------------- | --------- | ----------------------------------------- |
| `TRANSIENT_*`    | `warn`    | Expected transient failures               |
| `PERMANENT_*`    | `error`   | Unexpected issues requiring investigation |
| `UNCLASSIFIED`   | `error`   | Unknown errors need investigation         |
| Successful retry | `info`    | Audit trail                               |

---

## **12. Open Questions**

| # | Question                                         | Options                                          | Recommendation                                             |
| - | ------------------------------------------------ | ------------------------------------------------ | ---------------------------------------------------------- |
| 1 | Should we retry unclassified errors?             | A) Yes (conservative) B) No (strict)             | **A** - Be conservative until we understand error patterns |
| 2 | Max delivery count?                              | A) 5 B) 10 C) 20                                 | **B** - 10 attempts balances reliability vs resource usage |
| 3 | Should `PERMANENT_NOT_FOUND` always dead-letter? | A) Yes B) Retry 1-2 times (eventual consistency) | **A** - 404s rarely resolve themselves in our architecture |
| 4 | Log sensitive data in metadata?                  | A) Yes (full context) B) No (sanitize)           | **B** - Sanitize PII, credentials                          |
| 5 | Implement exponential backoff for throttling?    | A) Phase 1 B) Phase 2                            | **B** - Keep simple for initial release                    |

---

## References

### **1. Azure Service Bus Documentation**

* [Dead-Letter Queues](https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dead-letter-queues)
* [Message Sessions](https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-sessions)
* [Retry Patterns](https://learn.microsoft.com/en-us/azure/architecture/patterns/retry)

### **2. Industry Standards**

* [RFC 7807 - Problem Details for HTTP APIs](https://www.rfc-editor.org/rfc/rfc7807)
* [HTTP Status Codes](https://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml)
* [Node.js Error Codes](https://nodejs.org/api/errors.html#nodejs-error-codes)

---