> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://cla-demo.postmansolutions.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://cla-demo.postmansolutions.com/_mcp/server.

# Polling and HTTP Failures

## Summary

This spike defines a standardized error taxonomy and decision rules for polling and outbound HTTP calls across Fluxx. It focuses on transient vs. permanent failures, retry behavior, and observability for HTTP-based workflows.

## Goals

* Establish a clear error taxonomy for polling/HTTP failures.
* Define decision rules for retry or fail-fast.
* Ensure the approach is consistent, configurable, observable, debuggable, and testable.

## Current State (Reference)

* Polling and extraction workflows are implemented in src/services/extractionJobPollingWorker.ts and src/services/extractors/documentAIExtractor.ts.
* The polling flow treats certain errors as “processing” and continues retrying, but does not standardize error types or HTTP status interpretation.

## Proposed Error Taxonomy (HTTP/Polling Focus)

A standardized `HttpProcessingError` classification is attached to all HTTP and polling failures. The classification must be consistent across services and expose a small, stable set of types:

### Error Types

* **Transient**: Temporary failures likely to succeed on retry.
* **Permanent**: Failures that will not succeed on retry without code/data changes.
* **Unknown**: Cannot be determined; default behavior is limited retry, then fail.

### HTTP Status Classification (Baseline)

| Category  | Examples                                                                                                                                                                                                                               | Suggested Action |
| --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- |
| Transient | Network timeouts, DNS errors, HTTP Status Codes: 408, 429, 500, 502, 503, 504, 507, 509 (more can be added, except 501/505)                                                                                                            | Retry            |
| Permanent | Schema validation failures, missing required fields, unsupported document types, example HTTP Status code: 400, 401, 403, 404, 405, 406, 409, 410, 411, 412, 413, 414, 415, 422, 424, 426, 428, 431, 451, 501, 505 (more can be added) | Fail fast        |
| Unknown   | Unhandled exceptions, unexpected nulls. Any unexpected status or non-HTTP errors without clear mapping (e.g., thrown exceptions without status).                                                                                       | Retry            |

## Classification Contract

A shared, minimal error contract should be used by all polling/HTTP callers:

* `type`: `Transient` | `Permanent` | `Unknown`
* `action`: `Retry` | `Fail`
* `message`: string (concise summary; if possible, to include HTTP Code)

## Configuration

The classifier should be configurable per service (and optionally per handler):

* `transientHttpStatusCodes`: Default list for retryable HTTP status codes

## Observability & Debuggability

* **Structured logging** for all polling outcomes with fields:
  * `jobId`, `fileId`, `extractorId`, `classification.type`, `action`.

## Testability

Minimum test coverage should include:

* Classifier unit tests for each status mapping.
* Polling tests asserting correct behavior:
  * Transient -> retry
  * Permanent -> fail
  * Unknown -> retry until max attempt then fail

## Alignment with Framework

* The polling workflow in src/services/extractionJobPollingWorker.ts is the preferred integration point for standardized behavior.
* Extractor implementations (e.g., src/services/extractors/documentAIExtractor.ts) should throw classified errors or wrap unknowns with the shared contract.

## Possible User Story

* Introduce shared `HttpProcessingError` and classifier utility.
* Update polling workflow to apply classifier actions.
* Add configuration wiring for retryable status codes and thresholds.
* Add unit tests for decision logic and polling integration.
* Instrument logs and metrics for classification outcomes.