Polling and HTTP Failures

Summary

This spike defines a standardized error taxonomy and decision rules for polling and outbound HTTP calls across Fluxx. It focuses on transient vs. permanent failures, retry behavior, and observability for HTTP-based workflows.

Goals

Establish a clear error taxonomy for polling/HTTP failures.
Define decision rules for retry or fail-fast.
Ensure the approach is consistent, configurable, observable, debuggable, and testable.

Current State (Reference)

Polling and extraction workflows are implemented in src/services/extractionJobPollingWorker.ts and src/services/extractors/documentAIExtractor.ts.
The polling flow treats certain errors as “processing” and continues retrying, but does not standardize error types or HTTP status interpretation.

Proposed Error Taxonomy (HTTP/Polling Focus)

A standardized HttpProcessingError classification is attached to all HTTP and polling failures. The classification must be consistent across services and expose a small, stable set of types:

Error Types

Transient: Temporary failures likely to succeed on retry.
Permanent: Failures that will not succeed on retry without code/data changes.
Unknown: Cannot be determined; default behavior is limited retry, then fail.

HTTP Status Classification (Baseline)

Category	Examples	Suggested Action
Transient	Network timeouts, DNS errors, HTTP Status Codes: 408, 429, 500, 502, 503, 504, 507, 509 (more can be added, except 501/505)	Retry
Permanent	Schema validation failures, missing required fields, unsupported document types, example HTTP Status code: 400, 401, 403, 404, 405, 406, 409, 410, 411, 412, 413, 414, 415, 422, 424, 426, 428, 431, 451, 501, 505 (more can be added)	Fail fast
Unknown	Unhandled exceptions, unexpected nulls. Any unexpected status or non-HTTP errors without clear mapping (e.g., thrown exceptions without status).	Retry

Classification Contract

A shared, minimal error contract should be used by all polling/HTTP callers:

type: Transient | Permanent | Unknown
action: Retry | Fail
message: string (concise summary; if possible, to include HTTP Code)

Configuration

The classifier should be configurable per service (and optionally per handler):

transientHttpStatusCodes: Default list for retryable HTTP status codes

Observability & Debuggability

Structured logging for all polling outcomes with fields:
- jobId, fileId, extractorId, classification.type, action.

Testability

Minimum test coverage should include:

Classifier unit tests for each status mapping.
Polling tests asserting correct behavior:
- Transient -> retry
- Permanent -> fail
- Unknown -> retry until max attempt then fail

Alignment with Framework

The polling workflow in src/services/extractionJobPollingWorker.ts is the preferred integration point for standardized behavior.
Extractor implementations (e.g., src/services/extractors/documentAIExtractor.ts) should throw classified errors or wrap unknowns with the shared contract.

Possible User Story

Introduce shared HttpProcessingError and classifier utility.
Update polling workflow to apply classifier actions.
Add configuration wiring for retryable status codes and thresholds.
Add unit tests for decision logic and polling integration.
Instrument logs and metrics for classification outcomes.