Skip to navigation
Research and Spikes

Common Library for Standardization Topic and Subscription

Design a common pub/sub library that standardizes how topics, subscriptions, and related configurations are defined and consumed across services, prior to implementing or refactoring individual service setups. User Story 762533

Goals

  • Define a shared abstraction for pub/sub setup  

  • Standardize topic and subscription configuration  

  • Reduce duplication and configuration inconsistencies  

  • Improve discoverability and correctness of messaging setups  

Current State

We already have a shared TypeScript library at:

The library provides a shared abstraction over service bus, exposing the following core APIs but not limited to the following:

For more details:  

https://github.com/CLADevOps/poc-lib-autoextract-platform-pubsub/blob/main/README.md

Need Improvements

  • There is no explicit way to abandon a message when there is a transient error.  

  • There is no explicit way to move a message to the dead-letter queue (DLQ) when there is a permanent error.  

As a result, each service may implement error handling differently, reducing consistency and observability.  

Proposed Improvements (high level)

  • Extend the existing subscriber handling in the common pub/sub library with explicit message outcomes.  

  • Define a simple outcome model for message processing:

    • success → complete message  

    • transient_error → abandon message (allow retry)  

    • permanent_error → dead-letter message (DLQ)  

  • Keep service code focused on business logic while the library performs the corresponding Service Bus action.  

  • Ensure each path is logged consistently as part of discoverability and correctness (success, abandon, DLQ) for easier operations and troubleshooting.  

  • Keep this behavior optional and configurable so existing consumers can migrate safely.

  • By leveraging from a pubsub library, this will avoid code duplication, inconsistent topic/subscription implementation and minimize risk of misconfiguration.

  • (NICE-To-HAVE) Naming convention: (Topic and Subscription) Think of Topic = What happened; and Think of Subscription = Who cares on this topic. Having a standard naming convention will improve clarity on identifying which topic/subscription belongs to a specific service.

    • Topic name, suggested format <system>.<service name>.<action>
      • Example:
        • fluxx.file-intake.validated-file
        • fluxx.file-intake.accepted-file
        • fluxx.extractor.extracted-result
    • Subscription name, suggested format <service name>.<purpose (this is for multiple subs)>
      • Example:

        • file-intake

        • extractor

Follow-up Implementation Stories

  • Consider renaming the existing library poc-lib-autoextract-platform-pubsub to fluxx-pubsub or another preferred name.

  • Implement improvements in pub/sub library.

  • Integrate one pilot service using the shared definitions.

  • Add migration remaining services after pilot service implementation.