Skip to navigation
Services

Status Check Service

Overview

The Status Check Service is a key part of the system. It monitors and shows the status of files as they go through different processing stages. This service provides real-time updates, ensuring transparency and improving user experience.

In order to collect the needed information, the Status Check Service will subscribe to relevant topics in the AutoExtract Platform to compile status details for each file being processed. It will update the status details whenever an event message indicates a change. Some status changes are important to the end user, while others are only relevant to the platform itself. This service provides information about both types of changes.

When the Status Check Service determines that a file and all of the child-files we split from it have been processed, the file will be marked as complete. An event will be published on the extraction-complete topic, including the original file name, entry method, completion time, internal file identifier, and a list of child-files. The URLs listed for each child will be URLs to the Status Check Service to find status details for that specific child.

Key Components

  1. Event Listeners:

    • The service includes event listeners that watch for events such as new-external-file, valid-external-file and new-internal-file. More topics will be added as the platform grows. For example, the Classifier service will publish messages on classified-file, which will be monitored by the Status Check Service.

  2. Status Tracking:

    • When a new file enters the system, the service creates a record for the file which includes the file location and the creation time.
    • Upon creation of a file record the system will ALSO add the first status detail for the file. That status will be created and the detail record will include the time the system found the file.
    • If the system sees no validation problems with the file, the service updates the record with the status validated and it will have the timestamp for when validation completed successfully.
    • If the system sees no validation problems with the file, the service updates the record with the status failed validation and it will have the timestamp for when validation failed.
    • When a valid file has been copied to the internal storage container, the service updates the record with the status copied and it will have the timestamp for when copy completed successfully.
    • If the system cannot completely copy the file, the service updates the record with the status failed copying and it will have the timestamp for when validation failed.
    • As child-files are identified within the file, new file records are created for each one and each will get a created status detail and timestamp as well.
    • The child-file will be linked (as one of the children) to the file.
    • After each future step in the process, like extraction, there will be a message published on a related topic, like extraction-complete. Once those other steps are added the Status Check Service will record a status detail with a meaningful status, like extracted for example, and a time from the message.
    • Once all processing is complete and the file and all related children are in their final state, like complete, the file status is updated to complete for the “main” file. (We shoul make it so that when the final child is complete that we automatically complete the parent).
    • For any fatal error, like file validation for example, there will be a message on an error topic, like validation-failure, which gives details of what went wrong and when. The Status Check Service will look for failures and will record an appropriate status and time for the file, like validation failed.

  3. User Interface Support:

    • The service API has a GET /status/files/<id> endpoint that provides status details of a specific file for the user interface. The response will be JSON.
    • The details are not complete at this time, but it is likely that this service will also provide one or more endpoints to list which files are “in” the system. We will come back to that part once we have better requirements.

Workflow

  1. File Entry:

    • A new file enters the system, and the Status Check Service creates a record with the status created.

  2. File Validation:

    • A new file is validated successfully, and the Status Check Service creates a record with the status validated.

  3. File Copying:

    • A valid file is copied to the internal storage successfully, and the Status Check Service creates a record with the status copied.

  4. Child-File Identification:

    • As sub-files are split from a file, new records are created for each of the children with the status created.

  5. Raw Extraction:

    • When raw extraction is performed, the status of the corresponding file is updated to extracted.

  6. Post-Processing:

    • After post-processing, the status changes to postprocessed.

  7. Completion:

    • Once all processing steps are complete, the file status is updated to completed.

Benefits

  • Transparency: Users are informed about the status of their files at every stage of processing.
  • User Satisfaction: Real-time updates improve the user experience by providing clear visibility into the processing workflow.