DocumentationAutomationFailures and timeouts

Failures and timeouts

It happens that things don't go smoothly, and of course DataUnlocker implements best measures to keep systems running even in case of failures.

Timeouts

Automation pipeline steps won't run indefinitely and will eventually time out, issuing the timeout failure at this step. The default timeout is set to be 2 days, while many steps have this timeout configurable:

Configurable step timeout

We strongly recommend not to set custom timeouts over 2 days, as you risk to react on these failures slowly.

Exponential backoff

In case when the response status of the webhook is not 2xx, DataUnlocker will keep trying to deliver this webhook using the exponential backoff strategy, with the maximum of 16 minutes between invocations. Each next delivery will be scheduled in the following way:

  1. Deliver the webhook immediately.
  2. When failed, deliver it after 1 minute.
  3. When failed again, deliver it after 2 minutes.
  4. When failed again, deliver it after 4 minutes.
  5. When failed again, deliver it after 9 minutes.
  6. When failed again, deliver it after 16 minutes.
  7. Each next failure will deliver it after 16 minutes.

This will continue up to 12 hours of the maximum waiting time (be default), after which DataUnlocker will eventually fail the automation pipeline at this step and send email notifications to the automation pipeline owners.

Notifications

In case when the automation pipeline fails:

  • DataUnlocker sends email notifications to all users receiving "dev" notification types (can be configured in the property's user management page).
  • You will be able to examine the failed pipeline and all related info in the DataUnlocker's Admin Console UI.
  • You won't receive email notifications when you manually abort the pipeline by clicking the "Abort" button.

Failed automatic installation refresh pipeline most likely means that you will need to fix the issue and then start this pipeline manually from the DataUnlocker's Admin Console UI, waiting until it succeeds.

DataUnlocker Logo
DataUnlocker
Content blockers are friends