Retries, timeouts, cancellation, and recovery

Design workflows that handle interruptions without unsafe repeated effects.

A workflow can fail because an input is invalid, a service is temporarily unavailable, a budget is exhausted, or execution is interrupted. Configure recovery according to the work each step performs.

Configure retries and deadlines

Retry controls can include maximum retries and backoff between attempts. A retry is useful for a temporary service problem, but it does not repair a missing field or invalid configuration. Handler contracts can classify an error as non-retryable and impose stricter attempt or deadline limits.

Set a timeout that accommodates the expected operation. Increasing retries and timeout together can create long waits and repeated cost. Check model and connection usage before broadening either.

The error policy determines whether a failure aborts dependent work or is handled according to the job's supported behavior. With parallel branches, independent or already-running work may finish even when one branch fails.

Cancel safely

Cancel the specific active run when work should stop. Cancellation prevents new eligible work from starting, but a write already in progress may finish. Some operations stop at a safe boundary, such as the end of a chunk. Cancellation is not a transaction rollback of the entire workflow.

Inspect the destination after cancelling a write-heavy process. Do not assume a Cancelled status means nothing changed.

Make repeated work safe

Use stable natural keys, supported upserts, and idempotency mechanisms for destinations. A plain SQL insert can insert the same rows again after a retry; a configured natural-key upsert has different behavior. Re-running an HTTP request can also repeat its business effect.

Recovery can reuse retained outputs or resume a checkpointed operation. It should not silently repeat an operation whose completed outcome cannot be established. For a model operation reported as model_outcome_unknown, investigate before retrying: the provider may have processed the request even though the result was not recorded. This specific protection applies to structured model jobs; do not assume every external call has identical recovery guarantees.

Fix the cause, inspect completed steps and destinations, then choose a controlled retry or new run. See caching, batching, and troubleshooting.