A happy path is a demonstration, not a system

A tutorial usually starts with a trigger, calls two services and writes the result. That is a useful way to learn the interface. A production workflow must also answer what happens when the trigger repeats, the destination accepts the write but the response is lost, credentials expire, a provider returns 429, or an operator needs to replay yesterday’s failures.

The visual graph can hold these controls, call shared sub-workflows, or delegate them to a small service. The exact shape matters less than making the semantics explicit.

Give every business event an identity

A workflow execution ID identifies a technical run. An idempotency key identifies the business action across retries and separate runs. For an invoice, that might combine source tenant, vendor and invoice number; for a webhook, it may be the provider event ID.

Store that key before the consequential write with an atomic state transition. If the same event arrives again, return or resume the existing result. Do not hope the trigger fires once.

  • Correlation ID: traces one event across logs and services.

  • Execution ID: distinguishes each technical attempt.

  • Idempotency key: prevents a business action from happening twice.

Retries need a policy, not a loop

Retry transient failures such as timeouts, selected 5xx responses and rate limits. Do not retry invalid credentials, schema errors or rejected business rules as though time will fix them. Use bounded exponential backoff with jitter and respect provider retry headers.

After the final attempt, persist the event, failure class, safe input reference and prior side-effect state in a dead-letter queue. A dead letter is only useful if there is an owner, an alert threshold and a safe replay procedure.

Separate error workflow from error visibility

An error branch can catch a failure while still leaving operations blind. Logs should be structured and carry workflow version, step, correlation ID, tenant, duration, provider status and safe error class. Alerts should report conditions that need action, not every retry.

A useful run history answers: what entered, which decisions were made, which external writes completed, what remains pending and who owns the exception. Avoid logging secrets or entire documents merely because they are convenient during development.

Use staging, fixtures and contract tests

Pinning sample data helps build a node; it does not test production integrations. Maintain synthetic fixtures for normal, duplicate, malformed, missing and high-risk inputs. Run them against a staging workflow with non-production credentials and verify both outputs and forbidden side effects.

API contract tests should detect fields that disappear, enum values that change and permissions that expire. AI steps need a regression set because a prompt or model change can remain syntactically valid while changing downstream behaviour.

Version the things outside the canvas

Export workflow definitions to source control. Version schemas, routing tables, prompt templates, model configuration and approval policy alongside them. A run should record which versions shaped its outcome.

Credentials belong in a secret manager or the platform’s encrypted store, scoped to the smallest useful permissions. Production and staging should not share write credentials. A screenshot of a canvas is not a deployment record.

Reconciliation closes the crash window

No orchestrator can magically guarantee exactly-once behaviour when an external API performs a side effect but the network drops before confirmation. Design for the uncertainty. Record intent, use downstream idempotency where available, query by business key, and run reconciliation between source events and destination state.

Production means the team can recover from ambiguous outcomes without guessing. That is less glamorous than adding another node, and much more valuable.

The practical next step

Map one real execution and one failure.

That will reveal more about the right architecture than a tool comparison or model demo.

Let's build something real