ONE

The Workflow Engine Was Teaching Itself the Wrong Lessons

Clicking Test on a workflow ran it exactly like a real trigger, which meant every debugging click was quietly voting on which path the optimiser should trust. We found the bug during a gap analysis against novu's workflow runtime, fixed it, and shipped five more wires from the same read: a payload door, named skip reasons, bounded transient retry, step preview, and if/then/else route lanes on the canvas.

workflows engine reliability product

Every workflow in ONE runs on a graph. You draw the steps (trigger, skill, tool, agent, condition, human, delay, sell), connect them with edges, and activate it. Every real run deposits a signal on the edges it crossed: strength when the path worked, resistance when it didn’t. After ten runs the fastest path through your process is visible on the canvas. After fifty, the bottleneck lights up red. That’s the whole pitch: your SOP gets better every time it runs, because it remembers.

Here’s the bug. Clicking Test to try out a new workflow ran it exactly like a real trigger: same steps, same dispatch, same signal on every edge. Which meant every experiment you ran while building a workflow was quietly voting on which path the optimiser should trust. Ten test clicks while you debug a typo could out-vote your first three real customers. The evidence was lying, and nothing told you.

We found it during a gap analysis against novu’s workflow runtime (github.com/novuhq/novu: Nest, Mongo, BullMQ, structurally nothing like our Durable-Object-per-run design, but their runtime has shipped enough scars to be worth reading). Their architecture didn’t teach us anything about ours. Their failure modes did. We ported five of them as pure functions, fixed the pheromone bug as a sixth, and shipped all six last week.

Test runs stop lying

workflow:run now takes test: true. It threads through as isTest from the receiver into the executor, and both deposit points (the ones that write strength on success and resistance on failure) become no-ops when the flag is set. The run still executes for real: every step fires, every integration gets called, you see exactly what a live trigger would do. RunMonitor just tags it Test run, and the optimiser never hears about it. One D1 column (is_test on workflow_run), one guard at two existing closures. You can break a workflow ten times while you build it and the evidence stays real-traffic-only.

Bad payloads bounce at the door

A trigger step can now carry a payloadSchema: a small JSON-Schema subset, just type, required, and one level of properties. Nothing fancy. We looked at real trigger payloads and about 40 lines of pure validation covered all of them. No zod (it can’t read schema stored as JSON), no full draft-2020 validator. validateTriggerPayload runs before a run row is even inserted, in both places a workflow starts: the direct workflow:run call and the tag-matched triggerWorkflows fan-out. Miss a required field and you get { error: 'invalid_payload', issues: [...] } naming exactly what’s wrong, instead of a mysterious step:failed three steps into a run that should never have started. No schema set, the default, behaves exactly like it did before.

Every skip says why

A pruned branch (the one your condition step didn’t take) used to just vanish from the run log. Now it logs step:skipped with reason: 'pruned'. A step that never got a real payload because something upstream broke logs step:failed with reason: 'poisoned' instead of the same bare failure a genuine bug produces. The event kinds stayed exactly as coarse as they were: every consumer reading step:failed or matching on kind keeps working, unchanged. The reason just rides along in the payload nobody was reading closely enough. It’s the smallest fix in the batch and the one operators will notice first, because “why didn’t this step fire” used to mean opening the executor source.

Transient failures retry. Real bugs don’t.

Novu’s lesson here was specific: retry one narrow class of failure, never retry blind. isTransientFailure is a pure classifier (network errors, timeouts, 5xx shapes, nothing else) gating a bounded retry loop around the single dispatch call for tool, skill, and agent steps. Turn on Retryable on a step, set a max attempt count (default 3), and a 429 or a dropped connection backs off and tries again. A 4xx or a bad config still fails on attempt one, same as today. Only the final failed attempt deposits resistance. A retry that eventually succeeds counts as a clean pass, so the learning loop sees the outcome, not the flakiness it took to get there.

Preview before you commit

Every step’s arguments get resolved through the same substitution code a real run uses, substituteArgs plus stepBinding (now exported instead of file-private), against a mock trigger payload or your last real run’s. workflow:preview-step returns { receiver, data }. It dispatches nothing, reads no run history beyond what you feed it, and shows up as a Preview button right on the step you’re editing. You wire a step, click once, and see exactly what it’s about to send before you’ve armed anything.

Branches finally look like if/then/else

This was the one visible change, and the only one that touches the canvas. A condition step’s true and false edges have always existed in the data (edge-condition), but you had to read a diamond node and squint at labeled lines to find them. Now every condition step renders two lanes underneath it: Route 1 and Else, always both, even before you’ve built either one. Each lane is its own vertical column with a dashed placeholder where it’s empty. Click it, pick a step from the palette, and it lands wired into that lane’s condition automatically. Both lanes rejoin visually at the first step they share. None of it is new data. The pills are synthetic, render-only nodes derived by one pure function (branchLanes) from edges that already existed. Nothing new to keep in sync, nothing new to break.

What we didn’t build

Novu also has throttle steps, a digest system, and draft/publish environment syncing. We looked at all three and left them out on purpose. Throttle steps solve a rate-limit problem we don’t have yet. Build it when a real workflow hits a real ceiling, not before. Digest steps are parked with a target shape in the notifications plan, not this one. Environment sync solves a problem our simulate-first apply-diff already solves a different way. Novu’s runtime taught us what to fix. It didn’t get a vote on what to add.

The numbers

Six cycles, 31 files, +1,891/−162. Fourteen new test files, 54 assertions, all pure functions with unit tests, the same convention the existing edge-learning tests already used, so nothing about the executor’s shape changed to fit them in. Full suite: 671 passed, 6 skipped, five pre-existing failures we confirmed were unrelated by stashing the diff and re-running against trunk. Zero new TypeScript errors. One real regression caught in the last cycle: a coverage test asserting node types matched step kinds one-for-one, broken by the new render-only route-pill node. Fixed in the same branch before it shipped.

The bug that started this — test clicks corrupting the one signal an agent uses to learn — is fixed. The other five were the same shape: something the engine already did, done a little blind. Now it names its reasons, retries what’s worth retrying, and shows you what it’s about to do before it does it. Grand, so.