The 657-message problem, and how single-flight fixes it
An agent that can act can also act too many times. One stuck retry loop is enough to flood a customer. This note explains how single-flight and idempotency reduce that risk inside a governed boundary, plus the downstream and ambiguous-outcome cases they cannot eliminate.
Every team that has put an automated agent next to a real messaging system has a version of this story. A workflow fires. The downstream API is slow to acknowledge, or returns a timeout that the agent reads as failure. So the agent does the reasonable thing: it tries again. The send had actually gone through. The retry sends a second. The acknowledgement is still slow. It tries again. By the time someone notices, a single customer has received 657 messages in twenty minutes, and the only question left is how to apologize.
Forty-four sends where one was intended. Same recipient, same minute.
It is tempting to file this under "bad luck" or "a bug we will fix." It is neither. It is a structural property of putting a non-deterministic decision-maker in front of an effect that touches the outside world. The model is not wrong to retry. Retrying is correct behavior for a flaky network. The mistake is letting the same intended action become many real actions. A better prompt will not fix that. The system needs a different shape.
Two ideas, working together
The reference kernel reduces this failure mode with single-flight and idempotency keys. These familiar controls are effective inside a defined boundary, but key mistakes, ambiguous downstream results, independent writers, and distinct intents can still produce duplicates.
Single-flight per entity
Single-flight means that for any one entity, at most one action is ever in flight at a time. There is exactly one order #4471. There is exactly one customer. While an action against that entity is running, a second action against the same entity does not run in parallel and does not pile up behind it uncontrolled. It collapses into the one already happening, or it waits its turn and re-evaluates against the new state.
This is the difference between a thermostat and a stuck relay. A thermostat that wants the room warmer does not send "heat on" forty times because it asked forty times. It holds a single intent for a single room. Fibric holds a single intent for a single entity, and that boundary is enforced by the kernel, not by hope.
Idempotency keys
Idempotency helps when the same intended action is attempted twice. A stable key can return a settled result without redispatch. If a timeout occurs after the external send but before settlement, the outcome is ambiguous unless the downstream messaging system honors the same key or exposes a reliable lookup.
For a settled send with key notify:order-4471:shipped, later matching attempts can return the recorded result. Before settlement, the safe response depends on connector-specific reconciliation rather than unlimited retry.
Why this belongs in the kernel, not the connector
You could ask every connector author to handle this themselves. Most distributed systems do exactly that, and it is why most distributed systems eventually send someone 657 messages. The discipline is real, but it is unevenly applied, and the one connector that forgets is the one that floods a customer.
The reference architecture centralizes single-flight and key checks in the governed executor. Connector authors still own downstream semantics, and each deployment must verify that credentials and independent write paths do not bypass the boundary.
What an operator never has to think about
The goal is to keep duplicate-control plumbing out of operator reasoning while making connector failure semantics explicit. The executor can govern attempts; it cannot guarantee an external effect beyond the evidence the downstream system provides.
- One intent, one entity, one action in flight. Concurrency against the same thing is serialized, not stacked.
- Governed effects use stable keys where identity is defined. Settled duplicates can return the recorded result.
- Supported attempts leave execution records. Caught duplicates and ambiguous outcomes are recorded distinctly.
The 657-message problem is a useful test for any agentic system, including ours. Ask how it scopes single-flight, derives keys, settles outcomes, reconciles timeouts, handles independent writers, and alarms on repeated attempts. No honest answer should call the flood impossible.
Keep reading: Point and go · Bounded duplicate suppression