All writing Engineering

Webhooks in, receipts out

Fibric EngineeringOctober 28, 20255 min read

Strip an operator down to its interface and two things remain: signals arrive, decisions leave. The reference architecture is designed for at-least-once inputs and bounded duplicate suppression on supported action paths, with ambiguous external outcomes handled explicitly.

Reference architectureNo end-to-end exactly-once claim

Reference diagram of duplicate webhook deliveries being deduplicated before governed execution records are produced

Every system an operator watches speaks the same dialect of unreliable. Kustomer retries a webhook because your endpoint took four seconds to answer. Magento fires an order event twice during a deploy. A BACnet controller repeats a reading after a network blip, then drops the next three entirely. None of this is anyone’s bug. At-least-once delivery is the honest contract of distributed systems, and any platform that pretends otherwise is deferring the lesson, not avoiding it.

For a dashboard, duplicates may be cosmetic. For an operator, they can become an incident: a repeated refund, customer message, or relay command. The reference architecture therefore treats duplicate suppression and ambiguous outcomes as first-class control problems rather than promising an impossible end-to-end exactly-once effect.

The envelope: one target shape for supported signals

In the reference design, supported signals are normalized into a canonical envelope carrying tenant, source, entity, timestamp, and event identity before operator reasoning. Stable identity can help recognize known redelivery, but source quality, missing identifiers, and downstream ambiguity remain connector-specific boundaries.

In · signal

The envelope

Supported events use one target shape: tenant, source, entity, timestamp, and durable identity. Recognized redelivery can collapse to one record within the governed boundary.

Out · decision

The receipt

Supported attempts record the entity, policy, source envelope, key, and observed outcome. Duplicate suppression is bounded by the connector and downstream system.

Idempotency keys: the effect remembers itself

Deduplicating inputs is necessary but not sufficient. If an executor crashes after an external action but before settlement, the outcome is ambiguous. A stable key can suppress attempts after a known result; otherwise the connector must use downstream idempotency, status lookup, reconciliation, compensation, or stop for review.

Retry storms are not an edge case to survive. They are the weather. You build the roof before it rains.

Single-flight: one hand on each entity

The subtler failure is concurrency. Two individually sensible but jointly conflicting plans need more than an idempotency key. Single-flight can serialize covered execution for an entity so a later plan is revalidated against newer state. Its lock boundary must be explicit, and it cannot serialize independent writers outside the runtime.

The receipt: decisions leave as facts

On supported paths, the execution record links an entity, policy decision, source envelope, key, attempt, and observed response. It supports traceability through recorded events, while preserving gaps when an external outcome or causal link is unknown.

These are reference-kernel control objectives. BearScope is the live product proof point but currently implements its governed path directly through the BFF. Smart City Labs is an illustrative hotel pattern. Each connector still needs verified semantics, and nothing about external effects should be left implicit.

Keep reading: Idempotency keys · One envelope