Imagine a Dubai distributor connecting its online orders to an ERP and a courier. An order arrives, the integration creates a delivery booking, and the courier response times out. The workflow retries. Now the warehouse has two labels for one parcel. Every application can show a successful transaction. The customer still has one order.
Webhook reliability is the ability to survive that kind of disagreement without repeating or losing the business action. A webhook is a notification one system sends when something changes. It is useful plumbing. It becomes an executive concern when its failure books another collection, releases an order twice or leaves a paid customer waiting.
I would ask for a recovery demonstration before approving another integration. Show the same message arriving twice. Then show the message never arriving. The normal path tells you whether systems connect. These tests tell you whether the operation can depend on them.
Webhook reliability needs a business acceptance test
The acceptance question is not “Did the endpoint return success?” It is “Did the intended action happen, and can we prove its final state?” Receiving a message, accepting work, changing a downstream record and completing a customer promise are separate milestones.
Shopify's webhook documentation explicitly says delivery order is not guaranteed and recommends reconciliation because messages can be missed or mishandled. An integration that assumes every notification arrives once and in order is therefore relying on behaviour the provider does not promise.
Choose one consequential action as the test boundary. For the distributor, use courier booking after an order is released. Keep payment acceptance, inventory reservation and delivery booking as separate states. They may be related, but one successful state must not be used as evidence that all the others succeeded.
The wider ownership questions belong in API governance. Here the narrower job is to prove what happens when a notification is duplicated, delayed or absent. Give the operations manager and integration owner the same acceptance sheet.
1. Define what must happen only once
Write the business action precisely: create one courier booking for this shipment revision. “Process order” is too loose. One order can have several shipments, amendments and legitimate later actions. Blocking everything after the first order message would suppress useful work as well as duplicates.
Distinguish the notification's identity from the action's identity. A repeated delivery of one event is one problem. Two different events that both request the same booking are another. The implementation needs a durable record of the event received and the business operation it caused, scoped to the correct store, entity and shipment.
Stripe's webhook guidance describes both repeated event deliveries and cases where separate event objects refer to the same underlying object and event type. That distinction matters beyond payments: checking only whether a message ID looks familiar is not enough to establish that a commercial action is new.
Ask the team to send duplicates simultaneously, not just one after another. A check that works in a quiet demonstration can fail when two workers both see “not processed” before either records completion. The pass condition is one authorised booking and a visible explanation for every suppressed attempt.
2. Separate receipt from completed work
Verify that incoming notifications are authentic using the provider's documented signature mechanism. Then preserve accepted work durably before acknowledging it. Do not let a quick success response conceal a message held only in temporary memory that disappears during a restart.
The resulting work record should show received, pending, attempted, completed or awaiting investigation. It needs references and timestamps, not a copy of every customer detail. An operator should be able to locate a stuck shipment without searching raw payloads containing names, addresses and phone numbers.
Now interrupt the worker after receipt but before booking. Restart it and watch the pending work resume. Repeat the interruption after the courier accepts the booking but before your own system records success. This second test is where an apparently reliable queue can still produce two labels.
3. Treat an unknown result as unknown
A timeout does not tell you that the courier rejected the request. It tells you the caller did not receive a usable answer. If the business treats every timeout as failure and simply starts again, it can repeat a completed action.
Where the downstream provider supports it, use a stable idempotency key: a reference that lets a repeated request return the existing result instead of creating another action. Verify the actual endpoint's behaviour and retention window. Support elsewhere in the vendor's API proves nothing about this particular operation.
For example, Stripe documents idempotent requests and explains that keys may be removed after they are at least 24 hours old. That is a provider-specific boundary, not an indefinite protection guarantee. A business replaying old work still needs its own operation history.
If the courier offers no suitable duplicate protection, use its lookup capability to establish whether the booking exists. Where neither safe retry nor reliable lookup is possible, route the uncertain case to an owner. A small investigation queue is preferable to quietly creating another physical collection.
4. Recover what never reached the queue
A dashboard of failed messages cannot show a notification the integration never received. Reconciliation must compare source changes with expected downstream outcomes. For released shipments, identify those with no confirmed courier reference after the agreed operating window.
Do that comparison within each business entity and destination system. A group trading across Dubai and Riyadh should not join records using an order number that different subsidiaries can reuse. Keep source time, receipt time and action time distinct so delayed delivery does not masquerade as a new commercial decision.
Replaying missing work should use the same duplicate controls as normal processing. It also needs limits, an owner and an audit trail. A “retry all” button without those boundaries is a bulk instruction to repeat uncertainty.
Make the release decision on the evidence
Before launch, require proof for simultaneous duplicates, a delayed older event, a lost response, a worker restart and a missing notification. Agree the expected business state for each test before running it. Measure unresolved operations by age and commercial consequence, alongside duplicate actions and recovery time.
This is a concrete starting point for business automation in the UAE: one handoff whose normal execution and recovery can both be demonstrated. Webhook reliability is earned when the operation can explain every consequential action after the connection breaks. Until then, the integration is connected. It is not ready to carry the promise.