The order that never arrived
A customer emails asking where their confirmation is. You pull up Shopify. The order's sitting right there, paid, ready to ship. You check the fulfillment system your warehouse team uses. Nothing. The webhook that was supposed to push that order over never fired, or it fired and failed, and nobody in your company knew until a customer told you. That's the trouble with webhook failures: they're invisible right up until someone downstream pays for them.
That's what a silent webhook failure looks like from the outside. From the inside, it's worse. Your systems both think they did their job. Shopify shows the event as sent. Your endpoint, if you even logged the attempt, might show a 500 that got swallowed somewhere between the load balancer and the app server. Nobody gets paged. Nothing shows up red on a dashboard. The only signal is a customer, or a warehouse manager, or a bookkeeper, noticing three weeks later that numbers don't match.
If you've built any kind of integration between two SaaS tools (Stripe to QuickBooks, Shopify to your fulfillment app, your booking system to Slack) you've probably already had a version of this happen. The scary part isn't that webhooks fail. All webhooks fail sometimes. The scary part is how often nobody's watching.
Why webhook failures happen quietly instead of loudly
Webhooks are built on an assumption that doesn't hold up in practice: that the receiving side will always be up, always process fast enough, and always respond correctly. In reality, a dozen ordinary things break that chain.
Your endpoint returns a 200 before it's actually finished processing, because someone wanted the response to feel fast, and then the background job that does the real work throws an unhandled exception. The sending platform marks the delivery as successful. It has no idea anything went wrong on your end.
Or your server redeploys at exactly the wrong second and the incoming request gets dropped. Or a queue fills up and starts silently discarding messages under load. Or the webhook signature check fails because someone rotated an API key and forgot to update it in one of the three places it's stored, so every event gets rejected with a 401 that nobody's looking at.
None of these show up as an outage. Your app is up. Your website works. Support tickets don't spike right away. The gap just sits there, growing, until someone downstream notices missing data. Usually it's the person closest to the money: a customer who didn't get shipped, or a bookkeeper whose Stripe payouts don't match QuickBooks.
Most SMB teams don't have webhook monitoring because they never had a reason to build it. You wire up Zapier or a custom endpoint, watch it work for a week, and move on. Nobody budgets time for the failure path because the happy path worked in testing.
The fixes that feel right but don't hold
The first instinct is usually to add more webhooks, or point the same event at a second, redundant endpoint "just in case." That doubles your surface area for the same class of bug without fixing anything. If your endpoint silently eats errors, having two of them silently eats errors twice.
The second instinct is a manual reconciliation habit. Someone on the team spot-checks Stripe against QuickBooks every Friday, or eyeballs the last 20 orders in Shopify against the warehouse system. This works for about six weeks. Then that person goes on vacation, or gets busy during a launch, and the check quietly stops happening. Manual reconciliation isn't a system. It's a hope that a specific human remembers to do a boring task forever.
The third instinct, especially popular with technical founders, is to switch from webhooks to polling. Just ask the API every five minutes if anything changed, instead of waiting to be told. Polling does dodge the "silent drop" problem, but it trades it for a different one: latency, rate limits, and cost. If you're polling Stripe's API every five minutes across a dozen resource types just to catch what a webhook would have told you instantly, you'll hit rate limits before you hit reliability. Polling still needs the same thing webhooks need and usually don't have: a log of what was checked and what happened when it was.
None of these fixes address the actual gap, which is that nobody's system of record knows what it's supposed to have received. Without that, you're guessing at reliability instead of measuring it.
What actually needs to be true
There's a real tradeoff here, and it's worth being honest about it instead of pretending there's a free lunch.
Webhooks vs. polling. Webhooks are faster and cheaper at scale, but they fail silently by default and put the reliability burden on your endpoint. Polling is self-healing, since you'll eventually catch what you missed, but it costs more in API calls and adds delay. Most SMBs should default to webhooks for anything customer-facing (orders, payments, bookings) and reserve polling as a backup reconciliation pass, not the primary channel.
Retries vs. idempotency. If you ask the sender to retry failed deliveries, you now have to handle the same event arriving twice without double-processing it: charging a card twice, shipping an order twice. That means every webhook handler needs an idempotency key check before it does anything with side effects. Skipping this to save a day of dev time is how you end up refunding a customer for a duplicate charge.
Fast acknowledgment vs. honest acknowledgment. Returning a 200 immediately and processing in the background is good practice for speed, but only if failures in that background job get logged and surfaced somewhere. A 200 that lies about the outcome is worse than a slow 500 that tells the truth.
Here's a practical checklist we walk clients through when we're wiring up integrations:
- Every inbound webhook gets logged with a timestamp, payload hash, and processing status (received, processed, failed) in a table you can query, not just in application logs that roll off after a week.
- Signature verification happens before anything else, and failed verifications get flagged, not just silently rejected.
- Every handler is idempotent, keyed off the sender's event ID, so retries and duplicate deliveries can't cause double charges or double shipments.
- Failed events go to a dead-letter queue with an alert, Slack or email or whatever your team actually checks, not a log line nobody reads.
- A daily reconciliation job compares record counts between systems (orders in Shopify vs. orders received in fulfillment, charges in Stripe vs. entries in QuickBooks) and flags drift over a small threshold, say more than 1-2%.
- Someone owns this. Not "the team." One named person whose job includes checking the dead-letter queue and the daily reconciliation report.
That last one sounds obvious, but it's the item most companies skip, and it's the one that turns a good system into an actually reliable one.
Where this connects to automation and AI
This is the part people want to skip past, and it's the part that matters most if you're thinking about layering AI on top of your operations. An AI agent that's supposed to catch anomalies, auto-reconcile payments, or flag missing orders is only as good as the data it can see. If your webhook pipeline drops 2% of events silently, an AI model watching that data won't catch the gap. It'll learn to treat the gap as normal, because from where it sits, that's just what the data looks like.
AI is good at pattern-matching across data that's structured, logged, and reliably captured. It is not good at noticing the absence of data it was never shown in the first place. You can't ask a model to flag "the order that should have arrived but didn't" if your system has no record that an order was ever expected. That requires the plumbing (the logging, the reconciliation, the dead-letter queue) to exist before the intelligence layer does anything useful. Fix the pipe first. The smart layer on top is the easy part once the pipe is solid.
Where to start
If you're not sure how bad your own gap is, the fastest gut check is this: pick one integration that moves money or fulfillment data, and try to answer "how many events did we receive last month, and how many failed silently." If you can't answer that in five minutes, you don't have a reliability problem yet that you know about. You have one you haven't found.
We've done this kind of teardown for clients before, including turning disconnected order and fulfillment systems into a single reliable pipeline you can actually trust. You can see a few of those in our case studies if you want to see what that looks like in practice. If you want a second set of eyes on your own setup, we do a free 30-minute Process Teardown where we map one of your painful workflows and show you, in real numbers, how many hours it's quietly costing your team. No pitch, no obligation, just a clear look at where the gap is.
0 Comment