40 posts found
One Tenant's Webhook Flood Shouldn't Become Everyone's Outage
One Shopify store sent 30 million events in five minutes, and every other store's webhooks waited hours behind it. Here is why a single shared delivery queue turns one noisy tenant into a platform outage, and how per-tenant fairness keeps everyone else moving.
The Signature Header You Verify Just Stopped Arriving
On September 1, 2026, HighLevel stopped signing webhooks with its RSA X-WH-Signature header and moved to Ed25519 in X-GHL-Signature. Here is why a signature scheme cutover breaks receivers in two opposite ways, and how to build a verifier that survives the next one.
Every Webhook Payload You Store Is a Copy of Someone Else's Data
Your webhook inspection log is an uninventoried, forever-retained copy of your provider's customer data. Here is why payload retention is a decision you have to make on purpose, and how deletion is finally becoming a first-class operation.
A Shopify Event Field Changed From Array to Object, and Your Parser Didn't Notice
On September 16, 2026, Shopify reshaped its Events payload: fields_changed went from a flat array to an object, two delivery headers vanished, and parent triggers need a wildcard. Here is why a versionless payload change is the webhook break nobody's monitoring catches.
A Webhook Is Where Your Distributed Trace Goes to Die
A traceparent header rides through every internal call until it hits a webhook, then the trace splits in two: the sender's half and yours, with nothing joining them. Here is how to carry W3C Trace Context across the webhook boundary and stitch an event's whole journey back together.
mTLS for Webhooks: Proving Who Is Calling, Not Just What They Sent
Your HMAC signature proves the payload was not tampered with. It says nothing about who opened the connection. mTLS does, and Shopify just made receivers care by rotating the certificate under everyone's feet.
A Draft Protocol Wants to Replace Your Webhooks With a Change Feed
A new draft, SCROLL, proposes swapping push webhooks for a pull-based ordered change feed when you replicate state. Here is what the spec defines, why the reconciliation cron in your codebase proves its point, and where webhooks still win.
Transforming Webhook Payloads at the Gateway: The Signature You Just Invalidated
Reshaping webhook payloads at the delivery layer normalizes providers and strips PII before it reaches your logs. It also breaks signature verification, hides untested code in a vendor UI, and drops events silently. Here is what gateway transformations actually cost.
Your Webhook Signature Protects a Blob. RFC 9421 Protects the Request.
Most webhook signatures cover a concatenated string you rebuild yourself. RFC 9421 signs the actual HTTP request: method, path, and a body digest. Here is how HTTP Message Signatures work, why receivers are starting to see them, and where they bite.
Receiving Provider Events Without a Webhook Endpoint at All
Stripe, and a growing list of providers, can now deliver events straight to Amazon EventBridge or Azure Event Grid instead of POSTing to a URL you host. No public endpoint, no signature to verify, and a whole new set of failure modes. Here is what the event-bus model actually changes.
Allowlisting Webhook Sender IPs: A Control That Breaks Quietly
Locking your webhook endpoint to a provider's IP ranges feels like security, but the list changes without warning, retired ranges get reassigned to strangers, and a proxy in front of you hides the real source. Here is what IP allowlisting actually buys you and how it fails.
The Webhook Payload You Trust Is Already Out of Date
Thin events are winning. Stripe now ships lightweight notifications that carry an ID instead of the object, and you fetch the current state yourself. Here is why the fetch-on-notify pattern is the right default, what staleness it trades away, and the traps it creates.
When the Sender Gives Up on Your Webhook Endpoint
A four-second outage on your side can end with the sender switching your webhook endpoint off entirely. Then nothing arrives, nothing alerts, and you find out from a customer. Here is how endpoint disabling works, why it compounds, and how to survive it.
The Webhook You Never Got Is Sitting in an API You Forgot to Call
When a webhook evaporates between the sender and you, nothing notices until a customer complains. But the event was never lost. It is still in the provider's event log. Here is why pull-based Events APIs are becoming the backstop every serious webhook consumer needs.
Your Webhooks Arrive Out of Order, and Sending Them in Order Won't Save You
Regular webhooks are delivered on a best-effort basis, so events land out of sequence and your handlers process them that way too. Why ordering breaks, why FIFO endpoints are a real tradeoff, and how to build consumers that stop caring about order.
When Your Handler Processed It but Returned an Error: Taking Control of Webhook Retries
When your webhook handler completes but the HTTP response fails, the sender retries into a webhook you already processed. HTTP status codes are a blunt instrument for event delivery. Here is how receivers are gaining a more precise vocabulary to control what happens next.
Rotating Webhook Secrets After a Compromise: The GitHub Leak as a Case Study
GitHub inadvertently exposed webhook secrets in HTTP headers from September 2025 to January 2026. How to check your exposure, rotate secrets without dropping deliveries, and prevent the same mistake in your own sender.
SSRF in Webhook Senders: Protecting Your Infrastructure from User-Supplied URLs
When your platform sends webhooks to user-configured URLs, those URLs can point anywhere—including your cloud metadata service. DNS rebinding, private IP ranges, and redirect chains explained.
Webhooks Don't Keep Your State in Sync -- Here's Why and What to Do About It
Webhooks tell you when things change, but they can't bootstrap initial state, detect silent deletions, or verify no events were missed. Here is why most webhook consumers end up with a nightly reconciliation job and what the alternatives look like.
Webhook Signatures Are Converging on Standard Webhooks, and Moving to Public Keys
OpenAI, Anthropic, and more now sign webhooks with Standard Webhooks. Here is how the shared header format works, why an empty secret silently disables it, and what asymmetric ed25519 keys change for receivers.
When Webhook Failures Only Show Up at Scale
Connection pool exhaustion, load balancer timeouts, container restarts. Some webhook bugs hide until you hit production traffic. How to debug failures that vanish in staging.
When Webhooks Return 200 OK But Nothing Happens
Silent webhook failures are the worst kind. Your endpoint returns success, logs look clean, but business logic never runs. Here's how to catch what your monitoring misses.
Your Webhooks Are Processing Duplicates and You Probably Don't Know It
How idempotency keys prevent duplicate webhook processing, with real implementation patterns that actually work in production systems.
Your Webhook Logs Are Useless (And How to Fix Them)
Most webhook logging captures the wrong data at the wrong level. Stop logging everything and start logging what matters for debugging production incidents.
Your Webhooks Are Failing and Nobody Knows
Most webhook failures never trigger an alert. They just vanish. Here is how to catch the silent ones before your customers do.
Stop Processing Webhooks Synchronously. Your Database Will Thank You.
Why queue-based webhook processing beats synchronous handling, and the patterns that actually work at scale.
HMAC vs API Keys for Webhook Auth: What Actually Works
A practical comparison of HMAC signatures and API key authentication for webhooks, covering real tradeoffs most teams discover too late.
Killing Old Webhook Versions Without Killing Your Integrations
Everyone talks about launching new webhook versions. Nobody talks about sunsetting old ones. Practical deprecation strategies that actually work in production.
Your Webhook Tests Pass. Your Production Still Breaks. Time to Inject Some Chaos.
Happy-path webhook tests are table stakes. Chaos testing and failure injection reveal how your system actually behaves when webhooks arrive late, malformed, or not at all.
Webhook Replay Attacks: Why Signature Verification Alone Won't Save You
HMAC signatures stop forgery, but replay attacks bypass them entirely. Timestamp validation, nonce tracking, and idempotent handlers are what actually protect you.
Monitoring Webhook Health: Metrics, Alerts, and Dashboards That Actually Help
Generic monitoring misses webhook-specific problems. Track success rates, delivery latency, retry depth, and endpoint health scores to catch issues before your customers do.
Webhook Integration Patterns: From Direct Processing to Event-Driven Architecture
Most webhook handlers start as a single endpoint with inline processing. When that breaks down, you need queues, fan-out, gateways, and sagas. Patterns for scaling webhook consumers.
Webhook Delivery Guarantees: At-Least-Once, Exactly-Once, and the Outbox Pattern
Exactly-once delivery is a myth for webhooks. Understand why at-least-once with idempotent consumers is the practical choice, and how the transactional outbox pattern makes it work.
Rate Limiting Webhooks: Protecting Both Sides of the Connection
Rate limiting for webhooks works differently than for APIs. Token buckets, per-endpoint limits, backpressure, and handling 429 responses from the sender and receiver perspective.
Designing Webhook Payloads That Developers Actually Want to Consume
A well-designed webhook payload saves consumers hours of work. Event naming, envelope patterns, idempotency keys, and schema evolution rules that stand the test of time.
Testing Webhooks: From Local Development to Production Validation
Webhooks are hard to test because the sender controls the timing. Strategies for local tunneling, mock servers, integration suites, shadow mode, and production validation.
Webhook Versioning: How to Evolve Your Event API Without Breaking Consumers
Versioning webhooks is harder than versioning REST APIs because you push data to consumers. Strategies for additive changes, envelope versioning, and safe migrations.
Debugging Webhook Failures: A Systematic Approach to Finding What Went Wrong
When webhooks break in production, you need a system for finding the root cause fast. A step-by-step methodology for debugging delivery, parsing, auth, and timing issues.
Webhook Signature Verification: A Practical Guide to HMAC, Key Rotation, and Defense in Depth
Unverified webhooks are an open door. Learn HMAC-SHA256 verification for Stripe, GitHub, and Shopify, plus key rotation and timing-safe comparison.
Webhook Retry Strategies: Building Reliable Delivery Beyond Basic Exponential Backoff
Exponential backoff is just the starting point. Learn jitter, dead letter queues, retry budgets, and circuit breakers for production webhook systems.