Migrate webhooks to queue-based processing without breaking production
Inline webhook handlers fail under spikes, outages, and slow downstream dependencies.
This guide gives you a minimal-risk migration plan: ack fast, queue, add idempotency, and roll out safely.
No credit card required
TL;DR
- Goal: keep provider deliveries stable while moving side effects out of the HTTP request path.
- Stage 1: Ack fast (2xx/202), persist payloads durably, and process asynchronously in a worker.
- Stage 2: Add idempotency + error classification + dead-lettering before you scale throughput.
- Stage 3: Add replay/backfill tools and remove inline side effects.
- Cut over safely: dual-write/dual-consume only when necessary, and keep rollback simple.
- Measure end-to-end latency and backlog so you know if you are actually “safer” after migration.
For duplicates and retry semantics, see retries & backoff.
Anti-patterns
- Big-bang cutovers without a rollback plan or overlap window.
- Moving work to a worker without idempotency (you just moved the duplication problem).
- Changing the provider URL and the processing semantics in the same deploy.
See a concrete before/after in comms webhooks.
Core concepts
A safe migration changes one variable at a time: first delivery latency, then processing semantics, then the provider URL.
Queue boundary
Define what you persist: raw payload, headers, received_at, tenant, event ID. The queue boundary is your replay boundary.
Cutover strategy
Prefer cutovers that keep the provider URL stable at first. Dual-write only when you must; it creates duplicates.
Rollback
If you can’t roll back quickly, you are not done. Keep rollback paths boring and tested.
Simple flow
Provider
Sends events
retries on failure
Shim / Endpoint
Ack fast
forward/persist
Queue / Worker
Process async
idempotent
After the shim is stable, you can point the provider directly at Hooque and remove the shim.
Migration checklist
A staged plan that reduces risk. Do not skip the observability and idempotency steps.
- [ ] Inventory current inline side effects (DB writes, API calls, emails, billing actions)
- [ ] Add correlation IDs + structured logs for: request id, event id, tenant id
- [ ] Stage 1: Ack fast (2xx/202) and persist raw payload durably
- [ ] Stage 1: Start async worker processing from the persisted payloads
- [ ] Stage 1: Add a dead-letter/quarantine path (do not block the queue)
- [ ] Stage 2: Add idempotency (dedupe key + TTL/window + unique constraints)
- [ ] Stage 2: Classify failures (retryable vs permanent) and set retry/backoff policy
- [ ] Stage 2: Add dashboards for backlog, retry reasons, DLQ volume, latency
- [ ] Stage 3: Add replay tooling and verification queries to validate correctness
- [ ] Stage 3: Turn off inline side effects (keep only verification + enqueue)
- [ ] Cutover: Keep rollback (flip traffic back) and avoid schema changes mid-cutover
- [ ] Post-cutover: Backfill missing events and reconcile using provider APIs Reference implementation
A migration shim is a pragmatic first step: keep the provider URL stable, then forward payloads into a queue you control.
Node
Express shim forwarding to Hooque ingest
// Migration shim: keep your existing URL stable, but forward payloads to Hooque for queue-based processing.
// npm i express
import express from "express";
const app = express();
const HOOQUE_INGEST_URL =
process.env.HOOQUE_INGEST_URL ??
"https://app.hooque.io/hooks/wh_replace_me?token=replace_me";
app.post(
"/provider/webhook",
express.raw({ type: "*/*", limit: "2mb" }),
async (req, res) => {
const rawBody = Buffer.isBuffer(req.body) ? req.body : Buffer.from(req.body ?? "");
// TODO: keep your existing verification here (signature + timestamp) during the migration.
// If you change verification semantics, do it separately from “move to async”.
// Ack fast so the provider stops retrying.
res.sendStatus(202);
// Forward to Hooque asynchronously.
// IMPORTANT: forward the provider signature headers that Hooque needs for verification (if configured).
const forwardHeaders = {
"content-type": req.header("content-type") ?? "application/json",
// Example provider headers (adjust to your provider):
"stripe-signature": req.header("stripe-signature") ?? "",
"x-github-event": req.header("x-github-event") ?? "",
"x-github-delivery": req.header("x-github-delivery") ?? "",
};
try {
await fetch(HOOQUE_INGEST_URL, {
method: "POST",
headers: forwardHeaders,
body: rawBody,
});
} catch (err) {
// TODO: log + alert (forwarding failures mean you are dropping events)
console.error("forwarding to hooque failed", err);
}
}
);
app.listen(3000, () => console.log("migration shim listening on :3000")); Python
FastAPI shim forwarding to Hooque ingest
# Migration shim: keep your existing URL stable, but forward payloads to Hooque for queue-based processing.
# pip install fastapi uvicorn httpx
import os
import httpx
from fastapi import FastAPI, Request, Response
app = FastAPI()
HOOQUE_INGEST_URL = os.getenv(
"HOOQUE_INGEST_URL",
"https://app.hooque.io/hooks/wh_replace_me?token=replace_me",
)
@app.post("/provider/webhook")
async def webhook(request: Request) -> Response:
raw_body = await request.body()
# TODO: keep your existing verification here (signature + timestamp) during the migration.
# Ack fast so the provider stops retrying.
resp = Response(status_code=202)
# Forward to Hooque asynchronously.
forward_headers = {
"content-type": request.headers.get("content-type", "application/json"),
# Example provider headers (adjust to your provider):
"stripe-signature": request.headers.get("stripe-signature", ""),
"x-github-event": request.headers.get("x-github-event", ""),
"x-github-delivery": request.headers.get("x-github-delivery", ""),
}
try:
async with httpx.AsyncClient(timeout=10) as client:
await client.post(HOOQUE_INGEST_URL, content=raw_body, headers=forward_headers)
except Exception as err:
# TODO: log + alert (forwarding failures mean you are dropping events)
print("forwarding to hooque failed", err)
return resp Common failure modes
Migration failures usually come from changing too many things at once or losing observability during the move.
Events drop during the shim phase
Likely causes
- Forwarding errors not monitored.
- Shim acks 2xx but forwarding fails.
- Body limits or content-type handling differs from previous handler.
Next checks
- Alert on forwarding failures and non-2xx from Hooque ingest.
- Log correlation IDs for each forward.
- Match provider requirements (raw body, headers, size).
Duplicate side effects increase
Likely causes
- Dual-write/dual-consume overlap without dedupe.
- Workers lack idempotency keys.
- Provider retries + shim retries amplify duplicates.
Next checks
- Introduce dedupe store early.
- Make workers idempotent before scaling.
- Reduce overlap windows and avoid dual-consume if possible.
Cutover causes outages
Likely causes
- Provider URL changed alongside auth/processing changes.
- No rollback plan.
- Hidden dependencies in inline handler logic.
Next checks
- Split changes into separate deploys.
- Test rollback in staging.
- Keep old handler code path behind a flag.
How Hooque helps
If you are migrating because reliability is hurting you, a managed webhook-to-queue receiver cuts out a lot of operational surface area.
- Hosted webhook endpoints: move inbound exposure off your app servers.
- Durable queue persistence at ingest time: ack fast without losing payloads.
- Consumers pull/stream with explicit Ack/Nack/Reject so you control retries.
- Replay and inspection tools for migration validation and incident recovery.
- Metrics surface for backlog, deliveries, and processing outcomes.
Compare migration outcomes with payment webhooks and review pricing.
FAQ
Migration questions that come up when reliability issues are already impacting customers.
What is the safest way to migrate an inline webhook handler to a queue?
Start by acknowledging deliveries quickly and persisting raw payloads durably, then move side effects into a worker. Add idempotency and error classification before scaling. Keep the provider URL stable during the early stages to minimize delivery risk. With Hooque, you can move ingest to a hosted endpoint that persists instantly, then process from a durable queue.
Do I need idempotency before migrating to async processing?
You can start migrating without perfect idempotency, but you should add idempotency early (stage 2). Moving work to a worker does not remove duplicates; it makes duplicates more likely because retries are more visible. With Hooque, the delivery lifecycle (Ack/Nack/Reject) is explicit, so adding dedupe keys in your consumer is straightforward.
Should I dual-write during the cutover?
Only if you must. Dual-write and dual-consume add complexity and can create duplicates. Prefer cutovers that change one thing at a time and keep rollback simple. With Hooque, a common path is “provider → Hooque ingest → consumer queue” which avoids dual-consume in many migrations.
How do I validate the migration didn’t break correctness?
Use correlation IDs, compare counts and outcomes between old and new paths, and run replay/backfill validation queries. Track end-to-end latency and error rates before and after cutover. With Hooque, inspection and replay make it easier to reproduce failures and verify fixes without losing payloads.
What payload should I persist when migrating?
Persist the raw body bytes plus key headers (content-type, signature headers, provider request IDs) and received_at. This enables replay, signature debugging, and deterministic processing. With Hooque, payloads are persisted at ingest and delivered with metadata so consumers can Ack/Nack/Reject safely.
How do I roll back safely?
Keep the provider URL and handler signature unchanged, deploy the new worker path behind a flag, and be able to route traffic back to the old path quickly. Avoid coupling cutover to schema changes. With Hooque, rollback is often just swapping the provider URL back or pausing consumers while you keep ingest durable.
Start processing webhooks reliably
Ack fast, queue durably, and move side effects into workers with explicit delivery controls.
No credit card required