Verify PagerDuty webhook signature headers, replay defense, rotation
Minimal correct verification for PagerDuty: exact headers and signing inputs, replay protection, and a safe secret rotation workflow.
Includes runnable Node + Python snippets.
No credit card required. See pricing.
TL;DR
- Verify PagerDuty authenticity before parsing or side effects.
- Verify over the raw request body bytes; avoid JSON re-stringify.
- Required headers: "x-pagerduty-signature".
- Add replay protection: store a dedupe key for a TTL window and reject repeats.
- Rotate secrets without downtime: accept {current, previous} during overlap, then retire.
Debugging auth failures? Use the webhook debugging playbook .
Anti-patterns
- Parsing/transforming the body before verification (breaks signing inputs).
- Returning 2xx before authenticity is proven (you just acknowledged an attacker).
- Skipping replay protection (signed requests can still be replayed).
If verification fails in production, capture the raw body + headers and follow the debugging playbook .
Core concepts
Secure webhooks by verifying authenticity deterministically: exact headers, exact signing input, and no side effects until verification passes.
Threat model (start here)
Forgery, replay, leaked secrets/tokens, and tenant confusion are the common webhook incidents. Design verification and secret lookup to fail closed.
Exact verification inputs
This strategy verifies HMAC-SHA256 (hex) using HMAC_SHA256(secret, `v1:\${payload}`); header contains `v1=<sig>` entries. Verify before parsing or side effects.
Replay + rotation
Signed requests can still be replayed. Store a dedupe key for a TTL window and rotate secrets with an overlap window (current + previous).
Required headers (from the ingest strategy)
Confirm these are present before debugging signature math.
- Hooque supports multiple `v1=...` signatures in a comma-separated header.
Need workflows around tunnels/replay? See local development (pagerduty) and retries & backoff (pagerduty) .
Normal flow vs With Hooque flow
The goal is the same either way: authenticate first, then make retries/replays harmless with idempotency.
Normal security flow
- Request hits your endpoint.
- Verify signature/token before parsing or side effects.
- Add a dedupe key + TTL window to stop replays.
- Rotate secrets with overlap (current + previous).
- If your endpoint is down (deploy/offline/sleep), deliveries fail and you rely on retries/replays.
With Hooque
- Provider → Hooque ingest (verification at ingest time).
- Only verified requests are queued.
- Workers pull/stream events and ack/nack/reject explicitly.
- UI provides inspection + controlled redelivery after fixes.
- Ingest stays online when your worker is offline; events queue until you resume.
Production checklist
Use this when shipping PagerDuty webhook verification (and when debugging signature failures).
- [ ] HTTPS only; reject plain HTTP
- [ ] Body size limit + content-type allowlist (prefer strict in production)
- [ ] Capture the raw request body bytes for verification
- [ ] Verify required headers: `x-pagerduty-signature`
- [ ] Verify signature/token before parsing or side effects
- [ ] Constant-time compare for signatures/tokens (avoid timing leaks)
- [ ] Replay protection (dedupe key + TTL window)
- [ ] Dedupe on a stable key (provider event ID when available, otherwise hash of signature+body)
- [ ] Secret storage in a secret manager/KMS (avoid hardcoding)
- [ ] Secret rotation: accept `{current, previous}` during overlap, then retire
- [ ] Multi-tenant safety: scope secrets to the webhook/tenant identity
- [ ] Structured logs for verification failures (never log secrets)
- [ ] Alert on spikes in auth failures (misconfig or attack) Reference implementation
Minimal correct verification, replay protection, and secret rotation (Node + Python).
1) Minimal verification
Verify authenticity before parsing or side effects.
Node
Express raw body + verification
// Minimal PagerDuty verification (Express raw body)
// npm i express
import crypto from "crypto";
import express from "express";
const app = express();
app.post("/webhook", express.raw({ type: "*/*", limit: "2mb" }), (req, res) => {
const secret = process.env.WEBHOOK_SECRET ?? "replace_me";
const signatureHeader = req.header("x-pagerduty-signature") ?? "";
const rawBody = Buffer.isBuffer(req.body) ? req.body : Buffer.from(req.body ?? "");
const payload = rawBody.toString("utf8");
if (!signatureHeader) return res.status(401).send("missing x-pagerduty-signature");
const expected = crypto.createHmac("sha256", secret).update(`v1:${payload}`).digest("hex");
// Header can contain multiple signatures: v1=xxxx,v1=yyyy
const signatures = signatureHeader.split(",");
for (const sig of signatures) {
const [version, value] = sig.trim().split("=");
if (version !== "v1") continue;
if (timingSafeEqualString(value, expected)) return res.sendStatus(202);
}
return res.status(401).send("bad signature");
});
app.listen(3000, () => console.log("listening on :3000"));
function timingSafeEqualString(a, b) {
const aBuf = Buffer.from(a, "utf8");
const bBuf = Buffer.from(b, "utf8");
if (aBuf.length !== bBuf.length) return false;
return crypto.timingSafeEqual(aBuf, bBuf);
} Python
FastAPI raw body + verification
# Minimal PagerDuty verification (FastAPI raw body)
# pip install fastapi uvicorn
import hashlib
import hmac
import os
from fastapi import FastAPI, Request, Response
app = FastAPI()
@app.post("/webhook")
async def webhook(request: Request) -> Response:
secret = (os.getenv("WEBHOOK_SECRET") or "replace_me").encode("utf-8")
signature_header = request.headers.get("x-pagerduty-signature", "")
raw_body = await request.body()
payload = raw_body.decode("utf-8")
if not signature_header:
return Response("missing x-pagerduty-signature", status_code=401)
expected = hmac.new(secret, f"v1:{payload}".encode("utf-8"), hashlib.sha256).hexdigest()
for part in signature_header.split(","):
version, _, value = part.strip().partition("=")
if version != "v1":
continue
if hmac.compare_digest(value, expected):
return Response(status_code=202)
return Response("bad signature", status_code=401) 2) Replay protection
Use timestamp max-age when available, plus a dedupe window so retries and replays are harmless.
Node
Timestamp max-age + dedupe window
// Replay protection: timestamp max-age (when available) + dedupe window (in-memory example)
import crypto from "crypto";
const seen = new Map(); // key -> expiresAtMs
function purgeSeen() {
const now = Date.now();
for (const [key, expiresAt] of seen.entries()) if (expiresAt <= now) seen.delete(key);
}
// Inside your verified webhook handler:
purgeSeen();
const sig = req.header("x-pagerduty-signature") ?? "";
// Prefer a provider delivery/event ID when available. Otherwise hash signature + raw body.
const rawBody = Buffer.isBuffer(req.body) ? req.body : Buffer.from(req.body ?? "");
const key = crypto.createHash("sha256").update(sig).update(".").update(rawBody).digest("hex");
const ttlMs = 10 * 60 * 1000;
if (seen.has(key)) return res.status(409).send("replay detected");
seen.set(key, Date.now() + ttlMs); Python
Timestamp max-age + dedupe window
# Replay protection: timestamp max-age (when available) + dedupe window (in-memory example)
import hashlib
import time
seen: dict[str, float] = {} # key -> expires_at_epoch_seconds
def purge_seen() -> None:
now = time.time()
for k, exp in list(seen.items()):
if exp <= now:
del seen[k]
# Inside your verified webhook handler:
purge_seen()
sig = request.headers.get("x-pagerduty-signature", "")
raw_body = await request.body()
key = hashlib.sha256((sig + ".").encode("utf-8") + raw_body).hexdigest()
ttl_seconds = 10 * 60
if key in seen:
return Response("replay detected", status_code=409)
seen[key] = time.time() + ttl_seconds 3) Secret rotation (current + previous overlap)
Accept two secrets during an overlap window, then retire the previous secret.
Node
Try both secrets during overlap
// PagerDuty secret rotation: try {current, previous} during overlap
import crypto from "crypto";
function timingSafeEqualString(a, b) {
const aBuf = Buffer.from(a, "utf8");
const bBuf = Buffer.from(b, "utf8");
if (aBuf.length !== bBuf.length) return false;
return crypto.timingSafeEqual(aBuf, bBuf);
}
function verifyPagerDutyWithSecrets(payload, signatureHeader, secrets) {
const signatures = signatureHeader.split(",");
for (const secret of secrets) {
const expected = crypto.createHmac("sha256", secret).update(`v1:${payload}`).digest("hex");
for (const sig of signatures) {
const [version, value] = sig.trim().split("=");
if (version !== "v1") continue;
if (timingSafeEqualString(value, expected)) return true;
}
}
return false;
}
const secrets = [process.env.WEBHOOK_SECRET_CURRENT, process.env.WEBHOOK_SECRET_PREVIOUS].filter(Boolean); Python
Try both secrets during overlap
# PagerDuty secret rotation: try {current, previous} during overlap
import hashlib
import hmac
import os
def verify_pagerduty_with_secrets(payload: str, signature_header: str, secrets: list[bytes]) -> bool:
signatures = [p.strip() for p in signature_header.split(",") if p.strip()]
for secret in secrets:
expected = hmac.new(secret, f"v1:{payload}".encode("utf-8"), hashlib.sha256).hexdigest()
for part in signatures:
version, _, value = part.partition("=")
if version == "v1" and hmac.compare_digest(value, expected):
return True
return False
secrets = [os.getenv("WEBHOOK_SECRET_CURRENT"), os.getenv("WEBHOOK_SECRET_PREVIOUS")]
secrets = [s.encode("utf-8") for s in secrets if s] For idempotency patterns and safe backoff policies, read PagerDuty retries & backoff .
Common failure modes
Most security failures are configuration drift: wrong secrets, missing headers, or verifying the wrong bytes.
All PagerDuty webhooks start failing verification
Likely causes
- Wrong secret/token configured (test vs prod mismatch).
- Missing/renamed headers (x-pagerduty-signature).
- Body parsing/middleware changed the signing input.
Next checks
- Confirm the secret value and environment match the sender.
- Log which headers are present (names only) and verify casing/normalization.
- Verify over raw bytes and ensure no transformation before verification.
Duplicate deliveries cause duplicate side effects
Likely causes
- No dedupe store/TTL window.
- Handler is not idempotent.
- Retries/replays treated as new events.
Next checks
- Choose a stable dedupe key and store it with TTL.
- Make side effects idempotent (unique constraints, idempotency keys).
- Use a queue/worker model and explicit retry controls.
How Hooque helps
Verify at ingest and only process verified events — then use the queue to make retries/replays safe and auditable.
- Hosted ingest for PagerDuty with strategy-specific verification at ingest time.
- Only verified events enter the queue; unverified requests never reach your workers.
- Durable queue decouples ingest from processing (burst protection).
- Inspection + controlled redelivery after fixes (with an audit trail).
- Strategy-specific docs: local dev (pagerduty) and retries & backoff (pagerduty).
Start with signup, review pricing, and keep the debugging playbook handy for incidents.
FAQ
Short answers to the most common verification and incident-response questions.
Do I need the raw request body to verify PagerDuty webhook signatures?
General: Yes — signature schemes are computed over the exact bytes sent. Parsing JSON and re-stringifying can change whitespace/ordering and break verification. (x-pagerduty-signature) How Hooque helps: Hooque verifies at ingest before your app code runs, so you avoid framework-specific body parsing pitfalls.
What’s a webhook replay attack and how do I prevent it?
General: A replay attack is when a valid, signed request is captured and sent again. Use a timestamp max-age when available and store a dedupe key (event ID or a hash) for a TTL window. How Hooque helps: Only verified requests enter the queue, and you can implement dedupe/idempotency in workers while using UI redelivery for safe replays.
How do I rotate webhook secrets without downtime?
General: Accept `{current, previous}` secrets during an overlap window, verify against both, then retire the previous secret. Rotate per environment and per tenant. How Hooque helps: You can update verification secrets in one place (ingest) and keep workers stable while rotations happen.
Should I rely on IP allowlists to secure webhooks?
General: IP allowlists are brittle (IPs change; shared infrastructure). Prefer cryptographic verification (HMAC/public key) as the primary control, and use HTTPS + rate limits as defense-in-depth. How Hooque helps: Ingest-time verification reduces the attack surface of your own infrastructure and keeps unverified traffic out of your worker systems.
What status code should I return for invalid signatures?
General: Use `401`/`403` for authentication failures and avoid returning `2xx` before authenticity is proven. How Hooque helps: Hooque can reject invalid requests at ingest so only verified events are available to consumers.
How do I avoid tenant confusion in a multi-tenant webhook endpoint?
General: Bind secret lookup to the webhook identity/tenant and never “try secrets until something matches” across tenants. Scope verification to the correct tenant first. How Hooque helps: Webhooks are isolated per workspace/webhook, so verification config is naturally scoped and auditable.
Why does signature verification fail only in production?
General: Common causes are wrong secrets (test vs prod), proxies/middleware changing the body, header normalization differences, and clock skew for timestamp-based schemes. How Hooque helps: A consistent ingest layer reduces environment differences, and the dashboard makes it easier to inspect failed deliveries and debug safely.
Start processing webhooks reliably
Verify at ingest, queue durably, and consume with explicit ack/nack/reject — without building custom webhook infrastructure.
No credit card required