Guide

Verify GitLab webhook signature headers, replay defense, rotation

Minimal correct verification for GitLab: exact headers and signing inputs, replay protection, and a safe secret rotation workflow. Includes runnable Node + Python snippets.

No credit card required. See pricing.

TL;DR

  • Verify GitLab authenticity before parsing or side effects.
  • Verify over the raw request body bytes; avoid JSON re-stringify.
  • Required headers: "x-gitlab-token".
  • Add replay protection: store a dedupe key for a TTL window and reject repeats.
  • Rotate secrets without downtime: accept {current, previous} during overlap, then retire.
Also read the matching spokes: local development and retries & backoff .

Debugging auth failures? Use the webhook debugging playbook .

Anti-patterns

  • Parsing/transforming the body before verification (breaks signing inputs).
  • Returning 2xx before authenticity is proven (you just acknowledged an attacker).
  • Skipping replay protection (signed requests can still be replayed).

If verification fails in production, capture the raw body + headers and follow the debugging playbook .

Core concepts

Secure webhooks by verifying authenticity deterministically: exact headers, exact signing input, and no side effects until verification passes.

Threat model (start here)

Forgery, replay, leaked secrets/tokens, and tenant confusion are the common webhook incidents. Design verification and secret lookup to fail closed.

Exact verification inputs

This strategy verifies Shared secret token (constant-time compare) using x-gitlab-token === secret. Verify before parsing or side effects.

Replay + rotation

Signed requests can still be replayed. Store a dedupe key for a TTL window and rotate secrets with an overlap window (current + previous).

Required headers (from the ingest strategy)

Confirm these are present before debugging signature math.

x-gitlab-token

Need workflows around tunnels/replay? See local development (gitlab) and retries & backoff (gitlab) .

Normal flow vs With Hooque flow

The goal is the same either way: authenticate first, then make retries/replays harmless with idempotency.

Normal security flow

  1. Request hits your endpoint.
  2. Verify signature/token before parsing or side effects.
  3. Add a dedupe key + TTL window to stop replays.
  4. Rotate secrets with overlap (current + previous).
  5. If your endpoint is down (deploy/offline/sleep), deliveries fail and you rely on retries/replays.

With Hooque

  1. Provider → Hooque ingest (verification at ingest time).
  2. Only verified requests are queued.
  3. Workers pull/stream events and ack/nack/reject explicitly.
  4. UI provides inspection + controlled redelivery after fixes.
  5. Ingest stays online when your worker is offline; events queue until you resume.

Production checklist

Use this when shipping GitLab webhook verification (and when debugging signature failures).

- [ ] HTTPS only; reject plain HTTP
- [ ] Body size limit + content-type allowlist (prefer strict in production)
- [ ] Capture the raw request body bytes for verification
- [ ] Verify required headers: `x-gitlab-token`
- [ ] Verify signature/token before parsing or side effects
- [ ] Constant-time compare for signatures/tokens (avoid timing leaks)
- [ ] Replay protection (dedupe key + TTL window)
- [ ] Dedupe on a stable key (provider event ID when available, otherwise hash of signature+body)
- [ ] Secret storage in a secret manager/KMS (avoid hardcoding)
- [ ] Secret rotation: accept `{current, previous}` during overlap, then retire
- [ ] Multi-tenant safety: scope secrets to the webhook/tenant identity
- [ ] Structured logs for verification failures (never log secrets)
- [ ] Alert on spikes in auth failures (misconfig or attack)

Reference implementation

Minimal correct verification, replay protection, and secret rotation (Node + Python).

1) Minimal verification

Verify authenticity before parsing or side effects.

Node

Express raw body + verification

// Minimal GitLab token header verification (Express)
// npm i express
import crypto from "crypto";
import express from "express";

const app = express();
app.use(express.json({ limit: "2mb" }));

app.post("/webhook", (req, res) => {
  const secret = process.env.WEBHOOK_SECRET ?? "replace_me";
  const token = req.header("x-gitlab-token") ?? "";
  if (!token) return res.status(401).send("missing x-gitlab-token");
  if (!timingSafeEqualString(token, secret)) return res.status(401).send("bad token");
  return res.sendStatus(202);
});

app.listen(3000, () => console.log("listening on :3000"));


function timingSafeEqualString(a, b) {
  const aBuf = Buffer.from(a, "utf8");
  const bBuf = Buffer.from(b, "utf8");
  if (aBuf.length !== bBuf.length) return false;
  return crypto.timingSafeEqual(aBuf, bBuf);
}

Python

FastAPI raw body + verification

# Minimal GitLab token header verification (FastAPI)
# pip install fastapi uvicorn
import hmac
import os
from fastapi import FastAPI, Request, Response

app = FastAPI()

@app.post("/webhook")
async def webhook(request: Request) -> Response:
    secret = os.getenv("WEBHOOK_SECRET") or "replace_me"
    token = request.headers.get("x-gitlab-token", "")
    if not token:
        return Response("missing x-gitlab-token", status_code=401)
    if not hmac.compare_digest(token, secret):
        return Response("bad token", status_code=401)
    return Response(status_code=202)

2) Replay protection

Use timestamp max-age when available, plus a dedupe window so retries and replays are harmless.

Node

Timestamp max-age + dedupe window

// Replay protection: timestamp max-age (when available) + dedupe window (in-memory example)
import crypto from "crypto";

const seen = new Map(); // key -> expiresAtMs

function purgeSeen() {
  const now = Date.now();
  for (const [key, expiresAt] of seen.entries()) if (expiresAt <= now) seen.delete(key);
}

// Inside your verified webhook handler:
purgeSeen();
const sig = req.header("x-gitlab-token") ?? "";

// Prefer a provider delivery/event ID when available. Otherwise hash signature + raw body.
const rawBody = Buffer.isBuffer(req.body) ? req.body : Buffer.from(req.body ?? "");
const key = crypto.createHash("sha256").update(sig).update(".").update(rawBody).digest("hex");
const ttlMs = 10 * 60 * 1000;
if (seen.has(key)) return res.status(409).send("replay detected");
seen.set(key, Date.now() + ttlMs);

Python

Timestamp max-age + dedupe window

# Replay protection: timestamp max-age (when available) + dedupe window (in-memory example)
import hashlib
import time

seen: dict[str, float] = {}  # key -> expires_at_epoch_seconds

def purge_seen() -> None:
    now = time.time()
    for k, exp in list(seen.items()):
        if exp <= now:
            del seen[k]

# Inside your verified webhook handler:
purge_seen()
sig = request.headers.get("x-gitlab-token", "")

raw_body = await request.body()
key = hashlib.sha256((sig + ".").encode("utf-8") + raw_body).hexdigest()
ttl_seconds = 10 * 60
if key in seen:
    return Response("replay detected", status_code=409)
seen[key] = time.time() + ttl_seconds

3) Secret rotation (current + previous overlap)

Accept two secrets during an overlap window, then retire the previous secret.

Node

Try both secrets during overlap

// Secret rotation for token headers: accept {current, previous} during overlap
import crypto from "crypto";

function verifyTokenWithSecrets(headers, secrets) {
  const token = headers["x-gitlab-token"] ?? "";
  if (!token) return false;
  for (const secret of secrets) {
    if (timingSafeEqualString(token, secret)) return true;
  }
  return false;
}

const secrets = [process.env.WEBHOOK_SECRET_CURRENT, process.env.WEBHOOK_SECRET_PREVIOUS].filter(Boolean);

function timingSafeEqualString(a, b) {
  const aBuf = Buffer.from(a, "utf8");
  const bBuf = Buffer.from(b, "utf8");
  if (aBuf.length !== bBuf.length) return false;
  return crypto.timingSafeEqual(aBuf, bBuf);
}

Python

Try both secrets during overlap

# Secret rotation for token headers: accept {current, previous} during overlap
import hmac
import os

def verify_token_with_secrets(headers: dict, secrets: list[str]) -> bool:
    token = headers.get("x-gitlab-token", "")
    if not token:
        return False
    for secret in secrets:
        if hmac.compare_digest(token, secret):
            return True
    return False

secrets = [os.getenv("WEBHOOK_SECRET_CURRENT"), os.getenv("WEBHOOK_SECRET_PREVIOUS")]
secrets = [s for s in secrets if s]

For idempotency patterns and safe backoff policies, read GitLab retries & backoff .

Common failure modes

Most security failures are configuration drift: wrong secrets, missing headers, or verifying the wrong bytes.

All GitLab webhooks start failing verification

Likely causes

  • Wrong secret/token configured (test vs prod mismatch).
  • Missing/renamed headers (x-gitlab-token).
  • Body parsing/middleware changed the signing input.

Next checks

  • Confirm the secret value and environment match the sender.
  • Log which headers are present (names only) and verify casing/normalization.
  • Verify over raw bytes and ensure no transformation before verification.

Duplicate deliveries cause duplicate side effects

Likely causes

  • No dedupe store/TTL window.
  • Handler is not idempotent.
  • Retries/replays treated as new events.

Next checks

  • Choose a stable dedupe key and store it with TTL.
  • Make side effects idempotent (unique constraints, idempotency keys).
  • Use a queue/worker model and explicit retry controls.
Incident workflow: debugging playbook.

How Hooque helps

Verify at ingest and only process verified events — then use the queue to make retries/replays safe and auditable.

  • Hosted ingest for GitLab with strategy-specific verification at ingest time.
  • Only verified events enter the queue; unverified requests never reach your workers.
  • Durable queue decouples ingest from processing (burst protection).
  • Inspection + controlled redelivery after fixes (with an audit trail).
  • Strategy-specific docs: local dev (gitlab) and retries & backoff (gitlab).

Start with signup, review pricing, and keep the debugging playbook handy for incidents.

FAQ

Short answers to the most common verification and incident-response questions.

Do I need the raw request body to verify GitLab webhook signatures?

General: Yes — signature schemes are computed over the exact bytes sent. Parsing JSON and re-stringifying can change whitespace/ordering and break verification. (x-gitlab-token) How Hooque helps: Hooque verifies at ingest before your app code runs, so you avoid framework-specific body parsing pitfalls.

What’s a webhook replay attack and how do I prevent it?

General: A replay attack is when a valid, signed request is captured and sent again. Use a timestamp max-age when available and store a dedupe key (event ID or a hash) for a TTL window. How Hooque helps: Only verified requests enter the queue, and you can implement dedupe/idempotency in workers while using UI redelivery for safe replays.

How do I rotate webhook secrets without downtime?

General: Accept `{current, previous}` secrets during an overlap window, verify against both, then retire the previous secret. Rotate per environment and per tenant. How Hooque helps: You can update verification secrets in one place (ingest) and keep workers stable while rotations happen.

Should I rely on IP allowlists to secure webhooks?

General: IP allowlists are brittle (IPs change; shared infrastructure). Prefer cryptographic verification (HMAC/public key) as the primary control, and use HTTPS + rate limits as defense-in-depth. How Hooque helps: Ingest-time verification reduces the attack surface of your own infrastructure and keeps unverified traffic out of your worker systems.

What status code should I return for invalid signatures?

General: Use `401`/`403` for authentication failures and avoid returning `2xx` before authenticity is proven. How Hooque helps: Hooque can reject invalid requests at ingest so only verified events are available to consumers.

How do I avoid tenant confusion in a multi-tenant webhook endpoint?

General: Bind secret lookup to the webhook identity/tenant and never “try secrets until something matches” across tenants. Scope verification to the correct tenant first. How Hooque helps: Webhooks are isolated per workspace/webhook, so verification config is naturally scoped and auditable.

Why does signature verification fail only in production?

General: Common causes are wrong secrets (test vs prod), proxies/middleware changing the body, header normalization differences, and clock skew for timestamp-based schemes. How Hooque helps: A consistent ingest layer reduces environment differences, and the dashboard makes it easier to inspect failed deliveries and debug safely.

Start processing webhooks reliably

Verify at ingest, queue durably, and consume with explicit ack/nack/reject — without building custom webhook infrastructure.

Start for free

No credit card required