Payment Webhook Retry, Idempotency and Failure Handling | Adhya Enterprises

The Distributed Reality of Webhook Ingestion

In distributed systems, networks are inherently unreliable. When an authorized payment gateway (such as Razorpay, Cashfree, or Stripe) dispatches a webhook event confirming an order settlement, packet drops or brief application latency can delay the HTTP response.

To guarantee delivery, gateway providers implement automatic retry schedules with exponential backoff. Because network protocols cannot guarantee physical 'exactly-once' delivery across separate servers, the webhook receiver must be engineered for 'effectively-once' business effects through strict idempotency.

Achieving Effectively-Once Business Effects

An idempotent operation is one that can be executed multiple times without changing the initial outcome beyond the first application. In payment processing, receiving three identical payment.captured events must settle the invoice exactly once.

This is accomplished by recording incoming gateway event IDs in an immutable audit table with a unique database constraint before executing business logic.

// app/api/webhooks/payment/route.ts
import { NextResponse } from "next/server";
import prisma from "@/lib/db";
import { verifyWebhookSignature } from "@/lib/payments/crypto";

export async function POST(req: Request) {
  const rawBody = await req.text();
  const signature = req.headers.get("x-webhook-signature") || "";
  const secret = process.env.PAYMENT_WEBHOOK_SECRET || "";

  // 1. Cryptographic HMAC validation
  if (!verifyWebhookSignature(rawBody, signature, secret)) {
    return NextResponse.json({ error: "Invalid signature" }, { status: 401 });
  }

  const payload = JSON.parse(rawBody);
  const eventId = payload.event_id;
  const paymentId = payload.payload?.payment?.entity?.id;
  const invoiceId = payload.payload?.payment?.entity?.notes?.invoiceId;

  // 2. Atomic idempotency registration via database transaction
  try {
    const result = await prisma.$transaction(async (tx) => {
      // Check if event has already been recorded
      const existingEvent = await tx.paymentEvent.findUnique({
        where: { gatewayEventId: eventId },
      });

      if (existingEvent) {
        // Event already processed: safe duplicate acknowledgment
        return { duplicate: true };
      }

      // Record event receipt immediately
      await tx.paymentEvent.create({
        data: {
          gatewayEventId: eventId,
          paymentId: paymentId,
          payload: rawBody,
          status: "PROCESSED",
        },
      });

      // Update invoice status atomically
      await tx.invoice.update({
        where: { id: invoiceId },
        data: { status: "PAID", paidAt: new Date() },
      });

      return { duplicate: false };
    });

    // Return immediate HTTP 200 to acknowledge delivery
    return NextResponse.json({ received: true, duplicate: result.duplicate });
  } catch (error) {
    console.error("Webhook processing error:", error);
    return NextResponse.json({ error: "Processing failed" }, { status: 500 });
  }
}

Preventing Concurrency Race Conditions

When a gateway triggers rapid retries due to network jitter, two webhook requests containing identical payment data can arrive at your server instances within milliseconds of each other.

If both requests check the database simultaneously before either commits, both might determine the invoice is unpaid and trigger duplicate fulfillment actions. Wrapping event verification and invoice updates inside an atomic database transaction with unique constraints guarantees that only one request acquires the write lock.

Handling Out-of-Order Webhook Delivery

Gateway webhooks do not guarantee sequential ordering. During network congestion, an order.paid event could theoretically arrive after a refund.processed notification.

Design your invoice state machine with explicit valid transitions. An invoice in REFUNDED status should reject transitions back to PAID, logging an operational alert for manual review.

Dead-Letter Queues and Asynchronous Decoupling

Gateway webhook timeouts are typically strict (between 5 and 15 seconds). If your handler attempts to generate invoice PDFs, send WhatsApp notifications, and sync third-party CRMs synchronously inside the request loop, response latency will cause gateway delivery timeouts.

Adopt an asynchronous event-driven pattern: verify signature, register event idempotency, return HTTP 200 immediately, and dispatch downstream notifications to an asynchronous worker queue. If worker execution fails, failed jobs route to a Dead-Letter Queue (DLQ) for controlled inspection and retry.

Common Webhook Ingestion Pitfalls

Avoid these critical operational mistakes when handling payment events:

Webhook Reliability Checklist

Verify these controls before activating production webhook webhooks:

Related Engineering Capabilities & Policies

Review our foundational guide on Architecting Secure Payment Gateways & Webhooks in Next.js for end-to-end checkout flow patterns.

For technical automation and webhook microservice engineering, explore Custom API Integrations. Review our Payment Terms for official milestone, settlement, and dispute guidelines.