Designing Payment & Mutation APIs: Idempotency-Key Implementation Guide

If a user taps a "Pay Now" button, the mobile screen freezes for two seconds, and they tap it a second time—does their credit card get charged twice? In a well-engineered backend system, the answer is no: even if the exact same request arrives multiple times over an unreliable network, the underlying business side effect executes only once. This article explains the two building blocks behind that guarantee—Idempotency and the Idempotency-Key HTTP header—complete with database schema design, a NestJS interceptor implementation, and production pitfalls.

1. Understanding Idempotency Through an Everyday Analogy

Think of an elevator call button versus a vending machine coin slot. Pressing the elevator "Up" button once or pressing it ten times in a row produces the exact same state change: one elevator is dispatched to your floor. Inserting a coin into a vending machine ten times, on the other hand, adds money ten times. In software engineering, an operation where applying the same request once or multiple times leaves the server in the exact same state is called idempotent.

The HTTP specification (RFC 9110) defines GET, HEAD, PUT, and DELETE as idempotent methods by contract. Note that idempotency refers to the intended side effect on server state, not necessarily identical HTTP status codes on every call: calling DELETE /posts/42 a second time may return 404 Not Found instead of 200 OK, yet the server state ("post 42 does not exist") remains identical. The real challenge lies in POST endpoints—such as POST /payments, POST /transfers, or POST /orders—which create a brand-new resource or ledger entry on every invocation. Adding an Idempotency-Key header makes non-idempotent POST endpoints safely retryable.

2. Concrete Failure Scenario: The Network Timeout Retry

1. A user initiates a $50 payment from a mobile app.
2. The backend server charges the payment gateway and commits the transaction, but the user's cellular connection drops inside an elevator while the HTTP 200 OK response is travelling back.
3. Because the mobile client timed out without receiving a response, its HTTP client (or the user) automatically retries the exact same POST /payments request.
4. Without idempotency tracking, the server treats the retry as a second independent order and charges the customer another $50.

In a distributed network, a client can never distinguish between "the request never reached the server" and "the server succeeded, but the response was lost in transit." Instead of forbidding client retries, the server must recognize duplicate retries and replay the original result.

3. How the `Idempotency-Key` Protocol Lifecycle Works

Before sending a state-changing request, the client generates a high-entropy unique identifier (typically a UUID v4) per logical user intent, attaches it in the Idempotency-Key header, and reuses that exact same key across any network retries of the same operation. The server processes the header in four steps:

1. Atomic Key Claim: If the key does not exist yet, insert it atomically in an in_progress state (using a database PRIMARY KEY / UNIQUE constraint or Redis SET ... NX).
2. Request Payload Fingerprint Verification: Store a cryptographic hash (e.g., SHA-256) of the request parameters alongside the key. If a retry arrives with the same Idempotency-Key but a different payment amount or recipient, reject it immediately.
3. Execute & Persist Response: Once the business transaction completes, store the final HTTP status code and serialized response body in the idempotency record and transition its status to completed.
4. Short-Circuit Subsequent Retries: If a request arrives for a key already marked completed, skip the business logic entirely and return the stored status code and response body.

The IETF specification draft (draft-ietf-httpapi-idempotency-key-header) recommends returning 400 Bad Request when a required key is missing, 422 Unprocessable Content when a key is reused with a mismatched request payload, and 409 Conflict when a duplicate request arrives while the first request is still in_progress.

4. Relational Storage Schema (SQL)

For financial ledger operations, storing idempotency records in the same relational database and ACID transaction as the business tables eliminates split-brain failures where a payment commits in PostgreSQL but a separate Redis cache fails to record the completed key. Always scope the primary key by `(user_id, idempotency_key)` so one user can never collide with or probe another user's keys:

CREATE TABLE idempotency_keys (
  user_id          BIGINT       NOT NULL,
  idempotency_key  VARCHAR(255) NOT NULL,
  request_hash     CHAR(64)     NOT NULL,  -- SHA-256 hex digest of canonical request body
  status           VARCHAR(20)  NOT NULL,  -- 'in_progress' | 'completed'
  response_code    INT,
  response_body    TEXT,
  created_at       TIMESTAMP    NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (user_id, idempotency_key)
);

If two identical requests arrive at the exact same millisecond, the composite `PRIMARY KEY` guarantees that only one `INSERT` succeeds; the second transaction catches the unique constraint violation, reads the existing row, sees `status = 'in_progress'`, and returns `409 Conflict`. A scheduled background job purges keys older than your retention window (typically 24 to 72 hours).

5. NestJS Interceptor Implementation Example

Below is a streamlined NestJS `NestInterceptor` demonstrating the lifecycle. Notice two critical implementation details: we chain the asynchronous `store.complete()` call using RxJS concatMap (not map), and we release the `in_progress` claim inside catchError if a transient failure occurs before completion:

@Injectable()
export class IdempotencyInterceptor implements NestInterceptor {
  constructor(private readonly store: IdempotencyStore) {}

  async intercept(ctx: ExecutionContext, next: CallHandler): Promise<Observable<unknown>> {
    const req = ctx.switchToHttp().getRequest<Request>();
    const key = req.header('idempotency-key');
    if (!key) throw new BadRequestException('Missing Idempotency-Key header');

    const requestHash = sha256(JSON.stringify(req.body));
    const found = await this.store.begin(key, requestHash); // Atomic claim

    if (found.state === 'in_progress') throw new ConflictException('Request already in progress');
    if (found.state === 'hash_mismatch') throw new UnprocessableEntityException('Payload mismatch for key');
    if (found.state === 'completed') return of(found.body);

    return next.handle().pipe(
      concatMap(async (body: unknown) => {
        await this.store.complete(key, requestHash, body);
        return body;
      }),
      catchError(async (err: unknown) => {
        await this.store.release(key); // Release claim so the client can safely retry
        throw err;
      }),
    );
  }
}

A common RxJS bug in NestJS interceptors is writing `map(async (body) => ...)`—because `map` does not flatten Promises, the observable emits an unresolved Promise and database errors inside `store.complete()` bypass `catchError`. Always use `concatMap` or `mergeMap` for async side effects in RxJS pipelines.

6. When You Need `Idempotency-Key` (and When You Don't)

API Endpoint Category	Need for Idempotency-Key
Payments, bank transfers, point deductions, order creation	Mandatory
POST endpoints that trigger external emails, SMS, or webhooks	Strongly Recommended (prevents duplicate notifications)
Read-only GET queries, full-replacement PUT updates	Unnecessary (inherently idempotent)
Simple bookmark / like toggles	Usually satisfied by a database UNIQUE (user_id, post_id) constraint

7. Common Production Mistakes

• Non-atomic Check-Then-Act when claiming a key: Running a `SELECT` followed by a separate `INSERT` without a unique constraint or atomic lock allows concurrent duplicate requests to slip through together.
• Skipping request body hash validation: Returning a cached receipt when the client accidentally reused an old key for a completely different cart or amount causes silent data corruption.
• Leaving failed requests stuck in `in_progress` forever: If an unhandled exception aborts execution without rolling back or releasing the key, the user is locked out until the TTL expires.
• Generating a brand-new UUID on every retry loop in the client: If the frontend generates the UUID inside the retry wrapper instead of when the user clicks the button, every retry has a different key and idempotency is defeated.

Preventing two concurrent requests with the same key from executing simultaneously relies on atomic locking primitives. For deeper dives into locking strategies across databases and Redis, read Redis Distributed Locks & Fencing Tokens and Pessimistic vs. Optimistic Database Concurrency Locks.

8. Summary

References

• IETF Draft - The Idempotency-Key HTTP Header Field
• Stripe API Reference - Idempotent Requests
• RFC 9110 - HTTP Semantics: Idempotent Methods
• NestJS Documentation - Interceptors