Idempotency is not a theoretical property. It's the difference between a payment that charges once and one that charges three times when the client retries. Here's how to build it correctly.
A network request can fail in three ways: the server never received it, the server processed it but the response was lost, or the server processed it and the response arrived fine. From the client's perspective, the first two cases look identical: a timeout. If the client retries, it may process the request twice. Idempotency is the property that makes retrying safe: calling an operation multiple times has the same effect as calling it once. For most endpoints this doesn't matter much. For operations that send emails, charge credit cards, or transfer money, it matters enormously.
Stripe's approach is the industry standard for payment APIs: the client generates a unique idempotency key (typically a UUID) before making the request and includes it in the header. The server stores the key with the result of the operation. If the same key arrives again (retry), the server returns the stored result without re-executing. The key is unique per intended operation, not per HTTP request. If the client wants to charge a card twice for two separate purchases, it generates two keys. The implementation requires a durable key store (a database table with a unique constraint on the key), not an in-memory cache. The server may restart between the original request and the retry.
For operations that write to a database, the cleanest idempotency implementation is a unique constraint on the business key. An email notification table with a unique constraint on (user_id, template_id, date) cannot send the same daily digest twice. The second INSERT fails with a unique violation, which is caught and treated as a success. This approach works for inserts. For updates, compare-and-swap (UPDATE ... WHERE current_value = expected_value) achieves the same effect: the update only applies if the row is still in the state the caller expected, which means re-applying the same update to an already-updated row is a no-op.
Every Kafka consumer, SQS consumer, or queue worker must be idempotent. Queues guarantee at-least-once delivery: a message may be delivered more than once due to consumer crashes, rebalances, or network retries. If your consumer charges a payment, sends an email, or makes a database write on every message, duplicate delivery causes duplicate effects. The standard solution: before processing, insert a record into a processed_messages table with a unique constraint on the message ID. If the insert succeeds, process the message. If it fails (duplicate key), the message was already processed. Ack it and move on.
A subtle idempotency problem: you want to write to your database and publish an event to a queue atomically. If you write first and then publish, the service may crash between the two steps: the write happened but the event never fires. If you publish first and then write, the event fires but the write never happens. The outbox pattern solves this: write the database row and an outbox_events row in the same transaction. A background poller reads unpublished outbox events and publishes them to the queue, then marks them published. Because polling is idempotent (it re-processes any unacknowledged event), and the queue consumer is idempotent (it deduplicates by event ID), the entire pipeline is safe to retry at any step.
Idempotency prevents duplicate processing of the same operation. It does not prevent multiple distinct operations. If the client generates a new UUID for each retry instead of reusing the original key, each retry is treated as a new operation. Charges happen three times regardless of idempotency keys. Client-side idempotency key management is part of the contract. Idempotency also doesn't prevent race conditions between concurrent requests with different keys that are logically conflicting (two users booking the same seat). That requires separate concurrency control like pessimistic locking or compare-and-swap.