Designing Idempotency Keys for Safe Retries

October 9, 2026 · Java Backend Engineering API Design Distributed Systems Idempotency

The Silent Killer of Distributed Systems

You are building the next great fintech platform. Everything is going smoothly until a black Friday sale hits. Your infrastructure is scaling, but then the complaints start rolling in. Users are reporting that they were charged twice for the same purchase. Your database integrity checks are passing, but your business logic is bleeding money.

You pull up the logs and realize the culprit: network timeouts. A client sent a payment request, the network hiccuped, the client didn’t receive a response, and it retried the same request. Your backend, being stateless and eager to help, processed the payment again.

This is the classic problem of non-idempotent operations in distributed systems. The solution is not to fix the network (impossible) or to tell clients not to retry (bad advice). The solution is Idempotency.

In this post, we will dive deep into designing idempotency keys, implementing them in Java, and storing them efficiently to ensure your system remains safe even when the network fails.

What is Idempotency?

In mathematics and computer science, idempotency is the property of certain operations in which applying them multiple times yields the same result as applying them once.

For an HTTP API, this means:

• GET requests are naturally idempotent. Calling /users/123 five times returns the same user five times.
• POST requests are typically not idempotent. Calling /payments five times creates five payments.
• PUT and DELETE requests are idempotent. Deleting a resource five times results in the resource being deleted once (and subsequent deletes return 404).

The problem arises with POST requests that have side effects, such as charging a credit card, sending an email, or updating inventory. When a client retries a POST request due to a timeout, you must ensure the side effect happens exactly once.

The Role of Idempotency Keys

An idempotency key is a unique identifier provided by the client for a specific operation. It acts as a fingerprint for the request. When your server receives a request, it checks if it has already processed a request with that key.

• If the key exists and the request was successful, return the cached response immediately.
• If the key exists but the request failed, you can choose to retry or return an error, depending on your strategy.
• If the key does not exist, process the request, store the key, and return the response.

The key must be unique per operation and persistent across retries. It is usually generated by the client and passed in a header, such as Idempotency-Key.

Designing the Key

How should the idempotency key be generated? There are two main approaches:

1. Client-Generated UUIDs

The client generates a random UUID for each unique operation. This is the most common approach because it shifts the burden of uniqueness to the client. The server just validates the format.

String idempotencyKey = UUID.randomUUID().toString();

Pros:

• Simple to implement.
• No server-side coordination required for key generation.
• Universally accepted standard.

Cons:

• The client must ensure it does not reuse the same UUID for different operations.
• If the client crashes and restarts, it might generate a new UUID and retry with a different key, defeating the purpose.

2. Content-Hashed Keys

The client generates a hash of the request body and relevant headers. For example, if the request is to pay $100 to user 123, the key could be SHA256("pay:100:123").

String payload = "amount=100&userId=123";
String key = DigestUtils.sha256Hex(payload);

Pros:

• Deterministic. If the client retries the exact same operation, it generates the same key.
• Prevents accidental retries with different keys.

Cons:

• Complex to implement correctly. You must include all fields that affect the outcome.
• If the client changes a single character in the payload, it gets a new key and the operation is processed again.
• Not suitable for all scenarios (e.g., if the server adds timestamps or nonces).

Recommendation: For most fintech and e-commerce applications, UUIDs generated by the client are preferred. They are simpler, less error-prone, and the client can manage state to ensure it retries with the same key.

Storage Strategies

Where do you store the idempotency keys? The storage must be fast, durable, and support atomic checks.

Option 1: In-Memory Cache (Redis/Memcached)

This is the most popular choice for high-performance systems.

# Redis Configuration
redis:
  host: redis-cluster.internal
  port: 6379
  key-prefix: "idempotency:"
  ttl: 24h

Workflow:

1. Client sends request with Idempotency-Key: abc-123.
2. Server checks Redis for idempotency:abc-123.
3. If it exists, return the stored response.
4. If not, process the request.
5. Store the response in Redis with a TTL (Time To Live) of 24 hours.

Pros:

• Extremely fast (sub-millisecond latency).
• Simple implementation.
• TTL handles cleanup automatically.

Cons:

• If Redis goes down, you lose the ability to deduplicate. However, this is better than processing duplicate charges.
• Memory usage can grow if TTLs are long.

Option 2: Database with Unique Constraint

You can store idempotency keys in your primary database (PostgreSQL, MySQL).

CREATE TABLE idempotency_keys (
    idempotency_key VARCHAR(255) PRIMARY KEY,
    response_payload TEXT NOT NULL,
    status VARCHAR(50) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Workflow:

1. Insert the key with a UNIQUE constraint.
2. If the insert succeeds, process the request.
3. If the insert fails with a duplicate key error, the request has already been processed. Return the stored response.

Pros:

• ACID compliant. Guarantees no duplicates.
• No separate infrastructure to manage.

Cons:

• Slower than Redis (database locks, disk I/O).
• Can become a bottleneck under high load.
• Requires careful transaction management.

Option 3: Hybrid Approach

For critical financial transactions, you can use a hybrid approach. Use Redis for fast lookup and the database for durable storage. If Redis fails, fall back to the database. This adds complexity but provides the highest reliability.

Implementing Idempotency in Java

Let’s look at a practical implementation using Spring Boot and Redis. We will create a filter that intercepts requests, checks for the idempotency key, and either returns a cached response or allows the request to proceed.

The Idempotency Filter

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;

import javax.servlet.FilterChain;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
import java.util.Optional;
import java.util.concurrent.TimeUnit;

@Component
public class IdempotencyFilter extends OncePerRequestFilter {

    private static final String IDEMPOTENCY_KEY_HEADER = "Idempotency-Key";
    private static final String IDEMPOTENCY_KEY_PREFIX = "idempotency:";
    private static final long TTL_HOURS = 24;

    @Autowired
    private StringRedisTemplate redisTemplate;

    @Override
    protected void doFilterInternal(HttpServletRequest request, 
                                    HttpServletResponse response, 
                                    FilterChain filterChain) 
                                    throws ServletException, IOException {

        String idempotencyKey = request.getHeader(IDEMPOTENCY_KEY_HEADER);

        // Only apply to POST requests
        if ("POST".equalsIgnoreCase(request.getMethod()) && idempotencyKey != null) {
            
            // Check if already processed
            String cacheKey = IDEMPOTENCY_KEY_PREFIX + idempotencyKey;
            String cachedResponse = redisTemplate.opsForValue().get(cacheKey);

            if (cachedResponse != null) {
                // Return cached response
                response.setContentType("application/json");
                response.setStatus(HttpServletResponse.SC_OK);
                response.getWriter().write(cachedResponse);
                return;
            }

            // Wrap response to capture output
            CachedResponseWrapper responseWrapper = new CachedResponseWrapper(response);
            
            try {
                // Proceed with the request
                filterChain.doFilter(request, responseWrapper);
                
                // If successful, cache the response
                if (responseWrapper.getStatus() == HttpServletResponse.SC_OK) {
                    String body = responseWrapper.getContentAsString();
                    redisTemplate.opsForValue().set(
                        cacheKey, 
                        body, 
                        TTL_HOURS, 
                        TimeUnit.HOURS
                    );
                }
            } catch (Exception e) {
                // If an exception occurs, do not cache the error response
                // to allow retry
                throw e;
            }
        } else {
            filterChain.doFilter(request, response);
        }
    }
}

The Response Wrapper

To cache the response, we need to capture the output stream. Here is a simple wrapper:

import org.springframework.http.HttpStatus;
import javax.servlet.http.HttpServletResponse;
import javax.servlet.http.HttpServletResponseWrapper;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.PrintWriter;

public class CachedResponseWrapper extends HttpServletResponseWrapper {
    private final ByteArrayOutputStream buffer = new ByteArrayOutputStream();
    private final PrintWriter writer = new PrintWriter(buffer);

    public CachedResponseWrapper(HttpServletResponse response) {
        super(response);
    }

    @Override
    public PrintWriter getWriter() {
        return writer;
    }

    @Override
    public void flushBuffer() throws IOException {
        writer.flush();
        super.flushBuffer();
    }

    public String getContentAsString() {
        return buffer.toString(StandardCharsets.UTF_8.name());
    }

    public int getStatus() {
        return getResponse().getStatus();
    }
}

Handling Failures

What happens if the request times out after being processed but before the response is returned to the client? The client will retry. Since we cached the response in Redis, the retry will return the same successful response. This is the desired behavior.

However, what if the processing fails? We should not cache error responses. If we cache a 500 error, the client will retry and get the same 500 error, potentially causing an infinite loop. Instead, we should only cache successful responses (2xx). The client should handle retries for errors with exponential backoff.

Common Pitfalls

1. Caching Error Responses

As mentioned, never cache error responses. If a request fails, the idempotency key should not be stored. This allows the client to retry and potentially succeed.

2. Short TTLs

If your TTL is too short (e.g., 5 minutes), a client that retries after 10 minutes will not find the key and will process the request again. For financial transactions, a TTL of 24 hours is standard. For less critical operations, you can adjust based on your SLA.

3. Not Validating the Key Format

Always validate the idempotency key. Ensure it is not too long, contains only safe characters, and is not empty. This prevents abuse and storage issues.

4. Race Conditions

In a distributed system, two requests with the same key might arrive simultaneously. Use atomic operations in Redis (SETNX with EX) to prevent race conditions. Our implementation uses opsForValue().set() which is atomic.

Testing Idempotency

How do you test if your idempotency implementation works? You can write unit tests that mock the Redis template and integration tests that hit a real Redis instance.

import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.http.MediaType;
import org.springframework.test.web.servlet.MockMvc;

import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;

@SpringBootTest
@AutoConfigureMockMvc
public class IdempotencyTest {

    @Autowired
    private MockMvc mockMvc;

    @Test
    public void testDuplicateRequestReturnsCachedResponse() throws Exception {
        String idempotencyKey = "test-key-123";
        String payload = "{\"amount\": 100, \"userId\": 123}";

        // First request
        mockMvc.perform(post("/payments")
                .header("Idempotency-Key", idempotencyKey)
                .contentType(MediaType.APPLICATION_JSON)
                .content(payload))
                .andExpect(status().isOk());

        // Second request with same key
        mockMvc.perform(post("/payments")
                .header("Idempotency-Key", idempotencyKey)
                .contentType(MediaType.APPLICATION_JSON)
                .content(payload))
                .andExpect(status().isOk())
                .andExpect(result -> 
                    // Verify the response is the same
                    result.getResponse().getContentAsString().contains("success")
                );
    }
}

Key Takeaways

• Idempotency is crucial for safe retries in distributed systems, especially for operations with side effects like payments.
• Use client-generated UUIDs for idempotency keys to ensure uniqueness and simplicity.
• Store keys in Redis with a TTL of 24 hours for high performance and automatic cleanup.
• Only cache successful responses to avoid blocking retries on errors.
• Validate input and handle race conditions with atomic operations.
• Test thoroughly with unit and integration tests to ensure your idempotency logic works as expected.

By implementing these strategies, you can build robust APIs that withstand network failures and prevent costly duplicate operations. Remember, in distributed systems, failures are inevitable. Idempotency is your safety net.