Designing Idempotent APIs for Reliable Distributed Systems

· microservices distributed systems idempotency API design REST

Designing Idempotent APIs for Reliable Distributed Systems

Imagine your e-commerce platform processes a payment, but the client times out before receiving the response. The user clicks “Pay” again. Now what? Without idempotency, you might charge them twice. In distributed systems, network failures, retries, and message duplication are not exceptions—they’re the norm. Designing idempotent APIs is the cornerstone of building reliable, fault-tolerant systems that users can trust.

In this post, we’ll dive deep into idempotency: what it is, why it matters, and how to implement it effectively. We’ll cover key concepts, practical strategies, and code examples in Java. By the end, you’ll have a solid blueprint for designing APIs that gracefully handle retries and failures.

What Is Idempotency?

An operation is idempotent if performing it multiple times produces the same result as performing it once. In HTTP terms, GET, PUT, DELETE, and HEAD are inherently idempotent. POST is not—it’s designed to create resources. But in distributed systems, we often need idempotent POST operations (e.g., creating orders, processing payments).

Idempotency ensures that retries don’t lead to unintended side effects. It’s not about preventing retries (which are necessary for reliability) but about making retries safe.

Why Idempotency Matters in Distributed Systems

Distributed systems introduce complexities like network partitions, timeouts, and duplicate messages. Consider these scenarios:

Idempotency provides a contract: the client can safely retry, and the server guarantees no duplicate side effects. This reduces coupling and improves resilience.

Key Concepts

Idempotency Key

An idempotency key is a unique identifier that the client generates and sends with each request. The server uses this key to detect and reject duplicates. Typically, it’s a UUID or a hash derived from request parameters.

Idempotency Scope

Idempotency can be scoped to:

At-Least-Once vs. Exactly-Once Semantics

Idempotent APIs enable at-least-once delivery with exactly-once processing. The client may send the request multiple times, but the server processes it only once.

Designing an Idempotent API

Step 1: Require Idempotency Keys for Mutating Endpoints

For any POST, PATCH, or non-idempotent PUT, mandate an Idempotency-Key header. For example:

1
2
3
4
5
6
7
8
POST /api/orders
Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000
Content-Type: application/json

{
  "productId": "123",
  "quantity": 2
}

Step 2: Store Key-Response Mappings

On the server, store the idempotency key along with the response. Use a database or cache (like Redis) with a TTL. When a duplicate request arrives, return the stored response without reprocessing.

Step 3: Handle Concurrency

If two identical requests arrive simultaneously, use optimistic locking or a unique constraint on the key to prevent double processing.

Step 4: Return Consistent Responses

For duplicate requests, return the same HTTP status code and body as the original response. This allows clients to safely retry.

Implementation in Java with Spring Boot

Let’s build a simple idempotent payment API using Spring Boot and Redis.

Dependencies

1
2
3
4
5
6
7
8
9
<!-- pom.xml -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>

Idempotency Filter

Create a filter that checks for the Idempotency-Key header and handles duplicates.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
@Component
@Order(1)
public class IdempotencyFilter implements Filter {

    @Autowired
    private RedisTemplate<String, Object> redisTemplate;

    private static final String KEY_PREFIX = "idempotency:";

    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
            throws IOException, ServletException {

        HttpServletRequest httpRequest = (HttpServletRequest) request;
        HttpServletResponse httpResponse = (HttpServletResponse) response;

        String idempotencyKey = httpRequest.getHeader("Idempotency-Key");

        // Only apply to POST, PATCH, DELETE
        if (idempotencyKey != null && httpRequest.getMethod().equals("POST")) {
            String redisKey = KEY_PREFIX + idempotencyKey;

            // Check if key already exists
            Object cachedResponse = redisTemplate.opsForValue().get(redisKey);
            if (cachedResponse != null) {
                // Return cached response
                httpResponse.setStatus(200);
                httpResponse.getWriter().write(cachedResponse.toString());
                return;
            }

            // Wrap response to cache it later
            CachedBodyHttpServletResponse cachedResponseWrapper = new CachedBodyHttpServletResponse(httpResponse);
            chain.doFilter(request, cachedResponseWrapper);

            // Cache the response with a TTL of 24 hours
            redisTemplate.opsForValue().set(redisKey, cachedResponseWrapper.getBody(), 24, TimeUnit.HOURS);
        } else {
            chain.doFilter(request, response);
        }
    }
}

Cached Response Wrapper

We need a wrapper to capture the response body.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
public class CachedBodyHttpServletResponse extends HttpServletResponseWrapper {

    private ByteArrayOutputStream cachedBody = new ByteArrayOutputStream();

    public CachedBodyHttpServletResponse(HttpServletResponse response) {
        super(response);
    }

    @Override
    public ServletOutputStream getOutputStream() throws IOException {
        return new CachedBodyServletOutputStream(cachedBody, super.getOutputStream());
    }

    public String getBody() {
        return cachedBody.toString();
    }
}

Payment Controller

1
2
3
4
5
6
7
8
9
10
11
12
@RestController
@RequestMapping("/api/payments")
public class PaymentController {

    @PostMapping
    public ResponseEntity<String> processPayment(@RequestBody PaymentRequest request) {
        // Simulate payment processing
        String transactionId = UUID.randomUUID().toString();
        // In reality, you'd call a payment gateway here
        return ResponseEntity.ok("Payment processed. Transaction ID: " + transactionId);
    }
}

Handling Concurrent Requests

To prevent race conditions, use Redis’ SETNX (set if not exists) command:

1
2
3
4
5
6
7
8
9
10
11
Boolean acquired = redisTemplate.opsForValue().setIfAbsent(redisKey, "LOCK", 10, TimeUnit.SECONDS);
if (Boolean.TRUE.equals(acquired)) {
    try {
        // Process request
    } finally {
        redisTemplate.delete(redisKey);
    }
} else {
    // Another request is processing; wait or return conflict
    httpResponse.setStatus(409);
}

Best Practices

1. Use UUIDs for Idempotency Keys

UUIDs are universally unique and easy to generate. Avoid sequential IDs or timestamps that could collide.

2. Set Appropriate TTLs

Keys should expire to prevent storage bloat. Choose a TTL longer than the maximum expected retry interval (e.g., 24 hours).

3. Return 409 Conflict for In-Flight Requests

If a duplicate request arrives while the original is still processing, return 409 Conflict with a message indicating the request is in progress.

4. Include Idempotency in API Documentation

Clearly document which endpoints require idempotency keys, how to generate them, and what response to expect on duplicates.

5. Test with Chaos Engineering

Simulate network failures, duplicate requests, and race conditions to verify your implementation handles them gracefully.

Common Pitfalls

Advanced Considerations

Idempotency in Event-Driven Systems

In event-driven architectures, idempotency is often achieved using deduplication IDs in events. The consumer checks if it has already processed an event with the same ID.

1
2
3
4
5
6
7
8
public void handleOrderCreated(OrderCreatedEvent event) {
    String dedupKey = "dedup:" + event.getEventId();
    Boolean alreadyProcessed = redisTemplate.opsForValue().setIfAbsent(dedupKey, "1", 1, TimeUnit.DAYS);
    if (Boolean.FALSE.equals(alreadyProcessed)) {
        return; // Duplicate event
    }
    // Process event
}

Idempotency Across Microservices

When a service calls another, propagate the idempotency key. Use distributed tracing headers (e.g., X-Request-Id) to correlate requests.

Idempotency for Non-Idempotent Operations

Some operations are inherently non-idempotent (e.g., appending to a log). In such cases, use idempotency keys to detect duplicates and skip processing.

Key Takeaways

By designing idempotent APIs, you eliminate a whole class of failures and make your system more robust. Start small—add idempotency to your critical endpoints—and expand from there. Your users (and your on-call team) will thank you.