Designing Resilient Systems: Circuit Breaker and Retry Patterns

· Microservices Resilience4j Circuit Breaker Retry Pattern Resilience Patterns

Designing Resilient Systems: Circuit Breaker and Retry Patterns

In a distributed system, failures are not a matter of if but when. Network partitions, slow downstream services, database timeouts, and transient errors are the daily reality of microservices. Without deliberate design, a single failing dependency can cascade into a system-wide outage. This is where resilience patterns come in.

Two of the most fundamental patterns for building fault-tolerant systems are the Circuit Breaker and the Retry pattern. While they are often used together, they solve different problems and must be configured carefully to avoid making things worse.

In this post, I’ll share practical experience implementing these patterns in Java using Resilience4j, discuss real-world pitfalls, and provide code examples you can adapt today.

Why Resilience Matters

Imagine your e-commerce application calls a payment gateway. That gateway might be temporarily slow due to high load. If your service keeps waiting for a response, threads pile up, memory fills, and your entire application becomes unresponsive. This is the cascading failure problem.

Resilience patterns aim to:

Let’s dive into the two most effective patterns.

The Retry Pattern

The Retry pattern is straightforward: when a call to a remote service fails with a transient error (like a network timeout or a 503 Service Unavailable), automatically retry the operation after a short delay.

When to Retry

Not all failures are retryable. You should retry only when:

Basic Retry with Resilience4j

Resilience4j is a lightweight, easy-to-use library for Java applications. Here’s a basic retry configuration:

1
2
3
4
5
6
7
8
9
10
resilience4j.retry:
  configs:
    default:
      maxAttempts: 3
      waitDuration: 500ms
      retryExceptions:
        - org.springframework.dao.DataAccessException
        - java.net.SocketTimeoutException
      ignoreExceptions:
        - com.example.BusinessException

And in code:

1
2
3
4
5
6
7
8
9
Retry retry = Retry.of("paymentService", retryConfig);

Supplier<String> supplier = () -> paymentService.charge(order);

Supplier<String> decorated = Retry.decorateSupplier(retry, supplier);

Try<String> result = Try.ofSupplier(decorated);
result.onSuccess(System.out::println)
      .onFailure(e -> log.error("Payment failed after retries", e));

Exponential Backoff and Jitter

Using a fixed wait duration can cause a thundering herd problem: if many clients retry at the same interval, they all hit the recovering service simultaneously. Exponential backoff with jitter spreads out retries.

1
2
3
4
5
6
7
8
9
resilience4j.retry:
  configs:
    default:
      maxAttempts: 5
      waitDuration: 1s
      exponentialBackoffMultiplier: 2
      enableExponentialBackoff: true
      enableRandomizedWait: true
      randomizedWaitFactor: 0.5

This means the first retry waits 1s, second 2s, third 4s, and so on, with up to 50% randomness.

Common Pitfall: Retry Storm

A retry storm happens when a service is already struggling, and retries from multiple clients overwhelm it further. Always combine retries with a circuit breaker (next) and limit the max retry count to a small number (3–5).

The Circuit Breaker Pattern

The Circuit Breaker pattern prevents your application from repeatedly trying an operation that is likely to fail. It monitors for failures and once a threshold is reached, it opens the circuit and subsequent calls fail immediately without hitting the downstream service.

States of a Circuit Breaker

Circuit Breaker with Resilience4j

Configuration example:

1
2
3
4
5
6
7
8
9
10
11
resilience4j.circuitbreaker:
  configs:
    default:
      slidingWindowSize: 10
      minimumNumberOfCalls: 5
      failureRateThreshold: 50
      waitDurationInOpenState: 10s
      permittedNumberOfCallsInHalfOpenState: 3
      recordExceptions:
        - java.net.ConnectException
        - java.util.concurrent.TimeoutException

In Java:

1
2
3
4
5
6
7
8
CircuitBreaker circuitBreaker = CircuitBreaker.of("paymentService", circuitBreakerConfig);

Supplier<String> decorated = CircuitBreaker.decorateSupplier(circuitBreaker, 
    () -> paymentService.charge(order));

Try<String> result = Try.ofSupplier(decorated);
result.onSuccess(System.out::println)
      .onFailure(e -> log.warn("Circuit breaker open, fallback used", e));

Fallback Methods

When the circuit is open, you should provide a fallback:

1
2
3
4
5
Supplier<String> decorated = CircuitBreaker.decorateSupplier(circuitBreaker, 
    () -> paymentService.charge(order));

Supplier<String> withFallback = Fallback.decorate(decorated, 
    e -> fallbackPayment(order));

Fallbacks can return cached data, a default response, or redirect to an alternative service.

Combining Retry and Circuit Breaker

This is where many developers make mistakes. If you apply retry inside the circuit breaker, each retry counts as a separate call, potentially opening the circuit faster than intended. The correct order is:

  1. Circuit Breaker wraps the outer call.
  2. Retry wraps the inner call.
1
2
3
// Correct: Retry inside Circuit Breaker
Supplier<String> decorated = CircuitBreaker.decorateSupplier(circuitBreaker, 
    Retry.decorateSupplier(retry, () -> paymentService.charge(order)));

This way, if the circuit is open, the retry never executes. And if the circuit is closed but the call fails, the retry will attempt again, but each attempt is tracked by the circuit breaker.

Visual Flow

1
2
3
Request -> CircuitBreaker (CLOSED?) -> Retry (up to 3 attempts) -> Service
                                     |
                                     +-> If all retries fail -> CircuitBreaker records failure

Real-World Example: Payment Service

Let’s build a complete example using Spring Boot and Resilience4j.

Dependencies

1
2
3
4
5
<dependency>
    <groupId>io.github.resilience4j</groupId>
    <artifactId>resilience4j-spring-boot2</artifactId>
    <version>2.0.2</version>
</dependency>

Application.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
resilience4j.retry:
  instances:
    paymentRetry:
      maxAttempts: 3
      waitDuration: 500ms
      exponentialBackoffMultiplier: 2
      retryExceptions:
        - java.net.SocketTimeoutException
        - org.springframework.web.client.HttpServerErrorException

resilience4j.circuitbreaker:
  instances:
    paymentCircuitBreaker:
      registerHealthIndicator: true
      slidingWindowSize: 10
      minimumNumberOfCalls: 5
      failureRateThreshold: 50
      waitDurationInOpenState: 10s
      permittedNumberOfCallsInHalfOpenState: 3
      recordExceptions:
        - java.net.ConnectException
        - java.util.concurrent.TimeoutException

Service Layer

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
@Service
public class PaymentService {
    
    @CircuitBreaker(name = "paymentCircuitBreaker", fallbackMethod = "fallbackCharge")
    @Retry(name = "paymentRetry")
    public PaymentResponse charge(PaymentRequest request) {
        // call external payment gateway
        return restTemplate.postForObject("https://payment-gateway/charge", 
            request, PaymentResponse.class);
    }

    public PaymentResponse fallbackCharge(PaymentRequest request, Throwable t) {
        log.warn("Payment circuit breaker open, using fallback");
        return new PaymentResponse("FAILED", "Service unavailable, retry later");
    }
}

Monitoring with Actuator

Resilience4j exposes metrics via Spring Boot Actuator:

1
2
3
4
5
# Check circuit breaker state
GET /actuator/health

# Get metrics
GET /actuator/metrics/resilience4j.circuitbreaker.state

Best Practices

1. Set Realistic Timeouts

Circuit breakers work best when combined with timeouts. A call that hangs for 30 seconds is worse than a fast failure.

1
2
3
4
resilience4j.timelimiter:
  instances:
    paymentService:
      timeoutDuration: 2s

2. Use Separate Configurations per Dependency

Don’t use a single circuit breaker for all external calls. Each dependency (database, payment service, email service) should have its own configuration based on its typical latency and failure patterns.

3. Monitor and Tune

Start with conservative values (e.g., 5 calls, 50% failure rate, 10s open window). Monitor real traffic and adjust. If the circuit opens too often, increase the threshold. If it rarely opens, decrease it.

4. Test Failure Scenarios

Use tools like Chaos Monkey or Toxiproxy to simulate network failures, latency spikes, and service crashes. Verify that your circuit breakers and retries behave as expected.

Common Mistakes

Advanced: Bulkhead Pattern

For even better resilience, combine circuit breakers with the Bulkhead pattern, which limits the number of concurrent calls to a service. Resilience4j supports both thread-pool and semaphore-based bulkheads.

1
2
3
4
5
resilience4j.bulkhead:
  instances:
    paymentBulkhead:
      maxConcurrentCalls: 10
      maxWaitDuration: 500ms

Conclusion

Resilience patterns are not optional in distributed systems. The Retry pattern handles transient failures, while the Circuit Breaker prevents cascading failures and gives services time to recover. When combined correctly—retry inside circuit breaker—they form a powerful defense against the chaos of production.

Start small. Add circuit breakers to your most critical external dependencies. Monitor the results. Then layer on retries with exponential backoff. Your future self (and your users) will thank you.

Key Takeaways