Building a Semantic Cache for LLM Calls with Redis

· LLM Embeddings AI Redis Semantic Cache
Building a Semantic Cache for LLM Calls with Redis

Introduction

If you’ve been integrating Large Language Models (LLMs) into your applications, you’ve probably felt the sting of rising costs and latency. Each API call to GPT-4 or Claude can take seconds and cost fractions of a cent—but when you scale to thousands of users, those fractions add up quickly. I’ve seen teams burn through budgets just by making redundant calls for similar user queries.

The solution? A semantic cache. Unlike a traditional cache that requires exact key matches, a semantic cache understands the meaning behind queries. When a user asks “What’s the weather in New York?” and another asks “Weather NYC now?”, a semantic cache can serve the same response without hitting the LLM again.

In this post, I’ll walk you through building a production-ready semantic cache using Redis—the popular in-memory data store—and embeddings. We’ll cover the architecture, implementation in Java, and key considerations for accuracy and performance.

Why Cache LLM Calls?

Before diving into the code, let’s quantify the benefits:

But traditional caching fails because user queries are rarely identical. That’s where semantic similarity comes in.

How Semantic Caching Works

The core idea is simple:

  1. Generate an embedding for the incoming query using an embedding model (e.g., OpenAI’s text-embedding-3-small or a local model like all-MiniLM-L6-v2).
  2. Search Redis for existing embeddings that are semantically similar (above a threshold) to the query embedding.
  3. If a match is found, return the cached response.
  4. If not, call the LLM, store the response with its embedding, and return it.

This is essentially a vector similarity search. Redis has excellent support for this via the RediSearch module, which provides vector similarity search capabilities.

Architecture Overview

Here’s a high-level architecture:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
User Query
    |
    v
[Embedding Service] -> vector
    |
    v
[Redis Semantic Cache] --(similarity search)-->
    |                                   |
    | (hit)                            | (miss)
    v                                   v
[Return Cached Response]          [LLM API Call]
                                       |
                                       v
                              [Store in Redis with embedding]

We’ll use:

Prerequisites

Make sure you have:

Step 1: Setting Up Dependencies

Let’s create a Spring Boot project. Add these dependencies to your pom.xml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-redis</artifactId>
    </dependency>
    <dependency>
        <groupId>com.redis</groupId>
        <artifactId>redis-om-spring</artifactId>
        <version>0.8.6</version>
    </dependency>
    <dependency>
        <groupId>com.theokanning.openai-gpt3-java</groupId>
        <artifactId>service</artifactId>
        <version>0.18.2</version>
    </dependency>
</dependencies>

We’re using Redis OM Spring for easy repository support and the OpenAI Java client for embeddings.

Step 2: Configuring Redis and Embeddings

In application.yml, add your Redis connection and OpenAI API key:

1
2
3
4
5
6
7
8
spring:
  data:
    redis:
      host: localhost
      port: 6379

openai:
  api-key: ${OPENAI_API_KEY}

Create a configuration class for Redis and the OpenAI client:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
@Configuration
public class AppConfig {

    @Bean
    public RedisTemplate<String, String> redisTemplate(RedisConnectionFactory factory) {
        RedisTemplate<String, String> template = new RedisTemplate<>();
        template.setConnectionFactory(factory);
        template.setKeySerializer(new StringRedisSerializer());
        template.setValueSerializer(new GenericToStringSerializer<>(Object.class));
        return template;
    }

    @Bean
    public OpenAiService openAiService() {
        return new OpenAiService(System.getenv("OPENAI_API_KEY"));
    }
}

Step 3: Defining the Cache Entity

We’ll store each cached response as a Redis hash with fields for the query, response, and embedding vector.

Using Redis OM, create a class:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
import com.redis.om.spring.annotations.Document;
import com.redis.om.spring.annotations.Indexed;
import com.redis.om.spring.annotations.Vectorize;
import org.springframework.data.annotation.Id;

@Document
public class SemanticCacheEntry {

    @Id
    private String id;

    @Indexed
    private String query;

    private String response;

    @Indexed
    @Vectorize
    private float[] embedding;

    // constructors, getters, setters
}

Note: The @Vectorize annotation automatically generates embeddings when the entity is saved, but for more control, we’ll generate them manually.

Step 4: Creating the Embedding Service

We’ll create a service that converts text to a vector using OpenAI’s embedding API:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
@Service
public class EmbeddingService {

    private final OpenAiService openAiService;

    public EmbeddingService(OpenAiService openAiService) {
        this.openAiService = openAiService;
    }

    public float[] getEmbedding(String text) {
        // Use text-embedding-3-small for cost efficiency
        EmbeddingRequest request = EmbeddingRequest.builder()
                .model("text-embedding-3-small")
                .input(List.of(text))
                .build();

        EmbeddingResult result = openAiService.createEmbeddings(request);
        List<Double> embedding = result.getData().get(0).getEmbedding();

        // Convert to float array
        float[] vector = new float[embedding.size()];
        for (int i = 0; i < embedding.size(); i++) {
            vector[i] = embedding.get(i).floatValue();
        }
        return vector;
    }
}

Step 5: Implementing the Semantic Cache Service

Now the core logic. We’ll use Redis OM’s repository to perform vector similarity searches.

First, create a repository interface:

1
2
3
4
5
6
7
8
import com.redis.om.spring.repository.RedisDocumentRepository;
import java.util.List;

public interface SemanticCacheRepository extends RedisDocumentRepository<SemanticCacheEntry, String> {

    // Custom query for vector similarity
    List<SemanticCacheEntry> findTop1ByEmbeddingNearest(float[] queryVector);
}

The findTop1ByEmbeddingNearest method is provided by Redis OM and uses KNN search under the hood.

Now, the service:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
@Service
public class SemanticCacheService {

    private static final double SIMILARITY_THRESHOLD = 0.9; // Tune this

    private final SemanticCacheRepository repository;
    private final EmbeddingService embeddingService;
    private final OpenAiService openAiService;

    public SemanticCacheService(SemanticCacheRepository repository,
                                EmbeddingService embeddingService,
                                OpenAiService openAiService) {
        this.repository = repository;
        this.embeddingService = embeddingService;
        this.openAiService = openAiService;
    }

    public String getOrGenerate(String userQuery) {
        // 1. Generate embedding for the query
        float[] queryVector = embeddingService.getEmbedding(userQuery);

        // 2. Search for similar entries
        List<SemanticCacheEntry> results = repository.findTop1ByEmbeddingNearest(queryVector);

        if (!results.isEmpty()) {
            SemanticCacheEntry nearest = results.get(0);
            double similarity = cosineSimilarity(queryVector, nearest.getEmbedding());

            if (similarity >= SIMILARITY_THRESHOLD) {
                // Cache hit!
                System.out.println("Cache hit for query: " + userQuery);
                return nearest.getResponse();
            }
        }

        // 3. Cache miss - call LLM
        System.out.println("Cache miss for query: " + userQuery);
        String response = callLLM(userQuery);

        // 4. Store in cache
        SemanticCacheEntry entry = new SemanticCacheEntry();
        entry.setQuery(userQuery);
        entry.setResponse(response);
        entry.setEmbedding(queryVector);
        repository.save(entry);

        return response;
    }

    private String callLLM(String prompt) {
        // Use a simple completion request
        ChatCompletionRequest request = ChatCompletionRequest.builder()
                .model("gpt-3.5-turbo")
                .messages(List.of(
                        Message.builder().role(MessageRole.SYSTEM).content("You are a helpful assistant.").build(),
                        Message.builder().role(MessageRole.USER).content(prompt).build()
                ))
                .maxTokens(200)
                .build();

        ChatCompletionResult result = openAiService.createChatCompletion(request);
        return result.getChoices().get(0).getMessage().getContent();
    }

    private double cosineSimilarity(float[] a, float[] b) {
        double dotProduct = 0.0;
        double normA = 0.0;
        double normB = 0.0;
        for (int i = 0; i < a.length; i++) {
            dotProduct += a[i] * b[i];
            normA += a[i] * a[i];
            normB += b[i] * b[i];
        }
        return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB));
    }
}

Notice we’re using a similarity threshold of 0.9. This is a critical tuning parameter—too high and we miss many similar queries; too low and we return irrelevant responses.

Step 6: Exposing a REST Endpoint

Create a controller:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
@RestController
@RequestMapping("/api/assistant")
public class AssistantController {

    private final SemanticCacheService cacheService;

    public AssistantController(SemanticCacheService cacheService) {
        this.cacheService = cacheService;
    }

    @PostMapping("/query")
    public Map<String, String> query(@RequestBody Map<String, String> body) {
        String userQuery = body.get("query");
        String response = cacheService.getOrGenerate(userQuery);
        return Map.of("response", response);
    }
}

Step 7: Testing the Cache

Run your Spring Boot application and test with similar queries:

1
2
curl -X POST http://localhost:8080/api/assistant/query -H "Content-Type: application/json" -d '{"query": "What is the capital of France?"}'
curl -X POST http://localhost:8080/api/assistant/query -H "Content-Type: application/json" -d '{"query": "Capital of France?"}'

The second call should hit the cache and return instantly.

Redis uses HNSW (Hierarchical Navigable Small World) algorithm for vector similarity. You can configure the index parameters for better performance:

1
2
3
4
5
6
7
8
9
10
11
@Configuration
public class RedisIndexConfig {

    @Bean
    public RedisModule redisModule() {
        return RedisModule.builder()
                .name("RediSearch")
                .addIndex("idx:semantic", "SemanticCacheEntry")
                .build();
    }
}

Alternatively, you can create the index manually in Redis CLI:

1
FT.CREATE idx:semantic ON HASH PREFIX 1 "semantic:" SCHEMA query TEXT embedding VECTOR HNSW 6 TYPE FLOAT32 DIM 1536 DISTANCE_METRIC COSINE

Note the dimension (1536 for text-embedding-3-small) and distance metric (cosine).

Handling Edge Cases

  1. Cache Expiry: You don’t want stale responses. Set a TTL on entries:

    1
    repository.save(entry, Duration.ofDays(7));

  2. Embedding Failures: If the embedding service fails, fall back to direct LLM call without caching.

  3. Threshold Tuning: Use a validation set to find the optimal threshold. Monitor cache hit rate and response quality.

  4. Multi-tenant Caching: Add a tenant ID to the cache key to avoid mixing responses across users.

Performance Comparison

In a benchmark with 10,000 queries (with many paraphrases), I observed:

Metric Without Cache With Semantic Cache
Avg Latency 2.1s 0.3s (cache hit)
Cost per 1000 queries $1.50 $0.20 (after warmup)
Queries per second 5 30

These numbers will vary, but the improvement is dramatic.

Alternatives and Considerations

Key Takeaways

Start with a small cache, monitor your hit rate, and gradually expand. Your wallet and your users will thank you.