Agent Memory Architectures: Short-Term, Long-Term, and Episodic

September 11, 2026 · Java RAG LLM AI Agents Memory Architecture Vector Databases

The Missing Piece in Autonomous Agents

When we first started building AI agents in production, we assumed that a large context window was enough. We fed the entire conversation history into the prompt, attached a few documents via RAG, and called it a day. It worked—until it didn’t.

As conversations stretched beyond 50–100 turns, latency spiked. Token costs ballooned. And worse, the agent began to forget critical details from the beginning of the session or, even more concerning, failed to retain user preferences across separate interactions.

The problem wasn’t the model’s intelligence. It was its memory architecture. Just like humans, AI agents need different types of memory to function effectively: short-term working memory for immediate tasks, long-term semantic memory for knowledge, and episodic memory for personal experiences.

In this post, we’ll dive deep into these three memory architectures and show you how to implement them using Java, Spring AI, and modern vector databases.

Why Memory Matters in Agent Design

Before we architect anything, let’s understand why memory is a first-class citizen in agent systems. Consider these scenarios:

1. Context Window Limits: Even with 128K+ token contexts, there’s a hard limit. Once you hit it, you must truncate or summarize, risking information loss.
2. Cost Efficiency: Every token in a prompt costs money. Storing and retrieving only relevant memories reduces inference costs significantly.
3. Personalization: Users expect agents to remember their preferences, past interactions, and learned behaviors across sessions.
4. Reasoning Quality: Agents that can recall specific past events (episodic memory) make better decisions than those starting from scratch every time.

The solution isn’t to rely solely on the LLM’s context window. It’s to build a structured memory system that mirrors how humans organize information.

The Three-Tier Memory Model

Let’s define the three memory types we’ll implement:

1. Short-Term Memory (Working Memory)

Short-term memory holds the immediate context of the current interaction. It’s analogous to your working memory when solving a problem right now. For an AI agent, this includes:

• The current conversation turn
• Recent tool outputs
• Active goals and sub-goals
• Temporary variables and intermediate results

Characteristics:

• High volatility: Forgotten after the session ends (or summarized)
• Fast access: Available in the prompt context
• Limited capacity: Constrained by the LLM’s context window

2. Long-Term Memory (Semantic Memory)

Long-term memory stores general knowledge, facts, and learned concepts. This is the agent’s encyclopedia. It includes:

• Domain knowledge (e.g., company policies, technical documentation)
• User preferences and profiles
• Frequently accessed facts
• Procedural knowledge (how to do things)

Characteristics:

• Durable: Persists across sessions
• Sparse: Only the most relevant information is retrieved
• Indexed: Stored in vector databases for semantic search

3. Episodic Memory (Autobiographical Memory)

Episodic memory stores specific experiences and events. This is the agent’s diary. It includes:

• Past conversations and interactions
• Decisions made and their outcomes
• User feedback and corrections
• Specific incidents that shaped behavior

Characteristics:

• Time-stamped: Ordered chronologically
• Rich: Contains full context of events
• Retrieval-based: Accessed when relevant to current tasks

Architecture Overview

Here’s how these memory layers interact in a typical agent system:

┌─────────────────────────────────────────────────────────┐
│                    Agent Core                           │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐    │
│  │   Short     │  │   Long      │  │   Episodic  │    │
│  │   Term      │  │   Term      │  │   Memory    │    │
│  │   Memory    │  │   Memory    │  │             │    │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘    │
│         │                │                │           │
│         └────────────────┴────────────────┘           │
│                        │                              │
│              ┌─────────▼─────────┐                    │
│              │   Memory Router    │                    │
│              │  (Decision Engine) │                    │
│              └─────────┬─────────┘                    │
└────────────────────────┼──────────────────────────────┘
                         │
        ┌────────────────┼────────────────┐
        │                │                │
   ┌────▼────┐     ┌────▼────┐     ┌────▼────┐
   │ In-Memory│    │ Vector  │     │ Event   │
   │ Cache   │     │ DB      │     │ Store   │
   └─────────┘     └─────────┘     └─────────┘

The Memory Router decides which memory store to query based on the current task. Short-term memory is always in the prompt. Long-term memory is retrieved via semantic search. Episodic memory is fetched when historical context is needed.

Implementing Short-Term Memory

Short-term memory is the simplest layer. In Java, we can use a thread-local or session-scoped storage to maintain the conversation context.

The Conversation Buffer

We’ll create a ConversationMemory class that acts as a sliding window over recent messages:

@Component
@Scope("prototype")
public class ConversationMemory {
    
    private final Deque<Message> history = new ArrayDeque<>();
    private final int maxTurns;
    
    public ConversationMemory(int maxTurns) {
        this.maxTurns = maxTurns;
    }
    
    public void addAssistantMessage(String content) {
        history.addLast(new Message(Message.Role.ASSISTANT, content));
        trimIfNecessary();
    }
    
    public void addUserMessage(String content) {
        history.addLast(new Message(Message.Role.USER, content));
        trimIfNecessary();
    }
    
    public void addToolResult(String toolName, Object result) {
        history.addLast(new Message(
            Message.Role.TOOL, 
            String.format("Tool '%s' returned: %s", toolName, result)
        ));
        trimIfNecessary();
    }
    
    private void trimIfNecessary() {
        while (history.size() > maxTurns * 3) { // 3 messages per turn (user, tool, assistant)
            history.removeFirst();
        }
    }
    
    public List<Message> getRecentMessages(int n) {
        List<Message> recent = new ArrayList<>();
        Iterator<Message> it = history.descendingIterator();
        while (it.hasNext() && recent.size() < n) {
            recent.add(it.next());
        }
        Collections.reverse(recent);
        return recent;
    }
    
    public List<Message> getFullHistory() {
        return new ArrayList<>(history);
    }
}

This class maintains a bounded queue of messages. By limiting the window size, we ensure that the prompt never exceeds token limits. The trimIfNecessary() method removes oldest messages when the buffer is full.

Integration with Spring AI

Spring AI’s ChatClient works seamlessly with this memory class:

@Service
public class AgentService {
    
    private final ChatClient chatClient;
    private final ConversationMemory memory;
    
    public AgentService(ChatClient.Builder builder, ConversationMemory memory) {
        this.chatClient = builder.build();
        this.memory = memory;
    }
    
    public String processRequest(String userMessage) {
        memory.addUserMessage(userMessage);
        
        // Build prompt with recent context
        List<Message> recentContext = memory.getRecentMessages(10);
        
        Prompt prompt = new Prompt(
            recentContext.stream()
                .map(m -> new Message(m.getRole().toString(), m.getContent()))
                .toList()
        );
        
        // Call LLM
        ChatResponse response = chatClient.call(prompt);
        String assistantReply = response.getResults().get(0).getOutput().getText();
        
        memory.addAssistantMessage(assistantReply);
        return assistantReply;
    }
}

The key insight here is that short-term memory is ephemeral. It exists only for the duration of the conversation and is discarded afterward. This is intentional—short-term memory is meant for immediate reasoning, not permanent storage.

Implementing Long-Term Memory

Long-term memory requires persistent storage and semantic search capabilities. We’ll use a vector database to store embeddings of important facts and retrieve them based on relevance.

Choosing a Vector Database

For production Java applications, we recommend:

• PostgreSQL with pgvector: Great for relational data with vector search
• Redis with RediSearch: Fast in-memory vector search
• Pinecone: Managed service, excellent for scaling

We’ll use PostgreSQL with pgvector in our examples, as it’s widely adopted in enterprise Java stacks.

Storing Semantic Knowledge

First, let’s create a schema for our long-term memory:

CREATE EXTENSION vector;

CREATE TABLE semantic_memory (
    id SERIAL PRIMARY KEY,
    content TEXT NOT NULL,
    embedding vector(1536), -- OpenAI ada-002 dimension
    metadata JSONB,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX ON semantic_memory USING ivfflat (embedding vector_cosine_ops);

Now, let’s create a service to manage this memory:

@Service
public class LongTermMemoryService {
    
    private final JdbcTemplate jdbcTemplate;
    private final EmbeddingClient embeddingClient;
    private final int dimension = 1536;
    
    public LongTermMemoryService(JdbcTemplate jdbcTemplate, 
                                 EmbeddingClient embeddingClient) {
        this.jdbcTemplate = jdbcTemplate;
        this.embeddingClient = embeddingClient;
    }
    
    public void storeFact(String content, Map<String, Object> metadata) {
        float[] embedding = embeddingClient.embed(content);
        String embeddingJson = Arrays.toString(embedding);
        
        String sql = "INSERT INTO semantic_memory (content, embedding, metadata) " +
                     "VALUES (?, ?, ?)";
        
        jdbcTemplate.update(sql, content, embeddingJson, 
            JsonUtils.toJson(metadata));
    }
    
    public List<MemoryItem> retrieveRelevant(String query, int topK) {
        float[] queryEmbedding = embeddingClient.embed(query);
        String queryVector = Arrays.toString(queryEmbedding);
        
        String sql = "SELECT content, metadata, 1 - (embedding <=> ?) AS similarity " +
                     "FROM semantic_memory " +
                     "ORDER BY embedding <=> ? " +
                     "LIMIT ?";
        
        return jdbcTemplate.query(sql, (rs, rowNum) -> {
            MemoryItem item = new MemoryItem();
            item.setContent(rs.getString("content"));
            item.setMetadata(JsonUtils.fromJson(rs.getString("metadata")));
            item.setSimilarity(rs.getDouble("similarity"));
            return item;
        }, queryVector, queryVector, topK);
    }
}

When to Store in Long-Term Memory

Not everything should be stored. We need a filtering mechanism. Here’s a strategy:

1. User Preferences: Always store (e.g., “I prefer concise answers”)
2. Domain Facts: Store if they’re generalizable (e.g., “The API endpoint is /v2/data”)
3. Decisions: Store high-impact decisions with reasoning
4. Tool Patterns: Store successful tool usage patterns

We can implement an automatic extraction service:

@Service
public class MemoryExtractorService {
    
    private final ChatClient chatClient;
    private final LongTermMemoryService longTermMemory;
    
    public void extractAndStore(UserMessage userMsg, AssistantMessage assistantMsg) {
        // Ask LLM to identify memorable facts
        String extractionPrompt = """
            Extract any important facts, preferences, or knowledge from this 
            conversation that should be remembered long-term. Return as JSON:
            {
              "facts": ["string"],
              "preferences": ["string"],
              "decisions": ["string"]
            }
            """;
        
        ChatResponse response = chatClient.call(
            new Prompt(extractionPrompt + "\n\nConversation:\n" + 
                       userMsg.getContent() + "\n" + assistantMsg.getContent())
        );
        
        // Parse and store
        MemoryExtraction extraction = JsonUtils.fromJson(
            response.getResults().get(0).getOutput().getText()
        );
        
        extraction.getFacts().forEach(fact -> 
            longTermMemory.storeFact(fact, Map.of("type", "fact"))
        );
        extraction.getPreferences().forEach(pref -> 
            longTermMemory.storeFact(pref, Map.of("type", "preference"))
        );
    }
}

This approach lets the LLM decide what’s worth remembering, reducing noise in the long-term memory store.

Implementing Episodic Memory

Episodic memory is the most complex layer. It needs to store events with rich context, time stamps, and relationships. Unlike semantic memory (which stores facts), episodic memory stores experiences.

Schema Design

CREATE TABLE episodic_memory (
    id SERIAL PRIMARY KEY,
    event_type VARCHAR(50) NOT NULL, -- "conversation", "tool_call", "decision"
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    user_id VARCHAR(100),
    session_id VARCHAR(100),
    context JSONB, -- Full conversation snapshot
    summary TEXT, -- LLM-generated summary for quick retrieval
    embedding vector(1536),
    related_event_ids INTEGER[]
);

CREATE INDEX ON episodic_memory USING ivfflat (embedding vector_cosine_ops);
CREATE INDEX ON episodic_memory (user_id, timestamp DESC);

Storing Episodes

We’ll create an EpisodicMemoryService that captures complete interaction episodes:

@Service
public class EpisodicMemoryService {
    
    private final JdbcTemplate jdbcTemplate;
    private final EmbeddingClient embeddingClient;
    private final ChatClient chatClient;
    
    public void storeEpisode(String sessionId, String userId, 
                             List<Message> conversation, 
                             List<ToolCall> toolCalls) {
        // Generate a summary for quick retrieval
        String summary = generateSummary(conversation);
        
        // Store full context
        String contextJson = JsonUtils.toJson(Map.of(
            "messages", conversation,
            "toolCalls", toolCalls,
            "timestamp", Instant.now().toString()
        ));
        
        // Generate embedding for semantic search
        float[] embedding = embeddingClient.embed(summary);
        String embeddingJson = Arrays.toString(embedding);
        
        String sql = "INSERT INTO episodic_memory " +
                     "(event_type, user_id, session_id, context, summary, embedding) " +
                     "VALUES ('conversation', ?, ?, ?, ?, ?)";
        
        jdbcTemplate.update(sql, userId, sessionId, contextJson, summary, embeddingJson);
    }
    
    private String generateSummary(List<Message> conversation) {
        String conversationText = conversation.stream()
            .map(m -> m.getRole() + ": " + m.getContent())
            .collect(Collectors.joining("\n"));
        
        String prompt = "Summarize this conversation in 2-3 sentences. " +
                        "Focus on key decisions, outcomes, and user intent.";
        
        ChatResponse response = chatClient.call(new Prompt(prompt + "\n\n" + conversationText));
        return response.getResults().get(0).getOutput().getText();
    }
    
    public List<EpisodicEvent> retrieveRelated(String query, String userId, int limit) {
        float[] queryEmbedding = embeddingClient.embed(query);
        String queryVector = Arrays.toString(queryEmbedding);
        
        String sql = "SELECT id, event_type, timestamp, summary, context " +
                     "FROM episodic_memory " +
                     "WHERE user_id = ? " +
                     "ORDER BY embedding <=> ? " +
                     "LIMIT ?";
        
        return jdbcTemplate.query(sql, (rs, rowNum) -> {
            EpisodicEvent event = new EpisodicEvent();
            event.setId(rs.getInt("id"));
            event.setEventType(rs.getString("event_type"));
            event.setTimestamp(rs.getTimestamp("timestamp"));
            event.setSummary(rs.getString("summary"));
            event.setContext(JsonUtils.fromJson(rs.getString("context")));
            return event;
        }, userId, queryVector, limit);
    }
}

Retrieving Episodic Memory

When should the agent query episodic memory? Here’s a decision framework:

1. User asks about past interactions: “What did we discuss last week?”
2. Current task resembles a past task: “This looks like the bug we fixed before.”
3. User mentions a previous incident: “Remember when the API failed?”
4. Decision support needed: “What approach worked last time?”

We can integrate this into the agent’s reasoning loop:

@Service
public class AgentReasoningService {
    
    private final EpisodicMemoryService episodicMemory;
    private final LongTermMemoryService longTermMemory;
    
    public AgentContext enrichContext(String currentQuery, String userId) {
        AgentContext context = new AgentContext();
        
        // Retrieve relevant episodic memories
        List<EpisodicEvent> pastEpisodes = episodicMemory.retrieveRelated(
            currentQuery, userId, 3
        );
        context.setPastEpisodes(pastEpisodes);
        
        // Retrieve relevant semantic memories
        List<MemoryItem> relevantFacts = longTermMemory.retrieveRelevant(
            currentQuery, 5
        );
        context.setRelevantFacts(relevantFacts);
        
        return context;
    }
}

The Memory Router: Deciding What to Remember

The most critical component is the Memory Router. It decides:

1. What to store in each memory type
2. When to retrieve from each type
3. How to combine memories into the prompt

Here’s a simplified router implementation:

@Component
public class MemoryRouter {
    
    private final LongTermMemoryService longTermMemory;
    private final EpisodicMemoryService episodicMemory;
    private final ConversationMemory shortTermMemory;
    
    public Prompt buildPrompt(String userMessage, String userId) {
        List<Message> messages = new ArrayList<>();
        
        // 1. System prompt with memory instructions
        messages.add(new Message(Message.Role.SYSTEM, 
            buildSystemPrompt(userId)));
        
        // 2. Retrieve relevant long-term memories
        List<MemoryItem> semanticMemories = longTermMemory.retrieveRelevant(
            userMessage, 3
        );
        if (!semanticMemories.isEmpty()) {
            messages.add(new Message(Message.Role.SYSTEM, 
                "Relevant knowledge: " + formatMemories(semanticMemories)));
        }
        
        // 3. Retrieve relevant episodic memories
        List<EpisodicEvent> episodicMemories = episodicMemory.retrieveRelated(
            userMessage, userId, 2
        );
        if (!episodicMemories.isEmpty()) {
            messages.add(new Message(Message.Role.SYSTEM, 
                "Past experiences: " + formatEpisodes(episodicMemories)));
        }
        
        // 4. Add short-term conversation history
        messages.addAll(shortTermMemory.getRecentMessages(10));
        
        // 5. Add current user message
        messages.add(new Message(Message.Role.USER, userMessage));
        
        return new Prompt(messages);
    }
    
    private String buildSystemPrompt(String userId) {
        // Get user preferences from long-term memory
        List<MemoryItem> preferences = longTermMemory.retrieveRelevant(
            "user preferences", 5
        );
        
        String prefText = preferences.isEmpty() ? "" : 
            "User preferences: " + formatMemories(preferences);
        
        return "You are a helpful AI assistant. " + prefText + 
               "Use the provided context to give accurate, personalized responses.";
    }
}

Performance Considerations

When implementing memory systems, keep these performance tips in mind:

1. Caching Layer

Add a caching layer (e.g., Caffeine or Redis) in front of your vector database. Most queries are repetitive, and caching avoids redundant embedding generation and database lookups.

@Cacheable(value = "semanticMemory", key = "#query + '-' + #topK")
public List<MemoryItem> getCachedRetrieval(String query, int topK) {
    return longTermMemory.retrieveRelevant(query, topK);
}

2. Asynchronous Storage

Don’t block the response path on memory writes. Use async processing:

@Async
public void storeAsync(String content, Map<String, Object> metadata) {
    longTermMemory.storeFact(content, metadata);
}

3. Memory Compaction

Periodically summarize and compress old episodic memories. Merge related events and discard low-value episodes.

@Scheduled(cron = "0 0 2 * * *") // Run daily at 2 AM
public void compactOldMemories() {
    List<EpisodicEvent> oldEpisodes = episodicMemory.getOldEpisodes(30);
    for (EpisodicEvent episode : oldEpisodes) {
        episodicMemory.summarizeAndMerge(episode);
    }
}

4. Embedding Optimization

Reuse embeddings when possible. If the same content is stored multiple times, check for duplicates before generating new embeddings.

Real-World Example: Customer Support Agent

Let’s see how these memory types work together in a customer support scenario.

Scenario

A user contacts support about a billing issue. The agent needs to:

1. Remember the user’s account details (long-term)
2. Recall previous support tickets (episodic)
3. Maintain the current conversation flow (short-term)

Implementation

@Service
public class CustomerSupportAgent {
    
    private final MemoryRouter memoryRouter;
    private final EpisodicMemoryService episodicMemory;
    private final LongTermMemoryService longTermMemory;
    private final ConversationMemory shortTermMemory;
    
    public String handleSupportRequest(String userId, String message) {
        // Store the interaction in episodic memory
        episodicMemory.storeEpisode(
            UUID.randomUUID().toString(), 
            userId, 
            shortTermMemory.getFullHistory(), 
            List.of()
        );
        
        // Build enriched prompt
        Prompt prompt = memoryRouter.buildPrompt(message, userId);
        
        // Generate response
        ChatResponse response = chatClient.call(prompt);
        String reply = response.getResults().get(0).getOutput().getText();
        
        // Extract and store new knowledge
        memoryExtractor.extractAndStore(
            new UserMessage(message),
            new AssistantMessage(reply)
        );
        
        return reply;
    }
}

In this example:

• Short-term memory tracks the current billing conversation
• Long-term memory provides account details and billing policies
• Episodic memory recalls previous billing issues and resolutions

The result is an agent that feels personalized and context-aware, rather than starting from scratch every time.

Common Pitfalls and Solutions

Pitfall 1: Memory Overload

Problem: Storing too much information makes retrieval noisy and slow.

Solution: Implement relevance scoring and threshold-based filtering. Only store memories above a confidence threshold. Use summarization to compress low-value episodes.

Pitfall 2: Stale Memories

Problem: Old information becomes inaccurate but persists in the store.

Solution: Add expiration timestamps and update mechanisms. When storing, check for existing similar memories and update them instead of creating duplicates.

public void storeOrUpdateFact(String content, Map<String, Object> metadata) {
    // Check for similar existing memories
    List<MemoryItem> similar = longTermMemory.retrieveRelevant(content, 1);
    
    if (!similar.isEmpty() && similar.get(0).getSimilarity() > 0.85) {
        // Update existing memory
        longTermMemory.update(similar.get(0).getId(), content, metadata);
    } else {
        // Store new memory
        longTermMemory.storeFact(content, metadata);
    }
}

Pitfall 3: Context Window Bloat

Problem: Retrieving too many memories exceeds the context window.

Solution: Implement a budgeting system. Allocate token budgets to each memory type and truncate as needed.

public class MemoryBudget {
    private static final int SHORT_TERM_BUDGET = 4000;
    private static final int LONG_TERM_BUDGET = 2000;
    private static final int EPISODIC_BUDGET = 1000;
    
    // Truncate memories to fit budget
    public List<Message> truncateToFitBudget(List<Message> memories, int budget) {
        // Implementation using token counting
    }
}

Testing Your Memory System

Don’t forget to test your memory implementation. Here’s a simple test strategy:

@SpringBootTest
class MemorySystemTest {
    
    @Autowired
    private LongTermMemoryService longTermMemory;
    
    @Autowired
    private EpisodicMemoryService episodicMemory;
    
    @Test
    void testLongTermMemoryRetrieval() {
        // Store a fact
        longTermMemory.storeFact("The API rate limit is 100 requests per minute", 
            Map.of("type", "fact"));
        
        // Retrieve it
        List<MemoryItem> results = longTermMemory.retrieveRelevant(
            "What is the API rate limit?", 3
        );
        
        // Verify
        assertThat(results).isNotEmpty();
        assertThat(results.get(0).getContent())
            .contains("rate limit");
    }
    
    @Test
    void testEpisodicMemoryChronology() {
        // Store two episodes
        episodicMemory.storeEpisode("session1", "user1", 
            List.of(new Message(Role.USER, "First message")), List.of());
        episodicMemory.storeEpisode("session2", "user1", 
            List.of(new Message(Role.USER, "Second message")), List.of());
        
        // Retrieve in chronological order
        List<EpisodicEvent> episodes = episodicMemory.retrieveRelated(
            "messages", "user1", 10
        );
        
        // Verify ordering
        assertThat(episodes).hasSize(2);
        assertThat(episodes.get(0).getTimestamp())
            .isBefore(episodes.get(1).getTimestamp());
    }
}

Key Takeaways

1. Three memory types serve different purposes: Short-term for immediate context, long-term for persistent knowledge, episodic for personal experiences.

2. Vector databases are essential for semantic retrieval: They enable similarity-based search over unstructured memory content.

3. The Memory Router is critical: It decides what to store, what to retrieve, and how to combine memories into prompts.

4. Performance matters: Use caching, async storage, and compaction to keep memory operations fast and cost-effective.

5. Test your memory system: Verify retrieval accuracy, ordering, and relevance scoring with comprehensive tests.

6. Avoid common pitfalls: Prevent memory overload, stale data, and context bloat with proper filtering and budgeting.

Building effective agent memory is an iterative process. Start simple with short-term and long-term memory, then add episodic capabilities as your use case demands. The goal is to create agents that feel truly personalized and context-aware, rather than stateless conversationalists starting from scratch every time.

Remember: good memory architecture is the difference between an agent that forgets and one that remembers. Choose your memory types wisely, and your agents will thank you with better performance and happier users.