LLM Observability: Tracing, Evaluations, and Langfuse Deep Dive

· Observability LLM OpenAI MLOps Langfuse Python LLM Engineering
LLM Observability: Tracing, Evaluations, and Langfuse Deep Dive

The Black Box Problem in Production AI

If you have ever deployed a Large Language Model (LLM) application to production, you know the anxiety that comes with it. Unlike traditional software where a bug might throw a stack trace, LLMs are probabilistic. They hallucinate, they drift, and they behave inconsistently under load. When a user complains that your chatbot is “being weird,” you cannot simply look at the logs to understand why. The input might have been identical, but the output changed due to temperature fluctuations, context window shifts, or subtle prompt engineering tweaks.

This is the core problem of LLM observability. Traditional APM tools like Datadog or New Relic are excellent for tracking latency and error rates, but they are blind to the semantic content of an LLM call. They don’t know if the model answered correctly, if the prompt was truncated, or if the retrieval-augmented generation (RAG) pipeline pulled irrelevant documents.

In this post, we will move beyond basic logging and dive deep into LLM observability. We will explore the two pillars of reliable AI systems: Tracing and Evaluations. Then, we will build a practical implementation using Langfuse, an open-source LLM engineering platform that has become the industry standard for open-source observability. By the end of this guide, you will have a production-ready setup that gives you visibility into every token generated, every context window used, and every evaluation score computed.

Why Standard Logging Fails for LLMs

Before we install any tools, it is crucial to understand why print(response) is not enough. In a traditional CRUD application, logging is linear. You log the request, the database query, and the response. In an LLM application, the flow is non-linear and multi-layered.

Consider a RAG pipeline. A single user query triggers:

  1. A vector database search (retrieval).
  2. Prompt assembly (injecting context).
  3. An LLM API call (generation).
  4. Post-processing (extraction).

If you only log the final output, you have no idea if the failure came from poor retrieval (retrieving irrelevant docs) or poor generation (the model ignoring the context). This is known as the debugging gap. Without granular tracing, you are guessing. With observability, you are diagnosing.

Furthermore, LLMs are non-deterministic. Running the same prompt twice might yield two different answers. To detect regressions, you need to compare outputs over time, not just check for exceptions. This requires storing structured data about every interaction, including metadata like tokens used, latency, model version, and user IDs. This is the foundation of LLM observability.

The Two Pillars: Tracing and Evaluations

LLM observability rests on two distinct but interconnected pillars. Understanding the difference is key to building a robust system.

1. Tracing: The “What Happened”

Tracing is the practice of recording the sequence of events that occur during an LLM interaction. A trace is a collection of spans. In the context of LLMs, a span can represent:

Each span captures:

Tracing allows you to visualize the “journey” of a request. If a response is slow, you can see if the bottleneck was the vector search or the LLM inference. If the response is wrong, you can inspect the prompt that was actually sent to the model, including the injected context, to see if the retrieval step failed.

2. Evaluations: The “How Good Was It”

Tracing tells you what happened, but it doesn’t tell you if it was good. Evaluations are the mechanism for assessing the quality of LLM outputs. Unlike traditional software tests which are deterministic (pass/fail), LLM evaluations are often probabilistic or semantic.

There are three main types of evaluations:

Evaluations can be run asynchronously. You might generate a response, log it, and then run an evaluation pipeline in the background to score it later. This decouples the user-facing latency from the quality assurance process.

Why Langfuse?

The observability landscape for LLMs is crowded, with players like LangSmith, Weights & Biases, and Arize. However, Langfuse has emerged as a leading choice for several reasons:

  1. Open Source Core: Langfuse offers a robust open-source version that you can self-host. This is critical for enterprises with strict data privacy requirements, as you keep your prompt data and user interactions within your own infrastructure.
  2. Language Agnostic: While it integrates seamlessly with Python and JavaScript, Langfuse is not tied to a specific framework. It works with raw OpenAI clients, LangChain, LlamaIndex, and custom implementations.
  3. Cost Efficiency: The open-source model means you avoid the per-session pricing models that can explode as your usage scales.
  4. Feature Rich: It supports tracing, evaluations, prompt management, and analytics out of the box.

For this deep dive, we will use Langfuse because it provides the best balance of power, flexibility, and control for production systems.

Setting Up the Environment

Let’s build a practical example. We will create a simple RAG-based Q&A system and instrument it with Langfuse. First, ensure you have Langfuse installed. You can run it via Docker Compose, which is the recommended approach for local development and production.

Create a docker-compose.yml file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
version: '3.8'
services:
  langfuse:
    image: langfuse/langfuse:2
    ports:
      - "3000:3000"
    environment:
      - DATABASE_URL=postgresql://postgres:postgres@db:5432/postgres
      - NEXTAUTH_SECRET=secret
      - SALT=localsalt
      - HASHED_SALT=localsalthashed
    depends_on:
      - db
  db:
    image: postgres:15
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=postgres
    volumes:
      - pgdata:/var/lib/postgresql/data
volumes:
  pgdata:

Run docker-compose up -d to start the service. Once running, you can access the Langfuse UI at http://localhost:3000. Note the default credentials (admin@langfuse.com / password) for your first login.

Next, install the Python client:

1
pip install langfuse openai

Implementing Tracing

Let’s write a Python script that interacts with the OpenAI API and traces the interaction. We will simulate a simple chatbot that answers questions about a document.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
import os
import langfuse
from langfuse.decorators import observe
from openai import OpenAI

# Initialize Langfuse client
langfuse_client = langfuse.Langfuse(
    public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
    secret_key=os.environ["LANGFUSE_SECRET_KEY"],
    host=os.environ.get("LANGFUSE_HOST", "http://localhost:3000")
)

# Initialize OpenAI client
openai_client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

@observe()
def generate_response(user_input: str) -> str:
    # This decorator automatically creates a span in Langfuse
    # capturing input, output, and metadata
    
    # Simulate a retrieval step (in a real app, this would query a vector DB)
    retrieved_context = "The capital of France is Paris."
    
    prompt = f"Context: {retrieved_context}\n\nQuestion: {user_input}\n\nAnswer:"
    
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.7
    )
    
    return response.choices[0].message.content

# Run the function
result = generate_response("What is the capital of France?")
print(f"Response: {result}")

# Flush the client to ensure data is sent
langfuse_client.flush()

When you run this script, navigate to your Langfuse dashboard. You will see a new trace. Clicking on it reveals the span we created. You can see the exact prompt sent to OpenAI, the response received, the token usage, and the latency. This level of detail is invaluable for debugging. If the answer was wrong, you can inspect the prompt to see if the context was injected correctly.

Advanced Tracing: Nested Spans

In complex applications, you might have multiple LLM calls. Langfuse supports nested spans to represent hierarchical relationships. Let’s extend our example to include a reasoning step before generating the final answer.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
@observe()
def reason_about_query(user_input: str) -> str:
    # First LLM call: Reasoning
    reasoning_prompt = f"Analyze the following question and identify key entities: {user_input}"
    reasoning_response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": reasoning_prompt}]
    )
    return reasoning_response.choices[0].message.content

@observe()
def generate_response_with_reasoning(user_input: str) -> str:
    # Parent span
    reasoning = reason_about_query(user_input)  # Child span
    
    prompt = f"Reasoning: {reasoning}\n\nQuestion: {user_input}\n\nAnswer:"
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.3
    )
    return response.choices[0].message.content

In the Langfuse UI, you will now see a parent span generate_response_with_reasoning containing a child span reason_about_query. This tree structure allows you to drill down into specific sub-processes. If the final answer is poor, you can check if the reasoning step was flawed, indicating a need to improve the reasoning prompt or the model choice.

Implementing Evaluations

Tracing is passive; it records what happened. Evaluations are active; they judge the quality. Langfuse allows you to define evaluations in Python and run them asynchronously.

Let’s create a simple evaluation that checks if the answer contains the word “Paris”. In a production system, you might use an LLM-as-a-judge, but for this example, we’ll use a rule-based approach to keep it simple.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
from langfuse import Langfuse
from langfuse.api.resources.commons.types.observation_level import ObservationLevel

def evaluate_answer(trace_id: str, user_input: str, model_output: str):
    langfuse = Langfuse(
        public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
        secret_key=os.environ["LANGFUSE_SECRET_KEY"],
        host=os.environ.get("LANGFUSE_HOST", "http://localhost:3000")
    )
    
    # Define the evaluation logic
    is_correct = "paris" in model_output.lower()
    score = 1.0 if is_correct else 0.0
    
    # Create an observation for the evaluation
    langfuse.score(
        body=score,
        name="answer_correctness",
        trace_id=trace_id,
        level=ObservationLevel.OBSERVATION,
        comment="Rule-based check for keyword 'Paris'"
    )
    
    langfuse.flush()

# Integrate evaluation into our trace
@observe()
def generate_and_evaluate(user_input: str):
    trace = langfuse_client.trace(id="trace-123") # Get trace ID
    result = generate_response(user_input)
    evaluate_answer(trace.id, user_input, result)
    return result

When you run this, Langfuse will attach a score to the trace. You can then filter and sort your traces by this score. This allows you to monitor the quality of your application over time. If the average score drops, you know there is a regression, even if the application is not throwing errors.

Production Best Practices

Implementing tracing and evaluations is just the beginning. To make observability effective in production, consider these best practices:

  1. Instrument Early: Add Langfuse instrumentation to your codebase from day one. Refactoring legacy code to add tracing is difficult and error-prone.
  2. Anonymize PII: LLM applications often process sensitive user data. Ensure you are not logging Personally Identifiable Information (PII) in your traces. Langfuse allows you to mask sensitive fields using regular expressions.
  3. Cost Tracking: LLM API calls can be expensive. Always log token usage. Langfuse automatically tracks input and output tokens, allowing you to calculate the cost per trace and identify expensive patterns.
  4. Feedback Loops: Incorporate human feedback into your system. Allow users to rate responses (thumbs up/down) and store this feedback in Langfuse. This data is invaluable for fine-tuning models and improving prompts.
  5. Alerting: Set up alerts for critical metrics. For example, alert if the latency exceeds 5 seconds or if the evaluation score drops below 0.8 for a specific user segment.

Analyzing Data in Langfuse

Once you have collected data, the Langfuse UI becomes your command center. Here are some key views to explore:

For example, if you suspect that a new prompt is causing more hallucinations, you can create a new version of the prompt in Langfuse, route a percentage of traffic to it, and compare the evaluation scores between the old and new versions. This data-driven approach to prompt engineering is far more reliable than guesswork.

Conclusion

LLM observability is not a luxury; it is a necessity for any production AI system. Without tracing, you are flying blind. Without evaluations, you cannot measure quality. Tools like Langfuse provide the infrastructure to make your LLM applications transparent, debuggable, and improvable.

By implementing granular tracing, automated evaluations, and cost tracking, you can move from reactive firefighting to proactive optimization. You will be able to answer critical questions: Why did the model fail? Which prompt version performs best? How much is this feature costing us?

Start small. Instrument one function. Add one evaluation. Iterate from there. The journey to reliable LLMs is continuous, but with the right observability tools, you will always know where you stand.

Key Takeaways