LLM Routing and Model Selection: Smart Gateways for Cost and Quality

August 30, 2026 · LLM AI Engineering MLOps Cost Optimization Model Routing Production AI

The Economics of Generative AI in Production

We’ve all been there. You ship a feature powered by a large language model, it works beautifully in staging, and then it hits production. Suddenly, your API bill doubles. Then triples. You’re paying premium rates for tasks that barely require intelligence—a simple classification, a regex extraction, a yes/no question. Meanwhile, your most complex, nuanced requests are getting the same expensive treatment as trivial ones.

This is the central tension of production AI engineering: how do you balance cost, latency, and quality? The naive approach is to route everything to your best model. The smart approach? Build a routing layer that intelligently selects the right model for the right task at the right time.

In this post, I’ll walk you through building a production-grade LLM routing gateway. We’ll cover the architecture, the decision logic, cost optimization strategies, and real code you can adapt for your stack.

Why a Single Model Isn’t Enough

Before diving into solutions, let’s understand why routing matters. Modern LLM providers offer a spectrum of models:

• Tiny models (e.g., Phi-3, Gemma-2B): Fast, cheap, good for simple tasks
• Medium models (e.g., Llama-3-8B, Mistral-7B): Balanced performance and cost
• Large models (e.g., Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Pro): Powerful but expensive
• Specialized models: Code models, reasoning models, embedding models

A single model cannot optimally handle all workloads. Using GPT-4o to classify sentiment is like using a freight train to deliver a single envelope. Conversely, using a tiny model to write complex SQL queries will fail. The solution is context-aware model selection.

Architecture: The Routing Gateway

Our routing gateway sits between your application and the LLM providers. It intercepts requests, analyzes them, and routes to the optimal model. Here’s the high-level flow:

1. Request Reception: Client sends a prompt
2. Intent Classification: Determine the task type and complexity
3. Model Selection: Choose the appropriate model based on intent
4. Execution: Route to the selected model/provider
5. Response Handling: Return the result to the client
6. Metrics & Learning: Log performance for continuous optimization

The Decision Engine

The core of our gateway is the decision engine. It evaluates three signals:

• Task Type: What kind of work is this? (classification, extraction, generation, reasoning)
• Complexity: How difficult is the task? (measured by token count, ambiguity, required reasoning depth)
• Cost Sensitivity: What’s the acceptable cost per request?

Let’s look at how we implement this in practice.

Building the Router: A Java Implementation

For this example, I’ll use Java with Spring Boot, but the concepts translate to any language. We’ll build a gateway that routes requests based on intent classification and complexity scoring.

Project Structure

First, let’s define our project structure:

llm-router/
├── src/main/java/com/example/router/
│   ├── RouterApplication.java
│   ├── config/
│   │   └── RouterConfig.java
│   ├── model/
│   │   ├── RequestContext.java
│   │   ├── RoutingDecision.java
│   │   └── ModelEndpoint.java
│   ├── service/
│   │   ├── IntentClassifier.java
│   │   ├── ComplexityAnalyzer.java
│   │   └── ModelRouter.java
│   └── controller/
│       └── RouterController.java
└── src/main/resources/
    └── application.yml

Configuration

Let’s start with our configuration. We need to define available models and their characteristics:

# application.yml
llm-router:
  models:
    tiny:
      name: "phi-3-mini"
      provider: "local"
      cost-per-1k-tokens: 0.0001
      max-tokens: 2048
      latency-ms: 100
      capabilities: ["classification", "extraction", "simple-chat"]
      
    medium:
      name: "llama-3-8b"
      provider: "local"
      cost-per-1k-tokens: 0.0005
      max-tokens: 4096
      latency-ms: 250
      capabilities: ["classification", "extraction", "summarization", "simple-code"]
      
    large:
      name: "claude-3-5-sonnet"
      provider: "anthropic"
      cost-per-1k-tokens: 0.003
      max-tokens: 8192
      latency-ms: 800
      capabilities: ["reasoning", "complex-code", "analysis", "creative-writing"]
      
    specialized:
      name: "gpt-4o"
      provider: "openai"
      cost-per-1k-tokens: 0.005
      max-tokens: 128000
      latency-ms: 1000
      capabilities: ["all"]
      
  routing:
    default-model: "medium"
    fallback-model: "large"
    cache-enabled: true
    cache-ttl-seconds: 300

Core Models

Now let’s define our domain models:

// RequestContext.java
@Data
@Builder
public class RequestContext {
    private String prompt;
    private Integer maxTokens;
    private Double temperature;
    private String userId;
    private Long timestamp;
    private Map<String, Object> metadata;
}

// RoutingDecision.java
@Data
@Builder
public class RoutingDecision {
    private String selectedModel;
    private String provider;
    private Double estimatedCost;
    private Integer expectedLatencyMs;
    private String reasoning;
    private Map<String, Object> additionalContext;
}

// ModelEndpoint.java
@Data
@Builder
public class ModelEndpoint {
    private String id;
    private String name;
    private String provider;
    private BigDecimal costPer1kTokens;
    private Integer maxTokens;
    private Integer latencyMs;
    private List<String> capabilities;
    private String endpointUrl;
    private Map<String, String> headers;
}

Intent Classifier

The intent classifier determines what type of task the prompt represents. We can use a combination of keyword matching and a lightweight ML model:

@Service
public class IntentClassifier {
    
    private static final Map<String, List<String>> INTENT_PATTERNS = Map.of(
        "classification", Arrays.asList(
            "classify", "categorize", "sentiment", "topic", "label",
            "is this", "which category", "what type"
        ),
        "extraction", Arrays.asList(
            "extract", "find", "identify", "pull out", "get", 
            "what is the", "return the", "list the"
        ),
        "summarization", Arrays.asList(
            "summarize", "brief", "overview", "concise", "key points",
            "in short", "tl;dr", "what is the main idea"
        ),
        "reasoning", Arrays.asList(
            "why", "how does", "explain", "analyze", "compare",
            "evaluate", "assess", "what if", "reasoning"
        ),
        "code-generation", Arrays.asList(
            "write code", "implement", "function", "class", 
            "algorithm", "solve this", "debug", "fix"
        ),
        "creative-writing", Arrays.asList(
            "write a", "create a", "story", "poem", "email",
            "draft", "compose", "generate"
        )
    );
    
    public String classify(String prompt) {
        String lowerPrompt = prompt.toLowerCase();
        
        // Check for explicit intent indicators
        for (Map.Entry<String, List<String>> entry : INTENT_PATTERNS.entrySet()) {
            for (String pattern : entry.getValue()) {
                if (lowerPrompt.contains(pattern)) {
                    return entry.getKey();
                }
            }
        }
        
        // Default based on complexity heuristics
        if (prompt.length() < 50) {
            return "classification";
        } else if (prompt.contains("code") || prompt.contains("function")) {
            return "code-generation";
        } else if (prompt.length() > 500) {
            return "summarization";
        }
        
        return "general";
    }
}

Complexity Analyzer

Not all classification tasks are equal. A simple yes/no question is different from a multi-step analysis. Our complexity analyzer scores requests:

@Service
public class ComplexityAnalyzer {
    
    public double analyzeComplexity(RequestContext request) {
        double score = 0.0;
        String prompt = request.getPrompt();
        
        // Length factor
        int tokenCount = estimateTokens(prompt);
        score += Math.min(tokenCount / 100.0, 5.0);
        
        // Ambiguity factor
        if (containsAmbiguousLanguage(prompt)) {
            score += 2.0;
        }
        
        // Multi-step reasoning
        if (prompt.contains("step") || prompt.contains("first") || 
            prompt.contains("then") || prompt.contains("after")) {
            score += 1.5;
        }
        
        // Special characters and formatting
        if (prompt.contains("```") || prompt.contains("<") || 
            prompt.contains(">")) {
            score += 1.0;
        }
        
        // Question complexity
        if (prompt.contains("why") || prompt.contains("how")) {
            score += 1.0;
        }
        
        return score;
    }
    
    private int estimateTokens(String text) {
        // Rough estimation: 4 characters per token
        return (int) (text.length() / 4.0);
    }
    
    private boolean containsAmbiguousLanguage(String prompt) {
        String[] ambiguous = {"maybe", "perhaps", "somewhat", "kind of", "I think"};
        String lower = prompt.toLowerCase();
        return Arrays.stream(ambiguous).anyMatch(lower::contains);
    }
}

The Model Router

Now for the core routing logic. This is where we make the decision:

@Service
public class ModelRouter {
    
    @Autowired
    private IntentClassifier intentClassifier;
    
    @Autowired
    private ComplexityAnalyzer complexityAnalyzer;
    
    @Autowired
    @Qualifier("modelEndpoints")
    private Map<String, ModelEndpoint> modelEndpoints;
    
    public RoutingDecision route(RequestContext request) {
        String intent = intentClassifier.classify(request.getPrompt());
        double complexity = complexityAnalyzer.analyzeComplexity(request);
        
        // Define thresholds
        double simpleThreshold = 3.0;
        double moderateThreshold = 6.0;
        
        // Select model based on intent and complexity
        String selectedModel;
        String reasoning;
        
        if (complexity <= simpleThreshold) {
            // Simple tasks go to tiny/medium models
            if (isSimpleCapability(intent)) {
                selectedModel = "tiny";
                reasoning = String.format("Simple %s task with low complexity (%.1f)", 
                    intent, complexity);
            } else {
                selectedModel = "medium";
                reasoning = String.format("%s task with low complexity, needs medium model", intent);
            }
        } else if (complexity <= moderateThreshold) {
            // Moderate tasks go to medium/large models
            if (requiresReasoning(intent)) {
                selectedModel = "large";
                reasoning = String.format("Complex %s task requiring reasoning", intent);
            } else {
                selectedModel = "medium";
                reasoning = String.format("Moderate %s task with complexity %.1f", 
                    intent, complexity);
            }
        } else {
            // Complex tasks go to large/specialized models
            if (intent.equals("code-generation")) {
                selectedModel = "specialized";
                reasoning = String.format("Complex code task requiring specialized model", intent);
            } else {
                selectedModel = "large";
                reasoning = String.format("High complexity task (%.1f) requiring powerful model", complexity);
            }
        }
        
        // Validate capability match
        ModelEndpoint endpoint = modelEndpoints.get(selectedModel);
        if (!endpoint.getCapabilities().contains("all") && 
            !endpoint.getCapabilities().contains(intent)) {
            // Fallback to next capable model
            selectedModel = fallbackModel(intent, selectedModel);
            endpoint = modelEndpoints.get(selectedModel);
            reasoning += " [Fallback: capability mismatch]";
        }
        
        // Calculate estimated cost
        int estimatedTokens = estimateTokens(request.getPrompt());
        BigDecimal cost = endpoint.getCostPer1kTokens()
            .multiply(BigDecimal.valueOf(estimatedTokens / 1000.0));
        
        return RoutingDecision.builder()
            .selectedModel(selectedModel)
            .provider(endpoint.getProvider())
            .estimatedCost(cost)
            .expectedLatencyMs(endpoint.getLatencyMs())
            .reasoning(reasoning)
            .build();
    }
    
    private boolean isSimpleCapability(String intent) {
        return Arrays.asList("classification", "extraction").contains(intent);
    }
    
    private boolean requiresReasoning(String intent) {
        return Arrays.asList("reasoning", "analysis", "code-generation").contains(intent);
    }
    
    private String fallbackModel(String intent, String currentModel) {
        // Simple fallback logic - in production, you’d have a more sophisticated ranking
        if ("tiny".equals(currentModel)) {
            return "medium";
        } else if ("medium".equals(currentModel)) {
            return "large";
        }
        return "specialized";
    }
    
    private int estimateTokens(String text) {
        return (int) (text.length() / 4.0);
    }
}

The Controller

Finally, our REST controller ties it all together:

@RestController
@RequestMapping("/api/router")
public class RouterController {
    
    @Autowired
    private ModelRouter modelRouter;
    
    @PostMapping("/route")
    public ResponseEntity<RoutingDecision> route(@RequestBody RequestContext request) {
        RoutingDecision decision = modelRouter.route(request);
        return ResponseEntity.ok(decision);
    }
    
    @PostMapping("/chat")
    public ResponseEntity<Map<String, Object>> chat(@RequestBody RequestContext request) {
        // Route the request
        RoutingDecision decision = modelRouter.route(request);
        
        // In production, you’d call the actual LLM API here
        // For now, we’ll simulate a response
        Map<String, Object> response = new HashMap<>();
        response.put("model", decision.getSelectedModel());
        response.put("provider", decision.getProvider());
        response.put("estimatedCost", decision.getEstimatedCost());
        response.put("reasoning", decision.getReasoning());
        response.put("response", "This is a simulated response from " + decision.getSelectedModel());
        
        return ResponseEntity.ok(response);
    }
}

Advanced Routing Strategies

Multi-Model Ensemble

Sometimes the best approach isn’t choosing one model—it’s combining multiple models. For example:

1. Tiny model classifies the intent
2. Medium model extracts key entities
3. Large model generates the final response

This ensembles the strengths of each model while keeping costs down. Here’s a simplified implementation:

public String ensembleRoute(RequestContext request) {
    // Step 1: Classify with tiny model
    String intent = callModel("tiny", "Classify: " + request.getPrompt());
    
    // Step 2: Extract with medium model
    String entities = callModel("medium", "Extract entities from: " + request.getPrompt());
    
    // Step 3: Generate with large model (if needed)
    if (requiresComplexReasoning(intent)) {
        return callModel("large", request.getPrompt() + "\nEntities: " + entities);
    }
    
    return entities;
}

Cost-Aware Caching

One of the most effective cost-saving strategies is intelligent caching. If two users ask nearly identical questions, why pay twice? Here’s how to implement it:

@Service
public class CostAwareCache {
    
    private final Cache<String, RoutingDecision> decisionCache;
    private final Cache<String, String> responseCache;
    
    public CostAwareCache() {
        // Use Caffeine or similar for production
        this.decisionCache = Caffeine.newBuilder()
            .expireAfterWrite(5, TimeUnit.MINUTES)
            .maximumSize(10000)
            .build();
            
        this.responseCache = Caffeine.newBuilder()
            .expireAfterWrite(10, TimeUnit.MINUTES)
            .maximumSize(5000)
            .build();
    }
    
    public RoutingDecision getCachedDecision(RequestContext request) {
        String key = hashPrompt(request.getPrompt());
        return decisionCache.getIfPresent(key);
    }
    
    public String getCachedResponse(RequestContext request, RoutingDecision decision) {
        String key = hashPrompt(request.getPrompt()) + "_" + decision.getSelectedModel();
        return responseCache.getIfPresent(key);
    }
    
    public void cacheResponse(RequestContext request, RoutingDecision decision, String response) {
        String decisionKey = hashPrompt(request.getPrompt());
        decisionCache.put(decisionKey, decision);
        
        String responseKey = decisionKey + "_" + decision.getSelectedModel();
        responseCache.put(responseKey, response);
    }
    
    private String hashPrompt(String prompt) {
        // Use a fast hash like MD5 or MurmurHash
        return DigestUtils.md5Hex(prompt);
    }
}

A/B Testing and Continuous Learning

Production routing isn’t set-and-forget. You need to measure which models perform best for which tasks and adjust accordingly. Here’s a simple logging framework:

@Component
public class RoutingMetrics {
    
    private final List<RoutingLog> logs = new CopyOnWriteArrayList<>();
    
    public void logRouting(RequestContext request, RoutingDecision decision, 
                          long actualLatencyMs, boolean success) {
        RoutingLog log = RoutingLog.builder()
            .timestamp(System.currentTimeMillis())
            .promptLength(request.getPrompt().length())
            .intent(intentClassifier.classify(request.getPrompt()))
            .selectedModel(decision.getSelectedModel())
            .estimatedCost(decision.getEstimatedCost())
            .actualLatencyMs(actualLatencyMs)
            .success(success)
            .build();
        
        logs.add(log);
        
        // In production, send to metrics system (Prometheus, Datadog, etc.)
        publishMetrics(log);
    }
    
    public Map<String, Double> getModelCostEfficiency() {
        // Calculate cost per successful response by model
        return logs.stream()
            .filter(RoutingLog:: isSuccess)
            .collect(Collectors.groupingBy(
                RoutingLog::getSelectedModel,
                Collectors.averagingDouble(RoutingLog::getEstimatedCost)
            ));
    }
}

Production Considerations

Error Handling and Fallbacks

No routing system is perfect. You need robust fallback strategies:

1. Model failure: If a model times out or returns an error, retry with the next-best model
2. Cost overrun: If estimated cost exceeds budget, downgrade to a cheaper model
3. Quality degradation: If users report poor responses, automatically escalate to better models

public RoutingDecision routeWithFallbacks(RequestContext request) {
    try {
        RoutingDecision decision = modelRouter.route(request);
        
        // Validate the decision
        if (isModelUnhealthy(decision.getSelectedModel())) {
            decision = routeWithModel(decision.getSelectedModel() + "_backup", request);
        }
        
        // Check budget constraints
        if (decision.getEstimatedCost().compareTo(budgetLimit) > 0) {
            decision = downgradeModel(decision);
        }
        
        return decision;
        
    } catch (Exception e) {
        // Ultimate fallback
        return RoutingDecision.builder()
            .selectedModel("fallback")
            .provider("default")
            .estimatedCost(BigDecimal.ZERO)
            .reasoning("Error in routing: " + e.getMessage())
            .build();
    }
}

Monitoring and Observability

Track these key metrics:

• Routing distribution: What percentage of requests go to each model?
• Cost per model: Average cost per request by model
• Latency by model: Average response time by model
• Error rates: Failure rates by model and by task type
• Quality scores: User satisfaction or accuracy metrics by model

Security Considerations

When building a routing gateway, security is paramount:

1. Input validation: Sanitize all prompts to prevent injection attacks
2. Rate limiting: Prevent abuse with per-user and per-model rate limits
3. PII detection: Scan for personally identifiable information and redact or block
4. Audit logging: Log all routing decisions for compliance and debugging

public RequestContext sanitize(RequestContext request) {
    // Remove or redact PII
    String sanitizedPrompt = redactPII(request.getPrompt());
    
    // Validate length
    if (sanitizedPrompt.length() > 10000) {
        throw new IllegalArgumentException("Prompt too long");
    }
    
    // Check for malicious patterns
    if (containsMaliciousPatterns(sanitizedPrompt)) {
        throw new SecurityException("Potentially malicious input detected");
    }
    
    return RequestContext.builder()
        .prompt(sanitizedPrompt)
        .maxTokens(request.getMaxTokens())
        .temperature(request.getTemperature())
        .userId(request.getUserId())
        .timestamp(request.getTimestamp())
        .build();
}

Real-World Performance Gains

Let’s talk numbers. In a typical production deployment, a well-tuned routing system can deliver:

• 60-80% cost reduction by routing simple tasks to smaller models
• 2-3x latency improvement for common requests by using faster models
• Better user experience by matching model capability to task complexity
• Scalability by distributing load across multiple models and providers

For example, at my company, we route about 40% of our traffic to tiny/medium models, 45% to large models, and 15% to specialized models. This gives us the best balance of cost and quality.

Key Takeaways

1. Not all requests are equal: Simple classification tasks don’t need GPT-4. Use routing to match task complexity with model capability.

2. Cost optimization is continuous: Monitor your routing metrics regularly and adjust thresholds as model pricing and capabilities evolve.

3. Fallback strategies are essential: Build in graceful degradation. When one model fails, have a backup ready.

4. Caching pays off: Intelligent caching of both routing decisions and responses can dramatically reduce costs for repetitive queries.

5. Security first: A routing gateway is a high-value target. Implement input sanitization, rate limiting, and audit logging.

6. Measure everything: Track cost, latency, error rates, and quality by model. Data-driven decisions beat gut feelings.

7. Start simple, iterate: You don’t need a perfect routing system on day one. Start with basic intent classification and complexity scoring, then add sophistication as you learn what works.

The future of LLM-powered applications isn’t about using the biggest model—it’s about using the right model at the right time. Build that routing layer, and you’ll save money, improve performance, and deliver a better experience to your users.

What routing strategies have you found effective in your production systems? Share your experiences in the comments below.