AI & Agentforce

Hybrid LLM Routing: Send Each AI Request to the Right Model

Hybrid LLM routing sends each AI request to the right model: open-weight for sensitive data, frontier APIs for hard tasks. Here is how to set it up.

Most French companies that use generative AI seriously end up with the same problem. One frontier model handles everything, the invoice grows every month, and the data protection officer keeps asking why customer emails leave the EU to answer a question a much smaller model could handle. Hybrid LLM routing fixes this: instead of choosing one model for the whole company, you send each request to the model that fits its sensitivity, difficulty and budget. This guide explains how it works, what changed in 2026, and how to set it up without building a research project.

What hybrid LLM routing means in practice

A router sits between your applications and your models. Your CRM assistant, your support chatbot or your internal agent sends one ordinary API call. The router inspects the request, then decides which model answers it: a small open-weight model on your own servers, an EU-hosted open model, or a frontier API such as Claude, GPT or Gemini.

The open-source vLLM Semantic Router is a good example of the pattern. Its v0.2 release in March 2026 extracts signals from each request (intent, complexity, language, context), matches them against routing decisions, and then picks a model with a selection algorithm tuned for quality, latency and cost. One option, AutoMix, starts with a cheaper model and escalates only when needed. Jailbreak and personal-data (PII) detection run in the main routing path rather than as optional add-ons.

In July 2026 the project went a step further and described a Mixture-of-Models approach: several independent models, possibly with different owners, licences and hardware, exposed behind one stable model interface. The detail that matters for European companies is explicit: privacy constraints can rule out a stronger remote model, and the system then follows a declared fallback instead of quietly sending data to the cloud.

vLLM Semantic Router diagram of Mixture-of-Models execution topologies across cloud, data center and edge (vllm.ai)

Why 2026 makes hybrid easier for European companies

Two years ago, a hybrid setup meant running everything you did not send to a US API yourself. That is no longer the only option.

Regional processing from a European lab: Mistral made its Regional Endpoints generally available in August 2026, letting customers choose whether inference runs in Europe or the United States. Mistral notes that limited transfers to subcontractors outside the region can still occur, so read its Trust Center before you write "EU only" in a contract.

Open models served by someone else: in the same announcement, Mistral said it will host third-party open models on its infrastructure, starting with Z.ai's GLM-5.2, with the same regional controls as its own models. You get an open-weight model without buying GPUs.

A clearer regulatory calendar: according to the European Commission's AI Act page, obligations for general-purpose AI models have applied since 2 August 2025, transparency obligations from August 2026, and high-risk obligations for Annex III use cases (employment, education and similar) were moved to 2 December 2027 by the AI Omnibus. A router that logs which model handled which request makes this documentation much easier.

Mistral AI diagram of Mistral and third-party open models served on Mistral infrastructure with regional endpoints (mistral.ai)

If you are still deciding whether to self-host at all, start with our self-hosted LLM vs frontier API decision guide. Routing is the next step once you accept that the answer is usually "both".

How to split traffic: a routing policy you can defend

A routing policy is a short set of rules that IT, legal and the business can all read. Start with three tiers and resist the urge to add more.

Tier 1, local or EU-hosted open-weight model. Choose it if:

  • the request contains personal data, HR data, health data or contract terms you do not want to leave your perimeter;
  • the task is repetitive and well bounded: classification, field extraction, summarising a call note, tagging tickets;
  • volume is high and a few points of quality do not change the business outcome.

Tier 2, frontier API through an EU region or enterprise contract. Choose it if:

  • the task needs strong reasoning, long documents or multi-step tool use;
  • the data has been minimised or pseudonymised by the router before it leaves;
  • an error is costly enough that the extra price per request is worth it.

Tier 3, refuse or ask a human. Choose it if the router detects a jailbreak attempt, a request outside the use case, or sensitive data in a flow that is not approved for any model.

Write the policy before you pick tools. The same rules can be implemented with vLLM Semantic Router, a commercial AI gateway or a simple rules layer in your integration platform.

What to do now: a six-step rollout

  1. List your AI calls. Export one month of usage from each tool and API key: which application, which model, how many requests, which data categories. Most companies discover two or three flows that represent most of the cost.
  2. Classify each flow by sensitivity and difficulty. A simple two-axis view is enough: "contains personal data, yes or no" and "needs frontier reasoning, yes or no".
  3. Build an evaluation set per flow. Fifty to a hundred real, anonymised examples with expected answers. Run them on the candidate open model and on your current frontier model. Our guide to AI workflow evaluation before scaling shows how to score them.
  4. Start with one flow and a fallback. Route the easiest, highest-volume flow to the open model, keep the frontier model as an escalation path, and log every routing decision.
  5. Add guards in the router, not in each app. PII detection, prompt-injection checks and per-team budgets belong in one place so every application benefits.
  6. Review monthly. Compare cost per request, escalation rate and user complaints. New open models appear every few weeks; a router lets you swap one in without touching the applications.

Trade-offs to keep in mind

Routing is not free. You add a component that must be monitored, secured and versioned. A misclassified request can send sensitive data to the wrong model or give a weak answer where quality mattered, which is why evaluation and logs come first. Running your own open models also means owning updates, capacity planning and security patches; an EU-hosted provider removes that work but adds a contract to review.

For most mid-size French companies, the realistic target is modest: one router, two or three models, a written policy and a monthly review. That alone usually brings the cost and data-protection discussions back under control.

Conclusion

The question is no longer "open-weight or frontier?" but "which request goes where, and can we prove it?". Hybrid LLM routing gives you that control while keeping access to the best models when you need them. If you want help mapping your AI flows and writing a routing policy, our AI strategy and roadmap service starts there, and our AI implementation team can build and evaluate the router with your IT team.

Read next

More on AI & Agentforce

All 23 articles

Got a CRM project in mind?

Tell us where you are. We will tell you honestly what it takes, in a 30-minute call.