BitRouter

A context-aware LLM router that continuously improves agent workflows.

Key takeaways

Production agent workflows need routing that balances quality, cost, and latency.

Routing policies improve by learning from execution traces and user-defined evaluations.

Self-hosting can keep traces local; hosted optimization requires access to them.

Summary

BitRouter addresses the cost and latency of production agent workflows that rely heavily on frontier models. A team can bring its existing models, orchestration, and evaluations, then define the tradeoff it wants between quality and cost.

The proposed loop represents workflow steps as routing-policy files, observes execution traces, compares results with user-defined evaluations, and updates the policy toward the chosen objective. Observability is integrated into the gateway specifically to feed that loop.

The speaker reported a thirty-three-percent cost reduction without performance loss on a Terminal-Bench 2.1 configuration, along with presentation-time usage, repository-star, and download figures. Those numbers were not independently reproduced here. For privacy, the speaker said teams can self-host or use private deployment; the hosted optimization service needs the traces it learns from.

Why it matters

Optimization happens at the workflow level

Instead of treating every language-model call identically, the router can learn which parts of a specific agent workflow need more expensive capability and which can use cheaper routes.

Limits & caveats

The thirty-three-percent benchmark result is speaker-reported for one Terminal-Bench configuration and does not establish the same saving for other workflows or evaluation schemes.
Monthly-token, repository-star, and download figures are time-sensitive presentation metrics and were not independently verified in the packet.
Hosted optimization requires trace access according to the speaker; teams with stricter privacy requirements must evaluate self-hosted or private-deployment options and their own data-handling controls.
Production performance depends on the quality of the evaluations and objectives a team supplies.

How it works

Define the objective

A team supplies its workflow, models, evaluations, and a target tradeoff between quality and cost.

Encode each routing decision

Each workflow step gets a YAML routing-policy file that determines which model or route handles the call.

Observe and evaluate

The gateway records traces and compares outputs against evaluations supplied for the production task.

Update and evaluate again

The reinforcement-learning loop uses the evaluation results to update the routing policy, then tests the revised policy against the same goal.

Transcript

So now you guys have Smithers workflow running. Let's try to make it cheaper. So I'm Kelson. I'm one of the co-founders of BitRouter. We are building the self-improving LLM router that optimize your agentic workflow with everyone. So here's the problem. Almost everyone is tokenmaxing, and that's fine for early users. But it breaks when at the moment you are running production agentic workflow at scale, as cost and latency becomes unterrible, because tokenmaxing only achieves accuracy by sacrificing cost and latency. So to solve this, we built a LLM router that learns and adapts to every agent workflow. You can just bring your own models, bring your own harness, bring your own stability, bring, bring your own scales, bring your own orchestration.

BitRouter works with any loop, any workflows, and you just set a goal between between balancing quality and cost. The BitRouter will optimize toward that goal for you. So here's how it works. We use the reinforcement learning approach to make the self-improving optimization loop work. It... You can define every single step in the whole workflow as a, as a routing policy file, if, as a YAML file. The BitRouter will observe all those traces and, set evals to compare against the goal you are targeting. Then it will learn and keep fine-tuning the routing policy until it matches the goal. So we start where tokenmaxing is unsustainable, which is AI AI agent applications that are running production agentic loop at scale.

But we believe in the future when everybody loop maxing, then tokenmaxing will become a pain in the ass for everybody. So unlike other gateway, we integrate observability into our gateway solely to close this optimization loop. So other gateway will rot, but ours will grow with you. And as you saw in the last slide, we are fully open source. We use open source as our rules and as our distribution. We try to embed ourselves in the OSS AI communities. Sits above the open weight models and below the open source hunters. And the more agent trace runs through us, the better we can optimize for every workflow, because it's are our AOC. And so basically, every LLM, LLM call will become a data flywheel.

And so we have benchmarked BitRouter on Terminal Bench 2.1 by pairing it with base model, previous pre-proprietary models like GPT 5.6, Cloud Figment 5, and it achieves a stable thirty-three percent cost reduction without performance loss. And other And we are we are confident to go over to forty and fifty percent in near term. And also we have three billion monthly token usage, two hundred GitHub stars, fully open source, and over two thousand downloads right now. And here's us. We are the rotary we are building. We are trying to use our Chinese origin as our supply chain advantage to get cheaper token. And we have built our partnership with Tencent, Alibaba, ZAI, Staff, and more.

And also we want to use our OSS community, rules to help us to distribute. And also we have raised angel round, from Westlake China, OSG Ventures, China, et cetera. And we are looking for potential investor, partner, and contributor.

Okay. For Kelson, how do you do evaluation of, like, how... I don't remember if you said it, but, how much more optimization do you do on, like, on some example workflow? And then so I can just give the mic also, what privacy policy do you have regarding the traces? Like, does everyone opt in to letting you do improvement of BitRouter using their traces?

Okay. So for the second question, since we are, we're in our really early stage and our product is fully open source, so if you don't want to get to make us have your trace, you can just run it locally or deploy it yourself. We don't care. But if you use it with us, the, either you have to give us the trace because we have to get the trace. We, like, we will provide a privacy policy term of services, or if you are enterprise, you know, there's like a private, private deployment options for you. And for the, for the evals part, so right now, the way I was talking about, like, we benchmark on Terminal Bench 2.1 is, we pair those SOTA pro-proprietary models like GPT and Cloud with, solely with Kimi K 2.7 code.

That's, that's the way how we achieve this now, and it, achieves thirty-three percent reduction. And basically, the eval thing is that every time, it's not the agent that is evaluating right now. It's we have a code-based evaluation framework, and it's does it match the goal you are seeing? Because if you are you, if you are evaluating on benchmark, then the goal is very... The goal of the quality is very, real. Since it's like, how much goal can you get? Then it's, like, super easy. But if it's in production, then it's like your own evals and your own goals. You have to bring your own evals, and we will optimize toward it. But the, IO approach that I just introduced is adaptive to every kind of workflow.

I just done an integration with Smithers workflow, optimizing one of, two of the Smithers workflow template, and we achieve a stable twenty percent cost reduction by pairing it with Kimi. And, so if you can see it, works with every workflow. Yes.