The standard model routing question is which cloud provider to call. Anthropic vs. OpenAI vs. Google. Frontier model vs. open-weight. High-capability vs. cost-optimized. These are all cloud routing decisions — the model lives somewhere in a data center, and you're choosing which data center.
A new question is entering the routing conversation: should this call go to the cloud at all? Needle, a 26M-parameter function-calling model from Cactus Compute, runs at 6,000 tokens/second on-device with no API cost, no network round-trip, and no data leaving the edge. For simple tool dispatch — extracting structured data from a fixed schema, routing a user intent to a predefined action, classifying text into a known taxonomy — Needle performs equivalently to frontier models at a fraction of the cost.
The fraction, in this case, is zero. On-device inference has no per-call cost. The model runs locally. Nothing leaves the device.
Not every model call needs frontier capability. The use cases where Needle-class models are appropriate:
These use cases share a property: the output space is bounded and known in advance. The model doesn't need to reason about open-ended questions — it needs to pattern-match inputs to outputs from a finite set. Small specialized models excel at exactly this.
They're also high-volume in most production AI applications. Intent classification runs on every user interaction. Structured extraction runs on every document. Categorical classification runs on every content item. At scale, these are the calls that dominate the inference bill — and they're the calls where on-device routing saves the most.
The two-tier routing architecture:
The routing decision between tiers is based on task type, not cost alone. A call that requires multi-step reasoning, nuanced language generation, or access to broad world knowledge goes to the cloud tier. A call that requires structured output from a known schema stays at the edge.
This architecture has properties that cloud-only routing doesn't:
Function calling is a specific capability distinct from text generation. The model must parse a function schema, understand which function to call given the user's input, extract the correct arguments, and return a structured JSON response in the exact format the calling application expects. This requires reliability on a structured output task — not creativity or breadth of knowledge.
Frontier models support function calling because they were trained on it. Small general-purpose models often don't — or perform unreliably. Needle is specifically designed and trained for function calling: the 26M parameters are optimized for this task class, not for general-purpose generation. The result is a model that outperforms its size on function calling and underperforms its size on everything else — which is exactly the right trade-off for on-device tool dispatch.
The practical implication: Needle can serve as the routing brain for a local AI agent — deciding which tool to call, with which parameters — without sending the routing decision to the cloud. The cloud call only happens when the selected tool requires it. The tool dispatch itself is free.
Trimio's routing layer supports local endpoint configuration — a self-hosted or on-device inference endpoint that receives routing traffic alongside cloud providers. When a local endpoint is configured, Trimio includes it in the routing decision: if the task type is within the local model's capability range, and the local endpoint is available, the call routes locally at zero marginal cost.
The routing logic is the same as cloud LCR, extended to include on-device candidates. The eligibility criteria for a local endpoint routing decision:
When all three are met, the call routes locally. When any fails, it falls back to the cloud tier automatically. The application never handles this routing logic — it sends calls to Trimio, and Trimio routes to the best available endpoint for each call.
Edge vs. cloud routing is a new dimension in the AI cost optimization conversation — and it's one where the savings on high-volume simple workloads can be substantial. Not every call needs to go to the cloud. The routing layer should know which ones don't.
Trimio supports local endpoint routing alongside cloud providers — route simple tool calls to on-device models at zero marginal cost, fall back to cloud automatically when local capability is insufficient. See how it works.