Overview
OrcaRouter is a large language model (LLM) router designed for production environments. Its primary function is to determine which specific LLM should process an incoming request, considering the varied capabilities and inference costs associated with different models. The system integrates a LinUCB-based contextual bandit mechanism, processing features derived from lexical and sentence embeddings to inform its routing decisions.
Approach
OrcaRouter implements a hybrid offline-online learning protocol. During the offline phase, the system gathers full-information feedback. This involves evaluating each candidate model against a designated set of routing prompts. The outcomes of these evaluations generate a reward matrix, which is subsequently used to fit one ridge regressor per 'arm' (representing a candidate model).
Upon deployment, OrcaRouter initializes its operational parameters using the results from this offline learning phase. Once deployed, the system offers an optional capability for continuous learning, adapting to bandit feedback. In this online learning mode, updates are applied only to the selected model's arm after the observation of its reward, allowing for dynamic adjustments in routing strategy.
Findings
At the time of its submission to RouterArena, specifically on May 20, 2026, OrcaRouter-Adaptive attained the second position on the public leaderboard. Its performance was quantified by an arena score of 72.08. The system demonstrated an accuracy of 75.54% when routing queries. The associated operational cost for OrcaRouter was reported as USD 1.00 per 1,000 queries.
Why This Matters
The development of diverse large language models, each possessing distinct capabilities and incurring varying inference costs, necessitates efficient resource allocation in deployment scenarios. A production-oriented LLM router like OrcaRouter addresses the practical challenge of selecting the most appropriate model for a given request, aiming for effective utilization while managing operational expenses.