Pandora's AI Model Routing Box: Efficient Allocation Under Expensive Valuation
This paper investigates how to route queries efficiently to the most suitable AI expert models. Heterogeneous AI systems comprise multiple models, architectures, inference frameworks, or inference-time settings, and by assigning queries to the processor best suited and cheapest, one can improve both quality and efficiency simultaneously. The key to routing lies in estimating each expert's expected return, but valuation itself is costly: cheap estimators (such as embedding-based predictors) are fast yet noisy, while accurate estimators (such as fine-tuned models with access to retrieval results or partial reasoning chains) are expensive. The authors formalize this trade-off as the classic optimal search problem, Pandora's boxes. Under a Gaussian signal model, the resulting strategy admits a closed-form expression for the value of information, indicating whether it is worth paying to refine the valuation for each expert and input. The centralized policy is called the Pandora router, and its decentralized extension is the Pandora bidder, where experts autonomously decide whether to invest in self-assessment to accept a bid. Experiments across three domains show that the Pandora router matches exhaustive valuation in routing quality while using expensive estimators more sparingly.
Background and Context
Heterogeneous AI systems increasingly combine multiple models, distinct architectures, different inference frameworks, and variable inference-time settings within a single deployment. The central operational challenge is routing each incoming query to the processor best suited to answer it at the lowest cost. Doing so can improve both quality and efficiency simultaneously, but the prerequisite is estimating each expert's expected return on a given query.
The paper identifies a genuine tension in this estimation. Cheap estimators, such as embedding-based predictors, run quickly but produce noisy, low-accuracy outputs. Accurate estimators, such as fine-tuned models with access to retrieval results or partial reasoning chains, are reliable yet demand substantial computational or time costs. This friction between the cheap-and-noisy and the expensive-and-precise forms the core problem the authors aim to resolve.
Deep Analysis
The authors formalize this trade-off as the classic optimal search problem known as Pandora's boxes, in which checking a box incurs a cost. Under a Gaussian signal model, which renders the problem analytically tractable, the resulting policy admits a closed-form expression for the value of information. This quantity measures the expected gain from reducing uncertainty by acquiring additional information.
That expression converts an intuitive judgment into a quantifiable decision rule, determining whether it is worth paying to refine the valuation for each expert and each specific input. The centralized version of this policy is called the Pandora router, operated by a single controller that decides when to invoke cheap estimators and when to switch to expensive ones.
The framework extends to a decentralized setting called the Pandora bidder. Here, experts are no longer assigned centrally; instead, they autonomously decide whether to invest in self-assessment and whether to accept a bid to claim a query. This models strategic behavior in competitive environments, where each participant weighs the cost of self-evaluation against the payoff of claiming work.
Industry Impact
Validation occurred across three domains. The first is a standard multi-LLM benchmark measuring routing between large language models. The second is a retrieval-augmented expert system, testing expensive estimators with access to external retrieval results. The third involves models with variable inference-time reasoning, evaluating routing across different inference budgets.
Results show the Pandora router matches exhaustive valuation, which evaluates all experts at once, while using expensive estimators more sparingly and thereby saving significant valuation cost. Ablation studies reveal a dual role for the value-of-information reasoning: when competing estimates are accurate, it improves allocation efficiency, but when those estimates are noisy, it can instead benefit the strategic experts themselves at the expense of other participants.
For industrial deployments, where heterogeneous routing is already common, the closed-form solution and clear decision rules offer a practical theoretical basis for controlling inference cost while preserving quality. In the open-source community, the framework encourages researchers to incorporate valuation costs into routing design rather than optimizing routing accuracy alone.
Outlook
The Pandora's boxes perspective connects routing to optimal search and information-value theory, opening a path for introducing economic strategic analysis into multi-model systems. The decentralized Pandora bidder is particularly notable, providing a new tool for understanding expert behavior under competition and pointing toward future routing mechanisms that more closely resemble real markets.
The finding that estimation precision directly shapes the final outcome of mechanism design suggests that future work must treat estimator quality as a first-class design variable. As deployments scale, balancing the cost of valuation against the quality of routing will remain a central engineering concern, and this work supplies both the analytical machinery and the conceptual framing to address it.