PACE: Perception-Delay Routing and Filling Control for QoE-Efficient RAG Conversation Services

Background and Context

In conversational AI services, the dominant user experience bottleneck is frequently not answer accuracy but the length of the wait. When a user poses a question, the system must leave a processing window before producing a complete answer, and the emptiness of that interval directly shapes perceived quality. The PACE framework, introduced via an arXiv paper, addresses this gap by formalizing perceived first-response time (PTFR) as a quality-of-experience (QoE) objective for the first time and minimizing it under explicit quality and cost constraints.

Prior work has generally targeted a single stage of the pipeline: cascading routing only decides which model to invoke, semantic caching only tracks whether a query is already cached, and adaptive retrieval only tunes the search strategy. PACE departs from this narrow focus by jointly controlling two dimensions at once—the composition of answer sources and the content used to fill waiting windows. This combined perspective fills a genuine gap in the literature and reframes latency optimization for conversation services.

The framework was deployed in a real humanoid robot sales service, giving it direct engineering relevance. Its design rests on three cooperating mechanisms, each mapped to a distinct stage of the latency pipeline, from routing decisions through content filling to cache management.

Deep Analysis

The first mechanism is a load-adaptive cascading router that dynamically selects its strategy based on current system load, preferring a lightweight model under low load and switching to a stronger model under high load to preserve quality. The second is a joint path-filling controller that simultaneously decides the answer generation path and the filler content used during the wait, forcing the two to cooperate rather than operate independently. The third is a volatility-aware cache admission policy that decides which entries may enter the cache based on how rapidly the underlying data changes, avoiding stale hits when information is updated frequently.

Experiments were run on 75,000 real CarQA requests across varying load conditions. The cascading mechanism nearly halved the pure-LLM PTFR at P95, dropping it from 0.53 seconds to 0.29 seconds under a c16 load. The adaptive controller reached 0.41 seconds at P95, surpassing traditional RAG by 2.4 times at equal answer quality under high load. The filling controller cut backend call volume by 94% while maintaining zero conflicts, meaning the vast majority of waiting windows were filled safely without contradicting the final answer.

The volatility-aware admission policy reduced the proportion of expired answers from 86% to 0%, effectively eliminating the problem of stale cached responses. Ablation studies confirmed that each mechanism contributes independently and that only their combined operation achieves optimal results. A gating rule acts as a safety valve, guaranteeing that no control decision drops quality below the baseline while limiting risk exposure to a single waiting cycle.

Industry Impact

On the industrial side, PACE provides a reusable engineering template for the first latency-optimization scheme deployed in a real humanoid robot sales service, and its joint-control view can extend to other dialogue systems that demand low-latency responses. The framework's most consequential methodological contribution is converting perceived latency from a fuzzy user-experience notion into a quantifiable, optimizable engineering target, reminding researchers that delay is not merely a technical metric but a core experience driver.

For the open-source community, mechanisms such as load-adaptive routing and volatility-aware cache admission carry high reuse value and can inform other retrieval-augmented systems. The paper's first quantitative analysis of conflict risk in filled answers opens a fresh research dimension, allowing later work to refine filling strategies with explicit risk accounting.

As humanoid robots and intelligent assistants are deployed at scale, delivering smooth conversation under high interaction frequency becomes critical. PACE's joint-control approach offers a concrete, evidence-backed model for that challenge, grounded in measured results rather than abstract promises.

Outlook

The gating rule's guarantee that controller quality never falls below the baseline, combined with its confinement of risk to a single waiting cycle, gives PACE the predictability that real deployments require. This balance between aggressive optimization and reliability is likely to matter as the framework is adapted beyond its original sales-service context.

Future research can build directly on the newly quantified conflict-risk dimension, developing finer-grained filling strategies that trade off latency savings against the small residual chance of contradiction. The 94% reduction in backend calls also suggests substantial cost savings that operators could reinvest in richer retrieval or larger models.

Ultimately, PACE demonstrates that treating delay as a first-class, jointly-optimizable objective—rather than an unavoidable side effect—can yield large QoE gains without sacrificing quality, a principle that should reshape how retrieval-augmented conversation services are engineered going forward.