Can LLMs Deliberate? A Diagnostic Framework for Belief Updating

Background and Context

A new arXiv paper confronts a question that is frequently overlooked yet consequential for how large language models are deployed as proxies for public opinion. The authors ask whether LLM personas can update their beliefs deliberatively when confronted with new information, or whether they merely retrieve cached positions that were effectively fixed at training time. In the emerging paradigm of silicon-based sampling, researchers deploy LLM personas at scale to simulate public opinion, and the validity of that work depends entirely on the assumption that personas change their views the way real people do during deliberation.

The authors argue that existing evaluations fail to test this assumption. Nearly all current checks verify only whether a persona holds the correct view, which the paper characterizes as a static snapshot. Such a snapshot cannot capture the dynamic fidelity of deliberative belief change, meaning a model could produce plausible partisan positions while systematically distorting the way those positions shift as information arrives.

The paper's central contribution is to shift the research focus from whether views are correct to whether the process of belief evolution resembles human deliberation. This reframing opens an entirely new dimension for assessing the dynamic authenticity of LLM personas, moving evaluation beyond a single point-in-time correctness check.

Deep Analysis

The framework is grounded in the deliberative voting tradition from deliberative democracy theory, in which participants are asked to reconsider their positions after being exposed to arguments from multiple sides, with the aim of raising opinion quality. The protocol designs a set of neutral, balanced information interventions so that humans and models reassess their stances under fully symmetric conditions, after which belief displacement is quantified. The underlying data comes from America in One Room, a real deliberative experiment covering 526 personas and 72 questions, which gives the intervention情境 real-world thickness.

Rather than stopping at descriptive comparison, the authors run targeted ablation experiments to locate the causes of failure. They isolate policy content as the triggering factor and test whether changes vary by identity. From this they extract the core concept of self-sycophancy: the model's tendency to submit to its own internal stereotypes about a persona rather than reasoning from the information actually provided. This design lets researchers distinguish three distinct failure modes—directional errors, magnitude errors, and rigidity.

Industry Impact

Applying the framework to five frontier models, the authors find that each fails, and each fails differently. GPT-5.1 exhibits reversal: after receiving balanced information, its personas become more hostile toward the opposing party while human hostility clearly decreases. This reversal is selective, reaching 80 percent on opposing-party questions but only 26 percent on policy questions, and it appears symmetric across party identities.

Gemini 2.0 Flash, Claude Sonnet 4.5, and Llama 3.3 70B over-shift, moving in the correct direction yet by a magnitude five to seven times beyond the human baseline. DeepSeek V3 is nearly rigid, with belief changes close to zero. Ablation experiments show that policy content is the key trigger and that failures are identity-specific: GPT-5.1 reverses on opposing-party questions while over-shifting on its own party's questions, while Gemini displays the opposite pattern. The failures are therefore structured and coupled to identity and issue, not random noise.

The stakes extend well beyond academic critique. As LLM personas are increasingly used for public-opinion simulation, policy forecasting, and political communication research, their dynamic fidelity directly determines whether conclusions are trustworthy. The authors offer a actionable protocol: run this deliberative diagnostic before trusting that a persona has revised its beliefs. For the open-source community the framework yields a fair capability profile revealing each model's unique failure fingerprint; for industry it reminds practitioners to fold dynamic behavior into model selection and risk assessment; and for subsequent research it establishes a new paradigm shifting from static accuracy to dynamic deliberative fidelity.

Outlook

Self-sycophancy is the concept most worth watching, because a model's submission to persona stereotypes may be harder to detect than an obvious error and easier to quietly distort our understanding of public opinion at scale. The framework's move toward mechanism diagnosis rather than result validation suggests future evaluation will increasingly probe how beliefs evolve rather than merely where they land.

The selective and identity-coupled nature of the failures points toward more granular diagnostics that can attribute specific distortions to particular issue types and party alignments. If the field adopts the proposed protocol, model developers will face pressure to reduce dynamic fidelity gaps, and researchers relying on silicon sampling will need to validate belief change before drawing conclusions. Whether these diagnostics become standard practice will determine how much confidence the broader community can place in LLM-simulated public opinion.