Inflection Finds Synthetic Survey Answers Differed by About 22 Points From Real Users in Its Study

On October 30, 2026, Inflection AI published a comparison of synthetic survey respondents with answers from 1,003 real U.S. AI chatbot users. It generated six synthetic populations using Claude, ChatGPT and Gemini under two survey approaches and reported an average difference of about 22 percentage points from real answers on the selected questions; even the best synthetic population was nearly 19 points away. These are findings for the study's sample and questions, not a universal error rate for every synthetic survey, and this site's November 2 issue date does not change the original research date.

A synthetic respondent is a model-generated stand-in asked to simulate a person's demographics, habits or opinions. Such populations are attractive because they can answer quickly and cheaply, allowing researchers to test wording or explore hypotheses before recruiting people. Inflection's exercise shows why a convincing sentence should not be mistaken for a representative distribution of human behavior. The 1,003 real participants were U.S. chatbot users, not an automatically representative sample of every U.S. resident. The three models and two approaches also produced six different instruments rather than a single stable category called "synthetic users." A team changing a model, prompt or conversational setup might obtain a different market picture while leaving the nominal questions unchanged.

The sharpest examples concerned ordinary activities that may be underrepresented in online descriptions. In the previous week, 44% of real respondents said they had used wired headphones, compared with 6% of synthetic respondents in the reported comparison. For listening to music with other people, the figures were 45% and 2%; for listening to a podcast with others, 16% and approximately zero. Across the behaviors examined, Inflection says the average error was about six points for well-documented activities and 33 points for activities rarely captured online. Those two numbers summarize selected groups of behavior in this study. They are not fixed correction factors that can be applied to any future questionnaire.

A second gap involved why people stopped using a chatbot. Among real respondents, 9.9% cited values-based objections, including concerns about a company's conduct, privacy, political bias or environmental impact. Across the six synthetic populations, the corresponding share was 1.1%. For a product team, this matters because a model may produce calm, generic and plausible reasons while missing pointed criticism or distrust of a particular company. A synthetic panel used as a direct substitute for customers could therefore understate reputational problems. The study does not establish that every model fails equally on every values question; it gives a reason to test the specific issue with real people.

The models also disagreed about the chatbot market itself. Real respondents reported meaningful use of 13 different chatbots; one synthetic setup identified 11, while others identified as few as four. Keeping a simulated respondent's identity consistent across multiple questions improved some aspects of its answers, according to the report, but did not reliably improve overall accuracy. A longer conversation is therefore not, by itself, validation. Researchers should record the model version, prompt, demographic profile, question order, sampling procedure and scoring rule. They should inspect per-question errors and disagreement between synthetic populations instead of selecting whichever output best supports an existing narrative.

There is still a bounded role for synthetic research. Teams can use it to challenge survey wording, enumerate possible hypotheses and find areas worth deeper investigation. Before making a major product, pricing or safety decision, they should validate the relevant claims through human interviews or a survey with a transparent sampling design. The need is especially strong for emerging habits, marginalized groups, values conflicts and brand trust, where existing web text may be thin or skewed. A useful validation report would show real and synthetic sample sizes, absolute percentage-point gaps, uncertainty estimates and the model-to-model spread, not just a single average. Predefining an acceptable error and a stopping condition also limits the temptation to choose favorable results after seeing the answers.

The study's contribution is to separate responses that sound human from estimates that reproduce a measured human sample. Synthetic populations may assist early exploration, but they did not replace real respondents in these tested comparisons; any claim of improved accuracy should be supported by a same-question, same-sample comparison.