Google Walks Through the Agents CLI Eval Loop and Warns That Summary Scores Can Mislead

Google's developer blog published a walkthrough on October 28, 2026 of the evaluation commands in Agents CLI, a tool for building AI agents. The post uses a small demo agent to show a loop of running test cases, grading the results, comparing runs and acting on the difference. Its central warning is that a summary score can mislead, so teams should read per-case results. The score comparisons in the walkthrough come from a four-case demo, not a benchmark of real products; the post separately cites a Google study.

The eval namespace has five commands, and each does one job. The first, eval generate, runs the agent against a dataset of cases. The second, eval grade, scores the generated traces. The third, eval compare, diffs two result files side by side. The fourth, eval analyze, clusters failures into themes. The fifth, eval optimize, runs an automated pass that improves the agent's prompt. The authors call the whole routine hill-climbing: change the agent, rerun the cases, keep the change if the score rises and revert it if the score falls.

The demo uses a LangChain template that ships with Agents CLI v1.8.0. A custom judge metric, driven by gemini-3.8-flash, grades each answer from 1 to 5 for accuracy, relevance and clarity. The authors expand the default one-case dataset to four cases: a greeting, two weather questions and an off-topic question. The baseline mean is 4.0 out of 5 with a standard deviation of 1.41. The authors read the spread as healthy, because a flat 5.0 would suggest either a perfect agent or a dataset too easy to show differences.

Next they plant a realistic mistake: a developer pastes a chef persona into the weather agent's system prompt. Lint, unit tests and type checks pass, and the agent still runs. The mean falls to 2.75. Per-case comparison shows where the damage landed. The greeting lost a point on tone, the off-topic case lost two points on scope creep, and the sf_weather case lost two points because the agent stopped calling its weather tool and made up values. The authors' point is that only an end-to-end eval that scores behavior in context catches this kind of fault.

The second experiment swaps the agent's model from gemini-3.8-flash to gemini-2.5-flash. As printed in the post, the mean again goes from 4.00 to 2.75, but the per-case view differs. The cheaper model matched on the greeting and one weather case, lost a point on the other, and the visible drop traces to the off-topic case, where it refused with a hedge that the event had not happened yet. The authors warn that a prompt tuned for one model may lose capability on another, and that the judge model must stay fixed so the comparison is fair.

The post also covers scheduling and limits. Agents CLI does not ship a scheduler, so teams can run generate and grade nightly from cron or Cloud Scheduler and add a wrapper that exits non-zero when the mean drops past a threshold. Eval optimize works for ADK projects through the GEPA framework, not for the LangChain extension. It changes only the prompt, and it can raise a score by overfitting to the metric's wording, so a human reviews the candidates. Google also reports its own controlled study in which coding agents using the tool ran four times as many evals but changed the agent after an eval about as often as the baseline. That is Google's claim, with no data in the post.

The practical lesson is modest. Keep the daily dataset small enough to run on every change, save the full result files and not only the scores, and read the per-case diff before trusting a mean. A tool cannot supply the judgment about which cases matter, which the team still has to provide.