jevals: agent evals with typed verdicts, not an LLM judge

What the jevals README describes

jevals is an open-source Python library in the openlayer-ai GitHub organization. Its README calls it "evals and guardrails for agents" that use Jev-style decision models instead of an LLM judge. The main claim is about cost and speed. All the evals for one agent trace go out as a single request. The README says that request costs a few thousandths of a cent and comes back in a few hundred milliseconds. That is fast enough, it says, to run on every trace and inside the agent loop.

This article follows the README only. We did not install or run the tool. Every number below is the author's own claim, and we point out where the README itself marks a figure as an estimate or an illustration.

Key facts from the README

The model idea. The README says Jev does not generate text. You send it some state and a set of typed questions: yes/no, pick one of several options, or score on a rubric. It returns a calibrated probability for each question in a single forward pass. The questions are evaluated independently and in parallel, so asking 40 costs about the same latency as asking one. The README gives the price as $0.042 per million input tokens, with no output tokens, and reports p50 244 ms and p95 371 ms per request through Vercel's gateway. The quickstart. One call to `evaluate()` takes the messages and the tool schemas, plus a list of eval objects such as `ToolChoice`, `UsedToolResult`, `Grounded`, `StayedInScope`, `AnswerRelevancy`, `Completeness`, `IndirectInjection` and `PHI`. The README reports that this ran as one HTTP request for eight evals: 1,388 tokens, $0.00006 and 0.33 seconds. Backends. The library picks a backend from environment variables. The options are Jev directly through `TYPESAFE_API_KEY` (which needs waitlist access today, the README says), Jev through Vercel AI Gateway, a self-hosted Kev, Laya running in-process on Apple Silicon, or a regular chat LLM through OpenRouter that is emulated with a prompt asking for probabilities. The README warns that you should re-run `jevals calibrate` after you swap backends, because probabilities do not line up across models.

The catalog. The README lists agent evals (for example `ToolChoice`, `Grounded`, `StayedInScope`, `LoopDetection`, `GoalCompletion` and `ToolCallRisk`), security evals (`PromptInjection`, `IndirectInjection`, `Jailbreak`, `PII`, `PHI`, `SecretsExposure` and more) and Ragas-style quality metrics (`Faithfulness`, `AnswerRelevancy`, `ContextPrecision`, `ContextRecall` and more). It counts 37 evals in total. The benchmark table. The README compares jevals with Ragas on the same 20 rows of a small RAG dataset, measured on 2026-09-20. Ragas with gpt-4.1-mini takes 6.0 LLM requests per sample plus embeddings, about $2.60 per 1k samples, and 22 to 35 seconds for the 20 samples. jevals with Jev takes 1.0 request per sample, about $0.03 per 1k samples and 0.8 seconds. jevals with gpt-4.1-mini emulating Jev takes 1.0 request, $0.46 per 1k samples and 4 seconds. The Kev-4B and Laya rows are labelled estimates. Faithfulness came out at 0.90 to 0.92 in all three measured rows, and context precision and recall at 1.0.

How evals and gates work

An eval is a class with three methods. `state()` picks what the model should look at. `questions()` says what to ask about it. `reduce()` turns the returned probabilities into a score. When you pass several evals to `evaluate()`, the README says their states are merged and their questions are packed into one request. Because an eval depends only on the sample, the same class can serve as an offline metric, as a monitor on production traces, or as a gate inside the agent.

A gate is an eval plus a policy that maps the answers to allow, escalate or block. The README shows a YAML gate for tool-call risk. Its policy allows a call if `action.approve >= 0.85` and `grounded >= 0.7`, blocks it if `action.block >= 0.6`, and escalates otherwise. The README shows wiring for the OpenAI Agents SDK, for a plain async loop, for LangGraph and for a Claude Agent SDK `PreToolUse` hook. If the backend is still down after retries, a gate lets the call through by default. Passing `on_error="block"` reverses that for anything irreversible.

For PII and PHI the README describes two steps. Entity detection comes first. Then one question goes to the model: is this health information about an identifiable person, or is it a support email address? The README says entity detection alone cannot tell those apart, and that this is the source of most PII false positives.

Background and our reading

*This section is our analysis, not the README's.* An LLM judge means that you ask a general chat model to grade an output, usually through a prompt that asks for reasoning and a JSON answer. The README argues that most of what a judge decides fits three question types, and that the thing you keep from the judge is a label anyway. If a small model can return that label with a probability in one pass, you can afford to run it far more often. That changes what evals are for. A nightly sample becomes a check on every trace, and a check on every trace can move into the request path.

The README also raises variance. It cites a LangChain comparison in which GPT and Claude judges showed 92x to 913x the score variance of Jev on identical traces. We have not read that post. A probability, unlike a paragraph of text, can be thresholded, so you can set the trade-off per action. The `jevals calibrate` example in the README lists thresholds against wrong passes and missed passes, and the README advises thresholds per action, since a refund and a lookup should not share one.

The invented claim in the README's own example shows why several metrics help. The agent said it would "stay sunny all week" when the tool returned only today's forecast. The `grounded` eval gave that sentence p=0.05, while `answer_relevancy` stayed at 0.84. The README concludes that RAG-style metrics alone would have passed the trace.

Limits and open questions

The README is open about its limits. It says the project is alpha and a week old, and that the models underneath are a week old as well. The TypeSafe direct backend is written to the documented wire format and tested against a mock, not yet run live. The framework adapters are written to the SDK docs and tested against fakes. The Kev and Laya rows in the table are estimates, multiplied out from numbers that the authors of those models publish. Several output blocks are labelled "Illustrative output". The README also says the tool does not generate test sets, has no dashboard, and will not replace an LLM judge for work that needs multi-step reasoning or a written critique.

On accuracy, the README cites JevBench, an independent benchmark, which puts Jev around the accuracy of the smallest LLMs on classification tasks (83 to 87% on Banking77 and CLINC150) and finds that calibration varies by task. The LangChain result of 100% agreement with a human across 500 repetitions, against 80% for Claude, was on five traces, the README notes. It also reports that the gateway hung on some connections during one run, so p95 for that run was a minute, although the client retried and the run finished. We noticed small inconsistencies in the README. The intro shows eight evals at 1,388 tokens and 0.33 seconds, while the longer example shows nine at 1,423 tokens and 0.50 seconds. The text says "six hundredths of a cent" for a printed cost of $0.00006, which is six thousandths of a cent. These look like wording slips from different runs, not deep problems, but they show why you should re-measure. Our reading: the savings depend on your questions fitting the three types and on your data resembling the small sample used for the table. The README itself gives the sharpest caution: do not let the classifier become the authorizer. Whether a refund should run depends on account state and permissions that the model cannot see.

Practical takeaways

1. If you already run an LLM judge on a sample of traffic, the README's LLM-emulation backend is the lowest-friction way to try the API. The README says it works today but returns probabilities of 0.00 or 1.00 instead of calibrated values.

2. Label some of your own traces and run `jevals calibrate` before you trust a threshold.

The README says calibration data is the contribution that would help most.

3. Put a human on irreversible actions, and set `on_error="block"` on gates in front of them.

4. Follow the README's writing tips: one question per thing, describe the options instead of just naming them, keep the state small, and add an "insufficient evidence" option when the state might not hold the answer.

5. Treat the framework adapters as rough edges until someone runs them live, and treat the cost table as an author claim. Repeat the comparison on your own traces before you plan around it.