LogicTrack: Auditing LLM Reasoning with Formal Logic Solvers

A new paper titled "LogicTrack: Auditing Reasoning Trajectories of Large Language Models with Formal Logic Solvers" targets a blind spot that benchmark culture has largely ignored: in chain-of-thought (CoT) reasoning, a model can land on the correct final answer even while some of the intermediate reasoning steps along the way are logically flawed.

In plain terms, the answer is right, but the reasoning that produced it wasn't. The authors, Jingyu Hu, Shu Yang, Weiru Liu, and Di Wang, propose LogicTrack, a framework that auto-formalizes each reasoning step into symbolic, formal-logic representations and then verifies each step using automated theorem provers — actual formal logic solvers, not another LLM acting as a judge.

Why "right answer, wrong reasoning" is a serious hidden failure mode

This gap matters because it undermines the very metric the field uses to claim progress on reasoning. If evaluation stops at whether the final answer matches a gold label, then a model that stumbles onto the right output through a lucky shortcut, a memorized fragment, or a single unjustified logical leap looks identical on a leaderboard to a model that derived the answer through a rigorous, step-by-step chain.

That means published benchmark accuracy numbers can systematically overstate real reasoning reliability — the score looks good, but the reasoning underneath it may not survive scrutiny. And it is precisely that underlying reasoning process that we want models to possess, and that we need to be able to trust in consequential settings.

Formal solvers instead of an LLM judging an LLM

The methodological choice at the heart of LogicTrack is to verify with genuine automated theorem provers rather than the increasingly common "LLM-as-judge" pattern, where a second large language model is asked to assess whether the first model's reasoning holds up. That choice is not cosmetic.

A formal logic solver is deterministic and operates on strict logical rules; it cannot hallucinate a plausible-sounding but wrong verdict. An LLM-as-judge, by contrast, is fundamentally one probabilistic system checking another — the judge itself can be mistaken, and it can be persuaded by text that reads fluently but is logically broken underneath. Auditing potentially unreliable reasoning with a tool that cannot lie is a meaningfully stronger verification method than auditing it with another tool that can.

From catching mistakes to a self-improving loop

LogicTrack also introduces a Solver-Based Backtracking Reward (SBR) for step-wise scoring, and — notably — it converts the backtracking traces themselves into supervised fine-tuning data. In other words, examples where a flawed step was caught by the formal solver and then corrected are systematically turned into training material for the next iteration of the model.

That closes a self-improving verification loop: verification stops being a one-time, after-the-fact gate and becomes an ongoing source of data that feeds back into training. Across testing on eight reasoning benchmarks and seven different LLMs, the paper reports that the method "effectively improves both the verifiability of reasoning chains and final answer pass rate, thereby enhancing overall CoT quality and trustworthiness in high-stakes domains."

Why this matters specifically for high-stakes domains

In domains like medicine, law, and finance, a chain of reasoning that looks correct but cannot be verified is a liability in its own right. Practitioners in those fields depend not just on a model's conclusion but on whether the reasoning behind that conclusion can be inspected, traced, and trusted.

A model that happens to land on the right answer while its intermediate logic is broken carries a hidden risk: the same flawed step, applied to a slightly different question next time, can produce a wrong conclusion — and because the surface-level answer looked fine before, standard accuracy-based evaluation would never have caught it. Work like LogicTrack matters because it turns "does this reasoning actually hold up" from a fuzzy intuition into something that can be checked with a formal tool, which is exactly the kind of infrastructure high-stakes deployment of these systems requires.