Reasoning Models Fine-Tuned with Verifier for Thermal Energy Storage Control

Background and Context

The integration of building infrastructure with the electrical grid has emerged as a critical strategy for enhancing grid stability, particularly through the shifting of cooling loads in response to real-time grid conditions. At the heart of this architectural energy flexibility lies Thermal Energy Storage (TES), a technology that allows buildings to store cooling capacity during off-peak hours for use during peak demand. However, optimizing TES systems remains a formidable computational challenge. The core difficulty stems from the need to schedule energy storage and retrieval hours in advance, subject to complex physical constraints such as thermal losses, storage capacity limits, and fluctuating weather conditions. Traditional control methodologies, specifically Model Predictive Control (MPC) and conventional Reinforcement Learning (RL), have struggled to scale effectively across diverse building types. MPC often requires precise, physics-based models that are expensive to calibrate for each individual building, while standard RL methods frequently suffer from sample inefficiency and instability when applied to high-dimensional, continuous control spaces without extensive environment simulation.

To address these scalability and efficiency bottlenecks, recent research has introduced a novel framework leveraging Reinforcement Learning with Verifiable Rewards (RLVR) to fine-tune open-source reasoning models. This approach diverges significantly from previous methods that relied on massive datasets or high-fidelity digital twins of building environments. Instead, the study utilizes the mathematical certainty of Dynamic Programming (DP) as a ground-truth source. By converting the precise action values calculated by DP into dense reward signals, the research team created a learning environment where the model receives immediate, verifiable feedback on its decisions. This method bypasses the need for costly environmental simulations, allowing the model to learn directly from text-based states and weather forecasts. The ultimate goal is to transform general-purpose open-source reasoning models into specialized, high-level schedulers capable of outputting hourly heat pump setpoints with near-optimal efficiency.

Deep Analysis

The technical innovation of this study centers on the transformation of discrete action evaluations into continuous learning signals through a rigorous RLVR mechanism. The process begins with an offline Dynamic Programming calculation, which determines the exact optimal value for every candidate action within the defined state space. These precise values are then mapped to dense reward signals that guide the Reinforcement Fine-Tuning (RFT) process. Remarkably, this sophisticated control logic was achieved using only 30 carefully designed training prompts. This minimal data requirement drastically lowers the barrier to entry for deploying advanced control strategies, contrasting sharply with traditional deep reinforcement learning approaches that often require millions of interaction steps. The model is trained to interpret textual representations of building states and meteorological predictions, outputting specific control commands for heat pumps. Unlike black-box end-to-end models, reasoning models possess inherent logical capabilities that, when stabilized by RLVR, allow them to evaluate the long-term consequences of their actions rather than merely reacting to immediate rewards.

The experimental validation was conducted on a logically rigorous benchmark representing a simple office building with TES, a setup chosen because it allows for the calculation of global optimal solutions via Dynamic Programming. This objective baseline enabled precise performance measurement. The results demonstrated that the fine-tuned open-source reasoning model reduced carbon emissions from a baseline of 70.5 kg-CO2 down to 61.2 kg-CO2. This figure is exceptionally close to the theoretical DP optimum of 60.8 kg-CO2, validating the efficacy of the RLVR approach. In comparative analysis, non-reasoning models such as GPT-4o performed poorly, even generating emissions higher than a baseline with no storage capability. This highlights the critical importance of reasoning capabilities in solving complex planning problems. Conversely, advanced models like GPT-5 approached optimality without specific training, suggesting that while reasoning models have strong zero-shot potential, fine-tuning with verifiable rewards is essential for open-source models to reach competitive performance levels.

Further trajectory analysis revealed that the RFT process did not invent entirely new control strategies but rather stabilized existing planning modes inherent to the reasoning models. These stabilized modes include candidate comparison, lookahead reasoning, and feasibility checks. The model learned to systematically evaluate potential actions against future states, ensuring that its decisions were robust and logically sound. Additionally, robustness tests indicated that these reinforced planning modes remained effective even when faced with prediction errors and unseen storage conditions. While the study noted that the benefits could be partially transferred to battery storage tasks, the gains were limited due to structural differences between thermal and electrochemical storage systems. This finding underscores the specificity of the learned policies while affirming the generalizability of the reasoning framework across different energy storage modalities.

Industry Impact

This research establishes a new paradigm for deploying open-source large language models in vertical industrial applications, specifically within building energy management systems (BEMS). By demonstrating that a general-purpose reasoning model can be transformed into a domain-specific expert scheduler through minimal, high-quality fine-tuning, the study offers a scalable alternative to training bespoke small models for every unique building type. For the open-source community, this implies that powerful reasoning capabilities can be leveraged to solve complex physical constraint optimization problems without the need for extensive proprietary data or massive computational resources for pre-training. The ability to adapt the model to different building types and storage conditions significantly reduces the development and maintenance costs associated with traditional BEMS solutions, which often require manual calibration and physics-based modeling for each installation.

The implications for urban-level energy management are substantial. As cities strive for greater energy flexibility and carbon neutrality, the ability to efficiently coordinate thousands of buildings with TES systems becomes crucial. The RLVR framework provides a practical pathway for aggregating these distributed energy resources. By enabling open-source models to act as intelligent schedulers, building operators can achieve near-optimal energy usage and carbon reduction without relying on expensive, closed-source AI solutions. This democratization of advanced control algorithms could lead to widespread adoption of TES technologies, thereby enhancing the resilience of the electrical grid and facilitating the integration of renewable energy sources. The study’s emphasis on verifiable rewards ensures that the decisions made by these models are grounded in mathematical certainty, a critical requirement for industrial deployment where safety and efficiency are paramount.

Furthermore, the success of this approach challenges the prevailing notion that complex physical control problems require complex, data-hungry AI models. By proving that a small number of high-quality prompts can yield significant performance gains, the research encourages a shift towards more efficient and interpretable AI development in the energy sector. This efficiency is particularly valuable in the context of rapid deployment and continuous adaptation to changing grid conditions. The study also highlights the potential for cross-domain transfer, suggesting that the reasoning patterns learned in thermal storage could be adapted to other forms of energy storage with appropriate adjustments. This versatility positions reasoning models as a flexible tool for future energy management systems, capable of evolving alongside the increasing complexity of urban energy infrastructures.

Outlook

The immediate future for this technology lies in expanding the scope of validation from simple office building benchmarks to higher-fidelity, whole-building control tests. While the current results are promising, real-world buildings exhibit far more complex dynamics, including varying occupancy patterns, diverse HVAC configurations, and unpredictable internal heat gains. Future research must focus on adapting the RLVR framework to handle these higher levels of complexity and uncertainty. This will likely involve developing more sophisticated verifiers that can provide dense reward signals even in the absence of a global optimal solution, perhaps by leveraging surrogate models or hierarchical verification structures. Additionally, the development of scalable urban-level energy management verifiers will be critical. As the number of connected buildings increases, the ability to verify the collective impact of individual scheduling decisions on the broader grid will become a key area of innovation.

Another critical direction for subsequent studies is the exploration of multi-modal inputs and real-time adaptive control. While the current study relies on text-based states and weather forecasts, integrating real-time sensor data, electricity pricing signals, and grid frequency information could further enhance the model’s responsiveness and efficiency. The ability of the model to adapt its reasoning process in real-time based on live data streams will be essential for practical deployment. Moreover, investigating the transferability of the RLVR framework to other domains beyond energy storage, such as industrial process control or autonomous vehicle navigation, could unlock broader applications for reasoning models in physical system management.

Finally, the intersection of open-source AI and sustainable energy infrastructure presents a unique opportunity for collaborative innovation. By providing a clear, reproducible methodology for fine-tuning reasoning models with verifiable rewards, this study invites the broader research and engineering communities to build upon its foundations. The ultimate goal is to create a robust, scalable, and cost-effective ecosystem where open-source AI models serve as the brain behind intelligent, energy-efficient buildings. As the pressure to decarbonize urban environments intensifies, the ability to optimize energy use at the building level through advanced reasoning models will play a pivotal role in achieving global climate targets. The trajectory set by this research suggests a future where AI-driven energy management is not only a luxury for large commercial entities but a standard feature of sustainable urban living, powered by accessible and transparent open-source technologies.