Applying Verifier-Augmented Fine-Tuned Reasoning Models to Thermal Energy Storage Control

Background and Context

The modernization of urban energy infrastructure demands a fundamental shift in how buildings interact with the electrical grid. As renewable energy penetration increases, grid operators require greater flexibility to balance supply and demand, necessitating that commercial buildings adjust their cooling loads in response to real-time grid conditions. Thermal Energy Storage (TES) provides the physical mechanism for this load shifting, allowing facilities to store cooling capacity during off-peak hours and discharge it during peak periods. However, optimizing TES systems is not merely a matter of storage capacity; it is a complex, multi-variable optimization problem. The core challenge lies in scheduling thermal storage over horizons of several hours in advance, subject to strict physical constraints, fluctuating ambient temperatures, and dynamic electricity pricing. Traditional control methods, such as Model Predictive Control (MPC) and standard Reinforcement Learning (RL), have demonstrated efficacy in isolated settings. Yet, they face significant scalability hurdles when deployed across diverse building typologies. MPC often requires precise, building-specific physical models that are costly to develop and maintain, while standard RL agents frequently suffer from poor generalization and sample inefficiency when transferred between different environments. This limitation has created a bottleneck in the widespread adoption of intelligent energy management systems, leaving a gap for a more adaptable and scalable control paradigm.

To address these scalability challenges, recent research has introduced a novel approach leveraging Verifier-Augmented Fine-Tuned Reasoning Models, specifically utilizing Reinforcement Learning with Verifiable Rewards (RLVR). This method represents a paradigm shift from traditional black-box optimization to interpretable, logic-driven control. The central innovation involves adapting open-source reasoning models to act as high-level schedulers for TES systems. Instead of relying on complex neural network architectures to directly output control signals, this framework utilizes the logical reasoning capabilities of large language models (LLMs). The system is designed to process textual representations of building states and energy forecasts, then output precise heat pump setpoints. By framing the control problem as a reasoning task, the approach aims to harness the emerging ability of advanced models to plan, compare options, and verify feasibility, thereby offering a solution that is both robust and easily transferable across different building types without the need for extensive retraining or physical modeling.

Deep Analysis

The technical architecture of this RLVR approach is built upon a closed-loop framework that bridges the gap between discrete logical reasoning and continuous control actions. The core mechanism involves an offline dynamic programming (DP) solver that acts as a verifier. This solver calculates the exact action values for every possible control decision within a given state and forecast scenario. These precise values are then transformed into dense reward signals, which are fed back to the reasoning model during the fine-tuning process. This transformation is critical because it solves the sparse reward problem inherent in many RL applications, providing the model with immediate, clear feedback on the quality of its decisions. The fine-tuning process itself is remarkably efficient, requiring only 30 carefully designed training prompts. Through Reinforcement Fine-Tuning (RFT), the open-source reasoning model is converted into a sophisticated scheduler that interprets textual inputs regarding building status and energy predictions, and outputs specific heat pump setpoints. This design leverages the model's inherent pattern recognition and logical deduction skills, effectively turning a general-purpose language model into a specialized control agent.

The efficacy of this method was rigorously tested using a simplified office building benchmark environment, specifically chosen because it allows for the calculation of known optimal solutions via dynamic programming. This setup provided a ground truth against which the performance of the fine-tuned model could be measured. The results were significant: the RLVR-fine-tuned model reduced carbon emissions from an initial baseline of 70.5 kg-CO2 down to 61.2 kg-CO2. This figure is remarkably close to the theoretical DP optimum of 60.8 kg-CO2, demonstrating that the reasoning model can achieve near-optimal performance with minimal training data. The trajectory analysis further elucidated how this performance was achieved. Rather than inventing new control strategies, the RLVR process stabilized existing planning behaviors within the model. These behaviors include candidate comparison, where the model evaluates multiple potential actions; look-ahead, where it simulates future states; and feasibility checking, where it ensures constraints are met. By reinforcing these specific logical patterns, the model became more consistent and reliable in its decision-making, particularly when facing uncertainties in load forecasts or environmental conditions.

A crucial component of the study was the comparative analysis of different model architectures to isolate the impact of reasoning capabilities. The researchers tested both open-source models and proprietary closed-source models, including GPT-4o and GPT-5. The results highlighted a stark divergence in performance based on the model's inherent reasoning abilities. GPT-5, a model with advanced reasoning capabilities, performed exceptionally well even without any specific task training, approaching the performance levels of both the DP solver and traditional MPC. In contrast, GPT-4o, which lacks the same depth of reasoning architecture, performed poorly, producing carbon emissions that were actually higher than a baseline scenario with no storage facility at all. This comparison underscores a critical insight: for complex scheduling tasks that require long-horizon planning and strict constraint satisfaction, raw language modeling proficiency is insufficient. The ability to reason through logical steps, verify constraints, and plan ahead is the decisive factor in achieving optimal control. This finding suggests that the value of LLMs in industrial applications lies not in their ability to predict the next token, but in their capacity for structured, verifiable thought processes.

Industry Impact

The implications of this research extend far beyond academic benchmarks, offering a practical pathway for the industrial deployment of intelligent energy management systems. One of the most significant contributions is the demonstration that open-source reasoning models can be effectively adapted for complex control tasks using a verifier-based approach. This reduces the industry's reliance on expensive proprietary models and vast amounts of labeled training data. By using a dynamic programming solver as a verifier, the framework provides a robust and efficient method for fine-tuning models, lowering the barrier to entry for developers and energy service providers. This democratization of advanced control technologies could accelerate the adoption of TES systems in commercial real estate, where cost sensitivity and operational complexity have historically hindered innovation. Furthermore, the ability to use textual inputs and outputs aligns well with existing building management systems that increasingly utilize natural language interfaces, potentially simplifying integration and user interaction.

The study also highlighted the robustness and generalizability of the RLVR approach, which are critical factors for real-world deployment. Testing revealed that the reinforced planning patterns remained effective even when faced with prediction errors and unseen TES conditions. This resilience suggests that the models are not merely memorizing specific scenarios but have learned fundamental principles of energy scheduling. Moreover, the approach showed promise in transferring to other energy storage technologies, such as battery storage, although the gains were limited due to structural differences in the underlying physics. This cross-domain adaptability indicates that the core reasoning framework is versatile and can be applied to a wider range of energy management challenges. As cities move towards more integrated and complex energy grids, the ability to manage diverse storage assets with a unified, reasoning-based control layer becomes increasingly valuable. The research thus provides a scalable solution that can be adapted to various storage types, enhancing the flexibility of the overall energy system.

Additionally, this work has sparked interest in developing higher-fidelity control tests for entire buildings and推动了 the development of scalable verifiers for city-scale energy management. The current benchmark, while useful, is a simplified representation of a real office building. The next logical step is to apply this RLVR framework to full-scale, high-fidelity building simulations that account for more complex thermal dynamics, occupancy patterns, and grid interactions. The success of the verifier-augmented approach in a controlled environment provides a strong foundation for these more ambitious projects. It suggests that as urban energy systems become more decentralized and dynamic, the role of reasoning models in orchestrating these systems will grow. By providing a method to ensure that these models adhere to physical constraints and optimize for global objectives, the research addresses one of the primary concerns in deploying AI in critical infrastructure: safety and reliability. This paves the way for a new generation of energy management systems that are not only intelligent but also trustworthy and verifiable.

Outlook

Looking forward, the integration of RLVR techniques into building energy management represents a significant step towards more autonomous and efficient urban infrastructure. The immediate future of this technology lies in expanding the scope of verification and control. While the current study focused on thermal energy storage in office buildings, the underlying principles of using verifiable rewards to guide reasoning models can be applied to a broader array of control problems. This includes hybrid systems that combine thermal storage with battery storage and renewable energy generation. As the complexity of these systems increases, the ability of reasoning models to handle multiple constraints and long-horizon planning becomes even more critical. Researchers are likely to focus on developing more sophisticated verifiers that can handle the non-linear dynamics of real-world buildings, ensuring that the models remain accurate and reliable under a wider range of conditions.

Another key area of development will be the enhancement of the models' ability to generalize across different building typologies and climates. The current study demonstrated success in a specific office building benchmark, but real-world deployment requires models that can adapt to residential, industrial, and mixed-use buildings with varying thermal masses and operational schedules. Future research may explore meta-learning techniques or transfer learning strategies that allow the RLVR framework to adapt quickly to new environments with minimal additional training. This would significantly reduce the deployment time and cost, making the technology more attractive to building owners and operators. Additionally, the integration of real-time data streams from IoT sensors and smart meters will further enhance the precision of the control decisions, allowing for dynamic adjustments that respond to immediate changes in occupancy or weather conditions.

Finally, the broader impact of this research lies in its potential to contribute to city-scale energy optimization. As urban areas strive for net-zero emissions, the coordination of energy use across thousands of buildings will be essential. The RLVR approach offers a scalable framework for managing these distributed resources, enabling a more resilient and flexible grid. By treating each building as an intelligent agent capable of reasoning about its own energy needs and the broader grid context, cities can achieve greater efficiency and stability. The success of this method in reducing carbon emissions and approaching optimal control values provides a compelling case for its adoption. As the technology matures, it is likely to become a standard tool in the energy manager's toolkit, driving a new era of intelligent, sustainable, and responsive building energy systems. The journey from theoretical benchmark to widespread industrial application is underway, and the foundational work presented here provides a clear and promising path forward.