Selected Applications of Reinforcement Learning in Electricity Markets: Strategic Bidding and Bidding Objective Identification

Renewable energy and carbon reduction policies are creating new challenges for electricity markets. To achieve carbon-free goals, large-scale battery energy storage systems (BESSs) are necessary to ensure grid reliability and flexibility. The impact of BESSs on market and grid operation, as well as the optimal portfolio across the energy and ancillary services markets, must be analyzed to guide their operation. At the same time, the expansion of renewable and storage resources and the adoption of carbon reduction policies have introduced new complexities to the bidding behavior of market participants, which cannot be easily described by cost-based bidding objectives. In response to these challenges, this dissertation aims to achieve two research objectives: (I) enable BESS participation in energy and ancillary services markets under uncertainties, considering the battery's degradation cost; (II) identify robust bidding objectives for electricity market participants based on their historical bidding data. Three optimization frameworks are proposed in Part I to model a BESS as a price-maker in energy markets, evaluating its impact on market outcomes. The preliminary framework models automatic generation control signals, while the detailed framework proposes a participation factor for dispatching AGC signals and accounts for battery degradation costs. The stochastic framework models spinning reserve deployment with uncertainty and propose an optimization-based approximation method based on reinforcement learning. Case studies on proposed frameworks validate operational models for Battery Energy Storage Systems (BESS) and markets, showing the accuracy and efficiency of the approximation approach. Key findings include that accurate degradation cost modeling is essential, and participation in ancillary services markets is more profitable. Part II proposes a data-driven approach using Adversarial Inverse Reinforcement Learning to identify robust bidding objectives for electricity market participants. It introduces a tailored reinforcement learning model for bidding objective identification without data discretization, and a special policy structure compliant with multi-segment bidding rules. Two approaches are suggested for electricity market environment modeling in RL/IRL problems, ensuring the robustness of the identified bidding objective. Three case studies validated the accuracy and robustness of the proposed bidding objective identification method in various application scenarios.