In recent years, the adoption of Distributed Energy Resources (DERs) in power systems has been increasing, driven by technological advancements, development of monitoring and control techniques, policy guidance among various countries, and the benefits DERs bring to the power system. These benefits include low-cost energy production, environmental sustainability promotion, and enhanced operational efficiency of the power system. For instance, demand response (DR) can alleviate pressure during peak load periods, while solar PV units and wind turbines with smart inverters can improve grid reliability through grid regulation based on IEEE Standard 1547. Despite the opportunities DERs present, their adoption also poses challenges. The growing reliance on renewable sources introduces uncertainty, variability, and intermittency, directly impacting system stability and efficiency. Addressing these challenges necessitates comprehensive research to enhance stability, improve system operations, and maximize resource utilization. This dissertation concentrates on two primary research areas: analyzing prosumer (consumers and producers, as one) consumption behavior and developing AC optimal power flow (ACOPF) models. Firstly, understanding prosumer consumption behavior is important for reducing DERs' uncertainty, particularly DR programs. This study employs a proposed probabilistic algorithm to analyze the causal relationships between prosumer consumption behavior and other factors. Two causal-oriented approaches are utilized to establish accurate prediction models and assess demand flexibility. Causal artificial intelligence facilitates intervention and counterfactual analyses of prosumers’ DR participation and consumption behavior. Finally, a Conditional Hidden Semi-Markov Model (CHSMM) is applied to model and predict household appliance electricity consumption, further enhancing understanding of prosumer behavior. Secondly, the dissertation investigates optimization models for efficient, cost-effective power system operation and resource utilization maximization. A convex two-stage socially-aware and risk-aware Second-Order Cone Programming (SOCP)-based ACOPF model is introduced to mitigate DER uncertainty, enhance PV energy utilization, and reduce operational costs. Additionally, a convex SOCP-based ACOPF model is presented for three-phase unbalanced distribution systems, incorporating the Q-V characteristics of PV units with smart inverters based on IEEE Standard 1547. This model enables the participation of PV units with smart inverters in grid voltage regulation, enhancing power system stability and achieving efficient, cost-effective operation.

In this dissertation, a distribution system operator (DSO) framework is proposed to optimally coordinate distributed energy resources (DER) aggregators' comprehensive participation in the retail energy market as well as wholesale energy and regulation markets. Various types of DER aggregators, including energy storage aggregators (ESAGs), dispatchable distributed generation aggregators (DDGAGs), electric vehicles charging stations (EVCSs), and demand response aggregators (DRAGs), are modeled in the proposed DSO framework. An important characteristic of a DSO is being capable of handling uncertainties in the system operation. An appropriate method for a market operator to cover uncertainties is using two-stage stochastic programming. To handle comprehensive retail and wholesale markets participation of distributed energy resource (DER) aggregators under uncertainty, a two-stage stochastic programming model for the DSO is proposed. To handle unbalanced distribution grids with single-phase aggregators, A DSO framework is proposed for unbalanced distribution networks based on a linearized unbalanced power flow which coordinates with wholesale market clearing process and ensures the DSO's non-profit characteristic. When proposing a DSO, coordination with the ISO is important. A framework is proposed to coordinate the operation of the independent system operator (ISO) and distribution system operator (DSO). The framework is compatible with current practice of the U.S. wholesale market to enable massive distributed energy resources (DERs) to participate in the wholesale market. The DSO builds a bid-in cost function to be submitted to the ISO market through parametric programming. A pricing problem for the DSO is proposed. In pricing problem, after ISO clears the wholesale market, the locational marginal price (LMP) of the ISO-DSO coupling substation is determined, the DSO utilizes this price to solve the DSO pricing problem. The DSO pricing problem determines the distribution LMP (D-LMP) in the distribution system and calculates the payment to each aggregator. An efficient algorithm is proposed to solve the ISO-DSO coordination parametric programming problem. Notably, our proposed algorithm significantly improves the computational efficiency of solving the parametric programming DSO problem which is computationally intensive. Various case studies are performed to analyze the market outcome of the proposed DSO framework and coordination with the ISO.

Renewable energy and carbon reduction policies are creating new challenges for electricity markets. To achieve carbon-free goals, large-scale battery energy storage systems (BESSs) are necessary to ensure grid reliability and flexibility. The impact of BESSs on market and grid operation, as well as the optimal portfolio across the energy and ancillary services markets, must be analyzed to guide their operation. At the same time, the expansion of renewable and storage resources and the adoption of carbon reduction policies have introduced new complexities to the bidding behavior of market participants, which cannot be easily described by cost-based bidding objectives. In response to these challenges, this dissertation aims to achieve two research objectives: (I) enable BESS participation in energy and ancillary services markets under uncertainties, considering the battery's degradation cost; (II) identify robust bidding objectives for electricity market participants based on their historical bidding data. Three optimization frameworks are proposed in Part I to model a BESS as a price-maker in energy markets, evaluating its impact on market outcomes. The preliminary framework models automatic generation control signals, while the detailed framework proposes a participation factor for dispatching AGC signals and accounts for battery degradation costs. The stochastic framework models spinning reserve deployment with uncertainty and propose an optimization-based approximation method based on reinforcement learning. Case studies on proposed frameworks validate operational models for Battery Energy Storage Systems (BESS) and markets, showing the accuracy and efficiency of the approximation approach. Key findings include that accurate degradation cost modeling is essential, and participation in ancillary services markets is more profitable. Part II proposes a data-driven approach using Adversarial Inverse Reinforcement Learning to identify robust bidding objectives for electricity market participants. It introduces a tailored reinforcement learning model for bidding objective identification without data discretization, and a special policy structure compliant with multi-segment bidding rules. Two approaches are suggested for electricity market environment modeling in RL/IRL problems, ensuring the robustness of the identified bidding objective. Three case studies validated the accuracy and robustness of the proposed bidding objective identification method in various application scenarios.

Due to the new and old challenges, modern-day market management systems continue ‎to evolve, including market reformulations, introducing new market products, and ‎proposing new frameworks for integrating distributed energy resources (DERs) into the ‎wholesale markets. Overall, questions is regarding how to reflect these essential changes in ‎the market models (design, reformulation, and coordination frameworks), design market-‎based incentive structures to adequately compensate participants for providing ancillary ‎services, and assess these impacts on market settlements.‎First, this dissertation proposes the concept of securitized-LMP to solve the issue of how ‎market participants should be compensated for providing N-1 reliability services. Then, ‎pricing implications and settlements of three state-of-art market models are compared. The ‎results show that with a more accurate representation of contingencies in the market ‎models, N-1 grid security requirements are originally captured; thereby, the value of service ‎provided by generators is reflected in the prices to achieve grid security.‎ Also, new flexible ramping product (FRP) designs are proposed for different market ‎processes to (i) schedule day-ahead (DA) FRP awards that are more adaptive concerning ‎the real-time (RT) 15-min net load changes, and (ii) address the FRP deployability issue in ‎fifteen-minute market (FMM). The proposed market models performance with enhanced ‎FRP designs is compared against the DA market and FMM models with the existing FRP ‎design through a validation methodology based on California independent system operator ‎‎(ISO) RT operation. The proposed FRP designs lead to less expected final RT operating ‎cost, higher reliability, and fewer RT price spikes.‎ Finally, this dissertation proposes a distribution utility and ISO coordination framework ‎to enable ISO to manage the wholesale market while preemptively not allowing ‎aggregators to cause distribution ‎system (DS) violations. To this end, this coordination ‎framework architecture utilizes the statistical information obtained using different DS ‎conditions and data-mining algorithms to predict the aggregators qualified maximum ‎capacity. A validation phase considering Volt-VAr support provided by distributed PV smart ‎inverters is utilized for evaluate the proposed model performance. The proposed model ‎produces wholesale market awards for aggregators that fall within the DS operational limits ‎and, consequently, will not impose reliable and safety issues for the DS.‎

Power systems are undergoing a significant transformation as a result of the retirements of conventional coal-fired generation units and the increasing integration of converter interfaced renewable resources. The instantaneous renewable generation penetration as a percentage of the load served in megawatt (MW), in some areas of the United States (U.S.) sometimes approaches over 50 percent. These changes have introduced new challenges for reliability studies considering the two functional reliability aspects, i.e., adequacy and the dynamic security or operating reliability.

Adequacy assessment becomes more complex due to the variability introduced by renewable energy generation. The traditionally used reserve margin only considers projected peak demand and would be inadequate since it does not consider an evaluation of off-peak conditions that could also be critical due to the variable renewable generation. Therefore, in order to address the impact of variable renewable generation, a probabilistic evaluation that studies all hours of a year based on statistical characteristics is a necessity to identify the adequacy risks. On the other hand, the system dynamic behavior is also changing. Converter interfaced generation resources have different dynamic characteristics from the conventional synchronous units and inherently do not participate in grid regulation functions such as frequency control and voltage control that are vital to maintaining operating reliability. In order to evaluate these evolving grid characteristics, comprehensive reliability evaluation approaches that consider system stochasticity and evaluate both adequacy and dynamic security are important to identify potential system risks in this transforming environment.

The grand transition of electric grids from conventional fossil fuel resources to intermittent bulk renewable resources and distributed energy resources (DERs) has initiated a paradigm shift in power system operation. Distributed energy resources (i.e. rooftop solar photovoltaic, battery storage, electric vehicles, and demand response), communication infrastructures, and smart measurement devices provide the opportunity for electric utility customers to play an active role in power system operation and even benefit financially from this opportunity. However, new operational challenges have been introduced due to the intrinsic characteristics of DERs such as intermittency of renewable resources, distributed nature of these resources, variety of DERs technologies and human-in-the-loop effect. Demand response (DR) is one of DERs and is highly influenced by human-in-the-loop effect. A data-driven based analysis is implemented to analyze and reveal the customers price responsiveness, and human-in-the-loop effect. The results confirm the critical impact of demographic characteristics of customers on their interaction with smart grid and their quality of service (QoS). The proposed framework is also applicable to other types of DERs. A chance-constraint based second-order-cone programming AC optimal power flow (SOCP-ACOPF) is utilized to dispatch DERs in distribution grid with knowing customers price responsiveness and energy output distribution. The simulation shows that the reliability of distribution gird can be improved by using chance-constraint.

Nowadays, the widespread introduction of distributed generators (DGs) brings great challenges to the design, planning, and reliable operation of the power system. Therefore, assessing the capability of a distribution network to accommodate renewable power generations is urgent and necessary. In this respect, the concept of hosting capacity (HC) is generally accepted by engineers to evaluate the reliability and sustainability of the system with high penetration of DGs. For HC calculation, existing research provides simulation-based methods which are not able to find global optimal. Others use OPF (optimal power flow) based methods where

too many constraints prevent them from obtaining the solution exactly. They also can not get global optimal solution. Due to this situation, I proposed a new methodology to overcome the shortcomings. First, I start with an optimization problem formulation and provide a flexible objective function to satisfy different requirements. Power flow equations are the basic rule and I transfer them from the commonly used polar coordinate to the rectangular coordinate. Due to the operation criteria, several constraints are

incrementally added. I aim to preserve convexity as much as possible so that I can obtain optimal solution. Second, I provide the geometric view of the convex problem model. The process to find global optimal can be visualized clearly. Then, I implement segmental optimization tool to speed up the computation. A large network is able to be divided into segments and calculated in parallel computing where the results stay the same. Finally, the robustness of my methodology is demonstrated by doing extensive simulations regarding IEEE distribution networks (e.g. 8-bus, 16-bus, 32-bus, 64-bus, 128-bus). Thus, it shows that the proposed method is verified to calculate accurate hosting capacity and ensure to get global optimal solution.