Twin-delayed deep deterministic policy gradient algorithm for the energy management of microgrids

Abstract

The microgrid market is growing significantly due to several drivers, such as the need to lower greenhouse gas emissions by integrating higher shares of distributed renewable energy sources, falling costs of microgrid components, the need for more reliable power supply infrastructures, and new off-grid solutions to foster electricity access in developing economies. Coordinated management of the microgrid components is crucial for their effectiveness, and this can be very challenging when hosting solar or wind generation. This paper studies the energy management problem of a microgrid based on reinforcement learning algorithms. The advantage of using these algorithms against other optimization and machine learning techniques is that they do not need past experiences to learn a strategy. The learning is based on trial and error experiences, which facilitates its easy implementation to other microgrids while demonstrating their facility to be applied in real cases. In particular, this paper proposes an implementation for an Energy Management System (EMS) in microgrids using the Twin-Delayed Deep Deterministic Policy Gradient (TD3) algorithm. Moreover, it compares the proposed algorithm with the Deep Q-Network (DQN). This comparison evaluates the improvement over exploiting the continuous nature of the decision variables against a discretization of the same since the DQN cannot make actions over a continuous space.

The microgrid market is growing significantly due to several drivers, such as the need to lower greenhouse gas emissions by integrating higher shares of distributed renewable energy sources, falling costs of microgrid components, the need for more reliable power supply infrastructures, and new off-grid solutions to foster electricity access in developing economies. Coordinated management of the microgrid components is crucial for their effectiveness, and this can be very challenging when hosting solar or wind generation. This paper studies the energy management problem of a microgrid based on reinforcement learning algorithms. The advantage of using these algorithms against other optimization and machine learning techniques is that they do not need past experiences to learn a strategy. The learning is based on trial and error experiences, which facilitates its easy implementation to other microgrids while demonstrating their facility to be applied in real cases. In particular, this paper proposes an implementation for an Energy Management System (EMS) in microgrids using the Twin-Delayed Deep Deterministic Policy Gradient (TD3) algorithm. Moreover, it compares the proposed algorithm with the Deep Q-Network (DQN). This comparison evaluates the improvement over exploiting the continuous nature of the decision variables against a discretization of the same since the DQN cannot make actions over a continuous space.