Explainable post hoc portfolio management financial policy of a Deep Reinforcement Learning agent

Abstract

El artículo propone un enfoque novedoso para dotar de interpretabilidad al Aprendizaje por Refuerzo Profundo (DRL) aplicado a la gestión de carteras financieras. A diferencia de los modelos clásicos, como la teoría de Markowitz, poco fiables en mercados volátiles, los agentes DRL ofrecen soluciones más flexibles, pero opacas en su funcionamiento. Para superar esta limitación, los autores desarrollan un marco de Aprendizaje por Refuerzo Profundo Explicable (XDRL) que integra el algoritmo Proximal Policy Optimization (PPO) con técnicas de interpretación post hoc como SHAP, LIME y análisis de importancia de características. A través de experimentos con acciones tecnológicas, se muestra cómo estas herramientas permiten identificar las variables clave que influyen en las decisiones de inversión del agente en tiempo real. El trabajo constituye la primera propuesta de política financiera post hoc explicable basada en DRL, con implicaciones prácticas para aumentar la confianza de inversores y reguladores.

Financial portfolio management investment policies computed quantitatively by modern portfolio theory techniques like the Markowitz model rely on a set of assumptions that are not supported by data in high volatility markets such as the technological sector or cryptocurrencies. Hence, quantitative researchers are looking for alternative models to tackle this problem. Concretely, portfolio management (PM) is a problem that has been successfully addressed recently by Deep Reinforcement Learning (DRL) approaches. In particular, DRL algorithms train an agent by estimating the distribution of the expected reward of every action performed by an agent given any financial state in a simulator, also called gymnasium. However, these methods rely on Deep Neural Networks model to represent such a distribution, that although they are universal approximator models, capable of representing this distribution over time, they cannot explain its behaviour, given by a set of parameters that are not interpretable. Critically, financial investors policies require predictions to be interpretable, to assess whether they follow a reasonable behaviour, so DRL agents are not suited to follow a particular policy or explain their actions. In this work, driven by the motivation of making DRL explainable, we developed a novel Explainable DRL (XDRL) approach for PM, integrating the Proximal Policy Optimization (PPO) DRL algorithm with the model agnostic explainable machine learning techniques of feature importance, SHAP and LIME to enhance transparency in prediction time. By executing our methodology, we can interpret in prediction time the actions of the agent to assess whether they follow the requisites of an investment policy or to assess the risk of following the agent’s suggestions. We empirically illustrate it by successfully identifying key features influencing investment decisions, which demonstrate the ability to explain the agent actions in prediction time. We propose the first explainable post hoc PM financial policy of a DRL agent.