Simplicity in complexity: a probabilistic view Of Occam’s Razor

Abstract

Complex Systems are often characterized by their intricate structures and behaviors, which can be challenging to analyze and understand. However, the principle of Occam’s Razor suggests that the simplest explanation is often the best. This idea, encapsulated in the concept of the toy model, is often perceived as a parsimonious or exploratory approach when dealing with problems involving many unknown mechanisms or restricted to aggregated data, and not always they are taken as a serious approach to the problem.In this work, we connect the idea of model inference from data as a way to rationalize our intuition about minimal models. Connecting ideas from information geometry and Bayesian inference, we show that toy models are often the optimal way to exploit the information in the data and, hence, are not just amenable to handle mathematically or computationally but also optimal in the sense of information theory.We illustrate these ideas with examples from ecology [1], epidemiology, or astro-chemistry [2]. However, they can be applied to any field where the quality and quantity of the data are limited or poor in terms of diversity. We also discuss the implications of these ideas in the context of well-established ideas such as the renormalization group theory.