COVID-19 showed modelling is broken. This needs urgent fixing

As COVID-19 rapidly spread across the world, researchers and policymakers raced to create models predicting how the pandemic would unfold, including the potential loss of life.

For better or worse, these models controlled our basic rights. They advised governments and populations. They shut borders and separated families for months on end. They determined the stringency of isolation requirements. They indicated if and where we should wear masks. And they decided when loved ones could reunite and finally hug. They profoundly shaped our everyday lives.

Unfortunately, many of the models built to predict the pandemic were wrong, and not just by a bit – they were way off.

As we start to live in a ‘COVID-normal’ world, it’s time to take stock of how we model and how we can do it better, particularly in times of crisis.  This won’t happen until we fix the current problems plaguing modelling.

My research shows the problems with many COVID-19 models were due to three major factors. Addressing these three factors will help make all modelling, not just for COVID-19, better in the future.

Factor one: assumptions and values

Like other models, COVID-19 models were built on a specific set of assumptions about the nature of the virus, including its properties, rate of infection and rate of mortality. Different models may make different assumptions and yield different predictions.

Some of the assumptions made during modelling did not reflect the reality of the pandemic on the ground. For example, in many models there was an assumption that the population was relatively immobile and had a consistent level of susceptibility to the virus. In another example of failure, The Doherty Model – which guided Australia’s COVID-19 response – overlooked the virus’s impacts on different ethnic communities by assuming equal vaccine distribution, understating the risk to certain communities.

All models are built on assumptions, so this is not just a failing of those trying to represent COVID-19. However, it does question the values guiding any type of modelling and how a model is then deployed. This includes prioritising economic recovery over minimising COVID-19 cases, such as the ‘zero-COVID’ approach in Australia, or prioritising certain groups over others, like older adults over younger individuals during vaccination. The values that significantly influenced the model’s assumptions, boundaries and outcomes shaped decisions on when to lift lockdowns, how to allocate resources, and when it was safe to resume various activities.

Shouldn’t something that has the power to grant, suspend or deny human rights be guided by frameworks built on greater inclusion, transparency, and multiple perspectives and experiences?

Factor two: maths and uncertainty

Models operate differently based on the characteristic of their ‘engine’—that is, the maths behind them. For modelling COVID-19, various mathematical techniques were used to emulate how the virus spread and when the pandemic would peak, each with its own limitations and benefits. None of them were perfect. For example, the well-known SIR (susceptible-infected-recovered) models largely failed in predicting the epidemic in the long-term.

Most models also failed to deal with uncertainty. Trying to model an evolving beast changing faster than research can keep up with is no mean feat. COVID-19 models mostly failed because of too many ‘simplifications’.  For example, almost two years into the pandemic, the Victorian Government insisted that daily case numbers would not reach 25,000 based on worst case-scenario modelling. Just one week after this announcement, Victoria recorded 51,356 new coronavirus cases. The numerous unknown unknowns led some modelers from the early days of COVID-19’s global outbreak to conclude that an event like a pandemic cannot be numerically predicted.

Factor three: context

Models have and make their own context. A model is not only the representation of a situation, but also the product of many socio-political interactions.

The problem with COVID-19 models was that they were generally detached from local knowledge and history, while being attached to a global narrative framing COVID-19 and potential responses.

The question here is not how models should be locally translated, but how they should be deliberatively, inclusively, and democratically negotiated with different mentalities, rationales and alternative local models.

Responsible modelling should be our highest priority

First-hand experience of the pandemic clearly tells us that society needs to take the power of models seriously, particularly those models that are actively shaping policy decisions, and influencing basic rights. A model’s assumptions, maths, and context need to be understood as part of a wider conversation which takes place between modelers, non-modelers and the general public.

COVID-19 models incorporated more than just the science. They were shaped by political agendas and public demands. The idea of ‘apolitical science’ became a myth, as scientific models were tightly coupled with biases, social preconceptions and political agendas. This meant models weren’t just representing mathematics and data in isolation, they represented the whole range of choices we can make in the process of modelling.

Modelers, and the policymakers who use models, must take more responsibility in being transparent about a model’s objectives, assumptions, methodology and limitations.

COVID-19 models and the decisions made under their influence played a critical role in keeping us safe. But this is not without complications. We need to leverage the lessons of our failed COVID-19 modelling to adopt ‘responsible modelling’ in the post-pandemic world.

These lessons aren’t just for COVID-19 models. We must address the three factors I’ve outlined to build a future in which modelling is no longer only a technical challenge, but also a social, ethical, cultural and economic one.

After all, models are how we plan for the future; this is not a power to be treated lightly.