Why Robodebt failed

When Robodebt was announced by the federal government, it was positioned as a debt collection process that would result in net savings of more than a billion dollars over five years.  

The Services Australia scheme employed an algorithmic decision-making system (ADM), which compared the records of welfare recipients with income data from the Australian Tax Office, and pursued repayments based on its calculations. 

A total of $2 billion in debt notices were issued to 700,000 current or former welfare recipients. But the fact that the system operated upon averages resulted in large miscalculations regarding the amount of money people owed to the government. 

According to Professor Emerita Shirley Gregor from The Australian National University (ANU), it was primarily the way the technology was used — with a lack of responsible human oversight — that created problems. 

“Robodebt showed how even relatively well-known technologies can lead to disastrous effects when inadequate software is coupled with an inability of top management to take action when defects become obvious,” Gregor says. 

Through her research, Gregor and colleagues have found that while ADMs are a convenient way to manage data, minimising the level of human control has consequences.   

In the case of Robodebt, control of the system was managed by a central machine. This limited checks for accuracy when the system made a debt estimation.  

The final report by the Royal Commission into Robodebt found that roughly 443,000 people received false debt notices, with many recipients experiencing serious distress as a result. 

On top of this, recipients were also required to prove any inaccuracies themselves by acquiring old bank statements and pay slips from previous employers. When people rang Centrelink for assistance, staff were instructed to redirect them to an online portal, which was difficult to use.  

“Reversing the onus of proof is especially complicated when citizens are limited in their ability to obtain relevant information,” Gregor says. “If self-service through online portals is encouraged then user testing should be undertaken to ensure systems are accessible and user-friendly.”  

Despite Robodebt’s mismanagement — which led to a Royal Commission and a class action — Gregor says ADMs can be used effectively by government agencies. 

“There was a case reported in New Zealand with a child protection agency that showed, on balance, use of an algorithmic system gave better outcomes,” Gregor says. 

“But human operators were able to override decisions on a case-by-case basis. Not having ‘humans-in-the-loop’ and relying exclusively on a machine to make decisions can lead to bias and unacceptable levels of error.”   

Ultimately, Gregor says that while ADMs and other information technology systems can be highly effective, this shouldn’t remove human responsibility from the equation. 

“Emerging technologies such as generative artificial intelligence are now widely available, and their weaknesses are often not well understood,” she says.  

“In any use of IT for decision-making, thorough risk analysis, monitoring and user experience testing should be undertaken on an ongoing basis.”