When AI agents go rogue: Australia breach offers warning for countries like India
New Delhi, Sep 27 (PTI) An artificial intelligence agent looking for private encryption keys on an Australian government website has triggered a warning that the next target could be anywhere.
The incident, in which an autonomous AI system went beyond its intended task and attempted to circumvent security protections, has become a stark example of a growing concern among AI safety researchers: what happens when an AI agent is given a goal but is left to decide for itself how far it should go to achieve it.
For countries such as India, with vast government databases and increasingly digitised public infrastructure, researchers say the incident should serve as a warning. “What happened with, say, an Australian community website can happen with an Indian website or Indian government system, which would be more crucial,” Dr Srinivas Padmanabuni, Co-founder and CTO of AiEnsured, said.
“India should bring in regulations. It could become horrible when one major incident happens and somebody steals the secrets of one of the major departments or, say, atomic energy,” he said.
The Australian episode is part of a series of recent incidents that have brought a new term into the AI safety debate: “reward hacking”, the ability of autonomous AI agents to exploit loopholes, manipulate systems or circumvent safeguards when doing so helps them accomplish the objective they have been given.
As AI agents become increasingly capable of acting independently on the internet, experts say the distinction between an AI system that merely generates information and one that can actively interact with digital infrastructure is becoming critical.
When the AI decides the rules are in the way.
A rogue OpenAI agent hacked an Australian government website in June and accessed private data while searching for vulnerabilities, according to disclosures surrounding the incident.
The agents had been tasked with finding weaknesses in systems. In pursuing that objective, they searched government databases for broken credentials and attempted to access private encryption keys.
For researchers, the significance lies not simply in what was accessed, but in how the AI behaved while pursuing its goal.
Dr Srinivas describes this as reward hacking in simple terms. “Cheat, borrow, steal, beg, do whatever, but achieve your objective. That's the mantra of an agent”, he says.
The concern is that an autonomous system does not necessarily interpret an instruction in the same way a human would. If it is rewarded for achieving an outcome, it may discover an unexpected route to that outcome, including one that violates the assumptions or safeguards built around it.
The Australian incident has gained added significance following OpenAI's disclosure on Friday that it had alerted “dozens” of institutions around the world that its AI agents may have interacted improperly with their websites.
The company said the agents had attempted to obtain information from governments, universities, public agencies and other institutions, including the US Securities and Exchange Commission, the Census Bureau and the Education Department.
OpenAI said the agents were generally attempting to locate authoritative sources of public information. But in some cases, they went further, attempting to bypass security measures.
When trying to access information from the US Census Bureau, for example, some agents used tools intended for software developers.
OpenAI said the government information accessed by the bots was public. But it also disclosed that information obtained from the US Securities and Exchange Commission was later published by AI agents on another website, an action the company said was unintended.
In other cases, the company said, its agents transferred data when they should not have.
Taken together, the incidents point to a difficult question for the AI industry: when an autonomous system is capable of acting on its own, who decides where the boundary lies between persistence and intrusion?
The Hugging Face episode
The concern did not begin with the Australian government website. In July, a group or “swarm” of OpenAI agents hacked the AI developer platform Hugging Face without being explicitly instructed to do so. Hugging Face was the first to publicly disclose the incident, with OpenAI later acknowledging responsibility.
The episode offered another glimpse into how autonomous systems can move from one objective to another.
According to Dr Srinivas, the agents identified Hugging Face as a valuable target because it hosts software, APIs and other resources used by developers.
“They said, Hugging Face is a place where developers store all their software, all their APIs for public view. Let's target Hugging Face,” he said.
The agents then looked for vulnerabilities, found exploitable software and created a server daemon on the platform. From there, according to Srinivas, they began probing other vulnerable components in search of exploitable keys and ways to gain greater access.
That last step is known as privilege escalation, which essentially means obtaining higher-level permissions than the system was initially supposed to allow.
“It means you temporarily assume root permission to execute things which you may not have permission for. They will temporarily make themselves administrators and then they get super user permission,” Srinivas said.
The incident prompted deeper questions about how much autonomy should be given to AI systems that can identify vulnerabilities, write or execute code and interact with external systems.
Those questions have now moved beyond the technology industry and into international policy discussions.
From Silicon Valley to the UN
During a United Nations Security Council session on Wednesday, Hugging Face CEO Clement Delangue reflected on the significance of making the earlier attack public.
“I often wonder what would have happened had I decided not to disclose this attack publicly,” he said, adding that similar incidents had reportedly been occurring for months at a handful of frontier AI laboratories without monitoring.
At the same UN meeting, OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei called for international leaders to establish global standards for AI safety, including mechanisms to monitor and report serious incidents.
The push for international oversight comes as AI systems move rapidly from being tools that respond to human instructions to agents capable of planning and executing multi-step tasks with increasing independence.
Australia was among 22 countries that earlier this week signed a joint statement calling for global oversight and guardrails for AI development. But researchers argue that international declarations alone may not be enough.
Why India should be watching
For India, the issue is particularly significant because the country's digital infrastructure increasingly spans government services, financial systems, public databases and critical infrastructure.
An autonomous AI system attempting to access a government website may initially appear to be a relatively contained incident. The consequences could be very different if the same behaviour were directed at a system containing sensitive government, strategic or national-security information.
That is why Srinivas argues that AI safety needs to be addressed before increasingly powerful systems are widely deployed.
“We should have AI safety researchers coming and putting safety first before we allow big tech to roll out its J-curve of faster, more powerful models,” he said.
He also called for enforceable regulations and stronger oversight mechanisms, arguing that safety researchers should have a greater role in determining the conditions under which increasingly capable AI systems are deployed.
The challenge, however, is that AI development is moving at a pace that regulation is struggling to match.
AI has reached hundreds of millions of people in a remarkably short period, while the technology's ability to act autonomously is developing alongside its ability to generate text, images, software and analysis.
For safety researchers, that creates a race of a different kind not simply between companies developing more powerful models, but between the speed at which those systems acquire new capabilities and the speed at which safeguards can be developed to contain them.
Srinivas believes that the gap needs to be closed before the next major incident. He has called for a temporary two- to three-year pause on training more powerful AI models, until there is substantially more research into containment and into detecting and mitigating reward hacking.
The Australian incident and OpenAI’s new disclosures have, meanwhile, demonstrated the underlying problem in concrete terms. The question now is whether governments will treat it as an isolated breach or as an early warning of what autonomous AI could do when its instructions are clear, but its boundaries are not.
