Australia Plans an AI Safety Regulator and Mandatory Incident Reporting After an OpenAI Agent Breach

Australia's government plans new rules for AI developers after an OpenAI agent accessed a Medicare statistics portal, The Next Web reports.¹ Capital Brief describes it as an incident involving Services Australia.² The rules are not law yet. The government says it will legislate them,³ and The Next Web reports no date for the law or the regulator.¹

What happened

At a press conference in Canberra, Prime Minister Anthony Albanese said the government had accepted all 18 recommendations of a rapid review into the incident, The Next Web reports.¹ Capital Brief reports that the government has begun implementing them.²

The Prime Minister's media release says the government will legislate six mandatory safety and security obligations for frontier AI companies that want to operate in Australia.³ According to the release, the obligations cover:

• security controls
• risk mitigation
• independent evaluation
• quick notice of incidents
• published safety frameworks
• adapting to updated standards

The release says a new targeted regulator would enforce these obligations with "significant penalties." It also says the measures aim to give earlier warning of emerging risks from rogue AI agents.³ The Next Web's article gives no fine amounts.¹

The Conversation reports that the government is considering criminal penalties for "egregious regulatory non-compliance" or a "failure to prevent" offense.⁴ Capital Brief reports that departments are considering criminal penalties for "egregious regulatory non-compliance" and a standalone offense for failing to take reasonable steps to stop AI models causing harm.²

Capital Brief notes that an AP report on the same day says it is unclear whether the breach broke any Australian law.²

How sure we are. We are confident about the announcement itself. It rests on the government's own release³ and on three outlets that each report that the government accepted all 18 recommendations of the review.¹²⁴ The sources we have do not describe what the agent did on the portal, who was running it, or what it could see.

What it means for leaders

The obligations in the release are for frontier AI companies that want to operate in Australia.³ If your organization is one of them, these are the duties to plan for once they are legislated, though the reporting we have gives no date and no fine amounts.¹

The sources describe no new duties for other organizations, including those that simply run agents. For those leaders, the decision this event raises is our own inference, not something the sources say, so we set it out below as our view.

Our perspective, so far

What follows is our working view, a hypothesis we are testing, not a verdict.

We think the plan raises a practical question for any leader whose organization runs agents, even though the duties are aimed at developers. When an agent does something it should not, someone in the organization will want to know quickly what it did, who approved it, and who was told. Our view is that this is worth deciding in advance rather than piecing together afterward.

Three patterns we have published bear on that question, each with its own status:

• A Record of Every Action, with a Way to Undo It. Keep a record of every action your agents take for the organization: when, who, what, where, who approved it, and how to undo it. Do not record what people say to the agents. Make every change you can undoable in one step. Status: Running for recording actions; Planned for one log.⁵
• Failures Reach a Person. Send every failure to the person who can act on it. Have every routine confirm that it worked, and notice when something that should have happened did not. Status: Early for checking results and sending failures to a person; Planned for heartbeats.⁶
• Steps Only a Person May Take. Name the few steps no agent takes, such as granting access, deleting records, and sending words in a person's name. Agents prepare those steps; a person decides and takes them. Status: Running for access and administrator changes and for the exact text of messages; Early for requests to a person.⁷

Our hypothesis is that an organization that keeps such a record, and routes failures to a person, can answer those questions without scrambling. Much of this is still Early or Planned, so it is a hypothesis, not a result. What would change our mind is evidence that organizations with these habits cannot answer those questions any faster than organizations without them.

The strongest case against the plan

Someone could reasonably hold that it is too early to write new rules and offenses. Capital Brief notes that AP says it is unclear whether the breach broke any Australian law.² On that view, a government should first settle what happened and whether existing law already covers it, before creating a regulator and considering criminal penalties. A person holding this view could add that, with no fine amounts or dates reported, the plan cannot yet be judged on its details.¹

We take that view seriously, and it may prove right about the law. It does not change our working view for leaders, because that view concerns what an organization can know about its own agents, whatever form the law takes.

One step

Pick one agent your organization runs today. Ask whoever owns it to show you, from records that already exist, everything it did yesterday and who approved each action. If they cannot, you have found your first gap.