Most enterprise AI governance fails quietly. It fails not in an incident review but months earlier, in a policy document that no engineer ever opens, a risk register that lists controls no system implements, and a sign off gate that teams learn to route around because it arrives too late to change anything. By the time a model behaves badly in production, the governance that was meant to prevent it has already been decorative for a long while.
We work with Melbourne and interstate enterprises taking AI systems from pilot to production, and the pattern is consistent. Governance that is written as a compliance artefact stays an artefact. Governance that is designed into the delivery process becomes the thing that lets teams move faster, because the questions that would otherwise surface in a legal review at the end are answered while the design is still cheap to change.
Governance is a design problem
The instinct, when a board asks what the organisation is doing about AI risk, is to reach for a policy. A policy is necessary but it is not governance. Governance is the set of decisions an organisation makes about how a system may behave, who is accountable when it behaves otherwise, and how those constraints are enforced in the systems people actually build. That is a design activity. It sits closer to architecture than to legal drafting.
Treating it as design changes what good looks like. A good governance control is one that is legible to the team subject to it, that fails safe when a person forgets to invoke it, and that produces evidence as a by product of normal work rather than as a separate reporting chore. A control that depends on someone remembering to fill in a spreadsheet is not a control; it is a hope. When we design an assurance layer, we ask the same question we would ask of any other non functional requirement. What is the smallest mechanism that makes the wrong thing hard to do and the right thing the path of least resistance?
What ISO 42001 actually asks for
ISO/IEC 42001, the management system standard for artificial intelligence, is useful precisely because it refuses to prescribe specific model controls. It asks instead for a management system: a defined scope, leadership accountability, a way to assess AI specific risks and impacts, controls proportionate to those risks, and a cycle of measurement and improvement. If you have worked with ISO 27001 for information security, the shape is familiar. The novelty is what the standard treats as a risk.
Under 42001 the impact assessment looks beyond the organisation to the people affected by the system. A model that quietly disadvantages a cohort of applicants is a governance failure even if it never triggers a security or availability incident. That framing matters for Australian enterprises because it aligns with where domestic regulation is heading, and because it forces the assessment to name affected groups rather than reasoning about risk in the abstract.
The practical value of the standard is that it gives an organisation a defensible answer to a hard question: how do you know your AI is well managed? Not because a single model passed a single test, but because there is a system that decides which models get built, holds someone accountable for each one, assesses its impact before release, monitors it in production, and retires it when it drifts. Certification is optional. The operating discipline is the point.
“The organisation shall establish, implement, maintain and continually improve the AI management system, including the processes needed and their interactions.”
The Australian direction
Australia has not landed a single omnibus AI act, and enterprises should not wait for one before acting. The direction is already clear enough to design against. The government has published a Voluntary AI Safety Standard built around ten guardrails covering accountability, risk management, data governance, testing, human oversight, transparency and record keeping. It has consulted on mandatory guardrails for AI in high risk settings, and it has signalled that the settings most likely to attract obligations are the ones that affect a person's rights, safety, or access to services.
The reassuring part, for an organisation that has done the ISO 42001 work, is how little of this is new. The voluntary guardrails map almost cleanly onto a functioning management system. Accountable ownership, documented risk assessment, testing before deployment, meaningful human oversight, and an audit trail are the same controls whether they are framed as a standard, a guardrail, or a future obligation. The organisations that will struggle are the ones treating each new instrument as a separate project rather than as another view of one operating model. Layer in the Privacy Act reforms and the tightening expectations around automated decision making, and the message is consistent. Regulators want to see that a human remains accountable and that the reasoning behind a consequential decision can be explained.
An operating model teams use
The bridge from standard to reality is an operating model that engineering teams experience as infrastructure rather than as oversight. In our delivery work this reduces to a small number of load bearing mechanisms.
- A model and use case register that is the single source of truth for what is in production, who owns it, what risk tier it sits in, and when it was last assessed. Nothing reaches production without a record.
- Risk tiering at intake so the weight of governance is proportionate. A low stakes internal summariser should not carry the same burden as a system that shapes a person's access to a service, and forcing them through the same gate teaches everyone to resent the gate.
- Evaluation as a deployment gate with acceptance thresholds defined before the build and measured automatically, so quality and safety checks run on every change rather than once at launch.
- Human oversight that is specified, not assumed . Naming the decision a person must be able to make, the information they need to make it, and the point at which they can intervene turns a compliance phrase into an actual control.
- Evidence generated by the pipeline so that the audit trail an assessor or regulator asks for is a query against systems that already exist, not a scramble to reconstruct history.
None of these is exotic. What makes them work is that they are built where the work happens. The register is wired to the deployment system. The tiering question is asked in the intake form a team already fills in. The evaluation gate lives in the same pipeline as the unit tests. Governance stops being a meeting and becomes a property of the platform.
Where to start
If you are early, resist the temptation to begin with a policy. Begin with an inventory. You almost certainly have more AI in production than your governance function knows about, and you cannot govern what you have not counted. Once you can see the estate, tier it, and put your scarce assurance effort where the impact on people is highest. Then make one control real, end to end, on one high tier system, before you write it down as policy for all the others. A control proven in delivery is worth more than a chapter of aspiration.
The organisations that will handle whatever obligations arrive next are not the ones with the thickest policy. They are the ones who treated governance as part of how they build, sized it to the risk, and made the right thing the easy thing. That is design work, and it is the work that pays for itself long before a regulator ever asks.