AI Governance: We Used to Know Why, Now We Have to Find Out
How do you know if the answer you got from your AI assistant is actually correct?
If it tells you the weather forecast says there’s a 10% chance of rain between 9am and 11am, but it rains all day, does that mean it made a mistake?
What would you need to trust a prediction like that?
Would you need to know where the historical data came from (e.g. is it for your region, or maybe for the South Pole)? To understand the logic behind the prediction? Or maybe to see the last 10 forecasts and check if it was actually right in 9 out of 10 cases?
If your thought process looked like that, or you simply think it makes sense to understand more about how a decision was made, then you’ve just entered the world of AI governance.
The same thing happens when your AI assistant, or an application built on top of an LLM, predicts an outcome.
For example, a bidder could predict a 90% chance that an ad campaign will generate 500 conversions, but the campaign ends up generating only 400.
Same dilemma, but now, instead of getting wet shoes, it costs ad budget.
So what is AI governance?
AI governance is a set of rules and mechanisms that help you determine whether, and when, you can trust decisions made by AI.
It’s not a brake on innovation, and it’s not a compliance checkbox.
Done well, it’s what lets you deploy AI in the places that actually matter.
And it’s becoming increasingly urgent.
AI is being deployed at scale across all areas of a company. Regulations like the EU AI Act are coming into effect, and every new use case raises the cost of getting it wrong.
The three eras of decision-making approaches: From rules to the AI black box
Why has governance become such a hot topic recently?
It’s not like AI and machine learning (ML) are new to us, or that we’ve never needed to understand why these systems make the decisions they do, right?
The truth is that the current AI era is introducing a new set of governance challenges.
01. Rules
In traditional software development, decisions are made based on a set of predefined rules.
We notice patterns and gain some certainty about the outcome.
For example, a luxury car is more likely to be bought by people with an income above a certain amount and a history with the brand.
No matter how complex the rules get, we can follow the full decision tree and pinpoint exactly why a decision was made. In this case, why the luxury car brand decided to target that particular audience.
The problem with this approach is that rules only cover what their author could foresee.
Every new pattern means someone has to notice it first and document it.
02. Classic machine learning
Machine learning was introduced to move beyond hard-coded rules and start looking for patterns in specific datasets that aren’t immediately obvious.
Companies trained narrow models on their own historical data, with well-defined outcomes that could be checked against actual results.
These models were built by in-house data science teams that understood their purpose and limitations and applied the relevant guardrails.
We’ve built plenty of these machine-learning models for AdTech platforms over the last 15 years, including for bidders, fraud detection tools, and audience platforms.
And that’s exactly how it worked: a narrow scope, your own data, and your own team monitoring it.
The problem is that each model was built for a specific task, required your own data and expertise, and was expensive to build and often even more expensive to maintain.
03. Foundation AI models
So what has changed?
Models like Claude, Gemini, and those powering ChatGPT are trained by someone else, on data we have no control over.
The same model can write an email, analyse a contract, and generate code.
The output, in many cases, is no longer a simple right-or-wrong answer. And these models are no longer used only by data science teams, but across the whole company.
The problem here is that you get a general-purpose tool but lose much of the control you had in the previous two eras.
Now, we have limited visibility into what’s behind the answer.
A model can be right for the wrong reasons.
A well-known example is a model that “recognised” wolves by spotting snow in the background, rather than the animal itself, because many of the wolf photos it learned from had snow in them.
It looked great in testing but could fail in the real world.
We also can’t judge an answer by how confident it sounds, because a model can sound just as certain when it’s correct as when it’s making something up. And sooner or later, someone will ask “why?”—a regulator, a client, or your board.
“Because the model said so” is not an answer anyone will accept.
There’s one more shift.
Traditional software can be audited at release and generally behaves the same until someone changes the code or its operating environment.
An AI model can become less accurate without a single line of code changing, simply because the world and the data around it change.
Remember the rain example? You can only judge how reliable a forecast is by evaluating its predictions over time.
So you can’t check AI once and forget about it. You have to keep watching.
The dilemma isn’t new.
What’s new is that many of the safeguards we previously relied on no longer apply in the same way.
Four pillars to understand your AI – the OECD framework
Governance can be complex and painful, especially when the risks are significant.
So, before we decide how much time and money to spend on it, we need to understand what kind of system we’re dealing with and how serious the consequences of a wrong decision could be.
The OECD Framework for the Classification of AI Systems (OECD, 2022) provides a simple structure for doing that.
It was developed for policymakers and regulators, but it can also help companies understand and classify their own AI systems.
The original framework has five dimensions, but we’ve combined the first two (People and Planet + Economic Context) into one pillar (Context and Environment) to keep it practical.
The OECD names of the dimensions are kept inside each pillar.
Pillar 1: Context and Environment
People & Planet: The system’s impact on people – their rights, wellbeing, and work – as well as on society and the environment.
Economic context: The sector, business model, and scale of deployment. The same model can carry different levels of risk depending on whether it’s used in healthcare, finance, or advertising.
Example: A model supporting a diagnosis in a hospital or a credit decision in a bank, compared with a model recommending your next film.
Pillar 2: Data and Input
Data & Input: How the data was collected, its quality, its privacy implications, and the biases it may contain before it even reaches the model. Also, whether there’s enough data and whether it accurately represents the environment in which the model will operate.
Example: Weather data from one place on Earth is a poor basis for making predictions about another. Buying decisions made by people on one platform may not translate directly to another.
Pillar 3: The AI Model
AI Model: The model type, how it was trained and validated, and how explainable its decisions are. It’s also a moving target, as models change with every release and update, often without you deciding when. Even the same model version can give you a different answer to the same question tomorrow.
Example: With linear regression, you can see which factors contributed to a prediction and by how much. With deep learning models, including LLMs, understanding exactly how an output was produced is much harder.
Pillar 4: Task & Output
Task & Output: What the system actually does, e.g. generates content, scores people, makes decisions, or takes autonomous actions.
Example: A funny cat picture with three ears vs. an AI system that shuts down a power plant.
To sum up, once you understand these four factors, you can match the governance measures to the level of risk.
The data and model help you assess how much you can trust the result, while the context and output tell you how much it matters if you’re wrong.
Applying the same controls to everything means either paralysis or uncontrolled risk. The higher the risk, the more transparency, monitoring, and human oversight you need.
The mistakes you can’t see
The consequences of getting AI governance wrong can escalate quietly.
A bidder that burns through your budget? You’ll notice within a month.
A fraud model that blocks legitimate customers? You might only notice when they’re gone.
An ad system that starts promoting scams because scam ads generate great click-through rates and all the numbers add up? You may not notice at all until someone outside your company tells you.
Governance isn’t just there to catch the mistakes you can see.
It’s there to catch the ones you can’t.
We used to know why. Now we have to find out.
Self-assessment
Try this with your current understanding of the data and governance practices for your platform.
There are no points, just your own honest, internal assessment of where you stand.
Context
- Do you know who is affected by your model’s decisions and whether your use case falls under sector-specific regulations?
- Do you know what a wrong decision could cost and who would bear that cost?
Data
- Do you know what data the model was trained or fine-tuned on?
- Has anyone checked that data for quality, gaps, bias, and privacy issues?
Model
- Do you have documentation showing how the model was tested before deployment and how it performed?
- Can you show a client or an auditor how a specific decision was made – what went in, what came out, and on what basis?
Output
- For high-stakes decisions, is there appropriate human oversight before the output is acted on?
- Do you log inputs and outputs so you can reconstruct a decision from a month ago?
Ownership and monitoring
- Is there a person accountable when the model gets it wrong?
- Would you notice if the model started performing worse before your customers did?
If some of these questions made you uncomfortable, that’s the point.
In the next articles, we’ll go through each of these areas one by one – the model’s lifecycle, regulations and standards, and a practical framework you can actually apply.