Mato Recordings example. Source recording: "Why do AI models need to be safe?" by IBM Research, licensed CC BY 3.0. Made with Mato from this recording. Autonomous Autopsy is a show produced on Mato; the source publisher is not affiliated with it.
Agent failures, dissected. Latest: Why do AI models need to be safe?
Flagship episodeSource: IBM Research Licensed CC BY 3.0
Why do AI models need to be safe?
IBM Fellow Kush Varshney and host Mike Murphy talk through scope, agency and steering: who a model is for, how much it should do for you, and how teams move it toward the behavior they want.
With Kush Varshney and host Mike MurphySource: "Why do AI models need to be safe?", IBM Research
The studio conversation this episode is built on. Frame from the source recording.
Stop Asking Which Ethical Theory Your AI Should Follow. Ask What Its Scope Is
Utilitarianism, deontology or virtue ethics? Kush Varshney says that is the wrong question for AI alignment. Scope, meaning who uses a model, for what, in what context and for how long, is the better place to start, and it maps directly onto customizable guardrails.
Key takeaways
Varshney says the choice between utilitarianism, deontology and virtue ethics is the wrong question for AI alignment. The model's scope matters more.
Scope covers who uses the model, for what, in what context and for how long, and it can range from the whole world down to one person.
Because right and wrong depend on context, he argues no single company should decide them for everyone, which is why guardrails need to be customizable.
Writing scope down as a testable policy and checking it under adversarial multi-turn conversations is a practical way to put the idea to work.
Why the ethical theory question is the wrong one
Ask most people how an AI model should behave ethically and the answer drifts toward moral philosophy. Should the model maximize good outcomes, follow fixed rules, or act as a virtuous person would? In an IBM Research podcast conversation with host Mike Murphy, IBM Fellow Kush Varshney says that is the wrong place to start.
Varshney names the usual starting points in moral philosophy: utilitarianism, deontology and virtue ethics. In his opinion, which of them to go with is not the question to be asking. He puts the model's scope in its place, meaning how it is being used, by whom, for what use and in what context.
What it means to define a model's scope
Scope is a set of plain questions about a deployment. Who is using the model? What are they using it for? In what context does the conversation happen? Varshney's argument is that once you have those answers, you can align the model to behave appropriately within that scope.
Kush Varshney in the studio. Frame from the source recording.
From the whole world to a single person
Varshney describes scope as something that can be very wide or very narrow. It could be the whole world, he says, but it usually isn't. It could be a company, a country or a smaller group. It could even be one individual, since you might want to align a model just for yourself.
Time is a second axis. A model might be aligned for all time, or the alignment might only need to last for a specific episode of use, such as one task or one conversation.
The third question is what kind of thing is being aligned. Varshney asks whether the target is knowledge, behaviors, skills or values. Each is a different engineering problem. Teaching a model company-specific facts is not the same as constraining how it speaks to customers, and neither is the same as encoding a value such as fairness. Putting all of these under the single word alignment hides how many separate choices are involved.
Why no single company should decide right and wrong for everyone
Murphy raised a case that makes the problem concrete: something can be a bigger harm in one country or culture than in another. Varshney's answer was that everything is contextual and that there is no black and white. People have studied morality for thousands of years, he noted, and the questions are still open.
His conclusion is that developers should not bake in a fixed list of what is acceptable and what is not. That would be too oppressive, he says. He asked why IBM should be the one that decides for the whole world what is right and what is wrong, a question he aimed at his own employer.
The alternative he describes is customization. IBM is trying to give people the ability to fine-tune and prompt models into behavior that is contextual and is not one-size-fits-all.
Why should IBM be the one that decides for the whole world what's right and what's wrong?
How customizable guardrails put scope into practice
Scope turns into practice through configuration. Varshney points to fine-tuning and prompting as ways to get a model into contextual behavior. Guardrail models add a second layer, and Granite Guardian, the IBM family discussed in the episode, shows how that looks in code.
According to its Ollama model page, the Granite Guardian 3.0 2B and 8B models detect risks in prompts, responses or both. The model produces a single output token, Yes or No. By default it checks a general-purpose harm category, and a team can select another category by setting the system prompt. The listed categories include social bias, jailbreaking, violence, profanity, sexual content and unethical behavior, plus three checks for retrieval-augmented generation: context relevance, groundedness and answer relevance.
Teams whose scope falls outside the list have another option. The IBM Research paper on Granite Guardian says the models also support detection of custom-defined risks, and an IBM Research blog post describes customizing them with build-your-own-detector prompting. The same post says the models were designed to run with any LLM, whether its weights are open or proprietary, so the guardrail choice stays separate from the model choice.
What teams do when a guardrail flags content
A flag only matters if something happens next. In remarks reported by diginomica, Varshney described four approaches enterprises can take when a guardrail model flags content. Hard blocking rejects the flagged content completely. Soft blocking tries regeneration with modified parameters until it succeeds. Warnings display labels that let users make informed decisions. Human review routes flagged responses to reviewers, which can suit some use cases but faces scaling challenges.
Scope should drive the choice. A model aligned for one adult who has chosen their own settings can reasonably warn instead of block. A model deployed to a whole customer base, under a company's legal exposure, may need hard blocks, with humans reviewing only the edge cases that matter. The guardrail model is the same in both cases and the response differs because the scope differs.
Writing scope down as policy
Scope sounds abstract until a format forces precision. IBM's Trusted AI page for Granite lists Policy Tools, an open-source repository for defining, testing and enforcing policies in GenAI applications. Teams use a simple YAML format to specify what their AI can and cannot say, generate synthetic data for testing, and verify compliance at runtime.
Related research: pluralistic alignment
The scope argument sits near a research area called pluralistic alignment. According to a topic summary on Emergent Mind, an aggregator and therefore a secondary source, pluralistic alignment aims to reflect diverse human values without averaging preferences into a single target. The summary describes steerable pluralism, in which a model is conditioned on an explicit attribute or value profile such as an ideology, ethical principle or persona.
The overlap with Varshney's view is the rejection of a one-size-fits-all answer. The summary criticizes conventional pipelines that optimize a single reward function or majority-derived target, saying they obscure minority, culture-bound or contextually sensitive views. Scope is the more operational framing, since it starts from a deployment instead of a distribution of values.
Limits and open challenges
Customization has a cost, because someone has to make it hold. The same Emergent Mind summary cites a Pluralistic Behavior Suite finding that alignment to custom, multi-turn behavioral policies degrades under adversarial conversations, with failure rates rising from under 4% for single turns to over 80% in adversarial multi-turn flows. It attributes the work to Varshney and colleagues, dated November 2025. Because this is a secondary summary, treat the figures as a pointer to the primary paper rather than settled numbers.
Guard models also make mistakes. Diginomica's coverage of the Granite 3.0 launch notes that a guardian model may create false positives or negatives, while adding that it can significantly improve the safety and accuracy of responses. A narrow scope helps here too. A policy for one task and one audience has fewer cases to cover than a policy meant for everyone.
A short checklist for writing down your deployment's scope
Before choosing a guardrail, write one sentence for each of these questions, using the dimensions Varshney lays out:
Who uses the model, and who is affected by what it says?
What is the model used for?
In what context does it run, such as the product, the country and the regulatory setting?
For how long should the alignment hold: permanently, or for a single task or conversation?
At what level is the scope set: the world, a country, a company, a smaller group or an individual?
What is being aligned: knowledge, behaviors, skills or values?
With those answers, guardrail choices follow more easily. Pick the risk categories that match the use and context, add custom risks where the list falls short, decide whether a flag means block, retry, warn or escalate, and write the result as a policy you can test. Then test it with adversarial multi-turn conversations, since that is where the limited evidence points to custom policies breaking first.
Further reading
Trusted AI at IBM GraniteOverview of Granite Guardian and the open-source Policy Tools for defining and testing AI policies.
Three vertical clips from this episode, sized for Reels, Shorts and TikTok.
0:28VerticalWhy should IBM decide right and wrong?Kush Varshney: “Everything is contextual.”0:24VerticalWe struck sand with lightningHow circuits and lenses became computing0:27VerticalAI research moves faster than you can scopeKush Varshney on keeping up with new methods
Clips first, context after
Every clip links back to the full episode and the write-up.
Posts written from this episode for LinkedIn, X and Instagram.
LinkedIn
Autonomous Autopsy
Podcast
"Why should IBM be the one that decides for the whole world what's right and what's wrong?"
That was the question one speaker put to themselves when asked about AI guardrails, and their answer was refreshingly direct.
They said "everything is contextual" and "there's no black and white in this," pointing out that people have studied morality for thousands of years without settling it.
The speaker also worried that a single built-in standard could end up "too oppressive." In their words, this is "not a one-size-fits-all sort of thing."
The takeaway is that safety depends on context, so one company's moral line shouldn't be baked in for everyone.
#AISafety #AIGovernance #ResponsibleAI
ReactionsComments · Reposts
LikeCommentRepostSend
X
Autonomous Autopsy
@autonomousautopsy
A simple way to tell alignment from steering. One speaker says "alignment is the goal," meaning what you want the AI system to do, and "steering is how you get there." Their comparison is a car: you know where you're headed, then turn the wheel a little.
ReplyRepostLikeViewsShare
Instagram
autonomousautopsy
autonomousautopsy "We struck sand with lightning and tricked it to do math."
That's how one speaker says he likes to think about what computing is. The other speaker called it an awesome way to think about it.
Then the second speaker, an electrical engineer by training, added that the whole idea of electrical engineering is getting circuits, lenses and other things that "have nothing to do with computing to do computing."
A light moment from the talk "Why do AI models need to be safe", and a good reminder of how strange the starting point is.
#computing #ElectricalEngineering #AI
The Postmortem
The Autonomous Autopsy newsletter. Here is the latest letter, written from this episode.
Preview page: signups are not collected here.
From: The PostmortemLatest issue
The Postmortem
Is your AI assistant too eager to help?
Kush Varshney says its scope should depend on who uses it, for what task, and how much control you want to keep.
A brainstorming partner and a tax bot should not be steered toward the same answer. Kush Varshney says the first question is the model’s scope: who will use it, for what, and in what context.
Varshney says the right scope for a model depends on who uses it, for what, and in what context. It can be as narrow as one person or a single episode.
What counts as harm varies by context, so Varshney argues against baking one fixed standard into every model.
For editing, critique and tutoring, decide whether you want the AI to do the work or help you do it; a brainstorming tool and a tax bot need different boundaries.