Mato Recordings example. Source recording: "Why do AI models need to be safe?" by IBM Research, licensed CC BY 3.0. Made with Mato from this recording. Autonomous Autopsy is a show produced on Mato; the source publisher is not affiliated with it.

Autonomous Autopsy Subscribe

From the recording · A conversation, annotated

Why do AI models need to be safe?

IBM Fellow Kush Varshney and host Mike Murphy on scope, agency and steering: how much an AI assistant should do for you, and who gets to decide.

Two people seated across a steel desk in a dark studio, each speaking into a boom-mounted microphone.
Figure 1. Host Mike Murphy, left, and IBM Fellow Kush Varshney, right. A frame from the source recording.

The conversation

Lines from the conversation

What Kush Varshney and Mike Murphy said about agency, in their own words, with the episode player alongside.

Host Mike Murphy, bearded and in glasses, gestures while talking. Kush Varshney gestures with both hands while speaking.
Speaking now: Kush Varshney01
  1. 01
    Kush Varshney

    …it's about agency.

  2. 02
    Kush Varshney

    …how much control do you have over what goes on.

  3. 03
    Kush Varshney

    …maybe too eager for most use cases.

  4. 04
    Kush Varshney

    …and they're not gonna grow, they're not gonna learn, they're not going to have self-worth.

Annotation. Varshney names three activities where the human in an AI collaboration shouldn't have their agency taken away: critique, editing and tutoring.1Source recording, CC BY 3.0

  1. 05
    Kush Varshney

    I would be more than happy to give up all my agency.

  2. 06
    Mike Murphy

    Well, what do you think?

Lines quoted word for word in the articles and posts made from this recording. Numbers mark lines, not timestamps.

From the episode

When your AI assistant acts like a helicopter parent

Many chatbots are very eager assistants, maybe too eager. Kush Varshney's helicopter-parent analogy shows why the right amount of human agency depends on the task, from tutoring to tax filing.

By the Autonomous Autopsy desk · 6 minute read

What it means for an AI assistant to be a helicopter parent

Many chatbots are very eager assistants, and for most use cases they may be too eager. That is the view of Kush Varshney, an IBM Fellow, who described the problem in an IBM Research conversation about AI safety1Source recording, CC BY 3.0. His analogy comes from parenting. A parent who does everything for a child takes away the child's agency, and the child won't learn or build self-worth.

Who is Kush Varshney and why his view carries weight

Varshney is an IBM Fellow based at the Thomas J. Watson Research Center, where he directs Human-Centered Trustworthy AI research2Kush Varshney's site. His team built several open-source toolkits, including AI Fairness 360, AI Explainability 360 and Uncertainty Quantification 360.

A tutor sits back with folded hands while a teenage student works through a problem on paper at a library table.
Figure 2. A tutor leaves the work to the student. Image generated with Gemini.

IBM's profile of its 2024 Fellows3IBM 2024 Fellows adds that he co-founded the company's Science for Social Good initiative and has argued that accurate predictions are not enough, because these systems have consequences on people's lives.

Beyond style: from the “that’s great” debate to agency

He gives an example from before the LLM age. A customer working with IBM Research spent a whole day deciding whether their chatbot should use the phrase “that’s great.” That was a style question, and style still matters. But Varshney says the field is now headed past it, to agency. The harder question is what relationship an assistant should have with the person using it.

Why eager assistants are a poor fit for most tasks

An eager assistant defaults to finishing things. For plenty of work that default is fine, until the work is something the person was supposed to be doing for themselves. Then the assistant is playing the helicopter parent: smoothing the path, removing the effort, and leaving the person with less to do and less to learn.

Varshney frames the stakes broadly. In the IBM Research interview, he says that if AI agents take full control of tasks, humans can be left with no agency of their own4IBM Research interview, which he calls bad for human development and dignity. That is a large claim to attach to a chatbot's habits, but it follows from the parenting logic. Skills develop through use, and a tool that takes over every use stops that development.

Critique, editing and tutoring: where humans should keep control

Varshney names three activities where the human in an AI collaboration shouldn't have their agency taken away: critique, editing and tutoring. What they share is that the person's own thinking is the product. A writer who asks for feedback wants to become a better judge of the draft. A student who asks for help wants to understand the material.

Evidence from a tutor built to ask, not tell

One classroom study shows how this plays out. Economists at the University of Wisconsin, La Crosse built an AI tutor called Macro Buddy5The Conversation with ChatGPT’s custom GPT feature, trained on lecture transcripts, slides and homework questions from one macroeconomics course. Macro Buddy was designed to ask follow-up questions instead of giving complete solutions. If a student asked why lower prices might increase spending, it might ask what happens to purchasing power when prices fall.

The course enrolled 140 students across four sections, and the researchers assigned each section a different study format. By the third exam, students who combined Macro Buddy with group discussion5The Conversation had the highest average scores, and students who used Macro Buddy alone scored higher than those who worked alone without it. Average scores fell for all four groups on the second exam, so the gain appeared later.

The limits are plain. This was one course with 140 students, and a tool built from that course's own materials with internet access switched off. The authors say their experiment suggests that erosion of learning is not inevitable. It does not show that every chatbot tutor works.

Brainstorming: balancing agency between human and AI

Co-creative tools sit in the middle. When a human and an AI brainstorm together, Varshney says in the episode, agency has to be balanced among all of the parties. He notes that many customers are in the brainstorming business, so this is a common case.

The failure can run in either direction. In the IBM Research interview, Varshney describes the risk that an AI might ignore ideas contributed by humans4IBM Research interview. If that happens, the humans will stop contributing, and that could slow innovation. A brainstorming assistant that dominates the session gives people a reason to stop taking part.

The tax bot: when giving up agency is the point

Now take the opposite end. If an assistant is filling out your taxes, Varshney says he would be more than happy to give up all his agency. Host Mike Murphy added that he wouldn't want a tax bot to come back and ask, “Well, what do you think?”

Less agency is not a flaw here. Taxes are a task where most people want the result and have no interest in the process, so full takeover gives them what they asked for.

“I would be more than happy to give up all my agency.”

Kush Varshney, on an assistant that fills out your taxes

A spectrum of autonomy, not a switch

Between the tutor and the tax bot sits what Varshney calls a whole spectrum of possible activities, each with its own trade-offs. Research on security operations centers makes a similar point in a different field. One human-AI collaboration framework for security operations centers6arXiv defines five levels of AI autonomy, from manual to fully autonomous, and maps them to human-in-the-loop roles and task-specific trust thresholds. Its authors criticize existing approaches that assume static or binary autonomy settings, since tasks differ in complexity, criticality and risk.

What can go wrong when AI takes full control

The failures are easier to see in specific cases. Varshney gives the example of a human radiologist and an AI system reading a medical image together, where the radiologist's agency can be reduced. The IBM Research interview4IBM Research interview also notes that his team's list of bad behaviors associated with generative AI grew from 39 to more than 80 in two years, and his one-word answer for what changed was agentic AI. Of the risks the team has cataloged, he says the loss of human agency is top of mind for him.

How to decide how much agency users want

Deciding starts with defining the task. Varshney's questions in the episode are what the activity is, what you are doing, and how much agency a human really wants. He adds that the answer has to be customized to the precise use case and context. A short checklist turns that into design work:

  • Does the person want the result, or do they want to build a skill by doing the work?
  • If the assistant makes a mistake, who notices, and how quickly?
  • Should the assistant ask before acting, act and report, or simply act?
  • When several parties contribute ideas, whose get weight, and can the person see that?
  • Do users with different goals need different settings for the same product?

Steering: how teams tune an assistant's level of involvement

Once a team knows the target, steering is the mechanism for reaching it. Varshney says alignment is the goal and steering is how a team gets there. In the interview, he describes methods that act at different points. Prompting is the cheapest, fine-tuning and activation steering are more involved, and decoding is somewhere in the middle4IBM Research interview. IBM's open-source AI Steerability 360 toolkit7AI Steerability 360 on GitHub supports steering across four control surfaces (input, structure, state and output) and evaluates steering pipelines.

The design question to ask before building an assistant

A rule of thumb for builders, educators and everyday users: when the point of the task is the person's own growth or judgment, the assistant should ask, prompt and wait. When the point is only the finished result, it can do the work. Someone has to make that choice on purpose, because an eager default makes it for everyone.

Key takeaways

Eager by default
Varshney calls many chatbots very eager assistants and compares overly helpful AI to a helicopter parent who takes away a person's chance to learn.
Keep agency with the human
Critique, editing and tutoring should leave agency with the human, because the person's own thinking is the point.
Match the boundary to the task
Brainstorming tools need to balance agency between human and AI, and a tax bot can take nearly all of it.
Ask how much agency people want
Autonomy varies by task, so the working question is how much agency the person wants for this activity.
Steering sets the level
Steering methods such as prompting, fine-tuning, activation steering and decoding are how teams adjust an assistant's level of involvement.

Excerpts

Three moments, cut vertical

Three short vertical cuts from the conversation, each one leading back to the full episode.

0:28
Everything is contextual.Kush Varshney on why one company should not decide right and wrong for everyone
0:24
We struck sand with lightning and tricked it to do math.How circuits and lenses became computing
0:27
Just the pace of innovation is so fast right now.Kush Varshney on the pace of new methods

Field Notes, the newsletter

The letter written from this episode

The ideas from the conversation, written up as a short letter you can read in a minute and forward to your team.

Free. Plain text available. No tracking pixels.

Field Notes · Latest letter1 min read

Is your AI assistant too eager to help?

Kush Varshney says its scope should depend on who uses it, for what task, and how much control you want to keep.

A brainstorming partner and a tax bot should not be steered toward the same answer. Kush Varshney says the first question is the model’s scope: who will use it, for what, and in what context.

For editing, critique or tutoring, decide how much of the work you want the assistant to do. Do you want it to rewrite your draft, or point out what you could change and leave the choice to you?

Pick one AI tool you rely on. Ask what its real scope is, then how much agency you want to keep for this task.

Sent with Mato.

As shared

How the episode traveled

LinkedIn Company page

Autonomous Autopsy

Podcast

A customer spent a whole day deciding whether their chatbot should say "that's great."

That story comes from a talk called "Why do AI models need to be safe." The speaker used it to show how much care can go into the style of an assistant's voice.

Then the speaker said where things are headed goes "even beyond style." In their words, "it's about agency."

The speaker described agency as "how much control do you have over what goes on." The question moves from how an assistant sounds to how much control it takes and how much stays with the user.

#AI #AISafety #Chatbots

LikeCommentRepostSend

X @autonomousautopsy

Autonomous Autopsy

@autonomousautopsy

One speaker cautions against anthropomorphizing LLMs: "the thing isn't choosing to do anything." The developers, though, implicitly or explicitly are, maybe through how data sets get curated. How to understand that intent is "a really good question."

ReplyRepostLikeViewsShare

Instagram Feed post

autonomousautopsy

A small toy helicopter hovering over a scattered set of wooden building blocks on a table.

autonomousautopsy An AI that does everything for you is not helping you grow.

In this talk, one speaker says AI assistants can be "maybe too eager for most use cases." When the work is critique, or anything where a person and an AI collaborate, the human "shouldn't get their agency taken away from them."

The speaker compares it to helicopter parents. As a parent, they say they don't want to be doing everything for their kids, because it takes away their agency, "and they're not gonna grow, they're not gonna learn, they're not going to have self-worth."

A good assistant leaves room for the person to do the thinking.

#AI #AISafety #HumanAgency #AIEthics