User Intent in AI: Inference Under Uncertainty

Series: Thinking, Intention, and Action (3/4). Previous: How AI Systems Reason and Act; next: How Humans and AI Form a Shared Action Loop

When an AI system says that it has identified a user’s intent, it has not discovered a hidden object inside the user’s mind. It has selected an interpretation that is useful for deciding what to do next.

That distinction is fundamental. In product engineering, user intent is usually an operational variable: search for a flight, cancel an order, summarize a document, edit a file. In psychology and philosophy, intention can mean a practical commitment, a purpose in acting, or the mental organization of action. The engineering label is narrower and more provisional.

An AI system infers an actionable interpretation from available evidence. It does not directly observe the user’s full purpose.

“Intent” is a model of the task

Consider the prompt:

Check this proposal.

Several actions fit the words:

• summarize the proposal;
• verify its claims;
• identify logical gaps;
• edit the prose;
• judge whether it should be approved.

A system needs some representation of the requested task before it can respond. Traditional dialogue systems often assign a message to a predefined intent such as cancel_order, then extract slots such as an order number. A language model can infer and express a much wider range of tasks without a fixed list, but the underlying problem remains: which action does this utterance license in this context?

The inferred intent is therefore a working hypothesis about the task, not a complete theory of the person.

Prompts are often underspecified

Natural language relies on shared context. People omit information because another person can usually recover it from the situation.

“Make it shorter” presupposes a text and a relevant standard of brevity. “Use the first one” presupposes a previously presented set of options. “Publish it” may presuppose a particular site, account, version, audience, and approval state.

Linguistic meaning alone cannot supply all of this. A useful interpretation may depend on:

Research on ambiguity shows why forcing every utterance into one interpretation can be brittle. Ellipsis, polysemy, and missing constraints can leave several readings reasonable at the same time. EMNLP 2024: Making Language Models Explicitly Handle Ambiguity

Inference is better represented as a distribution

A simplified model is:

P(task interpretation | prompt, context, environment, rules)

The system compares candidate interpretations under the evidence it can access. One interpretation may dominate; several may remain close; all may be poor because a crucial fact is missing.

This yields three different conditions:

1. Clear enough to act. One interpretation is strongly supported and the action is reversible.
2. Ambiguous but manageable. The system can state an assumption, produce a draft, or preserve alternatives.
3. Ambiguous and consequential. The system should obtain clarification or confirmation before an irreversible or externally consequential action.

Confidence is decision-relative. The evidence needed to suggest a title is lower than the evidence needed to publish under someone’s name or transfer money.

Semantic similarity is only part of the problem

Language models learn statistical relations among expressions and contexts. That helps them recognize that “clean this up” may request editing, or that code followed by an error message probably requests diagnosis.

But intent inference also involves pragmatics:

• What is the speaker trying to accomplish by saying this now?
• Which earlier object does “it” refer to?
• Is the sentence a request, a question, a correction, or background information?
• Does a polite form conceal a firm requirement?
• Is the user authorizing execution or merely discussing a possibility?

Two prompts can be semantically similar while licensing different actions. “Can this be deleted?” asks about possibility. “Delete this” authorizes an action. “I am thinking about publishing it” does not necessarily authorize publication.

An effective system must therefore interpret language together with conversational commitments and action boundaries.

A deeper goal may remain hidden

Suppose a user asks for a resignation letter. Their immediate task may be clear: draft the letter. Their deeper purpose could be to resign, prepare for a negotiation, explore wording, write fiction, or test the system.

The system can often complete the immediate task without resolving the deeper goal. Confusing the two creates two errors:

• Overreach: claiming knowledge of motives that the evidence does not support.
• Unnecessary friction: demanding a personal explanation when the requested task is already clear and safe to perform.

A good assistant asks only for information that materially changes the work or the permission boundary. It can remain uncertain about a person’s deeper purpose while being precise about the action requested in the current turn.

Confirmation changes status, not metaphysics

If the assistant asks:

Should I identify problems only, or rewrite the proposal as well?

and the user answers:

Rewrite it.

then “rewrite the proposal” becomes an explicit instruction for the interaction. This is stronger evidence than the original ambiguous wording. It still does not prove every motive behind the request.

The most useful state model keeps these distinctions visible:

These labels describe evidence and workflow. They avoid a misleading field such as true_intent = true, which would turn an interpretation into an alleged psychological fact.

Intent and authorization must remain separate

A model may correctly infer what a user wants and still lack authorization to perform the action. It may also have general permission to edit a workspace while misunderstanding which file the user meant.

Reliable systems therefore ask two separate questions:

Interpretation: What action is most likely being requested?
Authorization: Is the system permitted to perform that action now?

This is especially important for publication, messages sent to other people, purchases, deletion, deployment, and access to private data. A confident guess does not create permission.

The inverse matters too. Repeatedly asking for confirmation after permission has already been granted adds friction and can obscure the actual uncertainty. The system should preserve prior authorization while remaining open to new corrections.

Action is a test of interpretation

Because inferred intent is fallible, systems should prefer actions that produce useful feedback at low cost:

• draft before publishing;
• preview before replacing;
• show a diff before merging;
• preserve previous versions;
• state a consequential assumption;
• make uncertain fields explicit rather than filling them with invented values.

The user’s response then supplies new evidence. Acceptance, correction, revision, or rejection updates the working interpretation.

This resembles Bayesian learning in a broad sense: begin with candidate interpretations, observe evidence, update their relative plausibility, and continue revising. But the analogy has limits. A production language model does not necessarily maintain a transparent table of hypotheses or calibrated posterior probabilities, and a plausible interpretation is not thereby the user’s private truth.

What an AI system can responsibly claim

An AI system can sometimes say:

• “The prompt explicitly asks for a rewrite.”
• “Given the previous turn, ‘the first one’ most likely refers to option one.”
• “The user confirmed that publication is authorized.”
• “Two interpretations remain plausible.”

It should be much more cautious about claims such as:

• “This is what the user really wants.”
• “The user’s deeper motive is X.”
• “The person had no intention to do Y.”

The defensible conclusion is precise:

AI can infer, test, and confirm operational interpretations of a request. It cannot turn limited linguistic evidence into certainty about a person’s complete or ‘true’ intention.

Sources

• EMNLP 2024: Making Language Models Explicitly Handle Ambiguity
• OpenAI: Our Approach to the Model Spec
• Stanford Encyclopedia of Philosophy: Intention
• Stanford Encyclopedia of Philosophy: The Pragmatics of How and Why Questions