Axiom Verity Independent AI governance research
Menu

Risk · R-059 · Agentic & Autonomous Systems

Function-calling hallucination

An agent invents a tool, a parameter or the result of a call, and the fabricated step is executed or relied on as though it had happened.

Severity
Medium
Audience
Practitioner
Frameworks
AML.M0033 · Confabulation · ASI02
Reviewed
2026-11-10

What it is

This is confabulation applied to actions. The model produces a call to a function that does not exist, supplies an argument it made up, such as an account number, or reports that a call succeeded when it was never made. It is separate from agent tool misuse, where a real tool is really called in a harmful way.

How it happens

  • The model fills a required parameter with a plausible value when it has no real one.
  • A tool name is generated from pattern, not from the list provided.
  • A failed or skipped call is summarised to the user as complete.
  • A long task exceeds the context, and earlier results are reconstructed from memory.

Why it matters

A fabricated parameter can direct a real operation at the wrong target. A fabricated result leaves the organisation believing something was done, such as a notification sent or a record updated, when it was not.

Warning signs

  • Tool calls fail validation for unknown names or malformed arguments.
  • Identifiers appear in calls that occur nowhere in the input.
  • The agent's summary lists steps absent from the execution log.
  • Failure rates rise with task length.

Mitigating controls

Primary

Output validation and guardrails Preventive
Validates that a called tool exists and that parameters and results are well formed. Every model output is validated against a schema and sanitised by deterministic code before any downstream system, tool or user interface acts on it.

Supporting

Agent action tracing Detective
Shows which calls were actually executed. Every agent action is recorded with the instruction, plan, tool call, result and identity that produced it, in a form that allows the sequence to be reconstructed.
Evaluation suite and regression testing Detective
Tests tool-use accuracy on each change. A maintained set of test cases with expected results is run on every change to model, prompt, data or tools, and the results are compared with the approved baseline.

Framework and clause references

FrameworkClauseTitle
MITRE ATLASAML.M0033Input and Output Validation for AI Agent Components Validation of agent inputs and outputs.
NIST Generative AI Profile (NIST AI 600-1)ConfabulationConfabulation Confabulation applied to tool use.
OWASP Top 10 for Agentic ApplicationsASI02Tool Misuse and Exploitation A cause of tool misuse.

Sources

NIST Generative AI Profile NIST · framework
MITRE ATLAS MITRE · framework

Related terms

How to cite

Axiom Verity (2026). Function-calling hallucination. Reference Library, R-059. https://axiom-verity.com/risks/function-calling-hallucination. Reviewed 2026-11-10.

Found an error or a better source? Report a correction.