Stale model knowledge

What it is

A language model knows the world as it was when its training data was collected. It does not know what has changed since and, unless told, does not flag that its answer may be out of date. For stable subjects this is immaterial. For regulation, prices, product terms and current events, the model's knowledge may be confidently wrong.

How it happens

• A question depends on a rule or rate that changed after the cut-off.
• The system relies on the model's memory instead of supplying current documents.
• A retrieval source is itself out of date.
• Users are not told the date of the model's knowledge.

Why it matters

Out-of-date information is presented as current. In regulated communications that can mean quoting a superseded rule, a withdrawn product or an old limit, with the consequences of any other inaccurate statement.

Warning signs

• Answers cite figures or rules that have been replaced.
• The system has no source of current information for time-sensitive topics.
• Nobody knows the model's training cut-off.
• Test questions do not include recent changes.