Generative AI used for disinformation

What it is

Generative AI lowers the cost of producing persuasive falsehood: articles, reviews, social media accounts, images and audio, in any volume and any voice. An organisation meets this in two ways. Its own public-facing tools may be used as the means of production, if they will write whatever they are asked. And it may be the target, with fabricated statements, fake executives or forged documents circulated under its name. The wider effect on public information is covered among the systemic risks.

How it happens

• A public assistant or interface produces misleading content on request, at volume.
• Automated accounts use the organisation's product to generate posts or reviews.
• A fabricated announcement or video of an executive is circulated.
• Generated content carries no marking, so its origin cannot be shown.

Why it matters

An organisation whose tool produced a disinformation campaign faces questions about its safeguards. One that is impersonated may see its share price, customers or staff affected before it can respond. Providers of the most capable models carry specific duties under the EU AI Act to assess and mitigate such risks.

Warning signs

• Usage shows high-volume generation of near-identical persuasive text.
• No monitoring exists for impersonation of the organisation or its leaders.
• The crisis communications plan does not cover fabricated media.
• Generated content from the organisation's tools is unmarked.