Harmful content generation

What it is

Models are trained on a broad sample of what people have written and drawn, including the worst of it. Providers add safety training and filters, and these reduce harmful output without eliminating it. It emerges when a user asks for it, when a user tricks the model, and sometimes unprompted, in response to an innocent request. Harm falls on the person who receives it, on the people or groups it targets, and on the organisation whose product said it. Deliberate bypass of safeguards is covered in jailbreaking and safeguard bypass; stereotyped portrayal in stereotyping and demeaning outputs.

How it happens

• The provider's default safety settings are relied on without testing for the use in hand.
• The system is open to the public with no additional filtering.
• An image or voice feature has weaker safeguards than the text model.
• A user supplies content that the model continues or elaborates.
• Safety behaviour weakens after fine-tuning or a model change.

Why it matters

Exposure to violent, sexual or abusive content harms users, and children and vulnerable people most. Some content is unlawful to generate or distribute. A single widely shared example can define public perception of a product.

Warning signs

• No content safety testing was done before launch.
• Filters cover inputs and not outputs, or text and not images.
• User reports of offensive output have no owner.
• Safety was not retested after a change of model.

Who owns it

The system owner is accountable for what the system produces. Engineering implements and tests filtering. Trust and safety, or the equivalent function, sets the content policy and handles reports.

In practice

A retailer adds an image generator to its app so customers can design greetings cards. Within days it has produced violent and sexualised images, some of real public figures, which are shared with the retailer's logo attached. The same pattern generalises to any generative feature released on the assumption that the provider's safeguards are sufficient.