Stereotyping and demeaning outputs

What it is

Generative models reflect the associations in their training data. Asked for a picture of a doctor or a cleaner, a story about a criminal, or a reference for a named person, they tend to return what was most common in that data, which includes its prejudices. The harm is representational: people are depicted in narrow or insulting ways, and those depictions are repeated in everything the model writes. Where the output is used in a decision, such as a summary of a candidate, it shades into discrimination.

How it happens

• Prompts are open-ended and the model fills in demographic detail.
• Names, dialect or other cues change the tone or content of the response.
• Image generation defaults to one group for a given role.
• Content filters address abuse and not subtler bias.

Why it matters

Published content of this kind damages reputation and offends the people depicted. Used internally, it can influence judgements about individuals in ways nobody intended. At scale, a widely used model narrows how groups are portrayed across much of what is written and drawn.

Warning signs

• Outputs change when only a name or pronoun is changed.
• Generated images for a role show one demographic.
• No test set probes for stereotyped responses.
• Users have reported offensive output.