Content safety filtering

What it does

It enforces the organisation's content policy whatever the main model was persuaded to do. Because the filter is separate from the model, a prompt that defeats the model's own restrictions still has to pass it. The system owner sets the policy with compliance; engineering operates the filter on every input and output.

How to implement it

• Define the harm categories and thresholds that apply to this system and its users.
• Use a separate classifier, not an instruction to the main model.
• Screen both the request and the response.
• Set stricter thresholds where children or vulnerable people may be present.
• Log blocked content with the category for monitoring.
• Review false positives and negatives and tune regularly.

Evidence it operates

Design: a content policy with categories and thresholds, approved by the system owner. Operation: filter logs by category; test results against a labelled set of harmful and benign content, with error rates; evidence of periodic tuning.

Where it fits

Runs in line on every exchange. It is a primary control for jailbreaking and safeguard bypass and abuse of a deployed AI service, alongside input filtering.

Limits

Classifiers miss novel phrasing and material in languages or formats they handle poorly, and they block legitimate content, which matters where users discuss sensitive subjects for good reason. Category definitions embed judgements that need an accountable owner. A filter that blocks too much drives users to unapproved tools, so its error rates need the same attention as its coverage. Blocked requests should receive a clear explanation and, where appropriate, a route to a person.