Risk · R-083 · Model Performance & Reliability
Self-reinforcing feedback loops
A model's outputs shape the data it is later trained or evaluated on, entrenching early errors and narrowing outcomes over time.
- Severity
- Medium
- Audience
- Heads-of · Practitioner
- Frameworks
- Article 10 · Article 15 · Harmful Bias
- Responsible AI domain
- Lifecycle stage
- Reviewed
- 2026-11-10
What it is
When a model's decisions determine what data is collected next, the model begins to learn from itself. A lender only observes repayment for the applicants it accepted. A recommender only records responses to what it showed. A model retrained on content that models wrote learns their habits. In each case the data confirms what the model already did, and alternatives are never tested.
How it happens
- Outcomes are recorded only for cases the model approved.
- Staff corrections are fed back, and staff mostly accept the model's suggestion.
- Model-generated content enters the training data for the next version.
- Performance is measured against labels the model influenced.
Why it matters
Errors and biases present at the start are reinforced with each cycle, while measured performance appears stable or improves. Groups the model disfavoured early generate no evidence that could correct it.
Warning signs
- The range of outcomes narrows across retraining cycles.
- No data is collected on cases the model rejected.
- Retraining data is not checked for the model's own influence.
- Measured accuracy rises while complaints or losses do not fall.
Mitigating controls
Primary controls
- Data quality and representativeness checks Detective
- Checks retraining data for the model's own influence. Training, test and reference data are assessed for accuracy, completeness, currency and coverage of the population the system will serve, before use and at each refresh.
- Drift and performance monitoring Detective
- Monitors outcome distributions for narrowing over time. Input distributions and output quality are measured in production against thresholds that trigger investigation, retraining or withdrawal.
Supporting controls
- Bias testing and fairness monitoring Detective
- Shows whether the loop is entrenching group differences. A system's performance and outcomes are measured separately for each relevant group before release and on a set cycle in production, against thresholds that trigger investigation.
Framework and clause references
| Framework | Clause | Title |
|---|---|---|
| EU AI Act | Article 10 | Data and data governance Data governance. |
| EU AI Act | Article 15 | Accuracy, robustness and cybersecurity Names feedback loops for systems that keep learning. |
| NIST Generative AI Profile (NIST AI 600-1) | Harmful Bias | Harmful Bias or Homogenization Homogenisation. |
Sources
- EU AI Act: Regulation (EU) 2024/1689 European Union (EUR-Lex) · official text
- NIST Generative AI Profile NIST · framework
Related terms
How to cite
Axiom Verity (2026). Self-reinforcing feedback loops. Reference Library, R-083. https://axiom-verity.com/risks/self-reinforcing-feedback-loops. Reviewed 2026-11-10.
Found an error or a better source? Report a correction.