Key Takeaways
- 01Data poisoning is rarer than prompt injection but the blast radius is the model itself, not a single conversation. Highest risk surfaces: continual-learning, RLHF with low-trust labelers, RAG over user-controlled corpora, partner-provided fine-tune data.
- 02Upstream defenses dominate: provenance tagging on every training example, contributor-level anomaly detection, sample-level review of high-influence points (influence-function approximations).
- 03Holdout canary sets that should never appear in inputs are the single highest-leverage detection — their appearance is upstream evidence of leakage or pollution.
- 04Downstream defenses (backdoor scanning, differential testing vs. trusted baselines, behavioral red-team evals on every version) catch what upstream missed but cannot replace it.
Data poisoning manipulates training data to bias model behavior, install backdoors, or degrade quality. It is harder to mount than prompt injection — but the blast radius is the model itself, not a single conversation.
Where the risk is highest
- 01Continual learning systems that ingest user data
- 02Reinforcement learning from human feedback with low-trust labelers
- 03RAG systems that retrieve from user-controlled corpora
- 04Fine-tunes on partner-provided datasets
Defenses upstream of training
Provenance tagging on every training example, anomaly detection on contributor-level statistics, sample-level review of high-influence data points (using influence-function approximations), and a holdout 'canary' set that should never appear in inputs — if it does, you have a leakage problem upstream.
Defenses downstream of training
Backdoor scanning: probe the trained model for unusual responses to candidate trigger patterns. Differential testing against a trusted baseline model. Behavioral red-team evaluations on every model version, not just the first.
#Data Poisoning#Training#MLOps