Data Poisoning

Data poisoning is an attack that corrupts a model by corrupting what it learns from. An adversary plants malicious examples in training data — scraped web pages, an open dataset, user-submitted feedback, or the documents a RAG system ingests — so the finished model misbehaves in ways the attacker chose. Variants include availability attacks (degrade accuracy broadly), targeted attacks (flip predictions for specific inputs), and backdoors, where a hidden trigger phrase activates planted behavior while the model looks normal on every benchmark. Research has shown poisoning can succeed with a surprisingly small absolute number of documents, and fine-tuning on user data or ingesting third-party content are the common exposure points for SaaS teams. Defenses are layered rather than absolute: provenance tracking for training sources, anomaly filtering before training, holding out clean eval sets, and treating any user-influenced corpus — including your RAG index — as untrusted input.