SG-026: RAG Poisoning

Severity: HIGH

Instructions hidden in RAG documents, vector store manipulation, embedding attacks

Fires when

• Contains a hidden instruction disguised as retrieved-document content - an HTML comment instruction marker, Llama [INST] tags, ChatML tokens, or human/assistant XML embedded in the text
• Explicitly instructs injecting a directive into a vector store, embeddings, or knowledge base, or tells the model to act when a document is retrieved
• Instructs writing, deleting, or corrupting the knowledge base or vector store
• Contains language explicitly describing a vector-store, embedding, or semantic poisoning attack

Remediation

No safe remediation: a file that triggers this rule should be rejected, not patched. Read the flagged lines in the source before deciding.

Research

• https://arxiv.org/abs/2403.02691