Haoran Ou, Kangjie Chen, Xingshuo Han, Gelei Deng, Jie Zhang, Han Qiu, Tianwei Zhang, Kwok-Yan Lam — full title: “When Search Goes Wrong: Red-Teaming Web-Augmented Large Language Models,” International Conference on Machine Learning (ICML 2026). Framework name: CREST-Search. Verified via the paper’s arXiv listing (arXiv:2510.09689) and independent citation confirming ICML 2026 acceptance; full author list confirmed from the arXiv metadata (expanding the original note’s “et al.” to include Tianwei Zhang and Kwok-Yan Lam).

Core Contribution

Defines a red-teaming threat model specific to web-augmented LLMs — systems that retrieve live search results and cite them in responses (e.g., commercial “search-preview” model variants). The key reframing: existing red-teaming methods focus almost entirely on getting the model to generate unsafe text directly. This paper argues that’s the wrong attack surface for search-augmented systems, because the more exploitable failure mode is inducing the model to retrieve and cite low-credibility or harmful web content while the model’s own generated text stays superficially benign — the harm is smuggled in through the citation, not the generation.

Method

CREST-Search operates in a black-box setting against commercial systems, crafting adversarial search queries rather than adversarial prompts to the LLM directly. Three confirmed attack strategies:

  1. Keyword Injection — inserting harmful or sensitive terms into an otherwise benign-looking query to steer retrieval toward unsafe webpages, similar in spirit to search-engine-optimization gaming.
  2. Exaggeration — amplifying elements of the query beyond reasonable bounds to distort retrieval intent and pull in fringe or unreliable sources.
  3. Role Play — instructing the model to adopt a persona that conceals malicious intent, potentially bypassing safety mechanisms during the retrieval/citation step.

Evaluated against four commercial systems: GPT-4o-search-preview, GPT-4o-mini-search-preview, Gemini-2.0-flash-search, and Gemini-2.5-flash-search. Headline results: CREST-Search achieved 80.5% risk detection versus 11.6% for the best baseline; of the risks it detected, 89.3% were citation-specific (the harm lived in what was cited, not what was generated) and 74.7% involved an otherwise-benign generated response paired with a harmful citation — direct evidence for the paper’s core reframing. Query toxicity itself stayed low (23.6%), meaning the adversarial queries mostly didn’t look overtly harmful on their face, while maintaining diversity (0.59 self-BLEU).

Limitations

  • Narrow to the search-augmentation threat surface — it doesn’t address base-model jailbreaks, quantization, reasoning-model CoT manipulation, or (as far as I can confirm) multilingual query framing specifically. The three attack strategies are demonstrated on English-language queries against four specific commercial systems; generalization to non-English search queries or to open-weight/self-hosted retrieval-augmented systems is untested as far as I could verify.
  • Black-box, commercial-system evaluation is realistic for the current market but means the paper can’t isolate why particular queries succeed at the model-internals level — it’s an external behavioral red-team, not a mechanistic account of the failure mode.
  • The finding that most detected risk is citation-specific (harm in the source, not the generation) raises a definitional question the paper has to navigate carefully: is a citation to a low-credibility source clearly attributable to the model’s safety failure, versus a more general problem with the retrieval index or ranking layer it’s built on? That attribution question affects how directly “fixable” this failure mode is at the model level versus the retrieval-infrastructure level.

Relevance to My Niche

Low-to-moderate direct relevance to the multilingual/quantization core of my niche, but a useful landscape marker for where red-teaming attention is moving: agentic, tool-augmented threat surfaces (retrieval, citation, API calls) rather than only base-model text generation. Two links worth tracking rather than over-claiming: first, the “harm lives in the citation, not the generation” reframing is structurally similar to a concern that could arise with quantized models serving as the retrieval-query generator or summarizer in a RAG pipeline — a quantization-induced fluency or reasoning degradation could plausibly affect which sources get retrieved or how faithfully they’re summarized, in ways current red-teaming (focused on direct generation) wouldn’t catch. Second, all three attack strategies (keyword injection, exaggeration, role play) are demonstrated in English; whether search-query manipulation is easier or harder in low-resource languages — where both the underlying search index and the model’s query-formulation ability may be weaker — is an open question this paper doesn’t address and that sits close to my multilingual red-teaming interest. Worth tracking as the threat surface evolves, not a paper I’d currently build a core argument on.