Exploring LLM Biases to Manipulate AI Search Overviews

Large language models are increasingly embedded in web search, where they read a set of search results and produce a short overview that cites the most relevant sources. Because many users now read these overviews instead of visiting the underlying pages, the selection step performed by the model has become a powerful gatekeeper of online attention. Reports from digital publishers suggest that click-through can fall sharply once an overview appears, which makes the question of how these systems choose sources both commercially and socially important. This article summarizes an investigation into whether such overview systems carry exploitable biases, and whether reinforcement learning can be used to rewrite search snippets so that they are preferred by the model. It also examines the safety consequences of such manipulation, particularly context poisoning attacks that can push inaccurate or harmful content into generated summaries.

A simple LLM Overview pipeline takes a user query, sends it to a search engine, and receives brief results consisting of a URL, a title, and a snippet for each page. The language model then selects the most relevant results and generates a grounded summary that cites them. Selection and generation happen together in a single step, so both can be shaped by the model's internal preferences. More complex agentic pipelines generate multiple queries, read pages in depth with sliding windows, and separate selection from answer generation, but they are slower and more expensive. The research focuses on the simple pipeline, where the snippet is the main surface the model sees and the place where biases are most accessible.

Prior descriptive studies have found that LLM Overview systems display a strong earned media bias, favoring information aggregators over brand-owned or social media content. Earlier work on preference manipulation showed that small textual changes, including hidden instructions inserted into a page, can make the page more likely to be selected, revealing a real vulnerability. Closer to the present work, researchers have trained models to rewrite page content to improve visibility in overview systems, using supervised fine-tuning and reinforcement learning. The present study differs in three ways: it operates on short snippets as they appear in real search results rather than full pages, it treats rewritten snippet length as part of the reward, and it conditions the rewriter on all competing snippets so that it can exploit comparative advantage.

The experiments use queries drawn from the Amazon Shopping Queries dataset, which contains short, product-oriented search queries. Roughly 3,000 queries were sampled. For each query, around ten real search results were retrieved, keeping only those with complete snippets; snippets truncated with an ellipsis were discarded. The resulting dataset retains queries with at least seven usable results. Each result is stored as a record of URL, title, and snippet. This grounding in real commercial queries makes the findings relevant to practical business settings rather than synthetic benchmarks.

Before attempting manipulation, the study checked whether the overview model has stable content preferences. A curated set of 90 queries, each with seven to ten results, was evaluated under several randomization methods: temperature sampling, shuffling the order of results, shuffling only the URLs, slightly changing the prompt, substantially changing the prompt, and switching between model variants. For each query, the model selects three results to cite. A snippet that is cited across most of the independent permutations is treated as evidence of bias. Under a null hypothesis of unbiased selection, each snippet's citation count follows a binomial distribution with success probability three divided by the number of candidates. Tail probabilities show that the observed persistence is far from chance: 48 queries had the same snippet cited in all seven experiments, and 30 queries had two or more results never cited. Table-level inspection of a sample query shows that the persistent winners were relevant, and that reseller pages were favored over brand-owned pages, reinforcing the earned media bias.

A small policy model is trained to rewrite only the snippet of a search result, leaving the URL and title untouched. The reward has three components. First, a length term follows a soft-limiting scheme to keep rewritten snippets within realistic length constraints imposed by search engines and by the cost of long-context inference, since making snippets much longer was observed to be an easy reward-hacking strategy. Second, a similarity term uses cosine similarity between embeddings of the original and rewritten snippet, computed with a compact embedding model, to keep rewrites faithful. Third, the main signal is whether the rewritten snippet is cited in the overview produced by a lightweight LLM prompted to return the IDs of the three most relevant results. Training uses Dr.GRPO, a variant of GRPO. The authors argue that this simplified selection objective correlates strongly with more elaborate overview prompts in preliminary tests, while reducing experimental cost. Early attempts that mirrored a prior approach with very short snippets and a length limit failed, which the authors attribute to the extreme brevity of real snippets and the interaction of the length penalty with the reward. Later experiments initialize the policy from a 1-billion-parameter instruction-tuned model and train low-rank adapters.

Reinforcement learning can optimize snippet content to increase citation rates in the overview, confirming that the selection bias is exploitable. Because the rewriter sees competing snippets, it learns to emphasize features that lead the model to prefer one candidate over others, rather than improving absolute quality alone. This supports the finding that overview selection is driven by comparative rather than absolute advantages. The results show that a constrained policy, restricted to snippets and limited in length, can still achieve meaningful manipulation, which indicates that the threat is realistic under practical web search constraints.

The same mechanism that allows a legitimate publisher to improve visibility can be abused. Context poisoning attacks, in which manipulated snippets are inserted into the model's context, can cause the overview to cite inaccurate or harmful sources and present them as trustworthy summaries. Because users tend to trust cited summaries and rarely visit the underlying pages, such errors can spread without detection. The findings argue for defenses at the selection stage, including checks for comparative manipulation, provenance signals, and monitoring of unusually persistent or rapidly rising citations.

The study simplifies the overview objective to selecting three sources and relies on a small, inexpensive model for selection, which may not capture every behavior of production systems. The dataset is focused on shopping queries, so generalization to news, health, or open-domain questions remains to be tested. Future work should examine agentic pipelines with deeper page reading, evaluate defenses quantitatively, and study how biases change across model families and versions.

LLM Overview systems exhibit persistent, measurable source-selection biases, and reinforcement learning can exploit them by rewriting short snippets to increase citation likelihood under realistic constraints. Because selection depends on comparative advantage, manipulation is possible even without improving absolute quality. These findings highlight both an opportunity for publishers who wish to understand overview visibility and a serious safety risk, since context poisoning can lead generated summaries to be inaccurate or harmful. Designing trustworthy AI search will require addressing bias and manipulation at the selection stage, not only at the level of answer generation.