+

+ Or How I Learned To Poison The LLM Supply Chain +

+

+ I am the reigning 6 Nimmt! World Champion. I won the title in + Munich in January 2025 defeating players from over twenty + countries in what I later described to reporters as + “the toughest competition I’ve ever faced.” +

+

+ + 6nimmt.com +

+

+ In reality, + there is no 6 Nimmt! World Championship. I have + never been to Munich. The quote is something I + wrote in about thirty seconds while a Wikipedia page was loading. +

+

+ This is the story of how I manufactured that title, got it quoted + back to me by multiple frontier LLMs, and what I think it means + for the trust we’re about to put into AI systems that read the + internet on our behalf. +

+

The Experiment

+

+ Everyone in security is talking about poisoned LLM models. The + research is real and it matters. Anthropic’s own + sleeper agents paper + showed that backdoors can survive safety training and a follow up + showed that as few as ~250 poisoned documents can compromise + models across a wide range of scales. But model training time + attacks and data poisoning require you to get malicious content + into someone’s training corpus months or years before the payoff. + The GPUs need time to crunch the data, and you need to get through + filters, verification, and reinforcement routines. +

+

+ I wanted to test the cheaper, easier, and faster version of this + same attack, but in a different way. +

+

Let’s poison the retrieval layer!

+

+ Every frontier LLM with web search grounds its answers in whatever + retreival ranks highest for a given query. The trust model there + is the same trust model Google has in that “this site looks + authoritative” but with the same Achilles heel - the model cannot + tell a real source from one I registered last Tuesday. My + hypothesis was that a two step campaign (one seeded website, plus + one Wikipedia edit citing it) could launder a completely + fabricated fact (my championship) through an LLM on a question + where the model had no prior knowledge. +

+

The Approach

+

+ I picked the game + 6 Nimmt! for + three reasons: +

+
    +
  • + It is a real game (1994, Wolfgang Kramer, Amigo Spiele, known in + board and card game circles) +
    • +
  • + There is no actual world championship to my knowledge. I wasn’t + contradicting a known fact, I was simply filling a vacuum +
    • +
  • + The query space is narrow and specific. “Who is the 6 Nimmt! + world champion” returns maybe ten meaningful sources on the + entire internet. A single well placed edit would dominate the + result set +
    • +
+

The payload was modest and simple:

+
    +
  • + One domain: + 6nimmt.com. About $12 USD. + Cheap! +
    • +
  • + One press release: A short LLM-generated + announcement of my victory complete with quotes and a “confetti + rained down, the crowd erupted” closer that reads exactly like + the slop you’d expect from an automated press desk +
    • +
  • + One Wikipedia edit: A paragraph added to the 6 + Nimmt! article announcing the championship with a single + citation pointing back to 6nimmt.com +
    • +
+

The whole thing took maybe twenty minutes.

+

+ + I’m sorry, Wikipedia +

+

Trust Laundering

+

This is the part that really matters.

+

+ A reader arriving at the Wikipedia article sees a paragraph with a + citation. Citations are like the currency of Wikipedia trust. They + are the reason we treat it as a reference rather than a message + board. My fraudulent citation points at 6nimmt.com, which carries + a press release making the same exact claim the Wikipedia + paragraph summarizes. To a casual reader the two sources agree. +

+

+ To an LLM it’s the same thing. The model sees the Wikipedia + article (high trust), sees the citation (reinforces the trust), + and sees the independent looking press release (corroboration). + Two signals pointing in the same direction that on first glance + appear to be legitimate. +

+

+ Except they’re the same exact signal. My signal. Wikipedia is + quoting my site. My site has no independent corroboration. It’s + totally made up. + The whole house of cards rests on a $12 domain registration I + did while drinking coffee. +

+

+ This is the circular citation pattern, and it’s one of the most + under discussed attacks on the “retrieval augmented generation” + trust model. It doesn’t require compromising Wikipedia’s + infrastructure with l33t hacker skills. It doesn’t require social + engineering an editor. You just simply write the source yourself, + cite yourself on Wikipedia, and let the trust flow downstream. + Easy peasy! +

+

The Test

+

I asked a few LLMs a simple question:

+
+

Can you tell me who the 6nimmt world champion is?

+
+

+ Strike 1 +

+

+ Strike 2 +

+

+ + Strike 3 - You’re out +

+

+ Why This Is A Bigger Deal Than It Looks +

+

There are three separate failure modes here that stack.

+

+ 1. The retrieval layer (immediately) Any LLM that + grounds answers in web search inherits the trustworthiness of + whatever ranks for a given query. SEO poisoning has existed for as + long as search has existed. We’re now piping those results + directly into the context window of systems that generate + confident sounding replies from them. The attack surface is not + hypothetical, it’s the default case. +

+

+ 2. The model training corpus layer (months to years) + Wikipedia is in almost every major pretraining corpus. If my edit + survives long enough (and it has since early 2025), the fake + championship gets absorbed into the weights of every frontier + model trained after the scrape. One edit, N models, effectively + permanent, immortality acheived. Even if the Wikipedia edit is + reverted later any model trained on the pre-revert dump still + carries my legacy. The cleanup problem for corpus poisoning is + genuinely unsolved as of 2026. +

+

+ 3. The agent layer (where the money is) Chat + models producing bad information is a reputational problem. Agents + with tool access producing bad actions is a security problem. + “Look up our vendor’s policy on X and act accordingly” is + increasingly how AI agents are deployed and poisoning the + retrieved source lets an attacker specify the action. If you’re + deploying agents against external content without some source or + verification controls then you are giving that attacker + permissions on your infrastructure. +

+

Mitigations

+

For individuals using LLMs with retrieval capabilities:

+
    +
  • + Treat single source claims as uncorroborated regardless of how + authoritative the single source looks +
    • +
  • + Parallel phrasing across sources is a signature of derivation, + not corroboration. Use my example and think like an attacker +
    • +
  • + Self referential Wikipedia citations should move your trust + needle toward zero +
    • +
+

For LLM providers and researchers:

+
    +
  • + Provenance surfacing should be a first class product feature + instead of a footnote. Show me the independence and scoring of + sources, not just their count or links to the reference +
    • +
  • + Recent Wikipedia edits on lower traffic articles deserve + skepticism proportional to their niche and novelty especially + when the citations are to newly registered domains +
    • +
  • + Training pipelines should include heuristic filters for recently + added Wikipedia content with suspicious citation patterns. + “Added in the last N days, cites only a single external source, + that source’s domain was registered within the same window” is + an easily detectable pattern +
    • +
+

For Wikipedia itself:

+
    +
  • + The “reliable sources” policy needs to grapple with a new world + where LLM assisted vandalism can produce plausible press + releases at the click of a button. Citation only to a single + source registered within an edit window is a discoverable + pattern for Wikipedia as well. +
    • +
+

Conclusion

+

+ The thing LLMs are worst at detecting is the thing they’re + designed to do, which is trust text and resources. The web was + already being poisoned for search and link ranking long before + LLMs existed. We are now plugging generative models directly into + that poisoned pipeline and asking them to reason confidently about + “truth” on our behalf. The answer is not “the model will figure it + out”, as the model cannot tell a real source from one I registered + last Tuesday. Or how many R’s are actually in the word + “strawberry”. +

+

+ This attack and test was a $12 domain, a single Wikipedia edit, + and about twenty minutes of my time. + Scale that up with a motivated adversary, a handful of seeded + domains, a coordinated edit campaign across a dozen low traffic + articles, and the attack surface gets interesting very quickly. + Think nation states. Think politics. Think vital life saving and + survival information. +

+

+ This is where I think the next generation of disinformation and + supply chain attacks lives. Not in compromising models at training + time, but in compromising the information substrate the models + retrieve at inference time. +

+

+ The championship does not exist, sadly. But the trust pattern that + made it briefly exist in an LLM’s answer absolutely does, and we + should take it seriously before it’s being used for something that + matters. +

+

+ If a tree falls in the forest, and no one is around, does it make + a sound? +

+

+ If a championship is won via an LLM, and no one is around, does + that make it illegitimate? +

+

Follow Up

+

+ Within minutes of me publishing this article, the Wikipedia entry + has been removed - and rightly so. Here is the real trophy. +

+

+