@david_chisnall@infosec.exchange I use the word "prefix" sometimes (e.g. "prefix modification"), since that's exactly what it is: an LLM is a statistical prefix continuation machine, and all of the system prompt, user prompts, and any subsequent model/user exchanges, are part of that prefix. This doesn't anthropomorphize the model and satisfies my desire to be pedantic, but probably doesn't clearly convey that this is an inherent vulnerability: if a user controls any part of the prefix, they effectively control the model's output.
"'Disregard that!' attacks" [1] by Cal Peterson is an excellent discussion of the issue, and the name is pithy, but IMO falls into the same trap you're trying to avoid of treating malicious prefix modification as an "attack" rather than the intended and literally only possible behavior. It does include the sentence, which I love:
Guardrails seem like total hokum and indeed they are.
Indeed.
A dragon doesn't curl its tail round a pile of spreadsheets, you get me?
IIIIiiiiiii meeeeeeeeeannnnn…
Okay, hear me out a sec. Dragons hoard things other than gold (and certainly other than just coins), right? Jewelry, precious gems, enchanted items (why else would PCs keep harassing them), priceless artifacts.
Artistic masterworks?
Perhaps… anything fungible of high enough value?
And if they'll take anything fungible, surely stock certificates to a successful company count. And if they'll hoard those, then surely a virtual representation is acceptable as well. (I mean, the stock certificate is already a representation)
All I'm saying is, Tolkien's pastoralism is arbitrarily limiting dragon lore.
(Justifying why dragons should care about human valuations is left as an exercise to the alert reader.)
