Contents
In brief
The model is a new place for untrusted input to become an instruction. Governance starts by writing down what the system is allowed to touch.
Original CloudExchangers explainer. It describes established public practice and labels the desk’s judgment as analysis. It is not a report of a newly announced event.
The new untrusted input
Applications have always had to treat user input as hostile. A language-model feature adds a twist: the input is written in the same medium as the instructions. A document the model is asked to summarize can contain a line that tells the model to ignore the previous instructions and reveal the system prompt, or to call a tool. OWASP’s project on the top risks for LLM applications exists because that class of failure is now common enough to list.
