Contents
In brief
Tokens, context, sampling and evaluation, for people who have to put a model behind an interface and live with the result.
Original CloudExchangers explainer. It describes established public practice and labels the desk’s judgment as analysis. It is not a report of a newly announced event.
A model consumes tokens, not paragraphs
A large language model does not read a document the way a person does. Text is split into tokens, and the model predicts a distribution over the next token. Attention Is All You Need showed how a transformer can relate positions in that sequence without the step-by-step bottleneck of earlier recurrent networks. The practical consequence is a context window: a hard budget of tokens for instructions, source material, and the answer.
Engineers feel that budget as a design constraint. A system prompt, three retrieved passages, a user question and a growing chat can exhaust it. When they do, something is dropped, often the thing you assumed was “in the prompt.” Truncation is a product bug even when the model call succeeds.
Sampling is not a truth setting
Temperature and related sampling controls change how sharply the model picks a likely token. Lower values tend to repeat the more probable continuation. They do not switch the model into a factual mode. A deterministic sample can still be wrong. A higher-temperature sample can still be right. If you need a fact, retrieve it or compute it. Do not turn a dial and call the result verified.
The same applies to a confident tone. The model is trained to produce fluent answers. Fluency is a property of the text, not a confidence interval.
Prompting is interface design
A prompt is the part of the program written in natural language. Treat it like an interface. Say what the output must contain, what it must not invent, and what to do when the input is insufficient. Give one or two examples if the format is easy to miss. Ask for a structured object when the next step is code, because a paragraph is a poor API.
Do not hide a policy in a paragraph of encouragement. “Be accurate” does not create a source. “If the passages do not support a number, write that the number is not in the source” is an instruction a later test can check.
Retrieval is a different system
Lewis and colleagues described retrieval-augmented generation: fetch relevant text, then condition the model on it. That pattern is how most enterprise assistants should meet private knowledge. It is not the same as fine-tuning, which changes weights, and it is not the same as hoping the base model memorized your wiki. Retrieval can still fetch the wrong passage. The model can still ignore the right one. Evaluation has to look at both stages. A companion piece on this desk walks through that test: Evaluate a retrieval pipeline before you trust it.
How to evaluate without a leaderboard
Public leaderboards compare models on someone else’s tasks. Your questions are different. Build a small set you can rerun: questions with known answers from your corpus, questions that should be refused, and questions that include a trap such as a number that is not in the source. Score three things. Did the system answer from the provided material? Did it invent a citation or a figure? Did a person accept the answer for the job it is supposed to do?
Run the set when you change the model, the prompt, the chunking, or the tools. A demo is not a regression test.
Analysis
The engineering unit of an LLM product is not the model name. It is the model plus the context you feed it, the tools you allow, the test set you refuse to delete, and the person who owns a wrong answer. Swap any one of those and you have a different system.
What the papers do not settle
The transformer paper and the retrieval paper explain mechanisms. They do not rank this year’s hosted models, and this desk will not invent a ranking. Measure the candidate on your tasks, under your data-handling rules.