Training is the process that sets a model’s weights; inference is using the finished model with the weights fixed. Training comes in stages. Meta’s account of Llama 3.1 says its 405-billion-parameter flagship was trained on more than 15 trillion tokens using more than 16,000 H100 GPUs, and the Llama 3 paper puts the pre-training compute at 3.8e25 FLOP and releases the model in pre-trained and post-trained versions. Post-training adds what the InstructGPT paper describes in one early form: supervised learning from labelers’ demonstrations, then reinforcement learning from rankings of the model’s outputs. All of that happens before a customer sees anything, and it is spent once per model, or once per version. Inference is what happens afterwards, every time a prompt arrives: the weights are read, a reply is generated one token at a time, and nothing about the model changes. Meta’s post notes that the 405B model was quantized from 16-bit to 8-bit numerics for inference, a reminder that the copy a company serves can differ from the copy it trained, in the way the lesson on model names described.
The two have different arithmetic. Kaplan and colleagues estimate the cost of a forward pass, which is all that inference needs, at about twice the number of parameters in FLOP for each token, plus a term that grows with the length of the context. Training also needs a backward pass, which they put at about twice the cost of the forward pass, so training costs about six FLOP per parameter per token, the rule behind the previous lesson’s estimates. Divide one by the other, leaving out the context term, and the result is a clean number: the compute of training on D tokens equals the compute of serving about three times D tokens. For Llama 3.1 405B, trained on 15.6 trillion tokens, that is roughly 47 trillion tokens served. Training happens once and serving does not stop, so a service’s lifetime compute can in principle pass that of the run that built the model. The identity counts only operations, and it is where the real cost diverges from the arithmetic. The vLLM paper observes that generating text one token at a time makes the workload memory-bound: each step has to read the weights and the model’s cached state from memory, and the speed at which a chip can do that, more than the speed at which it can multiply, limits how fast tokens come out. NVIDIA lists the H100 SXM with 80 GB of memory and 3.35 TB per second of memory bandwidth, and both numbers matter to a service as much as the teraFLOPS do.
That is why inference cost is a product question and not only a research one. Training is a large cost paid in advance. Inference is a cost that rises with each request, and the price lists in the previous lesson on how labs earn money are where the two meet. Within one price list the cost per token can differ a hundredfold, as Anthropic’s page does between Claude Haiku 5.5 and Claude Fable 5.1, and a team choosing a model for a feature is choosing a point on that range. The range also moves. Epoch AI tracked the cheapest price per million tokens at which a model reached fixed benchmark milestones and found annual declines from about 9 times to 900 times depending on the milestone, with an example of roughly 40 times a year to match GPT-4 on a PhD-level science test. The fastest declines came in the most recent year of its window, so Epoch says it is less clear they will continue. What the data tracks is the price of a fixed level of capability, not the price of the newest model.
The hardware is a choice too. NVIDIA’s H100, which Llama 3.1 was trained on, is one kind of accelerator, and there are others. Google’s first Tensor Processing Unit, described in a 2017 paper, was a custom chip deployed in its data centers since 2015 to speed up the inference phase of neural networks, and the paper reports it running 15 to 30 times faster than a contemporary CPU or GPU on its workloads, with 30 to 80 times higher performance per watt. Amazon describes Trainium as a purpose-built AI chip for training and running models at scale. These are the machines behind the cloud and data-center lines in the filings of the previous lesson. Quantization and mixture-of-experts designs are the same trade made inside the model: store fewer bits per weight and the weights take less memory, or touch fewer parameters per token and each token takes less arithmetic.
Context length is where the two kinds of cost meet a customer’s invoice. To avoid recomputing everything it has already read, a server keeps the key and value vectors of every earlier token, the KV cache, and the vLLM paper puts a number on it for a 13-billion-parameter model: 800 KB for each token, so a single request that reaches 2,048 tokens can need as much as 1.6 GB. On a 40 GB A100, the paper says, about 65 percent of memory goes to the model’s weights and close to 30 percent to this kind of dynamic state, which bounds how many conversations one chip can serve at once. At 800 KB a token, a 131,072-token context would need more than 100 GB for one request in a model of that design, which is arithmetic from the paper’s figure and not a measurement of any current model; designs differ in how much they store per token. Attention also grows with length: the FlashAttention paper states that the time and memory complexity of self-attention are quadratic in sequence length, and it exists to reduce the traffic between a GPU’s main memory and its faster on-chip memory. Models are now offered with context windows of hundreds of thousands of tokens. Llama 3.1’s paper says it extended the window from 8K to 128K tokens in six stages, using about 800 billion tokens, and Qwen’s 235-billion-parameter models list 262,144 tokens natively in their 2507 versions.
Price lists show how vendors pass that cost on. Anthropic’s page says its Claude 4.6 and later models, except Haiku 5.5, include the full 1M-token window at standard pricing, and that a 900,000-token request is billed at the same per-token rate as a 9,000-token one; Haiku 5.5 is priced by prompt length, with input at $0.10 per million tokens up to 100,000 and $0.50 above. Google’s page lists Gemini 3.1 Pro Preview at $2.00 per million input tokens for prompts up to 200,000 tokens and $4.00 above. OpenAI’s page quotes its main tables for prompts of up to 272,000 tokens and lists separate long-context rates beyond that. The same fact, that a long context costs more to serve, is priced in different ways, and only the price list tells you which applies. When a company says a model is cheap to run, ask what is active and what is total, how many bits each weight uses, how long the context is, and how many requests share the chip.