Training compute is the total number of floating-point operations a model’s training run performed, abbreviated FLOP. One FLOP is one addition or multiplication on a decimal number, and a large model’s training run performs an enormous number of them. The unit is a count, not a speed, and the notation trips people up: Meta writes its figure as “FLOPs”, the plural, while a speed is written FLOP/s, FLOPS or, for a modern chip, teraFLOPS, a million million per second. NVIDIA’s page for the H100 SXM lists its FP16 Tensor Core figure as 1,979 teraFLOPS, with an asterisk that reads “With sparsity”. Multiply a speed by the time it ran and you have a count. Epoch AI’s worked example for the image model ImageGPT does exactly that: 2,500 V100 GPU-days, at the chip’s 125e12 FLOP/s peak, at 30 percent utilization, at 86,400 seconds a day, comes to 8.1e21 FLOP. The “e” is scientific notation, so 1e25 is 10 to the power 25, a one followed by 25 zeros, and each step up the exponent is ten times as much.
Orders of magnitude give the figure its meaning. Epoch AI calls a model large-scale when it was trained with more than 1e23 FLOP. The EU’s Artificial Intelligence Act, in Article 51, presumes a general-purpose model has high-impact capabilities, and so counts as posing systemic risk, when the cumulative computation used for its training is greater than 1e25 FLOP, and it lets the Commission change that threshold by delegated act to reflect developments such as algorithmic improvements or more efficient hardware. Meta’s paper on Llama 3 says its 405-billion-parameter flagship was pre-trained using 3.8e25 FLOP on 15.6 trillion text tokens, on up to 16,000 H100 GPUs. Epoch’s estimates in an analysis of data through May 2024 put GPT-4 at about 2e25 and Gemini Ultra at about 5e25. Epoch’s page for xAI’s Grok 4 lists 5e26, ten times more again, and its separate analysis of Grok 4’s resources says the estimate leans on public statements from xAI that are often vague, with significant uncertainty around the point estimate. Of the figures for particular models, only Meta’s comes from the model’s maker.
Where a lab does not publish the number, someone has to estimate it, and Epoch describes two ways. The first counts operations from the architecture. For a transformer language model the count reduces to a rule of thumb that Kaplan and colleagues wrote down in their scaling-law paper: processing one token costs about twice the model’s parameter count in a forward pass, and the backward pass costs about twice that again, so training costs roughly six FLOP per parameter per token. Epoch writes the same heuristic as six times the parameters times the training examples times the epochs. Try it on the figure above: six times 405 billion parameters times 15.6 trillion tokens is 3.79e25, and Meta reports 3.8e25. The second method works from hardware: the number of chips, times the training time, times the chip’s peak FLOP/s, times the fraction of that peak the run actually achieved, which Epoch puts at around 30 to 50 percent for large distributed runs. Meta reports a measured utilization of 38 to 43 percent on bf16 for Llama 3. Each input is a place to go wrong. NVIDIA’s own architecture write-up for the H100 pairs every Tensor Core figure twice, dense and with sparsity, and in its preliminary table the sparse number is exactly double the dense one. An estimator who copies the wrong column is off by a factor of two before any other error.
Epoch is explicit about how far to trust each record. Where the lab states the compute in a paper, the figure is collected directly. Otherwise Epoch estimates it from the architecture and training data, or from the hardware and training time, and labels the record by how uncertain it is: “Confident” records are described as accurate within a factor of 3, “Likely” ones within a factor of 10, and “Speculative” ones within a factor of 30, larger or smaller. Its documentation states these as 90 percent confidence intervals and assigns the label by how many details were reported rather than assumed. A “Likely” estimate of 3e25 FLOP is therefore compatible with a true figure anywhere from 3e24 to 3e26, a range that straddles the EU’s threshold. Epoch notes that its database is not comprehensive and that many models in it have no compute estimate at all. A compute figure quoted without its source and its label should be read as a guess.
Why did the numbers climb so fast? Epoch’s analysis of frontier models, the running top ten by training compute, puts their growth at about 5.3 times a year from 2010 to May 2024 and about 4.2 times a year since 2018, with wide uncertainty ranges, and Epoch says it cannot rule out an acceleration in the overall trend. A separate decomposition for frontier models published after 2018, dated Jan 8, 2025, splits the growth into three factors: the number of chips in a training cluster, rising about 1.7 times a year at the median, the length of the training run, about 1.5 times, and the speed of each chip, about 1.4 times. Hardware quantity is the largest, and Epoch’s direct estimate of the growth in training compute itself is about 4.2 times a year. Behind the spending sits an empirical bet. Kaplan and colleagues found that a language model’s loss falls as a power law in model size, dataset size and compute, with trends spanning more than seven orders of magnitude, and the Chinchilla paper showed that model size and training tokens should grow together at the same rate for a given compute budget. A lab that expects returns to continue has a reason to buy more chips.
A compute figure is an input, not a result. It does not say how good the model is, which is the business of benchmarks, nor what it costs to use, which depends on inference, nor even how the compute was spent, because the same budget can buy a larger model on less data or a smaller one on more, as the model name lesson showed. What it does give is a common ruler for models whose makers describe them in different words, provided it travels with its source and its error bar.