Text Generation Temperature

Randomness

Temperature is only applied to output logit softmax

When T=0, it selects the token with highest probability (deterministic, greedy decoding)

When T approaches infinity, probability distribution becomes uniform, increasing randomness

Even with temperature = 0, you can still see nondeterminism in LLM inference because of batching. When the batch size changes, the GPU kernel may evaluate the same mathematical expression in a different order. For example, batch size 1 vs. batch size 2048 can lead to different tile sizes, different reduction orders, different Tensor Core instruction choices, and different parallelization strategies.

The main reason is not simply “GPU threads finish in random order so atomic-add ordering changes”; more precisely, the reduction strategy selected by the kernel itself can change with batch size. Because floating‑point arithmetic is not associative, changing the computation order can change results at the bit level, which can then cascade into different generated tokens.

RMSNorm, matmul, and attention all involve reductions. If the reduction order depends on the batch size, batch invariance breaks. The solution is to use a batch-invariant kernel (i.e., enforce a reduction strategy that does not change with batch size).

Defeating Nondeterminism in LLM Inference

Reproducibility is a bedrock of scientific progress. However, it’s remarkably difficult to get reproducible results out of large language models. For example, you might observe that asking ChatGPT the same question multiple times provides different results. This by itself is not surprising, since getting a result from a language model involves “sampling”, a process that converts the language model’s output into a probability distribution and probabilistically selects a token. What might be more surprising is that even when we adjust the temperature down to 0This means that the LLM always chooses the highest probability token, which is called greedy sampling. (thus making the sampling theoretically deterministic), LLM APIs are still not deterministic in practice (see past discussions here, here, or here). Even when running inference on your own hardware with an OSS inference library like vLLM or SGLang, sampling still isn’t deterministic (see here or here).