Purrx

Temperature, top-p and getting repeatable output

Temperature, topP and topK decide how the next token is picked. Learn what each knob does, which to touch, and how to make model output as repeatable as it gets.

9 min+25 XPHands-on

At each step the model gives you a list of possible next tokens, each with a chance (a ). Something has to pick one actual token from that list. That step is , and a few settings control how it picks. They are widely misunderstood, so let's be precise.

Temperature

Temperature changes the chances before a token is picked. A low temperature makes the likeliest token even more likely, so the model plays it safe. A high temperature evens the chances out, so unlikely tokens get picked more often.

temperature 0.1

almost always Paris

  • Paris99.9%
  • the~0%
  • situated~0%
  • anything else~0%

temperature 1.0

occasionally something else

  • Paris94%
  • the2%
  • situated1%
  • anything else3%

temperature 2.0

often picks something odd

  • Paris55%
  • the8%
  • situated6%
  • anything else31%
Same model, same prompt. Temperature reshapes the chances before one token is picked.
  • 0 – 0.3: extraction, classification, code, anything graded against a right answer.
  • 0.7 – 1.0: general assistant work, explanations, drafting.
  • Above 1.2: brainstorming and deliberate variety. Answers also start making less sense.
Under the hood — What is a "probability distribution" here?

At each step the model produces one raw score for every token in its vocabulary — a hundred thousand-odd numbers, on any scale, positive or negative. Those raw scores are called , and they are not yet probabilities.

A function called converts them: it exponentiates each score and divides by the total, which forces every value between 0 and 1 and makes them sum to exactly 1. Now they are a probability distribution, and "92% likely to be Paris" means something.

Temperature slots in just before that conversion: every logit is divided by it. Dividing by a small number (0.1) spreads the scores far apart, so softmax turns the leader into a near-certainty. Dividing by a large number (2.0) squashes them together, so unlikely tokens get a real chance. That is the entire mechanism behind the knob.

topP and topK

These shorten the list of tokens before one is picked:

  • topK keeps only the K most likely tokens.
  • topP keeps the most likely tokens until their chances add up to P (for example 0.95). When the model is confident, that might be two or three tokens; when it is unsure, dozens.

In practice, you will rarely need either.

Determinism, honestly

temperature: 0 means "always take the most likely token". The same prompt then usually gives the same answer, but not always. Tiny rounding differences between servers and quiet model updates can still change the output. Treat it as "very likely the same", never "guaranteed the same". In other words, model output is never fully .

Also worth knowing: maxOutputTokens

This isn't a sampling setting, but it goes in the same config object and often surprises people. It caps how long the answer can be. When the cap is hit, the answer stops mid-sentence and finishReason is "MAX_TOKENS". If a reply looks cut off, check this before blaming the prompt.

where the knobs go
await gemini.generateContent({
  contents: "Name three uses for a paperclip.",
  config: {
    temperature: 0.2,
    maxOutputTokens: 100,
    topP: 0.95,      // usually leave alone
  },
});

Key takeaways

  • Temperature controls randomness: near 0 for exact tasks, around 1 for open-ended writing.
  • Leave topP and topK at their defaults unless you have a specific reason to change them.
  • Even temperature 0 doesn't guarantee the same answer, so never test for exact model text.

Sign in to run the exercise

Reading is free. Writing code here needs an account so we have somewhere to keep your Gemini key and the +25 XP you are about to earn.

Continue with Google