The Effect of Sampling Temperature on Problem Solving in Large Language Models

In plain words

When a language model writes text, a setting called temperature controls how adventurous its word choices are. At a low setting it keeps to the words it finds most likely. At a higher setting it takes more chances, which can make the text more varied but also less reliable.

Developers often pick this setting by guesswork, because the advice they find is mostly anecdotal. The authors tested it systematically. They gave several well-known models multiple-choice questions drawn from standard benchmarks, tried five different prompt styles, and raised the setting step by step. From zero up to 1.0 the scores did not change in a statistically meaningful way. Above 1.0 the text became incoherent and the scores fell sharply.

Their practical advice is to use the lowest setting for problem-solving tasks, which also makes results easier to reproduce. The study covers only multiple-choice questions, so the advice may not carry over to open-ended tasks.

TL;DR

• Nine LLMs, five prompt styles and ten benchmark-derived question sets, with each question answered ten times per setting §2.1 §2.2 §2.3.
• Accuracy shows no statistically significant change from temperature 0.0 to 1.0 for GPT-3.5 on the 1,000-question exam (H(10)=10.439, p=0.403) §3.1.
• Above 1.0, text becomes incoherent and accuracy drops toward zero; the paper recommends 0.0 for problem-solving tasks §3.1 §4.1.
• Author-reported. The repo is BSD-2-Clause, and several code paths differ from the paper (see Paper vs code).

Problem

Advice on sampling temperature for problem-solving is largely anecdotal and rarely backed by experiments §1.3. Provider defaults differ between models and versions, and the reasons for them are not given §1.3, footnote. The question the paper asks is which temperature gives the best accuracy on problem-solving tasks, and whether that answer generalises across models, prompts and domains §1.3.

Method

Sampling temperature. Temperature divides each logit before the softmax converts them to probabilities eq. 1. Lower values make output more deterministic; higher values make it more varied §1.2.

# Temperature-scaled next-token distribution  [eq. 1]
# Input:  logits (V,) over the vocabulary, temperature tau > 0
# Output: probabilities (V,) that sum to 1

def token_probabilities(logits, tau):
    # Divide every logit by the constant temperature  [eq. 1]
    scaled_logits = logits / tau           # (V,)
    # Apply softmax to the scaled logits  [eq. 1]
    return softmax(scaled_logits)          # (V,)

Models (§2.1, tab. 1). Nine LLMs were chosen because their API let the caller set the temperature. tab. 1 lists them; the paper gives the release dates there, but not API snapshot identifiers.

Prompts (§2.1). Five single-request, one-shot prompt styles: baseline, domain expertise, self-recitation, chain-of-thought (CoT) and a composite that combines them. Multi-step agents, few-shot prompts and fine-tuning were excluded. Every prompt includes one worked problem-and-solution pair §2.1 app. A.

Data (§2.2, tab. 2). Ten problem sets from public benchmarks, covering science, maths, common sense, logic, law, English and medicine tab. 2. Two exams were built from them:

1. A 100-question exam with ten questions from each set §2.2.
2. A 1,000-question exam with 100 questions from each set §2.2.

Process (§2.3). The paper states the full design as 10 models, 5 prompts, 10 exams, 100 problems per domain and 10 attempts per problem, swept over all viable temperatures alg. 1 §2.3. Due to cost, the study ran a subset:

1. Sweep GPT-3.5 with CoT on the 100-question exam from 0.0 to 2.0 in steps of 0.1 to find the viable range. Stop at 1.6, because text became incoherent §2.3.
2. Run the other eight models with CoT on the 100-question exam from 0.0 to 1.0 §2.3.
3. Run GPT-3.5 with each of the five prompts on the 100-question exam, 0.0 to 1.0 §2.3.
4. Run GPT-3.5 with CoT on the 1,000-question exam, 0.0 to 1.0 §2.3.

Each question is answered ten times at each setting §2.3. Other sampling parameters (top-k, top-p, repetition penalty) are said to be fixed, but their values are not stated §2.3 §4.2.

Metrics (§2.4). The primary metric is correct-answer accuracy per question, the share of its ten attempts that match the gold letter. The reported accuracy is the mean of these values across questions §2.4. Secondary text-variability metrics are Jaccard, bag-of-words, TF-IDF, Levenshtein, BLEU and SBERT similarity §2.4. The paper does not say which responses are compared; the code groups the responses by question 2_process_text.py.

# Per-question accuracy and mean accuracy  [§2.4]
# Input:  answers[q] = 10 parsed choices for question q
#         gold[q] = the correct choice for question q

def question_accuracy(answers_q, gold_q):
    # Share of attempts whose choice equals the gold choice  [§2.4]
    matches = [answer == gold_q for answer in answers_q]
    return sum(matches) / len(answers_q)

def mean_accuracy(answers, gold):
    # Average the per-question values over all questions  [§2.4]
    scores = [question_accuracy(answers[q], gold[q])
              for q in range(len(gold))]
    return sum(scores) / len(scores)

Analysis (§2.5, §3.1). Per-question accuracy is not normally distributed; it is bimodal, with peaks at 0.0 and 1.0 §2.5. The authors therefore test the effect of temperature with a Kruskal-Wallis test, with one group per temperature. They use a significance threshold of 0.05 §3.1 fig. 5 and a Dunn-Bonferroni post-hoc test for the two exams that were significant §3.1.

# Effect of temperature on per-question accuracy  [§2.5] [§3.1]
# Input:  scores[t] = per-question accuracies at temperature t
# Output: H statistic and p-value (10 degrees of freedom for 11 groups)

def temperature_effect(scores, temperatures):
    # One group per temperature from 0.0 to 1.0  [§2.3]
    groups = [scores[t] for t in temperatures]
    # Kruskal-Wallis test across the groups  [§2.5]
    return kruskal_wallis(*groups)

Answer format. The prompts ask for a line of the form Action: Answer("[choice]") app. A. Unparseable answers count as wrong. Llama 2 7B often broke this format: 39% of its answers were badly formatted §3.1.

# Parse the choice from a model response  [app. A]
# Code: source/agents/actions.py#L4-L12

def parse_choice(text):
    # Find the first Answer("...") in the response  [app. A]
    match = find_first(r'Answer\("([^"]*)"\)', text)
    if match is None:
        return None                # never equals the gold choice
    # Keep only the first character, upper-cased  [code: actions.py#L9-L11]
    return match[0].upper()

Hyperparameters

Results

Author-reported. Setup: nine LLMs, 1,000-question and 100-question exams, temperatures 0.0 to 1.0 unless stated.

Headline results:

• GPT-3.5, CoT, 1,000 questions, 0.0 to 1.0: H(10)=10.439, p=0.403. No significant effect §3.1.
• GPT-3.5, CoT, 100 questions: stable from 0.0 to 1.0. After 1.0 the text becomes incoherent and accuracy falls to about zero near 1.6 §3.1 fig. 2.
• Text similarity decreases as temperature rises, with a rapid drop after 1.0 §3.2 fig. 6.

Per-model Kruskal-Wallis, CoT, 100 questions tab. 3:

Llama 2 7B's non-significant result comes from formatting: 39% of its answers were incorrectly formatted and 36% were correctly formatted but wrong. The authors say it did no better than chance §3.1.

Per-prompt Kruskal-Wallis, GPT-3.5, 100 questions tab. 4:

The paper says CoT outperformed the other prompts and used it for the single-prompt runs §3.1. The test shows no temperature effect for any prompt.

Per-exam Kruskal-Wallis, GPT-3.5, CoT tab. 5:

For LSAT-AR and SAT-Math the paper attributes the significance to 0.0 behaving differently from 0.1 to 1.0 (all-or-nothing answers at 0.0), while the mean accuracy stays similar §3.1.

Recommendation. For problem-solving tasks, set temperature to 0.0. This maximises reproducibility without a measured accuracy cost, and avoids the drop-off above 1.0 §4.1.

Paper vs code

Repository: https://github.com/matthewrenze/jhu-llm-temperature at b00cfb652897e8bdc28c63234def9b490fa84cfb, BSD-2-Clause (Copyright 2023, Matthew Renze). The repo's default run is GPT-3.5 Turbo with CoT on the 100-question exam, 0.0 to 1.0 1_run_experiment.py.

• Accuracy definition. The paper uses mean per-attempt accuracy §2.4. The statistics use that definition: 2_process_details.py computes the share of the ten answers that match the gold answer 2_process_details.py. The run-level results.csv accuracy is different. Each question's score is the majority vote, and accuracy is the share of questions where the vote is correct 1_run_experiment.py results_table.py.
• Majority vote details. Unparseable answers (None) enter the vote. Ties are broken by set iteration order, which the code does not control 1_run_experiment.py.
• Attempts. The paper says ten attempts §2.3. The OpenAI path makes one request with n=10 openai_model.py. The Anthropic path makes ten sequential requests anthropic_model.py.
• Temperature grid. Matches the text: 0.0 to 1.0 in steps of 0.1 §2.3 1_run_experiment.py.
• Algorithm 1 vs text. Algorithm 1 lists 10 models and 16 temperatures alg. 1. The text and tab. 1 list 9 models, and the text's range of 0.0 to 1.6 in steps of 0.1 gives 17 values §2.3 tab. 1.
• Large exam. The 1,000-question analysis pools the ten per-set 100-question files (data/exams/*-100.jsonl, excluding comprehensive-100) 3_analyze_accuracy_for_gpt_35.py. The paper describes this as one sampled exam §2.2.
• Chain-of-thought prompt. The paper names the CoT technique but gives no prompt text, only the composite prompt app. A §2.1. The code's CoT system prompt says to think step by step, then answer in the Action format chain_of_thought_agent.py. Its worked example has only a thought and an action chain_of_thought_agent.py.
• Composite prompt. The system prompt matches app. A composite_agent.py. The worked example differs: the appendix says Des Moines is the capital of Iowa, while the code says it is not app. A composite_agent.py. The code's version is factually wrong.
• Text-similarity pipeline. 2_process_text.py processes only claude-3-opus-20240229 responses, marked with a TODO 2_process_text.py. The plot script for Figure 6 filters gpt-35-turbo 4_plot_text_similarity_by_metric.py. As committed, the scripts cannot regenerate the GPT-3.5 figure the paper captions.
• Jaccard. The paper defines it over word sets §2.4. The code takes set(a) on a string, so it compares character sets 2_process_text.py.
• Levenshtein. The paper calls it a "similarity" §2.4. The code averages raw edit distances, so a higher value means more different 2_process_text.py.
• Max tokens. The paper does not state a limit. The code sets 1000 for all models model.py. The OpenAI path does not check for truncation, while the Anthropic path raises an error on it anthropic_model.py.
• Sampling parameters. The paper says the other parameters were fixed §2.3. The code sets only temperature, and n for OpenAI openai_model.py, so the provider defaults apply.
• API version. The Azure OpenAI version is 2024-02-01 openai_model.py. The paper does not state it.

Pitfalls

• "No significant effect" describes the average, not each question. The paper finds that per-problem accuracy varies significantly between temperatures in pairwise tests, and that the all-or-nothing behaviour at 0.0 explains the exam-level exceptions §3.1. Do not assume every item behaves like the mean.
• The test treats the groups as independent. The same problems are measured at every temperature, but the analysis calls the independent-samples Kruskal-Wallis test 3_analyze_accuracy_for_gpt_35.py. The paper chose this test because the data are not normal §2.5 and does not discuss the pairing. Treat the p-values as approximate.
• Use per-attempt accuracy. The run-level score (majority vote) is not the paper's metric. Compute accuracy from the ten answers per question.
• Parsing is fragile. Only the first character of the captured choice is kept, so Answer("10") becomes 1. Format failures are counted as wrong and can dominate a model's score §3.1.
• Output length grows with temperature. The authors say tokens per response rise as text becomes more random, which raises runtime and cost above 1.0 §4.4. With a 1000-token cap model.py, long answers may be cut off. The OpenAI path does not detect this.
• Derived cost of the full design: 100 problems × 10 attempts × 11 temperatures = 11,000 responses per model, prompt and exam combination. This figure is derived from the design, not stated by the paper.
• Scope. Only multiple-choice questions were tested §4.2 §4.4. Above 1.0, only one configuration was tested §4.2, so the drop-off may differ elsewhere.
• Model snapshots. The paper gives release dates but not API snapshot IDs tab. 1. Results may not reproduce with newer API versions.
• Paper internal inconsistency. Algorithm 1 (10 models, 16 temperatures) does not match the text (9 models, 0.0 to 1.6) alg. 1 §2.3 tab. 1. The paper does not explain the gap.

Lineage

• Builds on: softmax with a temperature scaling eq. 1. The paper cites prompt-engineering literature for the prompt styles §2.1 but no prior study of temperature and accuracy, which it describes as missing §1.3.
• Compared against: the paper's own temperature sweeps and prompt styles. No other sampling method is compared.
• Superseded by: not known from content in this wiki.