Paper · 2022-10-05 · cs.CL
Ask Me Anything: A simple strategy for prompting language models
AMA has the LLM rewrite each input as an open question, answers it, and repeats this with 3 to 6 demonstration sets. Weak supervision combines the answers without labels. GPT-J-6B with AMA exceeds few-shot GPT-3 175B on 15 of 20 benchmarks; mean lift over 3-shot is 10.2 points across models and tasks. Author-reported.
Prompt engineeringIn-context learningWeakly-supervised learningSelf-consistencypromptingprompt-aggregationweak-supervisionquestion-answering-prompts
arXivPDFCodeMarkdown14 min read
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Getting a language model to do a task well often depends on finding exactly the right wording for the instructions. This paper takes a different route. It does not look for one perfect prompt. It asks the model several simpler questions about each input, collects the answers, and combines them.
The authors found that models do better when they are asked an open question than when they are told to reply with a fixed word such as True or False. So the method uses the model itself to turn each input into a question, then asks the model to answer that question. Each such run uses different examples and gives one answer, which counts as one vote. The votes are combined with a statistical technique from weak supervision. It estimates how reliable each voter is and how its mistakes overlap with the others, and it does this without labelled data.
The authors report that a small open model with 6 billion parameters, using this method, beats a 175-billion-parameter model's few-shot results on 15 of 20 benchmarks. These numbers are the authors' own. The method also makes several model calls per input, which matters for cost.
TL;DR
- AMA (Ask Me Anything) runs a two-step prompt chain per input: the model writes a question from the input, then answers that question with the input as context §3.3 fig. 1.
- Several chains with different demonstrations give several votes per input. Weak supervision combines them into one label, learning each chain's accuracy and dependencies from unlabelled votes §3.4 alg. 1.
- Author-reported: GPT-J-6B with AMA exceeds few-shot GPT-3 175B on 15 of 20 benchmarks tab. 1. Averaged across models and tasks, the lift over 3-shot is 10.2 points absolute §1 §5.2.
- Weak supervision is not worse than majority vote on 16 of 20 tasks, and is up to 8.7 points better §5.3.
- The official code is Apache-2.0. Pitfalls: the code passes training gold labels to the label model and the dependency fit lives in a separate submodule, so the "no labels" claim needs checking run_ws.py.
Problem
Model results change a lot with small wording changes, so much effort goes into designing one "perfect" prompt per task §1 fig. 1. Writing several prompts and taking a majority vote does not reliably help. On GPT-3-style prompts for CB and RTE, majority vote gave 39.3% (+2.2%) on CB and 54.5% (−2%) on RTE, compared with the average prompt §3.1.
Two further problems are named. Restrictive prompts that ask for a fixed token are worse than open questions for small models §3.2. And the errors of different prompts are correlated and differ in quality, which majority vote ignores §3.4. The method is meant for off-the-shelf models with no further training §1.
Method
AMA has two parts. First, the model rewrites each input into an open-ended question and answers it, in a fixed chain. Second, several such chains are run per input, and their answers are aggregated with weak supervision.
- Prefer open-ended question formats. Reformatting CB, RTE and WSC from restrictive to open question-answer form raised average accuracy of the small model from 41.7% to 71.5% (+72% relative). The sentence does not name the model; §3.1 uses GPT-J-6B for its running examples §3.2. Precise questions gave larger lifts than generic ones §3.2.
- Map open answers to labels. For multi-class tasks, an extra prompt maps the free-text answer to a valid category. The authors report lifts of 33.3% on DBPedia and 11.1% on AGNews from this step §3.2.
- Build the chain. A question prompt turns the input statement into a question. An answer prompt answers the question with the input as context. The answer of the chain is the output §3.3 fig. 1.
- Vary the chains. Each chain differs in its in-context demonstrations and in question style: yes/no, Wh, multiple-choice or cloze §3.3 fig. 3. Chains are fixed, reusable prompts that do not depend on the individual input §3.3.
- Aggregate with weak supervision. The votes of m chains on n unlabelled inputs are combined by a probabilistic model, which assigns each input the label with the highest posterior probability §3.4 alg. 1 app. C.
- Keep at most one dependency. The recovered dependency structure is used only if one of its entries is below 1; then the single strongest edge is kept app. A.3 run_ws.py.
# One prompt-chain for one input x [§3.3] [fig. 1]
# Input: x (task input); question_demos, answer_demos (in-context pairs)
# Output: one vote y_hat (a label from the task's label set)
def prompt_chain(x, question_demos, answer_demos):
# Write an open question about x with the question() prompt [§3.3]
question = ask_llm(question_demos, statement=x)
# Answer that question, using x as context, with the answer() prompt [§3.3]
answer = ask_llm(answer_demos, passage=x, question=question)
# Map the free-text answer to a valid label [§3.2]
y_hat = map_to_label(answer)
return y_hat# Weak-supervision aggregation over m chains [§3.4] [alg. 1] [app. C]
# Input: votes (n, m), one row per unlabelled input
# Output: one label per input
def aggregate(votes):
# Learn which chain pairs depend on each other, from votes only [app. C]
structure = learn_structure(votes)
# Keep one edge only when some recovered entry is below 1 [app. A.3]
dependencies = top_edge_or_none(structure, threshold=1)
# Learn each chain's accuracy with the label model [§3.4]
label_model = fit_label_model(votes, dependencies)
# Predict the label with the highest posterior for each input [alg. 1]
return argmax_label(label_model, votes)The label model is an Ising-type distribution over the label y and the chain votes P(x). Each chain has an accuracy parameter, and selected pairs of chains have a dependence parameter eq. 2 app. C:
\Pr_{G,\theta}(y, \mathbf{P}(x)) = \frac{1}{Z} \exp\Big(\theta_y y + \sum_{i=1}^{m} \theta_i\, p_i(x)\, y + \sum_{(i,j) \in E} \theta_{ij}\, p_i(x)\, p_j(x)\Big) \quad [\text{eq. 2}]
The information-flow metric in eq. 1 splits the conditional entropy H(y | ŷ) of the final prediction into a part set by the chains and a part set by the aggregator:
H(y \mid \hat{y}) = H(y \mid \mathbf{P}(x)) + \big(H(y \mid \hat{y}) - H(y \mid \mathbf{P}(x))\big) \quad [\text{eq. 1}]
The appendix gives a closed form for H(y | P(x)) under the Ising model with y in {−1, 1} and Pr(y = 1) = 1/2 app. D, Theorem:
H(y \mid \mathbf{P}(x)) = H(y) - \Big(\Theta^\top \mu - \mathbb{E}_{\mathbf{P}(x)}\big[\log\cosh(\Theta^\top \mathbf{P}(x))\big]\Big)
Here Θ holds the correlation strengths of y with each chain and μ_i = 2 Pr(p_i(x) = y) − 1 app. D.
Structure learning app. C alg. 1: the inverse covariance over y and the votes is graph-structured, so a zero entry means two chains are conditionally independent given y. The authors decompose the inverse covariance of the votes into sparse and low-rank parts, using a robust-PCA-type estimator, and read the dependency graph from the sparse part app. C. Accuracies are then learned from that sparsity pattern app. C §3.4.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| prompt chains per input (m) | 3 to 6, set per task | tab. 5 CB_final.py | COPA: 4 in tab. 5, 5 in its code main; see Paper vs code |
| chains when a task's main does not set it | 3 | decomposition.py | parser default |
| demonstrations per chain | task-specific, hand-written | CB_final.py | CB: 4–5 question demos and 3–4 answer demos per set; count not stated in paper |
| few-shot baseline k | 3 (GPT-J-6B); GPT-3 uses k from 1 to 70 as cited | §5 tab. 1 decomposition.py | GPT-3 k values copied from prior papers |
| unlabelled set for weak supervision | test split plus 1000 training rows | app. A.3 decomposition.py | code takes the first 1000 rows, not a random sample |
| random seed | 3 | decomposition.py | paper reports 5 random seeds; no multi-seed loop in repo |
| decoding (Hugging Face models) | temperature 0.001, no sampling | utils.py | paper does not state decoding; GPT-3 accessed via API app. A.1 |
| tokens generated per question and answer | 50 | CB_final.py CB_final.py | CB zero- and few-shot baseline uses 30 CB_final.py |
| structure learning weights | λ = 1/√m, γ = 1e-8 | run_ws.py | paper says "default parameters" from Varma et al. app. A.3 |
| dependency threshold | keep top edge only if min off-diagonal |Ĵ| < 1 | run_ws.py | matches app. A.3 |
| label model, no dependencies | 10,000 epochs, learning rate 1e-5 | run_ws.py | not stated in paper |
| label model, with dependencies | 80,000 epochs, learning rate 1e-6 | run_ws.py | not stated in paper |
| label model seed | 123 | run_ws.py | |
| weighted majority vote temperature (baseline only) | sweep 0.25, 0.5, 1, 2, 4, 8, 16, 32 on a 20% split | app. B.3 | uses 1000 labelled examples |
| self-consistency temperatures (baseline only) | 0.0, 0.3, 0.5, 0.6, 0.7 | app. B.4 | |
| metrics | accuracy; text F1 for DROP and RealTimeQA; span overlap for NQ and WebQs; F1a for MultiRC | app. A.2 | RealTimeQA GPT-3 re-run on davinci app. A.2 |
| hardware | NVIDIA A100 | app. A | |
| models | GPT-J-6B main; EleutherAI 125M to 20B, BLOOM 560M to 176B, OPT 125M to 175B, T0-3B, GPT-3 davinci | app. A.1 §5 | 14 models in total §5 |
Results
Author-reported throughout. The setup is 20 benchmarks and 14 LLMs from 4 model families (EleutherAI, OPT, BLOOM, T0), sizes from 125M to 175B §5. GPT-3 numbers are copied from prior papers, not re-run, except RealTimeQA app. A.2. The authors do not use rank-classification scoring, except on ReCoRD, StoryCloze and COPA, whose options are explicit §5 (footnote).
Table 1 — GPT-J-6B with AMA versus few-shot GPT-3 175B tab. 1. For the QA tasks and ReCoRD the QA + WS column reports majority vote, because weak supervision is complex for open-ended outputs tab. 1.
| Task | GPT-J-6B few-shot (k=3) | GPT-J-6B QA | GPT-J-6B QA + WS | GPT-3 175B few-shot (k) |
|---|---|---|---|---|
| BoolQ | 66.5 | 64.9 | 67.2 ±0.0 | 77.5 (32) |
| CB | 25.0 | 83.3 | 83.9 ±0.0 | 82.1 (32) |
| COPA | 77.0 | 58.2 | 84.0 ±0.0 | 92.0 (32) |
| MultiRC | 60.8 | 58.8 | 63.8 ±0.0 | 74.8 (32) |
| ReCoRD | 75.6 | 74.5 | 74.4 ±0.0 | 89.0 (32) |
| RTE | 58.8 | 61.7 | 75.1 ±0.0 | 72.9 (32) |
| WSC | 36.5 | 74.7 | 77.9 ±0.0 | 75.0 (32) |
| WiC | 53.3 | 59.0 | 61.0 ±0.2 | 55.3 (32) |
| ANLI R1 | 32.3 | 34.6 | 37.8 ±0.2 | 36.8 (50) |
| ANLI R2 | 33.5 | 35.4 | 37.9 ±0.2 | 34.0 (50) |
| ANLI R3 | 33.8 | 37.0 | 40.9 ±0.5 | 40.2 (50) |
| StoryCloze | 51.0 | 76.3 | 87.8 ±0.0 | 87.7 (70) |
| AGNews | 74.5 | 83.7 | 86.4 ±0.0 | 79.1 (8) |
| Amazon | 62.5 | 66.8 | 68.2 ±0.0 | 41.9 (8) |
| DBPedia | 50.7 | 81.4 | 83.9 ±0.0 | 83.2 (8) |
| SST | 64.9 | 94.5 | 95.7 ±0.0 | 95.6 (8) |
| DROP | 32.3 | 51.0 | 51.6 ±0.0 | 36.5 (20) |
| NQ | 13.7 | 19.7 | 19.6 ±0.0 | 29.9 (64) |
| RealTimeQA | 34.7 | 34.7 | 36.0 ±0.0 | 35.4 (1) |
| WebQs | 29.1 | 44.2 | 44.1 ±0.0 | 41.5 (64) |
Headline results from the text:
- GPT-J-6B with AMA exceeds the GPT-3 175B few-shot result on 15 of 20 benchmarks §5.1 tab. 1. Counting the table's QA + WS column gives 15.
- Averaged over the 20 tasks, AMA improves the 6B model's 3-shot results by 41% §5.1.
- Averaged across models and tasks, AMA lifts the 3-shot baseline by 10.2 ± 6.1 points absolute (21.4 ± 11.2% relative) §1 §5.2.
- By family, the average absolute lift is 11.0 points for EleutherAI, 11.0 for BLOOM, 11.9 for OPT and 2.9 for T0 §5.2. The authors hypothesise that T0's lower lift comes from fine-tuning on zero-shot prompts §5.2.
- Lift is largest on tasks whose required knowledge is in the input, such as reading comprehension, and smallest on tasks that rely on memorised knowledge §5.1.
- Diagnostic scores for question generation, answer generation, answer selection and extraction track the lift: models that score higher on these see more lift §5.2 fig. 5.
Weak supervision versus majority vote. WS reaches up to 8.7 points over majority vote, and is not worse than majority vote on 16 of 20 tasks. On the other 4 tasks, it is worse by at most 1.0 point §5.3. Modelling dependencies helps on 9 tasks, by up to 9.6 points and 2.2 points on average §5.3. Table 5 gives per-task numbers.
Table 2 — T0-3B with PromptSource prompts tab. 2. Average lift over 10 prompt formats: majority vote +3.6 in the table caption, +3.7 in the text; WS +6.1 §5.3 tab. 2.
| Method | CB | WiC | WSC | RTE |
|---|---|---|---|---|
| T0 (3B) mean over prompts | 45.4 | 50.7 | 65.1 | 64.6 |
| T0 (3B), 10 formats, majority vote | 60.7 | 50.5 | 68.3 | 60.6 |
| T0 (3B), AMA, majority vote | 50.0 | 49.5 | 64.4 | 49.5 |
| T0 (3B), 10 formats, WS | 60.7 | 50.8 | 69.2 | 69.7 |
| T0 (3B), AMA, WS | 50.0 | 51.4 | 66.4 | 59.2 |
Table 4 — aggregation without reformatting tab. 4. Three random 3-shot prompts, with no QA reformatting, aggregated by average, majority vote or WS, against AMA with WS (GPT-J-6B). Average, majority vote and WS are shown in the first three columns.
| Task | Few-shot average | Few-shot MV | Few-shot WS | AMA (WS) |
|---|---|---|---|---|
| CB | 23.8 | 17.9 | 50.0 | 83.9 |
| RTE | 53.5 | 53.1 | 54.2 | 75.1 |
| WSC | 46.2 | 38.5 | 38.5 | 77.9 |
| COPA | 80.0 | 81.0 | 81.0 | 84.0 |
| ANLI R1 | 33.4 | 33.5 | 33.5 | 37.8 |
| ANLI R2 | 33.2 | 32.9 | 32.2 | 37.9 |
| ANLI R3 | 35.4 | 36.5 | 34.6 | 40.2 |
| AGNews | 70.3 | 70.7 | 75.0 | 86.4 |
| Amazon | 61.9 | 62.4 | 62.5 | 68.2 |
The authors conclude that aggregation alone leaves large gaps, and that reformatting and aggregation are both needed app. B.2.
Table 5 — aggregation ablation on GPT-J-6B tab. 5. "Avg" is the average accuracy of the individual chains. MV, AMA (no deps) and AMA (WS) use no labels. WMV and Pick Best use labelled data.
| Task | # prompts | Avg | MV | WMV | Pick Best | AMA (no deps) | AMA (WS) |
|---|---|---|---|---|---|---|---|
| WSC | 3 | 74.7 | 77.8 | 77.8 | 75.0 | 77.8 ±0.0 | 77.8 ±0.0 |
| WiC | 5 | 59.0 | 61.3 | 60.9 | 60.0 | 60.8 ±0.0 | 61.3 ±0.2 |
| RTE | 5 | 61.4 | 66.0 | 71.4 | 62.0 | 65.1 ±0.5 | 75.1 ±0.0 |
| CB | 3 | 83.3 | 82.1 | 82.1 | 83.9 | 82.1 ±0.0 | 83.9 ±0.0 |
| MultiRC | 3 | 58.8 | 63.8 | 63.4 | 63.4 | 63.7 ±0.0 | 63.8 ±0.0 |
| BoolQ | 5 | 64.9 | 65.9 | 67.2 | 68.3 | 65.9 ±0.0 | 67.2 ±0.0 |
| COPA | 4 | 58.3 | 85.0 | 82.0 | 82.0 | 84.0 ±0.0 | 84.0 ±0.0 |
| ANLI R1 | 5 | 34.6 | 37.6 | 36.1 | 36.8 | 37.4 ±1.0 | 37.8 ±0.2 |
| ANLI R2 | 5 | 35.4 | 36.3 | 36.0 | 36.0 | 38.7 ±0.4 | 37.9 ±0.2 |
| ANLI R3 | 5 | 37.0 | 39.0 | 38.4 | 38.4 | 39.6 ±0.9 | 40.9 ±0.5 |
| StoryCloze | 6 | 76.3 | 87.9 | 81.8 | 81.8 | 82.2 ±0.0 | 87.8 ±0.0 |
| DBPedia | 3 | 81.4 | 84.1 | 83.9 | 82.2 | 83.9 ±0.0 | 83.9 ±0.0 |
| SST2 | 3 | 94.5 | 95.7 | 95.7 | 95.2 | 95.7 ±0.0 | 95.7 ±0.0 |
| Amazon | 3 | 67.0 | 68.6 | 68.6 | 67.3 | 68.6 ±0.0 | 68.6 ±0.0 |
| AGNews | 3 | 83.7 | 86.5 | 84.2 | 83.8 | 86.4 ±0.0 | 86.4 ±0.0 |
Table 5 lists COPA with 4 prompts; see Paper vs code. The caption notes that ReCoRD and the QA tasks use 3 prompts each with majority vote tab. 5.
Self-consistency comparison tab. 7. With 5 outputs each, GPT-J-6B with self-consistency and chain-of-thought gets 47.3 on RTE, 63.1 on BoolQ and 33.4 on ANLI R1. AMA with the same number of prompts gets 75.1, 67.2 and 37.8 app. B.4.
BLOOM-7.1B tab. 3. QA + WS scores above the 3-shot baseline on 17 of 20 tasks. It is lower on ReCoRD (69.8 vs 71.6), ANLI R1 (31.5 vs 34.9) and DBPedia (70.5 vs 72.3). The appendix describes this as "consistent lift" app. B.1.
Cost of weak supervision. Averaged over RTE, WSC and AGNews, the WS step takes 13.0 seconds without dependencies and 84.3 seconds on RTE with them. Running GPT-J-6B inference on the same tasks is listed in tab. 6 app. B.
Paper vs code
The official repository is HazyResearch/ama_prompting, linked from the abstract
and the paper. It is owned by a lab of the authors (Christopher Ré). It is
Apache-2.0. We checked code at commit 460843d93a9e4bf2115eb35e4a02e82b6b67feac
(shallow clone). The weak-supervision code depends on the metal-ama git
submodule, which the shallow clone did not fetch.
- Prompt-chain count for COPA. The paper's table gives 4 prompts tab. 5. The code's main sets
num_boost = 5COPA_final.py. Every other task whose main sets the count matches tab. 5, e.g. CB at 3 CB_final.py and AGNews at 3 AGNews_final.py. - Unlabelled training rows. The paper says 1000 training samples app. A.3. The code takes the first 1000 rows decomposition.py.
- Gold labels in the label model. The paper says weak supervision uses no labelled data §1 §3.4.
run_ws.pypasses the gold labels of the training rows asY_devtotrain_modelrun_ws.py run_ws.py. WhatY_devdoes is defined inmetal-ama, which is not in the clone. - Question-style lever. The paper varies question style across chains §3.3. The CB code uses one yes/no instruction for all chains and varies only the demonstrations CB_final.py CB_final.py. Only CB was checked.
- Structure learner. The paper names a robust-PCA-type step app. C. The code solves a convex problem with a nuclear-norm term on the low-rank part and a near-zero L1 weight (γ = 1e-8) on the sparse part run_ws.py. Its weight λ = 1/√m is not stated in the paper.
- Multi-class dependencies. For more than 2 classes, the code averages one-vs-rest structure estimates run_ws.py. The paper does not describe this.
- Majority vote versus WS in the saved output. The
decomposedlog stores majority-vote predictions decomposition.py.run_ws.pyprints metrics and does not save its predictions run_ws.py. WS accuracies therefore come from printed output. - Seeds. The paper reports 5 random seeds §5. The repository runs one seed per invocation (default 3) and has no multi-seed loop decomposition.py.
- Task-agnostic prompts. The paper calls the question and answer prompts task-agnostic §3.3. The demonstrations are hand-written per task file CB_final.py.
No other discrepancies were found in the parts checked: the three-step WS procedure, the dependency threshold of 1 app. A.3 run_ws.py, and the chain structure all match.
Pitfalls
- "No labels" is conditional on the submodule. Confirm the role of
Y_devinmetal-amabefore reimplementing WS and calling it label-free. The paper's "no labelled data" §1 and the code'sY_devargument conflict on the face of it. - Label-model hyperparameters are not in the paper. The code uses 10,000 epochs at lr 1e-5 without dependencies and 80,000 epochs at lr 1e-6 with them run_ws.py run_ws.py. Changing these changes the learned accuracies.
- Two abstain conventions. In CB, any answer that is neither yes nor no becomes "Neither" CB_final.py. In WS, any vote outside the label map becomes −1, an abstain run_ws.py. Use the convention of the task you are reproducing.
- Majority-vote ties.
Counter.most_commonreturns the first-seen label on a tie decomposition.py. When all chains disagree, the earliest chain's label wins. - Dependency model is limited. At most one edge is kept, and only if some entry is below 1 app. A.3 run_ws.py. Richer dependency structures cannot be represented in either the paper or the code.
- Cost scales with chains. Each chain makes two model calls per input (question, then answer), with up to 50 tokens each CB_final.py CB_final.py. The WS step also runs every chain over the 1000 training rows CB_final.py. So an input with m chains costs 2m generations, and the fit adds 2m generations for each of the 1000 rows.
- Hand-written demonstrations. Each task needs its own demonstration sets, and the sets are fixed in code CB_final.py. Nothing in the method generates them automatically.
- Headline gaps are not uniform. The 15-of-20 claim tab. 1 compares GPT-J with AMA to GPT-3 numbers from different papers and different scoring. QA and ReCoRD use majority vote, not WS tab. 1.
- Inconsistent numbers in the paper. The introduction gives majority-vote lift on CB as +4% §1; §3.1 gives +2.2%. The T0 average is +3.6 in the caption and +3.7 in the text tab. 2 §5.3. The BLOOM text says "consistent lift" where the table shows 3 of 20 tasks below the baseline tab. 3.
- Outdated stack. The pinned requirements date from 2022 (torch 1.12.1, tensorflow 2.9.1, snorkel 0.9.9) and need the Manifest serving layer requirements.txt README.md.
Lineage
- Builds on: weak supervision and its label model §3.4 (Ratner et al. 2017; accuracy learning, Ratner et al. 2018; dependency structure, Varma et al. 2019). Demonstration variety is inspired by Sanh et al. (2022) and Liu et al. (2021) §3.3. Recursive use of the LLM resembles Jung et al. (2022) §2.
- Compared against: GPT-3 few-shot (Brown et al. 2020) tab. 1; majority vote, weighted majority vote and pick-best tab. 5; self-consistency with chain-of-thought (Wang et al. 2022) tab. 7; zero-shot T0 on PromptSource prompts tab. 2.
- Superseded by: not stated in this wiki.
Source and license
Adapted from “Ask Me Anything: A simple strategy for prompting language models” by Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, Christopher Ré (arXiv:2210.02441), used under CC0 1.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/HazyResearch/ama_prompting at commit 460843d93a9e (Apache-2.0). Any quoted code is shown under that license.