Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback

In plain words

Chatbots built on large language models often sound confident while making things up. This paper's fix does not retrain the model. It wraps the model in a few helper parts that work around it.

One helper looks up facts for the question, such as web pages, FAQs, reviews or Wikipedia tables, and links related pieces of text into short chains of evidence. The model is then asked to answer using that evidence. A second helper checks the answer against the evidence. If the answer does not match, the checker writes a short note such as "the response is inconsistent with the knowledge", and the question is sent to the model again with that note added. A small controller decides which helper to call next.

The point for an engineer is that the model itself stays a black box, so this works with a hosted model that cannot be fine-tuned. The cost is extra model calls for each answer, and the paper's own experiments are on ChatGPT, with news chat, customer service and open-domain question answering as the test tasks.

TL;DR

LLM-Augmenter sits between the user and a fixed LLM §2 fig. 2. It consolidates external evidence before prompting, then scores each candidate response with a utility function and re-prompts with feedback when the score is too low §1 §2.4. The main experiments use a hand-written rule policy that always consolidates evidence, with ChatGPT as the LLM; a T5-Base policy trained with REINFORCE is tested only in a learning-curve experiment §3.2 §3.3 fig. 3. Author-reported gains: Knowledge F1 about +10 points on news chat and +6 on customer service over ChatGPT alone §3.3 tab. 1 tab. 2, and Wiki QA F1 from 0.59 to 11.80 tab. 5. No code is released (see Paper vs code).

Problem

LLMs generate fluent text but can hallucinate, and their knowledge is fixed at training time and may be stale or private §1. Fine-tuning a large model to fix this is costly, so the authors want to improve a fixed LLM with plug-and-play modules §1. Two task types are used: information-seeking dialog, where replies must be grounded in external knowledge, and open-domain Wiki question answering over tables and passages, where multi-hop retrieval is needed §1 §3 §4. The authors treat the LLM as a black box §1 §2, and say ChatGPT's limited bandwidth forced a rule-based policy in their experiments §6.

Method

Overview and state

The system is a Markov decision process over dialog states, with actions that either consolidate evidence or query the LLM, and an external reward §2. A dialog state is the six-tuple (q, e, o, u, f, h_q): current query, consolidated evidence, candidate responses, utility scores, feedback, and dialog history §2.1. A query can take several iterations: each round produces a candidate from evidence, feedback and utility before the final response is sent §2.1 fig. 1.

The modules are Working Memory (the state above), Policy (chooses the next action), Action Executor (Knowledge Consolidator plus Prompt Engine) and Utility §2 fig. 2.

Policy §2.2

The policy chooses among (1) acquiring evidence, (2) calling the LLM for a candidate, and (3) sending a response to the user once Utility accepts it §2.2. It can be rule-based or learned. The learned form is a neural network with parameters θ, trained with REINFORCE to maximise expected reward, as in eq. 1 §2.2. The authors use a pre-trained model such as T5 for the policy §2.2. Learning can start from rules, then use a simulator, then real users §2.2.

# Policy objective, maximised with REINFORCE  [eq. 1] [§2.2]
# pi_theta: policy network, T5-Base in the experiments  [§3.2]
# R(s, a): external reward; KF1 in the RL experiment  [§3.3]
def policy_objective(theta, states):
    # Expected reward of actions sampled from the policy  [eq. 1]
    sampled = sample_actions(pi_theta, states)  # (s, a) pairs
    return mean(reward(s, a) for s, a in sampled)

Knowledge Consolidator §2.3.1

Three modules, following §2.3.1 (which cites CORE, Ma et al. 2022):

• Retriever: generates search queries from q and h_q, then calls one or more APIs for raw evidence (for example web search, or REST APIs over task databases) §2.3.1.
• Entity linker: adds context to raw evidence by linking each mentioned entity to a description, based on Wikipedia, to form an evidence graph §2.3.1.
• Chainer: prunes the graph and returns a short list of evidence chains most relevant to the query §2.3.1. The consolidated evidence e goes to Working Memory §2.3.1.

Experiment-specific retrievers: BM25 over web pages linked from Reddit posts (news chat), and BM25 over FAQs and Yelp reviews (customer service) §3.2. For Wiki QA, DPR is the retriever over Wikipedia passages and tables, with the CORE linker and chainer added §4.2. Golden (human-annotator) knowledge is also used as an oracle §3.2.

Prompt Engine §2.3.2

The prompt is a text string with the task instruction, the query, dialog history, evidence (when available) and feedback (when available) §2.3.2. Prompts are task-specific §2.3.2 tab. 7 tab. 8 tab. 9. The prompt has two variants: a plain chatbot prompt without evidence, and a prompt that adds a "Working Memory" block of evidence tab. 7 tab. 8. Wiki QA uses a question-answering prompt that asks for "Unknown" when the question has no clear answer tab. 9.

Utility §2.4

Given candidate o, Utility produces a score u and feedback f using task-specific utility functions §2.4. Two kinds are described: model-based (learned preference scores for fluency, informativeness, factuality) and rule-based (heuristics that test a rule) §2.4. Feedback comes from a text generator Q with parameters ψ that reads q, e, o and h_q eq. 2 §2.4. The experiments use one utility function at a time §2.4, footnote.

# Utility feedback  [eq. 2] [§2.4]
# Q_psi: text generator (seq2seq or LM); templates or ChatGPT also work  [§2.4]
def generate_feedback(query, evidence, candidate, history):
    # Feedback text f, given query, evidence, candidate and history  [eq. 2]
    return Q_psi(query, evidence, candidate, history)

The main experiments use three feedback sources: a template generator that emits the inconsistency message when the score is below a threshold (threshold not stated) §3.2; ChatGPT prompted to critique its own answer ("self-criticism") §3.2; and, in Wiki QA, the same template with recall as the score §4.2.

Control loop §1 §2 §3.2 fig. 1

The loop below is described in prose in §1 and §2.1. The paper has no formal algorithm box, so this pseudocode is our reconstruction from those sections; the stopping rule is not stated.

# Iterative response loop  [§1] [§2.1] [fig. 1]
# Rule-based policy of the main experiments: always consolidate first  [§3.2]
def answer(query, history, llm):
    # Consolidate external evidence for the query  [§2.3.1]
    evidence = knowledge_consolidator(query, history)
    feedback = None
    while True:  # no iteration cap is stated
        # Build prompt from query, history, evidence, feedback  [§2.3.2]
        prompt = build_prompt(query, history, evidence, feedback)
        candidate = llm(prompt)  # ChatGPT in all experiments  [§3.2]
        # Score the candidate; keep feedback for the next round  [§2.4]
        score, feedback = utility(query, evidence, candidate, history)
        # Accept when the score passes the threshold (value not stated)  [§3.2]
        if score passes threshold:
            return candidate

Hyperparameters

Values are as stated in the paper. Nothing here comes from the code (none is released).

Results

Author-reported throughout. Metric values are as printed in the tables (percent for dialog). Table numbers are inferred from source order, see Pitfalls.

Dialog: News Chat (DSTC7) tab. 1

Gold rows use human-annotator knowledge, which is not available at deployment.

Dialog: Customer Service (DSTC11) tab. 2

Human evaluation, customer service with BM25 tab. 3

Likert scores converted to win/tie/loss for reporting §3.4. All differences are reported significant at p < 0.05. Krippendorff's alpha is 0.15 (usefulness) and 0.07 (humanness).

Feedback ablation, customer service, BM25 tab. 4

Feedback helps, and the two feedback types are close; the authors expect self-criticism to matter more on complex tasks §3.5.

Wiki QA (OTT-QA) tab. 5

Recall of the gold answer inside CORE's top-5 evidence is 50.83 tab. 5. The table gives no units; the values are read as percent.

Figures

• fig. 3: RL learning curve on customer service with T5-Base policy and KF1 reward. The mean across 5 runs passes a random policy after about 600 interactions and reaches a KF1 near 37.5 §3.3.
• fig. 4: ablation of policies on customer service: no consolidator, Self-ask (consolidator only when the LLM asks), Always-use §3.5.
• fig. 5: ablation of utility and feedback combinations; the combination of utility and feedback-augmented prompting is best §3.5.

Paper vs code

No official code.

The abstract and its footnote [abstract] point to https://aka.ms/llm-augmenter, which redirects to github.com/pengbaolin/LLM-Augmenter. The owner is an author (Baolin Peng). That repository currently contains only a README (which says an implementation will be provided "soon"), a .gitignore and two images; there is no implementation, no config, no LICENSE and no commit SHA we could clone. We therefore set no code field. A GitHub search for "llm-augmenter" found no other official repository. Third-party reimplementations were not used.

Consequence: every hyperparameter, prompt and algorithm step on this card comes from the paper alone, and the pseudocode for the control loop is our reconstruction.

Pitfalls

• The stopping rule is not stated. The paper says the loop repeats until a candidate passes verification §1 but gives no maximum number of rounds, no fallback when no candidate passes, and no rule for the acceptance threshold on KF1 §3.2. An implementer must choose these; the paper does not say what it used.
• Feedback threshold value is not stated §3.2. The template message fires when KF1 falls "below a certain threshold"; the only mention of a score cut-off is this footnote.
• KF1 is not defined in the paper. It is cited to Shuster et al. 2021 and described as token overlap between the prediction and evidence §3.2. Tokenisation, stopwords and normalisation are not stated, so exact scores may not reproduce.
• Wiki QA utility is recall of token overlap with the evidence §4.2. This is a different score from KF1 used in dialog. Tokenisation not stated.
• Policy in the main results is not learned. The reported dialog and Wiki QA results use a rule that always calls the Knowledge Consolidator §3.2. The learned T5-Base policy appears only in the RL curve fig. 3 and uses KF1 as its reward §3.3, which is also the metric the paper reports, so the curve does not test a separate objective. The paper reports no learned-policy results on the main metrics.
• Which policy trains on which split is not stated. The RL setup does not say which data or split trains the T5-Base policy §3.3.
• Evaluation set for customer service is the validation set because the test set was not available §3.1. Results are on that split only.
• Gold-knowledge rows are an oracle. Human-annotator knowledge is used in the "gold" rows of tab. 1 and tab. 2 §3.2. Do not read them as deployable.
• Intro figures do not match the tables exactly.
  • "+10% in F1" §1: tab. 5 gives 0.59 → 11.80, which is +11.21 absolute. The 10% figure is not reproduced exactly.
  • "32.3% in Usefulness" §1 matches a relative gain from tab. 3 (45.07 vs 34.07), but the "12.9% in Humanness" figure does not: tab. 3 gives 35.22 vs 30.92, a relative gain of 13.9%.
  • "approximately 10 and 6 points" KF1 §3.3: this matches BM25 with feedback against ChatGPT in tab. 1 (+9.70) and tab. 2 (+6.08). The text does not say which rows it means.
• Human evaluation agreement is low. Krippendorff's alpha is 0.15 and 0.07 tab. 3. Results rest on a Likert scale that was converted to win/tie/loss, and the conversion rule is not stated §3.4.
• Model version and decoding are not stated. The paper names ChatGPT but gives no model snapshot, date, temperature or max tokens. Results may not reproduce on a different snapshot.
• Cost is high and the paper says so. The loop can call the LLM twice per response, and it slows interactive use §6. The authors suggest showing the first answer and offering the improved one as an option §6.
• Table and figure numbers are inferred. No compiled PDF was available; the numbers follow the order of the source files (main dialog tables, human evaluation, feedback ablation, Wiki QA, appendix examples and prompt templates). Check against the PDF before citing a number.
• Wiki QA gap to the fine-tuned state of the art. The authors say there remains a noticeable gap to the fine-tuned CORE model with top-50 evidence and give no number for it in the table §4.3. Do not use Wiki QA results as a claim of parity.
• Customer service and news chat use different consolidators. News chat uses web-page BM25; customer service uses FAQ and Yelp BM25 §3.2. Transfer of the retriever to a new domain is not tested.

Lineage

• Builds on: the Knowledge Consolidator follows CORE (Ma et al., 2022) for retriever, linker and chainer §2.3.1 §4.2. Wiki QA uses DPR (Karpukhin et al., 2020) as the dense retriever §4.2. Utility types are inspired by Sparrow §2.4. Policy learning uses REINFORCE (Williams, 1992) §2.2.
• Compared against: ChatGPT alone (zero-shot, closed-book) tab. 1 tab. 2 tab. 5; raw DPR evidence tab. 5; the fine-tuned CORE model with top-50 evidence, cited as state of the art but without a number in the table §4.3.
• Superseded by: not known from content in this wiki.