Large Language Models Can Learn Temporal Reasoning

In plain words

Many questions about a story depend on when things happened: who was married first, how long something lasted, what came right before what. Language models often get these wrong because the timing is spread over many sentences and they have to work out the order in their heads.

The authors split the job in two. First, a model reads the story and writes out a clean, ordered list of events, each with its start and end. Second, the same kind of model reads that list, plus a few computed facts such as how far apart two dates are, and writes a short step-by-step answer. Because the timeline is written down, each step has something concrete to point at.

To train these models without hand-labelled timelines, the authors built a synthetic story collection where the true timeline is known. They also gathered several reasoning traces per question, kept the ones that reach the right answer, and favoured traces that make the right answer stand out from the wrong ones. They further changed names and dates in the training data so the model learns the logic instead of memorising facts. The results are the authors' own.

TL;DR

TG-LLM is a two-step pipeline. A LoRA-tuned Llama-2 13B first translates a story into a temporal graph (a chronological timeline), then a second LoRA adapter reasons over that timeline and the question §3 fig. 1. The reasoning adapter is trained on chains of thought that were bootstrapped and scored with a contrastive weight, plus graph perturbations, on the synthetic TGQA dataset §3.2.1 §3.2.2 §2. Author-reported: on TGQA the 13B model reaches EM 0.797, below GPT-4 with CoT (0.821), but above GPT-4 with a standard prompt (0.772) tab. 4. Read the Pitfalls before using the headline claim.

Problem

Temporal reasoning needs ordering, duration, frequency and commonsense, all combined §1. Plain prompting and chain-of-thought prompting of LLMs do not reliably solve it §1. Most temporal benchmarks have no ground-truth timeline, so a model cannot be trained to produce one directly §1. Plain chain-of-thought distillation does not guarantee that the intermediate steps are correct, especially with little training data §1. In-context translation of a story into a timeline also misaligns with the benchmark questions: the model makes mistakes or focuses on irrelevant events §3.1.

Method

The paper has two parts: a dataset pipeline (TGQA) and the TG-LLM model that is trained on it. At inference, two LoRA adapters run in sequence §4.2.

Overview. Given a story and a question, the first adapter turns the story into a temporal graph (TG). The second adapter reasons over the TG, the computed external knowledge and the question, and outputs a chain of thought and an answer §3 §3.2 app. C. The external knowledge consists of temporal relations between time expressions in the story and the time gaps between them §1 §3.1 (2) tab. 7.

Step 1. Build TGQA §2. This is the training and evaluation data.

1. Split the temporal knowledge graph YAGO11k into subgraphs. For a seed entity, keep the events among its neighbours within three hops. No event is shared between the training, validation and test splits §2 Step 1.
2. Anonymise entities: each entity is replaced by a random name of the same type, using one global mapping per relation generated by GPT-3.5 §2 Step 1.
3. Ask GPT-3.5 to write a story that includes every event of the subgraph. The start and end of each event are stated separately in the prompt, because the model otherwise tends to drop end times §2 Step 2.
4. Generate question-answer pairs from the graph with rule-based templates covering the reasoning types in tab. 2 app. A.
5. Check the story-graph alignment semi-automatically: ask GPT-3.5 for the time of each event; events it cannot answer are flagged for manual review. Only the test stories were fully checked by hand §2 Step 3.

Step 2. Text-to-TG translation §3.1. This is the first adapter.

1. Entities and relations named in the questions are extracted, preferring rules over GPT-3.5 where rules apply §3.1 (1).
2. Time expressions in the story are found by GPT-3.5 and then filtered and normalised by a rule script. Temporal relations and time gaps between them become external knowledge §3.1 (2) tab. 7.
3. For TG construction, GPT-3.5 receives in-context examples (story, entities, relations, times, and the target graph) and writes a graph. It is then verified by asking GPT-3.5 the benchmark questions. Failures are checked by hand §3.1 (3) app. C.
4. Fine-tune Llama-2 with LoRA. Input is the story. Output is a chronological timeline in which the start and end of each event are separate entries. The chronological order was chosen after experiments [§3.1 "Supervised Fine-tuning"]. For TGQA the ground-truth graph is used directly §3.1.

Step 3. Temporal graph reasoning §3.2. This is the second adapter. It is trained with supervised fine-tuning on inputs made of the graph, the external knowledge and the question, with a chain of thought followed by the answer.

3a. Chain-of-thought bootstrapping §3.2.1 fig. 2.

1. For each training question, sample K chains of thought from GPT-3.5. Keep only the chains whose final answer matches the correct one §3.2.1.
2. Score each kept chain with a contrastive score: the length-normalised log-probability of the correct answer, plus a weighted plausibility growth against the wrong candidate answers eq. 2 eq. 3 eq. 4.
3. Turn the scores into a distribution with a softmax eq. 1 and sample one chain per training example. The paper says this keeps the data both useful and diverse, compared with best-of-N selection §3.2.1.

# CoT bootstrapping: keep correct CoTs, sample one by contrastive score
# Input:  context (TG, external knowledge, question), cots (K chains),
#         correct (answer a*), wrong_answers (candidate set A')
# Output: one chain of thought for the training example

def sample_cot(context, cots, correct, wrong_answers):
    # Drop chains whose final answer is wrong  [§3.2.1]
    kept = [c for c in cots if c.final_answer == correct]
    scores = []
    for cot in kept:
        # Mean per-token log-probability of the correct answer  [eq. 4]
        logp_right = mean_logprob(context, cot, correct)
        # Wrong-answer average (P-bar); the averaging rule is not stated
        logp_wrong = mean(
            [mean_logprob(context, cot, w) for w in wrong_answers]
        )
        # Plausibility growth G, a log-ratio  [eq. 3]
        growth = logp_right - logp_wrong
        # Contrastive score; gamma is a hyperparameter  [eq. 2]
        scores.append(logp_right + gamma * growth)
    # Softmax over scores gives the sampling distribution  [eq. 1]
    probs = softmax(scores)
    return sample_one(kept, probs)

3b. Graph data augmentation §3.2.2 fig. 3. The reasoning model is trained on verified graphs but tested on predicted graphs, which the authors see as a source of fragility §3.2.2. Four perturbations address this and the generalisation problem:

1. Remove irrelevant edges: a fact that appears in neither the question nor the chain of thought may be dropped eq. 5.
2. Replace relation names with a relation synonym eq. 6 rel_synonyms.txt.
3. Rename entities with one global mapping to random names of the same type eq. 7.
4. Shift all years by one global offset eq. 7.

The perturbations (3) and (4) must be applied to the question, chain of thought, candidates and answer together, so the example stays consistent app. B fig. 3.

# Graph augmentation for one TGQA training example  [§3.2.2] [eq. 5-7]
# Input:  example (tg, question, cot, candidates, answer), entity_map,
#         year_offset (both shared across one training run)
# Output: one perturbed training example

def augment_example(example, entity_map, year_offset):
    # (1) Drop irrelevant edges, not used by question or CoT  [eq. 5]
    facts = drop_irrelevant(example.tg, example.cot, example.question)
    # (2) Replace some relation names with a synonym  [eq. 6]
    facts = swap_relations(facts)
    fields = [
        facts,
        example.external_knowledge,
        example.question,
        example.cot,
        example.candidates,
        example.answer,
    ]
    # (3) Rename entities and (4) shift years, the same way in every field
    # [eq. 7]
    return [
        shift_years(rename_entities(f, entity_map), year_offset)
        for f in fields
    ]

Inference. The first adapter writes the timeline for the story. The reasoning adapter then reads the timeline, external knowledge and question §3 §4.2.

Equations. The paper's formulas, in the order they are cited above:

P_{sample}(c_k) = \operatorname{softmax}(score(c_k)) eq. 1

score(c_k) = \log P\left(a^* \mid q^{\dagger}, c_k\right) + \gamma G\left(c_k\right) eq. 2

G\left(c_k\right)=\log \frac{P\left(a^* \mid q^{\dagger}, c_k\right)}{\bar{P}_{\{a' \in A'\}}\left(a' \mid q^{\dagger}, c_k\right)} eq. 3

\log P\left(a \mid q^{\dagger}, c_k\right) = \frac{1}{|a|} \sum_{l=0}^{|a|} \log P\left(t_l \mid q^{\dagger}, t_{<l}\right) eq. 4

where q^{\dagger} := \{g, e, q\} (graph, external knowledge, question), \gamma is a weight the paper does not give a value for, a' ranges over the wrong candidate set A', and t_l is the l-th token of answer a eq. 2 eq. 4.

F_{irr} \in \{ F : P(c \mid g_{\setminus\{F\}}, e, q) \approx P_0 \}_{F \in g} eq. 5

F^{\prime} \in \{ f_R(F) : P(c \mid f_R(g), e, q) \approx P_0 \}_{F \in g} eq. 6

g^{\prime} = f_T(f_E(g)), \quad e^{\prime} = f_T(e), \quad q^{\prime} = f_T(f_E(q)) eq. 7

c^{\prime} = f_T(f_E(c)), \quad a^{\prime} = f_T(f_E(a)) eq. 7

Here P_0 = P(c \mid g, e, q), f_E is the entity mapping and f_T the time change eq. 5-7.

Hyperparameters

Values come from the paper where it states them, otherwise from the released code at code_commit. Where neither gives a value, the table says so.

Results

Author-reported. TGQA test set (100 stories, app. A). TimeQA and TempReason are evaluated on their test sets. T5 results are copied from the original papers and those models were fine-tuned on TGQA §4.1. Bold marks the paper's best results and is omitted here. A dagger (†) means "from the original paper", a double dagger (‡) means "answers parsed by GPT-3.5", and an asterisk (*) means "1000 random test samples". "–" means not reported.

Main results tab. 4

Headline numbers, all author-reported:

• TGQA: Llama2-13B SFT-TGR reaches EM 0.797 / F1 0.850 / Acc 0.819. GPT-4 with CoT reaches EM 0.821 / F1 0.865, and GPT-4 with a standard prompt reaches EM 0.772 / F1 0.829 tab. 4.
• TimeQA hard mode: SFT-TGR reaches EM 0.631 / F1 0.664, above GPT-4 with a standard prompt (EM 0.571 / F1 0.546) tab. 4.

Ablation on TGQA, Llama2-13B tab. 5

Abbreviations: TG, temporal graph; CoT, chain of thought; EK, external knowledge; bs, bootstrapping; aug, graph data augmentation tab. 5.

Human evaluation of 100 CoTs from the strategies, error rates tab. 3

ER-T1: wrong information used. ER-T2: logical inconsistency. ER-T3: external knowledge error. ER-T4: temporal-graph error tab. 3 §4.3. Lower is better. The paper reports that its strategies reduce every error type.

Per-question-type results on TGQA, exact match tab. 11 app. D. Q0 to Q8 are the nine question types in app. A. A dagger (‡) marks answers parsed by GPT-3.5; an asterisk (*) marks 1000 random test samples.

The paper states that TGQA results for zero-shot GPT rows are parsed by GPT-3.5 and may contain parsing errors app. D tab. 11. The 1-shot rows use one in-context example because of context-length limits app. D.

Paper vs code

Compared against the official repository (MIT licence) at code_commit.

• Eq. 1 has no scale factor; the code uses one. The paper's softmax is taken over the raw score eq. 1. The code samples with weights exp(-10 · s), where s is the negated score, so the effective distribution is sharper than Eq. 1 Models.py. This factor 10 is not mentioned in the paper.
• γ value. The paper says γ is a hyperparameter and gives no value eq. 2. The code sets γ = 0.5 Models.py.
• Wrong-answer average. Eq. 3 divides by a bar-P over the wrong candidates, without saying how they are averaged eq. 3. The code averages the length-normalised log-likelihoods of the wrong candidates, which is the mean of logs Models.py.
• CoT generation and filtering are not in the repository. The paper says GPT-3.5 generates K chains of thought per question and that chains with wrong final answers are discarded §3.2.1. The scoring script takes CoT lists that are already stored in each dataset row and does not generate or filter them CoT_bootstrap.py. K is not stated anywhere in the paper or the code.
• Augmentation scope and dataset size. The paper describes the four perturbations as applied during training §3.2.2. The code applies them only to TGQA and appends one augmented copy of each training example, so the training set is doubled SFT_TG_Reasoning.py.
• Irrelevant-edge removal. Eq. 5 defines irrelevant edges by their effect on the output probability eq. 5. The code uses string matching against the question and CoT, and drops a random 20% of the irrelevant facts utlis.py.
• Relation synonyms can keep the original relation. Eq. 6 describes replacing a relation with a synonym eq. 6. The synonym list in the code starts with the original relation, and the random choice runs over the whole list utlis.py [materials/TGQA/rel_synonyms.txt].
• Year offset. Eq. 7 has one global offset eq. 7. The code draws it uniformly from the integers −20 to 5 once per run [code: src/SFT_TG_Reasoning.py#L103]. The range is not explained in the paper.
• External knowledge receives the entity mapping too. Eq. 7 gives e′ = f_T(e), so the external knowledge gets only the time change eq. 7. The code applies both the entity change and the time change to it utlis.py.
• Base model in the released checkpoints. The paper's model list uses meta-llama/Llama-2-13b-hf app. B, and the code does the same SFT_TG_Reasoning.py. The README's config for the released checkpoints says base model "Llama2-13b-chat-hf" README.md. Pick one and state it.
• Training length. The paper gives no training length. The code sets max_steps = -1 and does not set an epoch count, so the training framework's default applies SFT_TG_Reasoning.py Models.py.
• Quantisation. The paper says LoRA on Llama-2 §3.1 and does not mention quantisation during training. The code loads the model in 8-bit for training SFT_TG_Reasoning.py.

The EM and F1 functions match the paper's description Evaluation.py. The perplexity-based accuracy was not compared line by line.

Pitfalls

• Headline claim is narrower than the abstract suggests. The paper says the 13B model is "comparable or even better than GPT-4 on all datasets" §4.3. In tab. 4 GPT-4 with a standard prompt is higher on EM and F1 for TimeQA easy (0.716 / 0.742 against 0.664 / 0.691), TempReason L2 (0.454 / 0.525 against 0.424 / 0.522) and TempReason L3 (0.431 / 0.485 against 0.356 / 0.469). On TGQA, GPT-4 with CoT is also higher (EM 0.821 against 0.797). Against GPT-4 with a standard prompt, the 13B model is higher only on TGQA (EM 0.797 against 0.772) and TimeQA hard mode (EM 0.631 against 0.571). Accuracy is not reported for the GPT rows, so the comparison uses EM and F1.
• The paper's Eq. 4 is off by one. It sums over l = 0 to |a|, which is |a| + 1 terms, and divides by |a| eq. 4. The code averages over the answer tokens only Models.py. Use the code's average when you reproduce the score.
• Tokenisation boundary. The code finds where the answer starts by tokenising the context and the full prompt separately and taking the minimum length Models.py. A boundary token can then leak into or out of the answer, and the code gives no check for this.
• Silent truncation. Scoring truncates prompts at 2048 tokens without warning Models.py. Long TGQA stories with many facts can be cut.
• Year replacement guard. The shifted years get an underscore inserted and all underscores are removed afterwards utlis.py. The code gives no comment. Our reading is that this stops one substitution from matching a value produced by an earlier one. A plain substring replacement over a mapping can apply two shifts to one year. Keep the guard, or use token-aware replacement.
• Augmentation is fixed for a run. The entity mapping and year offset are drawn once per data load and reused for every example SFT_TG_Reasoning.py. The code calls random without a seed, and no random.seed call appears in src/. Two runs will give different training data unless you fix the seed yourself.
• Stories are not fully verified. Only the test stories were checked by hand. Training stories were checked semi-automatically, so some misalignment can remain in the training data §2 Step 3.
• Generation is stochastic even at low temperature. The generation call uses sampling with top-k 10, top-p 0.9 and temperature 0.01 [code: src/Models.py#L252-L257]. Results can differ between runs. The paper reports single numbers without variance.
• Data depends on paid APIs at fixed versions. The stories, graphs, chains of thought and verification used GPT-3.5, and some examples used GPT-4 §4.2. The models are gpt-3.5-turbo and gpt-4-1106-preview app. B. Rebuilding TGQA needs access to those versions.
• Licensing of the data. The Ethics Statement says YAGO11k, TimeQA and TempReason are publicly available "for research purposes only" [Ethics Statement]. TGQA is built from YAGO11k with GPT-3.5 output. This card's licence covers the paper text only. Do not reuse the datasets or the trained adapters commercially without checking their terms.
• Licence of the code. The repository is MIT-licensed LICENSE. The base model's licence is not stated in the paper, so check it before you use or ship the adapters.
• Evaluation numbers come from the authors' own pipeline. Accuracy is the lowest-perplexity candidate among a candidate set built by the authors app. B. Results with another candidate set will differ.

Lineage

• Builds on: Llama 2 §4.2 and LoRA §3.1. The contrastive chain-of-thought score is inspired by SCOTT §3.2.1. The symbolic-reasoning direction is related to LogicLM §1.
• Datasets and benchmarks used: YAGO11k §2, TimeQA §4.1, TempReason §4.1.
• Compared against: Llama 2 at 7B, 13B and 70B, GPT-3.5, GPT-4, T5-based TimeQA and TempReason models, and REMEMO §4.1 tab. 4.
• Superseded by: not stated in this wiki.