Coreference resolution

Overview

Coreference resolution finds the mentions in a text that refer to the same entity and puts them into one cluster. A mention can be a name, a noun phrase or a pronoun. The survey treats it as the clustering side of a wider pair of tasks: anaphora resolution links a reference to one earlier item, while coreference resolution groups all mentions of an entity 1805.11824 §1. The two are often confused, so check which one a paper or benchmark measures 1805.11824 §1. The link view is on the anaphora resolution page.

The invariant all variants share: each mention either joins an existing cluster or starts a new one. Mention-pair and mention-ranking models make this choice by scoring candidate earlier mentions 1805.11824 §8.2 1805.11824 §8.3. Systems differ in how they find mentions, which features or learned vectors they use, and how they train 1805.11824 §8.

The survey covers English NLP only 1805.11824 §1. This page rests on two cards: the survey, which gives the taxonomy, metrics and reported scores, and Quoref, a benchmark whose questions need coreference to answer 1908.05803 §1. The systems themselves are described in their original papers, which have no cards here. Check those before implementing any of them.

Variants

Rule-based systems

Early systems use discourse properties and syntactic and semantic constraints. The survey names the BFP centering algorithm as an early discourse-based method for pronouns 1805.11824 §8.1. It cites a multi-sieve deterministic system as the best rule-based system on CoNLL 2012 1805.11824 §8.1 1805.11824 tab. 4. The survey's table gives the rule-based row (Lee et al. 2013) a CoNLL F1 of 54.82 1805.11824 tab. 7. Card: Anaphora and Coreference Resolution: A Review.

Statistical and machine-learning systems

These use mention-pair or mention-ranking models trained on hand-built features. The group also includes joint inference with integer linear programming and graph partitioning with conditional random fields 1805.11824 §8.2 1805.11824 tab. 5. Reported CoNLL F1 runs from 61.40 (Durrett and Klein 2013) to 63.02 (Clark and Manning 2015, entity-centric) 1805.11824 tab. 7. Card: Anaphora and Coreference Resolution: A Review.

Neural mention-ranking systems

A feed-forward network scores each candidate antecedent for a mention. Wiseman et al. 2015 uses tanh layers over mention and pair features and a latent-variable loss 1805.11824 §8.3 1805.11824 eq. 23 1805.11824 eq. 26. Wiseman et al. 2016 adds an RNN that reads each candidate cluster 1805.11824 §8.3. Clark and Manning 2016 pools mention-pair vectors into cluster-pair vectors and then chooses merge or pass actions. Its deep form replaces the heuristic loss with a reinforcement-learning objective 1805.11824 §8.3 1805.11824 fig. 4. CoNLL F1 across these rows runs from 63.39 to 65.73 1805.11824 tab. 7. See the Contradictions section for two numbers in this group that the text and the table disagree on. Card: Anaphora and Coreference Resolution: A Review.

End-to-end span models

Lee et al. 2017 detects mentions and links them in one model. A bi-LSTM reads the word vectors, every span up to width 10 is scored, the top-scoring spans are kept, and each is paired with the preceding 250 mentions 1805.11824 §8.3 1805.11824 tab. 6. The survey reports 68.80 CoNLL F1 for it, the highest figure in its table 1805.11824 tab. 7, and calls it the best system it knew of at the time of its posting 1805.11824 §8.3. The fraction of spans kept is not stated 1805.11824 §8.3. Card: Anaphora and Coreference Resolution: A Review.

Metrics

The survey defines seven scores. Each is a precision and recall over clusters or links 1805.11824 §5:

• B-cubed, per mention, with weights 1805.11824 §5.1 1805.11824 eq. 1.
• MUC, link-based: it counts the link changes needed to turn the output into the gold clusters 1805.11824 §5.2 1805.11824 eq. 3.
• CEAF, which first finds the best one-to-one mapping between output and gold clusters, then scores the mapped pairs 1805.11824 §5.3 1805.11824 eq. 6.
• ACE value, similar to CEAF but without normalisation 1805.11824 §5.4.
• CoNLL score, the unweighted mean of MUC, B-cubed and CEAF F1, as used in the CoNLL 2012 shared task 1805.11824 §5.5 1805.11824 eq. 12.
• BLANC, a Rand-index style score over coreferent and non-coreferent links 1805.11824 §5.6.
• LEA, link-based and weighted by entity size, which the survey introduces to avoid the mention-identification effect 1805.11824 §5.7.

The survey notes that MUC ignores singletons 1805.11824 §5.8. It also argues that published results are hard to compare because the metrics, splits and mention-detection setups differ 1805.11824 §5.8 1805.11824 §9. Several formulas in the survey need care before you implement them: the B-cubed weights are described differently in the text and the equation, the CoNLL formula does not name the CEAF variant, and one BLANC term carries the wrong label 1805.11824 eq. 1 1805.11824 eq. 12 1805.11824 eq. 16. Check each against the original paper.

# CoNLL 2012 headline score  [arxiv:1805.11824 §5.5] [arxiv:1805.11824 eq. 12]
# Input:  F1 values for MUC, B-cubed and CEAF, in the same units
# Output: CoNLL F1, the unweighted mean of the three

def conll_score(f1_muc, f1_b3, f1_ceaf):
    # Equal weight for each metric  [arxiv:1805.11824 §5.5]
    return (f1_muc + f1_b3 + f1_ceaf) / 3

When to use which

Benchmarks

Quoref: coreference inside span-QA

Quoref has about 24K questions over 4.7K Wikipedia paragraphs 1908.05803 tab. 1. Workers must find two or more co-referring spans and write a question whose answer depends on knowing that they refer to the same thing 1908.05803 §2. Each question is checked by an adversary, an uncased BERT base QA model trained on SQuAD 1.1. A question is accepted only if the adversary's answer does not match the worker's answer 1908.05803 §2. The paper does not define "match".

The authors judged 78% of 100 sampled questions to need coreference 1908.05803 §3. On test, the best baseline (XLNet QA, base) reaches 70.51 F1, BERT QA reaches 66.39, and a passage-only baseline reaches 28 to 29 1908.05803 tab. 3. The authors estimate human performance at 93.41 F1 on 400 test questions 1908.05803 §4.3. In an error analysis of 100 dev questions, about half of BERT QA's correct answers point at the first or most frequent entity in the paragraph, against 12% of its wrong answers 1908.05803 §4.4. Those shortcuts make the scores harder to read without a passage-only control; see benchmark validity. Card: Quoref.

Implementation sketch

Most neural systems in the survey share a mention-ranking core: score each earlier candidate for a mention, add a "no antecedent" choice, and link to the best one. The survey gives the scoring form for Lee et al. 2017 but not every term 1805.11824 eq. 28. It also does not say how scores become probabilities in the training objective 1805.11824 eq. 29.

# Greedy mention ranking  [arxiv:1805.11824 §8.2] [arxiv:1805.11824 §8.3]
# Input:  mentions in document order, pair scorer s(i, j), null score
# Output: cluster id per mention

def resolve(mentions, score, null_score):
    cluster = {}
    for i, m in enumerate(mentions):
        # Candidates: earlier mentions, plus a "start new entity" option
        best_j, best = None, null_score(m)
        for j in range(i):
            s = score(mentions[j], m)  # pair score  [arxiv:1805.11824 eq. 28]
            if s > best:
                best_j, best = j, s
        # Join the best antecedent's cluster, or open a new one
        cluster[i] = cluster[best_j] if best_j is not None else i
    return cluster

Contradictions

• Clark and Manning 2016 (improving), CoNLL F1. The text gives 65.39 1805.11824 §8.3. The table gives 65.29 for the same system 1805.11824 tab. 7. Use the table value and check the original paper.
• Clark and Manning 2016 (deep RL), component scores. The text assigns 65.73 and 63.4 to CEAF and B-cubed 1805.11824 §8.3. The table lists 65.73 as the CoNLL score, with CEAFe 59.23 and B-cubed 63.40 1805.11824 tab. 7. The table is internally consistent with the CoNLL average, so it is the more likely reading.
• Quoref learning rate. The paper typesets the learning rate as "3^{-5}". The intended value is probably 3e-5, but the exponent base is not written out 1908.05803 app. B. Check the stated settings against any released training configuration before you reuse them 1908.05803 app. B.

Papers

• Anaphora and Coreference Resolution: A Review: survey of reference types, the seven metrics, rule, statistical and neural systems, and datasets, with a CoNLL 2012 comparison table (author-reported).
• Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning: span-QA benchmark whose questions need coreference, filtered by a live adversary.