Automated Fact Checking: Task formulations, methods and future directions

In plain words

Checking whether a claim is true is work that journalists do by hand. They read sources, weigh what those sources say and reach a verdict. Researchers in several fields have tried to automate parts of this, but they often use different names for the same task and rarely read each other's work.

This survey looks at that research from the viewpoint of language processing. It sorts the work by three things: what goes in (a sentence, a fact written as a triple, or a whole document), what comes out (a true or false label, a score, or a label plus supporting sentences), and what evidence the system uses. Evidence is the main dividing line. Some systems look only at how a claim is worded, while others check it against a knowledge base, against text such as Wikipedia, or against claims that were already checked. The authors also list what is still missing, such as checking claims against open sources and handling images and video. The paper is a map of the field. It does not add a new system or run new experiments.

TL;DR

• A survey, not a method. It unifies fact-checking task definitions across papers by input, output and evidence §1 §3.
• Inputs are knowledge-base triples, textual claims or whole documents, and the claim-extraction step is the hard part for documents §3.1.
• Evidence types run from none (claim wording only) through metadata, knowledge graphs, text and past fact checks to social-network signals §3.2.
• Methods are mostly supervised. Entailment-based stance models assume the evidence is given, and the FEVER pipeline adds retrieval first §5.
• No experiments of its own. Results are dataset statistics and task definitions, not comparable scores §4 §3.3.

Problem

Fact checking is the assessment of whether claims made in speech or writing are true. Journalists do it by hand, and it can take anywhere from under an hour to a few days for a single claim §1. The volume of content makes automation attractive, and the paper's aim is to make research in NLP and related fields comparable §1.

Research on this topic is spread across communities that use inconsistent terms. One example from the paper: the same stance-like task has been called fake news detection, stance classification and incongruent headline detection §1. The paper treats evidence as the main axis of comparison, because it determines what a system can output and how it must be evaluated §1 §3.2.

The paper also separates verification from fact checking. Verification assesses the trustworthiness of sources, while fact checking assesses the logic and context of a claim §2. Misinformation is inaccurate information shared without intent to mislead, and disinformation adds that intent. Fact checking can detect misinformation but cannot tell it apart from disinformation §2. The paper avoids the term "fake news" because its meaning has become diluted §2.

Out of scope: how fact checks are communicated or used by social platforms §1.

Method

The paper has no method of its own. It defines the task along three axes and then describes the existing approaches on each §3.

1. Inputs §3.1. Three input types are considered.

1. Subject-predicate-object triples, such as (London, capital_of, UK). They allow lookup in structured knowledge bases such as Freebase, but turning text into triples is a natural-language understanding task in its own right §3.1.
2. Textual claims: short sentences cut from longer passages, as in PolitiFact and Full Fact. The paper calls them the most popular input in NLP §3.1. A four-part claim typology from the HeroX challenge covers numerical claims, entity and event properties, position statements and quote verification §3.1. Such claims often need entity disambiguation before checking §3.1.
3. Whole documents. The system must first extract the claims, either as triples through relation extraction or through sentence-level classification §3.1.

2. Evidence §3.2. The paper orders the approaches by the evidence they use.

1. None: the veracity is predicted from the claim wording alone, through surface linguistic features §3.2.
2. Metadata: originator, speaker profile and media source, which act as a prior and can serve as part of a justification §3.2.
3. Knowledge graph, retrieved: find the triple that supports or refutes the claim, then score the difference between claimed and stored values with rules §3.2. The stated limit is that the needed fact must already be in the graph §3.2.
4. Knowledge graph, topology: predict from the graph's structure how plausible a claim-edge is. The paper notes that an unlikely fact is not thereby false §3.2.
5. Text: article headlines for stance §3.2; whole articles, as in the Fake News Challenge §3.2; and sentences retrieved from Wikipedia, as in FEVER §3.2. Triple scoring against Wikipedia and a web-scale corpus appears in the WSDM cup §3.2.
6. Previously fact-checked claims: a new claim is matched to a repository and the stored label is returned. This allows labels beyond true or false, but limits checking to claims close to the repository §3.2.
7. Social-network behaviour: the spread pattern of posts is used to predict whether the claims in them are true §3.2.

The paper notes that the trustworthiness of the evidence sources is rarely considered, and that most work treats sources such as Wikipedia as given §3.2.

3. Output §3.3. The output types are these.

1. Binary true or false §3.3.
2. Ordinal labels on a multi-point scale, as Vlachos and Riedel 2014 proposed §3.3.
3. Multiclass labels on a multi-point scale that ignore the order between them §3.3.
4. A numeric score for each triple, evaluated by score differences and by rank correlation with annotators §3.3.
5. Supported, refuted or merely reported, with an added "irrelevant" label in the Fake News Challenge §3.3.
6. FEVER's output: a three-way label (supported, refuted, not enough information) plus, for the first two labels, the evidence sentences. A correct label with wrong evidence counts as wrong §3.3.

The paper also states that a label backed by Wikipedia alone is not a complete fact check, unless world knowledge is restricted to Wikipedia §3.3.

4. Methods §5. The paper groups existing methods as follows.

1. Supervised text classification on labelled claims or articles. The paper calls this the majority approach, and its main limit is the lack of world knowledge §5.
2. Path-based network analysis over knowledge graphs. A claim is scored by the cost of traversing a path between its entities, weighted by how connected the path is, with transitive closure applied §5.
3. Collective classification that combines subjectivity cues with the co-occurrence of a triple with other triples on the same topic §5.
4. Recognising textual entailment: the premise is an article, the hypothesis is the claim, and the label is for, against or observing. Most FNC teams used this form §5.
5. Pipelines for Wikipedia-scale evidence: document retrieval, then sentence selection, then the entailment component §5. The schematic below shows the flow, not a detailed algorithm from the paper.

# Pipelined fact verification  [§5]
# Input:  claim (text); corpus of documents (Wikipedia)
# Output: label in {supported, refuted, not enough info}; evidence sentences

def verify(claim, corpus):
    # 1. Retrieve candidate documents for the claim  [§5]
    documents = retrieve_documents(claim, corpus)

    # 2. Select sentences that may serve as evidence  [§5]
    evidence = select_sentences(claim, documents)

    # 3. Classify the claim against the selected evidence  [§5]
    label = entailment_model(claim, evidence)

    return label, evidence

6. Distant supervision for numerical claims: sentences with numbers close to values stored in a graph are taken as positive examples, and relation extraction finds the surface patterns §5.
7. Claim matching to already fact-checked claims, reduced to sentence similarity. The paper says this handles only repeated or paraphrased claims §5.
8. Speaker profiling, where the originator's credit history was the most influential feature in one study. The paper raises ethical concerns and notes the feature is missing for new sources §5.

Hyperparameters

Not applicable: the paper trains no model and states no hyperparameters §1 §7.

Results

Author-reported. The paper has no experiments of its own. The table gives the dataset sizes it reports, so the setup can be seen. Its figures are not comparable across rows because the tasks differ.

The paper says that FEVER claims average 8 words long §7. It reports no accuracy or other metric for any system.

Paper vs code

No official code. The paper and its arXiv metadata link no repository, and a survey has no implementation of its own to check. The paper's footnote gives the license only p. 1. Nothing from a third-party repository is used.

Pitfalls

• The paper is a survey. Its numbers are dataset sizes as reported by other papers, not measured results. Do not read the table as a leaderboard.
• Section numbers refer to the arXiv v2 source fetched for this card (updated 2018-09-05). Check v1 before citing a section by number.
• The LaTeX source contains commented-out passages (for example on stance detection, hate speech and a discussion of stance-based datasets). They are not part of the paper and are not cited here.
• "Verification" and "fact checking" are used with specific meanings, and the paper's terms (claim, evidence, verdict) are its own §2. Many cited systems use different labels for the same thing.
• Methods that assume evidence is given (entailment models) are inapplicable when retrieval is needed, as with Wikipedia-scale evidence §5.
• Most surveyed datasets are small or tied to one source (PolitiFact, Wikipedia). The paper names this as a barrier to training machine-learning systems §4 §7.
• The paper argues that open-world evaluation, meaning checking against sources not fixed in advance, was not handled by any surveyed work, and the HeroX challenge used manual evaluation §7.
• Verification of the evidence sources is left out under the closed-world assumption used by most of the surveyed models §6 §7.
• Multimodal evidence (images, video) is named as a gap but not reviewed §7.
• The paper dates from 2018. It cannot say which later approaches have been adopted, so verify any "current state" claim separately.

Lineage

• Builds on: none. The paper is a survey of prior work across journalism, NLP and related fields §1 §2.
• Covers the works that introduced the main task definitions: Vlachos and Riedel (2014, 2015), Ferreira and Vlachos (2016), the Fake News Challenge (2017), Wang (2017), Rashkin et al. (2017), Thorne et al. (2018, FEVER) and the WSDM cup triple-scoring task §3 §4. The arXiv ids are not given in the fetched source.
• Discusses related tasks in §6: source credibility (PageRank, TrustRank, knowledge-based trust), deception detection, rumour detection and click-bait detection.
• Superseded by: not stated in this paper.