Document-Level Event Argument Extraction by Conditional Generation

In plain words

An event extraction system reads text and fills in a form for each thing that happened: what kind of event it was, and who or what took part. Most systems only look at one sentence at a time. Real news often mentions a participant in one sentence and the event in another, or refers to someone as "he" when the name appeared earlier. Sentence-by-sentence systems miss those pieces or return a bare pronoun that tells the reader nothing.

The paper's idea is to treat the form-filling as writing. Each event type comes with a short sentence template with blank slots, such as "someone attacked someone with something at a place". The model reads that template together with the whole document and writes the template out again with the blanks filled in by words copied from the document. Because the model writes one filled sentence per event, it can leave a slot empty, put several people in one slot, or pick a name from far away, without a separate tuning step.

The authors also built a new benchmark, WikiEvents, where annotators marked every event across whole documents and linked mentions that refer to the same entity. They report that the model does better than the baselines on this benchmark and on two older datasets. A second part of the work tries to find events of types the model was never trained on, using a few keywords per type.

TL;DR

The model is a BART encoder-decoder. It is given an event template with generic argument slots, followed by the document with the trigger word marked, and it generates the filled template. Its output distribution is restricted to tokens that appear in the input §2.1, eq. 2. The author-reported gains are largest on the document-level and "informative mention" tasks §4.2, §4.3. The keyword-based trigger model is trained without mention-level labels and is reported to transfer to new types §2.2, §4.4.

Problem

Sentence-level event extraction misses arguments that sit in another sentence, and it returns pronouns where a name would be useful §1. The annotation standard of the older datasets favours the nearest mention of an argument. In WikiEvents, only 34.5% of the arguments found in the trigger's own sentence are the most informative mention of their entity §3.3. Existing document-level answers treat extraction as question answering, which needs thresholds to handle missing or repeated arguments §1, §4.2.

Assumptions: the event ontology supplies one template per event type §2.1. The argument extractor takes triggers as given §2.1. The paper's zero-shot part assumes a few keywords per type and no mention-level labels for the new types §2.2, §4.4.

Method

Overview. The argument extractor is a conditional generator §2.1. The condition is an unfilled template and a context. The output is the filled template, where a slot is either replaced by an argument or left as a generic placeholder when no argument is found §2.1, fig. 2.

Input and target construction §2.1, fig. 2.

1. Take the event template for the trigger's event type. Rewrite every numbered slot (arg1, arg2, ...) as one generic placeholder token §2.1.
2. Form the encoder input as the template, then the document. The trigger word is wrapped in two special marker tokens <tgr> §2.1, fig. 2.
3. Form the decoder target as the same template with each slot filled by the argument text found for its role. If one role has several arguments, join their texts with the word "and" §2.1.
4. If a role has more arguments than slots, the extra ones go into the last slot for that role KAIROS_data_module.py.

# Build encoder input and gold target for one event mention
# Input:  template with numbered slots; document tokens; trigger span
# Output: encoder input text; decoder target text

def build_example(template, document, trigger, arguments):
    # Replace numbered slots with one generic placeholder  [§2.1]
    input_template = replace_numbered_slots(template, "<arg>")
    # Wrap the trigger word in markers in the document  [§2.1] [fig. 2]
    marked_document = wrap_trigger(document, trigger, "<tgr>")
    # Fill each slot with its role's argument(s), joined by "and"  [§2.1]
    target = fill_slots(template, arguments, joiner=" and ")
    # Unfilled slots keep the generic placeholder  [§2.1]
    return input_template + marked_document, target

Generation with a restricted vocabulary §2.1, eq. 1, eq. 2. The decoder produces the target one token at a time, conditioned on all earlier tokens and on the encoder input. Each step scores every token that appears in the input by the dot product between the decoder state and that token's embedding, applies a softmax over those input tokens, and gives every other vocabulary item zero probability. The restriction is meant to stop the model from inventing arguments that are not in the text §2.1, eq. 2.

# One decoding step, restricted to tokens of the input  [§2.1] [eq. 2]
# Shapes: decoder_state (d,); input_tokens (n,)

def next_token_distribution(decoder_state, input_tokens):
    # Embed every input token; repeats get one score each  [eq. 2]
    input_embeddings = embed(input_tokens)               # (n, d)
    # Dot product between the decoder state and each input embedding  [eq. 2]
    scores = input_embeddings @ decoder_state            # (n,)
    # Softmax over input tokens only; all other tokens get probability 0  [eq. 2]
    return softmax(scores), input_tokens

Training objective §2.1, eq. 3. The model minimises the negative log-likelihood of the gold filled templates over the training set, which is an autoregressive product over target tokens eq. 1, eq. 3. The released code averages the per-token loss instead of summing it; see Paper vs code.

Type-constrained reranking §2.1, eq. 4. The paper also proposes appending type statements to each filled argument, such as " is a person", and reranking candidate outputs by the sum of their generation log-probability and the best log-probability of a valid type statement eq. 4. Roles with several allowed entity types take the maximum over those types. The paper argues that a type statement with low probability, such as "tax plan is a person", should prune a wrong filler §2.1. The released code does not implement this step.

Keyword-based trigger extraction (TapKey) §2.2. Trigger detection is sequence labelling with IO tags §2.2. Each event type gets a class vector from a few keywords (3 per type in the paper's example for StartPosition), computed by averaging BERT vectors of the keyword occurrences after masked-word filtering §2.2.

Scoring follows a linear-chain CRF §2.2, eq. 5. Emission scores use a TapNet-style projection that is solved in closed form from the class vectors and a set of reference vectors §2.2, eq. 6, eq. 7. Transition scores use two diagonal matrices, one for staying in the same event type and one for transitions involving O §2.2, eq. 8. Training minimises the negative log-probability of the tag sequences plus a term that keeps the reference vectors close to orthonormal §2.2, eq. 9. In the zero-shot setting, the model is first trained on pseudo-labels from cosine similarity between tokens and class vectors, with tokens of uncertain similarity marked as unknown §2.2.

# Pseudo-labels for zero-shot trigger training  [§2.2]
# Thresholds: appendix PDF in the code repo, §2.1 (cited as [app. §2.1])
# Input:  token embedding (d,); class vectors (k, d)

def pseudo_label(token_embedding, class_vectors):
    # Cosine similarity to every event class vector  [§2.2]
    similarities = cosine(token_embedding, class_vectors)    # (k,)
    best = argmax(similarities)
    # Accept the event type above 0.65  [app. §2.1]
    if similarities[best] > 0.65:
        return best
    # Label O when no class reaches 0.4  [app. §2.1]
    if similarities[best] <= 0.4:
        return "O"
    # In between: ignore the token in the loss  [app. §2.1]
    return "X"

Hyperparameters

Values come from the released scripts and code, and from the appendix PDF shipped in the code repository. The arXiv source of v1 refers to "Appendix A–D" but does not include them, so every appendix value below is cited to that PDF. "Not stated" means neither the paper source nor the repository gives it.

Results

Author-reported. Setup: the RAMS test set with the official scorer, the WikiEvents test set, and the ACE05 split defined in the paper §4.1, tab. 3. Hardware is not stated. Baselines: Two-step (RAMS) §4.2, BERT-CRF, BERT-QA and BERT-QA-Doc (WikiEvents) §4.2, §4.3, and Prototype networks plus BERT-QA for the zero-shot setting §4.4.

Datasets tab. 2, tab. 3.

RAMS, argument extraction tab. 4. Official scorer, test set.

WikiEvents, argument extraction (nearest mention) tab. 5. Test set.

WikiEvents, informative argument extraction tab. 6. Test set; the model is trained on informative mentions.

ACE zero-shot event extraction tab. 8. Trigger F1 (TI, TC) and argument head F1 (AI, AC), with the number of event types seen during training.

The abstract's ratios come from this table: 97% of supervised trigger F1 is 72.31 / 74.36, and 82% of supervised argument performance is 44.19 / 53.71 tab. 8, abstract. The abstract's gains of 7.6 F1 (RAMS) and 5.7 F1 (WikiEvents) match the Head F1 and Coref F1 columns respectively; see Pitfalls. The appendix PDF in the repository also has a fuller ACE table with precision and recall, plus a supervised ACE argument table; the arXiv source does not include them, so they are not reproduced here NAACL_2021_Appendix.pdf Tabs. 4–5.

Paper vs code

• Trigger model is not in the code. The repository has argument extraction only. No TapKey, CRF, prototype or pseudo-labelling code is present (a search for tapnet, tapkey, prototype and crf finds nothing). Trigger hyperparameters come only from the appendix PDF in the repository NAACL_2021_Appendix.pdf Tab. 2. Zero-shot results cannot be reproduced from this repository. §2.2, §4.4
• Type-constrained reranking (eq. 4) is not in the code. The test step runs greedy generation and takes the output as is model.py. Nothing in src/ mentions a type statement or reranking. §2.1, eq. 4
• "And" is not always available. The paper says multiple arguments are joined with "and" and that the vocabulary is restricted to input tokens §2.1. The released forward pass only scores input tokens constrained_gen.py. The helper that would always allow the "and" token, remove_unseen, is not called in either path; its call is commented out constrained_gen.py. So "and" can be generated only if it appears in the input text. §2.1, eq. 2
• Loss normalisation differs. Eq. 3 sums the log-likelihood over the dataset eq. 3. The code computes cross-entropy over target tokens with the default mean reduction, then averages over the batch constrained_gen.py model.py. A sum and a mean differ by the number of target tokens, which changes the effective learning rate; the paper does not say which it used.
• Decoding differs. Appendix Table 1 says beam size 4 NAACL_2021_Appendix.pdf Tab. 1. The released test step uses greedy decoding with one beam model.py. The paper's own example also uses greedy decoding §2.1, tab. 1.
• Max input length differs for ACE. The appendix says 512 for argument extraction, but the ACE data module truncates to 170 tokens ACE_data_module.py.
• The model used for RAMS differs. The RAMS script runs --model=gen train_rams.sh. That model is the unrestricted BART with a full-vocabulary output layer network.py network.py. The paper does not say which model produced the RAMS numbers §4.2.
• Informative training needs an undocumented flag. The paper trains on informative mentions for §4.3 §4.3. The code uses informative mentions only when --use_info is set train.py. No shipped script sets it, and the informative mention lists come from a coreference file that is not in the repository (the README points to an S3 bucket) README.md.
• Shipped scripts differ from each other. The WikiEvents training script uses 2 × 8 and the test script 4 × 4 (both 16 effective) train_kairos.sh test_kairos.sh.
• Appendix location. The paper refers to Appendix A to D, and the arXiv source has none of them. The appendix PDF in the repository numbers its sections 1 to 5 and has no "Appendix C" heading NAACL_2021_Appendix.pdf. Cite the repository copy, not the paper's letters.
• Pointer embeddings. Eq. 2 uses "Emb(w)" without saying which table. The code uses the encoder-side embedding of each input token, multiplied by embed_scale constrained_gen.py constrained_gen.py. Whether that is the same matrix as the shared output embedding was not checked.
• Checkpoint choice is hard-coded. The WikiEvents evaluation script loads a file named epoch=2-v0.ckpt test_kairos.sh. Training keeps the best two checkpoints by validation loss train.py, so the file name depends on which epoch won. Check which checkpoint was selected before comparing numbers.
• Shipped predictions. The repository contains predictions for WikiEvents and says the scorer reproduces Table 5 from them README.md predictions.jsonl. This card did not re-run the scorer.

Pitfalls

• The headline gains use different columns. The abstract's 7.6 F1 on RAMS equals 57.32 − 49.7, the Head F1 gap. The Span F1 gap is 48.64 − 41.8 = 6.84 tab. 4. The abstract's 5.7 F1 on WikiEvents matches the argument classification Coref F1 gap, 65.11 − 59.36 = 5.75 tab. 5. The 9.3 F1 informative gain matches Arg-C Coref F1, 57.04 − 47.70 = 9.34 tab. 6. Always say which column you are comparing.
• Output length is capped at 30 tokens at test time. Test generation uses max_length=30 model.py. Training targets may be up to 70 tokens KAIROS_data_module.py. Templates with several filled roles could be cut off. This card did not check whether any gold target exceeds 30 tokens.
• Scatter with repeated tokens is not well defined. Each input token's score is written into its vocabulary slot with a scatter. The code comments that this is technically incorrect when indices repeat constrained_gen.py. A token that appears several times in the input is therefore scored by whichever write lands last, and the result can differ between runs on GPU. Test this if you reimplement the model.
• Pointer-only output is not softmax over a full vocabulary. Non-input tokens get −1000 as a logit, not −∞ constrained_gen.py. Effectively zero probability, but not exactly zero.
• Context windows are built around sentences. The window is 400 words KAIROS_data_module.py. Adjacent sentences are added while they fit KAIROS_data_module.py. A sentence longer than the window is cut to 200 words on each side of the trigger start KAIROS_data_module.py. The same document is encoded once per event mention, so inference cost grows with the number of events.
• Training drops some events. Training events with no arguments are skipped. Dev and test keep them KAIROS_data_module.py. Event types missing from the ontology are skipped in all splits KAIROS_data_module.py.
• Pronoun informative mentions are dropped. With --use_info, an argument whose informative mention is a pronoun is left out of the target, so its slot stays empty KAIROS_data_module.py. The model then learns to leave those slots empty.
• Argument order follows the annotation file. Several arguments in one role are joined in the order the annotation lists them. The code does not sort them KAIROS_data_module.py. The paper does not say whether order matters.
• Metric definitions are specific. Informative mentions rank names over nominals and nominals over pronouns, and take the longest among equals. A prediction counts only if it is the most informative mention in the whole document §3.1. Head F1 scores the annotated head word; Coref F1 gives credit for any mention coreferent with the gold argument §3.1.
• Type constraints are not enforced at decode time. The paper's example has the model fill a slot that needs a person with a topic phrase tab. 1. The released decoder has no such check. Eq. 4 is the paper's proposed fix, and it is not in the code.
• Zero-shot depends on keyword quality. Class vectors come from a few seed keywords, filtered with a masked language model §2.2, code: NAACL_2021_Appendix.pdf §1.2. The paper's example uses 3 seeds per type. The appendix says nominal forms are added when triggers can be nominals. The seed keywords are in the ontology files (for example keywords in event_role_ACE.json). The model that uses them is not in the repository.
• Data is not in the repository. WikiEvents, RAMS and ACE must be obtained separately, and the coreference files are needed for informative mode README.md.
• Licensing. The code is MIT, with copyright held by "Zoey Li" LICENSE. The repository's license does not name the appendix PDF separately. This card uses only its numbers, cites them, and does not reproduce its tables or algorithm. §4.4, code: NAACL_2021_Appendix.pdf

Lineage

• Builds on: BART (Lewis et al., 2020) as the encoder-decoder backbone §2.1; T5 is named as an alternative backbone §2.1. Paper does not give arXiv ids for these in its source.
• Trigger model builds on TapNet (Yoon et al., 2019) and its extension to CRFs, with class vectors from masked category prediction §2.2.
• Compared against: BERT-CRF (sentence-level sequence labelling, Shi and Lin, 2019) §4.2; BERT-QA, a question-answering approach (Du et al., 2020) §4.2; Two-step head detection and expansion on RAMS (Zhang et al., 2020) §4.2; Prototype networks for zero-shot trigger detection (Snell et al., 2017) §4.4.
• Superseded by: not stated in the content available here.