Zero-Shot Transfer Learning for Event Extraction

In plain words

Event extraction means finding the things that happen in a sentence (the trigger word, such as "dispatched" or "conflict") and the people, places and objects involved in them. Most systems learn each event type from many labelled examples, so a new type needs new labelling work before it can be found.

This paper treats the task as matching. Each sentence fragment that might be an event is turned into a small structure of which words relate to which. Each event type from an existing ontology is also turned into a structure: the type name plus the roles it expects, such as a destination or an attacker. A shared network learns to place a fragment close to its correct type in one common space. Afterwards, a new type only needs its ontology entry, because the model can compare the fragment's structure with that entry and pick the closest one. The result is a system that handles types it has never seen labelled, at the cost of lower accuracy than a fully supervised system on the same types.

TL;DR

• Event extraction is recast as grounding: each candidate trigger or argument is mapped to its closest event type or role in an ontology §1 fig. 1 fig. 2.
• Event mentions (AMR-derived structures) and event types (type plus roles) are composed by learned functions and encoded by one weight-shared CNN, trained with a margin ranking loss §2.3 §2.4.
• Trained on a few "seen" ACE types and tested on unseen types with no annotation for those types, it reaches trigger Hit@1 of 33.47% and a trigger F1 that is lower than the supervised LSTM on the same classes (author-reported) tab. 5 tab. 7.
• The authors report that an LSTM needs about 500 annotated event mentions to match their zero-shot results on 23 types §3.3 fig. 4.

Problem

Supervised event extraction learns features from annotated mentions of each event type, so types with no annotations cannot be handled §1. The ACE ontology has 33 types and FrameNet alone adds more than a thousand frames, and annotating each would be costly §1 §3.3.

Method

The approach has four parts: candidate identification, structure construction and composition, a shared embedding trained with ranking losses, and zero-shot classification at test time §2 fig. 3.

Candidate identification §2.2. Each sentence is parsed to AMR. A word-sense disambiguation tool links concepts to OntoNotes senses. Every noun or verb concept that maps to an OntoNotes sense is a candidate trigger; concepts that match FrameNet verbs or nominal lexical units are also candidates §2.2. Candidate arguments for a trigger are the AMR relations listed in tab. 1: core roles (ARG0 to ARG4), non-core roles, temporal relations and spatial relations tab. 1.

Event mention structure §2.3. For a trigger t, the structure S_t is a set of tuples ⟨w1, λ, w2⟩: two AMR concepts and the AMR relation between them §2.3 fig. 3. Each relation λ has its own matrix M_λ. The two concept vectors are concatenated, multiplied by the matrix, and passed through a nonlinearity f, which yields updated vectors for both concepts §2.3, unnumbered equation.

Event type structure §2.3. For an ontology type y, the structure S_y has the type as root and one tuple ⟨y, r⟩ per predefined role r §2.3. The type vector and role vector are composed by one shared third-order tensor U, which takes the place of a per-relation matrix §2.3, unnumbered equation. The paper follows the recursive-tensor idea of Socher et al. and assumes one implicit relation between every type and role §2.3.

Joint embedding of mentions and types §2.4. Each tuple is a d × 2 block. Tuples are stacked top to bottom into a feature map, zero-padded on the right to h* tuples for mentions and p* for types §2.4. A single CNN with shared weights encodes both mention and type structures. The filter width is 2 and the stride is 2, so each filter reads one tuple at a time, with a tanh activation. Max-pooling over tuples gives the structure vector §2.4. The mention vector is the concatenation of the trigger's vector and its structure vector; the type vector is built the same way §2.4.

Loss §2.4. The model scores a mention and a type by cosine similarity and pushes the correct type above every other type by a margin m §2.4, L1. The authors found this plain hinge loss overfits the few seen types. Two changes follow. First, mentions labelled "Other" (no seen type applies) are added as negatives §2.4. Second, the loss takes the maximum violation over negatives, not the sum, and for Other mentions the reference type is the highest-ranked seen type §2.4, L^d_1.

Argument roles §2.5. For a trigger t and candidate argument a, the path S_a of tuples connecting them is encoded by the same CNN. Each role r of the predicted type is encoded as ⟨y, r⟩. The argument is assigned the role with the highest cosine similarity, trained with the same ranking loss §2.5, L^d_2. Trigger and argument models are trained separately, as a pipeline §2.5. Negative training instances for both are sampled §2.5.

Zero-shot classification §2.6. At test time a new mention is encoded by the same parameters, all ontology types are encoded, and the type with the highest cosine score is chosen §2.6. Top-k lists are scored for the Hit@k evaluation §2.6 tab. 5. Each argument is then matched to the roles of its chosen type with the same ranking §2.6.

Pseudocode for the mention tuple and the training loss follows. Equations in the paper are not numbered; they are cited by section.

# Compose one AMR tuple of an event mention  [§2.3]
# Input:  w1, w2 (d,) word vectors; relation: AMR label
# Output: updated w1, w2 (d,) each

def compose_mention_tuple(w1, relation, w2):
    # Concatenate the two concept vectors  [§2.3]
    pair = concat(w1, w2)                           # (2d,)
    # Multiply by the relation's own matrix and apply f  [§2.3]
    # f is not named in the paper
    composed = f(pair @ M[relation])                # (2d,)
    # Split back into the two updated concept vectors  [§2.3]
    return composed[:d], composed[d:]

# Compose one tuple of an event type structure  [§2.3]
# Input:  type_vec, role_vec (d,) each; U: tensor (2d, 2d, 2d)
# Output: updated type and role vectors (d,) each

def compose_type_tuple(type_vec, role_vec, U):
    # Stack type and role into one vector  [§2.3]
    pair = concat(type_vec, role_vec)              # (2d,)
    # Bilinear tensor slices (inferred: one scalar per slice)  [§2.3]
    composed = f(pair @ U @ pair)                  # (2d,)
    return composed[:d], composed[d:]

# Hinge ranking loss with a margin  [§2.4, L1]
# Input:  mention_vec (2d,), type_vecs (|Y|, 2d), true_index, margin
# Output: scalar loss

def ranking_loss(mention_vec, type_vecs, true_index, margin):
    # Cosine score of the mention against every type  [§2.4]
    scores = cosine(mention_vec, type_vecs)        # (|Y|,)
    loss = 0
    for j in range(|Y|):
        if j == true_index:
            continue
        # Penalise each wrong type that comes within the margin  [§2.4]
        loss = loss + max(0, margin - scores[true_index] + scores[j])
    return loss

Key equations, cited by section (unnumbered in the paper):

• Mention tuple composition: V_u = [V'_w1; V'_w2] = f([V_w1; V_w2] · M_λ), with M_λ in R^(2d × 2d) §2.3.
• Type tuple composition: V_u' = [V'_y; V'_r] = f([V_y; V_r]^T · U^[1:2d] · [V_y; V_r]), with U in R^(2d × 2d × 2d) §2.3.
• Pairwise score: C(t, y) = cos([V_t; V_St], [V_y; V_Sy]) §2.4.
• Plain loss: L1(t, y) = sum over j ≠ y of max{0, m − C(t,y) + C(t,j)} §2.4.
• Modified loss: L^d_1(t, y) = max over j ≠ y of max{0, m − C(t,y) + C(t,j)} for non-Other y; for Other y, the same with reference y' drawn from seen types §2.4.
• Argument loss: L^d_2(a, r) takes the same form over roles R_y (or R_Y' when Other) §2.5.
• Inference: ŷ(t, 1) = argmax over y in Y of cos([V_t; V_St], [V_y; V_Sy]) §2.6.

Hyperparameters

Results

Author-reported. The experiments use the ACE05 corpus (33 types) for seen types and unseen-type tests, plus FrameNet frames to build a 1,194-type ontology §3.3. Hardware is not stated. The baseline in tab. 5 is WSD-Embedding, which grounds mentions to types by lexical word-sense similarity alone §3.2.

Trigger and argument classification with gold boundaries, Hit@k on the 23 unseen ACE types tab. 5 tab. 4:

Trigger and argument extraction, setting D, all 23 unseen ACE types tab. 7 (precision, recall, F1 in %; supervised baselines trained on all 33 types):

Learning curve: the zero-shot model, with no annotations for the 23 types, is described as comparable to an LSTM trained on about 500 annotated mentions of those types §3.3 fig. 4. The comparison is shown as a plot, not a table, so no exact gap is given.

Manual evaluation on 200 sampled mentions with non-ACE types §3.4 tab. 8, judged by one linguistic expert: trigger identification + classification 40.9% accuracy; argument identification + classification 17.41% tab. 8.

Impact of the parser on a subset of ERE with perfect AMR, 6 seen types, 200 test sentences §3.5 tab. 9:

Paper vs code

Code not determined. During compilation no official repository was verified. The paper has no code link in its source, and its only URL is a FrameNet link §3.4. A GitHub repository search over the API returned HTTP 403, and profile pages for the authors' labs timed out (HTTP 504). Web search was not available. Retry the code search before relying on this card; until a repository is verified, no code field is set.

Pitfalls

• Equations are unnumbered. The paper's equations are displayed without numbers, so this card cites them by section, not by [eq. N].
• Notation of the CNN input is inconsistent. The text says the filter is in R^(nd) (a scalar output for width 2), yet the bias is in R^d and there are 500 filters §2.4 tab. 2. Implementers must choose a consistent output shape.
• Unstated values. Margin m, optimiser, batch size, epochs, stopping rule, the function f, tensor initialisation, and whether the word-sense embeddings are fine-tuned are not stated §2.3 §2.4 tab. 2.
• The learning rate is only an initial value. 0.1 is the initial rate tab. 2; the optimiser and any decay are not stated, so the rate cannot be reproduced exactly from the paper alone.
• Tensor size. With d = 200 (from tab. 2), U has 400 × 400 × 400 entries, about 64 million values; one tensor is shared across all types §2.3. Derived from the stated sizes, not stated by the paper. Each relation matrix M_λ has 400 × 400 entries §2.3.
• Loss definitions differ. The plain loss L1 sums over negatives; the modified L^d_1 takes the maximum over negatives §2.4. The paper states that L^d_1 replaces L1 to avoid overfitting, so use L^d_1 unless testing the plain form.
• Reference type for Other mentions is ambiguous. The text says y' ranks highest among "all event types", but the sum runs over seen types only §2.4. The paper does not resolve this.
• Argument role loss is typed inconsistently. The case for r|y = Other is written with a different reference set (R_Y') from the main case §2.5; implement both cases from the text and check against the claims.
• Test boundaries. The classification experiments assume gold trigger and argument boundaries §3.2; the extraction experiments use the AMR-based candidate pipeline §2.2 §3.3. The two cannot be compared directly.
• Ontology size is stated two ways. Section 3.3 gives 1,191 types and section 3.4 gives 1,194 §3.3 §3.4. The construction (33 ACE + 1,161 FrameNet frames) gives 1,194 §3.3.
• The LSTM training size is stated two ways. Section 3.3 gives 3,464 sentences with 905 mentions for the 23 types §3.3. The learning-curve text and caption give about 3,000 sentences with 500 mentions §3.3 fig. 4. The abstract uses 500 [abstract].
• Small evaluation. Extraction results use 150 sentences with 129 mentions §3.3. The manual accuracy in tab. 8 comes from one annotator, with no agreement figure §3.4.
• Hit@k is not defined in the paper. The paper reports Hit@k but does not define it precisely. Confirm the definition before comparing numbers across papers.
• Parser dependence. System AMR lowers trigger F1 from 59.1 to 46.7 tab. 9. Costs and results depend on the AMR and WSD tools, not only on this method.
• Argument performance is weak. Transfer argument F1 is below the LSTM's and below the Joint model's tab. 7. The authors attribute errors to roles such as Entity and Organization that are not informative enough to match argument concepts §3.3.

Lineage

• Builds on: Socher et al.'s recursive tensor composition §2.3; Huang et al. (2016, Liberal IE) for the cluster-based negative samples and the WSD and AMR pipeline §2.2 §3.2. No arXiv ids are given in the paper for these.
• Compared against: LSTM [Feng et al. 2016], Joint structured perceptron [Li et al. 2013] §3.3; WSD-Embedding, an in-house lexical baseline §3.2.
• Superseded by: not stated in this paper; check the wiki's concept pages for later work.