A Report on the Complex Word Identification Shared Task 2018

In plain words

Before a text can be simplified, something has to decide which words would be hard for the person reading it. This report covers a shared competition on that question: the organisers gave research teams annotated sentences, where people had marked the words they found difficult, and asked the teams to predict which target words are difficult for a reader.

The 2018 edition covered more languages (English, German and Spanish, plus a French test set with no French training data) and asked for a probability of difficulty as well as a yes or no answer. The traditional feature-based approaches, mostly built on length and frequency features, generally did better than neural and embedding-based ones. The report gives the datasets, the scores of the submitted systems, and a simple baseline the teams were compared against.

TL;DR

• Shared task on complex word identification (CWI), BEA workshop at NAACL-HLT 2018 §1.
• Four tracks (English, German, Spanish monolingual; multilingual with a French test set), each in a binary and a probabilistic task §1.1.
• 12 teams submitted; 11 wrote system papers that the report summarises §5.
• Organisers' baseline: nearest-centroid / linear regression on frequency and length features §4.1.
• Feature-engineering approaches beat neural and embedding-based ones overall; the best team (TMU) won 5 of 12 tracks §5 §4.2.

Problem

Lexical simplification pipelines usually begin by finding the words that a target population finds complex §1. Until the 2016 task there was no manually annotated and verified CWI dataset; that task's systems performed poorly, and the authors cite this as one reason for a second edition §1 §2. The report also notes that native and non-native annotators agree with each other to different degrees, which suggests the two groups may have different CWI needs §3.2. What counts as complex therefore depends on the reader group the data was collected for §3.1.

Method

This is a shared-task report, not a method paper. It defines the task, the data and the evaluation, then summarises each team's system as the team described it. Nothing here is a single algorithm of the authors'.

Task and tracks

• Binary task: label each target word or phrase in context as complex (1) or simple (0) §1.1.
• Probabilistic task: give the probability that each target is complex §1.1.
• Tracks: one monolingual track each for English, German and Spanish, and a multilingual track whose test set is French. French has no training data §1.1 §3.
• Participants chose their own track/task combinations §1.1.

The annotation target is a word or multi-word expression of up to 50 characters §3.1.

Datasets

• The English, German and Spanish data are the CWIG3G2 datasets from earlier work that the report cites for full details §3. The English data covers three genres: professionally written news, WikiNews, and Wikipedia §3.
• The French dataset was built for this edition from Wikipedia paragraphs taken from a comparable simplified corpus §3. For each article, paragraphs with 5 to 10 sentences were kept, and the best one was chosen by a ranking procedure. The ranking favoured longer sentences and lexical richness, and penalised named entities and foreign words §3. A subset of 100 paragraphs was then chosen by a greedy search that minimised pairwise vocabulary overlap (Jaccard) §3. A random test split of 24 paragraphs was annotated §3.

Instance counts (train / dev / test) tab. 3:

Complex-instance counts and shares tab. 4:

Single-word and multi-word complex annotations, by language tab. 5:

Annotators per language (native / non-native / total) tab. 2: English 134 / 49 / 183; German 12 / 11 / 23; Spanish 48 / 6 / 54; French 10 / 12 / 22.

Annotation process

• Crowdsourced on Amazon Mechanical Turk. Each HIT showed 5 to 10 sentences together, and annotators highlighted words or phrases that could be hard to understand §3.1.
• The instructions asked annotators to assume a given target reader, and the report names children, language learners and people with reading impairments as examples §3.1.
• Complex words were not pre-highlighted §3.1.
• A bonus was paid when an annotator's selection matched at least half of the other annotations, to encourage care §3.1.
• At most 10 annotations per HIT §3.1.
• English HITs had more than 20 annotators, at least 10 native and 10 non-native, so that the two groups can be compared §3.1.

Systems

The report summarises each team's system; the details below are as the report states them. Numbers in brackets are the team's own choices.

• UnibucKernel (monolingual English): character and lexical features, embedding-based features (cosine similarities to other sentence words, WordNet sense embeddings, and PCA from 300 to 2 dimensions), a kernel matrix, and an SVM (LibSVM for classification, v-SVR for regression). Parameters were tuned on the dev set §4.2.
• SB@GU (multilingual): features in five groups (surface, morphological, semantic, context, psycholinguistic). Extra Trees with 1000 and 500 estimators was their best classifier. For the English datasets, feature selection used scikit-learn's SelectFromModel. For French, two setups were used: majority vote of the English, German and Spanish classifiers, or the Spanish classifier alone §4.2.
• hu-berlin (multilingual): character n-grams with multinomial Naive Bayes. They found 2-gram and 4-gram features best for binary; 3- to 5-grams were used for English §4.2.
• NILC (monolingual English): three approaches. Feature-engineering models (38 features; the best was XGBoost for binary); average GloVe embedding fed to a network with two ReLU layers and a sigmoid output; and an LSTM over the context. A voting system combined the three §4.2.
• TMU (multilingual and cross-lingual): random forest classifiers (binary) and random forest regressors (probabilistic), with frequency features from a learner corpus and from Wikipedia/WikiNews §4.2.
• ITEC (English and Spanish): word2vec embeddings (300 dimensions, window 5, minimum count 20), morphological and psycholinguistic features, and a Keras / TensorFlow network §4.2.
• CoastalCPH (German, Spanish, French): random forests with 100 trees plus a feed-forward network; log-probability, inflection, surface, part-of-speech and embedding-similarity features §4.2.
• LaSTUS/TALN (monolingual English binary): word2vec features over the context, target and context (900 dimensions) plus cosine distances, lexical and semantic features; Weka SVM and random forest §4.2.
• CFILT_IITB (monolingual English binary): eight classifiers combined by hard voting, with a word-embedding classifier as tie-breaker §4.2.
• NLP-CIC (English and Spanish binary): tree ensembles and a 2D CNN over word embeddings §4.2.
• Camb (monolingual English, both tasks): feature union of lexical, dependency, psycholinguistic and frequency features; AdaBoost with 5000 estimators for binary; linear regression for probabilistic §4.2.

Other teams (SB@GU, hu-berlin and others) are summarised in §4.2; the report says these were the teams that wrote system papers §4.2 §5.

Evaluation

Binary track: accuracy and macro-averaged F1 are named as the metrics in the baseline section §4.1. Probabilistic track: mean absolute error §4.1.

# Binary track score  [§4.1]
# Input:  gold labels, predicted labels (1 = complex, 0 = simple)
# Output: accuracy and macro-averaged F1

def score_binary(gold, predicted):
    # Share of targets whose predicted label equals the gold label  [§4.1]
    accuracy = mean(gold == predicted)
    # Macro-averaged F1 over the two classes  [§4.1]
    f1 = macro_f1(gold, predicted)
    return accuracy, f1

# Probabilistic track score  [§4.1]
# Input:  gold probabilities, predicted probabilities, both in [0, 1]
# Output: mean absolute error (lower is better)

def score_probabilistic(gold, predicted):
    # Mean absolute error between gold and predicted probability  [§4.1]
    return mean(abs(gold - predicted))

Baseline

# Organisers' baseline  [§4.1]
# Input:  target words in context
# Output: label (binary) or probability (probabilistic)

def baseline(target):
    # Only frequency and length features are used  [§4.1]
    features = [frequency(target), length(target)]
    # Binary: Nearest Centroid; probabilistic: Linear Regression  [§4.1]
    return model_for_task(features)

The report does not say which frequency source the baseline uses. It refers to the earlier work for the feature definitions §4.1 §3.

Hyperparameters

Results

Author-reported. Setup: the organisers' test sets above; each team's own system; the baseline described in §4.1. All numbers are as printed in the report's tables. The tables below show the top three teams and the baseline per track. The full ranking is in the source tables tab. 6 to tab. 9.

Binary classification, F-1 (higher is better) tab. 6 tab. 7:

Probabilistic classification, mean absolute error (lower is better) tab. 8 tab. 9:

Headline findings, as the report states them: the traditional feature-engineering approaches, mostly built on length and frequency, performed better than neural and word-embedding approaches overall. More systems used deep learning than in 2016, and results improved, though the gap is less pronounced for the probabilistic task §5. The French cross-lingual results were described as promising, with scores superior or equivalent to the German and Spanish ones when trained on other languages' data; the report notes these cannot be compared directly §5. TMU won 5 of 12 tracks §4.2.

Paper vs code

No official code. The paper has no code or project link (its only URL is a word2vec footnote §4.2), and arXiv metadata has none. A GitHub repository search for "complex word identification" 2018 found only repositories whose owners are not authors, so none is used. Systems' own code is not part of this report.

Pitfalls

• Not a method. There is no single algorithm to implement. Each team system is described only briefly; for implementation details, the team's own system paper (cited in the report) is needed, and it is not in this wiki.
• Metric ambiguity. The text names accuracy and macro-F1 for the binary task §4.1, but the result tables show only "F-1" and do not say which metric produced the ranks, or whether "F-1" is macro-averaged tab. 6 tab. 7. The pseudocode above follows the text, not the tables.
• Ranks look inconsistent with scores. In the German binary table, the baseline F-1 (0.7546) is above the rank-1 system (TMU, 0.7451) tab. 7. In French binary, the baseline (0.6344) beats rank 3 (SB@GU, 0.6266) tab. 7. Ties also appear (e.g. Camb 0.8378 at ranks 2 and 3 in English WikiNews tab. 6). The report does not explain how ranks are assigned.
• Rows are submissions, not teams. Team names repeat (Camb, Gillin Inc., LaSTUS/TALN, CoastalCPH) and one team may appear several times with different runs. The tables do not say which run a row is. The text says hu-berlin submitted two English variants and four French runs §4.2.
• Gold values not explained. The report does not say how the probabilistic gold values are computed from the annotations §3 §4.1. Nor does it say how binary gold labels are derived beyond the task definition §1.1.
• Cross-lingual French numbers. French has no training data; the French test set has 657 complex instances out of 2,251 (29%), lower than the 39–42% in other sets tab. 3 tab. 4. The authors offer possible reasons (the 10-annotation cap per HIT, fewer annotators) but present them as plausible explanations, not established causes §3.2. French results are not directly comparable with the other tracks, as the report itself says §5.
• Annotation cap. A 10-annotation cap per HIT affects multi-word counts (the 3-gram+ share is highest for French, 19.03% tab. 5). The report says annotators sometimes merged adjacent single complex words into one sequence to avoid the cap §3.2.
• Agreement figures are the authors' own analysis. English: about 90% of complex phrases were selected by at least two annotators; agreement between native and non-native annotators was about 70% §3.2. German non-native agreement was 70.66% against 58.5% for native annotators §3.2.
• No variance or significance. The report gives single scores per run, with no confidence intervals or significance tests.
• Table numbers. Table numbers here are counted in source order from the LaTeX (tab. 1 to tab. 9). The compiled PDF was not checked against this numbering.

Lineage

• Builds on: the first CWI shared task at SemEval 2016 (Task 11), which the report compares against. The approaches of the 2016 systems are summarised in tab. 1 §2. An evaluation the report cites found that an ensemble of the 2016 systems degraded as more systems were added §1.
• Datasets: the English, German and Spanish data come from the authors' earlier CWIG3G2 work, cited in §3; that work is not compiled in this wiki, so nothing from it is used here.
• Compared against: the organisers' nearest-centroid and linear-regression baseline §4.1.
• Superseded by: not stated in this report.