The State and Fate of Linguistic Diversity and Inclusion in the NLP World

In plain words

Many language technologies work well only for a small set of widely spoken, well-documented languages, even though they are often described as language-agnostic. This paper asks who gets attention: which languages have labelled datasets and plenty of online text, which languages appear in the major NLP conferences, and whether that has changed over time.

The authors sort languages into six groups by how much labelled and unlabelled data they have. They count how often the languages of each group are mentioned in papers from the main NLP venues. They also train a small model that places conferences, authors and languages in one shared space. Languages with the least data sit near the bottom of the rankings, and some venues are more welcoming to them than others.

This is a measurement study, not a new model. It matters for anyone building multilingual tools, because the data gap affects how well those tools can serve each language.

TL;DR

• Analysis paper: no new model, no training objective for the main results.
• Languages fall into six resource classes, built from labelled (LDC, ELRA) and Wikipedia counts §2.2 §2.3.
• Class 0 has the largest inverse-MRR value (the least inclusive) in 10 of 11 venues tab. 4 §4.3.
• 549 of 1,139 WALS categories occur only in classes 0-2 §3.
• LREC and the workshops are the most inclusive venues by the authors' metrics §4.3.
• No code and no released dataset at compile time; see Paper vs code.

Problem

The authors pose five questions: how resources are distributed across languages and how that relates to speaker counts; which typological features current systems have been exposed to; how inclusive the ACL community has been over time; whether resource availability shapes research topics and venues; and what individual researchers and communities can do §1.

The starting premise is that neural methods need large data, and that zero-shot multilingual transfer still depends on large unlabelled corpora in many languages, so resource gaps may widen rather than close §1. Without a measurement, a claim that a system is language-agnostic cannot be checked.

Method

The paper has four parts. None of them is a learned model except the entity embedding in §5.

1. Language classes §2. Each language gets two counts. The labelled count comes from the LDC catalog and the ELRA map §2.2. The unlabelled count is the number of Wikipedia articles §2.2. A web-resource panel also appears in the paper, but the text does not say how its counts were obtained fig. 3. The two axes place each language into one of six classes §2.1:

• class 0 "Left-Behinds" (few or no resources);
• class 1 "Scraping-Bys" (some unlabelled data, almost no labelled);
• class 2 "Hopefuls" (a small set of labelled datasets);
• class 3 "Rising Stars" (strong web presence, little labelled data);
• class 4 "Underdogs" (large unlabelled and moderate labelled data);
• class 5 "Winners" (rich in both) §2.3.

The numeric cut-offs that assign a language to a class are not given in the text. Class sizes and speaker counts are in tab. 1.

2. Typology coverage §3. The authors use WALS §3. They take the categories of each WALS feature that occur only in classes 0-2 and not in classes 3-5. Each feature is counted as a set of categories, and the paper reports how many are "ignored" tab. 2 tab. 3.

3. Venue inclusion §4. Papers come from the ACL Anthology Reference Corpus, which ends in 2015, augmented with Semantic Scholar metadata and direct scraping of the Anthology §4.1. The text lists 11 venues, described as covering the period since 1990 §4.1. Two metrics measure how each venue mentions languages:

• Language occurrence entropy eq. 1. For each venue c and year y, a binary matrix marks which papers mention which languages. A language's share is the fraction of papers that mention it. Shares are normalised to a probability distribution and the Shannon entropy (natural log) is taken. Higher entropy means the mentions are spread more evenly across languages §4.2 fig. 4.
• Class-wise inverse MRR eq. 2 tab. 4. Languages are ranked by how often the venue mentions them. For each class, the mean reciprocal rank (MRR) of its languages is computed, and the table reports its inverse. Smaller values mean more inclusive.

4. Entity embeddings §5. A shared space holds four kinds of entity: authors, languages, conferences and conference-years §5.1. Each paper gives its entities an input vector. The model is trained to predict K words sampled from the paper's title and abstract, following the Skip-gram setup of Word2Vec §5.1 eq. 3 fig. 5. The learned vectors are projected to two dimensions with t-SNE §5.2 fig. 6. An author-language MRR then asks, for each language, how many of its K closest authors (by cosine distance) rank that language among their top 20 languages §5.2 tab. 5. A high value means a focused community, not a broad one §5.2.

Equations as printed in the compiled paper:

S^{c,y} = -\sum_{j=1}^{L} S'_j \log S'_j,\quad S'_j = \frac{S_j}{\sum_{j'} S_{j'}},\quad S_j = \frac{1}{P}\sum_{i=1}^{P} M_{ij} eq. 1

\mathrm{MRR} = \frac{1}{|Q|}\sum_{i=1}^{|Q|}\frac{1}{\mathrm{rank}_i} eq. 2

The entity objective, written as printed, is a sum of word probabilities over the papers and the sampled words eq. 3. The text does not state whether the probabilities use a softmax or negative sampling.

Pseudocode for the two metrics, in our own words:

# Language occurrence entropy for one venue and year  [eq. 1] [§4.2]
# Input:  mentions (P, L): 1 if paper i mentions language j, else 0
# Output: scalar entropy (natural log)

def occurrence_entropy(mentions):
    # Share of papers that mention each language  [eq. 1]
    share = sum(mentions, axis=0) / number_of_papers  # (L,)
    # Normalise the shares into a probability distribution  [eq. 1]
    prob = share / sum(share)  # (L,)
    # Shannon entropy; zero entries need a convention (not stated)  [eq. 1]
    return -sum(prob * log(prob))

# Class-wise inverse MRR for one venue  [eq. 2] [tab. 4]
# Input:  mention_counts: how often the venue mentions each language
#         class_languages: the languages of one resource class
# Output: 1 / MRR (smaller means more inclusive)

def class_inverse_mrr(mention_counts, class_languages):
    # Rank all languages by mention frequency  [§4.2]
    ranks = rank_by_frequency(mention_counts)
    # Reciprocal rank of each language of the class  [eq. 2]
    reciprocals = [1 / ranks[lang] for lang in class_languages]
    # Mean reciprocal rank over the |Q| languages  [eq. 2]
    mrr = sum(reciprocals) / len(class_languages)
    return 1 / mrr

The rank universe (all languages, or only the taxonomy languages) is not specified; see Pitfalls.

Hyperparameters

Results

Author-reported. These are the paper's own measurements on its own corpus. No external benchmark is involved.

Table 4 reports the inverse of Eq. 2's MRR, so its values are on the order of hundreds.

Paper vs code

No official code. The paper links a project page, https://microsoft.github.io/linguisticdiversity app. A.1 app. A.5. That page links no GitHub repository, gives no license, and marks the taxonomy file and the dataset as "coming soon" or "stay tuned". The paper also says the dataset is "in the process of" being released app. A.2. A GitHub search for "linguisticdiversity" (all repositories, and inside the microsoft organisation) returned no repository by the authors. A search for the title words returned none. The only hit for the first query is an unrelated Turkic-languages app.

The paper's own text and tables disagree in places; see Pitfalls.

Pitfalls

• Not reproducible from the paper alone. The main results depend on a language-mention detector, the class cut-offs and a dataset that is not released. An implementer cannot recover the tables from this card.
• Mention detection is unspecified. The compiled text never says how a paper is judged to mention a language, or which language list is used. The source contains commented-out text that mentions the WALS list and the title, abstract and body. That text is not in the compiled paper, so do not assume it applies §4.2.
• Class cut-offs are unspecified. The six classes are described qualitatively §2.3. The mapping from counts to classes is not stated.
• Language universe is unclear. Eq. 1 sums over L languages without defining the set. The class table covers about 2,485 languages when summed tab. 1, while WALS lists 2,679 §3 and the abstract refers to over 7,000 languages. The paper does not reconcile these numbers.
• Zero-count languages. Eq. 1 is undefined for zero shares. The 0 log 0 convention is not stated.
• Rank universe for Eq. 2. The text says rank is by mention frequency, without saying among which languages (all languages or only the class). Ties are not handled.
• Two MRR tables run in opposite directions. Table 4 is the inverse rank measure, where smaller means more inclusive. Table 5 is a normal MRR, where higher means a more focused community §5.2. Keep them apart.
• Text does not match Table 5. The text says MRR decreases from class 0 to class 5 except for class 2 §5.3. Table 5 also rises from class 2 to class 3 at MRR(5) and MRR(10), and from class 4 to class 5 at MRR(5), MRR(10) and MRR(15). Treat the trend as class 0 being highest, not as a monotone ordering.
• Speaker share in the text does not match Table 1. The text says class 0 holds 15% of all speakers §2.4. Table 1's rounded totals (1.0B out of about 7.5B) give about 13%.
• Venue count mismatch. The text lists 11 venues including SemEval §4.1. Table 6 has 10 rows and omits SemEval tab. 6.
• Symbol overload in §5. K is the number of sampled words in Eq. 3, and also the number of closest authors in §5.2 and in Table 5. The sum in Eq. 3 uses an index I that is not defined. The objective is written as a sum of probabilities rather than a log-likelihood, so do not treat it as a standard Word2Vec loss without checking.
• Missing training details. Learning rate, epochs, context window, number of negatives and the optimiser are not stated. The paper says the model uses Word2Vec's hyperparameters, but gives no values app. A.3.
• Only one validation check. The embedding is validated only by predicting publication year, with R² 0.6 and MAE 4.04 app. A.3. No check of the language-class mapping is given.
• Web counts source. Fig. 3's web panel is described, but its data source is not stated in the text fig. 3.
• Erratum not compiled. The source contains an erratum section with a corrected Table 1. Its values match the table in the main text, and the file is not included by the compiled main file. The embedding section used is v3; v1 and v2 files are present in the source but not compiled.

Lineage

• Builds on: the ACL Anthology Reference Corpus §4.1. The entity embedding follows the Skip-gram setup of Word2Vec, cited but not carded here §5.1.
• Compared against: not applicable; the paper compares venues and classes against each other, not against other methods.
• Superseded by: not known from wiki content.