Word Similarity Datasets for Indian Languages: Annotation and Baseline Systems

In plain words

Word embeddings turn each word into a list of numbers, and words with similar meanings should get similar lists. One common way to check this is to take pairs of words, ask people how similar they are, and see whether the embedding's similarity scores rank the pairs in the same order.

English has several such pair lists, but most Indian languages have none. This paper builds small lists of word pairs for six Indian languages. It starts from two English lists, has native speakers translate the words, and then has other native speakers rate how similar each pair is.

The authors also train three standard embedding methods on a little text in three of these languages and score them on the new lists. The results show that the FastText variant that builds vectors for unseen words beats plain FastText in all three, and is the top row in two of them. The lists are small and only three languages have baseline scores, so the work is best read as a starting point for resources.

TL;DR

• Six new monolingual word-similarity test sets (Urdu, Telugu, Marathi, Punjabi, Tamil, Gujarati) of 97 to 163 word pairs each §2.
• Built by translating RG-65 and WordSim-353 with three annotators, then rating each pair on a 0 to 10 scale with eight native speakers §4.1 §4.2.
• Baselines for Urdu, Telugu and Marathi: CBOW, Skip-gram and FastText, plus a FastText variant that handles unseen words. Author-reported Spearman rho x 100 tab. 2 tab. 3 tab. 4.
• Punjabi, Tamil and Gujarati have no baselines; the paper defers them to future work §7.

Problem

Word-similarity benchmarks are a cheap way to score word vectors, but they exist mainly for English. The authors say that for languages such as Marathi, Punjabi and Telugu, neither a test set nor a reliable corpus was available §1. The paper also notes that its test pairs contain rare words: the baseline models met unseen pairs at test time §6.

Method

The paper has two parts: dataset construction and a baseline evaluation.

Dataset construction §4:

1. Base lists. Both English RG-65 and WordSim-353 are used as the source for all six languages §4.1. The paper does not state how word pairs were selected from these base lists, and the new lists (97 to 163 pairs) are not the same size as either source §2.
2. Translation. Three native speakers, each fluent in English, translate the word pairs. Two translate first; the third breaks ties on disagreements. The final list must not repeat word pairs §4.1. The paper follows the translation procedure of Camacho-Collados et al. §4.1.
3. Scoring. Eight native speakers rate each pair on a 0 to 10 scale. A score of 10 means the words mean the same thing and 0 means they mean completely different things. The rating guidelines follow the SemEval-2014 Task 3 guidelines, which separate similarity from relatedness §4.2. Each pair's score is the mean of the eight ratings §4.2.
4. Agreement check. Inter-annotator agreement is reported as Fleiss' kappa per language §5.1 tab. 1.

Baseline evaluation §2 §6:

1. Train word vectors with CBOW, Skip-gram (word2vec) and FastText on a monolingual corpus of the target language, using 300 dimensions, a minimum count of 5 and negative sampling §2.
2. For each test pair, take the cosine similarity of the two word vectors. A word that is not in the vocabulary gets a zero vector §2.
3. Compute Spearman's rho between the human scores and the cosine scores, multiplied by 100 §2.
4. Report a "FastText w/ OOV" row alongside FastText. The paper says this variant generates vectors for missing words with a character model §6.

# Word-similarity evaluation  [§2]
# Input:  pairs: list of (word_a, word_b, human_score), score in 0-10
#         vectors: dict word -> vector of 300 floats
# Output: Spearman rho x 100

def evaluate(pairs, vectors):
    # Score each pair with the cosine of its two word vectors  [§2]
    human_scores = []
    model_scores = []
    for word_a, word_b, human_score in pairs:
        # A word not found gets a zero vector  [§2]
        vector_a = vectors.get(word_a, zeros(300))
        vector_b = vectors.get(word_b, zeros(300))
        # Cosine with a zero vector is undefined; not stated  [§2]
        model_scores.append(cosine(vector_a, vector_b))
        human_scores.append(human_score)
    # Rank correlation between human and model scores  [§2]
    rho = spearman(human_scores, model_scores)
    return 100 * rho

The paper's own equations: none. The Spearman formula and the Fleiss' kappa formula are not written out §5.1.1.

Hyperparameters

Results

Author-reported. Setup: six new test sets; three with baselines (Urdu, Marathi, Telugu). Models trained on the corpora in the table above §2. Score is Spearman's rho multiplied by 100 §2. The paper does not define the "OOV" column, so it is copied as printed tab. 2 tab. 3 tab. 4. The "Vocab" column is also not defined; the paper prints "-" for the FastText w/ OOV row.

Inter-annotator agreement (Fleiss' kappa) tab. 1: Urdu 0.887, Punjabi 0.821, Marathi 0.808, Tamil 0.756, Telugu 0.866, Gujarati 0.867. The paper describes Tamil's agreement as relatively weaker §4.2.

Analysis, as stated by the authors: the FastText w/ OOV row beats FastText in all three languages §6. The paper attributes this to character-based models handling unseen words §6.

Paper vs code

No official code. The paper has no code link, abstract link or project page, and the arXiv metadata has no comment field [raw metadata]. A GitHub repository search for "indian word similarity datasets" and "word similarity urdu telugu marathi" was run. It returned syedsarfarazakhtar/Word-Similarity-Datasets-for-Indian-Languages. It was not used as the official code, for these reasons:

• Its owner name matches the first author's name, but the paper does not link it.
• It has no LICENSE file, so no code_license can be recorded.
• Its About text cites an ACL Anthology 2017 workshop paper and not this arXiv id.
• It holds one plain-text word list per language (including Hindi, which the paper does not cover). It contains no training or evaluation code.

Pitfalls

• Base lists and pair selection. The paper names RG-65 and WordSim-353 as the bases §4.1 but does not say how 97 to 163 pairs were chosen from them. An implementer cannot reproduce the sampling from the text.
• Rating scale versus kappa. Each pair is scored on a 0 to 10 scale and averaged over eight raters §4.2, yet agreement is reported as Fleiss' kappa §5.1.1 tab. 1. The paper does not say how the scale was turned into categories for kappa.
• Zero vectors. Missing words get a zero vector §2. Cosine with a zero vector is undefined, and the paper does not say what score it assigns. The "FastText w/ OOV" row generates vectors for unseen words §6, so it is unclear whether the zero-vector rule applies to that row.
• OOV and Vocab columns. Both are unlabelled in the tables tab. 2 tab. 3 tab. 4. The "OOV" count is not defined, so it cannot be checked against the test pairs.
• FastText w/ OOV is not always best. The text says FastText w/ OOV beats FastText in every experiment §6, which holds for the three tables. Marathi Skip-gram (41.22) is higher than FastText w/ OOV (38.66) tab. 3, so the claim is not "best overall" for Marathi.
• Baselines for three of six languages only. Punjabi, Tamil and Gujarati have test sets but no baseline models §7.
• Corpus citation. The Urdu corpus is cited as the "untagged corpus" of Jawaid et al. §2, while the reference list gives the title "A Tagged Corpus and a Tagger for Urdu" with year 2014 [bib: BJ:15]. The Marathi IIT-Bombay corpus has no citation at all §2.
• Reference years. Several bibliography entries have years that do not match their keys or titles, for example word2vec (key 2013, year 2006 in the entry) [bib: MSG:13a] and RG-65 (2006 in the entry). Cite by arXiv id or the original venue, not by the entry year.
• Tokens, not sentences. Corpus size is given as tokens only §2. Whether the Telugu dump was cleaned is not stated; the paper's earlier cleaning step is commented out §2.
• Test size is small. At 97 to 163 pairs, a few pairs change Spearman's rho noticeably. The paper gives no confidence intervals §6.
• Section naming. Section 2 is titled "Datasets" but contains the training corpora and model settings, not the new test sets (those are in §4). Cite §2 for model settings and §4 for construction.

Lineage

• Builds on: FastText, Bojanowski et al., arXiv:1607.04606 §2 (no card yet); word2vec CBOW and Skip-gram, Mikolov et al., arXiv:1301.3781 §2 (no card yet).
• Base datasets: RG-65 and WordSim-353 §4.1 (no cards; not arXiv papers).
• Translation and annotation procedure follows Camacho-Collados et al. §4.1.
• Compared against: the CBOW, Skip-gram and FastText baselines in this paper §6.
• Superseded by: not known.