Paper · 2021-09-10 · cs.CL
IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization
Trains IndoBERTweet, a BERT model for Indonesian tweets, from scratch for 1M steps. It also tests a cheaper route: adapt IndoBERT to 26M tweets with a new vocabulary whose new tokens start as averaged IndoBERT subword embeddings, then train 200K steps. That route scores 86.5 average over seven Indonesian Twitter tasks, vs 86.1 from scratch. Author-reported.
Language model pretrainingDomain adaptationSubword tokenizationMasked language modelingindonesiantwitterbertpretrainingvocabulary-initializationnamed-entity-recognitionsentiment-analysishate-speech-detectionemotion-classification
arXivPDFCodeMarkdown15 min read
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Language models that read Indonesian are often trained on news and web text. Tweets look different: they are short, informal, full of slang and mentions, and they use words the model has never seen. The authors build a model for Indonesian Twitter. They start from an existing Indonesian BERT and extend its word list with words that are common in tweets but missing from the original.
The hard part is what to put in the new word slots. Starting them from random numbers works, but the model then has to learn them from scratch. The authors find that a simple rule works better than the alternatives: for each new word, take the old model's vectors for the smaller pieces the old tokenizer would split it into, and average them. The adapted model then needs far less extra training than a model built from zero.
The adapted model does better on average than the existing Indonesian models on seven tweet tasks, including sentiment, emotion, hate speech and named entity recognition. Results are reported by the authors. The paper matters to anyone who works on Indonesian social-media text, and to anyone who wants to adapt a pretrained model to a new domain without paying for full pretraining.
TL;DR
IndoBERTweet is a BERT-base model trained from scratch for 1M steps on 26M Indonesian tweets, with a 31,984-type WordPiece vocabulary §2.1 §2.2 §3. The paper also tests a cheaper route: extend IndoBERT's vocabulary and continue pretraining for 200K steps §2.3 §4. Of four ways to initialise the new entries, averaging the IndoBERT subword embeddings scores best on average §4 tab. 2. Results are author-reported, on seven Indonesian Twitter datasets tab. 1 tab. 2. Several claims are only averages, and no significance tests are reported §4. Read Pitfalls before reusing numbers.
Problem
Domain-general BERT models tokenise domain words poorly, and domain pretraining from scratch is expensive §1. The paper argues that domain-adaptive pretraining is cheaper but tends to be limited by the original vocabulary, which splits many domain words into many pieces §1. For Indonesian Twitter, the authors also note that Indonesian is underrepresented in NLP and that most earlier Indonesian Twitter work used traditional models §1. The question is how to add domain words to an existing vocabulary so that the adapted model learns quickly and performs well §1 §2.3.
Method
Twitter corpus §2.1. Indonesian tweets were crawled from December 2019
to December 2020 with the Twitter API, using 60 keywords in four topics:
economy, health, education and government. Non-Indonesian tweets were filtered
with Twitter's language identifier. In a sample of 100 tweets, 87 were
Indonesian, 12 Malay and 1 Swahili §2.1. After removing duplicate IDs, the
corpus has 26M tweets and 409M word tokens. 230K tweets are held out for
development §2.1. The vocabulary is 31,984 WordPiece types, and all text is
lower-cased §2.1. Preprocessing follows English BERTweet: user mentions
become @USER, URLs become HTTPURL, and emoticons are translated to text with
the emoji package §2.1.
Model §2.2. IndoBERTweet is trained with a masked language modelling
objective, following the same procedure as indobert-base-uncased §2.2
tab. 1. The architecture is BERT-base-sized: 12 layers, hidden size 768,
12 attention heads, and feed-forward size 3072 §2.2. The only architectural
change is a maximum sequence length of 128 tokens, chosen from the average
number of words per tweet §2.2. The paper trains five models: one from scratch
for 1M steps, and four adapted from IndoBERT with different vocabulary
initialisations §2.2 §3.
Datasets tab. 1. Seven Indonesian Twitter datasets in four tasks §3. Table 1 is reproduced under Hyperparameters below.
Vocabulary overlap §2.3. IndoBERT has 31,923 types and the IndoBERTweet
vocabulary has 31,984. Of the IndoBERTweet types, 14,584 (46%) are new and
17,400 (54%) are shared with IndoBERT §2.3. A footnote says that the
implementation trimmed the adaptive vocabulary to the same size as IndoBERT by
discarding some [unused-x] tokens §2.3.
Four initialisations for new types §2.3. Shared types keep their IndoBERT embeddings. The four strategies for new types are:
- Method 1: random values from the uniform distribution U(-1, 1) §2.3.
- Method 2: random values from a normal distribution whose mean and standard deviation are learned from IndoBERT embeddings §2.3. The paper does not say whether these statistics are global or per dimension.
- Method 3: a linear projection from 300-dimensional fastText vectors trained on the tokenised tweet corpus §2.3. The projection matrix W is fitted on the shared types only, by minimising the squared distance between fastText vectors times W and the IndoBERT vectors §2.3 (unnumbered equation). Method 3 follows Poerner et al. (2020), as the paper states §2.3.
- Method 4: the average of the IndoBERT embeddings of the WordPiece pieces that IndoBERT's tokenizer produces for the new word §2.3 (unnumbered equation).
# Build the new embedding table with subword averaging (Method 4) [§2.3]
# Input: ibt_vocab (list of new and shared types),
# ib_embeddings (V_IB, 768), ib_vocab (type -> row index),
# tokenize (IndoBERT WordPiece tokenizer, word -> list of row ids)
# Output: embeddings (V_IBT, 768)
def init_average(ibt_vocab, ib_embeddings, ib_vocab, tokenize):
embeddings = []
for token in ibt_vocab:
if token in ib_vocab:
# Shared types keep their IndoBERT row [§2.3]
embeddings.append(ib_embeddings[ib_vocab[token]])
else:
# New types: mean of the IndoBERT rows of their pieces [§2.3]
pieces = tokenize(token) # list of row ids
vector = mean(ib_embeddings[piece] for piece in pieces)
embeddings.append(vector)
return stack(embeddings) # (V_IBT, 768)# Fit and apply the fastText projection (Method 3) [§2.3]
# Input: shared_types (list), new_types (list),
# fasttext (word -> 300-d vector), ib_embeddings (word -> 768-d)
# Output: init vectors for new_types, (len(new_types), 768)
def init_projection(shared_types, new_types, fasttext, ib_emb):
# Stack fastText and IndoBERT vectors for shared types [§2.3]
sources = stack(fasttext[x] for x in shared_types) # (S, 300)
targets = stack(ib_emb[x] for x in shared_types) # (S, 768)
# Least-squares fit of W [§2.3] (unnumbered equation)
W = least_squares(sources, targets) # (300, 768)
# Map each new type's fastText vector through W [§2.3]
return stack(fasttext[x] @ W for x in new_types) # (N, 768)Pretraining §3. The from-scratch model is trained for 1M steps with a learning rate of 1e-4, the Adam optimiser and a linear schedule §3. Gradients are accumulated over 4 steps to give a batch size of 2048 §3. Experiments run on 4 V100 GPUs with 32GB each §3. Domain-adaptive pretraining runs for 200K steps and is done in three forms: without vocabulary change, with the new vocabulary and no further training (0 steps), and with the new vocabulary plus 200K steps §3 §4. The paper does not state the learning rate, batch size or warmup for the adaptation runs §3.
Fine-tuning §3. For sentiment, emotion and hate speech, an MLP layer reads the average-pooled IndoBERTweet output. For NER, the first subword of each word predicts the tag. Fine-tuning uses batch size 30, maximum length 128, learning rate 5e-5, Adam with epsilon 1e-8, and early stopping with patience 5 §3. IndoLEM, HS1 and HS2 use 5-fold cross-validation; the paper describes a canonical split for the hate speech tasks §3 tab. 1. The other datasets use their original held-out splits §3 tab. 1.
# Classification head for sentiment, emotion and hate speech [§3]
# Input: token_ids (B, T), token_types (B, T), mask (B, T), mask in {0, 1}
# Output: scores for each tweet, (B,) or (B, classes)
def classify(token_ids, token_types, mask):
# Final-layer token vectors from IndoBERTweet [§3]
hidden = bert(token_ids, token_types, mask) # (B, T, 768)
# Dropout 0.2 on token vectors
# [code: sentiment_indolem/indobertweet.py#L104]
hidden = dropout(hidden) # (B, T, 768)
# Average over unmasked positions, [CLS] and [SEP] included [§3]
# [code: sentiment_indolem/indobertweet.py#L110-L112]
pooled = sum(hidden * mask) / sum(mask) # (B, 768)
# Linear layer, then sigmoid; the paper calls this an MLP [§3]
# [code: sentiment_indolem/indobertweet.py#L103, #L113-L114]
return sigmoid(linear(pooled)) # (B,) binary tasks onlyNER §3. Each word is split into WordPieces, and the first piece carries the tag. The paper reports entity-level F1 §3 tab. 1. The train and dev partitions mix formal and informal tweets and are shared by the two test sets (formal and informal) §3.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Tweet corpus | 26M tweets, 409M word tokens | §2.1 | 230K tweets held out for dev |
| Vocabulary size | 31,984 types (IndoBERT: 31,923) | §2.1 §2.3 | footnote: implementation trimmed to IndoBERT's size |
| New / shared types | 14,584 (46%) / 17,400 (54%) | §2.3 | |
| Casing and preprocessing | lower-cased; mentions to @USER, URLs to HTTPURL, emoji to text | §2.1 | |
| Architecture | 12 layers, hidden 768, 12 heads, feed-forward 3072 | §2.2 | "3 feed-forward hidden layers" is worded oddly; read as BERT-base |
| Max sequence length | 128 | §2.2 §3 | |
| Pretraining steps, from scratch | 1M | §3 | |
| Adaptation steps | 200K | §3 | |
| Pretraining learning rate, from scratch | 1e-4 | §3 | Adam, linear schedule; warmup not stated |
| Effective batch size | 2048 | §3 | 4 accumulation steps; micro-batch not stated |
| Adaptation learning rate, batch size, warmup | not stated | §3 | |
| Masking rate, mask/random/keep split | not stated | §2.2 | "same procedure as IndoBERT" |
| fastText dimension | 300 | §2.3 | window, epochs, minimum count not stated |
| Projection solver | not stated | §2.3 | fit described as an argmin |
| Init range, Method 1 | U(-1, 1) | §2.3 | |
| Init statistics, Method 2 | mean and std learned from IndoBERT embeddings | §2.3 | global or per dimension not stated |
| Fine-tuning batch size | 30 | §3 | |
| Fine-tuning learning rate | 5e-5 | §3 | |
| Adam epsilon | 1e-8 | §3 | |
| Early-stopping patience | 5 | §3 | |
| Fine-tuning epochs | not stated | §3 | code default 20; see Paper vs code |
| Fine-tuning warmup, weight decay, grad clip | not stated | §3 | see Paper vs code |
| Seeds | not stated | §3 | see Paper vs code |
| Cross-validation | 5-fold for IndoLEM, HS1, HS2 | §3 tab. 1 |
Datasets tab. 1
| Task | Dataset | Label count | Train size | Dev size | Test size | Cross-validation | Metric |
|---|---|---|---|---|---|---|---|
| Sentiment analysis | IndoLEM | 2 | 3,638 | 399 | 1,011 | Yes | F1_pos |
| Sentiment analysis | SmSA | 3 | 11,000 | 1,260 | 500 | No | F1_macro |
| Emotion classification | EmoT | 5 | 3,521 | 440 | 442 | No | F1_macro |
| Hate speech detection | HS1 | 2 | 499 | 72 | 142 | Yes | F1_pos |
| Hate speech detection | HS2 | 2 | 9,219 | 2,633 | 1,317 | Yes | F1_pos |
| Named entity recognition | Formal | 3 | 6,500 | 657 | 1,122 | No | F1_entity |
| Named entity recognition | Informal | 3 | 6,500 | 657 | 1,227 | No | F1_entity |
Results
Author-reported. Setup: seven Indonesian Twitter datasets tab. 1; the baselines are mBERT, malayBERT and two IndoBERT models §3. Scores are in percent; the metric for each dataset is in tab. 1. The "Average" column is the unweighted mean of the seven scores. Bold follows the paper's marks, which are set within each block of rows tab. 2.
Table 2 tab. 2 (reproduced in full).
| Model | IndoLEM | SmSA | EmoT | HS1 | HS2 | Formal | Informal | Average |
|---|---|---|---|---|---|---|---|---|
| mBERT | 76.6 | 84.7 | 67.5 | 85.1 | 75.1 | 85.2 | 83.2 | 79.6 |
| malayBERT | 82.0 | 84.1 | 74.2 | 85.0 | 81.9 | 81.9 | 81.3 | 81.5 |
| IndoBERT (Wilie et al., 2020) | 84.1 | 88.7 | 73.3 | 86.8 | 80.4 | 86.3 | 84.3 | 83.4 |
| IndoBERT (Koto et al., 2020) | 84.1 | 87.9 | 71.0 | 86.4 | 79.3 | 88.0 | 86.9 | 83.4 |
| IndoBERTweet (1M steps) | 86.2 | 90.4 | 76.0 | 88.8 | 87.5 | 88.1 | 85.4 | 86.1 |
| IndoBERT + 200K steps of domain-adaptive pretraining | ||||||||
| Same vocabulary (V_IBT = V_IB) | 86.4 | 92.7 | 76.8 | 88.7 | 82.2 | 87.9 | 86.9 | 85.9 |
| IndoBERT + vocabulary adaptation + 0 steps | ||||||||
| Uniform distribution | 82.9 | 84.6 | 73.2 | 84.9 | 78.2 | 84.3 | 84.4 | 81.8 |
| Normal distribution | 83.5 | 86.7 | 71.1 | 85.2 | 77.4 | 85.0 | 86.3 | 82.2 |
| fastText projection | 84.4 | 83.6 | 72.2 | 85.5 | 80.9 | 85.4 | 85.6 | 82.5 |
| Average of subwords | 84.2 | 88.1 | 71.6 | 86.2 | 78.3 | 86.4 | 87.4 | 83.2 |
| IndoBERT + vocabulary adaptation + 200K steps | ||||||||
| Uniform distribution | 85.6 | 90.9 | 75.7 | 88.4 | 83.0 | 87.7 | 85.9 | 85.3 |
| Normal distribution | 87.1 | 92.5 | 75.4 | 88.8 | 82.5 | 88.7 | 86.6 | 85.9 |
| fastText projection | 86.4 | 89.7 | 78.5 | 88.7 | 84.4 | 88.0 | 86.6 | 86.0 |
| Average of subwords | 86.6 | 92.7 | 79.0 | 88.4 | 84.0 | 87.7 | 86.9 | 86.5 |
Table 1 tab. 1 is reproduced under Hyperparameters above.
Headline results, as stated by the paper §4.
- IndoBERTweet trained from scratch is about 3 points higher on average than the IndoBERT baselines (86.1 vs 83.4) §4 tab. 2.
- IndoBERT trained 200K more steps with the same vocabulary averages 85.9 §4 tab. 2.
- Adding the new vocabulary with no training (0 steps) lowers the average, and subword averaging has the smallest drop, 0.2 points below the original IndoBERT (83.2 vs 83.4) §4 tab. 2.
- With 200K adaptation steps after vocabulary adaptation, subword averaging gives the best average, 86.5. The paper says this beats the from-scratch IndoBERTweet trained for 1M steps §4 tab. 2.
- Subword averaging reaches this in 200K steps, which the paper describes as five times faster than training from scratch §4 §6.
English tab. 3 tab. 4 §5. The authors also adapt RoBERTa (base) to English tweets with subword averaging. They train a new BPE tokenizer on English tweets and use 42M tweets for adaptation §5. Averaged over seven tasks, RoBERTa scores 72.9, BERTweet 76.3 and adapted RoBERTa 74.1 tab. 3. The per-task scores are in tab. 4.
Table 3 tab. 3 (reproduced in full).
| Model | Average |
|---|---|
| RoBERTa | 72.9 |
| BERTweet (Nguyen et al., 2020) | 76.3 |
| RoBERTa + vocabulary adaptation + 200K steps | 74.1 |
Table 4 tab. 4 (reproduced in full). English results (%) over the test sets, as in the paper's appendix.
| Model | POS Ritter11 | POS ARK | POS TB-v2 | SemEval17 Sentiment | SemEval18 Irony | NER WNUT16 | NER WNUT17 | Avg. |
|---|---|---|---|---|---|---|---|---|
| RoBERTa | 88.6 | 91.0 | 93.4 | 70.8 | 71.6 | 45.5 | 49.9 | 72.9 |
| BERTweet | 90.5 | 93.2 | 94.8 | 72.6 | 75.5 | 51.9 | 55.9 | 76.3 |
| RoBERTa w/ BERTweet tokenizer (steps = 0) | 87.7 | 90.2 | 92.5 | 65.7 | 66.1 | 39.2 | 39.2 | 68.7 |
| RoBERTa w/ new tokenizer (steps = 0) | 87.4 | 89.6 | 92.8 | 66.5 | 70.7 | 42.5 | 42.4 | 70.3 |
| RoBERTa w/ new tokenizer (steps = 200K) | 90.1 | 90.9 | 94.9 | 72.1 | 73.3 | 47.2 | 50.0 | 74.1 |
Figure 1 fig. 1. A frequency chart, not a table. It compares how many
subword pieces each new type gets: BERTweet (English) with the RoBERTa
tokenizer, and IndoBERTweet with the IndoBERT tokenizer. A value of 1 means the
type becomes [UNK]. The average piece count of new types is 2.6 for
IndoBERTweet and 3.4 for BERTweet §5 fig. 1. The figure is described here
only; no image is included.
Paper vs code
Searched. The paper's abstract footnote links https://github.com/indolem/IndoBERTweet,
which is the official repository §Abstract. I cloned it at commit
371dde282d79b96bab9804b7e6794dee08dd4606. The repository has no LICENSE
file, so code_license is recorded as "none". Its README says the pretraining
tweets are not public (README, section 3).
What the repository contains. Fine-tuning scripts for six tasks, with data
splits: sentiment_indolem, sentiment_SmSA, emotion_EmoT,
hate_speech_HS1, hate_speech_HS2 and NER. The model is loaded from
Hugging Face as indolem/indobertweet-base-uncased indobertweet.py.
What the repository does not contain. No pretraining code: no masked language modelling loop, no vocabulary-initialisation code (Methods 1 to 4), no fastText training, no projection fit and no adaptation script. The paper's pretraining claims (1M steps, 200K steps, 5x speed, the averaging rule) cannot be checked against this code. Note that the repository's fine-tuning code does not contain the averaging rule either.
Head. The paper describes an MLP layer over average-pooled output §3.
The code has one linear layer with no hidden layer. Binary tasks apply a
sigmoid to that output indobertweet.py
indobertweet.py. I read the IndoLEM and NER
scripts in full. For the others I checked only the head and the hyperparameter
lines: HS1 and HS2 use nn.Linear(hidden, 1), and EmoT and SmSA use
nn.Linear(hidden, label count) indobertweet.py
indobertweet.py. Their loss functions were not read.
Pooling. Mean over unmasked positions, including [CLS] and [SEP],
because the mask is any non-zero id indobertweet.py
indobertweet.py. The paper says "average
pooled" but does not say this.
Dropout. Dropout 0.2 on the token vectors, before pooling. The paper does not state dropout indobertweet.py indobertweet.py.
Epochs and warmup. The paper gives no epoch count. The code's default is 20
epochs indobertweet.py. Warmup is set to 10% of
total steps, overriding the argparse default of 242 indobertweet.py
indobertweet.py. NER uses int(0.1 * t_total)
indobertweet.py.
Clipping, weight decay, seeds. Gradient clipping at norm 1.0 and weight decay 0 are code defaults, and the paper states neither indobertweet.py indobertweet.py. Seeds are 2021 for IndoLEM, SmSA and HS2, and 2020 for HS1, EmoT and NER indobertweet.py indobertweet.py. The paper gives no seed.
Model selection. The code reports test F1 at the epoch with the best dev F1 indobertweet.py. The paper says only "early stopping with patience of 5" §3.
Cross-validation. The code runs five folds and prints the mean of the best dev and test F1 over folds indobertweet.py. The paper does not say how the 5-fold numbers in tab. 2 are aggregated.
Batch steps. The schedule's total step count uses floor division of the dataset size by the batch size, so it ignores the final partial batch, which the training loop still runs. Each batch's loss is divided by the nominal batch size, not its real size indobertweet.py indobertweet.py.
NER labels. The code takes the label set from the training data. Tags are
rewritten to B-PERSON, B-ORGANIZATION and B-LOCATION for seqeval, with no
I- tags indobertweet.py indobertweet.py.
Only the first subword is labelled. The other subwords are ignored in the loss
and in scoring indobertweet.py indobertweet.py.
The paper's label count of 3 for NER tab. 1 was not checked against the
training file.
Preprocessing. Classification handles @, [username], [url] and
httpurl. NER handles @ and httpurl only indobertweet.py
indobertweet.py. The paper describes only the mentions and
URLs rules.
For the fine-tuning batch size (30), maximum length (128), learning rate (5e-5), Adam epsilon (1e-8) and early-stopping patience (5), the code matches the paper indobertweet.py.
Pitfalls
- Pretraining is not reproducible from the release. The repository has only fine-tuning code, and the tweets are not public [README §3]. The vocabulary-averaging rule, the 1M and 200K step counts and the 5x speed claim come from the paper alone §2.3 §3 §4.
- Unclear vocabulary size. The paper gives 31,984 types §2.1 but says the
implementation discards
[unused-x]tokens to match IndoBERT's 31,923 §2.3 footnote. Which size the released model uses is not stated here; check its config on the model hub. - Averaging is underspecified. The paper calls the piece set T_IB(x) a set
§2.3. Duplicate pieces, the handling of
##continuation pieces when a new type is tokenised as a stand-alone word, and the denominator are not stated. The paper also does not say how an empty tokenisation is handled. - Projection details missing. The solver for the least-squares fit, the fastText window, epochs and minimum count, and how out-of-vocabulary new types get fastText vectors are not stated §2.3.
- Normal init statistics. Whether the mean and standard deviation are learned per dimension or over all embeddings is not stated §2.3.
- Adaptation hyperparameters missing. Learning rate, batch size and warmup for the 200K adaptation steps are not stated §3. The gradient accumulation sentence sits in the setup section and reads as applying to all pretraining runs, but this is not explicit.
- "5 times faster" counts steps, not time. The claim follows from 200K vs 1M steps §4 §6. The paper reports no wall-clock time, and the per-step cost of the two runs is not stated.
- Averages hide task-level losses. The average-of-subwords model is not best on every task. For example, HS2 is 84.0 at 200K steps vs 87.5 for IndoBERTweet from scratch, and Formal NER is 87.7 vs 88.1 tab. 2. The paper's averages are unweighted means over seven tasks tab. 2.
- No variance or significance. The paper reports single numbers and no seeds or confidence intervals. Gaps of a few tenths, such as 86.5 vs 86.0 vs 85.9 between initialisation methods at 200K steps tab. 2, are not shown to be stable.
- The baselines are not the same data. The Wilie et al. IndoBERT was pretrained on 100M words of tweets, and the Koto et al. IndoBERT was not §4 footnote. This confounds the comparison with IndoBERTweet.
- Metrics differ across tasks. F1_pos, macro F1 and entity-level F1 are all used tab. 1. Match the metric to the task when comparing.
- MLP versus linear head. The paper says MLP; the code uses one linear layer §3 indobertweet.py. Use the code behaviour if you want to match the reported numbers, and say so.
- Model-selection rule. Test scores are taken at the best dev epoch [code: sentiment_indolem/indobertweet.py#L193-L196]. The paper does not say this.
- English results are not comparable. RoBERTa and BERTweet use different tokenisers, the authors used a much smaller adaptation corpus (42M vs 845M tweets), and the paper says the two models are not directly comparable §5.
- Wording slips. The paper's sentence "3 feed-forward hidden layers" §2.2 reads as a BERT-base feed-forward size of 3072, which is my interpretation, not a stated fact.
- Version. This card follows the arXiv v1 LaTeX source (2021-09-10). The published proceedings version may differ.
Lineage
- Builds on: IndoBERT (Koto et al., 2020), the monolingual Indonesian BERT used as the starting model §2.3. No card in this wiki yet.
- Builds on: BERTweet (Nguyen et al., 2020), which the paper follows for preprocessing and for its English comparison §2.1 §5. No card in this wiki yet.
- Compared against: mBERT, malayBERT, IndoBERT (Wilie et al., 2020) §3 tab. 2; and the fastText projection of Poerner et al. (2020) for initialisation §2.3.
- Superseded by: not known from content in this wiki.
Source and license
Adapted from “IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization” by Fajri Koto, Jey Han Lau, Timothy Baldwin (arXiv:2109.04607), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/indolem/IndoBERTweet at commit 371dde282d79 (none (no LICENSE file in repository)). Any quoted code is shown under that license.