wikillm

Paper · 2021-09-10 · cs.CL

IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization

Trains IndoBERTweet, a BERT model for Indonesian tweets, from scratch for 1M steps. It also tests a cheaper route: adapt IndoBERT to 26M tweets with a new vocabulary whose new tokens start as averaged IndoBERT subword embeddings, then train 200K steps. That route scores 86.5 average over seven Indonesian Twitter tasks, vs 86.1 from scratch. Author-reported.

Language model pretrainingDomain adaptationSubword tokenizationMasked language modelingindonesiantwitterbertpretrainingvocabulary-initializationnamed-entity-recognitionsentiment-analysishate-speech-detectionemotion-classification

arXivPDFCodeMarkdown15 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Language models that read Indonesian are often trained on news and web text. Tweets look different: they are short, informal, full of slang and mentions, and they use words the model has never seen. The authors build a model for Indonesian Twitter. They start from an existing Indonesian BERT and extend its word list with words that are common in tweets but missing from the original.

The hard part is what to put in the new word slots. Starting them from random numbers works, but the model then has to learn them from scratch. The authors find that a simple rule works better than the alternatives: for each new word, take the old model's vectors for the smaller pieces the old tokenizer would split it into, and average them. The adapted model then needs far less extra training than a model built from zero.

The adapted model does better on average than the existing Indonesian models on seven tweet tasks, including sentiment, emotion, hate speech and named entity recognition. Results are reported by the authors. The paper matters to anyone who works on Indonesian social-media text, and to anyone who wants to adapt a pretrained model to a new domain without paying for full pretraining.

TL;DR

IndoBERTweet is a BERT-base model trained from scratch for 1M steps on 26M Indonesian tweets, with a 31,984-type WordPiece vocabulary §2.1 §2.2 §3. The paper also tests a cheaper route: extend IndoBERT's vocabulary and continue pretraining for 200K steps §2.3 §4. Of four ways to initialise the new entries, averaging the IndoBERT subword embeddings scores best on average §4 tab. 2. Results are author-reported, on seven Indonesian Twitter datasets tab. 1 tab. 2. Several claims are only averages, and no significance tests are reported §4. Read Pitfalls before reusing numbers.

Problem

Domain-general BERT models tokenise domain words poorly, and domain pretraining from scratch is expensive §1. The paper argues that domain-adaptive pretraining is cheaper but tends to be limited by the original vocabulary, which splits many domain words into many pieces §1. For Indonesian Twitter, the authors also note that Indonesian is underrepresented in NLP and that most earlier Indonesian Twitter work used traditional models §1. The question is how to add domain words to an existing vocabulary so that the adapted model learns quickly and performs well §1 §2.3.

Method

Twitter corpus §2.1. Indonesian tweets were crawled from December 2019 to December 2020 with the Twitter API, using 60 keywords in four topics: economy, health, education and government. Non-Indonesian tweets were filtered with Twitter's language identifier. In a sample of 100 tweets, 87 were Indonesian, 12 Malay and 1 Swahili §2.1. After removing duplicate IDs, the corpus has 26M tweets and 409M word tokens. 230K tweets are held out for development §2.1. The vocabulary is 31,984 WordPiece types, and all text is lower-cased §2.1. Preprocessing follows English BERTweet: user mentions become @USER, URLs become HTTPURL, and emoticons are translated to text with the emoji package §2.1.

Model §2.2. IndoBERTweet is trained with a masked language modelling objective, following the same procedure as indobert-base-uncased §2.2 tab. 1. The architecture is BERT-base-sized: 12 layers, hidden size 768, 12 attention heads, and feed-forward size 3072 §2.2. The only architectural change is a maximum sequence length of 128 tokens, chosen from the average number of words per tweet §2.2. The paper trains five models: one from scratch for 1M steps, and four adapted from IndoBERT with different vocabulary initialisations §2.2 §3.

Datasets tab. 1. Seven Indonesian Twitter datasets in four tasks §3. Table 1 is reproduced under Hyperparameters below.

Vocabulary overlap §2.3. IndoBERT has 31,923 types and the IndoBERTweet vocabulary has 31,984. Of the IndoBERTweet types, 14,584 (46%) are new and 17,400 (54%) are shared with IndoBERT §2.3. A footnote says that the implementation trimmed the adaptive vocabulary to the same size as IndoBERT by discarding some [unused-x] tokens §2.3.

Four initialisations for new types §2.3. Shared types keep their IndoBERT embeddings. The four strategies for new types are:

  • Method 1: random values from the uniform distribution U(-1, 1) §2.3.
  • Method 2: random values from a normal distribution whose mean and standard deviation are learned from IndoBERT embeddings §2.3. The paper does not say whether these statistics are global or per dimension.
  • Method 3: a linear projection from 300-dimensional fastText vectors trained on the tokenised tweet corpus §2.3. The projection matrix W is fitted on the shared types only, by minimising the squared distance between fastText vectors times W and the IndoBERT vectors §2.3 (unnumbered equation). Method 3 follows Poerner et al. (2020), as the paper states §2.3.
  • Method 4: the average of the IndoBERT embeddings of the WordPiece pieces that IndoBERT's tokenizer produces for the new word §2.3 (unnumbered equation).
Pseudocode
# Build the new embedding table with subword averaging (Method 4)  [§2.3]
# Input:  ibt_vocab (list of new and shared types),
#         ib_embeddings (V_IB, 768), ib_vocab (type -> row index),
#         tokenize (IndoBERT WordPiece tokenizer, word -> list of row ids)
# Output: embeddings (V_IBT, 768)

def init_average(ibt_vocab, ib_embeddings, ib_vocab, tokenize):
    embeddings = []
    for token in ibt_vocab:
        if token in ib_vocab:
            # Shared types keep their IndoBERT row  [§2.3]
            embeddings.append(ib_embeddings[ib_vocab[token]])
        else:
            # New types: mean of the IndoBERT rows of their pieces  [§2.3]
            pieces = tokenize(token)  # list of row ids
            vector = mean(ib_embeddings[piece] for piece in pieces)
            embeddings.append(vector)
    return stack(embeddings)  # (V_IBT, 768)
Pseudocode
# Fit and apply the fastText projection (Method 3)  [§2.3]
# Input:  shared_types (list), new_types (list),
#         fasttext (word -> 300-d vector), ib_embeddings (word -> 768-d)
# Output: init vectors for new_types, (len(new_types), 768)

def init_projection(shared_types, new_types, fasttext, ib_emb):
    # Stack fastText and IndoBERT vectors for shared types  [§2.3]
    sources = stack(fasttext[x] for x in shared_types)  # (S, 300)
    targets = stack(ib_emb[x] for x in shared_types)    # (S, 768)
    # Least-squares fit of W  [§2.3] (unnumbered equation)
    W = least_squares(sources, targets)                 # (300, 768)
    # Map each new type's fastText vector through W  [§2.3]
    return stack(fasttext[x] @ W for x in new_types)    # (N, 768)

Pretraining §3. The from-scratch model is trained for 1M steps with a learning rate of 1e-4, the Adam optimiser and a linear schedule §3. Gradients are accumulated over 4 steps to give a batch size of 2048 §3. Experiments run on 4 V100 GPUs with 32GB each §3. Domain-adaptive pretraining runs for 200K steps and is done in three forms: without vocabulary change, with the new vocabulary and no further training (0 steps), and with the new vocabulary plus 200K steps §3 §4. The paper does not state the learning rate, batch size or warmup for the adaptation runs §3.

Fine-tuning §3. For sentiment, emotion and hate speech, an MLP layer reads the average-pooled IndoBERTweet output. For NER, the first subword of each word predicts the tag. Fine-tuning uses batch size 30, maximum length 128, learning rate 5e-5, Adam with epsilon 1e-8, and early stopping with patience 5 §3. IndoLEM, HS1 and HS2 use 5-fold cross-validation; the paper describes a canonical split for the hate speech tasks §3 tab. 1. The other datasets use their original held-out splits §3 tab. 1.

Pseudocode
# Classification head for sentiment, emotion and hate speech  [§3]
# Input:  token_ids (B, T), token_types (B, T), mask (B, T), mask in {0, 1}
# Output: scores for each tweet, (B,) or (B, classes)

def classify(token_ids, token_types, mask):
    # Final-layer token vectors from IndoBERTweet  [§3]
    hidden = bert(token_ids, token_types, mask)  # (B, T, 768)
    # Dropout 0.2 on token vectors
    # [code: sentiment_indolem/indobertweet.py#L104]
    hidden = dropout(hidden)                     # (B, T, 768)
    # Average over unmasked positions, [CLS] and [SEP] included  [§3]
    # [code: sentiment_indolem/indobertweet.py#L110-L112]
    pooled = sum(hidden * mask) / sum(mask)      # (B, 768)
    # Linear layer, then sigmoid; the paper calls this an MLP [§3]
    # [code: sentiment_indolem/indobertweet.py#L103, #L113-L114]
    return sigmoid(linear(pooled))               # (B,) binary tasks only

NER §3. Each word is split into WordPieces, and the first piece carries the tag. The paper reports entity-level F1 §3 tab. 1. The train and dev partitions mix formal and informal tweets and are shared by the two test sets (formal and informal) §3.

Hyperparameters

NameValueWhereNotes
Tweet corpus26M tweets, 409M word tokens§2.1230K tweets held out for dev
Vocabulary size31,984 types (IndoBERT: 31,923)§2.1 §2.3footnote: implementation trimmed to IndoBERT's size
New / shared types14,584 (46%) / 17,400 (54%)§2.3
Casing and preprocessinglower-cased; mentions to @USER, URLs to HTTPURL, emoji to text§2.1
Architecture12 layers, hidden 768, 12 heads, feed-forward 3072§2.2"3 feed-forward hidden layers" is worded oddly; read as BERT-base
Max sequence length128§2.2 §3
Pretraining steps, from scratch1M§3
Adaptation steps200K§3
Pretraining learning rate, from scratch1e-4§3Adam, linear schedule; warmup not stated
Effective batch size2048§34 accumulation steps; micro-batch not stated
Adaptation learning rate, batch size, warmupnot stated§3
Masking rate, mask/random/keep splitnot stated§2.2"same procedure as IndoBERT"
fastText dimension300§2.3window, epochs, minimum count not stated
Projection solvernot stated§2.3fit described as an argmin
Init range, Method 1U(-1, 1)§2.3
Init statistics, Method 2mean and std learned from IndoBERT embeddings§2.3global or per dimension not stated
Fine-tuning batch size30§3
Fine-tuning learning rate5e-5§3
Adam epsilon1e-8§3
Early-stopping patience5§3
Fine-tuning epochsnot stated§3code default 20; see Paper vs code
Fine-tuning warmup, weight decay, grad clipnot stated§3see Paper vs code
Seedsnot stated§3see Paper vs code
Cross-validation5-fold for IndoLEM, HS1, HS2§3 tab. 1

Datasets tab. 1

TaskDatasetLabel countTrain sizeDev sizeTest sizeCross-validationMetric
Sentiment analysisIndoLEM23,6383991,011YesF1_pos
Sentiment analysisSmSA311,0001,260500NoF1_macro
Emotion classificationEmoT53,521440442NoF1_macro
Hate speech detectionHS1249972142YesF1_pos
Hate speech detectionHS229,2192,6331,317YesF1_pos
Named entity recognitionFormal36,5006571,122NoF1_entity
Named entity recognitionInformal36,5006571,227NoF1_entity

Results

Author-reported. Setup: seven Indonesian Twitter datasets tab. 1; the baselines are mBERT, malayBERT and two IndoBERT models §3. Scores are in percent; the metric for each dataset is in tab. 1. The "Average" column is the unweighted mean of the seven scores. Bold follows the paper's marks, which are set within each block of rows tab. 2.

Table 2 tab. 2 (reproduced in full).

ModelIndoLEMSmSAEmoTHS1HS2FormalInformalAverage
mBERT76.684.767.585.175.185.283.279.6
malayBERT82.084.174.285.081.981.981.381.5
IndoBERT (Wilie et al., 2020)84.188.773.386.880.486.384.383.4
IndoBERT (Koto et al., 2020)84.187.971.086.479.388.086.983.4
IndoBERTweet (1M steps)86.290.476.088.887.588.185.486.1
IndoBERT + 200K steps of domain-adaptive pretraining
Same vocabulary (V_IBT = V_IB)86.492.776.888.782.287.986.985.9
IndoBERT + vocabulary adaptation + 0 steps
Uniform distribution82.984.673.284.978.284.384.481.8
Normal distribution83.586.771.185.277.485.086.382.2
fastText projection84.483.672.285.580.985.485.682.5
Average of subwords84.288.171.686.278.386.487.483.2
IndoBERT + vocabulary adaptation + 200K steps
Uniform distribution85.690.975.788.483.087.785.985.3
Normal distribution87.192.575.488.882.588.786.685.9
fastText projection86.489.778.588.784.488.086.686.0
Average of subwords86.692.779.088.484.087.786.986.5

Table 1 tab. 1 is reproduced under Hyperparameters above.

Headline results, as stated by the paper §4.

  • IndoBERTweet trained from scratch is about 3 points higher on average than the IndoBERT baselines (86.1 vs 83.4) §4 tab. 2.
  • IndoBERT trained 200K more steps with the same vocabulary averages 85.9 §4 tab. 2.
  • Adding the new vocabulary with no training (0 steps) lowers the average, and subword averaging has the smallest drop, 0.2 points below the original IndoBERT (83.2 vs 83.4) §4 tab. 2.
  • With 200K adaptation steps after vocabulary adaptation, subword averaging gives the best average, 86.5. The paper says this beats the from-scratch IndoBERTweet trained for 1M steps §4 tab. 2.
  • Subword averaging reaches this in 200K steps, which the paper describes as five times faster than training from scratch §4 §6.

English tab. 3 tab. 4 §5. The authors also adapt RoBERTa (base) to English tweets with subword averaging. They train a new BPE tokenizer on English tweets and use 42M tweets for adaptation §5. Averaged over seven tasks, RoBERTa scores 72.9, BERTweet 76.3 and adapted RoBERTa 74.1 tab. 3. The per-task scores are in tab. 4.

Table 3 tab. 3 (reproduced in full).

ModelAverage
RoBERTa72.9
BERTweet (Nguyen et al., 2020)76.3
RoBERTa + vocabulary adaptation + 200K steps74.1

Table 4 tab. 4 (reproduced in full). English results (%) over the test sets, as in the paper's appendix.

ModelPOS Ritter11POS ARKPOS TB-v2SemEval17 SentimentSemEval18 IronyNER WNUT16NER WNUT17Avg.
RoBERTa88.691.093.470.871.645.549.972.9
BERTweet90.593.294.872.675.551.955.976.3
RoBERTa w/ BERTweet tokenizer (steps = 0)87.790.292.565.766.139.239.268.7
RoBERTa w/ new tokenizer (steps = 0)87.489.692.866.570.742.542.470.3
RoBERTa w/ new tokenizer (steps = 200K)90.190.994.972.173.347.250.074.1

Figure 1 fig. 1. A frequency chart, not a table. It compares how many subword pieces each new type gets: BERTweet (English) with the RoBERTa tokenizer, and IndoBERTweet with the IndoBERT tokenizer. A value of 1 means the type becomes [UNK]. The average piece count of new types is 2.6 for IndoBERTweet and 3.4 for BERTweet §5 fig. 1. The figure is described here only; no image is included.

Paper vs code

Searched. The paper's abstract footnote links https://github.com/indolem/IndoBERTweet, which is the official repository §Abstract. I cloned it at commit 371dde282d79b96bab9804b7e6794dee08dd4606. The repository has no LICENSE file, so code_license is recorded as "none". Its README says the pretraining tweets are not public (README, section 3).

What the repository contains. Fine-tuning scripts for six tasks, with data splits: sentiment_indolem, sentiment_SmSA, emotion_EmoT, hate_speech_HS1, hate_speech_HS2 and NER. The model is loaded from Hugging Face as indolem/indobertweet-base-uncased indobertweet.py.

What the repository does not contain. No pretraining code: no masked language modelling loop, no vocabulary-initialisation code (Methods 1 to 4), no fastText training, no projection fit and no adaptation script. The paper's pretraining claims (1M steps, 200K steps, 5x speed, the averaging rule) cannot be checked against this code. Note that the repository's fine-tuning code does not contain the averaging rule either.

Head. The paper describes an MLP layer over average-pooled output §3. The code has one linear layer with no hidden layer. Binary tasks apply a sigmoid to that output indobertweet.py indobertweet.py. I read the IndoLEM and NER scripts in full. For the others I checked only the head and the hyperparameter lines: HS1 and HS2 use nn.Linear(hidden, 1), and EmoT and SmSA use nn.Linear(hidden, label count) indobertweet.py indobertweet.py. Their loss functions were not read.

Pooling. Mean over unmasked positions, including [CLS] and [SEP], because the mask is any non-zero id indobertweet.py indobertweet.py. The paper says "average pooled" but does not say this.

Dropout. Dropout 0.2 on the token vectors, before pooling. The paper does not state dropout indobertweet.py indobertweet.py.

Epochs and warmup. The paper gives no epoch count. The code's default is 20 epochs indobertweet.py. Warmup is set to 10% of total steps, overriding the argparse default of 242 indobertweet.py indobertweet.py. NER uses int(0.1 * t_total) indobertweet.py.

Clipping, weight decay, seeds. Gradient clipping at norm 1.0 and weight decay 0 are code defaults, and the paper states neither indobertweet.py indobertweet.py. Seeds are 2021 for IndoLEM, SmSA and HS2, and 2020 for HS1, EmoT and NER indobertweet.py indobertweet.py. The paper gives no seed.

Model selection. The code reports test F1 at the epoch with the best dev F1 indobertweet.py. The paper says only "early stopping with patience of 5" §3.

Cross-validation. The code runs five folds and prints the mean of the best dev and test F1 over folds indobertweet.py. The paper does not say how the 5-fold numbers in tab. 2 are aggregated.

Batch steps. The schedule's total step count uses floor division of the dataset size by the batch size, so it ignores the final partial batch, which the training loop still runs. Each batch's loss is divided by the nominal batch size, not its real size indobertweet.py indobertweet.py.

NER labels. The code takes the label set from the training data. Tags are rewritten to B-PERSON, B-ORGANIZATION and B-LOCATION for seqeval, with no I- tags indobertweet.py indobertweet.py. Only the first subword is labelled. The other subwords are ignored in the loss and in scoring indobertweet.py indobertweet.py. The paper's label count of 3 for NER tab. 1 was not checked against the training file.

Preprocessing. Classification handles @, [username], [url] and httpurl. NER handles @ and httpurl only indobertweet.py indobertweet.py. The paper describes only the mentions and URLs rules.

For the fine-tuning batch size (30), maximum length (128), learning rate (5e-5), Adam epsilon (1e-8) and early-stopping patience (5), the code matches the paper indobertweet.py.

Pitfalls

  • Pretraining is not reproducible from the release. The repository has only fine-tuning code, and the tweets are not public [README §3]. The vocabulary-averaging rule, the 1M and 200K step counts and the 5x speed claim come from the paper alone §2.3 §3 §4.
  • Unclear vocabulary size. The paper gives 31,984 types §2.1 but says the implementation discards [unused-x] tokens to match IndoBERT's 31,923 §2.3 footnote. Which size the released model uses is not stated here; check its config on the model hub.
  • Averaging is underspecified. The paper calls the piece set T_IB(x) a set §2.3. Duplicate pieces, the handling of ## continuation pieces when a new type is tokenised as a stand-alone word, and the denominator are not stated. The paper also does not say how an empty tokenisation is handled.
  • Projection details missing. The solver for the least-squares fit, the fastText window, epochs and minimum count, and how out-of-vocabulary new types get fastText vectors are not stated §2.3.
  • Normal init statistics. Whether the mean and standard deviation are learned per dimension or over all embeddings is not stated §2.3.
  • Adaptation hyperparameters missing. Learning rate, batch size and warmup for the 200K adaptation steps are not stated §3. The gradient accumulation sentence sits in the setup section and reads as applying to all pretraining runs, but this is not explicit.
  • "5 times faster" counts steps, not time. The claim follows from 200K vs 1M steps §4 §6. The paper reports no wall-clock time, and the per-step cost of the two runs is not stated.
  • Averages hide task-level losses. The average-of-subwords model is not best on every task. For example, HS2 is 84.0 at 200K steps vs 87.5 for IndoBERTweet from scratch, and Formal NER is 87.7 vs 88.1 tab. 2. The paper's averages are unweighted means over seven tasks tab. 2.
  • No variance or significance. The paper reports single numbers and no seeds or confidence intervals. Gaps of a few tenths, such as 86.5 vs 86.0 vs 85.9 between initialisation methods at 200K steps tab. 2, are not shown to be stable.
  • The baselines are not the same data. The Wilie et al. IndoBERT was pretrained on 100M words of tweets, and the Koto et al. IndoBERT was not §4 footnote. This confounds the comparison with IndoBERTweet.
  • Metrics differ across tasks. F1_pos, macro F1 and entity-level F1 are all used tab. 1. Match the metric to the task when comparing.
  • MLP versus linear head. The paper says MLP; the code uses one linear layer §3 indobertweet.py. Use the code behaviour if you want to match the reported numbers, and say so.
  • Model-selection rule. Test scores are taken at the best dev epoch [code: sentiment_indolem/indobertweet.py#L193-L196]. The paper does not say this.
  • English results are not comparable. RoBERTa and BERTweet use different tokenisers, the authors used a much smaller adaptation corpus (42M vs 845M tweets), and the paper says the two models are not directly comparable §5.
  • Wording slips. The paper's sentence "3 feed-forward hidden layers" §2.2 reads as a BERT-base feed-forward size of 3072, which is my interpretation, not a stated fact.
  • Version. This card follows the arXiv v1 LaTeX source (2021-09-10). The published proceedings version may differ.

Lineage

  • Builds on: IndoBERT (Koto et al., 2020), the monolingual Indonesian BERT used as the starting model §2.3. No card in this wiki yet.
  • Builds on: BERTweet (Nguyen et al., 2020), which the paper follows for preprocessing and for its English comparison §2.1 §5. No card in this wiki yet.
  • Compared against: mBERT, malayBERT, IndoBERT (Wilie et al., 2020) §3 tab. 2; and the fastText projection of Poerner et al. (2020) for initialisation §2.3.
  • Superseded by: not known from content in this wiki.

Source and license

Adapted from “IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization” by Fajri Koto, Jey Han Lau, Timothy Baldwin (arXiv:2109.04607), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/indolem/IndoBERTweet at commit 371dde282d79 (none (no LICENSE file in repository)). Any quoted code is shown under that license.