wikillm

Paper · 2021-04-01 · stat.ML, cs.LG

Regularized target encoding outperforms traditional methods in supervised machine learning with high cardinality features

Compares ways to turn high-cardinality categorical features into numbers for five learners on 24 OpenML datasets. Regularized target encoding, a per-feature random-intercept model fitted with 5-fold cross-fitting, ranked first for all learners except SVM (author-reported). Indicator, integer, frequency, hash, leaf and impact encoders trailed it.

Categorical encodingTarget encodingMixed-effects modelsGradient-boosted decision treestabularcategorical-featurestarget-encodingbenchmark

arXivPDFCodeMarkdown12 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Many tabular datasets have a column whose values are labels with a huge number of options, such as a zip code, a product ID or a job title. Most learning algorithms need numbers, so the label has to be converted first. The obvious trick of giving every label its own yes/no column becomes unwieldy when there are thousands of labels. Other tricks throw away labels or assign arbitrary numbers, and the study finds they did less well.

The strongest option in this study is target encoding: replace each label with a number learned from the outcome you want to predict, such as the average outcome for that label. Done naively, this overfits rare labels, because a label seen once gets its own outcome as its number. The regularized version shrinks each label's number towards the overall average, and it learns those numbers on other parts of the training data than the rows being encoded.

The authors compare these encoders across many datasets and five kinds of learner. Their finding is that the regularized target encoder should be a reasonable default. The ranking comes from one benchmark with limited tuning, and the best encoder is not the same for every learner.

TL;DR

  • For a categorical feature with many levels, the study compares seven encoders plus control conditions on 24 datasets with five learners §1.4 §3.1 §3.3.
  • The best performer is a target encoder that fits a random-intercept mixed model per feature and uses 5-fold cross-fitting for training rows §2.7 §4.3.
  • Author-reported: it ranked first for every learner except SVM, and no target-agnostic encoder convincingly beat it §4.3.
  • Runtime was not consistently worse than one-hot encoding. The largest increases were for the mixed-model encoders, especially with gradient boosting §4.4 tab. 1.
  • Several implementation details that matter are not stated in the paper, and the released code differs from the paper in places (see Paper vs code).

Problem

Categorical features with many unordered levels break simple encodings. One-hot or dummy coding creates one column per level, which becomes inefficient as the number of levels grows §1. Integer codes impose an order that does not exist §2.1. Target encoding can memorise rare levels, so a model trusts the encoded feature too much and generalises poorly §1.2. Earlier benchmarks were small, used only gradient boosting or only regression, or did not focus on high cardinality §1.3. The study assumes i.i.d. tabular data and encodes each categorical feature separately §1.1.

Method

The study compares encoders on the same ML pipeline. Each encoder maps a categorical feature to numbers. Features are encoded separately, and new levels at prediction time are handled per encoder §2 §2.1 §2.3.

The encoders (paraphrased from §2; the Supplementary Material with pseudocode for all encoders is not used here):

  1. Integer encoding maps observed levels to 1..L. Unseen levels are set to missing and then mode-imputed §2.1.
  2. Frequency encoding maps each level to its training-set count. Unseen levels get count 1 §2.2.
  3. Indicator encoding (one-hot or dummy). The HCT−1 most frequent levels get their own column and the rest share one "other" column. Unseen levels map to the zero vector for one-hot, and to the most frequent training level for dummy coding §2.3.
  4. Hash encoding maps each level to one of hash.size buckets, with the bucket size equal to HCT §2.4 §3.2.
  5. Leaf encoding fits a classification or regression tree on the feature alone, prunes it with 10-fold cross-validation, and uses the terminal node as the new level. The resulting factor is then one-hot encoded §2.5.
  6. Impact encoding (smoothed target encoding) replaces each level with its conditional target mean, or with class frequencies for classification, using a small smoothing constant ε = 0.0001 §2.6.
  7. GLMM encoding fits a random-intercept model per feature. For regression it is a linear mixed model; for classification it is a logistic mixed model, one one-vs-rest model per class. Each level's encoded value is its conditional mode. Unseen levels at prediction time use the fixed intercept §2.7.
    • Variants: no cross-fitting (glmm-noCV); 5-fold cross-fitting (glmm-5CV); 10-fold cross-fitting (glmm-10CV). With cross-fitting, each training row is encoded by the one model that did not see its fold, while the prediction-time model uses all training rows §2.7.

Control conditions §2.8: a featureless learner (predicts the training mean or majority class); a random forest without encoding; and removing high-cardinality features.

The GLMM encoder, as an implementer would read it §2.7 mlrCPOs.R mlrCPOs.R mlrCPOs.R:

Pseudocode
# Fit one level-encoding model on a set of rows  [§2.7]
# Input:  levels (n,) categorical feature; target (n,)
# Output: map from level to encoded value, plus a value for unseen levels
def fit_level_model(levels, target):
    # Random intercept per level: target ~ 1 + (1 | level)  [§2.7]
    # Regression: linear mixed model; classification: logistic mixed model
    model = fit_mixed_model(target, random_intercept=levels)
    # Fixed intercept, reused for levels not seen in this fit  [§2.7]
    global_value = model.fixed_intercept
    # Conditional mode per observed level  [§2.7]
    level_values = model.conditional_mode_by_level
    level_values[NEW_LEVEL] = global_value
    return level_values
Pseudocode
# Encode training rows with n_folds cross-fitting  [§2.7]
# Input:  levels (n,); target (n,); n_folds (1, 5 or 10)
# Output: encoded training values (n,); a model for prediction time
def fit_encode_train(levels, target, n_folds):
    # Model for prediction time, fitted on all training rows  [§2.7]
    full_model = fit_level_model(levels, target)
    # No cross-fitting: rows are encoded by a model that saw them  [§2.7]
    if n_folds == 1:
        return encode(full_model, levels), full_model
    encoded = empty(n)
    # Each row is encoded by the one model fitted without its fold  [§2.7]
    for fold_rows, other_rows in make_folds(n, n_folds):
        fold_model = fit_level_model(levels[other_rows], target[other_rows])
        encoded[fold_rows] = encode(fold_model, levels[fold_rows])
    return encoded, full_model

Pipeline shared by all encoders §3.3 (code order: pipeline.R):

  1. Impute: a new "missing" level for categorical features, and the mean for numeric features. The paper's description of the binary-feature case differs from the code (see Paper vs code).
  2. Encode the high-cardinality features with the chosen encoder.
  3. Drop constant columns (these can arise from encoding or from CV splits).
  4. One-hot encode any remaining categorical features.
  5. Fit the learner on the transformed training data. At prediction time the fitted steps are applied to the test data.

Learners §3.3: LASSO (glmnet, λ by internal 5-fold CV); random forest (500 trees, untuned); gradient boosting (xgboost, learning rate 0.01, number of rounds by early stopping on a 20% holdout); k-nearest neighbours (k = 15, standardised, information-gain filter to 25 features); RBF-kernel SVM (liquidSVM, bandwidth and regularisation tuned by internal 5-fold CV, one-vs-all for multiclass).

Hyperparameters

NameValueWhereNotes
High-cardinality threshold (HCT)10, 25, 125§3.2 global_params.RFeature is encoded if it has more than HCT levels; also the hash size.
Outer CV5-fold§3.4 global_params.RClassification stratified in code; the paper does not say.
Inner CV (tuning)3-foldglobal_params.RCode only; not in the paper.
MetricsRMSE (regression), AUC (binary), AUNU (multiclass)§3.4Author choices; AUNU from Ferri et al.
Smoothing ε (impact)0.0001§2.6 global_params.R
GLMM cross-fitting foldsnone (1), 5, 10§2.7 global_params.Rglmm-5CV is the headline variant.
Hash sizeequal to HCT§3.2 algorithms.R
Leaf pruning CV10-fold§2.5 mlrCPOs.RTree control defaults (minsplit 20, minbucket 10, cp 0, max depth 30) are code only.
Random forest trees500, untuned§3.3 pipeline.R
Gradient boosting learning rate0.01§3.3 pipeline.R
Gradient boosting early-stopping holdout20%§3.3 xgboost.earlystop.wrap.RModel is fitted on the other 80%.
Gradient boosting patience / max rounds10 / 6000pipeline.RCode only.
LASSOglmnet, α = 1, λ = lambda.min, 5-fold internal CV§3.3 pipeline.R
k-NNk = 15, standardised, information-gain filter to 25 features§3.3 pipeline.RCode also sets Minkowski distance 2 and a rectangular kernel.
SVMliquidSVM, Gaussian RBF, scaled, partition choice 6§3.3 pipeline.RPartition choice is code only. Inner folds default to 5 liquidsvm.R.
Benchmark seed250319global_params.RCode only.

The paper gives no numbered equations. The target encoding formula is written inline in the introduction and the frequency and hash definitions are inline in §2 §1.2 §2.2 §2.4. The only displayed equation is the consensus-ranking argmin in §4.2.

Results

Author-reported. Setup: 24 OpenML datasets (8 regression, 10 binary, 6 multiclass with 3 to 12 classes), 736 to 1,224,158 observations, 5 to 208 features, and between 1 and 20 categorical columns per dataset with over ten levels each §3.1. Five learners, 5-fold CV, RMSE, AUC and AUNU §3.3 §3.4. For six datasets some SVM conditions crashed, so those datasets were removed from SVM rank statistics §4. Per-dataset numbers are only in the figures fig. 2 to fig. 4 and these are not transcribed here.

Runtime relative to one-hot encoding (median across datasets of runtime ratio from 5-fold CV, best HCT per setting; the number in brackets is the min to max range across datasets) tab. 1:

EncodingLASSORFGBKNNSVM
integer0.4 [0, 1.4]0.59 [0, 1.1]0.67 [0, 0.9]0.85 [0.4, 1.5]1.01 [0.6, 1.6]
frequency0.34 [0, 2.1]0.54 [0, 1.7]0.54 [0, 1.5]0.79 [0.2, 1.1]1.11 [0.5, 1.7]
dummy1.13 [0.8, 2.8]0.91 [0.5, 1.7]0.97 [0.4, 5.7]1.05 [0.6, 1.6]1.02 [0.7, 1.5]
one-hot1 [1, 1]1 [1, 1]1 [1, 1]1 [1, 1]1 [1, 1]
hash0.87 [0.2, 2.6]1.03 [0.4, 3.3]0.98 [0.4, 7.3]0.93 [0.5, 2.2]1.16 [0.5, 2]
leaf0.46 [0, 1.7]0.67 [0, 2.3]0.85 [0, 9.1]0.97 [0.5, 2.2]1.01 [0.6, 1.9]
impact0.5 [0.1, 1.9]0.6 [0.1, 1.5]0.82 [0, 120.5]1.11 [0.3, 24]1.06 [0.7, 1.8]
glmm-noCV0.58 [0, 2.8]0.57 [0.1, 3.8]1.65 [0, 12]1.09 [0.3, 10.3]1.1 [0.7, 2.1]
glmm-5CV0.83 [0.1, 2.2]0.66 [0.1, 15.3]7.21 [0, 69.3]2.77 [0.7, 50.1]1.41 [0.6, 4.6]
glmm-10CV1.07 [0.1, 4.2]0.96 [0.1, 28.4]12.71 [0, 127.6]5.37 [0.8, 97.9]1.55 [0.5, 4.6]
none (RF only)—0.2 [0, 1.4]———
remove0.4 [0, 1.7]0.48 [0, 1.1]0.58 [0, 1.3]0.82 [0.2, 1.2]1.09 [0.6, 2]

tab. 1 Values are ratios of runtime to one-hot, as the table's title describes.

Headline findings (author-reported):

FindingWhere
glmm-5CV ranked first for every learner except SVM§4.3
In meta rankings, GLMM-plus-CV encoders beat all others across algorithms; the RF with no encoding was beaten by all except remove§4.2 fig. 5
Regularized target encoding was superior or competitive on every dataset; no target-agnostic method convincingly beat it§4.3
Often no gain from 10-fold over 5-fold cross-fitting§4.3
Regularized GLMM encoding was better than or equal to unregularized impact encoding; where GLMM was better, impact was sometimes worse than other encoders§4.3
Integer encoding did poorly with gradient boosting§4.3
For LASSO, indicator encoding was second best§4.3
One-hot was usually slightly better than dummy encoding for nonlinear learners§4.3
Regularized target encoding did not consistently increase runtime; glmm with gradient boosting was clearly slower§4.4 tab. 1
No consistent best HCT setting emerged across datasets§4.5
Target encoding was never outperformed in this benchmark§5

Paper vs code

The code is the authors' repository at the commit in the frontmatter. It has no LICENSE file, so no code is quoted here; code is cited by path and line only.

  • HCT counting. The paper counts levels in the training set §3.2. The code counts levels of the factor columns of the full task, which problems.R builds from the whole OpenML dataset, and it ignores the training split algorithms.R problems.R.
  • Which HCT is reported. The paper says it reports only the best HCT per learner and dataset §3.4 §4.1. The released driver defines no HCT job parameter main.R. Instead the encoder's HCT, folds and tree count are tuned inside each outer training fold by a 3-fold inner grid search pipeline.R. The post-processing that picks the best HCT by outer test mean is in the manuscript script manuscript.Rmd, and collect_results.R filters on a high.card.thresh column collect_results.R that the released driver does not create. Which procedure produced the published numbers cannot be determined from the released code.
  • Binary features in imputation. The paper imputes missing values in binary features with the mode §3.3. The code gives every factor a constant "missing" level, binary ones included pipeline.R.
  • Integer encoding, unseen levels. The paper imputes unseen levels with the mode §2.1. The code's integer encoder returns NA for unseen levels mlrCPOs.R and the pipeline has only one imputation step, before encoding pipeline.R. Whether NA reaches the learner is not determined from the code. Related Work describes integer encoding with random level order §1.2; the code assigns codes by a random permutation mlrCPOs.R.
  • Leaf encoding, unseen levels. The paper says unseen levels go to the terminal node with the most training observations §2.5. The code has no such fallback; it calls predict_nodes with na.pass mlrCPOs.R. How rpart treats an unseen factor level there is not verified here. Tree settings (minsplit 20, minbucket 10) are not stated in the paper mlrCPOs.R.
  • Impact encoder not in the repository. The benchmark calls cpoImpactEncodeRegr and cpoImpactEncodeClassif algorithms.R, but no definition exists in the released code. The logit link and per-class columns described in §2.6 cannot be checked.
  • GLMM values on classification are on the logit scale. The paper does not state the scale §2.7. The binomial model's per-level coefficients are log-odds mlrCPOs.R mlrCPOs.R.
  • GLMM optimiser. The paper names lme4 only §2.7. The encoder defaults to fast.optim = TRUE, which uses nloptwrap with derivatives turned off, and the algorithm code does not override it mlrCPOs.R mlrCPOs.R. The convergence messages are not checked. A helper function nlopt is defined in each encoder but never called.
  • Cross-fitting folds for classification. The encoder folds are not stratified, although the outer CV is stratified for classification mlrCPOs.R global_params.R. The paper does not state this.
  • Gradient boosting final model. The paper says early stopping uses a 20% holdout §3.3. The code fits the final model only on the other 80% xgboost.earlystop.wrap.R xgboost.earlystop.wrap.R. The patience of 10 rounds and the 6000-round cap are not stated in the paper pipeline.R.
  • Hash encoding output. The paper describes one indicator per level §2.4. The code uses FeatureHashing's model matrix, which yields hash.size columns and can produce values above 1 mlrCPOs.R.
  • Extra encoders and learner in code. The driver registers cluster, ranger, ranger_multiv, hash_multiv and embed encoders main.R. The global learner list includes a keras learner global_params.R. The paper reports seven encoders and five learners §1.4 §3.3. The card covers only the paper's set.
  • Excluded datasets. collect_results.R drops three OpenML ids marked unfinished collect_results.R. Whether the paper's count of 24 already excludes them is not verified.
  • Supplementary material. The paper points to a Supplementary Material for pseudocode and the software list §2 §2.8. The repository includes a supplementary PDF, but it is not used here.

Pitfalls

  • Training-row leakage without cross-fitting. The glmm-noCV variant encodes each training row with a model that saw its own target. The paper reports severe overfitting for the two datasets with the most levels when target encoding lacked strong regularisation (impact and glmm-noCV) §4.2.
  • Rare levels and unseen levels. Each encoder handles unseen levels differently: frequency uses count 1, GLMM uses the global intercept, indicator maps them to a collapsed "other" column, and integer produces NA in the released code §2 mlrCPOs.R. Reimplementations that do not match this will not reproduce the ranking.
  • Cost scales with folds and classes. By code structure, glmm-5CV fits one encoding model per feature per fold, and for multiclass it fits one per class too mlrCPOs.R. Table 1 shows the effect, with a median runtime of 7.21 times one-hot for gradient boosting at 5-fold and 12.71 at 10-fold tab. 1.
  • Selection bias in reported "best HCT" numbers. Picking the best HCT per dataset on outer test means is optimistic manuscript.Rmd. Use the nested variant if you reproduce this.
  • Tuning is minimal by design. The authors note the learners were barely tuned, that GB, KNN and SVM were likely suboptimal, and that 5-fold CV was run without repetitions §5.1. The ranking is claimed for this setting, not for tuned pipelines.
  • Results rest on 24 datasets. The paper itself says its dataset count was too low to reveal stable links between encoder ranking and dataset traits §5.
  • Dummy versus one-hot. The study found one-hot slightly better than dummy coding for nonlinear learners §4.3. Check which form your library produces.
  • Unlicensed code. The repository has no license. This card cites its code by path and line and quotes none of it.

Lineage

  • Builds on: target and impact encoding with smoothing [Micci-Barreca 2001, cited in §1.2]; generalized linear mixed models for shrinkage [Gelman and Hill 2006, cited in §2.7]. Neither has a card here.
  • Compared against: CatBoost's ordered target statistics [Prokhorenkova et al. 2018, cited in §1.2]; feature hashing [Weinberger et al. 2009, cited in §2.4]; random-forest categorical splits [Wright and König 2019, cited in §1.3]. None has a card here.
  • Leaf encoding simplifies the approach of Grąbczewski and Jankowski (2003) §2.5. No card here.
  • Superseded by: not stated in the paper. The published journal version is in Computational Statistics (2022), not checked here.

Source and license

Adapted from “Regularized target encoding outperforms traditional methods in supervised machine learning with high cardinality features” by Florian Pargent, Florian Pfisterer, Janek Thomas, Bernd Bischl (arXiv:2104.00629), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/slds-lmu/paper_2021_categorical_feature_encodings at commit ee62f3fdb78e (none (no LICENSE file in the repository)). Any quoted code is shown under that license.