wikillm

Concept · 1 paper · updated 2026-11-15

Categorical encoding

Turning a categorical feature with many levels into numbers. One card so far: a benchmark of seven encoders on 24 datasets with five learners. Regularized target encoding with cross-fitting ranked first for four of five learners (author-reported), from one lightly tuned study. Trade-off: it fits a mixed model per feature and costs more runtime, especially with gradient boosting.

tabularcategorical-featuresencodinghigh-cardinality

Markdown4 min read

AI-compiled overview, updated 2026-11-15, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

Overview

A categorical feature takes values from a set of labels. When the set is small, one indicator column per label is cheap. When it has thousands of labels, such as zip codes or product IDs, indicator columns become unwieldy, and simple numeric codes impose an order that does not exist 2104.00629 §1, §2.1. The task is to pick a mapping from labels to numbers, fit it on the training rows, and apply it to new rows. Each encoder also needs a rule for labels it has not seen 2104.00629 §2.

The main design split is whether the encoder uses the outcome. Encoders that ignore the target are safe from leaking it into the features. Target-based encoders can be more accurate, but when they are fitted and applied to the same rows, rare labels get numbers that reflect their own outcome. Cross-fitting, which encodes each training row with a model that did not see it, guards against this 2104.00629 §2.7, §4.2.

Only one card in this wiki covers this slug, so every variant below comes from one benchmark study. No alternative is summarised yet.

Variants

Target-agnostic: integer, frequency, indicator and hash

  • Integer. Each observed label gets a code from 1 to L. It is cheap, but the codes impose an arbitrary order, and it did poorly with gradient boosting 2104.00629 §2.1, §4.3. Card: Pargent et al..
  • Frequency. Each label becomes its count in the training set. Unseen labels get a count of 1 2104.00629 §2.2.
  • Indicator (one-hot or dummy). The most frequent labels get their own column and the rest share an "other" column. For nonlinear learners one-hot was usually slightly better than dummy coding 2104.00629 §2.3, §4.3.
  • Hash. Each label is mapped to one of a fixed number of buckets, set to the high-cardinality threshold in the study 2104.00629 §2.4, §3.2.

Tree-based: leaf encoding

A decision tree is fitted on the feature alone and pruned with 10-fold cross-validation. The terminal node a label falls into becomes its new level, which is then one-hot encoded 2104.00629 §2.5. Card: Pargent et al..

Target-based: impact and GLMM encoding

  • Impact (smoothed target encoding). Each label is replaced by its mean outcome, shrunk by a small smoothing constant of 0.0001. For classification the study uses class frequencies instead 2104.00629 §2.6. Without cross-fitting it overfits the datasets with the most labels 2104.00629 §4.2.
  • GLMM encoding. A random-intercept model is fitted per feature, and each label's encoded value is its conditional mode. Labels unseen at prediction time take the fixed intercept 2104.00629 §2.7. The study compares three settings: no cross-fitting, 5-fold and 10-fold. The 5-fold version is the headline variant, and 10-fold often gave no further gain 2104.00629 §4.3. Card: Pargent et al..

When to use which

Base the choice on the constraints. The numbers below are the study's own and are author-reported on 24 OpenML datasets.

SituationUseWhy
Accuracy matters most, and you can afford a mixed-model fit per featureGLMM with 5-fold cross-fittingRanked first for four of five learners, all but SVM 2104.00629 §4.3.
Gradient boosting with a tight runtime budgetFrequency encodingMedian runtime about 0.54 of one-hot with gradient boosting 2104.00629 tab. 1. Accuracy per dataset is only in the figures.
Gradient boosting with a large runtime budgetGLMM with 5-fold cross-fitting, after timing itIts median runtime with gradient boosting was about 7.2 times one-hot 2104.00629 tab. 1.
Linear model (LASSO)Indicator encoding as a strong baselineIt was second best for LASSO 2104.00629 §4.3.
Support vector machineCheck the figures before choosingThe overall ranking does not hold for SVM 2104.00629 §4.3.
Cannot cross-fitTarget-agnostic encodersWithout cross-fitting, target-based rows are encoded by a model that saw their own outcome 2104.00629 §2.7, §4.2.

The study tuned the learners only lightly and ran 5-fold CV once, so a tuned pipeline may rank the encoders differently 2104.00629 §5.1. With 24 datasets, the authors say they could not link the ranking to dataset traits, so do not assume it transfers to your data 2104.00629 §5.

Implementation sketch

Target-based encoding with K-fold cross-fitting. Two steps: fit a per-feature level model, then encode the training rows out of fold. The GLMM variant is shown; the impact variant replaces the model with a smoothed mean 2104.00629 §2.6, §2.7.

Pseudocode
# Fit one level model on a set of rows  [arxiv:2104.00629 §2.7]
# Input:  levels (n,) categorical; target (n,)
# Output: value per seen level, and a fallback for unseen levels
def fit_level_values(levels, target):
    # Random intercept per level; logistic form for classification
    model = fit_random_intercept_model(target, group=levels)
    # Fallback for levels not seen in this fit  [§2.7]
    fallback = model.fixed_intercept
    values = model.conditional_mode_per_level
    return values, fallback
Pseudocode
# Encode training rows out of fold, K folds  [arxiv:2104.00629 §2.7]
# Input:  levels (n,); target (n,); K (1 = no cross-fitting)
def encode_train(levels, target, K):
    # Model used at prediction time, fitted on all training rows  [§2.7]
    full_values, full_fallback = fit_level_values(levels, target)
    if K == 1:
        return lookup(full_values, full_fallback, levels), full_values
    encoded = empty(n)
    # Each row is encoded by a model that never saw its fold  [§2.7]
    for fold, rest in make_folds(n, K):
        vals, fb = fit_level_values(levels[rest], target[rest])
        encoded[fold] = lookup(vals, fb, levels[fold])
    return encoded, (full_values, full_fallback)

At prediction time, encode new rows with the full-data model and its fallback. Reimplementations should follow the paper's rule for unseen levels, because the ranking depends on it 2104.00629 §2.

Contradictions

None known within this wiki. The authors' released code differs from the paper in several details, such as how the high-cardinality threshold is counted and how gradient boosting's final model is fitted. Those differences are listed on the card's "Paper vs code" section, linked below, and may change the numbers one gets from the code.

Papers

  • Pargent et al.: compares seven encoders and control conditions on 24 datasets with five learners, and finds regularized target encoding with cross-fitting ranked first for most learners (author-reported).

Source and license

Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.