---
kind: concept
slug: categorical-encoding
title: Categorical encoding
summary: >-
  Turning a categorical feature with many levels into numbers. One card so far: a benchmark of seven encoders on 24 datasets with five learners. Regularized target encoding with cross-fitting ranked first for four of five learners (author-reported), from one lightly tuned study. Trade-off: it fits a mixed model per feature and costs more runtime, especially with gradient boosting.
tags: [tabular, categorical-features, encoding, high-cardinality]
papers: ["2104.00629"]
related: [gradient-boosted-decision-trees, mixed-effects-model, tabular-foundation-model]
status: draft
updated_at: 2026-11-15
---

## Overview

A categorical feature takes values from a set of labels. When the set is small,
one indicator column per label is cheap. When it has thousands of labels, such
as zip codes or product IDs, indicator columns become unwieldy, and simple
numeric codes impose an order that does not exist [arxiv:2104.00629 §1, §2.1].
The task is to pick a mapping from labels to numbers, fit it on the training
rows, and apply it to new rows. Each encoder also needs a rule for labels it
has not seen [arxiv:2104.00629 §2].

The main design split is whether the encoder uses the outcome. Encoders that
ignore the target are safe from leaking it into the features. Target-based
encoders can be more accurate, but when they are fitted and applied to the same
rows, rare labels get numbers that reflect their own outcome. Cross-fitting,
which encodes each training row with a model that did not see it, guards
against this [arxiv:2104.00629 §2.7, §4.2].

Only one card in this wiki covers this slug, so every variant below comes from
one benchmark study. No alternative is summarised yet.

## Variants

### Target-agnostic: integer, frequency, indicator and hash

- **Integer.** Each observed label gets a code from 1 to L. It is cheap, but
  the codes impose an arbitrary order, and it did poorly with gradient boosting
  [arxiv:2104.00629 §2.1, §4.3]. Card: [Pargent et al.](/papers/2104.00629).
- **Frequency.** Each label becomes its count in the training set. Unseen labels
  get a count of 1 [arxiv:2104.00629 §2.2].
- **Indicator (one-hot or dummy).** The most frequent labels get their own
  column and the rest share an "other" column. For nonlinear learners one-hot
  was usually slightly better than dummy coding [arxiv:2104.00629 §2.3, §4.3].
- **Hash.** Each label is mapped to one of a fixed number of buckets, set to the
  high-cardinality threshold in the study [arxiv:2104.00629 §2.4, §3.2].

### Tree-based: leaf encoding

A decision tree is fitted on the feature alone and pruned with 10-fold
cross-validation. The terminal node a label falls into becomes its new level,
which is then one-hot encoded [arxiv:2104.00629 §2.5]. Card: [Pargent et al.](/papers/2104.00629).

### Target-based: impact and GLMM encoding

- **Impact (smoothed target encoding).** Each label is replaced by its mean
  outcome, shrunk by a small smoothing constant of 0.0001. For classification
  the study uses class frequencies instead [arxiv:2104.00629 §2.6]. Without
  cross-fitting it overfits the datasets with the most labels [arxiv:2104.00629 §4.2].
- **GLMM encoding.** A random-intercept model is fitted per feature, and each
  label's encoded value is its conditional mode. Labels unseen at prediction
  time take the fixed intercept [arxiv:2104.00629 §2.7]. The study compares
  three settings: no cross-fitting, 5-fold and 10-fold. The 5-fold version is
  the headline variant, and 10-fold often gave no further gain [arxiv:2104.00629 §4.3].
  Card: [Pargent et al.](/papers/2104.00629).

## When to use which

Base the choice on the constraints. The numbers below are the study's own and
are author-reported on 24 OpenML datasets.

| Situation | Use | Why |
|---|---|---|
| Accuracy matters most, and you can afford a mixed-model fit per feature | GLMM with 5-fold cross-fitting | Ranked first for four of five learners, all but SVM [arxiv:2104.00629 §4.3]. |
| Gradient boosting with a tight runtime budget | Frequency encoding | Median runtime about 0.54 of one-hot with gradient boosting [arxiv:2104.00629 tab. 1]. Accuracy per dataset is only in the figures. |
| Gradient boosting with a large runtime budget | GLMM with 5-fold cross-fitting, after timing it | Its median runtime with gradient boosting was about 7.2 times one-hot [arxiv:2104.00629 tab. 1]. |
| Linear model (LASSO) | Indicator encoding as a strong baseline | It was second best for LASSO [arxiv:2104.00629 §4.3]. |
| Support vector machine | Check the figures before choosing | The overall ranking does not hold for SVM [arxiv:2104.00629 §4.3]. |
| Cannot cross-fit | Target-agnostic encoders | Without cross-fitting, target-based rows are encoded by a model that saw their own outcome [arxiv:2104.00629 §2.7, §4.2]. |

The study tuned the learners only lightly and ran 5-fold CV once, so a tuned
pipeline may rank the encoders differently [arxiv:2104.00629 §5.1]. With 24
datasets, the authors say they could not link the ranking to dataset traits,
so do not assume it transfers to your data [arxiv:2104.00629 §5].

## Implementation sketch

Target-based encoding with K-fold cross-fitting. Two steps: fit a per-feature
level model, then encode the training rows out of fold. The GLMM variant is
shown; the impact variant replaces the model with a smoothed mean
[arxiv:2104.00629 §2.6, §2.7].

```pseudocode
# Fit one level model on a set of rows  [arxiv:2104.00629 §2.7]
# Input:  levels (n,) categorical; target (n,)
# Output: value per seen level, and a fallback for unseen levels
def fit_level_values(levels, target):
    # Random intercept per level; logistic form for classification
    model = fit_random_intercept_model(target, group=levels)
    # Fallback for levels not seen in this fit  [§2.7]
    fallback = model.fixed_intercept
    values = model.conditional_mode_per_level
    return values, fallback
```

```pseudocode
# Encode training rows out of fold, K folds  [arxiv:2104.00629 §2.7]
# Input:  levels (n,); target (n,); K (1 = no cross-fitting)
def encode_train(levels, target, K):
    # Model used at prediction time, fitted on all training rows  [§2.7]
    full_values, full_fallback = fit_level_values(levels, target)
    if K == 1:
        return lookup(full_values, full_fallback, levels), full_values
    encoded = empty(n)
    # Each row is encoded by a model that never saw its fold  [§2.7]
    for fold, rest in make_folds(n, K):
        vals, fb = fit_level_values(levels[rest], target[rest])
        encoded[fold] = lookup(vals, fb, levels[fold])
    return encoded, (full_values, full_fallback)
```

At prediction time, encode new rows with the full-data model and its fallback.
Reimplementations should follow the paper's rule for unseen levels, because
the ranking depends on it [arxiv:2104.00629 §2].

## Contradictions

None known within this wiki. The authors' released code differs from the paper
in several details, such as how the high-cardinality threshold is counted and how
gradient boosting's final model is fitted. Those differences are listed on the
card's "Paper vs code" section, linked below, and may change the numbers one
gets from the code.

## Papers

- [Pargent et al.](/papers/2104.00629): compares seven encoders and control conditions on 24 datasets with five learners, and finds regularized target encoding with cross-fitting ranked first for most learners (author-reported).
