---
kind: concept
slug: algorithmic-fairness
title: Algorithmic fairness
summary: >-
  Makes a classifier's outcomes comparable across groups defined by a sensitive attribute. Start by choosing a group criterion, since no two common criteria can all hold together. Then pick where to intervene: repair or add data, constrain or reweight training, or adjust predictions after training. Each intervention trades some accuracy or robustness for fairness.
tags: [fairness, bias, group-fairness, data-quality, federated-learning, benchmarks, machine-translation, tabular-data, sentiment-analysis]
papers: ["2110.01167", "2112.06409", "1906.00591", "1803.07517", "2110.00857", "2110.08193", "2111.15366", "2305.08283", "1909.05088", "2004.09456", "2112.04359", "2206.15475", "2303.13379", "2306.11698", "1903.03862", "2107.10356", "2304.13917", "2012.04221", "2411.10915", "2311.14096", "1912.06883", "2103.09396", "2310.09219", "2005.00357", "2109.00725", "2110.00530"]
related: [dataset-curation, noisy-labels, challenge-set-evaluation, explanation-evaluation, federated-learning, llm-evaluation, controllable-text-generation, llm-ethics, toxicity-detection, causal-inference, personalized-federated-learning, adversarial-robustness, prompt-engineering, calibration, debiasing-word-embeddings, word-embedding-association-test, hallucination-detection, sentiment-analysis, generative-model-evaluation, benchmark-validity]
status: draft
updated_at: 2026-11-01
---

## Overview

A model is treated as fair toward a group when its outcomes do not depend on
a sensitive variable such as sex, race or age, in the sense the team has
defined. The variable is called $Z$ and the prediction $\hat{Y}$. Fairness is
a property of the whole system, so it has to be measured on the data and the
decision, not assumed from the model [arxiv:2110.01167 §2.5].

The problem is partly a data problem. Biased training data can lead to
discriminatory predictions, and the survey treats fairness as one step of the
data pipeline [arxiv:2112.06409 §5]. The same source separates three points
where a fix can go: before training (the data), during training (the loss or
constraints), and after training (the decision rule) [arxiv:2112.06409 §5.2].

### Measures

Measures come first, because every intervention is judged against one of
them. The survey states the common group criteria, with $Z$ a binary
sensitive attribute [arxiv:2112.06409 §5.1]:

- **Demographic parity**: the positive-prediction rate is equal across groups.
- **Equalized odds**: the rates of predicting positive are equal across groups
  within each true label.
- **Predictive parity**: among predicted positives, the share that is truly
  positive is equal across groups.
- **Individual fairness**: similar individuals get similar outcomes. This
  needs a task-specific distance, which the paper does not supply
  [arxiv:2112.06409 §5.1].
- **Causal fairness**: uses a causal graph and counterfactual comparisons
  [arxiv:2112.06409 §5.1].

The group criteria can be rewritten as three principles: independence (the
prediction does not depend on $Z$), separation (independence holds given the
true label) and sufficiency (the true label does not depend on $Z$ given the
prediction) [arxiv:2110.01167 §2.5] [arxiv:2112.06409 §5.1]. The survey
also says the choice of metric depends on the type of the variables, on the
loss used when a criterion cannot be met exactly, and on whether there is more
than one sensitive variable [arxiv:2110.01167 §2.5].

**Checks that can hide bias.** A fix can lower the one number it is judged by
and leave the bias in place. In word embeddings, projecting out a single
he-she direction shrinks the projection bias, but the most biased words still
cluster by gender, and an RBF-kernel SVM still predicts gender with 88.88%
accuracy after Hard-Debiased, against 98.25% before [arxiv:1903.03862 §4]. The
authors argue that the projection measures a direction, not the whole
structure [arxiv:1903.03862 §2] [arxiv:1903.03862 §5]. Card:
[Lipstick on a Pig](/papers/1903.03862). The result is for two embedding
methods on English word lists, so it does not cover other models or languages
[arxiv:1903.03862 §5].

The same gap appears in images. CNNs trained on nine medical imaging datasets
predict self-reported race, with one-vs-all ROC-AUC of 0.96 to 0.99 on chest
X-rays [arxiv:2107.10356 tab. 3]. The signal survives external validation on
other hospitals' data [arxiv:2107.10356 tab. 4]. Its strength falls under
masking, lung-only inputs and low resolution, but it stays above chance, and
the authors find no single cause [arxiv:2107.10356 §C2] [arxiv:2107.10356 §C3].
Their discussion calls for audits by race, sex and age [arxiv:2107.10356 p. 22].
For a team, this means that an audit must measure per-group results on the
decision, since the inputs may still carry the group. Card:
[Reading Race](/papers/2107.10356). The race labels are self-reported and used
as the only label, and the authors treat them as a social construct, not a
biological one [arxiv:2107.10356 p. 22].

**Impossibility.** Not all three principles can hold together in general.
The two surveys state this with different conditions, see Contradictions.
Neither gives a rule for choosing among them. The choice is a decision for the
team, and it should be written down with the reason [arxiv:2110.01167 §2.5].

**Individual and group measures.** Binns argues that the apparent conflict
between individual fairness and group measures is not a conflict of principle.
Both families can be defended from consistency and egalitarian justice. The
real disagreement lies in the assumption a team makes about where disparities
come from: unjust structure, or choices [arxiv:1912.06883 §3]
[arxiv:1912.06883 §5.3] [arxiv:1912.06883 tab. 2]. The paper is a philosophical
argument with no experiments. Its advice is to settle the purpose and the
assumption first, then pick the measure [arxiv:1912.06883 §6]. Card:
[On the Apparent Conflict Between Individual and Group Fairness](/papers/1912.06883).

**Counterfactual fairness.** A causal criterion asks that a predictor's
distribution for an individual stay the same when only the protected attribute
is changed, with the latent exogenous variables held fixed [arxiv:2206.15475 §6.1].
A path-specific version counts only the effects along causal paths the team has
marked as unfair [arxiv:2206.15475 §6.1.1]. The survey notes that the criterion
needs the latent variables and the structural equations, and gives no procedure
to fit or test them [arxiv:2206.15475 §6.1]. Card:
[Causal Machine Learning: A Survey and Open Problems](/papers/2206.15475).

For text, the causal survey of Feder et al. describes counterfactual swaps of
protected attributes as a fairness check, and says that invariance criteria are
rarely applied to text [arxiv:2109.00725 §4.2]. Swapping attribute words can
also add new spurious correlations, and keyword swaps can miss labels and
covariates [arxiv:2109.00725 §4.1.1]. Card:
[Causal Inference in Natural Language Processing](/papers/2109.00725).

**Challenge sets for generated output.** The criteria above assume a decision
with a label. For generated text, such as machine translation, one measure
uses a challenge set instead: templated sentences where the sensitive attribute
is fixed by the sentence. WinoMT joins Winogender and WinoBias into 3,888
English sentences. It reads the gender of the translated entity with
per-language rules and reports accuracy, plus the gap between pro- and
anti-stereotypical sentences [arxiv:1906.00591 §2] [arxiv:1906.00591 §3.1].
The authors report gender bias in all four commercial and both academic
systems they tested, with commercial accuracy from 28.9% to 74.1%
[arxiv:1906.00591 §3.2]. The paper calls the set a proxy, notes it may be
overfit, and uses only synthetic English [arxiv:1906.00591 §4]. The gap
Δ_S is defined as F1 in the text, but the released code subtracts accuracies,
so state which one a report uses [arxiv:1906.00591 §3.2].

**Templated sentences for sentiment.** The same template idea appears for
classifiers that output sentiment. The position paper by Poria et al. describes
the Equity Evaluation Corpus as 8,400 templated sentences, built from 7
templates, 60 names, 4 emotions and 5 words [arxiv:2005.00357 §3.9.2]. The
paper reports no experiments of its own [arxiv:2005.00357 §1]. It notes that few
de-biasing works exist, and that the context dependence of BERT makes bias harder
to measure than in static embeddings [arxiv:2005.00357 §3.9.3]. Card:
[Beneath the Tip of the Iceberg](/papers/2005.00357).

**Generated images.** Image generators are judged with their own measures. A
survey of GAN evaluation measures lists fairness and bias as one group of
measures, next to benchmarks and deepfake detection [arxiv:2103.09396 §4]. The
survey runs no experiments of its own, so its fairness coverage is a pointer to
other work, not a result [arxiv:2103.09396 §1.2]. Card:
[Pros and Cons of GAN Evaluation Measures: New Developments](/papers/2103.09396).

## Variants

Each variant below names what it changes. Papers are listed at the end of the
page.

### Pre-processing: repair, generate or acquire data

- **Repair the scores (Feldman et al.).** For each group, replace each score by
  the average, over groups, of the scores at the same percentile. The rank order
  inside each group is kept [arxiv:2112.06409 §5.2]. The survey does not say
  how quantiles are interpolated or how ties are broken [arxiv:2112.06409 §5.2].
- **Generate data.** Reweight the biased training set with a classifier that
  separates biased from unbiased examples, then train a generative model on the
  reweighted data [arxiv:2112.06409 §5.2].
- **Acquire data (Slice Tuner).** Keep a learning curve per data slice,
  estimate how much data each slice needs under a budget, and acquire it in
  rounds [arxiv:2112.06409 §5.2]. The budget value is not stated
  [arxiv:2112.06409 §5.2].

Data collection can also reduce bias at the source, through debiased sampling,
debiased annotation (choice of annotators) and recording provenance
[arxiv:2110.01167 §3.1.1].

### In-processing: constrain or reweight training

- **Covariance constraint (Zafar et al.).** The demographic-parity constraint is
  not convex, so the method uses a convex stand-in: the covariance between the
  sensitive attribute and each example's signed distance from the decision
  boundary. The survey says this does not readily extend to non-convex
  networks [arxiv:2112.06409 §5.2].
- **Adversarial debiasing.** Train the classifier against an adversary that
  tries to recover the sensitive attribute from the classifier's output.
  Training can be unstable [arxiv:2112.06409 §5.2].
- **Adaptive reweighting (FairBatch, OmniFair).** FairBatch adjusts the
  sensitive-group ratio within each batch based on intermediate fairness.
  OmniFair turns a constrained problem into an unconstrained one with a
  Lagrangian [arxiv:2112.06409 §5.2]. The survey's table lists these
  in-processing methods as tabular-only [arxiv:2112.06409 tab. 1].

### Post-processing: adjust the decision

- **Randomized group-specific rules (Hardt et al.).** For equalized odds, build
  a model that always predicts 0, one that always predicts 1, and the inverted
  model, then combine them with randomization per group. Pick the combination in
  the feasible region with the lowest expected loss [arxiv:2112.06409 §5.2]. The
  survey says this usually costs accuracy and does not quantify how much
  [arxiv:2112.06409 §5.2].

This variant needs the group membership at decision time. The survey lists it as
tabular [arxiv:2112.06409 tab. 1].

### Federated training: fair aggregation on the server

When data stays on the clients, the server cannot see per-group results, so
the group-level methods above do not transfer directly. FairFed keeps local
debiasing as a free choice per client and changes only the server's averaging
step. Each round the server obtains the global equal-opportunity gap and the
global accuracy through secure aggregation. Each client compares its own gap
with the global one, and clients whose gap is far from the mean lose weight in
the next average [arxiv:2110.00857 §4.2] [arxiv:2110.00857 §4.3]. The
mechanism needs no shared per-group statistics, apart from the group-positive
rates sent once at the start [arxiv:2110.00857 §4.3] [arxiv:2110.00857 alg. 1]. Card:
[FairFed](/papers/2110.00857).

Author-reported, with the paper's synthetic splits: on Adult at Dirichlet
alpha = 0.1, the equal-opportunity difference moves from -0.174 under FedAvg to
-0.017 with reweighting inside FairFed, and accuracy from 0.835 to 0.830
[arxiv:2110.00857 tab. 1]. At large alpha, gains over local debiasing are
small, except with FairBatch [arxiv:2110.00857 §5.2.1]. The paper
reports the best of several grid-searched learning rates on the test metric,
so the gains are upper bounds [arxiv:2110.00857 §5.1.5]. The experiments use
logistic regression with five clients, and the paper leaves other models
for future work [arxiv:2110.00857 §7].

### Federated training: personalize each device

Ditto uses "fair" in a different sense. Here the groups are the devices, and
fairness means that accuracy is even across them, measured as the standard
deviation of test accuracy over devices. The paper measures robustness as mean
accuracy on benign devices under attack [arxiv:2012.04221 §2]. Each device
keeps a personal model, pulled toward the shared global model by a penalty
weighted by λ. The global model is trained by an ordinary federated solver,
such as FedAvg, and the personal models stay on the device
[arxiv:2012.04221 §3.1] [arxiv:2012.04221 §3.2]. At λ = 0 the models are purely
local, and as λ grows the objective approaches the global one
[arxiv:2012.04221 §3.1]. Card: [Ditto](/papers/2012.04221).

Author-reported, on Fashion MNIST with clean data: Ditto reaches mean accuracy
0.943 (sd 0.06), against 0.909 (0.07) for the fair baseline TERM and 0.911 (0.08)
for the global model [arxiv:2012.04221 tab. 1]. With 50% of devices sending
corrupted labels (A1), TERM falls to 0.637 (0.13) while Ditto reaches 0.937
(0.07) [arxiv:2012.04221 tab. 1]. The paper argues that fair reweighting and
robust aggregation can work against each other, and it reports Ditto as both
robust and fair in its experiments [arxiv:2012.04221 §1] [arxiv:2012.04221 fig. 5].

Each device picks its λ from a grid on its own validation data
[arxiv:2012.04221 app. D.2]. The official code selects λ by a different rule
from the one in the appendix, so the appendix rule has to be implemented from
the paper if it is needed. See the [Ditto card](/papers/2012.04221) for the
differences.

### Clustering: proportional representation of groups

Fairness can also be stated for choosing cluster centers. The criterion
proportionally representative fairness (PRF) asks that any large enough group of
agents has about as many nearby centers as its size warrants. It needs no
protected attribute, since groups are defined by location [arxiv:2304.13917 §1].
Standard k-means, k-median and k-center can violate PRF [arxiv:2304.13917 §4]
[arxiv:2304.13917 §6]. The SEAR procedure always returns a PRF outcome in
polynomial time. Unconstrained SEAR is a 3-approximation to proportional
fairness, and the discrete version matches the best known (1+√2) guarantee
[arxiv:2304.13917 Lemma 2] [arxiv:2304.13917 Prop. 2]. Card:
[Proportionally Representative Clustering](/papers/2304.13917).

The paper is theoretical and reports no experiments, so it gives no measured
accuracy cost or runtime [arxiv:2304.13917 §1]. Its guarantees are about the
output rule, not about any classifier's decisions. Use it when the decision is
where to place centers, not a label.

### Input conditioning: give the model the group

Instead of changing the data, the loss or the decision, the group can be given
to the model as an input. In translation the speaker's gender decides some word
forms, so a model with no speaker information has to guess. Vanmassenhove et al.
add a speaker-gender tag to the English source of each sentence, then train the
same LSTM encoder-decoder with and without the tag [arxiv:1909.05088 §4.1]
[arxiv:1909.05088 §4.2]. Card: [Getting Gender Right in Neural Machine
Translation](/papers/1909.05088).

Author-reported BLEU on the general test set: the tag gives significant gains
for French (+1.44), Italian (+0.29) and Danish (+0.31), a significant loss for
Finnish (−0.47), and no starred change for the other six pairs
[arxiv:1909.05088 tab. 2]. On EN–FR test sets, the tagged system gains 1.13
BLEU on male speakers and 1.22 on female speakers. The authors report the
largest gain on female test sets [arxiv:1909.05088 §5] [arxiv:1909.05088 tab. 3].
The 0.09 BLEU difference between the two gains is small, and the significance
tests in the table compare each system with its baseline, not the two gains
with each other [arxiv:1909.05088 tab. 3].

The tag is a gender label, not a measure of the criteria in Measures. Its
labels are gender predictions from the MEP metadata, and the paper does not
measure how many are wrong [arxiv:1909.05088 §3]. Each system is trained once,
with no seeds or repeated runs reported [arxiv:1909.05088 §5]. The tag reflects
the speaker, so it applies only where the speaker's attribute is known at
translation time.

### Prompting an LLM for a decision: choose the demonstrations

When a language model makes the decision through its prompt, the in-context
examples are the lever. DecodingTrust tests fairness on the Adult income data,
converted from table rows into text. The sensitive attribute is sex or race,
and the paper reports accuracy plus two group gaps: a demographic parity gap
and an equalized-odds gap [arxiv:2306.11698 §10]. The paper varies the base
rate of the demonstrations and of the test set, and reports that accuracy and
unfairness both move with the demonstrations' base rate. Sixteen balanced
demonstrations already shift predictions toward fairness, at some cost in
accuracy [arxiv:2306.11698 §10]. Card:
[DecodingTrust](/papers/2306.11698).

The fairness run uses a small test set of 200 samples, and its token limit caps
the few-shot count at 32 [arxiv:2306.11698 §10] [arxiv:2306.11698 tab. 34]. The
paper's fairness results are one slice of a wider trust benchmark, so the
numbers are not a general ranking of models.

### Prompting an LLM as a member of a group: cultural prompting

When the question is whether a model's answers match a population, the
prompt can name the population. The respondent is described as an average
person born and living in a given country, and the model answers survey items
in that role. No training data or weights change [arxiv:2311.14096 §4.3]. Card:
[Cultural Bias and Cultural Alignment of Large Language Models](/papers/2311.14096).

The paper places five GPT models on a survey-based cultural map, for 107
countries and territories [arxiv:2311.14096 §4.1] [arxiv:2311.14096 §2]. With
the default respondent description, all five models sit in the same region,
near English-speaking and Protestant European countries [arxiv:2311.14096 §2]
[fig. 1]. Cultural prompting lowers the mean distance to the target country for
every model, from 2.42 to 1.57 for GPT-4o [arxiv:2311.14096 §2]. Author-reported,
the share of countries that move closer ranges from 71.0% to 81.3%
[arxiv:2311.14096 §2].

The mean hides losses. With GPT-4o, Finland's distance grows from 0.20 to 2.43,
Switzerland's from 0.45 to 2.48 and Taiwan's from 2.40 to 3.94 [arxiv:2311.14096
§2]. The authors put the share of countries that did not improve at 19 to 29%
[arxiv:2311.14096 §3]. Measure per country, and do not rely on the average.
The results rest on English prompts, ten wordings and one run per prompt at
temperature 0 [arxiv:2311.14096 §3] [arxiv:2311.14096 §4.2]. The paper does not
say whether the answers predict behaviour in real conversations, and it did not
test longer generation tasks [arxiv:2311.14096 §3].

### Generated documents: word choice, tone and invented detail

When a model writes a document about a person, bias can appear in three
places: word choice, tone, and details the model adds beyond its input. Wan et
al. test ChatGPT and Alpaca on recommendation letters, in a setting with only a
name, age and occupation, and in a setting with a biography
[arxiv:2310.09219 §3.1]. Word choice is scored per category as the odds of the
category in male letters against female letters, and in the biography setting
with WEAT [arxiv:2310.09219 §4.2.2] [arxiv:2310.09219 §4.3.5]. Tone is scored
per letter as the share of sentences that are formal, positive or agentic, and
the two genders are compared with a t-test [arxiv:2310.09219 §3.2.2]
[arxiv:2310.09219 tab. 4]. Invented sentences are flagged with an NLI model that
checks each sentence against the biography [arxiv:2310.09219 §3.3].

Author-reported. In the name-only setting, with 120 ChatGPT letters, the odds
ratio is 1.25 for masculine words and 0.85 for feminine words
[arxiv:2310.09219 tab. 2]. In the biography setting, ChatGPT letters show
male-higher gaps in positivity (t = 5.93) and agency (t = 10.47). Formality is
not significant at 0.05 (t = 1.48, p = 0.07) [arxiv:2310.09219 tab. 4]. For
Alpaca, formality (t = 3.04) and agency (t = 8.42) are significant, and
positivity is not (p = 0.07) [arxiv:2310.09219 tab. 4]. For both chatbots the
paper reports significant one-sided tests on the flagged sentences, which it
calls propagation and amplification of the bias [arxiv:2310.09219 tab. 6]
[arxiv:2310.09219 §3.3].

Three things limit these numbers. The ChatGPT model is an unpinned alias
[arxiv:2310.09219 app. A]. The agency classifier is trained and tested on
ChatGPT rephrasings, so its test accuracy of about 90% does not show performance
on real letters [arxiv:2310.09219 app. F.2]. The lexicon uses substring
matching in the released code, which changes the counts
[code: biases_string_matching.py#L47-L49]. Card:
[Gender bias in LLM reference letters](/papers/2310.09219).

### Tabular benchmarks: check the gaps against a constant classifier

Fairness studies on tabular data reuse a small set of public datasets. The
survey by Le Quy et al. covers 15 of them, from banking, criminal justice,
health, employment and education, with size, class ratio and protected
attributes for each [arxiv:2110.00530 tab. 1]. A dataset was included only if
it was public, in a known repository, and used in at least three
fairness-related papers [arxiv:2110.00530 §2.1]. Card:
[A survey on datasets for fairness-aware machine learning](/papers/2110.00530).

The authors train one logistic regression per dataset on a single 70/30 split,
and report accuracy, balanced accuracy and the fairness gaps
[arxiv:2110.00530 §4.1] [arxiv:2110.00530 tab. 15]. Ten of the 15 datasets
have an absolute statistical parity below 0.1 in that table
[arxiv:2110.00530 tab. 15]. In three rows the classifier predicts one class for
everyone: Credit card clients and Diabetes predict only the negative class, and
OULAD only the positive class. The gaps are 0.0 in those rows and balanced
accuracy is 0.5, so the zeros come from a constant classifier and not from a
fair one [arxiv:2110.00530 tab. 15] [arxiv:2110.00530 §4.2].

The numbers rest on one split with one seed and report no variance
[arxiv:2110.00530 §4.1]. The paper also notes that most datasets are old, and
the oldest is 48 years old [arxiv:2110.00530 §6]. Its data are a baseline for
logistic regression only, not a ranking of fairness methods
[arxiv:2110.00530 §4.2].

### Benchmarks for stereotyped outputs

Bias in a pretrained model can be measured with a benchmark of stereotype
pairs rather than a group gap. StereoSet has 16,995 context association tests
in gender, profession, race and religion, over 321 target terms
[arxiv:2004.09456 tab. 1]. Each item offers a stereotype, an anti-stereotype
and an unrelated continuation. The language-modeling score lms is the share
of items where the model prefers a stereotype or anti-stereotype over the
unrelated option. The stereotype score ss is the share where it prefers the
stereotype over the anti-stereotype, with 50 as ideal. ICAT combines the two,
with 100 as ideal and 0 for a fully biased model [arxiv:2004.09456 §6.2].
The test split is hidden, and only 25% of target terms are public
[arxiv:2004.09456 §6.1]. On the test split, the authors report GPT-2 (small)
with the highest ICAT among the pretrained models they score, 73.0. Every
model's ss is above 50, from 50.5 (RoBERTa-base) to 62.5 (the ensemble)
[arxiv:2004.09456 tab. 4]. The paper says lms and ss are strongly correlated
across models, so ICAT should be read with both sub-scores [arxiv:2004.09456 §8].
The code counts a tie between stereotype and anti-stereotype as an
anti-stereotype pick, and the paper does not define ties [code:
code/evaluation.py#L90-L94]. The masked-LM score is also computed differently
from the paper's text [arxiv:2004.09456 §7.1]. Card:
[StereoSet](/papers/2004.09456).

Bias in generated answers can also be measured with a hand-built test set. BBQ pairs an ambiguous context, where "unknown" is the
only correct answer, with a disambiguated context in which the answer is
stated [arxiv:2110.08193 §3]. Its bias score is twice the share of non-unknown
answers that follow the stereotype, minus one, and it is scaled by one minus
accuracy in ambiguous contexts [arxiv:2110.08193 §5]. The set has 58,492
examples across nine social-bias categories and two intersectional ones, in US
English only [arxiv:2110.08193 §3] [arxiv:2110.08193 tab. 3]. The best reported model, a
UnifiedQA-11B run in its RACE format, reaches 77.8% accuracy
[arxiv:2110.08193 §6]. The authors warn that a low bias score does not show a
model is unbiased, because each category rests on only 25 templates
[arxiv:2110.08193 §7] [arxiv:2110.08193 §9]. Card: [BBQ](/papers/2110.08193).

Where the bias enters matters for the fix. A study of political lean shows
that continued pretraining on partisan news or social-media text moves a
language model's leaning, mostly by small amounts, and that these changes
alter downstream hate-speech and misinformation classifiers by target group
and by news source [arxiv:2305.08283 §4.1] [arxiv:2305.08283 §4.2]. Overall
scores differ only slightly, so the group breakdown is the informative part
[arxiv:2305.08283 §4.2]. The partisan checkpoints are not released, so the
result can be reproduced only with a new pretraining run [arxiv:2305.08283 Ethics Statement]. Card:
[Political biases in pretraining to downstream tasks](/papers/2305.08283).

### Language models: map the bias to a stage

A review of bias in large language models splits the problem along two axes.
Intrinsic bias sits in the pretrained model and does not depend on a task.
Extrinsic bias shows up in downstream task results and outputs
[arxiv:2411.10915 §2] [arxiv:2411.10915 §3]. The intrinsic sources it names are
the training corpus, the collection method, the language itself, and
tokenisation with pretrained embeddings [arxiv:2411.10915 §2.1]
[arxiv:2411.10915 §2.4].

Evaluation is grouped by what it looks at: the data, the model, the output,
or a human reviewer. Data-level checks cover group representation, stereotype
detection in text and annotator agreement. Model-level checks cover fairness
metrics, interpretability tools and counterfactual fairness. Output-level
checks cover counterfactual tests on generated text, toxicity and sentiment,
and acceptance rates. Human review needs a written protocol with a rubric and
an agreement statistic [arxiv:2411.10915 §4.1] [arxiv:2411.10915 §4.2]
[arxiv:2411.10915 §4.3] [arxiv:2411.10915 §4.4].

Mitigation is grouped by when it acts [arxiv:2411.10915 §5]:

- **Before training.** Counterfactual data augmentation or substitution swaps
  attribute words in the corpus, for example "he" for "she" in a sentence about
  a professor [arxiv:2411.10915 §5.1].
- **During training.** An equalizing loss and a declustering loss added to BERT
  fine-tuning, pruning of attention heads, and dropout with extra pretraining
  [arxiv:2411.10915 §5.2].
- **After training.** Decoding-time methods such as Self-Debias, the projection
  methods SENT-DEBIAS and INLP, and causal prompting, which changes the prompt
  without touching weights or logits [arxiv:2411.10915 §5.3].

The review's cost table rates the pre-training option as low and the in-training
equalizing loss as high, and it rates Self-Debias as low because it needs only
inference. The authors call these estimates approximate and give no source
for them [arxiv:2411.10915 tab. 5]. The review runs no experiments of its own,
so its evidence comes from the works it cites [arxiv:2411.10915 §1]. Card:
[Bias in Large Language Models: Origin, Evaluation, and Mitigation](/papers/2411.10915).

### Fairness together with noisy data and robustness

Group membership and labels are often noisy, and a fix for one can hurt the
other [arxiv:2112.06409 §5.3]:

- Fairness methods that tolerate noisy or missing group membership, such as
  distributionally robust optimization and adversarially reweighted learning
  [arxiv:2112.06409 §5.3].
- FR-Train, which trains a classifier against a fairness discriminator and a
  robustness discriminator [arxiv:2112.06409 §5.3].
- Adaptive sample selection, which trains only on samples that are both clean
  and fair [arxiv:2112.06409 §5.3].

Removing noisy samples can worsen bias, because it can remove too many examples
from an underrepresented group [arxiv:2112.06409 §5.3]. See also
[noisy labels](/concepts/noisy-labels) for the label-cleaning side.

### Organizational measures

Fairness is also a lifecycle task. The survey covers fairness-related
documentation in datasheet and factsheet style [arxiv:2110.01167 §3.5.1],
auditing [arxiv:2110.01167 §3.5.2], and accountability for the design and
operation of the system [arxiv:2110.01167 §2.7].

A risk taxonomy comes before any metric. Weidinger et al. list 21 ethical and
social risks of language models in six areas. The first area covers social
stereotypes and unfair discrimination, exclusionary norms, toxic language, and
lower performance for some languages and social groups. For each risk the paper
gives the mechanism, the harm types and where the harm starts [arxiv:2112.04359
§2.1] [arxiv:2112.04359 tab. 1]. It reports no experiments and offers no method;
it is a checklist for risk assessment [arxiv:2112.04359 §1]. Card:
[Ethical and social risks of harm from Language Models](/papers/2112.04359).

Evidence that these checks are rarely done comes from a narrower field. A
scoping review of 118 studies that use language models in education finds that
privacy and equality are rarely addressed [arxiv:2303.13379 §4.2] [arxiv:2303.13379 §4.3].
Its results table reports no group-level metric, and the review runs no
experiments of its own [arxiv:2303.13379 §3.1] [arxiv:2303.13379 §4]. Card:
[Practical and Ethical Challenges of Large Language Models in Education](/papers/2303.13379).

**Explanations are not a fairness fix.** An explanation can show which learned
features a decision relied on, and that helps a separate bias analysis. The
explanation survey states that explanations alone do not solve bias or unfair
outcomes, and that other measures, such as regulation, are needed
[arxiv:1803.07517 §6] [arxiv:1803.07517 §7]. It is a book chapter with no
experiments, so use it for framing [arxiv:1803.07517 §7].

## When to use which

| Situation | Use | Why |
|---|---|---|
| You can change the training data and can collect more | Pre-processing: repair, generate or acquire data | Works without changing the learner. Repair keeps the rank order within each group [arxiv:2112.06409 §5.2]. Acquisition needs a budget [arxiv:2112.06409 §5.2]. |
| You control the training loop and the data is tabular | In-processing: constraints or adaptive reweighting | The survey lists these methods as tabular [arxiv:2112.06409 tab. 1]. |
| You train a deep network | Reweighting or adversarial training, with care | The convex covariance constraint does not readily extend to non-convex networks, and adversarial training can be unstable [arxiv:2112.06409 §5.2]. |
| You only have a trained model and group membership at decision time | Post-processing | Needs no retraining. Usually costs accuracy [arxiv:2112.06409 §5.2]. |
| Group membership or labels are noisy or missing | Methods that tolerate noisy groups, such as DRO or adversarially reweighted learning | Designed for this case [arxiv:2112.06409 §5.3]. |
| Fairness and robustness both matter | Check each fix against the other | The survey reports that fixes can conflict [arxiv:2112.06409 §5.3]. |
| The criterion is not yet chosen | Choose and record it first | No rule in the sources selects one criterion for a case [arxiv:2110.01167 §2.5]. |
| You generate text, such as translations, and need to measure bias | A challenge set with a per-group gap, such as WinoMT | Measures bias on templated sentences only; the set is synthetic English and a proxy [arxiv:1906.00591 §4]. |
| You can write a causal graph that shows how the sensitive attribute affects the outcome | Counterfactual fairness, if the latent variables can be modelled | The criterion needs the full causal model, and the survey gives no fitting procedure [arxiv:2206.15475 §6.1]. |
| You need to measure stereotype preference in a pretrained English language model | StereoSet, reporting lms, ss and ICAT together | Scores preference without fine-tuning, but the test split is hidden and tie handling differs between paper and code [arxiv:2004.09456 §6.1] [code: code/evaluation.py#L90-L94]. |
| You are planning a deployment and need the harms to assess | A risk taxonomy such as Weidinger et al. | Lists 21 risks with mechanisms and harm types, but has no method or results [arxiv:2112.04359 tab. 1] [arxiv:2112.04359 §1]. |
| A user or auditor asks why a decision was made | Explanations, plus a separate bias analysis | Explanations alone do not solve bias [arxiv:1803.07517 §7]. |
| Several clients train one model and cannot share data or per-group results | Fair aggregation on the server, such as FairFed, with local debiasing per client | Uses only the global gap, computed through secure aggregation [arxiv:2110.00857 §4.2] [arxiv:2110.00857 §4.3]. Gains are largest under strong heterogeneity [arxiv:2110.00857 §5.2.1]. |
| Devices hold different data, you need even accuracy across devices, and some devices may send corrupted updates | A personal model per device, pulled toward the global model, such as Ditto | Keeps the global solver unchanged. Adds per-device state that must be kept between rounds [arxiv:2012.04221 §3.2]. Fairness here is accuracy spread across devices, not a group gap [arxiv:2012.04221 §2]. |
| A generation task where a known group attribute changes the correct output, such as speaker gender in translation | Tag the input with the attribute and measure the result per group | Gains were small and mixed by language pair, and the labels are predictions [arxiv:1909.05088 tab. 2] [arxiv:1909.05088 §3]. |
| You debias word embeddings and want to know whether the bias is gone | Test beyond the projection: clustering, neighbour counts and a classifier | One projection can shrink while the structure remains [arxiv:1903.03862 §4] [arxiv:1903.03862 §5]. |
| You audit a medical imaging model for race-related results | Per-group results on the decision, with the group signal assumed present | CNNs recover self-reported race from images, including on external data [arxiv:2107.10356 tab. 4]. The mechanism is not identified [arxiv:2107.10356 p. 22]. |
| You need proportional representation when choosing cluster centers | SEAR, for PRF with a polynomial-time guarantee | Theory only; no experiments, so no measured cost [arxiv:2304.13917 §1] [arxiv:2304.13917 §4]. |
| You need to check a question-answering model for stereotyped answers | A hand-built benchmark such as BBQ, reported per category and context | Separates guessing under missing information from stereotypes that override a clear answer [arxiv:2110.08193 §1] [arxiv:2110.08193 §3]. Covers US English only [arxiv:2110.08193 §7]. |
| You want to know whether a pretrained model's leaning moves downstream results | Group-level breakdown after partisan pretraining | Overall scores may hide group differences [arxiv:2305.08283 §4.2]. The checkpoints are not released [arxiv:2305.08283 Ethics Statement]. |
| You prompt an LLM for a decision and can choose its in-context examples | Balance the base rate of the demonstrations, and report both group gaps | Sixteen balanced demonstrations shifted predictions toward fairness, at some accuracy cost; the test set has 200 samples [arxiv:2306.11698 §10] [arxiv:2306.11698 tab. 34]. |
| You need a model's answers to match a named country's survey values and you have no data to fine-tune on | Cultural prompting, with a per-country check | Lowered the mean distance for all five GPT models, but 19 to 29% of countries did not improve [arxiv:2311.14096 §2] [arxiv:2311.14096 §3]. Only tested in English [arxiv:2311.14096 §3]. |
| You report a single fairness or benchmark score as "general" | Disaggregated results by subgroup, plus test suites | Aggregate scores can hide poor subgroup performance [arxiv:2111.15366 §4.3.3] [arxiv:2111.15366 app. A]. |
| You need to choose between individual and group measures, and the team disagrees | Write down the assumption about where disparities come from, then pick the measure that fits it | Binns locates the conflict in such assumptions, not in the principles [arxiv:1912.06883 §5.3] [arxiv:1912.06883 §6]. |
| You audit a model that writes documents about people, such as letters or biographies | Score word choice, tone and invented details per gender, in a name-only and a biography setting | Gaps appeared in both settings for ChatGPT and Alpaca [arxiv:2310.09219 tab. 2] [arxiv:2310.09219 tab. 4]. The agency classifier is trained on synthetic data [arxiv:2310.09219 app. F.2]. |
| You debias a language model and must choose where to intervene | Pick the stage from your constraints: augmentation before training, a loss or pruning during training, decoding or projection after training | Cost follows the stage in the review's estimates, which are approximate [arxiv:2411.10915 tab. 5]. Measure the output after any change [arxiv:2411.10915 §4.3]. |
| You report fairness gaps on a tabular dataset | Report accuracy and balanced accuracy beside the gaps, and check the predictions are not constant | Zero gaps appear when the classifier predicts one class [arxiv:2110.00530 tab. 15] [arxiv:2110.00530 §4.2]. |
| You need public tabular datasets for a fairness study | Start from the survey's 15 datasets, matched on domain, class ratio and protected attribute | The survey lists size, class ratio and attributes per dataset, and includes only datasets used in at least three fairness papers [arxiv:2110.00530 tab. 1] [arxiv:2110.00530 §2.1]. |
| You test a text classifier for dependence on a protected attribute and can edit the text | Counterfactual swaps of the attribute, checked by hand where possible | Swaps can miss labels and add new correlations [arxiv:2109.00725 §4.1.1]. Invariance criteria are rarely applied to text [arxiv:2109.00725 §4.2]. |
| You need a templated test set for sentiment or emotion bias | An Equity Evaluation Corpus-style set, reported per template group | Templated sentences isolate the attribute, but the corpus is synthetic English and the paper runs no experiments [arxiv:2005.00357 §3.9.2] [arxiv:2005.00357 §1]. |

## Implementation sketch

Every variant needs the same audit first: a per-group gap for the chosen
criterion. The sketch below is a plain version of that check. The gap is our
own simplification, and the survey does not define one.

```pseudocode
# Per-group rates for a binary classifier  [arxiv:2112.06409 §5.1]
# Input: y (true labels), y_hat (predictions), z (group of each row)
# Output: gap per criterion; 0 means equal across groups

def fairness_gaps(y, y_hat, z):
    groups = unique(z)
    # Positive-prediction rate per group: demographic parity  [§5.1]
    rate = {g: mean(y_hat[z == g]) for g in groups}
    # True-positive rate per group: the separation criterion  [arxiv:2110.01167 §2.5]
    tpr = {g: mean(y_hat[(z == g) & (y == 1)]) for g in groups}
    # Precision among predicted positives: predictive parity  [§5.1]
    ppv = {g: mean(y[(z == g) & (y_hat == 1)]) for g in groups}
    return {
        "demographic_parity": spread(rate),
        "true_positive_rate": spread(tpr),
        "predictive_parity": spread(ppv),
    }

def spread(values):
    # Largest minus smallest group value  [own simplification]
    return max(values.values()) - min(values.values())
```

Report the gaps next to the accuracy of the same model. A small gap bought
with a large accuracy loss is a decision the team should record, not a
default.

## Contradictions

- **Scope of the impossibility result.** The lifecycle survey states that
  independence and separation are incompatible when the sensitive variable is
  linked to the true outcome [arxiv:2110.01167 §2.5]. The data-centric survey
  states that no two of independence, separation and sufficiency can be fully
  satisfied together, with no such condition [arxiv:2112.06409 §5.1]. Both
  statements are cited as given. Check the condition in the source before
  relying on either.
- **Fairness and noisy data.** The survey reports that removing noisy samples
  can make bias worse [arxiv:2112.06409 §5.3], and it also proposes adaptive
  sample selection that uses only clean and fair samples
  [arxiv:2112.06409 §5.3]. These are two different methods, not a settled
  answer. The paper does not compare them.

- **Within one paper (gender tag).** The text of Vanmassenhove et al. calls the
  Portuguese and Greek gains significant, but Table 2 does not star them and
  stars only French, Italian and Danish as gains [arxiv:1909.05088 §5]
  [arxiv:1909.05088 tab. 2]. The table is cited above. Their significance
  threshold also differs between Table 2 and Table 3 captions
  [arxiv:1909.05088 tab. 2] [arxiv:1909.05088 tab. 3].
- **Within one paper.** FairFed's text reports percentage gains that the
  paper's own table does not reproduce; the table values are the ones to cite
  [arxiv:2110.00857 §5.2.1] [arxiv:2110.00857 tab. 1].
- **Within one paper (Ditto).** Ditto's clean FEMNIST accuracy is 0.834 (0.09)
  in Table 1 and 0.836 (0.10) in Table 2, and its A1 50% value differs in the
  same way. The λ used for Table 2 is not stated, so the gap cannot be traced
  [arxiv:2012.04221 tab. 1] [arxiv:2012.04221 tab. 2].

- **Within one paper (tabular survey).** The text of Le Quy et al. says Bank
  marketing has the worst statistical parity, but Table 15 gives 0.0153 for it,
  and the largest absolute values are for Dutch census and COMPAS
  [arxiv:2110.00530 §4.2] [arxiv:2110.00530 tab. 15]. The table values are the
  ones to cite.

- **Group and individual measures.** Binns argues that their apparent conflict
  is not one of principle, and that real conflicts come from paired variants
  that rest on opposite assumptions [arxiv:1912.06883 §3]
  [arxiv:1912.06883 §5.3]. The surveys above state that some group criteria
  cannot all hold together [arxiv:2110.01167 §2.5] [arxiv:2112.06409 §5.1]. The
  page reports both and does not reconcile them.

None of the sources compares the mitigation methods numerically. The WinoMT
numbers measure bias in translation systems; they do not rank the methods
above [arxiv:1906.00591 §3.2].

## Papers

- [Trustworthy AI: From Principles to Practices](/papers/2110.01167): survey that places fairness, with the three group principles, in the AI lifecycle.
- [StereoSet: Measuring stereotypical bias in pretrained language models](/papers/2004.09456): crowdsourced context association tests in four domains, scored as lms, ss and ICAT.
- [Ethical and social risks of harm from Language Models](/papers/2112.04359): taxonomy of 21 risks in six areas, with mechanisms and origins; no method or experiments.
- [Causal Machine Learning: A Survey and Open Problems](/papers/2206.15475): survey that defines counterfactual and path-specific fairness on a structural causal model.
- [Data-centric AI survey of data collection and quality](/papers/2112.06409): survey with fairness measures and pre-, in- and post-processing methods.
- [Evaluating Gender Bias in Machine Translation](/papers/1906.00591): WinoMT challenge set and an automatic gender-bias metric for translation output.
- [Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges](/papers/1803.07517): book-chapter survey of explanation needs; explanations do not solve bias on their own.
- [FairFed: Enabling Group Fairness in Federated Learning](/papers/2110.00857): server-side aggregation that reweights clients by how close their local gap is to the global gap.
- [Ditto: Fair and Robust Federated Learning Through Personalization](/papers/2012.04221): one personal model per device, pulled toward the global model; reports even accuracy across devices and robustness to attacks.
- [Getting Gender Right in Neural Machine Translation](/papers/1909.05088): speaker-gender tag on the English source; small, mixed BLEU changes by language pair.
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](/papers/2110.08193): QA benchmark with ambiguous and disambiguated contexts and a bias score.
- [AI and the Everything in the Whole Wide World Benchmark](/papers/2111.15366): position paper on construct validity; argues for scoped benchmarks and disaggregated analysis.
- [From Pretraining Data to Language Models to Downstream Tasks](/papers/2305.08283): traces political lean from pretraining text to group-level differences in downstream classifiers.
- [Practical and Ethical Challenges of Large Language Models in Education](/papers/2303.13379): scoping review of 118 education studies; equality and privacy are rarely addressed, with no group-level metric.
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](/papers/2306.11698): trust benchmark with a fairness test on Adult; demonstration base rate shifts predictions.
- [Lipstick on a Pig](/papers/1903.03862): shows that two embedding debiasing methods hide gender bias on the he-she projection while the structure remains.
- [Reading Race: AI Recognises Patient's Racial Identity In Medical Images](/papers/2107.10356): CNNs predict self-reported race from medical images, across hospitals; the mechanism is not found.
- [Proportionally Representative Clustering](/papers/2304.13917): defines PRF for centroid selection and gives SEAR, a polynomial-time rule; theory only.
- [Bias in Large Language Models: Origin, Evaluation, and Mitigation](/papers/2411.10915): review that splits LLM bias into intrinsic and extrinsic and groups evaluation and mitigation by stage; no experiments of its own.
- [Cultural Bias and Cultural Alignment of Large Language Models](/papers/2311.14096): places five GPT models on a survey-based cultural map; cultural prompting moves most, not all, countries closer.
- [On the Apparent Conflict Between Individual and Group Fairness](/papers/1912.06883): philosophical argument that individual and group measures conflict only through unstated assumptions; no experiments.
- [Pros and Cons of GAN Evaluation Measures: New Developments](/papers/2103.09396): survey of GAN evaluation measures, with fairness and bias as one group; no new experiments.
- [Causal Inference in Natural Language Processing](/papers/2109.00725): survey that covers counterfactual swaps of protected attributes and the limits of counterfactual augmentation for text.
- [Beneath the Tip of the Iceberg](/papers/2005.00357): position paper on open sentiment-analysis problems, including bias and a templated test corpus; no experiments.
- [A survey on datasets for fairness-aware machine learning](/papers/2110.00530): 15 tabular datasets with group rates and fairness gaps for logistic regression; flags zero gaps from constant predictions.
- [Gender bias in LLM reference letters](/papers/2310.09219), with the paper's own title in quotation marks: "Kelly is a Warm Person, Joseph is a Role Model". The paper tests gender gaps in ChatGPT and Alpaca letters in word choice, tone and invented detail.