Bias in Large Language Models: Origin, Evaluation, and Mitigation

In plain words

Language models learn from huge amounts of human-written text, and that text carries stereotypes about gender, age, race, culture and other groups. The model can then repeat those stereotypes, for example by linking "doctor" with men and "nurse" with women, or by treating some dialects as more negative than others. The problem gets serious when such a model helps with hiring, medical or legal decisions.

This review sorts the problem into a few parts. Bias can be built into the model's internal representations, or it can show up only when the model is used for a particular task. Bias can be measured in the training data, inside the model, or in what the model writes. It can be reduced before training, during training, or after the model has produced an answer. The authors compare these options and list their costs and limits. They do not test a new method.

TL;DR

• A literature review with no new experiments. It covers about 200 papers from January 2016 to August 2024 §1.
• Bias is split into intrinsic (in the pretrained model, independent of any task) and extrinsic (in downstream task performance and outputs) §2 §3.
• Evaluation is grouped into data-level, model-level and output-level methods, with human-involved and domain-specific checks applied across them §4.
• Mitigation is grouped into pre-model (data), intra-model (training or architecture) and post-model (outputs at inference) methods §5 tab. 4.
• The cost comparison in tab. 5 is labelled by the authors as approximate. Every reader should treat it as a rough guide, not a measurement.

Problem

The authors say LLMs pick up social biases from their training corpora and can apply them in high-stakes settings such as healthcare, legal judgments and hiring §1. They argue that bias must be studied in stages of the model's life cycle, because it arises in the data, in the model and in the outputs §1 §4.

The paper does not state a single formal definition of bias. It works with the intrinsic/extrinsic split §2 §3 and with the representational and allocational harm categories used in the ethics section §6.

Method

This is a review, so the "method" is its classification scheme and its coverage rules. The steps below describe how the review is organised.

1. Literature collection. Searches covered academic search engines and digital libraries, including the ACM library and the ACL archive, plus the computational linguistics, AI and machine learning preprint categories. Query strings combined a model term ("large language model", "LLM", "language model") with a bias term ("bias", "fairness", "stereotype", "debiasing"). The window was January 2016 to August 2024. Inclusion needed a direct focus on bias origin, evaluation or mitigation, a peer-reviewed venue or substantive preprint, and English. About 200 papers survived screening §1.

2. Intrinsic bias (§2). Bias encoded in the pretrained model, independent of any task. The review names four sources: the training corpus (over- and under-representation, spatial and temporal skew) §2.1; the collection method (web, social media, academic text, proprietary and model-generated data) §2.2; the language itself (ambiguity, gendered pronouns, figurative language) §2.3; and tokenisation and pretrained word embeddings, where rare words are split into fragments and embeddings carry stereotypes §2.4.

3. Extrinsic bias (§3). Disparities in performance or behaviour on downstream tasks. The review splits tasks into natural language understanding (NLU) and natural language generation (NLG). For each task it lists gender, age and cultural or regional bias. The NLU examples are in tab. 1 and the NLG examples in tab. 2. Both tables are illustrative, not exhaustive §3.1 §3.2.

4. Bias evaluation (§4). Methods are grouped by the level they examine.

   • Data-level (§4.1, fig. 2): data distribution analysis (group representation, imbalance measures such as Gini and entropy, data source review); stereotype and bias detection in text (lexicons, contextual analysis, sentiment analysis); and annotation bias analysis (annotator guidelines, agreement statistics).
   • Model-level (§4.2, tab. 3): fairness metrics (equal opportunity, predictive parity, calibration); interpretability tools (SHAP, LIME); and counterfactual fairness, which changes a sensitive attribute in the input and checks whether the model's decision stays the same.
   • Output-level (§4.3, fig. 3): counterfactual testing of generated text, stereotype detection in generations, sentiment and toxicity analysis, and acceptance and rejection rates (for example, in simulated hiring decisions).
   • Human-involved (§4.4, fig. 4): expert or crowd review and qualitative methods. The review asks for an explicit protocol: evaluator population and diversity, guidelines and rubrics, pilot calibration, a pre-set disagreement rule, and reported agreement statistics.
   • Domain-specific checks are described as a cross-cutting mode, not a separate level §4.

5. Bias mitigation (§5, tab. 4, fig. 5). The review sorts methods by when they act.

   • Pre-model (§5.1): counterfactual data augmentation (CDA) and counterfactual data substitution (CDS) swap bias-carrying attribute words in the training corpus; for example, "he" becomes "she" in a sentence about a professor §5.1. Other pre-model options are resampling, and an iterative human-in-the-loop process in which an expert reviews dataset reports and asks an LLM for new data §5.1.
   • Intra-model (§5.2): an equalizing loss and a declustering loss added to BERT fine-tuning; movement pruning of attention heads; transfer learning and multilingual training; dropout regularisation with extra pretraining; and causal-inference methods that trace where bias is stored in a model.
   • Post-model (§5.3): reinforcement-learning calibration of generation (GPT-2 is the example); Self-Debias, which uses the model's own judgement to lower the probability of biased continuations at decoding time; projection methods SENT-DEBIAS and INLP (see pseudocode below); and causal prompting, which changes the prompt without touching weights or logits.

6. Ethics and law (§6). Harms are grouped into representational harms (stereotyping, misrepresentation) and allocational harms (disparate treatment in hiring or healthcare). The section cites US, New York City, US healthcare and EU rules that apply to such tools §6.

Projection methods, step by step

The review describes two post-model projection methods §5.3. The pseudocode below restates those descriptions. The review gives no further detail.

# SENT-DEBIAS  [§5.3]
# Input:  attribute words (e.g. he/she); general sentences
# Output: sentence representations with a bias subspace removed

def sent_debias(attribute_words, general_sentences, encoder):
    # 1. Identify words that carry a biased attribute  [§5.3]
    words = attribute_words
    # 2. Embed them in attribute sentences, generated by CDA  [§5.3]
    sentences = counterfactual_sentences(words)
    reps = [encoder(sentence) for sentence in sentences]
    # 3. Estimate the bias subspace with PCA  [§5.3]
    # Number of components: not stated  [§5.3]
    subspace = pca(reps)
    # 4. Remove each general sentence's projection onto that subspace  [§5.3]
    return [remove_projection(encoder(g), subspace) for g in general_sentences]

# INLP (iterative nullspace projection)  [§5.3]
# Input:  representations (N, d); protected-attribute labels (N,)
# Output: representations from which a linear probe can no longer read
#         the attribute

def inlp(representations, protected_labels, rounds):
    # Number of rounds: not stated  [§5.3]
    for step_index in range(rounds):
        # 1. Train a linear classifier to predict the protected attribute  [§5.3]
        classifier = train_linear(representations, protected_labels)
        # 2. Project onto the nullspace of its weight matrix  [§5.3]
        projector = nullspace_projector(classifier.weights)
        representations = representations @ projector
    return representations

Hyperparameters

This is a review. It reports no training or evaluation settings for a model it built. The values it does give are about its own literature search.

Results

The review reports no benchmark numbers of its own. Author-reported content:

The GPT-4 row is a claim the review reports from a cited study. It is not checked by the review and is not a result of this paper.

Paper vs code

No official code. The paper has no code link: its only URLs are DOIs and references. GitHub searches for the arXiv id and for the title words (bias LLM survey Guo) returned no repositories. Repositories that link a survey's reference list are not code for this paper.

Pitfalls

• No experiments. Nothing in the paper was measured. The only numbers are literature counts and the cost estimates in tab. 5, which the authors call approximate and which have no cited source in the table.
• Secondhand results. Most numbers and examples come from cited works (for example, the 16,800 prompts in §4.3 and the GPT-4 finding in §6). Check the original before you rely on them.
• Gaps in the taxonomy. The introduction §1 says the review is organised around intrinsic versus extrinsic bias, but the body also uses data-, model- and output-level evaluation and pre-, intra- and post-model mitigation. Those are separate axes. The review says so explicitly §4.
• Cross-cultural gaps. §1 lists cross-cultural and multilingual gaps as a contribution, but the body has no dedicated section on them. The examples in tab. 1, tab. 2 and the appendix are the main place where non-Western cases appear.
• Projection methods. Both SENT-DEBIAS and INLP are described only as steps. The review gives no PCA component count, no INLP stopping rule and no handling of the sentence set beyond the outline above.
• Cost table. Relative costs (low, medium, high) are the authors' judgement. Use them to compare categories, not to predict run time or GPU hours.
• Intrinsic bias and hallucination. §4.3 argues that some hallucinations can be read as bias-driven. This is an interpretation offered by the authors, not a finding of a cited study.
• Date. The literature window ends in August 2024, while the paper's arXiv record was updated in May 2026. Newer models (for example, the GPT-4 and Llama 3 remarks in the conclusion) are discussed without systematic evidence §7.
• Appendix. The examples appendix ("Examples of Extrinsic Biases") is illustrative. The source does not state an appendix letter, so cite it by its title.

Lineage

• Builds on: the prior reviews Mehrabi et al., Navigli et al. and Gallegos et al. The authors say these covered bias phenomena and mitigation, and that their own review adds the taxonomy, the cost comparison and the ethics framing §1.
• Compared against: the same three reviews §1.
• Superseded by: not known from this wiki.