Machine Learning with a Reject Option: A survey

In plain words

A normal model always gives an answer, even when it is likely to be wrong. In many settings a wrong answer is costly, for example a medical flag or a safety decision. A model with a reject option can instead say "I will not decide this one" and pass the case to a person.

This survey maps the field of models that abstain. It separates two reasons to abstain: the case sits where the right answer is unclear, or the case looks unlike anything the model was trained on. It then covers how to score a model that abstains, the three common ways to build the abstain step, how to set its threshold, how to combine several abstain rules, where the idea is used in practice, and what it has in common with neighbouring fields such as anomaly detection and active learning.

The survey runs no experiments of its own. It is useful as a map and as a checklist of design choices, not as a recipe with tuned numbers.

TL;DR

A model with rejection returns a label or a "reject" symbol. The paper formalises two rejection types, ambiguity and novelty §2.2, and gives three architectures: a separated rejector that filters inputs before the predictor, a dependent rejector that thresholds a confidence score from the predictor, and an integrated rejector trained as one model with an extra reject output §4–§6. Use it to choose an architecture and evaluation metric; it reports no new numbers, and it names the lack of a standard benchmark as an open problem §10.2.

Problem

Standard supervised learning returns a prediction for every input, which is a problem where mistakes are costly §1. The paper frames the goal as a predictor h plus a rejector r, where the model returns h(x) only when the rejector accepts §2.1 eq. 2.

The paper states that the abstain decision is needed for two broad kinds of uncertainty. Ambiguity rejection covers inputs where the target is not determined by the features, for example overlapping classes or a poor hypothesis space §2.2.1. Novelty rejection covers inputs that are rare or absent in the training data §2.2.2. The list of four underlying causes (R1–R4) is in §2.2.

Method

The survey does not propose a method. Its contribution is a taxonomy and a set of formal definitions. The key pieces are below.

Model and metrics. Let the expected risk be R(h) = \int L(h(x), y)\, dP eq. 1 §2. A model with rejection is a pair (h, r) that returns a label or the reject symbol eq. 2 §2.1. Performance is read from a prediction-versus-rejection confusion matrix (true accept, false accept, false reject, true reject) tab. 1 §3. Coverage is the fraction of inputs the model answers §3. Accuracy on answered inputs and coverage trade off against each other §3.

Architecture 1: separated rejector. A rejector r(x) sees only the input. If r(x) < \tau, the model rejects; otherwise it calls the predictor eq. 3 §4. The rejector is a density estimate (for example a Gaussian mixture or a variational autoencoder), a one-class classifier (for example a one-class SVM), or a novelty score that is squashed into a probability §4. Typical use is novelty rejection; ambiguity rejection needs the predictor's output and so does not fit this architecture §4.

Architecture 2: dependent rejector. A confidence function c_h(x) in [0, 1] is computed from the predictor §5. The model rejects when r(x; h) < \tau eq. 4 §5. Four ways to build c_h are listed §5:

• the largest class posterior \max_k P(Y=k \mid X=x) eq. 5, used for ambiguity, with posteriors taken from softmax, sigmoid calibration, or agreement between several predictors §5;
• the largest class-conditional density \max_k p(X=x \mid Y=k) eq. 6, used for novelty, estimated with a GMM or with nearest-neighbour heuristics §5;
• sensitivity analysis: the stability of the prediction under small parameter or input perturbations, for example 1 - \mathrm{Var} over perturbed predictors, or P(h(x+\varepsilon) = h(x)) §5 (ambiguity only);
• predictor properties, such as tree-leaf configurations or a self-organising map's weights §5.

Choosing the threshold \tau (dependent rejector). Four ways are given §5:

• a target rejection rate \rho, with P_X(c_h(x) < \tau) = \rho §5;
• a rejection-rate bound \mathfrak{R}: minimise the error on accepted inputs subject to the rejection rate staying under the bound §5;
• an error bound \mathfrak{M}: minimise the rejection rate subject to the error on accepted inputs staying under the bound §5;
• an objective-based choice. Chow's rule minimises \int_{c_h \ge t}(1 - c_h)\,p\,dx + t \int_{c_h < t} p\,dx over t eq. 7 §5. The two-threshold rule of Tortorella chooses a reject band \tau_1 \le c_h \le \tau_2 eq. 8 using ROC-based cost terms eq. 9 eq. 10 §5.

Several local thresholds, one per region (per class, Voronoi cell, or class cost), are also covered §5.

Architecture 3: integrated rejector. The model outputs a label or the reject symbol: m(x) \in \mathcal{Y} \cup \{\text{reject}\} eq. 11 §6. The model-agnostic option uses a cost-based loss, with C_r the cost of rejecting and C_r \in (0, 1/2] for classification §6. The loss is replaced by a convex surrogate (logistic or hinge) for optimisation §6. An extra output class for rejection is also used §6. Model-specific options include:

• integrated SVMs with two hyperplanes, one per class, or two parallel hyperplanes around the decision boundary, so inputs between them are rejected fig. 8 §6;
• a sigmoid rejection head m_r(x) in a neural network, rejecting when m_r(x) < 0.5 §6;
• two networks, rejecting where they disagree in sign §6.

Combining rejectors. If rejection regions do not overlap, reject when any rejector fires §7. If they overlap, order the rejectors and apply them one after another, recording which one fired §7.

# Separated rejector  [§4] [eq. 3]
# Input:  example x, rejector r, threshold tau, predictor h
# Output: a label, or REJECT

def predict_separated(x, rejector, tau, predictor):
    # The rejector sees only the input, never the predictor's output  [§4]
    score = rejector(x)
    if score < tau:
        return REJECT
    return predictor(x)

# Dependent rejector  [§5] [eq. 4]
# Input:  example x, predictor h, confidence function c_h in [0, 1], tau
# Output: a label, or REJECT

def predict_dependent(x, predictor, confidence, tau):
    # Confidence is derived from the predictor's output or internals  [§5]
    if confidence(x, predictor) < tau:
        return REJECT
    return predictor(x)

# Threshold for a target rejection rate  [§5]
# Input:  confidence values on training examples, rho in [0, 1]
# Output: tau

def threshold_for_rate(confidences, rho):
    # Sort training confidences from lowest to highest  [§5]
    ranked = sort(confidences)
    # Choose tau so that a fraction rho of examples falls below it  [§5]
    # (empirical quantile; the paper states P_X(c_h < tau) = rho)
    tau = ranked[floor(rho * len(ranked))]
    return tau

Hyperparameters

The survey runs no experiments and gives no tuned settings. Values below are the constraints and defaults that the text states.

Results

Author-reported, not from experiments by this survey. The paper contains no results table. It gives the following numbers for a single illustrative application.

The caption attributes the sleep-stage figure to the cited work (VanderPlas 2021). It says the novelty rejector mainly rejects the child cases fig. 6. The paper gives no numbers for the accuracy-reject curve comparison in fig. 7 §3.2, which is described only qualitatively.

Paper vs code

No official code. The paper's code availability section says "Not applicable" (Declarations section, after §10). Searches of GitHub repository names for "learning with rejection", "reject option" and "reject option survey" found no repository belonging to the authors.

Pitfalls

• No experiments, no benchmark. The paper itself says there is no standard benchmark for models with rejection, and that most papers use custom or proprietary data §10.2. Numbers from the cited works are not comparable.
• Single metrics are misleading. Accuracy on answered inputs alone favours low coverage. The rejection quality ratio alone favours low rejection rates §3.1. Report coverage together with accuracy, or use the combined quality measure §3.1.
• Metric formulas are unnumbered. The accuracy, coverage, rejection quality and combined quality formulas are not numbered in the paper §3.1. Take them from the text, and check the denominators: the rejection quality ratio divides by the misclassification rate on all inputs, not on accepted inputs §3.1.
• Rejection cost bounds depend on normalisation. The bounds in §3.3 and §6 are stated for normalised costs. The paper's remark that C_r \le 1/2 holds for classification §6 is about the convex hinge surrogate. Check which bound applies to your loss.
• Chow's rule needs calibrated posteriors. The paper says Chow's rule is not suitable when posteriors are estimated, and that a cost-based choice of threshold should be used instead §5.
• Sign convention. The rejector rejects when its score is below \tau eq. 3 eq. 4. Novelty scores in which higher means more novel must be inverted or transformed first, for example by a sigmoid §4. The paper does not give a canonical transform.
• Density proxies are heuristic. The nearest-neighbour density estimates §5 are heuristics, not calibrated densities. The paper gives no error analysis for them.
• Separated rejectors filter, dependent rejectors need the predictor. A separated rejector cannot reject ambiguous inputs in its own architecture §4. A dependent rejector can change its rejection rate only by changing \tau, but some rejectors cannot set a rate directly and may need retraining to change coverage §3.2.
• Figure 6 is a third-party figure. The caption credits the cited work fig. 6. Do not reuse it. Describe it only, as this card does.
• Hinge surrogate formula. The piecewise surrogate in §6 is given without derivation. Check the cited surrogate-loss sources before implementing it.

Lineage

• Builds on: Chow's minimum-risk rule (1970) §5, §1; Hellman (1970) §1.
• Compared against: not applicable (survey).
• Related areas, with their own sections: uncertainty quantification §9.1, anomaly detection §9.2, active learning §9.3, class-incremental learning §9.4, delegating classifiers §9.5, meta-learning §9.6.
• Superseded by: not stated.