---
kind: concept
slug: decision-threshold-tuning
title: Decision threshold tuning
summary: >-
  Turning scores into yes or no decisions with a cut-off set on validation data, from error costs, or from clean-data tails. A tuned global cut-off raised 19-value macro-F1 from 0.282 to 0.315. In RAG poison filtering, a high AUROC hid a dead deployed cut-off. Two sources, author-reported.
tags: [threshold-calibration, multi-label-classification, class-imbalance, macro-f1, operating-point, cost-sensitive-learning]
papers: ["2601.14172", "2610.33849"]
related: [calibration, multi-label-classification, ensembles, seed-variance-evaluation, data-poisoning, benchmark-validity]
status: draft
updated_at: 2026-11-21
---

## Overview

A classifier or filter that outputs a score still has to decide which items are
"yes". The cut-off between yes and no is the decision threshold. It is a
separate choice from training, and it can be set on validation data before
anything is scored on test. Two papers show how much this one choice matters.
In 19-value human value detection, moving the cut-off moves macro-F1 more than
adding extra input features does [arxiv:2601.14172 §1.1] [§6.3]. In query-time
filters that screen retrieved documents for RAG poisoning, a filter can rank
documents well while its yes-or-no decision fails. On NQ with the question
prefix kept, one filter had an AUROC of 0.953 and still missed 0.191 of poison
at its deployed point [arxiv:2610.33849 §VI-A] [§VI-C]. Every number on this page
comes from one of these two papers and is author-reported.

## Variants

### Fixed cut-off at 0.5

The default that most training code uses. It needs no validation split. The
paper uses it as the reference point for every comparison [arxiv:2601.14172
§4.1]. With it, a multi-label DeBERTa-base model under-predicts the rare values,
and its test macro-F1 is 0.282 ± 0.001 against 0.315 ± 0.004 once tuned
[arxiv:2601.14172 tab. 6]. Card: [Yeste and Rosso](/papers/2601.14172).

The poisoning paper also compares a fixed 0.5 that ignores error costs. It is
one of three decision rules scored on the same filter [arxiv:2610.33849 tab. I].
It matches the cost-derived rule below only when the two error costs are equal
[arxiv:2610.33849 §IV-C]. Card: [Kumar and Sahaym](/papers/2610.33849).

### One global cut-off chosen on validation

A single threshold is shared by all labels. It is the value, from a grid
between 0 and 1, that maximises macro-F1 on validation. It is then frozen and
applied once to test [arxiv:2601.14172 §4.1] [§5.3]. The paper uses this rule
for every reported 19-value DeBERTa result. The gain was +0.033 for DeBERTa-base
and +0.043 for RoBERTa-base, averaged over three seeds, with paired per-seed
p-values of 0.004 and 0.033 [arxiv:2601.14172 tab. 6]. The chosen value differs
by feature set, between 0.15 and 0.30 in the same table [arxiv:2601.14172
tab. 5]. Card: [Yeste and Rosso](/papers/2601.14172).

### One cut-off per label with a precision floor

Each label gets its own threshold. The rule takes the setting with the highest
recall whose precision on validation is at least 0.40, searched on a grid from
0.10 to 0.89 in steps of 0.01 [arxiv:2601.14172 §4.1] [§5.3]. The paper applies
this rule to the presence gate, which is a single binary label, and to the LLM
baselines. For the presence gate, tuning moved test F1 by between 0.000 and
+0.011 across the five rows of its table, and every row chose the lowest grid
value [arxiv:2601.14172 tab. 3]. The paper does not compare this rule against
the global rule on the same model. Card: [Yeste and Rosso](/papers/2601.14172).

### Cost-derived cut-off on a calibrated score

Here the cut-off comes from the price of errors, not from a search. Let rho be
the cost of a missed item divided by the cost of a wrongly blocked one. If the
score is a calibrated probability, blocking at or above 1 / (1 + rho) minimises
expected harm [arxiv:2610.33849 eq. 2]. The paper credits this rule to Elkan and
says each ingredient predates its work [arxiv:2610.33849 §IV-C] [§II-C]. Its
illustrative rho values were 1, 5, 10 and 20, treated as governance inputs and
swept in an appendix [arxiv:2610.33849 §IV-C]. A per-source trust multiplier
kappa, clipped to [0.25, 20], moves the cut-off to 1 / (1 + rho × kappa)
[arxiv:2610.33849 eq. 3] [§IV-D]. Card: [Kumar and Sahaym](/papers/2610.33849).

### Cut-off from the clean-score tail

Each defense in the poisoning study sets its cut-off from clean documents
rather than from poison. Perplexity and DRS use the 99th percentile of clean
scores. RAGuard uses a 2.5% tail on each of its three tests, combined by OR, for
a nominal 10% false-positive rate. Two defenses use fixed numbers instead, 0.88
and 0.25 for TrustRAG stage 1 and 0.95 cosine for near-duplicates [arxiv:2610.33849
§II-B] (tab. I). The threshold is fitted on clean data, so its realised rate on
other clean data can differ from the nominal one. Across four corpora, RAGuard's
nominal 10% became 0.098 on average and DRS's nominal 1% became 0.103, but DRS
was 0.035 on medical [arxiv:2610.33849 §II-B]. Card: [Kumar and Sahaym](/papers/2610.33849).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Comparing two models or losses on the same data | One global cut-off tuned on validation, applied the same way to each model | Without tuning, the comparison mixes architecture with threshold choice [arxiv:2601.14172 §1.1] [§4.1] |
| Labelled validation data exists, and rare labels matter | Global cut-off tuned on validation | It raised macro-F1 for both encoders tested, by 0.033 and 0.043 [arxiv:2601.14172 tab. 6] |
| No validation split is available | 0.5, and report it as the untuned baseline | Both tuned rules in the paper pick the cut-off on validation, and it reports no tuned result without that split [arxiv:2601.14172 §4.1] |
| Each label needs a recall target under a precision floor | Per-label cut-off with precision at least 0.40 | This is the paper's rule for the presence gate and LLM baselines [arxiv:2601.14172 §4.1]. It is not tested head to head against the global rule |
| Model outputs are close to 0 or 1, such as LLM answers | Expect little gain; tune anyway, or keep 0.5 | Label-wise tuning changes little when outputs are nearly binary [arxiv:2601.14172 §4.1] |
| Misses and wrong blocks have known, unequal costs, and scores are calibrated | Cost-derived cut-off, 1 / (1 + rho) | It is the harm-minimising rule under calibration. With equal costs it reduces to 0.5 [arxiv:2610.33849 eq. 2] [§VI-E] |
| Scores may drift away from the calibration data | Check calibration at the deployment prevalence before trusting any fixed cut-off | On one shifted medical setting the learned scorer's ECE at realised prevalence was 0.471, and AUROC cannot show that kind of failure [arxiv:2610.33849 §VI-C] |
| Blocking benign items has a fixed budget | Cut-off from the clean-score tail, checked on held-out clean data | The realised false-positive rate differed from the nominal one in the paper [arxiv:2610.33849 §II-B] |

## Implementation sketch

The rules share one final step: compare scores with frozen thresholds on test.
They differ in how the thresholds are chosen. The first two use validation
data, the cost rule uses error costs, and the tail rule uses clean data.

```pseudocode
# Global cut-off: one threshold for all labels, chosen on validation  [§4.1]
# Input:  val_scores (N, L), val_gold (N, L), candidate grid in [0, 1]
# Output: one threshold t, applied unchanged to test  [§5.3]
def global_threshold(val_scores, val_gold, grid):
    best_t, best_f1 = 0.5, -1.0
    for t in grid:
        pred = val_scores >= t                  # (N, L) booleans
        f1 = macro_f1(val_gold, pred)           # mean over the L labels
        if f1 > best_f1:
            best_t, best_f1 = t, f1
    return best_t
```

```pseudocode
# Per-label cut-off: highest recall at precision >= floor  [§4.1] [§5.3]
# Input:  scores (N,) and gold (N,) for one label, on validation
# Output: one threshold for that label, or None if no grid value qualifies
def label_threshold(scores, gold, floor=0.40):
    grid = arange(0.10, 0.90, 0.01)             # 0.10 to 0.89  [§5.3]
    best_t, best_recall = None, -1.0
    for t in grid:
        pred = scores >= t
        precision, recall = precision_recall(gold, pred)
        if precision >= floor and recall > best_recall:
            best_t, best_recall = t, recall
    # The paper does not say what to do when None is returned  [§4.1]
    return best_t
```

```pseudocode
# Cost-derived cut-off on a calibrated score  [eq. 2] [eq. 3]
# Input:  score (calibrated P(positive)), rho = C_FN / C_FP, kappa (1.0 if unused)
# Output: True (block) or False (pass)
def cost_rule(score, rho, kappa=1.0):
    # kappa is clipped to [0.25, 20] before use  [§IV-D]
    threshold = 1.0 / (1.0 + rho * kappa)       # eq. 3; kappa = 1 gives eq. 2
    return score >= threshold
```

```pseudocode
# Tail cut-off from clean scores at a nominal false-positive rate  [§II-B]
# Input:  clean_cal (M,) scores of clean items on a calibration split
# Output: threshold t; confirm the realised rate on a separate clean split
def tail_threshold(clean_cal, fpr=0.01):
    # With M = 200 the 99th percentile is the second-largest value  [§VIII]
    return quantile(clean_cal, 1.0 - fpr)
```

```pseudocode
# Apply frozen cut-offs to test scores; test is not touched before this  [§5.3]
# thresholds: a scalar (global rule) or one value per label (per-label rule)
def predict(test_scores, thresholds):
    return test_scores >= thresholds
```

## Contradictions

- **Which rule produced the 19-value DeBERTa numbers.** The paper says a
  global cut-off was used [arxiv:2601.14172 §4.1] [§5.3]. A code review of the
  authors' release, recorded on the card's Paper vs code notes, found that
  their evaluation function tunes one cut-off per label with the precision rule.
  The same notes say the global rule is the one needed to reproduce the paper's
  tables [arxiv:2601.14172 §4.1] [§5.3]. Which rule a reader should use to match
  the reported numbers is therefore not settled by the sources here.
- **Gains from the two papers are not comparable.** One reports a gain from a
  tuned cut-off on 19-value multi-label scores [arxiv:2601.14172 tab. 6]. The
  other proposes no new threshold rule. It reports that existing per-document
  rules lose most of their detection when the poison's construction changes
  [arxiv:2610.33849 §VI-A] [§VI-B]. Neither paper's numbers contradict the other's.

## Pitfalls

- **Tuned and untuned numbers are not comparable.** The cut-off gain alone is
  larger than any lightweight feature in the paper. Compare models at the same
  tuning state, and say which state a published number uses
  [arxiv:2601.14172 §6.3] [§7].
- **A high ranking score can hide a dead cut-off.** When the question prefix was
  removed from the NQ poison, RAGuard's AUROC fell from 0.953 to 0.758 and its
  miss rate at the deployed point rose from 0.191 to 0.763 [arxiv:2610.33849
  §VI-A]. Report the miss rate and the false-alarm figures at the cut-off you
  actually ship, not AUROC alone. The paper says AUROC cannot tell calibration collapse
  from signal collapse [arxiv:2610.33849 §VI-C].
- **Calibration sets can be too small for a tail quantile.** Every fitted
  threshold in the poisoning study except DRS is a quantile of at most 200 clean
  documents, so its 99th percentile is the second-largest value. The calibration
  split is fixed across seeds, so seed spread does not show this variance
  [arxiv:2610.33849 §VIII].
- **Nominal and realised false-positive rates differ.** Compare miss rates only
  at the realised rate measured on held-out clean data [arxiv:2610.33849 §II-B].
- **Equal costs quietly give 0.5.** When rho = 1, the cost-derived rule equals the
  0.5 classifier. Five of the twelve cells where all three rules gave identical
  actions had rho = 1, and the paper counts none of them as evidence
  [arxiv:2610.33849 §VI-E].
- **The presence optimum sits on the grid edge.** Every presence row chose
  0.10, the lowest value in the search range [arxiv:2601.14172 tab. 3]. The
  true optimum may lie below the grid. Widen the grid before trusting the value.
- **Grid step differs between rules.** The label-wise search uses 0.01 steps.
  The global sweep script in the authors' code uses 0.05 steps by default. Use
  the same grid for every model you compare [arxiv:2601.14172 §4.1] [§5.3].
- **Thresholds do not transfer across feature sets.** The tuned global value
  was 0.15 for one model and 0.30 for another in the same study
  [arxiv:2601.14172 tab. 5]. Re-tune for each configuration instead of reusing
  one value.
- **Feature gains sit inside seed noise.** Once the threshold is tuned, the
  gains from context, LIWC-22 and topic features are small, and none passes the
  paired per-seed test over three seeds [arxiv:2601.14172 §6.3] [tab. 5].
- **Rules with no qualifying value are undefined.** If no grid value reaches
  the precision floor for a label, the paper does not say what to use
  [arxiv:2601.14172 §4.1]. Decide and document a fallback before running.

## Papers

- [Human Values in a Single Sentence](/papers/2601.14172): compares a fixed
  0.5 cut-off, a global cut-off tuned on validation, and a per-label rule with
  a precision floor on 19-value value detection, with DeBERTa-base and LLM
  baselines.
- [Ranking Is Not Deciding](/papers/2610.33849): shows that query-time RAG
  poison filters can rank well while their deployed cut-offs fail, derives a
  cost-based cut-off on calibrated scores, and reports false-positive rates
  against nominal targets.
