Concept · 2 papers · updated 2026-11-21
Decision threshold tuning
Turning scores into yes or no decisions with a cut-off set on validation data, from error costs, or from clean-data tails. A tuned global cut-off raised 19-value macro-F1 from 0.282 to 0.315. In RAG poison filtering, a high AUROC hid a dead deployed cut-off. Two sources, author-reported.
threshold-calibrationmulti-label-classificationclass-imbalancemacro-f1operating-pointcost-sensitive-learning
Markdown9 min read
AI-compiled overview, updated 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
Overview
A classifier or filter that outputs a score still has to decide which items are "yes". The cut-off between yes and no is the decision threshold. It is a separate choice from training, and it can be set on validation data before anything is scored on test. Two papers show how much this one choice matters. In 19-value human value detection, moving the cut-off moves macro-F1 more than adding extra input features does 2601.14172 §1.1 §6.3. In query-time filters that screen retrieved documents for RAG poisoning, a filter can rank documents well while its yes-or-no decision fails. On NQ with the question prefix kept, one filter had an AUROC of 0.953 and still missed 0.191 of poison at its deployed point 2610.33849 §VI-A §VI-C. Every number on this page comes from one of these two papers and is author-reported.
Variants
Fixed cut-off at 0.5
The default that most training code uses. It needs no validation split. The paper uses it as the reference point for every comparison [arxiv:2601.14172 §4.1]. With it, a multi-label DeBERTa-base model under-predicts the rare values, and its test macro-F1 is 0.282 ± 0.001 against 0.315 ± 0.004 once tuned 2601.14172 tab. 6. Card: Yeste and Rosso.
The poisoning paper also compares a fixed 0.5 that ignores error costs. It is one of three decision rules scored on the same filter 2610.33849 tab. I. It matches the cost-derived rule below only when the two error costs are equal 2610.33849 §IV-C. Card: Kumar and Sahaym.
One global cut-off chosen on validation
A single threshold is shared by all labels. It is the value, from a grid between 0 and 1, that maximises macro-F1 on validation. It is then frozen and applied once to test 2601.14172 §4.1 §5.3. The paper uses this rule for every reported 19-value DeBERTa result. The gain was +0.033 for DeBERTa-base and +0.043 for RoBERTa-base, averaged over three seeds, with paired per-seed p-values of 0.004 and 0.033 2601.14172 tab. 6. The chosen value differs by feature set, between 0.15 and 0.30 in the same table [arxiv:2601.14172 tab. 5]. Card: Yeste and Rosso.
One cut-off per label with a precision floor
Each label gets its own threshold. The rule takes the setting with the highest recall whose precision on validation is at least 0.40, searched on a grid from 0.10 to 0.89 in steps of 0.01 2601.14172 §4.1 §5.3. The paper applies this rule to the presence gate, which is a single binary label, and to the LLM baselines. For the presence gate, tuning moved test F1 by between 0.000 and +0.011 across the five rows of its table, and every row chose the lowest grid value 2601.14172 tab. 3. The paper does not compare this rule against the global rule on the same model. Card: Yeste and Rosso.
Cost-derived cut-off on a calibrated score
Here the cut-off comes from the price of errors, not from a search. Let rho be the cost of a missed item divided by the cost of a wrongly blocked one. If the score is a calibrated probability, blocking at or above 1 / (1 + rho) minimises expected harm 2610.33849 eq. 2. The paper credits this rule to Elkan and says each ingredient predates its work 2610.33849 §IV-C §II-C. Its illustrative rho values were 1, 5, 10 and 20, treated as governance inputs and swept in an appendix 2610.33849 §IV-C. A per-source trust multiplier kappa, clipped to [0.25, 20], moves the cut-off to 1 / (1 + rho × kappa) 2610.33849 eq. 3 §IV-D. Card: Kumar and Sahaym.
Cut-off from the clean-score tail
Each defense in the poisoning study sets its cut-off from clean documents rather than from poison. Perplexity and DRS use the 99th percentile of clean scores. RAGuard uses a 2.5% tail on each of its three tests, combined by OR, for a nominal 10% false-positive rate. Two defenses use fixed numbers instead, 0.88 and 0.25 for TrustRAG stage 1 and 0.95 cosine for near-duplicates [arxiv:2610.33849 §II-B] (tab. I). The threshold is fitted on clean data, so its realised rate on other clean data can differ from the nominal one. Across four corpora, RAGuard's nominal 10% became 0.098 on average and DRS's nominal 1% became 0.103, but DRS was 0.035 on medical 2610.33849 §II-B. Card: Kumar and Sahaym.
When to use which
| Situation | Use | Why |
|---|---|---|
| Comparing two models or losses on the same data | One global cut-off tuned on validation, applied the same way to each model | Without tuning, the comparison mixes architecture with threshold choice 2601.14172 §1.1 §4.1 |
| Labelled validation data exists, and rare labels matter | Global cut-off tuned on validation | It raised macro-F1 for both encoders tested, by 0.033 and 0.043 2601.14172 tab. 6 |
| No validation split is available | 0.5, and report it as the untuned baseline | Both tuned rules in the paper pick the cut-off on validation, and it reports no tuned result without that split 2601.14172 §4.1 |
| Each label needs a recall target under a precision floor | Per-label cut-off with precision at least 0.40 | This is the paper's rule for the presence gate and LLM baselines 2601.14172 §4.1. It is not tested head to head against the global rule |
| Model outputs are close to 0 or 1, such as LLM answers | Expect little gain; tune anyway, or keep 0.5 | Label-wise tuning changes little when outputs are nearly binary 2601.14172 §4.1 |
| Misses and wrong blocks have known, unequal costs, and scores are calibrated | Cost-derived cut-off, 1 / (1 + rho) | It is the harm-minimising rule under calibration. With equal costs it reduces to 0.5 2610.33849 eq. 2 §VI-E |
| Scores may drift away from the calibration data | Check calibration at the deployment prevalence before trusting any fixed cut-off | On one shifted medical setting the learned scorer's ECE at realised prevalence was 0.471, and AUROC cannot show that kind of failure 2610.33849 §VI-C |
| Blocking benign items has a fixed budget | Cut-off from the clean-score tail, checked on held-out clean data | The realised false-positive rate differed from the nominal one in the paper 2610.33849 §II-B |
Implementation sketch
The rules share one final step: compare scores with frozen thresholds on test. They differ in how the thresholds are chosen. The first two use validation data, the cost rule uses error costs, and the tail rule uses clean data.
# Global cut-off: one threshold for all labels, chosen on validation [§4.1]
# Input: val_scores (N, L), val_gold (N, L), candidate grid in [0, 1]
# Output: one threshold t, applied unchanged to test [§5.3]
def global_threshold(val_scores, val_gold, grid):
best_t, best_f1 = 0.5, -1.0
for t in grid:
pred = val_scores >= t # (N, L) booleans
f1 = macro_f1(val_gold, pred) # mean over the L labels
if f1 > best_f1:
best_t, best_f1 = t, f1
return best_t# Per-label cut-off: highest recall at precision >= floor [§4.1] [§5.3]
# Input: scores (N,) and gold (N,) for one label, on validation
# Output: one threshold for that label, or None if no grid value qualifies
def label_threshold(scores, gold, floor=0.40):
grid = arange(0.10, 0.90, 0.01) # 0.10 to 0.89 [§5.3]
best_t, best_recall = None, -1.0
for t in grid:
pred = scores >= t
precision, recall = precision_recall(gold, pred)
if precision >= floor and recall > best_recall:
best_t, best_recall = t, recall
# The paper does not say what to do when None is returned [§4.1]
return best_t# Cost-derived cut-off on a calibrated score [eq. 2] [eq. 3]
# Input: score (calibrated P(positive)), rho = C_FN / C_FP, kappa (1.0 if unused)
# Output: True (block) or False (pass)
def cost_rule(score, rho, kappa=1.0):
# kappa is clipped to [0.25, 20] before use [§IV-D]
threshold = 1.0 / (1.0 + rho * kappa) # eq. 3; kappa = 1 gives eq. 2
return score >= threshold# Tail cut-off from clean scores at a nominal false-positive rate [§II-B]
# Input: clean_cal (M,) scores of clean items on a calibration split
# Output: threshold t; confirm the realised rate on a separate clean split
def tail_threshold(clean_cal, fpr=0.01):
# With M = 200 the 99th percentile is the second-largest value [§VIII]
return quantile(clean_cal, 1.0 - fpr)# Apply frozen cut-offs to test scores; test is not touched before this [§5.3]
# thresholds: a scalar (global rule) or one value per label (per-label rule)
def predict(test_scores, thresholds):
return test_scores >= thresholdsContradictions
- Which rule produced the 19-value DeBERTa numbers. The paper says a global cut-off was used 2601.14172 §4.1 §5.3. A code review of the authors' release, recorded on the card's Paper vs code notes, found that their evaluation function tunes one cut-off per label with the precision rule. The same notes say the global rule is the one needed to reproduce the paper's tables 2601.14172 §4.1 §5.3. Which rule a reader should use to match the reported numbers is therefore not settled by the sources here.
- Gains from the two papers are not comparable. One reports a gain from a tuned cut-off on 19-value multi-label scores 2601.14172 tab. 6. The other proposes no new threshold rule. It reports that existing per-document rules lose most of their detection when the poison's construction changes 2610.33849 §VI-A §VI-B. Neither paper's numbers contradict the other's.
Pitfalls
- Tuned and untuned numbers are not comparable. The cut-off gain alone is larger than any lightweight feature in the paper. Compare models at the same tuning state, and say which state a published number uses 2601.14172 §6.3 §7.
- A high ranking score can hide a dead cut-off. When the question prefix was removed from the NQ poison, RAGuard's AUROC fell from 0.953 to 0.758 and its miss rate at the deployed point rose from 0.191 to 0.763 [arxiv:2610.33849 §VI-A]. Report the miss rate and the false-alarm figures at the cut-off you actually ship, not AUROC alone. The paper says AUROC cannot tell calibration collapse from signal collapse 2610.33849 §VI-C.
- Calibration sets can be too small for a tail quantile. Every fitted threshold in the poisoning study except DRS is a quantile of at most 200 clean documents, so its 99th percentile is the second-largest value. The calibration split is fixed across seeds, so seed spread does not show this variance 2610.33849 §VIII.
- Nominal and realised false-positive rates differ. Compare miss rates only at the realised rate measured on held-out clean data 2610.33849 §II-B.
- Equal costs quietly give 0.5. When rho = 1, the cost-derived rule equals the 0.5 classifier. Five of the twelve cells where all three rules gave identical actions had rho = 1, and the paper counts none of them as evidence 2610.33849 §VI-E.
- The presence optimum sits on the grid edge. Every presence row chose 0.10, the lowest value in the search range 2601.14172 tab. 3. The true optimum may lie below the grid. Widen the grid before trusting the value.
- Grid step differs between rules. The label-wise search uses 0.01 steps. The global sweep script in the authors' code uses 0.05 steps by default. Use the same grid for every model you compare 2601.14172 §4.1 §5.3.
- Thresholds do not transfer across feature sets. The tuned global value was 0.15 for one model and 0.30 for another in the same study 2601.14172 tab. 5. Re-tune for each configuration instead of reusing one value.
- Feature gains sit inside seed noise. Once the threshold is tuned, the gains from context, LIWC-22 and topic features are small, and none passes the paired per-seed test over three seeds 2601.14172 §6.3 tab. 5.
- Rules with no qualifying value are undefined. If no grid value reaches the precision floor for a label, the paper does not say what to use 2601.14172 §4.1. Decide and document a fallback before running.
Papers
- Human Values in a Single Sentence: compares a fixed 0.5 cut-off, a global cut-off tuned on validation, and a per-label rule with a precision floor on 19-value value detection, with DeBERTa-base and LLM baselines.
- Ranking Is Not Deciding: shows that query-time RAG poison filters can rank well while their deployed cut-offs fail, derives a cost-based cut-off on calibrated scores, and reports false-positive rates against nominal targets.
Source and license
Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.