wikillm

Paper · 2026-01-20 · cs.CL, cs.AI

Human Values in a Single Sentence: Moral Presence, Hierarchies, and Transformer Ensembles on the Schwartz Continuum

Detects the 19 refined Schwartz values in English ValueEval'24 sentences with DeBERTa-base on one 8 GB GPU. Presence-gated pipelines do not beat direct prediction. Tuning one decision threshold on validation gives most of the gain (macro-F1 0.282 to 0.315). A soft-voting DeBERTa ensemble reaches 0.332. Zero- and few-shot 7-9B LLMs and QLoRA trail. Author-reported.

Multi-label classificationDecision threshold tuningEnsemblesSeed-variance evaluationvalue-detectionmulti-label-classificationthreshold-calibrationensemblesclass-imbalance

arXivPDFCodeMarkdown16 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

The paper asks a machine to read one sentence from a news article or a political manifesto and say which human values it expresses. There are 19 values, such as concern for nature or wanting to keep rules, and most sentences express none or only one or two. Because the common ones dominate the data, a model's default cut-off between "yes" and "no" misses many of the rare values.

The authors test several designs under the same budget of one consumer GPU with 8 GB of memory. One design first checks whether a sentence contains any value at all and only then picks the values. Another adds extra signals such as word-category counts or topic scores. A third mixes several trained models into an ensemble, and a fourth uses general-purpose language models that are prompted or fine-tuned. The main finding is that moving the cut-off, which is chosen on validation data, improves results more than any of the other changes. The ensemble comes second, and the language models come last.

TL;DR

  • Fine-tuned DeBERTa-base detects whether a sentence contains any value with test F1 about 0.73 tab. 3.
  • A gate that checks for presence first, then hands the sentence to the value detector, does not beat direct 19-way prediction. The gate's recall limits the result §6.2 tab. 4.
  • Tuning one global decision threshold on validation raises the text-only DeBERTa from macro-F1 0.282 to 0.315 (author-reported, three seeds). Lightweight features add at most about 0.005 and do not pass the paired per-seed test §6.3 tab. 5.
  • A soft-voting ensemble of three DeBERTa models reaches the best reported macro-F1, 0.332, on the reference seed §6.4 tab. 7.
  • Zero-shot, few-shot and QLoRA 7-9B models reach at most 0.238 single and 0.277 as an ensemble §6.4 tab. 7.

Problem

Detecting values in short text has three difficulties. Value cues are often implicit §2.2. Values are very unevenly distributed: about 50% of sentences have at least one value, while rare values such as Humility appear in roughly 0.2% of sentences §3.2 §3.3 app. A. Neighbouring values on the Schwartz circle are easily confused §2.2 §6.8.

With a 0.5 cut-off, a multi-label DeBERTa model under-predicts rare labels, so macro-F1 is low even when the architecture is sound §6.3. Comparisons between architectures are therefore confounded with threshold choice unless the threshold is tuned for every model in the same way §1.1 §4.1. Under the 8 GB budget, the paper also considers how far a hierarchy with a presence gate or a 7-9B LLM can close the gap §5.4.

Method

Task. Each sentence s has 19 binary value labels y[s,v]. An attained label and a constrained label are merged into one label per value §3.1. The sentence-level presence label z_s is 1 if any of the 19 values is set §3.1. Both are studied, but the 19-way task is the main target. Presence is never used to replace the 19-way labels §1.1 §3.1.

Decision rules. A model outputs a probability per value. Evaluation uses macro-F1 over the positive class of the 19 values §3.1 §4.1. Three threshold rules are used:

  1. A fixed threshold of 0.5 §4.1.
  2. One global threshold for all 19 values, chosen on validation to maximise macro-F1 and then frozen for test. This is the rule behind the tuned 19-value numbers §4.1 §5.3.
  3. A label-wise threshold for each value, chosen on validation to maximise recall subject to precision of at least 0.40. This rule is used for the presence gate and for the LLM baselines. The DeBERTa value detectors use the global rule §4.1 §5.3.
Pseudocode
# Direct 19-value DeBERTa detector, global threshold  [§4.4] [§4.1]
# Input:  train, val and test sentences with 19 binary labels each
# Output: 19 binary predictions per test sentence
def direct_detector(train, val, test):
    # DeBERTa-base encoder with one linear head over 19 logits  [§4.4]
    model = deberta_base_with_linear_head(num_labels=19)
    # Multi-label binary cross-entropy on the 19-dim target  [§4.4]
    model.fit(train, loss="bce_with_logits", select_on=val)
    # Probabilities on validation, one per sentence and value  [§4.1]
    probs_val = model.predict_proba(val)
    # Pick the single threshold that maximises validation macro-F1  [§5.3]
    threshold = best_global_threshold(probs_val, val.labels)
    # Apply the frozen threshold to test; no test data used before this  [§5.3]
    probs_test = model.predict_proba(test)
    return probs_test >= threshold
Pseudocode
# Label-wise threshold: most recall at precision >= 0.40  [§4.1] [§5.3]
# Input:  scores (N,) and gold 0/1 labels (N,) on validation for one label
# Output: one threshold for that label
def label_threshold(scores, gold, min_precision=0.40):
    # Search grid 0.10 to 0.89 in steps of 0.01  [§5.3]
    # [code: core/evaluation.py#L36]
    best_threshold = 0.5
    best_recall = 0.0
    for threshold in grid(0.10, 0.89, step=0.01):
        predicted = scores >= threshold
        precision, recall = precision_and_recall(gold, predicted)
        # Keep the threshold with the highest recall that meets the
        # precision floor  [§4.1]
        if precision >= min_precision and recall > best_recall:
            best_recall = recall
            best_threshold = threshold
    return best_threshold

Model families.

  1. Direct DeBERTa detectors. The sentence is encoded by microsoft/deberta-base, and a linear head with 19 outputs is trained with binary cross-entropy. No class weights are used §4.4 §5.2. The code takes the first token (CLS) of the encoder output as the sentence representation [code: core/models.py#L262]. The paper calls this the pooled representation and does not name the token §4.4.
  2. Lightweight auxiliary signals. Four signals can be attached to the sentence representation before the head §4.5:
    • (a) The two previous sentences joined with separators, plus a 38-dim binary vector of their 19-value labels. During training this vector holds gold labels. Evaluation uses predictions. The vector goes through a linear layer to 16 dimensions with ReLU §4.5 models.py.
    • (b) Lexicon features: LIWC-22, the extended Moral Foundations Dictionary, the Moral Justification Dictionary, the Schwartz value lexicon and several affective lexica. For count-based lexica each feature is the share of tokens found in the category. Each lexicon is projected to 128 dimensions §4.5 dataset_utils.py.
    • (c) Topic probabilities from LDA (60 topics), NMF (90 topics) and BERTopic, each projected to 128 dimensions with ReLU §4.5.
    • The fused vector is [text ‖ prior ‖ lexicon ‖ topic], passed through dropout and the 19-way head §4.5 models.py.
  3. Presence-gated hierarchy. A binary gate predicts z_s with either a DeBERTa model or a sentence-embedding model (all-MiniLM-L6-v2) followed by a class-weighted logistic regression. The value head is trained only on presence-positive sentences. At test time, value probabilities are zeroed where the gate score is below t_gate. t_gate is chosen on validation to maximise macro-F1 of the full pipeline §4.6 main.py.
  4. LLM baselines. Gemma 2 9B, Llama 3.1 8B, Ministral 8B and Qwen 2.5 7B are loaded in 4-bit NF4 and run with greedy decoding and at most 200 new tokens §4.7 main.py. They are tested in four zero-shot prompt styles and in few-shot prompts with k between 1 and 20 exemplars. The output is a JSON array of value names §4.7 §6.5 app. B. QLoRA is applied to Gemma 2 9B only §4.7.
  5. Ensembles. Models are combined by hard or soft voting. Forward selection starts from the best single model and adds a candidate only if the one-sided lower 95% bootstrap bound of the macro-F1 gain is above 0 and the gain is at least 1% relative §4.8 ensemble_voting.py. Soft voting averages probabilities, optionally weighted by validation macro-F1, and sweeps one global threshold on validation in steps of 0.01 §4.8 [code: core/ensemble_soft_threshold.py#L58].
Pseudocode
# Fused DeBERTa representation with auxiliary branches  [§4.5]
# Input:  sentence tokens; optional prior-sentence labels, lexicon vector,
#         topic vector
# Output: 19 logits
def forward(tokens, prior_labels, lexicon, topics):
    # Sentence representation from the first token of the encoder  [§4.4]
    text = encoder(tokens)[0]                        # (B, 768)
    parts = [text]
    # Prior-sentence labels: 38 binary values -> 16 dims with ReLU  [§4.5]
    parts.append(relu(linear_38_to_16(prior_labels)))   # (B, 16)
    # Lexicon vector: 128-dim projection, see the paper-vs-code note  [§4.5]
    parts.append(lexicon_branch(lexicon))           # (B, 128)
    # Topic vector: 128-dim projection with ReLU  [§4.5]
    parts.append(relu(linear_to_128(topics)))       # (B, 128)
    fused = concatenate(parts)
    # Dropout on the fused vector, then the 19-way linear head  [§5.2]
    return linear_19(dropout(fused))

Evaluation protocol. Training uses the train split, selection uses validation, and each configuration is scored once on test. Every DeBERTa configuration is run with seeds 42, 7 and 1701 §5.1 §5.2. Results are reported as means with per-seed standard deviations. Presence and 19-way results come from paired tests across seeds, bootstrap tests over sentences (2000 resamples) and McNemar tests with Benjamini-Hochberg correction per value §4.9.

Hyperparameters

Values marked "not stated" do not appear in the paper. Code citations show what the published scripts set.

NameValueWhereNotes
Encodermicrosoft/deberta-base§4.4linear head over 19 labels
Lossbinary cross-entropy with logits, no class weights§4.4 §5.2 models.py
Max tokens512§4.4 §5.2 dataset_utils.pyleft truncation when previous sentences are used runner.py
Batch size4 per device§5.2 main.pythe paper gives this for all transformer models; the code changes it for previous-sentence runs, see Paper vs code
Gradient accumulation4 (effective batch 16)§5.2 main.py8 for previous-sentence runs in the code training.py
Learning rate2e-5§5.2 main.pyno learning-rate schedule or warm-up set in the code training.py
Weight decay0.15§5.2 main.py
OptimiserAdamW§5.2not set explicitly in the code; not verified
Dropout (head)0.1§5.2the head uses the encoder's hidden dropout models.py
Dropout (auxiliary branches)0.4not stated in the paperhard-coded in the code models.py models.py models.py models.py
Epochsup to 10§5.2 main.py
Early stoppingpatience 4 on validation macro-F1§5.2 main.py training.py training.pybest checkpoint loaded at the end
Warm-up evaluationevaluation and checkpoint saving skipped for the first 2 epochsnot stated in the papertraining.py models.py
Seeds42, 7, 1701§5.2run-revision-A1-A3.sh
Hyperparameter searchOptuna, 50 trials; epochs 3-10; batch 2 or 4; learning rate 5e-6 to 5e-5 (log); weight decay 0.1 to 0.3§4.4in approaches/moral-values/main.py the search space sits in a string literal main.py; see Paper vs code
Lexicon projection128§4.5code uses a two-layer block with 256 hidden units models.py
Previous-label projection38 to 16, ReLU§4.5models.py
Topic projection128, ReLU§4.5models.py
Global threshold grid[0, 1] on validation§4.1 §5.3the shell script's default grid is 0.05 steps eval-threshold.sh
Label-wise threshold grid0.10 to 0.89 in 0.01 steps; minimum precision 0.40§4.1 §5.3evaluation.py
Soft-vote threshold grid0 to 1 in 0.01 steps§4.8ensemble_soft_threshold.py
Forward selectionone-sided lower 95% bootstrap bound > 0 and gain of at least 1% relative§4.8the code also requires an absolute gain of at least 0.001 ensemble_voting.py
Bootstrap resamples2000§4.9the ensemble script uses 5000 ensemble_voting.py
Presence gate (SBERT)all-MiniLM-L6-v2 embeddings, class-weighted logistic regression§4.6main.py
LLM precision4-bit NF4 via bitsandbytes§4.7 §5.4main.py
LLM decodinggreedy, max 200 new tokens§4.7main.py
QLoRA (direct)rank 16, alpha 32, learning rate 2e-4, 3 epochs, gradient accumulation 8, max length 512§4.7main.py main.py main.py
QLoRA (hierarchical)rank 8, alpha 16, max length 256§4.7not checked in the code
LoRA targetsq, k, v and o projections§4.7main.py
LoRA dropout, warm-up rationot stated in the paper0.05 and 0.03 in the code main.py main.py
Few-shot exemplars1, 2, 4, 8, 12, 16, 20§4.7not checked in the code

Results

Author-reported. Setup: English machine-translated ValueEval'24 splits (44,758 train, 14,904 validation, 14,569 test sentences), one 8 GB GPU, DeBERTa-base or 4-bit 7-9B LLMs §3.2 §5.4.

Presence (RQ1), positive-class F1, mean over three seeds tab. 3. Tuned threshold t* = 0.10 for every row.

Presence modelAux. featuresVal F1Test F1, threshold 0.5Test F1, tuned threshold
Text-only detectornone0.7290.728 ± 0.0130.732 ± 0.005
LIWC-22 + linguisticLIWC-22 and linguistic categories0.7250.724 ± 0.0120.732 ± 0.003
Prev-2 + LIWC-22two previous sentences, labels, LIWC-220.7310.730 ± 0.0140.730 ± 0.011
Prev-2 + EmoLextwo previous sentences, labels, EmoLex0.7200.721 ± 0.0060.732 ± 0.004
Prev-2 + eMFDtwo previous sentences, labels, eMFD0.7150.719 ± 0.0140.730 ± 0.007

Direct vs presence-gated (RQ2), test macro F1 over 19 values tab. 4.

ModelArchitectureFeaturesThreshold(s)Test F1
Direct baselinedirecttext onlyt = 0.300.315 ± 0.004
Direct + LIWC-22directtext + LIWC-22t = 0.250.320 ± 0.003
Direct + MJDdirecttext + MJDt = 0.250.316 ± 0.001
Presence-gated (MJD)hierarchicaltext + MJD; gate on Prev-2 + LIWC-22t = 0.40; t_gate = 0.100.302 ± 0.009

Lightweight signals on the direct detector (RQ3) tab. 5. Each test cell gives a mean and a standard deviation over the three seeds. The tuned threshold is the reference-seed value.

Direct modelAux. featuresVal F1 (threshold 0.5)Test, threshold 0.5Test, tunedTuned threshold
Text-only detectornone0.2790.282 ± 0.0010.315 ± 0.0040.30
Prev-2 sentences + labelscontext0.2770.280 ± 0.0080.317 ± 0.0010.15
LIWC-22psycholinguistic lexicon0.2840.288 ± 0.0040.320 ± 0.0030.25
BERTopic topicstopic distribution0.2790.284 ± 0.0020.320 ± 0.0010.20

Threshold calibration on a second encoder tab. 6. Text-only models. For each seed, the threshold is picked on validation and then used as is on test.

EncoderTest F1 @ 0.5Test F1 @ t*Calibration gainPaired p
DeBERTa-base0.282 ± 0.0010.315 ± 0.004+0.033 ± 0.0040.004
RoBERTa-base0.267 ± 0.0090.309 ± 0.005+0.043 ± 0.0140.033

Supervised encoders vs LLMs and ensembles (RQ4) tab. 7. The single transformer row is a three-seed mean. The ensemble and LLM rows use reference seed 42, because each LLM was run once and the ensemble is built from reference-seed models.

Model classWhat was combinedStructureTest F1
One encoderDeBERTa with LIWC-22 features, threshold chosen on validationflat0.320
Encoder ensemblethree DeBERTa runs (plain, LIWC-22, previous sentences), averaged, threshold 0.29flat0.332
One LLMGemma 2 9B with few-shot prompting, behind a presence gategated0.238
LLM ensemblethe gated few-shot Gemma run and a QLoRA Gemma run, majority voteflat0.277

The paper reports that the three-model soft-voting ensemble improves on its best single component by 0.012 (one-sided lower 95% bound 0.007) and on the best LLM ensemble by 0.055 §6.4. The gated Gemma few-shot model is significantly better than the ungated one (0.233) but well below the DeBERTa models §6.4.

Zero-shot prompt sensitivity, validation macro F1 tab. 8. Each model keeps only its best prompt for test.

Modeldirectcot_hiddendefinitionqaRange
Gemma 2 9B0.1840.1820.2150.1760.039
Llama 3.1 8B0.1190.1000.1630.0970.066
Ministral 8B0.1060.0870.1770.1020.090
Qwen 2.5 7B0.0910.0370.0680.0610.054

The definition prompt, which gives a short gloss for each value, is the best style for three of the four models. The chain-of-thought-hidden style is never the best §6.5.

Source language of the test sentences tab. 9. Calibrated text-only DeBERTa, mean ± std over seeds, each seed's global threshold applied unchanged.

Source languagenMacro F1Source languagenMacro F1
Hebrew1,4800.361 ± 0.006Bulgarian1,3020.291 ± 0.009
Greek1,4020.333 ± 0.013French9310.264 ± 0.019
Turkish2,1870.310 ± 0.009Dutch2,1500.262 ± 0.002
German1,8740.291 ± 0.003English (original)2,0050.254 ± 0.014
Italian1,2380.231 ± 0.007

Machine-translated sentences score higher on average (0.320 ± 0.004) than the English-original ones (0.254 ± 0.014). The paper attributes this to the English subset having fewer values per sentence, and reports that matching on the number of values removes the gap §6.9 app. A.

Paper vs code

Checked against the authors' repository at commit 0f384c2d8c01e929c2a75b688ef271d9b7586af9, licensed Apache-2.0 [code: LICENSE.txt]. The repository is named in the paper's data-availability text and its owner is the first author §3.2.

  • Previous-sentence inputs at validation and test. Paper: validation and test use fixed predictions of the frozen direct model, and the model's training is leakage-free by design §4.5 §5.2. Code: validation uses a callback that re-runs the model being evaluated over each document in sequence, feeding its own predictions back as the previous-label inputs models.py dataset_utils.py. Test prediction does the same with the trained model prediction.py. The paper's leakage argument does not describe this code path. Implement the paper's procedure or the code's, not both, and state which one.
  • Lexicon projection. Paper: one linear projection to 128 dimensions and ReLU per lexicon §4.5. Code: a two-layer block (linear to 256, ReLU, dropout 0.4, linear to 128, ReLU) for the lexicon branch models.py. The linguistic and topic branches are single-layer models.py models.py.
  • Dropout in auxiliary branches. Paper: a grid over 0.1 and 0.2 for dropout is described §5.2. Code: 0.4 is hard-coded in every auxiliary branch [code: core/models.py#L126] models.py. The grid search is not visible in the paths read.
  • Batch size with previous sentences. Paper: batch 4 and accumulation 4 for all transformer models §5.2. Code: when previous sentences or augmented data are used, the batch is set to 2 and accumulation to 8, which keeps the effective batch of 16 training.py.
  • Global versus label-wise threshold. Paper: DeBERTa 19-value results use one global threshold §4.1 §5.3. Code: eval_labels in core/evaluation.py tunes one threshold per label with the precision rule [code: core/evaluation.py#L81-L87]. The global macro-F1 sweep is in approaches/moral-values/eval-threshold.sh, whose default grid runs in 0.05 steps eval-threshold.sh. Tables 4 and 5 use the global rule; use that rule to reproduce them.
  • Hyperparameter search. Paper: Optuna with 50 trials over four hyperparameters §4.4. Code: in approaches/moral-values/main.py the search space is inside a string literal, so --optimize runs the fixed values [code: approaches/moral-values/main.py#L54-L60]. approaches/presence/main.py has the same layout main.py.
  • Forward-selection rule. Paper: 1% relative gain and a lower bound above zero §4.8. Code: also requires an absolute gain of at least 0.001 [code: core/ensemble_voting.py#L63-L64].
  • Bootstrap resamples. Paper: 2000 §4.9. Ensemble script: 5000 [code: core/ensemble_voting.py#L60].
  • Default run parameters. run_training defaults (9 epochs, learning rate 2.07e-5, weight decay 1.02e-5, accumulation 2, patience 3) differ from the values the paper reports. The published scripts pass the paper's values explicitly runner.py main.py.
  • Lexicon normalisation. Paper: count-based lexicon features are divided by the number of tokens §4.5. Code matches: scores are divided by the token count dataset_utils.py. No discrepancy.
  • Presence gate. Paper: class-weighted logistic regression on sentence embeddings §4.6. Code matches main.py. No discrepancy.

Pitfalls

  • The threshold is most of the gain. Choosing the threshold on validation lifts the text-only detector by about 0.033 macro-F1. That is more than any lightweight feature in the paper. A model trained with a 0.5 cut-off and compared to published numbers at a tuned threshold is not like for like §6.3 §7.
  • Thresholds sit on the grid edge for presence. Every presence row selects t* = 0.10, the lowest value in the search range tab. 3 [code: core/evaluation.py#L36]. The reported optimum may be bounded by the grid. Test a lower grid before trusting it.
  • Feature gains are inside seed noise. The gains from context, LIWC-22 and topics are +0.002 to +0.005 with per-seed standard deviation up to 0.008. None passes the paired per-seed test §6.3 tab. 5. Three seeds give a t-test with two degrees of freedom §4.9.
  • Gate recall is a ceiling. The hierarchy can never recover a value the gate filters out. The value head is trained only on presence-positive sentences, and the feature-matched direct model beats the gated one §4.6 §6.2 tab. 4.
  • Mixed reporting bases in the summary table. Tab. 7 compares a three-seed mean with reference-seed values. The 0.332 ensemble and the LLM rows are one seed each, so the comparison is not seed-matched tab. 7 caption.
  • Ensemble weighting is not fully specified. The paper says averaged or weighted probabilities were used and does not say which one gives 0.332 §4.8 §6.4.
  • Previous-sentence features are asymmetric. Training uses gold prior labels, while evaluation uses predictions. The paper reports this as a train-inference mismatch that flatters the feature §4.5 §6.3. See Paper vs code for which prediction source the code uses.
  • Warm-up evaluation hides early epochs. The code does not evaluate or save checkpoints during the first two epochs. Early stopping and checkpoint selection therefore start late, which the paper does not mention training.py.
  • Default run_training values are not the paper's. Calling the function with its defaults changes epochs, learning rate, weight decay and patience [code: core/runner.py#L27-L31].
  • Data is restricted. The ValueEval'24 texts come under a Data Usage Agreement that forbids redistribution. Obtain them from Zenodo before you can rerun the experiments. The authors release code, thresholds and per-model predictions §3.2.
  • Machine translation. About 86% of the English release is machine-translated from eight other languages. Lexicon features depend on word forms and are the most exposed to translation effects. The paper finds no detectable degradation after controlling for value counts, but does not audit translation quality §3.2 §6.9 §6.10.
  • Attainment is collapsed. Attained and constrained labels are merged into a single binary label, so stance information is not modelled §3.1.
  • LLM rows are not tuned like the encoders. LLM outputs are nearly binary, so label-wise tuning changes little §4.1. Few-shot prompt choice used the validation split only, so the gap to encoders is not an artefact of one prompt §6.5.

Lineage

  • Builds on: the authors' ValueEval'24 English cascade system (Yeste et al., 2024), whose direct DeBERTa component is reimplemented here without its rule-based post-processing §1.1 §4.4. The paper reports this system at macro-F1 about 0.28 §2.2 tab. 1.
  • Compared against: text-only and feature-augmented DeBERTa-base detectors, RoBERTa-base for the calibration check, and Gemma 2 9B, Llama 3.1 8B, Ministral 8B and Qwen 2.5 7B as LLM baselines §4.3 §4.7.
  • Superseded by: not stated.

Source and license

Adapted from “Human Values in a Single Sentence: Moral Presence, Hierarchies, and Transformer Ensembles on the Schwartz Continuum” by Víctor Yeste, Paolo Rosso (arXiv:2601.14172), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/VictorMYeste/human-value-detection at commit 0f384c2d8c01 (Apache-2.0). Any quoted code is shown under that license.