wikillm

Paper · 2026-10-26 · cs.CL

RAGing Against Misogyny: Improving Implicit Misogyny Explanations

Retrieval-augmented misogyny classifier: Llama-3.1-8B-Instruct gets evidence from Wikipedia, Wiktionary slang or two feminist books, compressed by similarity to the post. Author-reported: recall on ImplicIT-Mis rises to 0.923 with literature (best no-retrieval 0.891); no source beats the baseline on SBIC+; annotators prefer RAG explanations in both languages.

Retrieval-augmented generationKnowledge injectionToxicity detectionHuman evaluationmisogyny-detectionhate-speechretrieval-augmented-generationexplanation-generationmultilingual

arXivPDFCodeMarkdown16 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Calling a comment misogynistic often depends on what it leans on without saying it: a public figure, a community's slang, or an idea from feminist writing. A language model that reads only the comment can miss that. This paper gives the model short pieces of outside text first. It looks up the named people and topics in Wikipedia, the unusual words in a slang dictionary, and the ideas in two feminist books.

The model then gets those pieces together with the comment. It writes down the assumption the comment leaves unstated, and it gives a yes or no label. The outside text is trimmed to the sentences that match the comment, so the prompt stays short.

The results are mixed. On the Italian set of news comments, the outside text helped the detection score. On the English set it did not. Human raters, who saw the explanations without knowing which system wrote each one, usually preferred the explanations that had outside text. The text from the feminist books was the most useful source.

TL;DR

A retrieval-augmented pipeline that detects and explains implicit misogyny in two languages, English and Italian. It draws on three knowledge sources: Wikipedia, a Wiktionary slang lexicon, and two feminist books. The generator is Llama-3.1-8B-Instruct, and no fine-tuning is described §5 §6. Author-reported, recall on ImplicIT-Mis reaches 0.923 with literature retrieval, against 0.891 for the best no-retrieval prompt, and no retrieval source beats the baseline on SBIC+ tab. 3. Annotators prefer RAG explanations over no-RAG ones in both languages tab. 5. The official repository has no LICENSE file, and it lacks the Wikipedia and literature retrieval code, so parts of the pipeline cannot be checked (see Paper vs code).

Problem

Implicit misogyny needs background knowledge: a named public figure, a cultural reference, community vocabulary, or a theoretical idea §1 §3. The authors say that LLMs fall back on shallow semantic associations and produce generic or unsupported explanations §1 §2. The paper asks four research questions: does the posting context help detection (RQ1); does RAG improve detection and explanation (RQ2); which knowledge source helps most (RQ3); and do RAG models fix known failure modes (RQ4) §1.

Method

The pipeline runs in five stages, in order: restrict the domain, retrieve by named entity, compress once, rerank the candidates, and compress again §5. Each knowledge source is run separately, so each prompt uses one source §6. The model returns an implied assumption and a label §6 app. A. Figure 1 is a worked example for one comment, showing the Wikipedia page, the compression step and the baseline and RAG explanations fig. 1.

Per source:

  • Wikipedia §3.1 §5. Wikidata entities are filtered to humans and three domain topics. The retained pages go into a local index, and a live Wikipedia search fills in when the index has no match. Named entities come from spaCy, expanded by two rules and filtered by a fixed keyword list §5. Each candidate page is scored and kept or dropped by similarity §5. The first compression scores the sentences of each page and keeps the best ones §5.
  • Slang §3.2 §5. Input tokens are matched against Wiktionary entries for slang categories of each language. Only senses marked vulgar, derogatory or offensive are kept §3.2. Each word-and-sense pair is scored against the input and kept above a similarity cut-off §5. Slang is not compressed a second time §5.
  • Feminist literature §3.3 §5. The source is two books: Manne (2018) and Ging (2019). The books are split into two-sentence units. The top two units by similarity are kept if they exceed a cut-off. There is no first compression for this source §5.

Second compression, for Wikipedia and literature only: keep each sentence whose similarity to the input exceeds 0.3 §5.

Overall procedure. The input is the message plus its posting context. The context is available only for ImplicIT-Mis §5. Retrieval is run for the chosen source. The prompt includes the retrieved text, and the model is called at temperature 0.3 with at most 256 new tokens §6.

Pseudocode
# Overall pipeline for one knowledge source per run  [§5] [§6]
# Input: text, context ("" for English), source in {wiki, slang, literature}
# Output: implied assumption and YES/NO label

def classify_and_explain(text, context, source, llm):
    # Query is the message plus its posting context when there is one  [§5]
    query = join_nonempty(text, context)
    # Candidate evidence from the chosen knowledge base  [§3] [§5]
    evidence = retrieve(query, text, source)
    # Second compression applies to Wikipedia and literature, not slang  [§5]
    if source != "slang":
        evidence = keep_sentences_above(evidence, query, threshold=0.3)
    # Prompt with the evidence appended; temperature 0.3, 256 new tokens  [§6]
    prompt = build_prompt(text, context, evidence)
    answer = llm.generate(prompt, temperature=0.3, max_new_tokens=256)
    # The model returns an implied assumption and a label  [§6] [app. A]
    return parse_json(answer)
Pseudocode
# Wikipedia: first compression of one page  [§5]
# Input: page (str), entities (list of str), keywords (list of str)
# Output: retained text, at most 8 sentences and 1,500 characters

def first_compress(page, entities, keywords):
    # Split the page into sentences  [§5]
    sentences = split_sentences(page)
    scored = []
    for position, sentence in enumerate(sentences):
        # +3 if the sentence mentions a target entity  [§5]
        score = 3 * mentions_any(sentence, entities)
        # A keyword hit from the domain list adds 2  [§5]
        score += 2 * contains_any(sentence, keywords)
        # A sentence in the opening three positions adds 1  [§5]
        score += 1 * (position < 3)
        scored.append((score, position, sentence))
    # Keep the top 8 sentences; the paper does not state the order  [§5]
    kept = top_sentences(scored, n=8)
    # Cut the result at 1,500 characters  [§5]
    return truncate("".join(kept), 1500)
Pseudocode
# Slang: match, pick a sense, keep pairs above the similarity cut-off  [§5]
# Input: text, context, lexicon (dict term -> list of senses)
# Output: list of (term, sense, similarity)

def retrieve_slang(text, context, lexicon, embed):
    query = join_nonempty(text, context)
    query_vector = embed(query)
    hits = []
    # The matching rule (exact or substring) is not stated in the paper  [§5]
    for term in matched_terms(query, lexicon):
        # Prefer the offensive sense when a term has several  [§3.2]
        sense = pick_offensive_sense(lexicon[term])
        # Similarity between the input and the "term: sense" pair  [§5]
        score = cosine(query_vector, embed(term + ": " + sense))
        # Keep the pair only if the similarity is above 0.3  [§5]
        if score > 0.3:
            hits.append((term, sense, score))
    return hits
Pseudocode
# Literature: two most similar two-sentence units  [§5] [app. B]
# Input: query (str), units (list of two-sentence strings from the books)
# Output: up to two units

def retrieve_literature(query, units, embed):
    query_vector = embed(query)
    scores = [cosine(query_vector, embed(u)) for u in units]
    # Take the two most similar units  [§5]
    best = top_indices(scores, k=2)
    # Keep a unit only if its similarity exceeds 0.6  [§5] [app. B]
    return [units[i] for i in best if scores[i] > 0.6]

Hyperparameters

NameValueWhereNotes
GeneratorLlama-3.1-8B-Instruct§6Chosen as the best model in the authors' earlier study §6. Its licence was not checked here.
Temperature0.3§6Sampling is on when temperature > 0 run.py
Max new tokens256§6Stated to fit the JSON output §6 run.py
Other decoding settings"left at default"§6Not stated beyond that. The code sets none beyond the two above run.py
Precisionnot stated in paper—Code default bfloat16 run.py
Embedding modelparaphrase-multilingual-MiniLM-L12-v2§5Same model in the code compression.py slang_pipeline.py
Wikidata entity classesQ5 (human); Q7252 (feminism); Q1342425 (femicide); Q1800556 (violence against women)§3.1Wikidata filtering code is not in the repository
Local indexWhoosh; live Wikipedia search as fallback§3.1Index code is not in the repository
Entity keyword filternine fixed terms, alphabetically: abuse, femicide, harassment, misogyny, rape, sexism, victim, violence, woman§5Chosen before inspecting the data §5
Wikipedia candidates per entityat most 3§5
Wikipedia admission rulesimilarity to the entity at least 0.3, or the entity in the title or extract§5
Wikipedia candidate scorecosine(entity string, page) + cosine(input, page)§5
First compression scores+3 entity mention; +2 domain keyword; +1 among the first three sentences§5
First compression capsat most 8 sentences; 1,500 characters per page§5
Wikipedia field cap4,500 characters, for up to 3 candidates§5
Slang categoriesItalian: one Wiktionary category (about 530 entries). English: three Wiktionary categories for general slang, internet slang and incel slang (about 22,000 entries)§3.2Live Wiktionary categories slang_pipeline.py
Slang senses keptvulgar, derogatory, offensive§3.2 §5slang_pipeline.py slang_pipeline.py
Slang similarity cut-off0.3 (paper: "above")§5 app. BCode keeps >= slang_pipeline.py
Literature unitstwo-sentence units; top 2; similarity above 0.6§5 app. BBooks not released § Ethical Considerations
Second compressionsentence similarity above 0.3; Wikipedia and literature only§5 app. BThreshold 0.3 chosen from 0.2, 0.3, 0.4 app. B
Threshold selectionmanual review of 200 instancesapp. BNo held-out set reported
Input contextheadline plus comment for ImplicIT-Mis only; English has none§5 §6
Prompt examplesthree worked examples per prompt (Italian few-shot in code)app. A run.pyPrompt wording differs between paper and code (see Paper vs code)
Human evaluation sample100 Italian and 100 English texts; priority to more retrieved contexts§8
Faithfulness scale0, 1 or 2 per explanation; two annotators§6.1Faithfulness = Sum / N, with N = 2 × number of explanations §6.1
Preference labelsprefer system X, tie, noneapp. D
Smaller modelsLlama-3.2-1B and 3B-Instructapp. CSubset only app. C

Results

Author-reported. Setup: ImplicIT-Mis (Italian, 1,120 Facebook comments, all misogynistic, with news headlines) and SBIC+ (English, 2,344 messages, all misogynistic). The metric is recall, because every instance is misogynistic §4 §6. The main model is Llama-3.1-8B-Instruct §6.

Coverage (share of instances with any retrieved evidence) tab. 1 §5.1:

Knowledge baseItalianEnglish
Wikipedia87.4%3.4%
Slang9.7%57.1%
Literature29.6%40.8%

RQ1, posting context (recall) tab. 2 §7:

DatasetPrompt settingRecall
ImplicIT-Mistext-only, Italian0.838
ImplicIT-Mistext-only, English0.771
ImplicIT-Mis+ headline, Italian0.865
ImplicIT-Mis+ headline, English0.891

The headline helps in both prompt languages. The gain is +0.120 with English prompts and +0.027 with Italian prompts §7.

RQ2, retrieval (detection recall) tab. 3 §7:

DatasetSystemPromptRecall
ImplicIT-Mis+ headlineEN0.891
ImplicIT-MisWikipediaIT0.881
ImplicIT-MisWikipediaEN0.853
ImplicIT-MisSlangIT0.916
ImplicIT-MisSlangEN0.895
ImplicIT-MisLiteratureIT0.923
ImplicIT-MisLiteratureEN0.666
SBIC+baselineEN0.966
SBIC+WikipediaEN0.962
SBIC+SlangEN0.806
SBIC+LiteratureEN0.723

The authors say the language and the dataset change together, so the Italian and English gap cannot be attributed to language alone §7.

Faithfulness (0 to 2 scale, mean per explanation, two annotators) tab. 4 §8:

DatasetSystemNSumMeanStd.
ImplicIT-MisWikipedia1941280.6590.774
ImplicIT-MisSlang94720.7650.835
ImplicIT-MisLiterature1741610.9250.826
ImplicIT-Misno RAG2001180.5900.745
SBIC+Wikipedia36391.0830.841
SBIC+Slang1842401.3040.833
SBIC+Literature1862531.3600.802
SBIC+no RAG2002691.3450.774

On ImplicIT-Mis, RAG explanations are more faithful than no-RAG ones, with a Wilcoxon p below 0.0001 and Cohen's d of 0.272 (author-reported) §8. On SBIC+ the difference is not significant §8.

Pairwise preference, RAG vs no-RAG. Across both annotators, the pairwise counts over 200 texts are: on ImplicIT-Mis, 121 preferences for the RAG explanation and 51 for the no-RAG one; on SBIC+, 154 for RAG and 95 for no-RAG. A text can count in both columns after a tie or a disagreement tab. 5 §8.

Mutually exclusive breakdown. Counting each text once: on ImplicIT-Mis, RAG wins 88 texts, the baseline wins 18, 33 are ties, and 61 have no selected explanation. On SBIC+, RAG wins 86, the baseline wins 27, 68 are ties and 19 have no selection tab. 6 §8.

Preference by knowledge source (preferred ÷ explanations produced) tab. 7 §8:

DatasetSystemPreferredNormalised
ImplicIT-MisWikipedia46/1940.237
ImplicIT-MisSlang24/940.255
ImplicIT-MisLiterature71/1740.408
ImplicIT-Mis+ headline43/2000.215
SBIC+Wikipedia13/360.361
SBIC+Slang92/1840.500
SBIC+Literature100/1860.538
SBIC+baseline92/2000.460

Failure modes (RQ4) §8 tab. 8: the authors identify known failure modes in the no-RAG explanations and count how many RAG explanations resolve them. For Italian, 21 issues are resolved, nine of them wrong inference and six of them opposite intention; the rest are denial of misogyny (3), sarcasm or irony (2) and figurative language (1). For English, 9 are resolved: four wrong inference, four sarcasm or irony, and one lack of reference §8. The paper says RAG still relies on lexical association in some cases §8. Table 8 shows four examples (lack of reference, wrong inference, sarcasm or irony, figurative language) tab. 8.

Inter-annotator agreement (Cohen's kappa) tab. 10 §6.1 app. D:

SubsetItalianEnglish
General (faithfulness)0.5050.454
Wikipedia0.7100.452
Slang0.6800.504
Literature0.6500.421
Baseline0.7000.437
Human preference0.2860.333

After the discussion and re-annotation step, the authors report 0.925 (Italian) and 0.702 (English). The metric for that figure is not named §6.1.

Smaller models (recall, subset of configurations) tab. 9 app. C:

ModelDataKBSettingRAGNo RAG
1BITWikipediafew-shot0.9490.754
1BENWikipediazero-shot0.8870.882
3BITWikipediafew-shot0.8930.868
3BENWikipediazero-shot0.9640.965
1BENSlangfew-shot0.9460.944
3BITSlangfew-shot0.7120.666

The authors say the retrieval effect varies with model size and is not consistent across settings app. C.

Paper vs code

Repository: https://github.com/ivana2003/rag-implicit-mis, cloned at commit 58b095d2d5e7d16ea60b36d7ef967c6358a36787. The paper links it directly §1. The repository has no LICENSE file, so code_license is recorded as none. The code is cited for comparison only, and no code is quoted.

What the repository contains: the generation script, slang lexicon retrieval, second compression, merging, output parsing, and annotation handling. It does not contain the Wikipedia pipeline (Wikidata filter, Whoosh index, Wikipedia search, entity extraction, reranking, first compression) or the literature retrieval. The README also describes a 0.3 compression threshold for Wikipedia and a 0.6 cut-off for literature README.md. These are described in the paper, and the code that would produce them is absent.

  • Wikipedia and literature retrieval are not in the repository. Every Wikipedia and literature step in §5 (filtering, index, reranking, the +3/+2/+1 scores, the 8-sentence and 1,500-character caps, the 4,500-character field cap, the 0.6 literature cut-off) cannot be checked against code. The merge script expects columns named lit_compressed and retrieval_text_RAG, but no script in the repository writes them merge.py merge.py.
  • Slang keeps one sense per term. The paper keeps every vulgar, derogatory or offensive sense of a matched term §3.2 §5. The code keeps one definition per term, the first one containing a priority tag, and falls back to the first definition when no tag matches slang_pipeline.py slang_pipeline.py.
  • Slang is compressed a second time. The paper leaves slang untouched by the second compression §5. The slang script applies the same sentence filter as for Wikipedia, but it splits on the | character while the retrieved entries are joined with blank lines slang_pipeline.py slang_pipeline.py slang_pipeline.py. Each entry therefore probably passes or fails as one unit, though I did not run it.
  • Slang term matching is loose. Candidates are added when a token is a substring of a lexicon term or the reverse slang_pipeline.py. Tokens shorter than three characters are dropped slang_pipeline.py. The paper does not state a matching rule §5.
  • Slang tag filter may miss labels. The cleaner removes every {{...}} template from a definition slang_pipeline.py, and the tag test runs on the cleaned text slang_pipeline.py. Wiktionary usually writes usage labels in such templates. If so, the tag test would miss them. I did not check this against live Wiktionary pages.
  • Cut-offs use >=. The sentence filter and the slang cut-off keep a score equal to 0.3 compression.py slang_pipeline.py. The paper says "exceeds" §5.
  • Extra rules in the second compression. The code drops sentences of 20 characters or fewer compression.py, strips headings compression.py, and cuts each passage at a set of Italian Wikipedia section headings compression.py compression.py. None of these rules is in the paper.
  • Slang output cap. The slang script cuts its output at 4,500 characters slang_pipeline.py. The paper gives that cap for Wikipedia only §5.
  • Prompts. The paper gives English prompts for each setting, plus several Italian variants app. A. The code builds only Italian few-shot prompts, with three examples and a JSON output template run.py. It has no English prompt builder and no text-only baseline. The no-RAG baseline includes the headline, which matches the paper's "+ headline" setting run.py.
  • Mode flag is unused. --mode is accepted and never read, so every run produces both the RAG and the no-RAG output run.py run.py.
  • Sampling. Output depends on the model's own decoding defaults for everything except temperature and token count run.py. The paper says the other settings are left at default, so those are the same.
  • Human-evaluation sample. The paper says the 100 texts per language were prioritised by retrieval count §8. The script orders rows by how many of the three sources returned evidence and takes the first 100. Its seed argument is never used, so the selection is deterministic, not random merge.py. The Italian merge also inserts one hard-coded row (id 1092) if it is missing merge.py.
  • Agreement and significance are not in the code. The Cohen's kappa values, the Wilcoxon test and the effect size are not computed in any script I read (code/annotation_analysis.py, code/build_results.py). The README gives different kappa values (faithfulness 0.528 Italian, 0.520 English; preference 0.842 Italian, 0.838 English) README.md. The paper gives 0.505, 0.454, 0.286 and 0.333 §6.1. The two sources disagree, and the paper's values are the ones used in this card.
  • Label parsing. The summary count matches the exact string SÌ (with the accent) or YES parser_rag.py. Recall is not computed in any script I read, so the handling of an unaccented SI answer is unknown.
  • Data column names differ across scripts. The generator reads retrieval_compressed run.py, the compressor writes it by default compression.py, and the merger reads retrieval_text_RAG merge.py merge.py. The glue between the steps is not in the repository beyond these scripts.

Pitfalls

  • Recall-only metric on all-positive data. Every instance in both datasets is misogynistic, so the detection score is recall alone §4 §6. It cannot show false positives. A prompt that answers YES to everything would score 1.0.
  • Margins across prompt languages. The best RAG recall (0.923, Italian prompt) is 0.032 above the best baseline (0.891, English prompt). Over 1,120 positives that is about 36 instances. Against the Italian headline baseline (0.865) it is 0.058, about 65 instances tab. 2 tab. 3. The paper compares across prompt languages, so this is not a like-for-like gap. The compression check gives 0.881 without compression and 0.883 with it app. B. That is 0.002, about two instances. The paper keeps the compression step on that basis, and no test is reported for it.
  • Thresholds came from one qualitative review. All retrieval cut-offs were set by reading the retrieved evidence on 200 instances, not on a held-out split app. B §5. Reusing them on other data is untested.
  • Language and dataset are confounded. The authors say the Italian and English results differ in dataset, implicitness, coverage and the type of knowledge needed, so they cannot attribute the gap to language §7.
  • Coverage drives the result. Wikipedia covers 3.4% of SBIC+ tab. 1, and literature covers 40.8% of it. Per-source results therefore rest on different subsets of the data tab. 1 §5.1.
  • Human evaluation is small and authored by the paper's authors. There are 100 texts per language, and the two annotators are authors of the paper §8 § Limitations. The preference kappa is low (0.286 Italian, 0.333 English) §6.1.
  • Paper-internal numbers differ. The text gives the aggregated RAG faithfulness mean as 0.781 in one place §8 and as 0.788 in another §8. Some printed means do not match Sum/N to the last digit: 128/194 is 0.660 and 72/94 is 0.766, and the table prints 0.659 and 0.765 tab. 4. Check the annotation file before reusing these numbers.
  • RQ4 counts do not map cleanly onto sentences. The paper reports 21 resolved issues for Italian against 20 identified problem sentences §8. It does not say whether one sentence can carry more than one failure mode.
  • Feminist books are not released. The source texts are copyrighted and were used under a text-and-data-mining exception. The authors do not release the texts or the derived index § Ethical Considerations. An implementer must bring their own copies, and the retrieval results cannot be reproduced from this repository alone.
  • Wikipedia and Wiktionary are live sources. The Wikipedia search API and the Wiktionary category lists change over time. The slang lexicon is cached only if the cache file is kept slang_pipeline.py.
  • Reproduction needs gaps filled. The Wikipedia index, the literature retrieval and the Wikipedia reranking are not in the repository. Retrieval coverage tab. 1 was not recomputed from the data files.
  • Model choice. The results depend on one 8B model. The smaller-model results show retrieval can help or hurt, depending on setting tab. 9 app. C.

Lineage

  • Builds on: compression of retrieved evidence before in-context use, following the RECOMP approach of Xu et al. (2024) §5; the Llama-3.1-8B model and the ImplicIT-Mis results of an earlier study by Muti et al. (2024) §6 §4; the SBIC and Implicit Hate datasets §4; and the failure-mode taxonomy of Muti et al. (2025) §8.
  • Compared against: the text-only prompt, the text with headline, and the no-RAG prompt in each language tab. 2 tab. 3 tab. 5.
  • Superseded by: not known from the paper.

Source and license

Adapted from “RAGing Against Misogyny: Improving Implicit Misogyny Explanations” by Ivana Crescenzi, Arianna Muti, Maarten Sap, Tommaso Caselli, Debora Nozza (arXiv:2610.35094), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/ivana2003/rag-implicit-mis at commit 58b095d2d5e7 (none (no LICENSE file in repository)). Any quoted code is shown under that license.