wikillm

Paper · 2024-01-27 · cs.CL, cs.AI

Improving LLM Reliability with RAG in Religious Question-Answering: MufassirQAS

MufassirQAS answers Turkish questions on the Quran and hadith with retrieval-augmented generation: 1500-token chunks, OpenAI embeddings in Faiss, the top five chunks and a system prompt sent to GPT-3.5 Turbo, with the source passages shown beside the answer. Evaluation is four qualitative screenshot comparisons against ChatGPT, with no accuracy metric reported.

Retrieval-augmented generationHallucination mitigationText embeddingPrompt engineeringretrievalragquestion-answeringhallucinationreligious-textturkish

arXivPDFMarkdown8 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Chatbots built on large language models sometimes make things up, and that is risky when people ask about religion. This paper builds a chatbot that answers questions about the Quran and hadith from a fixed set of Turkish books instead of from memory.

When a question arrives, the system first looks up the passages in those books that seem most related to it. It then hands those passages, a set of standing instructions and the question to a language model. The answer is shown together with the passages it was based on, so a reader can check the source. The authors also tuned the instructions to keep answers respectful and on-topic.

The idea is a common pattern called retrieval-augmented generation. It matters because the model's answer is tied to text the system can point to, rather than to whatever the model remembers.

TL;DR

  • MufassirQAS is a retrieval-augmented chatbot for Turkish questions on the Quran and hadith §1 §4.
  • It splits a set of Turkish books into 1500-token chunks, embeds them, and sends the five most relevant chunks with the question to GPT-3.5 Turbo §5 alg. 1.
  • The source passages are shown beside each answer §4 fig. 5.
  • The evaluation is four screenshot comparisons against ChatGPT, judged by the authors; no accuracy, error or hallucination metric is reported §6 fig. 3 fig. 4 fig. 6 app. 4.
  • Use it as a reference for the pipeline and its design choices, not as evidence that the system is reliable.

Problem

Large language models state wrong facts, present old or generic information, and rarely name their sources §2.1 §2.2. The authors argue that this is especially harmful for religious questions, where misinformation has serious consequences §1. Fine-tuning on the same books is considered costly and less effective than retrieval for new information §4 §2.2. The paper assumes that the answer should be grounded in a known, authoritative set of Islamic texts, and that the answer should show which texts it used §1 §4.

Method

The system follows a standard retrieval-augmented generation design fig. 1 alg. 1. It has an offline indexing step and an online answering step.

Offline indexing. Source books are converted to text, cleaned and normalised, and tokenised §4, data pipeline paragraph. The text is split into chunks by a recursive character splitter with a chunk size of 1500 tokens and an overlap of 100 tokens §5. Each chunk is embedded with an OpenAI embedding model and stored in a Faiss vector index §5 §4.2. Each chunk carries metadata such as the book title, the author, the publisher and the source line range fig. 5.

Online answering. The user question is embedded with the same kind of model alg. 1, steps 2–3 §4 "Step 2: Embed Question". Retrieval combines two signals: a keyword score (TF-IDF) and embedding similarity §4 §4.3. The candidates are then re-ranked by a neural model §4. The top five chunks are kept §5. The system prompt, the retrieved chunks and the user question are combined into one prompt fig. 1 §4.4. The language model writes the answer from that prompt alg. 1, step 4 §5. The interface shows the answer with the chunks it used, including book, lines and, in the examples, the source references alg. 1, step 5 fig. 6 fig. 3b.

Knowledge base. The paper uses three open-access Turkish sources: a Turkish translation and commentary of the Quran §4.1, the Kütüb-i Sitte hadith collections §4.1 §2.3, and an Islamic catechism (İlmihal) §4.1 §2.3. Token statistics for each source are in tab. 2 and fig. 2.

Prompt design. The system prompt is meant to limit the answer to the given passages, set the tone, specify the response format and say when the model should decline to answer §4.4. The actual prompt text is not published §4.4.

Sample questions. Examples used in the paper are whether vaccination breaks a fast, whether men or women are superior, whether evolution is accepted in Islam, and the conditions for entering heaven fig. 3 fig. 4 fig. 6 app. 4.

Pseudocode of the online and offline steps, following alg. 1 and §5:

Pseudocode
# Offline: build the index  [alg. 1, steps 1]  [§5]
# Split each book into 1500-token chunks with 100 tokens of overlap
def build_index(books, embed_model):
    chunks = split_recursively(books, size=1500, overlap=100)
    # Embed each chunk and keep its book metadata  [fig. 5]
    vectors = [embed_model(chunk.text) for chunk in chunks]
    return vector_index(chunks, vectors)  # Faiss  [§5]

# Online: answer one question  [alg. 1, steps 2–5]
def answer(question, index, embed_model, llm, system_prompt):
    # Embed the question  [alg. 1, step 2]
    question_vector = embed_model(question)
    # Score candidate chunks by embedding similarity and TF-IDF  [§4]
    candidates = index.nearest(question_vector, limit=20)  # limit not stated
    candidates = rerank(question, candidates)  # re-ranker  [§4]
    # Keep the five most relevant chunks  [§5]
    top_chunks = candidates[:5]
    # Build the prompt: system text, chunks, question  [fig. 1] [§4.4]
    prompt = build_prompt(system_prompt, top_chunks, question)
    # Generate the answer and return it with its sources  [alg. 1, steps 4–5]
    answer_text = llm(prompt)  # GPT-3.5 Turbo, 16k context  [§5]
    return answer_text, top_chunks

The limit=20 value and the re-ranker are not stated in the paper; they are placeholders that mark a step the paper leaves open.

Hyperparameters

NameValueWhereNotes
Chunk size1500 tokens§5Chosen from token statistics tab. 2 fig. 2; tokeniser not stated
Chunk overlap100 tokens§5Higher overlap is motivated as keeping context §4.2
Text splitter separators"\n\n", "\n", ""§5Recursive character splitter; third separator printed as an empty string, whether it is a space is not stated
Chunks retrieved (top-k)5§5§4.3 says the count is the number of highest-scoring chunks but gives no fixed value there
Embedding modelOpenAI embeddings§5Model name and vector dimension not stated
Vector indexFaiss§5§6 says a Chroma store was used in the Colab proof of concept
Similarity metricnot stated§4.3
Keyword retrievalTF-IDF§4Weighting and how keyword scores are combined with embedding scores not stated
Re-rankerneural network re-ranker§4Model, input length and cut-off not stated
LLMChatGPT-3.5 Turbo, 16k context§5Name as written in §5; appendix shows ChatGPT 3.5 app. 1 app. 2
Temperature0.5§5Written in the same sentence as the embeddings; whether it applies to the LLM is not stated
Max output tokensnot stated
System promptnot stated§4.4Text not published
FrameworkLangChain, Flowise§5 [ref. 38] [ref. 39]Flowise used to display the retrieved chunks §5
HostingRailway, then Hugging Face Spaces§6Moved due to API-call issues §6
Hardwarelocal machine, Intel Core i5-12450H, 16 GB RAM§5No GPU used §5

Results

Author-reported. The paper compares MufassirQAS with ChatGPT on sensitive questions §6 fig. 3 fig. 4 fig. 6 app. 1–4. No quantitative metric is reported. The judgement is the authors' reading of the screenshots.

SettingMetricBaselineThis paperSource
Vaccination while fasting, Turkish and English versionsnone reported; qualitative readingChatGPT (version in the main text not stated)Says vaccination does not break the fast as a general ruling, advises consulting a scholar; source chips shown, no verse cited in the text fig. 3bfig. 3 app. 1
Whether men or women are superiornone reported; qualitative readingChatGPT (version not stated)States no inherent superiority; cites Quran 49:13 and 4:34 fig. 4b; the Turkish version also cites 4:32 and 58:1–4 app. 2fig. 4 app. 2
Whether evolution is accepted in Islamnone reported; qualitative readingNone shown; the text says ChatGPT tends to hedge, without a figureGives a direct answer and cites Quran 21:30 and 49:13 fig. 6 app. 3fig. 6 app. 3
Conditions for entering heavennone reported; qualitative readingNoneShows the source chunk used for the answer, no answer text reproduced in the cardapp. 4

The Related Works classification compares MufassirQAS with three other systems on training data, source display and hallucination handling tab. 1. This is a qualitative table, not a measurement.

Dataset token statistics (not results, but needed to set chunk size; tab. 2 fig. 2):

SourceMinMeanMedian90th percentileMax
Islam İlmihali27.00778.25782.50918.001585.00
Diyanet Kuran Yolu (Turkish translation and commentary)37.00737.19741.00900.001005.00
Hadislerle İslam5.001022.921155.001375.302914.00

Paper vs code

No official code. The paper has no code or project link §1–§8. A GitHub search for "mufassirqas" and "MufassirQAS RAG Quran" returned no repositories. A search for "mufassir" returned 43 repositories; the first 19 read were owned by accounts not matching any of the three authors, and no author-owned repository was found. The arXiv metadata has no code link. No discrepancies can be checked.

Pitfalls

  • No quantitative evaluation. The abstract says the system "demonstrated superior performance", but the paper reports no metric, no question set size, no labelled answers and no error rate §6 §8. The four examples were chosen by the authors and the baseline is a single chatbot whose version and settings are not fully stated §6 fig. 3–6.
  • Prompt text is missing. The system prompt that controls refusals, tone and format is described in words only §4.4. An implementer must write their own and test it; the behaviour in the screenshots cannot be reproduced exactly.
  • Retrieval details are unstated. The re-ranker model, the way TF-IDF and embedding scores are combined, the similarity metric and any score threshold are not described §4 §4.3. There is no minimum score for "I don't know": the top five chunks are always passed to the model §5.
  • Separator detail. The third splitter separator is shown as an empty string §5; the paper does not say whether this means no separator or a space. Test the split output on sample texts.
  • Page numbers versus line numbers. The abstract refers to source page numbers [abstract]. The chunk metadata in fig. 5 stores a line range (loc.lines.from, loc.lines.to), and the example shows no page number. Plan citations around line ranges unless the source format carries pages.
  • Dataset naming. The text names the hadith source as Kütüb-i Sitte §4.1; tab. 2 labels a column "Hadislerle İslam" and the mapping between the two is not stated. Check which book each column is before reusing the statistics.
  • Context fragmentation. The authors state that long chunks can hurt coherence and that multiple retrieved chunks can produce fragmented answers §7 §8.
  • Dataset licensing. The paper says the books are open access [Data Availability]. The Turkish translation and commentary is listed as a Diyanet publication in the chunk metadata fig. 5, and the licence of each text is not stated. Do not redistribute the text without checking each source's terms.
  • Infrastructure drift. §5 describes Faiss, while §6 describes a Chroma proof of concept on Colab and a hosted Flowise deployment. The final configuration is partly described §5 §6.
  • Cost and latency are not reported. No per-query cost, token count per answer or response time is given §5 §6.
  • Language. Only Turkish is used. Appendix versions are translated into English for reading; their wording is not the system's output in English app. 1–4.

Lineage

  • Builds on: retrieval-augmented generation as described by Lewis et al. 2020 [ref. 14] and Kulkarni et al. [ref. 27] for a RAG chatbot built on FAQ data.
  • Compared against: ChatGPT in the four sensitive-question comparisons §6 fig. 3–6; Al Usouly, QuranGPT and the Quran and Hadith dataset system of Altammami and Atwell tab. 1 [ref. 28] [ref. 32] [ref. 33].
  • Superseded by: not known from content in this wiki.

Source and license

Adapted from “Improving LLM Reliability with RAG in Religious Question-Answering: MufassirQAS” by Ahmet Yusuf Alan, Enis Karaarslan, Omer Aydin (arXiv:2401.15378), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.