Artificial Intelligence-Enabled Intelligent Assistant for Personalized and Adaptive Learning in Higher Education

In plain words

A university course scatters its information across many places: slides, reading lists, forum threads, announcements, assignment pages and hours of recorded lectures. Students who want a quick answer ("when is the midterm?", "what does the lecture say about engineering design?") have to dig through all of it.

This paper describes VirtualTA, a chatbot that sits inside the Canvas learning platform and answers from the course's own material. It turns lecture recordings into text with a speech recogniser, cuts every document into short pieces, and stores a numerical fingerprint (an embedding) of each piece. When a student asks something, the question gets a fingerprint too, the closest pieces are pulled out, and a GPT-3.5 model writes the answer from them, showing which file it used. If nothing close enough is found, it says the answer was generated on its own and suggests asking an expert.

Around this core the authors built study tools (flashcards, self-graded quizzes, summaries, a code runner in the chat) and instructor tools (a grader that scores student answers against a key, question generation, and a guard that refuses to solve problems the instructor marked as homework).

It is a description of a built system, not an experiment: the paper shows screenshots and one grading example but reports no accuracy measurements or student studies.

TL;DR

VirtualTA is a retrieval-augmented course assistant: LMS content and Whisper transcripts are chunked (~800 characters), embedded with text-embedding-ada-002 (1536-d), retrieved by cosine similarity (top 10, keep those above 75%), and passed as context to OpenAI GPT-3.5-family models, with a fine-tuned Davinci query classifier routing each request to a feature (QA, summary, flashcards, quiz, code, question generation, homework refusal). The paper is useful as an architecture checklist for an LMS-integrated tutor bot. It contains no benchmark, ablation or user study, the prompts and fine-tuning data are not published, and there is no official code.

Problem

Text-heavy, qualitative courses need individual, on-demand support that instructors cannot give at scale §1. The authors list gaps in prior work on virtual teaching assistants: little integration with learning management systems (LMS), class recordings and interactions are not used as knowledge, tools rarely serve both students and instructors, and cheating prevention is rarely addressed §2.

Method

VirtualTA (the paper also calls the framework AIIA) has four layers §3.2 fig. 1:

1. Data retrieval. Course resources come from Canvas through a JavaScript LMS library; recordings are transcribed with Whisper and diarised with pyannote §3.2 §3.2.4. Resource types and their roles are listed in tab. 1 (reproduced below).
2. Core framework. Embedding management, LLM fine-tuning and context generation on a NodeJS backend with relational and vector databases fig. 1 §3.2.4.
3. Intelligent services. Student-facing (QA, flashcards, quizzes, summaries, code sandbox, context-aware chat) and instructor-facing (auto-evaluator, homework detection, question generation) features §4.
4. Communication. A web chat embedded in Canvas, plus a React instructor dashboard §3.2.4.

Models §3.1: text-davinci-003 (described as a GPT-3 model built on InstructGPT), gpt-3.5-turbo for text completions, a fine-tuned Davinci model for query classification, and a fine-tuned Curie model for open-ended question generation. Embeddings come from text-embedding-ada-002 with 1,536 dimensions §3.1.1.

Course resources tab. 1:

Resources are categorised and prioritised: primary resources such as lectures rank above secondary ones such as external readings, so answers are sought in the most important sources first, and the source used is disclosed to the user §3.2.1. The priority mechanism itself is not specified.

Knowledge base generation §3.2.2 fig. 2:

# Knowledge base build  [§3.2.2] [fig. 2]
# Input:  course resources pulled from the LMS, plus lecture recordings
# Output: list of (chunk_text, embedding, metadata)

def build_knowledge_base(course):
    knowledge_base = []
    # Acquire files, recordings, assignments, discussions, quizzes,
    # announcements from the LMS  [§3.2.2]
    resources = lms_fetch_all(course)
    for resource in resources:
        # Recordings become text via Whisper; diarisation via pyannote
        # [§3.1.2] [§3.2.4]
        if resource.is_recording:
            # Whisper API limit is 25MB: split or compress first  [§5.1]
            parts = split_or_compress(resource.media, max_bytes=25 * 2**20)
            resource.text = join(whisper_transcribe(p) for p in parts)
        # Structured LMS JSON (announcements, assignments) skips parsing
        # [§3.2.2]
        if resource.is_structured_json:
            chunks = [resource.json_text]
        else:
            # ~800-character chunks that keep words and sentences intact
            # [§3.2.2]
            chunks = split_text(resource.text, target_chars=800,
                                keep_sentences=True)
        for chunk in chunks:
            # 1536-d text-embedding-ada-002 vector  [§3.1.1]
            vector = embed_ada_002(chunk)
            knowledge_base.append((chunk, vector, resource.metadata))
    return knowledge_base

Query handling §3.2.3:

# Query pipeline  [§3.2.3]
# Input:  user query string, knowledge base, chat history
# Output: response text plus disclaimer (source file, confidence)

def answer(query, knowledge_base, history):
    # Classify type and intent with a fine-tuned Davinci model  [§3.1] [§3.2.3]
    intent = finetuned_davinci_classify(query)
    # Embed the query and score every chunk by cosine similarity  [§3.2.3]
    query_vector = embed_ada_002(query)
    scored = [(cosine(query_vector, v), chunk, meta)
              for chunk, v, meta in knowledge_base]
    # Keep the 10 best, then only those above 75% similarity
    # [§3.1.1] [§3.2.3]
    top = sorted(scored, descending=True)[:10]
    context = [item for item in top if item.score > 0.75]
    # Homework guard: refuse and point to a resource instead  [§4.2.2]
    if intent_is_flagged_homework(query, context):
        return refuse_and_point_to(context)
    # Prompt the GPT-3.5 model with context; prompts are designed to
    # say "I'm not sure" instead of guessing  [§3.1.1] [§3.2.3]
    response = gpt35_generate(intent, query, context, history)
    # Attach disclaimer with source and confidence; with no matching
    # documents, say the answer was generated autonomously  [§4.1.3]
    return with_disclaimer(response, context)

Steps 3 and 4 are run as written: the paper says it builds a list of the ten documents with the highest similarity and then keeps those above 75% §3.1.1 §3.2.3. How "confidence" in the disclaimer is computed is not stated; screenshots show a percentage next to the source filename fig. 5 fig. 7.

Features §4:

• Flashcards: true/false and open-ended cards on a requested topic, each answer with an explanation §4.1.1 fig. 3.
• Quiz and auto-grading: true/false and open-ended questions; the system grades a submitted answer and explains when it is wrong §4.1.2 fig. 4.
• Question answering on administrative and content topics, with a disclaimer giving source and confidence §4.1.3 fig. 5.
• Code sandbox: the chat detects the programming language and, on "run code", opens an in-chat environment §4.1.4; the screenshot shows an online compiler credited to JDoodle running MATLAB code fig. 6.
• Summarisation of a topic from the knowledge base, with the same disclaimer §4.1.5 fig. 7.
• Context-aware conversation: the bot mirrors the student's communication style, detects emotional state, and manages history within the token limit by "dynamic rewriting and rotation" §4.1.6 fig. 8.
• Auto-evaluator (instructor): given questions, a key and student answers (PDF upload supported), it scores each question with an explanation §4.2.1. The display colours scores ≤ 2 red, ≤ 5 yellow,

  5 green §4.2.1.

• Homework detection: instructors mark assignments as off-limits; matching queries get pointers to resources instead of answers §4.2.2 fig. 10.
• Question generation: instructors request True/False, Multiple Choice or Open-Ended items §4.2.3.

Infrastructure §3.2.4: NGINX web server, NodeJS backend, PostgreSQL, caching, user and course management modules, a service-oriented design that plugs into any web platform supporting webhooks, and a Python Flask Speech API (Whisper + pyannote) with endpoints for Canvas file URLs and YouTube or other URLs, each with or without timestamps.

Hyperparameters

Results

The paper reports no quantitative evaluation: no retrieval accuracy, answer correctness, latency, cost or user study. "Results" §4 consists of feature descriptions and screenshots fig. 3 to fig. 10.

The one worked example is the auto-evaluator on ten answers from one student fig. 9:

No human grades are given for comparison; the authors only note that the scoring "may occasionally be lenient" §4.2.1. Limitations reported: no OCR for scanned PDFs, LMS APIs lacking standard export formats, the Whisper 25MB limit, and having to keep up with model and API updates §5.1.

Paper vs code

No official code.

Pitfalls

• Retrieval order is ambiguous but stated. Top-10 first, then the > 0.75 filter §3.2.3, so at most 10 chunks reach the prompt and possibly none. The 0.75 value is not justified or tuned in the paper; calibrate it for your own embedding model and corpus.
• Prioritisation is undefined. Lectures are said to outrank external readings §3.2.1, but nothing says whether this is a score boost, a tiered search or a filter. The pseudocode above does not implement it.
• Structured JSON is embedded unchunked §3.2.2. Long announcements or assignment descriptions may exceed what fits well in one embedding; the paper does not address this.
• "Confidence" is unexplained. Disclaimers show a percentage fig. 5 but its source is not stated. Do not assume it is the cosine score.
• Hallucination control is prompt-only ("I'm not sure" responses) §3.1.1 §3.2.3; prompts are not published and its effectiveness is not measured.
• Homework detection is described only as "a sophisticated algorithm" §4.2.2. The screenshots show the refusal citing an assignment file with a confidence value fig. 10, which suggests retrieval against flagged assignments, but the paper does not say so.
• Fine-tuned classifier and generator need training data that is not described; the OpenAI text-davinci-003, Davinci and Curie fine-tuning endpoints the paper uses §3.1 may need substitutes in a reimplementation.
• Auto-grader leniency. The authors say scores can be lenient §4.2.1 and that the tool should not replace instructor grading.
• Long recordings must be split or compressed before the Whisper API §5.1; splitting can cut words or speaker turns at boundaries, and diarisation alignment across parts is not discussed.
• Conversation memory ("dynamic rewriting and rotation") is named but not specified §4.1.6.
• Scanned PDFs yield no text without OCR, which the system does not have §5.1.

Lineage

• Builds on: the authors' earlier GPT-3-based, curriculum- and syllabus-oriented educational assistant framework (Sajja et al., no arXiv id given) §2; Whisper ASR (Radford et al., 2022, arXiv:2212.04356) §3.1.2; InstructGPT (Ouyang et al., 2022, arXiv:2203.02155) via text-davinci-003 §3.1.
• Compared against: no baselines; related virtual teaching assistant work is discussed qualitatively §2 §5.