Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey

In plain words

A general-purpose chatbot model knows a bit about everything but is rarely expert in one field. Law, medicine, finance and similar areas use their own vocabulary, rules and facts, and those change over time. This survey asks how to make a general model good at one such field and sorts the known ways into three groups, by how much access you have to the model.

The first group leaves the model untouched and gives it outside help: documents to read from, or tools it can call. The second group also leaves the weights alone but changes what you write to the model, through carefully designed instructions, or through learned "soft" input vectors. The third group changes the model itself, by training small add-on modules or by updating parts of its weights.

The survey does not add a new method or run experiments. Its value is the map: a taxonomy, a common four-step way of thinking about any specialization project, a tour of example application areas, and a list of problems that remain open.

TL;DR

A survey, not a method paper. It classifies LLM domain specialization by the access one has to the model: black box (external augmentation), grey box (prompt crafting) and white box (model fine-tuning) §2.2 fig. 1. It describes each class with example techniques §3 §4 §5, gives a four-stage framework of definition, augmentation, optimization and evaluation §2.2, and lists open challenges §7. No benchmark numbers are reported.

Problem

Three problems are named as the reason domain specialization is hard §1. First, a model's knowledge stops at its training cutoff, and keeping it current is costly. Second, one model cannot hold deep knowledge of many specialized domains, since widely discussed topics dominate training data, and models can hallucinate on specialized queries. Third, adapting a large model needs task-specific data, careful choice of training settings, and large compute, and fine-tuning may cause catastrophic forgetting of earlier knowledge §1.

Method

The survey proposes an organising scheme rather than an algorithm.

Taxonomy by access §2.2 fig. 1 fig. 2.

1. External augmentation (black box). Only the model's input or output is changed. Domain knowledge or domain tools are added from outside §3. The authors state that this needs no access to the inner parameters §2.2.
2. Prompt crafting (grey box). Prompts are designed with some access to model signals such as gradients or loss values §2.2 §4.
3. Model fine-tuning (white box). Parameters are updated, either through adapters or through task-oriented training §5. This needs the most access and resources §2.2.

The survey also mentions other ways to classify the same work: by training strategy, by intervention stage (pre-training, fine-tuning or inference), and by evaluation or feedback type §2.2. It does not use these for its main structure.

Common framework §2.2. Any specialization project goes through four stages:

1. Definition. Fix the domain, its objectives and its constraints, and identify the data and resources.
2. Augmentation. Add domain knowledge to the model or to its inputs and outputs.
3. Optimization. Improve the augmented system, by gradient descent, by prompt engineering, or by post-processing, depending on the class.
4. Evaluation. Test against domain benchmarks or expert feedback, and refine.

External augmentation §3. Two sub-classes.

• Domain knowledge augmentation §3.1. Explicit knowledge is a document collection or a relational store, read through a retriever: the query and the knowledge are embedded and matched by similarity. The paper notes that retrieved context should win over the model's memorised knowledge when they conflict §3.1. Implicit knowledge is held as latent vectors or a memory, and is read with attention over stored entries §3.1. The authors say implicit augmentation is less used in practice §3.1.
• Domain tool augmentation §3.2. The LLM calls external software such as APIs, solvers, simulators or search engines. The basic loop is in the pseudocode below. The survey also covers LLMs as task planners that decompose work across several tools §3.2, and LLMs embedded as agents in robots or simulated environments §3.2.

# LLM calls a domain tool  [§3.2]
# Input:  query; LLM f; domain tool T; prompt p
# Output: answer y
def call_domain_tool(query, llm, tool, prompt):
    # 1. Ask the LLM for an executable command  [§3.2]
    command = llm(prompt, query)
    # 2. Run the command in the domain tool  [§3.2]
    result = tool(command)
    # 3. Post-process the result by rule or by the LLM  [§3.2]
    return post_process(result)

Prompt crafting §4.

• Discrete prompts §4.1. Plain-language instructions. Zero-shot prompts give only the task description. Few-shot prompts add worked examples that show the output format §4.1 fig. 4 fig. 5. Named techniques include chain-of-thought, its zero-shot form that adds a "think step by step" cue, automatic chain-of-thought generation, ensemble-based instructions that aggregate several outputs, and recursive instructions that split a task into subtasks §4.1.
• Continuous (soft) prompts §4.2. Learned token vectors attached to the input. They are trained on downstream data while the model stays frozen. The survey splits them into task-dependent and instance-dependent prompts §4.2 fig. 6.

Model fine-tuning §5.

• Adapter-based fine-tuning §5.1. Small trainable modules are inserted while the original weights stay frozen. Neural adapters are bottleneck feed-forward blocks, first with down-projection, GeLU and up-projection §5.1. Low-rank adapters (LoRA) learn a low-rank update; DyLoRA and KronA are named as variants §5.1. Integrated frameworks combine several adapters, for example AdapterFusion, UniPELT and Polytropon §5.1.
• Task-oriented fine-tuning §5.2. Two kinds are described.
  • Instruction-based knowledge update: fine-tuning across many tasks written as explicit instructions, the same idea as instruction tuning §5.2 fig. 8.
  • Partial knowledge update: only a chosen subset of weights is updated, through a mask eq. 1. The survey covers knowledge editing, gradient masking, and knowledge distillation into smaller models under this heading §5.2.

\tilde{\Theta} = \Theta + \nabla f_{\Theta}(D) \odot T

where T^{(i)} = 1 if \theta^{(i)} \in \Theta_T and 0 otherwise eq. 1. The mask T selects the parameters \Theta_T that may change in each step §5.2.

Applications §6. The survey groups domains into social sciences (education, finance, law), natural sciences (biomedicine, earth science) and formal sciences (human-computer interaction, software engineering, cybersecurity) §6. The main text gives one paragraph per domain. Per-domain detail is in the supplementary material, which is commented out in the source (see Pitfalls).

Hyperparameters

Not applicable. The survey trains nothing and states no hyperparameters of its own §1 §2.2. Learning rates, ranks, adapter sizes and similar settings appear only as examples inside the cited works, and this card does not carry them over.

Results

Not applicable. The paper reports no experimental results of its own. Its contributions are the taxonomy, the application overview and the discussion of open problems §1. The paper gives no quantitative comparison between the three classes; trade-offs are described qualitatively: augmentation and prompting are cheaper but may give smaller gains, and fine-tuning gives larger gains at higher cost and risk of overfitting §2.2.

Paper vs code

No official code. The paper names no code repository and no project page. A GitHub repository search for "domain specialization llm survey" returned zero results. A second query on the same API returned HTTP 422, so no further search was made. A commented-out line in the introduction source points to a third-party awesome-list of LLM resources §1. That is not the authors' code, and it is not used here.

Pitfalls

• Nothing to implement. This is a map of techniques. Each cited method has its own paper, which is the place to look for its details.
• Supplementary application text is absent. The per-domain sections of the application chapter are commented out of the main source §6. The main text covers each domain in one paragraph only.
• Taxonomy is by access, not by performance. The three classes say how much of the model one needs to touch. The survey does not rank them on accuracy, and it says some adapter methods were not claimed as domain methods by their authors §5.1.
• Mask equation is loose as printed. Equation 1 uses a plus sign with a gradient of f_\Theta(D) and does not define a loss. An implementer should read it as a masked gradient step. The sign convention and loss are not stated in the paper §5.2 eq. 1.
• Literature cut-off. The arXiv version used here is v7, revised 2024-03-29 [raw metadata]. Techniques released after that are not covered. The survey's own date-sensitive claims about models should be checked against current sources.
• Open-challenge lists are the authors' view. Challenges are listed per class §3.1 §3.2 §4.1 §5.1 §5.2 and across classes §7. They are not sourced from measurements.
• Cross-class interaction is asserted, not tested. The paper says the three classes can be combined, for example retrieval with a fine-tuned model §2.2. It gives no experiment for this.

Lineage

• Builds on: general LLM and pre-trained language model surveys and domain-adaptation surveys, cited in the related-work section §1. Names are not repeated here.
• Compared against: the related surveys in §1 are compared by scope. The paper states that none gives a technical taxonomy of LLM domain specialization.
• Superseded by: not known from content in this wiki.