Paper · 2024-02-04 · cs.CL, q-fin.GN
A Survey of Large Language Models in Finance (FinLLMs)
Survey of large language models in finance. Traces BERT- and GPT-style models to finance versions (FinBERT, FLANG, BloombergGPT, FinMA, InvestLM, FinGPT), compares five build techniques, reviews six benchmark tasks and lists eight advanced ones with datasets. It runs no experiments: headline results are compiled from other papers.
Language model pretrainingDomain adaptationInstruction tuningLLM evaluationsurveyfinance
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Large language models, the systems behind chatbots, have been studied widely in general, but much less for finance. This survey follows how language models moved from small general models to versions adapted to money-related text. It sorts the ways teams have built finance models: training further on finance text, training from scratch on finance text, mixing general and finance text, or teaching a general model to follow finance instructions. It then lists the tests people use to score these models, plus harder tasks that lack good benchmarks. The authors close with the problems they see: made-up answers, the privacy of private data, and the cost of running large models.
TL;DR
This is a survey, not a method: it runs no experiments and gives no code to reimplement. It traces general models (BERT, the GPT series, LLaMA, BLOOM) to finance models (FinBERT, FLANG, BloombergGPT, FinMA, InvestLM, FinGPT) §2 tab. 1, compares five build techniques §3, and summarises six benchmark tasks plus eight proposed advanced ones §4 §5. Use it to choose a model family or a benchmark. Do not take its numbers as measured here: they are compiled from other papers §4.
Problem
General-purpose LLMs are widely studied, but financial LLM research is at an early stage §1. The survey asks how finance models evolved from general ones, which techniques they use, how they are evaluated, and what remains open §1. Two problems it names are using internal data without breaching privacy or security, and getting answers that users can trust §6. The scope is English-language models only §1.
Method
The paper gives no algorithm and no equations. Its implementable content is the training and tuning recipes it describes, which this section maps.
- Lineage of general models §2.1.
- Pre-training is either discriminative (masked language modelling, encoder-only or encoder-decoder) or generative (next-token prediction, decoder-only) §2.1. The text says the discriminative case predicts "the next sentence", which conflates two objectives; see Pitfalls.
- GPT series: GPT-1 (110M parameters), GPT-2 (1.5B), GPT-3 (175B), which the authors credit with in-context learning §2.1.1. ChatGPT combines GPT-3, Codex and InstructGPT, which uses RLHF §2.1.1. GPT-4 is given as an estimated 1.7T parameters §2.1.1.
- Open models: BERT base (110M) §2.1.2; LLaMA (7B, 13B, 33B, 65B), released February 2023, followed by many variants that use instruction fine-tuning and chain-of-thought prompting §2.1.2; BLOOM (176B), trained on 46 natural and 13 programming languages §2.1.2.
- Finance models §2.2 fig. 1. The paper groups them into four FinPLMs and four FinLLMs. The FinPLMs are FinBERT-19, FinBERT-20 and FinBERT-21 (BERT-based) and FLANG (ELECTRA-based). The FinLLMs are FinMA, InvestLM and FinGPT (LLaMA or other open bases) and BloombergGPT (BLOOM-style, closed) §2.2.
- Five build techniques §3 fig. 2 tab. 1.
- 3.1 Continual pre-training. Take an existing general model and train it further on domain data. FinBERT-19 does this in three steps: start from general BERT, continue pre-training on a financial corpus, then fine-tune on a financial task §3.1.
- 3.2 Domain-specific pre-training from scratch. Train on an unlabelled domain corpus with the original architecture and objective. FinBERT-20 is pre-trained on a 4.9B-token financial communication corpus and ships a financial vocabulary §3.2.
- 3.3 Mixed-domain pre-training. Train on general and domain corpora together. FinBERT-21 uses six self-supervised pre-training tasks §3.3. FLANG masks financial keywords and phrases and follows ELECTRA's training strategy §3.3.
- 3.4 Mixed-domain LLM with prompt engineering. The model is pre-trained on general and finance data and then used frozen, with task descriptions and examples only. BloombergGPT draws on 345B general tokens and 363B financial tokens §3.4. The survey does not cover prompt engineering itself §3.4.
- 3.5 Instruction-tuned LLM with prompt engineering. Instruction tuning comes in two parts: building instruction datasets, and fine-tuning LLMs on them §3.5. Existing financial datasets are converted into instruction data. FinMA fine-tunes LLaMA 7B and 30B on a 136k-sample instruction set built from nine public financial datasets across five tasks §3.5. InvestLM fine-tunes LLaMA-65B on a curated instruction set; its footnote says the dataset is unavailable and the model is released §3.5. FinGPT is a data-centric framework with data APIs, an instruction dataset and fine-tuned models; its experiments use LoRA on six open-source LLMs §3.5.
# Instruction-tuned finance LLM, as the survey describes it [§3.5]
# Input: base_model (a general LLM); datasets (labelled finance tasks)
# Output: tuned_model (a finance LLM)
def build_finance_llm(base_model, datasets):
# Convert each labelled finance dataset into instruction pairs [§3.5]
instruction_data = []
for dataset in datasets:
instruction_data += to_instructions(dataset)
# Fine-tune the base model on the instructions; FinGPT uses LoRA [§3.5]
tuned_model = finetune(base_model, instruction_data)
return tuned_model- Six benchmark tasks §4. Each task lists its datasets and the results compiled for them (see Results). Sentiment analysis uses FPB and FiQA-SA §4.1. Text classification uses the gold-news Headline set, with FedNLP, FOMC and Banking77 added §4.2. Named entity recognition uses FIN §4.3. Question answering uses FinQA and ConvFinQA, not FiQA-QA §4.4. Stock movement prediction uses StockNet, CIKM18 and BigData22 §4.5. Summarisation uses ECTSum, with MultiLing 2019 added §4.6. Extra datasets for SA, TC and NER are listed as well §4.1 §4.2 §4.3.
- Eight advanced tasks §5: relation extraction (FinRED), event detection (EDT), causality detection (FinCausal20), numerical reasoning (FiNER-139, FinQA, ConvFinQA), structure recognition (FinTabNet), multimodal understanding (MAEC, MONOPOLY), machine translation (MINDS-14, MultiFin) and market forecasting (StockEmotions, EDT, MAEC, MONOPOLY). The paper gives dataset descriptions only and no results for these.
- Outlook §6: datasets (instruction-tuned and multimodal finance data), techniques (retrieval-augmented generation for private data, reliability and fewer hallucinations), evaluation (expert review and finance-specific metrics), implementation (cost against performance, parameter-efficient tuning, CI/CD) and applications (robo-advisors, quantitative trading, low-code tools, report generation).
Hyperparameters
The paper specifies no training hyperparameters: learning rate, batch size, epochs, LoRA rank and warm-up are not stated §3. The table below gives the model sizes and pre-training data as the survey reports them tab. 1.
| Model | Backbone | Parameters | Pre-training data (as stated) | Tuning (as stated) | Source |
|---|---|---|---|---|---|
| FinBERT-19 | BERT | 0.11B | general 3.3B words; financial 29M words | post-pre-training, fine-tuning | tab. 1 §3.1 |
| FinBERT-20 | BERT | 0.11B | financial 4.9B tokens | pre-training, fine-tuning | tab. 1 §3.2 |
| FinBERT-21 | BERT | 0.11B | general 3.3B words; financial 12B words | pre-training, fine-tuning | tab. 1 §3.3 |
| FLANG | ELECTRA | 0.11B | general 3.3B words; financial 696k documents | pre-training, fine-tuning | tab. 1 §3.3 |
| BloombergGPT | BLOOM | 50B | general 345B tokens; financial 363B tokens | pre-training, prompt engineering | tab. 1 §3.4 |
| FinMA | LLaMA | 7B, 30B | general 1T tokens | instruction tuning, prompt engineering | tab. 1 §3.5 |
| InvestLM | LLaMA | 65B | general 1.4T tokens | instruction tuning, prompt engineering, PEFT | tab. 1 §3.5 |
| FinGPT | six open-source LLMs | 7B | 2T tokens for the base model (e.g. LLaMA2) | instruction tuning, prompt engineering, PEFT | tab. 1 §3.5 |
Results
Not measured by the authors. The survey compiles results from original studies and from analysis studies §4, and it takes task-specific state-of-the-art (SOTA) results from the task-specific models §4. Shot settings are given in the rows where the survey states them; prompts and the evaluation protocol are not stated.
| Setting | Metric | Model | Value | Source |
|---|---|---|---|---|
| Sentiment analysis (FPB, FiQA-SA) | F1 | FLANG-ELECTRA | 92% | §4.1 |
| Sentiment analysis | F1 | FinMA-30B; GPT-4 (5-shot) | 87% each | §4.1 |
| Text classification (gold-news Headline), avg | F1 | FLANG-ELECTRA; FinMA-30B (5-shot) | 98% | §4.2 |
| Text classification, avg | F1 | BERT; FinBERT-20 | 97% | §4.2 |
| Named entity recognition (FIN) | entity F1 | GPT-4 (5-shot) | 83% | §4.3 |
| Named entity recognition | entity F1 | FLANG-ELECTRA | 82% | §4.3 |
| Named entity recognition | entity F1 | other FinLLMs | 61–69% | §4.3 |
| Question answering (FinQA, ConvFinQA) | exact match | GPT-4 (zero-shot) | 69–76% | §4.4 |
| Question answering | exact match | human experts (avg); general crowd | 90%; 47% | §4.4 |
| Question answering | exact match | BloombergGPT | 43% | §4.4 |
| Stock movement, avg of 3 datasets | accuracy | task SOTA; GPT-4 (zero-shot); FinMA | 58%; 54%; 52% | §4.5 |
| Summarisation (ECTSum) | ROUGE-1 | task SOTA; GPT-4 (zero-shot) | 47%; 30% | §4.6 |
Author-reported, compiled from other papers §4. The discussion says finance-tuned BERT-style models are strongest on SA, TC and NER, task SOTA models beat all LLMs on QA, SMP and Summ, and GPT-4 is strong everywhere except Summ §4.7.
Paper vs code
No official code. The abstract and a footnote link https://github.com/adlnlp/FinLLMs. Fetching it shows a curated list of resources, papers, benchmarks and datasets, with no model or training code, and the GitHub search for "FinLLMs" under that account returns only that repository. The paper names no project page. Other finance models it covers have their own repositories, which are third-party to this survey and not used here.
Pitfalls
- Compiled, not reproduced. Every result in §4 is taken from other papers, and the survey does not re-run any of them §4. The rows mix models, tasks and shot settings, and the survey does not state a common evaluation protocol or prompt templates. Verify a number at its original source before you cite it.
- Stale snapshot. The open-source column reflects availability as of December 2023 tab. 1. The FinBERT-21 link did not resolve as of December 2023 §3.3.
- Technique count is ambiguous. The abstract and contributions list five techniques across four FinPLMs and four FinLLMs §1. Section 3 says three techniques for the FinPLMs and two for the FinLLMs §3. Use the five headings in §3.1 to §3.5, not a count.
- Units differ within the paper. The general BERT corpus is "3.3B tokens" in the text §3.1 but "3.3B words" in the table tab. 1. The FPB dataset is called "articles" in the text, while annotation is said to cover "each sentence" §4.1. Treat both as unresolved.
- Objective description. §2.1 says masked language modelling predicts the next sentence. Masked modelling predicts masked tokens; next-sentence prediction is a separate objective. This card follows the objectives as the paper names them without adopting that sentence.
- FinBERT-21 details are thin. The text mentions six self-supervised pre-training tasks but does not name them, and it gives no corpus size §3.3; the 12B-word financial corpus appears only in the table tab. 1. Implementing FinBERT-21 requires the original paper.
- Two citation keys for FinGPT. The table and §2.2 cite the experiment paper for FinGPT, while §3.5 cites the framework paper and the experiment paper separately. Check which one you intend to cite.
- Metrics mismatch. The authors say accuracy and F1 are not enough for stock movement prediction, and that financial metrics and backtests are needed §4.5. The survey reports none of these.
- Hidden costs. The survey names a cost-versus-performance trade-off between general prompting, task-specific models and building a finance LLM, but gives no cost figures §6.
- Figure and table numbers are inferred. The source has one table and two figures in order of appearance; "Table 1", "Figure 1" and "Figure 2" follow that order. Figure 1 is the timeline; Figure 2 compares the techniques fig. 1 fig. 2.
- Unavailable datasets. The InvestLM instruction dataset is unavailable §3.5. Advanced-task datasets are listed without results §5.
Lineage
- Builds on: the general models in §2 (BERT, the GPT series, LLaMA, BLOOM), cited by the paper's own keys; no arXiv ids appear in the text.
- Compared against: ChatGPT and GPT-4 on the six benchmark tasks §4.
- Superseded by: not stated.
Source and license
Adapted from “A Survey of Large Language Models in Finance (FinLLMs)” by Jean Lee, Nicholas Stevens, Soyeon Caren Han, Minseok Song (arXiv:2402.02315), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.