Revolutionizing Finance with LLMs: An Overview of Applications and Insights

In plain words

Finance runs on a lot of text: news, earnings reports and analyst reports. Large language models (LLMs) are good at reading and writing text. This paper asks where that skill helps in finance, and tests one such model on a set of finance tasks.

The authors walk through four areas: trading and portfolio work, forecasting (mergers, bankruptcy, market direction), risk work (credit, ESG ratings, fraud, compliance) and answering finance questions for learners and customers. For most areas they describe ideas and imagined examples rather than measured results.

For the test, they give a single model, GPT-4, a fixed instruction format and ask it to label sentiment, pull out names, answer questions over earnings reports and predict stock direction. The model does best on text labelling and lower on numeric question answering and on price direction. The results are a snapshot of one model and one prompt style, not a benchmark that other systems can be compared against.

TL;DR

• A survey of LLMs in finance, organised into financial engineering, forecasting, risk management and financial question answering §3.
• A small evaluation of GPT-4 on six datasets, with zero-shot or one-shot prompts and no baseline §4 §5 tab. 2.
• Strongest on text tasks (sentiment 0.78 to 0.79 accuracy, NER 0.81 entity F1); weaker on numeric QA (0.64 and 0.73) and stock direction (0.53) tab. 2.
• Prompts, model snapshot, decoding settings and sample selection are not stated, so the numbers cannot be reproduced from the paper alone §4.2.

Problem

The paper's concern is that finance needs accuracy and reliability, and that ensuring LLM output meets that bar is a major challenge §1. Traditional models are task-specific, transfer poorly across tasks, and struggle with unstructured text such as news and social media §3.1.1 §3.3.1. The authors argue that a general LLM could cover many finance tasks through natural-language instructions, and they ask how well GPT-4 does on them §1 §4.

The problem statement has no formal definition. The survey does not state a selection rule for the 11 tasks it names §7 or for the six datasets it evaluates §4.1 tab. 1.

Method

This is a survey plus a light evaluation, so there is no training procedure and no new model. The "method" is the procedure below.

1. Scope the survey §3. Organise finance work into four groups: financial engineering (quantitative trading, portfolio optimisation, robo-advisors) §3.1; financial forecasting (merger and acquisition, insolvency, market trend) §3.2; financial risk management (credit scoring, ESG scoring, fraud detection, compliance checks) §3.3; and real-time question answering, illustrated by financial education §3.4. Each area is a narrative with hypothetical examples §3.2.1 §3.2.2.

2. Choose datasets §4.1 tab. 1. Six public or licensed datasets cover sentiment (Financial PhraseBank and FiQA-SA), named entity recognition on financial agreements, question answering over earnings reports (FinQA and ConvFinQA) and stock movement (BigData22) §4.1. Table 1 lists raw and "instruction" counts per dataset. How the instruction counts were produced is not described.

3. Define labels and metrics §4.1 tab. 2. Financial PhraseBank uses three sentiment classes. FiQA-SA is described as a continuous score from -1 to 1, but is reported as accuracy. NER uses three entity types: location, organisation and person. FinQA and ConvFinQA use exact-match accuracy. The stock task is binary: a move above +0.55% is positive and a move below -0.5% is negative §4.1. See the pseudocode below.

4. Build the prompt §4.2. The prompt has three parts, a role description, a required output format and an example with its expected answer. The first three parts go into the system message for each query. The authors also tried extra parts, which did not substantially change results; they do not quantify this §4.2. Prompt text itself is not published.

5. Choose the prompting variant §4 §4.2. The paper names three strategies: plain zero-shot, zero-shot with chain-of-thought wording, and one-shot. Section 4 describes the evaluation as one-shot learning and zero-shot prompting. Which strategy produced each number in Table 2 is not stated.

6. Query the model §3.2.3 §4. GPT-4 is called through its API §3.2.3. Model snapshot, temperature, maximum output length and retry rules are not stated.

7. Report. One score per dataset tab. 2. Several figures show single model answers as case studies fig. 3 fig. 4 fig. 5 fig. 6 fig. 7 fig. 8 fig. 9; these are illustrations, not measurements, except where a caption gives a count.

The loop below reconstructs the evaluation from the description in §4. It is not code from the paper.

# Zero-shot or one-shot scoring of GPT-4 on one dataset  [§4] [§4.2]
# Input:  dataset: list of (input_text, gold_label)
# Output: one score for the dataset (accuracy, entity F1 or EmAcc)  [tab. 2]

def evaluate_task(dataset, system_message, metric):
    # Collect one model answer per query  [§4.2]
    predictions = []
    for input_text, gold_label in dataset:
        # One request per query; the paper gives no batching or retry rule
        reply = call_gpt4(system_message, input_text)
        predictions.append(parse_label(reply))
    # Parsing rule for free-text replies is not stated  [§4.2]
    gold_labels = [label for _, label in dataset]
    return metric(predictions, gold_labels)

The stock-movement labelling rule from §4.1 is written out below.

# Binary label for one stock move, in percent  [§4.1]
# Input:  move_percent: price change over the window  [§4.1]
# Output: "positive", "negative", or "not stated"

def stock_label(move_percent):
    # Positive above +0.55%  [§4.1]
    if move_percent > 0.55:
        return "positive"
    # Negative below -0.5%  [§4.1]
    if move_percent < -0.5:
        return "negative"
    # Moves between -0.5% and +0.55% are not assigned a class  [§4.1]
    return "not stated"

Key equation, from the background section, the scaled dot-product attention used by transformer LLMs eq. 1. The equation is written in lowercase and without the transpose on the key matrix; see Pitfalls.

Attention(q, k, v) = softmax\left(\frac{qk}{\sqrt{d_k}}\right) v eq. 1

Hyperparameters

Results

Author-reported. Setup: GPT-4 with zero-shot or one-shot prompting on six finance datasets. No baseline model is in the table, and no hardware is stated §4 tab. 2.

Table 2 has one model column, labelled GPT 4, although its caption refers to "different LLMs" tab. 2.

Paper vs code

No official code.

Searched: the LaTeX source has no code, repository or project-page link (raw/src/2401.11641/main.tex); the arXiv abstract page lists no code repository; a GitHub repository search for the title's distinctive words returned only unrelated third-party projects. The paper gives no prompts or code for its evaluation, so the procedure above is a reading of the text.

Pitfalls

• Not reproducible from the text. The prompts are described only in words §4.2, the model snapshot and decoding settings are absent, the sample selection per dataset is not stated, and the instruction-data construction in Table 1 is not explained tab. 1. An implementer can rebuild the three-part prompt but will not get the reported numbers.

• Figure and table disagree on stock accuracy. Table 2 gives 0.53 for BigData22 tab. 2. The caption for the 1,470-point stock run gives 51% fig. 8. The paper does not say whether these are the same run.

• Stock labels leave a gap. The positive threshold is +0.55% and the negative threshold is -0.5% §4.1. Moves between them have no class and the paper does not say how they were scored, so the 0.53 figure depends on an unstated decision. The thresholds are also asymmetric.

• FiQA-SA scale versus accuracy. The text describes a continuous score from -1 to 1 §4.1, yet Table 2 reports accuracy. How scores became classes is not stated. Captions write the percent sign as "$", so 79$ means 79%.

• Counts do not match. The conclusion refers to 11 financial tasks §7, but the evaluation covers six datasets §4.1 tab. 1. The paper does not map one to the other.

• Instruction counts. Table 1 lists instruction counts ten times the raw count for some datasets and equal counts for others tab. 1. The construction is not described.

• Fraud claim rests on five cases. The "5 out of 5" result is a hand-picked illustration on the PaySim dataset fig. 9 §3.3.3. It supports no rate, precision or recall.

• Finance-specific claims without data. The results section says LLM recommendations were checked against real-world financial data and market history §5. No such comparison is shown; the reported results are the six accuracy-style scores and the case figures.

• Equation notation. Equation 1 writes the query and key product without a transpose and uses lowercase symbols for matrices eq. 1. Implementers should take the standard form with the key matrix transposed, and treat the paper's equation as notation only.

• Shot setting unclear. Section 4 describes one-shot and zero-shot §4; Table 2's caption says "zero-shot and few-shot"; §4.2 lists plain zero-shot, chain-of-thought zero-shot and one-shot. Which setting each row used is not stated.

• Author list differs. The LaTeX title block lists Wei Ruan among the authors, but the arXiv metadata does not. This card follows the metadata [raw/arxiv/2401.11641.json].

• Numbering comes from source, not the PDF. Section, table, figure and equation numbers were counted in the LaTeX source. The compiled PDF was not built here, so check them against a rendered copy.

• Dataset licenses. Table 1 lists "Public" for two datasets. That is not a license. Check the dataset's own terms before reuse.

Lineage

• Builds on: the datasets and evaluation framework it uses, FinQA, ConvFinQA, BigData22, the FLUE sentiment framework and the PaySim fraud dataset §4.1 §3.3.3; chain-of-thought prompting, which it cites for question answering §2.4.
• Compared against: none reported tab. 2.
• Superseded by: not stated.