Paper · 2023-08-13 · cs.CL, cs.AI, cs.CE, cs.IR
Transforming Sentiment Analysis in the Financial Domain with ChatGPT
Zero-shot GPT-3.5-turbo labels forex news headlines as positive, neutral or negative, with prompts for single headlines and for daily batches. Against FinBERT on 2,291 annotated headlines, the best prompts reach about 0.79 accuracy versus 0.56 (author-reported). Daily summed scores are compared with market returns.
Sentiment analysisPrompt engineeringsentiment-analysisfinancepromptingzero-shot
arXivPDFCodeMarkdown12 min read
AI-compiled by claude-haiku-5-5 on 2026-11-22, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Financial news is hard to score for sentiment. A headline can be good for one currency and bad for the pair it belongs to, and older sentiment models reflect the view of the experts who labelled their training data. This paper asks whether a general chat model, GPT-3.5-turbo, can do the job without any training, just by being asked in plain language.
The authors collected forex headlines for five currency pairs over about three months in 2023 and labelled each one by how it was likely to move its pair in the short term. They then asked the model, one headline at a time or in daily batches, to say whether the pair should be bought, sold or held. Each prompt was tried as a variant. The model answers were compared with the labels, with a finance-tuned model called FinBERT, and with how the pair's price moved on the same day.
The main message is that the wording of the question matters a lot. The prompts that did best named the currency pair and asked for an expert's judgement or a trader's buy, sell or hold call. The card's pitfalls section lists where the paper leaves gaps and where its code differs from its text.
TL;DR
- Zero-shot GPT-3.5-turbo, no fine-tuning, labels forex headlines as positive, neutral or negative §3.3.
- Best single-headline prompts reach 0.784 to 0.790 accuracy, against 0.561 for FinBERT tab. 4 (author-reported).
- Prompt wording changes results a lot; the trader-framed prompt P4 is best on headlines that do not name the pair tab. 6.
- The shipped code differs from the text in temperature, token limits and prompt wording, and the paper does not state its metric averaging (the code uses weighted averages); see Paper vs code.
Problem
Financial sentiment models based on lexicons or fine-tuned encoders often miss which instrument a headline is about and do not adapt to a use case §1 §2.1. The paper also notes that FinBERT reflects the view of its annotators §2.1. The question is whether an instruction-following LLM used zero-shot can match or beat FinBERT, and whether its daily scores track market returns §1 §3.4.2.
Method
Data §3.1. Headlines come from two forex news sites (Forex Live and FXstreet), collected every ~15 minutes over 86 days from January to May 2023 §3.1. The dataset has 2,291 unique headlines for five pairs: AUDUSD, EURCHF, EURUSD, GBPUSD and USDJPY §3.1. Each headline has a pair, timestamp, source, author, URL and full article text §3.1. Each headline gets one manual label, one of positive, negative or neutral, which the authors defined as bullish, bearish or hold for the named pair §3.1 tab. 1. The label reflects the likely short-term impact on the pair, not the wording alone §3.1. Token statistics are in tab. 2: headlines average 11.69 tokens (SD 2.82) and articles 262.58 (SD 158.99) tab. 2.
Baseline FinBERT §3.2. FinBERT is run through the Hugging Face Transformers library and gives a probability for each of the three classes §3.2. The predicted class comes from those probabilities §3.2. The numeric score, FinBERT-N, is the positive probability minus the negative probability §3.2. The checkpoint is not named in the paper.
ChatGPT prompts §3.3 tab. 3. Six prompt families are tested (P1 to P6), each with a numeric twin (P1N to P6N) that asks for a score in [-1, 1] §3.3.
- P1: a financial expert "holding" the pair asks how they feel about the headline. Answer as one token: positive, negative or neutral tab. 3.
- P2: a financial expert classifies the pair's sentiment from the headline alone tab. 3.
- P3: a sentiment model trained on financial headlines classifies the headline (no pair named) tab. 3.
- P4: a forex trader holding the pair asks whether to buy, sell or hold it short term, based only on the headline. Positive means buy, negative sell, neutral hold tab. 3.
- P5: the same trader question, but over the list of all headlines for one pair on one day tab. 3.
- P6: a sentiment service gives a daily summary for all pairs, as JSON with one label per pair tab. 3.
For the numeric twins, the model returns a number in [-1, 1], e.g. inside a list §3.3. Each prompt is one user instruction, with the pair and headline inserted §3.3. The model is GPT-3.5-turbo through OpenAI's Python library §3.3. Output token limits and temperature are in Hyperparameters. Extra fields recorded per call are latency, completion tokens and prompt tokens §3.3. Errors are handled with retries §3.3.
Classification and scoring. Each label maps to an integer: positive 1, neutral 0, negative -1 §3.4.1. Each prompt's output goes through the same mapping. P1 to P4 and FinBERT are scored per headline. P5 and P6 are scored per day and pair §3.4.2. Classification uses accuracy, precision, recall and F1 on the labels, plus the mean absolute error on the integer labels (S-MAE, eq. 1) §3.4.1.
# Zero-shot label for one headline [§3.3] [tab. 3]
# Input: ticker (str), headline (str); Output: label in {-1, 0, 1}
def classify_headline(ticker, headline):
# Insert the pair and headline into the chosen prompt [§3.3]
prompt = fill_template(template, ticker, headline)
# One short chat reply; temperature 0.2 per the text [§3.3]
reply = chat_model(prompt, max_tokens=1, temperature=0.2)
# Map the reply to the integer label [§3.4.1]
return to_label(reply)# Daily score for one pair on one day [§3.4.2]
# Input: headline rows with (date, pair, value); Output: float
def daily_score(rows, date, pair):
# Sum, not average, the values of that day's headlines [§3.4.2]
values = [row.value for row in rows
if row.date == date and row.pair == pair]
return sum(values)# Sentiment mean absolute error [eq. 1]
# Input: true and predicted integer labels; Output: float
def s_mae(true_labels, predicted_labels):
total = 0.0
for y_true, y_pred in zip(true_labels, predicted_labels):
# Penalise each miss by its distance on the -1..1 scale [eq. 1]
total += abs(y_true - y_pred)
return total / len(true_labels)# Market check: correlation and directional accuracy [§3.4.2]
# Input: daily scores, daily returns, headline counts per day
def market_check(scores, returns, counts):
# Keep only days with at least one headline [§3.4.2] [code: results.py#L221]
days = [d for d in scores if counts[d] > 0]
# Pearson correlation between score and simple return [§3.4.2]
corr = pearson(scores[days], returns[days])
# Share of days where score and return have the same sign [§3.4.2]
hits = [sign(scores[d]) == sign(returns[d]) for d in days]
return corr, sum(hits) / len(hits)Market evaluation §3.4.2 §4.2. Daily scores per pair are compared with that pair's daily return, using Pearson correlation §3.4.2. Directional accuracy (DA) is the share of days where the score's direction matches the return's direction §3.4.2. Integer label predictions also enter this check as scores §3.4.2.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| model | GPT-3.5-turbo (OpenAI API) | §3.3 | snapshot/version not stated; code: gpt-3.5-turbo helper_functions.py |
| temperature | 0.2 | §3.3 | code does not set it and falls back to 0 helper_functions.py; see Paper vs code |
| top_p | not stated | — | code sets 1 in each prompt entry, e.g. prompts.json |
| max_tokens, P1 to P5 | 1 | §3.3 | matches code prompts.json prompts.json prompts.json prompts.json prompts.json |
| max_tokens, P1N to P4N | 10 (literal reading of "P1N-P4N" = 10) | §3.3 | code: P1N 20, P2N 20, P3N 10, P4N 10 prompts.json prompts.json prompts.json prompts.json |
| max_tokens, P5N | 20 | §3.3 | matches code prompts.json |
| max_tokens, P6 and P6N | 200 | §3.3 | matches code prompts.json prompts.json |
| retries | not stated | — | code: 5 attempts, 0.1 s sleep, then NaN helper_functions.py |
| FinBERT score (FinBERT-N) | P(positive) − P(negative) | §3.2 | checkpoint not named; inference code not in repo |
| dataset | 2,291 headlines, 5 pairs, 86 days (Jan–May 2023) | §3.1 | labels from manual annotation; annotator count not stated |
| S-MAE | labels −1, 0, 1 | eq. 1 | code: mean of absolute differences helper_functions.py |
| Pearson / DA | per pair and per day | §3.4.2 | DA code not in repo |
Results
Author-reported. Setup: 2,291 headlines, five forex pairs, zero-shot GPT-3.5-turbo, FinBERT baseline §3.1 §3.2 §3.3. The paper does not report run-to-run variance.
Headline classification on the full dataset tab. 4 §4.1:
| Model | Accuracy | Precision | Recall | F1 | S-MAE |
|---|---|---|---|---|---|
| FinBERT | 0.561 | 0.560 | 0.562 | 0.556 | 0.540 |
| GPT-P1 | 0.730 | 0.760 | 0.730 | 0.725 | 0.300 |
| GPT-P2 | 0.790 | 0.797 | 0.790 | 0.790 | 0.227 |
| GPT-P3 | 0.735 | 0.780 | 0.735 | 0.737 | 0.282 |
| GPT-P4 | 0.784 | 0.804 | 0.784 | 0.789 | 0.221 |
Headlines that do not name the pair tab. 6 §4.1:
| Model | Accuracy | Precision | Recall | F1 | S-MAE |
|---|---|---|---|---|---|
| FinBERT | 0.543 | 0.543 | 0.543 | 0.538 | 0.539 |
| GPT-P1 | 0.663 | 0.715 | 0.663 | 0.672 | 0.410 |
| GPT-P2 | 0.711 | 0.728 | 0.711 | 0.714 | 0.326 |
| GPT-P3 | 0.665 | 0.657 | 0.665 | 0.657 | 0.384 |
| GPT-P4 | 0.765 | 0.772 | 0.765 | 0.763 | 0.249 |
Best model per pair, classification tab. 5 §4.1:
| Pair | Accuracy | Precision | Recall | F1 | S-MAE |
|---|---|---|---|---|---|
| AUDUSD | P2 | P4 | P2 | P2 | P2 |
| EURCHF | P2 | P4 | P2 | P2 | P4 |
| EURUSD | P4 | P4 | P4 | P4 | P4 |
| GBPUSD | P4 | P4 | P4 | P4 | P4 |
| USDJPY | P2 | P2 | P2 | P2 | P2 |
Directional accuracy, non-numerical and numerical models tab. 7 tab. 8 §4.2.2:
| Model | DA | Model | DA | |
|---|---|---|---|---|
| FinBERT | 0.595 | FinBERT-N | 0.599 | |
| GPT-P1 | 0.583 | GPT-P1N | 0.672 | |
| GPT-P2 | 0.607 | GPT-P2N | 0.644 | |
| GPT-P3 | 0.607 | GPT-P3N | 0.660 | |
| GPT-P4 | 0.640 | GPT-P4N | 0.611 | |
| GPT-P5 | 0.652 | GPT-P5N | 0.623 | |
| GPT-P6 | 0.640 | GPT-P6N | 0.652 |
Directional accuracy per pair, numerical models tab. 9 §4.2.2:
| Model | AUDUSD | EURCHF | EURUSD | GBPUSD | USDJPY |
|---|---|---|---|---|---|
| FinBERT-N | 0.518 | 0.250 | 0.654 | 0.698 | 0.714 |
| GPT-P1N | 0.648 | 0.500 | 0.673 | 0.679 | 0.696 |
| GPT-P2N | 0.648 | 0.437 | 0.576 | 0.679 | 0.678 |
| GPT-P3N | 0.630 | 0.500 | 0.635 | 0.755 | 0.678 |
| GPT-P4N | 0.537 | 0.188 | 0.500 | 0.660 | 0.536 |
| GPT-P5N | 0.593 | 0.469 | 0.615 | 0.660 | 0.589 |
| GPT-P6N | 0.593 | 0.344 | 0.731 | 0.792 | 0.607 |
Market correlation §4.2.1 fig. 8: the paper reports that the correlation between true labels and daily returns is about 0.49 §4.2.1. The correlations of the other models are shown only in the heatmap fig. 8. The text says that GPT-P6N correlates more with returns than the true labels do, that numerical (N) models align better with market moves than categorical ones, and that P5 has the lowest correlation §4.2.1 §5.1. The table values are not printed in the paper.
Time and tokens per call tab. 10 §4.2.3:
| Prompt | Time per call (s) | Tokens per call | Time per headline (s) | Tokens per headline |
|---|---|---|---|---|
| P1 | 0.94 | 63.0 | 0.94 | 63.17 |
| P2 | 0.81 | 67.0 | 0.81 | 67.15 |
| P3 | 0.97 | 66.0 | 0.97 | 65.91 |
| P4 | 0.82 | 84.0 | 0.82 | 84.39 |
| P5 | 0.83 | 193.0 | 0.11 | 24.74 |
| P6 | 5.99 | 592.0 | 0.22 | 22.22 |
| P1N | 2.29 | 90.0 | 2.29 | 90.34 |
| P2N | 1.42 | 85.0 | 1.42 | 84.79 |
| P3N | 1.23 | 80.0 | 1.23 | 79.98 |
| P4N | 1.06 | 91.0 | 1.06 | 90.81 |
| P5N | 1.85 | 201.0 | 0.24 | 25.68 |
| P6N | 6.10 | 600.0 | 0.23 | 22.54 |
The paper's cost example uses 600 tokens per article for P6N, at 0.002 USD per 1K tokens, for 5,000 articles a day, which it puts at about 6 USD a day §4.2.3. See Pitfalls for why this basis is unclear.
The paper says ChatGPT beats FinBERT under every prompt tested and that prompt wording is decisive §5.1. The reported gains over FinBERT are about 35% in classification and 36% in correlation with returns § Abstract. The text does not say how these percentages were computed.
Paper vs code
Official repository: giorgosfatouros/Financial-Sentiment-Analysis-with-ChatGPT
at commit bd9ce58ca5283cff19219340b44ecab3fcfbaf21, MIT license. Its README cites
this paper, and the LICENSE names "George Fatouros", the first author [code:
README.md#L1-L7] LICENSE. The repository contains the prompts,
the helper and results scripts, prediction CSVs and price data. It does not
contain the FinBERT inference code or a directional-accuracy or cost calculation.
- Temperature. Paper: 0.2 §3.3. Code: the parameter is not set in
prompts.json, so the helper falls back to 0 helper_functions.py prompts.json. Nothing in the shipped scripts sets 0.2. - Token limits. Paper's sentence ("P1N-P4N, P5N, P6-P6N ... 10, 20, and 200") is ambiguous. Code: P1N and P2N use 20, P3N and P4N use 10 prompts.json prompts.json prompts.json prompts.json. Other limits match tab. 3 text.
- Prompt structure. Paper: each prompt is a single user instruction §3.3. Code: most prompts are split into system, assistant and user messages, and the question sits in the assistant turn prompts.json. Replicating the code means copying the role layout.
- Prompt P2 wording. Paper's P2 says "Act as a financial expert" tab. 3. Code's P2 system message says "Act as an expert at forex trading" prompts.json.
- Extra prompts in code. The code has article-based variants P4A and P4NA (16 tokens) and a naive baseline column. The results script reports P4A and "Naive Sent" results.py prompts.json prompts.json results.py. Neither appears in the paper's tables; the naive baseline is the most frequent label per pair.
- Metric averaging. The paper does not say how precision, recall and F1 are averaged across the three classes. The code uses scikit-learn's
average='weighted'results.py. Weighted F1 can differ from macro F1 when classes are unbalanced. - Error handling. Failed calls return NaN after 5 attempts helper_functions.py. The function that maps free text to labels,
extract_sentiment, is defined but not called by the shipped scripts helper_functions.py. The paper does not say how off-format replies are scored. - Daily aggregation. Matches the text: sums per pair per day results.py §3.4.2.
- Correlation scope. The heatmap pools all pairs and filters to days with news results.py. Per-pair correlations are computed separately results.py. The paper does not say which set gives the 0.49 figure.
- Returns. Daily returns are simple close-to-close changes in
forex_price_data.csvforex_price_data.csv. The price source and the cut-off time are not documented. - Directional accuracy. Not computed anywhere in the repository. The sign rule for zero returns or zero scores is not stated.
- Timing. The repository does not compute the latency or token tables. Those columns exist only in the prediction CSVs.
No discrepancies found in the S-MAE definition: the code matches eq. 1 helper_functions.py.
Pitfalls
- Temperature and token limits are not what the code runs. Use 0.2 only if the paper's setting is the target. The shipped code runs at temperature 0 with a different P1N and P2N limit. Pin both explicitly.
- Model version is not pinned. The paper uses
gpt-3.5-turbowith no dated snapshot §3.3. Results can change when the hosted model changes. The paper also warns that latency depends on load §5.3 §4.2.3. - Evaluation may reuse tuning data. The paper says the prompts were refined through "extensive prompt engineering" §3.3. It does not describe a held-out split, so the reported accuracies may be optimistic for prompts chosen on the same headlines.
- Labels measure expected impact, not wording. Annotators labelled each headline by its likely short-term effect on the named pair §3.1. A model that reads the text literally will score lower. The number of annotators and any agreement statistic are not stated.
- The benchmark is small and uneven. Five pairs, 86 days, two news sites §3.1. The paper itself notes that EURCHF has less data and that its DA is lower §4.2.2.
- FinBERT's setup is underspecified. The checkpoint, the tie rule for the predicted class, and the exact FinBERT-N formula are not in the paper's code §3.2. Reproduce it with the same checkpoint before comparing.
- Accuracy and S-MAE reward different errors. By construction of eq. 1, a positive-versus-negative error counts twice as much as a neutral-versus-polar miss. The paper motivates this in §3.4.1; it gives no trading-cost analysis to check it against.
- Correlation with the same-day return. Headlines and returns are matched on the calendar date results.py. Headlines published after the close are then credited to the same day. The paper does not discuss this timing; a stricter test would use next-day returns.
- Correlation is not prediction. The paper says its alignment "does not necessarily imply predictive power" §5.2. Treat the correlation as descriptive.
- The cost example mixes units. It multiplies P6N's per-call token figure (600) by 5,000 articles §4.2.3 tab. 10. Table 10 gives 22.54 tokens per headline for P6N. Using that figure (own calculation: 22.54 × 5,000 × 0.002 / 1,000) gives about 0.23 USD a day, not 6 USD. The paper's price (0.002 USD per 1K tokens for GPT-3.5) is not checked here.
- The 35% and 36% headline gains have no stated basis. The tables give P2 accuracy 0.790 against FinBERT 0.561 tab. 4, a relative gain of about 41% (own calculation). The paper does not say which metric or average gives 35%.
- Correlation values live only in a figure. Use fig. 8 or the repository's
fig_res_corr.png; the text gives only about 0.49.
Lineage
- Compared against: FinBERT §3.2 (not compiled in this wiki).
- Mentioned as future work: GPT-4 and open models such as BLOOM §5.3 (not compiled in this wiki).
- Related prompting background: see the prompt-engineering and sentiment-analysis concept pages.
Source and license
Adapted from “Transforming Sentiment Analysis in the Financial Domain with ChatGPT” by Georgios Fatouros, John Soldatos, Kalliopi Kouroumali, Georgios Makridis, Dimosthenis Kyriazis (arXiv:2308.07935), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/giorgosfatouros/Financial-Sentiment-Analysis-with-ChatGPT at commit bd9ce58ca528 (MIT). Any quoted code is shown under that license.