Large Language Models for Travel Behavior Prediction

In plain words

Transport planners often predict how people travel, for example whether they take a train, a car or a fast new underground system, or why they are on a trip. The classic tools are statistical models fitted to survey data. This paper asks whether a large language model can make these predictions by reading a plain-text description of the trip and the traveller.

The authors test two ways of doing it. In the first, the model gets no training examples at all. The prompt lists the options and their times and costs, describes the traveller, and gives some common-sense rules about how people usually choose, and the model answers in text with a reason. In the second, the model turns each description into a list of numbers, and a normal statistical model is trained on those numbers. This second route is aimed at cases where only a few labelled examples exist.

The authors report that the first route does well when there is little or no training data, and that the second helps the classic models when training data is scarce. When plenty of labelled data is available, the classic models trained on it do best.

TL;DR

The paper compares two uses of LLMs for travel behavior prediction on mode choice (Swissmetro survey) and trip purpose (2017 US NHTS). Zero-shot prompting sends a structured text prompt with no training data and asks for a JSON answer with a reason §3.2, §3.3, app. A. Embedding-as-features takes LLM embeddings of the same text and fits MNL, random forest or a neural network on them §3.4. Results are author-reported, on 200-row balanced test sets averaged over 5 random samples §4.1, tab. 3. Costs and reproducibility details are sparse in the paper; the released code is a later, different revision [see Paper vs code].

Problem

Traditional travel behavior models are numerical discrete choice models calibrated on observed data §1. Their weakness is that they need labelled samples, and they encode preferences only through hand-built features §1, §3.4. The authors ask whether LLMs can predict travel behavior at all, and whether they can do it without task-specific training data §1, §2.

The paper states its assumptions as: the prompt carries all needed information, the LLM can reason about numeric comparisons if the prompt gives them explicitly §3.3.4, and the test set is balanced across labels §4.1. Known limitations the authors report are reasoning errors and hallucinated explanations §4.4.2, tab. 4.

Method

Preliminaries. An LLM is treated as a conditional next-token model. A prompt "(Input, Task)" is turned into an answer §3.1, eq. 2. The same model can return an embedding vector h for the prompt, taken from an internal layer before the output head §3.1. The paper's stated dimension H depends on the architecture and is not given for the embedding model used §3.1.

Framework 1: zero-shot prediction §3.2, fig. 1. The prompt is built from six parts, in this order §3.3.6, fig. 7, app. A:

1. Task description: the task, the allowed options, and what information follows §3.3.1, fig. 2.
2. Structural data: travel time and cost per option, as a nested dictionary §3.3.2, fig. 3.
3. Descriptive data: traveller attributes written as plain sentences, not as dummy variables §3.3.3, fig. 4.
4. Domain knowledge and comparison sentences §3.3.4, fig. 5, app. A. These include generic rules about cost and time trade-offs, and sentences that state which option has the lowest time or cost and the percentage saving relative to each other option.
5. Output format: a JSON object with keys "prediction" and "reason" §3.3.5, fig. 6, app. A.

No training examples are included §3.3.6. The model is called with temperature 0 §4.3.

Framework 2: embeddings as features §3.2, §3.4. Each sample n gets an embedding h_n of its text. A supervised model with parameters beta is trained on h_n and the labels of the training set, minimizing the summed per-sample loss. For classification the loss is cross-entropy §3.4. The paper names multinomial logit, random forest, gradient boosting and neural networks as possible heads, and in the experiments uses MNL, RF and NN §3.4, §4.2.

# Zero-shot prediction  [§3.2] [§3.3] [app. A]
# Input:  travel_options (per mode: time, cost), traveller_attributes
# Output: predicted_label, reason

def predict_zero_shot(travel_options, traveller_attributes, llm):
    # 1. Assemble the prompt from task text, structured travel data,
    #    descriptive traveller data, domain rules and output format
    #    [§3.3.1-§3.3.6] [app. A]
    prompt = build_prompt(travel_options, traveller_attributes)

    # 2. Query the LLM with temperature 0; no training data is used  [§4.3]
    raw_text = llm.generate(prompt, temperature=0.0)

    # 3. Read the JSON object with keys "prediction" and "reason"  [app. A]
    #    (parser not described in the paper)
    answer = parse_json(raw_text)
    return answer["prediction"], answer["reason"]

# Embedding as features  [§3.2] [§3.4]
# Input:  train_samples (B rows with labels), test_samples (N rows)
# Output: predicted label for each test sample

def predict_with_embeddings(train_samples, test_samples, embed, fit):
    # 1. Embed each sample's prompt text with an LLM embedding model
    #    [§3.2] [§3.1]  -> (B, H) and (N, H)
    train_vectors = [embed(prompt_of(s)) for s in train_samples]
    test_vectors = [embed(prompt_of(s)) for s in test_samples]

    # 2. Fit a supervised head (MNL, RF or NN) on the embeddings  [§3.4]
    model = fit(train_vectors, labels_of(train_samples))

    # 3. Predict labels for the test embeddings  [§3.4]
    return model.predict(test_vectors)

Mode choice (Swissmetro). Options are train, car and Swissmetro. The attributes are travel time and cost per option, whether the traveller is a regular train user, and whether they own a train annual pass §4.1, tab. 1, app. A. The paper's prompt example writes the percentage saving as the gap to the best option divided by the other option's value; this formula is inferred from the example numbers and is not stated in the text app. A.

Trip purpose (NHTS). Four labels: working, social, recreation and others §4.1. The prompt in the appendix lists working, shopping, social and others instead app. A. See Pitfalls.

Hyperparameters

Feature statistics:

Results

Author-reported. Setup: Swissmetro mode choice and NHTS trip purpose, 200-row balanced test sets, averages over 5 random samples, benchmark models trained on 1,000 or 10 samples §4.1, tab. 3. Accuracy and weighted F1, mean ± spread as printed tab. 3.

Authors' reading: large-sample supervised models give the best overall results; with 10 samples they drop; zero-shot LLMs outperform small-sample supervised models; embeddings improve small-sample supervised models on both tasks §4.4.1, §5.

Interpretability §4.4.2, tab. 4: four cases. Case 1 is correct with a reasonable reason. Case 2 is wrong with a reasonable reason. Case 3 is wrong with a reasoning error. Case 4 is wrong with a hallucinated reason (a preference for time saving with no such input) tab. 4.

Ablation, mode choice only, accuracy with decrease from full prompt §4.4.3, tab. 5:

The paper reads GPT-3.5 as sensitive to prompt design and GPT-4 as robust to it §4.4.3.

Paper vs code

The code at code_commit is mbc96325/llm-travel-prediction, owned by the GitHub account named Baichuan Mo (first author). It was found by a GitHub repository search for the title and linked from neither the paper nor an abstract. The paper's own source has no code link [main.tex]. The repository's README describes itself as code for a later revision (Revision 1), in a journal the paper's source does not name, and adds TextGrad prompt optimisation, TabPFN and a London (LPMC) dataset README.md. Several results below therefore compare the paper against code that is not the paper's method.

• Models. Paper: GPT-3.5, GPT-4, Llama 3.1 8B and 70B §4.3. Code: the discovery and default model is gpt-4o-mini E00_textgrad_prompt_optimization.py; the evaluation script names the paper's GPT-3.5, GPT-4-turbo and Llama 3.1 results E04_faithfulness_eval.py. Temperature is 0 in both E00_textgrad_prompt_optimization.py README.md.
• Zero-shot method. The paper's prompts are fixed hand-written text §3.3, app. A. The code's main contribution is an automatic prompt search: an LLM critiques errors on a labelled budget and rewrites the prompt README.md E00_textgrad_prompt_optimization.py. The code's minimal and handcrafted prompts are separate conditions, and the paper's zero-shot prompt is not shown in the code as one fixed string.
• Baselines. Paper: sklearn default hyperparameters §4.3. Code: random forest and neural network are tuned by grid search with stratified cross-validation E03_tuned_baselines.py E03_tuned_baselines.py E03_tuned_baselines.py. The code's MNL is logistic regression with C grid {0.1, 1.0, 10.0} and cross-validation E06_nhts_baselines.py.
• Embedding heads. The paper gives no head settings §3.4, §4.2. The code uses logistic regression with C=1.0, a random forest with 200 trees and a small grid over depth and leaf size, and an MLP with one hidden layer of 64 units, alpha 1e-2 and up to 1,500 iterations E09_embedding_experiments.py E09_embedding_experiments.py.
• Embedding input. The code embeds each row's rendered descriptive text with text-embedding-3-small E09_embedding_experiments.py E09_embedding_experiments.py. The paper's prose does not name the model §3.2; see Pitfalls.
• Trip purpose labels. The code uses working, shopping, social and others E00_textgrad_prompt_optimization.py E06_nhts_baselines.py, matching the appendix prompt app. A. The paper's §4.1 text lists "recreation" instead of "shopping" §4.1.
• Data splits. Code expects seeds 0 to 4, budgets 10, 100 and 1,000, and a fixed 200-row test set README.md. The paper uses 1,000 and 10 training samples §4.1; the budget of 100 is an addition.
• Significance. The paper reports no significance tests §4. The code adds McNemar tests E07_significance_mcnemar.py E12_mcnemar_from_cache.py.
• Licence. The repository has no LICENSE file. We cannot publish code from it, so this card cites by path and line only and quotes nothing.

No further discrepancies were checked. The prompt-builder code for the percentage sentences was not traced to confirm the formula inferred in Method.

Pitfalls

• Trip purpose label set. Section 4.1 lists working, social, recreation and others, while the appendix prompt and the code use working, shopping, social and others §4.1 app. A. An implementer must pick one; the prompt used in the reported runs is the appendix version app. A.
• Degenerate baseline. RF with small training sets on trip purpose reports accuracy 0.250 ± 0.000 and F1 0.100 ± 0.000 tab. 3. This pattern is consistent with one predicted class, but the paper does not say so. Treat the row as unexplained.
• Spread of results. The ± values are printed without a definition tab. 3. Five random samples are a small basis for these spreads §4.1.
• Balanced test sets. The test sets are balanced by label §4.1. Accuracy on real, unbalanced data may differ.
• Embedding model name. Prose says "text-embedding model" §3.2; the bibliography key names text-embedding-3-small. The embedding dimension H is not stated §3.1.
• Percentage sentences. The prompt states percentage savings computed from the inputs app. A. The formula is inferred from the example numbers only; the paper does not state it, and an implementer should check that rounding matches the example.
• Output parsing. The paper asks for JSON "so that we can easily extract the results" §3.3.5, but the parser and failure handling are not described. Reasons are free text and can be wrong or hallucinated §4.4.2, tab. 4.
• Sensitivity to prompt. The ablation shows GPT-3.5 losing up to 5.8% accuracy when the guide of thinking is removed, and GPT-4 losing under 1% §4.4.3, tab. 5. A prompt written for one model may not transfer.
• Scope of evidence. The ablation covers mode choice only, and only GPT-3.5 and GPT-4 §4.4.3. The authors name more trip-purpose and departure-time tasks as future work §5.
• Small samples. The "small" pipelines use 10 training samples §4.1. The embedding-based results depend on that size, and no range of sizes is shown.
• Equation numbers. Only eq. 2 carries a label in the source. Other equations are cited by section.
• Cost. The paper gives no token counts, API prices or latency for the zero-shot runs.

Lineage

• Builds on: chain-of-thought prompting and plan-and-solve prompting, cited in the paper as Wei et al. 2022 and Wang et al. 2023 §3.3.4 (no arXiv ids recorded here).
• Compared against: multinomial logit, random forest and neural networks, all with sklearn defaults in the paper §4.2, §4.3.
• Related work on LLMs in mobility prediction: §2. Those papers are named by the authors and are not used here.
• Superseded by: not known from content on this wiki.