LLM as GNN: Graph Vocabulary Learning for Text-Attributed Graph Foundation Models

In plain words

Text-attributed graphs are networks in which every node carries a piece of text, such as a paper's title and abstract. Graph neural networks learn from such networks by letting each node take in information from its neighbours, but they work on numbers, not words, so combining them with a language model is awkward. PromptGFM keeps the whole process in words. A language model rewrites each node's summary over a few rounds, and each rewrite folds in short summaries of some sampled neighbours, which is roughly how a graph network passes messages along edges. The last summary of each node becomes that node's ID, made of ordinary words the model already knows.

Because the IDs are readable text, a language model can be fine-tuned on prompts that list a node's ID together with its neighbours' IDs and ask a question, such as which node a given node should connect to. When the model has to name a node, a decoding constraint keeps its answer to IDs that exist in the graph. The aim is one model that transfers across graphs and tasks, instead of a separate vocabulary for each graph.

TL;DR

• PromptGFM is a text-only graph foundation model for text-attributed graphs, built around a graph vocabulary made of language-based node IDs §1 §3.2.
• A language model summarises each node, then refines the summaries over L rounds using sampled one-hop neighbours; the final summaries are the IDs §3.1.
• Flan-T5 or Llama3-8B is fine-tuned on templated node-classification and link-prediction instructions, and link prediction is decoded with a prefix tree over valid IDs §3.2.
• Author-reported: Llama3-8B has the top node-classification accuracy in the paper's table on six of seven datasets, not on PubMed tab. 1. Code is partial (see Paper vs code).

Problem

Existing GNN-LLM methods either feed GNN embeddings to an LLM or let an LLM supervise a GNN in separate stages, so the two parts are only loosely coupled §1 fig. 1. Other methods treat each node as an out-of-vocabulary token; this causes graph-specific semantics, growing vocabularies, and mismatch with natural-language task prompts, which limits transfer across graphs and tasks §1. Heuristic graph-to-text prompts also omit the message passing of a real GNN, which the authors argue is needed for structure §1 fig. 2. Prior graph foundation models rely on domain-specific vocabularies such as per-dataset IDs or quantised computation trees §5.

Method

PromptGFM has two modules §3 fig. 3.

Graph understanding module §3.1 fig. 6. The authors frame a GNN layer as neighbour sampling, aggregation and update, and optimisation §2 eq. 1 eq. 2. They replicate this in text:

1. Initialise each node's textual representation by summarising its own title and abstract with an LLM. The summary prompt is described in words in app. D.
2. For each layer l = 1..L, sample one-hop neighbours of node i from the previous round eq. 3.
3. Prompt the LLM to aggregate those neighbours' previous summaries and the node's own previous summary into a new summary eq. 4. No fixed mean or weighted operator is used §3.1.
4. The optimisation signal (neighbours similar, distant nodes dissimilar) is written into the prompt as natural-language instruction, and no separate training loss is added §3.1.
5. After L rounds the final representation is T_i = X_i^(L) §3.1.

The paper analyses input-order sensitivity app. E.1 and prompt length limits app. G.

Graph inference module §3.2 fig. 4. The final summaries are tokenised with the LLM's own tokenizer and used directly as IDs, with no clustering or remapping §3.2 eq. 5. Task templates insert the centre node's ID and its one-hop and two-hop neighbours' IDs into a readable prompt eq. 6 app. D. The model is trained across graphs and tasks with a standard token-level loss eq. 7. For generative link prediction, a prefix tree is built from the IDs of all candidate nodes, and only tokens that extend a valid prefix are allowed during decoding §3.2.

Auxiliary tasks §4.3. Node discrimination (choose the one of three nodes in a different class) and link discrimination (choose the one of three candidates not linked to a centre node) are added as extra instruction tasks, individually or together §4.3 tab. 8.

Other settings described in the paper. Adaptive prompting prepends a domain instruction at inference time app. H tab. 18. A dynamic variant regenerates only the affected nodes when a node, an edge or a node text changes app. I tab. 19 §5 Limitations (3).

# Round 0: each node is summarised from its own text  [§3.1] [app. D]
def initial_summaries(texts):
    summaries = {}
    for node in texts:
        # One LLM call per node  [app. D]
        summaries[node] = llm(summary_prompt(texts[node]))
    return summaries

# Rounds 1..L: the LLM replicates message passing in text  [§3.1]
def message_passing_round(summaries, neighbours, ratio):
    updated = {}
    for node in summaries:
        # Sample one-hop neighbours from the previous round  [eq. 3]
        sampled = sample_neighbours(neighbours[node], ratio)
        # Aggregate and update; every input is last round's text  [eq. 4]
        updated[node] = llm(
            aggregate_update_prompt(
                summaries[node],
                [summaries[j] for j in sampled],
            )
        )
    return updated

# Run L rounds; the last output is the node's textual representation  [§3.1]
def graph_understanding(texts, neighbours, rounds, ratio):
    summaries = initial_summaries(texts)
    for _ in range(rounds):
        summaries = message_passing_round(summaries, neighbours, ratio)
    return summaries

# Language-based ID: the final summary as LLM tokens  [eq. 5] [§3.2]
def language_id(summary, tokenizer):
    # No clustering or remapping; the token sequence is the ID  [§3.2]
    return tokenizer(summary)

# Instruction: a task template filled with language-based IDs  [eq. 6]
def build_instruction(template, centre, one_hop, two_hop):
    # IDs are written inside angle brackets in the prompts  [app. D]
    return template.format(
        centre=language_id(centre),
        one_hop=[language_id(n) for n in one_hop],
        two_hop=[language_id(n) for n in two_hop],
    )

# Multi-instruction fine-tuning: negative log-likelihood of the target  [eq. 7]
def training_loss(model, instruction, target):
    loss = 0
    for s in range(len(target)):
        # Next-token probability given instruction and earlier target tokens
        # Summed over every target position s  [eq. 7]
        probs = model.next_token_probs(instruction, target[:s])
        loss = loss - log(probs[target[s]])
    return loss

# Generative link prediction: prefix-tree constrained decoding  [§3.2]
def build_prefix_tree(candidate_token_lists):
    # Each root-to-leaf path is one valid node ID  [§3.2]
    tree = Trie()
    for tokens in candidate_token_lists:
        tree.add(tokens)
    return tree

def allowed_next_tokens(tree, generated_so_far):
    # Only tokens that extend a valid prefix may be emitted  [§3.2]
    return tree.children(generated_so_far)

Key equations, as printed in the paper eq. 3–eq. 7:

\{X_j^{(l-1)}, v_j \in \mathcal{N}_i^{(l)}\} \leftarrow \text{Prompt}_{\text{sample}}(X^{(l-1)}, \mathcal{N}_i) eq. 3

X_i^{(l)} \leftarrow \text{Prompt}_{\text{agg-update}}(\{X_j^{(l-1)}, v_j \in \mathcal{N}_i^{(l)}\}, X_i^{(l-1)}) eq. 4

\mathcal{F}: V \rightarrow T, \quad T_i = (t_i^1, \ldots, t_i^{m_i}), \; t_i^k \in \mathcal{X} eq. 5

\mathcal{T} \leftarrow \text{Prompt}_{\text{template}}(T, G) eq. 6

\mathcal{L} = - \sum_{s=1}^{|Y|} \log \Pr(y_s \mid \mathcal{T}, Y_{<s}) eq. 7

Equation numbers are counted in source order from the arXiv v4 LaTeX; cross-check them against the PDF.

Hyperparameters

Settings the paper states are marked with their source. Settings only the code states are marked "code". "not stated" means neither the paper nor the code gives a value.

Results

Author-reported throughout. Setup: each model is trained from scratch on a single graph for node classification and link prediction §4.1; the evaluation is 10-fold cross-validation with standard deviations described in app. C, but the tables below show means only.

Node classification accuracy (%), higher is better tab. 1 (bold and colour highlights in the source are omitted; the authors' significance test is against an unnamed strongest baseline):

Discriminative link prediction, accuracy (%) tab. 2:

Generative link prediction, HR@1 (%) tab. 3: backbone not stated in the table.

Intra-domain transfer to Citeseer, accuracy and Macro-F1 (%) tab. 4 (source = graphs used for pre-training, or co-training):

Inter-domain transfer, e-commerce to biomedical tab. 5: the caption does not name the target; PubMed is the only biomedical dataset in the paper app. A.

Inter-domain transfer, computer science to biomedical tab. 6: target not named in the caption.

Cross-task generalisation, accuracy (%) tab. 7: the backbone is not stated. The ST rows match the Flan-T5 NC row of tab. 1 and the Llama3 LP row of tab. 2.

Auxiliary pre-training tasks on Citeseer and PubMed, accuracy and Macro-F1 (%) tab. 8:

Controlled neighbour study, Cora and Citeseer, accuracy and Macro-F1 (%) tab. 9:

Efficiency tab. 10: API cost in USD and tokens are for graph understanding; time and power for inference.

Other reported results, in words and in the appendix:

• Dataset statistics, Table 11 app. A: Cora 2,708 nodes, 10,858 edges, 7 classes; Citeseer 3,327, 9,464, 6; Ogbn-arxiv 169,343, 2,332,486, 40; WikiCS 11,701, 431,726, 10; Photo 48,362, 873,782, 12; History 41,551, 503,180, 12; PubMed 19,717, 88,648, 3.
• Permutation sensitivity app. E.1 tab. 12: shuffling neighbours gives 90.64 ± 1.12 on Cora and 83.79 ± 0.89 on Citeseer; shuffling position-tagged tokens gives 90.55 ± 1.03 and 84.28 ± 1.06.
• Layer-wise cosine similarity on Cora app. E.2 tab. 13: connected pairs 0.171, 0.223, 0.308 at layers 1–3; disconnected pairs about 0.03–0.04 at all depths.
• Self-retention and neighbour similarity app. E.2 tab. 14: self-retention falls from 0.837 to 0.632 on Cora over four layers; the margin over neighbours narrows from 0.673 to 0.295.
• Depth app. E.2: the best depth is 3 on Cora and 2 on Citeseer; over-smoothing is reported beyond that fig. 7.
• Vocabulary app. F tab. 15: coverage is 100% for every dataset; entropy ranges from 8.46 (PubMed) to 12.28 (merged graphs).
• Degree app. G.1 tab. 16: average degrees from 2.84 (Citeseer) to 36.90 (Ogbn-arxiv); nodes with more than 300 neighbours are 0.19–0.25% on three datasets and 0 elsewhere.
• Sampling ratio app. G.1 tab. 17: on Cora, 100% gives 81.36, 60% gives 89.35 and 30% gives 88.36; the configuration is not stated.
• Adaptive prompting app. H tab. 18: adds a domain-change sentence at inference; it improves every cell of the table, by up to about 3 points.
• Dynamic graphs app. I tab. 19: on a DBLP co-authorship graph, PromptGFM-dyn reaches 59.46 accuracy and 56.23 Macro-F1; OFA is the best baseline at 55.17 and 52.54. Graph statistics are not stated.
• Ablations on node classification §4.3 fig. 5: removing the graph understanding module gives the smallest drop, removing instruction fine-tuning a larger one, and removing both the largest; the full model is highest. The dataset is not named in the text, and the values are only in the figure, so they are not reproduced here.

Paper vs code

Code: official repository linked from the abstract, github.com/agiresearch/PromptGFM, cloned at commit f2e70fe46a5d68960094bcb77a2f7e3bd16574f3. The repo has no LICENSE file, so nothing from it is quoted; this card uses only pseudocode and citations. The README says the repo is still under maintenance.

• Training loss. Paper: the loss is the negative log-likelihood of the target tokens Y given the instruction eq. 7. Code: the JSON has an output key Train_PromptGFM.py, but SFTTrainer is given dataset_text_field="instruction" Train_PromptGFM.py, so training text is the input prompt alone. The target is never placed in the training text, so the code does not train on eq. 7 as written.
• Training length. Paper: early stopping with a per-dataset patience app. C. Code: num_train_epochs is 10 by default Train_PromptGFM.py but max_steps=100 overrides it Train_PromptGFM.py; there is no early-stopping callback.
• Learning rate and batch. Paper: 3e-4 and batch 4 app. C. Code: 2e-4 Train_PromptGFM.py and batch 2 Train_PromptGFM.py.
• Adaptation method. Paper: not stated. Code: LoRA with r=64 Train_PromptGFM.py. The 4-bit quantisation config is built but never passed to the trainer Train_PromptGFM.py.
• Neighbour sampling in the generator. Paper: each round samples one-hop neighbours of each node and aggregates them eq. 3 eq. 4. Code: a single used_nodes set is created before the round loop and never reset, and a neighbour is removed from the pool once sampled by any centre node in any round generate_textual_id_V3.py generate_textual_id_V3.py. A node with no unused neighbours keeps its previous text generate_textual_id_V3.py.
• Default sampling ratio. Paper: 60% is best overall tab. 17; search set {0.3, 0.6, 1.0} app. C. Code default: 0.3 generate_textual_id_V3.py.
• Default rounds. Paper: best depth 3 on Cora, 2 on Citeseer app. E.2. Code default: 2 generate_textual_id_V3.py.
• Neighbour cap. Paper: cap 20 app. G.1. Code: one-hop capped at 12 and two-hop at 5, chosen at random per prompt and not seeded NC_preprocess.py NC_preprocess.py. The LP script has no cap.
• Prompts. Paper: the node-initialisation prompt asks to "summarize the paper" app. D. Code: one of five random variants, each asking for about 8 words generate_textual_id_V3.py. Paper: the update prompt includes a note that connected nodes should be similar app. D. Code: that note is absent; the prompt asks for a summary of under 8 words generate_textual_id_V3.py.
• Language-based ID source. Paper: T_i is the final round's representation §3.1. Code: NC and LP scripts read <dataset>_text_feature.csv by default NC_preprocess.py LP_prepocess.py, which is the round-0 file written in generate_textual_id_V3.py. Round-N files need --generator_outcome passed explicitly generate_textual_id_V3.py. The ID is the whole summary with periods removed NC_preprocess.py.
• Evaluation. Paper: 10-fold cross-validation app. C. Code: node classification uses one 80/20 split with seed 42 NC_preprocess.py; link prediction uses one 70/30 node-level split LP_prepocess.py. No random seed is set for sampling of neighbours or candidates in either script.
• Instruction templates. Paper: templates in app. D. Code: the node-classification prompt reads "has one hop connections with" and "also has two hop connection with" NC_preprocess.py. The wording differs from the paper but the structure matches.
• Decoding. Paper: prefix-tree constrained decoding §3.2. Code: 100 beams and 50 returned sequences Decoding.py, with the trie built from candidate2.csv Decoding.py. The T5 checkpoint is loaded from a hard-coded local path Decoding.py.
• Scope of the repo. Paper: seven datasets app. A. Code: only Cora and Citeseer data are in the repository [file listing under dataset/]. Missing from the repo: the five other datasets, Flan-T5 training code, and code for node and link discrimination, adaptive prompting, the dynamic variant, the cross-graph and cross-task experiments, and the controlled neighbour study.
• README. The README tells users to run python train.py README.md. No file of that name exists; the closest files are Train/Train_PromptGFM.py and Train/llama3_finetune.py, which are identical.
• OpenAI client. The generator uses the legacy openai.ChatCompletion call generate_textual_id_V3.py with an empty API key generate_textual_id_V3.py. It needs an older OpenAI SDK and a key set by the user. The environment file pins Python 3.8 environment.yaml.

No discrepancies found: not applicable; the discrepancies above were found.

Pitfalls

• Message passing degrades across rounds in the released generator. Because used_nodes is shared across all rounds generate_textual_id_V3.py, each node can be sampled as a neighbour only once in the whole run. Later rounds have fewer neighbours to aggregate, and some nodes keep their earlier summary. An implementation that follows eq. 3 and eq. 4 must reset the pool per round and per centre node.
• Retry temperature is unbounded within one node. For round-0 summaries, a duplicate reply raises the temperature by 0.1 and a new reply resets it to 0; the retry loop has no cap generate_textual_id_V3.py generate_textual_id_V3.py. A node that keeps repeating its summary is retried indefinitely at rising temperature.
• Summaries are not reproducible. The IDs come from a closed hosted model; the paper does not pin a model version or sampling settings for the summaries app. C. Cache the IDs once generated §3.3.
• Duplicate summaries are checked only in round 0. Distinct nodes can produce the same short summary; the generator retries only in the round-0 pass generate_textual_id_V3.py, and round updates are not checked. Check for duplicates before building prefix trees.
• Prompt length on dense graphs. The paper estimates about 400 tokens per neighbour in round one, so a 128k window fits over 300 neighbours app. G.2. The cap used in code (12 one-hop + 5 two-hop) differs from the cap the paper names (20) app. G.1. Measure prompt length for your graph before training.
• Generative link prediction is hard. Its HR@1 values are between 0.28 and 8.90 tab. 3, even for the best method. The candidate set size is not stated.
• Cross-task and generative-LP backbones are not stated. The ST rows of tab. 7 match the Flan-T5 NC row and the Llama3 LP row, and tab. 3 names no backbone. Compare numbers only within a table.
• Sampling-ratio table does not match the headline. On Cora, tab. 17 gives 89.35 at 60% sampling, but tab. 1 reports 91.72 for Flan-T5. The paper does not say which setup tab. 17 uses.
• A shuffle baseline differs from the headline. Shuffling neighbours on Cora gives 90.64 tab. 12, against 91.72 in tab. 1; the paper does not explain this.
• PubMed is not a win. GraphPrompter scores 94.80 against 94.65 for PromptGFM (Llama3-8B) tab. 1.
• Target of cross-domain tables is unnamed. tab. 5 and tab. 6 say "biomedical" but not which dataset; PubMed is the only biomedical dataset in the paper app. A, so it is the likely target, but check before comparing.
• Standard deviations are missing from the tables. The paper says 10-fold CV with standard deviations app. C, but only means appear; significance is against an unnamed strongest baseline tab. 1.
• The released training script does not match eq. 7. It trains on the prompt text only and stops at 100 steps (see Paper vs code). Fix both before comparing to the reported numbers.
• Compute and cost. Graph understanding on Ogbn-arxiv cost about $2.69 and 17.91M tokens §4.3. Inference time grows roughly linearly as graphs are added §4.3 tab. 10.
• Figures carry numbers that are not in the text. Ablation values are only in fig. 5; read them from the PDF.

Lineage

• Builds on: Flan-T5 as one of the two backbones; Llama3-8B is cited as backbone and is not carded here.
• Compared against: GCN, GAT, GraphSAGE, RevGNN, SGFormer, NodeFormer app. B; BERT, Sentence-BERT, Llama3-8B app. B; OFA, LLaGA, GraphPrompter, ENGINE, GraphText app. B; GFT app. B; InstructGLM, KGT5-context for generative link prediction app. B.
• Superseded by: not known from content already in this wiki.