Molecular representation learning

Overview

A model that predicts a molecule's behaviour needs the molecule as numbers. A compound encoder maps the molecule to a vector, and the rest of the model works on that vector. In DeepPurpose the compound input is a SMILES string, a text notation for the molecule 2004.08919 §2.1. BOOM uses several inputs: descriptors, SMILES, a 2D bond graph, and 3D atom positions or distances 2505.01912 tab. 3. GeoDiff goes the other way: it takes a bond graph and produces 3D atom coordinates 2203.02923 §3.1.

The encoders differ in what they read. They use precomputed fingerprints or descriptors, fingerprints that are learned end to end with the predictor, learned sequence models over characters or substructure tokens, message passing over the bond graph, attention over all atom pairs with graph-structure biases, or 3D models that also read geometry 2004.08919 app. A 2505.01912 tab. 3 2106.05234 §3.1 2103.09430 §2.3.

The shared invariant is that each predictive encoder outputs a vector. In DeepPurpose that vector is joined with a protein embedding and passed to an MLP 2004.08919 §2.2. In BOOM the models are used off the shelf, and only non-architectural settings were tuned 2505.01912 app. C.1. GeoDiff does not fit this pattern. Its network predicts noise on coordinates inside a sampling loop, and the conformation that loop produces is the output 2203.02923 alg. 1. Protein encoders are out of scope for this page.

The sources test different things. DeepPurpose compares encoder pairs on drug-target binding benchmarks 2004.08919 §3. The learned-fingerprint paper tests three regression sets: solubility, antimalarial drug efficacy and organic photovoltaic efficiency 1509.09292 §4.2. BOOM compares 15 models on 10 molecular properties, with a split that holds out the molecules with the most unusual property values 2505.01912 §2. OGB-LSC regresses the HOMO-LUMO gap of PubChem molecules from 2D graphs, split by compound ID, with a fixed inference budget 2103.09430 §2.3. GeoDiff measures generated conformations against reference ensembles on GEOM-QM9 and GEOM-Drugs 2203.02923 §5.1. Their numbers are not comparable.

Variants

Fingerprints or descriptors with an MLP or random forest

DeepPurpose computes Morgan, PubChem, Daylight and RDKit 2D fingerprints from the molecule and passes them to an MLP. The paper's sizes are Morgan with radius 2 and 1024 bits, PubChem with 881 bits, Daylight with 2048 bits, and RDKit 2D with 200 values 2004.08919 app. A. Card: DeepPurpose: A Deep Learning Library for Drug-Target Interaction Prediction.

BOOM feeds RDKit molecular descriptors to a random forest and to an MLP. The descriptor set is 125 chemistry features and 86 functional-group fractions 2505.01912 app. B.2. The MLP has 153k parameters 2505.01912 tab. 3. On the 10 OOD tasks, the random forest's error is higher than the MLP's on 8 2505.01912 tab. 7. Card: BOOM.

OGB-LSC uses a 2048-bit Morgan fingerprint as the input of an MLP baseline for PCQM4M, with a grid over radius 2 and 3 and a 16.1M-parameter model 2103.09430 §2.3 tab. 6. Its test MAE is 0.2070 eV, against 0.1494 for the best single GNN in the same table 2103.09430 tab. 6. Card: OGB-LSC.

Learned fingerprints trained end to end

Neural graph fingerprints replace the hashing and indexing steps of circular fingerprints with differentiable ones 1509.09292 §3 alg. 2. Each layer lets every atom take in its neighbours' features, and each atom then gives a softmax over the fingerprint slots. The votes are summed into the fingerprint, so the fingerprint is trained together with the prediction head 1509.09292 §3 alg. 2. In the paper's Table 1, the neural fingerprint with a one-hidden-layer head gives 0.52 on solubility against 1.40 for circular fingerprints with the same head, and 1.43 against 2.00 on photovoltaic efficiency 1509.09292 tab. 1. The table does not name its metric, and the values are author-reported. On drug efficacy, the circular fingerprint with a linear head (1.13) is slightly better than the neural fingerprint with a linear head (1.15) 1509.09292 tab. 1. The paper's depth search says "up to 6"; the released code searches depths 1 to 4 1509.09292 §4.2. Card: Convolutional Networks on Graphs for Learning Molecular Fingerprints.

CNN over SMILES characters

A one-dimensional CNN runs over the characters of the SMILES string, followed by global max pooling over positions 2004.08919 app. A. The learned part is the network, not the input features. Card: DeepPurpose.

CNN with a bidirectional recurrent layer

The CNN output feeds a bidirectional GRU or LSTM 2004.08919 app. A. The paper does not state the recurrent layer sizes. Card: DeepPurpose.

Transformer over substructure tokens

The input is split into substructure tokens rather than characters, and a Transformer runs over them 2004.08919 app. A. The paper does not state the Transformer sizes. Card: DeepPurpose.

Pretrained Transformers over SMILES

BOOM evaluates ChemBERTa (83M parameters), MoLFormer (48M), Regression Transformer (27M) and ModernBERT (111M), all reading SMILES 2505.01912 tab. 3. Pretraining improved ID RMSE by 31% for ChemBERTa, 35% for MoLFormer and 12% for the Regression Transformer. OOD RMSE did not change significantly, and binned OOD R² fell by 53% for MoLFormer and 39% for ChemBERTa 2505.01912 §4.2 fig. 4. ModernBERT had lower OOD RMSE than the other Transformers on heat of formation and C_v 2505.01912 §4.6 fig. 8; the size of that gain is disputed, see Contradictions. Card: BOOM.

Message passing on the 2D graph

DeepPurpose's MPNN runs message passing on the molecular graph 2004.08919 app. A. BOOM's Chemprop and TGNN use atom and bond features with permutation symmetry 2505.01912 tab. 3. Chemprop's OOD error fell after a random search over 50 configurations per property, for density (−60%), heat of formation (−23%) and ZPVE (−50%) 2505.01912 §4.3 fig. 6. That search selected each configuration on the OOD test set, so these gains are optimistic 2505.01912 app. C.4. Cards: DeepPurpose, BOOM.

OGB-LSC's GIN and GCN models run on the 2D graph, with a virtual node linked to every atom in both directions and sum pooling over atoms [arxiv:2103.09430 §2.3]. The GIN with a virtual node has validation MAE 0.1396 eV and test MAE 0.1494 eV, the best single model in its table 2103.09430 tab. 6. The paper's model-size sweep gives 0.1410 for the same width and depth 2103.09430 tab. 7, so cite one source and say which. The KDD Cup winners reached about 0.121 on test, but they used ensembles [arxiv:2103.09430 §2] tab. 6. Card: OGB-LSC.

Graph Transformer with structural encodings

Graphormer runs a standard Transformer over all nodes of a graph, plus one virtual graph token whose final state is the readout 2106.05234 §3.2. Three structural terms enter the attention: a learned embedding for node degree, a learned bias for each shortest-path distance, and a learned bias from the edge features along the path 2106.05234 §3.1 eq. 6 eq. 7. In the authors' PCQM4M-LSC ablation, validation MAE is 0.2276 with no node relation and no centrality, and 0.1304 with spatial relation, centrality and the attention bias 2106.05234 tab. 5. Author-reported single-model results include MolHIV AUC 80.51 with FLAG, MolPCBA AP 31.39 with FLAG and ZINC test MAE 0.122 for the Slim model 2106.05234 tab. 2 tab. 3 tab. 4. Attention cost grows with the square of the node count, which the authors name as a main limitation 2106.05234 §D §6. Card: Do Transformers Really Perform Bad for Graph Representation?.

Learnable positional vectors on message passing

LSPE adds a separate positional vector to each node. It starts from random-walk return probabilities (RWPE) or Laplacian eigenvectors, and the vector is updated in every layer alongside the node features 2110.07875 §3.1 §3.2. The authors apply it to two sparse GNNs, GatedGCN and PNA, and to two fully connected Transformer GNNs, SAN and GraphiT 2110.07875 tab. 1. With RWPE, GatedGCN's ZINC test MAE falls from 0.251 to 0.090 with the optional Laplacian loss, and PNA's from 0.141 to 0.095 2110.07875 tab. 1. On the two OGB molecular sets the authors report results comparable to the state of the art, not clear wins: GatedGCN-LSPE reaches 0.7754 test ROC-AUC on MOLTOX21, against 0.7757 for GIN-VN 2110.07875 §4.2 tab. 2. The Laplacian term in the released code is detached, so the Laplacian-loss rows may not reflect the written loss 2110.07875 §3.1. Card: GNN with learnable structural and positional representations.

3D equivariant and invariant networks

BOOM's 3D models read atom positions or pairwise distances. EGNN and IGNN each have 217k parameters, MACE has 3.9M and GotenNet has 6M 2505.01912 tab. 3. The hybrid models Graphormer-3D, GeoFormer and Equivariant Transformer also read 3D inputs 2505.01912 app. B.5. GotenNet has the lowest OOD RMSE on 7 of 10 tasks, and MACE on two 2505.01912 §4.1 tab. 7. The paper's text does not mention EGNN, which has the lowest OOD RMSE on density 2505.01912 tab. 7. Card: BOOM.

Diffusion over 3D coordinates

GeoDiff models the distribution of 3D atom coordinates given the bond graph. Sampling starts from Gaussian noise and reverses a diffusion process in many small steps 2203.02923 §4.1 2203.02923 alg. 1. Its noise network is built from equivariant layers 2203.02923 eq. 5 to 2203.02923 eq. 7. The prior has no centre-of-mass drift, and the paper proves that the generated density is then invariant to rotation and translation if every reverse kernel is equivariant 2203.02923 §4.2. The paper argues that predicting distances first and solving for coordinates accumulates error, which direct generation avoids 2203.02923 §2.

Sampling needs one network evaluation per step, 5000 steps at the paper's default 2203.02923 tab. 4. On GEOM-Drugs without force field, GeoDiff-C reports a mean COV-R of 89.13%, against 67.16% for GeoMol, the best baseline in that table 2203.02923 tab. 1. The main model is the chain-rule variant GeoDiff-C, not GeoDiff-A 2203.02923 §5.2. The force-field rows are a different protocol and should not be compared with the rows without it 2203.02923 tab. 2. Card: GeoDiff.

When to use which

The sources do not rank encoders for every case. The choices below follow from each encoder's inputs and from the reported results.

Two cautions apply across rows. First, ID and OOD performance are only moderately correlated across tasks, so a model's ID score is a weak guide to its OOD score 2505.01912 §4.7. Pick models on OOD results. Second, BOOM's Lipophilicity check uses five models, and its split procedure is not stated 2505.01912 tab. 9 app. C.9. MoLFormer had the lowest OOD RMSE there (0.956), and Chemprop had 1.051 2505.01912 tab. 9.

DeepPurpose reported evidence. The results are on DAVIS, with values read from a low-resolution figure, so check them against the PDF. Of the two DeepPurpose pairs listed for DAVIS, MPNN with AAC has the lower MSE and the higher concordance index than CNN with CNN 2004.08919 §3 fig. 1 D. The pairs also differ in protein encoder, so the gap cannot be attributed to the compound encoder alone 2004.08919 §3.

BOOM LLM check. Llama 3.1 8B without fine-tuning had very large OOD errors, for example 262.99 for heat of formation. The authors caution against comparing these numbers with the main table, because of possible training-data overlap 2505.01912 app. D tab. 11.

Implementation sketch

The common core is a function that maps one compound to a vector. The variants differ only in the branch taken for the molecule's input.

# Compound encoder, one call per molecule
# Input:  smiles (str); positions (n, 3) for 3D variants only
# Output: a vector; its length depends on the variant

def encode_compound(smiles, variant, positions=None):
    if variant == "fingerprint":
        # Computed bit vector, then an MLP  [arxiv:2004.08919 app. A]
        bits = fingerprint(smiles)
        return mlp(bits)
    if variant == "descriptors":
        # RDKit descriptor vector, then a random forest or MLP
        # [arxiv:2505.01912 app. B.2]
        return descriptors(smiles)
    if variant == "learned_fingerprint":
        # Differentiable fingerprint, trained with the head  [arxiv:1509.09292 alg. 2]
        return mlp(learned_fingerprint(molecular_graph(smiles)))
    if variant == "cnn":
        # 1D convolutions over SMILES characters, then global max pooling
        # over positions  [arxiv:2004.08919 app. A]
        return global_max_pool(conv1d(embed_chars(smiles)))
    if variant == "cnn_rnn":
        # Same CNN, then a bidirectional GRU or LSTM  [arxiv:2004.08919 app. A]
        return rnn_summary(conv1d(embed_chars(smiles)))
    if variant == "transformer":
        # Input is substructure tokens, not characters  [arxiv:2004.08919 app. A]
        return transformer_summary(embed_tokens(substructure_tokens(smiles)))
    if variant == "smiles_transformer":
        # Tokens are SMILES characters; pretraining is optional
        # [arxiv:2505.01912 app. B.3] [§4.2]
        return transformer_summary(embed_tokens(smiles_tokens(smiles)))
    if variant == "graph":
        # Message passing on the 2D bond graph, then a readout
        # [arxiv:2004.08919 app. A] [arxiv:2505.01912 tab. 3]
        return readout(message_passing(molecular_graph(smiles)))
    if variant == "geometric":
        # Needs atom positions as well as the graph  [arxiv:2505.01912 app. B.4]
        return readout(equivariant_passing(molecular_graph(smiles), positions))
    if variant == "graph_transformer":
        # Attention over all node pairs, plus structural biases  [arxiv:2106.05234 §3.1]
        return readout(transformer(embed_nodes(graph), structural_bias(graph)))
    if variant == "learnable_pe":
        # Positional vector per node, initialised from RWPE, updated each layer
        # [arxiv:2110.07875 §3.1]
        return readout(message_passing(graph, pos=rwpe(graph)))
    raise ValueError(variant)

The generative branch does not fit this function. GeoDiff's network returns a noise estimate inside a sampling loop, and the coordinates are the output, so it needs its own sampler 2203.02923 alg. 1.

Where the variants differ

• Input units. Character CNNs, SMILES Transformers and substructure Transformers read different tokens from the same string 2004.08919 app. A 2505.01912 tab. 3.
• Output. Predictive encoders return a vector for a downstream head. A generative model returns coordinates from a sampling loop 2203.02923 alg. 1. OGB-LSC clamps its predicted gap to [0, 50] eV for v1 and [0, 20] eV for v2, and the code applies the clamp only at evaluation 2103.09430 §2.3 app. C gnn.py.
• Geometry. The 3D variants need atom positions or distances. A SMILES string alone does not provide them, and this page does not cover how to produce them 2505.01912 tab. 3. GeoDiff generates positions, so it does not need them as input 2203.02923 §3.1. In the PCQM4M v2 dataset, 3D structures exist only for training molecules, so a 3D model cannot use them at test time 2103.09430 app. C.
• Readout. Fingerprints and descriptors need no readout. The MPNN readout is named as mean or sum in DeepPurpose; see Contradictions. Graphormer reads out from its virtual graph token 2106.05234 §3.2.
• Batch dependence. The learned fingerprint says batch normalisation was used 1509.09292 §4.2. The released code normalises over every atom in the input array and keeps no running statistics, so the same molecule can get a different fingerprint depending on its batch. The card's Paper vs code section records this. Keep batch composition fixed when comparing models, or use running statistics and say so.
• Length caps. The papers do not state the maximum SMILES or token length for every encoder. Check the reference implementation before relying on a cap 2004.08919 app. A.
• Paper and code. GeoDiff's released code differs from its paper in ways that matter for reimplementation. The paper's noise network takes the step t 2203.02923 eq. 4, and its loss has per-step weights [arxiv:2203.02923 eq. 9]. The released denoiser takes no timestep input, scales its coordinates differently, and hard-codes its loss weights, as read from the repository in this wiki's card. A time-conditioned reimplementation will not match the released weights.

Contradictions

• MPNN readout. DeepPurpose names the readout as "mean/sum" 2004.08919 app. A. This page does not resolve which one an implementation should use.
• Descriptor models labelled SMILES. BOOM's results table labels the random forest and MLP as SMILES, while its model summary and appendix say they use RDKit descriptors 2505.01912 tab. 7 tab. 3 app. B.2. This page follows the descriptor description.
• Graph versus SMILES. The paper's text says graph models perform comparably to Transformer models if the Regression Transformer is discounted 2505.01912 §4.4. The averaged table, which includes that model, shows graph models with lower OOD RMSE than SMILES models on 9 of 10 columns 2505.01912 tab. 1. The Regression Transformer's OOD errors are very large 2505.01912 tab. 7, so the averages depend on it.
• ModernBERT gain. The text says ModernBERT cuts OOD RMSE for heat of formation by more than 58% against the best other Transformer 2505.01912 §4.6. The table values give about 52%: 44.49 for ModernBERT against 93.6 for MoLFormer, the lowest other Transformer on that task 2505.01912 tab. 7. For C_v the table values give about 78% against ChemBERTa, which agrees with the text. This page reports the table values and does not resolve the gap.
• Cross-paper ZINC comparison. LSPE reports GatedGCN-LSPE at 0.090 test MAE and compares it with 0.122 for Graphormer, claiming a 26.23% gain 2110.07875 §4.2 tab. 2. The Graphormer card reports 0.122 for its Slim model 2106.05234 tab. 4. Each paper runs its own training setup, so this page does not rank the two.

Papers

• Convolutional Networks on Graphs for Learning Molecular Fingerprints: differentiable circular fingerprints trained end to end, compared with circular fingerprints on solubility, drug efficacy and photovoltaic efficiency.
• DeepPurpose: A Deep Learning Library for Drug-Target Interaction Prediction: an open-source library with fingerprint, CNN, recurrent, Transformer and message-passing compound encoders, and DAVIS and KIBA comparisons.
• BOOM: a benchmark of 15 models across descriptor, SMILES, 2D graph and 3D representations, with ID and OOD results on 10 properties.
• GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation: a diffusion model over 3D atom coordinates with equivariant noise networks, evaluated on GEOM-QM9 and GEOM-Drugs.
• Do Transformers Really Perform Bad for Graph Representation?: Graphormer, a Transformer over graph nodes with degree, shortest-path and edge-path structural biases, evaluated on PCQM4M-LSC, MolHIV, MolPCBA and ZINC.
• Graph Neural Networks with Learnable Structural and Positional Representations: LSPE, learnable per-node positional vectors for GNNs and Transformer GNNs, initialised from RWPE or Laplacian eigenvectors, evaluated on ZINC and two OGB molecular sets.
• OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs: three large graph benchmarks, including PCQM4M, a 2D-graph HOMO-LUMO regression with GCN, GIN and MLP-fingerprint baselines and a KDD Cup 2021 summary.