Convolutional Networks on Graphs for Learning Molecular Fingerprints

In plain words

A molecular fingerprint is a fixed list of numbers that describes a molecule, and prediction models often take that list as input. The classic recipe fixes the list in advance: each piece of a molecule's structure is hashed into a slot, so the list has to be long to keep pieces apart, and it never changes to suit the task.

This work keeps the same shape of computation but makes every step learnable. Each atom takes in information from the atoms bonded to it, and each round of this adds a soft tally of which atoms look like which slot into one shared vector. Because every step is differentiable, the fingerprint is trained together with the prediction model on labelled molecules, so the features come from the data.

The learned fingerprints can be shorter, and each feature responds to a family of related fragments rather than one exact substructure, which makes them easier to read. On the three datasets the authors test, the learned version does about as well as the classic one or better.

TL;DR

The paper turns circular (ECFP-style) molecular fingerprints into a neural network: atoms exchange features with their neighbours, and each layer adds a softmax-weighted count of atoms into a fixed-length vector alg. 2 §3. Trained end-to-end with a small regression head, it matches or beats circular fingerprints on aqueous solubility, antimalarial drug efficacy and organic photovoltaic efficiency, with the best results when a one-hidden-layer network sits on top tab. 1 §4.2. Numbers are author-reported. Read the Pitfalls section before reimplementing: the paper omits several details that the code settles, and the code departs from the paper in a few places.

Problem

Standard molecular property models need a fixed-size input, but molecules vary in size and shape §1. The usual answer is a hand-designed circular fingerprint, whose bits each mark one substructure, and such fingerprints must be very large to avoid collisions: one cited model used a fingerprint of length 43,000 after removing rare features §1. Fixed fingerprints also cannot be tuned for the task, and each bit means exactly one fragment, so similar fragments do not share activation §1 §4.1.

Method

The paper replaces each discrete step of the circular fingerprint algorithm with a differentiable analogue §3 alg. 1 alg. 2. The network runs in three parts:

1. Input features. Each atom gets a vector built from its element, degree, number of attached hydrogens, implicit valence and aromaticity. Each bond gets a vector of bond type, conjugation and ring membership. Hydrogens are implicit §4.2 features.py.
2. Neighbourhood update (replaces hashing) §3 "Hashing" alg. 2. In each layer, every atom sums the features of its neighbours, each neighbour concatenated with the feature of the bond to it. That sum is multiplied by a weight matrix chosen by the atom's degree, which lets the network distinguish bonding patterns. The atom's own features are added through a separate matrix build_convnet.py [code: neuralfingerprint/build_convnet.py#L58-L70]. The result goes through a smooth nonlinearity; the paper's experiments use ReLU §4.2.
3. Soft indexing and summation (replaces indexing) §3 "Indexing" alg. 2. After each layer, every atom produces a softmax over the L fingerprint slots. The atoms' vectors are summed into a per-layer fingerprint, and the per-layer fingerprints are summed into the final one [code: neuralfingerprint/build_convnet.py#L81-L96].
4. Permutation invariance (replaces canonicalisation) §3 "Canonicalization". Circular fingerprints sort neighbours into a canonical order. The paper instead sums over neighbours, which is invariant to ordering and simpler to implement; the authors report that they tried sorting and all-permutation alternatives §3.
5. Why this generalises circular fingerprints §3, last paragraph. With large random input weights, tanh approaches a step function, which acts as a hash, and softmax approaches an argmax, which acts as indexing. So random, large-weight networks behave like circular fingerprints, and training moves them away from that regime §3 §4.

The forward pass is written in our own words below. It follows alg. 2, then adds the bias, batch normalisation and degree-specific matrices that the code uses build_convnet.py.

# Neural graph fingerprint, forward pass  [alg. 2]
# Input:  molecule graph; depth R; width F; fingerprint length L
# Output: real-valued fingerprint, shape (L,)

def neural_fingerprint(molecule, R, F, L):
    # Initial atom states from the atom features  [§4.2]
    states = atom_features(molecule)                   # (N, F0)
    fingerprint = zeros(L)
    for layer in range(R + 1):
        # Each atom softmaxes its vector over the L slots; sum over atoms
        # [§3 "Indexing"] [alg. 2] [code: build_convnet.py#L84-L88]
        votes = softmax(states @ W_out[layer] + b_out[layer])  # (N, L)
        fingerprint = fingerprint + sum_over_atoms(votes)
        if layer < R:
            states = update_states(molecule, states, layer)
    return fingerprint

# One message-passing layer  [alg. 2] [§3 "Hashing"]
# Input:  molecule graph; states (N, F); layer index
# Output: new states (N, F)

def update_states(molecule, states, layer):
    new_states = zeros(N, F)
    for atom in molecule.atoms:
        # Sum neighbour states, each concatenated with its bond features
        # [code: build_convnet.py#L25-L30]
        message = sum_over_neighbours(
            concat(states[neighbour], bond_features(bond))
        )
        # The weight matrix depends on the atom's degree, 0 to 5
        # [code: mol_graph.py#L5] [code: build_convnet.py#L55-L56]
        new_states[atom] = (
            message @ W_degree[layer][degree(atom)]
            + states[atom] @ W_self[layer]
            + bias[layer]
        )
    # Normalise each feature over all atoms in the input, then ReLU
    # [§4.2] [code: neuralfingerprint/build_convnet.py#L68-L70]
    # [code: neuralfingerprint/build_vanilla_net.py#L7-L13]
    return relu(batch_normalize(new_states))

Hyperparameters

Values fixed by the paper are cited to the paper; values searched or set only in the code are marked as such. The paper gives no numeric ranges for the searched values, so the code's ranges are the only source for them.

Results

Author-reported. Setup: three regression datasets §4.2 (aqueous solubility, antimalarial EC50, organic photovoltaic efficiency), linear or one-hidden-layer prediction heads, and circular fingerprints as the baseline §4.2. Hardware is not stated. The paper says training took about an hour on the larger datasets §5. The table's ± is not defined in the paper, so do not compare it against confidence intervals from other papers.

Bold marks the paper's own best entries per column tab. 1. On drug efficacy the circular fingerprint with a linear head (1.13) is numerically slightly better than the neural fingerprint with a linear head (1.15), so "matched or beat" is the accurate summary there.

Similarity analysis §4: on the solubility set with fingerprints of length 2048, the continuous Tanimoto-style distance eq. 1 between neural and circular fingerprints correlates at r = 0.823 §4 fig. 3. With large random weights, neural fingerprints track circular fingerprints in predictive performance fig. 3; with small random weights they do better than either §4.

Interpretability §4.1: the authors show, per feature, the fragments that most activate it fig. 4 fig. 5. For solubility, the most pro-solubility feature fires on hydrophilic R–OH groups and the most anti-solubility feature on non-polar repeated rings §4.1. For toxicity, one feature picks out sulphur on an aromatic ring and another fused aromatic rings fig. 5. These fragment visualisations are generated automatically, without a hand-made list of substructures §4.1.

Paper vs code

Repository: https://github.com/HIPS/neural-fingerprint, checked at commit d26dae3a293a736261fee307b28b37e548c5f218. The repository's license is MIT (the copyright line names Harvard, where the authors work) license.txt. Code paths below are relative to the repository root.

• Fingerprint depth range. The paper searches depth "up to 6" §4.2. The search bounds in the code are [1, 4] for both the neural and the Morgan models launch_experiments.py [code: experiment_scripts/launch_experiments.py#L59]. Run the search you intend to reproduce, and do not assume depth 6.
• Self term in the update. Algorithm 2 sums the atom's own state into the neighbour sum and applies one degree-specific matrix alg. 2. The code keeps the atom's own state separate, through a single matrix that is not degree-specific, and adds it to the degree-specific neighbour term [code: neuralfingerprint/build_convnet.py#L62-L67].
• Bias and softmax. Algorithm 2 writes softmax(r_a W_L) with no bias alg. 2. The code adds a per-layer output bias before the softmax [code: neuralfingerprint/build_convnet.py#L82-L84].
• Number of fingerprint writes. Algorithm 2 writes once per layer L = 1 to R, after each update alg. 2. The code also writes the input layer, before any update, so it writes R + 1 times build_convnet.py.
• Degree-specific matrices. The paper says one matrix per possible number of bonds, "up to 5" §3. The code has six degree buckets, 0 to 5 [code: neuralfingerprint/mol_graph.py#L5] build_convnet.py.
• Batch normalisation. Algorithm 2 has no normalisation step, and §4.2 says batch normalisation was used. The code computes (x − mean)/(std + 1) over all atoms in the input array and uses those batch statistics at prediction time too build_vanilla_net.py [code: neuralfingerprint/build_convnet.py#L68-L69].
• Fingerprint L2. The paper says the L2 penalty is searched §4.2. The code applies it to the prediction head only; the fingerprint weights get no penalty in the experiments because the default fingerprint penalty is 0 [code: neuralfingerprint/build_convnet.py#L126-L129] [code: experiment_scripts/eval_methods.py#L76].
• Solubility size. The paper says 1144 molecules §4.2. The data file has 1128 rows, and the cross-validation code uses 1124 [code: data/2015-05-24-delaney/delaney-processed.csv] [code: experiment_scripts/launch_experiments.py#L48].
• Drug efficacy units. The table says EC50 in nM tab. 1. The preprocessing takes the natural log of the raw value, so reported losses are on the log scale preprocess_data.py.
• Toxicity. The validation sweep also runs the SR-MMP toxicity set [code: experiment_scripts/launch_experiments.py#L47-L48]. The paper reports no numbers for it; it appears only as Figure 5 fig. 5.
• Predict-mean baseline. The code learns the constant by gradient descent on the same loss, rather than computing the training mean [code: neuralfingerprint/build_vanilla_net.py#L111-L120]. The paper does not say which.
• Hyperparameter draws. The random draws use a fresh RandomState(0) for each dataset, fold and model, so every fold of a model sees the same 50 configurations launch_experiments.py. The paper does not say.
• Model selection. The code picks the configuration with the lowest loss at the last training step on the validation chunk, then retrains on train plus validation with seed 0 util.py launch_experiments.py.

Pitfalls

• Predictions depend on the batch. Batch normalisation uses the statistics of the whole array passed in, both in training and at prediction, and no running averages are kept build_convnet.py [code: neuralfingerprint/build_vanilla_net.py#L48-L51]. The same molecule can get a different fingerprint depending on what else is in the batch. Keep the batch composition fixed when comparing models, or implement running statistics and say so.
• Batch normalisation epsilon. The denominator is std + 1, not std + eps. For features with small spread the normalised values are shrunk toward zero. The paper does not state this; implement the code's form if you want to match it build_vanilla_net.py.
• Fingerprints scale with molecule size. The softmax is taken per atom over the L slots, so each atom contributes a total of 1 per layer and each layer's fingerprint sums to the number of atoms [code: neuralfingerprint/build_convnet.py#L84-L88]. Large molecules get larger fingerprint values. Neither the paper nor the code normalises this.
• Degree above 5 crashes. Atoms with more than five neighbours have no degree bucket, so both the sort and the feature code fail [code: neuralfingerprint/mol_graph.py#L22-L26] features.py. The code's own comment calls the degree sort "brittle" [code: neuralfingerprint/mol_graph.py#L71]. Filter or handle such atoms before use.
• Unknown atom values are silently mapped. Element symbols outside the list go to the "Unknown" bucket; total H count above 4 maps to 4; valence above 5 maps to 5 features.py [code: neuralfingerprint/util.py#L114-L118]. The paper does not describe this.
• Target scaling. Regression targets are z-scored with training-split statistics, and the loss is computed on the scaled values. Predictions must be un-scaled before any metric is reported util.py [code: experiment_scripts/eval_methods.py#L103-L108].
• Metric is not named. The paper does not name the metric in Table 1. The code reports mean squared error on the original scale, not RMSE [code: neuralfingerprint/build_vanilla_net.py#L18-L19]. The table's values may be MSE; this card does not confirm that.
• Minibatches are fixed. Minibatches are contiguous cyclic slices of the already shuffled data file, with no reshuffling between passes [code: neuralfingerprint/util.py#L126-L131] preprocess_data.py.
• Cost. Cost is O(RNFL + RNF²) for N atoms, R layers, F features and length L §5, so the hidden-to-hidden terms matter for wide networks. Training took about an hour on the larger datasets §5.
• Limits the abstract hides. Information travels only as far as the depth R, so the model cannot tell apart graphs needing N/2 layers in the worst case §5. It does not distinguish stereoisomers §5. Those limits are stated in the paper's limitations section.
• Interpretation by search. The fragment visualisations are the most-activating examples for each feature, found by searching data §4.1, not a causal account.

Lineage

• Builds on: circular fingerprints (ECFP) and the Morgan algorithm §2; graph-structured neural networks, including Lusci et al. (2013) §6; spectral graph convolution by Bruna et al. (2013) §5 §6. Circular fingerprints are the direct ancestor: the paper replaces each of their steps §3.
• Compared against: circular fingerprints with a linear head or a one-hidden-layer network, and a predict-mean baseline tab. 1. The paper also cites multitask and deep-network QSAR work on circular fingerprints as related, without comparing against it §1 §6.
• Superseded by: none listed in this wiki.