Logic Tensor Networks

In plain words

Neural networks learn well from raw data, but they find it hard to use general rules such as "friends of smokers are probably smokers". Logic Tensor Networks (LTN) let you write rules like this in first-order logic and train the network so that the rules hold as much as possible. Each statement gets a degree of truth between 0 and 1 instead of true or false, so a whole set of rules turns into one number that gradient descent can push up.

The same mechanism covers several tasks. A classifier is a predicate, a regression is a function that must agree with the target values, and clustering is a set of rules about which points belong together. One example from the paper: the network is never told the digit in each image, only the sum of two images, and it still learns to read digits because only digit pairs that add up to that sum are allowed by the rule.

The cost is that the logic is soft. How the operators behave, and how their gradients behave, depends on which fuzzy operators and which aggregation exponent you choose. The card lists the choices the paper used, and where its released code differs from the text.

TL;DR

Logic Tensor Networks (LTN) let one deep network learn from data and from first-order logic rules at the same time. Every formula gets a truth value in [0, 1], so a knowledge base has one satisfiability score that training maximises by gradient descent. The paper uses this one language for classification, multi-label constraints, regression, clustering, embedding learning and reasoning. Only the MNIST addition results are compared with a baseline; the other experiments are qualitative. The official code is MIT-licensed, but several example settings differ from the paper's text.

Problem

Sub-symbolic deep learning is data-hungry and struggles to generalise out of distribution. Symbolic AI is brittle with noisy data and can be computationally costly §1. The paper wants one formalism in which learning from data and reasoning over rich first-order knowledge are specified together §1.

Standard Boolean first-order logic cannot express degrees of confidence on noisy data, so the formulas are made fuzzy, with truth values in [0, 1] §2.1 §2.2. Gradient-based training also breaks with many standard fuzzy operators, which show vanishing or exploding gradients §2.4 [App. C].

Assumptions: every domain is interpreted as a set of real tensors, and variables are grounded on finite sequences of instances §2.2 §2.2.1.

Method

LTN has two parts: Real Logic, the language and its grounding, and a learning, query and reasoning procedure on top of it §2 §3.

1. Language. A signature has constants, variables, function symbols and predicate symbols. Each symbol has a domain type, and terms and atomic formulas must respect those types §2.1. Formulas use negation, conjunction, disjunction, implication and bi-conditional, plus universal and existential quantifiers §2.1.
2. Grounding. A grounding G maps each domain to a set of tensors. A variable is grounded on an ordered sequence of instances, not a set, so the same value may appear more than once §2.2.1 [Def. grounding]. Constants map to tensors, functions to tensor operations, and predicates to functions with output in [0, 1] §2.2.1.
3. Free variables become axes. A term or formula with n free variables gives a tensor with one axis per variable, sized by the number of instances of that variable. A predicate's feature axis has size 1 and is dropped §2.2.2.
4. Connectives and quantifiers. Each connective is a fuzzy operator applied element-wise over the joint axes of its operands. Each quantifier is an aggregator applied along the axes it quantifies §2.2.3. The paper's operators are the stable product configuration in step 6.
5. Diagonal quantification. Diag(x, y) quantifies only over matched pairs, so the i-th instance of x is paired with the i-th instance of y. This needs the two variables to have the same number of instances §2.2.3. The paper stresses that this is not the same as stacking the pairs into a new variable, because the stacking needs reshapes when the ranks differ [§2.2.3, footnote].
6. Stable product configuration (the one used in all examples). Conjunction is the product t-norm, disjunction the probabilistic sum, negation the standard complement, and implication the Reichenbach operator. Each operator first shifts its inputs away from the edge values 0 or 1 §2.4. The universal quantifier is a p-mean of errors (approximating min); the existential quantifier is a p-mean (approximating max); both need p ≥ 1 §2.4. Pseudocode below.
7. Guarded quantifiers. A mask selects which instances an aggregator sees. Masked-out instances get no gradient. The guarded form ∀x:m(x) φ(x) is not the same as ∀x (m(x) → φ(x)) §2.3.
8. Knowledge base and satisfiability. A theory is a triple of formulas K, a parametric grounding G_θ, and a hypothesis space. Satisfiability aggregates the truth values of all formulas in K with a formula aggregator §3.1 §3.1.4.
9. Learning. Learning picks θ that maximises Sat, optionally with a regulariser λR(θ) such as L1 or L2 §3.2. Learning the grounding of constants gives embeddings. Learning functions gives generative models or regression. Learning predicates gives classification §3.2.
10. Querying. Four query types: truth values of a formula, value of a term, truth values of a formula on unseen instances, and value of a term on unseen instances §3.3.
11. Reasoning. A formula φ is a consequence of K at threshold q (with 1/2 < q < 1) if every grounding with Sat ≥ q also gives φ ≥ q §3.4. Two approximations are given. (a) Query after learning checks φ on the groundings that maximise Sat §3.4. (b) Refutation searches for a grounding that satisfies K but makes φ fall below q §3.4. Pseudocode below.

Learning objective:

\theta^\ast = \arg\max_{\theta}\ \mathrm{Sat}_{\phi\in\mathcal{K}}\ \mathcal{G}_\theta(\phi) §3.2

In the examples, training minimises the loss 1 − Sat on mini-batches of data §4.1 §4.2.

Stable product connectives and quantifiers

# Stable product configuration  [§2.4]
# EPS is a small positive constant. The paper does not state its value.
# Code uses 1e-4  [code: ltn/fuzzy_ops.py#L12]
def not_zero(a):
    # shift values away from 0  [§2.4]
    return (1 - EPS) * a + EPS

def not_one(a):
    # shift values away from 1  [§2.4]
    return (1 - EPS) * a

def negate(a):
    # standard negation  [§2.4]
    return 1 - a

def conj(a, b):
    # conjunction: product t-norm on shifted values  [§2.4]
    return not_zero(a) * not_zero(b)

def disj(a, b):
    # disjunction: probabilistic sum on shifted values  [§2.4]
    return not_one(a) + not_one(b) - not_one(a) * not_one(b)

def implies(a, b):
    # implication: Reichenbach on shifted values  [§2.4]
    return 1 - not_zero(a) + not_zero(a) * not_one(b)

def forall(values, p):
    # universal: p-mean of errors, approximates min, p >= 1  [§2.4]
    errors = 1 - not_one(values)
    # aggregate over the quantified axes  [§2.2.3]
    mean_error = mean(errors ** p)
    return 1 - mean_error ** (1 / p)

def exists(values, p):
    # existential: p-mean, approximates max, p >= 1  [§2.4]
    shifted = not_zero(values)
    # aggregate over the quantified axes  [§2.2.3]
    return mean(shifted ** p) ** (1 / p)

Learning and refutation

# Learning by maximising satisfiability  [§3.2]
# Each step uses a mini-batch rather than the full data set  [§4.1]
def train(theta, knowledge_base, batches, epochs):
    for epoch in range(epochs):
        for batch in batches:
            # ground the data variables on this batch only  [§3.2]
            sat = satisfiability(knowledge_base, theta, batch)
            # loss = 1 - Sat, minimised with Adam, lr 0.001  [§4] [§4.1]
            loss = 1 - sat
            theta = adam_update(theta, gradient_of(loss, theta))
    return theta

# Soft penalty for the knowledge base, gap = q - Sat(K)  [§3.4] [fig. 6]
def soft_penalty(alpha, beta, gap):
    # linear while K is violated (gap >= 0)  [§3.4]
    if gap >= 0:
        return beta * gap
    # saturates towards -alpha once K holds  [§3.4]
    return alpha * (exp(gap) - 1)

# Refutation: find a grounding that satisfies K but falsifies phi  [§3.4]
def refutation_objective(theta, knowledge_base, phi, q, alpha, beta):
    # satisfiability of the knowledge base under theta  [§3.1.4]
    sat_kb = satisfiability(knowledge_base, theta)
    # minimise truth of phi, with the soft penalty on K  [§3.4]
    return truth(phi, theta) + soft_penalty(alpha, beta, q - sat_kb)

How the examples are encoded

• Binary classification §4.1. Knowledge base: ∀x₊ A(x₊) and ∀x₋ ¬A(x₋). A(x) = sigmoid(MLP(x)), where the MLP has two hidden layers of 16 units with elu and one output unit tab. 3. Data: 100 points uniform in [0, 1]²; a point is positive if it lies within 0.09 of (0.5, 0.5) (Euclidean distance per the footnote; see Paper vs code). Split 50/50. Stable product, ∀ = p-mean error with p = 2, formula aggregator p-mean error with p = 2, Reichenbach implication. 1000 epochs, batch 64. Accuracy threshold 0.5 §4.1.
• Multi-class single-label §4.2. One predicate P(x, l) with a softmax over three classes, so classes are mutually exclusive. Axioms: ∀x_A P(x_A, l_A), and the same for B and C. Iris data, 4 features. ∀ with p = 2, formula aggregator p = 2, batch 64, 500 epochs. Accuracy is top-1 §4.2.
• Multi-label §4.3. Predicate P(x, l) with a sigmoid over four labels (blue, orange, male, female). Axioms: four positive-example axioms, plus ∀x ¬(P(x, blue) ∧ P(x, orange)) and ∀x ¬(P(x, male) ∧ P(x, female)). No negative examples are given. Crabs data, 5 features, 200 examples. Accuracy is 1 − Hamming loss at threshold 0.5. Queries use ∀ with p = 5 §4.3.
• Single-digit addition §4.4. digit(x, d) = onehot(d)ᵀ · softmax(CNN(x)). Axiom: ∀Diag(x, y, n) ∃d₁, d₂ : d₁ + d₂ = n (digit(x, d₁) ∧ digit(y, d₂)). The mask keeps only pairs whose digits sum to n. Images are 28×28 grayscale, scaled to [0, 1]. Batch 32. Only the sums are used as labels, not the digits. The ∃ exponent follows a schedule (see Hyperparameters).
• Multi-digit addition §4.4. Same setup with four digit variables and the condition 10d₁ + d₂ + 10d₃ + d₄ = n.
• Regression §4.5. f = MLP with a linear output on 6 features. Smooth equality eq(u, v) = exp(−α·‖u − v‖), with α = 0.05 in the text. Axiom: ∀Diag(x, y) eq(f(x), y). Real-estate data: 414 examples, split 330 train and 84 test.
• Clustering §4.6. C(x, c) with a softmax over four clusters. Axioms: ∀x ∃c C(x, c), ∀c ∃x C(x, c), and two guarded rules. If |x − y| < 0.2 then C(x, c) ↔ C(y, c) for every c. If |x − y| > 1.0 then ¬(C(x, c) ∧ C(y, c)). Data: four Gaussian centres with 50 samples each. 1000 epochs §4.6.
• Smokes-friends-cancer embeddings §4.7. 14 individuals, each an embedding in R⁵. Predicates S, F and C are small sigmoid MLPs. Knowledge: known smoking facts in both groups, known cancer facts in group 1 only, and known friendship facts. Friendship is anti-reflexive and symmetric, everyone has a friend, smoking propagates along friendship, smoking implies cancer, and not smoking implies no cancer. The paper calls the knowledge base not satisfiable in the strict logical sense §4.7. 1000 epochs. Queries after training: ∀p (C(p) → S(p)) and ∀p,q ((C(p) ∨ C(q)) → F(p, q)).
• Reasoning by refutation §4.8. Knowledge base A ∨ B with propositional variables A and B, grounded on two real parameters initialised to 0, using probabilistic sum for ∨. Query: (A ∨ B) ⊨_q A with q = 0.95. Counterexample: A = 0, B = 1.

Implementation: TensorFlow 2 [App. A]. The networks used in each example are listed in tab. 3.

Hyperparameters

Results

Author-reported. Setup: MNIST addition (single and multi-digit), with the Siamese CNN baseline of the DeepProbLog paper and the DeepProbLog numbers as reported there §4.4. Other experiments are reported as figures or in words only.

Qualitative findings: binary classification can reach 100% accuracy before satisfaction is maximised §4.1. In multi-label training, the query "blue implies not orange" rises towards true, while "blue implies orange" and "blue implies male" fall towards false §4.3 fig. 12.

Paper vs code

Repository: https://github.com/logictensornetworks/logictensornetworks. The paper links it in the introduction, §4 and App. A §1 §4 app. A. It is MIT-licensed LICENSE. The card compares the text against code at commit f3027f3, which is the default branch at clone time. Files checked: the operator library, the core (predicates, quantifiers, diagonal, masks), the binary, single-digit, multi-digit, multi-class (single- and multi-label), regression, smokes and clustering examples, and the reasoning and operator tutorials. The paper's related-work section was not read.

• Binary data label. The paper's footnote defines the positive class as Euclidean distance below 0.09 §4.1. The code compares the squared distance with 0.09, which gives a radius of 0.3 [code: examples/binary_classification/binary_classification.py#L27].
• Regression equality. The text uses exp(−α·d) with α = 0.05 §4.5. The active line in the code is 1/(1 + 0.5·d), the paper's alternative form with α = 0.5. The exp form with 0.05 is commented out [code: examples/regression/regression.py#L42-L43]. The code's learning rate is 0.0005 regression.py, while the text sets 0.001 for all examples §4.
• Smokes embedding size and MLPs. The paper uses R⁵ embeddings §4.7 and predicates with Dense(8) hidden layers tab. 3. The code uses embedding size 10 smokes_friends_cancer.py and hidden layers of 16 units smokes_friends_cancer.py.
• Smokes ∀ exponent. The paper uses p = 6 for anti-reflexivity and symmetry §4.7. The code uses p = 5 smokes_friends_cancer.py [code: examples/smokes_friends_cancer/smokes_friends_cancer.py#L76].
• Smokes non-friendship pairs. The paper's axiom ranges over pairs with u > v §4.7. The code ranges over pairs with x < y [code: examples/smokes_friends_cancer/smokes_friends_cancer.py#L61-L62]. These are different orientations.
• MNIST ∃ schedule. The paper says p goes from 1 to 6 "gradually" §4.4. The single-digit code steps through 1, 2, 4, 6 at epochs 0, 4, 8 and 12, for 20 epochs by default [code: examples/mnist/single_digits_addition_ltn.py#L118-L126] [code: examples/mnist/single_digits_addition_ltn.py#L10-L11]. The multi-digit code steps through 1, 2, 4, 6 at epochs 0, 6, 12 and 18, for 30 epochs by default multi_digits_addition_ltn.py [code: examples/mnist/multi_digits_addition_ltn.py#L10]. It also has fixed-p options that match the paper's three scenarios [code: examples/mnist/multi_digits_addition_ltn.py#L14].
• Clustering ∃ schedule. The text says p = 1 for the first 100 epochs §4.6. The notebook uses p = 1 for epochs 0 to 100 inclusive [code: examples/clustering/clustering.ipynb#L263-L266].
• Output layers. Table 3 lists a single final Dense(1) for the multi-class and clustering rows, but the text says three and four output neurons tab. 3 §4.2 §4.6. The code uses three outputs for single-label [code: examples/multiclass_classification/multiclass-singlelabel.py#L44] multiclass-singlelabel.py. Multi-label uses four [code: examples/multiclass_classification/multiclass-multilabel.py#L66].
• Checked and matching. The empty-selection rule (∀ gives 1, ∃ gives 0) §2.3 is implemented core.py. Stable operators follow §2.4 [code: ltn/fuzzy_ops.py#L12-L154]. MNIST pixels are divided by 255 [code: examples/mnist/data.py#L8]. The reasoning penalty uses α = 0.05, β = 10, q = 0.95 4-reasoning.ipynb.

Pitfalls

• EPS is not stated. The stable configuration needs an arbitrarily small positive constant §2.4. The library uses 1e-4 fuzzy_ops.py. A tutorial suggests 1e-5 as an example [code: tutorials/2b-operators_and_gradients.ipynb#L262]. Pick one and document it.
• The stable product is not a t-norm. Its conjunction does not satisfy the identity element for values below 1 §2.4. Don't assume standard fuzzy-logic identities hold.
• Diagonal quantification is not a reshape. It only pairs the i-th instances and needs equal counts §2.2.3. Stacking variables into pairs looks similar but is not equivalent when the ranks differ §2.2.3.
• Guarded versus implication. ∀x:m(x) φ(x) and ∀x (m(x) → φ(x)) give different values. Masked-out instances get no gradient in the guarded form §2.3. For empty masks, ∀ returns 1 and ∃ returns 0 §2.3 core.py.
• Cost grows with the number of free variables. A formula with two free variables produces |G(x)| × |G(y)| values §2.2.2. Use diagonal quantification or masks to keep this down.
• The ∃ exponent is sensitive. A fixed p = 2 from the start gives poor results on multi-digit addition, and p = 1 gives satisfaction values that are hard to interpret for reasoning. A schedule that rises to 6 is preferred §4.4.
• Local optima. The MNIST optimiser stalls at initialisation in about one run in five §4.4. The paper reports the mean of the 10 best of 15 runs, so the headline figures involve selection §4.4.
• Satisfaction is not accuracy. A model can reach 100% accuracy while the knowledge base is not fully satisfied, if its confidence stays at 0.7 §4.1.
• Querying after learning is not refutation. Re-optimising Sat from several starts converged to the same grounding and missed the counterexample §4.8 fig. 22. Refutation found it §4.8 fig. 23. The refutation penalty depends on q, α and β §3.4.
• Smooth equality scale. α sets how strict the smooth equality is §4.5. The active code line uses α = 0.5 in the alternative form, see Paper vs code.
• Unstated details. The paper does not state the Iris and crabs train/test splits §4.2 §4.3, the MNIST test size or epoch count §4.4, the regression batch size §4.5, or the smokes batch size §4.7. The code uses 120/30 Iris rows [code: examples/multiclass_classification/iris_training.csv] [code: examples/multiclass_classification/iris_test.csv] and 160/40 crabs rows [code: examples/multiclass_classification/multiclass-multilabel.py#L37-L38].
• Exclusivity. Multi-class single-label uses softmax, so one-hot exclusivity is built into the predicate. Adding explicit exclusion axioms is unnecessary there §4.2. Multi-label uses sigmoid and needs the exclusion axioms written out §4.3.

Lineage

• Builds on: Real Logic and Logic Tensor Networks were first introduced in an earlier paper by Serafini and colleagues §1 (not compiled in this wiki). The fuzzy operator analysis is from van Krieken et al.'s differentiable fuzzy logic §1 §2.4 (not compiled in this wiki).
• Compared against: DeepProbLog on MNIST addition §4.4 (not compiled in this wiki). Markov logic networks are discussed as the origin of the smokers example §4.7 (not compiled in this wiki).
• Superseded by: not known from content in this wiki.