Differentiable fuzzy logic

Overview

Logical rules are exact, but gradient descent needs a number that changes smoothly with the parameters. Differentiable fuzzy logic gives every formula a truth value in [0, 1] and defines each connective and quantifier as a smooth function of its operands 2012.13635 §2.1 2012.13635 §2.2. A knowledge base then has one satisfiability score, and training maximises it 2012.13635 §3.2. In practice the loss is 1 minus that score, averaged over mini-batches 2012.13635 §4.

The shared pattern is: ground the variables on data, evaluate each formula to a truth value, aggregate the axioms, and take a gradient step. Every variant differs in which operators it uses and how strictly the quantifiers aggregate. The operators must keep gradients usable, which many standard fuzzy operators do not 2012.13635 §2.4.

Variants

The only compiled paper is the Logic Tensor Networks (LTN) framework. Its design is the stable product configuration, used in every example in the paper 2012.13635 §2.4 2012.13635 §4. Card: Logic Tensor Networks.

Stable product configuration

• Connectives. Conjunction is the product of truth values. Disjunction is the probabilistic sum a + b − ab. Negation is 1 − a. Implication is Reichenbach's, 1 − a + ab 2012.13635 §2.4. Before each operator, inputs are shifted away from exactly 0 or 1 by a small constant 2012.13635 §2.4.
• Quantifiers. The universal quantifier is a p-mean of the errors 1 − a, which approximates a minimum. The existential quantifier is a p-mean of the truth values, which approximates a maximum. Both need p ≥ 1 2012.13635 §2.4.
• Free variables become axes. A formula with several free variables is evaluated on the product of their instance sets, and each quantifier aggregates over the axes it binds 2012.13635 §2.2.2 2012.13635 §2.2.3.
• Diagonal quantification. Diag(x, y) pairs the i-th instance of x with the i-th instance of y, so it needs equal counts and is not the same as stacking the variables into one 2012.13635 §2.2.3.
• Guards. A mask restricts a quantifier to the instances it selects. Masked-out instances give no gradient. The guarded form is not the same as writing the mask as an implication inside the quantifier 2012.13635 §2.3.

Learning and reasoning on top

• Satisfiability. A formula aggregator combines the truth values of all axioms into one score 2012.13635 §3.1.4. Learning maximises it over the grounding parameters, optionally with a regulariser 2012.13635 §3.2.
• Query. After training, the paper defines four query types, for formulas and terms, on the training instances or on unseen ones 2012.13635 §3.3.
• Reasoning. A formula follows from a knowledge base at threshold q, with 1/2 < q < 1, if every grounding that satisfies the knowledge base at least to q also gives the formula at least q 2012.13635 §3.4. Two approximations are given. Querying after learning checks the formula on the best grounding found. Refutation searches for a grounding that satisfies the knowledge base but makes the formula fall below q, using a soft penalty on the knowledge base 2012.13635 §3.4.

When to use which

Implementation sketch

The common core is the operator set, the quantifier aggregation, and the satisfaction loss. Variants differ in the operators, the exponents and the masks. The sketch uses the stable product configuration.

# Shift a truth value away from its boundary  [arxiv:2012.13635 §2.4]
# The paper does not state the constant; the card records the released code's value
def shift(a, eps):
    # keeps the value strictly inside (0, 1) so gradients do not vanish
    return (1 - eps) * a + eps

# Connectives on truth values in [0, 1], stable product  [arxiv:2012.13635 §2.4]
def conj(a, b):
    # product of the shifted values
    return shift(a, eps) * shift(b, eps)

def disj(a, b):
    # probabilistic sum on values shifted away from 1
    x, y = (1 - eps) * a, (1 - eps) * b
    return x + y - x * y

def neg(a):
    # standard complement
    return 1 - a

def implies(a, b):
    # Reichenbach: 1 - a + a*b
    return neg(shift(a, eps)) + shift(a, eps) * (1 - eps) * b

# Quantifiers over the instances of a variable  [arxiv:2012.13635 §2.4]
def forall(truths, p):
    # p-mean of the errors, approximates min; p >= 1
    errors = 1 - truths
    return 1 - mean(errors ** p) ** (1 / p)

def exists(truths, p):
    # p-mean of the values, approximates max; p >= 1
    return mean(truths ** p) ** (1 / p)

# Train by maximising satisfaction of the knowledge base  [arxiv:2012.13635 §3.2]
for batch in data:
    # ground the data variables on this batch and evaluate each axiom
    truths = [truth(axiom, theta, batch) for axiom in kb]
    # formula aggregator over the axioms  [arxiv:2012.13635 §3.1.4]
    sat = aggregate(truths)
    # loss = 1 - Sat, minimised with Adam at learning rate 0.001  [arxiv:2012.13635 §4]
    theta = adam_step(theta, grad(1 - sat, theta))

Contradictions

None between papers, as only one card is compiled for this concept. The paper's text and its released code differ in several settings, such as the existential schedule in the digit-addition experiments and the exponents in the smokes-friends example. Those differences are listed on the card's Paper vs code section; check them before reimplementing. Logic Tensor Networks

Papers

• Logic Tensor Networks: the Real Logic language with stable product operators, p-mean quantifiers, diagonal quantification, satisfaction-based learning and refutation-based reasoning, with author-reported MNIST addition results.