Euclidean-equivariant network

Overview

A Euclidean-equivariant network maps 3D points or agents to 3D outputs. If the whole input is rotated, reflected or shifted, the outputs are transformed the same way, so the network does not have to learn each orientation separately 2303.10876 §1 2303.10876 §3.2. The second card, NeuSE, covers rotation and translation, plus uniform scale. It does not claim reflection 2303.07308 §3.1 2303.07308 eq. 1-2. The third card, NDF, gives rotation and translation for a field of point descriptors used in manipulation 2112.05124 §II-A 2112.05124 eq. 5.

The usual recipe keeps two kinds of state per element. A geometric feature holds coordinates that transform with the scene. A pattern feature is a plain vector built only from invariant quantities, such as speed, the distance between two elements, or the norm of a row 2303.10876 §4.1 2303.10876 §4.2 2303.07308 §3.5. Any step that decides which elements affect each other must also read only invariant inputs. Otherwise the routing changes under the transform and the output no longer follows it 2303.10876 app. A.

The designs reach the property in different ways. EqMotion builds the layers so that the proof covers them as written 2303.10876 §4.6. NeuSE uses a vector-neuron encoder for rotation and a learned centre for translation, and the paper measures residual error rather than claiming exact equivariance 2303.07308 §3.2 2303.07308 §3.3 2303.07308 §6. NDF centres the cloud on its mean for translation and uses Vector Neurons for rotation, so its descriptor field is SE(3)-equivariant by construction 2112.05124 eq. 5-7. Unlike the other two, NDF builds its descriptors from the activations of an occupancy network, not from a latent point cloud or a sequence of positions 2112.05124 §II-A.

Other equivariant architectures that EqMotion compares against (EGNN, TFN and SE(3)-Transformer) have no card here, so this page does not describe them.

Variants

Two-stream equivariant and invariant message passing (EqMotion)

The network keeps a geometric feature G with C coordinates per agent and a pattern feature h with D entries per agent 2303.10876 §4.1. It runs four stages: an initial embedding, an invariant interaction step, L layers of equivariant geometry and invariant pattern updates, and an equivariant readout 2303.10876 §4 2303.10876 eq. 1-5.

• Initial features. The geometric feature is a learned linear map over the time axis of the past positions, taken around their common mean. The pattern feature embeds the speed and the turning angle between consecutive velocity steps 2303.10876 §4.1 2303.10876 eq. 1. The paper argues that both quantities are invariant to rotation and translation 2303.10876 app. A.
• Interaction categories. For each ordered pair of agents, a softmax over K categories is computed from the pattern features and the column-wise distances between geometric features, with a temperature tau 2303.10876 §4.2 2303.10876 eq. 2. The paper uses only invariant inputs here, so the categories are invariant 2303.10876 app. A.
• Geometry update. Each layer does three things in order 2303.10876 §4.3 2303.10876 eq. 6-7:
  1. Rescale each coordinate around the mean, with an attention weight that depends on the pattern feature.
  2. Move each agent by a category-weighted sum of its differences to the other agents. The per-pair weights come from small MLPs of the pattern features and the distances.
  3. Apply an equivariant non-linearity. Build a query and a key from the centred geometry. Keep the query where its inner product with the key is non-negative, and otherwise remove the part of the query that is parallel to the key.
• Pattern update. Messages from pattern features and distances are summed over the other agents and merged with the agent's own pattern feature 2303.10876 §4.4 2303.10876 eq. 4. The geometry and pattern updates of one layer read the same previous-layer features.
• Readout. A linear map over the coordinate axis of the centred geometry gives the future positions, with the mean added back 2303.10876 §4.5 2303.10876 eq. 5.

The paper proves that the initial geometry, the readout and each geometric update are equivariant, and that the pattern features and the interaction categories are invariant. It also proves that the whole network is equivariant 2303.10876 §4.6 2303.10876 Theorem 1. The proof covers the layers as written in the paper.

Reported results (author-reported). Each number below is from the paper's own tables.

• Particle reasoning, Springs: 97.6 ± 1.1 accuracy against 93.0 ± 1.1 for NRI. Consistency over 20 random Euclidean transforms is 100.0 ± 0.0, against 93.7 ± 1.2 for NRI 2303.10876 tab. 1.
• Molecules (MD17), aspirin: ADE / FDE 5.95 / 8.38 against 10.41 / 14.67 for NMMP, on the paper's ×10⁻² scale 2303.10876 tab. 4.
• H3.6M skeleton, short-term: 55.0 mm at 400 ms against 58.3 for SPGSN 2303.10876 tab. 2.
• ETH-UCY pedestrians, single prediction: average ADE / FDE 0.49 / 1.03 m against 0.53 / 1.11 m for Traj++ 2303.10876 tab. 5.

Ablations (single runs, no variance). On H3.6M short-term, the full model scores an average MPJPE of 32.0. Removing inner-agent attention gives 32.4, removing the non-linearity gives 34.0 and removing inter-agent aggregation gives 34.5 2303.10876 tab. 7. Using L = 1, 2, 3 or 5 layers gives 34.0, 33.0, 32.4 and 32.1, against 32.0 for L = 4 2303.10876 tab. 8.

Card: EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoning.

Latent point-cloud code with SE(3) and scale equivariance (NeuSE)

Each partial object point cloud P is encoded into a latent point cloud z with k = 512 points of three coordinates 2303.07308 §3.1 [arxiv:2303.07308 app. A1]. A decoder maps a query point and the latent code to an occupancy value, so the full shape can be recovered from the code 2303.07308 §3.2. The training target asks for the code to move with the object: a rigid transform of the object moves z by the same transform, and a uniform scale scales z 2303.07308 eq. 1-2 2303.07308 §3.1.

• Rotation comes from an encoder built on vector neurons, a rotation- equivariant layer type 2303.07308 §3.2 2303.07308 app. A1.
• Translation is handled by centring. The encoder runs on the centred cloud, and the partial centre is added back. The paper says that the centre of the latent code is learned to match the true object centre, not built in 2303.07308 eq. 5 2303.07308 §3.3.
• Scale comes from normalising the encoding by the cloud's norm, so the following layers are scale-equivariant. The exact normalisation is not written out 2303.07308 §3.2.
• Relative pose. For two views of the same object whose latent codes share correspondences, a closed-form Horn fit gives the relative SE(3) transform. No render-and-optimise step is needed 2303.07308 eq. 6 [arxiv:2303.07308 §3.3]. Several visible objects are used together by concatenating their codes 2303.07308 §3.4 2303.07308 §4.2.
• Shape descriptor. The rotation-invariant part of the code gives one norm per row. A batch-hard triplet loss on cosine similarity makes descriptors of one object match across views and differ across objects [arxiv:2303.07308 §3.5] 2303.07308 eq. 10.
• Scale. The norm of the code is trained to match the diagonal of the partial cloud's bounding box 2303.07308 §3.6 2303.07308 eq. 11.
• Ambiguous objects. Symmetric objects have several valid poses, so the pose loss depends on the class. Unambiguous objects use a distance to the ground-truth transform. Ambiguous ones use a Chamfer distance between the aligned full clouds, plus extra samples that rotate about the symmetry axis 2303.07308 §3.4 2303.07308 eq. 7-9. This needs full object clouds, which exist only in simulation 2303.07308 §3.4 [arxiv:2303.07308 §3.7].
• Training. The total loss adds four terms with weight 1 [arxiv:2303.07308 eq. 12]. Training data are rendered in PyBullet from ShapeNet models, one model per category 2303.07308 §5.1 2303.07308 app. A1.

Use in object-based SLAM. The codes drive three steps:

• Data association runs in order: same semantic label, then similar scale, then shape similarity, then spatial proximity from a Horn transform of the concatenated codes 2303.07308 §4.2 2303.07308 fig. 5.
• Pose constraints link each keyframe to its earlier neighbours that share objects, in a sliding window of K = 10 keyframes, and link long-range loops when an object returns after a gap 2303.07308 §4.3 [arxiv:2303.07308 §5.1].
• Change detection runs only on objects that match in shape but sit apart. It compares relative layout to matched anchor objects, not absolute position 2303.07308 §4.4 2303.07308 eq. 13.

Equivariance measured (author-reported). These are residual errors on synthetic samples, not zero 2303.07308 §6.

• Rotation: normalised RMSE with mean 0.83% 2303.07308 §6.1.
• Translation: RMSE mostly below 0.01 m for offsets up to 0.4 m 2303.07308 §6.2.
• Scale: ratio error averages about 0.011 over scales from about 1 to 6.75 2303.07308 §6.3.

Localisation results (author-reported). On real sequences with a RealSense camera and Vicon ground truth:

• 4-Round, full trajectory, ORB-SLAM3 in non-stop mode: ATE RMSE 0.128 m alone, 0.119 m with the SE(3)-only variant, and 0.093 m with the full scale-equivariant codes 2303.07308 tab. 5.
• Hand-held indoor sequence, full trajectory: 0.192 m for ORB3 alone and 0.151 m with the full codes 2303.07308 tab. 6.
• Speed: codes take 6 ms per object to infer and 1 ms for the inter-frame constraint on the evaluation laptop 2303.07308 §7.

Card: NeuSE: Neural SE(3)-Equivariant Embedding for Long-Term Object-Based Simultaneous Localization and Mapping.

Point descriptor field with pose search (NDF)

Neural Descriptor Fields map each 3D point, conditioned on an object's point cloud, to a descriptor. The descriptor is the concatenation of the activations of every layer of an occupancy network 2112.05124 §II-A [arxiv:2112.05124 eq. 4]. The network is trained by reconstructing the object from a partial cloud, so no keypoint or correspondence labels are needed [arxiv:2112.05124 §II-A]. Translation equivariance comes from subtracting the cloud's mean from both the query point and the cloud. Vector Neurons give rotation equivariance of the encoder and occupancy network 2112.05124 eq. 6-7.

• Pose descriptor. A rigid body is a set of query points in a canonical pose. The pose descriptor concatenates the point descriptors of those points after the pose is applied 2112.05124 §II-B 2112.05124 eq. 10. The query points are sampled uniformly in the body's bounding box, and their placement changes which feature the pose is aligned to [arxiv:2112.05124 §II-B] 2112.05124 fig. 6.
• Pose search, not closed form. The pose of a new object is found by starting from random poses and minimising an L1 distance between the pose descriptor and the demonstration's, using Adam 2112.05124 §II-B 2112.05124 eq. 11. The search is non-convex, so results depend on the starts and steps. The paper does not give their number [arxiv:2112.05124 §II-B].
• Few-shot pick and place. Each demonstration gives a grasp pose and a relative place pose, and the place pose is the relative pose applied to the grasp. The pose descriptors are averaged over the demonstrations, and a new cloud is solved against those averages 2112.05124 §II-C. Inverse kinematics and motion planning then execute the result [arxiv:2112.05124 §II-C].

Results (author-reported). Simulated pick and place on unseen instances, with overall success per object 2112.05124 tab. I:

• Upright poses: 0.87 to 0.91 for NDF, against 0.11 to 0.45 for Dense Object Nets (DON).
• Arbitrary SE(3) poses: 0.58 to 0.78 for NDF, against 0.00 to 0.17 for DON.

Descriptor ablation on the upright mug: all layers give 0.88 overall, while the last layer alone gives 0.65 and the first layer alone 0.65 2112.05124 tab. II. The query-point scale is best at 1.0 of the 0.1 to 10.0 values tested 2112.05124 tab. III. The real-robot part has qualitative examples only 2112.05124 §IV-C.

Card: Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation.

When to use which

The rows compare the two designs against the constraints they address. Only claims the cited results support are listed.

Implementation sketch

The sketches follow each paper's equations. They are not the official code. Shapes are for one batch element. The first sketch is EqMotion's forward pass. The second is NeuSE's relative pose step, which is the part the paper uses across all its SLAM modules.

# EqMotion forward pass, one batch element  [arxiv:2303.10876 §4] [eq. 1-5]
# Input:  pos (M, T, 3) past positions, vel (M, T, 3) velocities
# Output: future (M, T_f, 3), transforms with the input

def predict(pos, vel):
    # Invariant pattern feature from speed and turning angle  [§4.1]
    speed = norm(vel)                        # (M, T)
    turn = angle(vel[t], vel[t - 1])         # (M, T), first step unstated
    h = embed(concat(speed, turn))           # (M, D)
    # Equivariant geometry: linear over time, centred  [§4.1] [eq. 1]
    mu = mean(pos)                           # (3,)
    g = linear_time(pos - mu) + mu           # (M, C, 3)
    # Categories from invariant inputs only  [§4.2] [eq. 2]
    cat = softmax(reason(h, dists(g)) / tau) # (M, M, K)
    for _ in range(L):
        # Both updates read the previous layer  [§4.3] [§4.4]
        h_next = pattern_update(g, h)
        g = geometry_update(g, h, cat)
        h = h_next
    # Readout: linear over the coordinate axis  [§4.5] [eq. 5]
    mu = mean(g)
    return linear_coords(g - mu) + mu        # (M, T_f, 3)


def geometry_update(g, h, cat):
    # Rescale each coordinate with an attention weight  [§4.3] [eq. 6]
    w = attention(h)                         # (M, C)
    mu = mean(g)
    g = w[..., None] * (g - mu) + mu
    # Move agents by category-weighted differences  [§4.3] [eq. 7]
    d = dists(g)                             # (M, M, C)
    e = sum_k cat[..., k] * mlp_k(h, d)      # (M, M, C)
    diff = g[:, None] - g[None, :]           # (M, M, C, 3)
    g = g + sum_j e[..., None] * diff        # (M, C, 3)
    # Equivariant non-linearity: query vs key  [§4.3]
    mu = mean(g)
    c = g - mu
    q, k = proj_q(c), proj_k(c)              # (M, C, 3) each
    keep = sum(q * k) >= 0                   # (M, C)
    k_hat = k / norm(k)
    q_perp = q - sum(q * k_hat) * k_hat
    return where(keep, q, q_perp) + mu

# NeuSE relative SE(3) transform between two views of one object
# [arxiv:2303.07308 §3.3] [eq. 6]
# Input:  z_a, z_b: latent codes (k, 3) with shared correspondences
# Output: rotation (3, 3), translation (3,)

def relative_pose(z_a, z_b):
    # Closed-form Horn solution; no rendering or iterative refinement  [§3.3]
    # Several objects: stack their codes first  [§3.4] [§4.2]
    rotation, translation = horn(z_a, z_b)
    return rotation, translation

Implementation notes:

• EqMotion: use the paper's readout shape, C to T_f, because the paper's stated shape of W_out does not fit its geometric feature 2303.10876 §4.5 2303.10876 eq. 5.
• EqMotion: the paper leaves tau unstated 2303.10876 §4.2. Pick a value and report it.
• EqMotion: the first turning angle is undefined in the paper 2303.10876 §4.1.
• EqMotion: the paper's non-linearity divides by the key's squared norm with no epsilon. The official code adds a small one 2303.10876 §4.3.
• EqMotion: the turning angle is an arccos. Its gradient is unbounded at ±1, so near-parallel consecutive velocities can make training unstable. Clamp the cosine and guard the norms 2303.10876 §4.1.
• NeuSE: the exact scale normalisation and the cross-product encoding are not written out, so an implementation has to choose them 2303.07308 §3.2.
• NeuSE: the batch-hard shape loss in the paper has an undefined index, so follow the text: the least similar same-object view and the most similar other-object view 2303.07308 §3.5 2303.07308 eq. 10.
• NDF: the paper gives no selection rule for the best start, and no learning rate for the search 2112.05124 §II-B. Pick a rule, such as the lowest final loss, and report it.
• NDF: the query-point scale is tested from 0.1 to 10.0 with no stated unit 2112.05124 tab. III. Fix the unit in your own code before comparing scales.

Contradictions

No cross-paper disagreement is recorded. The two cards solve different tasks, multi-agent forecasting and object-based SLAM, and use different metrics, so their numbers should not be compared.

Inside each card, a few headline numbers do not match the body:

• EqMotion: the abstract's reductions (24.0, 30.1, 8.6 and 9.2%) are not all traceable to a sentence in the body. The text gives 8.6% for H3.6M short-term 2303.10876 §5.3. A recalculation from the printed table gives about 8.8% against SPGSN, which does not match the paper's figure. The averaging rule is not stated 2303.10876 tab. 2 2303.10876 §1.
• NDF: the introduction says overall success is above 85%. That holds for upright objects only (0.87 to 0.91). With arbitrary poses the overall success is 0.58 to 0.78 2112.05124 §I 2112.05124 tab. I. Use the table.
• NeuSE: the text gives a 65% ATE improvement on one synthetic trajectory 2303.07308 §7.1. The table gives a full-trajectory reduction against the point-cloud baseline, which by our calculation from the printed values is about 58% for the planar layout and 66% for the non-planar one 2303.07308 tab. 3. Use the table values.

Papers

• EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoning: two-stream equivariant and invariant motion predictor with a learned interaction-category step.
• NeuSE: Neural SE(3)-Equivariant Embedding for Long-Term Object-Based Simultaneous Localization and Mapping: latent object code that is SE(3)- and scale-equivariant, used for data association, pose constraints and change detection in object SLAM.
• Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation: occupancy-network point descriptors made SE(3)-equivariant with Vector Neurons, and a pose search that matches demonstration descriptors without keypoint labels.