Hand-eye calibration

Overview

A camera mounted on a robot hand moves with the hand, but its pose relative to the hand is fixed and unknown. Hand-eye calibration estimates that fixed transform by moving the hand several times and recording how the camera and the hand each moved 2311.12655 §1 §2.1. Every variant splits the problem into a rotation part and a translation part 2311.12655 §4 §5. At least two independent motions are needed to fix the unknown uniquely 2311.12655 §3. The intrinsic camera parameters are assumed constant during the calibration 2311.12655 §8.

Variants

A method has two independent choices: the equation that relates the motions, and the solver for that equation.

Formulations

Classical: AX = XB

The sensor motion A, the hand motion B and the unknown offset X satisfy AX = XB 2311.12655 §2.1. For a camera, A comes from two calibrations, and each calibration is split into intrinsic and extrinsic parameters 2311.12655 eq. 5. The paper argues that this split is unstable 2311.12655 §2.2. Card: Hand-Eye Calibration.

New: MY = M'YB

This form works directly on the 3x4 projection matrices M and M' of the camera at two positions, relative to a calibration frame. The unknown Y is the hand-to-calibration-frame transform 2311.12655 §2.2. Because the intrinsic and extrinsic parts are never separated, it avoids the split that the paper calls unstable 2311.12655 §2.2 §8. The paper shows the two forms have the same mathematical structure, so a solver written for one applies to the other 2311.12655 §8. Card: Hand-Eye Calibration.

Solvers

Linear, Tsai-Lenz (TL)

The earlier solvers decouple rotation from translation and solve each with linear algebra 2311.12655 §3. The paper uses the Tsai-Lenz method as its linear baseline in all three real data sets 2311.12655 tab. 1 tab. 2 tab. 3. Card: Hand-Eye Calibration.

Closed-form rotation, then translation (CF)

The rotation is a unit quaternion. The rotation residual becomes a quadratic form in the quaternion, and its minimiser is the eigenvector of the smallest eigenvalue of a 4x4 matrix 2311.12655 §5.1 eq. 29. Translation then follows from a linear least-squares problem with the rotation fixed 2311.12655 §5.1. The rotation step is the closed form of Faugeras and Hebert, named here only 2311.12655 §8. Card: Hand-Eye Calibration.

Joint non-linear (NL)

Rotation and translation are estimated together, as seven parameters: four for the quaternion and three for the translation 2311.12655 §5.2 eq. 30. The cost sums rotation and translation residuals and adds a penalty that keeps the quaternion near unit length. The penalty weight is 2 × 10^6 in all the paper's experiments. The minimisation uses Levenberg-Marquardt 2311.12655 §5.2. The initial guess and the stopping rule are not stated 2311.12655 §5.2. Card: Hand-Eye Calibration.

When to use which

Implementation sketch

The rotation and translation equations come from pairs of motions. For the classical form, each motion gives a unit-vector pair (v, v'), a translation pair (p, p') and a matrix K. The paper states the general forms 2311.12655 eq. 25 eq. 26. The mapping of the classical and new formulations onto these symbols is our reading, not written out in the paper 2311.12655 eq. 16 eq. 18 eq. 22 eq. 23 eq. 24.

# Closed-form rotation, then translation  [arxiv:2311.12655 §5.1]
# Input: per motion, v, v', p, p' and K  [eq. 25] [eq. 26]
# Output: unit quaternion q and translation t
def closed_form(motions):
    # 1. Accumulate the 4x4 matrix A over all motions  [eq. 29]
    A = zeros(4, 4)
    for m in motions:
        D = Q(m.v_prime) - W(m.v)            # Q, W  [app. A]
        A = A + transpose(D) @ D
    # 2. Rotation: eigenvector of A for its smallest eigenvalue  [§5.1]
    #    q is defined up to sign, which does not change R
    q = smallest_eigenvector(A)
    R = rotation_matrix(q)                   # [app. A]
    # 3. Translation: stack (K - I) t = R p - p'  [eq. 26], solve linearly
    M = vstack([m.K - I for m in motions])
    b = concat([R @ m.p - m.p_prime for m in motions])
    t = least_squares(M, b)                  # [§5.1]
    return q, t

# Joint non-linear estimate  [arxiv:2311.12655 §5.2]  [eq. 30]
# Input: the same motion data, starting values q0 and t0 (not stated)
# Output: unit quaternion q and translation t
def joint(motions, q0, t0):
    # 1. Cost over the seven parameters  [eq. 30]
    def cost(q, t):
        total = 0
        for m in motions:
            # rotation residual, weight 1  [§5.2]
            total += norm2(m.v_prime - rotate(q, m.v))
            # translation residual, weight 1  [§5.2]
            total += norm2(rotate(q, m.p) - (m.K - I) @ t - m.p_prime)
        # penalty keeping q near unit length, lambda = 2e6  [§5.2]
        total += 2e6 * (1 - dot(q, q)) ** 2
        return total
    # 2. Levenberg-Marquardt  [§5.2]
    return levenberg_marquardt(cost, q0, t0)

Here rotate(q, x) is the quaternion product q x q-bar app. A. The paper notes that a trust-region method would also suit the joint problem 2311.12655 §5.2.

Contradictions

• Real-data rotation of the NL method. The paper's text describes the rotation cost of NL on real data as small 2311.12655 §7. In 2311.12655 tab. 3, classical NL has a rotation error of 0.258 on set 3, against 0.014 for classical TL. The paper does not discuss this gap.
• Overall ranking. The discussion says the non-linear method gives the most accurate results 2311.12655 §8. On real data, the linear method has the lowest classical rotation error on set 3 2311.12655 tab. 3, and the paper notes that linear and closed-form rotation can be better on real data 2311.12655 §7.
• Trade-off within one table. On set 2, classical NL has a higher rotation error than TL and CF (0.0017 against 0.0014), but a lower translation error (0.015 against 0.023 and 0.036) 2311.12655 tab. 2.

Papers

• Hand-Eye Calibration: a second formulation, MY = M'YB, built on projection matrices, and two solvers: closed-form rotation then translation, and joint non-linear optimization.