Hand-Eye Calibration

In plain words

A robot that carries a camera on its hand needs to know exactly where that camera sits relative to the hand. Finding this fixed offset is called hand-eye calibration. The usual approach moves the hand, measures how the camera and the hand each moved, and solves an equation of the form A X = X B.

This paper points out a second equation that works directly on the camera's projection matrices. The camera then does not have to be split first into its internal settings and its pose, a split the paper argues is unreliable in practice. It also gives two ways to solve for the offset: one finds the rotation first in closed form and the translation second, and the other solves both at once by numerical optimization. In the tests, the joint optimization is the most robust to measurement noise.

This matters when a robot needs to map what its camera sees into its own coordinate frame, for example to grasp an object. The choice of equation and solver changes how much measurement error ends up in the result.

TL;DR

Hand-eye calibration recovers the fixed rigid transform between a sensor on a robot hand and the hand, from several pairs of motions. The paper shows that the classical equation AX = XB has a second form, MY = M'YB, built from the 3x4 projection matrices, which avoids decomposing the camera into intrinsic and extrinsic parameters §2.2. It solves either form in two ways: a closed-form unit-quaternion rotation followed by a linear translation, or a joint non-linear optimization of rotation and translation §5. Author-reported: the joint optimization is the most accurate and noise-robust method in simulation and is the most accurate overall on real data §6 §7 §8.

Problem

The offset between a sensor and the hand is needed in two tasks: mapping sensor measurements into the robot workspace, and moving the sensor precisely §1. The classical formulation writes the transform from the sensor motion A, the hand motion B and the unknown offset X as A X = X B eq. 1 §2.1. For a camera, A comes from two calibrations, and each calibration is decomposed into intrinsic parameters (scales and image centre) and extrinsic parameters (pose) eq. 5. The paper argues this decomposition is unstable and adds error §2.2 §8.

Previous solvers decouple rotation from translation and use linear algebra for each; the paper argues that the linearization of the rotation equation fails under noise, which makes the linear solution unreliable §3. The problem assumes the camera's intrinsic parameters stay constant over the calibration §8, and that each camera position sees the calibration pattern §7. The paper states that at least two independent motions (three positions) are needed to determine X uniquely §3.

Method

Two equivalent formulations are used. The classical one is A X = X B. The new one is M Y = M' Y B, where M and M' are the 3x4 projection matrices of the camera at two positions relative to a calibration frame, and Y is the hand-to-calibration-frame transform §2.2 eq. 2. In the new formulation, each 3x4 matrix M_i is split into its left 3x3 block N_i and its last column n_i §4. For two positions, N = N_1^{-1} N_2 §4.

Both formulations split into a rotation equation and a translation equation §4 §5. Writing the rotation as R and the translation as t, the common form is

v' = R v eq. 25

(K - I) t = R p - p' eq. 26

For the classical formulation, the rotation equation is n_A = R_X n_B, where n_A and n_B are unit eigenvectors of R_A and R_B for eigenvalue 1, and the translation equation is (R_A - I) t_X = R_X t_B - t_A eq. 16 eq. 18. For the new formulation, the same structure holds with N in place of R_A and the translation t_N = N_1^{-1}(n_2 - n_1) eq. 22 eq. 23 eq. 24. Our reading, not written out in the paper: for each motion i, v = n_B, v' = n_A (or n_N), K = R_A (or N), p = t_B and p' = t_A (or t_N). The paper states eqs. 25 and 26 only in generic form.

Rotation then translation §5.1. Rotation is a unit quaternion q. The rotation residual is ||v' - q v q-bar||^2, which is a quadratic form q^T A q with A the sum of 4x4 matrices A_i = (Q(v') - W(v))^T (Q(v') - W(v)) eq. 29. Q and W are the left and right quaternion-product matrices app. A. The minimizer over unit q is the eigenvector of A for its smallest eigenvalue §5.1. Translation then follows from a linear least-squares problem on eq. 26 with R fixed §5.1.

Rotation and translation jointly §5.2. The paper minimizes a penalized cost over 7 parameters: 4 for q and 3 for t eq. 30 eq. 32. The cost is rotation residuals plus translation residuals plus the penalty λ(1 - q^T q)^2 that keeps q near unit length. The paper uses Levenberg-Marquardt and states that trust-region methods are also suitable §5.2.

# Rotation then translation  [§5.1]
# Input:  per motion i: v[i], v_prime[i] [eq. 25]; p[i], p_prime[i], K[i]
#         [eq. 26]
# Output: unit quaternion q (rotation), 3-vector t (translation)

def rotation_then_translation(v, v_prime, p, p_prime, K):
    # 1. Accumulate the 4x4 symmetric matrix A over all motions  [eq. 29]
    A = zeros(4, 4)
    for i in range(number_of_motions):
        D = Q(v_prime[i]) - W(v[i])              # Q, W  [app. A]
        A = A + matmul(transpose(D), D)
    # 2. Rotation: eigenvector of A for its smallest eigenvalue  [§5.1]
    q = eigenvector_of_smallest_eigenvalue(A)
    # 3. Rotation matrix of q, the rotation that maps v to v_prime  [app. A]
    R = rotation_matrix(q)
    # 4. Stack the translation equations (K[i] - I) t = R p[i] - p_prime[i]
    #    [eq. 26]
    rows = []
    rhs = []
    for i in range(number_of_motions):
        rows.append(K[i] - identity3)
        rhs.append(matvec(R, p[i]) - p_prime[i])
    # 5. Translation: linear least squares  [§5.1]
    t = least_squares(vstack(rows), concat(rhs))
    return q, t

# Rotation and translation jointly, non-linear  [§5.2]  [eq. 30] [eq. 32]
# Input:  the same per-motion data as above
# Output: unit quaternion q, 3-vector t
# Initial q0, t0: not stated

def joint_estimate(v, v_prime, p, p_prime, K, q0, t0):
    # 1. Cost over the 7 parameters  [eq. 30]
    def cost(q, t):
        rotation = 0
        translation = 0
        for i in range(number_of_motions):
            # rotation residual, weight lambda_1 = 1  [§5.2]
            rotation = rotation + norm2(v_prime[i] - q_rot(q, v[i]))
            # translation residual, weight lambda_2 = 1  [§5.2]
            translation = translation + norm2(
                q_rot(q, p[i]) - matvec(K[i] - identity3, t) - p_prime[i]
            )
        # penalty keeping q near unit length, lambda = 2e6  [§5.2]
        penalty = 2e6 * (1 - dot(q, q)) ** 2
        return rotation + translation + penalty
    # 2. Minimize with Levenberg-Marquardt  [§5.2]
    q, t = levenberg_marquardt(cost, q0, t0)
    return q, t

Here q_rot(q, x) is the rotation q x q-bar, i.e. the quaternion product app. A. The quadratic expansion of the translation term, which turns it into eq. 31 and gives the 5-term form of eq. 32, is derived in app. B.

Hyperparameters

Results

Author-reported. Setup: three real data sets from two labs, with 17, 7 and 6 robot positions §7. Rotation error is the sum of squared norms of R_A R_X - R_X R_B (or N R_Y - R_Y R_B), and translation error is a ratio of squared residuals, as defined in the table captions tab. 1 tab. 2. "TL" is Tsai-Lenz, the baseline; "CF" is the closed-form rotation then translation; "NL" is the joint non-linear method.

Simulation results §6 fig. 3 to fig. 12. With two motions and noise from 1% to 6%, the joint NL method is the most accurate in all reported settings. TL and CF behave almost identically, which the paper attributes to both decoupling rotation from translation §6. For 4 motions with Gaussian noise, the translation error is about 4% for NL and 6.5% for the other two methods §6. Figures 11 and 12 plot the errors against the square root of the number of motions at fixed noise.

Headline, author-reported: the new formulation with NL gives the lowest errors on data set 2, with a translation error of 0.0006 against 0.036 for classical TL tab. 2.

Paper vs code

No official code.

Pitfalls

• Source date. The LaTeX source is a 1995 journal template (\volumeyear{1995}). The arXiv record is version v2, updated 2023-11-22 (arXiv metadata, not the paper); the card follows the arXiv record.
• Equation numbering. Equation numbers here follow the order of numbered equations in the LaTeX source. The paper's own cross-reference "eq. (31)" in its appendix confirms this count up to eq. 31 app. B. Equation numbers after that are our count.
• Quadratic form needs unit q. The step that rewrites the residual as q^T A q, and the expansion of the translation term app. B, multiply by ||q||^2 and then drop the q-dependence of that factor. The expansions are exact only for unit q. The paper keeps q near unit length only through the penalty λ(1 - q^T q)^2 §5.2 eq. 30 eq. 31.
• Units in the joint cost. λ_1 = λ_2 = 1 weights rotation residuals (unitless) against translation residuals. The units of the points p_i and the translations are not stated, so the effective relative weight depends on units. Translations are reported in mm §6 §7.
• Symbol clash. λ is the Lagrange multiplier in the closed-form derivation §5.1 and the penalty weight in the joint cost §5.2.
• Initialization and convergence. The initial guess for the joint solver, the convergence test and the iteration cap are not stated. The cost is non-linear, so results may depend on the start. The paper does not say whether the closed-form solution is used as a start.
• Sign of q. q and -q give the same rotation. The closed-form eigenvector is defined only up to sign. This does not change R.
• Translation solve is not written out. Section 5.1 says translation is "a linear least-squares problem" §5.1. The stacked system from eq. 26 is our reading.
• Per-motion N_i. The paper defines N = N_1^{-1} N_2 for two positions §4. For n motions, the summary lists M_i Y = M_1 Y B_{1i} eq. 14. Taking the per-motion N as N_1^{-1} N_i is our reading.
• Index conventions. Section 2 uses B = B_2^{-1} B_1 for two positions eq. 4. Section 6 writes the general motions as B_{i i+1} = X^{-1} A_{i+1} A_i^{-1} X and B_{1i} = Y^{-1} A_1^{-1} A_i Y. Use the two-position equations when in doubt.
• Degenerate motions. The paper notes that two motions are needed and that the linear rotation equation is rank-deficient for one motion §3. It does not discuss near-degenerate motion sets, such as small rotation angles or shared rotation axes.
• Noise definition. The noise "level" is a ratio, and the paper says it is represented by C or by 2σ §6. Figures are labeled by percent noise fig. 3 to fig. 10. The paper does not say which of C or 2σ the percentages refer to in each case beyond this definition.
• Simulation nominal vs real. The simulation normalizes translation error by 157 mm §6. The real data have translation norms of 93 mm and 681 mm §7. The paper does not explain the 157 mm value.
• Table 3 vs text. The text says the NL method gives "sometimes slightly less accurate rotation" §7. In tab. 3, classical NL has rotation error 0.258, against 0.014 for classical TL. The paper does not comment on this gap.
• New formulation is not uniformly better. In tab. 3, new TL has rotation error 0.038, against 0.014 for classical TL. Set 1 has extrinsics only, so only the classical formulation is tested there §7.
• Real data robustness. The paper says that real-data rotation can be better with linear methods, because the robot's translation parameters have uncalibrated errors that simulations do not model §7.
• Constant intrinsics. The new formulation still assumes the camera's intrinsic parameters are constant during calibration. The paper says this assumption is "not very realistic" §8.
• Typo in appendix. Appendix B writes W(Q) where W(q) is meant app. B.
• Task scope. The paper covers camera-to-hand calibration only. It names robot-plus-hand-eye calibration as future work §8.

Lineage

• Builds on: the classical AX = XB formulation and its linear solvers, Shiu and Ahmad 1989, Tsai and Lenz 1989 §3, Chou and Kamel 1991, Chen 1991 and Wang 1992 §3. Named only; no content used from them.
• Closed-form rotation: Faugeras and Hebert 1986 and Horn 1987 §5.1 app. A. Named only.
• Compared against: Tsai-Lenz (the linear baseline), the closed-form rotation-then-translation method and the joint non-linear method, all in this paper §6 §7.
• Superseded by: not known from content already in this wiki.