Simultaneous Robot-World and Hand-Eye Calibration

In plain words

A robot that looks at the world with a camera has to know two fixed offsets. One is where the camera sits on the robot's hand. The other is where the robot's base sits relative to the room, or to the pattern the camera looks at. Both are unknown until someone measures them, and getting them wrong makes every later grasp or move slightly off.

The usual approach solves one offset, then the other, and lets errors from the first step leak into the second. This paper treats both as unknowns in the same equation, so they are estimated together. It offers two solvers. The first is a formula-style solution that works on two rotations at once. The second starts from one of those answers and adjusts all the numbers in one iterative fit. Under the noise the authors add to their simulated data, the iterative fit is the most stable of the three methods they compare.

TL;DR

• Problem: find the hand-eye transform X and the robot-to-world transform Z from several motions, given A X = Z B §2 eq. 1.
• Closed-form: a rotation solver that minimises a positive quadratic form in two unit quaternions, followed by a linear least-squares fit for translations §3.1.
• Non-linear: a joint Levenberg-Marquardt fit of 24 parameters with penalty terms that keep the rotation matrices orthogonal §3.2.
• Author-reported: non-linear gives the most accurate results in simulation; closed-form is slightly better than the linear method of Zhuang et al. in simulation, but the real data do not clearly show it §4 §5 §6.

Problem

Given n pairs of hand and camera (or tool and robot) poses, the unknown rigid transforms X and Z satisfy A X = Z B for every motion i §2 eq. 1. In the camera case A is the camera pose relative to a calibration pattern and B is the hand pose from forward kinematics, which the paper assumes known §2. Figure 1 shows the camera-on-gripper case and Figure 2 the tool-on-gripper case, where the world frame equals the camera frame fig. 1 fig. 2.

The earlier linear method of Zhuang et al. cannot solve some special configurations, namely when a scalar component of one quaternion is zero §2.1 §3.1.1. A unique solution needs at least two motions with distinct rotation axes, which the linear and closed-form methods both require §1.

Method

The paper splits A X = Z B into a rotation equation and a translation equation §2 eq. 2 eq. 3:

R_A R_X = R_Z R_B eq. 2

R_A\, t_X + t_A = R_Z\, t_B + t_Z eq. 3

For known rotations, eq. 3 is linear in t_X and t_Z, and each motion gives three such equations §2 §2.1.

Baseline (linear, Zhuang et al.). Write the rotations as unit quaternions and turn R_A R_X = R_Z R_B into a quaternion product §2.1 eq. 4. Its scalar and vector parts give two constraints eq. 5 eq. 6. If a_0 ≠ 0, the scalar part of X solves for x_0 eq. 7. Substituting that into the vector part gives a 3x6 linear system in the six unknowns (the vector parts of q_X and q_Z) eq. 8. The remaining components of q_X and q_Z follow from the unit-norm constraints and eq. 7 §2.1. This step needs a_0 ≠ 0 and z_0 ≠ 0 §2.1.

Closed-form method (this paper). The paper builds one error function over all motions instead §3.1:

1. For each motion i, write the quaternion products as matrices, Q(q_Ai) q_X and W(q_Bi) q_Z, with Q and W defined in §3.1 (printed in the source as 4x4 matrices). Their squared error is a quadratic form in the stacked 8-vector v = [q_X, q_Z] with an 8x8 matrix S_i §3.1 eq. 9.
2. Sum over motions. The total error is f = v^T S v, where S has identity blocks scaled by n and the off-diagonal block C is the sum of the per-motion blocks C_i §3.1 eq. 10.
3. Minimise f under the unit-norm constraints of q_X and q_Z with two Lagrange multipliers λ_1 and λ_2 §3.1 eq. 11. Setting the derivatives to zero gives two linear relations eq. 12 eq. 13.
4. Solving them gives q_X = C q_Z / (λ_1 - n) eq. 14 and (C^T C) q_Z = (λ_1 - n)(λ_2 - n) q_Z eq. 15. So q_Z is an eigenvector of C^T C. Unit norm of q_X forces λ_1 = λ_2 = λ, and (λ - n)^2 = α_i for the eigenvalue α_i. Hence λ = n ± sqrt(α_i) §3.1.
5. Pick the eigenvalue whose n ± sqrt(α_i) is the smallest positive value §3.1. At the stationary point f = λ_1 + λ_2 §3.1.
6. Translations come from a linear least-squares fit once R_X and R_Z are known §3.1.

# Closed-form rotation solver  [§3.1] [eq. 9] [eq. 14] [eq. 15]
# Input:  unit quaternions q_A[i], q_B[i] for i = 1..n (n >= 2)
# Output: unit quaternions q_X, q_Z

def solve_rotations(q_A, q_B):
    # Sum of the 4x4 blocks C_i = -Q(q_Ai)^T W(q_Bi)  [§3.1] [eq. 9]
    C = zeros(4, 4)
    for i in range(n):
        C = C - Q(q_A[i]).T @ W(q_B[i])
    # Eigen-decomposition of the symmetric matrix C^T C  [§3.1] [eq. 15]
    alphas, vectors = eig_symmetric(C.T @ C)
    # Candidates lambda = n +/- sqrt(alpha_i);
    # keep the smallest positive one  [§3.1]
    best_i, best_lam = None, None
    for i in range(4):
        for lam in (n + sqrt(alphas[i]), n - sqrt(alphas[i])):
            if lam > 0 and (best_lam is None or lam < best_lam):
                best_i, best_lam = i, lam
    # q_Z is the matching eigenvector  [§3.1]
    q_Z = vectors[:, best_i]
    # q_X follows from the stationarity condition  [§3.1] [eq. 14]
    q_X = C @ q_Z / (best_lam - n)
    return q_X, q_Z

Special configuration (a_0 = 0). When some q_Ai has zero scalar part, q_Ai and -q_Ai describe the same rotation. The paper then builds two matrices, S_i^+ with +C_i and S_i^- with -C_i in the off-diagonal blocks, and computes two solutions. It keeps the one with the smaller minimum §3.1.1.

Non-linear method. The rotations and translations are estimated together. There are 24 parameters: two 3x3 rotation matrices (18) and two translation vectors (6) §3.2. The error is a weighted sum of rotation residuals, position residuals and two orthogonality penalties §3.2:

f = \mu_1 \sum_{i=1}^{n} \| R_{Ai} R_X - R_{Bi} R_Z \|^2 + \mu_2 \sum_{i=1}^{n} \| R_{Ai} t_X + t_{Ai} - R_Z t_{Bi} - t_Z \|^2 §3.2, unnumbered

\quad + \mu_3 \| R_X R_X^T - I \|^2 + \mu_4 \| R_Z R_Z^T - I \|^2 §3.2, unnumbered

The paper writes this as a sum of squared terms Φ_j with f = ½ Σ Φ_j² and minimises it with Levenberg-Marquardt §3.2. The initial guess comes from either the linear or the closed-form method §3.2 §6.

# Joint non-linear refinement  [§3.2]
# Input:  pairs (R_A[i], t_A[i], R_B[i], t_B[i]), initial estimate
# Output: refined R_X, R_Z, t_X, t_Z

def refine(init, R_A, t_A, R_B, t_B):
    # 24 unknowns: two rotation matrices and two translation vectors  [§3.2]
    params = pack(init)
    # Rotation and position residuals, weighted by mu_1, mu_2  [§3.2]
    # Orthogonality penalties for R_X and R_Z, weighted by mu_3, mu_4  [§3.2]
    # Levenberg-Marquardt minimisation  [§3.2]
    params = levenberg_marquardt(residuals, params)
    return unpack(params)

Hyperparameters

Results

Author-reported. Setup: real data only, with two robots, 17 and 7 hand-eye positions. The simulation results are shown only as plots, and the paper gives no numbers from them in the text §4 fig. 3 fig. 4 fig. 5.

Errors in the table are the paper's E_R (sum of squared rotation residuals over positions) and E_t (relative translation error) §5 eq. 16 eq. 17.

Simulation headline (qualitative, from the text): the non-linear method yields the most accurate results §6, and the abstract calls it the most stable under noise. The closed-form method is more accurate than the linear method in simulation §4 §6. The text gives no numbers for the Fig. 5 trend over position count, so none are reported here §4.

Paper vs code

No official code.

Search performed on 2026-11-03: the paper has no code link in its LaTeX source and no project page. A GitHub repository search for "hand-eye calibration robot-world" and "AXYB calibration" returned repositories by other owners (none under an author's name); a GitHub user search for "dornaika" returned no account that belongs to the author. No code was cloned.

Pitfalls

• Notation slip in the introduction. The first paragraph of §1 writes the hand-eye equation as A X = X B §1. From §2 on, the paper uses A X = Z B eq. 1. Use the A X = Z B form, where X is hand-to-camera (or hand-to-tool) and Z is robot-base-to-world.
• Quaternion convention. The paper says it uses the notation of Walker, Shao and Volz for quaternion products §2.1 §3.1. The Q and W matrices in §3.1 are printed in the source with that convention. Check them numerically against your own quaternion product before use. A different convention changes Q and W.
• Full-rank assumption. The paper assumes C has full rank "without loss of generality" §3.1. If it is rank-deficient, one α can be zero, which gives λ = n in eq. 14 and a division by zero. The paper does not handle this case.
• Eigenvector sign. Flipping the sign of q_Z flips q_X as well, because q_X = C q_Z / (λ - n) eq. 14, and the pair describes the same rotations. The paper does not state a sign convention, which is harmless here.
• Choice of λ. The text says to minimise λ_1 + λ_2 and also to pick the smallest positive n ± sqrt(α_i) §3.1. Both give the same choice, since λ_1 + λ_2 = 2λ at the stationary point. Only candidates with λ > 0 count.
• Special configuration. When some a_0 = 0, the paper requires two candidate solutions, one per sign of C_i §3.1.1. Measured data almost never give an exact zero scalar part. The paper does not say whether the two-candidate treatment is needed for near-zero values.
• Translations are not written as a system. §3.1 and §2.1 say each motion gives three linear equations in six unknowns, but do not write out the stacked matrix. The implementer must stack eq. 3 for all motions. The pseudocode above reads eq. 3 as R_A t_X - t_Z = R_Z t_B - t_A.
• Non-linear weights and scaling. The penalty weights are 10^6, against 1 for the data terms §3.2. This is a large scale mismatch and may affect conditioning. The paper does not say how the residual vector is stacked, so the scaling of each Φ_j with the square roots of μ is implied, not stated.
• No projection after optimisation. The penalties only softly enforce orthogonality. The paper does not say how to project R_X and R_Z back to rotation matrices after the fit §3.2.
• Stopping rules. The Levenberg-Marquardt damping, tolerance and maximum iterations are not stated §3.2.
• Minimum number of motions for the non-linear fit. §3.2 counts 2n non-linear constraints for n positions and 24 unknowns, but does not state how many positions are needed for a unique solution. The simulations use n = 3 to 8 §4.
• Noise model. The noise ratio is the noise amplitude divided by the nominal value. For uniform noise the amplitude is C in [-C/2, C/2]. For Gaussian noise the ratio is 2σ, so σ is half the ratio times the nominal value §4. The paper does not say whether noisy quaternions are renormalised before solving.
• Translation noise scale. The nominal translation magnitude is the mean of |t_Ai| and |t_Bi| over all motions §4. Noise is added to each component of each vector, and the paper does not say how this interacts with normalisation.
• Figure captions. The caption of Fig. 3 lists only the linear and non-linear curves, while §4 defines dotted curves as the closed-form method. Whether the closed-form method appears in Figs. 3 and 5 is unclear fig. 3 fig. 5.
• Error metric. E_R is a sum over positions with no normalisation, and the norm is not specified eq. 16. The two data sets use different robots and position counts, so the values are not comparable across them tab. 1 tab. 2.
• Real data disagree with simulation. The real data do not show the closed-form method beating the linear method, and the non-linear method has slightly larger rotation error §5 §6. The authors attribute this to unmodelled robot kinematic errors §5. For data set 2, the linear method has a large rotation error (0.12174) tab. 2.
• Date and venue. The LaTeX running head says IEEE Transactions on Robotics and Automation, vol. 14, no. 4, 1998 §1. The arXiv version is dated 2023-11-20. This card follows the arXiv version.
• Real data availability. The real data sets, camera model, robot model and calibration pattern are not described beyond the robot types and number of positions §5. They are not stated.

Lineage

• Builds on: Zhuang, Roth and Sudhakar's linear solution of A X = Z B with unit quaternions (IEEE TRA 1994), which this paper restates and compares against §1 §2.1. Also the earlier closed-form and non-linear hand-eye literature cited in §1 (Tsai and Lenz 1989; Wang 1992; Park and Martin 1994; Shiu and Ahmad 1989; Horaud and Dornaika 1995).
• Compared against: the linear method of Zhuang et al. §4 §5.
• Uses the quaternion-product notation of Walker, Shao and Volz §2.1 §3.1.
• Superseded by: none in this wiki.