Robot Hand-Eye Calibration using Structure-from-Motion

In plain words

A camera fixed to a robot arm sees the world from a position that the robot knows only through its joints. To use the camera for grasping or guiding the arm, you need the fixed offset between the camera and the robot's hand. This offset is called the hand-eye transform. The usual way to find it is to show the camera a printed target with known geometry, which is awkward on a rover or in space.

This paper drops the target. The camera watches an unknown scene while the arm moves, and a structure-from-motion step turns the image sequence into camera motions. Those motions lose their overall size, so one unknown scale factor is carried along. The authors show that, once this is set up correctly, the offset can be found by solving linear equations rather than iterating a non-linear fit. Because the algebra is linear, they can also see which kinds of arm motion are enough and which are not: pure slides, pure turns, motion in a plane, and general motion each reveal different parts of the offset.

The method is aimed at robots that cannot carry a calibration object and need to recalibrate in place.

TL;DR

• Hand-eye calibration (camera-to-hand transform) without a calibration rig.
• Camera motions come from structure-from-motion, so they are known only up to one global scale factor λ, which is solved together with the transform.
• Rotation is solved linearly as a null-space problem over 3×3 matrices, then projected back onto a rotation; translation and λ follow by least squares.
• The paper's own experiments report rotation accuracy close to rig-based linear methods and translation accuracy lower than them. All numbers are author-reported.

Problem

Classical hand-eye calibration needs a calibration rig, because the camera's pose relative to the rig is computed from known 3D target points §1 §2.3. The authors want to avoid the rig for on-site and remote robots §1. Using structure-from-motion instead removes the need for 3D target knowledge, but the camera motions then carry an unknown scale §2.3 fig. 4. The earlier calibration formulations need the rotation estimated with an orthogonality constraint, and the paper argues that minimal rotation parameterisations are ill-conditioned for small rotations §1. The assumptions are: two motions with non-parallel rotation axes are needed for a unique solution in the general case §2.2, and the scene points must be tracked across the motion §6.

Method

Set-up. The robot motion between two arm poses is B_i = (R_b_i, t_b_i). The camera motion is A_i = (R_a_i, t_a_i). The unknown hand-eye transform is X = (R_x, t_x), and the classic relation is A_i X = X B_i eq. 2 §2.2. With structure-from-motion the camera translation is known only up to scale, so the camera motion is written with unit-direction vector u_a_i and unknown scale λ, with t_a_i = λ u_a_i eq. 5 eq. 6 §2.3. The relation becomes A_i(λ) X = X B_i eq. 7 §3.1.

Splitting the relation. Writing the rotation and translation parts separately gives R_a_i R_x = R_x R_b_i eq. 8 and R_a_i t_x + λ u_a_i = R_x t_b_i + t_x eq. 9 §3.1.

Linear form. The rotation equation is a Sylvester-type equation. Using the Kronecker product and the vec operator eq. 1 §2.1, the rotation constraint becomes (R_a_i ⊗ R_b_i) vec(R_x) = vec(R_x) eq. 12. The translation equation becomes (I3 ⊗ t_b_i^T) vec(R_x) + (I3 − R_a_i) t_x − λ u_a_i = 0 eq. 13 §3.2. Stacking all motions gives one homogeneous system in the 13 unknowns (vec(R_x), t_x, λ) eq. 14 §3.2.

# Linear hand-eye self-calibration  [eq. 14] [§4.2] [app. A]
# Input:  per motion i: camera (R_a, u_a) and robot (R_b, t_b); n >= 2
# Output: R_x (3, 3), t_x (3,), lam (scalar)

def solve_hand_eye(R_a, u_a, R_b, t_b):
    # 1. Stack the rotation constraints  [eq. 12] [eq. 14]
    # (I9 - kron(R_a_i, R_b_i)) vec(R_x) = 0 for every motion i
    M_rot = vstack([eye(9) - kron(R_a[i], R_b[i]) for i])  # (9n, 9)

    # 2. Take the null-space vector and reshape it  [§4.2] [app. A]
    v = null_space_vector(M_rot)     # e.g. SVD, smallest singular value
    V = unvec(v)                     # (3, 3)

    # 3. Rescale V to a rotation with det = +1  [§4.2, Lemma 1]
    R_x = sign(det(V)) / abs(det(V)) ** (1 / 3) * V

    # 4. Stack the translation constraints  [eq. 14] [§4.4]
    # (I3 - R_a_i) t_x - lam * u_a_i = -R_x t_b_i
    A = vstack([hstack([eye(3) - R_a[i], -u_a[i]]) for i])  # (3n, 4)
    rhs = vstack([-R_x @ t_b[i] for i])                     # (3n,)

    # 5. Least squares for t_x and lam  [§4.4]
    t_x_lam = lstsq(A, rhs)
    return R_x, t_x_lam[:3], t_x_lam[3]

Lemma (rotation from the null space). If two motions have non-parallel rotation axes, the 9-unknown rotation system has a one-dimensional null space spanned by a matrix V proportional to R_x, and the scale of that matrix is fixed by det(R_x) = 1 §4.2, Lemma 1 app. A. The formula used in step 3 is the one stated there. The proof is in Appendix A and uses preliminary results on commuting rotations app. A.

Pure translations (closed form). When every robot motion is a pure translation, R_b_i = I and camera motions are pure translations too §4.1. Then t_a_i = R_x t_b_i eq. 15. Three linearly independent translations give a full-rank 9×9 system eq. 18. Its inverse has the cross-product form [eq. 20] §4.1, and this gives R_x up to the scale λ, with R̃_x = (λ/Δ) Σ_k u_a_k (t_b_(k+1) × t_b_(k+2))^T eq. 23 §4.1, where Δ = det(t_b_1, t_b_2, t_b_3) and indices wrap. The paper then uses det = 1 to fix λ and says the result is a rotation for exact data §4.1 eq. 24.

# Pure translations: rotation and scale in closed form  [§4.1] [eq. 23]
# Input:  three linearly independent robot translations t_b (3, 3),
#         unit camera directions u_a (3, 3)
# Output: R_x (3, 3), lam (scalar)

def rotation_from_three_translations(t_b, u_a):
    # 1. Cross products of successive robot translations  [§4.1]
    c_1 = cross(t_b[1], t_b[2])
    c_2 = cross(t_b[2], t_b[0])
    c_3 = cross(t_b[0], t_b[1])

    # 2. Determinant of the three translations  [§4.1]
    delta = dot(t_b[0], cross(t_b[1], t_b[2]))

    # 3. Shape of R_x with lam = 1  [eq. 23]
    shape = (outer(u_a[0], c_1) + outer(u_a[1], c_2)
             + outer(u_a[2], c_3)) / delta

    # 4. Fix lam from det(R_x) = 1 (inferred from the unity constraint)
    lam = sign(det(shape)) / abs(det(shape)) ** (1 / 3)
    R_x = lam * shape
    return R_x, lam

Pure rotations. When t_b_i = 0, the system decouples eq. 26 eq. 27 §4.2. The rotation part is the null-space problem above, which needs two rotations with non-parallel axes §4.2, Lemma 1. The translation part (I3 − R_a_i) t_x = λ u_a_i has a one-dimensional solution family: t_x is known only up to λ §4.2 The paper states this as its third proposition (hand-eye translation up to the scale t_x/λ, rotation and translation decoupled) §4.2.

Two pure translations. Two linearly independent translations are enough, but the solution is no longer linear: λ is found from norm equalities ‖R_x t_b_i‖ = ‖t_b_i‖, which give λ ‖u_a_i‖ = ‖t_b_i‖, then R_x comes from a full-rank system that uses the cross product of the two translations §4.1.

Planar motion. If the camera and robot rotate about the same axis (normal to the plane of motion) and one pure translation is added that is not parallel to that axis, R_x and λ are recovered. The translation component along the plane normal is not determined and is written t_x(α) with free α §4.3, Lemma 2 fig. 7 fig. 8. Two planar motions suffice under the three conditions of the planar proposition, including a non-zero combined translation (I3 − R_b_2) t_b_1 − (I3 − R_b_1) t_b_2 ≠ 0 §4.3.

General motion (two-step solution). Step 1: the rotation is the null-space estimate of the stacked rotation constraints, followed by an orthogonalisation to counteract noise §4.4. Step 2: the translation and λ come from the stacked translation equations, solved by linear least squares once R_x is fixed §4.4. The paper says the two-step solution keeps R_x orthogonal, whereas solving the full 13-unknown null space does not §4.4 eq. 14.

Structure-from-motion step. The camera motions are computed with an iterative Euclidean factorisation method (Christy and Horaud), which gives rotations exactly and translations up to a global scale §2.3. In the real experiments the same tracked points are used for the self-calibration method, without a 3D model §5.3.1.

Simulation model. Hand-eye transform and robot motions are random: Roll-Pitch-Yaw angles and translation coefficients are drawn from Gaussian laws §5.2.1. Noise on camera translations is t̃ = t + ν ‖t‖ n, with n a zero-mean unit-variance Gaussian 3-vector. Noise on Roll-Pitch-Yaw angles is α̃ = (1 + ν r) α, with r a scalar white noise §5.2.2. So ν is a relative noise level §5.2.2.

Hyperparameters

Results

Author-reported. Setup: real images of a calibration grid (Experiment 1, 33 positions) and four positions with tracked points (Experiment 2), on the authors' 5-DOF robot §5.3 fig. 1. Methods compared: axis/angle method of Tsai and Lenz (M1), dual-quaternion method of Daniilidis (M2), non-linear minimisation of Horaud et al. (M3), this linear formulation with pose-based camera motions (M4), and this self-calibration with structure-from-motion (M5) §5.

Simulations. Results are described only in words; the plots are not tabulated in the text. For rotation, M1 and M4 are the most accurate under noise. For translation, M1 and M4 are very accurate at low noise but less accurate than M2 and M3 at higher noise §5.2.3 fig. 9. With small motions (2 cm, 10°), the authors report that M4 is less sensitive than the others §5.2.3 fig. 10. With small motions and ν = 0.01, the authors report higher accuracy for M4 across the motion-count range §5.2.4 fig. 11.

Real data, Experiment 1. RMS error between A_i X and X B_i, and the comparison to the reference estimate:

The text says M4 has the smallest rotation error and that M5 is larger but still acceptable for visual servoing §5.3.1 fig. 13.

Real data, Experiment 2. Four images, tracked points §5.3.2 fig. 14:

The authors attribute M3's failure here to the small number of data points §5.3.2 fig. 15.

Three of the paper's headline claims: the linear method gives the smallest rotation error in Experiment 1 §5.3.1; self-calibration has the largest translation error among the five in Experiment 1 (0.322) tab. 2; and the self-calibration translation error in Experiment 2 is 0.20 tab. 3.

Paper vs code

No official code.

Searched: the paper and its arXiv record contain no code or project link §1, and the arXiv metadata has no comments. GitHub repository searches for "hand-eye structure-from-motion", "hand-eye calibration self" and "hand-eye calibration linear kronecker" returned no results.

Pitfalls

• Scale normalisation is garbled in the text. The paper fixes ‖u_a_1‖ = 1 and writes λ = ‖t_a_1‖ §2.3, but the displayed formula for u_a_i reads as t_a_i divided by ‖u_a_1‖, which is t_a_i itself under that constraint. Use u_a_i = t_a_i / λ with λ = ‖t_a_1‖, which matches eq. 6.
• Null-space vector is not exactly a rotation under noise. The lemma assumes noise-free data. With noise, V is not orthogonal, and the paper says only that the orthogonalisation "has not been quantified" and that results before it were "close to orthogonal" §4.2. The orthogonalisation method (projection to the nearest rotation, or another) is not specified.
• Pure translations need three linearly independent translation vectors. The 9×9 system is singular otherwise, because Δ = 0 §4.1. Two independent translations give a non-linear solution, not the closed form §4.1.
• Pure rotations never fix the physical scale of the hand-eye translation. t_x is available only up to λ, which in the self-calibration setting is itself unknown §4.2. With pose-based camera motion (λ known), the translation is recoverable, as the paper notes, citing Li §4.2.
• Planar motion leaves a height ambiguity. The component of t_x along the normal to the plane is undetermined. The paper gives no way to fix it from the data alone §4.3 fig. 8.
• Full null-space solution for the general case is not used. The authors say it does not give an orthogonal R_x and that correcting it would be non-linear. They therefore use the two-step solution §4.4. An implementer who solves the full 13-unknown null space must add orthogonalisation and should expect it to disturb t_x.
• Simulation noise scales with magnitude. Translation noise is relative to ‖t‖, and rotation noise multiplies each Roll-Pitch-Yaw angle, so large angles get large absolute errors §5.2.2. Series 1 uses motions up to 1 m and 180°; the results cannot be transferred to other scales without re-running.
• Real-data reference is not reproducible from this paper. The "mean" rigid transform of 100 RANSAC-style estimates is referred to a 1999 thesis §5.3. The card does not know its exact method.
• "Close to 3 %" claim. The text says M5's relative translation error in Experiment 2 is close to 3 % §5.3.2. The reference is not defined further, and the table lists 0.20 as the translation error for M5 in that experiment tab. 3. Use the table value and the definition in §5.1.
• Quaternion-norm rotation metric. It is strictly increasing in the residual angle from 0 to 2π, but it is not a true angle metric and should not be compared to other papers' degree-based errors §5.1.
• Non-linear baseline (M3) under few data. The authors report very large errors for M3 in Experiment 2 and explain it with the small number of data §5.3.2. Treat that as a property of this experiment, not of non-linear hand-eye methods in general.
• Versioning. The arXiv record is v2 (2023-11-21), and the source is an IJRR-style SAGE journal template. The paper does not state changes from v1. The licence recorded by wikillm fetch is CC BY 4.0.

Lineage

• Builds on: the classic hand-eye formulation with the Tsai and Lenz axis/angle method, the Daniilidis dual-quaternion method, and the Horaud non-linear quaternion method, all used as baselines §2.2 §5. For the camera-motion step, it builds on the Christy and Horaud iterative Euclidean factorisation §2.3.
• Compared against: M1 to M3 above, and the pose-based linear variant M4 §5.
• Related work cited in the text: Wei et al. (joint hand-eye and camera calibration without a rig, non-linear) §1; Ruf et al. (projective kinematic calibration for visual servoing) and a stereo-based Euclidean reconstruction as an alternative to tracking, both named as future directions §6.
• Superseded by: not stated.