Time-of-flight depth sensing

Overview

Every ToF variant turns a round trip of light into a distance. With travel time τ and speed of light c, depth is d = c τ / 2 2012.06772 §4.3. Pulsed sensors measure τ directly. Continuous-wave (CW) sensors measure a phase φ = 2π f τ for a modulation frequency f 2012.06772 §1 2012.06772 §4.1.

The shared constraint is timing precision. A centimetre of depth needs timing better than a nanosecond, because light covers about 30 cm per nanosecond 2012.06772 §2.1. CW sensors trade direct timing for phase, which repeats every full cycle, so depth is ambiguous beyond one unambiguous range 2012.06772 §4.3.

Only one card (the survey by Horaud et al.) covers this concept. The paper has no experiments of its own, so the figures below are formulas and device specifications the paper reports. They are not measured performance.

Variants

Pulsed-light range scanners

A laser or laser diode emits pulses, and single-photon avalanche diodes (SPADs) with per-pixel time-to-digital converters record arrival times 2012.06772 §2 2012.06772 §2.2. The scanner uses one laser on a rotating mirror, or a head of 16, 32 or 64 laser-detector pairs at fixed vertical angles 2012.06772 §2. The Velodyne HDL-64E, HDL-32E and VLP-16 are the models in the paper's table, at 905 nm with 2–120 m or 2–100 m range and 2 cm accuracy 2012.06772 tab. 1. Card: Horaud et al. survey.

Pulsed-light flash cameras

One diffused laser pulse floods the scene, and a 2-D SPAD array records depth in one shot 2012.06772 §2 2012.06772 §3.3. The paper's table lists TigerEye and DragonEye units at 128×128 pixels with ranges up to 1100 m and 1500 m, all outdoor-capable 2012.06772 tab. 2. The MIT Lincoln prototype used a 1 ns pulse at 532 nm 2012.06772 §3.3. Card: Horaud et al. survey.

Continuous-wave cameras

The emitter sends a sinusoidally modulated signal, and each pixel demodulates the return 2012.06772 §4.1. A real pixel takes four samples, each through a shutter open for a quarter of the modulation period, and sums them over many periods, up to hundreds of thousands, to raise the signal-to-noise ratio 2012.06772 §4.2. The paper's table lists SR4000, SR4500, DS311, DS325, E70, E40 and Kinect v2 at 160×120 to 512×424 pixels and 30 to 60 FPS 2012.06772 tab. 3. Most are indoor-only; E70 and E40 are marked outdoor-capable there. Card: Horaud et al. survey.

When to use which

Implementation sketch

The common core is: get a per-pixel quantity (phase or arrival time), convert it to a radial distance, then back-project through a calibrated camera model. Calibration covers intrinsics, distortion and a depth-wiggling correction 2012.06772 §6.

The code below follows the paper's CW formulas. The paper's LaTeX source contains no code link.

# Four-bucket demodulation of one CW pixel  [arxiv:2012.06772 §4.1] [§4.2]
# Input:  c0..c3 = samples at 0, pi/2, pi, 3pi/2, each summed over many
#         periods  [arxiv:2012.06772 §4.2]
#         a1, a2 = emitted offset and amplitude (source not stated)  [§4.1]
# Output: phase (rad), amplitude, offset
def demodulate_pixel(c0, c1, c2, c3, a1, a2):
    # Phase from the arctan; atan2 picks the quadrant (implementer's choice)
    phase = atan2(c3 - c1, c0 - c2)
    # Map into one period; the paper does not say how to wrap  [§4.1]
    phase = phase % (2.0 * pi)
    # Amplitude, scaled by the emitted amplitude a2  [§4.1]
    amplitude = sqrt((c3 - c1)**2 + (c0 - c2)**2) / a2
    # Offset, scaled by the emitted offset a1  [§4.1]
    offset = (c0 + c1 + c2 + c3) / (4.0 * a1)
    return phase, amplitude, offset


# Depth from phase  [arxiv:2012.06772 §4.3]
# Input:  phase (rad) in [0, 2*pi), modulation frequency f (Hz)
# Output: depth (m) inside the unambiguous range [0, d_max)
def depth_from_phase(phase, f):
    # Unambiguous range: depth at phase 2*pi  [§4.3]
    d_max = C_LIGHT / (2.0 * f)
    # d = c * phase / (4*pi*f), equal to d_max * phase / (2*pi)  [§4.3]
    return d_max * phase / (2.0 * pi)


# Pixel to camera-frame point, pinhole model  [arxiv:2012.06772 §6.1]
# Input:  (u, v) pixel after undistortion  [§6.3], radial distance d (m),
#         A 3x3 intrinsic matrix (no skew term in the paper)
# Output: point (X, Y, Z) in camera coordinates
def pixel_to_point(u, v, d, A):
    # Ray direction from the pixel  [§6.1]
    ray = inverse(A) @ [u, v, 1]
    # Scale the ray so its length equals the radial distance d  [§6.1]
    return d / norm(ray) * ray

Notes for implementers, by variant:

• Pulsed sensors skip the demodulation step. They use arrival time τ directly, with d = c τ / 2 2012.06772 §4.3.
• CW sensors must unwrap phase. The paper cites several methods but describes none. Multi-frequency unwrapping is cited without detail 2012.06772 §4.3.
• Depth-wiggling correction. A systematic nonlinear error ("wiggling") can be corrected by a lookup table or a cubic B-spline correction whose coefficients come from least squares on known depths 2012.06772 §6.4. The number of coefficients and calibration depths is not stated, so calibrate per device.
• Colour alignment. Projective alignment solved by SVD, refined by bundle adjustment, is one option. The paper states the method without equations 2012.06772 §7.
• Practical calibration. The amplitude-plus-offset image of a CW camera can be calibrated with standard colour-camera tools. Because ToF images are low resolution, the paper uses a corner detector that fits two pencils of lines to the pattern 2012.06772 §6.5.

Pitfalls the card records:

• Radial distance, not z-depth. The pinhole model in 2012.06772 §6.1 assumes the sensor reports radial distance from the optical centre. The paper's footnote says the sensor measures to the image plane and assumes a small offset 2012.06772 §6.1. Many SDKs return z-depth, so check the device before using pixel_to_point.
• No skew term. The intrinsic matrix has none, and the paper does not justify dropping it 2012.06772 §6.1.
• Scale constants. The amplitude and offset formulas need a1 and a2, and the paper does not say how to get them. Using the sensor's own amplitude-plus-offset output is the practical route 2012.06772 §6.5, but the paper does not give that as a formula.
• Motion blur. Long integration means moving objects blur, and fast shutter speeds are not possible 2012.06772 §4.2 2012.06772 §4.3.
• Error sources. The paper lists temperature, motion blur, distance, background light, phase wrapping, scattering and multi-path errors 2012.06772 §4.3. It gives magnitudes for few of them.

Contradictions

• Phase formula, within the paper. The four-sample phase is arctan((C3 − C1) / (C0 − C2)) 2012.06772 §4.1. The depth formula writes arctan((C3 − C1) / (C0 − C1)) 2012.06772 §4.3. The paper does not reconcile them, and this page does not pick one. Check the derivation before implementing.
• SR4500 range. The paper states a 9 m maximum at 13.5 MHz 2012.06772 §5.1 2012.06772 tab. 3. Its own relation c / (2f) gives about 11.1 m at that frequency 2012.06772 §4.3. Other SR4000 values match the relation: 5.17 m at 29 MHz 2012.06772 §5.1.
• HDL-64E pulse width. The text gives 5 ns 2012.06772 §3.1. Table 1 gives 10 ns 2012.06772 tab. 1.
• HDL-64E minimum range. Figure 2 gives 1 m 2012.06772 fig. 2. Table 1 gives 2 m 2012.06772 tab. 1.
• SPAD timing jitter. The introduction says about 10 ps 2012.06772 §1. Section 2.2 says a few tens of picoseconds 2012.06772 §2.2.
• DragonEye frame rate. The text gives 10 FPS 2012.06772 §3.3. Table 2 gives 10–30 FPS 2012.06772 tab. 2.
• SR4000 frame rate. Table 3 gives 30 FPS maximum 2012.06772 tab. 3. The text says 10–15 FPS is the most accurate setting 2012.06772 §5.1.
• CW outdoor use. The paper says CW systems do not work under adverse outdoor light 2012.06772 §8. Table 3 marks E70 and E40 as outdoor-capable 2012.06772 tab. 3. The paper does not explain the difference, and "adverse" is not defined.

Papers

• Horaud et al. survey: survey of pulsed-light and CW ToF sensing, with depth-from-phase formulas, device specifications and a pinhole-plus-distortion calibration model for ToF cameras. No new experiments.