An Overview of Depth Cameras and Range Scanners Based on Time-of-Flight Technologies

In plain words

A time-of-flight camera measures how far away things are by sending out light and timing how long it takes to come back. Some devices time a short flash directly. Others send a light that flickers at a steady rate and measure how far the flicker has shifted when it returns. The first kind is the sort of spinning sensor used on vehicles and robots, and it works outdoors. The second kind is the small camera type, and it works best indoors over short distances.

The second kind has a catch. A shift of one full flicker cycle looks the same as no shift at all, so distances repeat after a fixed span. The paper explains how these sensors work, lists the commercial devices, and shows how to turn their raw readings into 3-D points by treating the sensor like an ordinary camera that needs calibration.

TL;DR

This is a survey and tutorial, not a method paper. It covers two sensing principles for time-of-flight depth: pulsed-light (direct timing, outdoor, long range) and continuous-wave (phase shift, indoor, short range). It gives the four-sample phase formulas, the depth-from-phase relation with its wrap limit, and a pinhole-plus-distortion calibration for ToF cameras. Use it as a reference for equations and device specifications, and check the pitfalls below before implementing.

Problem

Depth from a single laser beam needs scanning, a mirror or many beams, to give a full image §1. Pulsed sensors need very fine timing: a centimetre of depth needs timing better than a nanosecond, because light covers about 30 cm per nanosecond (footnote in §2.1). Continuous-wave sensors measure a phase, which repeats every full cycle, so depth is ambiguous beyond one unambiguous range §4.3. Stereo cameras fail on untextured surfaces such as walls, and ToF sensors give depth regardless of texture §7. Pulsed-light sensors are the systems of choice for autonomous driving and robot navigation, because continuous-wave systems do not work under adverse outdoor light §8. The paper compares these trade-offs and points to calibration as the step that turns raw ToF data into usable coordinates §6.

Method

The paper is organised as an overview. No new method is proposed. The sections below restate its technical content.

Principle. A ToF sensor emits light and measures the travel time τ from the source to the object and back. Depth is d = ½ c τ, with c the speed of light §4.3. Pulsed-light sensors measure τ directly, and continuous-wave sensors measure a phase φ = 2π f τ for modulation frequency f §1 §4.1.

Step 1: pulsed-light sensors §2. A laser or laser diode emits pulses. Single-photon avalanche diodes (SPADs) fire on one photon, and per-pixel time-to-digital converters record the arrival time §2 §2.2. Two setups exist:

• Range scanner: one laser on a rotating mirror, or a head of 16, 32 or 64 laser-detector pairs at fixed vertical angles §2.
• 3D flash camera: one diffused laser pulse floods the scene and a 2-D SPAD array records depth in one shot §2 §3.3.

Step 2: continuous-wave sensors §4. The emitter sends a sinusoidally modulated signal, and each pixel demodulates the return. With emitted power s(t) = a1 + a2 cos(2πft) and received power r(t) = A cos(2πft − φ) + B, where A is the received amplitude and B the background offset §4.1, the cross-correlation of the two gives a value that depends on the phase. Four samples at ψ = 0, π/2, π and 3π/2 are enough to recover phase, amplitude and offset §4.1 fig. 8:

• phase: arctan of (C3 − C1) over (C0 − C2) §4.1;
• amplitude: the root of the sum of squares of the two differences, divided by a2 §4.1;
• offset: the sum of the four samples, divided by 4·a1 §4.1.

In a real pixel, the four samples come from four shutters, each open for T/4 of a modulation period T. The shutter cycle repeats for an integration time that covers hundreds of thousands of periods to raise the signal-to-noise ratio §4.2.

Step 3: depth from phase §4.3. Depth is d = c φ / (4π f). The phase wraps at 2π, so the unambiguous range is d_max = c / (2f). The measured depth is d_measured = d − n · d_max, where n is an integer that can differ from pixel to pixel. The paper cites several phase-unwrapping methods but does not choose one §4.3.

Step 4: pinhole model and intrinsics §6.1. A ToF camera is modelled as a pinhole camera. The intrinsic matrix A has α_u, α_v (scale times focal length) and the principal point (u0, v0). The paper's matrix has no skew term. The measured quantity is the radial distance d from the optical centre to the point, not the depth along the optical axis. The camera-frame point is P = d / ‖A⁻¹q‖ · A⁻¹q, where q = (u, v, 1) §6.1.

Step 5: extrinsics and lens distortion §6.2 §6.3. The rigid pose has six parameters (rotation and translation) §6.2. The distortion model has a radial polynomial l(r) = 1 + ρ1 r² + ρ2 r⁴ and a tangential term with τ1, τ2, so four distortion parameters in total §6.3. The image is undistorted by numerically inverting this model §6.3. The paper notes that the outgoing infrared signal also vignettes the intensity image, which can be modelled by a bivariate polynomial when full photometric calibration is needed §6.3.

Step 6: depth-wiggling correction §6.4. Systematic nonlinear error ("wiggling") is removed either by a lookup table or by a cubic B-spline correction d′ = d − Σ_i β_i B_{i,3}(d). The coefficients β_i come from least squares over known target depths, and the number of coefficients depends on the number of depth planes used in calibration §6.4.

Step 7: practical calibration §6.5. The amplitude-plus-offset image of a CW camera can be calibrated with standard colour-camera tools. Because ToF images are low resolution, the paper uses a special corner-detection method that fits two pencils of lines to the calibration pattern §6.5 fig. 12.

Step 8: combining ToF and colour cameras §7. Three configurations are covered: one ToF camera plus one colour camera (the Kinect v2 and DS311/DS325 have one built in); one ToF camera plus two colour cameras, which gives a stereo pair; and multiple ToF units. Cross-calibration can use a projective alignment solved by SVD and refined by bundle adjustment §7. The paper states these methods without giving their equations.

Pseudocode 1: four-bucket demodulation and depth §4.1 §4.3.

# Four-bucket demodulation of one CW-TOF pixel  [§4.1] [§4.2]
# Input:  c0..c3 = correlation samples at 0, pi/2, pi, 3pi/2  [fig. 8]
#         a1, a2 = emitted offset and amplitude (not stated how obtained)
# Output: phase (rad), amplitude, offset
def demodulate_pixel(c0, c1, c2, c3, a1, a2):
    # Phase from the two quadrature differences  [§4.1]
    # The paper writes arctan; atan2 keeps the quadrant
    phase = atan2(c3 - c1, c0 - c2)
    # Wrap into [0, 2*pi): the paper does not say how  [§4.1]
    phase = phase % (2.0 * pi)
    # Received amplitude, scaled by the emitted amplitude a2  [§4.1]
    amplitude = sqrt((c3 - c1) ** 2 + (c0 - c2) ** 2) / a2
    # Background offset, scaled by the emitted offset a1  [§4.1]
    offset = (c0 + c1 + c2 + c3) / (4.0 * a1)
    return phase, amplitude, offset


# Depth from phase  [§4.3]
# Input:  phase (rad) in [0, 2*pi), freq (Hz) of the modulation
# Output: depth in metres, inside the unambiguous range [0, d_max)
def depth_from_phase(phase, freq):
    # Unambiguous range: the depth at phase 2*pi  [§4.3]
    max_depth = SPEED_OF_LIGHT / (2.0 * freq)
    # Same as c / (4*pi*f) * phase  [§4.3]
    depth = phase / (2.0 * pi) * max_depth
    return depth

Pseudocode 2: pixel to 3-D point and wiggling correction §6.1 §6.4.

# Radial depth and pixel to camera-frame 3-D point  [§6.1]
# Input:  (u, v) pixel, d radial distance (m), A intrinsic 3x3 matrix
# Output: point (X, Y, Z) in camera coordinates
def pixel_to_point(u, v, d, A):
    # Ray direction from the pixel: p = A^-1 q  [§6.1]
    # (lens undistortion is applied to the pixel first  [§6.3])
    ray = inverse(A) @ [u, v, 1]
    # Scale the ray so its length is the measured radial distance  [§6.1]
    point = d / norm(ray) * ray
    return point


# Depth wiggling correction with a cubic B-spline  [§6.4]
# Input:  d measured depth, beta coefficients, centres of spline knots
# Output: corrected depth
def correct_wiggling(d, beta, centres):
    # Coefficients are fitted by least squares on known depths  [§6.4]
    bias = 0.0
    for i in range(len(beta)):
        # Cubic B-spline basis B_{i,3} evaluated at d  [§6.4]
        bias = bias + beta[i] * cubic_bspline(i, d, centres)
    return d - bias

Hyperparameters

Values are as the paper states them. Nothing here comes from code.

Results

This is a survey with no new experiments. The numbers below are device specifications the paper reports, and they are author-reported. The paper gives no accuracy figures for most commercial devices, because it says they depend on surface, lighting and frame rate tab. 2 tab. 3.

Table 1: pulsed-light scanners tab. 1.

Table 2: commercial pulsed-light flash cameras tab. 2.

Table 3: commercial continuous-wave cameras tab. 3.

Other reported figures: the SR4000 runs at 10–15 FPS for the most accurate depth §5.1; the Kinect v2 has 0.5 % accuracy in its range §5.2; the paper reports no accuracy number for the flash cameras tab. 2.

Paper vs code

No official code. The paper has no code link. Its only links are to vendor websites: Velodyne, Advanced Scientific Concepts, Odos Imaging, Mesa Imaging, SoftKinetic, Fotonic and PMD §2 §3 §5. A GitHub repository search for time-of-flight horaud returned no results. The paper is a survey, so the only "code" would be the vendors' own SDKs, which are not covered here.

Pitfalls

• Phase formula typo. The four-sample phase is arctan((C3 − C1)/(C0 − C2)) §4.1, but the depth formula in §4.3 writes (C0 − C1) in the denominator. The derivation in §4.1 gives C0 − C2 (C0 − C2 = a2·A·cos φ and C3 − C1 = a2·A·sin φ, inferred from the paper's C(ψ, φ) expression). Use C0 − C2.
• Scale constants unknown. The amplitude and offset formulas need the emitted amplitude a2 and offset a1 §4.1. The paper does not say how to get them. Using the sensor's own amplitude-plus-offset output is the practical route §6.5, but that is not stated as a formula.
• Radial, not z-depth. The pinhole model in §6.1 assumes the sensor measures the distance d from the optical centre to the point. The footnote says the sensor really measures to the image plane and that the offset is assumed small §6.1. Many SDKs return z-depth. Check which one your device gives before using the back-projection in Pseudocode 2.
• No skew term. The intrinsic matrix §6.1 has no skew parameter, and the paper does not justify dropping it.
• Phase wrap. The unambiguous range is c/(2f), 5 m at 30 MHz §4.3. Lowering f extends range and reduces accuracy §4.3. Multi-frequency unwrapping is cited but not described §4.3.
• SR4500 range does not match the formula. The paper states a 9 m maximum at 13.5 MHz §5.1 tab. 3. Its own relation c/(2f) gives about 11.1 m. The paper does not reconcile the two. Other SR4000 values match the formula, for example 5.17 m at 29 MHz.
• Velodyne pulse width. The text says 5 ns §3.1; Table 1 says 10 ns tab. 1.
• Velodyne minimum range. Figure 2 gives 1 m as the minimum fig. 2; Table 1 gives 2 m tab. 1.
• SPAD timing. "About 10 ps" §1 and "a few tens of picoseconds" §2.2 disagree.
• Frame rates. The DragonEye is given as 10 FPS in the text §3.3 but as 10–30 FPS in Table 2 tab. 2. The SR4000 is listed at 30 FPS maximum tab. 3 but the paper says 10–15 FPS is more accurate §5.1.
• Integration time and motion. CW sensors integrate over very many periods §4.2, so motion blur appears on moving objects and fast shutter speeds are not possible §4.3.
• Error sources. The paper lists demodulation, integration, temperature, motion blur, distance, background light, phase wrapping, scattering and multi-path errors §4.3. It does not give magnitudes for most of them. It also says depth error from unknown materials and complex scenes is hard to predict §6.5.
• Wiggling correction parameters. The number of B-spline coefficients and the calibration depths are not stated §6.4. An implementer must choose them and calibrate per device.
• Multi-camera interference. Cameras can run together only if each has its own modulation frequency. SR4000 units are limited to six such frequencies, and SR4500 units encode their frequency §5.1 §4.3. Many devices do not let the user set the frequency §7.
• Vendor data are dated. The tables describe devices at the paper's time. The paper notes that the D-imager was discontinued in March 2015 §5.3. Re-check any spec before relying on it.
• Stereo versus ToF. The paper argues that ToF data help stereo on untextured surfaces §7. It gives no quantitative comparison.
• Figures. The paper's figures are not reproduced here. They are cited by number only.

Lineage

• Builds on: the cited foundations rather than cards in this wiki. Lange and Seitz (2001) for the CW principle §4; Cova et al. (1981) for SPAD timing §2.2; Glennie and Lichti for the Velodyne HDL-64E model §3.1.
• Compared against: the paper compares the pulsed scanners, the flash cameras and the CW cameras against each other through its tables tab. 1–3.
• Superseded by: not stated.