Paper · 2023-02-24 · cs.CV
Decoupling Human and Camera Motion from Videos in the Wild
SLAHMR recovers global 3D human trajectories from moving-camera videos. It combines DROID-SLAM camera motion, a scale factor fitted by optimization, SMPL smoothness and shape/pose priors, and the HuMoR motion prior with contact terms. Author-reported: EgoBody world MPJPE 141.1 mm versus 387.8 mm for PHALP+; PoseTrack ID switches fall from 450 to 420 with the recovered scale.
Human mesh recoveryParametric body modelSimultaneous localization and mapping (SLAM)Tracking by detectionhuman-motionworld-coordinatescamera-motionslamsmplmotion-priormulti-person-tracking
arXivPDFCodeMarkdown13 min read
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
A video taken with a moving camera shows people moving in the frame, but the frame itself moves too. A person who stands still while the camera pans looks like they are walking. This paper recovers where each person actually walked in a shared world frame, so that the camera's motion is taken out of the picture.
It works in three passes. First, a standard tool estimates how the camera moved from the static background. Second, the people are placed in the world using that camera motion, and their bodies are made smooth and plausible. Third, a learned model of how people move fixes the one thing the camera cannot tell us, the true scale of the scene, and checks that each footstep meets the floor. The method is aimed at in-the-wild videos, where the scene was never reconstructed in advance.
The authors report lower world-frame error than their baselines on the EgoBody dataset. They also report fewer identity switches on PoseTrack when the recovered scale is given to a tracker. These numbers are the authors' own and come from those specific datasets.
TL;DR
Use this when you have monocular video with a moving camera and need people's trajectories in one world frame, not per-frame 3D poses. The method needs a per-frame human pose tracker and a SLAM system as inputs, and runs an optimization of roughly 40 minutes for about 100 frames with 4 people app. C. Results are author-reported on EgoBody and PoseTrack.
Problem
Per-frame 3D human pose estimators place each person in the camera frame. The person's motion in that frame mixes their own movement with the camera's movement §3, eq. 3. Methods that ignore the camera do not recover global trajectories §2. Methods that rely on background pixels usually need a full scene reconstruction, which in-the-wild videos often lack §3.
A monocular SLAM system gives the camera's relative rotation and translation, but not the metric scale of the scene §3. The scale is what turns a plausible body displacement into a real one, so it has to come from somewhere. Human motion priors are defined in the body's own frame and have so far been limited to known or static cameras §2.
Method
The input is a video of T frames with N people. Each person is an SMPL-H model with a global orientation, 22 body joint angles, a shape vector shared over time and a root translation §3, eq. 1. The mesh and joints come from the differentiable SMPL function §3, eq. 2. Per-frame poses and identity tracks come from PHALP, and the camera transforms (world to camera, R_t and T_t) come from DROID-SLAM §3. A scale alpha links the camera's translation to the people's world translation §3.1.
Stage 1: place people in the world §3.1.
- Start the global orientation and root translation of every person in the world frame from the camera-frame estimate, using the camera rotation and translation. Set alpha = 1 §3.1.
- Take the body shape and local pose from PHALP unchanged §3.1.
- Project the joints into each image with the camera intrinsics, and compare with the 2D keypoints weighted by their confidences. The per-keypoint error uses the Geman-McClure function §3.1, eq. 6.
- Optimize only the global orientation and root translation for 30 iterations, with the data weight 0.001 §3.1, eq. 7.
Stage 2: smooth trajectories and fit body and scale §3.2. 5. Add a joint smoothness term on the world joints, a shape prior and a pose prior in the VPoser latent space §3.2, eq. 8. 6. Optimize all world parameters and alpha together, using the same keypoint term from stage 1, for 60 iterations. The weights are 5 for smoothness, 0.05 for shape and 0.04 for pose §3.2, eq. 9.
Stage 3: learned motion prior, ground and contacts §3.3. 7. Replace the smoothness term with HuMoR, a transition model. Each state holds the SMPL pose, velocities and joint locations. A transition is a latent z of size 48 with a learned Gaussian conditional prior, and the energy is the negative log-likelihood of the latents §3.3, eq. 10. 8. Initialize each latent from the pretrained HuMoR encoder on consecutive states, and roll out the trajectory with the decoder §3.3, eq. 11. 9. Also optimize a free ground plane g, and use the decoder's per-joint contact probabilities c(j) for two terms: a zero-velocity term for likely contacts (skate) §3.3, eq. 12, and a term that keeps likely contacts within a threshold distance of the plane §3.3, eq. 13. The weights are 100 and 10 §3.3. 10. Optimize the initial states, the scale alpha, the plane g and the latents together, with the data, shape and pose terms and the three stage-3 terms combined as in §3.3, eq. 14. 11. Grow the horizon in chunks of 10 frames, H = 10 tau. After each chunk, roll out 10 more frames when the loss falls by less than a threshold gamma, with at least 5 and at most 20 iterations per chunk §3.3. The paper does not give gamma.
Implementation details §3.4; app. A-C. Missing detections are filled by interpolating orientations on SO(3) and translations linearly; motion priors keep the body plausible through those frames §3.4. Tracks are stored from their first appearance and optimized in one batch, with invisible frames masked out app. C. Several floor planes can be used when people stand at different heights app. C. The stronger tracker PHALP+ replaces PHALP as the input detector §3.4.
# SLAHMR, three optimization stages [§3]
# Input: PHALP poses per person and frame (camera frame), 2D keypoints
# with confidences, DROID-SLAM rotations R_t and translations T_t
# Output: world trajectories, camera scale alpha, ground plane g
def slahmr(people, keypoints, cam_R, cam_T):
# Start each person in the world frame from the camera estimate;
# alpha starts at 1 [§3.1]
alpha = 1.0
world = init_world(people, cam_R, cam_T, alpha)
# Stage 1: fit only global orientation and root translation,
# 30 iterations, data weight 0.001 [§3.1] [eq. 7]
world = fit_root(world, keypoints, lambda_data=0.001, iters=30)
# Stage 2: add smoothness, shape and pose priors, and alpha;
# 60 iterations [§3.2] [eq. 8] [eq. 9]
world, alpha = fit_smooth(world, alpha, keypoints, iters=60)
# Stage 3: learned transition prior, ground plane and contacts [§3.3]
world, alpha, ground = fit_motion(world, alpha, keypoints)
return world, alpha, ground# Stage 3 horizon loop [§3.3]
# T: number of frames; latents: one per transition
def fit_motion_horizon(states, latents, T):
# Horizon H = 10 * tau, tau = 1 .. ceil(T / 10) [§3.3]
for tau in range(1, ceil(T / 10) + 1):
horizon = 10 * tau
# Keep iterating until the loss drop is below gamma, with
# 5 to 20 iterations per chunk; gamma is not stated [§3.3]
latents = optimize_chunk(states, latents, horizon)
return latents# Latent initialization and rollout [§3.3] [eq. 11]
# states: (T, D) augmented HuMoR states; latents: (T - 1, 48)
def init_latents(states):
# Encode each consecutive pair of states with the pretrained encoder
# [§3.3] [eq. 11]
return [encoder(states[t], states[t - 1])
for t in range(1, len(states))]
def rollout(initial_state, latents):
# Decode each transition: s_t = s_{t-1} + decoder(z_t, s_{t-1})
# [§3.3] [eq. 11]
states = [initial_state]
for z in latents:
states.append(states[-1] + decoder(z, states[-1]))
return states# Ground-contact terms on joints [§3.3] [eq. 12] [eq. 13]
# joints: (T, J, 3); contact: (T, J) in [0, 1]; ground: plane g
def env_loss(joints, contact, ground, delta):
# Likely contacts should not move between frames (skate) [eq. 12]
step = norm(joints[1:] - joints[:-1], axis=-1) # (T-1, J)
skate = sum(contact[:-1] * step)
# Likely contacts should lie within delta of the ground [eq. 13]
height = sum(contact * relu(dist(joints, ground) - delta))
return 100.0 * skate + 10.0 * height # weights from [§3.3]Hyperparameters
Paper values are cited to the paper. The "Code" column gives the value
in the official code at code_commit, cited to the file, where it differs or
adds detail. "not stated" means the paper and the code do not give it.
| Name | Value | Where | Notes |
|---|---|---|---|
| Stage 1 data weight lambda_data | 0.001 | §3.1, eq. 7 | Code uses 0.001 in all three stages optim.yaml |
| Stage 1 iterations | 30 | §3.1 | Code: 30 optimizer steps, each running up to 20 inner L-BFGS iterations optim.yaml optim.yaml |
| Stage 2 iterations | 60 | §3.2 | optim.yaml |
| lambda_smooth | 5 | §3.2, eq. 9 | Code weight 10 on 0.5 times the sum of squares, so effectively 5 optim.yaml losses.py |
| lambda_beta (shape) | 0.05 | §3.2, eq. 9 | Code also scales this by the sequence length T losses.py |
| lambda_pose | 0.04 | §3.2, eq. 9 | optim.yaml |
| VPoser latent size | 32 | §3.2 | Pose prior is the squared latent norm |
| HuMoR latent size | 48 | §3.3 | |
| lambda_CVAE | 0.075 | §3.3, eq. 10 | optim.yaml |
| lambda_skate | 100 | §3.3, eq. 12 | Code uses squared distances times 0.5 optim.yaml losses.py |
| lambda_con | 10 | §3.3, eq. 13 | optim.yaml |
| Contact threshold delta | not stated | §3.3, eq. 13 | Code uses 0.08 m on the z coordinate losses.py |
| Horizon chunk size | 10 frames | §3.3 | optim.yaml |
| Chunk iterations | min 5, max 20 | §3.3 | Code max is 20; also advances on a loss drop below 20 after 5 steps and on a negative loss with a drop below 100 optim.yaml optimizers.py |
| Chunk stop threshold gamma | not stated | §3.3 | See above |
| Optimizer | L-BFGS, learning rate 1 | §3.3 | Strong-Wolfe line search optim.yaml optimizers.py |
| Robust function | Geman-McClure | §3.1, eq. 6 | Sigma 100, not in the paper optim.yaml losses.py |
| Initial scale alpha | 1 | §3.1 | params.py |
| Ground plane default | not stated | §3.3 | Code starts from (0, -1, 0, -0.5) dataset.py |
| Runtime | about 40 min for about 100 frames and 4 people | app. C | Hardware not stated |
| EgoBody evaluation clip length | 100 frames | app. A | Sequences split into 100-frame chunks |
| Camera for EgoBody | DROID-SLAM with ground-truth intrinsics | app. A |
Results
Author-reported. Setup: EgoBody validation set, with the camera from DROID-SLAM and 3D ground truth for one person app. A. PoseTrack tracking uses the DROID-SLAM cameras, with and without the recovered scale §4.2. Lower is better in every column.
| Setting | Metric | Baseline | This paper | Source |
|---|---|---|---|---|
| EgoBody, world | W-MPJPE (mm) | PHALP+ 387.8; GLAMR 416.1 | 141.1 | tab. 3 |
| EgoBody, world | WA-MPJPE (mm) | PHALP+ 204.9; GLAMR 239.0 | 101.2 | tab. 3 |
| EgoBody, world | Acc error (mm/s^2) | PHALP+ 307.6; GLAMR 173.5 | 25.78 | tab. 3 |
| EgoBody, local | PA-MPJPE (mm) | PHALP+ 72.16; GLAMR 114.3 | 79.13 | tab. 3 |
| EgoBody, ablation | W-MPJPE / WA / Acc | w/o last stage 234.0 / 152.9 / 275.9; PHALP+ with SfM 253.6 / 150.3 / 302.1 | full 141.1 / 101.2 / 25.78 | tab. 1 |
| EgoBody and 3DPW, local | PA-MPJPE (mm) | PHALP+ 72.16 (EgoBody), 64.68 (3DPW dagger), 56.70 (3DPW star) | 79.13 (EgoBody), 62.60 (3DPW dagger), 55.86 (3DPW star) | tab. 2 |
| PoseTrack tracking | ID switches | T3DP 655; PHALP 541; PHALP+ 450; PHALP+ with DROID cameras 446 | PHALP+ with DROID cameras and alpha 420 (-6.7%) | tab. 4 |
The paper's own reading of these: the method is best on world-frame metrics in EgoBody, but PHALP+ is best on the local PA-MPJPE, which the paper says does not describe global trajectories app. A §4.1. On 3DPW the full system beats PHALP+ in both variants, with and without ground-truth tracks tab. 2. On PoseTrack, DROID cameras without rescaling do not change ID switches, while the scale alpha lowers them §4.2 tab. 4.
Paper vs code
Official code: the project page linked from the abstract §3.4 has a code
link to https://github.com/vye16/slahmr, and the page lists the same four
authors. The repository is MIT-licensed, cloned at code_commit. Searched:
the abstract and paper links, then the project page, which named this
repository. I did not run a GitHub search because the link was enough.
The following differ between the paper and the code at code_commit.
- The smooth-stage scale flag is ignored. The config sets
smooth.opt_scale: Falseoptim.yaml andrun_optpasses it on run_opt.py. ButSmoothOptimizeradds the scale whenevermodel.opt_scaleis true optimizers.py. So the code optimizes the scale in stage 2, as §3.2 says. The flag and the effective behavior disagree; the effective behavior matches the paper. - Reprojection confidence is squared. The paper weights each keypoint residual by its confidence psi §3.1, eq. 6. The code weights it by confidence squared losses.py.
- Shape prior is scaled by sequence length. The paper's shape term has no length factor §3.2, eq. 9. The code multiplies it by T, the number of frames in the whole sequence, in stages 2 and 3 losses.py run_opt.py. With lambda 0.05 and T = 100, the effective weight is 5.
- Smoothness has a factor of one half. The code's smoothness term is 0.5 times the sum of squared differences losses.py, with weight 10 optim.yaml. The effective weight equals the paper's 5 on the unhalved sum, so the two agree.
- Skate uses squared distances. The paper's skate term is the unsquared norm of joint displacement §3.3, eq. 12. The code uses squared displacement, times 0.5, with weight 100 losses.py. These are different functions, not just different constants.
- Contact height uses the z coordinate. The paper measures each contact's distance to the free plane g §3.3, eq. 13. The code thresholds the absolute z coordinate at 0.08 m and does not reference the optimized plane in that loss losses.py losses.py. I did not trace how the world frame relates z = 0 to the plane. The two agree only if the world frame is floor-aligned.
- Stage 3 has terms the paper does not write out. The code adds a GMM prior on the initial state with weight 0.075 optim.yaml losses.py, a joint-consistency term with weight 100 and a bone-length term with weight 2000 optim.yaml losses.py. The paper defers its extra stabilizing terms to HuMoR and gives no weights §3.3.
- Cameras are not optimized in stage 3 by default. The motion stage sets
opt_cams: Trueoptim.yaml, but the effective flag isopt_cams and model.opt_camsand the model config setsopt_cams: Falseoptimizers.py config.yaml. The paper does not claim camera optimization in stage 3 §3.3. - Stage 1 iteration counts differ in meaning. The paper's "30 iterations" is 30 outer optimizer steps in the code. Each step can run up to 20 inner L-BFGS iterations optim.yaml optim.yaml.
- Per-frame SMPL stage is off. The optimizer file describes an SMPL-fit stage between root and smoothing optimizers.py. Its iteration count is 0 optim.yaml, so the paper's stage 2 is the code's smooth stage.
- Neck keypoint is ignored. The code zeros the confidence of the neck keypoint in the 2D term specs.py. The paper says nothing about this. A commented-out line in the same file would also ignore the hips.
- Default floor is a fixed plane. With
est_floor: Falseconfig.yaml, the floor starts from (0, -1, 0, -0.5) dataset.py dataset.py. The paper says g is free but gives no start value §3.3. - Multi-floor grouping is optional. The appendix's multi-floor clustering
app. C exists as a
groupoption. The default is one shared floor config.yaml.
Agreements, checked against the code: the three stage data weights are 0.001 optim.yaml; pose 0.04 and shape 0.05 for stage 2 optim.yaml; motion prior 0.075, skate 100 and contact 10 for stage 3 optim.yaml; the Geman-McClure robust function losses.py; the conditional motion prior, on by default humor_model.py; and initial alpha 1 params.py.
Pitfalls
- Stage 2 scale. An implementation that reads the config flag would hold alpha at 1 during stage 2, which departs from the paper and the code's effective behavior. Follow the paper, or fix the flag. See the Paper vs code section.
- Shape prior grows with length. Because the shape weight is multiplied by T, results shift with clip length. Compare runs only at equal length, or normalize.
- Undefined stopping rule. Gamma and the chunk stopping rule are not in the paper. The code uses 20, with extra rules for negative losses. An implementation without these will chunk differently and can change results.
- Squared skate term. The code's squared form weights large foot displacements more heavily, and small ones less, than the paper's norm does.
- Plane and z coordinate. A new implementation must decide whether the contact height is measured to the free plane or to z = 0 in a floor-aligned frame. The code does the second, implicitly.
- Stage 3 stabilizers. The paper does not say whether the joint-consistency and bone-length terms and the GMM initial-state prior are needed. The code uses them with the weights in the Paper vs code section. Leaving them out is an untested deviation.
- HuMoR and camera motion. The authors report that HuMoR fails on 30 percent of EgoBody sequences when the camera is treated as static, and that their method succeeded on 99 percent app. D. These are author-reported.
- Approximate intrinsics. For in-the-wild videos the camera uses approximate intrinsics, and SfM struggles to separate translation from rotation when the focal length is long app. E.
- Identity switches. Tracking errors that merge two people into one track can propagate into the optimization app. E.
- Co-linear motion. When a person moves toward or away from the camera, the method can underestimate their displacement, so they appear to run in place app. E.
- Single floor. Most optimization failures on PoseTrack come from the single-floor assumption, which the authors address with floor clustering app. D app. C.
- Cost. Runtime grows linearly with the number of tracked people app. C. The hardware is not stated.
- Local metrics. The method is not better than PHALP+ on PA-MPJPE in EgoBody tab. 2. Do not judge it on local pose accuracy alone.
Lineage
- Builds on: PHALP for tracking and per-frame 3D pose §3; DROID-SLAM for relative camera motion §3; HuMoR for the transition prior and contact terms §3.3; VPoser for the pose latent space §3.2; SMPL-H for the body model §3. Each is cited in the paper's reference list, not as a card here.
- Compared against: GLAMR, VIBE and VIBE-opt on EgoBody tab. 3; PHALP and T3DP for tracking on PoseTrack tab. 4.
- Superseded by: not known from content in this wiki.
Source and license
Adapted from “Decoupling Human and Camera Motion from Videos in the Wild” by Vickie Ye, Georgios Pavlakos, Jitendra Malik, Angjoo Kanazawa (arXiv:2302.12827), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/vye16/slahmr at commit 58518fec9918 (MIT). Any quoted code is shown under that license.