Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigation

In plain words

Robots that move among people should behave in a socially compliant way. Describing that behaviour as rules or a reward is hard, so one approach is to record a person driving the robot and let a learning method copy that behaviour.

This paper releases such recordings. A person steered two wheeled or four-legged robots around a university campus, in both indoor and outdoor places, while the robots logged their sensors. Each recording is tagged with the kinds of social situations it contains, such as crossing a street or passing a group that is talking. The authors then show two things can be done with the data: a model can tell which of two people drove a robot, and a simple copy-the-demonstrator policy can plan paths that people rate as more socially compliant than a standard planner.

TL;DR

• SCAND is a public dataset of 138 human-teleoperated social navigation trajectories, 8.7 hours, 25 miles, on two robots §I tab. I.
• Each trajectory carries sensor streams and labels for 12 social interaction types §III-A §III-B tab. II.
• Demonstrator classification reaches 74.48% accuracy on 16 trajectories with two demonstrators, against a 50% chance level §IV-A.
• A behaviour-cloned planner has a lower average Hausdorff distance to the demonstrated path than move_base (0.26 vs 1.25), and 14 participants rated it higher for social compliance and safety on the Spot §IV-B fig. 7.
• The dataset is the main contribution. The analyses are small, and the paper says the learned policies handle only simple scenarios §IV-B.

Problem

Imitation learning and inverse reinforcement learning need demonstrations of socially compliant navigation, and the paper states that large-scale datasets of such demonstrations in the wild are lacking §I. The authors state that existing datasets fall short in one or more ways: simulated only §II-B; small or indoor-only §II-B; collected with a scripted social scenario rather than naturally occurring interactions §II-B; or teleoperated without social intent §II-B tab. I. The paper's own assumption is that a human demonstrator can show social navigation more easily than a reward function can be written for it §I.

Method

Data collection §III-A

1. Four human demonstrators, including the first two authors, drive each robot with a joystick. The campus is the setting §III-A.
2. For each of the 138 trajectories, the demonstrator walks behind the robot and keeps about two metres away on average. The demonstrator does not deliberately interact with the crowd §III-A.
3. Data is collected in the wild, indoors and outdoors, on sidewalks, roads, lawns and inside buildings, during peak foot traffic. Two outdoor sessions took place near a football stadium on game days §III-A.
4. Velocity commands are kept within the ranges given in the hyperparameter table §III-A.
5. Sensor data is recorded as rosbag files with the sensor-to-body transforms for each robot §III-A fig. 3.

Robots and sensors §III-A fig. 3

• Both robots carry a VLP-16 Velodyne lidar, a 6-DoF IMU and a front-facing Azure Kinect RGB camera §III-A.
• The Jackal (wheeled) adds a front stereo camera and wheel odometry §III-A.
• The Spot (legged) adds five body-mounted monocular cameras and records visual odometry and leg joint angles from its own API §III-A.

Labelled interaction tags §III-B tab. II fig. 2

• Every trajectory has a list of tags. The tags come from a fixed set of 12 labels §III-B.
• The tag set and per-tag counts are in tab. II. The paper gives no annotation procedure or agreement figure §III-B.
• The authors state the labels are meant for studying specific scenarios, and they do not add annotations for human detection or tracking. For those, they point to the JRDB dataset §III-A.

Analysis 1: demonstrator classification §IV-A fig. 4

Question: can the demonstrator be identified from behaviour alone, which would mean more than one social strategy exists?

1. Use 16 trajectories driven by two demonstrators on the same route (Speedway road) and split them 12 for training and 4 for validation §IV-A.
2. Input is a 10-second window of: lidar subsampled to 1 Hz and rendered as a grayscale bird's-eye-view image; the robot's positions relative to the first lidar frame; the human's future trajectory of 200 points in the latest lidar frame; and inertial and joystick values at each lidar frame §IV-A.
3. A four-layer convolutional encoder processes the image. A three-layer fully connected network processes the other inputs. A three-layer fully connected classifier head then predicts the demonstrator. The loss is binary cross-entropy §IV-A fig. 4.

# Demonstrator classifier  [§IV-A] [fig. 4]
# Input:  10 s of observations per sample
# Output: score for one of two demonstrators (output form not stated)

def classify(bev_lidar, other_obs):
    # Encode 1 Hz grayscale BEV lidar with a 4-layer CNN  [§IV-A]
    # bev_lidar: (T, H, W) grayscale images, H and W not stated
    lidar_features = conv_encoder(bev_lidar)
    # Encode positions, future path, IMU and joystick values  [§IV-A]
    # other_obs: concatenated, 3-layer MLP
    other_features = mlp_encoder(other_obs)
    # Fuse both encodings and classify with a 3-layer MLP head  [§IV-A]
    fused = concat(lidar_features, other_features)
    return classifier_head(fused)

Analysis 2: behaviour cloning for global and local planning §IV-B fig. 5

Question: can a standard imitation learner trained on SCAND produce a socially compliant global and local planner?

1. The inputs are two seconds of observations: lidar subsampled to 2 Hz as bird's-eye-view images, the positions of earlier lidar frames relative to the first, and inertial data at each lidar frame §IV-B.
2. A global path and desired velocities from the move_base planner are added, computed for a goal 10 m ahead of the robot, as prior information §IV-B.
3. A shared encoder (not pretrained, per the paper) feeds two three-layer fully connected heads, one for each planner §IV-B.
4. The global head predicts 200 points of the demonstrated path, within a 10 m horizon. The local head predicts 20 timesteps of (v, ω) joystick commands §IV-B.
5. The two mean-squared-error losses are summed and both heads are updated together, with behaviour cloning as the learning method §IV-B §II-A.

# Joint behaviour cloning, global and local planner  [§IV-B]
# Input:  2 s of observations, a move_base global plan and desired velocities
# Output: scalar loss

def training_step(observations, move_base_plan, desired_velocities,
                  demo_path, demo_commands):
    # Encode the 2 s window with one shared encoder  [§IV-B]
    shared = encoder(observations, move_base_plan, desired_velocities)
    # Global head: 200 points of the demonstrated path, within 10 m  [§IV-B]
    predicted_path = global_head(shared)
    # Local head: 20 timesteps of (v, omega) joystick commands  [§IV-B]
    predicted_commands = local_head(shared)
    # Sum the two mean-squared errors and update both heads  [§IV-B]
    loss = (
        mse(predicted_path, demo_path)
        + mse(predicted_commands, demo_commands)
    )
    return loss

Real-robot human trials §IV-B fig. 6 fig. 7

• The local planner was run on a Spot indoors, with 14 participants and two scenarios: a static one and a dynamic one §IV-B.
• Each participant rated the robot on two 1-to-5 scales, social compliance and safety, after each scenario and for each algorithm. The algorithm order was randomised §IV-B.

Anticipated use cases §V

The paper suggests future work on representation learning, trajectory prediction, inverse reinforcement learning §V and real-to-sim transfer §V. It notes that the data comes from one city and may carry local conventions, such as which side of the road people walk on §V.

Hyperparameters

Results

Author-reported. Setup is given per row.

Statistical tests reported in §IV-B: one-way ANOVA gives F(1, 55) = 38.67, p < 0.001 for social compliance and F(1, 55) = 47.87, p < 0.001 for safety. See Pitfalls for the degrees-of-freedom question.

Comparison of datasets tab. I:

Tag set tab. II:

Paper vs code

No official code. Searched: the paper text and arXiv metadata (no code link), the authors' project page (lists the dataset download and no code or license), and GitHub repository search for SCAND, SCAND socially compliant navigation and scand with the owner xuesu (no matching repositories). The paper cites the AMRL autonomy stack (ut-amrl/graph_navigation, §III-A) for recording the sensor data. That is robot software used for collection, not code for the classifier, the behaviour-cloning agent or the statistics. It was not cloned or checked here.

No training code for the classifier or the behaviour-cloning agent was found. The paper states that it does not pretrain the encoders with any representation learning algorithm §IV-B.

Pitfalls

• Dataset licence is not verified. The paper's CC BY 4.0 licence covers the paper, not the data. The project page states no licence. The Texas Data Repository record (linked on the project page) returned HTTP 403, so its terms could not be read. This card describes the data and does not redistribute it. Check the record's terms before using the data.
• Demonstrator classification is small. It uses 16 trajectories from two demonstrators on one route. The 12/4 split is train/validation; the paper then reports accuracy on a "held-out test set" without giving its size or saying whether it is the same 4 trajectories. No variance or confidence interval is given. The paper notes that many 10 s windows contain no differentiating social interaction, and it reads 74.48% as successful on that basis §IV-A.
• Chance level. The 50% reference assumes balanced classes, which the paper does not state.
• Only Spot data in the analysis. The analyses use Spot data only §IV intro. The Jackal data is not used in any reported result.
• Hausdorff numbers have no units. The paper gives 1.25 and 0.26 without units and without saying whether the metric is directed or symmetric, or how the paths were sampled. Do not compare these numbers with other papers' Hausdorff values.
• Degrees of freedom in the human study do not reconcile from the text. A one-way ANOVA with two groups and F(1, 55) implies 57 observations. The design described (14 participants, 2 algorithms, 2 scenarios, one rating per question) gives 56 ratings, so 54 error degrees of freedom. The paper does not explain the unit of analysis or whether repeated measures per participant were modelled. Treat the p-values with that uncertainty.
• Small human study. It uses 14 participants, one robot (Spot), one indoor location, and one baseline (move_base). The authors call the behaviour-cloned agent able to handle only simple social scenarios §IV-B.
• Hidden preprocessing. The paper refers to "processed sensor observations" (bird's-eye-view rendering, frame alignment, normalisation) but does not give the image size, range or resolution, or how the lidar points are projected. Reproducing the analyses needs these details, and they are not stated.
• Loss weighting and training details are missing. The behaviour-cloning loss is a sum of two MSE terms §IV-B. Their weights, the optimiser, learning rate, epochs and batch size are not stated for either analysis.
• Annotation process is not described. It is not stated who assigned the 12 tags, how many annotators there were, or how disagreements were resolved. Treat the tags as the authors' labels, not as a validated annotation.
• Demonstrator mix. Two of the four demonstrators are authors §III-A. The paper does not say how the other two compare in style or experience.
• Version. The card is compiled from the arXiv v2 LaTeX source (updated 2022-06-08). Version 1 was not checked.

Lineage

• Builds on: move_base (ROS navigation stack) as the baseline planner §IV-B; behaviour cloning §IV-B (cited to Bain and Sammut, 1995).
• Closest prior imitation work for social navigation: Tai et al. (simulated, GAIL, social force model as expert) §II-A.
• Compared against in the dataset table: CoBot, L-CAS, NCLT, FLOBOT, JRDB, THÖR tab. I.
• Related work on inverse reinforcement learning for social cost functions: Kretzschmar et al. (one-hour dataset) §II-A; and MEDIRL on static scenarios §V.
• Superseded by: not known from content already in this wiki.