Talk2Car: Taking Control of Your Self-Driving Car

In plain words

A passenger in a self-driving car might say, "park behind the silver car next to the lamppost," and the car has to work out which object in its camera view the words mean. This paper builds a dataset for that task. It takes recorded driving videos, has crowd workers write free-form instructions about one object in each scene, and records which object each instruction points to.

The dataset is then used to test several existing AI models on the same job. The simple models do poorly, which shows the task is hard. The best model gets about half the answers right. Models do worse when instructions are long, and most of them drop when many similar objects are in view; two multi-step reasoning models hold up better in that case.

TL;DR

Talk2Car is a dataset of 11,959 natural-language driving commands, each linked to one object in a street scene from nuScenes. Seven models are evaluated: three simple baselines and four adapted or original visual-language models. On the test split, MAC has the best accuracy at IoU above 0.5 (50.51%), and the non-proposal models are roughly twice as fast as the proposal-based ones (author-reported). Read it if you need a referring-expression dataset for driving or a reference point for command grounding; note the NC-SA data license and the gaps between paper and released code below.

Problem

Object referral asks for the image region that a sentence describes. Earlier referral datasets use short, curated captions from everyday photos §2 §4.2. The authors argue that passenger commands are longer, less structured, and often contain irrelevant or even wrong spatial information §4.2. The task is also time-sensitive for a car, so predicted location and speed both matter §1. The paper restricts each command to exactly one referred object §1.

Method

The paper contributes a dataset and an evaluation, not a new model.

Dataset collection §3.1. Crowd workers on Amazon Mechanical Turk watched the front-camera view of a 20-second nuScenes video. They picked a frame and an annotated object in it, then wrote a command that referred to that object. Each video had five workers. Commands were checked by two reviewers, and workers were paid a bonus for approved work §3.1. The paper says there is no limit on wording as long as the car could follow the command §3.1.

Splits §3.3. The data is divided into train, validation and test sets of roughly 70%, 10% and 20%. Samples from one video stay in one split. Left-hand (Singapore) and right-hand (Boston) traffic and scene conditions are kept balanced across splits. Every object category in the test set also appears in training. The authors sampled random splits 10,000 times and kept one that met these constraints §3.3. The sampling seed is not stated.

Test subsets §3.3 fig. 3. Subsets select by the number of same-category objects in the scene (ambiguity), by command length, and by object distance.

Object evaluation §5.4. A prediction is correct when its box has IoU above 0.5 with the ground-truth box. The 3D annotations are converted to 2D boxes first. The conversion method is not described in the paper §5.4.

Region-proposal models §5.1.

• OSM (Object Sentence Mapping). An SSD-512 detector proposes 64 regions. A ResNet-18 encodes each region, and a bidirectional GRU encodes the command. The region whose embedding has the highest inner product with the command embedding is returned §5.1 app. A, OSM.
• SCRC (Spatial Context Recurrent ConvNet). Adds a global image encoding and an 8-number spatial vector per box: corner coordinates, height, width and centre. The original SCRC formulation is referenced rather than restated §5.1.

Non-proposal models §5.2.

• MAC. A recurrent cell with read and write units, borrowed from visual question answering. A Bi-LSTM encodes the command, and a ResNet-101 encodes the image. The authors feed the pre-softmax attention map of the last read unit to a fully connected layer that predicts the grid cell holding the box centre, then use a convolution to predict the box offsets §5.2 app. A, MAC.
• Stack-NMN. Chains neural modules, each conditioned on attended command words, over ResNet-101 features. Like MAC, it predicts a grid cell and then offsets §5.2 app. A, STACK-NMN.

Simple baselines §5.3.

• RS. Random choice among 64 proposals, averaged over 100 runs.
• BOBB. Returns one fixed box that overlaps the most training boxes (Algorithm 1) alg. 1 app. A, BOBB.
• RNM. A dependency parser extracts nouns; a proposal is chosen at random from those matching a category name. If no noun matches, a random proposal is used. Averaged over 100 runs §5.3.

# Object Sentence Mapping (OSM)  [§5.1] [app. A, OSM]
# Input:  image; command tokens
# Output: one predicted box

def predict_osm(image, command_tokens):
    # 1. Propose 64 candidate regions with an SSD-512 detector  [§5.1]
    boxes = ssd_proposals(image, count=64)           # (64, 4)
    # 2. Encode each region with a ResNet-18  [§5.1] [app. A, OSM]
    region_feats = resnet18(crop_regions(image, boxes))  # (64, d)
    # 3. Encode the command with a bidirectional GRU  [app. A, OSM]
    command_vec = bigru(command_tokens)              # (d,) d not stated
    # 4. Score every region by inner product with the command  [§5.1]
    scores = region_feats @ command_vec             # (64,)
    # 5. Return the box with the highest score  [§5.1]
    return boxes[argmax(scores)]

# Biggest Overlapping Bounding Box (BOBB)  [alg. 1]
# Input:  training ground-truth boxes; image 1600 x 900; IoU threshold 0.5
# Output: one box [x1, y1, w, h]

def find_best_bbox(train_gt_boxes, img_w=1600, img_h=900, thresh=0.5):
    # 1. Sample candidate coordinates on a grid with step 5  [alg. 1]
    xs = linspace(0, img_w, step=5)
    ys = linspace(0, img_h, step=5)
    best_count = 0
    best_box = None
    # 2. Enumerate boxes with x2 > x1 and y2 > y1  [alg. 1]
    for x1 in xs:
        for x2 in xs:
            if x2 <= x1:
                continue
            for y1 in ys:
                for y2 in ys:
                    if y2 <= y1:
                        continue
                    # 3. Candidate box as [x1, y1, w, h]  [alg. 1]
                    box = [x1, y1, x2 - x1, y2 - y1]
                    # 4. Count training boxes with IoU above thresh  [alg. 1]
                    count = count_iou_above(box, train_gt_boxes, thresh)
                    # 5. Keep the box with the highest count  [alg. 1]
                    if count > best_count:
                        best_count = count
                        best_box = box
    return best_box

Hyperparameters

Only values stated in the paper or its appendix are listed. "not stated" means the paper gives no value. Appendix A is the reproducibility section app. A.

OSM

SCRC

Stack-NMN

MAC

Baselines

Dataset

Results

Author-reported, on the Talk2Car test split (2,447 commands) with IoU above 0.5 §5.4 tab. 2. Speed was measured on a TITAN Xp tab. 2.

Other reported results, all on the test split §5.4:

• Pre-training the proposal network on all training keyframes, not only annotated frames, raised OSM from 35.31 to 40.78 and SCRC from 38.70 to 41.15 §5.4.
• Zeroing the command vector gave SCRC 40.37 and OSM 21.65 §5.4.
• Using ground-truth boxes with the RNM heuristic gave 54% §5.4.
• GloVe embeddings did not help; ELMo and BERT gave small gains for some models §5.4.
• Test subsets: performance falls for longer commands and for more same-category objects, except for Stack-NMN and MAC on ambiguity. Shown only as plots, without numbers fig. 3.

Paper vs code

What was searched. The paper names a project website, macchina-ai.eu (in the abstract). Fetching it failed with a TLS error. The arXiv record has no code link. A GitHub repository search for "talk2car" returned talk2car/Talk2Car, which describes itself as the official dataset repository for this paper, and cites the paper and the same KU Leuven grant number as the acknowledgements. That repository was cloned at the commit in the frontmatter and is the one used here. Other Talk2Car repositories, by the first author, are extensions (Talk2Car-Destination, Talk2Car-Trajectory, Talk2Car-Expr, VL-BERT-Talk2Car, vilbert-Talk2car) and are not used.

The official repository does not contain the four evaluated models. OSM, SCRC, MAC and Stack-NMN come from other papers. The repository contains a dataset devkit (talk2car.py, eval.py) and a challenge baseline (baseline/). The baseline is a different method: CenterNet region proposals matched with a command embedding README.md. Do not treat it as the paper's models.

Licence of the code. The repository's LICENSE is CC BY-NC-SA 4.0. It states that the dataset and its extensions are non-commercial, following the nuScenes terms LICENSE. The code is cite-only; nothing is quoted from it.

Evaluation. The paper's metric is accuracy at IoU above 0.5. The released code computes mean IoU and "AP" at thresholds 0.50 to 0.95 in steps of 0.05, and counts a hit when IoU is strictly greater than the threshold [code: eval.py#L26-L36] eval.py. At 0.5 the two agree. The paper does not state whether it used this scorer.

3D-to-2D conversion. The paper converts 3D ground-truth boxes to 2D for evaluation, but gives no method §5.4. The released get_2d_bbox returns a stored 2D box talk2car.py. The old projection-based conversion is commented out talk2car.py. The stored boxes are therefore the only conversion in the release. Its method is not documented.

Splits. The paper describes the constraints behind the split §3.3 but gives no seed. Use the released split files, not a fresh random split, if you need comparable numbers. Whether the released split is the one used for the table was not checked.

Pitfalls

• Dataset licence. The annotations are CC BY-NC-SA 4.0 non-commercial in the released repository, and that licence follows the underlying nuScenes data. Using the dataset commercially may be prohibited even though this card is published under CC BY-SA. Check before any commercial training run.
• Ambiguous "mean IoU" in the loss masking for OSM app. A, OSM. Implement it as the paper's wording says, and record the choice.
• Stack-NMN backbone cut point is described as "at the fourth channel" app. A, STACK-NMN. Confirm against the original implementation before you rely on the 32 x 32 x 1024 shape.
• Feature-space size of OSM. The paper does not state the shared dimension between region and command embeddings, or whether a projection is applied before the inner product §5.1 app. A, OSM.
• BOBB is expensive. With the 5-pixel grid, the algorithm has about 321 x values and 181 y values. That gives roughly 8 x 10^8 candidate boxes, each scored against every training box alg. 1. This count is derived from the algorithm; the paper gives no runtime. Also, the box format says "lower left corner" for [x1, y1], but the image y-axis direction is not stated app. A, BOBB.
• BOBB counting. The paper names the helper getAmountOfIoUAboveThresh but does not define it. The loop above treats it as a count of training boxes with IoU above the threshold, which the name suggests.
• MAC reasoning steps differ from Stack-NMN (10 versus 4) and are chosen on validation app. A, MAC app. A, STACK-NMN. Use the stated values, not the defaults of the original MAC code.
• Learning-rate schedule for MAC uses a loss-change threshold of 0.2 whose unit is not stated app. A, MAC.
• Random baselines (RS and RNM) average 100 draws. The seed and variance are not reported §5.3.
• Inference speed was measured on a TITAN Xp, with one value per model. Speeds are not directly comparable across hardware.
• Blanking the command raised SCRC. The authors read this as a bias in the global image encoding §5.4. Keep it in mind when interpreting SCRC against the other models.
• Proposal quality matters. Changing only the proposal network's pre-training data changes OSM by about 5 points §5.4. Compare models with equal proposal settings.

Lineage

• Builds on the object-referral datasets ReferIt, RefCOCO, RefCOCO+, RefCOCOg, Cityscapes-Ref and CLEVR-Ref, all compared in §4.1 tab. 1.
• Built on nuScenes for the source videos and 3D boxes §3.1.
• Adapted models: OSM after Karpathy et al. (2014); SCRC after Hu et al. (2016); MAC after Hudson and Manning (2018); Stack-NMN after Hu et al. (2017). Names and years as cited in the paper's bibliography §5.1 §5.2.