wikillm

Paper · 2016-12-01 · cs.CV

TorontoCity: Seeing the World with a Million Eyes

Introduces TorontoCity, a benchmark over the Greater Toronto Area (712.5 km², about 400,000 buildings) with aerial, street-level panorama, LiDAR and car-sensor data. Ground truth comes from high-precision maps aligned to the imagery by an automatic method plus a short human check. Nine tasks are defined; the baseline FCN and ResNet pilot is weak on instances, contours and height.

Semantic segmentationInstance segmentationdatasetbenchmarkaerial-imagerystreet-viewurban-mapping

arXivPDFMarkdown10 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Teaching a computer to understand a city needs a lot of labelled pictures, and drawing labels by hand over a whole metropolis is far too slow. TorontoCity is a very large collection of city data: photos taken from planes and drones, street-level panoramas, and laser and camera scans from a car that drove around Toronto. Instead of people drawing the labels, the authors used accurate city maps (buildings, roads, zoning) and lined them up with the photos.

Lining up is not free. Street-level panoramas are placed on the map only roughly, so the authors wrote software that finds the right position using the aerial photos, and then a person only has to say yes or no to each result. The dataset comes with a set of tasks, such as finding building outlines, estimating building height, or telling road edges from the middle of the road.

In a first test, standard image-labelling networks did well on some tasks, such as road and building labelling, and poorly on others, such as separating individual buildings, drawing clean outlines and guessing heights. The authors present the dataset as a starting point for research rather than a finished answer.

TL;DR

TorontoCity is a city-scale benchmark (712.5 km² of land, 8,439 km of road, about 400,000 buildings) built by aligning high-precision maps with aerial, panoramic, LiDAR and car-sensor data, with minimal human checking §1 §3. The paper defines eight benchmark tasks and reports one pilot study on a 125 km² subset. Baselines (FCN, ResNet) do well on road and building segmentation and poorly on instance segmentation, contours and building height §6. All numbers are author-reported. Dataset, code and training details are not in the paper §6 §7.

Problem

Large benchmarks such as ImageNet, PASCAL, COCO, KITTI and Cityscapes have driven progress, but their data comes from similar viewpoints and they lack rich semantics and 3D information at scale §1. Labelling a city by hand is not feasible at this size §1 §4. Online maps such as OpenStreetMap and Google Maps are less accurate than the high-precision maps used here §3.2.

The main obstacle is alignment. Aerial images line up with the maps, but street-level panoramas have geolocation errors of up to 5 m, with an average of 1.5 m, while their rotation is accurate §4.1. Projecting the maps directly onto panoramas therefore gives poor ground truth §4.1.

Method

The paper has two parts. First, the dataset is assembled from several sources §3. Second, two alignment procedures produce the ground truth §4.

Data sources §3.1:

  • Aerial images: full GTA coverage in 2009, 2011, 2012 and 2013, orthorectified to 10 cm/px for 2009 and 2011 and 5 cm and 8 cm for 2012 and 2013. They have four channels (RGB and near-infrared), 16-bit for 2011 and 8-bit otherwise. They are projected to UTM zone 17 (WGS84) and tiled into non-overlapping 500 × 500 m images. The images are not true orthophotos, so facades are visible.
  • Airborne LiDAR: captured in 2008 with a Leica ALS sensor at 6.8 points per m², covering 22 km².
  • Street-level panoramas: about 520 full 360° panoramas per km² on average, with metadata (geolocation, address, and pitch, yaw and tilt). Resized to 3200 × 1600 px.
  • Car platform: GoPro Hero 4 RGB camera, Velodyne HDL-64E LiDAR, PointGrey Bumblebee3 stereo camera and Applanix POS LV positioning, driven for about 90 km including repeats.

Maps used as annotations §3.2: building footprints and 3D building models (about 400,000 buildings, mean height 4.7 m, mean footprint area 148 m²), road centrelines with attributes and intersections, road curbs, urban zoning categories (residential, commercial, industrial, institutional, with multiple categories per zone allowed), and point features such as poles, traffic lights, street lights and street trees. The paper does not say who produced the maps or under what licence they are available.

Panorama alignment, coarse §4.1. Each panorama is first projected onto the ground plane as a 400 × 400 m patch at 10 cm/px, with three degrees of freedom for the camera offset. A score combines normalised cross-correlation (NCC) with the aerial image and a Gaussian prior centred at (0, 0, 2.5) m with diagonal covariance (2, 2, 0.2) m. Both images are rescaled to [0, 1] before NCC. The search covers [-10, 10] × [-10, 10] × [2.2, 2.6] m in 0.1 m steps, by exhaustive search with NCC computed via FFT. The prior is a fixed lookup table.

Panorama alignment, fine §4.1. Road curbs are aligned to boundary edges detected in the panorama by the structured-edge detector. The search covers [-1, 1] × [-1, 1] m in 5 cm steps. A human then accepts or rejects each image. About 34.35 % of images pass. Verifying one takes under 2 s, while annotating the alignment by hand takes 20 s §4.1.

Road surface from curbs and centrelines §4.2. Curb segments and centreline segments are not aligned in the maps. Each curb segment chooses one of its k nearest centreline segments, or state 0 for no match. A Markov random field scores the choices: a unary term from distance and angle to each centreline (weights not stated in the text), and a pairwise smoothness term between consecutive curb segments. Its job is to avoid holes at intersections. Multiple curbs may match one road. Because the chains have a simple structure, exact inference by dynamic programming runs per chain. The authors mark errors as "don't care" regions by manual inspection, then convert each curb-centreline assignment to road polygons §4.2.

Pseudocode
# Coarse alignment of one rectified panorama to the aerial image  [§4.1]
# Input:  aerial patch; ground patch (400 m x 400 m, 10 cm/px)
# Output: camera offset (x, y, z) in metres
def coarse_offset(aerial, ground):
    # Rescale both images to [0, 1] before NCC  [§4.1]
    aerial_01 = rescale_to_unit(aerial)
    ground_01 = rescale_to_unit(ground)
    best_score = -inf
    # Exhaustive search over a 0.1 m grid  [§4.1]
    for (x, y, z) in search_window_coarse:
        # Combination of appearance and prior is not stated in the
        # active text; the weighted form is commented out  [§4.1]
        score = appearance(aerial_01, ground_01, x, y, z)
        score = score + prior_lookup(x, y, z)
        if score > best_score:
            best_score = score
            best_offset = (x, y, z)
    return best_offset
Pseudocode
# Fine alignment: road curbs against boundary edges  [§4.1]
# Input:  curb polylines; boundary edge map from a structured-edge detector
# Output: refined (dx, dy) offset in metres
def fine_offset(curbs, edge_map, coarse):
    # Search +-1 m in 5 cm steps around the coarse solution  [§4.1]
    for (dx, dy) in fine_window(coarse):
        # Score function is not stated in the active text  [§4.1]
        score = match_curbs_to_edges(curbs, edge_map, dx, dy)
    return best (dx, dy)
Pseudocode
# Assign curb segments to centrelines and build road polygons  [§4.2]
# Input:  ordered curb chain; centreline segments; k candidates per segment
# Output: label per curb segment (0 = no match)
def assign_curbs(curb_chain, centrelines, k):
    # Candidate states: k nearest centrelines, plus state 0  [§4.2]
    candidates = nearest_k(curb_chain, centrelines, k)
    # Unary: weighted distance and angle to each candidate; weights not
    # stated  [§4.2]
    # Pairwise: smoothness between consecutive curb segments  [§4.2]
    # Exact dynamic programming on the chain  [§4.2]
    labels = viterbi(candidates, unary, pairwise)
    # Manual check marks errors as "don't care"  [§4.2]
    return labels

Benchmark tasks §5: semantic segmentation of building footprints and roads (mean IoU, top-down); building instance segmentation (weighted coverage, AP, precision and recall at 50 % IoU); building structured contours (IoU multiplied by a turning-function similarity; formula deferred to a supplement not in this source); road topology from curb and centreline polylines (precision and recall at 0.5 m and 2 m, after 10 cm discretisation); ground-view urban zoning classification (top-1 accuracy); bird's-eye urban zoning segmentation (IoU, with merged classes); ground-view road segmentation (IoU); and building height (log-RMSE, formula not in active text). Planned tasks include building reconstruction, facade parsing, trees, traffic lights and signs §7.

Hyperparameters

NameValueWhereNotes
Panorama ground patch400 × 400 m at 10 cm/px§4.1
Panorama resize3200 × 1600 px§3.1
Aerial tile size500 × 500 m, no overlap§3.1
Coarse search window[-10, 10] × [-10, 10] × [2.2, 2.6] m, 0.1 m step§4.1exhaustive search
Gaussian priormean (0, 0, 2.5) m, diagonal covariance (2, 2, 0.2) m§4.1fixed lookup table
Fine search window[-1, 1] × [-1, 1] m, 5 cm step§4.1
Discretisation for road topology10 cm§5
Road topology thresholds0.5 m and 2.0 m§5 tab. 3
Contour simplification (RDP)0.5 m threshold, about 13 vertices§6
Morphological filter (instances)opening: erosion then dilation, same size§6kernel size not stated
Morphological filter (roads)closing (dilation then erosion)§6kernel size not stated
Pilot subset125 km²: 50 train, 25 val, 50 test§6train, val and test regions do not overlap or touch
Pilot street-view images56K: 22K train, 18K val, 16K test§6
Ground-view road subset1000 train, 200 val, 800 test§6
Learning rate, optimiser, batch size, epochs, augmentationnot stated§6"More details are in supp. material", which is not in this source
Network depthResNet with 56 conv layers§6see pitfalls: other parts of the paper say ResNet-32

Results

Author-reported. Setup: the 125 km² pilot subset §6. The paper reports one pilot study and no comparison with prior datasets. The comparison table is commented out in the source §3.2.

Aerial segmentation §6, mean and per-class IoU tab. 1:

MethodMeanBuildingRoad
FCN77.64 %70.44 %73.32 %
ResNet (56 conv layers)78.46 %69.15 %76.44 %

Building instance segmentation §6 tab. 2:

MethodWeightedCovAPRe-50Pr-50
FCN39.74 %8.04 %19.64 %18.38 %
FCN + Open43.19 %16.45 %24.55 %36.09 %
ResNet38.70 %10.47 %21.30 %21.93 %
ResNet + Open41.10 %22.92 %22.78 %43.78 %

Road centrelines and curbs §6 tab. 3. F1 only; precision and recall are in the source table.

MethodCentreline F1 @0.5 mCentreline F1 @2 mCurb F1 @0.5 mCurb F1 @2 m
FCN0.1690.6260.4440.778
FCN + Close0.1730.6390.4440.781
ResNet0.1620.6130.5750.796
ResNet + Close0.1620.6440.5680.799

Building contours §6 tab. 5. Metric names as in the table; the text does not define them.

MethodWeightedCovPolySim
FCN0.4560.323
ResNet0.4010.292

Ground-view urban zoning classification, top-1 §5 tab. 4:

ModelInitialisationTop-1
AlexNetImageNet fine-tuned75.49 %
VGG-16ImageNet fine-tuned79.12 %
GoogleNetImageNet fine-tuned77.95 %
ResNet-152ImageNet fine-tuned79.33 %
AlexNetfrom scratch66.48 %
ResNet-32from scratch75.65 %
GoogleNetfrom scratch75.08 %
NiNfrom scratch79.07 %

Aerial urban zoning segmentation (merged classes, per-class scores) §6 tab. 6:

MethodResidentialOpen spaceOthers
FCN60.20 %32.20 %5.57 %
ResNet51.71 %33.63 %1.49 %

Ground-view road segmentation §6: IoU 97.21 %, pixel accuracy 98.64 %, top-down IoU 87.53 % on 800 test panoramas. The network is not named in the active text.

Building height §6: no network estimated height from aerial images alone. No numbers are reported.

Paper vs code

No official code. The paper links no repository or project page, and the LaTeX source contains no URL. Searched the GitHub API for "torontocity" (10 hits, none by an author or an author's lab), for "TorontoCity" under org uber-research (0 hits), and for the user urtasun (no public repositories). The paper says nothing about releasing the data or the evaluation server, so no release is recorded.

Pitfalls

  • Dataset and maps are not licensed in the paper. The CC BY 4.0 licence covers the paper text and figures. It does not give a licence for the TorontoCity imagery, the panoramas, the LiDAR, the car data or the maps. Check these separately before reuse. Street-view panoramas in particular come from a third party §3.1.
  • Several formulas are missing from the active text. The coarse score's weight λ, the road-assignment unary weights, the building-height log-RMSE formula, and the contour metric (the turning-function similarity) appear only in commented-out drafts or in a supplement that is not part of this source §4.1 §4.2 §5.
  • Training details are not stated. Learning rate, optimiser, batch size, epochs and augmentation are deferred to "supp. material" §6. A commented draft mentions Adam and larger tiles, but the active text does not.
  • ResNet depth is inconsistent. The text says 56 conv layers §6, figure captions say ResNet56 fig. 2, and the from-scratch ground-view zoning row says ResNet-32 §5 tab. 4. The paper does not explain the mismatch.
  • Rejection rate. Only about 34 % of panoramas pass the fine alignment check §4.1. Any reuse needs that filtering step, and the pilot numbers are on accepted images only.
  • Alignment accuracy. The coarse alignment works at aerial resolution, so it is coarse. Fine alignment only refines a 5 cm grid within ±1 m §4.1. The paper claims high accuracy for maps, but gives no quantitative figure for the final panorama label error.
  • Instance segmentation is weak. The baseline uses connected components after semantic segmentation, which merges touching buildings. Morphological opening helps a little §6. The paper says the task is open.
  • Building height. No model learns it from aerial images, and no numbers are given §6.
  • Table 6 caption. It says "qualitative results" but the values are percentages. The metric is not restated, though §5 implies IoU.
  • Source inconsistencies. The intro refers to a dataset comparison figure (fig:dataset) whose table is commented out. The source reuses the label tab:zone_seg for two tables, so references to it resolve to the second one. Figure and table numbers in this card are inferred from source order.
  • Scale of the pilot. The experiments use a 125 km² subset and one year of aerial imagery (2011) with the ground panoramas §1 §6. The other years and the car data are not used in the experiments.
  • Labelling cost. The paper quotes 20 s per manual alignment and under 2 s per verification §4.1. These are the authors' own timings with no procedure described.

Lineage

  • Builds on: FCN [fcn] and ResNet [resnet] as the baseline networks; NCC and the structured-edge detector [structedge] for alignment; the Ramer–Douglas– Peucker algorithm [rdp] for contours §6.
  • Compared against: ImageNet, PASCAL, COCO, KITTI and Cityscapes as earlier benchmarks §1; ISPRS and aerial KITTI are in the commented-out comparison only §3.2.
  • Superseded by: not stated in this source.

Source and license

Adapted from “TorontoCity: Seeing the World with a Million Eyes” by Shenlong Wang, Min Bai, Gellert Mattyus, Hang Chu, Wenjie Luo, Bin Yang, Justin Liang, Joel Cheverie, Sanja Fidler, Raquel Urtasun (arXiv:1612.00423), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.