Places: An Image Database for Deep Scene Understanding

In plain words

Recognising the kind of place in a photo (a kitchen, a harbour, a bedroom) needs many labelled example photos, and the big image collections mostly show objects. The authors built Places, a collection of about 10 million photos, each labelled with the kind of place it shows. They collected candidate photos from web image search, using a list of place names combined with many descriptive adjectives. Human workers then confirmed or rejected each label, and a trained classifier helped find more candidates for the rarer places.

The paper also measures how varied the collection is compared with two other datasets, and it trains standard image networks on it to give baseline scores. Places is meant as a common training and test ground for scene recognition.

TL;DR

• Dataset paper: 10,624,928 images from 434 scene categories after four rounds of construction and labelling §2.2.4.
• Two benchmark families are defined: Places205 and Places365 §2.3.1.
• AlexNet, GoogLeNet and VGG-16 are trained on it, and a ResNet-152 is fine-tuned on Places365 §3.
• Relevant if you need a scene-centric training or evaluation set, or scene-recognition baselines. Check the image terms before use: the paper says nothing about image-level licences (see Pitfalls).

Problem

A supervised model needs a training set with enough coverage of the classes and enough samples per class §1. The authors argue that object-centric datasets such as ImageNet do not cover scenes, and that scene classification needs a dataset built around place categories §1. A good dataset is dense (many images) and diverse (many appearances and viewpoints), and the authors note that density alone can hide low diversity §2.3.2. They also note that single top-1 accuracy can be ill-defined for scenes, since a scene often fits several labels §3.1.

Method

The contribution has three parts: how the dataset is built §2.2, how its diversity is measured §2.3.2, and how CNN baselines are trained and evaluated on it §3. The category list is inherited from the SUN scene database §2.1.

1. Category space. Start from the SUN scene category names, which the authors describe as a quasi-exhaustive list of environments §2.1.
2. Download candidates (Step 1). Each category name is combined with each of 696 common English adjectives and sent to Google Images, Bing Images and Flickr §2.2.1. About 60 million image URLs were found §2.2.1. Only colour images of at least 200 x 200 px are kept §2.2.1. PCA-based duplicate removal is run within each category, both in Places and in SUN, so that the two datasets share no images §2.2.1.
3. Human labelling (Step 2). On Amazon Mechanical Turk, each HIT shows 750 candidate images plus 30 positive and 30 negative control images from SUN. Workers with at least 90% control accuracy are kept §2.2.2. Round 1 defaults to "No", so workers only mark the positives §2.2.2. Round 2 runs on the round-1 positives with the default set to "Yes" §2.2.2. 25.4% of round-2 images were relabelled "No" §2.2.2. A third iteration was tried on a few exemplars and not pursued §2.2.2. The result is 7,076,580 images in 476 categories §2.2.2.
4. Classifier-assisted bootstrapping (Step 3). The remaining images (about 53 million) were not assigned to any category §2.2.3. An AlexNet trained on 1,000 images per category (50 for validation), for the 413 categories with at least 1,000 images, reached 32% validation accuracy §2.2.3. It scores the leftover images. For categories with too few exemplars, the top-ranked images with predicted score above 0.8 go to a third human round with default "No" §2.2.3. About 3 million images were added §2.2.3.
5. Separating similar classes (Step 4). 46 synonym pairs are merged §2.2.4. 53 ambiguous pairs are found, and workers classify images as one of the two categories or neither §2.2.4 fig. 4. The final database has 10,624,928 images in 434 categories §2.2.4.
6. Diversity measure. For sets A and B, the diversity of A relative to B is 1 minus the probability that two random images from A are closer than two random images from B §2.3.2. For more than two sets the definition is eq. 1. Workers pick the most similar pair among 12 pairs (4 from each dataset) §2.3.2. There were 40 trials per category, two observers per trial, over 88 categories shared by Places, SUN and ImageNet §2.3.2.
7. Benchmarks. Places205 has 2,448,873 training images from 205 categories, with 100 validation and 200 test images per category §2.3.1. Places88 is the 88 categories shared with ImageNet and SUN §2.3.1. Places365-Standard keeps the 365 categories with more than 4,000 images each, and has 1,803,460 training images §2.3.1. Places365-Challenge has the same categories and about 8 million training images §2.3.1.
8. CNN baselines. AlexNet, GoogLeNet and VGG-16 are trained on Places205 and Places365-Standard with Caffe; a ResNet-152 is fine-tuned on Places365-Standard §3. Scores are the class score averaged over 10 crops of each test image tab. 1 tab. 2.
9. Generic features. The paper uses fc7 activations (4096-d) of AlexNet and VGG, and the globally pooled layer before the softmax of GoogLeNet (1024-d) §3.3. A linear SVM with C = 1 is trained on these features §3.3. A Hybrid1365-VGG is trained on the 1000 ImageNet classes plus the 365 Places365 classes §3.3.

The construction procedure, condensed, in our own words. The paper has no algorithm box, so this is a paraphrase and not an alg. citation:

# Places construction pipeline (paraphrase; no algorithm box in the paper)
# Input:  scene names from SUN [§2.1]; 696 adjectives [§2.2.1]
# Output: per category, images confirmed by human labellers [§2.2.4]

def build_places(scene_names, adjectives):
    # 1. Search with name + adjective, keep unique URLs  [§2.2.1]
    urls = search_unique_urls(scene_names, adjectives)
    # Keep colour images of at least 200 x 200 px  [§2.2.1]
    images = [img for img in download(urls) if is_color_200px(img)]
    # Remove PCA-based near-duplicates within each category  [§2.2.1]
    images = remove_duplicates_pca(images)

    # 2. Round 1 crowd check, default answer No  [§2.2.2]
    positives = crowd_check(images, default_answer="no")
    # Round 2 on round-1 positives, default answer Yes  [§2.2.2]
    labelled = crowd_check(positives, default_answer="yes")

    # 3. Train a classifier on categories with >= 1000 images  [§2.2.3]
    classifier = train_alexnet(per_category_samples(1000, 50))
    for category in sparse_categories:
        # Rank leftover images by predicted score for this category  [§2.2.3]
        ranked = rank_by_score(classifier, leftover_images, category)
        # Send top images with score above 0.8 to a round with default No
        top = [img for img in ranked if score(img) > 0.8]  # [§2.2.3]
        labelled += crowd_check(top, default_answer="no")

    # 4. Merge 46 synonym pairs, then split 53 confusable pairs  [§2.2.4]
    labelled = merge_synonyms(labelled)
    labelled = crowd_pairwise_check(labelled, ambiguous_pairs)
    return labelled

Key equation for the diversity measure eq. 1:

\mathrm{Div}_{A_2,\dots,A_N}(A_1) = 1 - p\left(d(a_{11}, a_{12}) < \min_{i=2:N} d(a_{i1}, a_{i2})\right) eq. 1

Hyperparameters

Values in the first block come from the paper. Rows marked "code only" are from the released Caffe or PyTorch files, not from the paper. The paper does not state a learning rate, batch size, iteration count, momentum, weight decay or dropout for its CNNs §3.

Results

Author-reported. Setup: Caffe-trained CNNs on Places205 and Places365-Standard §3; top-1 and top-5 accuracy with 10-crop averaging tab. 1 tab. 2; generic features scored with a linear SVM on eight benchmarks §3.3.

Places205 test set and SUN205 test set tab. 1

The two leaderboard rows are other groups' entries, listed in the paper's table as the top two at the time of writing (2 Oct 2016) tab. 1 §3.1.

Places365 validation and test set tab. 2

A footnote to tab. 2 in §3.1 says the ResNet result may reflect fine-tuning or under-training, because it is not trained from scratch.

Generic features, linear SVM, 8 benchmarks tab. 3

Each value is the top-1 accuracy or precision on that benchmark; the caption says all values are top-1 tab. 3. Average is the mean of the eight columns tab. 3. The authors say the Places365-VGG feature gives the best reported single feature plus linear SVM on SUN397, and the Places205-VGG feature on MIT Indoor67, as far as they know §3.3. The Hybrid1365-VGG has the best average over the eight benchmarks §3.3.

Other reported results: a public demo built on Places205-AlexNet reached about 72% top-5 accuracy on 9,925 anonymous user feedback entries from October 2014 to May 2016 §3.2. This is uncontrolled user data, not a benchmark.

Paper vs code

The official repository is metalbubble/places365, linked in the paper's footnote §3, at commit 8a953ed56438726dc98bdef3796d042e7f1f171e. Its LICENSE is MIT LICENSE. The repository ships the Places365 solvers, the Caffe deploy files and PyTorch scripts. It does not contain the training images or the Places205 training configuration.

• Test-time crops. The paper averages class scores over 10 crops tab. 1 tab. 2. The released code uses a single crop: the training script resizes to 256 and takes a 224 centre crop [code: train_placesCNN.py#L114-L115]. The demo does the same [code: run_placesCNN_basic.py#L32-L33]. The Caffe validation layer takes one 227 crop with mirroring off train_val_alexnet.prototxt. No 10-crop evaluation code was found.
• Learning rate and batch size. The paper states none §3. The Caffe Places365 solvers use base learning rate 0.01 [code: solver_alexnet.prototxt#L4] and batch 128 [code: train_val_alexnet.prototxt#L18]. The separate PyTorch script defaults to learning rate 0.1 and batch 256 train_placesCNN.py, with step decay by 10 every 30 epochs train_placesCNN.py. The paper does not say which training code produced its reported models.
• Which training data the solvers cover. The paper trains Places205 and Places365-Standard models §3. The repository's Caffe solvers are for Places365 only solver_alexnet.prototxt. No Places205 solver is included.
• Category count. The paper says Places365 has 365 categories §2.3.1. The category file has 365 entries categories_places365.txt. No discrepancy.
• Later releases. The README lists models not in the paper, including ResNet-152 trained from scratch, ResNet-50 and DenseNet-161 [code: README.md]. These are not described in this card's results and are not used.

Pitfalls

• The counts do not reconcile. The introduction gives 476 categories §1, and Step 2 also gives 476 §2.2.2. Merging 46 synonym pairs §2.2.4 would leave 430, but the paper's final count is 434 §2.2.4. The paper does not explain the difference. Use 434 and 10,624,928 as stated, and check counts against the released data, not against this card.
• Image-level licences are not stated. The paper's CC BY 4.0 licence covers the paper text. The photos come from web search engines §2.2.1, and the paper says nothing about their licences or redistribution. Check the dataset's own terms before downloading or redistributing any image.
• Step 3 has no defined threshold for "too few exemplars" §2.2.3. The number of images sent to the third human round per category is not stated. An implementer must choose these values and say so.
• Candidates for the third round come from a classifier that reached 32% validation accuracy §2.2.3. Only images the classifier scored above 0.8 were sent to humans §2.2.3. The bootstrap is therefore not independent of the classifier's own errors. This is an inference from the described procedure.
• Crowd quality is only checked with control images §2.2.2. The number of workers, per-image agreement and the rate of wrong labels are not stated.
• The diversity score is relative. It is computed among the three datasets only §2.3.2. The averages (0.83 Places, 0.67 ImageNet, 0.50 SUN) must not be compared with other datasets. The paper gives 2/3 as the expected value when three datasets are identical §2.3.2.
• Single crop versus 10 crops changes the reported accuracy. The released code evaluates a single centre crop (see Paper vs code). If you want to match Table 2, use 10-crop averaging and say so.
• Top-1 is ill-defined for scenes, as the authors themselves note §3.1. Multi-label annotation is listed as future work §3.1.
• Places205 training schedule is not stated in the paper §3. The only Places365 schedule is in the code (see Hyperparameters).
• Figures are not tabulated. The SUN397 curve over training-set sizes fig. 11 and the diversity histogram fig. 7 are only in figures, so their values are not in this card.
• "State of the art" claims are from 2016 and are the authors' claims at that time §3.3.

Lineage

• Builds on: the SUN scene database for the category list §2.1 and the SUN397 and SUN Attribute benchmarks §3.3. The Places205 benchmark is from an earlier work that the paper cites as zhou2014learning §2.3.1. The paper gives no arXiv id for it.
• Compared against: ImageNet-trained CNNs with the same architectures §3.3, and hand-designed features on SUN397 fig. 11.
• Architectures used: AlexNet, GoogLeNet, VGG-16 and ResNet-152 §3.
• Superseded: not stated in this card's sources.