Weakly Supervised Object Localization and Detection: A Survey

In plain words

Teaching a computer to draw boxes around objects is much easier when the training data only says which objects appear in each picture, not where they are. This survey collects the many research attempts at that problem from the last two decades. It groups them into older statistical methods, methods that reuse features from deep networks trained on other tasks, and methods trained end to end with deep learning.

It also lists the public image collections people test on, the scores they report, and the areas where the method is used, such as video, medical images and satellite imagery. It ends with open problems the authors think matter. It is a map of the field, not a method you can run.

TL;DR

A 2021 survey of weakly supervised object localization (WSOL) and detection (WSOD), where training uses only image-level labels §1. It is useful for choosing a family of approaches, finding the benchmark and metric definitions, and locating open problems. It contains no new experiments and no algorithm to implement, and it has no official code.

Problem

Fully supervised detectors need boxes for every object. Image-level labels are cheaper, but the learner must work out the location itself §1. The survey names four failure modes the learner has to handle §1:

• inaccurate instance locations, because object parts and context are ambiguous;
• noisy samples, because a box contains background and similar categories;
• domain shift between the object samples found in training and the test data;
• too few instance samples, with long-tailed category distributions.

The survey treats WSOL and WSOD as one task. Both learn from the same image-level labels and usually output boxes. WSOL usually localizes one object per image, while WSOD detects every instance §1, footnote 1.

Method

This is a survey, so there is no method to reproduce. The survey frames the field as two stages §1:

1. Initialization uses prior knowledge to turn image-level labels into instance-level boxes. These boxes are noisy in label, sample coverage and location §1.
2. Refinement uses the boxes from stage 1 to learn a detector, then mines new instances from it and repeats. The survey does not give a stopping rule §1.

The taxonomy has three main categories and eight subcategories §2, fig. 2:

• Classic models (§3): initialization only (§3.1), refinement only (§3.2), or both (§3.3). These use hand-crafted descriptors such as HOG, SIFT and bag-of-words with models such as SVMs, DPMs, MIL and topic models.
• Off-the-shelf deep models (§4): pre-trained deep features (§4.1), inherent cues in deep models (§4.2), and fine-tuned deep models (§4.3). Non-deep components remain in these methods §4.
• Deep weakly supervised learning (§5): single-network training, where a deep network is trained end to end, often with MIL in the head (§5.1), and multi-network training, which combines several function-specific networks (§5.2) §5.3.

The survey says the three categories can be converted into one another, by replacing components with deep or non-deep ones §3, §4, §5. It says the field's history runs from a few classic approaches around 2002, through a slow period until 2009, to deep feature-based methods from 2014 onward. Recent work is mostly deep-only §2, fig. 3. The per-year counts are only in the image file, not in the text fig. 3.

Pseudocode of the two-stage framing from §1. This is a summary, not an algorithm from the paper:

# Two-stage weakly supervised detector, as framed by the survey  [§1]
# Input:  images with image-level labels only
# Output: a detector trained on instance boxes

def weakly_supervised_pipeline(images, image_labels):
    # 1. Initialization: turn image labels into instance boxes  [§1]
    #    The boxes carry label noise, sample bias and location error  [§1]
    initial_boxes = propose_instances(images, image_labels)
    # 2. Refinement: train, mine new instances, repeat  [§1]
    #    The survey gives no stopping rule and no box-merge rule  [§1]
    detector = train_detector(images, initial_boxes)
    detector = refine_with_mined_boxes(detector, images)
    return detector

Hyperparameters

Not applicable. The survey trains no model and reports no training settings of its own §2. Settings of the surveyed methods are not collected in the paper.

Results

No experiments of the authors' own. The survey reports benchmark statistics §6, tab. 9 and definitions of metrics §6. These numbers come from the dataset papers and the metric definitions, and are not results of this work.

Source: tab. 9 (tab:dataset) §6. Split and metric details are in §6:

• PASCAL VOC. Train on trainval: 5,011 (2007), 10,869 (2010), 11,540 (2012) images. Evaluate on the rest §6. WSOD is scored with mAP, where a detection counts if IoU with ground truth is above 0.5 §6. WSOL on VOC uses only trainval for both learning and localization, and is scored with CorLoc, where the top-scoring box per image must overlap ground truth by more than 50% §6.
• ILSVRC 2016. 1.2 M training images, 1,000 classes, validation set of 50,000 images used for testing §6.
• CUB-200-2011. 5,994 training and 5,794 test images §6.
• Localization metrics §6: GT Loc counts a result as correct when IoU with ground truth is at least 50%. Top-1 Loc also needs the top-scoring class to be the correct one. Top-5 Loc accepts the true class among the top five predictions. Top-1 Loc is the hardest of the three.

The survey describes VOC images as more cluttered, with several instances and scale variation, and ILSVRC images as mostly one object on a simpler background §6, fig. dataset.

Paper vs code

No official code. The paper's only project link is the authors' laboratory homepage in the author affiliation note [title block], which does not host code. Search attempted: a GitHub repository search for the title keywords returned no repositories. Survey papers are not code, so any awesome-list collection elsewhere was not used.

Pitfalls

• Not a method. An implementer cannot build "the survey's method". The card is for picking a family of approaches and reading its benchmarks.
• Challenge count. Section 1 says "five-fold challenges" §1, but the list of challenges has four items. The fifth item is commented out in the source, so the paper's own count and list disagree.
• Table entries are approximate. The tables mix general-object and category-specific methods. A method counts as general when it was tested on more than five categories tab. 1 to tab. 8 captions. "None" in the detector column marks WSOL-only methods tab. 1 to tab. 8 captions.
• Metric text vs table. The text says Top-1 and Top-5 Loc apply to both CUB and ILSVRC §6. The dataset table lists only GT Loc for ILSVRC tab. 9. Treat the table as the authors' summary and check the metric per paper.
• Boundary inequality. CorLoc is written as "more than 50%" overlap §6, GT Loc as "no less than 50%" IoU §6. Ties at exactly 50% are not treated consistently.
• Dataset role. VOC is used for WSOD with a train/test split, but for WSOL with trainval only §6. Scores from the two settings are not comparable.
• Counts not stated. The per-year publication counts fig. 3 are an image only, and the survey gives no numbers in text.
• Age. The arXiv record is v1, dated 2021-04-16. Work published after that date is outside this survey.
• Table hygiene. A row in the refinement table has no citation key (Rosenberg, BMVC 2002) tab. 2. Check references against the paper's bibliography before citing that row.

Lineage

• Builds on: not applicable (survey). It draws on the cited classic and deep WSOD papers, which are not compiled here.
• Compared against: not applicable.
• Superseded by: not stated in wiki content.