Paper · 2020-07-07 · cs.CV
Re-thinking Co-Salient Object Detection
Co-salient object detection finds objects shared by a group of images. The paper argues that existing datasets assume look-alike objects and adds CoSOD3k (3,316 images, 160 groups). It benchmarks 18 methods and proposes CoEG-Net: EGNet saliency times an unsupervised co-attention map from the top principal component of group CNN features. Author-reported: E-measure 0.825 on CoSOD3k.
Salient object detectionSegmentation evaluation (scoring foreground, saliency and segmentation masks)Benchmark validityConditional random fieldco-saliencysaliency-detectiondatasetbenchmark
arXivPDFCodeMarkdown15 min read
AI-compiled by claude-haiku-5-5 on 2026-11-22, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Salient object detection asks which object in a picture a person would look at first. Co-salient object detection asks a harder question: given a set of pictures, which object appears in all of them and stands out in each? The authors found that the datasets used for this task mostly contain groups where the shared object looks the same every time, such as a row of near-identical football players. Real photo collections are messier. The shared thing may be a dog in one picture and a dog-walker in another, with different backgrounds.
The paper does three things. It builds a new, larger dataset, CoSOD3k, where the shared objects belong to the same semantic category but look different. It runs a fair comparison of many existing methods on it. And it proposes a simple baseline, CoEG-Net. That baseline takes an existing single-image saliency model and multiplies its output by a map that highlights what the group has in common. The map comes from finding the main direction of variation in the image features across the whole group.
The authors report that this baseline beats the other methods they tested on the new dataset. They also find that single-image saliency models do surprisingly well on these benchmarks.
TL;DR
- Co-salient object detection (CoSOD) finds the objects shared by a group of images. The paper argues that existing datasets assume look-alike objects, which hides how well methods work in the wild.
- It introduces CoSOD3k (3,316 images, 160 groups, hierarchical annotations) and benchmarks 18 methods on iCoSeg, CoSal2015 and CoSOD3k.
- CoEG-Net multiplies an EGNet saliency map by an unsupervised co-attention map. That map is the projection of each image's last-layer VGG16 features onto the first principal component of the group's features.
- Author-reported on CoSOD3k: E-measure 0.825, S-measure 0.762, F-measure 0.736, MAE 0.092. The best competitor on E-measure is AGC at 0.823 in tab. IV; the text names CSMG (0.804) as second best.
- The released code differs from the paper in several ways. Read Paper vs code before implementing from this card.
Problem
- Most CoSOD datasets pick groups whose shared object has a similar appearance. Models trained and tested on them can look better than they would on real collections where the shared object varies in texture, scene and background §I.
- The datasets are also small. CoSal2015 has 2,015 images in 50 groups and iCoSeg has 643 images in 38 groups. Most of them have only object-level labels tab. I §II-A.
- F-measure and MAE have known limits. Papers rarely state how they binarise saliency maps, so reported F-measures are hard to compare §I.
- Co-salient objects differ in size across images, but current CoSOD metrics are built from single-image SOD metrics. The paper argues they would likely favour large objects §VI.
Method
The paper has two parts: the benchmark (see Dataset and Results) and the baseline model, CoEG-Net §IV.
Architecture §IV.A fig. 9. For a group of N images, the model runs two branches and multiplies their outputs element-wise:
- Top branch: produces a per-image map A^n from the group's features (see below). It uses no labels.
- Bottom branch: a single-image saliency prior S^n from EGNet, trained on DUTS. It knows nothing about the other images §IV.A.
- Final prediction: A^n ⊗ S^n §IV.A.
Co-attention map §IV.B fig. 6. The paper starts from class activation mapping (CAM), which needs keyword labels. Since CoSOD has no such labels, it instead takes the direction of greatest variance among the group's local descriptors §IV.B:
- Take the last-convolution activations X^n of each image. These have K channels tab. III.
- Compute the group mean x̄ over every position of every image, and centre each descriptor §IV.B eq. 3.
- Build the K×K covariance matrix of the centred descriptors, normalised by Z = N×H×W eq. 3.
- Let ξ* be the eigenvector with the largest eigenvalue §IV.B.
- Project each centred descriptor onto ξ* to get the map A^n eq. 4.
The paper motivates this with weakly supervised localisation and PCA §IV.B. The text says the top eigenvector localises the common objects, and fig. 10 compares the top and second eigenvectors on one group §IV.B fig. 10.
Post-processing §IV.C. Before the product with S^n, the authors refine A^n with DenseCRF and manifold ranking. The paper does not state the parameters of either step.
# Co-attention map for one group [§IV.B] [eq. 3] [eq. 4]
# Input: feats[n] for n in 1..N, each (H, W, K) from the last conv layer
# Output: attention[n], each (H, W)
def co_attention_maps(feats):
# 1. Pool every position of every image into one descriptor set [§IV.B]
descriptors = concat(flatten(f) for f in feats) # (N*H*W, K)
# 2. Centre the descriptors with the group mean [eq. 3]
mean = descriptors.mean(axis=0) # (K,)
centred = descriptors - mean # (N*H*W, K)
# 3. Covariance over the group, normalised by Z = N*H*W [eq. 3]
cov = centred.T @ centred / len(centred) # (K, K)
# 4. Eigenvector of the largest eigenvalue; sign not fixed [§IV.B]
xi = leading_eigenvector(cov) # (K,)
# 5. Project each image's centred descriptors onto xi [eq. 4]
attention = [(f - mean) @ xi for f in feats] # each (H, W)
return attention# CoEG-Net for one group [§IV.A] [fig. 9]
# Input: images (list of N images)
# Output: co-saliency maps (list of N maps, each (H, W))
def coeg_net(images, backbone, egnet):
# 1. Top branch: last-layer features of each image [§IV.B] [§IV.C]
feats = [backbone(image) for image in images] # (H, W, K)
# 2. Unsupervised co-attention map for the group [eq. 4]
attention = co_attention_maps(feats) # per image (H, W)
# 3. Refine the attention map (DenseCRF, manifold ranking) [§IV.C]
attention = refine(attention) # params unstated
# 4. Bottom branch: single-image saliency prior from EGNet [§IV.A]
prior = [egnet(image) for image in images] # (H, W)
# 5. Element-wise product of the two branches [§IV.A] [fig. 9]
return [a * s for a, s in zip(attention, prior)]Dataset
CoSOD3k §III tab. I. 3,316 images in 160 groups (about 21 images per group). Images come from ILSVRC, which the paper says was collected from Flickr with scene-level queries §III.A. The groups belong to 13 super-classes, which contain 160 sub-classes. Each group is labelled with a category, a bounding box, an object-level mask and an instance-level mask §III fig. 2.
Labelling protocol §III.B.
- Category: each sub-class is taken from the ILSVRC categories, grouped under a middle level and then under 13 super-classes fig. 4.
- Bounding boxes: three viewers redraw the boxes around the objects that dominate their attention. The boxes are merged, and two senior CoSOD researchers double-check them. Images with more than six objects are discarded.
- Masks: 20 annotators, trained on 100 examples, label object-level and instance-level masks. These take about 8 and 15 minutes per image. Three volunteers cross-check the work fig. 5.
- Counts: the object-level labels number 3,316 and the instance-level labels 4,915. The final boxes are tightened using the instance masks.
Statistics §III.C fig. 8. The instance count ratio is 1 : 2 : ≥3 = 7 : 2 : 1. Instance sizes run from 0.02% to 86.5% of image pixels, with an average of 13.8%. The paper notes that the overall mask is centre-biased §III.C fig. 7.
Comparison with earlier datasets tab. I. The table lists MSRC (240 images), iCoSeg (643 images, 38 groups), Image Pair (210 images), CoSal2015 (2,015 images, 50 groups), WICOS (364 images, single-image groups) and CoSOD3k. Only CoSOD3k is marked as having instance-level labels and bounding boxes for each image tab. I.
Hyperparameters
Values marked "code only" are not in the paper. They come from the released repository, which may differ from what the paper's experiments used.
| Name | Value | Where | Notes |
|---|---|---|---|
| Backbone | VGG16 with the top layer removed | §IV.C | Paper gives no pretraining details. The released demo uses a CAM-trained VGG16 Caffe model demo.m |
| Training epochs | 30 | §IV.C | |
| Learning-rate schedule | Divided by 10 after epoch 15 | §IV.C | Initial learning rate not stated |
| Optimiser, batch size, weight decay, input size | not stated | - | |
| Saliency branch | EGNet settings and loss | §IV.C | Trained on DUTS §IV.A |
| CoEG-Net training set | DUTS, 10,553 images | tab. II | Listed as "S" supervision and CRF post-processing |
| Co-attention feature dimension K | Not stated | tab. III | Symbol defined, value not given |
| Spatial size H × W | Not stated | tab. III | |
| Eigenvector sign | Not stated | §IV.B | |
| Map normalisation before the product | Not stated | §IV.A | Code min-max normalises each map demo.m |
| DenseCRF parameters | Not stated | §IV.C | |
| Manifold-ranking parameters | Not stated | §IV.C | |
| F-measure β² and binarisation grid | Not stated | §V.A | Text describes both a maximum F and a best fixed threshold |
| Released CRF (confident labels) | Foreground threshold 0.30, background threshold 0.05 | code only | run_sample.py |
| Released CRF kernel | 10 iterations, gt_prob 0.7; Gaussian sxy 3 compat 3; bilateral sxy 50 srgb 5 compat 10 | code only | imutils.py |
| Released random walk | β = 10; 2^8 steps (exp_times = 8); background threshold 0.25 | code only | run_sample.py |
| Released IRN training | 4 epochs; learning rate 0.1; batch 12; weight decay 1e-4 | code only | run_sample.py |
Results
Author-reported. Setup: 18 CoSOD methods run with released code or author-supplied maps, on iCoSeg, CoSal2015 and CoSOD3k §V.A tab. IV. The four metrics are E-measure (E), S-measure (S), F-measure (F) and MAE (ε). The paper does not state the F-measure's β² or its threshold grid.
Benchmark, all methods tab. IV. "∘" means code or results were not available. A dash is printed in the table without a definition. The paper also underlines models trained on the dataset in question; those underlines are omitted here.
| Model | iCoSeg E | S | F | ε | CoSal2015 E | S | F | ε | CoSOD3k E | S | F | ε |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CBCS | .797 | .658 | .705 | .172 | .656 | .544 | .532 | .233 | .637 | .528 | .466 | .228 |
| ESMG | .784 | .728 | .685 | .157 | .640 | .552 | .476 | .247 | .635 | .532 | .418 | .239 |
| RFPR | .841 | .744 | .771 | .170 | ∘ | ∘ | ∘ | ∘ | ∘ | ∘ | ∘ | ∘ |
| CSHS | .841 | .750 | .765 | .179 | .685 | .592 | .564 | .313 | .656 | .563 | .484 | .309 |
| SACS | .817 | .752 | .770 | .154 | .749 | .694 | .650 | .194 | ∘ | ∘ | ∘ | ∘ |
| CODR | .889 | .815 | .823 | .114 | .749 | .689 | .634 | .204 | .700 | .630 | .530 | .229 |
| UMLF | .827 | .703 | .761 | .226 | .769 | .662 | .690 | .271 | .758 | .632 | .639 | .285 |
| DIM | .864 | .758 | .797 | .179 | .695 | .592 | .580 | .312 | .662 | .559 | .495 | .327 |
| CODW | .832 | .750 | .782 | .184 | .752 | .648 | .667 | .274 | ∘ | ∘ | ∘ | ∘ |
| MIL | .799 | .727 | .741 | .186 | .720 | .673 | .620 | .210 | ∘ | ∘ | ∘ | ∘ |
| IML | .895 | .832 | .846 | .104 | – | – | – | – | .773 | .720 | .652 | .164 |
| GONet | .864 | .820 | .832 | .122 | .805 | .751 | .740 | .160 | ∘ | ∘ | ∘ | ∘ |
| SP-MIL | .843 | .771 | .794 | .174 | ∘ | ∘ | ∘ | ∘ | ∘ | ∘ | ∘ | ∘ |
| CSMG | .889 | .821 | .850 | .106 | .842 | .774 | .784 | .130 | .804 | .711 | .709 | .157 |
| CPD | .900 | .861 | .855 | .057 | .841 | .814 | .782 | .098 | .791 | .757 | .699 | .120 |
| GSPA | .818 | .784 | .718 | .098 | .855 | .797 | .779 | .099 | .800 | .736 | .682 | .124 |
| AGC | .897 | .821 | .837 | .079 | .890 | .823 | .831 | .090 | .823 | .759 | .729 | .094 |
| EGNet | .911 | .875 | .875 | .060 | .843 | .818 | .786 | .099 | .793 | .762 | .702 | .119 |
| CoEG-Net (ours) | .912 | .875 | .876 | .060 | .882 | .836 | .832 | .077 | .825 | .762 | .736 | .092 |
Ablation: the co-attention map on three SOD backbones tab. VI. Each row pair shows the backbone and the same backbone with the co-attention projection added (Ours-A, Ours-P, Ours-E).
| Dataset | Metric | Amulet (SOD) | Amulet + co-attention | PiCANet (SOD) | PiCANet + co-attention | EGNet (SOD) | EGNet + co-attention |
|---|---|---|---|---|---|---|---|
| iCoSeg | E | .877 | .878 | .906 | .907 | .911 | .912 |
| iCoSeg | S | .828 | .829 | .869 | .870 | .875 | .875 |
| iCoSeg | F | .829 | .829 | .854 | .854 | .875 | .876 |
| iCoSeg | ε | .088 | .087 | .065 | .064 | .060 | .060 |
| CoSal2015 | E | .772 | .831 | .859 | .870 | .843 | .882 |
| CoSal2015 | S | .719 | .744 | .801 | .825 | .818 | .836 |
| CoSal2015 | F | .684 | .758 | .799 | .818 | .786 | .832 |
| CoSal2015 | ε | .147 | .125 | .090 | .084 | .099 | .077 |
| CoSOD3k | E | .752 | .803 | .780 | .819 | .793 | .825 |
| CoSOD3k | S | .685 | .692 | .750 | .758 | .762 | .762 |
| CoSOD3k | F | .629 | .700 | .682 | .724 | .702 | .736 |
| CoSOD3k | ε | .145 | .122 | .137 | .095 | .119 | .092 |
Per super-class E-measure on CoSOD3k tab. V. Only the ten methods with released code are included, plus the proposed model. The paper says the other methods were excluded for this table.
| Model | Veg-class | Food-class | Fruit-class | Tool-class | Necessity-class | Traffic-class | Cosmetic-class | Ball-class | Instrument-class | Kitchen-class | Electronic-class | Animal-class | Other-class | Whole set |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Sub-class count | 4 | 5 | 9 | 11 | 12 | 10 | 4 | 7 | 14 | 9 | 9 | 49 | 17 | 160 |
| ESMG | .577 | .635 | .735 | .625 | .546 | .673 | .633 | .559 | .655 | .631 | .629 | .687 | .592 | .635 |
| CBCS | .680 | .621 | .739 | .617 | .603 | .666 | .664 | .619 | .627 | .625 | .640 | .672 | .594 | .637 |
| CSHS | .613 | .591 | .733 | .677 | .585 | .691 | .677 | .563 | .637 | .651 | .665 | .715 | .624 | .656 |
| CODR | .682 | .682 | .774 | .679 | .634 | .756 | .678 | .580 | .671 | .686 | .695 | .771 | .638 | .700 |
| DIM | .622 | .687 | .773 | .650 | .604 | .708 | .633 | .577 | .665 | .612 | .641 | .709 | .623 | .662 |
| UMLF | .781 | .777 | .781 | .694 | .779 | .836 | .714 | .668 | .711 | .763 | .748 | .810 | .690 | .758 |
| IML | .802 | .725 | .808 | .740 | .714 | .867 | .753 | .653 | .734 | .795 | .729 | .855 | .663 | .773 |
| CPD | .805 | .763 | .818 | .734 | .758 | .894 | .763 | .629 | .638 | .848 | .784 | .892 | .693 | .791 |
| EGNet | .833 | .761 | .815 | .746 | .767 | .890 | .769 | .632 | .654 | .841 | .771 | .893 | .697 | .793 |
| CSMG | .755 | .872 | .854 | .722 | .744 | .908 | .766 | .778 | .690 | .849 | .840 | .885 | .690 | .804 |
| CoEG-Net (ours) | .802 | .842 | .840 | .811 | .790 | .897 | .795 | .780 | .746 | .844 | .842 | .881 | .739 | .825 |
Runtime tab. VII §V.E. The unit is listed as seconds, and the paper does not say whether this is per image. Traditional methods ran on a laptop CPU. Deep models ran on a workstation with an RTX 2080 Ti GPU. Languages differ across rows.
| Model | Time (s) | Language |
|---|---|---|
| CBCS | 0.3 | Matlab |
| ESMG | 1.2 | Matlab |
| CSHS | 102 | Matlab |
| CODR | 35 | Matlab |
| UMLF | 87 | Matlab |
| DIM | 25 | Matlab |
| CSMG | 3.2 | Caffe |
| CPD | 0.016 | PyTorch |
| EGNet | 0.034 | PyTorch |
| CoEG-Net (ours) | 2.3 | PyTorch |
Author observations.
- CoEG-Net has the best E, F and ε on CoSOD3k, and ties EGNet on S tab. IV §V.B. The text calls CSMG (0.804) the second-best method on E-measure, but tab. IV gives AGC 0.823.
- On CoSal2015, the text says CoEG-Net is best. The table shows AGC at E 0.890 against CoEG-Net at 0.882 tab. IV §V.B.
- The paper says SOD models such as EGNet score much lower on CoSOD datasets than on SOD datasets: EGNet has F 0.786 on CoSal2015 and 0.702 on CoSOD3k, against 0.937 and 0.943 on HKU-IS and ECSSD §VI.
Paper vs code
The official repository is https://github.com/DengPingFan/CoEGNet, owned by the first author. Its README gives the paper's citation. It contains a MATLAB stage 1, a Python stage 2 and a demo of four CoCA images at commit 143e45ad1485f198d70819c68665800bc665c290.
- Eigenvector and centring. The paper uses centred descriptors eq. 3 eq. 4. The code projects uncentred features returnCAMmap.m. The difference is one constant for the whole group (the projected group mean). The code then min-max normalises each map demo.m, which removes that constant, so the released map is unaffected.
- Crops. The paper does not mention crops. The code takes ten 224×224 crops prepare_image.m prepare_image.m and averages their features before the PCA demo.m. It then projects each crop's features separately demo.m.
- Backbone. The paper uses a generic VGG16 with the top layer removed §IV.C. The code uses a VGG16 with a CAM head (layers CAM_conv and CAM_pool) deploy_vgg16CAM.prototxt demo.m.
- Group definition. The code runs the PCA once per folder of images demo.m. The paper defines the group only as N associated images.
- Refinement. The paper's refinement is DenseCRF and manifold ranking §IV.C. The released stage 2 instead builds confident foreground and background labels with two CRF thresholds cam_to_ir_label.py and then runs an IRNet-style random walk make_sem_seg_labels.py run_sample.py. The IRNet stage is not described in the paper. The repository contains no manifold-ranking code.
- Final product. The paper multiplies a soft co-attention map A by the saliency prior S §IV.A. The released final output multiplies the random walk's label map by the EGNet map run_sample.py run_sample.py. The label map takes only the values 0 and 1, because the class keys are 0 and 1 make_cam.py make_sem_seg_labels.py make_sem_seg_labels.py. The output is therefore a hard mask times EGNet, not a soft product.
- Inputs. The paper computes CAM-style maps from the network. The released stage 2 does not compute them. It reads precomputed masks from
mask_rootmake_cam.py run_sample.py. Theresnet50_camnetwork named in the config is not called by the pipeline run_sample.py. - Saliency prior. The paper trains EGNet on DUTS and uses its output §IV.A. The repository has no EGNet training code and no weights. The demo reads precomputed EGNet maps run_sample.py.
- Training. The paper gives 30 epochs and a learning rate divided by 10 after epoch 15 §IV.C. The repository contains no training code for the co-attention network. The IRNet stage trains for 4 epochs at learning rate 0.1 run_sample.py. That is a different model from the one in the paper.
- Demo data. The defaults point to a four-image CoCA demo, not to CoSOD3k run_sample.py .
- Project page. The paper names http://dpfan.net/CoSOD3K/ as the project page. A fetch during compilation returned unrelated content, so the GitHub repositories are the only usable source.
Searches: the paper's GitHub links were checked first, then the owner's profile and a search on the method name. The paper's linked toolbox (https://github.com/DengPingFan/CoSODToolbox) was fetched with a web fetch during compilation, and it showed only a README and a LICENSE. Its contents were not cloned.
Pitfalls
- Text and tables disagree. The CoSal2015 text says CoEG-Net is best. The table gives AGC a higher E-measure (0.890 against 0.882). The CoSOD3k S-measure is a tie with EGNet (0.762), but only CoEG-Net is bolded. The text calls CSMG (0.804) second-best on CoSOD3k E-measure, while the table gives AGC 0.823.
- Competitor count. The text says 16 methods were compared (7 traditional and 9 deep) §V.A. The caption and abstract say 18, and tab. IV has 18 competitor columns. GSPA and AGC appear in the table, but §II-B calls them concurrent work.
- Ablation text. The text gives CoSal2015 S-measure gains of 2.5%, 1.4% and 1.8% for the three backbones §V.D. Table VI gives +2.5 (Amulet), +2.4 (PiCANet: 0.801 → 0.825) and +1.8 (EGNet). The middle figure does not match.
- Training data differs across competitors. Table II lists the training sets, and they vary a lot: some use MSCOCO, some use ImageNet pretraining, and others are listed as N/A. The table also lists CSMG's training count as "25,00", which looks like a typo. Comparisons are therefore not like-for-like.
- Metric definitions. The paper names maximum F-measure, but its text describes a best fixed threshold for the whole dataset §V.A. It states neither β² nor the threshold grid. E-measure, S-measure and MAE are cited, not written out.
- Eigenvector sign and scale. The paper does not say how the sign of ξ* is fixed or how A is scaled before the product. The released code min-max normalises each map but never fixes the sign demo.m demo.m. A sign flip would therefore invert the map. This follows from the code, not from a test.
- Hidden inference cost. The paper admits that CoEG-Net is a two-stage pipeline and costs more than one SOD model §VI. Its runtime is 2.3 s against 0.034 s for EGNet tab. VII.
- Runtime comparisons are not controlled. Languages, hardware and per-image timing differ across rows §V.E tab. VII.
- Metric bias. The paper says current CoSOD metrics favour large objects because they assume similar sizes across images §VI.
- Dataset rights. The images come from ILSVRC §III.A. The paper states no licence for the CoSOD3k images or annotations. Check before redistributing the dataset.
- Code licence. The top level of the repository has no LICENSE file. stage2/LICENSE is MIT, and its copyright belongs to Jiwoon Ahn, the author of IRNet. stage1/LICENSE is empty. Do not copy code from this repository without checking.
- Boundaries. The paper concedes that CoEG-Net's boundaries are coarse and that it misses fine shape detail §VI fig. 13.
Lineage
- Builds on: EGNet, the single-image SOD model used as the saliency prior §IV.A [/papers/1908.08297]. The co-attention idea draws on CAM §IV.B.
- Extends: the authors' CVPR 2020 preliminary version §I.
- Compared against: the 18 methods in tab. IV. Amulet and PiCANet serve as backbones for the ablation tab. VI.
- Superseded by: not stated in the paper.
Source and license
Adapted from “Re-thinking Co-Salient Object Detection” by Deng-Ping Fan, Tengpeng Li, Zheng Lin, Ge-Peng Ji, Dingwen Zhang, Ming-Ming Cheng, Huazhu Fu, Jianbing Shen (arXiv:2007.03380), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/DengPingFan/CoEGNet at commit 143e45ad1485 (none at top level; stage2/LICENSE is MIT (Jiwoon Ahn, IRN-derived files); stage1/LICENSE is empty). Any quoted code is shown under that license.