wikillm

Paper · 2021-03-30 · cs.CV

Locate then Segment: A Strong Pipeline for Referring Image Segmentation

Locate-Then-Segment (LTS) splits referring image segmentation in two. Fused image and sentence features give a heatmap from one language-conditioned 1x1 filter, and ASPP with a 2x deconvolution refines it into a mask. No DenseCRF. Author-reported IoU gains of about +3 over CGAN on RefCOCO+ and RefCOCOg val.

Referring segmentationDynamic convolutionCross-attentionMultimodal fusionsegmentationvision-language

arXivPDFMarkdown13 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Referring image segmentation means: given a photo and a sentence such as "the person on the left holding a cup", draw the outline of exactly the object the sentence names. Earlier systems mixed the picture and the words inside one tangled network and then cleaned up the rough result with a slow post-processing step.

This paper splits the job in two. First the system works out where the named object is, and stores that answer as a rough heatmap, bright where the object probably sits. Then a small second network uses the heatmap as a hint and draws the outline. It is like first pointing at the thing and then tracing it.

The heatmap makes the model easier to inspect, because you can see where it thinks the object is before it draws the mask. The authors present the design as a simple baseline that others can build on.

TL;DR

  • Decouples referring segmentation into "locate" (a heatmap of the referent) and "segment" (a mask refined from that heatmap) §1.
  • Locating uses one sentence-level kernel: a 1x1 filter built from the averaged bi-GRU sentence vector and applied at every location §3.2 eq. 8. A transformer decoder can replace it eq. 9.
  • Segmenting concatenates the fused features with the heatmap, applies ASPP and a 2x deconvolution, and outputs a mask at H/4 x W/4 §3.3 eq. 10.
  • Trained with sigmoid BCE on the mask plus 0.1 times BCE on the heatmap; no DenseCRF or other post-processing at inference §3.4.
  • Author-reported: highest IoU in seven of the eight benchmark splits in tab. 1; the exception is RefCOCO testA. The largest margins are on RefCOCO+ and RefCOCOg val tab. 1.

Problem

Referring segmentation needs two things at once: find the right region for the language, and produce a fine mask for the object §1. Earlier methods fuse the two modalities with recurrent or implicit interaction, and do not model where the referent is explicitly §1 §2.2. The authors also note that many of them depend on time-consuming DenseCRF post-processing and grow complex §1. This cost is not quantified in the paper §1.

Method

The model has four parts: visual and text feature extraction, a localization module with two interchangeable designs, a segmentation module, and a loss §3 fig. 2.

  1. Visual features. A visual backbone gives three levels, F_v1 at H/32, F_v2 at H/16 and F_v3 at H/8. Their channel widths d1, d2 and d3 are not stated §3.1. The backbone is Darknet53 in the experiments §4.1.

  2. Text features. Each word is embedded with 300-d GloVe vectors and read by a bi-directional GRU §3.1:

    \overrightarrow{h_{t}} = \overrightarrow{GRU}(x_{t}, \overrightarrow{h_{t-1}}), \quad h_{0} = 0 eq. 1

    \overleftarrow{h_{t}} = \overleftarrow{GRU}(x_{t}, \overleftarrow{h_{t+1}}), \quad h_{m+1} = 0 eq. 2

    The sentence vector is the average of the word states eq. 3, where each word state concatenates both directions eq. 4:

    f_{text} = avg(h_{1}, h_{2}, \dots, h_{m}) eq. 3

    h_{t} = concat(\overrightarrow{h_{t}}, \overleftarrow{h_{t}}) eq. 4

  3. Fusion. The text vector is multiplied with the coarsest visual level, and each finer level is merged with the upsampled previous fusion §3.1:

    f^{l}_{m_{1}} = g(f^{l}_{v_{1}} W_{v_{1}}) \cdot g(f_{text} W_{t}) eq. 5

    F'_{m_{i-1}} = UpSample(F_{m_{i-1}}) eq. 6

    F_{m_{i}} = concat(g(F'_{m_{i-1}} W_{m_{i-1}}), g(F_{v_{i}} W_{v_{i}})), \quad i \in \{2, 3\} eq. 7

    Here g is Leaky ReLU, and the upsampling stride is 2x2 §3.1. The result F_m3 is at H/8 x W/8 and is the input to localization and segmentation §3.1 §3.2.

  4. Localization, option A: relevance filter (the default). A kernel K = f_text W_k with K in R^{d_k} is reshaped to d_k x 1 x 1 and applied to F_m3. The result is a heatmap at H/8 x W/8, where higher values mean the location more likely matches the expression §3.2 eq. 8:

    H_{mask} = conv(K, F_{m_{3}}) eq. 8

    The filter dimension d_k is 1024 §4.1. The paper uses one filtering pass in the main results; a second pass is reported separately tab. 5.

  5. Localization, option B: transformer. The text vector is the encoder output, with no text encoder of its own. A standard transformer decoder takes F_m3, flattened to a sequence of d x HW/64, and produces the heatmap. Fixed positional encodings from [bello2019attention] are added to the input of each attention layer §3.2 eq. 9:

    H_{mask} = decoder(F_{m_{3}}, f_{text}) eq. 9

  6. Segmentation. The fused features and the heatmap are concatenated, passed through ASPP, and upsampled by 2x with a deconvolution. The mask logits are at H/4 x W/4 §3.3 fig. 3:

    P_{mask} = Seg(concat(F_{m_{3}}, H_{mask})) eq. 10

    Seg is ASPP, which probes its input with filters at several sampling rates and fields of view §3.3.

  7. Loss. Both terms are sigmoid binary cross-entropy. The segmentation term uses the down-sampled ground-truth mask at H/4 x W/4. The locating term uses the heatmap at H/8 x W/8 §3.4:

    L_{seg} = \sum_{l=1}^{\frac{H}{4} \times \frac{W}{4}} [y_{l}\log(p_{l}) + (1-y_{l})\log(1-p_{l})] eq. 11

    L_{loc} = \sum_{l=1}^{\frac{H}{8} \times \frac{W}{8}} [y_{l}\log(h_{l}) + (1-y_{l})\log(1-h_{l})] eq. 12

    L = L_{seg} + \lambda L_{loc}, \quad \lambda = 0.1 eq. 13

  8. Inference. The predicted mask is upsampled to the original H x W and binarized at 0.25. No other post-processing is used §3.4.

Pseudocode
# Locate-then-segment forward pass  [§3.1-§3.3]
# Input:  image (B, 3, 416, 416); token ids per expression (B, m)
# Output: mask logits (B, H/4, W/4); heatmap (B, H/8, W/8)

def lts_forward(image, token_ids):
    # Word vectors from GloVe, read by a bi-GRU  [§3.1] [eq. 1-2]
    words = glove_embedding(token_ids)                # (B, m, 300)
    states = bi_gru(words)                            # (B, m, 2h)
    # Sentence vector: average over the word states  [§3.1] [eq. 3-4]
    text = mean(states, axis=words)                   # (B, 2h)
    # Visual features at strides 32, 16 and 8  [§3.1]
    v1, v2, v3 = darknet53(image)
    # Fuse the coarsest level with the sentence  [§3.1] [eq. 5]
    m1 = leaky_relu(v1 @ W_v1) * leaky_relu(text @ W_t)
    # Upsample and merge the next level, twice  [§3.1] [eq. 6-7]
    m2 = concat(leaky_relu(upsample2(m1) @ W_m1), leaky_relu(v2 @ W_v2))
    m3 = concat(leaky_relu(upsample2(m2) @ W_m2), leaky_relu(v3 @ W_v3))
    # Sentence kernel, then a 1x1 filter over m3  [§3.2] [eq. 8]
    kernel = text @ W_k                               # (B, d_k)
    heatmap = conv1x1(kernel, m3)                     # (B, H/8, W/8)
    # Alternative to the filter: transformer decoder  [§3.2] [eq. 9]
    # heatmap = decoder(flatten(m3) + pos_enc, text)
    # Concatenate m3 with the heatmap, refine with ASPP  [§3.3] [eq. 10]
    refine_in = concat(m3, heatmap)
    features = aspp(refine_in)
    # Deconvolution x2 gives mask logits at H/4 x W/4  [§3.3]
    mask_logits = deconv_x2(features)                 # (B, H/4, W/4)
    return mask_logits, heatmap


def training_loss(mask_logits, heatmap, mask, lam=0.1):
    # Segmentation BCE against the mask at H/4 x W/4  [§3.4] [eq. 11]
    loss_seg = bce_sum(mask_logits, downsample(mask, 4))
    # Locating BCE against the mask at H/8 x W/8  [§3.4] [eq. 12]
    loss_loc = bce_sum(heatmap, downsample(mask, 8))
    # Weighted total, lam = 0.1  [§3.4] [eq. 13]
    return loss_seg + lam * loss_loc


def predict(mask_logits, height, width):
    # Upsample to the image size; sigmoid implied by the BCE  [§3.4]
    probs = sigmoid(upsample(mask_logits, height, width))
    # Binarize at 0.25; no DenseCRF or other post-processing  [§3.4]
    return probs > 0.25

Hyperparameters

NameValueWhereNotes
Visual backboneDarknet53, pre-trained on MSCOCO with val and test images of the three benchmarks removed§4.1pre-training schedule not stated
Input resolution416 x 416§4.1 tab. 6aspect-ratio handling not stated
Max sentence length15 (RefCOCO, RefCOCO+); 20 (RefCOCOg)§4.1padding and truncation not stated
Word embedding300-d GloVe, initialised from GloVe§3.1fine-tuning and GloVe version not stated
bi-GRU width1024§4.1per direction or total not stated
Sentence vectormean of word states§3.1 eq. 3padding in the mean not stated
Filter dimension d_k1024§4.1 eq. 8channel count of F_m3 not stated
Feature channels d1, d2, d3, dnot stated§3.1the paper names them but gives no values
Activation in fusionLeaky ReLU§3.1 eq. 5negative slope not stated
Upsampling in fusionstride 2 x 2§3.1 eq. 6upsampling operator not stated
Transformer decoder (option B)1 layer, 4 heads, 1024 hidden units§4.1feed-forward width, dropout and output projection not stated
Segmentation headASPP, then 2x deconvolution§3.3 fig. 3ASPP dilation rates not stated
Heatmap resolutionH/8 x W/8§3.2 eq. 8
Mask resolutionH/4 x W/4§3.3 eq. 10
OptimizerAdam§4.1betas, weight decay, epsilon not stated
Learning rate0.001, multiplied by 0.1 at epoch 30§4.1warm-up not stated
Batch size18§4.1number of GPUs not stated
Epochs45§4.1
Loss weight lambda0.1§3.4 eq. 13"empirically set"
Binarization threshold0.25§3.4applied after upsampling to H x W
Metric thresholdsprec@X for X in {0.5, 0.6, 0.7, 0.8, 0.9}§4.1
Augmentation, hardware, training timenot stated

Results

Author-reported. Setup: RefCOCO, RefCOCO+ and RefCOCOg (UNC split for RefCOCOg), IoU and prec@X, Darknet53 at 416 x 416, one sentence-level filter, no DenseCRF and no ASNLS post-processing §4.1 §4.2. Whether the baseline numbers were re-run or copied is not stated tab. 1.

Headline margins over the strongest other row in each column of tab. 1 (computed from the table, not stated by the paper):

SplitLTS (this paper)Best otherBest other methodDifference
RefCOCO val65.4364.86CGAN+0.57
RefCOCO testA67.7668.04CGAN-0.28
RefCOCO testB63.0862.07CGAN+1.01
RefCOCO+ val54.2151.03CGAN+3.18
RefCOCO+ testA58.3255.51CGAN+2.81
RefCOCO+ testB48.0244.69MCN+ASNLS+3.33
RefCOCOg val54.4051.01CGAN+3.39
RefCOCOg test54.2551.69CGAN+2.56

Full comparison, IoU, tab. 1. "-" means the original paper does not report the value. DCRF and ASNLS are post-processing steps. MCN+ASNLS marks ASNLS in its name, not in the DCRF column.

MethodBackboneDCRFRefCOCO valtestAtestBRefCOCO+ valtestAtestBRefCOCOg valtest
RMIResNet101yes45.1845.6945.5729.8630.4829.50--
DMNResNet101no49.7854.8345.1338.8844.2232.29--
RRNResNet101yes55.3357.2653.9539.7542.1536.11--
MAttNetMRCN-ResNet101no56.5162.3751.7046.6752.3940.0847.6448.61
NMTreeMRCN-ResNet101no56.5963.0252.0647.4053.0141.5646.5947.88
CMSAResNet101yes58.3260.6155.0943.7647.6037.89--
Lang2segResNet101no58.9061.7753.81---46.3746.95
BCAMResNet101yes61.3563.3759.5748.5752.8742.13--
CMPCResNet101yes61.3664.5359.6449.5653.4443.23--
MCN+ASNLSDarkNet53no62.4464.2059.7150.6254.9944.6949.2249.40
LSCMResNet101yes61.4764.9959.5549.3453.1243.50--
CGANDarkNet53no64.8668.0462.0751.0355.5144.0651.0151.69
LTS (this paper)DarkNet53no65.4367.7663.0854.2158.3248.0254.4054.25

Ablation on RefCOCO tab. 2. Check marks show which parts are on. Filter means the relevance filter, Seg the segmentation module, and L_loc the locating loss. Values are IoU and prec@0.5 through prec@0.9.

SplitFusionFilterSegL_locIoU@0.5@0.6@0.7@0.8@0.9
val✓55.0860.5851.2641.5527.156.76
val✓54.5360.4350.0039.0324.955.85
val✓✓56.9463.4954.2044.1329.438.36
val✓✓63.5072.8465.8557.6142.3313.41
val✓✓63.6472.9466.5258.0142.8413.20
val✓✓✓65.0575.0168.4860.6944.9314.03
val✓✓✓✓65.4375.1669.5160.7445.1714.41
testA✓56.8262.4953.6042.8728.276.65
testA✓56.0561.6651.7640.3225.615.67
testA✓✓58.5364.5255.6545.5930.517.94
testA✓✓65.3175.3269.4560.4644.7611.38
testA✓✓66.4177.0071.2762.6346.6512.82
testA✓✓✓67.4978.1072.8764.4748.3113.24
testA✓✓✓✓67.7678.4773.1364.5647.9812.92
testB✓53.5259.2049.6439.2026.228.58
testB✓52.5556.4546.9137.4524.108.03
testB✓✓55.1260.6151.2141.3229.0710.64
testB✓✓60.9769.1362.2453.3939.6515.68
testB✓✓60.7268.0361.1452.6039.5916.68
testB✓✓✓62.4370.9964.6555.2742.4117.48
testB✓✓✓✓63.0871.8264.5955.7442.7917.35

Segmentation module added to CMPC, RefCOCO IoU tab. 3: CMPC 61.36 / 64.53 / 59.64 becomes CMPC+Seg 62.75 / 65.34 / 61.08 (val / testA / testB).

Transformer instead of filter, IoU on val tab. 4: LTS 65.43 / 54.21 / 54.40 (RefCOCO / RefCOCO+ / RefCOCOg) against LTS-Trans 66.15 / 54.52 / 54.51.

Filter count and word-level filtering, RefCOCO IoU tab. 5. n is the number of filtering passes:

ModelnvaltestAtestB
LTS165.4367.7663.08
LTS266.0468.6863.27
LTS365.5467.8262.97
LTS-WordFilter (one kernel per word, as DMN)164.9267.3162.50

Input resolution, RefCOCO IoU tab. 6:

ResolutionvaltestAtestB
320 x 32063.0165.4060.78
352 x 35264.0466.2461.76
384 x 38464.4567.0262.47
416 x 416 (used elsewhere)65.4367.7663.08
448 x 44865.7167.9063.23
480 x 48065.9068.1663.45

Combined settings, val IoU tab. 7: LTS 65.43 / 54.21 / 54.40 against LTS* 66.75 / 54.94 / 54.51 (RefCOCO / RefCOCO+ / RefCOCOg). LTS* uses the transformer, more filtering and a larger input. Its exact values are not stated §4.4.

Localization quality: the maximum of the heatmap falls inside the ground-truth mask in 81.74% of cases. The split on which this was measured is not stated §4.3.

Qualitative: heatmaps and final masks are shown in fig. 4; the comparison with MCN is in fig. 5. The paper describes them in words only, and this card does not reproduce them §4.2 §4.3.

Paper vs code

No official code.

Searched: the paper text and LaTeX source (no code links); the arXiv abstract page (comment field "CVPR 2021", no code or project link); GitHub repository search with the queries "locate-then-segment", "referring segmentation LTS", "referring segmentation darknet" and "referring segmentation". None of the results belongs to the authors (for example MCN, CMPC-Refseg and CMSA-Net are other groups' code and were not used).

Pitfalls

  • Headline wording versus table. The text says the method achieves the best IoU across all datasets §4.2. On RefCOCO testA, CGAN is higher (68.04 against 67.76) tab. 1. The "+3.2 %" and "+3.4 %" headlines are on the val splits of RefCOCO+ and RefCOCOg. On RefCOCOg test, the gain is +2.56 tab. 1.
  • Post-processing in the comparison. Some baselines use DenseCRF and ASNLS tab. 1. The paper's own comparison is therefore not all like-for-like. Which numbers were re-run is not stated.
  • Eq. 5 operator. The "·" between two vectors is a scalar product if read literally, which cannot give a spatial map F_m1. The likely reading is an element-wise product broadcast over positions, but the paper does not say. Check shapes before implementing.
  • Channel widths. d1, d2, d3, d and the channel count of F_m3 are not stated. Eq. 8 implies F_m3 has d_k = 1024 channels, and the W matrix shapes follow from those widths. The paper does not write them out.
  • Bi-GRU width. "1024 dimensional bi-GRU" does not say whether 1024 is per direction. Eq. 4 concatenates both directions, so the per-word width doubles.
  • Sign of the loss. Eqs. 11 and 12 are written as sums of log-likelihood terms, which are the negative of BCE. Implement them as minimised BCE. Whether the sums are normalised (sum or mean) is not stated, and the weight 0.1 in eq. 13 depends on that choice.
  • Ground-truth down-sampling. The method for down-sampling the mask to H/4 and H/8 is not stated. Nearest-neighbour and area averaging give different targets.
  • Sigmoid at inference. The text says sigmoid BCE is used, but eqs. 8 to 10 do not show a sigmoid, and the inference step does not state whether the 0.25 threshold applies to sigmoid probabilities. The pseudocode assumes it does.
  • Upsampling operator. Upsampling in fusion (eq. 6) and output upsampling to H x W are both not stated.
  • Transformer option. The output projection that turns decoder states into one heatmap channel is not stated. Fixed positional encodings are cited to another paper and their formula is not given here. Dropout, feed-forward width and layer norm placement are not stated.
  • Commented source text. The arXiv LaTeX source contains a commented-out remark on a 512-wide feed-forward layer and commented-out tables. These are not part of the compiled paper and are not used in this card.
  • LTS is underspecified.* The text says LTS* uses more filtering and a larger input, but does not state the pass count or the resolution §4.4 tab. 7. Table 5 shows n = 2 gives the best val score, while the main results use n = 1 tab. 5.
  • Speed claim. The paper says DenseCRF and similar steps are time-consuming, but gives no runtime, parameter count or FLOPs §1. The benefit of dropping post-processing is therefore not measured.
  • Pre-training overlap. The backbone is pre-trained on MSCOCO after removing val and test images of the three benchmarks §4.1. Whether training-split images are kept is implied, not stated.
  • IoU definition. The paper does not say whether IoU is averaged per image or computed on pooled intersections and unions §4.1. Prec@X is described as "higher than" the threshold, so the comparison is strict §4.1.
  • Dataset statistics. RefCOCO: 19,994 images, 142,210 expressions, 50,000 objects, splits 120,624 / 10,834 / 5,657 / 5,095. RefCOCO+: 19,992 images, 141,564 expressions, 49,856 objects, splits 120,191 / 10,758 / 5,726 / 4,889. RefCOCOg (UNC): 26,711 images, 104,560 expressions, 54,822 objects §4.1.

Lineage

  • Builds on: the backbone and training setup follow MCN §4.1 (Luo et al., CVPR 2020; no arXiv id in the source). No card in this wiki.
  • Related filtering methods: DMN §2.2 §4.2 and Lang2seg §4.2, which the paper says also filter with the language; neither has a card in this wiki.
  • Compared against: MAttNet, NMTree, CMSA, Lang2seg, BCAM, CMPC, MCN+ASNLS, LSCM and CGAN tab. 1 (none has a card in this wiki).
  • Superseded by: not stated in this wiki.

Source and license

Adapted from “Locate then Segment: A Strong Pipeline for Referring Image Segmentation” by Ya Jing, Tao Kong, Wei Wang, Liang Wang, Lei Li, Tieniu Tan (arXiv:2103.16284), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.