Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval

In plain words

You draw a rough sketch of a thing, such as a pear or a helicopter, and the system returns photos of that thing. This paper asks how to make that work for things the system was never trained on, and when the sketches are fast, loose doodles rather than careful drawings. A "zero-shot" setting means the test categories are disjoint from the training categories.

The authors first built a large dataset of quick doodles from a public drawing game, paired with a photo gallery, so that the sketch-to-photo gap is as wide as it is in real use. Their model has two networks, one for sketches and one for photos, that map both into one shared space where a sketch lands near photos of the same thing. Three training signals shape that space: one pulls matching pairs together and pushes mismatched ones apart, one pushes sketches and photos to look alike so they do not form two separate regions, and one ties each embedding to the word-vector meaning of its category. The authors report higher scores than earlier methods on existing benchmarks and on their new dataset.

TL;DR

Zero-shot sketch-based image retrieval (ZS-SBIR) ranks photos by distance to a query sketch, with no test category seen in training. The method trains two VGG-16 encoders with attention under a triplet ranking loss, a gradient-reversal domain loss and a semantic reconstruction loss against word2vec class vectors §4.2 §4.3. It also contributes QuickDraw-Extended, 330,000 sketches and 204,000 photos over 110 categories §3.1. Results are author-reported. The released code differs from the paper in several places (see Paper vs code), most importantly the domain-loss schedule.

Problem

Existing benchmarks use sketches that are either too photorealistic or drawn from a remembered reference photo, which narrows the sketch-photo gap §1 §3. Real queries are amateur, abstract drawings, and the gallery can be large. The paper argues that a practical ZS-SBIR system must handle both a large domain gap and large-scale retrieval §1 §3.1. The formal setup uses disjoint seen and unseen category sets for training and test, and assumes the model must embed sketches and photos into a space where unseen categories are still ranked correctly §4.1.

Method

The encoders are two CNNs with attention, one for sketches (ψ) and one for photos (φ). A shared distance function (L2) ranks photos for a sketch query. Training combines a triplet ranking loss, a domain-adversarial loss that uses a gradient reversal layer (GRL), and a semantic loss that tries to rebuild each category's word vector from its embedding. A negative photo also passes through a GRL in the semantic branch, so the model learns to separate similar categories §4.2 §4.3 fig. 2.

1. Setup. Photos \mathcal{X} and sketches \mathcal{Y} carry category labels; seen and unseen category sets are disjoint §4.1.
2. Encoders. Each encoder is an ImageNet pre-trained VGG-16 with attention §4.2 §5.1. The code applies the attention block to the last convolutional map encoder.py. The attention mask is computed with 1\times1 convolutions, and the block output is f + f \cdot att. The last fully connected layer is replaced to give a D-dimensional embedding §4.2. The paper does not state D.
3. Triplet loss. For an anchor sketch a, a positive photo p of the same category and a negative photo n of another category, the loss uses L2 distances and a margin \mu>0 §4.3 eq. 1.
4. Domain loss. A binary MLP predicts sketch (0) or photo (1) from the embedding. A GRL placed before it makes the encoders learn embeddings that the classifier cannot separate by domain. The GRL weight \lambda_d ramps up over training §4.3 eq. 2.
5. Semantic loss. A decoder maps each embedding to 300 dimensions and is scored against the category's word vector s with a half cosine distance. The word vectors are word2vec, with GloVe and fastText named as alternatives §4.3 eq. 3. The negative photo's embedding goes through a GRL with fixed weight \lambda_s = 0.5 §4.3.
6. Total objective. The three losses are summed with weights \alpha_1, \alpha_2, \alpha_3, which the paper says are equal, without giving the value eq. 4 §4.3.
7. Batches. Each sample is an anchor sketch, a positive photo and a negative photo. The negative category is chosen by semantic similarity, so close categories are more likely to be drawn §5.1 alg. 1.
8. Optimisation. SGD is used end-to-end; the learning rate is fixed at 1\times10^{-4} and training runs for about 40 epochs with early stopping on validation §5.1 alg. 1.

Key equations, as written in the paper:

\mathcal{L}_{t} = \frac{1}{N} \sum_{i=1}^{N} \max\{0,\ \mu + \delta_+^i - \delta_-^i\} eq. 1

\mathcal{L}_{d} = \frac{1}{3N} \sum_{i=1}^{N} \big( l_0(\psi(a_i)) + l_1(\phi(p_i)) + l_1(\phi(n_i)) \big) eq. 2

\mathcal{L}_{s} = \frac{1}{3N} \sum_{i=1}^{N} \big( l_c(\psi(a_i), s_i) + l_c(\phi(p_i), s_i) + l_c(R_{\lambda_s}(\phi(n_i)), s_i) \big) eq. 3

\mathcal{L} = \alpha_1 \mathcal{L}_{t} + \alpha_2 \mathcal{L}_{d} + \alpha_3 \mathcal{L}_{s} eq. 4

Here \delta_\pm are L2 distances from the anchor to the positive or negative embedding §4.3. The cosine term is l_c(e,s) = \tfrac{1}{2}\big(1 - \cos(g(e), s)\big) with g the semantic decoder §4.3. The GRL is the identity on the forward pass and multiplies the gradient by -\lambda on the backward pass §4.3. The domain-loss sign convention is discussed under Pitfalls.

The training loop is Algorithm 1 alg. 1: sample a mini-batch of anchor, positive and negative triples; compute \lambda_d; evaluate the full loss eq. 4; take one gradient step on all encoder and head parameters; stop at convergence or at a maximum iteration count T that the paper does not state alg. 1.

# Training step on one mini-batch  [alg. 1]
# Inputs: sketches, positive and negative photos, class vectors s (B, 300),
#         iteration index step

def training_step(sketches, positives, negatives, class_vectors, step):
    # grl(x, l): identity on forward, gradient times -l on backward  [§4.3]
    # Embed anchor sketches with psi and both photo sets with phi  [§4.2]
    anchor_emb = psi(sketches)                    # (B, D)
    positive_emb = phi(positives)                 # (B, D)
    negative_emb = phi(negatives)                 # (B, D)

    # Triplet ranking loss on L2 distances, margin mu (value not stated)
    # [eq. 1] [§4.3]
    d_positive = l2_norm(anchor_emb - positive_emb)
    d_negative = l2_norm(anchor_emb - negative_emb)
    loss_triplet = mean(max(0, mu + d_positive - d_negative))

    # Domain-GRL weight: 0 up to step 5, then (step - 5) / 20, capped at 1
    # [§4.3] [alg. 1]
    lambda_d = clip((step - 5) / 20, 0, 1)

    # Domain loss: sketches labelled 0, photos labelled 1  [eq. 2] [§4.3]
    loss_domain = (
        bce(domain_head(grl(anchor_emb, lambda_d)), 0)
        + bce(domain_head(grl(positive_emb, lambda_d)), 1)
        + bce(domain_head(grl(negative_emb, lambda_d)), 1)
    ) / 3

    # Semantic loss: rebuild each class vector from its embedding; the
    # negative photo passes a GRL with fixed lambda_s = 0.5  [eq. 3] [§4.3]
    loss_semantic = (
        half_cos_dist(decoder(anchor_emb), class_vectors)
        + half_cos_dist(decoder(positive_emb), class_vectors)
        + half_cos_dist(decoder(grl(negative_emb, 0.5)), class_vectors)
    ) / 3

    # Equal weights alpha_1 = alpha_2 = alpha_3; value not stated  [eq. 4]
    return loss_triplet + loss_domain + loss_semantic

# Retrieval at test time  [§4.2] [§5.1]
# Queries: sketches of unseen categories; gallery: photos of the same

def rank_gallery(sketches, photos):
    # Embed queries with psi and gallery photos with phi  [§4.2]
    query_emb = psi(sketches)                     # (Q, D)
    gallery_emb = phi(photos)                     # (G, D)
    # Rank photos by L2 distance to each query  [§4.2]
    distances = l2_cdist(query_emb, gallery_emb)  # (Q, G)
    # A photo is relevant if it shares the query's category; average
    # precision is then computed over the full ranking  [§5.1]
    return argsort(distances, axis=1)

Hyperparameters

Values in the paper are cited to the paper. Values marked "code" come from the released repository only, at the commit in the frontmatter; the paper does not state them.

Results

Author-reported. Setup §5.1: each encoder is an ImageNet-pre-trained VGG-16, implemented in PyTorch and trained on one Pascal Titan X. Benchmarks are Sketchy-Extended, TU-Berlin-Extended and the new QuickDraw-Extended, with class splits as in tab. 1. Metrics are mAP over the full gallery, plus mAP@200 and P@200 §5.1. Baselines are ZSIH [shen2018zero] and CVAE [yelamarthi2018zero], with the ZSIH split not reported by its authors tab. 2.

Dataset comparison tab. 1:

State-of-the-art comparison tab. 2 (ZSIH's Sketchy score uses a random 25-class test partition; under that partition the paper reports 0.3521 for its own model):

Ablation tab. 3. The triplet-only row is the baseline; rows add attention, the domain loss and the semantic loss:

Qualitative retrieval examples (top-8, correct in green, wrong in red) compare CVAE with the full model on Sketchy-Extended and QuickDraw-Extended fig. 4. The paper notes that confusions such as door with window are plausible even for humans fig. 4.

The text says a reduced model already beats previous results §1. The paper does not name that model; the triplet-only row of tab. 3 is the likely candidate. The TU-Berlin discussion says results fluctuate strongly across random splits, and that the paper reports the median over random splits §5.2, whereas the table caption says the same split was used for CVAE and the model tab. 2. The text also says QuickDraw-Extended's poor scores show the problem is still unsolved §5.2.

Paper vs code

Code: github.com/sounakdey/doodle2search at commit 07680c4 (read only, not executed). The owner is the first author, and the README links the authors' profiles. The project-page footnote in the abstract could not be opened (TLS certificate error). A code link in a commented-out line of the LaTeX source (results.tex, line 45) points to a repository that returns 404. This is the only official repository found.

Differences between the paper and the code, each with both citations:

• Domain GRL schedule. The paper ramps \lambda_d with the iteration index i §4.3 alg. 1. The training loop passes the batch index within the current epoch, so the schedule restarts every epoch train.py train.py. Inside the loss, that argument is named epoch and sets \lambda_d=0 for the first five batches and 1 from batch 26 on loss.py.
• Semantic-loss scale. The paper averages the three semantic terms with 1/(3N) eq. 3. The code adds the three per-sample terms and then takes the batch mean, which is three times the paper's value loss.py. The domain loss does divide by 3 loss.py.
• Semantic decoder depth. The paper says all heads have 3 FC layers §5.1. The semantic decoder has 2 linear layers loss.py. The domain classifier has 3 loss.py.
• Backbone. The paper says VGG-16 §5.1. The code uses vgg16_bn, the variant with batch normalisation encoder.py.
• Attention. The paper describes 1\times1 convolutions and f + f\cdot att §4.2. The code uses two 1\times1 convolutions with no nonlinearity between them and a softmax over the spatial positions of the 7×7 map encoder.py encoder.py encoder.py.
• Evaluation. The paper's tables report mAP@200 and P@200 tab. 2 tab. 3. The released test script computes only mAP over the full ranking; the top-200 metrics are commented out test.py test.py test.py. The similarity 1/(1+d) test.py gives the same ranking as d, so AP is unchanged.
• QuickDraw training classes. The paper says 80 training categories and 30 test categories §3.1 tab. 1. The code trains on 72 of the 80 non-test categories and holds out 8 for validation quickdraw_extended.py.
• Domain-loss sign. The paper defines l_t(e) as t\log f + (1-t)\log(1-f), which is the negative of binary cross-entropy §4.3. The code minimises standard BCE loss.py.
• Input normalisation. The paper gives no preprocessing §5.1. The code converts images with ToTensor only, without mean and standard-deviation normalisation, for both training and testing generator_train.py train.py test.py.
• Word vectors. The paper names word2vec, GloVe and fastText §4.3. The code builds vectors only from word2vec-google-news-300 through gensim class_word2vec.py. The GloVe and fastText arrays in the repository are not loaded by the Sketchy or QuickDraw loaders, as far as the loader files show quickdraw_extended.py sketchy_extended.py.
• Hard negatives. The paper says semantically closer categories are more likely to be drawn as negatives §5.1, without a formula. The code uses \exp(-d^2/0.1) over Euclidean distances between category vectors, excluding the anchor's category quickdraw_extended.py quickdraw_extended.py.
• Pretrained flag. The code flag --nopretrain uses store_false, so pre-training is on by default options.py train.py. The name is inverted.

Pitfalls

• Domain-loss schedule. Implement \lambda_d from the global step, as in the paper, not from the batch index within an epoch as the code does. The two give different adversarial curricula, and the paper's ablation is only interpretable with the paper's version.
• Semantic-loss weight. The code's semantic term is three times the paper's 1/(3N) scale. With the paper's equal \alpha, the code effectively gives the semantic loss more weight. Pick one convention and state it.
• Attention is off by default. --attn defaults to false, so running the script without it trains the variant without attention options.py. The paper's full model uses attention.
• Undocumented choices. The paper leaves the embedding size, batch size, margin, input size, dropout and the \alpha value unstated. The code fixes each one, so it is not an unambiguous reference for them either.
• Domain-loss sign. The paper's l_t has the sign of a log-likelihood. Minimising it as written would push the classifier the wrong way. Use BCE, as the code does loss.py.
• Scoring is not the paper's. Tables report mAP@200 and P@200, but the released code only produces full-gallery mAP. Any comparison in the tables needs an independent top-200 implementation.
• Gallery size. QuickDraw-Extended has about 166M sketch–photo comparisons in test tab. 1. The paper does not discuss the cost of full-ranking AP at that size or any approximation.
• Pre-processing and encoder mismatch. Pretrained VGG expects normalised inputs, but the code does not normalise generator_train.py. Scores may move if you add normalisation, so measure before and after.
• Split and selection. Model selection uses the 8 held-out training categories in the code, and the paper does not say how its validation split is built quickdraw_extended.py. Test categories are never used for selection train.py.
• Loader path quirk. The QuickDraw loader tests for ./data/semantic_labels_sketchy.npy but then loads ./data/semantic_labels_quickdraw.npy, which is not in the repository quickdraw_extended.py. I did not run it. Behaviour depends on the working directory, and an unrelated file can satisfy the check.
• TU-Berlin. The paper itself says its TU-Berlin numbers fluctuate across random splits, and that the ZSIH split is unknown, so that comparison is weak §5.2 tab. 2.
• Dataset and code licences. The dataset is linked from the README and the code has no LICENSE file. This card does not redistribute either. Check the dataset's terms before reuse.
• Commented-out results. The LaTeX source contains a commented-out table that lists dimension 64 for the full model (results.tex, line 315). It is not in the published paper and is not used here.

Lineage

• Builds on: the triplet ranking model for sketch retrieval, cited as commonly used in SBIR §1 (no card); gradient-reversal domain adaptation §4.3 (no card); soft attention §4.2 (no card).
• Compared against: CVAE [yelamarthi2018zero] and ZSIH [shen2018zero]. Neither has a card (no card, no arXiv id given in the source).
• Superseded by: not known from content already in this wiki.