JAFAR: Jack up Any Feature at Any Resolution

In plain words

Modern image encoders turn a picture into a grid of feature vectors, but the grid is coarse. A 448-pixel image can end up as a 32 by 32 grid, so each cell covers a 14-pixel patch and object edges get lost. Stretching that grid with plain interpolation makes it bigger but blurry.

JAFAR is a small learned module that fixes the grid. It looks at the original photo as a guide. Each cell of the big output grid collects information from the coarse cells whose content matches it, so edges come back and the features follow the shapes in the picture. It is trained without any task labels, so the same output can feed segmentation, depth estimation or class activation maps.

Training needs no labels and no high-resolution targets. The same photo is shrunk, both versions go through the frozen encoder, and the module learns to rebuild the larger map from the smaller one. Its authors report that a module trained on small factors still works at much larger output sizes.

TL;DR

• JAFAR is a small cross-attention upsampler. It maps features from any frozen vision encoder to an arbitrary output grid, using the full image as high-resolution guidance §3.1.
• It is trained without annotations, on ImageNet. Each image is compared with a copy shrunk by a random factor between 2 and 4, with a cosine and L2 loss against the encoder's own high-resolution features §3.2 eq. 3.
• Author-reported: on frozen DINOv2 ViT-S/14 linear probes it has the highest mIoU on all four segmentation sets, e.g. 60.78 on COCO-Stuff, and the highest ADCC for Grad-CAM tab. 1 tab. 2.
• The official code (Apache-2.0) differs from the paper in the loss, the query and key construction, and the guidance resolution used at evaluation. See Paper vs code before implementing.

Problem

Foundation vision encoders downsample the image by 14 to 16 times, which leaves semantically rich but spatially coarse maps §1. Plain bilinear interpolation gives blurry output, and upsampling the input image costs a lot because self-attention is quadratic §1. Task-specific upsamplers need labels from the target task §1. LiFT is task-agnostic but fixed to 2x §1 §3.2. FeatUp's JBU variant over-smooths, and its implicit variant is trained per image §1. JAFAR aims for four properties: a task-agnostic objective, any output resolution, compatibility with any encoder, and low inference overhead §1.

Method

JAFAR takes a high-resolution guidance image I (3 x H x W) and the low-resolution feature map F_lr = f(I) from a frozen encoder f, of size C x h x w. It returns features at any requested size h_q x w_q §3.1. The output is a convex combination of the low-resolution features, weighted by attention between a query per output cell and a key per input cell §3.1 eq. 1.

Shared image encoding. The image goes through a light encoder E_theta that lifts it to d channels, then RoPE positional encoding is added §3.1. The paper does not give d numerically; the code uses 128 jafar.yaml.

Query branch. A small encoder processes the shared representation. Adaptive average pooling reduces it to the output grid h_q x w_q §3.1. The paper describes the pooling as training-only, so that inference keeps full output resolution; the code does not do this (see Paper vs code).

Key branch. A second encoder processes the shared representation, then average pooling brings it to the low-resolution grid h x w §3.1. The keys are then modulated by the encoder features with spatial feature transform (SFT): K = gamma * K_tilde + beta, with gamma and beta spatially varying and obtained from F_lr by linear projections §3.1 eq. 2.

Attention and upsampling. Attention weights are a scaled dot product of queries and keys, a softmax per head, averaged over heads after the softmax §3.1 eq. 1. The output is the matrix product of these weights with the raw low-resolution features. The paper uses no learned value projection, so the output keeps the encoder's feature content §3.1.

RoPE. Rotary positional embedding is applied to both queries and keys. The authors say this gives an inductive bias for spatial relations and removes the need to pick neighbouring keys by hand, as earlier similarity-based upsamplers do §3.1.

Training. The frozen encoder gives high-resolution targets F_hr from a 448-pixel input, so 32 x 32 for patch size 14 §4.1. A random factor delta in [2, 4] shrinks the image; the same encoder gives F_lr from the shrunken image §3.2. JAFAR sees the image (downsampled to 224 for training efficiency §4.1) and F_lr, and predicts F_hr. The loss is a cosine term plus an L2 distance, equation 3 §3.2 eq. 3. No high-resolution supervision is used §3.2.

Pseudocode of the paper's method. Code-only steps are marked and cited in Paper vs code:

# JAFAR upsampling  [§3.1] [eq. 1] [eq. 2]
# Input:  image (B, 3, H, W) guidance; low_feats (B, C, h, w)
# Input:  out_size = (h_q, w_q), the requested output grid
# Output: features (B, C, h_q, w_q)

def jafar(image, low_feats, out_size):
    # 1. Encode the guidance image, then add RoPE to every cell  [§3.1]
    image_repr = image_encoder(image)                  # (B, d, H, W)
    image_repr = rope(image_repr, grid_coords(H, W))   # (B, d, H, W)

    # 2. Queries: encode, then average-pool to the output grid  [§3.1]
    queries = query_encoder(image_repr)                # (B, d, H, W)
    queries = adaptive_avg_pool(queries, out_size)     # (B, d, h_q, w_q)

    # 3. Keys: encode, then average-pool to the low-res grid  [§3.1]
    keys = key_encoder(image_repr)                     # (B, d, H, W)
    keys = adaptive_avg_pool(keys, (h, w))             # (B, d, h, w)

    # 4. SFT: modulate keys with gamma and beta from low_feats  [§3.1] [eq. 2]
    keys = gamma(low_feats) * keys + beta(low_feats)   # (B, d, h, w)

    # 5. Softmax per head, then average the heads  [§3.1] [eq. 1]
    weights = head_averaged_attention(queries, keys)   # (B, h_q*w_q, h*w)

    # 6. Values are the raw low-res features, no projection  [§3.1]
    values = flatten(low_feats)                        # (B, h*w, C)
    out = weights @ values                             # (B, h_q*w_q, C)
    return reshape(out, (B, C, h_q, w_q))

# Training step  [§3.2] [§4.1] [eq. 3]
# Input:  images (B, 3, 448, 448) from ImageNet train
# Output: scalar loss

def training_step(images, encoder, jafar):
    # 1. High-resolution targets from the frozen encoder  [§3.2] [§4.1]
    hr_feats = encoder(images)                         # (B, C, 32, 32)
    # 2. Random factor delta in [2, 4]; low-res view of the same image  [§3.2]
    delta = sample_uniform(2, 4)
    small = resize(images, size_divided_by(delta))    # (B, 3, H/d, W/d)
    lr_feats = encoder(small)                         # (B, C, h/d, w/d)
    # 3. Guidance is the image downsampled to 224 px  [§4.1]
    guide = resize(images, 224)                        # (B, 3, 224, 224)
    # 4. Predict high-res features from guidance and low-res features  [§3.1]
    pred = jafar(guide, lr_feats, out_size=(h, w))     # (B, C, h, w)
    # 5. Paper form of the alignment loss  [eq. 3]; code differs
    return 1 - cos(pred, hr_feats) + l2(pred - hr_feats)

Hyperparameters

Results

Author-reported. The setting is a frozen DINOv2 ViT-S/14 encoder with 448-pixel inputs and 32 x 32 feature maps, linear probes, and upsamplers trained on ImageNet §4.1 §4.3. Task-agnostic baselines are pre-trained on the same backbone, and task-dependent ones are trained per task §4.3 app. A.2. "COCO" in Table 1 means COCO-Stuff with 27 classes §4.3.1.

Table 1. Linear probing with DINOv2 ViT-S/14 tab. 1

The "--" cells are empty in the source table tab. 1. The text reports an average gain of 1.63 mIoU over the next-best method across the four segmentation sets, 2.78 mIoU over FeatUp (4.8%), and a peak gain of 5.41 mIoU (9.7%) over FeatUp on Cityscapes. JAFAR is second on depth, behind ReSFU's 66.14 delta1 and 0.56 RMSE §4.3.1 §4.3.2.

Table 2. Grad-CAM faithfulness, ViT-B/16 tab. 2. Setup: 2,000 correctly predicted ImageNet validation images; metrics defined in app. A.1.

The text says JAFAR beats the second-best method (Strided, 65.3) by 8 ADCC points, a 12.5% relative gain §4.3.3.

Table 3. Zero-shot open-vocabulary segmentation, CLIP ViT-B/16 with MaskCLIP tab. 3

On ADE20K accuracy JAFAR ties FeatUp (33.28) tab. 3.

Table 4. Bird's-eye-view vehicle IoU on nuScenes, frozen DINOv2 backbone tab. 4 §4.3.5

Table 5. Key-strategy and head-count ablation, linear probing tab. 5 §4.4

Further results, all author-reported and summarised here with citations:

• Resolution generalisation (linear probing, mIoU, JAFAR trained at small ratios): from an 8 x 8 input to 896 x 896 output, JAFAR stays at 72.55 to 73.13 on VOC, while bilinear stays near 64.9. On Cityscapes it stays at 35.5 to 36.5 from 8 x 8 input, while bilinear stays near 31.7 app. D.1. At 32 x 32 input, JAFAR reaches 83.55 on VOC at 448 output app. D.1. The full table is in the source app. D.1.
• RoPE (linear probing, mIoU): removing RoPE lowers the 448-output VOC score from 83.55 to 67.22, and the Cityscapes score from 60.86 to 36.09 app. D.2.
• Runtime (ms, batch 1, A100): at 448 output, JAFAR takes 94.0 ms, FeatUp 64.9 ms, LoftUp 145.1 ms and LiFT 1.5 ms. At 224 output, JAFAR takes 16.6 ms and FeatUp 14.9 ms app. E. The full table is in the source app. E.
• Training memory (GB, backward pass, 448 output): JAFAR 26.0, LoftUp 26.5, FeatUp 7.4 app. E.
• Bird's-eye-view cost (GFLOPs): JAFAR adds 18.2 for SimpleBeV (594.00 to 612.24), 17.1 for PointBeV (154.30 to 171.41) and 24.3 for BeVFormer (352.40 to 376.74) app. E.
• Task-agnostic comparisons with official FeatUp and LiFT checkpoints, and with LoftUp, show JAFAR ahead on most settings, with a few exceptions, e.g. FeatUp at 448 on ADE20K app. C.1 app. C.2.

Paper vs code

The code is the official repository PaulCouairon/JAFAR, linked from the authors' project page, at commit 25321a2def9b00d06a8a8794ca0a799dcfd18d86, Apache-2.0. The compiled paper gives no code link. Everything marked "code" below cites this repository.

• Loss. Paper: cosine term plus the L2 distance of the difference eq. 3. Code: a cosine embedding loss (1 minus cosine, averaged over cells) plus a mean squared error of min-max-normalised features, where each cell is normalised by its own target minimum and maximum (+1e-6) loss.py. The MSE is squared and averaged, not an L2 norm, and the normalisation is missing from the paper.
• Query pooling. Paper: pooling is applied only during training, so inference keeps full output resolution §3.1. Code: the query grid is always set to output_size in upsample, with no training-mode check jafar.py. Training and inference use the same path.
• Query and key processing, not in the paper. Code adds a GroupNorm without affine parameters on queries jafar.py jafar.py; a 3x3 convolution without bias after the query branch attentions.py attentions.py; and an L2 normalisation of F_lr along channels before the key-modulation encoder jafar.py. The SFT step normalises the keys with GroupNorm before modulation other.py.
• Key modulation. Paper: gamma and beta come from linear projections of F_lr §3.1 eq. 2. Code: F_lr goes through a 1x1 convolution to d channels and two residual blocks, then gamma and beta are 1x1 convolutions jafar.py other.py.
• Attention form. Paper: softmax of QK^T over sqrt(d), averaged over heads eq. 1. Code: nn.MultiheadAttention with learned query and key projections (with PyTorch's default bias), scaled by the per-head dimension, returning head-averaged weights attentions.py. RMS normalisation is applied to queries and keys before projection attentions.py.
• Value projection. Paper: no learned value projection §3.1. Code: the output uses the raw features as values attentions.py jafar.py, which matches. The attention module also creates value and output projections whose results are discarded, so they never affect the loss attentions.py.
• Image encoder kernel. Paper: not stated. Config: kernel size 1 jafar.yaml. The code comment says "kernel_size=3 for spatial context" jafar.py, which contradicts the default.
• Guidance at evaluation. Paper: all experiments except Large Image use 448-pixel input images §4.3. Training used a guidance image of 224 §4.1 train.py. Probing passes the full 448 image as guidance train_probes.py.
• Probe settings. Paper: 5 epochs on COCO-Stuff, 20 on the other datasets, batch 4 §4.3.1. Code defaults: 20 epochs, training batch 2, AdamW at 5e-4 with cosine decay eval.yaml eval.yaml eval.yaml train_probes.py. The per-dataset epoch count is not visible in the code I read.
• Learning-rate schedule. Paper: lr 2e-4 with AdamW, no schedule stated §4.1. Code: constant learning rate, no scheduler, betas (0.9, 0.999), weight decay 1e-5 adamw.yaml train.py.
• Augmentation. add_blur_noise (Gaussian blur and noise) is defined but not called in the training loop train.py train.py. The paper mentions no augmentation.
• Downscale factor. Paper: factor delta in [2, 4] §3.2. Code: factor uniform in [0.25, 0.5] on the 448 image, rounded to a multiple of the patch size train.py. Grid sizes are therefore approximate.
• Feature layer. Paper: the encoder is frozen, and no layer is stated. Code: last block output with the final norm vit_wrapper.py.
• Repository checkpoints. The README lists checkpoints for several other backbones, including DINOv3 and RADIO README.md. The paper does not report them.

Pitfalls

• Guidance size mismatch. The guidance image is 224 px in training and 448 px in probing. Compare results only at the same guidance size, or test both.
• Loss scale. Implementers following equation 3 literally get a different loss from the code's cosine and min-max MSE. Choose one and say which in the reproduction notes.
• Query pooling is always on. The code pools queries to the requested output size in every call. An implementer who follows the paper and pools only in training has no defined query grid at inference, so use the code's behaviour.
• Convex output. Each row of the attention weights is averaged over heads after softmax, so it sums to one. Each output vector is therefore a convex combination of the low-resolution feature vectors. JAFAR cannot create values outside their range, and feature norms are not learned directly. This follows from the paper's formulas; the authors do not discuss it.
• Per-head scaling. Equation 1 writes sqrt(d). PyTorch scales by the per-head dimension, here 128/4 = 32 attentions.py. The paper does not define d.
• RoPE coordinates. Code builds coordinates by evenly spacing [0, 1] over the guidance-encoder grid, then applies RoPE before pooling img.py jafar.py. Pooling averages rotated vectors. The paper does not say RoPE is applied before pooling or how coordinates are scaled. RoPE theta is 100 in code positional_encoding.py.
• Cost grows with input resolution. Each query attends to every key, and the key count grows with the guidance grid. The authors say this becomes prohibitive for key grids of 64 x 64 or larger, and suggest localised attention as future work app. F app. E.
• Parameter count is inconsistent in the paper. The text gives 700K parameters app. C.2; the runtime table gives 0.7M app. E; an unreferenced parameter table gives 0.5M (file tables/parameters.tex, not included in the compiled text). Counting the unused value and output projections would change the total, so a reimplementer should count from their own model.
• GFLOPs claim does not match the table. The text says JAFAR needs 3 to 10% more GFLOPs than no upsampling app. E. For PointBeV the table gives 154.30 to 171.41, which is about 11% app. E.
• Baseline training differs. FeatUp is retrained on ImageNet for 50,000 steps, 25 times the schedule in its original paper app. A.2. Comparisons therefore depend on the authors' retraining choices.
• Fixed-seed training. The code sets seed 0 train.py. Results from other seeds are not reported.

Lineage

• Builds on: LiFT (Suri et al.), whose training objective is similar but whose architecture is convolutional and fixed to 2x §3.2 §1. RoFormer's rotary position embedding (Su et al.) for positional encoding §3.1. Spatial feature transform (Wang et al.) and SPADE (Park et al.) for feature modulation §3.1. The loss family follows Denoising ViT (Yang et al.) §3.2.
• Compared against: CARAFE, SAPA, DySample and ReSFU, task-dependent upsamplers; FeatUp and LiFT, task-agnostic upsamplers; LoftUp in app. C.2; and Large Image and Strided encoder baselines app. A.2.
• Superseded by: none known from content already in this wiki.