wikillm

Paper · 2022-12-27 · cs.CV

DAE-Former: Dual Attention-guided Efficient Transformer for Medical Image Segmentation

A U-Net-shaped pure transformer for medical segmentation. Each block pairs linear-cost spatial (efficient) attention with channel (transpose) attention, and skip paths fuse encoder and decoder features by cross-attention. On Synapse multi-organ CT it reports the highest Dice of the compared methods, 82.63, without pretraining (author-reported).

Linear attentionChannel attentionCross-attentionDice lossmedical-image-segmentationtransformerefficient-attentionu-net

arXivPDFCodeMarkdown16 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Medical segmentation means colouring every pixel of a scan by the organ or lesion it belongs to. The network that does this well so far is usually a convolutional U-Net. Transformers see long-range context better, but their standard attention gets expensive as the image grows, so many designs restrict attention to small windows or shrink the image first.

This paper keeps attention global and makes it cheap in a different way. Each block looks at the image in two passes: one pass compares positions across the whole image, and a second pass compares feature channels with each other, so the block sees both where things are and what kinds of features matter. The skip connections, which carry fine detail from the encoder to the decoder, are not simply joined to the decoder features. Instead, one set of features asks the other set what it needs, using cross-attention.

The result is a U-shaped transformer that is meant to segment organs in CT scans without pretraining. The reported gains are on one multi-organ CT benchmark and one skin lesion benchmark, so treat them as evidence for those settings only.

TL;DR

  • DAE-Former is a U-Net-shaped pure transformer with three encoder and three decoder levels §2.
  • Each encoder block has a linear-cost spatial attention followed by a channel attention, both with feed-forward layers §2.1–§2.3.
  • Skip connections fuse encoder and decoder features with a cross-attention module (SCCA) §2.4.
  • Author-reported: 82.63 mean Dice on Synapse (8 abdominal organs), the highest Dice among the methods in Table 1, with 48.01 M parameters. HiFormer has a lower HD tab. 1.
  • The official code covers the Synapse experiments only; the skin lesion results have no code here README.md.

Problem

Standard self-attention has quadratic cost in the number of tokens, which limits high-resolution images §2.1. Local-window or reduced-resolution transformers cut that cost but capture global context only partially §1. Pure CNN U-Nets lack long-range context, and hybrid CNN-transformer models keep a heavy convolutional backbone §1. The paper argues that a medical segmenter needs both spatial and channel context across the full feature map, at a cost that stays manageable §1, §2.

Assumptions: the paper trains from scratch, without pretrained weights §1; the experiments use a 224×224 input app. B.

Method

The network is a hierarchical encoder-decoder built from one block type, the efficient dual attention block §2, fig. 1. The encoder has three stages. Each stage applies two dual blocks and then a patch merging step that halves the token grid §2. The decoder has three levels that expand the tokens by a factor of 2 each time. The two upper decoder levels fuse their input with the matching encoder output through the SCCA module, and then run two dual blocks §2, §2.4. A final linear projection gives the per-pixel class map §2.

Efficient attention §2.1. Spatial attention is computed in the order "keys and values first, then queries", which removes the N×N attention map. Equation (2) gives the form with normalisation functions ρ on queries and keys. The paper states that softmax normalisation makes it equal to dot-product attention eq. 1, eq. 2.

Transpose attention §2.2. Channel attention is computed from a d×d cross-covariance matrix. Queries and keys are L2-normalised and divided by a temperature τ before the softmax eq. 3.

Efficient dual attention block §2.3. The block chains efficient attention, a feed-forward layer, transpose attention and a second feed-forward layer, each with a residual connection eq. 4–eq. 8, fig. 2. The feed-forward layer is a Mix-FFN, which adds a depthwise convolution inside the MLP eq. 9.

SCCA §2.4, fig. 1b. The decoder features are projected to the skip width. Queries come from the skip features; keys and values come from the projected decoder features. The attention is the efficient form, so its cost stays linear. The output width is not stated in the compiled text.

Equations, as written in the paper (normalisation functions and τ are left unspecified beyond the notes in each section):

S(Q,K,V) = \operatorname{softmax}\!\left(\frac{QK^{T}}{\sqrt{d_k}}\right)V eq. 1

E(Q,K,V) = \rho_q(Q)\,\big(\rho_k(K)^{T} V\big) eq. 2

T(Q,K,V) = V\,C_T(K,Q), \qquad C_T(K,Q) = \operatorname{Softmax}\!\big(K^{T}Q/\tau\big) eq. 3

E_{block} = E(Q_1,K_1,V_1) + X eq. 4

MLP_1(E_{block}) = MLP\big(LN(E_{block})\big) eq. 5

T_{block} = T\big(MLP_1(E_{block}) + E_{block},\,Q_2,K_2,V_2\big) + MLP_1(E_{block}) eq. 6

MLP_2(T_{block}) = MLP\big(LN(T_{block})\big) eq. 7

\operatorname{DualAttention}(T_{block}) = MLP_2(T_{block}) + T_{block} eq. 8

MLP(X) = FC\big(GELU(\text{DW-Conv}(FC(X)))\big) eq. 9

Pseudocode follows the equations. Shapes assume tokens of width d and N tokens. The width d, the head counts and the normalisation axes are not given in the paper's text; the values the code uses are in Hyperparameters and in Paper vs code.

Pseudocode
# Efficient attention, linear in N  [§2.1] [eq. 2]
# Keys normalised over tokens, queries over channels
# [code: networks/DAEFormer.py#L123-L125]
def efficient_attention(x):                  # x: (B, N, d)
    queries = softmax_over_channels(x @ W_q)  # (B, N, d_k)
    keys = softmax_over_tokens(x @ W_k)       # (B, N, d_k)
    values = x @ W_v                          # (B, N, d_v)
    # Global context first: (d_k, d_v); no N x N matrix  [§2.1]
    context = keys.transpose(1, 2) @ values
    return queries @ context                  # (B, N, d_v)
Pseudocode
# Transpose (channel) attention  [§2.2] [eq. 3]
# Queries and keys L2-normalised before the cross-covariance  [§2.2]
def transpose_attention(x, tau):             # x: (B, N, d); tau: temperature
    queries = l2_normalize(x @ W_q, dim=1)   # (B, N, d)
    keys = l2_normalize(x @ W_k, dim=1)      # (B, N, d)
    values = x @ W_v                         # (B, N, d)
    # d x d cross-covariance, softmax over channels  [eq. 3]
    weights = softmax(keys.transpose(1, 2) @ queries / tau)
    return values @ weights                  # (B, N, d)
Pseudocode
# Mix-FFN  [eq. 9] [§2.3]
def mix_ffn(x):                              # x: (B, N, d)
    # Expand the width, then mix neighbouring tokens  [eq. 9]
    hidden = fc1(x)                          # (B, N, 4d); 4x is the code value
    hidden = gelu(depthwise_conv3x3(hidden)) # spatial mixing  [eq. 9]
    return fc2(hidden)                       # (B, N, d)
Pseudocode
# Efficient dual attention block  [§2.3] [eq. 4] to [eq. 8] [fig. 2]
def dual_block(x):                           # x: (B, N, d)
    # Spatial attention with residual  [eq. 4]
    e_block = efficient_attention(x) + x
    # First feed-forward layer on the layer-normed output  [eq. 5]
    mlp_1 = mix_ffn(layer_norm(e_block))
    # Channel attention on the sum, plus a residual  [eq. 6]
    t_block = transpose_attention(mlp_1 + e_block) + mlp_1
    # Second feed-forward layer and residual  [eq. 7] [eq. 8]
    return mix_ffn(layer_norm(t_block)) + t_block
Pseudocode
# Skip connection cross attention  [§2.4]
# Queries from the skip; keys and values from the decoder  [§2.4]
def scca(skip, decoder):                     # (B, N, d2), (B, N, d1)
    # Project the decoder features to the skip width  [§2.4]
    projected = linear(decoder)              # (B, N, d2)
    queries = project_q(skip)
    keys = project_k(projected)
    values = project_v(projected)
    # Efficient attention on the three inputs; output is 2 * d2 wide  [§2.4]
    return efficient_context(queries, keys, values)
Pseudocode
# Full forward pass  [§2] [fig. 1a]
# Three encoder stages; decoder fuses two skips  [§2] [§2.4]
def dae_former(image):                       # (B, 1 or 3, 224, 224)
    # Overlapping patch tokens  [§2]
    tokens = patch_embed(image)
    skips = []
    for stage in range(3):
        # Two dual blocks per stage  [§2]
        for layer in range(2):
            tokens = dual_block(tokens)
        # Stage output is the skip for the matching decoder level  [§2]
        skips.append(tokens)
        # Halve the token grid and widen the channels  [§2]
        tokens = patch_merge(tokens)
    # Lowest level: two dual blocks, then expand  [§2] [code: DAEFormer.py#L479]
    x = expand_2x(dual_blocks(tokens, count=2))
    # Middle level: fuse the skip, two dual blocks, expand  [§2] [§2.4]
    x = scca(skips[1], x)
    x = expand_2x(dual_blocks(x, count=2))
    # Top level: fuse the skip, two dual blocks, expand 4x  [§2] [§2.4]
    x = scca(skips[0], x)
    x = expand_4x(dual_blocks(x, count=2))   # code value, see Paper vs code
    return conv1x1(x, classes=9)             # (B, 9, 224, 224)

Training objective and schedule:

Pseudocode
# One training step  [§3] [code: trainer.py#L132-L142]
def train_step(model, image, label, step, total_steps):
    logits = model(image)                    # (B, 9, H, W)
    # Weighted sum of cross-entropy and Dice loss  [§3]
    loss = 0.4 * cross_entropy(logits, label)
    loss = loss + 0.6 * dice_loss(logits, label)
    # SGD, momentum 0.9, weight decay 1e-4, base lr 0.05  [§3]
    sgd_update(loss)
    # Poly decay per iteration; the paper states no schedule
    # [code: trainer.py#L140-L142]
    set_lr(0.05 * (1 - step / total_steps) ** 0.9)
    return loss

Hyperparameters

Values in the paper come with the section where they appear. Where the paper is silent, the table says so and gives the code value separately.

NameValueWhereNotes
OptimiserSGD§3
Base learning rate0.05§3 app. ACode scales it for batch sizes divisible by 5 other than 24 train.py
Momentum0.9§3
Weight decay0.0001§3
Batch size24§3README example command uses 20 README.md
Epochs400§3Code also has a max-iterations argument that is not used by the trainer train.py
Loss0.6 × Dice + 0.4 × cross-entropy§3Dice formula not given in the compiled text
LR schedulenot stated—Code: poly decay, power 0.9, per iteration trainer.py
Warm-upnot stated—Not in code either
Input size224 × 224app. BCode default is 224 train.py
Input normalisationnot stated—Code: mean 0.5, std 0.5 trainer.py
Input channelsnot stated—Code repeats a 1-channel input to 3 DAEFormer.py
Encoder stages3, each with 2 dual blocks§2Then patch merging
Patch tokens (stage 1)"overlapping 4 × 4" tokens§2Code kernel 7, stride 4, padding 3 DAEFormer.py
Patch merging2 × 2 tokens merged, channels doubled§2Code uses overlapping 3 × 3 stride-2 convs; channels 128 → 320 → 512, not doubling DAEFormer.py
Decoder expansion2× per block§2Code: final 4× expansion then 1×1 conv to 9 classes DAEFormer.py
Channel width dnot stated—Code: 128 / 320 / 512 per stage DAEFormer.py
Key and value widthsnot stated—Complexity remark assumes d_k = d/2 §2.1; code uses d_k = d DAEFormer.py
Heads, spatial attentionnot stated—Code: 1 head DAEFormer.py
Heads, channel attentionnot stated—Code: 8 heads by default DAEFormer.py
Temperature τnot stated—Code: learnable per head, initial value 1 DAEFormer.py
FFN expansionnot stated—Code: 4× DAEFormer.py
Dice smoothingnot stated—Code: 1e-5 utils.py
Seednot stated—Code default 1234 train.py
Hardwaresingle RTX 3090 GPU§3
FrameworkPyTorch§3
Augmentationnot stated—Code: 50% rotation by a multiple of 90° plus flip; otherwise 50% rotation within ±20° dataset_synapse.py
Pretrainingnone§1Paper says results are obtained without pretrained weights

Results

Author-reported. The paper does not repeat the protocol in the results text. The Synapse setup follows TransUNet's evaluation §3.1. The data are 30 abdominal CT cases with 3779 axial images §3.1. The code lists 12 test volumes test_vol.txt. Scores cover 8 abdominal organs (background excluded). The ISIC 2018 setup follows TMU-Net §3.1; the split and input size are not stated in this paper.

Table 1: Synapse multi-organ, DSC and HD tab. 1. Columns: parameters in millions, DSC (higher is better), HD (lower is better), then per-organ DSC. The paper marks best and second-best with colour; those marks are not reproduced here.

ModelSize, millions of parametersMean Dice ↑Hausdorff distance ↓AortaGallbladderLeft kidneyRight kidneyLiverPancreasSpleenStomach
U-Net14.876.8539.7089.0769.7277.7768.6093.4353.9886.6775.58
Att-UNet34.8877.7736.0289.5568.8877.9871.1193.5758.0487.3075.75
TransUNet105.2877.4831.6987.2363.1381.8777.0294.0855.8685.0875.62
Swin-Unet27.1779.1321.5585.4766.5383.2879.6194.2956.5890.6676.60
LeVit-Unet52.1778.5316.8478.5362.2384.6180.2593.1159.0788.8672.76
MT-UNet79.0778.5926.5987.9264.9981.4777.2993.0659.4687.7576.81
TransDeepLab21.1480.1621.2586.0469.1684.0879.8893.5361.1989.0078.40
HiFormer25.5180.3914.7086.2165.6985.2379.7794.6159.5290.9981.08
MISSFormer42.4681.9618.2086.9968.6585.2182.0094.4165.6791.9280.81
EffFormer (baseline)22.3180.7917.0085.8166.8984.1081.8194.8062.2591.0579.58
DAE-Former w/o SCCA40.7581.5917.3187.4169.5785.2280.4694.6863.7191.4778.23
DAE-Former48.0182.6316.3987.8471.6587.6682.3995.0863.9391.8280.77

The text reports a Dice gain of 0.67 over MISSFormer §3.2. This matches 82.63 − 81.96 from Table 1. The text says the improvement is most visible for gallbladder, kidney, liver and spleen §3.2.

Table 2: ISIC 2018 skin lesion tab. 2. Columns: DSC, SE, SP, ACC (decimals as printed).

MethodDSCSESPACC
U-Net0.85450.88000.96970.9404
Att U-Net0.85660.86740.98630.9376
TransUNet0.84990.85780.96530.9452
MCGU-Net0.8950.8480.9860.955
MedT0.83890.82520.96370.9358
FAT-Net0.89030.91000.96990.9578
TMU-Net0.90590.90380.97460.9603
Swin U-Net0.89460.90560.97980.9645
TransNorm0.89510.87500.97900.9580
EffFormer0.89040.88610.96980.9519
DAE-Former w/o SCCA0.89620.86340.98300.9578
DAE-Former0.91470.91200.97800.9641

The text also gives a parameter comparison with TMU-Net (about 165.1 M against 48.1 M for DAE-Former) and states that TMU-Net uses extra boundary and foreground inputs §3.2. These are author statements; no table in the paper lists TMU-Net's parameters.

Table 3: Dual attention variants on Synapse tab. 3, app. A. Ordering of the two attention blocks and the fusion rule.

Dual attention strategyParams (M)DSCHD
Sequential (chosen)48.182.6317.46
Simple additive48.079.5123.83
Complex additive61.181.4919.36
Concatenation64.080.1127.20

Table 4: Number of skip connections on Synapse tab. 4, app. B.

Skip connectionsDSCHD
079.7122.46
181.8121.66
2 (base)82.6317.46

Table 5: Input resolution on Synapse tab. 5, app. B. The 224 row is the setting used throughout; the paper marks it with a dagger.

Image sizeDSCHD
128 × 12879.2015.90
224 × 224 (base)82.4317.46
288 × 28882.6817.85

Table 6: Parameter comparison tab. 6, app. D. Values as printed; DeepLabv3+ is listed by the paper as "DeepLapv3+".

MethodParams (M)DSCHD
DeepLabv3+ (CNN)59.5077.6339.95
Swin-Unet27.1779.1321.55
TransUNet105.2877.4831.69
LeVit-Unet52.1778.5316.84
MISSFormer42.581.9618.20
HiFormer25.5180.3914.70
DAE-Former48.182.6317.46

The text says DAE-Former is comparable in size to MISSFormer and has slightly better performance app. D. It also reports a Dice gap of 4.95 and an HD gap of 14.23 against TransUNet app. D. See Pitfalls.

The paper also reports a statistical check: 10 runs each for the SOTA methods and the proposed model, with per-organ mean Dice shown as a figure fig. 4. The text describes low variance for gallbladder, kidney and liver; no numbers are given in the text §3.2. Qualitative masks are in fig. 3 and fig. 8, app. C; Grad-CAM maps are in fig. 5. These are figures only.

Paper vs code

Code: github.com/mindflow-institue/DAEFormer, commit a13909b66bc6844e69e969eba33b5133c96e55e4, MIT License, copyright 2022 Amirhossein Kazerouni LICENSE. The paper and the README both link this repository. The repository contains only the Synapse training and evaluation code; there is no skin lesion code train.py.

  • Patch embedding. Paper: overlapping patch tokens of size 4×4, and 2×2 token merging that doubles channels §2. Code: stage 1 is a 7×7 convolution with stride 4 and padding 3; later stages use 3×3 convolutions with stride 2 and padding 1; channel widths are 128, 320 and 512 DAEFormer.py. The later stage-embedding class is in segformer.py.
  • SCCA roles. Paper: queries from the skip features X₂, keys and values from the projected decoder features X₁′ §2.4. Code: keys and queries both come from the skip features, and values come from the decoder features DAEFormer.py. The paper's form is what this card describes; an implementer copying the code gets the other form.
  • SCCA residual and FFN. The SCCA equation has no residual or feed-forward layer §2.4. Code concatenates the two inputs as a residual, then adds an attention output and a feed-forward layer DAEFormer.py.
  • Dual block order. Paper: efficient attention, then a Mix-FFN, then transpose attention, then a second Mix-FFN, with residuals as in eq. 4–8 §2.3. Code applies a LayerNorm before each sub-block. Transpose attention sees the normalised sum (eq. 6 has no LayerNorm there), and its residual includes the block input as well as the first Mix-FFN output; eq. 6 adds only the Mix-FFN output DAEFormer.py.
  • Feed-forward variant. Paper: plain Mix-FFN, eq. 9 §2.3. Code default mix_skip: the depthwise-conv output is added to the first projection before GELU, with a LayerNorm in the path DAEFormer.py.
  • Efficient-attention key width. Paper: the complexity remark assumes d_k = d/2 as typical §2.1. Code: d_k = d DAEFormer.py.
  • Transpose attention heads and temperature. Paper: τ is a temperature and no head count is in the compiled text §2.2. Code: 8 heads by default, with a learnable per-head temperature initialised to 1 DAEFormer.py. The L2 normalisation of queries and keys matches DAEFormer.py.
  • Upsampling. Paper: each decoder block expands by 2×, and a linear projection gives the map §2. Code: the top level expands by 4× and then applies a 1×1 convolution to 9 classes DAEFormer.py.
  • Greyscale input. Code repeats a one-channel input to three channels before the encoder DAEFormer.py. The paper does not say how inputs are formed.
  • Loss. Weights 0.6 Dice and 0.4 cross-entropy match §3 and trainer.py. The Dice implementation sums over all 9 classes, including background, uses squared terms in the denominator and averages over classes utils.py. The paper gives no Dice formula in the compiled text.
  • Learning-rate schedule. Paper: none stated §3. Code: poly decay with power 0.9 per iteration trainer.py.
  • Batch size and learning rate. Paper: batch 24, base lr 0.05 §3. README example: --batch_size 20 README.md. The training script scales the learning rate to 0.05 × 20/24 for that batch size train.py.
  • Metric. Paper: "average Hausdorff Distance" §3.1. Code: 95th-percentile Hausdorff (HD95) per organ and per volume utils.py.
  • Evaluation split. The trainer evaluates on the 12 test volumes every 20 epochs from epoch 200 trainer.py. There is no validation split. The paper does not say which checkpoint it reports.
  • Augmentation. Not in the paper. Code applies 90° rotations with flips, or ±20° rotations, to training slices dataset_synapse.py.

Pitfalls

  • The paper's numbers disagree with each other. Table 1 gives HD 16.39 for DAE-Former, but Tables 3, 4 and 6 give HD 17.46 for the same DSC 82.63 setting. Table 5 gives DSC 82.43 for the 224 setting, not 82.63 tab. 1, tab. 3–6. The appendix Dice gap to TransUNet is written as 4.95, but Table 1 gives 82.63 − 77.48 = 5.15; the HD gap of 14.23 uses HD 17.46 app. D. Pick one source per number and note which.
  • "HD" is HD95 in code. The paper's table column is HD. The code reports the 95th-percentile distance utils.py. Numbers from a reimplementation using the average distance will not match.
  • Degenerate metric cases bias the numbers. If the prediction is empty for an organ and the reference is not, the code returns Dice 0 and HD 0, so a total miss gets a perfect distance utils.py. If the prediction is non-empty and the reference is empty, the code returns Dice 1 and HD 0 utils.py. These cases rarely decide the Synapse mean, but they distort any per-slice reanalysis.
  • Reporting protocol. Periodic test-set evaluation is the only evaluation in the training script; best_performance is set and never used trainer.py. The paper does not say which epoch's numbers are in Table 1.
  • Single runs versus the variance claim. The paper reports 10 runs for the statistical figure but no numbers in the text, and the code fixes a seed of 1234 train.py. Whether Table 1 is a single run is not stated.
  • Complexity text. The space formula for transpose attention in §2.2 contains an N² term, while the same paragraph says transpose attention scales with d rather than N. The formula as printed does not match the d×d cross-covariance the method describes. Treat the space claim as unverified.
  • Ambiguous skip position. The paper does not say whether the skip is taken before or after patch merging. The code takes it before the merge, after the last block of each stage DAEFormer.py.
  • Input size is fixed in the code. The decoder sizes are hard-coded for 224 input through d_base_feat_size = 7 DAEFormer.py, and the expansion layer asserts the token count DAEFormer.py. The 128 and 288 settings in Table 5 therefore need code changes to run tab. 5.
  • Source contains uncompiled text. commented_passages.tex and several commented blocks in main.tex contain a Dice formula, a "single head" claim, and software versions. None of it is in the compiled paper and this card does not use it.
  • Parameter counts are not checked. The code was not executed, so the 48.01 M / 48.1 M figures are the paper's own.
  • Pretraining. The paper says it does not use pretrained weights §1. The training script has no weight loading path other than an unused --resume argument train.py, so training from scratch is what the code does.

Lineage

  • Builds on: efficient attention, linear cost through key-value context first §2.1, Shen et al.; cross-covariance (transpose) attention §2.2, XCiT, Ali et al.; Mix-FFN from MISSFormer §2.3, arXiv:2109.07162.
  • Compared against: U-Net, Attention U-Net, TransUNet, Swin-Unet, LeVit-Unet, MT-UNet, TransDeepLab, HiFormer, MISSFormer, EffFormer tab. 1; on ISIC 2018 also TMU-Net, TransNorm, MedT, FAT-Net, MCGU-Net tab. 2.
  • Superseded by: not stated; nothing in this wiki covers it.

Source and license

Adapted from “DAE-Former: Dual Attention-guided Efficient Transformer for Medical Image Segmentation” by Reza Azad, René Arimond, Ehsan Khodapanah Aghdam, Amirhossein Kazerouni, Dorit Merhof (arXiv:2212.13504), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/mindflow-institue/DAEFormer at commit a13909b66bc6 (MIT). Any quoted code is shown under that license.