Paper · 2022-04-03 · cs.CV
Style-Based Global Appearance Flow for Virtual Try-On
Parser-free virtual try-on that warps a garment with a dense appearance flow predicted by a StyleGAN-style generator. One global style vector, modulated convolutions, then a local refinement per scale, so large person-garment misalignments are handled. Author-reported on VITON: SSIM 0.91 and FID 8.89, against 0.89 and 10.09 for the previous parser-free state of the art.
Knowledge distillationImage-to-image translationThin-plate spline (TPS) warpDense correspondencevirtual-try-onappearance-flowgarment-warpingstyleganknowledge-distillation
arXivPDFCodeMarkdown12 min read
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Virtual try-on means showing a shop garment on a photo of a person. The hard step is bending the flat garment image so that its sleeves, collar and hem land where the person's body is. Earlier methods look at small neighbourhoods of the two images to find matching points. That works when the person is roughly where the garment is expected, and fails when the person stands off-centre or in an unusual pose.
This paper uses a different kind of network, borrowed from StyleGAN, a well-known image generator. The network looks at the whole person and the whole garment at once, condenses them into one summary vector, and uses that vector to steer the prediction of where each garment pixel should move. A second, local step then tidies up fine details such as the arms and hands.
The model also does not need a human-parsing tool at test time, so it does not depend on that tool's mistakes. The authors first train a teacher model that does use a parser, and the parser-free model learns from it. The authors report that the result beats earlier parser-free methods on the standard VITON benchmark, and especially on test photos where the person is shifted.
TL;DR
The method predicts a dense appearance flow that moves garment pixels onto the person's body. A global style vector, computed from the lowest-resolution features of both images, modulates a convolutional flow generator §3.4 eq. 2. A local refinement at each scale then corrects fine detail §3.4 eq. 4. The model is parser-free at inference and is trained with a parser-based teacher §3.2. Author-reported: FID 8.89 and SSIM 0.91 on VITON, better than all compared methods tab. 1. Check the Paper vs code section before implementing. The code differs from the paper in how flows are combined, which loss terms are used and which learning rate and epoch counts it uses.
Problem
Appearance-flow warping predicts, for every output pixel, where to sample the garment image from. Earlier flow estimators compute matches only within local receptive fields, so they assume the corresponding garment and body regions already sit close together §1. Large misalignments, unusual poses and occlusions break this assumption §1 fig. 1.
Parser-based try-on methods need a human-parsing model at inference, and their results inherit its errors §2. Parser-free methods remove that dependency, usually by distilling a parser-based teacher §2 §3.2.
The paper assumes VITON-style data: a person image and a garment image at 256×192 with paired ground truth for training §4 Datasets.
Method
The model has a parser-based teacher F_PB and a parser-free student F. Only F is used at test time §3.2 eq. 1. The teacher receives a semantic representation (segmentation, keypoints, DensePose) and produces a try-on image that the student takes as its person input during training §3.2.
The student F has three parts §3.2: an encoder for the person, an encoder for the garment (both convolutional, written E_p and E_g), a warping module W, and a generator G. E_p and E_g share one architecture of stacked residual blocks and produce N feature maps each §3.3. The paper uses N = 5 §4 Implementation details.
The warping module W has N stacked blocks §3.4. Each block has a style-based coarse flow and a local refinement.
- Build a global style vector s from the lowest-resolution person and garment features, through one fully connected layer per side and concatenation §3.4 eq. 2. A footnote says the two-sided vector gave better results than the person side alone, as an empirical finding; no ablation is shown §3.4 footnote.
- In block 1, modulate a convolution on the lowest-resolution garment map with s to get a coarse flow §3.4 eq. 3.
- In every later block, upsample the previous flow, sample the garment feature with it, and modulate a convolution with s to get the coarse flow at that scale §3.4 eq. 3.
- Refine locally: sample the garment feature with the coarse flow, concatenate it with the person feature at the same scale, and predict a refinement flow with a convolution §3.4 eq. 4.
- Add the coarse and refinement flows to get that block's flow §3.4 eq. 5.
- After the last block, sample the garment image with the final flow to get the warped garment §3.4 eq. 6.
- Concatenate the warped garment with the person image and generate the try-on image with an encoder-decoder with skip connections §3.4 eq. 7.
The pseudocode follows the paper's equations. The released code combines flows differently; see Paper vs code.
# Style-based appearance flow, coarse to fine [§3.4] [eq. 2-6]
# Input: garment[k], person[k] for scale k = 1..N (k = N is lowest)
# style s, shape (B, 256)
# Output: flow (B, 2, H, W), a sampling grid over the garment image
def style_flow(garment, person, style):
# Block 1: style-modulated coarse flow from the lowest scale [eq. 3]
coarse = modulated_conv(garment[N], style)
# Block 1: local refinement after sampling the garment [eq. 4]
refine = refine_conv(concat(sample(garment[N], coarse), person[N]))
# Block 1: coarse and refinement flows are added [eq. 5]
flow = coarse + refine
for k in range(N - 1, 0, -1):
# Upsample the previous flow to the next scale [eq. 3]
up = upsample(flow, scale=2)
# Coarse flow at scale k, from the sampled garment [eq. 3]
coarse = modulated_conv(sample(garment[k], up), style)
# Refinement at scale k, from the sampled garment [eq. 4]
refine = refine_conv(concat(sample(garment[k], coarse), person[k]))
# Coarse and refinement flows are added [eq. 5]
flow = coarse + refine
# Final flow is the output of the last block [eq. 6]
return flowTraining combines four terms §3.5 eq. 8-11. The paper gives the weights λ only symbolically.
# Training objective [§3.5] [eq. 8-12]
# The lambdas are not stated in the paper; see Paper vs code.
def total_loss(try_on, person_gt, warped, mask, flow, teacher, student):
# Perceptual loss between the try-on and the ground truth [eq. 8]
loss_p = vgg_distance(try_on, person_gt)
# Warped garment against the masked ground truth [eq. 9]
loss_g = norm(warped - mask * person_gt)
# Smoothness of every block's flow, generalised Charbonnier [eq. 10]
loss_r = sum(charbonnier_gradient(f) for f in flow)
# Feature distillation from the parser-based person encoder [eq. 11]
loss_d = sum(norm(t - s) for t, s in zip(teacher, student))
# Weighted sum of the four terms [eq. 12]
return lam_p * loss_p + lam_g * loss_g + lam_r * loss_r + lam_d * loss_dPretraining order §3.2: first the parser-based model F_PB, then F. F_PB is used only to produce person images for F's input and to supervise F §3.2.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Training pairs | 14,221 | §4 Datasets | VITON train; code draws indices from 14221 aligned_dataset.py |
| Test pairs | 2,032 | §4 Datasets | VITON test |
| Resolution | 256 × 192 | §4 Datasets | Both person and garment |
| Blocks N | 5 | §4 Implementation details | Code: five scales afwm.py |
| Style and feature width c | 256 | §4 Implementation details | Style vector is 128 + 128 in code afwm.py |
| Batch size | 4 | §4 Implementation details | Matches --batchSize 4 in the training scripts train_PFAFN_e2e_fs.sh |
| Epochs | 100 | §4 Implementation details | Code stage 1: 100 + 100; end-to-end default 50 + 50, see Paper vs code |
| Optimiser | Adam | §4 Implementation details | Betas not stated in the paper. Code: β1 = 0.5, β2 = 0.999 train_options.py train_PFAFN_e2e_fs.py |
| Initial learning rate | 5e-4 | §4 Implementation details | Code default is 5e-5, see Paper vs code train_options.py |
| LR schedule | constant, then linear decay to zero after 50 epochs | §4 Implementation details | Code decays by lr / niter_decay each epoch after niter afwm.py |
| Warp learning rate | not stated | Code uses 0.2 × lr for the warp model afwm.py | |
| Loss weights λ_p, λ_g, λ_R, λ_D | not stated | §3.5 eq. 12 | Code uses different weights, see Paper vs code |
| Hardware | 1 × RTX 2080 Ti | §4 Implementation details | |
| Charbonnier ε, exponent | not stated | §3.5 eq. 10 | Code: ε = 0.001, exponent 0.45 train_PFAFN_e2e_fs.py |
| Demodulation ε | not stated | §3.4 | Code: 1e-5 afwm.py |
| Refinement conv width | not stated | §3.4 eq. 4 | Code: 512 → 128 → 64 → 32 → 2 with LeakyReLU 0.1 afwm.py |
Results
Author-reported. Setup: VITON test set (2,032 pairs), person and garment at 256 × 192. Baselines are the models listed in §4 Evaluation metrics and baselines. Metrics are SSIM and FID; the paper drops IS for try-on §4 Evaluation metrics and baselines. The code repository contains no FID or SSIM script; the README points to external implementations.
| Setting | Metric | Best baseline | This paper | Source |
|---|---|---|---|---|
| VITON test | SSIM ↑ | 0.89 (PF-AFN, Cloth-flow*) | 0.91 | tab. 1 |
| VITON test | FID ↓ | 10.09 (PF-AFN) | 8.89 | tab. 1 |
| Augmented VITON test | SSIM ↑ / FID ↓ | 0.87 / 12.19 (PF-AFN) | 0.91 / 9.91 | tab. 3 |
| Augmented VITON test, drop from standard | ΔSSIM / ΔFID | 0.002 / 2.10 (PF-AFN) | 0 / 1.02 | tab. 3 |
| Human preference (AMT), ours share vs CP-VTON++ / ACGPN / Cloth-flow* / PF-AFN | % | 87.3 / 79.8 / 61.5 / 56.8 | tab. 2 | |
| Ablation, SM + RF (full) | SSIM / FID | 0.91 / 8.89 | tab. 4 | |
| Ablation, SM only (global style flow) | SSIM / FID | 0.89 / 9.84 | tab. 4 | |
| Ablation, RF only (local correspondence) | SSIM / FID | 0.89 / 10.73 | tab. 4 |
The paper states the FID reduction against PF-AFN is 11.9% §4 Main results. The arithmetic from tab. 1 agrees (10.09 → 8.89). The human study used 15 AMT workers, each comparing 100 images per model pair §4 Evaluation metrics and baselines.
Paper vs code
Code: official repository SenHe/Flow-Style-VTON at ed25d525a53fadaf3ea794d0beb6affbb75f8160. The owner matches the first author, and the paper's abstract links it. The repository is CC BY-NC-SA 4.0, so this card paraphrases the code without quoting it.
- Flow combination. The paper adds the coarse flow and the refinement flow eq. 5. The code does not add them. It applies each predicted offset to the identity grid, then samples the previous flow at that grid, so the flows compose afwm.py. A reader implementing eq. 5 literally will get a different model.
- Refinement input. The paper samples the garment feature with the coarse flow alone eq. 4. The code samples it with the composed flow, which includes the previous blocks afwm.py.
- Gradient stop. The code detaches the previous flow before sampling the person feature in each block afwm.py. The paper does not mention this.
- Smoothness. The paper uses the gradient of the flow eq. 10. The code uses second-order filters in x, y and both diagonals afwm.py with a generalised Charbonnier penalty train_PFAFN_e2e_fs.py. It adds a first-order total-variation term on all predicted offsets, weighted 0.01 train_PFAFN_e2e_fs.py train_PFAFN_e2e_fs.py.
- Style vector. The paper says "fully connected layers" §3.4. The code uses 8 × 6 convolutions, which act as fully connected layers only when the lowest-resolution map is 8 × 6, i.e. the input is 256 × 192 afwm.py. This is inferred from the shapes, not stated.
- Downsampling. The paper says each residual block is followed by a pooling layer §4 Implementation details. The code uses a stride-2 convolution with batch norm and ReLU before each residual pair afwm.py afwm.py.
- Modulated convolution. The code demodulates the weights, with ε = 1e-5, and multiplies activations by √2 afwm.py afwm.py. The paper does not mention either.
- Learning rate. The paper gives 5e-4 §4 Implementation details. The code default is 5e-5 train_options.py. The stage-1 script uses 3e-5 train_PFAFN_stage1_fs.sh. The warp optimiser uses 0.2 × lr train_PFAFN_e2e_fs.py.
- Epochs. The paper trains for 100 epochs §4 Implementation details. The stage-1 script uses 100 epochs at constant learning rate and 100 with decay, 200 in total train_PFAFN_stage1_fs.sh. The end-to-end defaults give 50 + 50 train_options.py.
- Two-phase stage 1. The stage-1 script trains only parameters whose names contain
cond_plus the refinement convolutions for the first 100 epochs, then all warp parameters train_PFAFN_stage1_fs.py train_PFAFN_stage1_fs.py. The name filter also picks up the person-side style layer through operator precedence. The paper does not describe this schedule. - Generator output. The paper's generator takes the warped garment and the person image and outputs the try-on eq. 7. The code's generator takes the masked try-on from the teacher, the warped garment and the warped edge (7 channels). It outputs RGB and a mask. The try-on is a learned blend of the warped garment and the rendered image, restricted to the garment region train_PFAFN_e2e_fs.py. The generator is a residual U-Net train_PFAFN_e2e_fs.py networks.py.
- Loss terms. The paper has four terms with unspecified weights eq. 12. The code has about a dozen weighted terms: L1 and VGG at five scales, edge-map loss, second-order smoothness, cond-feature and flow supervision, mask L1, skin-region losses and background losses train_PFAFN_e2e_fs.py train_PFAFN_e2e_fs.py. The total is 0.25 × warp loss + generator loss train_PFAFN_e2e_fs.py. The generator weights change at epoch niter train_PFAFN_e2e_fs.py.
- Teacher weighting. The code supervises the student's cond features and flow only on samples where the teacher's warp has lower L1 error than the student's train_PFAFN_e2e_fs.py train_PFAFN_e2e_fs.py. The paper does not describe this.
- Distillation form. The paper's distillation term is a norm of the difference eq. 11. The code uses squared error, summed over five scales and weighted by (5 − k) × 0.04 train_PFAFN_e2e_fs.py.
- Garment mask. The paper takes the garment mask from an off-the-shelf parser §3.5. The code reads the dataset's label map, class 4 train_PFAFN_e2e_fs.py.
- Augmentation. The augmented test set uses person shifts and zooms §4 Datasets. The code has an augmented dataset class, but the training entry point uses the non-augmented one unless the user changes the import train_PFAFN_e2e_fs.py aligned_dataset.py.
- Consistent with the paper. Batch size 4 train_PFAFN_e2e_fs.sh. The 14,221 training indices aligned_dataset.py. The decay after 50 epochs, given the default counts train_options.py.
Pitfalls
- Equation numbers. The source labels f_ci, f_ri and f_i as eq:2, eq:3 and eq:4, and the text cites "Equation 2" for f_ci. Counting the numbered equations in the source gives 3, 4 and 5 §3.4. Check the PDF before citing these numbers. The numbers here are counted from the source.
- Norms. The paper writes ‖·‖ in eq. 1, 8–11 without saying L1 or L2 §3.2 §3.5. The code's choices differ by term, see Paper vs code.
- Sign of the flow. Flows are in normalised coordinates in [-1, 1], as in the code's grid helper afwm.py. Sampling with pixel offsets instead will not match.
- Input size is fixed. The 8 × 6 style convolutions only work for 256 × 192 inputs, as inferred above. Other sizes need a different style head.
- Teacher required for training. The parser-based teacher must be trained first and its person images generated before the student can be trained §3.2. This is a multi-stage pipeline, not one training run.
- Hidden loss weights. λ_p, λ_g, λ_R and λ_D are unstated §3.5. Taking the code's weights gives a different model. Use the code's values only with that caveat.
- Old dependencies. The end-to-end training script uses NumPy aliases (
np.float,np.int) that current NumPy releases removed train_PFAFN_e2e_fs.py train_PFAFN_e2e_fs.py. The model hard-codes a CUDA tensor type afwm.py. The README asks for Python 3.6.13 and PyTorch 1.1.0 [README.md]. - Table naming. The paper writes PF-AFN in some places and AF-PFN in others (Table 2) tab. 2. Figure 3's caption cites the DCTON paper for PF-AFN fig. 3. The text cites the PF-AFN paper instead §2.
- Metrics not reproducible from the repository. FID and SSIM are computed by external code not in this repository [README.md].
- Augmented results. The augmented test images are an external download; the README says to point the test at them and to switch the augmented training dataset on [README.md].
Lineage
- Builds on: appearance flow for warping, introduced for try-on by Han et al. (ClothFlow, ICCV 2019) and used in the parser-free PF-AFN (Ge et al., CVPR 2021) §2. Style modulation follows StyleGAN §2 §3.4. None of these has a card in this wiki.
- Compared against: VTON, CP-VTON, CP-VTON++, Cloth-flow, ACGPN, DCTON, ZFlow and PF-AFN §4 Evaluation metrics and baselines. Only those with cards in this wiki can be linked from here.
- Superseded by: not known from this wiki.
Source and license
Adapted from “Style-Based Global Appearance Flow for Virtual Try-On” by Sen He, Yi-Zhe Song, Tao Xiang (arXiv:2204.01046), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/SenHe/Flow-Style-VTON at commit ed25d525a53f (CC BY-NC-SA 4.0 (non-commercial; LICENSE.txt). Paraphrased only, no snippets.). Any quoted code is shown under that license.