Compressed Sensing MRI Reconstruction using a Generative Adversarial Network with a Cyclic Loss

In plain words

An MRI scanner builds its image from many measurements, and collecting them takes time. Compressed sensing lets you skip many of them and still recover a usable image, but the usual recovery methods are slow iterative solvers. This paper trains a neural network to do that recovery in one pass.

The network is trained as a GAN: a generator produces the full image from the incomplete data, and a discriminator tries to tell its outputs apart from real scans. On top of that, a "cycle" check is added. If the generated image is measured again with the same skipped pattern, it should match the measurements that were actually taken, and a fully scanned image that is skipped and reconstructed should come back close to itself. The method also stacks a second copy of the generator to clean up what the first one produces.

The point is speed and image quality at low sampling rates: a trained network needs a single forward pass per image, so it can run in near real time.

TL;DR

RefineGAN is a fully residual encoder-decoder generator trained adversarially against a discriminator with a cyclic data-consistency loss, followed by a second chained generator for refinement §III-B, §III-G. The authors report reconstruction times of about 0.06 to 0.11 s per 256x256 image and better PSNR, SSIM and NRMSE than the baselines they compare against, using radial masks at 10% to 40% sampling §IV-A, tab. 1. All quality results are author-reported and mostly shown as plots.

Problem

MRI acquisition is slow, which hurts patient comfort and time-critical diagnoses §I. Compressed sensing (CS-MRI) reconstructs an image from under-sampled k-space by solving a regularised optimisation problem with a sparsity prior §III-A, eq. 3. The paper's point is that these solvers are slow, and that dictionary-learning methods are slow too because of dictionary training and sparse coding §I, §II.

The paper assumes a static target and radial sampling masks at 10%, 20%, 30% and 40% of k-space, which correspond to 10x, 5x, 3.3x and 2.5x acceleration §IV-A. It does not study other masks for the main results; it reports a qualitative comparison of masks on the knee data only §IV-B, fig. 13.

Method

Notation. m is the measured under-sampled k-space, R the sampling mask and F the Fourier operator. The zero-filling image is s0 = F^H R^H m §III-A, eq. 1. Full images are s, and RF is the forward under-sampling operator §III-A, eq. 2.

Generator G. G is a convolutional autoencoder with residual blocks §III-E. It takes the zero-filling image and predicts a residual. The reconstruction is the zero-filling input plus G's output, so G learns the aliasing artefact rather than the image itself §III-E. Its encoder halves resolution with stride-2 3x3 convolutions and its decoder doubles it with transposed convolutions; the convolution mode is "same" §III-E. For complex data the input may have 2 channels (real and imaginary) §III-B. The experiments on real data use one channel §IV-A.

Discriminator D. D has the same architecture as G's encoder path. The output of its last residual block is used for the adversarial loss, with D(real) pushed toward 1 and D(fake) toward 0 §III-E.

Adversarial loss. G and D are trained as a minimax game §III-C, eq. 4. The paper writes the value function as

L_{adv}(G,D) = \mathbb{E}_{m}\left[1 - \log D(G(s_0))\right] + \mathbb{E}_{s}\left[\log D(s)\right] eq. 5

and states that in practice a Wasserstein GAN energy is used for stability §III-C. The variant is not named beyond that reference.

Cyclic data-consistency loss. The adversarial loss alone does not force G to respect the measurements §III-D. The cyclic loss has two terms, each computed on its own samples: §III-D, eq. 6

L_{cyc}(G) = d\left(m[i],\, RF(G(s_0[i]))\right) + d\left(s[j],\, G(F^HR^H RF\, s[j])\right) eq. 6

• L_freq: G's output for a measured sample is under-sampled again and compared with the measurements m[i].
• L_imag: a fully sampled image s[j] is under-sampled, zero-filled, passed through G, and compared with s[j].
• d can be MSE, MAE or another pixel-wise distance §III-D. The paper does not say which one its experiments use.
• This loss updates G only, not D §III-D.

Full objective. The generator minimises the total objective §III-F, eq. 7, with α = 1.0 and γ = 10.0 for all experiments §III-F:

L_{total} = L_{adv}(G,D) + \alpha\, L_{freq}(G) + \gamma\, L_{imag}(G) eq. 7

Optimisation. Adam, starting at a learning rate of 1e-4 that falls steadily across a 500-epoch run §III-F. The implementation uses the tensorpack wrapper over TensorFlow §III-F.

Chaining (RefineGAN). Two generators with identical architecture are connected in series §III-G, fig. 5. The first is called ReconGAN and the second RefineGAN. The whole chain is trained jointly, and each generator has its own variable scope, so the weights are not shared §III-G. The paper describes the second stage as a refinement of the first stage's output §III-G.

# One training step of RefineGAN  [§III-C] [§III-D] [§III-F] [§III-G]
# measured_batch: m[i] (independent samples); full_batch: s[j]  [§III-D]
# The weights alpha = 1.0 and gamma = 10.0 are fixed  [§III-F]

def train_step(measured_batch, full_batch):
    # Zero-filling input s0 = F^H R^H m for the measured samples  [eq. 1]
    zero_filled = zero_fill(measured_batch)
    # Chained generators: the second refines the output of the first  [§III-G]
    # Input form of the second generator is not stated in the text  [§III-G]
    fake = refinement_generator(first_generator(zero_filled))
    # Discriminator scores the real and the generated images  [§III-C]
    real_score = discriminator(full_batch)
    fake_score = discriminator(fake)
    # Adversarial term as written in the paper  [eq. 5]
    adv_value = adversarial_value(real_score, fake_score)
    # Frequency cycle term: re-sample G's output on the measurements  [eq. 6]
    freq_loss = distance(measured_batch, sample(fake))
    # Image cycle term: under-sample a full image, then reconstruct it  [eq. 6]
    image_loss = distance(full_batch, reconstruct(sample(full_batch)))
    # Total generator objective with fixed weights  [eq. 7] [§III-F]
    gen_loss = adv_value + 1.0 * freq_loss + 10.0 * image_loss
    return adv_value, gen_loss

Notes on the sketch: sample means RF and zero_fill means F^H R^H. The sketch does not say whether reconstruct in the image term is the full chain or only the first generator; the paper does not state it. The sign convention for D (maximising) and G (minimising) follows §III-C.

Hyperparameters

Results

Author-reported. Setup: real-valued magnitude-style images at 256x256 from the IXI brain set and the Data Science Bowl chest set, 100 training and 100 test images each, radial masks at 10% to 40% §IV-A. The paper does not describe how the baselines were run.

Running time, seconds per image tab. 1. The table does not state the sampling rate used for timing.

Image quality (PSNR, SSIM, NRMSE, plotted per sampling rate, not tabulated):

• Over 42 dB average PSNR at 40% sampling §I. The per-rate values are only in fig. 8, fig. 9 and fig. 10.
• The cyclic loss increases PSNR by about 1 dB over SingleGAN; the refinement stage reduces error by a similar amount §IV-A. These are text-only figures.
• Visual comparisons at 10% and 30% with 10x error maps are in fig. 11. The paper says RefineGAN has "less error" on the knee data at 10% fig. 12, §IV-B. This is qualitative.
• On knee data at 20%, radial masks gave the lowest NRMSE among radial, Cartesian, random and spiral masks fig. 13, §IV-B. The paper says RefineGAN beats SingleGAN and ReconGAN across these patterns §IV-B.

Complex-valued data §IV-B: RefineGAN accepts a 2-channel complex input and returns a 2-channel reconstruction. The results are qualitative only (no metric table).

Paper vs code

The repository hvcl/RefineGAN at commit 4895e2ea34d3350c3f0b5f4cca717507d0991636 is an official-looking repo. It belongs to the HVCL organisation, whose lab page lists Won-Ki Jeong (an author of this paper) as the principal investigator. The repo's README cites this paper and names the same authors.

The code is not used for this card. The repository has no LICENSE file, and its README says the code is for research use only and not for commercialisation. CLAUDE.md allows only pseudocode written in our own words and cited code from MIT, Apache-2.0 or BSD repositories. This one is neither, so no code was read into this card and no [code: ...] citations are given. The comparison between paper and code was therefore not done. A reader who needs the code's details should check the repository directly and its terms.

Other GitHub repositories named RefineGAN (for example tmquan/RefineGAN, hellopipu/RefineGAN, nikhilpinnaparaju/RefineGAN) were not used. Their owners are not verified as authors, and the tmquan account's profile name does not match an author.

Pitfalls

• Citation numbering. Section citations use the paper's IEEE numbering: Method is §III with subsections A to G, Results is §IV, Related work is §II. Equation numbers are the displayed equations of the compiled (clean) version: eq. 1 is the zero-filling image, eq. 2 is the forward under-sampling, eq. 3 is the constrained CS-MRI problem, eq. 4 is the minimax game, eq. 5 is L_adv, eq. 6 is L_cyc and eq. 7 is L_total. Figure numbers follow the order of the captions. The source LaTeX is the only reference used here; the compiled PDF was not checked.
• Speed claim versus table. The introduction says reconstruction of a 256x256 image takes under 100 ms §I. The table gives 0.106 s (brain) and 0.112 s (chest) for RefineGAN tab. 1. ReconGAN (about 61 to 69 ms) and SingleGAN (about 65 to 76 ms) are under 100 ms; RefineGAN is not.
• Timing is not comparable. Hardware is not stated for any method. CSCMRI and DLMRI timings depend on their numerical solvers §IV-A. The sampling rate used for timing is not stated.
• Epochs versus iterations. The LR schedule runs "over 500 epochs" §III-F, and Fig. 6 is captioned "over 500 epochs", but the text says the curves converge in "a few hundred training iterations" §IV-A. Which one a reimplementation should follow is not clear.
• Adversarial loss form. Eq. 5 is written with the term 1 - log D(G(s0)). The constant 1 does not change the gradient, so the generator term behaves like the non-saturating loss -log D(G(s0)). The text then says a Wasserstein energy is used "in practice" §III-C without giving the variant. An implementer has to pick one and document the choice.
• Which distance d. The paper allows MSE, MAE or others §III-D but never states which one its experiments use. The results depend on this choice.
• Chained training is underspecified. The text does not say whether the cyclic loss applies to the output of each stage or only to the final output. It also does not say whether the second generator's input is the first generator's full output, the first generator's residual, or the zero-filling image plus that. fig. 5 shows the architecture but is not described in words.
• Residual block description. The text says conv_i (stride 1, nb_filters features) "reduces the dimension of the tensor by half", but then conv_m has nb_filters/2 features and conv_o returns nb_filters. Read literally, conv_i does not halve the spatial size, since stride is 1. The intended meaning is not stated; the figure may settle it.
• Missing architecture numbers. Depth, number of residual blocks per level, nb_filters values, decoder stride and padding, and the discriminator's pooling are not given in the text. The values in fig. 4 are not reproduced here.
• Data are magnitude-style real images. The main quantitative results use real-valued images. The paper says extra preparation is needed, naming range normalisation and adding an imaginary channel §IV-A, but does not describe either step. Complex-valued results have no metrics §IV-B.
• Test versus validation. The 100 "test (validating)" images are the only held-out set mentioned §IV-A. Model selection is not described.
• Baseline fairness. The paper says SingleGAN, ReconGAN and RefineGAN use the same generator and discriminator sizes §IV-A. It does not say whether the baselines were re-trained on the same splits or reused published numbers.
• Plots, not tables. PSNR, SSIM and NRMSE values are shown only as plots fig. 8 to fig. 10. Numbers should be read from the source figures, not inferred.
• arXiv version. This card reads arXiv v2 (2018-03-15). A later journal version may differ and is not covered here.

Lineage

• Builds on: CycleGAN (Zhu et al., unpaired image-to-image translation with cycle-consistent GANs), which the paper calls the inspiration for its cyclic loss §I. The paper also uses Wasserstein GAN energy §III-C and residual learning (He et al.) §III-E. None of these have a card in this wiki.
• Compared against: CSCMRI (convolutional sparse coding, ADMM), DLMRI (patch dictionary learning), DeepADMM (Sun et al.), DeepCascade (Schlemper et al.), and SingleGAN (Mardani et al.; Yu et al.) §II, tab. 1.
• Superseded by: not known from content in this wiki.