Medical image synthesis

Overview

Medical image synthesis predicts a missing imaging modality from one that was acquired. The motivating case is MRI to CT: CT is needed for dose calculation in radiotherapy and for PET attenuation correction, but it irradiates the patient, while MRI does not 1612.05362 §1. The shared requirement is paired training data: the card's method assumes MR and CT volumes of the same subjects, voxel-aligned 1612.05362 §3.

Two cards on wikillm list this concept. The first is Medical Image Synthesis with Context-Aware Generative Adversarial Networks, which is the only one with a method and measurements. The second is ChatGPT for Shaping the Future of Dentistry, a review with no method or evaluation of its own; it touches synthesis only through one generated-image example and one text-record idea. The two cards measure different things, so the sections below keep their evidence separate. Earlier MR-to-CT approaches named in the paper are tissue segmentation, atlas registration and feature-based regression 1612.05362 §1. None of them has a card here, so they are named but not summarized.

Variants

Patch-based 3D GAN for paired MR-to-CT

A 3D fully convolutional generator maps a 32³ MRI patch to the matching 16³ CT patch, with no pooling 1612.05362 §2.1 1612.05362 §4. It is trained as a GAN generator with an adversarial term, a squared-error term and a 3D gradient difference term, weighted 0.5, 1 and 1 1612.05362 eq. 7 1612.05362 §4. The gradient term asks edges in the output to match edges in the real CT. The authors add it because a squared-error loss alone tends to give blurry output 1612.05362 §1 1612.05362 §2. Full volumes are assembled from overlapping patch predictions that are averaged 1612.05362 §2.

Reported results, author-reported and leave-one-out:

• Brain (16 subjects): MAE 92.5 and PSNR 27.6 for the proposed method, against 26.3 for the structured random forest baseline (SRF+) and 21.3 for sparse representation (SR) 1612.05362 tab. 1.
• Pelvis (22 subjects): MAE 39.0 and PSNR 34.1 for the proposed method, against 32.1 for SRF+ 1612.05362 tab. 2.
• The one numeric ablation is a generator alone versus the same generator trained as a GAN, reported as PSNR 24.7 versus 25.9. The card records that the dataset is not stated 1612.05362 §4.1.

Two-stage auto-context refinement

The same generator can be stacked: the second stage receives the MRI patch and the first stage's full-volume CT estimate as a second channel. The paper uses two such iterations on both datasets 1612.05362 §2.2 1612.05362 §4.2. This variant has its own concept page, Auto-Context Model, which also covers the cost of building the context and the gains reported only as plotted curves.

Text-prompted synthetic dental X-rays (diffusion, unmeasured)

The dentistry review describes a diffusion model, which the authors name as DALL-E 2, that makes synthetic panoramic dental X-rays from a short text prompt such as "an alveolar cleft" 2304.03086 p. 8 2304.03086 fig. 8. The figure is one output the authors chose to show. The paper gives no evaluation, no baseline and no dataset 2304.03086 p. 8 2304.03086 fig. 8. Treat it as a demonstration of the idea, not as evidence that the generated images are clinically faithful. Card: ChatGPT for Shaping the Future of Dentistry.

Synthetic dental records (text, not images)

The same review proposes a text model that writes artificial patient records, so that real records need not be shared 2304.03086 p. 4-5 2304.03086 fig. 4. These are text, not images, so this page does not treat them as image synthesis. The paper reports no results for them. Its human-evaluation box is empty, and five of the six generation settings have no value given 2304.03086 p. 4-5 2304.03086 fig. 4.

Variants not covered

No card covers 2D slice-wise synthesis, other modality pairs, or unpaired training. Diffusion-based synthesis appears only as the unmeasured dental example above. These are not discussed further here. For unpaired image translation in general, see Image-to-image translation.

When to use which

Implementation sketch

The core loop is a patch-based GAN with a three-term generator loss. The shapes and weights are from the card's method description. Preprocessing, stride and training length are not stated in the paper.

# Patch-based 3D GAN for paired MR -> CT  [arxiv:1612.05362 §2.1] [§4]
# Input:  paired mr_volumes, ct_volumes (voxel-aligned, same subjects)
# Output: generator G mapping MRI patches to CT patches

def train_patch_gan(mr_volumes, ct_volumes):
    G = new_3d_fcn_generator()        # no pooling, BN + ReLU  [§2.1]
    D = new_3d_cnn_discriminator()    # scores real vs generated CT  [§2.1]
    for step in range(num_steps):     # number not stated  [§4]
        # discriminator: real CT -> 1, generated CT -> 0  [eq. 3]
        mr, ct = sample_patches(mr_volumes, ct_volumes,
                                in_size=32, out_size=16)
        fake = G(mr)
        loss_d = bce(D(ct), 1) + bce(D(stop_grad(fake)), 0)
        update(D, loss_d)
        # generator: adversarial + squared error + gradient difference  [eq. 7]
        mr, ct = sample_patches(mr_volumes, ct_volumes,
                                in_size=32, out_size=16)
        fake = G(mr)
        loss_g = 0.5 * bce(D(fake), 1)           # adversarial weight  [§4]
        loss_g += 1.0 * sum_sq(ct - fake)        # L2 weight  [§4]
        loss_g += 1.0 * gradient_difference(ct, fake)  # GDL weight  [§4]
        update(G, loss_g)
    return G

def predict_volume(G, mr_volume):
    # overlapping 32^3 inputs, 16^3 outputs, averaged where they overlap  [§2]
    # stride not stated in the paper  [code: 3dganversion/g_model.py#L178]
    out = zeros_like(mr_volume)
    count = zeros_like(mr_volume)
    for centre in sliding_window(mr_volume):
        out[crop16(centre)] += G(crop32(mr_volume, centre))
        count[crop16(centre)] += 1
    return out / count

Contradictions

No numeric conflict exists between the two cards, because only the GAN card reports numbers. The difference is in evidence: the GAN card gives leave-one-out metrics on 16 and 22 subjects 1612.05362 tab. 1 1612.05362 tab. 2, while the review reports none for its synthetic images or records 2304.03086 p. 8.

Within the GAN card, the paper and its released code disagree on several points. Neither side is resolved here.

• Auto-context. The paper describes it 1612.05362 §2.2; the released 3D code has a single-channel input and no such stage 1612.05362 code: 3dganversion/g_model.py#L56.
• Generator layers. The paper gives filters 32, 32, 32, 64, 64, 64, 32, 32 1612.05362 §2.1; the code uses 32, 32, 32, 32, 64, 64, 64, 32 and a final single-channel layer 1612.05362 code: 3dganversion/g_model.py#L91-L101.
• Discriminator. The paper gives three conv, BN, ReLU and max-pooling stages with 5³ kernels and FC sizes 512, 128, 1 1612.05362 §2.1; the code uses 3³ convolutions, no pooling, and omits the 512-unit layer 1612.05362 code: 3dganversion/g_model.py#L109-L116.
• Learning rate. The paper states 1e-6 1612.05362 §4; the code's flag is 2e-7 1612.05362 code: 3dganversion/main.py#L10.
• Minibatches. The paper updates G on a separate minibatch 1612.05362 §2.1; the code feeds the same batch to both updates 1612.05362 code: 3dganversion/g_model.py#L146-L154.

The paper also gives an inconsistent discriminator description internally: it lists four convolution filter counts for three stages 1612.05362 §2.1.

Papers

• Medical Image Synthesis with Context-Aware Generative Adversarial Networks: a 3D patch-based GAN for MRI-to-CT synthesis, with auto-context refinement, evaluated on 16 brain and 22 pelvis subjects.
• ChatGPT for Shaping the Future of Dentistry: a review of multi-modal LLM uses in dentistry, with one diffusion example of synthetic panoramic X-rays and a proposal for synthetic records; no evaluation.