---
kind: concept
slug: medical-image-synthesis
title: Medical image synthesis
summary: >-
  Generates a medical image from another modality, such as a CT volume predicted from an MRI of the same patient, or from a text prompt. Two cards: a patch-based 3D GAN trained on paired, voxel-aligned MR and CT, with metrics on 16 and 22 subjects against older baselines; and a dentistry review whose one diffusion example of synthetic X-rays has no measurement.
tags: [medical-imaging, image-synthesis, gan, mri, ct, diffusion, dentistry]
papers: ["1612.05362", "2304.03086"]
related: [image-to-image-translation, generative-adversarial-network, auto-context-model, image-registration, diffusion-models, multimodal-language-model]
status: draft
updated_at: 2026-11-01
---

## Overview

Medical image synthesis predicts a missing imaging modality from one that was
acquired. The motivating case is MRI to CT: CT is needed for dose calculation
in radiotherapy and for PET attenuation correction, but it irradiates the
patient, while MRI does not [arxiv:1612.05362 §1]. The shared requirement is
paired training data: the card's method assumes MR and CT volumes of the same
subjects, voxel-aligned [arxiv:1612.05362 §3].

Two cards on wikillm list this concept. The first is
[Medical Image Synthesis with Context-Aware Generative Adversarial Networks](/papers/1612.05362),
which is the only one with a method and measurements. The second is
[ChatGPT for Shaping the Future of Dentistry](/papers/2304.03086), a review
with no method or evaluation of its own; it touches synthesis only through one
generated-image example and one text-record idea. The two cards measure
different things, so the sections below keep their evidence separate.
Earlier MR-to-CT approaches named in the paper are tissue segmentation, atlas
registration and feature-based regression [arxiv:1612.05362 §1]. None of them
has a card here, so they are named but not summarized.

## Variants

### Patch-based 3D GAN for paired MR-to-CT

A 3D fully convolutional generator maps a 32³ MRI patch to the matching 16³
CT patch, with no pooling [arxiv:1612.05362 §2.1] [arxiv:1612.05362 §4]. It is
trained as a GAN generator with an adversarial term, a squared-error term and
a 3D gradient difference term, weighted 0.5, 1 and 1 [arxiv:1612.05362 eq. 7]
[arxiv:1612.05362 §4]. The gradient term asks edges in the output to match
edges in the real CT. The authors add it because a squared-error loss alone
tends to give blurry output [arxiv:1612.05362 §1] [arxiv:1612.05362 §2]. Full volumes are assembled from
overlapping patch predictions that are averaged [arxiv:1612.05362 §2].

Reported results, author-reported and leave-one-out:

- Brain (16 subjects): MAE 92.5 and PSNR 27.6 for the proposed method, against
  26.3 for the structured random forest baseline (SRF+) and 21.3 for sparse
  representation (SR) [arxiv:1612.05362 tab. 1].
- Pelvis (22 subjects): MAE 39.0 and PSNR 34.1 for the proposed method, against
  32.1 for SRF+ [arxiv:1612.05362 tab. 2].
- The one numeric ablation is a generator alone versus the same generator
  trained as a GAN, reported as PSNR 24.7 versus 25.9. The card records that the
  dataset is not stated [arxiv:1612.05362 §4.1].

### Two-stage auto-context refinement

The same generator can be stacked: the second stage receives the MRI patch and
the first stage's full-volume CT estimate as a second channel. The paper uses
two such iterations on both datasets [arxiv:1612.05362 §2.2] [arxiv:1612.05362 §4.2].
This variant has its own concept page,
[Auto-Context Model](/concepts/auto-context-model), which also covers the cost
of building the context and the gains reported only as plotted curves.

### Text-prompted synthetic dental X-rays (diffusion, unmeasured)

The dentistry review describes a diffusion model, which the authors name as
DALL-E 2, that makes synthetic panoramic dental X-rays from a short text prompt
such as "an alveolar cleft" [arxiv:2304.03086 p. 8] [arxiv:2304.03086 fig. 8].
The figure is one output the authors chose to show. The paper gives no
evaluation, no baseline and no dataset [arxiv:2304.03086 p. 8]
[arxiv:2304.03086 fig. 8]. Treat it as a demonstration of the idea, not as
evidence that the generated images are clinically faithful. Card:
[ChatGPT for Shaping the Future of Dentistry](/papers/2304.03086).

### Synthetic dental records (text, not images)

The same review proposes a text model that writes artificial patient records,
so that real records need not be shared [arxiv:2304.03086 p. 4-5]
[arxiv:2304.03086 fig. 4]. These are text, not images, so this page does not
treat them as image synthesis. The paper reports no results for them. Its
human-evaluation box is empty, and five of the six generation settings have no
value given [arxiv:2304.03086 p. 4-5] [arxiv:2304.03086 fig. 4].

### Variants not covered

No card covers 2D slice-wise synthesis, other modality pairs, or unpaired
training. Diffusion-based synthesis appears only as the unmeasured dental
example above. These are not discussed further here. For unpaired image
translation in general, see
[Image-to-image translation](/concepts/image-to-image-translation).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Paired, voxel-aligned MR and CT of the same subjects | Patch-based 3D GAN (this card) | This is the setting the paper evaluates [arxiv:1612.05362 §3] |
| Only unpaired MR and CT | Not this card's method | The card's losses compare each output with its own ground-truth CT [arxiv:1612.05362 eq. 5] |
| Target depends on structure wider than one 32³ patch | Add the auto-context stage | The paper uses two stages and reports that more iterations help, only as plotted curves [arxiv:1612.05362 §4.2] [arxiv:1612.05362 fig. 5] |
| Tight training budget | Single stage, no auto-context | Each extra stage needs full-volume predictions for every training subject first [arxiv:1612.05362 §2.2] |
| Need a clear gain attributed to auto-context | Run your own on/off comparison | The card has no auto-context on/off number [arxiv:1612.05362 §4.1] [arxiv:1612.05362 §4.2] |
| Need a runnable reference implementation | Build from the paper text, not the released code | The released 3D code omits auto-context and differs in layers and losses [arxiv:1612.05362 code: 3dganversion/g_model.py#L56] |
| Need to compare against the reported baselines | Re-run them under the same protocol | The SRF+ numbers are copied from its paper, not re-run [arxiv:1612.05362 §4] |
| Need synthetic dental X-rays as training data | Not supported by either card | The only example is one unmeasured output with no evaluation [arxiv:2304.03086 p. 8] [arxiv:2304.03086 fig. 8] |
| Need to share patient data via synthetic records | Treat as an open idea | The review proposes it and reports no results or privacy test [arxiv:2304.03086 p. 4-5] [arxiv:2304.03086 p. 11] |

## Implementation sketch

The core loop is a patch-based GAN with a three-term generator loss. The
shapes and weights are from the card's method description. Preprocessing,
stride and training length are not stated in the paper.

```pseudocode
# Patch-based 3D GAN for paired MR -> CT  [arxiv:1612.05362 §2.1] [§4]
# Input:  paired mr_volumes, ct_volumes (voxel-aligned, same subjects)
# Output: generator G mapping MRI patches to CT patches

def train_patch_gan(mr_volumes, ct_volumes):
    G = new_3d_fcn_generator()        # no pooling, BN + ReLU  [§2.1]
    D = new_3d_cnn_discriminator()    # scores real vs generated CT  [§2.1]
    for step in range(num_steps):     # number not stated  [§4]
        # discriminator: real CT -> 1, generated CT -> 0  [eq. 3]
        mr, ct = sample_patches(mr_volumes, ct_volumes,
                                in_size=32, out_size=16)
        fake = G(mr)
        loss_d = bce(D(ct), 1) + bce(D(stop_grad(fake)), 0)
        update(D, loss_d)
        # generator: adversarial + squared error + gradient difference  [eq. 7]
        mr, ct = sample_patches(mr_volumes, ct_volumes,
                                in_size=32, out_size=16)
        fake = G(mr)
        loss_g = 0.5 * bce(D(fake), 1)           # adversarial weight  [§4]
        loss_g += 1.0 * sum_sq(ct - fake)        # L2 weight  [§4]
        loss_g += 1.0 * gradient_difference(ct, fake)  # GDL weight  [§4]
        update(G, loss_g)
    return G

def predict_volume(G, mr_volume):
    # overlapping 32^3 inputs, 16^3 outputs, averaged where they overlap  [§2]
    # stride not stated in the paper  [code: 3dganversion/g_model.py#L178]
    out = zeros_like(mr_volume)
    count = zeros_like(mr_volume)
    for centre in sliding_window(mr_volume):
        out[crop16(centre)] += G(crop32(mr_volume, centre))
        count[crop16(centre)] += 1
    return out / count
```

## Contradictions

No numeric conflict exists between the two cards, because only the GAN card
reports numbers. The difference is in evidence: the GAN card gives
leave-one-out metrics on 16 and 22 subjects [arxiv:1612.05362 tab. 1]
[arxiv:1612.05362 tab. 2], while the review reports none for its synthetic
images or records [arxiv:2304.03086 p. 8].

Within the GAN card, the paper and its released code disagree on several points.
Neither side is resolved here.

- **Auto-context.** The paper describes it [arxiv:1612.05362 §2.2]; the
  released 3D code has a single-channel input and no such stage
  [arxiv:1612.05362 code: 3dganversion/g_model.py#L56].
- **Generator layers.** The paper gives filters 32, 32, 32, 64, 64, 64, 32, 32
  [arxiv:1612.05362 §2.1]; the code uses 32, 32, 32, 32, 64, 64, 64, 32 and a
  final single-channel layer [arxiv:1612.05362 code: 3dganversion/g_model.py#L91-L101].
- **Discriminator.** The paper gives three conv, BN, ReLU and max-pooling
  stages with 5³ kernels and FC sizes 512, 128, 1 [arxiv:1612.05362 §2.1]; the
  code uses 3³ convolutions, no pooling, and omits the 512-unit layer
  [arxiv:1612.05362 code: 3dganversion/g_model.py#L109-L116].
- **Learning rate.** The paper states 1e-6 [arxiv:1612.05362 §4]; the code's
  flag is 2e-7 [arxiv:1612.05362 code: 3dganversion/main.py#L10].
- **Minibatches.** The paper updates G on a separate minibatch
  [arxiv:1612.05362 §2.1]; the code feeds the same batch to both updates
  [arxiv:1612.05362 code: 3dganversion/g_model.py#L146-L154].

The paper also gives an inconsistent discriminator description internally: it
lists four convolution filter counts for three stages [arxiv:1612.05362 §2.1].

## Papers

- [Medical Image Synthesis with Context-Aware Generative Adversarial Networks](/papers/1612.05362): a 3D patch-based GAN for MRI-to-CT synthesis, with auto-context refinement, evaluated on 16 brain and 22 pelvis subjects.
- [ChatGPT for Shaping the Future of Dentistry](/papers/2304.03086): a review of multi-modal LLM uses in dentistry, with one diffusion example of synthetic panoramic X-rays and a proposal for synthetic records; no evaluation.