Generated Faces in the Wild: Quantitative Comparison of Stable Diffusion, Midjourney and DALL-E 2

In plain words

Text-to-image tools draw whole scenes from a sentence, and faces inside those scenes are often the hardest part to get right. This paper asks how realistic the faces are in three popular tools: Stable Diffusion, Midjourney and DALL-E 2.

The method is simple. Captions from the COCO image dataset that mention a person are fed to each tool. A face detector cuts out the faces in the resulting pictures, the cut-outs are shrunk to small squares, and the whole set is compared with a set of real faces using a standard realism score (FID). A lower score means the generated faces look more like the real ones.

The author finds that Stable Diffusion's faces score best, and that Midjourney's score worst. The paper also releases the collection of generated faces it built, called GFW. The comparison is an early, face-focused check, and the score it uses was not designed for faces.

TL;DR

• Generated faces from Stable Diffusion, Midjourney and DALL-E 2 are scored with FID against real faces, using COCO captions as prompts §2.1 §2.2.
• Each FID is computed on sets of 5,000 faces, repeated 10 times; DALL-E 2 has too few faces for that and is also scored on 676 faces §2.3 §2.4 fig. 1.
• Stable Diffusion gives the lowest FID of the three; Midjourney the highest §2.4 fig. 1.
• The official code shows that the DALL-E 2 "5K" score was computed on far fewer unique faces than 5,000 (see Paper vs code).

Problem

Standard generative-model scores measure whole images or whole scenes. This paper states that fine-grained checks on categories such as faces are missing §1. It scores faces that appear in cluttered scenes, not portraits §1. That choice depends on a face detector and on a set-level score (FID) whose reference network was trained on ImageNet §3.

Method

Models §2.1. Three generators are compared:

1. Stable Diffusion: run with the Hugging Face Colab notebook; 512x512 output; a frozen CLIP ViT-L/14 text encoder; an 860M-parameter UNet and a 123M text encoder; the paper says it runs on a GPU with at least 10 GB VRAM.
2. Midjourney: images taken from a third-party collection on Kaggle; the paper does not say how many images were drawn from it beyond the total in §2.2.
3. DALL-E 2: prompts entered by hand in OpenAI's portal, because no code was available for large-scale generation; the paper says it is trained on about 650 million image-text pairs §2.1.

Prompts and face extraction §2.2.

1. Take COCO 2017 training captions as prompts. Keep a caption only if one of ten words appears in it: person, man, woman, men, women, kid, child, face, girl, boy §2.2.
2. Run a MediaPipe face detector on the whole image. Run it again on each crop to remove false positives. Then remove faces by hand that are heavily occluded, too dark or blurred, dull, animal, drawn or cartoon. Faces with eyeglasses are kept §2.2.
3. Resize every kept face to 100x100 pixels §2.2.

Real faces come from the same detector run on COCO 2017 training images, plus 13,233 faces from Labeled Faces in the Wild (LFW), which the paper crops to their central 100x100 area §2.2. The paper states 30,000 real faces in total §2.2.

Scoring §2.3. The paper uses the pytorch-fid implementation of FID §2.3 and gives no equations. For each model:

# FID protocol for one generated set  [§2.3]
# Inputs:  real_faces, generated_faces (lists of 100x100 crops)
# Output:  mean and standard deviation over 10 runs

def fid_protocol(real_faces, generated_faces):
    # Repeat the whole draw 10 times  [§2.3]
    scores = []
    for run in range(10):
        # Shuffle each pool and take 5,000 faces  [§2.3]
        real_sample = random_subset(real_faces, 5000)
        # Paper: sample with replacement when fewer than 5,000 exist
        # (DALL-E 2 only). See Paper vs code for what the code does.
        gen_sample = random_subset(generated_faces, 5000, replace=True)
        scores.append(fid(real_sample, gen_sample))
    return mean(scores), std(scores)

Real-versus-real reference §2.3:

# Reference score: two disjoint real sets  [§2.3]
# Shuffle the real pool, keep its first 10,000 faces, split them in two

def real_reference(real_faces):
    shuffled = random_permutation(real_faces)
    first_10k = shuffled[:10000]
    # Alternate faces between two sets of 5,000  [§2.3]
    set_a = first_10k[0::2]
    set_b = first_10k[1::2]
    return fid(set_a, set_b)

Scoring at 676 faces fig. 1, right: the same protocol, but every model is limited to 676 faces, the DALL-E 2 count §2.4.

Hyperparameters

Results

Author-reported. Setup: COCO-caption prompts; faces cropped to 100x100 §2.2; FID on sets of 5,000 faces, 10 runs §2.3; second panel at 676 faces fig. 1. The text gives no FID numbers. The values below are read from the bars of fig. 1 and are approximate. The paper's own text reports only the ranking.

Other reported counts §2.2: 15,076 generated faces in total (8,050 Stable Diffusion, 6,350 Midjourney, 676 DALL-E 2). The faces are taken from scenes with several objects, not from portraits §1.

Saved outputs in the official notebook match the bars for two models (see Paper vs code): Midjourney scores 80.2 to 85.7 over the 10 runs [code: Scoring.ipynb#L362-L452], and DALL-E 2 scores 63.7 to 65.5 [code: Scoring.ipynb#L560-L650]. The notebook has no saved output for Stable Diffusion or for the real-versus-real reference.

Qualitative findings §2.4 fig. 2 fig. 3: most generated faces look convincing at a glance, but many fail in eyeglasses, eyeballs, occlusions, profile views and symmetry §2.4 fig. 4 fig. 5 fig. 6. The paper attributes Midjourney's poor score partly to surreal or anime-style faces §2.4. For DALL-E 2 it gives three reasons: deepfake safeguards, a single focus of attention that favours portraits, and fewer images §2.4.

Paper vs code

The official repository is https://github.com/aliborji/GFW, owned by the paper's author, at commit 21c5b4505e80a3ae16ab49dd7c6552578587446b. It holds four Colab notebooks and a README. Its LICENSE file is GPL-3.0, so the code is cited here and not quoted. The README's license line still reads "[INSERT LICENSE TYPE HERE]"; the LICENSE file governs.

• DALL-E 2 "5K" uses far fewer than 5,000 faces. The paper says that because DALL-E 2 has fewer than 5K faces, it samples "with replacement" §2.3. The code copies each shuffled file under its own name, so a repeated face overwrites the earlier copy and the folder keeps only unique files [code: Scoring.ipynb#L679-L686] Scoring.ipynb. The saved outputs show 14 progress batches for the generated set against 100 for the real set in every DALL-E 2 run Scoring.ipynb Scoring.ipynb, which fits about 676 images, not 5,000. Figure 1 agrees: the DALL-E 2 bar is about the same in the 5K and 676 panels fig. 1. The Midjourney cell does reach 100 batches, so it is scored on 5,000 unique faces [code: Scoring.ipynb#L360-L361].
• LFW crop is not 100x100 from 250x250. The paper says it cropped the central 100x100 area of each 250x250 face §2.2. The code removes 50 pixels from each side, leaving 150x150, then resizes to 100x100 [code: lfw.ipynb#L70-L71]. The 250x250 source size is taken from the paper; the notebook does not state it.
• Two-pass detection matches; manual cleanup does not appear. The second detector pass over each crop, which removes faces with no detection, is in the code Stable_Diffusion_faces.ipynb. The manual removal of occluded, dark, blurred, animal and drawn faces §2.2 is not in any notebook.
• Generation settings are not the paper's. The paper names no Stable Diffusion version. The code uses CompVis/stable-diffusion-v1-4 with the fp16 branch Stable_Diffusion_faces.ipynb. Prompts are lower-cased Stable_Diffusion_faces.ipynb. Generation uses captions from index 20,000 on, and real faces use captions from index 30,000 on [code: Stable_Diffusion_faces.ipynb#L1787] Stable_Diffusion_faces.ipynb. The code sets a target of 10,000 generated faces, while the paper reports 8,050 for Stable Diffusion after cleanup Stable_Diffusion_faces.ipynb §2.2.
• Real-face counts cannot be reproduced. The notebooks do not show the 30,000 total or the split between COCO and LFW faces §2.2.
• No seeds. The shuffles in Scoring.ipynb do not set a random seed, so a rerun gives different subsets. The paper does not mention seeds either.
• FID network version. The code installs pytorch-fid 0.2.1 [code: Scoring.ipynb#L140] Scoring.ipynb and downloads its pt_inception-2015-12-05 weights Scoring.ipynb. The paper names only pytorch-fid.
• Data availability. The paper and the README point to a Google Drive file for the dataset §1, abstract. The README gives no license for the data. Midjourney faces come from a third-party Kaggle collection §2.1. Neither is copied into this card.

Pitfalls

• Sample size changes the number. FID depends on how many images are compared. The paper says its values are higher at 676 faces §2.4, and the real, Stable Diffusion and Midjourney bars rise between the two panels fig. 1. A 676-face score cannot be compared directly with a 5,000-face score.
• Do not use the DALL-E 2 5K bar as a 5,000-face result. Per Paper vs code, it is computed on unique faces from about 676 images. Compare DALL-E 2 with the 676 panel fig. 1.
• FID measures distribution, not individual faces. The paper notes that the reference network was trained on ImageNet and may be a poor judge of faces, and that alignment or a face-trained classifier might help §3. Crops are not aligned in the code either Stable_Diffusion_faces.ipynb.
• Face detection shapes the sample. Crops depend on the MediaPipe detector and its 0.8 confidence threshold Stable_Diffusion_faces.ipynb. Other thresholds would change which faces count.
• Hand steps block exact reruns. Manual removal of faces and manual prompting of DALL-E 2 §2.1 §2.2 cannot be reproduced from the notebooks.
• Stable Diffusion inference settings are missing. Steps, guidance scale and seed are not stated in the paper and are not set in the code [code: Stable_Diffusion_faces.ipynb#L1799]. A rerun will not match the paper's generated set.
• Results are approximate. Exact FID values are not in the paper text. The table above reads them from bars, and the saved outputs give only two of the four models.
• Error bars are shown but not tabulated. Figure 1 draws error bars over 10 runs fig. 1 §2.3. The paper does not state the standard deviations.

Lineage

• Builds on: FID (Heusel et al.), the score used §2.3; Stable Diffusion (Rombach et al.), one of the three models §2.1; DALL-E 2 (Ramesh et al.), one of the three models §2.1.
• Compared against: Midjourney, a text-to-image service in open beta at the time §2.1; real faces from COCO and LFW §2.2. The paper mentions StyleGAN as a portrait model that is not compared here §1.
• Superseded by: not stated in this wiki.