wikillm

Paper · 2023-07-18 · cs.AI, cs.HC

PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation

Visual analysis system for text-to-image prompt refinement. It generates Stable Diffusion images over a guidance-scale range, retrieves similar prompt-image pairs from DiffusionDB with CLIP, clusters them, and recommends cluster-level TF-IDF keywords shown in a 2-D map. A 12-person user study reports better creativity support than two baselines (author-reported).

Prompt engineeringText-to-image generationHierarchical clusteringDimensionality reductionprompt-engineeringtext-to-imagevisual-analyticskeyword-recommendationcliptf-idft-snediffusiondbuser-study

arXivPDFCodeMarkdown12 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Text-to-image models turn a sentence into a picture, but finding the sentence that gives the picture you want is usually trial and error. Small changes to the words, and to settings such as how strongly the image must follow the text, can change the result a lot. PromptMagician is a tool that makes this search less blind. It generates a batch of images with several settings at once, looks up pictures that other people made from similar sentences, and shows everything on one map so that similar pictures sit together.

Words that tend to appear with a group of pictures are shown near that group, so you can see which words push the style or subject in a given direction. You can also sort pictures by a pair of opposing words, such as "cute" and "ugly", and keep only the ones near one end. The tool then suggests words to add to your sentence. The authors tested it with a small user study and compared it with two baseline setups.

TL;DR

PromptMagician is an interactive visual system for prompt engineering with Stable Diffusion. It generates images across a guidance-scale range, retrieves similar prompt-image pairs from DiffusionDB with CLIP, and clusters them. It recommends keywords per cluster with TF-IDF and shows images and keywords in a multi-level 2-D map §4 §5. The authors report better creativity support than two baselines in a 12-person study; no numbers are given in the text §7.4.4 fig. 7.

Problem

Text-to-image models are sensitive to wording and to hyper-parameters such as the guidance scale and the random seed. Users, especially novices, struggle to write prompts that give the wanted result, and it is hard to tell whether the prompt or the setting is at fault §1. Automatic prompting can help, but image creation depends on human judgement, so the authors keep a human in the loop §1. Published keyword guidelines for prompts exist, but the authors judge them too general for personal goals §1 §2.

Method

The system has four views: a model input view (prompt, guidance range, number of images), an image browser (generated and retrieved images with keywords placed near their clusters), an image evaluation view (filter by opposing words) and a local exploration view (details, keyword-by-image table, guidance histogram) §5.1 fig. 1. The pipeline behind it has five stages, labelled A to E in the paper: retrieve, embed, cluster, mine keywords and match keywords to clusters §4 fig. 3.

Generation. The system calls Stable Diffusion with the prompt and a guidance range §3.1 §5.1. The paper says a guidance value is sampled randomly from the range §5.1. The batch-level sampling is only in the code (see Hyperparameters and Paper vs code).

Retrieval (A). The prompt is encoded with CLIP. The DiffusionDB image features are the search space, because the authors found that prompts alone return many images generated from the same prompt §4.1. Items are ranked by cosine similarity between the prompt and the images §4.1.

Embedding (B). Each retrieved image is represented by its CLIP image feature and the CLIP feature of its prompt, concatenated to a 1024-dimensional vector §4.2. A t-SNE projection to 2-D uses cosine distance, to match the CLIP training objective §4.2.

Clustering (C). Images are grouped with agglomerative (bottom-up) hierarchical clustering, which avoids choosing the number of clusters in advance §4.3. A cluster is only used for keyword mining if its number of children is in a set range (the paper gives 3 to 20) §4.3. Clusters built from children that lie far apart in the 2-D layout are dropped; the distance cut-off is not stated §4.3.

Keyword mining (D). For each usable cluster, words and multi-word phrases from its prompts are scored by TF-IDF at cluster level. The term frequency is the count in the cluster divided by the total count in the cluster eq. 2 eq. 3. The inverse document frequency compares the cluster with the whole retrieved prompt set eq. 4. Stop words are removed with NLTK; n-grams such as "unreal engine" are kept §4.4. The top keywords of each cluster are kept §4.4.

Matching (E). Each keyword is attached to the cluster where its TF-IDF score is highest eq. 5. Multi-word phrases that map to the same cluster are merged, and the single words they contain are dropped §4.5.

Positioning. A keyword sits at the average position of the images whose prompts contain it, weighted by how often it appears in each prompt eq. 6 §5.2. Overlapping keywords get a small random shift §5.2.

Multi-level view. At overview level, each cluster is shown by the image nearest its centre; other images are hidden behind translucent rectangles. Zooming reveals the images and keywords of finer clusters §5.2.

Evaluation of images (criteria). The user gives one or two words, for example "simple" and "complex". Each image gets a score from CLIP similarity to the two phrases "[word] image" §5.3. A softmax over the two similarities gives a value between 0 and 1 eq. 7. If the second word is left empty, the opposite is "not" plus the first word §5.3. The score distribution is shown with brushing for filtering §5.1 §5.3.

Pseudocode
# Keyword recommendation as the paper describes it  [§4] [fig. 3]
# Input:  prompt (str); image_db (CLIP image features per item)
# Output: keywords, each with a best cluster and a 2-D position

def recommend_keywords(prompt, image_db, n_search, top_k):
    # Retrieve the database items closest to the prompt  [§4.1]
    retrieved = top_k_cosine(clip_text(prompt), image_db, n_search)
    # Concatenate the CLIP image and prompt features (512 + 512)  [§4.2]
    features = concat(clip_image(retrieved), clip_text(retrieved))
    # Project to 2-D with t-SNE using cosine distance  [§4.2]
    layout = tsne_2d(features, metric="cosine")
    # Build a bottom-up merge tree over the images  [§4.3]
    tree = agglomerative_tree(layout)
    keywords = {}
    for cluster in tree.clusters():
        # Skip clusters outside the child-count range  [§4.3]
        if not 3 <= cluster.size <= 20:
            continue
        # Skip clusters merged from distant children; cut-off not stated  [§4.3]
        if cluster.merge_distance_too_large(layout):
            continue
        # Score each term in the cluster's prompts by cluster TF-IDF  [§4.4]
        scores = {t: tfidf(t, cluster, retrieved) for t in cluster.terms()}
        # Drop stop words, keep n-gram phrases  [§4.4]
        scores = drop_stop_words(scores)
        # Keep the top terms of this cluster  [§4.4]
        keywords[cluster] = top_n(scores, top_k)
    # Attach each keyword to the cluster with its highest score  [§4.5]
    best = {kw: argmax_cluster(kw, keywords) for kw in keywords}
    # Place each keyword at its occurrence-weighted image mean  [§5.2]
    return {kw: weighted_mean(layout, kw) for kw in best}

The TF-IDF definitions, as written in the paper:

\mathrm{tfidf}_{i,x} = \mathrm{tf}_{i,x} \times \mathrm{idf}_{i} eq. 1

\mathrm{tf}_{i,x} = \frac{n_{i,x}}{\sum_{k} n_{k,x}} eq. 2

n_{i,x} = \sum_{d_j \in c_x} n_{i,j} eq. 3

\mathrm{idf}_{i} = \lg \frac{|D|}{|\{ j : t_i \in d_j \}|} eq. 4

cb_{t_i} = \underset{c_x \in c_{t_i}}{\arg\max}\ \mathrm{tfidf}_{i,x} eq. 5

Keyword position and the image rating:

p_{t_i} = \frac{\sum_j n_{i,j}\, p_j}{\sum_j n_{i,j}} eq. 6

\bar{s} = \frac{e^{s_1}}{e^{s_1} + e^{s_2}} eq. 7

Pseudocode
# Rate one image between two opposing words  [§5.3] [eq. 7]
# Input:  image; word_1; word_2 (the caller passes "not " + word_1 if empty)
# Output: value in [0, 1], near 1 when the image matches word_1 (per eq. 7)

def rate_image(image, word_1, word_2):
    # Cosine similarity of the image with each filled-in template  [§5.3]
    s_1 = cosine(clip_image(image), clip_text(word_1 + " image"))
    s_2 = cosine(clip_image(image), clip_text(word_2 + " image"))
    # Two-way softmax over the similarities  [§5.3] [eq. 7]
    return exp(s_1) / (exp(s_1) + exp(s_2))

Hyperparameters

Values marked "paper" come from the paper. Values marked "code" are from the authors' repository at the commit in the frontmatter and are not stated in the paper. Where the two disagree, see Paper vs code.

NameValueWhereNotes
Image and text encoderCLIP, 512-dim each, concatenated to 1024§4.2Variant not named in the paper. Code: ViT-B/32 workflow.py
Retrieved items per query400config.pyNot stated in the paper
Similarity for retrievalcosine§4.1Code ranks by softmax over CLIP logits, same order workflow.py
t-SNE output dimension2§4.2Paper gives cosine metric only
t-SNE perplexity, init10, randomworkflow.pyNot stated in the paper
Cluster child count3 to 20§4.3Code uses strict bounds, 3 < n < 20 workflow.py
Distance cut-off for merged clustersnot stated§4.3Code: position range divided by 15 server.py
Linkage methodnot stated§4.3Code: SciPy default (single linkage, Euclidean) on the 2-D layout server.py
Keywords per cluster"top", number not stated§4.4Code: 1 for the overview workflow.py; 10 in the detail panel server.py
Stop wordsNLTK English list§4.4Code removes single words only workflow.py
Minimum cluster TF-IDFnot stated§4.4Code: 0.01 workflow.py; weight filter 0.02 util.py
Prompt cap for the encodernot stated§4Code: at most 40 words, each word at most 3 times workflow.py server.py
DiffusionDB subsetnot stated§3.1Code: first 100,000 items of 2m_first_100k config.py
Stable Diffusion checkpointnot stated§3.1Code: CompVis/stable-diffusion-v1-4 config.py
Image sizenot stated§3.1Code: 512 x 512 config.py
Denoising steps, schedulernot stated§3.1Code: 50 steps config.py; DPM-Solver++ multistep scheduler, fp16 sd_server.py
Images per prompt call7config.pyScenario 1 generates 100 images in total §6.1
Devicesnot stated§3.3Code: 8 GPUs, one server each config.py
Guidance-scale samplingrange from user input; sampled on a 0.5 grid§5.1Code: one value per 7-image batch sd_server.py
Guidance range, scenario 15 to 30, then 5 to 10§6.1Front-end default range 0 to 50 parameter.ts
Guidance histogram10 bins over 0 to 50§5.1Code server.py
Rating scalesoftmax, no temperatureeq. 7Code uses CLIP logits, which include CLIP's logit scale server.py
Keyword position jitternot stated§5.2Code: 1 to 3 percent of layout range, random sign, both axes util.py
Formative study9 ordinary users, interviews over four months§3.2
User study12 participants, three systems, 20 minutes per system§7.1 §7.3Order counterbalanced §7.3
Expert interviews5 creators from a Midjourney community§8Qualitative only

Results

Author-reported. The setup is a single-university user study with 12 participants, comparing the full system with two baselines on creativity support §7.1 §7.2 §7.3. The text gives no numeric scores; the Likert results and the creativity support bars appear only in figures.

StudySetupHeadline, as reportedSource
Keyword usefulness (Q1 to Q3)12 users, five-point scaleMost users found keywords meaningful, related and helpful§7.4.1 fig. 6
Visual system (Q4 to Q7)same usersBrowser, evaluation view and detail panel judged useful for exploring and validating keywords§7.4.2 fig. 6
Usability (Q8 to Q10)same usersAll agreed the system is easy to learn and use, and all would use it again§7.4.3
Creativity support indexthree systems, collaboration dimension excludedFull system higher than both baselines on every dimension§7.4.4 fig. 7
Expert interviews5 creatorsQualitative design implications; open issues on knowing when to stop§8
Usage scenariostwo single-user walkthroughsQualitative: users reached a target image after keyword changes§6.1 §6.2

The authors also describe three prompt patterns seen in use: adding detail step by step, using summary words, and replacing one word §7.5. In the discussion they report that negative keywords can backfire, an effect they call semantic contamination §8.1.

Paper vs code

Repository: https://github.com/YingchaojieFeng/PromptMagician, linked from the paper's introduction §1. The owner matches the first author, so it is used. It was cloned at commit 06b95add88788646f2bb586c02d5c2333db44abb. The repository has no LICENSE file, and its README states no licence, so code_license is recorded as none. No verbatim code is quoted in this card.

Discrepancies, each with both sides:

  • Clustering corpus. The paper runs clustering and keyword mining on the retrieved set §4. The code runs them on the images the user marked as satisfied, after the brushing step on the front end server.py index.tsx.
  • Clustering space and linkage. The paper clusters the image representation §4.3. The code clusters the 2-D t-SNE coordinates with SciPy's default linkage server.py.
  • Child-count limit. The paper ties the allowed range to how many images were retrieved §4.3. The code uses fixed, strict bounds of 3 and 20 workflow.py.
  • Keywords per cluster. The paper keeps "the top prompt keywords for each cluster" §4.4. The overview keeps one keyword per cluster workflow.py workflow.py.
  • TF-IDF formula. The paper writes the idf with "lg" (base not restated) over all prompts as the document set, with no normalisation eq. 4. The code builds one document per non-cluster prompt plus one joined cluster document, applies scikit-learn TF-IDF with L2 row normalisation and no smoothing, and takes the cluster's row workflow.py util.py. The idf is therefore not the value in eq. 4.
  • Weighting and filters. The code multiplies each score by a factor that grows with the number of underscores in the term util.py. It drops terms that share a word with an already chosen term workflow.py. None of this is in the paper.
  • Phrases. The paper uses n-grams §4.4. The code joins phrases with a pre-trained gensim model whose file is not in the repository util.py util.py.
  • Cut-off. The distance cut-off is not stated in the paper; the code uses the layout range divided by 15 server.py.
  • Retrieval count. Not stated in the paper; the code retrieves 400 config.py server.py.
  • Database subset. The paper describes the full DiffusionDB and its 2M subset §3.1. The code uses the first 100,000 items of 2m_first_100k config.py workflow.py.
  • Image rating. Eq. 7 uses raw cosine similarities eq. 7. The code feeds the CLIP model's logits into the softmax server.py, so the scores are sharper. The returned value is the probability of the second word server.py, while eq. 7 gives the first.
  • Negation. The paper says "not" plus the first word §5.3. The code concatenates without a space, giving "notcute image" server.py.
  • Guidance sampling. The paper says values are sampled randomly within the range §5.1. The code draws from a 0.5 grid, one value for each batch of 7 images from one device sd_server.py.
  • Generation count. The code generates 8 devices times 7 images times ceil(N / 56) batches, then keeps N server.py.
  • Prompt cleaning. The code drops prompts with Chinese characters, a few listed strings and unusual characters, and caps prompt length workflow.py util.py. The paper does not mention this.
  • Stable Diffusion settings. The paper does not name the checkpoint, size, steps or scheduler. The code uses v1-4 at 512 x 512, 50 steps and a DPM-Solver++ scheduler config.py sd_server.py.
  • CLIP variant. The paper gives 512-dim features §4.2; the code loads ViT-B/32 workflow.py, which is consistent with that size.

The user study and expert interviews have no code in the repository.

Pitfalls

  • Two corpora. Following §4 gives a different cluster set and TF-IDF scores than the shipped code, which works on the user's selection. Decide which corpus you mean before comparing results.
  • Count, not k. The overview number of keywords follows the number of clusters that pass the size filter, not the "top k" setting workflow.py.
  • Crash path. If a keyword has no substring match in its cluster, the helper returns a two-item list of zeros util.py. The caller then indexes a float and fails server.py. I did not run this; the path is read from the code.
  • Substring matching. Keyword counts use substring search, so "cat" also matches "category" util.py.
  • Unseeded jitter. Keyword positions change between requests because the shift is random and no seed is set in the files read util.py.
  • Wasted generations. For N = 100 the code creates 112 images on 8 GPUs and discards 12 server.py.
  • Shared guidance per batch. All 7 images from one device share one guidance value sd_server.py sd_server.py.
  • Rating convention. Check the front-end labels before using the value from the code, since it is the second word's probability server.py.
  • Negative words. The authors report that adding negative keywords can remove a body part instead of fixing it §8.1. Expect some keywords to backfire.
  • Study scope. The user study has 12 students from one university; the paper names this as a limit of who was tested §7.1. Scores are only in the figures.
  • Infrastructure. The default needs 8 GPUs and a precomputed CLIP feature file for the 100,000-item subset config.py workflow.py.
  • Licence. The repository has no LICENSE file. Re-implement from this card and the paper, not from the code text.

Lineage

  • Builds on: Stable Diffusion, which the paper uses for generation §3.1 (no arXiv id given in the bibliography); CLIP, arXiv:2103.00020, for retrieval, embedding and rating §4.1 §5.3; DiffusionDB, arXiv:2210.14896, the search corpus §3.1; t-SNE for the 2-D layout §4.2; TF-IDF for keyword scoring §4.4 §1.
  • Compared against: Baseline A, similarity search without keyword recommendation, similar to browsing a public gallery §7.2; Baseline B, Promptist, arXiv:2212.09611, an automatic prompt refiner §7.2.
  • Related, not compared: RePrompt, an explainable prompt-optimisation method for text-to-image models §2. Named only; no card on this wiki.

Source and license

Adapted from “PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation” by Yingchaojie Feng, Xingbo Wang, Kam Kwai Wong, Sijia Wang, Yuhong Lu, Minfeng Zhu, Baicheng Wang, Wei Chen (arXiv:2307.09036), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/YingchaojieFeng/PromptMagician at commit 06b95add8878 (none (no LICENSE file in repository)). Any quoted code is shown under that license.