Paper · 2021-11-29 · cs.CL
GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injection
GALAXY pre-trains a UniLM-based dialog model with a dialog-act prediction head on labelled dialogs and on unlabelled dialogs. A dropout consistency KL term is gated by act-distribution entropy. Author-reported end-to-end combined scores: 107.45 (In-Car), 110.35 (MultiWOZ2.0), 110.76 (MultiWOZ2.1).
Dialogue generationSemi-supervised learningConsistency regularizationdialoguepre-trainingsemi-supervised
arXivPDFCodeMarkdown12 min read
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
A task-oriented chatbot, such as one that books a table, has to do three things in each turn: understand what the user said, decide what to do next, and write a reply. Pre-trained chat models learn the first and third from large piles of plain dialogue text, but they never learn the "what to do next" part explicitly. Labelled data for that part exists, but it is small and each dataset labels it differently.
GALAXY adds the missing step to pre-training. Each turn gets a set of labels (such as "ask a question" or "confirm"), and the model learns to predict them along with the reply. Most dialogue data has no such labels, so the authors also let the model train on unlabelled dialogue. They ask the model to give consistent label guesses when the same input runs through it twice with different random dropout. A gate turns that consistency signal down on unlabelled dialogues whose label guesses look unreliable, since many of them do not fit the label set at all. The result is a model that can be fine-tuned on a specific chatbot task with less labelled data.
TL;DR
GALAXY is a UniLM-based dialog model pre-trained on labelled dialog acts (UniDA) and unlabelled dialogs (UnDial) with a dialog-act prediction head, a dropout consistency KL term and an entropy gate on the unlabelled rows §4.3. Author-reported end-to-end combined scores beat the best listed prior models on In-Car, MultiWOZ2.0 and MultiWOZ2.1 §6.1. Before you implement it, read the Paper vs code section: the released pre-training code has no response-selection loss, and its loss normalisation and gate differ from the equations.
Problem
Dialog models pre-trained on plain text or on generic dialog corpora capture conversational language but not the dialog policy that a task-oriented system needs §1. Dialog acts are the usual way to label that policy, but labelled act data is small and inconsistent across corpora §1 app. A.1. Most dialogs have no act labels, and naive multi-task training on the few labelled ones overfits §1. Methods that add a latent act variable do not use explicit act labels §1.
Method
GALAXY uses UniLM as its backbone. The model has a bidirectional encoder for understanding and a unidirectional decoder for generation, with shared weights §4.1. Each input token gets token, role, turn and position embeddings, as in PLATO §4.1 app. B.1 fig. 8. The final hidden state at the [CLS] position is the sentence representation h_cls, and the act and selection heads read it §4.2 app. B.1.
Pre-training runs on a mixed batch of UniDA rows (tag 0, acts known) and UnDial rows (tag 1, acts unknown) §4.3 preprocess.py. Four objectives apply §4.2:
- Response selection (binary classification of matched and random responses) §4.2 eq. 1 eq. 2.
- Response generation (negative log-likelihood of the reply) §4.2 eq. 3.
- DA prediction, multi-label with one sigmoid per act, on labelled rows only §4.2 eq. 4 eq. 5.
- Dropout consistency: a bidirectional KL between two dropout passes of the same input, softmax over the acts §4.2 eq. 6 eq. 7 fig. 3.
On unlabelled rows the KL term is multiplied by a gate g in [0, 1] computed from the entropy of the first pass's act distribution §4.3 eq. 9. Labelled rows keep weight 1 §4.3 eq. 8 eq. 10. Fine-tuning keeps the DA loss when the task has act labels (alpha = 1) and drops it otherwise (alpha = 0) §4.4 eq. 12. For tasks with labels, the target sequence is the label sequence followed by the reply; labels include belief state, database result, act and delexicalised response §4.4 app. B.2 fig. 9.
Data
- UniDA (labelled): 20 unified dialog acts app. A.1 built by aligning eight existing task-oriented corpora plus DailyDialog §3.1. Totals: 70,726 dialogs and 975,780 utterances tab. 1. Corpora: MultiWOZ, Frames, MSRe2e, SGD, DSTC2, SimJoint, STAR, DailyDialog tab. 1.
- UnDial (unlabelled): 14 dialog corpora, about 35M utterances, average 2.5 utterances per dialog §3.2 tab. 2 tab. 10. Reddit is the largest source, with 31,908,317 of the 35,529,276 utterances tab. 10. Filters: drop instances with URLs, with a word repeated in at least three-word runs, non-English sentences, markup characters, offensive language, and non-unicode characters, which are replaced app. A.2.
- Leakage control: the MultiWOZ validation and test dialogs are removed from UniDA during pre-training §6. In the low-resource runs, the unused part of the MultiWOZ training set is also removed from UniDA §6.4.
Equations
The equations below are the paper's, in our notation. Eq. 9 is the paper's gate exactly as printed; see Paper vs code for the implemented version.
\mathcal{L}_{\mathrm{RS}} = -\log p(l{=}1 \mid c, r) - \log p(l{=}0 \mid c, r^{-}) eq. 1
p(l{=}1 \mid c, r) = \operatorname{sigmoid}(\phi_a(h_{cls})) \in \mathbb{R}^1 eq. 2
\mathcal{L}_{\mathrm{RG}} = -\sum_{t=1}^{T} \log p(r_t \mid c, r_{<t}) eq. 3
\mathcal{L}_{\mathrm{DA}} = -\sum_{i=1}^{N} \{\, y_i \log p(a_i \mid c) + (1-y_i) \log(1 - p(a_i \mid c)) \,\} eq. 4
p(a \mid c) = \operatorname{sigmoid}(\phi_b(h_{cls})) \in \mathbb{R}^N eq. 5
q(a \mid c) = \operatorname{softmax}(\phi_b(h_{cls})) \in \mathbb{R}^N eq. 6
\mathcal{L}_{\mathrm{KL}} = \tfrac{1}{2} \big( D_{\mathrm{KL}}(q_1 \| q_2) + D_{\mathrm{KL}}(q_2 \| q_1) \big) eq. 7
\mathcal{L}_{\mathrm{label}} = \mathcal{L}_{\mathrm{RS}} + \mathcal{L}_{\mathrm{RG}} + \mathcal{L}_{\mathrm{DA}} + \mathcal{L}_{\mathrm{KL}} eq. 8
g = \min\!\Big\{\max\!\Big\{0,\ \frac{E_{max} - (E + \log E)}{E_{max}}\Big\},\ 1\Big\}, \quad E_{max} = \log N eq. 9
\mathcal{L}_{\mathrm{unlabel}} = \mathcal{L}_{\mathrm{RS}} + \mathcal{L}_{\mathrm{RG}} + g\,\mathcal{L}_{\mathrm{KL}} eq. 10
\mathcal{L}_{\mathrm{pre}} = \mathcal{L}_{\mathrm{unlabel}} + \mathcal{L}_{\mathrm{label}} eq. 11
\mathcal{L}_{\mathrm{fine}} = \mathcal{L}_{\mathrm{RS}} + \mathcal{L}_{\mathrm{RG}} + \alpha\,\mathcal{L}_{\mathrm{DA}} eq. 12
The VAE and pseudo-labelling baselines have their own losses in app. C.1 (numbered eq. 13 to eq. 15 in the compiled paper).
Pre-training step (as the paper describes it)
# One pre-training step on a mixed batch [§4.3]
# Input: contexts, responses, da_labels (B, N) in {0, 1}, tags (B,)
# Output: scalar loss; tag 0 = UniDA (labelled), tag 1 = UnDial
def pretrain_step(contexts, responses, da_labels, tags):
# Two dropout passes over the same batch [§4.2]
first = model(contexts, responses)
second = model(contexts, responses)
labelled = tags == 0
# Response selection, binary cross-entropy [§4.2] [eq. 1]
loss_rs = selection_bce(first, second)
# Response generation, negative log-likelihood [§4.2] [eq. 3]
loss_rg = generation_nll(first, responses)
# DA loss on labelled rows: one sigmoid per act [§4.2] [eq. 4]
loss_da = da_bce(first.act_logits, da_labels) * labelled
# Symmetric KL between the two dropout passes [§4.2] [eq. 7]
kl = symmetric_kl(softmax(first.act_logits),
softmax(second.act_logits))
# Gate on unlabelled rows; weight 1 on labelled rows [§4.3] [eq. 9]
gate = entropy_gate(softmax(first.act_logits))
kl_weight = where(labelled, 1.0, gate)
# Batch normalisation of the sum is not stated [§4.3] [eq. 8] [eq. 10]
return mean(loss_rs + loss_rg + loss_da + kl_weight * kl)# Gate from the entropy of the first pass's act distribution [§4.3]
# Input: probs (B, N), softmax over N acts
# Output: gate (B,) in [0, 1]
def entropy_gate(probs):
# Maximum entropy of N acts, E_max = log N [§4.3] [eq. 9]
max_entropy = log(num_acts)
# Entropy as a positive number; see Paper vs code [§4.3] [eq. 9]
entropy = -sum(probs * log(probs))
# Linear in entropy and log entropy, clipped to [0, 1] [§4.3] [eq. 9]
gate = (max_entropy - entropy - log(entropy)) / max_entropy
return clip(gate, 0, 1)Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Backbone | UniLM; 12 transformer blocks; hidden size 768 | app. B.2 | Pre-training starts from UniLM weights app. B.2 |
| Number of acts N | 20 | app. B.2 app. A.1 | Multi-label |
| Pre-training context length | 256 tokens | app. B.2 | Code keeps the last 256 tokens pretrain_field.py pretrain_field.py |
| Pre-training response length | 50 tokens | app. B.2 | Code default pretrain_field.py |
| Context turns (pre-training) | 16, including the reply turn | not stated in paper | Code default pretrain_field.py pretrain_field.py |
| Batch size | 128 | app. B.2 | Released script uses 32 train_single.sh; meaning (per process or total) not stated |
| Optimiser | AdamW | app. B.2 | |
| Learning rate | 1e-5 | app. B.2 | Same in released script train_single.sh |
| Warmup | not stated | — | Code default: 10% of steps pretrain_trainer.py |
| Weight decay | not stated | — | Code default 0.0 unified_transformer.py |
| Gradient clipping | not stated | — | Code default max norm 5.0 unified_transformer.py |
| Dropout (pre-training) | 0.3 | app. B.2 | Script sets hidden, attention and feed-forward dropout to 0.3 and embedding dropout to 0.0 train_single.sh |
| KL weight | unit weight in the equations | eq. 8 eq. 10 | Released script sets kl_ratio 5.0 train_single.sh |
| Gate | Eq. 9, computed on the first pass | §4.3 | Gate is detached by default pretrain_unified_transformer.py |
| DA decision threshold | 0.5 for F1 | not stated | Code uses 0.5 pretrain_unified_transformer.py |
| Random seed (pre-training) | 11 | app. B.2 | Not set in released pre-training script |
| Checkpoint | epoch 14 | app. B.2 | Script trains 60 epochs train_single.sh; no epoch-selection code found |
| Fine-tuning optimiser and lr | AdamW; 1e-4 | app. B.2 | Same in MultiWOZ2.0 script train.sh |
| Fine-tuning warmup | not stated | — | MultiWOZ2.0 script uses 2000 steps train.sh |
| Fine-tuning epochs | not stated | — | MultiWOZ2.0 script uses 60 train.sh |
| Fine-tuning lengths | context 1024; response 100 | app. B.2 | Script max_len 1024 train.sh |
| Fine-tuning batch | MultiWOZ 32; In-Car 64 | app. B.2 | MultiWOZ script 32 train.sh; no In-Car script in repo |
| Fine-tuning dropout | MultiWOZ 0.1; In-Car 0.35 | app. B.2 | Code default 0.1 unified_transformer.py; In-Car value not in code |
| Fine-tuning seed | not stated | — | MultiWOZ2.0 script uses 10 train.sh |
| Hardware | not stated | — | Released scripts assume 8 GPUs train.sh |
Results
Author-reported. Setup: MultiWOZ2.0 and MultiWOZ2.1 end-to-end (E2E) with 8438/1000/1000 train/validation/test dialogs; In-Car with 2425/302/304 dialogs; delexicalised responses §5.1. Baselines are copied from their original papers, not re-run tab. 3 tab. 4. Comb = (Inform + Success) x 0.5
- BLEU on MultiWOZ and (Match + SuccF1) x 0.5 + BLEU on In-Car §5.2.
In-Car tab. 4
| Model | Match | SuccF1 | BLEU | Comb |
|---|---|---|---|---|
| SEDST | 84.50 | 82.90 | 19.30 | 103.00 |
| TSCP | 84.50 | 81.10 | 21.90 | 104.70 |
| LABES | 85.80 | 77.00 | 22.80 | 104.20 |
| FSDM | 84.80 | 82.10 | 21.50 | 104.95 |
| GALAXY (w/o pre-train) | 81.90 | 83.30 | 22.00 | 104.60 |
| GALAXY | 85.30 | 83.60 | 23.00 | 107.45 |
MultiWOZ E2E tab. 3
| Model | 2.0 Inform | 2.0 Success | 2.0 BLEU | 2.0 Comb | 2.1 Inform | 2.1 Success | 2.1 BLEU | 2.1 Comb |
|---|---|---|---|---|---|---|---|---|
| SimpleTOD | 84.40 | 70.10 | 15.01 | 92.26 | 85.00 | 70.50 | 15.23 | 92.98 |
| DoTS | 86.59 | 74.14 | 15.06 | 95.43 | 86.65 | 74.18 | 15.90 | 96.32 |
| SOLOIST | 85.50 | 72.90 | 16.54 | 95.74 | — | — | — | — |
| MinTL | 84.88 | 74.91 | 17.89 | 97.79 | — | — | — | — |
| PPTOD | 89.20 | 79.40 | 18.62 | 102.92 | 87.09 | 79.08 | 19.17 | 102.26 |
| UBAR | 95.40 | 80.70 | 17.00 | 105.05 | 95.70 | 81.80 | 16.50 | 105.25 |
| GALAXY (w/o pre-train) | 93.10 | 81.00 | 18.44 | 105.49 | 93.50 | 81.70 | 18.32 | 105.92 |
| GALAXY | 94.40 | 85.30 | 20.50 | 110.35 | 95.30 | 86.20 | 20.01 | 110.76 |
Headline gains over the best prior row, excluding the GALAXY without pre-training row: In-Car +2.5 Comb (FSDM), MultiWOZ2.0 +5.3 (UBAR), MultiWOZ2.1 +5.5 (UBAR) §6.1 tab. 3 tab. 4.
Comparison with other PCMs, MultiWOZ2.0 tab. 5: UniLM Comb 105.35; PLATO 100.88; TOD-UniLM 106.53; TOD-PLATO 102.98; GALAXY 110.35. TOD-UniLM and TOD-PLATO are the baselines retrained on UniDA and UnDial §6.2.
Other semi-supervised methods, MultiWOZ2.0 tab. 6: Pseudo-labelling Comb 101.99; VAE 99.18; GALAXY without the KL term (multi-task) 107.27; GALAXY 110.35 §6.3 app. C.1.
Low-resource, MultiWOZ2.0 Inform / Success / BLEU tab. 7. Rows marked * are the authors' re-implementations. Comb is not reported in this table.
| Model | 5% data | 10% data | 20% data | 50% data |
|---|---|---|---|---|
| DAMD | 56.60 / 24.50 / 10.60 | 62.00 / 39.40 / 14.50 | 77.90 / 70.30 / 12.10 | 83.00 / 72.90 / 16.90 |
| SOLOIST | 69.30 / 52.30 / 11.80 | 69.90 / 51.90 / 14.60 | 74.00 / 60.10 / 15.24 | — |
| MinTL | 75.48 / 60.96 / 13.98 | 78.08 / 66.87 / 15.46 | 82.48 / 68.57 / 13.00 | 90.10 / 78.60 / 17.90 * |
| PPTOD | 79.86 / 63.48 / 14.89 | 84.42 / 68.36 / 15.57 | 84.94 / 71.70 / 17.01 | — |
| UBAR | 73.04 / 60.28 / 16.03 * | 79.20 / 68.70 / 16.09 * | 82.50 / 66.60 / 17.72 * | 91.50 / 78.20 / 17.05 * |
| GALAXY | 80.59 / 67.43 / 17.39 | 87.00 / 75.00 / 17.65 | 89.55 / 75.85 / 17.54 | 93.35 / 82.35 / 18.37 |
Ablation, MultiWOZ2.0 Comb tab. 8: full GALAXY 110.35; without the gate 108.11; without the DA loss 103.27; without the KL term 107.27; without DA loss and KL 106.79 §7.
Oracle belief states, MultiWOZ E2E tab. 11: GALAXY Comb 110.18 (2.0) and 110.79 (2.1). Excluding the GALAXY without pre-training row, the best listed prior Comb is 109.4 (HDNO and JOUST, 2.0) and 106.87 (HDNO, 2.1) app. D.
Paper vs code
The code is at ee1cf33a0cabac31c8323d0bcbddb8bd7491852f (Apache-2.0). It
contains the pre-training model, the loss criterion and the pre-training and
MultiWOZ scripts. The processed data is not in the repo; the README links it on
Google Drive README.md.
- Response selection is missing. The paper's objectives include response selection §4.2 eq. 1 eq. 2. The pre-training loss in the released code has only the generation NLL, the DA BCE and the KL term pretrain_unified_transformer.py. A search of the repository for negative sampling and response selection found nothing.
- Gate entropy sign. The paper defines E as the sum of q log q eq. 9. That is negative entropy, and log E is undefined for it. The code uses positive entropy H = -sum(q log q) and computes (H_max - H - log H) / H_max, clipped to [0, 1] pretrain_unified_transformer.py. Use positive entropy.
- Gate detached. The gate is detached from the graph by default pretrain_unified_transformer.py pretrain_unified_transformer.py. The paper's equations do not state whether gradients pass through g §4.3.
- DA loss scale. Eq. 4 sums over the N acts. The code averages the BCE over the N acts, then averages over labelled rows pretrain_unified_transformer.py. Per row, the DA term is 1/N of the Eq. 4 sum, with N = 20. The code also averages over the labelled rows in the batch, which Eq. 4 does not do.
- Generation loss scale. Eq. 3 sums token NLL per response. With
token_loss true, which the released scripts set, the code divides the summed NLL by the number of target tokens in the batch pretrain_unified_transformer.py train_single.sh. - Single mixed-batch loss. Eq. 11 adds L_unlabel and L_label. Read literally, that counts L_RS and L_RG twice. The code computes one loss over the mixed batch, with the gate applied to the KL term pretrain_unified_transformer.py.
- KL weight. The equations give the KL term unit weight, but the released pre-training script sets kl_ratio 5.0 train_single.sh.
- Dropout consistency. The two passes come from concatenating the batch with itself model_base.py. Dropout is set for the whole model (0.3) rather than only for the consistency term train_single.sh.
- Batch size. The paper states 128 app. B.2. The script uses 32 train_single.sh. The script runs under distributed launch on eight processes, but whether 32 is per process is not stated in the paper or script.
- Checkpoint and seed. The paper picks epoch 14 with seed 11 app. B.2. The pre-training script trains 60 epochs train_single.sh and sets no seed. No epoch selection is in the code.
- Warmup. Not stated in the paper. The code uses 10% of total steps when the warmup flag is -1 pretrain_trainer.py.
- Unlabelled rows. Unlabelled rows carry an all-zero act vector pretrain_field.py, and the DA BCE is masked to tag 0 pretrain_unified_transformer.py.
- In-Car and CamRest. The README says the DA task is dropped for these two datasets because they have no useful acts README.md. The MultiWOZ2.0 script keeps it on train.sh. There is no In-Car script in the repo; its B.2 settings are not checked.
Beyond these points, the files we read (pre-training model, criterion, pre-training and MultiWOZ2.0 scripts, data builder) showed no further discrepancies. Not read: the MultiWOZ2.1, CamRest and KVRET scripts, the inference code, and the batch sampler that mixes UniDA and UnDial.
Pitfalls
- Pre-training data is large and dominated by Reddit. Reddit is about 31.9M of the 35.5M UnDial utterances tab. 10. The paper does not give GPU hours or memory.
- Two forward passes per batch. Each step runs the model twice for the consistency term, so compute per step is roughly double a standard pre-training step. The paper does not report the cost.
- Gate source and gradients. The gate is computed from the first pass only, while the KL term uses both passes. Whether gradients reach the gate is not stated in the paper; the released code blocks them by default.
- Epsilon in the gate. The paper gives no epsilon. The code uses 1e-12 inside the logarithms pretrain_unified_transformer.py. The gate formula has log of entropy, so zero entropy must be guarded.
- Few-shot claim. The text says GALAXY outperforms all baselines in every low-resource setting §6.4. At 20% data, the BLEU of UBAR is higher (17.72 vs 17.54) tab. 7. The UBAR and MinTL rows at 50% data are re-implementations.
- Few-shot pool. Low-resource runs remove the unused MultiWOZ training data from UniDA §6.4. Without that removal, pre-training would see data the low-resource setting is meant to exclude.
- Baseline numbers are copied, not re-run. Table 3 and Table 4 say results come from the original papers tab. 3 tab. 4. Different papers used different preprocessing, so Comb values are not strictly comparable.
- Data licences. Data is downloaded from links in the README. The corpora come from other sources, and their licences are not checked here.
- The DA label set is a merge. Acts from eight corpora are mapped into one
set of 20 app. A.1 tab. 1. The paper describes the mapping in prose only.
We did not check the repo's
tools/mapping.pairagainst it.
Lineage
- Builds on: UniLM as the backbone §4.1; the token, role, turn and position embedding scheme of PLATO §4.1; a bidirectional KL term from the paper's reference liang2021r §4.2; dropout-based representations as in the paper's reference gao2021simcse §4.2.
- Compared against: UniLM and PLATO retrained on the same data as TOD-UniLM and TOD-PLATO §6.2; pseudo-labelling, VAE and multi-task baselines §6.3; SimpleTOD, DoTS, SOLOIST, MinTL, PPTOD, UBAR (MultiWOZ) and SEDST, TSCP, LABES, FSDM (In-Car) tab. 3 tab. 4.
- Superseded by: not stated in the wiki's content.
Source and license
Adapted from “GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injection” by Wanwei He, Yinpei Dai, Yinhe Zheng, Yuchuan Wu, Zheng Cao, Dermot Liu, Peng Jiang, Min Yang, Fei Huang, Luo Si, Jian Sun, Yongbin Li (arXiv:2111.14592), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/siat-nlp/GALAXY at commit ee1cf33a0cab (Apache-2.0). Any quoted code is shown under that license.