MS-MDA: Multisource Marginal Distribution Adaptation for Cross-subject and Cross-session EEG Emotion Recognition

In plain words

An EEG headset reads brain signals, and the same emotion looks different in every person and on every day the headset is worn. A model trained on earlier people or sessions often does badly on a new one. The usual fix is to pool all the old recordings into one big training set and adapt that pool to the new person. This paper argues that the pool blurs the differences between people and sessions, which are the very thing that makes each recording distinct.

MS-MDA keeps each old person or session as its own source. A small shared network first extracts features common to all recordings. Then each source gets its own small branch that is nudged to look like the new person's data and has its own classifier. A penalty keeps the branches' answers close to one another. For a new recording, the model averages what all the branches say. The authors also find that how the signal is normalised changes results a lot.

TL;DR

• Multi-source domain adaptation for EEG emotion recognition: one branch per source subject or session, each aligned one-to-one with the target §I fig. 3.
• Input is a 310-dimensional vector of differential entropy features (62 channels × 5 bands) §III-D. The network is a shared 3-layer MLP plus small per-branch layers §IV §V-A.
• Author-reported: 88.56 ± 7.80 (cross-session) and 89.63 ± 6.79 (cross-subject) on SEED, 3 classes tab. II.
• Code is MIT-licensed and linked from the paper, but it differs from the paper in several places that change the loss, the optimiser and the reported numbers. See Paper vs code.

Problem

Signals from one EEG session and from one subject have non-stationary, biased distributions, so a model does not transfer across them without adaptation §I. Labelled data is costly to collect. Earlier domain-adaptation work for EEG usually merged all source subjects or sessions into one source domain before aligning it with the target §I §II. The authors argue this breaks the marginal distribution of each source and makes the merged distribution hard to interpret, so the adaptation becomes harder §I.

Method

Setting. Inputs are N labelled source domains, each a subject (cross-subject) or a session (cross-session), and one unlabelled target domain. The target labels are used only for testing §IV alg. 1.

Architecture fig. 3 §IV:

1. Common feature extractor (CFE). Shared by all domains. It maps the 310-dimensional input to a 64-dimensional common feature, using a 3-layer MLP with a LeakyReLU after each linear layer §IV §V-A.
2. Domain-specific feature extractor (DSFE), one per source. A single linear layer maps 64 to 32, followed by LeakyReLU §IV §V-A. Each DSFE sees the common features of its own source and of the target.
3. Domain-specific classifier (DSC), one per source. A single linear layer maps 32 to the number of classes (3 for SEED, 4 for SEED-IV), with no activation §V-A.

Losses. Each branch i compares its source features with the target features using MMD eq. 2. The classifiers are trained with cross-entropy on their own source eq. 3. A discrepancy term penalises disagreement between the branches' predictions on the target eq. 4. The total loss is a weighted sum eq. 5 alg. 1.

\operatorname{MMD}_{\left(X^{S}, X^{T}\right)}=\left\|\frac{1}{N^{S}} \sum_{i=1}^{N^{S}} \Phi\left(\mathrm{x}_{i}^{S}\right)-\frac{1}{N^{T}} \sum_{j=1}^{N^{T}} \Phi\left(\mathrm{x}_{j}^{T}\right)\right\|_{\mathcal{H}}^{2} eq. 2

\mathcal{L}_{c l s}=\sum_{i=1}^{N} \mathbf{E}_{x \sim X_{S}} J\left(\hat{\mathbf{Y}}_{i}^{S}, \mathbf{Y}_{i}^{S}\right) eq. 3

\mathcal{L}_{disc} = \sum_{i \neq j}^{N} \mathbf{E}_{x \sim X_{t}} \left|\hat{\textbf{Y}}_{i}^{T}-\hat{\textbf{Y}}_{j}^{T}\right| eq. 4

\mathcal{L} = \mathcal{L}_{cls} + \alpha \mathcal{L}_{mmd} + \beta \mathcal{L}_{disc} eq. 5

The paper uses a weight of 0.01 for the discrepancy term relative to α, applied over the whole training run tab. V §V-E. The α schedule is given in §V-A. Training uses the loop in Algorithm 1 alg. 1. The pseudocode below follows the paper's text and equations. The code differs in several places (see Paper vs code).

# Training loop of MS-MDA  [alg. 1] [§IV]
# Input:  N labelled source sets (X_i, Y_i) and one unlabelled target X_T
# Output: N branches; prediction = average of their softmax outputs

def train(sources, target, num_iterations, batch_size, lr):
    # Build the shared extractor and one branch per source  [§IV] [§V-A]
    cfe = common_feature_extractor()                       # 310 -> 64
    dsfe = [domain_specific_extractor() for _ in sources]  # 64 -> 32
    dsc = [linear(32, num_classes) for _ in sources]       # 32 -> C
    optimizer = adam(all_params, lr=lr)                    # lr 0.01 [§V-A]

    for step in range(1, num_iterations + 1):
        # Ramp the MMD weight alpha from 0 towards 1 [§V-A]
        alpha = 2 / (1 + exp(-10 * step / num_iterations)) - 1
        # Discrepancy weight is 0.01 times alpha [tab. V] [§V-E]
        beta = 0.01 * alpha
        # Draw m samples per source and m unlabelled target samples [alg. 1]
        source_x, source_y = sample_per_source(sources, batch_size)
        target_x = sample_target(target, batch_size)

        # Common features for every domain [§IV] [alg. 1]
        q_src = [cfe(x) for x in source_x]
        q_tgt = cfe(target_x)

        # Domain-specific features, one branch per source [§IV] [alg. 1]
        r_src = [dsfe[i](q_src[i]) for i in range(num_sources)]
        r_tgt = [dsfe[i](q_tgt) for i in range(num_sources)]

        # MMD between each source branch and the target [eq. 2]
        mmd = 0
        for i in range(num_sources):
            mmd += mmd_distance(r_src[i], r_tgt[i])

        # Cross-entropy on each source branch [eq. 3]
        cls = 0
        for i in range(num_sources):
            cls += cross_entropy(dsc[i](r_src[i]), source_y[i])

        # Pairwise gap between the target predictions [eq. 4]
        pred_tgt = [dsc[i](r_tgt[i]) for i in range(num_sources)]
        disc = 0
        for i in range(num_sources):
            for j in range(num_sources):
                if i != j:
                    disc += mean(abs(pred_tgt[i] - pred_tgt[j]))

        # Total loss, then one gradient step [eq. 5] [alg. 1]
        loss = cls + alpha * mmd + beta * disc
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

# Inference: average the N branch predictions [§IV] [alg. 1]
def predict(x):
    q = cfe(x)                             # (B, 64)
    probs = []
    for i in range(num_sources):
        r = dsfe[i](q)                     # (B, 32)
        probs.append(softmax(dsc[i](r)))   # (B, C)
    return mean(probs)                     # (B, C)

Hyperparameters

Results

Author-reported. Setup: SEED (15 subjects, 3 sessions, 3 classes) and SEED-IV (4 classes), DE features, shared-MLP model. Rows without a citation (DDC, DAN, DCORAL) were run by the authors with their settings; rows with a citation in tab. II are, per the text, copied from the original papers §V-B. Model size, hardware and training time are not stated. The ± values are standard deviations as described in §III-B. Accuracy in %.

The text says the margins are about 7% and 3% on SEED and about 7% and 18% on SEED-IV, for cross-session and cross-subject §V-B. Against the best baseline in tab. II, the SEED margins are 7.0 and 2.9 points. For SEED-IV the table gives 3.8 points (cross-session, against DDC) and 21.9 points (cross-subject, against DCORAL, the best of the three baselines there). The SEED-IV numbers in the text do not match the table.

Ablation (SEED and SEED-IV) tab. III

On SEED-IV cross-session, the variant without both losses scores higher than the full model tab. III.

Normalisation study tab. IV

Three normalisations are compared: electrode-wise, sample-wise and global-wise fig. 4. Two orders are compared for DAN: normalise and then concatenate (order A, DAN¹) or concatenate and then normalise (order B, DAN²) fig. 5 §V-D.

Discrepancy weight study tab. V

Training percentage 1 means the loss is used over the whole run, 0.2 means only the last 20% of training. Weight is the ratio to α tab. V caption. The first row has no discrepancy loss.

The chosen setting (whole run, weight 0.01) is the one reported in tab. II §V-E.

Other experiments

• Batch size and epochs: both models drop as batch size grows; the drop is largest at batch 512 on SEED-IV. Longer training gives little gain, but MS-MDA converges faster §V-E fig. 6 fig. 7.
• t-SNE of raw, normalised, DAN-adapted and MS-MDA-adapted features, cross-subject case, target shown in black fig. 8 §V-E.

Paper vs code

Code: VoiceBeer/MS-MDA at commit a2946c38dc6b8d75e9a81bc1aa3651b1cb70f240 (MIT). The README notes that its LOSO runs differ from the paper's setup and reports separate numbers for them README.md.

• Discrepancy term runs on the wrong tensors. The paper's eq. 4 uses the classifier predictions Ŷᵀ. The code applies softmax to the 32-dimensional DSFE features and takes the mean absolute difference, so classifiers are not compared on their predictions. Paper: eq. 4. Code: F.softmax(data_tgt_DSFE[...]) models.py. The code averages over samples and classes. eq. 4 writes an absolute value with an expectation and does not say whether the class dimension is summed or averaged, so the scale may differ.
• MMD kernel. The paper's eq. 2 is an RKHS distance. The code uses the linear-kernel form mmd_linear, which equals the squared distance between the two batch means utils.py. The Gaussian multi-kernel MMD is in the same file but commented out msmdaer.py.
• Optimiser state and update schedule. The paper's Algorithm 1 does one update per iteration alg. 1. The code makes one optimiser step per source per iteration, and it creates a new Adam optimiser at the start of each iteration, which resets Adam's moments every iteration msmdaer.py msmdaer.py.
• Batch size default. The paper states 256 §V-A. The main script's default is 128 msmdaer.py. The ablation script's default is 256 msmdaer_wodisc.py.
• Number of iterations. The code computes iterations as ceil(epoch × sample count / batch), using 3394 for SEED and 820 for SEED-IV msmdaer.py. The paper gives 851, 832 and 822 samples for the three SEED-IV sessions §III-D.
• α schedule. The paper's i is undefined. The code uses the step index over the total number of iterations msmdaer.py. Paper: i/epoch §V-A.
• Learning rate. The paper gives an initial lr of 0.01 §V-A. The code keeps it constant; the decay formula is commented out msmdaer.py.
• DSFE has BatchNorm. The paper describes a linear layer and LeakyReLU §V-A. The code adds BatchNorm1d models.py.
• Source / target split. The paper says cross-session uses the first two sessions as sources and cross-subject uses the first 14 subjects §III-B. The code runs leave-one-out: every subject (cross-subject) and every session (cross-session) takes a turn as target msmdaer.py msmdaer.py msmdaer.py. That gives 45 runs per scenario, and the mean and std are taken over all 45 runs msmdaer.py. The paper says cross-session results are averaged over 15 subjects and cross-subject over 3 sessions §III-B.
• Normalisation. The paper calls the best method "electrode-wise" §V-D. The code min–max normalises each of the 310 feature columns separately, per subject and session matrix, before training msmdaer.py utils.py. Target data are included in these statistics.
• Test accuracy. The code evaluates the target set every 200 iterations and returns the highest accuracy seen msmdaer.py. The paper does not say how the reported checkpoint is chosen.
• Test batching. The target loader uses drop_last=True and is also the loader used by test msmdaer.py msmdaer.py. Accuracy is divided by the full target size msmdaer.py, so the dropped samples count as errors.
• Discrepancy weight in an extra script. The main script uses β = γ/100 msmdaer.py. The additions script uses γ/1000 msmdaer_additions.py.
• Baselines. The code includes separate DDC, DAN and DCORAL files DDC.py DAN.py DCORAL.py. The paper says some baseline numbers are copied from original papers §V-B.

No discrepancies found in the CFE linear layers (310 → 256 → 128 → 64), the DSC size, the Adam choice, or the 0.01 discrepancy ratio.

Pitfalls

• Discrepancy loss is the most fragile term. The text says it must be tuned carefully and can hurt when most classifiers are wrong §VI. Table V shows that its effect depends on the schedule and dataset. On SEED-IV cross-session the best row is 0.2 / 0.1 (64.75), not the chosen setting (61.43) tab. V.
• Hyperparameters were picked on the reported test numbers. The discrepancy weight is chosen from tab. V, which reports test accuracy on the same target sets §V-E. Normalisation is chosen the same way §V-D. Expect the headline numbers to be optimistic.
• Checkpoint selection on the target. The code returns the best checkpoint accuracy on target labels (see Paper vs code). Do not compare these numbers with methods that use a fixed stopping rule without saying so.
• Standard deviations are large. Several cells have ±10 or more on SEED-IV tab. II tab. III. Differences of 1 to 2 points are within noise. The paper does not state how many seeds were run.
• Cost grows with the number of sources. Every iteration passes the target through all N branches, and the authors report training time that grows linearly with N §VI. Cross-subject uses 14 sources and cross-session 2 §V-B.
• Band list is incomplete. The text lists four frequency bands but says there are five. The fifth range is not stated §III-D.
• Figure and table numbers. IEEE prints section and table numbers as Roman numerals and subsection letters, so this card cites them that way (for example §V-B, tab. II). Figure and equation numbers are arabic.
• Text margins do not match the table on SEED-IV (see Results).
• Unclear pieces that the code resolves differently or not at all: the MMD weight's ramp variable i, whether the MMD terms are summed over branches (the code does, by stepping through them), the LeakyReLU slope, the Adam betas, and the random seed.

Lineage

• Builds on: DAN (Long et al., 2015) and its MMD-based adaptation §II; the multi-source style transfer mapping of Li et al. (2019), which is similar but has no domain-invariant features §II; the multi-source aligning idea of Zhu et al. (2019), which motivates the one-to-one branches §II.
• Compared against: DDC, DAN, DCORAL, DANN and PPDA on SEED and SEED-IV tab. II.
• Superseded by: not known from the content of this wiki.