wikillm

Paper · 2024-07-22 · cs.LG, cs.AI

Enhancing Cognitive Workload Classification Using Integrated LSTM Layers and CNNs for fNIRS Data Analysis

A lightweight 1D CNN followed by two LSTM stages classifies 30-second fNIRS windows from the Tufts dataset into four n-back workload levels. Author-reported test accuracy is about 97.9%, against 97.4% for the same CNN without LSTMs. The released code feeds the LSTMs a single time step, and the checkpoint behind the reported numbers is not in the repository.

Convolutional neural networkLong short-term memory (LSTM)Time series classificationfnirsneuroimagingcognitive-workloadtime-series

arXivPDFCodeMarkdown12 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Functional near-infrared spectroscopy (fNIRS) shines light into the forehead and measures how much oxygen the blood carries there. A busy brain changes those levels, so the signal hints at how hard a person's working memory is working. This paper uses that signal to tell apart four levels of a memory game (the n-back task), labelled 0-back to 3-back.

The model has two parts. A small convolutional part scans the signal for short local patterns. Then a pair of LSTM layers, which are built to follow things over time, process the result. The authors report that adding the LSTM layers gives a small accuracy gain over the same network without them.

Read the code side with care. The published code does not feed the LSTM layers a sequence over time, and the best model the paper reports cannot be reloaded from the repository. The numbers are the authors' own and have not been reproduced here.

TL;DR

  • Task: classify 30-second fNIRS windows from the Tufts dataset into 0-, 1-, 2- or 3-back workload §1 §4 §4.1.
  • Model: 1D CNN (two conv-pool blocks) followed by two LSTM stages and a dense head tab. 1 models_finals.ipynb.
  • Author-reported: 97.89% test accuracy, against 97.40% for the same CNN without LSTMs tab. 2.
  • Problems: the code runs the LSTMs over one time step, the reported checkpoint is not in the repo, and the test set drives checkpoint selection. See Paper vs code.

Problem

fNIRS measures changes in oxygenated (HbO) and deoxygenated (HbR) haemoglobin by sending near-infrared light into the scalp, mostly over the forehead, and reading how much returns §1. Cognitive load is the amount of information working memory must hold §1. Earlier fNIRS studies mostly separated two load levels, often on hand-crafted features, and the paper argues that the four-level Tufts dataset is under-used §1 §2. Classical classifiers such as Naive Bayes, Nearest Centroid, Decision Trees and k-NN are the paper's baselines §2 tab. 2. The paper also argues that CNNs alone tend to overfit and miss temporal dependencies, which motivates adding LSTMs §1 §4.2.

Method

The model is a 1D CNN whose flattened output feeds two LSTM stages. Each window has 150 time steps of eight measurements §4.1. Two conv-pool blocks reduce the length from 150 to 75 and then to 37 tab. 1. The paper's Table 1 shows two LSTM stages (each with hidden size 64), a 64→128 dense layer with ReLU, and a 128→4 output tab. 1. The code uses two LSTM layers per stage models_finals.ipynb.

The convolution and pooling operations are standard eq. 1 eq. 2 eq. 3. The paper's LSTM equations eq. 4–eq. 9 use the usual input, forget and output gates, a candidate cell state, and a hidden state. They also state that the gate weights are 2D convolution kernels §3.1. The code does not use convolutional gates; it uses the standard nn.LSTM models_finals.ipynb. See Paper vs code.

The paper has no algorithm box. The pseudocode below is reconstructed from tab. 1, §4.1, §4.2 and the notebook. It follows the architecture as the code runs it.

Pseudocode
# Input: a batch of windows, shape (B, 150, 8) [§4.1]
#        B = batch size; 150 time steps; 8 measurements per step
# Output: logits for 4 classes (0-back, 1-back, 2-back, 3-back) [tab. 1]

def forward(windows):
    # Channels first for Conv1d: (B, 8, 150)  [code: models_finals.ipynb#L22138]
    x = transpose(windows, 1, 2)
    # Conv1d 8 -> 16, kernel 3, padding 1, then ReLU  [tab. 1] [code: #L22115]
    x = relu(conv1(x))                         # (B, 16, 150)
    # Max pooling, kernel 2, stride 2  [tab. 1] [code: #L22117]
    x = max_pool(x)                            # (B, 16, 75)
    # Conv1d 16 -> 32, kernel 3, padding 1, then ReLU  [tab. 1] [code: #L22119]
    x = relu(conv2(x))                         # (B, 32, 75)
    # Max pooling, kernel 2, stride 2  [tab. 1] [code: #L22121]
    x = max_pool(x)                            # (B, 32, 37)
    # Flatten channels and steps into one vector  [tab. 1] [code: #L22123]
    x = flatten(x)                             # (B, 1184) = 32 * (150 // 4)
    # The code gives the LSTM ONE time step; see Paper vs code
    # [code: models_finals.ipynb#L22147]
    x = reshape(x, (B, 1, 1184))
    # First LSTM stage: hidden 64, 2 layers  [tab. 1] [code: #L22127]
    x = lstm_stage_1(x)                        # (B, 1, 64)
    # Second LSTM stage: hidden 64, 2 layers  [tab. 1] [code: #L22128]
    x = lstm_stage_2(x)                        # (B, 1, 64)
    # Keep the output of the last time step  [code: models_finals.ipynb#L22156]
    x = x[:, -1, :]                            # (B, 64)
    # Dense 64 -> 128, ReLU, then dense 128 -> 4  [tab. 1] [code: #L22131]
    x = relu(fc1(x))                           # (B, 128)
    return fc2(x)                              # (B, 4)

Training uses Adam with a learning rate of 0.001, cross-entropy loss, batches of 1024 and 1000 epochs §4.2 §4.1. The code evaluates on the test split after every epoch and saves a checkpoint when test accuracy or test loss reaches a new best models_finals.ipynb models_finals.ipynb.

Pseudocode
# Training: Adam, lr 0.001, cross-entropy, batch 1024, 1000 epochs
# [§4.2] [§4.1] [code: models_finals.ipynb#L21658-L21660]
optimizer = adam(model.parameters(), learning_rate=0.001)
loss_fn = cross_entropy

for epoch in range(1000):
    # Shuffle the training windows and visit them in batches of 1024
    # [§4.1] [code: models_finals.ipynb#L433-L434]
    for batch_x, batch_y in shuffled_batches(train_set, batch_size=1024):
        logits = model(batch_x)
        loss = loss_fn(logits, batch_y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    # Evaluate on the TEST split after every epoch; there is no validation split
    # [code: models_finals.ipynb#L21716-L21762]
    test_loss, test_acc = evaluate(model, test_set)
    # Save a checkpoint when test accuracy or test loss hits a new best
    # [code: models_finals.ipynb#L21763-L21769]
    if test_acc >= best_test_acc or test_loss <= best_test_loss:
        save_checkpoint(model)

Figures are described here and not reproduced. fig. 2 is the block diagram. fig. 3 shows training and test curves for the CNN-LSTM model. fig. 4 is its confusion matrix. fig. 5 and fig. 6 show the same for the CNN without LSTMs. fig. 7 shows the classical classifiers.

Hyperparameters

NameValueWhereNotes
Window length30 s, overlapping, stride 0.6 s§4.1Windowing code is not in the repository
Time steps per window150§4.1150 steps over 30 s; the sampling rate is not stated
Measurements per step8§4.1The paper does not name the eight measurements
Band-pass filter3rd-order zero-phase Butterworth, 0.001–0.2 Hz§4Filter code is not in the repository
Input normalisationnot stated§4.1No scaling step is visible in the notebook
Splitrandom 80/20, random_state 10§4.1 models_finals.ipynb80,947 train and 20,237 test samples §4.1
Batch size1024§4.1 models_finals.ipynbTrain and test loaders both shuffle
Epochs1000§4.1 models_finals.ipynbNo early stopping in the code
OptimiserAdam, lr 0.001§4.2 models_finals.ipynbBetas and epsilon are not set in the code
Losscross-entropy over 4 classes§4.2 models_finals.ipynb
Conv18→16, kernel 3, stride 1, padding 1tab. 1 models_finals.ipynbKernel and padding are not in the paper
Conv216→32, kernel 3, stride 1, padding 1tab. 1 models_finals.ipynb
Poolingmax, kernel 2, stride 2 (both blocks)tab. 1 models_finals.ipynb150→75→37 with floor division
LSTM stages2 stages, hidden 64, 2 layers eachtab. 1 models_finals.ipynbLayer count is only in the code
Dense head64→128, ReLU, 128→4tab. 1
Dropoutnone§4.2Confirmed by the code (no dropout layer)
Learning-rate schedule, weight decay, early stopping, random seednot stated§4.2None present in the notebook

Results

Author-reported. Setup: the open Tufts fNIRS dataset with 68 participants and four n-back levels §4 §1. One random 80/20 split with random_state 10 §4.1. Hardware is an Intel i9-12900K CPU and an NVIDIA RTX 3090 Ti GPU §4.2. The baseline is the same CNN without the LSTM stages tab. 2.

Table 2 (author-reported; all rows cite tab. 2):

ClassifierAccuracyF1AUCPrecisionRecallSource
Naive Bayes0.33730.28590.57120.36710.3373tab. 2
Nearest Centroid0.33250.32070.58030.33160.3325tab. 2
Decision Trees0.69720.69720.61050.69720.6972tab. 2
k-NN0.97020.97020.66280.97020.9702tab. 2
CNN without LSTM layers0.97400.97400.66380.97400.9740tab. 2
Proposed CNN-LSTM0.97890.97890.67040.97900.9789tab. 2

The paper gives three different headline figures for the proposed model: 97.92% in the abstract, 97.82% in §5 §5, and 0.9789 in Table 2 tab. 2. Summing the diagonal of the CNN-LSTM confusion matrix (5020 + 4936 + 4982 + 4874 = 19,812) over 20,237 test samples gives 0.9790. That matches Table 2 fig. 4. This is our arithmetic on the paper's figure, not a separate result.

Timing, author-reported: training took 7.79 min for the CNN and 9.76 min for the CNN-LSTM tab. 3. Mean inference time over 100,000 timed runs after 1000 warm-up runs §4.2:

SettingMean ± SD (ms)Min (ms)Max (ms)Source
GPU, CNN0.0891 ± 0.02660.03170.7516tab. 4
GPU, CNN-LSTM0.2153 ± 0.04130.0871.4888tab. 4
CPU, CNN0.21535 ± 0.041320.0871.4888tab. 4
CPU, CNN-LSTM0.83560 ± 0.14630.6432.4185tab. 4

Paper vs code

The repository is https://github.com/mehshankhan/fNIRS_tufts_CNN_LSTM at commit d7a735da37488a68a1e2a1b4afb7a45f3239544b. It is linked from the paper's data-availability statement §5. It has one notebook (models_finals.ipynb) and a README. It has no LICENSE file, so no code is quoted here.

  • The LSTM sees one time step. The paper says the LSTMs capture temporal dependencies across the window §3.2 §5. The code flattens the CNN output into one vector, reshapes it to (batch, 1, 1184), and runs the LSTMs over that single step models_finals.ipynb models_finals.ipynb. It keeps the output at that step models_finals.ipynb. As written, the LSTM stages cannot model order across the 150 time steps.
  • Gates are dense, not convolutional. The paper's gate equations use convolution kernels §3.1 eq. 4–eq. 9. The code uses nn.LSTM models_finals.ipynb.
  • The reported checkpoint is not in the repository. The CNN-LSTM evaluation loads results_final/folder2/.../epoch_759_tacc0.98_tloss_0.11.pth from a path on the author's machine models_finals.ipynb. The file is not in the repository.
  • The notebook's own log does not reproduce the claim. The only 1000-epoch training log follows the CNN-LSTM instantiation models_finals.ipynb. Its test accuracy reaches 0.9982 at epoch 100 models_finals.ipynb. At epoch 300 it is 0.2487, which is chance for four classes, with training and test loss at about 1.3863, the value for uniform guessing over four classes models_finals.ipynb. It stays there until epoch 999 models_finals.ipynb. At epoch 759, the epoch in the checkpoint's name, the logged test accuracy is 0.2487 models_finals.ipynb. The paper's Figure 3 shows accuracy near 0.98 for the whole run fig. 3. Figure 3 does not match the notebook's log.
  • Checkpoint selection uses the test set. The code saves a checkpoint whenever test accuracy or test loss hits a new best models_finals.ipynb. The reported model is therefore picked on the same split it is scored on. The paper says nothing about a validation split or early stopping §4.2.
  • The CNN baseline is also not reproducible from the repository. Its checkpoint, named epoch_965_tacc0.97, is loaded from a results_final/folder1 path models_finals.ipynb. The notebook prints an accuracy of 0.97406 for it models_finals.ipynb. No training log for that run is in the repository.
  • The preprocessing is missing. The notebook loads features from a pickle produced by featues_save.py models_finals.ipynb. That script is not in the repository. The band-pass filter, the windowing and the construction of the 150×8 input cannot be checked.
  • Matching details. The layer sizes in Table 1 match the code tab. 1 models_finals.ipynb. The code's flattened size is 32 × (150 // 4) = 1184 models_finals.ipynb. The paper writes L/4, which is 37.5 for L = 150 tab. 1. The sample counts match: training classes are about 20,200 each and total 80,947; test classes total 20,237 §4.1 models_finals.ipynb.
  • Stale comment. The notebook comment says the training split is 70% models_finals.ipynb. The code uses test_size = 0.20 models_finals.ipynb, which matches the paper's 80/20 §4.1.
  • Notebook syntax error. The first copy of the model class has nn.ReLU()32 models_finals.ipynb. That line is not valid Python. The second copy, at lines 22110 onward, is clean and is the one cited here.
  • Stale comments. Some cells say "ResNet-18" and num_classes = 10 while building the CNN models_finals.ipynb models_finals.ipynb. These are copy-paste leftovers.
  • Timing claims have no code. The notebook has no timing code: no perf_counter, timeit or time.time. Tables 3 and 4 cannot be checked. In Table 4 the CPU CNN minimum and maximum (0.087 and 1.4888) are identical to the GPU CNN-LSTM row tab. 4. This could be a copy error.
  • Timing conclusion. §5 says adding the LSTMs "did not significantly increase the inference time" §5. Table 4 gives a mean of 0.0891 ms for the GPU CNN and 0.2153 ms for the GPU CNN-LSTM, about 2.4 times as long tab. 4.
  • Decision Trees. The text says Decision Trees reach 69.72% on "accuracy, F1-score, AUC, precision, and recall" §4.2. Table 2 gives an AUC of 0.6105 tab. 2.
  • AUC is unexplained. AUC values of 0.57–0.67 sit far below accuracies of 0.97 tab. 2. The paper does not state how AUC was computed. The notebook has no AUC code, and no roc_auc call appears in it.
  • Ablation. The paper calls the no-LSTM result "marginally lower" §5. The gap is 0.9789 against 0.9740 tab. 2, about half a point, from one split with no seeds or variance. The paper says results were "consistent across multiple runs" §4.1, but gives no run count or spread.
  • Code licence. The repository has no LICENSE file and the README states no licence. The code cannot be quoted and must be implemented from the paper and the pseudocode above.

Pitfalls

  • The LSTM stages do not model order over time as the code runs. The one-step input means their contribution is a gated dense transform of the flattened CNN features. If you want the temporal modelling the paper describes, you must keep the time axis, for example by passing the 37 pooled steps to the LSTM. That is a design choice the paper does not make, so do not assume it.
  • Reproduce with a validation split. The published checkpoint was picked on test accuracy or test loss models_finals.ipynb. Hold out a validation split from the training data and report test results once.
  • Expect training collapse without guards. In the notebook log, training loss is already about 0.0002 by epoch 25 models_finals.ipynb, then the run falls to chance by epoch 300 models_finals.ipynb. The code has no scheduler, weight decay, dropout or early stopping. The paper's explanation for avoiding overfitting, which relies on max pooling and a small learning rate §4.2, is qualitative and not tested.
  • Random windowing leaks. Windows overlap heavily (30 s windows with a 0.6 s stride), and the split is random over windows §4.1. Near-duplicate windows can therefore sit in both train and test. This is an inference from the paper's stated method, not something the paper reports. Split by participant or by session.
  • Floor division and the flattened size. Two stride-2 poolings on 150 steps give 37, not 37.5. The flattened size is 32 × 37 = 1184 models_finals.ipynb. Anything that changes the window length must recompute this.
  • Channel order. Conv1d expects channels first. The code permutes the (batch, time, measurement) input before the first convolution models_finals.ipynb.
  • Input scaling and the eight measurements are not stated. Neither the paper nor the code says how the eight per-step values are scaled or what they are §4.1.
  • Hardware-specific timings. Table 4 numbers come from one i9-12900K and RTX 3090 Ti setup §4.2. Do not use them as expected speeds on other hardware.
  • Class balance. The training counts are near-equal across the four levels models_finals.ipynb. The paper reports only overall and weighted scores, not per-class results tab. 2.

Lineage

  • Builds on: standard 1D CNN blocks eq. 1–eq. 3 and LSTM cells eq. 4–eq. 9 §3.1.
  • Compared against: Naive Bayes, Nearest Centroid, Decision Trees and k-NN tab. 2, and the same CNN without LSTM stages tab. 2 §5.
  • Dataset: the open Tufts fNIRS mental-workload dataset §4, cited as reference [32] in the paper, which gives no arXiv id.
  • Superseded by: not known.

Source and license

Adapted from “Enhancing Cognitive Workload Classification Using Integrated LSTM Layers and CNNs for fNIRS Data Analysis” by Mehshan Ahmed Khan, Houshyar Asadi, Mohammad Reza Chalak Qazani, Adetokunbo Arogbonlo, Siamak Pedrammehr, Adnan Anwar, Asim Bhatti, Saeid Nahavandi, Chee Peng Lim (arXiv:2407.15901), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/mehshankhan/fNIRS_tufts_CNN_LSTM at commit d7a735da3748 (none stated (no LICENSE file in repository)). Any quoted code is shown under that license.