Evaluation of Conversational Agents: Understanding Culture, Context and Environment in Emotion Detection

In plain words

Chatbots and voice assistants need to notice how a person feels, but most emotion-recognition systems were built on data from a narrow set of people. This paper asks what goes wrong when the same tools meet faces and voices from Black African communities, and it builds a simple emotion detector that uses both the face and the voice.

The detector reads the face in each video frame and the sound in each short clip, and gives each a ranked list of likely emotions. A small extra step then checks whether the two lists agree. When the face says one thing and the voice says something very different, the system takes that as a sign of sarcasm.

The authors report high accuracy on their mixed test set, but the paper has several internal inconsistencies, so treat the headline numbers with care.

TL;DR

• Multimodal emotion classifier: a 3-block CNN for face images and a 3-block CNN for audio features, each trained on seven emotion classes §III.B §III.D.
• The added Audio-Frame Mean Expression (AFME) step compares the top-3 emotions from each modality on short sub-clips and reports sarcasm when no emotion matches §III.E alg. 1.
• Author-reported accuracy of the CNN+AFME model is 95.7% on a mix of FER-2013, RAVDESS, SAVEE, TESS and local data, against 70.14 to 95.6 for the compared models tab. 3.
• Per-class results are internally inconsistent between Table 1 and Table 2, and the sarcasm claim rests on one YouTube example §III.F fig. 10.
• No official code was found.

Problem

Emotion detectors built mostly on public image sets can fail on faces and voices they did not see, and the authors argue this matters for conversational AI used in African settings §I. Their stated gaps are diversity, micro expressions, and real-time detection §II.

Single-modality systems miss emotions that a face shows but a voice does not, or the reverse. Sarcasm is the case the authors single out, because the words, the tone and the face can disagree §III.G.

The method assumes that each modality yields a usable ranking of emotions for every short sub-clip. The paper does not test the other direction, where the two modalities agree on a wrong emotion §III.E.

Method

Overview §III. The model combines two CNN classifiers, one trained on image frames and one on audio features, then applies AFME on top of their outputs. Both channels predict the seven basic emotions: anger, disgust, fear, happy, sad, surprise and neutral §III.B tab. 1 tab. 2.

Image branch §III.B §III.C §III.D.

1. Images are loaded from FER-2013 (48 x 48 grayscale) and from locally collected images, which are converted to grayscale with the weighted sum written in the paper §III.C.
2. Face landmarks are found with the Dlib predictor (68 points); the paper keeps 5 points per brow, 8 per eye, 20 for the mouth and 17 for the jaw §III.C fig. 5.
3. A 3-block CNN classifies the frames. Each block is a convolution, batch normalisation, ReLU, 2 x 2 max-pooling and dropout, in that order. Block width starts at 64 and grows by a factor of two in later blocks §III.D §IV.A.

# Image branch: one convolutional block, applied three times  [§III.D]
# Input:  frames (B, 1, 48, 48) grayscale; kernel size not stated
# Output: features (B, units, H', W')

def conv_block(frames, units):
    # 2D convolution; kernel size and padding not stated  [§III.D]
    features = conv2d(frames, units)
    # Batch normalisation on the convolution output  [§III.D]
    features = batch_norm(features)
    # ReLU activation  [§III.D]
    features = relu(features)
    # Max pooling with pool size 2  [§III.D]
    features = max_pool(features, size=2)
    # Dropout; rate not stated  [§III.D]
    features = dropout(features)
    return features

# Stack: units = 64, then 128, then 256 (paper: "multiple of two";
# exact widths for blocks 2 and 3 are not stated)  [§IV.A]

Audio branch §III.B §III.E.

1. Speech clips are converted to WAV, then transformed into a time-frequency representation and passed on for speech recognition and text classification §III.C fig. 6 fig. 7.
2. Audio features are computed per clip: zero-crossing rate, a 12-bin chroma vector, chroma deviation, MFCCs, root-mean-square energy, and a mel spectrogram §III.E.
3. A CNN classifies the audio features. The paper says LSTM and CNN were both trained with different hyperparameters, but no LSTM results are reported §III.A §III.D.

AFME §III.E alg. 1. Each video is cut into sub-videos of 5 to 8 seconds, so that micro-expressions are not missed §III.E. Each sub-video gets a top-3 emotion list from the audio and from the image frames. The algorithm then looks for one emotion that appears in both lists. If it finds one, it returns that emotion. If no emotion matches across all sub-videos, it reports that sarcasm is detected.

# Audio-Frame Mean Expression (AFME)  [alg. 1]
# Input:  speech_scores, image_scores: per sub-video, emotion -> percent
# Output: the first shared emotion, or None when sarcasm is flagged

def afme(speech_scores, image_scores):
    # Top-3 emotions for every audio sub-video  [alg. 1 lines 5-7]
    speech_top = [top_three(scores) for scores in speech_scores]
    # Top-3 emotions for every image sub-video  [alg. 1 lines 8-10]
    image_top = [top_three(scores) for scores in image_scores]
    # Return the first emotion that both modalities share  [alg. 1 lines 12-18]
    for speech_set in speech_top:
        for image_set in image_top:
            for emotion in speech_set:
                if emotion in image_set:
                    return emotion
    # No shared emotion anywhere: flag sarcasm  [alg. 1 lines 20-21]
    print("sarcasm detected")
    return None

The paper says the emotion spectrum of Plutchik [30] is used to assign weights, so that neighbouring emotions on the wheel count as partial matches. No weights are given, so the pseudocode above uses exact matches only §III.E alg. 1.

Training §III.D §IV.B. Adam is used with a learning rate of 0.001. The paper says the other optimiser settings are left at their defaults. The paper says it tested time-based and exponential decay and then moved to Adam §IV.B. Oversampling and re-weighting are used to balance classes §IV.B §III.B. Epochs: 50 in the curve of fig. 8 and 70 in fig. 9.

Hyperparameters

Results

Author-reported. Setup: CNN+AFME on mixed image and speech data; compared against published FER and JAFFE models in Table 3 tab. 3 §IV.B.

Headline accuracies stated in the paper:

Per-class results of the CNN on the test data:

Per-class results after AFME:

Comparison with other models, as printed in the paper:

Training curves: loss falls and accuracy rises over the epochs. The text says that with more features and epochs, accuracy reached 82.59% and loss was "decreasing to 47.21%"; whether that is a relative reduction or a final value is not clear §IV.C fig. 9. The paper reports a 52% accuracy for an earlier NLP-based prosody model; this is not part of the present method §II.

Paper vs code

No official code.

Searched: the paper, abstract and reference list contain no repository or project URL. GitHub repository search for "AFME", "audio frame mean expression emotion", "Teye" and "Missah" returned no repository that belongs to the authors. Third-party projects with the same short names were ignored.

Pitfalls

• Accuracy is not consistent across the paper. The abstract gives 85% to 96%. Table 1 gives 83.18% to 98.00% for the CNN alone. The text gives 82.59% and 65.23%. Table 3 gives 95.7% for the same method. Which model produced each number is not always stated, so use these values only as ranges §III.D §IV.A §IV.C tab. 1 tab. 3.
• Table 1 and Table 2 disagree on the same emotions. The AFME results (Table 2) are higher than the CNN-only results (Table 1) for Anger (93.18 vs 83.18), Disgust (94.64 vs 89.64), Sad (97.34 vs 88.38), Surprise (98.38 vs 87.34) and Neutral (95.80 vs 86.01). The paper gives no reason for these gaps. Table 2's Happy row shows accuracy 100 with recall 0.94, which the paper does not explain tab. 1 tab. 2.
• Column meanings change between tables. Table 1 labels its count column "N(classified)". Table 2 has "N (truth)" and "N (classified)". The Happy row's count is 2719 in Table 1 and in Table 2's truth column. The paper does not say which meaning applies to which table tab. 1 tab. 2.
• Dataset sizes do not add up. The local set is stated as 1000 images, but its per-class counts sum to 990. The speech class counts in Figure 2 are not tied to a stated split §III.B fig. 2.
• Split and leakage are unstated. The train/test split is not described, and the 20% share of "localized" data is ambiguous. With the same people in train and test, the accuracy could be inflated; the paper does not rule this out §III.D.
• The AFME pseudocode in the paper has defects. Line 9 gives C = {p1, p3, p3}, probably a typo for p1, p2, p3. Line 4 loops over i while the body uses m and n. Line 14 compares k with b, which is never defined; it should be the current element of B. The paper says the weights come from Plutchik's wheel, but no weights are given. The output is an emotion in line 16 and a boolean in line 23, while the caption says it returns the emotion "of the audio and image process completion". The implementation above follows the intent and returns the matched emotion or flags sarcasm alg. 1 §III.E.
• Sarcasm is shown on one example. The sarcasm claim rests on a single public video (Figure 10), with no labelled sarcasm set and no sarcasm metric §III.F fig. 10.
• Epoch counts conflict. Figure 8's caption says 30 epochs, but its axis runs to 50 and the text refers to the 46th to 50th epochs. Figure 9 is 70 epochs. The paper does not say which run produced the final numbers §IV.B fig. 8 fig. 9.
• The learning-rate decay constant is missing. The paper says a decay constant was increased, but gives no value and no schedule §IV.B.
• Oversampling and augmentation are named but not specified. The section heading lists data augmentation, random oversampling and re-weighting, but no factor, method or class weight is stated §IV.B.
• Table 3 compares different datasets. The rows use different training and test sets, so the ranking is not like-for-like. The Khaireddin cell reads "73.28 and 98" as printed tab. 3.
• VGG16 is mentioned but not reported. The paper says a second model was trained for the speech case, but no VGG16 results appear §III.D.
• Dates differ. The PDF header shows IEEE Access, volume 10, 2022, and a 2022 DOI, while arXiv records v1 on 2026-05-28. This card uses the arXiv metadata.
• Grayscale weights differ from the common values. The paper uses 0.3, 0.59 and 0.11, and says averaging would not work in all cases. Implement the weights as written §III.C.
• Single run, no error bars. Results are reported without variance, seeds or repeated runs.

Lineage

• Builds on: Plutchik's emotion wheel [30] for the AFME weighting; Dlib landmarks and OpenCV face tools §III.C; VGG16 §III.D. None of these are arXiv papers in this wiki.
• Compared against: the models in tab. 3 (Syed, Giannopoulos, Dachapally, Jaiswal, Khaireddin). None of them have cards.
• Superseded by: not stated.