Deepfake Video Detection Using Convolutional Vision Transformer

In plain words

Deepfakes are fake videos made by swapping or re-animating faces, and they are easy to make. This paper builds a detector that looks at cropped face images taken from a video rather than whole frames. A small convolutional network first picks out local visual detail in each face. A Vision Transformer, which lets every part of its input look at every other part, then decides whether the face is real or fake. The authors add the convolutional front end to a plain Vision Transformer, arguing that the pair learns both local detail and wider structure. Data handling gets as much care as the model: faces are found with detectors, cropped to a fixed size, and augmented during training. A video is judged from a set of its face crops, and the authors report about 91.5% accuracy on 400 test videos from the DFDC challenge.

TL;DR

A face-crop deepfake classifier: a 17-convolution, VGG-style stack turns each 224x224 face into a 512x7x7 map, which a Vision Transformer encoder with a two-way MLP head classifies. Trained on DFDC with Adam. The paper reports 91.5% accuracy, AUC 0.91 and loss 0.32 on 400 DFDC test videos, all author-reported. The released code departs from the paper in several places, and by reading its training script would fail as shipped (not run). See Paper vs code before implementing.

Problem

Deepfake detectors often generalise poorly to manipulation methods they were not trained on, and the authors argue that data preprocessing is under-described in earlier work §1. They cite a shallow three-layer CNN, FakeCatcher, that does not describe its face preprocessing §1. The paper's aim is a detector that uses a deeper network and an explicit, extensive preprocessing pipeline, trained on a large and varied dataset §1 §5.

Method

The pipeline has two parts: preprocessing §3.1 and detection §3.2. The detector is a convolutional feature stack (FL) followed by a Vision Transformer (ViT) encoder and an MLP head. Training and validation use the same CViT model; testing applies it to face images from each video §3.2.

1. Face extraction. Faces are cropped from video frames and stored as 224x224 RGB images §3.1 §4.1. The paper names three detectors (BlazeFace, MTCNN and face_recognition) and uses face_recognition as a filter on BlazeFace output §4.1 §4.3. Figures 4 to 6 show non-face regions that the detectors produce fig. 4 fig. 5 fig. 6.
2. Augmentation (training only). About 90% of face images are augmented §4.1. The paper does not list the transforms.
3. Feature learning (FL). 17 convolutions with 3x3 kernels, stride 1 and padding 1. Each is followed by batch normalisation and ReLU. Five 2x2 max-pools with stride 2 sit between blocks. Channels start at 32 and double after each pool, up to 512. Blocks have three convolutions each, except the last two, which have four §3.2 fig. 1. The output is a 512x7x7 map. The paper gives 10.8 million parameters for this stack §3.2.
4. Tokens. The paper says the feature map is split into seven patches and embedded to a 1x1024 sequence, with a 2x1024 position embedding §3.2. Code uses one 7x7 patch (see Paper vs code).
5. Transformer encoder. Encoder-only, with multi-head self-attention (8 heads) and MLP blocks, each with normalisation §3.2. The paper does not state the number of blocks or the MLP hidden size.
6. MLP head. Two linear layers with ReLU: the first has 2048 units, the last has two outputs, for the fake and real classes §3.2. The paper says softmax is applied to the output §3.2.
7. Training. Binary cross-entropy loss, Adam, batch size 32, learning rate 0.1e-3, weight decay 0.1e-6, 50 epochs, with the rate multiplied by 0.1 every 15 epochs §4.2.
8. Video decision. Each video is classified from 30 of its face images. The paper does not state how per-image outputs are combined §4.2.

Key equation, used as an evaluation metric and not as the training loss §4.2:

LogLoss = -\frac{1}{n}\sum_{i=1}^{n}\left[y_i\log(\hat{y}_i) + \log(1-y_i)\log(1-\hat{y}_i)\right] eq. 1

Forward pass as the paper describes it, with the code citations for the points where the code shows what the paper leaves open:

# Forward pass of CViT  [§3.1] [§3.2]
# Input:  face crops (B, 3, 224, 224), normalised
# Output: two logits per face (B, 2)

def cvit_forward(images):
    # Feature learning: five conv blocks with max-pool  [§3.2]
    features = feature_stack(images)           # (B, 512, 7, 7)
    # Flatten the whole 7x7 map into one token
    # [code: model/cvit.py#L171-L172]
    tokens = linear_to_1024(flatten(features)) # (B, 1, 1024)
    # Prepend a learned class token  [code: model/cvit.py#L175-L176]
    tokens = concat(class_token, tokens)       # (B, 2, 1024)
    # Add the position embedding  [§3.2] [code: model/cvit.py#L178]
    tokens = tokens + position_embedding[:2]   # (B, 2, 1024)
    # Transformer encoder, 8 heads; depth not stated in paper  [§3.2]
    tokens = transformer_encoder(tokens)       # (B, 2, 1024)
    # MLP head on the class token, 1024 -> 2048 -> 2  [§3.2]
    return mlp_head(tokens[:, 0])              # (B, 2)

Video-level decision as the code implements it. The paper does not state this rule §4.2:

# Video decision in the released code  [code: model/pred_func.py#L183-L200]
# Input:  a video file
# Output: label "FAKE" or "REAL", and a score

def predict_video(video):
    # Sample 15 frames evenly  [code: cvit_prediction.py#L291]
    # [code: model/pred_func.py#L207-L212]
    frames = sample_frames(video, num_frames=15)
    # Detect faces, pad 10 px, resize to 224x224
    # [code: model/pred_func.py#L133]
    faces = detect_and_crop_faces(frames)      # (N, 3, 224, 224)
    # No face found: the code returns class index 0, which is FAKE
    # [code: cvit_prediction.py#L253-L256]
    if len(faces) == 0:
        return "FAKE", 0.5
    # Two outputs per face, each through a sigmoid
    # [code: model/pred_func.py#L185]
    probs = sigmoid(cvit(normalise(faces)))    # (N, 2)
    # Average over faces, then take the argmax
    # [code: model/pred_func.py#L188-L200]
    mean_probs = mean(probs, axis=0)           # (2,)
    index = argmax(mean_probs)                 # 0 = fake, 1 = real
    return real_or_fake(index)

Hyperparameters

Results

Author-reported. Setup: DFDC dataset, 400 unseen test videos §4.2; CViT trained on the DFDC training images §4.1; baselines as listed in the paper; hardware and training time not stated.

Paper vs code

Code: erprogs/CViT at commit cdb80248c76a1cbd8fa48e93d86b48119fc1444f, GPL-3.0. The repository's README names all five authors and links this paper README.md README.md. Its last commit is dated 2025, and the README describes a later CViT2 (2024) version README.md. Code citations below are relative to the repository. Because the license is copyleft, the code is cited by path and line only, with no quoted code.

1. Which architecture. Training (cvit_train.py) imports model/cvit.py cvit_train.py. The default inference network is cvit2, which also uses model/cvit.py cvit_prediction.py. --n cvit loads model/cvit_old.py pred_func.py. The two files share the feature stack, patch embedding, class token and head. They differ in the positional table (size and how it is added, item 3) and in the sequence-length bookkeeping. The paper does not say which file its results come from.
2. Patches. The paper says the feature map is split into seven patches §3.2. In both files, the 512x7x7 map is one patch: the rearrange maps a 7x7 grid with a 7x7 patch size to a single token cvit.py. The positional table in cvit.py is sized for 32x32 = 1024 patches, but only its first two rows are used cvit.py cvit.py.
3. Position embedding. The paper gives 2x1024 §3.2. cvit.py has a learned 1025x1024 table and uses its first two rows, one per token cvit.py cvit.py. cvit_old.py has a 32x1x1024 tensor, added by batch row index, so the same vector is added to both tokens cvit_old.py cvit_old.py. It is not a per-token embedding.
4. Class token. Code prepends a learned class token and classifies from it cvit.py cvit.py cvit.py. The paper does not mention a class token §3.2.
5. Loss and output. The paper says binary cross-entropy §4.2 with softmax on the head output §3.2. Training uses CrossEntropyLoss on the two logits cvit_train.py. The model applies no softmax cvit.py. Inference applies a sigmoid to each output pred_func.py.
6. Decision rule. The paper gives a 0.5 threshold, with the bands garbled in the text §4.2. Code takes the argmax of the frame-averaged sigmoids pred_func.py. Class index 0 is fake, because ImageFolder sorts folder names and the training data folders are fake and real . The mapping is in pred_func.py. The README gives "< 0.5 REAL, >=5 FAKE" README.md; the "5" is probably a typo.
7. No face found. The code sets the label to index 0 (FAKE) with score 0.5 cvit_prediction.py. The paper does not say what happens.
8. Frames. The paper uses 30 face images per video §4.2. The default at inference is 15 frames cvit_prediction.py, and training extraction takes 10 random frames per video extractfaces.py.
9. Face detection. The paper uses BlazeFace with face_recognition as a filter, and names MTCNN as a third option §4.1 §4.3. Training extraction takes the best BlazeFace face, then runs face_recognition on that crop extractfaces.py. Inference uses face_recognition alone, on the full frame with 10 px padding; the BlazeFace and MTCNN paths are commented out pred_func.py pred_func.py.
10. JPEG quality. The paper says a 90% compression ratio §4.1. Code saves JPEG quality 85 extractfaces.py.
11. Augmentation order and recipe. The paper says faces are normalised, then augmented §4.2. Code augments the PIL image first, then converts and normalises loader.py. The transforms and their probabilities are in augmentation.py, with a 0.9 overall probability augmentation.py.
12. Learning rate and epochs. Paper: 0.1e-3 and 50 epochs §4.2. Code defaults: 1e-4 cvit_train.py, 1 epoch cvit_train.py; the README gives 0.001 as the learning-rate default README.md.
13. Checkpoint. Code keeps the weights with the lowest validation loss cvit_train.py cvit_train.py. The paper does not say which checkpoint it reports.
14. Split. The paper splits 70:15:15 by image §4.1. Code splits by DFDC folder number: folders 0 to 34 and 45 go to train, 35 to 44 to validation, 46 and up to test extractfaces.py.
15. Test set. json_file/dfdc_files.json lists 400 .mp4 files, matching the 400 test videos dfdc_files.json cvit_prediction.py. The paper does not say that this list is its test split.
16. Metrics. Training and test code print accuracy only cvit_train.py. No AUC or log-loss computation is in the repository (searched all Python files).
17. Training script. As shipped, it cannot run, by reading. load_data returns three values loader.py, but train unpacks two cvit_train.py. gen_parser uses optparse, which is never imported cvit_train.py cvit_train.py. I did not run it.
18. Weights. The README points to Hugging Face for weights README.md. The paper does not name a checkpoint. The shipped training script saves checkpoints under names that do not match the README's.

Pitfalls

• Video decision rule is not in the paper. Implement the code's rule pred_func.py only if you accept it. The score it returns is not a probability: it is mean[0] when class 0 wins, and 1 - mean[1] otherwise pred_func.py.
• Accuracy hides frame noise. The paper notes that fake videos can contain real frames, so per-frame labels are not reliable video labels §4.2.
• Architecture choice changes the model. The position-embedding difference between cvit.py and cvit_old.py (Paper vs code, item 3) changes what the model can learn about position. The paper's 2x1024 wording suits a per-token embedding, not the batch-indexed one.
• Parameter count. From the layer shapes in cvit_old.py (my arithmetic, not run): the conv stack with batch norm is about 10.81M, the patch embedding (25,088 to 1024) about 25.69M, and the MLP head about 2.10M. Together that is about 38.6M, the paper's total. One transformer block at width 1024 with an MLP of 2048 is about 8.40M, so depth 6 would give about 89M. The "20 weighted layers" count (17 convolutions, one patch linear and two head linears) also excludes the transformer. The paper's count therefore appears to exclude the transformer, but it does not say so.
• Depth and MLP size are unstated in the paper. Depth 6 and MLP 2048 come from the code only cvit.py.
• Loss equation. As printed in the source, the second term is log(1-y_i) log(1-yhat_i). The usual binary log-loss has (1-y_i) in front of the second log. Treat eq. 1 as a metric and use the standard form.
• Decision threshold text is garbled. "0>y<0.5" and "0.5 ≥ y < 1" §4.2 cannot be implemented as written. Use the code rule or a documented 0.5 threshold on a stated score.
• Validation and test disagree. In tab. 2, CViT scores 87.25 on validation and 91.5 on test, with no explanation. The validation unit is not stated.
• Table 3 is hard to read. Its caption says AUC on UADFV, but the FaceSwap and Face2Face columns are FaceForensics++ (per the table's footnote), and the CViT entries there look like accuracy. Do not combine columns without checking.
• Unclear evaluation sets. Only DFDC is described as a training source §4.1. The paper does not say whether the FaceForensics++, UADFV and FaceShifter numbers come from the DFDC-trained model or from models retrained on them.
• Dataset counts. 162,174 images are split 112,378 / 24,898 / 24,898. The stated total after augmentation is 308,130. The paper does not say whether augmented copies count toward the split sizes.
• Preprocessing matters a lot. Without removing non-face crops, accuracy falls to 69.5% and loss rises to 0.4 §4.3. Choice of detector alone moves DFDC accuracy from 83.40% (BlazeFace) to 91.50% (face_recognition) tab. 4. Training and inference use different face pipelines (Paper vs code, item 9).
• Weakness the authors acknowledge. FaceShifter (46%) and NeuralTextures (60%) are low tab. 1 §4.2. The paper attributes this to hard-to-learn artefacts.
• Hardware and time are not stated. A 224x224 input with 15 to 30 faces per video makes inference cost scale with the frame count.

Lineage

• Builds on: the Vision Transformer, which the paper adopts as is §3.2, and a VGG-style convolution stack §3.2. Neither has a card here.
• Compared against: CNN+RNN-GRU of Montserrat et al. (cited as [47]) on DFDC tab. 2; MesoNet and MesoInception tab. 3.
• Later version: the code README describes CViT2 (2024), trained on five datasets README.md README.md. That version is not covered by this card.