Multimodal Deepfake Detection via Audio-Visual Fusion

Architected a scalable multimodal deepfake detection pipeline utilizing a Transformer-based attention mechanism to fuse spatial, temporal, and acoustic embeddings, achieving 95% accuracy and 0.97 AUC.

Classification Accuracy

95.0%

ROC-AUC Score

0.970

Benchmark Dataset

FakeAVCeleb

Inference Latency

Real-time edge compatible

Read Full Paper (PDF)IEEE Explore DOI Code Repository

Abstract & Overview

Deepfake generation techniques have matured to generate hyper-realistic manipulations across both facial visual dynamics and acoustic speech. Conventional unimodal detectors fail when adversaries tamper with cross-modal synchronization. This paper introduces an end-to-end multimodal deepfake detection framework fusing spatial, temporal, and acoustic embeddings using cross-attention transformers. By enforcing phoneme-viseme alignment verification via custom synchronization loss, our system achieves 95% classification accuracy and 0.97 ROC-AUC on the benchmark FakeAVCeleb dataset.

Problem Formulation

Existing deepfake detection systems often operate in unimodal silos (analyzing only facial video frames or only audio waveforms), leaving them susceptible to adversarial perturbations and cross-modal desynchronization. Furthermore, unimodal architectures struggle with high false positive rates in compressed, in-the-wild video streams.

Architecture & Key Mechanisms

1Feature Extraction: Employs ResNet-50 for frame-level facial feature extraction and BiLSTM for modeling long-term facial landmark dynamics.

2Acoustic Modeling: Converts audio streams to high-resolution Mel-spectrograms processed through a 1D-CNN encoder.

3Cross-Modal Transformer Fusion: Synchronizes spatial, temporal, and acoustic embeddings into unified multimodal attention vectors.

4Synchronization Loss: Integrates a contrastive cross-modal sync loss penalizing phoneme-viseme temporal discrepancies.

Empirical Highlights

Integrated ResNet-50 visual encoder, BiLSTM temporal encoder, and 1D-CNN audio encoder processing Mel-spectrograms.
Implemented custom synchronization loss function to enforce phoneme-viseme alignment verification.
Evaluated on FakeAVCeleb benchmark dataset demonstrating superior cross-modal robustness.

Technologies & Tooling

PyTorchResNet-50BiLSTM1D-CNNMel-spectrogramsTransformers

BibTeX Citation

@inproceedings{ghosh2025multimodal,
  title={Multimodal Deepfake Detection via Audio-Visual Fusion},
  author={Ghosh, Debjyoti and Collaborators},
  booktitle={2025 22nd IEEE India Council International Conference (INDICON)},
  year={2025},
  organization={IEEE},
  doi={10.1109/INDICON68490.2025.11392906}
}