Self-Supervised Learning with ViT for Noisy Real-World Data
Self-supervised pretraining with masked spectrogram reconstruction followed by supervised ViT-style classification for robust mental stability detection from noisy, unprocessed real-world voice data.
Workflow at a glance
Prepare noisy voice data
Define voice-learning experiments.
Study self-supervision
Explore representation learning objectives.
Train and compare
Investigate robustness under noise.
Ongoing evaluation
Research remains in development.
Project facts & reported results
Implementation fact
Implementation fact
Implementation fact
Implementation fact
Project Overview
Problem Statement
Real-world voice data collected outside laboratory conditions is often noisy, untrimmed, and distribution-shifted, which significantly degrades the performance of fully supervised models trained on curated datasets. Labeled data is expensive and scarce in the mental health domain. Robust classification under real-world noisy conditions remains a key challenge.
Approach & Methodology
Introduced a two-stage framework: (1) Self-supervised pretraining on unlabeled voice spectrograms using a masked reconstruction objective with MSE loss — randomly masks time–frequency regions and trains a reconstruction network to recover the original spectrogram. (2) Supervised fine-tuning of a ViT-style classifier with frequency band attention, Conv1D blocks for local temporal features, and Transformer blocks (Multi-Head Self-Attention + FFN) for global dependencies. Pretrained SSL weights are transferred to the classifier for fine-tuning on unprocessed/noisy data. Evaluated with 5-Fold Cross-Validation and ensemble of top-performing folds.
Outcome & Scope
Ensemble accuracy of 84.90% on noisy real-world data (precision 89.79%, recall 82.69%, F1-score 86.10%). Demonstrated that self-supervised pretraining significantly improves robustness under distribution shift. Stable convergence across folds under noisy conditions. Research carried out under Dr. Md. Taimur Ahad at the 4IR Research Cell, DIU. Manuscript in preparation. Dataset not included due to privacy/ethical constraints.
System Architecture
Two-stage SSL pretraining + supervised ViT-style classification for noisy voice data
Key Components:
Log-mel spectrogram extraction from raw voice segments at 48 kHz, 2-second segments
Masked spectrogram reconstruction on unlabeled data using MSE loss for representation learning
Learns importance weights across frequency regions of the spectrogram
Local temporal feature extraction from spectrogram representations
Multi-Head Self-Attention + Feed-Forward Networks for global dependency modeling
Combines top-performing folds for robust final prediction on noisy test data
Key Features & Capabilities
Self-supervised masked spectrogram reconstruction pretraining (MSE loss)
ViT-style Transformer classifier with frequency band attention
Conv1D blocks for local temporal feature extraction
Multi-Head Self-Attention + FFN for global dependencies
Transfer learning from SSL pretrained weights to supervised classifier
5-Fold Cross-Validation with ensemble of top-performing folds
Robust classification on unprocessed, noisy real-world voice data
Log-mel spectrogram input representation
Current Scope & Limitations
Research use only. These experiments do not establish a clinically validated diagnostic or screening tool.