Stochastic Covariance Regularization for Imbalanced Datasets

Imbalanced classification remains an important challenge in machine learning, where models often exhibit biased performance toward majority classes, leading to poor generalization on minority classes. To address this issue, we introduce SCoR, a novel stochastic and self-supervised regularization method for imbalance. Unlike existing approaches that rely on imbalance-dependent hyperparameters or explicit class labels, SCoR operates in a self-supervised manner without any class- or distribution-dependent hyperparameters. Extensive experiments on benchmark datasets, including MNIST and CIFAR-10, as well as real-world datasets, demonstrate that SCoR performs comparably to or better than popular methods such as focal loss and label smoothing, particularly in large datasets or when the number of target classes is large. Furthermore, our spectral analysis shows that SCoR is associated with lower minimum singular values in classifier weight matrices, a property that correlates with improved generalization. We also find that combining SCoR with label smoothing can yield additional performance gains in certain datasets. These results highlight SCoR’s potential as a robust regularizer and motivate further research into spectral regularization methods for imbalanced learning. Code for SCoR and all experiments is available at https://github.com/ahmeterdem1/scor.