A new cluster-aware regularization of neural networks

Inherent clusters formed by observations used for the training of a classification model is a frequently encountered case. These clusters differ in certain characteristics, however in classical modelling techniques no information on these differences is fed into the model. Differentiations in purchasing styles of e-commerce customers may be a good example for this case. While some customers like to do research and comparisons on price, functionalities and comments, some others may need a shorter examination to decide on their purchase. In a similar manner, purchasing journey of a deal seeker customer would differ from a luxury buyer customer. In this paper, we propose a neural network model which incorporates different cluster information in its hidden nodes. Within the forward propagation and backpropagation calculations of the network, we use a non-randomized Boolean matrix to assign hidden nodes to different observation clusters. This Boolean matrix shuts down a hidden node for observations which do not belong to the cluster that the node is assigned to. We performed experiments for different settings and network architectures. Also, analyses are conducted to study the influence of alternative application patterns of the Boolean matrix on the results – expressed in terms of iterations and epochs for an Adam (adaptive moment estimation) optimization. Empirical results demonstrate that our proposed method works well in practice and compares favorably to fully randomized alternatives.