Deep Learning for Channel Estimation in RIS-NOMA-assisted THz Communications over Generalized Fading Channels

This paper investigates the channel estimation problem in a multiple-input single-output (MISO) reconfigurable intelligent surface (RIS) non-orthogonal multiple access (NOMA)-assisted terahertz (THz) communication system where users experience mobility and varying small-scale fading characterized by the generalized α − μ distribution. Reliable channel estimation becomes challenging when passive RIS elements and superimposed NOMA signals operate under high attenuation of THz band. We propose a novel deep learning framework, THz RIS-NOMA Channel Estimation (TRiNCE), for cascaded channel estimation. TRiNCE is designed as a GRU-based conditional Wasserstein Generative Adversarial Network with Gradient Penalty (cWGAN-GP). TRiNCE performance is evaluated under various α − μ fading conditions, the number of RIS elements, NOMA power allocation factors, and the number of BS antennas. The model achieves an R² score of 0.85 under Rayleigh fading and up to 0.95 under milder α–μ conditions. It further demonstrates strong robustness, maintaining high estimation accuracy with less than 2% performance variation across different RIS sizes and NOMA power allocation factors, and only ∼ 7% degradation when the number of BS antennas increases from 1 to 5. Results show that TRiNCE not only provides accurate and reliable channel estimation across all tested network configurations but also significantly outperforms convolutional neural network (CNN), long short-term memory (LSTM), and gated recurrent unit (GRU) baseline models while requiring substantially fewer trainable parameters. This establishes TRiNCE as a computationally efficient and effective solution for channel estimation in RIS-NOMA-assisted THz communication systems.