3PS-RAN: A Real-Time Framework for Securing the O-RAN RACH Against DDoS Attacks Toward NextG

Even with the introduction of additional security features, the progression toward 6G and Open Radio Access Network (O-RAN) architectures expands security risks, increasing both the scale and complexity of mobile networks. Consequently, robust defenses against threats are crucial to ensure the secure and reliable operation of these evolving systems. However, Distributed Denial-of-Service (DDoS) attacks continue to exploit weaknesses in the RAN signaling protocols. Specifically, the Random Access Channel (RACH) procedure remains vulnerable to flooding attacks that target resource management and the finite capacity of the CU-UE-ID list, effectively blocking legitimate User Equipment (UE) access. In this paper, the possible security concerns during the RACH procedure within an O-RAN environment are investigated. Based on discovered security concerns, various RACH flooding attack scenarios and intelligent attacks are designed to perform DDoS attacks on the system. To counteract such attacks, a novel Three-Phase Security (3PS) mechanism is designed to enhance protection during the RACH procedure in the O-RAN architecture. In this regard, both statistical and machine learning-based novel detection algorithms have been developed. Through these detection algorithms, the location of the attacker UEs, the attack type, its duration, and period are identified. Consequently, these detected attacks are mitigated more efficiently by enhanced RACH protocol steps. The adverse effects of these attacks and the security improvements introduced by the 3PS mechanism are evaluated in a real-Time setup using the OpenAirInterface (OAI) 5G platform. Experimental results demonstrate that the proposed framework improves the average connection time for legitimate UEs by a factor of 9 to 25 under heavy attack conditions. Additionally, system profiling reveals that the proposed mechanism introduces a negligible operational CPU overhead of less than 0.2%, while maintaining a remarkably stable memory footprint. Furthermore, this implementation confirms the practical relevance of the designed attack scenarios and the solution's compliance with 5G 3GPP standards within an O-RAN context.