Cross-Network Generalization Failure in Machine Learning-Based Network Intrusion Detection

Abstract

Machine learning increasingly powers network intrusion detection systems (NIDS) that flag cyberattacks in network traffic, and published detectors routinely report near-perfect accuracy on standard benchmarks. This accuracy can mislead, as a model may reach it by memorizing the specific network it trained on rather than learning what attacks look like. Prior studies show that detection collapses when a model meets a different network, but they stop at aggregate accuracy and do not identify which attacks fail, explain why, or test a fix. We aligned two benchmark datasets, CIC-IDS2017 and CSE-CIC-IDS2018, to a common 76-feature schema and trained random forest classifiers, evaluating them within and across networks over 13 shared attack classes. Same-network detection was strong (macro F1 0.86), yet cross-network detection fell to macro F1 0.14, and 11 of 13 attack classes dropped to zero recall. The reverse direction failed similarly. Only slow-rate attacks, Slowloris and GoldenEye, kept measurable detection across networks. Features the model weighted most also shifted most between networks: its top feature, the initial forward window size, had a median of 251 on one network and 8192 on the other, which indicates that the model encoded the environment rather than the attack. Restricting training to low-drift features did not recover cross-network detection (macro F1 0.14) and lowered same-network performance. These preliminary results suggest that reported NIDS accuracy overstates real-world reliability and that feature selection alone cannot close the gap.