Explainable Spark-based PSO Clustering for Intrusion Detection

Abstract

Given the exponential growth of available data in large networks, the existence of rapid, transparent and explainable intrusion detection systems has become of high necessity to effectively discover attacks in such huge networks. To deal with this challenge, we propose a novel explainable intrusion detection system based on Spark, Particle Swarm Optimization (PSO) clustering and eXplainable Artificial Intelligence (XAI) techniques. Spark is used as a parallel processing model for the effective processing of large-scale data, PSO is integrated for improving the quality of the intrusion detection system by avoiding sensitive initialization and premature convergence of the clustering algorithm and finally, XAI techniques are used to enhance interpretability and explainability of intrusion recommendations by providing both micro and macro explanations of detected intrusions. Experiments are conducted on several large collections of real datasets to show the effectiveness of the proposed intrusion detection system in terms of explainability, scalability and accuracy. The proposed system has shown high transparency in assisting security experts and decision-makers to understand and interpret attack behavior.