Detecting cyber-attacks in cyber systems is essential for protecting cyber infrastructures from cyber-attacks. It is very difficult to detect cyber-attacks in cyber systems due to their high complexity. The accuracy of the attack detection in the cyber systems depends heavily on the completeness of the collected sensor information. In this thesis, two approaches are presented: one to detecting attacks in completely observable cyber systems, and the other to estimating types of states in partially observable cyber systems for attack detection in cyber systems. These two approaches are illustrated using three large data sets of network traffic because the packet-level information of the network traffic data provides details about the cyber systems.

The approach to attack detection in cyber systems is based on a multimodal artificial neural network (MANN) using the collected network traffic data from completely observable cyber systems for training and testing. Since the training of MANN is computationally intensive, to reduce the computational overhead, an efficient feature selection algorithm using the genetic algorithm is developed and incorporated in this approach.

In order to detect attacks in cyber systems in partially observable environments, an approach to estimating the types of states in partially observable cyber systems, which is the first phase of attack detection in cyber systems in partially observable environments, is presented. The types of states of such cyber systems are useful to detecting cyber-attacks in such cyber systems. This approach involves the use of a convolutional neural network (CNN), and unsupervised learning with elbow method and k-means clustering algorithm.