Improving Automated Penetration Testing with Reinforcement Learning

In today's digital world, keeping information systems secure is crucial. One effective way to check the security of a computer system is through penetration testing (PT). This process helps identify potential weaknesses that could be exploited by harmful actors. Traditional PT requires skilled professionals, making it time-consuming and labor-intensive, sometimes taking days or even weeks. Furthermore, manual testing can lead to considerable downtime of systems. Therefore, there is a strong demand for Automated Penetration Testing techniques (AutoPT).

Several advanced tools and frameworks for AutoPT have been created to improve testing efficiency. For instance, Metasploit is a widely used tool that helps gather information and exploit vulnerabilities. Despite these advancements, many current tools have limited capabilities, focusing only on specific tasks and not being able to perform comprehensive evaluations independently, unlike human testers.

A promising avenue to enhance PT is using Reinforcement Learning (RL), a branch of artificial intelligence (AI). RL involves a computer program, or agent, making decisions within an environment to achieve specific goals. The agent learns from its actions and adjusts based on the rewards it receives, similar to how humans learn through experience. RL has already shown success in various applications, including self-driving cars, robotics, and game-playing AI.

In recent years, research has increased on using RL in PT for information systems. Some studies reformulated the PT process as decision-making problems, allowing agents to learn optimal strategies using algorithms. For example, one approach used deep Q-learning to automate post-exploitation tasks. Others have integrated RL with existing industrial PT frameworks to minimize manual work.

#Challenges Faced in Automated Penetration Testing

Despite progress, existing RL-based PT approaches encounter several challenges. One significant issue is sampling efficiency, where the agent needs many interactions with the environment to learn the best strategies. This need arises from the large action space, where a pen-tester has multiple actions to choose from for each scenario.

Another challenge is the complexity of defining rewards for the agent. Successful actions typically receive positive rewards, while invalid actions face penalties. However, creating a single reward function that captures all the necessary rules can become complicated, making it harder for the agent to learn effectively.

Moreover, RL-based PT often struggles with interpretability. After training, agents may not clearly indicate their current phase or next steps in the testing process. This lack of clarity can undermine trust in the agent's decisions and performance.

#Introducing a Knowledge-Informed Approach

To tackle these challenges, we propose a new framework called DRLRM-PT, which combines knowledge from cybersecurity with RL. This approach helps the agent decompose complex tasks into smaller, manageable subtasks, improving learning efficiency.

The framework employs a "reward machine" (RM) to encode domain knowledge from recognized cybersecurity knowledge bases. The RM outlines a set of events during PT and breaks the process down into distinct subtasks. It also provides tailored reward functions based on the current phase of the PT, enhancing the flexibility of rewards assigned to the agent during training.

In this study, we focus on Lateral Movement as a case study. Lateral movement refers to the actions taken after gaining initial access to a network, moving deeper to take control of valuable assets. To guide this process, we formulate it as a partially observable decision-making problem using RMs.

#The DRLRM-PT Framework Explained

Our proposed framework DRLRM-PT involves an agent acting as a pen-tester, engaging with a target network system. The target environment comprises various components, including hosts, firewalls, and routers. The agent can select from a range of PT actions, such as scanning for vulnerabilities and attempting exploits.

As the agent interacts with the environment, it makes observations based on the outcome of its actions. The immediate rewards reflect how well the agent is achieving its goals, particularly taking ownership of critical resources in the network. The agent aims to maximize overall rewards through its learning experiences.

In this framework, the agent is supported by the RM that encodes cybersecurity knowledge. The RM works as a state machine, helping to outline subtasks and specify reward functions for each action the agent takes. By tracking events detected during PT, the RM transitions its state, guiding the agent's learning process effectively.

#Action and Observation Spaces in Lateral Movement

In our study, we consider three main types of actions related to lateral movement:

1. Scanning: This involves collecting essential information about the network by discovering machines, their connections, and vulnerability data.

2. Vulnerability Exploitation: This can be further classified into local and remote exploitation. Local exploitation occurs when the agent operates on a connected node, while remote exploitation targets nodes currently discovered but not yet accessed by the agent.

3. Connection: This allows the agent to connect to a node using specific credentials and ports.

The observations made by the agent are obtained through scanning operations after executing actions. The observation space consists of several subspaces, including discovered node counts, privilege levels of nodes, discovered properties, leaked credentials, and whether the agent successfully made lateral moves.

#Designing Reward Machines for Improved Learning

We utilize RMs to guide the agent's actions and help it learn more efficiently. One simplified RM focuses on three main subtasks:

1. Discover new credentials.
2. Connect to new nodes using those credentials.
3. Elevate the privileges of connected nodes.

This phase of the process will repeat until the agent reaches specific goals, such as accessing critical data.

We also examine a more detailed RM that includes a broader set of tasks. In this RM, the agent is first guided to discover new nodes before looking for credentials, then connecting to new nodes, and finally elevating privileges. The increased complexity of this RM allows for more precise guidance and support during the learning process.

#Objectives and Methodology in Testing

The main goal of lateral movement is to gain control over as many nodes as possible within the network. By maximizing the accumulated rewards linked to the RM during PT, we can guide the agent toward achieving this goal effectively.

To train the agent and enhance the learning process, we adopt the Deep Q-learning algorithm with RMs (DQRM). This approach allows the agent to refine its strategy and improve its overall performance over time.

#The Simulation Platform and Experimental Setup

For our experiments, we use CyberBattleSim, an open-source simulator developed for testing and evaluating lateral movement strategies within networks. This platform creates simulated networks modeled by graphs with interconnected nodes and vulnerabilities.

Two network environments are set up for testing: CyberBattleChain (a sequential structure) and CyberBattleToyCtf (a more complex mesh structure). Each node is designed with specific properties, including vulnerabilities that may lead to credential exposure or privilege escalation.

The agent's objective in the simulation is to capture as many important resources, referred to as 'flags', while using the fewest number of actions possible.

#Experimental Analysis and Results

We designed experiments to validate our framework and address two research questions:

1. Can the agent guided by RM improve the learning efficiency of PT compared to the agent without RM?
2. How will different RM designs affect the PT performance?

To evaluate these questions, we compared four configurations of agents-two using the DQRM algorithm with distinct RMs and two using a traditional approach without RMs. Agents were trained in both environments to assess their performance across different phases.

#Training Efficiency Results

In both environments, agents utilizing the DQRM framework demonstrated improved training efficiency compared to those using traditional methods. The results indicated that the RM-guided agents could achieve higher average rewards with fewer actions taken.

#Evaluation Performance Findings

The testing revealed that DQRM agents outperformed traditional agents in terms of efficiently capturing flags and accomplishing goals. The differences in average steps taken by agents showed that the RMs indeed provided a valuable advantage during the testing process.

#Impact of RM Designs on Performance

Analyzing the performance of agents guided by different RMs showed that those with more detailed and structured guidelines performed better than those with simpler designs. The agents with nuanced RMs could navigate the PT process more effectively and achieve goals in fewer actions.

#Conclusion and Future Directions

In summary, our proposed knowledge-informed AutoPT framework, DRLRM-PT, effectively integrates domain knowledge into the reinforcement learning process, enhancing the capabilities of automated penetration testing. Our study highlights the importance of employing structured guidance through RMs to improve the learning efficiency and performance of agents during testing.

Future work will involve investigating more sophisticated RMs informed by additional cybersecurity knowledge bases, aiming to increase the adaptability and efficacy of the system across various PT scenarios. The goal is to broaden the scope of AutoPT beyond lateral movement to encompass other critical applications in penetration testing.