New Zero Day in Palo Alto GlobalProtect VPN Appliances
Introduction
On April 12th, Volexity announced they discovered zero day exploitation occurring against Palo Alto GlobalProtect VPN appliances running PAN-OS 10.2 and above. CISA has issued an alert and added it to their KEV catalog. For those keeping track, this is the fourth vulnerability in an SSL VPN appliance in as many months of 2024. At the time of this writing, Palo Alto has assigned CVE-2024-3400 and is still in the process of releasing patches. On April 16th, Watchtowr released a blog explaining that the vulnerability is…yet Another Path Traversal. PAN-OS has also fallen victim to the vulnerability class which has impacted devices from other industry leaders: Ivanti, Citrix, F5 and Cisco.
Description
The vulnerability involves a critical command injection flaw that allows an attacker to execute arbitrary code with root privileges. This vulnerability is exploitable when telemetry features are enabled on the device, and it arises from improper validation of user inputs in SESSID cookies. Attackers can manipulate these inputs to modify file paths and commands, enabling the execution of malicious code. Shortly after the Watchtowr blog was released, TrustedSec confirmed they had seen this PoC being exploited in the wild:
In addition to abusing path traversal, this vulnerability also leveraged an ancient UNIX hacking trick; Internal Field Separator or (IFS) is an environment variable that defines a delimiter or set of delimiters the shell uses to split text into words. The IFS’s default value typically includes the space, tab, and newline characters. When a script processes user input or the output from commands, IFS determines how strings are broken into individual pieces or tokens, affecting command parsing and the execution of shell scripts. IFS can be abused in scenarios where scripts do not securely handle inputs that can be controlled by an attacker. If an attacker can influence the contents of variables that are read into commands or scripts that rely on IFS to parse input, they can introduce delimiters through those inputs to manipulate the command execution or data handling. It’s worth pointing out that the BMC&C vulnerabilities discovered by Eclypsium in December 2022 and July 2023 also leveraged IFS in the exploits we developed.
Post Exploitation Techniques
After exploiting the devices, Volexity observed attackers leveraging multiple techniques to gain persistence. They first added a cron entry to retrieve and execute a Python script; this script contained additional payloads and was protected by ACLs in an attempt to prevent IP’s other than the PAN-OS device from accessing it, a defensive measure to impede incident responders from identifying what it contained. The attackers changed the contents of this file numerous times to send Python reverse shells, a Golang tunneling tool named GOST and an SSH based reverse shell.
The most unique malware they deployed was dubbed UPSTYLE by Volexity, and consists of a Python module /usr/lib/python3.6/site-packages/system.pth which ensured it was run at every Python startup. This backdoor allows attackers to communicate with the compromised system by requesting non-existent web pages containing specific patterns. When the requests failed, they would be logged to the web server error log; the backdoor will then parse the error log looking for these failures, find the pattern and decode the data added to the non-existent URI to determine the commands to run. Finally, the backdoor removes the offending log messages from the logs and modifies the timestamp to avoid detection (lines 72-78 below):
Summary
This attack campaign continues the growing trend observed over the last two years of attacks against network appliance firmware. As attackers move lower into the computing stack to evade security tooling, their focus has shifted towards infrastructure systems whose functionality requires internet exposure. As with other attacks in this category, the devices targeted are widely deployed, infrequently updated, mission critical and unfortunately, outside the view of most security tooling. As with most appliance vendors, PAN-OS devices have been architected as black boxes. Our lab device running PAN-OS 10.2.5 is based on Centos version 8.3.2011, a relatively new release but access to the bash shell has been replaced with a restricted CLI; this inhibits forensics response as standard techniques such as listing all running processes, checking the filesystem, filtering log files and looking for files with more recent timestamps are all unavailable to administrators.
These restrictions are commonplace in the appliance ecosystem, especially firewalls and VPN appliances but they pose less of a hurdle for attackers than they are an obstacle to defenders. As part of our investigation into this vulnerability, Eclypsium Research deployed a virtual PAN-OS device but in order to fully understand what an attack looks like, we had to modify the system through non-standard techniques to provide us with a full bash shell before the system was fully analyzable. This is outside the scope of most DFIR practices and would be infeasible – possibly impossible – on a hardware appliance where mounting the virtual disk isn’t an option. Appliance vendors should reconsider their approach to device management; there are no signs the attack against the infrastructure supply chain is slowing down and comprehensive response requires all available tools at the defenders disposal.
During our testing process, we initially used a virtual machine trial we had downloaded in January 2024, which we knew would be vulnerable. As a test, we requested a new virtual trial on April 15th, and we discovered that the virtual machine trial versions being distributed by Palo Alto were still of a vulnerable version (10.2.5). After testing the vulnerability, we wanted to see what a patched version looked like and since we couldn’t upgrade our trial VM, we decided to spin up an instance in Google Cloud Platform; Watchtowr noted in their blog that “partway through our investigations, Palo Alto took the step of removing the vulnerable version of their software from the AWS Marketplace” so presumably, the same had been done in GCP. However, when we tested the virtual appliances in Google Cloud Platform we determined they too were still vulnerable (version 10.2.8). In the spirit of protecting customers, we immediately notified Palo Alto so they can take the necessary steps to address this.
Palo Alto trial download email, April 15th
Contents of virtual machine trial zip file
Successful exploitation of Google Cloud Platform version of PAN-OS v10.2.8
How Eclypsium can Help
Eclypsium customers can view the vulnerability status of their PAN-OS devices in the Eclypsium console.
Additionally, Eclypsium has developed detection techniques that will alert customers to exploitation attempts and indicators of compromise:
At Eclypsium, our goal is to help organizations defend their IT infrastructure against increasingly frequent and sophisticated supply chain attacks against network appliance firmware. As we shift to a posture where third-party code is suspect and should be validated and monitored, Eclypsium can assist with multiple techniques for analyzing IT infrastructure code.
More broadly, the Eclypsium platform helps customers to quickly identify systems affected by certain widely used components. In the case of the Palo Alto VPN appliance vulnerability, the vector was the manipulation of user inputs in SESSID cookies. But the same type of supply chain complexity affects other IT infrastructure. Request a demo and we’ll be happy to share how our platform helps organizations quickly assess their exposure to supply chain incidents using the Palo Alto VPN appliance example.
Additional Resources
