Last week CISA took a major step in the fight against ransomware and state-sponsored attacks by issuing Binding Operational Directive 23-02. This directive directly calls out an area where threat actors are the most active and successful in the wild—the exploitation of network infrastructure appliances such as VPNs, switches, routers, and firewalls as well as the out-of-band management interfaces (baseboard management controllers or BMCs) that support server infrastructure. This is a critical step because it requires the 100+ federal agencies under CISA’s purview to take action to protect the internet-facing management interfaces of these critical devices. Days later CISA and the NSA issued new joint guidance on the importance of hardening BMCs. Even if you’re not in the federal government, you should take the CISA and NSA message to heart: Attacks on network devices and BMCs are now part of the mainstream threat landscape, and action is required.
To protect themselves, organizations need to understand two critical things—how threat actors are targeting these assets in the real world, and the practices and controls that are available to counter them. We will cover both topics in this blog.
How the Landscape Has Changed and Why
The CISA directive provides important context: “As agencies and organizations have gained better visibility of their networks and improved endpoint detection and response, threat actors have adjusted tactics to evade these protections by targeting network devices supporting the underlying network infrastructure.”
This calls for a somewhat shameless plug. This class of threats was one of the reasons we founded Eclypsium, and likewise was one of the first pieces of research we published. The core issue both then and now is that these are incredibly high-impact threats that largely are not addressed by traditional security controls.
Attackers Go Where the Defenders Are Not
Network devices and BMCs are different from most enterprise assets. They don’t run standard commodity operating systems, and thus, don’t support standard security tools such as EDR. Instead, they rely on specialized code that is almost entirely defined in the supply chain. For example, network devices run custom integrated OS/firmware that is unique to each vendor. These are often full-fledged operating systems based on Linux or BSD, such as F5 TMOS, Cisco IOS, Citrix Netscaler OS, Fortinet FortiOS, etc. BMCs likewise run on their own separate integrated hardware and firmware that is fully independent of the host hardware and OS. These systems have a number of built-in management functions that, if not configured properly, can enable attackers to compromise the system, elevate their privileges, establish persistence, move laterally within the network, and evade detection.
The resulting lack of standard endpoint security not only leaves these assets more open to attack, it also makes it far harder for security teams to detect when a device has been compromised. Even laptops are increasingly targeted at the level of supply chain components and integrated code that lives below the level of the operating system. But network devices and BMCs are at an even much greater risk because organizations have limited visibility and control over what is going on in the devices themselves. Mandiant reported that the APT group UNC3524 persisted undetected in victim networks for at least 18 months by tunneling through compromised network devices.
The reliance on specialized, low-level code also introduces challenges for vulnerability management. The assets themselves are often not part of the regular vulnerability scanning process, which typically focus on commodity laptops and workstations. The low-level nature of the code means that vulnerabilities can be hard to see even when organizations know to look for them. And once problems are found, updates are often slow due to the challenges of taking critical devices offline. Together this means that network devices and BMCs are a great place for attackers to go looking for exploitable vulnerabilities.
Attackers Go Where the Value Is High
Once compromised, these assets are incredibly valuable to threat actors. Network devices and gateways often need to be exposed to the Internet and naturally provide incredible reach across an organization. This has made them an ideal beachhead that attackers can use to gain initial access into an organization and spread to other systems. BMCs provide near-omnipotent control over the servers they manage. They have their own integrated code, power, network stack, and interfaces yet have full control over the server resources. This gives an attacker access to all the virtual machines, applications, and data running on the device. Worse still, attackers can easily “brick” a compromised server, taking it or even entire datacenters offline. And by residing in code off of the main system, attackers can easily maintain persistence and evade any security controls that may be present. The recent CISA/NSA alert put it clearly:
Traditional tools and security features including endpoint detection and response (EDR) software, intrusion detection/prevention systems (IDS/IPS), anti-malware suites, kernel security enhancements, virtualization capabilities, and TPM attestation are ineffective at mitigating a compromised BMC. For these reasons, NSA and CISA recommend organizations pay attention to the security of their BMCs and apply the hardening actions detailed in the following section.
In the end, you have a near-perfect recipe for risk: High-value assets where security is weak.
Accelerating Attacks in the Wild
We don’t have to look far to see how these risks are playing out in the real world. The reality is that new attacks are popping up so quickly that any list we put together will likely be out-of-date soon after it is published.
But it’s important to recognize that this situation didn’t just come out of the blue. The wave of attacks we see today is actually an acceleration of trends that began in earnest well over three years ago. CISA and other agencies began sounding the alarm in 2020 that ransomware and APT groups were actively exploiting VPNs, routers, switches, and firewalls. These attacks proved to be highly successful, and attackers immediately set out to replicate the formula, which in turn has led to even more threat actors, discovering more vulnerabilities, and targeting more vendors and asset types.
- Volt Typhoon and Attacks on Fortigate Devices – Researchers recently identified new attacks attributed to a state-based Chinese threat actor known as Volt Typhoon. This campaign exploited Fortinet security devices in order to gain initial access to critical infrastructure targets in the U.S. According to Fortinet research, the Volt Typhoon targeted CVE-2022-40684, although their research uncovered even more related vulnerabilities. Additional research identified Chinese threat actors exploiting CVE-2022-42475, although Fortinet has not attributed those attacks to the Volt Typhoon group.
- LockBit Attacks on Network Devices – LockBit continues to be one of the most prolific and widely-deployed ransomware families in the wild. LockBit has gone through several iterations and has even included code and techniques originally found in Conti. Much like other popular ransomware, LockBit heavily targets network devices in order to gain initial access into organizations, specifically targeting F5 BIG-IP devices and Fortinet’s FortiOS-based devices.
- Barracuda Email Security Gateway (ESG) – Just last week, researchers identified a global operation targeting Barracuda ESG appliances. These attacks exploited CVE-2023-2868, which is a vulnerability in the firmware of physical Barracuda appliances. After gaining access, the threat actor uses the position to perform espionage and steal data.
- Ransomware Attacks on BMCs and Data Centers – Researchers have identified ongoing attacks targeting Cloud Service Providers (CSPs) and Managed Service Providers (MSPs), specifically by targeting remote management and out-of-band management components such as BMCs and IPMI. Ransomware operators have also been observed targeting data centers and exploiting BMCs as a method for maintaining persistence.
- Attacks on VMware ESXi Hosts – Threat actors have also exploited 0-day vulnerabilities in VMware’s bare-metal ESXi hypervisors. Like other forms of infrastructure we have discussed, ESXi are typically unprotected by endpoint security tools, and once compromised, attackers are able to evade other security controls such as the ESXi firewall.
These are just some of the most recent examples of attacks in the wild and serve as an illustration of just how broad the problem is and the many types of assets that can be affected. The important point is that while these assets are not your standard Windows laptop, they are not rare or some sort of corner-case risk. Virtually every organization will have multiple types of high-value assets that won’t be covered by traditional security tools.
Taking Action to Safeguard Your Organization
Ultimately all these examples share a common thread – threat actors are pivoting to softer targets that have the least protection. To close the gap, organizations need the right tools and processes that align with real-world risk. Naturally, the details will vary from organization to organization, but there are some key steps and requirements that will apply in almost every case.
- Ensure Interfaces Are Not Exposed to the Internet – One of the most fundamental steps is to get control over how network interfaces, BMCs, and other assets can be accessed. These interfaces should not be exposed directly to the Internet and ideally should be placed on tightly-controlled, internal management networks. While this is and has always been security best practice, it is a step that is consistently overlooked. It’s important to remember that network and security devices in particular are often targeted as an initial access vector, and any exposed interfaces will play directly into an attacker’s hands. Given that attackers are regularly discovering and exploiting 0-day vulnerabilities, even fully patched and up-to-date systems will be at risk if they are directly accessible to the outside world. With an infrastructure supply chain security platform, you can easily tally your Internet-facing management interfaces.
- Know Your Network Infrastructure and Supply Chain Assets – Inventory management is a standard part of good security practice in general, but often gets overlooked when it comes to network devices and other infrastructure. Organizations need to know exactly what assets they have and the components they contain both in terms of code and physical components. As attackers increasingly target vulnerabilities in supply chain components, it is critical that organizations know what is inside their critical devices. For example, if a new BMC vulnerability is discovered, organizations need to be able to immediately know which devices have the affected component.
- Add Threat Detection for Network Devices and BMCs – Devices are actively under attack today, yet most organizations have no standardized way to tell if their devices have been compromised. To solve this problem, organizations need to be able to verify the integrity of their devices and the code they run. In addition to looking for known threats, security teams need tools that know exactly what code should be on each device and component. Any unexpected changes or code that doesn’t match authorized vendor code—such as unexpected binaries or processes—can be signs that a device has been compromised either in the supply chain or by external threat actors.
- Add Network Devices and BMCs to Your Vulnerability Management – Network devices and BMCs can pose additional challenges when it comes to vulnerability assessment and management. They may (and should be) on isolated management network segments that might be missed by external vulnerability scans. Vulnerabilities in internal code and components in these devices will often not be visible via a passive or even active network vulnerability scan. By focusing on this internal software and firmware running inside network infrastructure devices and their components, a supply chain security tool can understand risks that are often missed and can then help staff to prioritize the vulnerabilities that are being exploited by threat actors and patches to apply to affected devices.
- Establish Patching of Network Devices and BMCs – All network devices and BMCs run complex native operating systems and software/firmware stacks consisting of hundreds of components and libraries. Network device and server manufacturers regularly publish updates in response to critical vulnerabilities. Some organizations may want to deploy patches quickly in response to active threat campaigns, while other organizations may want to fully test updates and deploy during regular update windows. While these decisions are up to each organization, it is important that patching critical network infrastructure equipment becomes a standard part of an organization’s existing patch management programs. Patching of network devices and BMCs can be done with dedicated enterprise software tools remotely (over vendor-specific or Redfish interfaces) and locally via software installed on a server.
- Integrate and Coordinate Security – Attackers are already taking advantage of a blindspot in enterprise security. So as organizations close the gap, it is important that their new insights don’t get stuck in an information silo. Instead, organizations may want to make sure that the posture and integrity of network and server infrastructure become a standard part of the overall security strategy. For example, a Zero Trust access decision may want to consider the posture of the network device involved in the request. Likewise, IR or threat hunting programs may regularly inspect the integrity of infrastructure devices as they look for threats or determine the scope of an event.
Together, these steps can give organizations a solid foundation for dealing with the threats they are facing today. If you would like to learn more, we recommend you review the Ultimate Guide to Supply Chain Security, or as always, you can reach out to the Eclypsium team at [email protected].
