For years, Hewlett Packard Enterprise’s (HPE) “integrated lights out” or “iLO” technology has provided out-of-band management facilities for multiple series of HP ProLiant and Integrity servers and microservers, including functions like reset, power-up, log management, authentication, CLI access and more.
Recent research from Iranian security firm Amnpardaz disclosed how threat actors are actively using BMC implants against HPE Servers in the wild. Known as iLOBleed, the implant affects HPE Gen8 and Gen9 servers using iLO4 and potentially earlier models that do not provide an iLO BMC hardware root of trust. Even newer Gen10 and iLO5 systems that implement a hardware root of trust can be affected if they are vulnerable to firmware downgrades as described here (PDF).
(For additional vulnerability and threat details, see CVE-2018-7078, CVE-2018-7113.)
iLOBleed has been observed in the wild since 2020 and has proven to be stealthy, persistent, and damaging. A BMC implant can provide virtually omnipotent control over a compromised server, and in this case, attackers used iLOBleed to completely wipe the victim server’s disks. Just as importantly, iLOBleed used the unique powers of firmware to do this repeatedly. Since the malicious code was hidden within the BMC firmware, the implant was able to persist even after the server operating system was reinstalled; enabling the attacker to repeat the cycle of destroying data after the server was recovered. Additionally, the implant took the added steps of silently preventing the system from updating the BMC firmware, while spoofing results to make it appear that the firmware had been updated.
While the specific threat actor(s) tied to the Amnpardaz research are still unclear, it is important to note the vulnerabilities in question apply to enterprise servers that are very common and are used in nearly all verticals. The vulnerabilities themselves enable a wide range of malicious impacts. Taking these two points together, the potential risks tied to this issue extend to virtually any industry and will be relevant to any number of threat actors and underlying motivations.
Eclypsium has done extensive research on BMC attacks, including demos on Remotely Bricking a Server which describe related attack techniques and highlight the need for tighter security controls over BMC firmware in servers.
Analyzing the HPE iLO Attack Surface
As a best practice, server BMCs should not be directly exposed to the internet but this is often not a reality in the field. Eclypsium research has identified thousands of vulnerable HPE iLO BMCs that are publicly accessible, with dozens showing obvious signs of ransomware infection.
Analysis using Shodan identified 7,799 HPE iLO BMCs ranging from iLO 3 version 1.26 to iLO 5 version 2.6. Without the protections of a hardware root of trust, all of these devices would be exposed to iLOBleed. Notably, the data further highlights how BMC firmware is rarely updated. 5,126 devices were running versions of iLO 4 and another 2,322 were running versions of iLO 3.
Of these devices, 44 showed obvious signs of ransomware infection based on data displayed on their publicly accessible login screen, such as the example seen below.
Source: https://twitter.com/M_Shahpasandi/status/989157283799162880
Additionally, it is important to note that these are only the BMCs that are publicly exposed to the internet. There are undoubtedly far more vulnerable devices that are not publicly visible. These devices would make high-value targets to attackers after gaining initial access to an enterprise network or datacenter.
The Risks of BMCs and Out-of-Band Management
Unfortunately iLOBleed is the latest example of attackers targeting BMCs and out-of-band management features built into modern devices. It is critical that organizations know what these components are, how they work, and their potential impact in the context of a cyberattack.
First, BMCs implement the Intelligent Platform Management Interface or IPMI protocol to provide a standardized interface for managing servers. Various remote management tools can interact with the BMC over IPMI or Redfish remotely manage a wide range of functions on the server. The specific functions that can be managed can vary from vendor to vendor or platform to platform. HPE iLO is HPE’s branded version of BMC and other vendors likewise have their own brands such as Dell’s iDRAC. However the functionality is largely the same across these versions and includes the ability to monitor and manage virtually everything about the server including firmware updates, changes to the host operating system, cycling the host, and so on.
It is important to note that the BMC is independent from the rest of the server in virtually every way. They will have their own independent networking capabilities and even their own power, allowing them to function even when a server is powered down. This “lights-out” management is actually the LO in HPE’s iLO.
Attackers can abuse these capabilities in a variety of ways. iLOBleed has demonstrated the ability to use the BMC to wipe the disks of a server. The attacker could just as easily steal data, install additional payloads, control the server in any way, or disable it entirely. It is also important to note that compromising physical servers can put not only workloads but entire clouds at risk. A recent analysis provided a real-world example of how BMCs could be used to gain full control of an organization’s cloud of over 25,000 virtualized hosts. As such, organizations should consider the risk that BMC vulnerabilities pose not just to disrupting assets, but to disrupting entire operations. iLO also supports the ability to update the system BIOS/UEFI firmware, enabling attackers to install additional implants within UEFI.
Other real-world threats such as JungleSec ransomware have used IPMI as a way to infect devices and spread ransomware payloads. This same strategy can apply to laptops as well as servers. Intel’s AMT technology provides a similarly independent out-of-band management interface, which has been used by PLATINUM as a command-and-control channel capable of evading the host-based firewall.
BMC vulnerabilities are also incredibly common and often overlooked when it comes to updates. Vulnerabilities and misconfigurations can be introduced early in the supply chain before an organization ever takes ownership of a server. Supply chain issues can still exist even after deployment due to vulnerable updates or if adversaries are able to compromise a vendor’s update process. Ultimately, this creates a challenge for enterprises in which there are many vulnerable systems, very high impacts in the case of an attack, and adversaries actively exploiting the devices in the wild.
What Organizations Should Do Today
Organizations with HPE servers should take steps to assess their devices for the presence of iOBleed. Next there are steps to help mitigate the risk going forward, and these recommendations are applicable to any organization with servers that use BMCs.
Detecting iLOBleed
Detecting iLOBleed can be challenging since the implant will interrupt any BMC update attempts but report fake results to make it seem like the firmware was successfully updated. However, researchers at Amnpardaz noted a very visible indicator. In short, while the implant falsely reports an upgraded version of iLO firmware, it does not include the new visual style used in the latest versions of the iLO user interface. The image below from Amnpardaz shows a valid version of the UI on the right and an infected version on the left.
The malware showing a fake version of 2.55 on a 2.30 UI.
The original 2.55 iLO login page.
Source: https://threats.amnpardaz.com/en/2021/12/28/implant-arm-ilobleed-a/
Eclypsium is including new behavioral checks to better monitor BMCs and detect inconsistencies in compromised systems, whether attacked by iLObleed or something else.
Reducing Risk of iLOBleed and BMC Implants
Organizations should also take the following steps to reduce their exposure to iLOBleed and other BMC implants.
- Isolate BMCs or other out-of-band management interfaces from production networks using VPNs, VLANS, firewalls, and other security technologies. Do not expose BMCs directly to the Internet.
- Monitor iLO Audit logs via Redfish or XML interfaces to catch malicious changes early. Securely configure the management interfaces by changing default passwords and configuring authentication and encryption mechanisms like certificates. For Gen10/iLo5 systems upgrade to the latest available version and disable the ability to downgrade firmware to prevent the vulnerability from being re-introduced to install implants.
- Record BMC version information and check against relevant CVEs. Eclypsium customers can use automatic vulnerability checks and change detection to identify and mitigate firmware vulnerabilities in critical server components including BMC (iLO etc) and UEFI.
- Frequently check manufacturer support pages for vulnerabilities and updates in all device components, especially BMC. Eclypsium customers can leverage built-in firmware update management capabilities to automatically discover available updates and deploy them to servers.
Establishing Trust and Next Steps
iLOBleed provides an incredibly clear case study not only on the importance of firmware security in BMCs, but for firmware security in general. Many organizations today have adopted concepts such as Zero Trust which defines the need to independently assess and verify the security of every asset and action. Yet, in most cases, these ideas have not made their way to the most fundamental code of a device.
Whether evaluating devices in the technology supply chain or those that are already deployed, organizations can not afford to simply trust that their firmware assets are free of threats and vulnerabilities. As with all their critical assets, organizations need to have visibility into their firmware, the ability to find and resolve vulnerabilities and weaknesses that put the asset at risk, and the ability to detect and mitigate the presence of an active threat.
BMC threats such as iLOBleed are real-world case studies into the importance of maintaining secure state of firmware in device fleets. The fact that a BMC is a physically separate and independent component makes it easier to visualize how a change in firmware can undermine the integrity of the entire device. However, many of the same issues we see in BMCs apply to all firmware. The logical differences and capabilities of firmware are far more important than its physical location on the motherboard. Firmware whether in a BMC, the system UEFI, or component such as a harddrive, sits beneath the operating system and if compromised can deceive and subvert the operating system. In the same way that iLOBleed lies to the user about the BMC firmware version, other firmware threats can lie to or subvert the operating system. This is why firmware and machine code is so powerful and so valuable to attackers – it is the end of the layers of abstraction that all computing is built upon. When this layer is compromised, the OS and higher layers can no longer have a trusted view of the system. This is precisely why security teams need the ability to independently Identify, Verify, and Fortify their firmware.
If you’d like to learn more, please contact the Eclypsium team at [email protected]. If an existing Eclypsium customer is experiencing an incident related to iLOBleed they can contact dedicated customer support at [email protected].
