Infrastructure Attacks Part 2: Hidden Risks to the Data Center in Remote System Management
In a recent blog we looked at a wave of attacks targeting the switches, routers, and firewalls within an organization’s network. This time we are going to extend this theme of threats to IT infrastructure to include servers and data centers.
Of course, servers drive the data and applications that truly define most organizations, and as such, you might expect that they are likewise the most secured assets in IT. Certainly, most security teams spend significant time and effort to properly harden their servers and ensure they remain patched and protected. But as we’ve discussed previously, there is a world of attack surface that lies beneath the level of operating systems and applications. System-level firmware can subvert the operating system and traditional security controls, and the firmware in hardware components such as drives, network adapters, memory, and processors can give attackers direct access to critical data. At Black Hat, we demonstrated remote attacks on UEFI system firmware. Vulnerabilities in the firmware update capabilities of commodity systems made it possible for a remote attacker to execute arbitrary code in firmware on certain systems.
However, when it comes to servers and the data center, the rabbit hole goes deeper still. With hardware dedicated to out-of-band management, modern servers contain a fully independent “computer within the computer” with incredible control over the host OS and all its workings. This is the world of baseboard management controllers (BMCs) and IPMI, and here we will analyze how this is one of the most high-value, yet least protected targets in an enterprise.
The Hardware Supporting a Virtual Revolution
Before getting into the details of the risks and threats to server hardware, it’s probably worth mentioning why this is a massive issue even in an age of virtualization and cloud. For the better part of two decades computing has evolved toward increasing layers of abstraction with virtualization, containers, dockers ruling the day. It was easy to simply think of hardware as a base commodity, and as a result, much of the security thinking has focused on how to bring security up into these rapidly evolving layers of virtualization.
Yet while virtualization has made hardware somewhat of an operational commodity, the hardware is still of critical importance to security. Data is ultimately stored in physical drives, processors must perform the actual computing, and the IO bus and network adapters must move the data. Attacks against hardware and firmware can thus provide direct access to data, allow an attacker to control the host OS and virtualized layers, or even disable the machine.
And when a weakness is discovered, this can apply to an entire data center or cloud. For instance, when attackers discovered an IPMI flaw in SuperMicro devices, they were able to build a botnet of up to 100,000 servers to drive massive DDoS campaigns. So while virtualization is the first thing that comes to mind when most people think of data centers and cloud, we have to remember that security still starts at the hardware. If you can own the hardware and firmware, you can own all the layers above.
The Computer Within the Computer
While all enterprise devices have a hardware/firmware attack surface, the problem goes even deeper when it comes to servers. Needless to say, data centers can consist of racks and racks of hardware, and managing all of those devices, operating systems, and other images is a considerable amount of work, especially when something breaks. As a result, modern servers typically possess a fully integrated yet independent system dedicated to management, with its own power, board, processors and interfaces.
Collectively, this functionality is known as the Intelligent Platform Management Interface (IPMI) and its main processor is the Baseboard Management Controller or BMC. Hardware vendors have their own branded version of IPMI (HPE iLO, Dell iDRAC, etc) but the functionality is largely the same. This includes the ability to monitor and manage virtually everything about the server including the ability to update firmware, change the host operating system, and boot or cycle the host. And since no one wants to physically run to a server and cable everytime there is an issue, these functions are available over the network, often on its own dedicated management subnet.
Weaknesses and Impacts
While IPMI and BMCs serve a critical need for data centers, they also present a massive risk. A remotely accessible interface that can control virtually every aspect of the host even when the host is powered off would be an obvious goldmine for attackers. Unfortunately, this critical area is often overlooked when it comes to security. IPMI can rely on default, widely-known passwords, and administrative access is often not logged. An attacker who gained access to the server via IPMI, could easily gain full control over the power of the data center. This include using the data center as part of a botnet as seen in the previous example, silently embedding malware to steal information, or even taking down the device or the entire data center altogether.
Given this importance, the BMC has become a particularly hot area of investigation for security researchers. One of the more highly anticipated talks at this year’s Black Hat will focus on new issues uncovered in BMCs.
Remote Attacks Against Firmware
Unfortunately the problem doesn’t end at the BMC. As UEFI system firmware has become more robust, network functionality has been added here as well, enabling wide range of remote management features such as full HTTP boot from a remote server over the Internet. Various hardware vendors have also added “update over the Internet” functionality, which allows the firmware to open a connection to a remote server and download firmware updates before the operating system is loaded. In our research, we have even found system firmware that can send emails from UEFI with the ability to mount NTFS partitions and attach any file from the system as part of the email message being sent.
ASRock, ASUS, and HP all have their own version of a “update over the internet” feature and as part of our research, we discovered that ASRock and ASUS are both vulnerable to a basic buffer overflow vulnerability allowing arbitrary remote code execution. In our recent talk at Black Hat we further described the ASRock vulnerability and showed off a proof-of-concept exploit which we wrote to demonstrate this issue.
Defending Your Infrastructure
As we saw in the case of networking devices, most organizations lack the ability to properly deal with threats at this layer of their infrastructure. Eclypsium gives security teams the visibility and control to proactively find problems and weaknesses, actively manage them, and even detect and respond to implants at the hardware level. By bringing real security to the areas where the stakes are highest, Eclypsium is truly able to secure the foundation of the enterprise.
So far we have seen how some of our most valuable devices in the enterprise can and are being attacked at the hardware level. In the next installment we will shift to how attacks against firmware can be performed both remotely, based on physical access to the device, and even in the supply chain. Stay tuned, and let us know if you have any comments or questions.
