Check our discussion of this between Paul Asadoorian and Scott Scheferman.
In the cybersecurity industry, we hear about “the root of trust” (RoT) quite a bit. It’s the most important thing after all, right? Yet, very seldom do SecOPs, risk management, or board level conversations revolve around the topic or action the enterprise or mission to directly address it.
Why is that? Quite simply, RoT is fundamentally important and typically and necessarily (by definition even) relegated to the makers of devices, certificate authorities, firmware authors, and other aspects of infrastructure that are meant to be trusted by the enterprise. As time rolls on, we forget about it entirely. We assume that this trust is immutable, trust-worthy, and that the implementation of said trust is designed properly enough to earn our trust. Yet, here we are in the year 2022 and we are being forced to re-examine the real-world impact associated with attacks that violate, break, or bypass the root of trust. What’s at stake is nothing less than the trust required to secure our entire compute stack ranging from IoT and mobile devices, to the cloud, and traditional IT and OT/ICS environments.
Nowhere is this more apparent than in the area of firmware security and in particular, the ability of the end-user operating system (Windows, Linux, Mac, etc…) to trust the software that lies beneath via the Secure Boot process. The vast majority of every security control we implement relies entirely on this trust. Every network security appliance. Every OS-level security control. Every AV, EDR, XDR, SOAR, IDM, MDM, SIEM <insert cyber industry acronym soup here>…relies upon the root of trust to function in a reliable, trusted, effective and pragmatic way to insure the integrity of the services provided. As often mentioned in this threat report, this is why attackers benefit from subverting technologies like Secure Boot; because they can then disable the security controls that rely upon and trust the secure boot process. For a more in-depth understanding look no further than this video podcast covering the jaw-dropping talk Eclypsium’s Jesse and Mickey delivered at DefCon 30 earlier this month, hosted by Paul Asadoorian. Attendees were stunned during the demo portion of the talk when the attacker sent a simple spearphishing email that resulted in the installation of a malicious bootloader allowing the attacker to connect to the machine via a Remote Access Trojan (aka RAT or Command-and-Control software) all while Windows continues to think Secure Boot is still working. Judging by reactions in the room, 95% of the audience had no idea such a remote attack scenario was even possible, let alone that it works on the vast majority of operating systems ever shipped. “One Bootloader to Load Them All”, indeed. For an extremely well-written piece on one person’s journey to understand Secure Boot in the context of running Linux as their primary OS, check out this blog by Paul Asadoorian.
One would think that the software developed to support the RoT mechanism itself and the management of the keys and their revocation mechanisms would both be developed and implemented with the utmost care, and that the downstream OEM supply chain that designs and builds computing devices would demand as much. Yet this is sorely not the case. The software written to support this function is every bit as vulnerable, complex and “messy” as software at the OS and application layers. What’s more, in recent years, this same software (the stuff we call firmware) has also become much more extensible, and creates a bi-directional pathway between the user OS and the firmware underneath. The same OEMs forced to trust a vulnerable RoT, are the same OEMs that author drivers that help the end user at the OS level interact with and update the firmware beneath it, opening up further avenues of attack. Taken together (drivers, firmware, and inherited RoT mechanisms from upstream providers), a single device represents an entire ecosystem of software that is commonly leveraged by adversaries to compromise a device at the lowest levels.
The narrow definition of RoT is normally tied to cryptographic systems. It’s meant to be a source that can always be trusted. More often than not the RoT scheme is generally tied to a hardened hardware security module (HSM) that generates and protects keys and performs cryptographic functions securely. This is normally what we mean when we say ‘Root of Trust’.
While this narrow definition of RoT is the industry standard there are other ways to frame this problem and the resulting attack surface. One way is to turn it upside down and view RoT from a different direction in terms of how trust ‘flows’. In the classic sense above, it flows ‘from’ the hardware and firmware and ‘up’ to the OS and applications sitting on top of it. But it is equally important to ask ourselves: “Should the RoT itself trust the software above it to not serve as an IV (Initial vector) that can be used to attack the RoT?” Like a tree with roots, diseases can flow from soil up to the branches, but can also flow from bugs on the leaves all the way down to the roots. Either way, the tree can be attacked. Either way “trust” is ultimately a two-way street.
Another way to expand the definition of RoT for pragmatic purposes is to think of it as trust in the firmware itself, not just the cryptographic system that assures its integrity. In this sense a user-level OS might be able to know the firmware is signed cryptographically, but does it know, or even have the means to know, whether it can trust the quality of firmware, the configuration of the firmware, or the vendor implementations of a specific feature designed to protect the firmware from attack? Does it know whether a backdoor has been implanted from the factory or the rest of the supply chain, or from a firmware update process that can be abused by attackers like ShadowHammer or via a decades old https bug like this?
If we don’t expand what we mean by RoT in these additional senses of the word trust, we will continue to make the incorrect assumption that signed software indicates secure and benign software. We’ll continue to make this assumption at the technical level (as well as at the national security and policy levels) when addressing third party IT supply chain risks. Somehow the entire IT supply chain vertical has been making the wrong assumption about the essence of what signed software provides, which is ultimately nothing more than the integrity of that software, whether or not it is securely written, implemented correctly, or backdoored (compromised earlier in the supply chain).
Once we come to this expanded view of RoT, we can finally begin to address the true core problems with technologies like Secure Boot and every other piece of firmware shipping with a device. In the end, there are no cryptographic systems left that aren’t ultimately reduced to nothing more than a software stack running on hardware. The truth is, the software we trust on any modern device is authored by dozens upon dozens of authors from different locations around the world. It is sloppily written. Much of it uses OSS (Open Source Software) libraries that can be backdoored and have been the subject of much debate and research. There is no raw accountability in place to help ensure it was written securely, and there is no economic incentive to deliver it as such, either punitive or rewarding. On the contrary, the entities that sit at the very top of the IT supply chain are incentivized to build the most flexible, feature-rich, and ‘default-enabled’ set of features possible, usable by the widest set of downstream OEMs. This is precisely why many firmware bugs disclosed to their authors end up re-appearing in subsequent versions of the firmware. This also exemplifies another key reason why firmware remains so vulnerable over time: It is dynamic…constantly being developed and expanded in both functionality and complexity. Quite frankly, there is nothing firm about it!
One of the talks that garnered attention at Black Hat was called Glitched on Earth by Humans: A Black-Box Security Evaluation of the SpaceX Starlink User Terminal, given by researcher Lennert Wouters, who spent $25 in parts to hack a Starlink disk’s firmware and subvert its boot process in order to run arbitrary unsigned code. The attack serves as a quintessential reminder that firmware lies at the intersection of hardware and software, and that bugs in either can result in unintended behavior. In this case, physical voltage is applied to trigger a fault injection flaw allowing an attacker to bypass the signature verification process and run customized firmware. In turn, the attacker gains access to any of the functionality present and could potentially lead to access to the Starlink network’s backend servers. Our own Wes Dobry describes it like this:
“The Starlink User Terminal Modchip shows that with enough time and physical access anything is achievable. Even with a walled garden, a sophisticated and well-timed voltage fault can be leveraged to bypass the fundamental security check allowing an attacker to have complete access to the terminal and thereby the ability to perform attacks and reconnaissance on the entire Starlink network.” – Wes Dobry, Eclypsium
One of the foremost challenges surrounding the expanded RoT definition put forth above is simply the size, scope, complexity, shortage of expertise and absence of accountability within the supply chain itself. Suppliers at the top of the IT supply chain do not interact directly with end-users and enterprises. Instead, they ship firmware and hardware to mid-tier OEMs and those in turn supply the larger well-known OEMs, which end up being the brands the end-users interface with. This unfortunate structure means that there is little incentive for upstream suppliers to spend time and money on shipping secure code or hardware. Yet, when it comes to the foundational software for the entire supply chain below it, there is no shortage of high-impact vulnerabilities. What a security researcher might call a vulnerability, the upstream providers might call a feature. What a researcher might call a backdoor, the supplier might call an operational requirement in the context of support services and SLAs with downstream vendors. Regardless, this software collectively forms the expanded Root of Trust paradigm that plagues the industry. The end result is that the enterprise is faced with tremendous supply-chain risk for which they have little ability to see, let alone mitigate.
Even when large OEMs like Microsoft take measures to protect the RoT with technologies like Secure Boot and Secured-Core PC’s, the rub still lies in the way OEMs implement and configure devices they ship. Here’s a question: has there ever been a PC shipped from an OEM with the entire set of Secured-Core technologies enabled and configured properly? If there has been, we haven’t seen it yet. But we would like to! Remember the Everyone Gets a Rootkit blog showing how every Windows computer since 2012 can be rooted via a single vulnerability? The demo was performed on a brand new Secured-Core PC. The demos at the DEF CON 30 talk given by Jesse and Mickey? The same. Our lab has a Secured-Core laptop in it running a firmware implant as a simple conversation starter. Just the UEFI alone would need every one of these items in place and configured correctly as a suite of technologies. But other than being an Eclypsium customer how does a typical enterprise even begin to assess this below the surface risk, or even just start the conversation with executives and the IT/security staff in charge of protecting it? They don’t. And that’s why the very worst cyber adversaries that stand to do the most harm specifically target firmware and have been doing just that.
It’s not only SecureBoot and Secured-Core technologies, but critical technologies like BMC’s and Intel AMT/ME that present challenges. An attacker able to abuse their position inside of Intel ME can read all open files, examine all running applications, track all keys pressed and mouse movements, and even capture or display images on the screen. Effectively, the operating system above it doesn’t even have a say in the matter and is trusting the device and all of its firmware to not spy on the end-user’s running processes. The entire Root of Trust paradigm ends up being broken because Intel ME subverts it by design. It’s meant to provide its added functionality across the network, independently of the primary user OS. Once again Paul Asadoorian was intrigued by this and wrote up another excellent blog exploring these issues and related vulnerabilities.
The biggest challenge in any cryptosystem is often the secure storage of keys. Intel SGX is a technology meant to protect secrets even if the device is compromised by creating a secure enclave that cannot be accessed. Countless technologies trust this SGX enclave to function reliably. Casually put, this is the same enclave that allows a messaging app like Signal to protect the user’s private keys. However, Recent research has uncovered an architectural hardware vulnerability that jeopardizes the secure enclave SGX provides to these other services. This allows attackers to access things like AES secret keys and RSA private keys, and in-turn break trusted environments and cryptosystems. This serves as an example of how we ultimately ground RoT in hardware, and if that hardware has flaws, it affects every other trust system built on it.
Another way RoT can be effectively subverted is for the attacker to simply leverage a fully functioning RoT process to their own advantage. Why attempt to break RoT systems when you can simply bring along a driver sourced from a video game that is already signed and trusted (and therefore completely whitelisted by Windows and AV/EDR), and use it to run privileged commands for user land? We call this BYOD (Bring Your Own Driver), and it’s an extremely effective and evasive way to act as root on a system even if you are not root. Ransomware actors have been doing this for many years, but the first high-profile attack was on the City of Baltimore in 2019, when Robbinhood ransomware actors leveraged a motherboard driver from Gigabyte to gain kernel level access. A fully signed driver that was never revoked by Microsoft’s RoT mechanism despite a vulnerability that had been disclosed a year prior. This driver remained both vulnerable and trusted (not revoked) for years after the attack. BYOD style attacks have become quite common recently, and even last week a new one was discovered in the wild. This time the attackers used a driver meant to accompany a video game (Genshin) to gain root-level access just by bringing the driver along for the ride during the attack chain. Once again, a perfectly working RoT mechanism in Windows is simply used to both elevate privileges and evade defenses as a result.
The best way to summarize the above examples of broken trust is for us to realize that at every level of the stack there are ways to move up or down it that present-day RoT systems are ill-equipped to prevent. Collectively, these ‘trust problems’ are the result of a disjointed, unaccountable supply chain structure that is presenting a clear danger to the entire computing landscape. We’ve grounded our Root of Trust in sloppy code, flawed hardware, and a vertical IT supply chain that passes flaw after flaw all the way down to the computing device you are reading this on. The only way quantitatively and qualitatively ascertain what this risk presents to enterprises and missions alike is to tackle it head on. In effect, we are being forced to take back security control of the devices under our area of responsibility. Today, that can only be done with an enterprise and mission grade solution like Eclypsium. To help you begin this conversation with your stakeholders and other third parties, we suggest looking over the results of this extensive study of firmware security. We learned that 76% of you confess to significant gaps in your awareness of your firmware footprint, and that 10 out of 10 of you fully acknowledge you have the very same “firmware blind spot” in preventative and reactive detection of firmware vulnerabilities and threats jeopardizing trust across the environment. But don’t worry, we’re rooting for you and here any time to talk about how to find these flaws before the bad guys do.
