PODCASTS

BTS #62 - Unpacking the F5 Breach, Framework UEFI Shells

In this episode, the hosts discuss the recent F5 breach, exploring the implications of the attack, the tactics used by threat actors, and the importance of vulnerability disclosure. They delve into the complexities of securing network edge devices, the challenges posed by Linux security, and the need for standardization in security practices. The conversation also touches on the future of firmware security and the necessity for proactive measures in incident response. We also close out the show talking about the recent Framework UEFI shell vulnerability.

Subscribe

Transcript

Paul Asadoorian (00:30.14): Alrighty, here we go.

This week we discussed the recent framework UEFI Secure Boot Bypass and the recent F5, the breach, attack campaigns, vulnerabilities, threats and more. Stay tuned below the surface. Coming up next.

Paul Asadoorian (00:52.34): Welcome to the Below the Surface. It’s episode number 62 being recorded on Thursday, October 16th, 2025. I’m of course, Paul Isidorean joined by my co-workers, Mr. Chase Snyder is here with us. Chase, welcome.

Chase Snyder (01:05.42): Hey guys, hey Paul, hey Vlad.

Paul Asadoorian (01:07.752): Vlad Babkins here with us, Vlad, welcome.

Vlad Babkin (01:12.411): Hello

Paul Asadoorian (01:13.138): And as always, before we dig into the topics today, below the surface listeners can learn more about Eclypsium by visiting Eclypsium. com forward slash go. There you’ll find the ultimate guide to supply chain security, an on demand webinar I presented called Unraveling Digital Supply Chain Threats and Risk, a paper on the relationship between ransomware and the supply chain, and a customer case study with Digital Ocean. If you’re interested in seeing the product in action, you can sign up for a demo. All that at Eclypsium. com.

forward slash go. I take it you guys probably want to start with the F5. I know. Is a new story going on?

Chase Snyder (01:48.482): What happened with that five? There’s some sort of new story going on.

Chase Snyder (01:54.636): I haven’t caught up. I haven’t caught up on the news lately.

Paul Asadoorian (01:57.8): You know, it’s interesting. Oftentimes when there’s breaches, because breaches happen all the time, I don’t often cover them on my podcasts because oftentimes there’s not interesting stuff to talk about. hey, we got breached and we’re not really telling you how. And yeah, we were breached. I’m like, so what are we going to talk about? But in this case, it’s kind of interesting because we got a little more information, perhaps than we normally do from a breach.

and F5 disclosed earlier this week that they had suffered a breach that, well, we’ll talk about it, what they have confirmed that what the attackers have done, they engaged with, was it both IOACTIVE and NCC Group, like heavy hitters in our industry to determine exactly what the attackers got. And so we know at this juncture,

We know that the attackers were able to exfiltrate some source code for F5 products. We know that they were able to get some customer configuration data. believe it was like the knowledge base or help support system that they were able to access. We know, least we don’t have evidence to the contrary, that F5’s software supply chain had not been hampered with.

In other words, if you are applying firmware or got firmware on your F5 devices, we’re reasonably sure from everything I’ve read that it has not been backdoored. I say that with a little bit of hesitancy because attackers were alive and active in F5’s network. Did they say for how long? Were they able to put a time frame on that?

Vlad Babkin (03:42.209): One year, I think.

Chase Snyder (03:43.124): I don’t remember having read a timeframe. mean, the disclosure came out on, it was reported in the news yesterday or two days ago, and they said that it happened or they figured out on August 9th. So yeah, I don’t, I don’t, I don’t know. I didn’t see any other sort of timeframe information in there. Did you just say one year, Vlad?

Vlad Babkin (04:08.617): Yeah, so from what I have read in multiple different sources, it was one year, I think.

Chase Snyder (04:13.282): Looks like you might be on mute or something, I’m not hearing you.

Vlad Babkin (04:16.757): What’s a… Why? Do you hear me?

Paul Asadoorian (04:17.388): you can’t hear Vlad. Vlad, put the mic a little closer. Put the mic a little closer to your mouth, Vlad. You need to adjust the boom arm.

Vlad Babkin (04:22.593): Can’t. This is as close as it goes. Give me a moment.

Paul Asadoorian (04:28.99): Can you hear him now? Okay, go ahead Vlad.

Vlad Babkin (04:32.673): I changed nothing just in case. Anyways, from multiple sources I have… Yeah, I think my connection is lagged, so I’m lagging behind you guys. Anyways, from what I have read in multiple different sources, it is one year. So they were backdoored for one year by a thread pack.

Paul Asadoorian (04:37.352): I think there’s a little bit of lag on your connector, but.

Vlad Babkin (04:56.033): So in at least one news source…

Paul Asadoorian (04:56.5): We believe that threat actor to be UNC.

We believe that threat actor to be UNC 5-2-2-1.

Vlad Babkin (05:06.717): Yes.

Paul Asadoorian (05:07.284): Suspected China Nexus threat clusters is what the Mandate report that came out. I believe today was that today Oh, no, this is this is from a previous September 24th report from Mandy and about brick storm the stealthy backdoor Enabling espionage and we believe there’s a link where it’s been published that these threat actors UNC 5 2 2 1 because we can’t come up with better names than that In brick storm is the malware activity

Vlad Babkin (05:13.513): Yeah, we do.

Paul Asadoorian (05:35.673): and the MO of this group is long-term access, makes the year, you know, time frame make more sense.

Paul Asadoorian (05:47.508): just pausing for the lag with the lathe hopefully it improves as we record.

Vlad Babkin (05:51.873): Yeah, so the lag is huge. So from what I can tell, so yes, we do suspect that it’s China-based attackers. At least I have read it in one news source. So…

again, do not drop any names, I honestly don’t remember which one of them was mentioned in China. So it was China and it was UNC 5221, think. yeah, so this was a malicious activity that they backdoored of F5 for a long time. So they were in the infrastructure for a while. So a lot of things could have happened. So from what we read, I think a bunch of

code and documentation was disclosed for F5, potential unreleased vulnerabilities, et cetera, et cetera. So there is quite a lot going on in there.

Vlad Babkin (06:47.777): So at the same day, F5 published…

Paul Asadoorian (06:49.342): Chase, do you want to add anything to the, like what’s been, oops.

Vlad Babkin (06:54.81): Yeah, that’s a leg turbo.

Chase Snyder (06:55.35): Yeah, I was looking around just to see if I could get to that verification of that. Yeah, it looks like 44 vulnerabilities. People are speculating about a year. Yeah, the I mean, there’s a trend happening, right? OK, it looks like Brickstorm was previously known or this UNC 5221 that we’re talking about was previously known to be targeting Ivanti devices. All of the sort of Typhoon branded.

Attack groups also are well known for targeting network devices, network infrastructure, network edge stuff. There’s a trend happening of the supply chain for network edge devices being targeted. And this is sort of a new thing happening in that trend where previously these attack groups were focusing on known vulnerabilities, accelerating how quickly they would exploit.

vulnerabilities that had been disclosed. Now this is like the provider of the network stuff has been compromised. It’s not just accidental vulnerabilities that they shipped. It’s like the provider has been compromised. Maybe they didn’t get back to it yet, but it’s like if we think about the solar wind situation a couple of years ago where the update pipeline of solar winds was compromised and can be used to deliver malicious binaries or you know.

malicious updates. That may or may not have been the end game in this situation, but it attackers are going to places where they can get more leverage, right? The blast radius of a successful attack on a global network device provider like F5 is potentially huge. and so was just more bang for their buck to hack F5 than it is to hack any individual organization that uses F5.

Paul Asadoorian (08:48.616): Right, and also what they got were undisclosed vulnerabilities. I glossed over that. I didn’t mention that yet, but there were a number of vulnerability disclosures that F5 was working on that fell into attacker hands, which is why we see, would you say 44 different, I haven’t read through all the CVEs. Did they, they pushed like 44 updates or fixed 44 vulnerabilities? Is that true?

Chase Snyder (09:14.956): Yeah, that’s the number that they said, but I haven’t seen any details about what the actual vulnerabilities are yet. And maybe that’s out. I mean, this is, you know, coming out in real time, like new stuff being published on the hourly today.

Paul Asadoorian (09:24.38): Yeah. Go ahead, bud.

Vlad Babkin (09:26.625): There is actually a link with those. Like a F5 and there are F5 incident link. I don’t actually have it on my hand right now. There is a link to a bunch of vulnerabilities which they published at the same time. The link is right on the incident. So if you click and go there, there are all of the vulnerabilities with details on them. I’m not sure if there are any public exploits for any of them, but we know which vulnerabilities exist. So from what I understood, F5 is not actually…

Well, marketing is probably not the right term in here, but they don’t think that those vulnerabilities are anyhow related to attacker activities or etc.

Paul Asadoorian (10:01.458): Yeah, I have a theory. I have a theory as to why. So if you’re a threat actor and you’re inside F5’s network and you’re like, ooh, look, a whole bunch of vulnerabilities that haven’t been disclosed yet. Well, let me take those. Now, as an attacker, are you going to go and write exploits for those and start flinging around on the Internet? Because if you do, then F5 and others are going to be like, wait, no one was supposed to know about those vulnerabilities yet. Something is up. So if the attackers were smart.

they would collect those vulnerabilities, write exploits for them, not release them anywhere until such time maybe the CVEs get pushed and now they’re like, okay, now we can go forward. It’s the secret thing. It ties back to World War II, right? We broke the Enigma codes, but we had to be careful how we used the information because if we used it in certain ways, then the adversary would know that we were able to break the codes. And this is the same situation, I think, with the attackers. They didn’t want everyone to know that they had stolen

vulnerabilities that hadn’t been released yet. So they held on to the zero days. That’s why we’re not seeing them exploited in the wild just yet. It’s kind of like, I mean, now the cat’s out of the bag. So now, of course, the attackers, in my mind, will start flinging them on the internet and compromising things as people work really fast to patch these issues. But also now with access to the source code, they could potentially find more vulnerabilities potentially more easily and more reliably with access to the source code.

Vlad Babkin (11:30.305): they could have already found some which we don’t know about yet and that they’re just keeping, I mean, why the hell exploit every single zero-day you find? If you can keep it for a while until people start patching up and then use zero-days once people stop looking.

Paul Asadoorian (11:39.283): Right.

Chase Snyder (11:45.41): Yeah, information theory. Pioneered by Claude Shannon in World War II, the namesake of the Claude AI chat bot from Anthropic.

Vlad Babkin (11:55.979): Yep. Information theory is actually pretty amazing. According to that, there is no such concept as disinformation. just doesn’t exist. Like, Claude actually merged information starting with zero bits. There is no minus one bit. This implies that whatever information you give that’s above zero, even if you try to deceive your attacker, if the attacker is clever, he will be able to see through you and get more information that he would have otherwise gotten if you kept quiet.

Paul Asadoorian (12:28.02): It’s interesting too, the Ars Technica article that was published today reminded me of another event that happened. I think it was, was it Ryan Dewhurst that was commenting on LinkedIn about this? It’s kind of funny. I saw that fly by somewhere and I’m like, wait, I remember Ryan. He’s the guy that wrote Damn Vulnerable Web App. And he’s at Watchtower now, which is a company that I love. And I haven’t spoken with Ryan in a while, but.

Hi Ryan, miss you man. anyway, he’d pointed to the F5 signing certificate and key rotation that happened on October 13th and F5 says as a result Big IP and big IQ TMOS product releases are signed with all new certificates and keys and It you were speculating is this due to an overabundance of caution

Or was there evidence that would make it so that this was a required activity?

I’m kind of leaning, I don’t know, maybe I’m usually I’m not on this side, but I’m thinking it’s more of an overabundance of caution. They were like, look, we just had this big breach, probably a good time to rotate our signing keys. Just in case. I mean, just it was probably a just in case. If I’m at five, I’m like, you know what? Like regardless, like just in case, let’s just rotate the signing keys.

Vlad Babkin (13:58.493): If I would be a 5, and I would be operating for this many years, all of my signing keys would be on actual physical hardware. I wouldn’t actually store them as key files. I would store them in the hardware security modules. Like, you know, at the very least, a Yubi key. And store it there. Because, storing them as files at this level of the game is just your security decision. So really hope they didn’t leave them as files for attackers to just grab and potentially do it.

Paul Asadoorian (14:10.674): You know, right.

Paul Asadoorian (14:27.806): Right. Right.

There’s so many things about this article, right, about this story in this incident. CISA has ordered all federal agencies to remediate and patch everything.

Chase Snyder (14:44.61): Yeah, emergency directive 2601, which this is the, this is the second emergency directive we’ve covered on this podcast. Maybe this month, the last one was the big Cisco ASA sweeping range of attacks. And I feel like we’re getting a lot of signals to be able to effectively predict future attacks lately. and how, don’t know how many of these have to happen before people start taking more proactive action.

Paul Asadoorian (14:44.724): Which is also interesting.

Paul Asadoorian (14:59.485): Right.

Chase Snyder (15:12.77): But the Cisco thing was very effectively predicted by grain noise. They put out that alert that was like, hey, by the way, Cisco ASA devices are getting scanned at a much higher rate than they had been recently.

Paul Asadoorian (15:24.552): Right. And SNMP fell into that as well in the articles I read this week pointed to SNMP being exploited in the wild on Cisco devices. Yep.

Chase Snyder (15:33.902): They did? Okay, yeah, it’s happening. And then, okay, so right now, here’s something you could do to take proactive action because GreyNoise had just published another article that said that Palo Alto devices had like a massive multi-hundred percent increase in scanning too. So if the pattern holds, you know, it’s no guarantee, but there seems to be a pattern that increased scanning that looks like that is followed by attacks. And obviously the F5 situation is a little bit different.

But yeah, just because those vulnerabilities have been patched doesn’t mean everybody’s going to get the patch out in time. And as you said, now that the patches and the CVEs are out there, there’s no information theoretical reason not to use the exploits. know, cats out of the bag. So likelihood of exploits against these in organizations that are slow to patch seems high.

Paul Asadoorian (16:20.979): Right.

Paul Asadoorian (16:31.346): Yeah, and I don’t want to knock F5 because we cover this kind of thing all the time. Companies get breached. Vendors have issues with security and they’re constantly fixing things. Cisco was the one we were covering even in recent episodes. Now it’s F5. I think what we’re the thread we’re pulling on is more so than ever now.

Vlad Babkin (16:31.964): if not already happened.

Chase Snyder (16:34.124): Yeah, sure.

Paul Asadoorian (16:59.218): Given the recent news, even just looking at Cisco and F5, where the threat actors are concentrating their efforts certainly is targeting at network edge devices or network security appliances. What do we want to call these? Do want to call them edge devices or do we want to call them network security appliances? mean, edge devices is easier to say, but they’re not.

Chase Snyder (17:19.608): Yeah, they’re not all security either. Like Big IP has load balancers or application delivery controllers, a lot of it. I mean, it does a lot of VPNs and firewalls, so that’s security, but it’s also routers and switches, which are not, you know, foundationally security. I think network edge or network infrastructure is the best catchall bucket for it.

Vlad Babkin (17:35.582): mod

Vlad Babkin (17:41.218): In modern network, every single device has little bit of security, if you think about it. Like, if you think about modern routers, like I have a router at home, which is just a home model of Mikrotik, which costs like, what, $200? It’s supposed to be a home router and nothing else, but it is actually VPN gateway. It’s a file server. It has a web server in there, so I can host something right on the router. Like, it can even host Docker containers.

Paul Asadoorian (17:45.394): Yeah, sure.

Paul Asadoorian (17:54.632): Mm-hmm.

Paul Asadoorian (18:03.06): Yeah.

Vlad Babkin (18:06.401): So not calling it a security appliance would be an understatement, but it’s also not purely security appliance. So it’s like edge device or internet facing device, whatever you want to call it.

Paul Asadoorian (18:20.06): It seems to me attackers have doubled down on this activity. We’ve been tracking this activity for a couple of years now and we’ve seen indications that, like they’ve got a zero day for this platform, zero day for that platform, or they’re patch-diffing and coming out with an exploit really fast and compromising. You know, their entry point into the network, and I think the Verizon DBR was the latest data point that pointed to this, their entry, attacker entry point into the network is increasingly…

a network edge device. And I think we’ve talked about this before, Normally you’d fish or use some other mechanism to get in, but this is the initial access vector that now attackers are doubling down on for all the reasons that we’ve talked about. And I say doubling down because if you look at the two Cisco, you look at ASA and you look at SNMP, those were big events, right, in the threat landscape. Now we’re moving to

F5 and attackers decided to spend an entire year, I mean roughly, I don’t know if we know for certain, right, but let’s go with a year, lurking in a major enterprise vendor’s network to collect intelligence to continue their campaigns that are targeting these devices to get into organizations. And so there’s been several of these wake up calls and

I feel like defenders were still at a disadvantage today. You guys agree that it’s my pulp fiction briefcase analogy? Don’t look inside the briefcase. I dropped these devices on my network. I can’t look inside them. So attackers do, and attackers are exploding them.

Vlad Babkin (19:56.044): under.

Vlad Babkin (20:03.405): The problem with this Defender is that the Defender must be aware of every single device. Like, you cannot just drop a random device into a network and just forget about it, especially if it is facing the outside. But because you have all the limited tools to look into it, you have a problem. Like, if I put it straight. But that’s not even the worst part, right? So the Defender must be aware of every single device, of every single service on it, and devices are getting more complex.

Paul Asadoorian (20:33.266): Yeah.

Vlad Babkin (20:33.749): Like, not only can you run an antivirus for them, there are lot more services running in every single device. most home routers nowadays that you can see are probably running some kind of complete Linux distribution or maybe BSD or something like this. It’s not a single binary like it was before. So there is a lot more surface, a lot more passes for attacker to actually abuse that device. And even if you have full visibility into that, it will still be at a disadvantage.

Like, there is just too much you have to worry about. And consider that, again, in modern world, it’s not just enterprise hardware. It’s also normal consumer stuff.

Paul Asadoorian (21:04.723): Yeah.

Paul Asadoorian (21:17.83): It’s interesting, I go back to my Windows analogy, that when we bring Windows devices or Windows operating systems into our environments, I think there’s two things that we rely on to provide robust security, that’s not the right word, but enable really good security architecture. And one is we bring in vulnerability management and we might put an agent on our Windows devices.

Whose sole job it is to identify vulnerabilities and report them back so that we can remediate them. And we also add some type of, however you want to say it, antivirus, anti-malware, EDR. Those could be three different things, by the way, three different products. We tend to use EDR as all-encompassing kind of thing, but we add on software to Windows devices that provide some level of protection against malicious code, a level of detection. For that and the R enables some type of response. But when we bring in an edge network device appliance, we don’t have those type of capabilities. not by default going, yep, going to put the agent on there that’s going to tell me when it’s vulnerable and help me patch it. And I’m going to add some other piece of software to it that’s going to prevent it, hopefully, in some cases, from being exploited.

If it is exploited, it’s going to hopefully detect some type of malware or malicious activity on it. And by the way, it’s going to give me great telemetry so that I can go back and do incident response and forensics. We don’t have those tools. I think that’s the number one reason why attackers are like, well, this is great. All that stuff isn’t there. So why am going to go after a Windows system as my first entry point? I’m going to go after these devices. I can dwell. I can collect telemetry. can collect traffic. I can even collect credentials.

Because I’m on this choke point inside the network that’s not being monitored the same way we monitor other infrastructure devices in our environment.

Vlad Babkin (23:23.841): level open is secret if you think about your average website like most administrators don’t install any anti malware on the web server for example or EDR like there is barely anything running in there that’s the same kind of problem even though you have full visibility into it you still face the same issue and this makes lot of sense why why Linux attacks are on the rise

Paul Asadoorian (23:31.39): Right.

Paul Asadoorian (23:43.25): Yeah.

And so most of these devices are Linux and or BSD based. And the over the years, there’s been this notion, and I’ve heard many different analysts, practitioners, security researchers say this over the years. And I’ll give you a recent example as well. But over the years, I hear that anti-malware EDR vendors make a great product for Windows and put a lot of R &D into that Windows product.

Now organizations adopted Linux over the years and to meet compliance requirements, which I’ve firsthand seen this, your Linux devices have to run some type of EDR or antivirus or anti-malware in order to check that box for compliance. And what many of the vendors did was just kind of poured over basic functionality to support Linux. And it is not as comprehensive as

of coverage as it is on Windows. And there was a recent post, I believe it was from Praetorian, that tested this even recently and stated in their report that the state of EDR on Linux systems is still really poor. And I don’t want to pick on the EDR vendors. I’m just, this is like something I’m observing. As someone who runs Linux, I want you to be awesome at protecting Linux. But I think also that’s really hard.

I think Mac OS is probably easier to defend because it’s such a solid platform that you really can’t be tampered with very easily. It’s a known set of standards. don’t let you… I think they even killed kernel modules. You can’t even write your own kernel module and load it on Mac OS anymore. So it’s a very tightly closed system. You get to Windows, still a closed system, but slightly more open because it has to run on all kinds of different hardware.

Paul Asadoorian (25:44.072): But you get to Linux, and every Linux installation is conceivably different. companies have standardized, of course, standardized on the hardware, standardized on their Linux systems, right? But oftentimes, Linux is so customizable. I think I saw a meme that was like Dr. Strange, right? And the Linux user is like, so I want to modify the kernel source code.

And it’s like, okay, that’s weird, but I’ll allow it. It’s like, you can do that in Linux. And every Linux is its own special unique snowflake. Therefore, if you want to add some protection to that, those options are vast and heavily dependent on how your Linux system is configured. And by the way, if you want to update your Linux system, all that stuff has to update along with it, which can be, I think, more complex on Linux than on other operating systems.

Because you’re on this kernel version, you want to go to that kernel version, well that module is not compiled, and now I’ve got dependency issues, I we could go on and on, right, about the challenges with defending Linux systems because it’s so highly customizable, all open source.

Vlad Babkin (26:54.913): have to make a small comment here, like which is both agree and disagree. I agree with you that a lot of users can customize the hardware, but how many users actually do customize the kernel? Like, besides implementing a bunch of kernel modules, right? Kernel module is probably something you might install just because, oh, hey, you need the kernel module for this network card that you have. Sure, here you go, kernel module installed, but that’s the point, it’s a module. Everything else is not.

Paul Asadoorian (26:56.894): Mmm.

Paul Asadoorian (27:16.819): Mm-hmm.

Vlad Babkin (27:24.001): and your average user will just install something like Ubuntu. And… Yup. So there is a lot of very standard templates for Linux. So while you cannot protect them all, you can protect the most common ones and still hit the good mark and make, oh hey, so this Ubuntu configuration, well, for it we can do this verification because it’s very standard. Like, oh hey…

Paul Asadoorian (27:29.736): Yeah, and people do standardize on that too.

Paul Asadoorian (27:41.757): Right.

Vlad Babkin (27:52.017): There is only this many kernels that are released by Ubuntu and if Ubuntu is not running one of them, that’s something to flag, like to inform the user, or hey, are you sure that that’s fine? That’s okay? And whatnot.

Paul Asadoorian (27:59.155): Mm-hmm.

Paul Asadoorian (28:06.45): Right. The problem is, if you’re maintaining software that is tied to a Linux kernel version, you have to support a number of different kernels. Because that’s going to vary. Even on the same distribution, someone could be running a different kernel. I’ve run into that challenge here at Eclypsium too, right? Everyone runs a different kernel version.

Chase Snyder (28:30.092): And this, so the thing of like not everybody customizes the kernel is super true for consumers. But in the case of these like network vendors where their devices are running some flavor of Linux, how likely is it? Do you think that they’re like, we talked about F5 big IP, the operating system underneath that is a flavor of the CentOS Linux. So what’s what level of difference or customization?

Paul Asadoorian (28:52.361): Mm-hmm.

Chase Snyder (28:58.062): Is there in that that they use to change that from CentOS Linux to a big, I think, IPOS.

Paul Asadoorian (28:59.977): Yeah.

Paul Asadoorian (29:04.766): They’re all different, Even the user lane is different on all these appliances. It’s Linux, but it’s F5’s Linux. It’s Palo Alto’s Linux. It’s whatever vendor has customized Linux for their platforms.

Chase Snyder (29:21.666): which then stops being open source per se. It’s not like you can go out and get, yeah.

Paul Asadoorian (29:24.852): It’s still open, but yeah, it’s not ubiquitous. It’s their own flavor. Basically, everyone has their own flavor. And that can vary across all of their different products and versions of their products too.

Chase Snyder (29:35.042): Yeah. And their customer, their customizations are not,

necessarily documented and available for the Xerta.

Paul Asadoorian (29:43.08): Yeah, but that’s a great point. That’s why they don’t want you messing, mucking with the platform because it’s not like a regular Linux distro where you can go in and do stuff. It’s highly tuned for that specific platform. And if you go messing with stuff, now they’re going to have more support tickets, right? I think support is certainly one reason why they’re like, you can’t get in there and start mucking around with the Linux underneath. know, Palo Alto has been publicly stated, like, we don’t want you rooting your devices and gaining that. That’s not something they want customers doing. And I think that’s a support issue.

Chase Snyder (30:11.63): Yeah. There’s a perception of security, not actual security by obscurity, but a perception of security that is created by the obscurity where they’re like, don’t look inside the box. Don’t look in the briefcase. We are this giant company with a skyscraper with our name on it. Don’t worry about it. And there’s sort of cracks in that facade coming through now where it’s like, okay, the more times that we see a big attack.

Paul Asadoorian (30:22.972): Yes. Yes.

Paul Asadoorian (30:33.94): Right.

Chase Snyder (30:41.9): that leverages a vulnerability that’s inside that sort of walled off area, but we know it’s just Linux in there, the more it’s like, okay, well, we need to be able to do our own security on it. We’re no longer willing to trust and not verify that the guts of these systems we’re buying from you are secure.

Paul Asadoorian (30:49.606): Mm-hmm.

Paul Asadoorian (31:03.028): There’s also like there’s so many knobs and dials for securing a Linux system. Because it’s open source, people have created their, I mean you’ve got GR security, you’ve got SE Linux, you’ve got, I mean the list could go on in all the different ways to make a Linux system more resilient. I think one thing that would be really nice is if in the S-bomb that companies don’t…

like this don’t put in alongside their devices and ship with it. But if we had an S-bomb, one thing I would want was, what have you do? What security controls are enabled? Which ones are optional? Like make some available to the user. There’s great ways to provide a more reliable and secure Linux installation. We just don’t have access to the knobs and dials, nor do we even have access to a list of, hey, like what did you can did you configure as a Linux on there or not? Like that should be in an S-bomb.

in my mind. And that should drive, I mean, ultimately that should drive adoption and purchasing is the security of the platform. We want the more robust, secure platforms to be on our network.

Vlad Babkin (32:14.433): There are at least three problems to this. Problem number one. No matter how many security controls you implement, if your web application will just execute commands, the user passes as part of its normal flow. All of those security controls are busted. And there are a of platforms that do exactly this flow from one time and again. And most of the time, it’s not an intended feature like a bus shell. We should provide the shell, and that’s a feature. OK, that’s one thing.

Paul Asadoorian (32:25.844): That’s the problem.

Paul Asadoorian (32:38.483): Yeah.

Vlad Babkin (32:42.593): But when you run a PIN command, don’t check that you are pinging an IP address or a domain, just allow users to pass semicolon with whatever command following it. That’s definitely not normal. Problem number?

Paul Asadoorian (32:53.628): Yeah, you’re right, Vlad. It comes down to your SDLC. You got to harden your application. No matter how hardened your operating system is, I think here’s your point, right? You’re going to put an application on there that also has to be hardened equally as well. Yeah.

Vlad Babkin (32:58.815): Yeah.

Vlad Babkin (33:03.137): As I have said, are three problems. Problem number two is that customers are not willing to pay more for this. Customer wants the device as cheap as imaginable. And here’s problem number three. Every single vendor

Paul Asadoorian (33:13.118): Right.

Vlad Babkin (33:22.209): every single vendor is inventing their own way to write web servers. So instead of standardizing on something like, again, Fast API is just an awesome framework which showcases how validation should be done in the right way. So I’m going to use it and use it again and again and again. And if your framework is not as good as it, I’m going to compare it to this. And that’s going to be 99 % of frameworks out there. So if your validation is not as good as Fast API, you’re not doing it right.

Paul Asadoorian (33:46.994): Mm-hmm.

Vlad Babkin (33:51.425): That’s the point. And most languages, most frameworks just don’t get it right. Like, in most cases, you rely on bunch of… Instead of declaring something as an int and having it handle it behind the scenes, so that there is only one place where it handles an int and it is guaranteed to be correct with unit tests, you make an if constantly. Every single time you make an if, you have a chance to make a typo. And that’s the dumbest example. It gets much worse the more complex your validation becomes.

Paul Asadoorian (33:56.274): Yeah, and every time I look at

Paul Asadoorian (34:12.39): Mm-hmm.

Vlad Babkin (34:21.109): And eventually we miss something, whether you like it or not.

Paul Asadoorian (34:22.642): Yeah, 100%. Yeah, and every time I look under the hood at the firmware behind a device, they’ve implemented the web management interface completely different from the last dozen devices that I’ve looked at. I mean, you’ve led the same way, right? Every web interface on these devices is a different implementation. And that’s part of the problem.

Vlad Babkin (34:39.457): Mm-hmm.

Yup. And I’m not even saying about, hey, we need something unique in our interface. Sure, that’s your decision. At least standardize on how you validate the API parameters. Like, there is a known good way to do this. And most devices just deviate from it for no reason. And then get beaten by it. Like, time and again, time and time again. Same for escaping shells, same for sanitizing data. There are known good paths to do this, which vendors completely ignore.

Paul Asadoorian (34:53.289): Yeah.

Vlad Babkin (35:11.649): for whatever reason they do it. Either legacy software or speed of development, perceived speed of development because it’s actually slowing you down, I’ll open you a small secret, or whatever other reasons they invent for them.

Paul Asadoorian (35:13.523): Mm-hmm.

Paul Asadoorian (35:24.882): And that can even vary per product from a single vendor. They’ve implemented an entirely different development environment and application. If you look at any vendors that have multiple products, some run Linux, some don’t run Linux, some have this as an application, some has that as an application. And now they have to maintain all of that and provide good secure code quality across several different platforms.

Vlad Babkin (35:28.129): Mm-hmm.

Paul Asadoorian (35:50.992): in in they’re not getting it right obviously attackers are finding zero days all time

Vlad Babkin (35:54.209): Yep, and the same company has different developers, different departments, all of them using different standards. And that’s actually a massive problem. So what we should do is not just, hey, let’s stop using C++ because it’s so unsafe. It’s let’s standardize what we require for validation for production web framework. And it should be at least at the level of fast API.

Paul Asadoorian (36:01.266): Mm-hmm. Right.

Paul Asadoorian (36:17.425): Mm-hmm.

Vlad Babkin (36:22.035): maybe not one-to-one, but at least analogous where it’s all declarative. Like, you declare something to be an integer, something behind the scenes in the framework handles that. If your framework is not behaving like this, there is something wrong with your framework. Because if you have to write it in a way where you have to imperatively validate everything through if-else and other paths and just raise exceptions on your own, you’re not doing it right.

Paul Asadoorian (36:46.152): Yep. Was there a third thing, Vlad? Sorry.

Vlad Babkin (36:49.697): That’s the soul 3 actually, name of the soul 3.

Paul Asadoorian (36:51.839): okay. Anything else on F5? Did we miss anything?

Vlad Babkin (36:58.273): Not really, but there is one slight problem. I have to drop for a meeting. It got added after we started this conversation. So I might not be able to join you for the framework.

Paul Asadoorian (37:07.657): No worries.

Yeah, Chase and I will finish covering the framework UEFI thing.

Chase Snyder (37:14.902): Yeah, good stuff, Vlad. Thank you.

Paul Asadoorian (37:17.566): Thanks, bud.

Vlad Babkin (37:17.845): Yeah, goodbye everybody.

Paul Asadoorian (37:20.628): Thanks, bud.

So framework, framework. I love framework. By the way, I love framework. It was difficult for me to do this project and this disclosure because I’m like, I love you guys. But what happened is, and actually Brian Richardson and our team gave an interesting point that I don’t think, I didn’t make it into the blog, so you get some inside baseball. So your UEFI shells.

Chase Snyder (37:26.222): framework. Find UEFI shells. Me too. It’s a great, they make great stuff.

Paul Asadoorian (37:53.296): are typically included in utilities that allow you to do system updates and maintenance. In Framework’s case, for Linux users, because the Framework, people who buy Framework are people like me, we run Linux, we like to tinker. If I would need to take apart my laptop completely and put it back together, I can, and I can swap out components, right? And that’s why we buy Framework. And so a lot of Framework users will run Linux. And Framework wanted to provide, props to them,

provide a way for Linux users to update the UEFI BIOS on their systems. And the manufacturers and OEMs will often do this with a UEFI shell. So you boot on a thumb drive, conceivably, right? It’s usually the path, boot on a thumb drive, it loads a shell, that shell loads a shell script, which is basically like a batch file in Windows, because it’s very Windows-like in the EFI application. It’ll execute some commands.

It’ll use that UEFI shell to update your BIOS since you’re not running Windows, which has better facilities to do that in utilities. Oftentimes, manufacturers will sign code such as UEFI shells as part of the maintenance and service packages that allow users to run diagnostics to update UEFI and do various things. So Microsoft…

won’t sign a UEFI shell because it can be used to bypass secure boot and it’s too much of a risk. So OEMs use their keys. So like when you buy a laptop, let’s say you got your laptop from Dell or HP or whatever OEM, that Microsoft has a key in the root of trust for secure boot and your OEM has a key in there for your root of trust. So the OEM can sign software that’s valid on secure boot. And since Microsoft won’t sign the shells,

That means the manufacturer will assign the shell. In this case, Framework distributed UEFI shells that contained commands that allow the user or an attacker to manipulate memory, and in that memory manipulation, it allows an attacker to bypass secure boot. This was covered in Mickey and Jesse’s DEF CON presentation, One Boot Loader to Load Them All, where they showcased this technique. It’s been around, I think, even before that.

Paul Asadoorian (40:18.822): And it’s very similar to other boot loaders that we’ve seen that enable things like Black Lotus and Hybrid Petya. So Microsoft will sign a boot loader, not a shell, but a boot loader. If that boot loader has a vulnerability, it’s a very similar attack. You can leverage that vulnerability as an attacker to disable secure boot and run code of your choosing. I also believe that when you do the memory manipulation with the shell, that the system still thinks secure boot’s enabled, even though it’s not.

which is kind of dangerous. if you’re inventorying your systems and you’re like, secure what’s enabled in all these. An attacker shouldn’t be able to run code, but this is kind of your backdoor method. So we discovered that Framework had signed these UEFI shells, went through disclosure, identified them. Framework has since fixed most of the software out there. So they have released new update packages for Linux users that contain shells that are still signed, but don’t contain any dangerous…

memory manipulation commands, which is good. They’re also working on pushing out a revocation list that will revoke the older shells that contain those commands. And the other third workaround is if you contact Framework, they’ll give you instructions. You can actually remove Framework’s key from the root of trust or rebuild the root of trust on your own, however you want to slice and dice that. Talk to Framework before you do that on your laptop. And so you can change the root of trust, which will invalidate

these shells. So not only did they fix it, but they also provided extra remediation steps. They’re like, know, for those users that want to remediate now, you can do this, but we’re also working on updates. And through the disclosure process, they did release more and more updates. Where, like most of their platforms now, they’ve released new code that contains shells that don’t contain these dangerous commands.

Chase Snyder (42:12.278): Yeah, it’s a strange situation. mean, kind of like we’ve talked a lot about baseboard management controller vulnerabilities and these like management systems or like remote management systems that are intentionally baked in. There’s this constant tug of war, a constant tension between manageability and simplicity for both the customer and the seller of various technology. And then the risks that it introduces. And I think, mean, relevant both to this and to the F5 situation.

Paul Asadoorian (42:20.989): Mm-mm.

Paul Asadoorian (42:26.44): Yes, yes.

Chase Snyder (42:41.742): I was just looking back at the Verizon data breach investigations report for 2025, which is always a spectacular source of information about the general state of security. But it looks like the involvement of third parties in security incidents or data breaches doubled since last year. And they include software vulnerabilities in that, meaning that if you have

Vulnerable software that you got from a third party and that leads to a breach. They count that as third party involvement but Certainly the f5 situation and this framework situation would involve a third party, you know, you have a business or you or person, know, just you’re you and you buy this technology and because of what you bought from them your you know information is at risk and

I don’t know, just the way that the overall ecosystem, technology ecosystem is there is an incredible amount of interdependency between all of the all of the companies and all of their customers.

Paul Asadoorian (43:48.148): Right. Yeah, just people. And people were asking the right questions. I think it was on Reddit and some other places as well of, this just isn’t a framework issue. What if another OEM has a UEFI shell that they’ve signed that means their platforms that trust that shell that may or may not contain memory manipulation commands that allow an attacker to bypass secure boot, do those exist? And I’m like, well,

I’m sure they do, but I have not tested every signed UEFI shell in existence. And that’s the worrying thing about this disclosure to me, which is like not really about framework. And I think there’s other researchers working on this too. Like how do we determine what code has been signed in order to evaluate it to determine if it can be used maliciously or not? We don’t know because that software doesn’t often come with the system.

Chase Snyder (44:23.352): Yeah.

Paul Asadoorian (44:47.026): This is like a utility that you have to go find and download that is authorized in the secure root of trust. How do we know that? And I’m kind of pushing for, when I talked about this in my Security Weekly show, you know, I’m pushing for, I’d love OEMs to have, like disclose what software they’ve signed with their keys. Like maybe there’s an attestation API where you can go query it and say, hey, I’ve got this key on my system, right? That establishes the root of trust.

or part of it, and what software has been signed with this key to run on my system, and I want a list of that software, and then I want to be able go get that, so identify and go get that software so that we can do our own validation against it. So it’s kind of similar to an SBOM situation where I want the SBOM for all of the root of trust in all the software that’s authorized in that root of trust. That’s something I believe is

consumers and enterprises we should have access to.

Chase Snyder (45:46.924): Yeah, super relevant to that question that you got in the talk that we gave together earlier about someone asked about the cryptographic bill of materials, which is a new item in the SPDM DX standard where, yeah, it’s asking questions about what cryptographic systems or libraries have been used. And this seems very adjacent and relevant to that to have, yeah, giving a bill of materials.

Paul Asadoorian (45:51.815): Mm.

Chase Snyder (46:15.286): It’s not exactly the same as the bill of materials, but yeah, list of all the software that’s been signed using this key that’s present. Yeah.

Paul Asadoorian (46:20.402): Yeah, it’s interesting. Another interesting defensive measure that’s being proposed, and I don’t believe it’s been implemented, but it’s being proposed in the UEFI standard in the reference, EDK2 reference implementation, is that if secure boot is enabled and you’re trying to load a UEFI shell, it just won’t do that. Like the UEFI code could be modified, and this is what I’ve read in my research, could be modified to if…

Secure Boot is enabled and oh hey, you’re trying to load a shell. Uh-uh, I’m not going to do that. So you can still do the maintenance. You just have to go in, you have to disable Secure Boot, run your maintenance utility, and then re-enable Secure Boot. So as an attacker, it would make their job much more difficult because you need physical access to go disable Secure Boot. Or the attacker would need some other way to disable Secure Boot.

But if that was the case, they could run code of their choosing anyway. They wouldn’t really need the shell that let them disable secure boot. So it’s an interesting kind of change in the way UEFI works to prevent against this situation. So I that was really interesting.

Chase Snyder (47:28.142): Yeah, yeah. It’s like you can’t. There’s always going to be potential risk or vulnerability, but it’s all about introducing friction or barriers or there’s a guy, Andrew Thompson, I think always says imposed cost. Just make it harder and more expensive for the attacker to get past these or to circumvent the security measures. I like the other. You don’t have to be.

Paul Asadoorian (47:38.633): Yes.

Chase Snyder (47:56.546): You don’t have to be faster than the bear, you just have to be faster than the next slowest guy. Make it harder to get you, and you won’t get God as much.

Paul Asadoorian (48:08.05): I did present some of my utilities that I use to validate this process, right, to evaluate a shell to see if it has this capability to get the signature table from it. None of it is rocket science. It’s all well documented. I actually posted to my social media. If you want to go check that out, in any of my social media, I made a post on how to use ChemU virtualization.

to run a UEFI shell in the commands that you need from that. I think one person already commented, like, this just saved me hours of research. So I gave everyone the command to do that.

Chase Snyder (48:45.154): Nice, I love that you’ve published some really great just firmware security stuff. know people are always looking for even just the basic command your your guide to like evaluating Linux firmware. Yeah, tons of useful info there we should. Yeah, we should get that stuff distributed posted in the comments of this and stuff.

Paul Asadoorian (48:55.646): Yeah.

Paul Asadoorian (49:03.068): Yeah, I like people that have the tools and the techniques to be able to evaluate some of this stuff on your own. So if you’re like, I have XYZ laptop, OEM is giving me this utility. Like, how do I know if it’s vulnerable or not? Well, maybe you can check, right? So we can kind of crowdsource some of this stuff. Because again, trying to acquire all the software that has been signed, I don’t know why we don’t do that. Like even Microsoft with drivers, we know Microsoft has signed.

tons and tons of drivers, but Microsoft won’t tell us what software they signed. Like, why isn’t that public? Like, why doesn’t Microsoft have an API where I can query and get a list of software that has been signed and the hashes for that software? We get a list of hashes, but we don’t know what software that correlates back to. It’s like shrouded in mystery. I don’t understand it. So it’s left up to us to go find all the software and then validate and go, wait, is this one signed?

Chase Snyder (49:53.678): You

Paul Asadoorian (50:01.97): Yeah, it is. does have a vulnerability, right? It just makes our jobs more difficult.

Chase Snyder (50:08.238): Quit shrouding me in mystery, bro. Yeah, I was going to ask you when you said that, and I’m sorry. I mean, what’s your theory for why we don’t why that doesn’t already exist? It’s just not been a priority. It’s like it’s fairly esoteric, right? This is like you have to be pretty deep in computer land to be thinking about hardware root of trust at all. Ninety nine point nine nine nine nine percent of computer users have never heard of UEFI.

Paul Asadoorian (50:09.928): Yeah, know. Crazy.

Paul Asadoorian (50:20.723): Yeah, I-

Paul Asadoorian (50:30.462): Yeah.

Chase Snyder (50:36.852): It’s not, it’s just not something they’re thinking about.

Paul Asadoorian (50:37.724): It could, but you know, it could be, I think it’s more of like security through obscurity. If we did have a global, a list that anyone could access from Microsoft of every piece, just say drivers, of every driver they’ve ever signed. As an attacker, they would go acquire all that software and they would go look for all the flaws in it, right? And they would know exactly which software has been signed.

So now it’s more difficult for both defenders and attackers without that list to go and, well, now I got to go find this and see if it is signed. don’t know what all the software is. I got to go find it, right? And so I think it’s just a security, security through obscurity. It’s really what it is.

Chase Snyder (51:19.992): Yeah, which I think I mean, I feel like security through obscurity gets a bad rep and there is like a sort of reputation. It’s like that’s not not real security at all. And it’s I mean, it’s not not there are cases where obscurity is is the thing, but that stops you from getting attacked. But it’s it’s not enough in any case. And it’s also one of the easiest. It’s one of the most rapidly erased motes. You know what I mean? Like

Paul Asadoorian (51:26.014): Mm.

Paul Asadoorian (51:37.022): Yeah.

Paul Asadoorian (51:43.165): No.

Chase Snyder (51:49.964): The whole situation being able to scan for, you know, BMC management interfaces exposed to the internet. It’s like, okay, the obscurity, nothing that’s internet facing has any security by obscurity anymore. Maybe there was a time when it did, but it doesn’t anymore. Like security by obscurity is an easily erased mode. It’s not nothing. It imposes a little friction. Like you said, it’s like having to go out and sort of manually cobble together that information does impose friction that might be just enough to, you know,

Paul Asadoorian (52:04.531): Right.

Paul Asadoorian (52:12.585): Mm-hmm.

Chase Snyder (52:20.184): have a malicious person take a different tack. But yeah, in this case, clearly once it gets cracked, once it’s at the point where it’s like we have these vulnerabilities, this stuff is out there, I feel like organizations should move pretty quickly to stop depending on security through obscurity at all and to say like, okay, here’s all the information you need to stay safe. We know that this, like the bad guys know about this lane now. So go protect yourself.

Paul Asadoorian (52:46.516): Right. Well, cool. I don’t I don’t have anything else to you, Chase.

Chase Snyder (52:50.104): Game Theory, maybe.

No, those were the two hot potatoes. And I feel like the F5 thing is going to continue. There’s already been new sort of stuff disclosed today about it that we didn’t know yesterday. And I think it’s going to continue to be a story. And I hope that, yeah, I mean, for our customers, for Eclypsium customers, we can definitely help you find vulnerable assets and sort of, we have a lot to offer as far as helping investigate and mitigate this, this potential risk. So, but I hope that.

Paul Asadoorian (52:59.334): It is.

Chase Snyder (53:21.624): people out there getting what they need to be able to handle this. Cause I’m sure, I mean, I’m in Seattle. There’s the F5 tower here. It’s like the number of people who are suffering because of this and having to manage this both from a technology and just a PR standpoint. God, my heart goes out to you. But you know, we’ll get through it.

Paul Asadoorian (53:37.524): For sure. Awesome. Well, thanks, Chase. Thanks everyone for listening and watching this edition of Below the Surface. We’ll see you next time.

Back to all Podcasts