BTS #73 - Uncovering Firmware Risks: From Y2K to Modern Malware
In this episode of Below the Surface, hosts Paul Asadoorian, Chase Snyder, and guest Brian Richardson explore the evolution of firmware security, the risks of supply chain vulnerabilities, and the latest threats targeting network edge devices like Cisco ASA and FTD. They discuss historical malware like the Chernobyl virus, modern malware campaigns such as Firestarter, and the challenges of securing complex network infrastructure in a rapidly evolving threat landscape.
Transcript
Paul Asadoorian (00:51.999): Welcome to Below the Surface. This is episode number 73, being recorded on April 30th, 2026. I’m your host, Paul Asadoorian, joined by Mr. Chase Snyder. Chase, welcome.
Chase Snyder (01:04.45): Hey Paul, always a joy.
Paul Asadoorian (01:06.795): Returning guest who’s now an employee, Mr. Brian Richardson is here with us making his second appearance on Below the Surface, but his first time as an Eclypsium employee.
Brian Richardson (01:17.884): Yep. Yeah, that was episode like seven or something. That was single-digit stuff. Yeah. So I am your stunt Vlad today.
Chase Snyder (01:28.824): We got him, folks.
Paul Asadoorian (01:28.824): Yes, Vlad couldn’t join us, unfortunately, but we’re gonna party without him and hopefully he can make it next time. Just a quick announcement, Below the Surface listeners can learn more about Eclypsium by visiting Eclypsium.com/go. You can find on that site the ultimate guide to supply chain security. Also an on-demand webinar I presented called Unraveling Digital Supply Chain Threats and Risk, a paper on the relationship between ransomware and the supply chain, and a customer case study with DigitalOcean. Of course, if you’re interested in seeing our product in action, you can do that and more at Eclypsium.com/go.
Paul Asadoorian (01:50.717): Alrighty, I am excited for this episode. We got three succinct topics. I like it, although we’ll probably branch off into other things as we sometimes do. You’ll get me started talking about my new Tesla. I know that happens. We go off on tangents.
Brian Richardson (02:16.594): Really? That happens?
Paul Asadoorian (02:20.555): Brian, well I want to start with Brian. Obviously we already mentioned that you transitioned from Intel to Eclypsium, but tell us about how that came to be and what you do here at Eclypsium for our audience.
Brian Richardson (02:29.33): Yeah, so when we last spoke, I was doing firmware evangelism at Intel. I think at the time I was still the TianoCore community manager. And for a while I was also doing some promotion of Chipsec, which is actually where I ran into the people that are now my corporate overlords. I mean, very, very competent managers at Eclypsium. And I’ve kind of took the journey from firmware developer to sales engineer to firmware evangelist over the path of my career. And I, like many people, graduated from Intel back in the summer of 2025. And fortunately Eclypsium for me is literally like right up the road. And it’s kind of a weird skill set mashup of like all the stuff you do with firmware. And for a while I was doing security marketing at Intel. So this is kind of like a weird intersection of all the things and it takes me back in time.
Brian Richardson (02:57.364): You know, do the shimmering effect in post, Paul. I don’t think you can do that as part of the live, but when I was a baby AMI developer, I started out at AMI, Prestolite Intersetup. And actually, if you look me up on YouTube, I have a talk on Y2K. So that gives you kind of a time when I was in the, yeah. So I, yeah, I shaved.
Paul Asadoorian (03:47.551): Now you have to explain for our younger audience that was born after the year 2000 what Y2K was.
Brian Richardson (03:56.123): I actually had to do that. So I’ve actually given a Nerd Night talk twice now on the history of Y2K, why it relates to the BIOS, how it dates back to the reverse engineering of the original PC BIOS. And if that wasn’t haunting enough, we could probably throw the link in the show notes, I don’t know. I censored most of the bad words out of it except for the four-letter word BIOS, which comes up quite a lot in that program. But back when I was a baby as of 1999, while we’re all trying to fix this Y2K nonsense, one of the functions I had at AMI when I was a BIOS developer is also release coordinator. So that means like when we at AMI would develop a BIOS package for a customer, we would typically do like periodic releases. And so we’d put everything in source control and then package it up.
Brian Richardson (04:23.252): Back in the day, this was like, we would send you a link to a zip file or like an FTP or something archaic. Like it wasn’t quite gopher, but it was somewhere in that sort of like, if we did that today, you know, that kind of level of file security, we might end up in court. But back then FTP with a password was the gold standard. And I started noticing that the releases weren’t matching. We had a pretty decent release process where we would build locally and then we would check everything in and then check it back out and do bit compares. Now this is before UEFI. This is back when BIOS was 16-bit nonsense and we were doing stuff like just blindly yeeting to the MBR on the hard drive. And the hard drive made the spin emotion.
Paul Asadoorian (05:25.173): I feel like you’re gonna talk about how you noticed a Tencent accounting error. That’s what it feels like, right?
Brian Richardson (05:33.127): It’s pretty close to that. I noticed that the files didn’t compare. This isn’t quite like Superman III level accounting, but another dated reference. Anyway, basically I’m like, hey, these files don’t match up. And I was talking to one of the QA engineers, he’s like, I’m seeing the same thing. And so this is when we started noticing that we were finding a virus, as they say, in the wild. In other words, we had picked up a virus before any of the EDR scanners of the day had found it.
Brian Richardson (06:02.548): So we started noticing a pattern across this that it had a header and the header always said CIH and had a version number and that was popping up in the free space of an executable. So we’re a Windows DOS house. 1999, so this is like spring of 1999. The reason we were early on the case is because one of our engineers from Taiwan who was now like a vice president at a software company, he shall remain nameless—it wasn’t his fault really, but like he had brought his laptop from Taiwan and at some point he had gotten it on his laptop and brought it into the company firewall. Back when you could get viruses by like actually sitting next to the notebook and putting it on your physical network, not just like, “I got owned through a webpage.”
Brian Richardson (06:31.252): And we noticed that it had this header and consistent versioning and it’s hiding in essentially what is white space in Windows PE executables. So things in like Windows executable files are usually page aligned so that you can say, like the first kilobyte of this is always gonna be some amount of information with variable length, and then the next K is gonna be something else. There’s enough space in there to put a small payload. And people still try to pull this nonsense on executables on Windows all the time. And we noticed that a consistent header, a consistent version number…
Paul Asadoorian (07:20.659): And this was in your BIOS update from AMI.
Brian Richardson (07:23.572): Well, this wasn’t in the BIOS update. This was in the files we were giving people to build the BIOS. So it was attaching itself to executables. And so we started, this payload is attached to executables. We don’t know what it is. All we know is that in a couple of occasions, it actually did try to wipe the master boot record on our drives, which is essentially before you had GPT style partitioning. If somebody wanted to, the typical virus payload of the 80s and 90s was that they would try to overwrite the boot drive, either with a vector that sent it to an alternate location, like we’re gonna hook the boot vector before you actually load the OS, get something in ahead of it, or we’re gonna end up maybe just wiping out the first four kilobytes of your drive, which effectively kills your hard drive. You’re not able to boot. So we noticed that was occasionally triggering and trying to infect some of our systems.
Brian Richardson (07:53.128): So we ended up reporting this to one of the many antivirus vendors of the day. And what we didn’t realize was that once we got more information about it, this actually had a BIOS payload. And it’s very rare for that to have happened back at the time. So the way this worked is Intel had a, so back in the day when Pentium was the main processor and Pentium was not the discount brand for Intel, so way before Core, this thing called Pentium MMX, which was like the gaming processor of the day. And it was trying to, on this very specific model of Pentium MMX board, overwrite the first four kilobytes of the BIOS, what we call a boot block, which would have…
Paul Asadoorian (09:01.321): Yep. So it was not attacking the MBR on the drive. It was attacking the… Was it SPI Flash? It was still SPI Flash back then. No?
Brian Richardson (09:08.421): No, it wasn’t SPI Flash. This was like PLCC 32. So, or even like older, it wasn’t quite EEPROM yet. It was still FlashROM. So like you could program it the way that you program SPI today. Because what happened on old boards, and there’s a Tom’s Hardware article that’s gonna be in the show notes that kind of brings up the history of this. It had its anniversary on April 26. It happens to coincide with the Chernobyl event, but it’s not related to it. The C in CIH doesn’t stand for Chernobyl, it’s the dude’s initials in Taiwan who wrote it, which is its own wacky story.
Paul Asadoorian (09:40.935): Right. And someone named it the Chernobyl, someone from what McAfee or one of those early companies, right? Yeah.
Brian Richardson (09:45.349): Somebody, yeah, somebody, I think it was maybe Dr. Dobb’s or somebody like named it Chernobyl because they noticed the date and they thought the C was Chernobyl. And then later on, it’s like, well, no, this dude’s English name initials were C-I-H and it was a Taiwanese student who wrote it as an example because he had noticed that antivirus didn’t look below the surface, name of the podcast. And so everyone’s like trying to figure out why is it targeting this particular set of motherboards?
Brian Richardson (10:12.39): So back in the day, if you wanted to protect the BIOS, there was a physical jumper. Like you had to go inside the case and pop the jumper out. And that would unlock the first kilobyte or so of the flash chip. And that’s where…
Paul Asadoorian (10:24.011): I think Chromebooks today use a similar method. Yeah. It’s a hardware thing. Yeah.
Brian Richardson (10:27.188): Chromebooks have a similar mechanism for overriding the boot process. If you want to do custom payloads, basically. It’s a similar process. So this is like a, so this is a staging problem. If you’re starting to do more flash updates, because we’re leading up to Y2K and people are doing BIOS updates to fix Y2K problems, then you ended up with this problem of, you have to like physically intervene with the system. You got to take the lid off, move the jumper or do the thing.
Paul Asadoorian (10:56.427): To do your traditional BIOS update where you booted on a floppy, but even some systems you had to move a jumper or jumper pins to. Yeah, okay.
Brian Richardson (10:56.679): Exactly. Yeah. So, when Intel came up with a new chip design, they would end up making what they call a reference board. So they would just say, hey, this is the 430TX chipset, the Pentium MMX processor. We’re going to give any of our customers a fully designed motherboard as a starting point. So this is your chili recipe and you get to figure out how much spice you want to put in it, but here’s the base level of what we think of, like they validated memory, devices. There was like a test BIOS that came with it.
Brian Richardson (11:32.148): So they had this whole thing wrapped up. And what a lot of the Taiwanese ODMs did is they just Xeroxed this thing and took the entire design as is, made a couple of small changes just like branding strings and whatever. So this is what like white boxing is the term for it. Like you wouldn’t necessarily know where your motherboard came from, but it was super compatible with everything. So Intel said, hey, we think this is gonna be a problem [having to move a physical jumper]. So in the reference design, just as an example, you all can change it if you want, we’re gonna move this jumper to a GPIO, a software controlled GPIO so that you can trigger this on or off in your BIOS update utility.
Brian Richardson (12:01.681): And everybody just copied that too. So that meant if you knew where the bit was, even if you weren’t the BIOS update, you could flip it. So high probability that a 430TX motherboard—and the virus actually checked for this, “If it’s this PCI device and vendor ID, I’m gonna go play with these bits.”
Paul Asadoorian (12:07.081): Yep. Everyone’s like, that’s great. Our users don’t have to move a jumper anymore. Beautiful. Ship it. You could flip it.
Brian Richardson (12:28.179): Most of us actually weren’t using 430TX motherboards on our development systems, so it didn’t affect any of us in development. So not upgrading immediately, I guess, you know, a value-oriented company, you know, not immediately jumping on the bandwagon saved us, question mark. I did accidentally release it to a customer and I had to go apologize to them, but you know, we didn’t know any better at the time. But this is an interesting thing because if you look at like the Tom’s Hardware article, they’re like, yeah, this couldn’t happen on BIOS today. And Chase and I were discussing it. We’re like, no, that’s not really how that works. This is why we have a business.
Paul Asadoorian (12:56.779): They did say that and we were talking about that. Let’s address that. But the first thing it reminded me of before we even got to that statement in the article, which we wholeheartedly disagree with with much evidence, is when you were talking about the jumper and then the GPIO setting, it reminded me of manufacturing mode. It’s like a very similar problem where you’ve got to give the OEMs full access to customize the platform, and then they’re supposed to lock it down to prevent others from tampering with the platform, it sounds like even the early days that similar situation was happening.
Brian Richardson (13:42.42): Yeah, it’s the difficult problem. Being at AMI when we were developing bias reference kits for customers, it’s the same thing. You can’t fully lock down your example system because then no one can debug and develop on it. So you’ve got to like leave a couple of doors open for CPU debugging, SPI debugging, serial port access for remote, whatever the thing is on your platform, and then leave enough breadcrumbs to lock that down.
Brian Richardson (14:10.567): You know, you’ll still run into like, so if you do any system inventories, like you open up Windows system information, if you ever see something where there’s a string and it says “to be filled in by OEM,” that means that somebody didn’t read the “how to lock this platform down.” They didn’t like do all the steps of like, let me turn this from a reference design to my own design. So they put no spice in the chili, basically. They left it at factory defaults. And that’s still a huge problem, which is why like things like Chipsec or Eclypsium or other methods where we’re like, hey, check to make sure that you have turned on the security bits, that you’ve turned on boot block protection. Intel will call it Boot Guard, AMD’s got an equivalent, Qualcomm’s got an equivalent. On your cell phone, it’s secure boot. Like these little lockdowns of did you protect your memory, did you protect your flash part, is the flash descriptor writable? It shouldn’t be. These are all checks that people still have to do.
Paul Asadoorian (15:01.619): Right. Now, but I just want to point out, I want to point out to people that locking your SPI descriptor, I’ve covered it in articles now that I wrote when I joined the company three, four years ago. There’s a lot to unpack there. Like it’s not just the, you protected your SPI flash where your UEFI BIOS is stored. No, there’s a whole bunch of things to, yeah, there’s multiple configurations that each have their own different ways to protect that, but also some of them chain together and it gets very confusing. And obviously when it’s that confusing, OEMs can make mistakes. They can not properly lock it down, which leads to like, yes, this could happen today, 100%. I mean, I think we’ve done it. I mean, we still do it today in test systems.
Brian Richardson (15:44.655): Yeah, we found a system less than a year ago that, and the OEM was asking us to test this, that we were like, hey, you actually didn’t fully lock down your flash descriptor. And you protected the BIOS image, but the payload on systems now is more than just BIOS. Like in that same flash part, you have things like AMD PSP code or Intel Manageability Engine (CSME) code that are sharing space there or option ROMs for secondary devices that are not covered by the main BIOS. So you can still, like you can do all this stuff to protect the main UEFI BIOS image and still miss a trick and have somebody else be able to edit that and put extra entries into the flash part or like keep you from running your Intel ME code which will essentially break your system. So this is still an issue.
Paul Asadoorian (16:41.003): Yeah, people have tried to do that safely. People were really paranoid about, you know, we used to work for Intel, right? So you know the community reaction to the Management Engine which got renamed to CSME.
Brian Richardson (16:55.796): Yeah, it’s Converged Security and Management Engine, I think.
Paul Asadoorian (17:02.571): But people found ways to basically disable that. I talked about it the article that I wrote four years ago. I think that you had to do that through hardware, I want to say. That’s the other way. I mean, OK, so we’ve talked about software. Yeah, software-wise, through the operating system, you can potentially write to the SPI Flash. The other way is to put a clip on it or some other mechanism right but a clip on it and then write it and don’t do that on your systems unless you know what you’re doing because you can easily break your system.
Brian Richardson (17:36.018): Yeah, 100%. So the short version of that is this is still an issue and it’s something that you need to be diligent as a manufacturer to make sure you’re locking down. But then even like over updates, could be version one could get it right and then you could do an update and somebody could make a mistake and version two would be wrong. So it’s like it’s a constant monitoring thing. It’s not a fixed point in time, which I think people think of firmware as kind of a fixed point in time problem.
Brian Richardson (18:09.18): Like, yeah, we got it right at shipping and then this isn’t malleable, but like the average enterprise laptop’s gonna live in service probably three, maybe four years, maybe now five since DDR5 is, you know, you’re making four installments on your online payment.
Paul Asadoorian (18:30.475): If you can afford DDR5 RAM.
Brian Richardson (18:33.992): Yeah, but yeah, so you’re probably gonna have your units stay in service longer because of the market forces. So servers could be seven to 10 years, client device could be three to five, industrial could be seven to 10. So you’re gonna take dozens of firmware updates over that time if you’ve got a vendor who’s staying on top of things. And not checking in between those, you could accidentally expose yourself to one of these issues. It’s not malice necessarily, it’s just things happen. You just need to stay on top of it.
Paul Asadoorian (18:58.623): Yes. There’s certainly an operational risk. There is certainly a security risk. We’ve seen several different strains of malware try to attack, use UEFI to attack the system, to get in before the operating system. I think now the trend that hybrid Petya largely set was boot kits, and that’s living inside where your bootloaders live, in the ESP partition on Windows systems, right? And living in that partition. But still it is a huge threat landscape today. I mean we can go all the way back to 1999 and talk about the threat that then would destroy your system. Now, of course, we have threat actors that have more sophistication that aren’t just looking to destroy your system. My concern is what if an attacker were to go, hey, I just want to destroy a bunch of systems—not Petya, right?—but if I’m gonna do it from the UEFI level… To me that is, I don’t wanna give anyone ideas, but to me that’s really frightening because we talked about it many moons ago on the show, right? But the recovery from that is potentially extremely difficult, if not impossible in some circumstances, to physical hardware.
Brian Richardson (20:44.894): Yeah. And that was the thing with the original CIH and a lot of the issues now is that CIH didn’t try to introduce any kind of malicious code. It just bricked and it was purely a demonstration, purely a show off of “I think there’s a gap in this market. The only way I can show it is to nuke it.” And you’re nuking this area, the boot block is the first bit of instructions. And in a normal BIOS, the first, depending on the era, the first 4 to 64K contains enough code to recover, which is why the boot block is usually not malleable after it ships. So that’s why like the Boot Guard and those kinds of things are not protecting the entire part necessarily, they’re protecting like a key area. So this was bad enough to where you couldn’t recover this unless you physically had like a clip. You know, either the thing was socketed or you had a giant PLCC32 clip to chonk onto your motherboard. And most people turns out did not have that thing.
Brian Richardson (21:14.042): Now what the concern is, if you’re what we call a high value target, you’re a bank, you’re a government agency, you’re an insurance company, there’s enough incentive to say, I’m going to develop a specialized payload. So much like CIH was targeting 430TX or even a later version of it, like CIH versioned his headers. CIH had better software control than some software I was developing at the time and I was kind of mad about it. version one, two actually specifically had targeting in the payload to look for Chinese actors. It was looking for a Chinese, like a mainland Chinese version of the code. And the whole East Coast, West Coast thing that China and Taiwan have are a topic for a completely different podcast. But you can see where that and the good old Israeli associated firmware for certain countries, spinny bits that made the uranium better (Stuxnet), was very machine targeted. So people can say, I’m going to make a machine targeted payload that goes after a particular model of system, a particular model of system with these other properties, depending on your target, is worth enough of somebody’s time to pick that system apart and make it specialize for a particular version model vendor that targets an industry. And that’s where the real risk come from.
Paul Asadoorian (22:32.541): It’s interesting too. We talk about how malware can be somewhat sophisticated, even going back to CIH, right? Making decisions based on geolocation. When we talk about ransomware, ransomware doesn’t necessarily want to destroy the system. They want to preserve it in order to extort and get the ransom. However, just like any other software, it can have bugs. And if this is persisting in these lower layers, as we just talked about how dangerous it is if someone were to wipe your SPI flash or mess with it so your system’s not bootable, if they get that wrong, it ends up just being, basically wipe-ware malware. And I have an example of this. I did some forensics for a friend of mine. And I pulled the ESP partition, and that’s the partition in Windows that basically holds all your boot loaders. So the problem that my friend described to me was like, I’m booting my system and it’s telling me it can’t, basically it can’t find the bootloader. And I’m like, well, let me tell you how the boot process works on PCs today. And he was like, wow, that was a lot of information. Like send me the drive.
Paul Asadoorian (23:29.311): When I looked at the ESP partition, I could see that the files were encrypted in a certain way. And a quick search in AI was like, it’s this ransomware group. And I’m like, I know what happened, at least I think. The ransomware group had a little bug in their code that it encrypted the ESP partition as part of the ransomware. But what happens is when your system boots, when your BIOS goes to go, okay, I gotta go find the bootloader now, BIOS/UEFI goes, I can’t find the bootloader, and I just stop, and you don’t see the ransomware message. So I’m like, they messed up, basically. Yeah, just crazy.
Chase Snyder (24:22.562): I don’t know if I’ve ever heard the phrase “chonk onto your motherboard” before on the pod. So that’s good. We’re hitting all kinds of firsts.
Brian Richardson (24:29.396): It’s the worst crane game ever. Cause you’re just like, kinda this little clippy thing, and you’re hoping you’re getting like lined up just right on the SPI part or that the part’s readable. There’s this whole sidebar of like, if they do a pull-up resistor or not determines if you can clip it with the power on or off. Like there’s this whole thing, which is why we don’t normally get frustrated and de-solder the part.
Chase Snyder (24:55.198): Dude, we got to make this actual crane game and have it at the booth. We got to make the game. We’re in the age of because you can just vibe code a game and put it on Steam in like three hours. I saw one that was a data center wiring simulator and it was like, this is you’re being instrumented by the algorithm right now. They built the video game to train you to be a data center tech.
Brian Richardson (25:21.896): They PowerWash Simulator to data center? Wow. It is a beautiful day in the data center and you are a horrible goose. Yeah, so now I’m imagining just like the claw chooses who stays and who goes and you have to like get it just right to read the data. And so you could actually do like a live read. Like if you could get the Dediprog to read it, you get a prize.
Chase Snyder (25:43.758): We’re doing this. We’re doing it. We’re gonna get an old timey claw machine. We’re gonna put the like giant motherboard on.
Brian Richardson (25:49.778): We should call Kenny. Paul, I think Kenny wants in on this one. I’m just gonna call it right now.
Paul Asadoorian (25:51.849): Yeah, yeah. I did want to, Chase, I wanted to get your take on the Firestarter malware because you and I both have researched this a lot. Not this particular malware, but campaigns, threat actors, and malware specifically targeting Cisco ASA and FTD devices. We’ve even somewhat using Graynoise’s data, right, predicted that things were going to happen and then they happened. And I’ve been kind of doubling down on my research here at Eclypsium going, we should pay attention to this because in my research, it’s like the number one most deployed platform by revenue, by number of installs for firewalls and VPNs. And so all of this leads up to, by the way, there’s this whole new strain of malware that’s attacking these platforms. So I wanted to get your take on it too.
Chase Snyder (26:50.338): Yeah, sure. I don’t know. We talk all the time about how the network devices and security appliances are becoming kind of the battlefront for cyber attacks. Like they’re being targeted increasingly by both more advanced APT type attack groups and just like ransomware gangs and less sophisticated groups. And it’s because they don’t have, it’s like when your security box, you know, ASA, that’s Adaptive Security Appliance. FTD is Firepower Threat Defense. You would think that these would have the security internals that would befit their name, but they just don’t. They’re opaque to defenders and people who buy them and deploy them think that when they buy a security appliance from a major name like Cisco, like the household name of technology, that the box itself would be secure. And it just isn’t.
Chase Snyder (28:21.675): It seems like it’s taking a really long time for the mindset to shift and start thinking of those devices as devices that you have to secure unto themselves. Like you ask people how they have secured their stuff and they said, “We’ve got the firewalls in place.” Like, what are you doing to secure your firewalls? “What do you mean? I don’t secure the firewall. The firewall secures me.”
Paul Asadoorian (28:21.675): It’s a firewall. And you’re like, well, what do you do secure your Windows systems? Well, we put all this EDR and stuff on it. And like, what do you do to secure your Linux servers? And like, oh, we do a lot of monitoring and we keep them patched. Well, on your edge security devices, what are you doing? Because it’s just a Linux box, except you can’t really manage the Linux box because the vendors don’t want you to, except when, as we’ve said before, a threat actor has an exploit for it. Now they’re managing your Linux box for you, essentially.
Chase Snyder (28:53.282): Yeah. It’s like a framing that we’ve used in the past that just sticks with me a lot is like for you to gain root access to your firewall or your VPN appliance, or your routers and switches and stuff, you would probably have to jailbreak it or something.
Paul Asadoorian (29:13.843): Right, but with ASA and FTD, you can get into the underlying Linux. They provide you a facility for that. However, big caveat, this is not a Linux that you manage. And so if it has an outdated library, if it runs Python 3.9, like the one I was just looking at today, like you can’t just go upgrade that. I mean, maybe you could. It’s Linux. Like if you want to delete your bootloader, Linux is like fine. Right. If you’re on Windows and you want to delete Edge, it’s like, can’t help you with that, right? So there’s differences here. But you will more than likely break something or even not be able to manipulate software packages on these Linux systems because they’re finely tuned for what’s running on the firewall or VPN appliance.
Chase Snyder (29:42.803): Yeah, at least you can look at it because some of them you get no visibility. It’s like a whole thing that you can’t put an EDR agent on a firewall. And there’s this—ever watch New Girl, the show? I’m dating myself a little bit, maybe. But there’s a scene where they figure out one of the roommates has not washed his towel in like maybe ever. They’re like, dude, what are you doing? Why do you you’re not washing your towel? And he’s like, “I don’t clean the towel, the towel cleans me.” It’s like, “I don’t secure the firewall, the firewall secures me.”
Paul Asadoorian (30:36.907): I like that analogy a lot, actually.
Brian Richardson (30:38.48): Yeah, and keep in mind you’re saying “I’m dating myself” to a guy who just told you he did a Y2K talk. So let’s get the range in here. My doctor has asked me if Cologuard is right for me. So like that’s where we are demographically.
Chase Snyder (30:54.434): We’ve got the range. We got range.
Paul Asadoorian (31:25.995): So just really quickly, listen to this part first, then go read more about Firestarter and you’ll have a much better time. Arcane Door is the campaign that I believe started as far back as 2023. And that is just the name of the campaign. It’s also referred to as MITREC0046 to make things extra confusing. It is not an actor, is just the moniker for the campaign, which largely is a campaign targeting Cisco and other edge devices. The threat actors that I found are UAT 4356 or Storm-1849, depending on who gave it the name.
Paul Asadoorian (31:55.339): Now the malware: Line Dancer in-memory shellcode loader. Line Runner is a persistent HTTP Lua implant on ASA devices. Ray Initiator is a multi-stage boot kit that delivers another payload called Line Viper. Line Viper is user-mode shellcode loader and post-exploitation implant. Firestarter is a Linux backdoor that hooks the “Lina” process for long-term persistence.
Paul Asadoorian (31:55.339): Now, the next logical question that everyone asks is, what is Lina? And if you’re not familiar with the Cisco ASA/FTD platform, you may not know. Lina essentially is the software that runs on Linux that provides the user with the firewall VPN technology. So they’re using the standard Linux stack as the platform, developing additional software that provides you with the firewalling, the VPN, like your traditional IPsec VPN, and the web VPN. So when we say that an attacker has hooked that process, this is the MO of these groups. This is the holy grail of firewall and VPN hacking. Because if I can hook or get inside the process that’s doing that VPNing functionality, the attacker can do things like: hey, every time someone authenticates to the web VPN, write their credentials in clear text to this file, and then shoot that file back over the C2 that I have on it. Essentially that same kind of TTP persists through multiple similar campaigns against other network vendors with other groups. UNC3886 is notorious for doing that kind of stuff as well, which I think has dotted lines to this malware. So we’ve seen this evolution of all this malware from almost three years, evolution of malware that’s essentially, to me it almost looks like they’re perfecting their craft to implant these firewalls.
Brian Richardson (35:21.234): Yeah. And it’s different technologies too, like Cisco is not consistent across those lines. So you have different ways of managing and different amounts of visibility into the products, which is a challenge for us trying to keep on top of it, and a challenge for the customers that are managing it. Because there’s some inconsistencies in between them, but you’re talking about this Lina process. If that’s common to a lot of the devices, then once you figure out how to hook that, that’s why you see this happening across vendors as well as across product lines within a vendor. You know, because it’s a supply chain thing. A lot of people relying on the same libraries components under the hood pieces. Because these things are really just Linux with a lot of SFP ports sitting on them.
Chase Snyder (36:04.088): A thing that we hear from customers all the time is that they already have whatever the management tool is from a particular vendor, their Cisco shop. They have the Cisco management product, whatever that happens to be called.
Paul Asadoorian (36:18.091): Which, by the way Chase, I just want point out that you buy all of these Firewall VPN appliances and then you’re like, I have to manage them. So you have to buy another appliance that is basically just Linux with some custom software running on it to manage all of it. FMC, Firepower Management Console.
Chase Snyder (36:46.03): Yeah, 100%. And so it’s like the amount of effort, cause almost no company will have only one vendor. Arguably you shouldn’t because you should have a sort of supply chain mitigations in place by having different vendors fulfilling the same sort of basic requirements. But then if you have different types of network devices or the same types of network devices from different vendors, and then you’ve got to have the different management appliance and console. And then you got to have the person or people who knows how to use that different management appliance or console. It gets muddy real quick. And when an organization has several of those different things and they’re trying to keep their stuff up to date to N-1 or whatever firmware across multiple products from multiple vendors through different management planes, they just can’t keep up. Then that’s how you end up with super out of date firewall managers and your firewall getting pwned.
Paul Asadoorian (38:02.699): There’s also the, I think that you could easily have, let’s say you have 200,000 employees. Any one of these platforms that’s providing VPN access especially is providing access for a user base of a hundred thousand or more people. Just think about that for a moment. Like, oh, I have to bring down access for a hundred thousand people or even better, I’m frustrated with vendor XYZ for any number of reasons—security, I’m told, is one of the reasons today that CISOs are making these decisions—and I want to move to a different platform. Oh, let’s just migrate 200,000 users to a new platform, because that’s not hard, is it? Of course it is. And that’s how they get stuck. Enterprises are stuck on these platforms.
Brian Richardson (38:59.977): Yeah. Geographic region support is one of them, right? You end up with like a solution, either through like vendor support or sanctions or politics or whatever. Like your Asia customers can run one firewall, but your US customers can’t. And so you’ve got different versions of agents places or you’re in the middle of a rollout. And if you’re still doing that phased approach, cause I’ve been in companies where we’ve migrated the firewall and it’s been months of that transition. And at that point like you’re probably not maintaining the system you’re transitioning from so you still have a gap even when you’re moving from that vendor to have something that’ll persist after you put up the new guard but the thing already got in before you switched out the firewall.
Paul Asadoorian (39:52.555): The other interesting thing about Firestarter is in order to get rid of the persistence, you have to physically power it off.
Brian Richardson (40:03.925): Ooh, oh no, no. IT people love going into server farms and playing with racks. That’s great.
Paul Asadoorian (40:11.115): Not only that, you have to power it off long enough so that all the data leaks out on the floor or whatever analogy you want to use, but essentially the malware lives in areas that would persist through soft reboots and those areas aren’t cleaned out or removed unless there’s a full power off.
Brian Richardson (40:20.437): Drain those caps. Yeah.
Paul Asadoorian (40:39.947): Cisco comes right out and says in their advisory you have to power it off. So even if you’ve patched it, even if you’ve tried to clean it, like rebooting it or patching it doesn’t remove the infection. So like applying a firmware update doesn’t remove the malware. The authors thought of that and it persists. But also it could write into areas that are persisting beyond powering off, powering it back on as well. So you have to do some forensics on these systems.
Brian Richardson (41:13.097): Yeah. And sometimes when you power off a firewall, a lot of these things are running some kind of virtual machine management or like F5, I think they call it, TMM. Essentially they’re using virtual machine management or other types of containerization that when they reboot something, what they’re really doing is like flipping the off switch on something virtual. And so there may be certain management layers where you have to hit it with the hard reset hand. Which just means if you’ve ever gone to a data center, that’s an hour of your life just getting in and out the door, getting past the gates. And then you’ve got to go in all the racks and get carted in. Those places have better security than some military bases have.
Paul Asadoorian (41:57.395): And then hoping when you power it back on that it comes back on and comes back on in the same state from when you powered it off. And those of us that have worked in IT for a long time know that’s not always the case.
Brian Richardson (42:08.661): It doesn’t know what happened. And that’s minutes per box. Like this is not like flipping on your laptop. Like these are essentially servers with a lot of network cards attached to them. So you bring a sandwich, you’re gonna be there for a while. But don’t eat the sandwich in front of the rack.
Paul Asadoorian (42:33.119): Did you want to touch on the Graynoise report, Chase, too? I thought that was super interesting. Basically, Graynoise is pointing out that we can notice scans for certain targets, and then a certain number of days after the increase in that scanning, that target, we learn of a new vulnerability, a new zero-day exploit, and new malware that is infecting it.
Chase Snyder (43:31.564): Yeah, they’ve been really effective at predicting these things over the past year or so. So shout out to Graynoise. And that sort of general lane of information, like the Verizon Data Breach Investigation Report had a solid chunk of network device facing vulnerability exploit incidents that they had as part of the data. And the median time to exploit a vulnerability across those was zero days. So a lot of the time these things are getting exploited before they’re disclosed. We find out about them because they get exploited and then they get disclosed. But it kind of shifts the whole timeline of how you need to be thinking about protecting yourself against exploitation in those network edge devices because vulnerability management and threat detection tools, anything that relies on an existing CVE or anything like that, it’s not going to find it for a while.
Chase Snyder (44:29.356): And even then, to the point of how long it takes or how hard it is to address these things, I think the median time for an organization to patch one of those vulnerabilities was like 30 days. And so there’s just a lot of operational friction and rolling out any sort of a fix to this enterprise network infrastructure layer versus the time that it takes to exploit those things just keeps getting faster. I think, yeah, there’s a whole paradigm shift coming in how folks secure their network edge because the current trend is things are getting exploited way faster. The exploits are becoming more commodified. It’s becoming easier and easier for less sophisticated groups to target your whole network infrastructure. And it’s happening super fast. I think the pain and the sort of operational downtime and financial costs that end up hitting these companies is gonna cause some of them to totally radically overhaul how they’re trying to secure these things.
Chase Snyder (46:01.003): One more data point that I just can’t stop myself from quoting at every opportunity is, it’s not just nation states. For certain vendors of on-premises VPNs, there was this insurance company called App-based Cyber Insurance that said that certain vendors were associated with a 6.8x higher likelihood of being targeted with a successful ransomware attack. So having that VPN, you think it’s like a security product, right? The VPN, it’s not really a security product. People treat it that way.
Paul Asadoorian (46:29.363): Right. It’s like I parked my car in the garage so my insurance would be cheaper, right?
Chase Snyder (46:33.152): Yeah, yeah, it’s like you’re six times more likely to be robbed if you put it in the garage. That’s a great analogy.
Paul Asadoorian (46:43.847): It’s so nuts. Also my other concern is, well, we’ve established that threat actors know about the vulnerabilities and exploits before we do. My other concern, and I don’t think it gets talked about enough, is what if the InfoStealer—it’s all the rage today, right? Threat actors are just going after data, wherever, however they can get data through supply chain attacks. I’m like, hey, I got an account. Someone’s just telling me like, they’re just going to log into your Salesforce. They’re going to push the export button and they’re going to dump everything. And then this is so common and they’re stealing so much data. Shout outs to Tammy at Flair. She was just telling me about this. There’s so much data that there’s a site called Leak Bizarre and their service, their niche is to go through the info stealer logs or data dumps and pull out what’s interesting because attackers are getting hundreds of gigabytes, terabytes worth of data. And it’s a lot of work to analyze that and figure out what you got, right? We saw it with the MOVEit breach that had a long tail as they were sifting through all of the data. And my concern is that in some of these dumps and leaks is information about vulnerabilities. I mean, we had that concern with F5, right, when they disclosed that their network was breached.
Brian Richardson (48:52.558): Yeah, it was one of the reasons why they did a massive key rotation. Like one of the big things in that update was not just, we patched a ton of vulnerabilities, which they probably were already in the pipeline for, it may have accelerated the release. If you read the change log on those releases post-disclosure, they’re long. They fixed a lot of stuff. It was very comprehensive, but the biggest thing was like, we’re gonna rotate some keys. And that was like trying to prevent them from like being able to take advantage of build infrastructure, for instance, hoping that they didn’t have the new keys because they hopefully had locked them out by the time they did that regeneration.
Paul Asadoorian (49:14.667): Sure. Which is why we put so much effort into, I mean, not to plug our product, right? But this is why we are entrenched in this topic. One reason is we’re trying to help people defend their network edge devices, right? So that’s important. Mythos was interesting. We talked about it a lot. Everyone’s talking about Mythos. But the latest thing, Brian, I think you put it is: The White House doesn’t want Anthropic to broaden its access, but Anthropic is doing that anyway.
Brian Richardson (49:47.508): Yeah! Yeah, so Mythos, if you want background on it, there’s probably 4,000 podcasts that have already covered it, but source level tool for trying to find exploits. And Mythos is not a general release. It is most likely a derivative of one of the existing Claude models, but it probably has different guardrails and different prompting techniques to look for things that maybe the standard model doesn’t because Anthropic does a very good job of guard railing. And we spend a lot of time trying to massage that for our purposes. “We’re the good guys, trust us.”
Brian Richardson (50:45.545): But so Anthropic, it could be the fact that some of the current government agencies were not first on Anthropic’s list for giving them the Mythos model, but they want to be able to expand it out. So the administration, they’re having these conversations around it. And this is a Wall Street Journal article. So they really want to, they want to up the number of customers that have access to this. And they went to some big people like CrowdStrike and Intel with the initial model, so they could use it internally on their source code and look for things. And this is where everybody was kind of freaking out about Mythos and thinking like, this is gonna kill every security company out there. And it probably really won’t because again, you need source level access to work on this.
Brian Richardson (51:41.501): The government’s concern, according to the article, is that if Anthropic spreads its resources too wide supporting Mythos or even other new products, it reduces the number of engineers that can respond to the government’s needs for Anthropic models. Because as we’ve heard in the news recently, several government agencies heavily rely on Anthropic for models they use for a variety of purposes—and the discussions about what purposes those should be, it’s a topic for a different podcast. So this is of interest and it’s hard to say if it’s motivated by the concern over, “We weren’t first on your list, what gives?” And more of the, “You’re someone we rely on, so let’s make sure that…” Yeah. Yeah, yeah. Sorry about that. There was a glitch in the Matrix. I was talking about Anthropic in the government. I should have turned on my VPN first.
Paul Asadoorian (52:23.532): Brian froze.
Chase Snyder (52:26.1): He was in the middle of making such a good point. Yeah, that systems level thinking.
Paul Asadoorian (52:31.298): He’s back. Good. Glitch in the matrix here.
Brian Richardson (52:39.861): But I’m just going to assume that disconnection is completely unrelated to the topic—
Paul Asadoorian (52:43.498): He’s frozen again.
Brian Richardson (52:47.679): Am I still getting…yeeted?
Paul Asadoorian (52:53.475): I did read something where NSA was still, despite it being kind of poo-pooed by the government, that NSA was still using Anthropic’s models. And I mean, let’s be frank, Anthropic has the best models for a lot of tasks.
Chase Snyder (53:11.682): I mean, there was the big drama in the news, Anthropic getting designated as a supply chain risk, first ever American company to get that designation. But I think that the level of integration that they already had into government systems, it’s much like when a massive vulnerability is discovered inside of some network appliance and CISA issues a binding operational directive that’s like, hey, you guys have to rip this out of all your systems or patch it or whatever. They give them a certain amount of time and it may or may not be possible. My sense is that Anthropic was so involved and being so heavily used and was so integral to certain operations that were happening that even though that designation happened, they couldn’t get pulled out in the middle of the action. I don’t really have any evidence about that other than reading publicly available internet stuff and reading between the lines of it. But I think they’re still using it and that it is foundational enough to certain systems and operations that it would be hard to pull Anthropic models out of some government action that’s happening right now.
Paul Asadoorian (54:45.227): Brian’s back.
Brian Richardson (54:45.885): Yeah, I’m back. Now, turns out you talk about AI and used in the government enough and somehow your Wi-Fi goes down at the same time and I’m just going to call it a coincidence.
Chase Snyder (54:55.15): Spooky action at a distance.
Brian Richardson (54:58.397): Yay. This is probably in a normal podcast, the part where the sponsor segment would come up and they try to sell you a VPN, but we’re not that podcast.
Chase Snyder (54:59.736): Don’t talk about goblins.
Paul Asadoorian (55:08.235): We’re not. We’re not. Yeah. Well, I thought it was a great discussion. I thought we covered some great topics. I wanted to thank everyone for listening and watching this edition of Below the Surface. We’ll see you next time.
