PODCASTS

BTS #78 - Patching: The Race Against Time

In this episode of Below the Surface, host Paul Asadoorian is joined by Vlad Babkin and Chase Snyder for a wide-ranging discussion on modern vulnerability management, network appliance visibility, AI-assisted exploitation, Linux kernel bugs, cold boot attacks, and software supply chain risk.

The conversation begins with a perfect CVSS 10 vulnerability in Ubiquiti’s UniFi Connect application, but quickly expands into a larger question: what should enterprises actually do when the number of vulnerabilities, patches, dependencies, and exploit paths keeps increasing faster than security teams can respond?

Paul, Vlad, and Chase return to a recurring theme for the show: defenders need more than version checks and patch deadlines. They need visibility into device configuration, firmware integrity, exposed services, package provenance, and whether a vulnerability is actually reachable in their environment. The episode is especially focused on network appliances, where attackers often gain the benefits of a Linux system, a privileged management plane, and poor visibility for defenders.

Key Topics Covered

  • Ubiquiti and the CVSS 10 problem: The episode opens with Ubiquiti’s UniFi Connect vulnerability and the difficulty of treating every maximum-severity score as an equal emergency. The hosts discuss how Ubiquiti’s product line has moved from home and small-office networking into broader “enterprise of things” territory, making patching and exposure decisions more consequential.
  • Auto-updates in the enterprise: Vlad argues that the absence of auto-update options on many enterprise devices is a serious weakness, while Chase points out the operational risks of automatically updating routers, switches, and other foundational hardware. The debate lands on a practical tension: enterprises need faster update paths, but infrastructure updates can carry real downtime and compatibility risks.
  • AI and the accelerating vulnerability cycle: The hosts connect bug bounty programs, AI-assisted vulnerability discovery, and the growing volume of patches. Paul asks whether AI can help defenders patch more effectively, while Vlad warns that attackers may gain powerful vulnerability discovery and exploit development capabilities before defenders can operationalize the same tools.
  • Compensating controls before the next wave: Vlad argues that teams should not assume patching alone will be enough. Strong validation, authentication, content security controls, telemetry, and better prevention layers matter because AI may compress the time between vulnerability discovery and working exploit.
  • Visibility into security appliances: A central claim of the episode is that a security appliance that cannot be inspected is not giving defenders enough security. The hosts discuss why access to a real shell, device files, logs, firmware state, and integrity checks can help defenders investigate compromise, even though the same Linux-like openness can also help attackers.
  • Network devices as attacker infrastructure: The conversation repeatedly returns to routers, firewalls, switches, VPNs, and gateways as high-value targets. These systems sit at privileged points in the environment, often expose management services, and may give attackers credential access, packet capture capabilities, or persistence that survives normal endpoint remediation.
  • Citrix Bleed as a class, not a one-off: Paul discusses Citrix NetScaler memory-disclosure issues as more than a single vulnerability. The hosts emphasize that a version check is not enough when exploitability depends on whether a feature such as SAML IdP is enabled and how the device is configured.
  • Why configuration-aware checks matter: The episode uses Citrix and Arista examples to explain why vulnerability management needs to combine software version, enabled features, exposure, and configuration. A scanner that only says “this version is affected” may either overstate risk or miss the condition that actually matters.
  • The cost of zero-CVE compliance: Vlad criticizes compliance programs that require every vulnerable package in a container or appliance image to be removed, regardless of whether the vulnerable code path is reachable. The hosts argue that teams should prioritize exploitable classes such as authentication bypass, command injection, and remotely reachable RCE over inert library or kernel findings that do not change real-world risk.
  • Firmware and old kernels in appliances: Paul notes that reverse engineering firmware often reveals old Linux kernels with many CVEs, but many kernel bugs require local access or privilege escalation that may not matter if services already run as root. The more important question is often how an attacker gets initial access and what privilege boundaries exist once they land.
  • Open source dependency growth inside appliances: Chase describes how network device update packages have grown dramatically, including the addition of many Python dependencies. The hosts connect this to NPM, Python, Go, and other software supply chain risks, where a compromised dependency could be pulled into a device image during a narrow but damaging window.
  • Defending the package supply chain: Vlad recommends two practical controls: pin dependencies by cryptographic hash rather than trusting version tags, and stagger routine updates so defenders have time to detect poisoned packages before they enter production. For urgent security fixes, teams can still make exceptions with manual validation.
  • Bad Epoll and Linux privilege escalation: Paul discusses CVE-2026-46242, also referred to as Bad Epoll, as part of a broader uptick in Linux kernel privilege escalation research. The hosts note that kernel bugs may matter more on servers and desktops than on many appliances, but they still expose the lack of privilege separation inside network and security devices.
  • Mythos missed one, but AI still helped: The episode uses Bad Epoll to show that frontier models are not magic vulnerability finders. They can miss important bugs, but their reasoning traces may still point human researchers toward suspicious behavior, and AI can reduce the time required to build proof-of-concept exploits once a bug is understood.
  • Cold boot attacks and pre-OS tools: Paul explains a new lightweight RAM dumping tool that revives the 2008 cold boot attack model through legacy BIOS behavior. Vlad widens the discussion: once an attacker has physical access and is willing to freeze memory or manipulate hardware, mitigation becomes difficult, and memory encryption may be the more relevant defense direction.
  • A preview of Paul’s Linux update tool: Paul closes with a preview of a Python tool he has been building with AI assistance. The tool is intended to help assess Linux updates, package integrity, orphaned packages, maintainer changes, microcode status, and supply chain risk before applying updates.

Timestamps

00:00 – Pre-show setup and episode intro
00:53 – Welcome to Episode 78 with Paul Asadoorian, Vlad Babkin, and Chase Snyder
01:13 – Eclypsium resources and the episode’s first topic: Ubiquiti vulnerabilities
02:49 – Home routers, Ubiquiti, MikroTik, and separating routing from Wi-Fi
05:30 – UniFi Connect, CVSS 10, and the role of auto-updates
06:54 – Auto-updating network gear in enterprise environments
08:20 – Ubiquiti’s expanding “enterprise of things” product line
09:08 – SaaS auto-updates versus firmware updates on critical infrastructure
10:20 – Why enterprises are falling behind the speed of vulnerability discovery and exploitation
12:22 – Bug bounties, AI, and the increasing volume of patches
13:23 – Vlad’s warning about attacker access to frontier-level AI capabilities
14:38 – Patching, prevention, telemetry, and defense across the full attack surface
15:58 – Why closed security appliances can block defenders more than attackers
17:01 – Linux-based appliance malware and the defender’s shell advantage
19:28 – Citrix Bleed as a recurring class of NetScaler memory-disclosure issues
20:54 – Why version checks alone do not determine real exposure
22:25 – Compliance programs, Docker images, and the limits of zero-CVE thinking
23:13 – Prioritizing remotely reachable vulnerability classes over raw CVSS scores
24:17 – Library dependencies, unused vulnerable features, and appliance risk scoring
25:55 – Firmware analysis, old Linux kernels, and why exploitability context matters
28:52 – Open source dependency growth inside network device update packages
30:10 – How poisoned software dependencies could enter IoT and edge devices
31:07 – Hash pinning and staggered updates as supply chain defenses
33:52 – Bad Epoll, Linux kernel privilege escalation, and Google kernelCTF
35:34 – Privilege separation failures inside network and security appliances
37:05 – Mythos, AI vulnerability discovery, and what models still miss
38:31 – AI-assisted exploit development and persistence frameworks
39:55 – FortiBleed, credential abuse, and living off the land on firewalls
40:40 – Fine-grained privilege levels for appliance management functions
44:21 – Cold boot attacks, RAM dumping, legacy BIOS, and CSM
48:44 – Paul’s preview of an AI-assisted Linux update and supply chain checking tool
53:43 – Detecting orphaned packages, maintainer changes, and risky build scripts
56:19 – Closing thoughts

Links & References

Core References

Vulnerabilities & Research Discussed

Concepts & Frameworks

Tools & Platforms Mentioned

  • Ubiquiti UniFi Connect
  • UniFi OS
  • MikroTik
  • Cisco NX-OS
  • Citrix NetScaler
  • Arista EOS
  • Fortinet / FortiGate
  • F5 BIG-IP
  • EMBA firmware analyzer
  • NPM
  • Python / pip
  • Go modules
  • Arch User Repository
  • yay
  • Lua hooks
  • apt
  • Flatpak
  • Snap
  • Topgrade
  • Fleet
  • Claude

Transcript

Paul Asadoorian (00:33.630): This week, a perfect CVSS ten, Mythos missed one, Citrix Bleed again, a new take on cold boot attacks, and perhaps a sneak preview. Stay tuned. Below the Surface, coming up next.

Paul Asadoorian (00:53.118): Welcome to Below the Surface. This is episode number seventy-eight, being recorded on July ninth, twenty twenty-six. I’m Paul Asadoorian, joined by Mr. Vlad Babkin. Vlad, welcome.

Vlad Babkin (01:05.467): Hey.

Paul Asadoorian (01:07.250): Mr. Chase Snyder is here with us. Chase, welcome.

Chase Snyder (01:10.254): What’s up, Paul? What’s up, Vlad? Always a pleasure.

Paul Asadoorian (01:13.438): Below the Surface listeners can learn more about Eclypsium by visiting eclypsium.com/go. There you’re going to find the ultimate guide to supply chain security, an on-demand webinar I presented called Unraveling Digital Supply Chain Threats and Risk, a paper on the relationship between ransomware and the supply chain, and a customer case study with DigitalOcean. If you are interested in seeing our product in action, you can sign up for a demo — all that at eclypsium.com/go.

I guess we start with the perfect CVSS ten and a slew of other vulnerabilities affecting Ubiquiti devices. It’s also interesting how much we talk about Ubiquiti amongst us nerds. For some time, many of us have started with Ubiquiti in our homes. They’ve certainly persisted into small business and small enterprise. I hear some indications that they are breaking into enterprises — enterprises are actually adopting them for their technology as routers or gateway firewalls, and of course access points, which is kind of their bread and butter. They’ve expanded their portfolio to include VPN functionality, cameras, building access, doorbells, and some of the smart home stuff they support as well.

Chase Snyder (02:49.294): You guys don’t just use the combination router modem that came with your internet service?

Paul Asadoorian (02:53.246): No. A lot of people ditched that. I’ve always been a huge fan of separating things — you need a router firewall for routing and firewalling, and then you need a Wi-Fi access point system that is separate from that, because combining all of it in one device is typically underpowered and has a pretty poor track record of security.

I like the Ubiquiti story. From what I can remember, it was someone who worked for Cisco who said, “Hey, we can use cheaper hardware based on open source Linux and provide folks with a great alternative to enterprise networking products.” I like the backstory. This isn’t the first time we’ve talked about issues with Ubiquiti. Most of their stuff is based on Debian Linux. You can log in via SSH. They have their own proprietary protocols and systems to manage everything — you’ve got your cloud gateways and you can host that in Ubiquiti’s cloud as well. Most of my stuff auto-updates. Do you guys use Ubiquiti?

Chase Snyder (04:15.467): I don’t.

Vlad Babkin (04:16.038): What I use are MikroTiks. Where I am right now I just have the router from the provider, but for my home infrastructure I use those. It’s a really powerful device for not a lot of money, and it has a relatively good track record on vulnerabilities, especially recently. They got some, but they fixed them quickly.

Paul Asadoorian (04:42.036): Which one do you use, Vlad?

Vlad Babkin (04:46.252): MikroTik. I don’t remember the exact model number.

Paul Asadoorian (04:47.621): Yeah, MikroTik makes good stuff too.

Vlad Babkin (04:51.056): Yeah, it’s relatively good. The point about them is that if you configure it right, it’s not exposing anything to the internet that you don’t want exposed. The only correct configuration for your home router is to only ever expose VPN ports. You don’t want anything else.

Paul Asadoorian (05:02.057): Right.

Paul Asadoorian (05:13.812): Yeah.

Chase Snyder (05:13.870): I’m embarrassed to tell you guys what router I use at home.

Vlad Babkin (05:19.782): Let me guess — TP-Link?

Chase Snyder (05:23.232): No, not anymore, I should say.

Vlad Babkin (05:28.527): Understandable.

Paul Asadoorian (05:30.868): The CVSS ten is in their UniFi Connect application, which I’m assuming is the same one hosted in their cloud that you can also deploy locally. I think it’s the same application that came with the cloud keys, which would let you license and manage your Ubiquiti gear — basically a web application that lets you manage all your Ubiquiti stuff. I’m not sure if that translates to their mobile app as well.

What I’ve found is that the Ubiquiti stuff just auto-updates. I’m not sure how that setting translates across their entire product fleet. I know I have a Dream Machine for my house, and it was just set to auto-update out of the box, which is pretty amazing. It’s interesting — we talk about Ubiquiti and how it’s used in smaller home and SOHO environments, and security feature-wise it’s kind of got a leg up on some of the enterprise gear.

Vlad Babkin (06:53.221): Yep.

Paul Asadoorian (06:54.902): Customers don’t want that auto-update functionality, unlike Cisco, Fortinet, Arista, or similar devices. There’s maybe a lot more going on on those devices. How do you guys feel about auto-updating in the enterprise? Is there pushback from customers? That’d be my guess.

Vlad Babkin (07:17.389): I can understand why you would disable this in the enterprise, but not having it at all is a massive problem. I suspect they don’t have it because they really want you to have a support contract. If your device auto-updates and provides the firmware for free, you’re not going to sign a support contract. They want to have this whole ceremony around it to get more money. Actually, smaller vendors…

Paul Asadoorian (07:24.575): Yeah.

Chase Snyder (07:44.064): Yes.

Vlad Babkin (07:45.829): …handle this better. Believe it or not, MikroTik also has auto-updates.

Chase Snyder (07:52.182): It’s more of an operational thing than a security thing, though. A lot of places don’t want to be on the latest — the latest version might be totally broken. Any sort of update event can cause downtime or make it incompatible with whatever legacy stuff you have. So any sort of auto-update is kind of incompatible with the enterprise operating model.

Paul Asadoorian (08:20.888): That’s interesting. Maybe I was confusing Ubiquiti’s product line, which can easily happen. They’ve got a lot under the UniFi umbrella. They say UniFi Connect is a modern “enterprise of things” — that might be the first time I’ve heard that term. It’s interesting how they’re using the word “enterprise,” which speaks to their marketing…

Chase Snyder (08:37.507): Yeah.

Chase Snyder (08:41.570): Good marketing, yeah.

Paul Asadoorian (08:48.426): …and their go-to-market strategy of breaking into the enterprise. It consists of professional audio-visual products, lighting, electrical, and electric vehicle charging stations. They’ve really expanded their product line beyond where I thought, which is interesting.

Vlad Babkin (09:08.485): Beyond that, Chase, I have to disagree that enterprise is incompatible with auto-updates. If you look at software, it does it. Slack, which is the enterprise messenger, has auto-updates.

Chase Snyder (09:23.928): Okay, this is fair, but it’s different in that it’s typically delivered as a SaaS. I don’t know if Slack has an on-prem version, but enterprises use tons of SaaS. Updating your routers, switches, and network infrastructure is different — those are based in hardware and are more foundational. An auto-update to Slack is not going to cause downtime in any of your enterprise systems or products. Whereas a firmware update on a router could…

Paul Asadoorian (09:59.306): Yeah. Most user software auto-updates — your browsers and several applications — and that impacts one user potentially. Although if you’ve got tens of thousands of users and they’re all on a version of Slack and it auto-updates to a version that breaks something, that could be a bad day for IT.

Vlad Babkin (10:06.691): Yeah.

Chase Snyder (10:19.884): Yeah, it’s a continuum.

Vlad Babkin (10:20.143): Yeah, I can see your point, but with how modern CVEs and the modern threat landscape are looking, enterprises will have to change. There are basically two options: update or get exploited. It’s just that bad.

Paul Asadoorian (10:39.860): Yeah.

Chase Snyder (10:41.910): Yeah.

Chase Snyder (10:45.368): This is the sort of main mega-trend that is driving so many things in security and otherwise — everything changes faster than enterprises can keep up. And it’s only getting faster. The most visible example is AI adoption, where AI is changing how every single person at an enterprise works, and it’s possible that the whole structure of how that company is built just can’t evolve to accommodate it. Some can — many big companies have published about how they’re using AI. Similarly with the threat landscape, we talk about how patching timelines have slowed since 2024. It used to be that enterprise security teams were patching faster and faster. The Verizon Data Breach Investigations Report this year found that it had gotten faster and faster until 2024, and then patching started to lose ground — enterprises were not able to patch as quickly as they previously had been. It’s only gotten worse since 2024. That’s everything: vulnerability discovery, the release of patches, the development of proof-of-concept attacks, and the use of exploits…

Paul Asadoorian (10:55.679): Yeah.

Vlad Babkin (10:57.177): Yeah.

Paul Asadoorian (11:50.985): Interesting.

Chase Snyder (12:13.024): …all going faster, and enterprises in their current model are maxing out their speed while everything else gets faster and they’re falling behind. Something fundamental is going to have to change.

Vlad Babkin (12:22.733): Yeah, and…

Paul Asadoorian (12:22.932): Yeah. So many segues from that. One related to this article — to try and stay on topic with Ubiquiti — is that when I researched these CVEs, it seemed that most of them were reported through HackerOne. Not to say that AI wasn’t involved, but this came through a bug bounty program, which begs a lot of interesting questions. Is Ubiquiti using AI in some form to flesh out bugs and vulnerabilities in their own code? Are they relying on bug bounties, or some combination of both? In each of those scenarios, AI and bug bounties are introducing more vulnerabilities, which result in more patches that people have to apply. And now we’re getting into one of my favorite topics of late: how do we use AI to patch more effectively in our environments?

Vlad Babkin (13:23.365): That’s the neat part — you don’t. Attackers are very likely getting missus-level capability within maybe half a year. It will take only a short while for open models to catch up. Missus does not have as massive a lead as the market thinks. So attackers are probably getting something missus-level within half a year tops. You are not getting that at the same time, because it’s gated behind a trust program.

Paul Asadoorian (13:35.851): Yeah.

Paul Asadoorian (13:48.502): Yeah.

Vlad Babkin (13:51.566): And thus, you cannot use it to patch your stuff. The operational reality is you have to clean up the code and add compensating measures now to not have an apocalypse in half a year. If your product doesn’t have compensating measures — for web applications, that could be really strong validation, strong authentication systems, and extra prevention layers. For example, to prevent XSS, you can use content security policy. You will have a lot of pain once attackers actually get frontier model access and you still don’t. That’s the scary part.

Paul Asadoorian (14:38.240): That’s interesting, Vlad, because I tend to go to “we still need to patch, we need to get better at patching.” But you’re correct that it’s a multi-pronged approach. I still have to be good at patching, but I also have to get better at defense, detection, and prevention across all of my devices — not just my Windows devices, as we’ve talked about before. I need better…

Vlad Babkin (14:51.043): Mm-hmm.

Paul Asadoorian (15:06.880): …telemetry, visibility, better detection and prevention methods implemented across all of my attack surface. That entire attack surface is now under a microscope, and threat actors are iterating much faster. Even if they don’t have frontier models, they’re still using models to discover vulnerabilities and develop exploits — and that’s been the case since day one. There are so many AI models available, and threat actor skill is going to increase. Whether they get access to Mythos or keep getting better at using local models and developing harnesses, we’re going to see more vulnerabilities and exploits. The big question is: what do we do about it?

Vlad Babkin (15:09.231): Yep.

Vlad Babkin (15:58.992): And the whole point is — if your security appliance does not give you visibility into it, it’s not actually a security appliance. It’s giving you vulnerabilities you cannot see, cannot monitor, cannot do anything about. At that point, you’re only using it because you feel insecure, but it’s not giving you what you’re paying for.

Paul Asadoorian (16:08.246): Mm-hmm.

Vlad Babkin (16:28.176): If this appliance cannot start with an open shell where you can actually navigate all its files, then I have questions. And we have proof that a lot of routers can be really open. For example, NX-OS by default provides you with a bash shell with root access. NX-OS is a relatively complex ecosystem where you can have a huge box potentially housing multiple different appliances in one.

Paul Asadoorian (16:36.790): Yeah.

Vlad Babkin (16:56.995): But the whole point is you can still drop to a bash shell and do a deep scan of those devices — as one example — if you suspect they’ve been attacked.

Paul Asadoorian (17:01.832): Right. But it’s also an advantage for the attackers. A lot of the malware samples I’m looking at are targeting Linux-based devices. When I look at the malware, I think, “That would work on a regular Linux server or desktop.” And because it’s cross-compiled for multiple architectures, it’s going to work on those devices that have a raw Linux bash shell accessible.

Vlad Babkin (17:07.428): Yep.

Paul Asadoorian (17:31.946): That’s an advantage to the attackers because they can live off the land — basic Linux persistence and command-and-control right from Linux. But it’s also an advantage for the defenders if we choose to take advantage of it. I have a bash shell too, so I can do detection, threat hunting, look for IOCs, maybe automate some of my defenses. That’s harder because every Linux flavor and device is different — and that’s a lot of where our engineering efforts are here at Eclypsium, doing things like that. It’s a double-edged sword, but also a great disadvantage on platforms where access to Linux is greatly limited or restricted. All that does is limit the defenders’ visibility, while the attackers have a remote exploit executing commands and living off the land.

Vlad Babkin (18:31.343): Yep. I don’t want to call out any specific vendors, but quite a few of the closed platforms have had major RCE problems — weekly. Not just once, fixed, and gone. Literally weekly or daily new vulnerabilities. Yet another surprise in this platform.

Paul Asadoorian (18:53.280): Well, yeah.

Vlad Babkin (18:58.957): And they still insist on staying closed and saying, “That’s the attacker’s advantage if you become open” — but attackers get a shell every week. They just have to wait seven days for someone to release an exploit. That’s not really stopping anyone in the grand scheme of things. Defenders are stopped by not seeing anything. Imagine as a defender you can monitor this activity and see it early, not have to wait a month to find out you got exploited when you see a ransomware message somewhere.

Paul Asadoorian (19:28.042): Yeah. We have an example of that with Citrix — not to pick on any particular vendor, but when we talk about Citrix NetScalers and utter the phrase “Citrix Bleed,” it’s no longer just one vulnerability. As Watchtower writes, it’s an entire class of memory-disclosure vulnerabilities in Citrix NetScaler devices. Many have played roles in breaches and incidents in recent memory. They link to four previous posts — this being their fifth — on Citrix NetScalers. They say they want to scream “I told you so,” and you can’t blame them. Here’s yet another memory over-read that they describe in great detail, and I’m assuming they publish a script to help you detect it in your environment, which is great.

This vulnerability is also kind of interesting. We saw this with Arista, and now we’re seeing it again with Citrix. Cisco has had similar ones — where just a version check doesn’t give you the whole picture. You may have a CVSS nine or ten, but if you compare the version, it’s going to tell you…

Vlad Babkin (20:47.439): Yep.

Paul Asadoorian (20:54.378): …based on version, you’re vulnerable to that CVSS nine or ten. But if you read the fine print, it says SAML IdP has to be enabled on your Citrix NetScaler device in order for you to be vulnerable. With resources as precious as they are and as many patches as we have to apply, misprioritizing this as an emergency when it’s really not is something we need to discuss. You’re only vulnerable if you’re running the vulnerable version and have that feature enabled — and sometimes it also has to be configured in a specific way. So you now have to combine traditional vulnerability scanning by version with a smart configuration audit to determine if you are in fact vulnerable. I’m not sure how…

Vlad Babkin (21:09.509): Yes. Yep.

Paul Asadoorian (21:51.058): …vulnerability management vendors today are truly tying those things together. When we develop a check for a situation like this, we do tie those things together — as we did with Arista, which we wrote about on our blog. Those are the kinds of checks that give you peace of mind: I don’t have to drop everything and address this one. I can work on other things and continue to monitor, using tools like the ones we provide that tell you if you’re truly vulnerable.

Vlad Babkin (22:25.040): Yep. Beyond that, there’s also one other vector for this: compliance programs for Docker containers. “Your Docker container must not have any vulnerable packages. It must have zero CVEs.” Nobody’s going to exploit a vulnerability in a disk partition manager that’s bundled in a Docker image, because nobody’s doing disk partitioning inside Docker. There are potentially a hundred CVEs like this in an image, and now you’re forcing your software vendor to fix them because of your compliance program — actually diverting their resources from the things they should be defending. A lot of zero-CVE compliance programs are not doing anything useful. They’re actually detrimental to the effort.

Paul Asadoorian (23:13.610): Yeah. We have to focus on classes. When I think about this, it’s looking at services that can listen on a port — whether exposed to the internet or not is another factor. But if you’re focusing on classes like authentication bypass, command injection, and remote code execution on services that are remotely accessible, that’s where you have to start. That’s what really needs your attention. The CVSS score sometimes doesn’t matter — couple in “is it exploited in the wild” and there’s a Fortinet one with a 4.8 medium that’s exploited in the wild. That’s something you have to pay attention to. So you really have to do what we call vulnerability remediation prioritization, and we’re still struggling with it because there are so many factors…

Vlad Babkin (23:50.426): Yep.

Paul Asadoorian (24:12.298): …to take into consideration. What do I patch first? It’s still a huge question.

Vlad Babkin (24:17.626): Yep. There’s also a question about library dependencies and how to score them. Let’s say you have a framework and you find a CVSS ten in it because it’s potentially network-exposable. Then you come to a specific vendor who uses that framework in an older version, and you say they need to be in compliance and need to patch it. But the vendor can do the analysis and say, “We’re not even using the vulnerable feature, so for us it’s a CVSS zero.” But they still need to be in compliance, and vendors are now forced to do a wild goose chase. This also applies to software libraries, hardware images, and quite a few appliances sitting on really old Linux kernels.

Paul Asadoorian (24:49.930): Right.

Vlad Babkin (25:11.918): I believe — and I don’t want to call out any specific vendor — but F5 BIG-IP was sitting on a really old Linux kernel, like Linux 3.10 or something. I might be wrong, or they might have updated by now. That probably has a lot of CVEs. The question to the regulators is: why are you applying all of the regulations to Docker containers that might never get exposed…

Paul Asadoorian (25:31.894): Mm-hmm.

Vlad Babkin (25:40.230): …versus an appliance that’s actually connected to the internet, where exploitation is more than likely to happen, and you allow it to sit on a really old Linux kernel for ten-plus years? Linux 3.10 is not a young kernel by any means. There is a very good question here.

Paul Asadoorian (25:55.688): That’s common. When we reverse engineer firmware, there’s a high probability that when we look at the kernel, it’s going to have some kind of vulnerability — probably multiple. EMBA is great for listing all your kernel vulnerabilities. As a security researcher, I have this firmware and it’s got a vulnerable kernel. What I can’t really do much with is that I need to find some kind of initial access vector. Most Linux kernel vulnerabilities require that I’m already on the box. Many are also privilege escalation vulnerabilities, and with a lot of these devices, services are running as root — you’re already root; there’s nothing to escalate. Compliance and regulation say, “Fix all your vulnerabilities.” But we need to fix the ones that matter. That’s the big question: how do you determine what matters?

Vlad Babkin (26:34.245): Yeah.

Vlad Babkin (26:39.718): You are root anyway.

Vlad Babkin (26:44.602): Yeah.

Vlad Babkin (26:53.392): Yep. And to not speak without data — F5 BIG-IP version 21.x is using base OS CentOS 7.3, and the Linux kernel base version is 3.10.0. CentOS is end-of-life by now.

Paul Asadoorian (27:21.814): Mm-hmm.

Vlad Babkin (27:23.536): So we have this appliance facing the internet in many enterprises, using a really outdated operating system and a really outdated kernel version. And I don’t know how they apply patching. No one is observing.

Paul Asadoorian (27:35.870): No one’s observing remote exploitation of F5 devices via a kernel vulnerability. There could be vulnerabilities in the TCP/IP stack and things of that nature. I haven’t done the study to see if they are reliably remotely exploitable — that could be one concern — but I don’t observe a lot of that in the threat landscape. Not that I see everything, but I see a lot of things.

Vlad Babkin (27:43.141): Yeah.

Vlad Babkin (27:47.515): Yeah.

Vlad Babkin (28:01.360): A lot of the landscape is in their services code, if I’m not mistaken. Also, to give credit where it’s due — F5 is doing a really good job of staying as an open platform. That I can actually Google the kernel version is already a massive step up compared to a lot of the platforms we usually discuss on the show. On top of that, they provide you access to a root bash shell. So if you get exploited, you have a lot of tools for visibility.

Paul Asadoorian (28:14.806): Mm.

Paul Asadoorian (28:24.223): A hundred percent.

Vlad Babkin (28:31.172): And I believe they even partnered with an antivirus vendor at some point. To give credit where it’s due — while we’re speaking about compliance, F5 is actually doing a decent job of giving us this information and giving us access to actually discuss it. That’s a lot more than quite a few other vendors do.

Chase Snyder (28:52.232): Another thing I don’t think we touched on yet is the inclusion and reliance on other open source packages that don’t fall directly inside Linux or CentOS. I won’t name the vendor, but one of the update packages we disassembled and analyzed basically 100x’d in the number of different binaries and files included over the past six years. The sheer amount of stuff going on inside this network device update package increased dramatically — it went from having no Python dependencies in older versions to having a hundred-plus Python package dependencies. We’ve heard about NPM and Python supply chain attacks recently this year. I think it’s only a matter of time before one of those affects, in a traceable way, a network device — where those node or Python packages are getting into…

Paul Asadoorian (29:59.339): Yeah.

Paul Asadoorian (30:10.676): Yeah, I agree. I think it’s just timing. There’s a time window when a dependency for any of those frameworks is malicious, and you have to update or install during that window in order to be impacted. You’re right, Chase — it’s only a matter of time before something is tampered with in a supply chain attack and then gets included in some IoT or enterprise edge device. Hopefully you catch it before you ship it, but if you don’t, you just shipped a backdoored library or dependency inside your devices.

Chase Snyder (30:54.828): Yeah. It’s like if someone put poison in the water stream, and then a downstream bottler happened to be bottling water at that time — they just shipped you poison bottled water.

Vlad Babkin (30:55.407): Let me…

Paul Asadoorian (30:58.474): Yes.

Paul Asadoorian (31:05.610): That’s a great analogy.

Vlad Babkin (31:07.077): Let me raise a hand and say it’s not black and white. Let me give you a little data. First, there are a few companies who became very adept at detecting supply chain attacks within minutes. So monitoring is quite doable with software supply chain attacks — more so than with hardware.

There are two approaches you can use to defend yourself. Approach one: pin all your dependencies by hash. Your production pipeline should install from a pinned list of hashes. You’re not installing by version — you’re installing a specific package hash, and if it doesn’t match, you fail your pipeline.

Paul Asadoorian (31:49.695): Mm-hmm.

Vlad Babkin (32:06.569): Yes, it will lead to a little downtime in your CI pipeline. But that downtime is worth it if the attack is someone messing with a version tag, because you’re not installing by version tag. This applies to Python, NPM, and Go. Second, the actual killer for such practices: stagger your updates. Don’t update to the latest version — update to the version that’s one week old. Let companies detect the poisoned packages. Give the security vendor a time window to react and notify that a package has been breached. Of course, this is at war with “what about the latest vulnerability patch?”

Paul Asadoorian (32:42.302): Yeah.

Vlad Babkin (33:03.747): If something critical comes out, you can do manual validation — download the hash and do your own diligence to validate that after you manually update, you didn’t get any poison. For one package with a critical vulnerability, a vendor can get some people to look at exactly what got delivered in that update within that one-week window. The main point is, if you combine these two approaches and apply them to your software supply chain, hardware supply chain, and firmware supply chain, the attack surface for NPM, Python, pip, and whatever other package manager you’re using is reduced by a lot.

Vlad Babkin (33:51.651): Yes, you can still get beaten, but it’s going to be harder.

Paul Asadoorian (33:52.032): Yeah. Switching gears — going back to the Linux kernel for a moment. It is slightly supply chain related, right? You’re including the Linux kernel, a bunch of modules, and likely inheriting a lot of vulnerabilities. As new kernel vulnerabilities come out — and there have been several making headlines — bad epoll is one, linked in the show notes. That’s CVE-2026-46242. There were several other Linux privilege escalation exploits. This particular one came from Google’s kernel CTF. It works inside Chrome’s renderer sandbox, which is interesting — you could chain a Chrome bug for full privilege escalation to root on a desktop, server, or Android. A six-instruction race window may not sound like much until someone builds that retry loop and gets a better working exploit. It’s interesting how many Linux kernel exploits we’re seeing. But as we established, a lot of times it’s kind of a nothing burger in the embedded and edge device world — that’s not our primary concern. However…

Paul Asadoorian (35:23.402): …there is an onslaught of privilege escalation vulnerabilities, and you should prioritize those accordingly on your Linux servers especially.

Vlad Babkin (35:34.608): Yep. Your network device vendor is doing a really bad job at privilege separation on the device. A lot of so-called security appliances are running all of their externally facing services as root, breaking the very first rule: don’t run your externally facing services as root.

If you add a few security boundaries inside the device, exploiting quite a few of the vulnerabilities would become a lot harder. Suddenly, these kernel vulnerabilities would matter, but you would have to land on the device first. Why do we not hold our appliances that face outside to the same standard we hold our servers that face outside?

The industry is supposed to start asking these questions pretty soon, because even if you call the vendor out, they need time to react, design solutions, and implement them. With how fast exploitation is going to go once attackers get good harnesses, good models, and get better at using all of them, the window is closing.

Paul Asadoorian (36:47.701): Mm.

Vlad Babkin (36:53.967): With how fast exploitation is going to be going once attackers get good harnesses, good models, and get better at using all of them — the window is closing.

Paul Asadoorian (37:05.364): A hundred percent. There’s definitely an uptick in Linux kernel vulnerabilities and privilege escalation. Mythos actually missed this bad epoll one, which is interesting. As much as we talk about frontier models and their ability to identify vulnerabilities, they’re still not finding all of them. Are they helping people find the ones they miss? Certainly. In this write-up, I think that’s kind of…

Vlad Babkin (37:17.887): Mm-hmm. Yep.

Vlad Babkin (37:27.011): Yeah, it’s…

Paul Asadoorian (37:33.726): …what happened — it missed the spawn, but it provided information that allowed a human to find it.

Vlad Babkin (37:42.650): Yeah. It’s also not even about missing the vulnerability. Something we use AI for at Eclypsium — it sees a resource leak that’s not really supposed to be there, but it’s probably fine because it’s in the code. That’s from actual AI reasoning traces. So beyond just code output, read its reasoning, because sometimes it sees bugs and just doesn’t flag them. And another important point…

Paul Asadoorian (37:59.765): Mm.

Paul Asadoorian (38:09.493): Right.

Vlad Babkin (38:12.113): AI accelerates. Okay, you found this bad epoll thing. How do you make an exploit for it? Usually it took humans months, but if you use AI to help you code the exploit once you’ve found it, it might take days.

Paul Asadoorian (38:25.268): Yep. We’re definitely reducing that time from vulnerability to exploit using AI.

Vlad Babkin (38:31.471): Yeah, and it’s not just that. Imagine persistence frameworks — AI can also help you code those. In the past, coding a good implant was hard. Now you can probably code it over a weekend if you have a model tuned to work for you. Frontier models are trying to defend against that, but we often see bypasses. Anthropic had a data distillation attack where many accounts were using their models to distill reasoning traces. If they cannot reliably defend against that, we cannot rely on them reliably blocking every attempt to get their models to do nefarious things. And this is just frontier models. We also have open models that aren’t restricted by anything. An attacker can run their own open model, or even train their own. A well-funded attacker — like a state actor — can actually get an internal AI lab to train them an attack model.

Paul Asadoorian (39:15.862): Mm.

Vlad Babkin (39:30.341): And get their models to do nefarious things. And this is just frontier models. We also have open models which aren’t restricted by anything.

Paul Asadoorian (39:55.892): Right. Yeah. I want to come back to: do attackers need root in a lot of cases to achieve their goals? FortiBleed is an excellent example — it was just credential abuse to gain access to Fortinet devices. Built into Fortinet, they didn’t need to escalate privileges. You can just run a command that sniffs packets. Normally I’d need to become root to sniff packets on a Linux machine, but if that functionality is baked into the appliance, attackers are going to live off the land.

Vlad Babkin (40:32.153): Yep.

Vlad Babkin (40:40.783): Yeah, the whole point is to have better privilege separation — and I’m not just speaking about Linux-level separation. Devices usually have network admin, which has full access, and network operator, which has read-only access to configuration. That’s basically it. That’s not really how you do access control.

Paul Asadoorian (40:54.474): Yeah.

Vlad Babkin (41:10.361): You need a lot of account levels to actually prevent this. By default, your admin does not need the full tooling. You can lock the packet sniffer behind a singular high-privilege account and not give it to admins unless they specifically need it. The admin can use an account with much lower privileges for day-to-day use — if you have good privilege control.

Because if you don’t — and the current industry standard is you don’t own those devices, you’re forced to have full access to everything or nothing. This applies to vendor-specific shells, the Linux kernel, everything. We need more fine-grained controls.

Paul Asadoorian (41:58.580): Yeah. As a regular user on some of these devices, why do I need to see the password hash that represents the admin user?

Vlad Babkin (42:09.018): Yep.

Paul Asadoorian (42:10.314): That’s what they did in FortiBleed — another great example — it was just cracking hashes.

Vlad Babkin (42:21.190): Yep. Why do you display the hash? If you export your config file, maybe encrypt it, or make hashes exportable only under the highest privilege account. Everyone else does not get hash export — or just don’t export hashes at all. Okay, this forces the user to recreate all account passwords if they have to re-import the backup. But that’s a small price to pay for attackers not being able to crack the hashes so easily.

Paul Asadoorian (42:24.661): Mm-hmm.

Paul Asadoorian (42:34.676): Right.

Paul Asadoorian (42:50.155): Right.

Vlad Babkin (42:51.024): So privilege separation is non-existent — not just at the Linux level, but also in the device shell. To the point: Cisco has account levels 1 through 15, but in practice it’s just level 15 and level 1, and everything else is unused. You have room for 15 different privilege levels and you don’t use them. That’s Cisco — and it’s very common.

Paul Asadoorian (43:02.038): Mm-hmm.

Paul Asadoorian (43:08.222): Yeah.

Paul Asadoorian (43:15.080): Yeah, it’s common. Yep.

Vlad Babkin (43:20.538): Why? At this point the industry needs to start asking really hard questions to get vendors to move. It’s not that hard — I’m not saying they should remake their products from scratch. At the very least, add some soft walls. Does your admin need regular access to packet sniffing? No. Admin can have a high-privilege account for rare occasions and a normal privilege account for day-to-day use, with a lot less access.

Vlad Babkin (43:57.425): And some automated tools need really high-privilege accounts to collect relevant data — fine. But those accounts are automated. You can set a really hard password, it’s not being used in a credential manager, and it’s harder to leak from a user device.

Paul Asadoorian (44:17.450): Mm.

Paul Asadoorian (44:21.392): One of the other items we had was on cold boot attacks. I thought this was interesting — kind of in our wheelhouse. The first paper on cold boot attacks was from 2008. The reason they call it that is if you freeze your RAM down to negative sixty degrees Celsius, you can quickly reboot the system, preserve that RAM, and make a copy of it. What this tool represents is a very small, purpose-built way to boot your system that lives entirely in BIOS, because you don’t need much code to boot a system and access RAM.

This uses the legacy BIOS — the CSM, Compatibility Support Module — which mimics an older traditional legacy BIOS environment. CSM still exists on quite a few systems today for backward compatibility. They’re using that capability to have a tiny bootloader that loads just enough code to dump the contents of RAM. This does require physical access. While disabling legacy BIOS probably isn’t really a defense against this — since you can just go back in and enable it — I guess that would make you reboot and lose your RAM contents. Best practice, which I think we even recommend: don’t enable legacy CSM on your computers to start. I think that mitigates a lot of this attack, no?

Vlad Babkin (46:52.228): Yeah, this mitigates this specific tool, but the idea still stands. Consider: what stops the attacker if they’re dead set on attacking this laptop — ready to freeze it to minus 60 degrees and write a custom tool? The attacker might actually look at the pinout for the RAM and just…

Paul Asadoorian (46:56.822): Mm.

Paul Asadoorian (47:20.478): Mm-hmm.

Vlad Babkin (47:22.936): …add wires to it. If they have physical access to the level of freezing it below 60 degrees and booting from a USB stick with a custom tool, at that point I would question if they’d instead rip the RAM out for laptops that don’t have soldered RAM, or add wires to the RAM to keep power to it, shut the laptop down…

Paul Asadoorian (47:24.678): Right. If they have physical access, all bets are off.

Paul Asadoorian (47:47.231): Yeah.

Vlad Babkin (47:52.612): …and the RAM still gets power and retains all the values.

Paul Asadoorian (47:56.234): So there’s really not much you can do about that — it’s going to persist a reboot anyway. They can change BIOS settings; there are a lot of ways, as you described, Vlad. Protecting against this is very difficult.

Vlad Babkin (48:11.738): Yeah. In this case, maybe RAM encryption is the answer. Linux had something like this for a while so that even if you dump RAM, you still have a lot of pain trying to decrypt it. TPM might actually help with keys for this. The biggest question in my brain is how exactly does this tool solve the problem — it’s a really cool tool, kudos to the author. But…

Paul Asadoorian (48:17.780): Mm-hmm.

Paul Asadoorian (48:22.281): Mm.

Paul Asadoorian (48:37.386): Yeah, it is.

Vlad Babkin (48:40.814): The attack itself is much wider than this one tool.

Paul Asadoorian (48:44.670): Right. Agreed. A sneak preview. I’ve been working on a tool. I’ve written lots of scripts to help me update Linux systems — I’ve been in Unix systems administration for longer than I probably care to admit, and I’ve had lots of cracks at this. I wrote a couple of shell scripts I used to update my systems, and one that was a supply chain checker based on our Eclypsium cheat sheet. Recently I had this idea — since I still have to maintain Linux systems and I want visibility into both software supply chain attacks through packaging systems and hardware-based supply chain attacks, like whether my microcode is up to date.

I basically rewrote everything using Claude’s help in Python, which was great because I had tested examples to build off of. It was a great usage of Claude — take these already working scripts and convert everything to Python. There are no external dependencies. It’s still in testing, so there’s a link in the show notes. I’m going to talk more about it next week. Some of the cool things I built in include telemetry about the integrity of the packages you’re installing. Atomic Arch was a big one, and really any of these package systems — apt, Flatpak, Snap — are all susceptible to some form of supply chain attack. Some have better protections than others, and different tools for you to evaluate the state of a package: did the maintainer change? Has the package been orphaned? How long since it was last updated? Does the package build script contain malicious commands? Your mileage will vary whether you’re Arch-based or Debian/Ubuntu-based. I built a lot of those checks into this tool.

Paul Asadoorian (51:08.852): The other thing, to put my money where my mouth is, is that I wanted to use AI to help ease the pain of updating. Updating something can have negative consequences — how do you minimize those? I found myself collecting system artifacts, collecting information about what the upgrade is going to do, and using AI to say, “Hey, I have this system, I’m applying these updates. Tell me if I’m going to have a good or a bad experience.” I actually built some of that into the tool. I estimate right now that each query costs about 30 to 60 cents per run using the models I’m using. What that gives you is whether AI believes this is going to be a good or bad experience, plus tips on things you can do before and after the upgrade to ease your pain and make the upgrade successful.

It’s still in beta. Use at your own risk — if your system blows up, don’t blame me. There’s a huge disclaimer on it right now. It’s still just in testing. I think it’s a pretty fun, neat little utility. Topgrade is also really good — to my knowledge it doesn’t have that kind of supply chain checking built in, so that was one of the big feature differences. Fleet is a complete, fully open-source framework, but it’s a pretty big lift to install — it’s a whole ecosystem. They do have a free version, so you can check that out as a similar tool. I just wanted something I could maintain and control myself, and build in features as I see fit with a flair for security. I’m greatly concerned when we update things — as we talked about with supply chain security, these package ecosystems are a nightmare right now. We need tools to give you some telemetry and indications that…

Vlad Babkin (53:29.700): Yep.

Paul Asadoorian (53:37.544): …that update could have some security implications to it.

Vlad Babkin (53:43.239): Yeah. The hardest problem is still: this package was actually taken over by an attacker — how do you detect this? I believe this tool doesn’t really solve that, but it solves a lot of other questions, so it’s really cool. Detecting orphaned packages is a massive one. In the Arch ecosystem, orphaned packages can be taken over by anybody.

Paul Asadoorian (53:53.142): Mm.

Paul Asadoorian (54:08.734): Yeah, and I hope they change that. I didn’t like that either — it’s a little too open of an ecosystem. The other package systems don’t do that. I hope AUR, the Arch User Repository, follows suit and adds some validation process before anyone can take over a package. What my tool does, though, is that when the maintainer of a package changes, it gives you an indication — an early warning system. Maybe the package isn’t malicious; maybe someone just legitimately took it over. But at least it tells you: “Hey, this package was orphaned and hasn’t been updated in 958 days, and by the way, the maintainer just changed and there’s an update.” That could be completely benign, but at least it gives you an indicator to look into it, because that’s not happening to every single package.

Vlad Babkin (55:04.378): Yeah. That would be cool. I’ll give you a suggestion — I’m not sure if Arch has hooks you can run pre-install, but it would be interesting to actually block the install in such cases.

Paul Asadoorian (55:16.507): It does. Yep.

Paul Asadoorian (55:20.606): The tool yay is the package updater, and it has Lua hooks. Claude wrote me some Lua hooks that give you pre-warning indications. It doesn’t actually block the install, but it halts and gives you a warning: “Hey, by the way, these pre-installation hooks ran, and this package build file is looking at your SSH keys. Maybe you don’t want to install that package.”

Vlad Babkin (55:50.554): Yeah. Having a bunch of hooks like this would actually be helpful, and having them cross-platform — if you add this to the tool, it will become really valuable. Just hunting for suspicious stuff in your libraries and packages. It won’t save us from every type of attack or every way to take over an account, but let’s at least cover the obvious ones.

Paul Asadoorian (56:19.454): Right. Well, Vlad, Chase — thank you very much for appearing on this edition of Below the Surface. Thanks everyone for listening and watching. We’ll see you next time.

Vlad Babkin (56:20.166): So.

Chase Snyder (56:33.154): Good talk, guys.

Back to all Podcasts