BTS #69 - Navigating Network Edge Vulnerabilities
In this episode of Below the Surface, Paul Asadoorian, Vlad Babkin, and Adrian Sanabria discuss the ongoing vulnerabilities in network edge devices, the implications of legacy systems like Ivanti, and the strategies employed by threat actors. They explore the importance of monitoring and detection in cybersecurity, as well as innovative deception techniques to enhance security measures against exploitation. In this conversation, the speakers delve into various aspects of cybersecurity, including innovative strategies to enhance security, the challenges posed by vendor cooperation, the implications of cyber insurance, and the importance of visibility in threat detection. They discuss the use of canary tokens, the exploitation of edge devices, and the reality of zero-day vulnerabilities. The conversation also touches on the need for firmware updates, the shift towards open-source solutions, and the role of AI in developing cybersecurity tools.
Subscribe
Transcript
Paul Asadoorian: Welcome to Below the Surface. This is episode number 69 being recorded on Friday, February 27th, 2026. I’m your host, Paul Asadoorian, and joined by Mr. Vlad Babkin. Vlad, welcome.
Vlad Babkin: Hello.
Paul Asadoorian:And our very special guest for this episode, no stranger to podcasting, Mr. Adrian Sanabria. Adrian, welcome.
Adrian Sanabria:Hey, Paul, nice to be here. I’m excited.
Paul Asadoorian: Network edge devices, which just blows me away that we’re still here talking about it, but somewhat also not surprised that we’re still talking about it. Because this trend, I think, is still on the upswing. Adrian, I want to get your thoughts in general on the just general trend of attackers going after these network edge devices.
Adrian Sanabria: I mean, you know, on the one hand, I do have a blog post where I kind of talk about this. I refer to the asbestos of IT, which all these file transfer servers, FTP VPN, like old school IP sec and SSL VPN exposed to the public internet. Like we don’t have to expose all that attack surface in 2026. have stuff like tail scale. We’ve got, you know, wire guard.
Paul Asadoorian: I love tail scale, dude. It’s so awesome.
Adrian Sanabria: ZTNA technologies that allow us to access remote resources without exposing TCP services or UDP services to the public internet, which is how most of this happens. know, most of these edge vulnerabilities we see, and I don’t know SD-WAN a little bit. We’re going to talk about that. I’m not sure I understand the, whether or not that is absolutely necessary to expose to the public internet.
Paul Asadoorian: Right.
Adrian Sanabria: Because it looks like an authentication, an issue in the authentication system that is exploited there. But generally, think there’s a lot of work we can do to just re-architect to remove a lot of attack surface.
Paul Asadoorian: Yeah, agreed. We’ve touched on that before, Adrian, and I’m just puzzled as to why enterprises are still kind of stuck on these somewhat legacy VPN models. If you read the Bloomberg article on Ivanti, it covers some of that history, So go ahead, Vlad.
Vlad Babkin: I can add a little bit of oil into the fire actually. If you take a look at modern devices, one example is Microtik that I use for my home network. And it’s not that hard to configure them to not expose anything but the OpenVPN. Like Microtik actually can run the OpenVPN server on top of it or WireGuard or whatever, both of them. And in this case, if you configure it correctly, firewall it off and everything nicely. The only port that you will expose to the public is literally OpenVPN. and whatever you want to expose to the public, like maybe forward some service inside of the network. certain devices, like which probably are not very enterprisey, already even have a solution to this. So an insistence of enterprises of using stuff that literally exposes management interfaces is strange to me.
Paul Asadoorian: Yeah, I’m somewhat ignorant to exactly how tail scale works, because you don’t have to expose a port to the internet, but Adrian, go ahead.
Adrian Sanabria: Yeah, I think the other factor here is that these all tend to be legacy products where their code base has existed for 10, 20 years in some cases, you know, that are not really receiving a lot of quality of life updates, you know, just a bare minimum to be able to keep selling them. And as we see from the Bloomberg article about Ivanti here,
Paul Asadoorian:Mm-hmm.
Adrian Sanabria: They’re not even interested in doing that. They’re not even interested in going after the security issues and fixing them. They’re kind of just letting them, letting whatever is going to happen happen.
Paul Asadoorian: No.
Adrian Sanabria:So maybe avoid some of those vendors and products as well, right?
Paul Asadoorian: Yeah, for sure. So there is a Bloomberg article. I do want to state, and I talked about it on my other podcast too, Bloomberg wants to charge like $2 a month, which would be fine, except I need to read a Bloomberg article like maybe once or twice a year. Like if they had a cybersecurity section and it was at least somewhat, if it was value-wise worth $2 a month, 100%, I would subscribe for $2 a month, right?
Adrian Sanabria: So I recently found, believe it or not, I got a new library card for the first time in 40 years. And you can access a lot of magazine subscriptions with a library card without having to pay for them. Yeah. They have like an app you can put on your phone. I forget the name of the app…
Paul Asadoorian: Yeah. I don’t want to take away from, like, I understand why they want to charge $2 a month. And look, I would rather have you charge $2 a month than try and do some, like, crazy advertising thing that, from cybersecurity standpoint, is somewhat risky. So I appreciate it. I get it. Everyone’s going to make a living. But the audience, like, when you put together one really good or once or twice a year, a really good cybersecurity article, well, I can’t justify it.
Adrian Sanabria: It’s called Press Reader is the name of the app. And you just put, yeah, you just plug in your, what library it is and your library card number and boom, can read like that’s the economist right there, which is a really expensive subscription. yeah, yeah. So that’s, that’s a, hot tip. You can get a library card for free.
Paul Asadoorian: Oh yeah, I used to subscribe to The Economist. But I haven’t read that in a long time. And of course, like, I don’t know – we’re hackers. It’s not like it’s public knowledge how to get around paywalls. Like it’s not even most LLMs will tell you some ways to get around paywalls. And so no knock to Bloomberg. But I really want to read the article. And I wasn’t the target market for the subscription. So in any case, I’m reading the article. It’s a really good article, though. Bloomberg interviewed seemed like a pretty large number of ex-employees of Ivanti.
Adrian Sanabria: And you. Yeah.
Paul Asadoorian: Ivanti bought PulseSecure, right? Because that was, I always forget how Ivanti was created. Did Ivanti ever have their own software or were they just a company that was created to acquire other companies?
Adrian Sanabria: So they were a combination. It was just a new name they gave because it was Heat Software and, you know, Shavlik and like all these different companies.
Paul Asadoorian: Mm-hmm.
Paul Asadoorian: It was Shavlik, the patch. Is that what their EPM is? That’s Shavlik. Okay, that makes sense. It makes sense that there’s nothing truly new. Then when I hear these things and I talk about them, I’m like, what’s an Ivanti EPM? If someone had just told me, by the way, that’s Shavlik, it would have saved me a lot of reading. Because I’m like, I already what that product does. I’m just old.
Adrian Sanabria:Yeah. Yeah, it was.
Adrian Sanabria:you Yeah, I just happened to be an industry analyst when all that was happening. So I covered a lot of that. I wrote up a lot of that merger and acquisition stuff.
Paul Asadoorian: Yes. right. So Ivanti is really just the shell company, if you will, and it acquired MobileIron for older cybersecurity folks like us, we recognize those names, and it acquired PulseSecure. And then rebranded it to Connect Secure. So you’ve got this really confusing naming thing where you’re like, wait, what is this VPN? Is it Ivanti? Is it Connect Secure or is it PulseSecure? Today, it’s Ivanti Connect Secure. is the product. You can run it as a virtual appliance or a hardware appliance. And then again, they have other offerings. But it is the Pulse Secure/Connect Secure line that this article is focused on. Do they touch on MobileIron a little bit in here? I’m not sure. They look like they’re primarily focused on the Pulse Secure/Connect Secure line. And it really shed light as to why we’ve got some of the problems that we have. If you believe me, some people are skeptical. Some people are like, well, you know, this article’s overblown. It’s not really, know, Ivanti tried to do some hand waving. And I think some other analysts were like, it’s not as bad as that Bloomberg article painted it out to be. But they do make a pretty compelling case. Vlad, Adrian, I don’t know if you got a chance to read this article, but I thought it was really good.
Adrian Sanabria:Yeah, mean, 119 organizations breached. there’s a handful, I can count on one hand the number of companies whose products got over 100 companies breached because of one vulnerability, right? Like, move it to a DMZ.
Paul Asadoorian: And that was the supply chain. That was the supply chain breach, Where the Russian threat actors breached Pulse Secure in like 2021, backdoored the product and it shipped to 119 customers. I think we knew that, but we didn’t know how many customers, right?
Adrian Sanabria: Yeah, this is huge. Like it just keeps happening.
Paul Asadoorian: And then have other vulnerabilities that get exploited by threat actors, 100%.
Adrian Sanabria: And the one thing that this article doesn’t drop in there is the fact that Ivanti was one of the signers of the Secure by Design CISA pledge back in 2023, right in between all this stuff. And we do have some inside knowledge here from the former CISO of Ivanti saying, yeah, there is no effort to fix any of this stuff.
Paul Asadoorian:Right. Yeah.
Paul Asadoorian: Yeah, so we have first slash secondhand accounts, however you want to count them of that kind of thing happening, which is telling but what was even more, I think, like important and carried a lot of weight was the not just number of customers that they lost, but the who those customers were. So basically, at one time, like we had pre pandemic, then pandemic hit, and everyone went out and bought VPNs. And VPN spending was on the rise during the pandemic. Then, as the pandemic kind of wound down, people started ditching their VPNs. And also it should be important from a business standpoint at that time, Ivanti made that decision to go from you buy it and pay us upfront to a subscription model, which further hurt them financially. There’s some numbers in there financially that are just astonishing.
Adrian Sanabria:Yeah, they went from like 300 million in revenue on this product to 150 million in revenue in like 18 months. Even the US government was saying, there’s no other advice we can give than stop using this product, which is rare.
Paul Asadoorian: And they did. They lost the Navy, Air Force, Army, and several private companies. It’s here in the article. I’m trying to find it. But they lost a huge amount of DoD customers and a huge amount of private companies. And actually, funny, I remember talking to a CISO. And I’m not saying that this is the case, because you can go look it up in the PSW archives. But we were having this conversation and he kind of hinted towards on live on the air that we’re making decisions to get rid of technology based on security incidents. And now my guess is he was referring to Avanti after this article came out. And again, that’s just speculation on my point.
Adrian Sanabria: And there’s some stats out there. Looks like most people jump to Palo Alto, something like 70 % of all VPN use is, what do they call it? Global protect. Yeah. That’s where it came from.
Paul Asadoorian: Yes. It is the global protect. Yeah, in the latest GreyNoise Report Global Protect was was huge. Yeah, huge in numbers on that. Sorry, Vlad, you were trying to say something.
Vlad Babkin: It’s not even that surprising. You’re running a cybersecurity company. I’m not speaking just about Ivanti, just every one of them. If you refuse to fix vulnerabilities in your product over years and years and years, and they keep piling up, and people start talking, hey, there is no effort to fix them inside your company. Well, it’s going to eventually translate into a huge reputational hit, after which just lose customers. Because de facto, all of the VPN companies Like they use WatchGuard, OpenVPN maybe, maybe. And it’s literally just two, three protocols in play. And in general, nobody of them invented anything radical in you. Like, I don’t know about tail scale in this case specifically, but most of the appliance-based stuff is literally OpenVPN nicely bundled in the box, right? So this is…
Paul Asadoorian: Yeah. And some other weird VPN binaries that I haven’t gone back and looked at the lineage of them. I think it would be an interesting study. But in a lot of these network edge appliances are, so you’ve got Ivanti, Fortinet, Palo Alto, and Cisco. You can trace back these, the buffer overflow vulnerabilities, right, since they’re Linux under the covers, to a Linux service that’s listening on.
Vlad Babkin:Yeah.
Paul Asadoorian: Typically the default port of 8.443, and typically that’s like slash SBIN slash SSL VPN D. And it’s that binary that contains the memory corruption flaws that have been, when, you know, that’s just one class, obviously, of bugs that we have in there. It’s perhaps, I don’t want to say it’s the most detrimental, because it’s memory corruption, keep in mind, you have to tune your exploit to your target. So these devices can be run as virtual appliances on x86. So you need a specific kind of exploit for that. They can also be targeted at a physical appliance. But those physical appliances, as is the case in Fortinet, they’re ARM32. And then I think like newer models are ARM64. So you take all those three platforms as an attacker. If I’m going after memory corruption and I need a ROP chain, I need to execute a payload. It has to be tuned for that platform, right? And tuned for that version that is installed on that platform. So if you’ve looked at, and I’m not picking on Fortinet, I just happened to look into it. They’ve got a lot of different versions of FortiOS. Like, I think version five, six, seven, and I don’t know if they’ve gone to eight yet, but six and seven are some of their major versions that are still supported, even in six. But there’s a lot of different versions of them.
Vlad Babkin:That’s it.
Paul Asadoorian: And if that binary changed in between those versions, you have to adjust your exploit for that version that’s running either on x86 or ARM32 or ARM64. So you have those problems. But keep in mind, a memory corruption exploit gives you code execution on that Linux layer, which is super useful for attackers, right? Then what I noticed threat actors doing… is they’ll go into the bootloader on those devices and maintain persistence. They’ll somehow embed themselves either through SimLinks in different partitions or right in the bootloader like on Raman on Cisco FDD devices, they’ll actually infect the bootloader. And I’m not just saying like that’s a thing and it’s possible. That’s based on I’ve looked at the vulnerability in Fortinet, which threat actors have used it. How have they used that? How have people observed those threat actors doing it? And that’s where that information comes from, which is actually, it’s so much fun, I’m overwhelmed, because you can do this for all the major platforms. You take the major vulnerability, and then you go start tracking threat actors. then you start drawing parallels to what they’re doing. And they’re basically all using this as jumping off points to collect credentials and conduct operations. It’s C2 infrastructure as well. Vlad, go ahead.
Vlad Babkin: Anyways. Massive case in point, all of the added value for these devices does not come from VPN features itself. It comes from all of the added services on top of the binaries they just keep reusing. And like, if you suddenly stop providing that added value…
Paul Asadoorian: Yes, I agree.
Vlad Babkin: Bye bye, company.
Paul Asadoorian: If you just needed one device, right? Like you’re an individual or even a small company, need one device, you could use tail scale stand up, a PF Sense, Open Sense, OpenWare, and you’d be fine. The enterprise use case is they need tens, hundreds, thousands of these potentially deployed. So not only do I need something that can be deployed very reliably, but I need a management layer on top of that. And if you look at Palo Alto, Fortinet, and Cisco as example, their management platforms are also like huge beasts of software that help you manage all that infrastructure, which by the way, have also had vulnerabilities that are exploited by threat actors. you just keep increasing your attack surface in this area in an effort to provide security for your company, which is crazy.
Vlad Babkin:Yep, and in this case, like, the real solution to this probably would be when somebody comes out with a service which allows to deploy all of this Kubernetes containers and whatnot, and just containerize this OpenVPN without any extra interfaces so that, like, it’s managed with normal infrastructure as code kind of stuff. But that doesn’t exist yet in my brain.
Paul Asadoorian:Sure. Yeah, I know there’s some, and I think they’re for like training purposes, but I know that when I looked at yesterday, I was looking at Juniper platforms, it was like, how can I virtualize them? And they had one platform that it looked like it spun up a container, a Docker container, and inside the Docker container, it runs Linux KVM and virtualizes one of their platforms. And I was like, Really? inside a Docker container? They said they did that to make it easier for people to deploy. said, our customers told us they wanted containerized stuff, so we made it deployable in a container. Those of us that are good with Linux or watch my technical segment on Wednesday know a little more about how to set up Linux KVM now. And I’m like, just give me a QCOW2 image and deploy it. I that’s what Cisco does. That’s what Fortinet does and a lot of other folks as well.
Adrian Sanabria: Inside a Docker container?
Paul Asadoorian:And I mean, not so secretly, right? I’m building a lab. I want to be able to run this stuff. I want to be able to test this stuff. I want to be able to infect these things, watch that infection, and then develop ways to help detect that infection in an effort to help enterprises with visibility. Because the mind-blowing thing, and we talked about it in our webinar, right? The mind-blowing thing, I came to this point, I’m like, look, it’s interesting how these vendors put out a platform. and they’re like, here’s an appliance, virtual or physical, you get, as the customer, you get access to like the operating system layer, which for those more technical, right, Cisco iOS, Cisco FortiOS, like that’s your interface to the device. Underneath is Linux, but many vendors are like, no, no, no, no, you can’t, as Mr. Customer, you can’t go into that Linux layer because that would cause support issues. or whatever the reasons are, support being probably the number one. And cybersecurity companies like us are like, I want to give you, like I had literally people on the webcast asking me, Paul, why can’t we enable eBPF monitoring on the Linux subsystem on these enterprise appliances so that we can monitor what’s happening in Linux? I’m like, you’re making so much sense right now. It’s a great, like you’re spot on. I’m like. But as cybersecurity companies, we don’t have access to that layer either, unless we exploit a vulnerability. But we’ve all the time, we can’t put an exploit in our product, okay? That’s not what we do, right? We can’t be exploiting vulnerabilities to provide visibility, so cybersecurity companies are limited. So what does an attacker do? An attacker finds a buffer overflow, vulnerability, and exploit, and they live inside that Linux layer where the user and cybersecurity companies have limited visibility. And that’s the frustrating thing.
Adrian Sanabria: It is not unheard of for customers to hack a vendor’s product to make it more secure. That is not unheard of.
Paul Asadoorian: Yeah, it’s happened, right? I get it. I know we’ve had the discussion many, many times here at Eclypsium. maybe someday, who knows? I know. It’s an ongoing discussion too. We haven’t closed the books on it so that makes it hard, right?
Vlad Babkin:We should have opened them.
Paul Asadoorian: I know that most of these are really just Linux underneath. When we talk about Cisco SD-WAN, that’s just Linux underneath. And I was not familiar with the Cisco SD-WAN platform. And also SD-WAN is kind of this like weird catch-all term, I think, for different, slightly different technologies. But when I, it was NCSC, someone published, I put it in the show notes. Someone published a really great guide. Oh, it was like the Australian counterpart to Sysa maybe? Produced the Cisco SD-WAN Threat Hunt Guide. I well, this is great. When you look through the Threat Hunt Guide, they’ve got you looking at things like slash, var, log, and looking at SSH configuration. So I’m like, this box is just Linux. I can tell from the hardening guide, this box is just Linux. So, but it looks like they might give you access to that Linux layer if the hunt guide is like you need to go look at this file.
Adrian Sanabria: Yeah, yeah, I mean most Cisco devices you do get some kind of CLI, but most of what I’m used to from most of my experience goes way back is iOS CLI, not Linux based.
Paul Asadoorian: Mm-hmm. Yeah, was, there was like some juncture where everyone started building in Linux to the underlying layer. And I think that allowed them to virtualize the iOS and FortiOS operating systems. And there was a lot of benefits to doing it. I remember reading about it, forget what all the, but there was like benefits to doing it that way and changing the architecture. Yeah, right.
Adrian Sanabria: It just makes development cheaper and quicker.
Vlad Babkin:And some of the Cisco products actually allow you access to underlying bash, like an Exos does, et cetera, et cetera.
Paul Asadoorian: Right. And so FTD does as well, which is super handy. Then we can provide better visibility into those platforms. We do that in FTD. can tell you the Arcane Door campaign, if you haven’t looked at that, they targeted FTD devices, did some pretty amazing things with them in terms of credential gathering. One of their line dancer or line runner payloads, has a network sniffer and also can harvest credentials locally from the VPN appliance itself. if you’re not patching, but patching, this is interesting because Adrian and I have talked a lot about vulnerability management with each other in the past. The thing that gets me is if you’re an organization that’s relying on vulnerability management to solve this problem with your network edge devices, it’s too late. The traditional way of doing vulnerability management, is not really going to help this problem. We saw with the Cisco FTD, the most recent one, that Cisco came out and said, threat actors have been exploiting this vulnerability since 2023. Right? And this is not the only one.
Adrian Sanabria: I have stats on this. So I’m putting together a blog post on this, basically I’ve been kind of obsessed for the last year about speed of exploitation. So the answer to the question, how fast do you have to be to fix your stuff before attackers can exploit it, can use it? And the answer is you can never be fast enough. So it kind of shifts focus to like passive mitigations, to hardening, to things that generally make exploits harder to pull off or harder to leverage or better detection on lateral movement. because according to Mandiant, the average time to exploit in 2023 was five days. And then in 2024, it dropped to negative one days, which means the majority of active exploitation that we’re seeing is before disclosure. These are zero days, right? And in this case, this is…
Paul Asadoorian:Yep. There’s several sources that that backup your statements, Adrian, 100%.
Adrian Sanabria:And this is exactly that case where, you know, now that we know what we’re looking for, you know, we can see it happening for two years already.
Paul Asadoorian: Yep. so monitoring these devices is super important as well. Because as we march through the different classes of bugs, authentication bypass is also big too. Now, that doesn’t give them a Linux root shell, but authentication bypass can do a couple of things. One, it gives you the attackers, threat actors access to your firewall. And so what they’ll do is if they can just bypass authentication…
Adrian Sanabria: Look for Shabby Stuff.
Paul Asadoorian: They’ll set up an SSL VPN tunnel for themselves and put themselves on your internal network. And they really only need the authentication bypass to do that. And the nice part about the auth bypass is it’s not dependent on architecture. It just works natively within the application, and you’re just using it. Similarly, command injection might be the most damaging class of attacks on these devices because it is platform.
Vlad Babkin:Yeah.
Paul Asadoorian: You know, CPU architecture is agnostic and gives the attacker not just the capabilities in your VPN, but also the capabilities to execute commands that could lead to a Linux implant being deployed on the system. So the combination, of course, of I’ve seen combinations of authentication bypass to command injection. The latest FTD demo that I built uses an authentication bypass exploit. to find the URL endpoint that doesn’t require authentication. And it just needs that to go to the next stage to do the buffer overflow exploit to gain a Linux shell on the device. So it’s chaining these exploits together, which is hard for defenders, right? How do you assign a severity to multiple vulnerabilities now on any system, specifically edge devices?
Vlad Babkin:Yeah, there are practical cases where like three or four low to medium vulnerabilities when combined are becoming one critical one. And you cannot really do anything about it. And also there’s another beautiful thing about like authentication bypassing these devices, you don’t necessarily even need to make an SSL VPN tunnel. So like if your strategy is, hey, I will just monitor if there are new SSL VPN tunnels, what the attacker can do is just open themselves support that goes into some internal service.
Paul Asadoorian:Mm.
Vlad Babkin: And, oh hey, now I will monitor open ports. Okay, they can modify routing tables and that’s another surprise they can do. So there is a lot of stuff that they can do with configuration there which you will never be able to detect and see.
Paul Asadoorian: Yeah, I mean also if SSH is on the Linux system, that’s another avenue that attackers could use as well.
Adrian Sanabria: Yeah, which is why I think deception is really interesting. Put some fake SSH on there. Put some fake binaries on your system. Stuff that you know the attacker is going to reach for immediately, that living off the land type of stuff. Yeah, make detection easy by putting fake stuff out there.
Paul Asadoorian: Yeah, you know, this is a case I don’t often recommend people go towards things like canary tokens in honey pots. I mean, I love canary tokens. I think you should have them right. But do we now put canary tokens on our network edge devices if we can like do we have developed custom ones? I this is a case where you that given the lack of visibility, you need anything that you can reach out for to put on here to get some visibility. So yeah, I would do that. It’d be interesting.
Adrian Sanabria: Absolutely. They’re the ones getting hacked.
Paul Asadoorian: Like I’d wonder how you could do that inside like the FortiOS or iOS layer if you don’t have access to Linux.
Adrian Sanabria: I gotcha. So if you have access to bash or some kind of shell like that, the way that I do it is I create wrappers for binaries. So I will create an alias that loads when you log in, use your bash profile, your bash RC, and that alias will basically run a script that triggers a canary token and then runs the actual binary. So instead of…
Paul Asadoorian:Mm-hmm.
Adrian Sanabria:And you can replace the actual binary with a canary binary, but I find it much easier to just use an alias because nobody’s going to look to see if they’re being run through an alias first. And by pointing that alias at a script that you hide somewhere in the file system, everything looks normal when they run the command. Like there’s no difference between the command running. if it goes through an alias or doesn’t go through an alias because most of these canary tokens are just, you know, curl this HTTP or, you know, trigger this DNS, do a lookup on this DNS.
Paul Asadoorian:I see. So in the script you trigger the canary token and then you just run the command that was alias. Yeah.
Adrian Sanabria: It’s literally just a curl line and you hit that HTTP destination with curl or Wget or whatever you have on the box.
Paul Asadoorian: Yeah.
Paul Asadoorian: Right. And then you know, hey, an attacker ran this binary. Or someone ran the binary.
Adrian Sanabria: Yeah. Well, and not only that, but you can use, so when I do this, when I write these scripts, you can grab information about them. So like one of the things you can grab is the IP address that they’re coming from. And you can use the user agent. Every time you touch a web server, there’s a user agent field and you can use that to smuggle out all kinds of details like what account is it, what IP address are they coming from, like any details you want.
Paul Asadoorian: Interesting. Yeah, yeah. Right, right.
Adrian Sanabria:You can put in that so not only do you get an alert saying somebody just ran this command on this box But you can also see what IP address that they SSH-ed.
Paul Asadoorian:That’s awesome. That’s awesome. Yeah, we definitely see, I think we need this level of creativity to help defend this attack surface because I don’t think we’re going to get anything better anytime soon. I mean, there is the, if you want follow the headlines, right. Crowd strike partner with F5 to do some stuff, but I haven’t seen much else in the way of providing great visibility. I mean, other than when we do it at Eclypsium, we do some pretty amazing things to basically, we have to live off the land that’s given to us. And we do some amazing things there too, to give people visibility. And that’s the level of creativity we’re down to, the vendors aren’t going to give us more access. I don’t see that happening.
Vlad Babkin: Yeah, well some vendors actually are cooperating well, like Cisco is one example. But some vendors are just a nightmare. Customers will want visibility into Palo Alto. But Palo Alto is a fully closed ecosystem. They’re not giving you any kind of access. So the only access you can hope for, you cannot even get aliases. So you don’t even get that. So there is like a question how to monitor them and like, answer there is not easy.
Adrian Sanabria: Well that should go into your build versus buy decision. You know, when picking something up is our ability to detect attacks on this appliance are limited or, or, you know, you just need different strategies. Maybe.
Vlad Babkin:Yeah, it was very funny to me when I heard the news that cyber insurers actually increase the insurance cost when you use SSL VPN appliances. it was very fun.
Adrian Sanabria: Oh, really?
Vlad Babkin: Yeah. Yeah. Because that’s now in their actuarial data. When they look at their actuarial data, they’re saying, okay, like 30 % of what we’re paying out on start with these edge devices. if you’re gonna again, the asbestos, that’s why I call it asbestos, right? You know, like, like, the industry knows it to be dangerous. There are alternatives.
Paul Asadoorian: Yes.
Adrian Sanabria: So as time goes on, you have less and less excuse to use it. You’re going to get fined. You’re going to pay more for insurance.
Vlad Babkin: Because it’s not that hard to actually exploit your device, even though it looks like it should make exploitation harder, but no. It makes exploitation easier.
Paul Asadoorian: Right. The way attackers are hiding on these platforms is, you know, I mean, they don’t even have to be that stealthy because we don’t have great visibility. So that’s why they’re there. Yeah, go ahead, Adrian.
Adrian Sanabria: Can I share something real quick, Paul? Yeah, so this is a blog post. I put it in the chat that I wrote up back when I worked at Thinkst. And you can do this with… Yeah, I did.
Paul Asadoorian: Mm. that’s right. You worked for Haroon at Thinkst, which I love Haroon, the whole team and the company. You guys are great.
Adrian Sanabria: So this is just using an HTTP token, which is pretty easy to use. Thinkst is not the only one with these tokens. There’s a couple of vendors now that you can go with. Of course, they do have free ones at canarytokens.org if you just want to play around with this. And so yeah, it’s really important to have the token reminder. I think this is the biggest mistake I see people make, is they try to create a single token and then put it on their whole production infrastructure. But then when you get an alert, you don’t know where it’s coming from unless you capture that. Yeah, you gotta capture that host name. But here’s where I’m using the user agent. And basically, I forget the command switch, but with curl or wget, there is a switch you can use to put whatever text you want into the user agent. And this is where I use these variables, these system variables to capture the command that they used, the username.
Paul Asadoorian: You don’t know which host which host it came from. Yeah.
Adrian Sanabria: That they used on which host and then the source IP address that they were coming from. So there you go. You get a lot of useful information to start your investigation.
Paul Asadoorian: That’s awesome. You have more stuff to consider. Yeah, to protect the stuff.
Adrian Sanabria: And this is just Netcat I was using as the example here. And so it’s all in a little netcat.sh, probably two lines, three lines long. I think I have it in a gist somewhere.
Paul Asadoorian:That’s great. And you can have your own infrastructure, like open source, right? Like you can do it all yourself with the Canary tokens.
Adrian Sanabria: Yeah, I think a lot of it is open source. They have an open source canary, open canary. And with the tokens, I think you can, but I’ve never tried it. I don’t know what all is involved there. But yeah, it’s not super sophisticated stuff. It’s like the reason you pay for things is you want to deploy a million of these things across your infrastructure.
Paul Asadoorian: Right. Yes.
Adrian Sanabria: Using their API, right? You don’t want to have to do all that by hand.
Paul Asadoorian: Yeah—and you want something low-interaction too, which is much safer.
Adrian Sanabria: Low interaction. I just think you’re wasting a whole lot of time when you go high interaction. It’s a lot of setup time, it’s a lot of work. And every company I run into is like, “I don’t want to study the attacker’s behavior. I just want them gone. I just want to keep them out.”
Paul Asadoorian: Mm.
Paul Asadoorian: Yeah—you want to leave that to us, cybersecurity companies, right? We’ll do the honeypot stuff. I mean, most cybersecurity companies today—if they’re doing any kind of research—either partner or have some of their own honeypots to observe attacker behavior. Look, quite frankly, it’s the best way to get threat intelligence, because I don’t have to rely on third-party sources. If I have my own honeypot system, I can observe attacker behavior, report things, and build detections for it. To me, it’s one of the most accurate—actually, it is the most accurate—sources of threat intelligence.
Adrian Sanabria: Exactly.
Vlad Babkin: One fun idea I have as a defender: if you’re defending an SSL VPN appliance, you can make a fake VPN connection into a fake network, deploy 5–10 devices inside it named “CEO laptop” or “CCU laptop” or whatever, and just let the attacker go in there and start trying to interact. Then: “Oh, hey—somebody is interacting with all of that network.”
Paul Asadoorian: Mmm. Yeah. I saw that with ArcaneDoor. The post-exploitation makes so much sense, right? You can read the high-level news and they’ll tell you roughly what the threat actors do. But I love digging down into the details.
What I found with ArcaneDoor is: since the attackers have access to your firewall/VPN device, they’re going to look at all the interfaces on it. Enumerating the interfaces gives them all the subnets it can connect to, and the VLANs it can connect to. Then they initiate scans.
Inside their malware, they have scanners that run based on what they collect from the device. They’re like, “Great—you have this subnet.” Then: “I enumerated all these Windows hosts on that subnet.” And then maybe they sniff credentials, or hook—this is my favorite one—they hook the authentication process in the underlying Linux of the SSL VPN device and write passwords in cleartext.
It doesn’t happen on every platform, but on certain platforms they were able to hook that auth process, write it to a file, exfiltrate that file, and run the next stage of the attack with credentials—plus knowledge of your internal network mapping.
Adrian Sanabria: What.
Adrian Sanabria: You can do it even easier than that. You could skip writing to the local file system entirely and just ship it off—like syslog it across the internet to some syslog server you’re running in real time.
Paul Asadoorian: Mm-hmm.
Paul Asadoorian: Yeah. And when we talk about ArcaneDoor specifically—I can’t remember if it’s Line Dancer or Line Runner—but one of them is a stage. There’s an early-stage payload that drops the second stage payload. The second stage payload is an in-memory resident piece of malware that does collection of passwords and credentials. Then they also infect the bootloader for persistence as an extra measure. The initial implant is in memory.
Adrian Sanabria: One thing we should mention: as much as this sounds really tough, something like 70% of exploited vulnerabilities were exploited as zero-days. So now your patch and vulnerability management is only concerned with 30% of the problem, right?
Paul Asadoorian: That’s crazy.
Paul Asadoorian: Yes.
Adrian Sanabria: Another part of my research is going back in time and saying: “Okay—let’s make a list of all the vulnerabilities that caused damage in previous years.” And it’s an extremely short list—like a couple hundred vulnerabilities that were actually exploited, used in attacks, and resulted in ransomware or other damage.
Then when you look further at those couple hundred, something like over half are in edge devices. The rest are things you can use to get an infostealer installed—something in a browser, web attacks, web-based products where you can get into an email system or similar. Once you look at these, you see patterns and you’re like, “Maybe I should just get rid of really old legacy products that aren’t maintained anymore,” and a big chunk of the problem goes away.
Paul Asadoorian: Yeah—but they’re also exploiting newer stuff too. When I prepped to do a demo—I’ve been doing a lot of attack demos lately—I’ll look at Fortinet and pick a specific product. I’m not picking on Fortinet; we just did a Fortinet demo, so it’s fresh on my mind.
I took FortiOS / FortiGate firewalls and said: “In the past couple years—I don’t want to go anywhere before 2024—what are the vulnerabilities that have actually been exploited?” It’s not necessarily the most critical ones.
I wrote a tool that I hope to release that helps enumerate this data. I find the CVEs, then look at which threat actor campaigns used those CVEs, and paint the picture.
For Fortinet, I came up with six recent ones from 2024 to today. I think four were buffer overflows and two were authentication bypasses—the auth bypasses are very new. And you can tie them back to specific threat actor campaigns. But again: that’s six out of hundreds released for Fortinet products. At least for FortiGate, there were six.
Adrian Sanabria: All—
Vlad Babkin: And also: attackers don’t want to exploit complicated vulnerabilities if they have a plain auth bypass with one URL. It’s a lot easier to build your information gathering from that than from some mega-complicated five-vulnerability chain that gets you onto a Windows machine—where you still have AV to bypass.
We all know they’re bypassable, but that’s extra time. If you can get all of that with one curl command and not worry about antivirus, why wouldn’t you?
Paul Asadoorian: Yep.
Paul Asadoorian: Right. Another reason they target these platforms is we don’t have the same hardening. We deploy Windows systems, we harden them, we put extra software on them like EDR and vulnerability management agents. We don’t have that for these network edge devices.
Vlad Babkin: Too.
Paul Asadoorian: Adrian, you got some data on Fortinet.
Adrian Sanabria: This is an old copy of CISA KEV—eight months old or so—but it’s an overview of everything in CISA KEV for Fortinet as of then.
Paul Asadoorian: Yeah. It’s interesting how older vulnerabilities end up on there. One of my theories about why attackers hang onto older exploits and still use them: they’re valid because getting the latest firmware/software updates for many network edge devices requires a support contract. You have to pay the vendor.
A lot of organizations—especially smaller ones, or budget-constrained ones—won’t update firmware because they don’t have a support contract, so they can’t get the latest firmware. And it’s tempting because you can go to eBay, buy these devices cheap, and they work great. Even older ones have capability and capacity. But you need that support contract.
People get stuck in procurement, or they deploy it to smaller sites without support, so they can’t upgrade firmware—which is crazy. And that’s how we end up with these exposed to the internet. That’s most of the use cases out there.
Adrian Sanabria: That makes sense. People are told “do more with less.” IT directors are told they’re not getting more budget—they have to make it happen.
Paul Asadoorian: Yep. You need the latest firmware. If you can’t do that, you might consider alternatives that use open source. You don’t want to run older vulnerable firmware on these devices—because if you’re exposing it to the internet, you’re probably already owned, quite frankly.
Adrian Sanabria: And don’t put the management interface on the public internet.
Paul Asadoorian: Yeah—the problem is the SSL VPN service itself often has to be exposed to the internet. Which is why I love the Tailscale model: it’s WireGuard under the covers, and you don’t have to expose a port to the internet. Connections are initiated from inside the network to what they call a relay server instance in the cloud.
Adrian Sanabria: Yeah—that is how it works.
Paul Asadoorian: Tailscale has been making it available so you can run those on your own now.
Adrian Sanabria: And you can run that yourself. You don’t have to depend on their infrastructure.
Vlad Babkin: This is a problem with relay servers run by Tailscale themselves—they’re run by a third party. Now that you can deploy one yourselves, that’s great. Latency can still be a factor, depending on what two points you’re trying to connect, so you have to be careful where you place the relay server, but…
Paul Asadoorian: Mmm.
Adrian Sanabria: I deployed this on my network. I used it from airplane Wi-Fi while on an airplane, and I was able to use RustDesk to get the GUI of some of my systems at home. It felt like I was sitting right in front of it. It was great.
Paul Asadoorian: How do you like RustDesk? Do you use the open source version, or do you pay them?
Adrian Sanabria: Open source. I did a bunch of research—think it might’ve been NetworkChuck who recommended it out of a bunch of options. I love it. On my iPad, on an airplane.
Paul Asadoorian: And you run the RustDesk server locally, right? You don’t have to run it in their cloud.
Adrian Sanabria: Exactly. I’m using Tailscale to get to my network, and then RustDesk client/server inside it.
Paul Asadoorian: Nice—your own Tailscale infrastructure and your own RustDesk infrastructure. I’m lazy: I spun up the free Tailscale cloud one a while ago and haven’t done my own yet, but it sounds like I need to.
Tailscale is pretty amazing. On the personal version, you can do subnet routing—one device on your local network advertises routes, and when you connect via Tailscale you can access that subnet as if you were on it.
And you can configure an exit node—one of your nodes shares its internet access, so when you enable the exit node, all your internet traffic goes over Tailscale and then out through the exit node. You can use it like a more traditional VPN.
I had to get around restrictions once at a soccer event—the local place was filtering cybersecurity and hacking sites. I needed those sites, so I used Tailscale and it worked great. And that’s on the free tier. I can’t believe they still give that away for free.
So beyond updating firmware, consider these technologies. There have to be other alternatives too. Adrian—you’re close to the vendor space—any others?
Adrian Sanabria: ZeroTier is another good one. Those are the two main ones I’ve used. I can’t give you a full list beyond those two, but I do have one somewhere. Most of these are essentially WireGuard under the hood.
And I threw a Tailscale blog link in the chat with diagrams that explain how it works.
Paul Asadoorian: Yeah—you have to do a little reading to get up to speed, but once deployed it runs really well. I haven’t done a local deployment, but it sounds like you have, which is cool.
Adrian Sanabria: Yep—building up the home lab.
Paul Asadoorian: Building labs could be a whole episode. I’m working on one too—mixing virtual and physical systems in the same lab, while protecting it. If you want to run intentionally vulnerable stuff, you have to segment it from the rest of your network—but still have access to it. And you have to treat it carefully; you don’t want to expose it to the internet.
I predict a shift—especially after reading the Ivanti article—where the security posture of these devices drives more purchasing decisions. And I get it: it’s not easy to switch platforms. If you have a large deployment, it’s a lot of work because you have to touch every user. We’ve all worked in IT—that’s hard.
Adrian Sanabria: Ivanti is not doing great. By February 2025 they’d lost a third of their customers, but they still have two-thirds—which is amazing to me. We’ve got the US government and the EU telling people to stop using these products, and they went from 50,000 customers to 34,000 customers. They don’t have a lot of cash reserves.
Paul Asadoorian: Right.
Adrian Sanabria: They’re taking on more debt.
Paul Asadoorian: They have something like $2.5 billion in debt on the books… is it over $3 billion at this point? And I forget their projected annual revenue—not enough to cover the debt. It’s crazy.
Adrian Sanabria: Yeah—over $3 billion. I think it’s about $3.1 billion total debt. Not enough.
Paul Asadoorian: You mentioned the EU. One thing I like about the EU is they adopt open source. I’ve seen articles about governments—France, Denmark—moving from commercial products toward open source. I appreciate that.
It’s a double-edged sword: you might have more maintenance and more work to implement and keep running. But if it’s open source, you have access to that layer to do monitoring. If you’re running pfSense, OPNsense, or other alternatives, you can do that.
I think the culture in the EU is more attuned to open source than the US—they always have been.
Adrian Sanabria: You need people who know what they’re doing.
Paul Asadoorian: Yeah.
Adrian Sanabria: A lot of it comes from Sweden and Finland. Linux came out of Finland—it jives with the mindset there.
Paul Asadoorian: Yes.
Adrian Sanabria: Everything isn’t profit-driven.
Paul Asadoorian: One other thing: I saw a Trail of Bits article about a tool that allows memory forensics without published symbol tables—which can be frustrating.
The article says it analyzes Linux memory dumps without requiring external debug information. That’s pretty awesome. I sent it to our product research team. If we can get a memory dump from a system, we don’t have to worry about which kernel or symbol tables it uses.
Adrian Sanabria: That was really cool.
Vlad Babkin: This is a game changer for edge devices. Sometimes you only have a compiled kernel—no debug symbols, no info about what it was compiled from. If this approach works beyond x86—like random ARM32 architectures—that’s even more amazing. Suddenly these devices become analyzable.
Adrian Sanabria: And not just that—it provides an interface for analyzing the dump. They said they were inspired by osquery, but they give you an interactive SQL interface for exploring the dump.
Paul Asadoorian: Right—to query it. That’s very cool.
There are a lot of new tools released every week. Trail of Bits does great work; this one really caught my eye.
It’s interesting how AI has shaped tools and content. I’ve noticed a trend: when I see an article on Medium now, 90+% of the time it’s AI-generated and not very useful. Medium used to have great explanations. Over the past couple years it feels like it shifted to a lot of AI slop.
Adrian Sanabria: Really?
Adrian Sanabria: Yeah—companies are pivoting because of that. A lot of people stopped using Pinterest because it got flooded with AI images. Now companies are trying to build filters to detect and filter out AI stuff.
Paul Asadoorian: Yep. Users police it too. I follow a lot of Reddit threads. I’ll see a headline in my RSS reader and think, “That sounds like a really cool tool.” Then I go to the Reddit thread and the moderator already removed it, and people commented that it was AI slop. Great—you saved me time.
But some AI-coded projects are really good. We covered a stat that 4% of commits are Claude in GitHub right now, and it’s expected to rise to 26%. Claude gets better every day at writing code and doing projects. If you prompt it correctly, you can get a really good project out of it.
If you go to my GitHub—github.com/pasadoorian—I put up a repository called linux_hacks. It’s like the traditional technical segments we’ve done on podcasts for years: problems I need to solve, scripts I write, and then I’ll have Claude go through them, clean them up, add error checking, and so on.
One of them was a little AI coding thing. If you follow Eclypsium, you may remember I produced supply chain cheat sheets for Windows, Android, and Linux—commands and utilities you can use to query a device/OS for hardware and configuration. For example: do you have a TPM, what version, what chip, and so on.
I asked Claude: “Based on my Linux cheat sheet document, create a shell script that implements everything in that cheat sheet.” It did—and it works really well. If you want a taste of our philosophy—understanding hardware, firmware, and configuration—that script is a great starting point. It’s free and open source on my GitHub.
Adrian Sanabria: Thanks.
Paul Asadoorian: I was impressed. It gives you a lot of great information. You still have to do the manual work—like if it tells you you’re running an old BIOS, you still have to go get the latest update and apply it. But it also tells you about microcode vulnerabilities that might exist in your CPU. There are a couple ways to query those, and Linux can update microcode—or it might come from a BIOS update.
It’s a cool script and a good way to flesh out issues we work on at Eclypsium.
That’ll bring us to time. Adrian, thanks for appearing on the show this week. Vlad, thank you as always. Thanks everyone for listening and watching this edition of Below the Surface. We’ll see you next time. Over and out.
Adrian Sanabria: Bye.
