BTS #65 - Exploring AI in Firmware Analysis
In this episode, special guest Matt Brown joins us to discuss the integration of AI in firmware analysis, exploring its benefits and challenges. We delve into the transition from traditional methods to AI-driven approaches, emphasizing the importance of prompt specificity for effective vulnerability discovery. The conversation also covers the role of open-source components, the need for guardrails in AI use, and the implications of AI-generated reports in cybersecurity. Additionally, they touch on man-in-the-middle techniques and the future of AI in firmware development, highlighting the creative monetization of vulnerabilities in IoT devices.
Subscribe
Transcript
Paul Asadoorian(00:39.196)
This week, Matt Brown from Brown Find Security joins us to talk about firmware analysis using AI. Stay tuned below the surface, coming up next.
Paul Asadoorian(00:51.474)
Welcome to Below the Surface. This is episode number 65 being recorded on December 11th, 2025. I’m your host, Paul Asadorian. Joined by my coworker, Mr. Vlad Babkin. Vlad, welcome.
Paul Asadoorian(01:06.694)
Mr. Matt Brown has rejoined us. Matt’s no stranger to the show. Of course has an amazing YouTube video channel on firmware analysis and reverse engineering, which if you haven’t checked it out, you should definitely go check it out. Matt, welcome back to Below the Surface.
Matt Brown(01:22.198)
Thanks for having me.
Paul Asadoorian(01:24.008)
Yeah, man, it’s good to have you on the show. Just a quick announcement before we dig into the topic for today. Below surface listeners can learn more about Eclypsium by visiting eclipseum.com forward slash go. There you’ll find the ultimate guide to supply chain security and on demand webinar I presented called Unraveling Digital Supply Chain Threats and Risk, a paper on the relationship between ransomware and the supply chain and a customer case study with Digital Ocean. If you’re interested in seeing our product in action, you can sign up for a demo.
all that at Eclypsium.com forward slash go. I believe collectively the three of us have been, and probably a lot more people that have been doing especially any kind of reverse engineering for some time, are expanding our usage of AI tools. I’ve tested quite a large number of them. think I usually come back to Claude Code. Matt, I noticed you had some videos that you were using Claude Code in your videos in some interesting ways.
What I guess what first kind of led to the initial discovery and kind of switch from our more traditional tools in adding on these AI tools.
Matt Brown(02:32.49)
Yeah, that’s a great question. So I gotta admit, I was a bit of an LLM AI skeptic at first. so maybe I wasn’t the first adopter of these tools, more just started off with that, I’ll stick to my manual analysis approach to reverse engineering, firmware analysis, things like that. But as I started to see people posting use cases out there, and especially
Paul Asadoorian(02:39.786)
Mm-hmm.
Matt Brown(03:02.222)
LLM platforms with tool calling, Clawed code, cursor, that can interact and fully understand the shell of my computer, just as I would be using all of my open source tools on the command line to take apart firmware, to reverse engineer binaries. I started to see some use cases form and yeah, just been exploring ever since.
Paul Asadoorian(03:32.021)
You know, it’s interesting, I used EMBA really heavily. And what I found is it’s kind of like the opposite approach. I let EMBA run on the firmware and it just collects all the information, right? And of course you’ve got some tuning that you can do with EMBA, but you just get this huge report and then I would go through the report and I would go look for specific things. And what I find that I do using something like Claude Code is I typically unpack the firmware first.
I have a script that runs on blob, which handles 90 plus percent of the firmware out there, right? So basically to save tokens, I unpack it, plop it in a directory, run Claude code, and then I ask it specific questions of what I want, right? So I’m not starting off with this massive amount of data and trying to sift through it. I’m creating the data that I want to see and build and building on it. And I think that’s the thing that really impressed me the most about this approach.
Matt Brown(04:28.233)
Yeah, and that’s been, that’s been my experience as well. I’m not trying to completely replace myself in the analysis process. I’m oftentimes walking the LLM on target, as we were kinda talking about before the show. Sometimes you gotta hold its hand and be like, no, hey, I want you to go do this thing. And you have to be insistent. providing,
Paul Asadoorian(04:36.532)
Mm-hmm. Right.
Matt Brown(04:55.751)
the context, right? So you’re like saying, okay, I’m going to run unblog, binwalk, I’m going to, I’m going to, or maybe some manual unpacking of the, of the firmware ahead of time. And then you’re going to hand it all of these different resources and maybe some custom tools to help, help you to learn about how the system is acting and maybe do some, vuln, vuln discovery.
Paul Asadoorian(05:09.054)
Mm-hmm.
Paul Asadoorian(05:20.328)
Yeah, what I found too, and like I almost find this is like cheating, right? But if the firmware has any open source components that are on GitHub, I tend to pull those down. Like anything related to the firmware or project that is open source, I pull that down and I stick it in that directory too. And I’m like, my God, the results are just so much better. Because if any, you know, rather than decompiling, like here’s the source code that they’re using.
It can just do such amazing things with that. I I feel like it’s cheating, but there’s no real cheating in hacking, right? Hacking and cheating have this relationship, right? It’s whatever gets us there faster. I also find that, you know, folks listening, and Matt, you just kind of alluded to this, is I’m not just taking the firmware and just prompting it saying, hey, go find all the vulnerabilities for me. Like, that doesn’t, it doesn’t work. You can try it, but I’m telling you right now, it doesn’t work hardly at all when you do it that way.
Matt Brown(05:55.833)
yeah.
Vlad Babkin(06:11.534)
Mm-hmm.
Paul Asadoorian(06:18.799)
You have to be very specific, as you were saying, man.
Matt Brown(06:22.39)
yeah, yeah, definitely. so, often times, I have, I think all hackers, we get the Spidey sense, right? We’re like, okay, I think there’s this class of vulnerability that’s on a system or in a firmware file or something like that. And so, you’re giving it that very targeted…
Paul Asadoorian(06:30.847)
Mm-hmm.
Paul Asadoorian(06:37.044)
Yes.
Matt Brown(06:44.86)
inclination of, go try to find this vulnerability. And I’ve had a lot more success when you give it that direction.
Paul Asadoorian(06:50.442)
Yeah, me too. Even specific, like you take a class of vulnerabilities or vulnerability class, like authentication bypass, and then what I’ll do too is I’ll be even be more specific. And I’ll be like, okay, what are the various authentication bypass techniques? And I’ll say, go test for just this one first, right? And so even being more specific than the main vulnerability class, specify a technique that you want it to test and it does way better.
way better. What about you Vlad? Are you using it in a similar kind of way or you have slightly different techniques?
Vlad Babkin(07:25.996)
Yeah. So I have slightly different in general approach to this. So I like to have the firmware actually running somewhere, so not just statically analyzing it. And in this case, I like to actually go ahead and see what’s up with the running device or a virtualized version of it. So this way, hey, so there is this huge binary. You can start to analyze it statically and take a few days. Or if you manage to get it running, you can understand what it does in a few minutes.
because, hey, this is a web server, and hey, it has all of these queries, I guess. So while it’s maybe not as complete, it’s much faster. And also, you can get the AI to actually, like, as I said, spidey sense for vulnerability classes, right? So again, depending on what you just found. And also, I’m kinda using it as a developer as well. So what I found it really good at is when you give it a very specific task, again.
Paul Asadoorian(07:55.967)
Mm-hmm.
Paul Asadoorian(08:18.11)
Mm-hmm.
Vlad Babkin(08:23.874)
you cannot ask it to, hey, please refactor my code. It will. You will not like the results. But if you ask it for something more specific, like, hey, here is a well-defined interface for you now called the function, that works much better.
Paul Asadoorian(08:28.26)
Mm, yeah. Yup. Agreed, yeah.
Paul Asadoorian(08:41.544)
Yeah, I’ve also found that using the keywords create a plan with phases and milestones that I can review has been super helpful, both in vulnerability research and using it to just write code for me. tend to say, especially, you know, even a smaller feature, I’ll be like, just don’t go make these modifications yet, but show me a plan.
Give me multiple phases and milestones and explain what you’re going to do to do this thing. then that’s the rest of the prompt is what I want it to do. And then I can kind of cherry pick because you can see it get off the rails before it goes off the rails. And you can be like, no, know what? Do phase one milestone one, two, and three, but not four, but do five. Right. And I’ve actually told it to do stuff like that. like, no, no, skip that step. You’re going way too crazy there.
You guys found that too, that the LLMs in general try to be too helpful almost to their detriment. You have to kind of scale it back. I’m not the only one who experiences this, right?
Vlad Babkin(09:46.158)
Yup. Yup.
Yep, also…
Matt Brown(09:50.421)
yeah, yeah, it does fill in the blanks when you don’t give it like only like do this thing and then I like using negative prompts of like don’t go past this point until I tell you that you kind of give it like a don’t don’t do this not just like a do this because then it will interpret it very broadly
Paul Asadoorian(09:54.1)
Right.
Paul Asadoorian(10:02.122)
Mm-hmm.
Right, right.
Paul Asadoorian(10:10.484)
Correct. And it’ll also, it does the opposite thing too. Like I was just explaining before we started recording, it’ll leave stuff out. Cause again, you’re talking to a computer, right? Like you haven’t explicitly told it, like it’ll build a function and not call the function. I’m like, did you, did you actually use that? no, I defined it. I’m like, you did a great job defining it. Don’t get me wrong. Like, but you have to call it so that we can use the data that results from the function. So I found it does that as well. You have to watch for that.
Vlad Babkin(10:20.131)
Yup.
Vlad Babkin(10:40.398)
Yep, also it likes to refactor the entire class instead of like, hey, changes this one function. Yes, I will only do just that, goes ahead and changes like five files along with this function you actually to change. So, it allows us to do that.
Paul Asadoorian(10:52.242)
Right. And it’s not great at regression testing after it does that. And I’ve often found that it’ll remove functionality or introduce or reintroduce a bug while it’s recoding that, you know, that entire method or function.
Vlad Babkin(11:06.35)
Yep. Also, what I found it really helpful for is like two specific tasks. Specific task one, generate documentation for a function. Like especially great with one function is semi-readable like let’s imagine Python, which exists in firmware, right? So, hey, document this and it does great on that. So almost no extra configuration needed, you get something semi-decent. So I used it on the code and not just in firmware, but in like our own code base.
Paul Asadoorian(11:14.91)
Mm-hmm.
Vlad Babkin(11:35.308)
And that works great. Like 99 % of the time it produces documentation which you’re happy with. And also what I found great for it is unit tests. It sometimes goes ahead and generates way too many detailed unit tests, so you might want to remove some of them. But the whole point, if you ask it to make unit tests and then remove a few, that’s still much faster than writing the manual.
Paul Asadoorian(11:35.369)
Mm-hmm.
Paul Asadoorian(11:50.922)
Right.
Paul Asadoorian(11:58.772)
Yeah, and you gotta make sure it actually does the unit test and isn’t just passing it. Because sometimes I feel like LLMs were created to just appease us humans. And I’m like, no, it’s okay if it fails. Don’t just make it pass just for the sake of passing. I find it does that if you’re not paying attention.
Vlad Babkin(12:04.376)
Yup.
Vlad Babkin(12:13.506)
Yep, yep, and this is where also all of the problems come with it. Like, hey, it just casually drops production database after being specifically instructed not to. Well, that’s because it’s trying to be way too helpful.
Paul Asadoorian(12:28.041)
Right. I also find that I don’t have to dig, if I’m analyzing source code with it, I don’t have to dig through the code myself. Like I know there’s code in there that does this specific piece of functionality. And yeah, could I go dig and find it? But that would take me a long time. If I prompt Claude, I’m like, hey, go find me in the code that handles this. Right. And then you can kind of follow onto that. Well, show me once that function returns a value, where does that go and how is it processed? So you can just guide Claude to give you just the code snippets that
Vlad Babkin(12:40.108)
Mm-hmm. Yep.
Paul Asadoorian(12:57.813)
that you want to see, which is awesome.
Vlad Babkin(13:00.677)
So there is one comment to that. If you have actual code where you can do this, you can probably use grep and get results much faster. But again, for grep, you actually need some keywords. So if you don’t exactly know what keywords you’re looking for, you probably should ask Claude. But what we have with LLMs is that it often omits results. So even if you get results from Claude, my recommendation would be to actually do grep afterwards to make sure that the…
Paul Asadoorian(13:09.461)
Hmm.
Paul Asadoorian(13:26.419)
Yeah, to verify.
Vlad Babkin(13:28.31)
It didn’t miss like three other functions which do the same thing.
Paul Asadoorian(13:32.297)
Right. Matt, you had an interesting use case where you had Claude interact with a TTY serial console for a device. Tell us about that. That was an interesting exercise.
Matt Brown(13:47.027)
Yeah, the Claude skills are kind of a key part of my automation stack. so Claude skills, mean, the TLDR is it’s just a markdown file with a, you know, kind of a header section where it will read the header, the description of your tool, your skill, and then it will decide.
Paul Asadoorian(13:54.603)
Mm-hmm.
Matt Brown(14:09.951)
to use that and then it doesn’t read the whole description into the context window so it saves you some tokens. So it only reads the rest of your tool description and how to use it if it needs to.
Paul Asadoorian(14:15.422)
Mm-hmm.
Paul Asadoorian(14:21.416)
Is a skill like a GPT or space and perplexity?
Matt Brown(14:27.293)
You know, I’m not as familiar with those tool sets, so I’m not sure.
Paul Asadoorian(14:31.154)
Yeah, so like like Proplexity, for example, like I trained it to write a blog post as me. And all I do is I create a space and then you customize the prompt. Like every time I prompt you in this space, you also prompt it this way. So I’m like, basically like go look at Eclypsium.com to pull content and writing style. And then you can upload files to it. And it’s kind like a poor man’s way. Like you’re not training the model. You’re just giving it context. And then in the files thing, I’m like, here’s
Matt Brown(14:37.269)
Mm.
Matt Brown(14:41.301)
Hmm.
Paul Asadoorian(14:58.568)
you know, PDFs of blog posts that I wrote use that also for the writing style. And it does a pretty good job of writing stuff like as Paul for Eclypsium, right? I mean, it’s by no means perfect. Like, could I just prompted to write an entire blog post? Not really. I mean, sometimes it does better than others and saves me more time, you know, than other times, but it works well. But in Claude, I haven’t really explored the…
What did you call those in Claude? Skills? Skills, yeah. And so what is it, just an XML file that you’re providing context to or?
Matt Brown(15:28.871)
skills.
Matt Brown(15:34.262)
It’s a markdown file with a… I forget. It’s not a YAML, but there’s some kind of front matter. Front matter? There’s been a front matter at the top of the markdown file. It loads into the context window when you start Claude code. It will read all of your skills. You put them in .Claude slash skills.
Paul Asadoorian(15:51.498)
Mm-hmm.
Matt Brown(15:59.752)
Slack and then and then under there you have all your tools and so it will go into every one of those files and it will just read the front matter the description of the tools and then if it decides to use the tool so I have my Pico com tool so it’s just the terminal emulator that’s and then in the further description of the tool it tells it how to you know how to set the baud rate how to interact the default you know path of the tty to use and
Paul Asadoorian(15:59.777)
Mm-hmm.
Paul Asadoorian(16:12.885)
Mm-hmm.
Paul Asadoorian(16:22.827)
Mm-hmm.
Matt Brown(16:29.417)
to use a different one if you’re specified, if you’re prompted to. And so what this allows me to do, yeah.
Paul Asadoorian(16:31.155)
Mm-hmm.
Paul Asadoorian(16:34.684)
yeah, and I can see that that saves a lot of tokens too, because it’s not trying to go figure that stuff out burning tokens.
Matt Brown(16:40.565)
Exactly. Yeah. Yeah. That’s the whole idea. And so you can obviously build this. So in this, in this case, PicoComp’s a custom tool or sorry, an open source tool, but you can also build skills for custom tools, which is a really cool kind of development cycle that I have gone through recently where sometimes I will manually find a Vaughan without any LLM help. And then I will, you know, create a, you know, cloud code session. I will basically walk the LLM on target to be like, Hey, look,
Paul Asadoorian(16:56.267)
Right.
Paul Asadoorian(17:09.503)
Yep, this is the vulnerability you’re looking for,
Matt Brown(17:10.111)
there’s this vulnerability. And then I’ll be like, hey, go write a tool that would find that vulnerability or that class of vulnerabilities. And now what I do is I go one step further and I say, OK, now write a Claude skill to integrate that into my stack. so that process is helpful. Yep.
Paul Asadoorian(17:16.618)
Right.
Paul Asadoorian(17:29.215)
Yeah, because if you don’t, it forgets it, right? I’ve noticed that, like, I had it create a shell script that just reloads my Docker application, right? It was Python Flask with Docker, you know, basic setup. And it was running all these wacky commands, like, restore my app. I don’t like a little timeout. Like, write me a script that does this. And I added, extra features, like, give me a log parameter so I can just run, like, this script logs, and it just tails the logs automatically from the running container.
And then I’m like, okay, use that from now on. But then like the next session, it forgot that I told that. But the skill is the way to help it remember like how you want it to behave during that session basically, right?
Matt Brown(18:10.479)
Exactly, exactly. And I think the cool thing is that in those sessions, if it ever trips up, then I’m always saying, okay, hey, you messed, something got messed up along here, go rewrite, go amend the skill. And then that way the next time, it’s always going to, again, it’s always going to go into that dot-clawed skills, that path, and it’s going to pull that new definition in when it chooses to execute a skill.
Paul Asadoorian(18:23.966)
This scale, yeah.
Paul Asadoorian(18:32.146)
Mm-hmm.
Paul Asadoorian(18:40.35)
Right. That’s awesome.
Matt Brown(18:40.531)
Yeah, and so this specific skill again was interacting with my UART tooling. So that way I can give Claude either a live shell on a device, so sometimes I’ll get the TTY logged into like a shell on a Linux device, and then I will say, hey, you can…
access this TTY interface and you can have a shell on this device to poke around with. There’s been other situations where you don’t have a shell but maybe you have standard error and standard out of the main program that’s running and that’s super helpful too if it’s writing that out to the console. And there’s been situations where I’ve had a binary protocol that I was reverse engineering and it would write its logs to the console so I could have
Paul Asadoorian(19:09.322)
Yeah.
Paul Asadoorian(19:17.578)
Mm-hmm.
Matt Brown(19:33.279)
Clawed send payloads over the network, and it could literally monitor the results, the error messages that were being generated. And so it was able to craft, eventually, a successful payload.
Paul Asadoorian(19:34.698)
Yep. And then monitor. Yep.
Mm-hmm.
Paul Asadoorian(19:48.651)
Yeah, that’s awesome. Yeah. And I love it. We would still explore the command line ourselves, but it’s kind of nice. You get dropped into some of these limited shells or bootloaders or whatever, and you’re like, what is this? And what commands do I have at my disposal? It’s, LLMs are really great at like, hey, I get this command prompt. Go try a bunch of stuff and tell me as much as you can about it. The prompt could probably be just like that, because you just let it go. That’s where…
like letting it explore, letting it somewhat hallucinate actually kind of helps in this type of research,
Paul Asadoorian(20:25.906)
That’s That’s great. So what kind of firmware have you been looking at recently with Claude? I know you had a recent video. I don’t remember what the firmware was. You’ve analyzed so many different types of devices,
Matt Brown(20:37.365)
Yeah, so a lot of the recent Claude stuff has been targeting an IP camera. And so this project of mine to build this hack bot.
Paul Asadoorian(20:45.673)
Mmm.
Matt Brown(20:50.685)
just my Clodcode infrastructure, started around a camera that implemented the on-vif protocol. So in the commercial security world, the IP camera world, there’s a standard called on-vif that defines a bunch of APIs, and they’re SOAP-based, so it’s like gross XML.
Paul Asadoorian(20:57.556)
Mm-hmm.
Paul Asadoorian(21:10.181)
yeah.
Matt Brown(21:11.269)
data going back and forth and I didn’t want my eyes to bleed out reading that spec. And so it was really great is you just say, hey, Claude, go read this spec and create me a tool. And so there’s in the on this specification, there are a number of pre-auth API endpoints, like endpoints that you can access without authentication. And then the rest of them should require authentication, the keyword being should. So I wrote a tool.
Paul Asadoorian(21:17.994)
Right.
Paul Asadoorian(21:22.633)
Yep.
Paul Asadoorian(21:38.015)
Right.
Matt Brown(21:40.969)
that would go and try to access every API endpoint without authentication and kind of match it up with the spec and try to find if there were any out of spec. Yeah. sweet.
Paul Asadoorian(21:47.913)
Right. It’s so funny. I did the same thing. I the same thing for another device. Yeah. And it’s really great if there’s API documentation. And in some cases, the back end code for the API is also open source. So you can get clone that. And then Cloud can create anything you want that interfaces with the API. It’s really great because it has an API that’s very structured. And the documentation is very structured. So LLMs are
I think really great at reading that documentation and then allowing you to implement the API, which if you’ve ever coded against an API, yes, you could know how to program really well in Python, but the specifics of what you’re trying to do are in that API documentation and having an LLM to help you code that is, I’m like, my God, I wish I had this sooner. Because I’ve spent so many hours of my life like reverse engineering an API and writing the Python code for it.
Vlad Babkin(22:38.638)
So…
Vlad Babkin(22:44.748)
Yes, there is also a use case where we tried to integrate with a bunch of devices. what we did was at one point, hey, so there is this request which gives information that we want from the device. And it is like some kind of random request which gets XML as a response. And again, it’s painful to write a parser when you don’t exactly know the structure or anything like that. Or even if you know it, it’s sometimes painful.
Paul Asadoorian(23:09.876)
Right.
Vlad Babkin(23:13.262)
So what I did was, oh, hey, let me drop that as a comment. And like, this request exists. And then, hey, I was using Juni in this case, like JetBrains version of Claude. And hey, Juni, can you please implement me a function which interacts with this and collects this type of data so that it’s present in the response for the device? like, okay. And it just did it. And like, I repeated this like for like, what, 10, 15 times for various devices.
and I had a lot of success with it. In this case, you actually do need to drop stuff like…
Example requests responses if you don’t have full documentation. So the researcher still has to go ahead and poke the device with a stick But overall I had a lot of success with this when you have like not a full API But like let’s say web UI is just making interactions with the device and oh, hey like this request which I would like to extract Well, you can just copy the body for it and like oh, hey, there is this request Can you can you please make a message that sends it out and that saves a ton of time usually?
Paul Asadoorian(23:57.864)
Yeah, yeah.
Paul Asadoorian(24:22.802)
I’ve found that in general it saved me a lot of time. As we’ve said, it doesn’t magically do your research for you, find vulnerabilities and disclose them. mean, OpenAI has kind of experimented with that in open source using very large models. I think to a pretty high degree of success, but the tools that we have at our disposal really save us a lot of time. I feel like that’s the…
Vlad Babkin(24:31.214)
Mm-hmm.
Paul Asadoorian(24:50.218)
I want to go to Cloud Code though, and speaking of saving time, that doesn’t come without pros and cons. And one of the cons is this LLM is running on my system. I probably should be better on letting it run in a VM, but I don’t know if either of you have experimented with giving it some more guardrails, right? Because I’ve heard horror stories, right? Someone in another podcast host was like…
it wiped out all my VMs. I was using Vagrant or something and it created a script and it wiped out all my VMs. And someone else was like, it ran like a kill all command for all the bash sessions on the system, so I lost access. So how do you guys put the rails around, some guard rails around it?
Matt Brown(25:34.037)
That’s great question and so far I haven’t so probably need to look into that.
Paul Asadoorian(25:42.667)
Yeah, the defaults do ask you, like, hey, do you give it permission to run this command? do you give it permission to run the grep command during this entire session? And I’m like, that’s fine. I’m like, but what if you’re piping grep to some other command? Are you still just going to run grep and run other commands after it? Are you going to ask me if you’re going to run the rm command along with grep? Are you going to ask me or not? Or because it started with grep, are you just going run grep? I’m like, yeah, it’s dangerous.
Vlad Babkin(26:12.206)
So there is also what I found specifically in JetBrains. I’m not sure if it exists elsewhere, but you can actually specify reg access for commands, won’t allow it running automatically. So this answers your question exactly. So if you would like to allow it to run grep, you can totally do that.
Paul Asadoorian(26:27.242)
Yeah, I know there’s a rules, right? Yeah. Yeah, there is like a rules file that I haven’t fully explored. I kind of just rely on the it manually prompting me for like, you sure you want to do this kind of thing. And it’s done okay with that. But I do the next step is I want some rules around it. For sure. My next question was, you know, we’ve talked about cloud code. The model that I’m using now is the Opus 2.5 or 4.5? Is it 4.5?
Vlad Babkin(26:46.839)
Yeah.
Matt Brown(26:56.643)
4.5, think, yeah.
Paul Asadoorian(26:56.906)
I it’s 4.5, yeah. They kind of touted that as like, is the next greatest coding model, but have you experimented with other models from other vendors like Gemini or anyone else?
Matt Brown(27:11.284)
A little bit. actually started because like I said, I’m not like an LLM expert. I use this as a tool to help me do the thing that I am good at, which is the firmware and reverse engineering. So I started just using curse, just playing around with LLMs and cursor. And obviously they give you access to a bunch of different models. And so I played around with Grok one code. They’re like fast coding agent, which is really fast, but it just wasn’t as
Paul Asadoorian(27:19.998)
Mm-hmm.
Matt Brown(27:41.844)
I guess powerful as cursor has been.
Paul Asadoorian(27:43.199)
Yeah, I tried the Gemini CLI for coding and it was the same thing. I just didn’t find it was anywhere close to what Claude gives me. Vlad, you said you were using some of the JetBrains models. How have those been working? I mean, I’m a huge fan of JetBrains, by the way. I’ve developed on it before. It’s great.
Vlad Babkin(28:00.789)
So in this case they have at least three different AI things. So first one is inline suggestions. So this is mostly useful when you’re actually coding and it can just suggest you the whole line or the whole little bit of code that you’re trying to finish and it just does it inline without any prompts. So at first it was very noisy but recently I would say that it is doing a pretty good job. Like in many cases it actually suggests well
Paul Asadoorian(28:25.642)
Mm-hmm.
Vlad Babkin(28:27.95)
The only thing to be aware of as a developer is if it tries to suggest a lot of the same code, at one point you need to notice this and, hey, I probably should do a loop here instead of trying to unroll it with AI, right? So it might lead to bad architecture if you don’t watch yourself. Second thing they have is classic AI chat in IDE. And in this case, they do automatically add context with the current file, but you can actually add files to context manually. And it is de facto classic AI chat.
Paul Asadoorian(28:37.418)
You’re right.
Vlad Babkin(28:57.644)
and they don’t really have their own models there, so you got a pick of the models, let me check which ones exist. So yeah, they allow you to communicate with Claude, GPT, Grog, and there are also more models. I believe they have O3, O4, so Grog, Gemini, GPT. They don’t have Claude in… No, they have Claude in there actually. So yeah, so you just pick your model. And they have a default pick for you, which is, in my case, Gemini 3 Pro.
So all of that shenanigans exist with AI chat. And then they have Juni, which is like a cloud code but in JetBrains ID. And I don’t remember what model it uses, but I think that it also uses something like…
not their own model, but probably something open source or commercial as well. And I had a lot of success with it as well. So I would say this is very strong alternative to cloud code. the only issue with it is that it is JetBrains IDE specific stuff. So they do have this AI assistant integrated with other IDs already as well. yeah, JetBrains ecosystem is kinda closed. So even if you install it in another ID, it will not be quite the same.
Paul Asadoorian(29:51.55)
Interesting.
Paul Asadoorian(30:09.53)
Right, right.
Vlad Babkin(30:12.876)
But within JetBrains IDE, these tools are hard to beat. Even if you install stuff like GitHub Copilot, it’s just gonna be slightly worse because of user experience. Like for example, to generate documentation, I’m not even writing prompts. I have a nice button that pops up whenever I just open the documentation comment in Python, for example, and it just has a generate documentation right there. And you can predefine a bunch of actions for yourself like this.
Paul Asadoorian(30:19.251)
Right.
Paul Asadoorian(30:35.561)
Mm.
Vlad Babkin(30:42.72)
In this case, they also have really good user experience in their browser than just a chat. So, yeah, this is the main upside for it. But I would say that Juni and Cloud Code are pretty close to each other in functionality.
Paul Asadoorian(30:49.684)
Mm-hmm.
Paul Asadoorian(30:59.09)
Nice. It’s awesome. You know, it’s interesting, you you spend the first part of the show talking about how we use it for our own coding experiences, our own vulnerability research. I’m curious, like, will it catch up an adoption where the people who are writing this firmware are using it? Let’s see the show going away. Like, the security researchers are using it in this way. Wouldn’t it be nice?
Vlad Babkin(31:01.166)
So hard for me to tell which one is better. Depends on the task.
Paul Asadoorian(31:25.01)
if the manufacturers and vendors used it and flushed that use it to flush those bugs out before they shipped it and i think that that’s not a technology problem i think technology is the least of the concern there
Vlad Babkin(31:30.964)
Vlad Babkin(31:36.782)
there is one important point about it all. By default, AI can actually produce vulnerable code. So the code it produces sometimes does contain vulnerabilities. I have seen that more than once. It’s trained on stuff like GitHub. GitHub is not the place where it contains perfect code, right? So because of this, it has like code with vulnerabilities in its training. And because of it, if it tries to replicate it thoughtlessly, it might produce it. So…
Paul Asadoorian(31:43.274)
Mm.
Paul Asadoorian(31:53.47)
Right.
Vlad Babkin(32:04.854)
Even if you produce code with it, you have to be careful about it dropping vulnerabilities in.
Paul Asadoorian(32:09.694)
Mm-hmm.
Matt Brown(32:11.292)
Yeah, I know there’s definitely a lot of startups out there in kind of the AI powered, you know, software, software code review space that are trying to, I’m sure what they’re doing is they have some kind of a rag setup where they have, you know, good, like, like they have examples of like Vaughn’s like, like good code, bad code, right? Where they’re
Paul Asadoorian(32:35.571)
Yeah, yep.
Matt Brown(32:37.63)
They’re helping to fine tune the model and those models to, again, they’re walking it on target to be like, hey, this is what’s bad, this is what’s good.
Paul Asadoorian(32:49.298)
Right. Yeah, and I think they’re making progress in those models, right? Because you’ve got vulnerable code, you’ve got not vulnerable code, and then when you have a patch, can like train the LLM to go, this is just what the fix looks like, right? So this is the result, this is before the patch, and using just those data points alone, you can create some pretty powerful stuff. Probably what went into, you know, OpenAIs, I forgot what tool they call that, but the one that’s looking at open source projects.
Matt Brown(33:03.7)
Hmm, yeah.
Paul Asadoorian(33:18.686)
finding vulnerabilities, creating a patch, and then issuing the pull request, which is pretty awesome, right?
Matt Brown(33:26.866)
Yeah, that’s fascinating. It’ll be interesting to see in the next 10 to 20 years if certain bug classes start to kind of go away. Obviously, there’s certain bug classes that will be harder, like business logic vulnerabilities. I don’t see those getting fixed. Like wholesale, we’re not going to solve that problem anytime soon, right?
Paul Asadoorian(33:33.63)
Just go away.
Paul Asadoorian(33:51.018)
Yeah, yeah, because logic stuff, it’s not it’s still a computer, right? And it’s still a bunch of conditional statements. It doesn’t have consciousness. So oftentimes, like, I’ll look at something and I’ll be like, like, you were really, like, technically focused on this one thing. But if you just step back and look at it, like the vulnerabilities right there, and the LMS aren’t going to see that a human is going to see that. I’m like, oh, I’m like, wait, do it this way.
And it’s like, oh yeah, I can do it that way. It doesn’t get excited like I’m excited. I’m like, no, we just found a vulnerability. This is great. And it’s just like, it doesn’t see that. It doesn’t get logic at all, because it doesn’t have consciousness. It’s just basically a bunch of conditional statements. don’t to sugar coat that too much, because it is helpful and does more than just conditional statements. it doesn’t understand the code that it’s looking at or writing.
Matt Brown(34:44.314)
Absolutely. Right.
Paul Asadoorian(34:45.064)
making decisions on what it was trained on.
Vlad Babkin(34:46.83)
Yep. I like to think about it like as a statistical parrot, right? So it’s parroting whatever it was trained on, literally. And in this case, it’s just really, really good at parroting. But like a parrot will repeat whatever you tell it. It might actually construct something on its own, but it’s not gonna have a deep understanding on whatever you are trying to get it to parrot, right? So LLMs are kind of the same exact way.
Paul Asadoorian(34:52.628)
Mm-hmm.
Vlad Babkin(35:15.81)
So if you ask it to find something and don’t indicate that you want to also cover the case where this isn’t found, well, it will try to find something everywhere. And to be successful, it find it everywhere, even if it is not there. This is something that we observe constantly. Or, for example, it might actually try to please the user way too much, and because of it, will just, again, hallucinate something that’s not even in the source code in the first place and tells that it exists there.
This is how we get the problem that people are writing about. hey, so we got a lot of submissions of false positive vulnerability finds. Like how you get there? Well, whomever is doing the bounty really wants to find the vulnerability and get the bounty. So AI just goes and finds vulnerabilities where there aren’t any. So there is also the other end of the spectrum for the problem.
Paul Asadoorian(35:56.852)
Right.
Paul Asadoorian(36:08.744)
Hmm.
Paul Asadoorian(36:14.698)
at and it is used to find web application vulnerabilities specifically
Matt Brown(36:20.18)
I’m less focused in the web app space. I know that tools like Kaido are trying to do a lot of like LLM integration with their web scanning, tooling, even manual augmentation, like augmenting manual testing with AI. So they’ve been doing some stuff, but that’s less of my skill set. So I haven’t like got like a…
Paul Asadoorian(36:32.234)
Mm-hmm.
Paul Asadoorian(36:46.282)
Yeah.
Matt Brown(36:48.397)
golden use case bug of like I just let AI run on the web and find this thing.
Paul Asadoorian(36:54.398)
Yeah, I’ve used it too, like when you unpack the firmware, the web application code will be in there typically. And so I like telling it, hey, go find the web, any web application code in here. And I’m like, then tell me all about the architecture. Like tell me what language, what framework, what versions of libraries. if it’s NPM, does it have a manifest? Let’s see what packages it has, what libraries it’s implementing. even just that is a great starting point.
before you run burp suite and actually manually go find the vulnerabilities, then you can kind of tell it to document the web application. You can tell it to find some vulnerabilities. And I find it’s a great starting point. Now when I fire up, I’m going to do manual analysis. I’ve got the areas that I really want to focus on. And it saved me a whole bunch of effort in grepping and discovery to figure out what’s going on. And just a few prompts, I can get a feel for what the web application is and areas that I might want to focus on. And that’s a huge time saver for me.
Vlad Babkin(37:53.55)
So I didn’t specifically try to use it to detect vulnerabilities, but it’s helpful when you try to understand what you’re looking at, especially if you have something really big you’re looking at. Again, I’m using a lot more of traditional tooling in this case. Like for example, okay, I just found an old JS app. How do I find its packages? Well, there is package.json. So I don’t really need an LLM to get me a list of packages if I know the exact files that contains it. So…
Paul Asadoorian(38:00.937)
Mm.
Vlad Babkin(38:21.004)
I would say that relying on LLM in this case might actually be detrimental because it might hallucinate extra packages. So what we tried to use LLM for is parsing out advisories for vulnerable versions and details on that. And what LLMs do, at least I had lot of pain with it, is when I ask it to get me versions out of it, it will just produce either extra version numbers or version numbers with components missing.
Paul Asadoorian(38:34.493)
Mm-hmm.
Paul Asadoorian(38:49.929)
Mm-hmm.
Vlad Babkin(38:50.346)
or like some letters missing, for example. And then it becomes very critical very fast. So asking it to extract a specific piece of package.json might actually not extract everything. So if you want a list of libraries, I would rely a lot more on like more static tooling. Like what LLM is good for is, hey, does it even have package.json or is it even based on Node.js in the first place? Right? And it’s also…
Paul Asadoorian(39:04.488)
Right.
Paul Asadoorian(39:09.107)
Yeah.
Paul Asadoorian(39:16.453)
Right, right. Yeah, because then I would probably use a tool like gripe to go in and enumerate all the vulnerabilities in the React app.
Vlad Babkin(39:22.242)
Yup, yup, yup, yup, yup.
But at the same time it’s also really good at, hey, so you have a compressed blob of front-end source code. Well, you can first of all, prettify it using just the standard public tooling. And once it’s prettier, you can actually use LLM to try to find interesting functions in it. Because LLM is, again, as I have said, really good at documenting code. So instead of trying to read through whatever, you just prettify it. You can ask an LLM to try to produce reasonable documentation for it.
Paul Asadoorian(39:43.965)
Mm-hmm.
Vlad Babkin(39:55.982)
And suddenly you have lot easier time navigating the code.
Paul Asadoorian(40:00.616)
And it still hallucinates though. it’s still, have one app that, Claude is convinced no matter what I tell him, there’s an authentication bypass. And I’m like, dude, that’s not an authentication, stop saying that. It’s not, it’s not. I validate it is not. And it just believes what it wants to believe, which is sometimes frustrating.
Matt Brown(40:20.275)
Oh yeah. Yeah, that’s an interesting thing that I’ve noticed that I have yet to get it to do well. It’s less important because obviously this is easy to do manually, but like when it does find things, the risk assessment is always overhyped. feel like it’s like everything’s Italy, like all caps. I found a critical vulnerability, you know, and you’re like, whoa, calm down. you know, like context, like this isn’t like the end of the world. So.
Paul Asadoorian(40:30.931)
Mm.
Paul Asadoorian(40:35.975)
Yes, yes.
Paul Asadoorian(40:42.163)
Yeah.
Yeah, yeah.
Paul Asadoorian(40:49.755)
Right. Yeah, because you got to take into account the functionality of the device. Like you mentioned in IP camera, it may make a big deal if someone can get a hold of the video stream even after they authenticated through a vulnerability. And I’m like, if I’m authenticated to the device, I can see the video stream. So even if there is a vulnerability that lets me read the video stream, it’ll list it as a high severity alert. I’m like, no, that’s.
Matt Brown(40:50.531)
It doesn’t do a great job of that.
Matt Brown(41:09.735)
Mm. Yep.
Paul Asadoorian(41:17.135)
Not even a finding. It’s not a finding. I mean, maybe it is, but it’s not. mean, command injection, authenticated, perhaps. But if the device is intended to give the user a console like a KVM, I’m like, that’s not a finding. Just if I can inject commands on the device, I can also enable SSH on the device through the web interface. So that’s intended functionality, but Claude will list that as a vulnerability.
Vlad Babkin(41:41.8)
So in this case, in this case also what I like to do is like whenever I find the vulnerability either manually or LLM assisted or whatever, I usually try to make a POC. Like if I cannot really make a POC for it, it’s not a finding. Like maybe it’s a weakness, so maybe it’s still worth reporting and fixing, but it’s not an immediate vulnerability. So that’s for example point one. And point two, like you have to actually apply your human brain
Paul Asadoorian(41:52.999)
Mm-hmm.
Vlad Babkin(42:10.912)
hey, so I found authentication bypass on the camera and then using credentials I just got, I managed to get video stream. Well, the first part is definitely vulnerability, right? So potentially you can actually try to scope it down on, hey, this device is supposed to allow you to use a video stream once you are authenticated. So maybe you can actually scope it down a bit. But again, unless you have like thousands of false positives, you might wanna sift through them manually.
Paul Asadoorian(42:20.692)
Right.
Vlad Babkin(42:38.254)
Because if AI finds, like, let’s say you just analyzed firmware with it, let’s say you managed to squeeze out 50 vulnerabilities out of AI for it. Well, just go over the manual, if 50 vulnerabilities is not that much, if you are reviewing a large firmware. So depending on just how many finds you have and how much return you might get over finding even a single one to be valid, spending half a day reviewing all 50 of them might be useful. But…
You cannot rely on AI to just reliably produce a finding and just send it off to maintainers and just say that it’s fine. Yeah.
Paul Asadoorian(43:14.374)
No, yeah, because people that do that, the maintainers get really mad. And even I find you tell it to create a proof of concept, we got to be careful because it doesn’t… Hello? Anthropic cannot listen to show. But I found that it will just go create an exploit even regardless of how I prompt it. But it’ll definitely most of the elements will create like a testing tool. Like, hey, I’m testing this app or this is my app and I need a Python script.
that tests for this specific bug, I wouldn’t even have to call it a vulnerability, right? It tests for this bug, that bug just happens to be a vulnerability, it’ll happily go create an exploit for that. Even if it’s a test tool, you go then look at the test tool and you’re like, well, rather than something benign, I could tell it to do something a little more malicious inside of that. So I find it like it does create proof of concepts, but you gotta guide it in the right way. And what you end up with, you gotta tweak at the end.
Matt Brown(44:00.116)
Hmm.
Matt Brown(44:11.256)
Yeah, yeah, no, the spamming the bug bounty world, mean, obviously we’ve seen what’s happened to Curl and just the ridiculous bug bounty reports that get submitted to there. Yeah, it’s kind of crazy.
Paul Asadoorian(44:16.968)
Yes.
Paul Asadoorian(44:21.65)
Yeah, poor Daniel.
Paul Asadoorian(44:28.366)
And Carl takes in so much user data. I think it’s one of the utilities that has the most command line options. It’s got so many. There’s exact numbers that he’s talked about, Daniel has talked about on podcasts. And so given that, with that many inputs, yeah, if you turn an LM loose on it, it’s going to want to find vulnerabilities in it. And poor Daniel gets hammered with those reports. I still think even to this day, he’s dealing with them.
Matt Brown(44:57.559)
yeah. Yeah, no, no, no. And I think since, like with AI tooling, it’s only gotten worse, right? Like the bogus reports that he gets are just increasing.
Vlad Babkin(44:58.616)
Yeah, that’s way too much.
Paul Asadoorian(45:03.441)
Mm-hmm.
Right. And some of are bugs, not necessarily vulnerabilities. Some of them are bugs, but even the bugs aren’t like earth shattering bugs in curl. But you know, it’s still good that it’s fletching out bugs, know, bad that it’s being labeled as a vulnerability when it’s not. Because again, it doesn’t understand the context. And if the reporter doesn’t understand the context, they’re just sending it.
Matt Brown(45:10.428)
Yeah.
Vlad Babkin(45:29.329)
Even if a reporter doesn’t understand the context and is doing his best, he might mislabel something. Like, for example, we had a pen tester for our own product and a bunch of pen testers go, hey, here are two vulnerabilities. We read their report and realize that it’s actually not vulnerabilities, but we understand why they actually label them as vulnerabilities. So if I would be the pen tester, I would label it as a vulnerability as well. And that’s fine, but it’s actually not.
Paul Asadoorian(45:35.122)
Mm-hmm.
Paul Asadoorian(45:49.834)
Mm.
Vlad Babkin(45:57.27)
if you actually start to read product details and internal details, it was actually the other way around. So like the bug was like something was exposing way too much access per their perspective, but it’s actually the other way around where UI was not showing all of the access you are supposed to have. So, and Curl might be in the same position many times, right? So because of how many options it has, it’s nearly impossible to track down all of the options which are supposed to work as they are.
Paul Asadoorian(46:06.846)
Mm-hmm.
Paul Asadoorian(46:15.255)
Mm.
Vlad Babkin(46:26.68)
versus what’s actually security vulnerability.
Paul Asadoorian(46:30.769)
Right. I wanted to chat a little bit about the man in the middle scripts that you have, Matt, because I have updates to those. So I actually took your your scripts because you have two repositories on like man in the middle kind of stuff. And it was always like on my plate to like build something that I had a completely like not completely isolated, but a separate network on one side of like a Raspberry Pi. And then on that pie, I have some tools.
to intercept DNS to route all the traffic to and from the internet so you can look at it, manipulate it, use man in the middle proxy or SSL strip against the traffic as well. And so I took what you had and then I asked Claude to create me a new script with all this extra functionality. So I’m working on seeking permission to open source that, but I’ve been using that in our lab environment and I’m like, this is super useful.
Matt Brown(47:18.793)
that’s cool.
Nice.
Paul Asadoorian(47:27.625)
But thank you for producing those tools. I’ve used them. And if you want to our audience how those tools came to be,
Matt Brown(47:30.631)
Yeah.
Matt Brown(47:35.462)
Yeah, well, like I said, it’s not any amazing thing I did. It’s really taping together in a shell script multiple different open source Linux tools that are awesome. so the general philosophy behind the tool is that I want a separate network.
Paul Asadoorian(47:42.109)
Mm-hmm.
Matt Brown(47:55.56)
that I can test devices on. I might want, know, like one device might be testing might use wifi, another one might use ethernet. And so at the heart of my man in the middle router setup is it’s just a shell script. It creates a bridge interface and then attached to that bridge interface is a wireless network using host APD to spin up a wireless network and
Paul Asadoorian(48:04.361)
Mm-hmm.
Paul Asadoorian(48:14.387)
Mm-hmm.
Paul Asadoorian(48:20.339)
Mm-hmm.
Matt Brown(48:22.943)
This lets me tweak a lot of different settings. You’ll see some edge case vulnerabilities out there with some different Wi-Fi settings. I’ve found on a couple of microcontroller SDKs where a device that is configured for like a WPA2 access point, it will happily connect to an open access point of the same SSID.
Paul Asadoorian(48:47.136)
interesting
Matt Brown(48:47.835)
Yeah, yeah. So that’s something I always test and it’s just really easy with the minimal proxy because it’s just there in the script and I can just go in and change the configuration of the network through host EPD. So, but that’s to create the LAN network that the devices will live on, whether Wi-Fi or Ethernet. You can have both set up. If you want like a device that’s on the LAN that’s going to like communicate with your IoT device, you can do that. And then…
Paul Asadoorian(48:54.27)
Mm-hmm.
Paul Asadoorian(49:06.011)
ethernet. Yep.
Paul Asadoorian(49:11.335)
Right. And then if you set up a packet sniffer on the bridge interface, you get all the traffic coming through, which is convenient.
Matt Brown(49:17.677)
Exactly, all the traffic. So you mentioned trying to set it up on the Raspberry Pi, and I used to do that, but then I was all the time, I was running TCB dump on the Raspberry Pi, and I was SCP-ing the PCAPs back so that I could look at them, and I was like, I should just run this on my Linux desktop, and then I can kind of cut out the middle man there.
Paul Asadoorian(49:24.232)
Mm-hmm.
Paul Asadoorian(49:28.711)
Yeah.
Yeah, yeah.
Paul Asadoorian(49:36.069)
Right, right. Yeah, I was going to set up like a Samba share or an SSH, like mount the thing over SSH to that directory so I could just easily pop them over or have some kind of script that just automatically sends me back the PCAPs. I had the same thoughts. It was really funny.
Matt Brown(49:42.899)
Yeah.
Matt Brown(49:53.136)
Right, Yeah. So, and then that all gets routed out through, you know, basically like IP forwarding out to whatever your interface on your computer is that has internet access, right? It’ll follow the default route on your machine. And then optionally, I’ve got a set of IP tables rules that can redirect.
certain traffic flows. So let’s say I want to redirect all outbound traffic to a destination port of 443, all that TLS traffic. Let’s say I want to try to a man in the middle attack on that communications to test whether the device is actually doing certificate validation. So.
Paul Asadoorian(50:40.22)
Isn’t it crazy? did this, so man in the middle proxy or SSL strip is the same. I have both of them, so you can choose which one you want to run. And so IPTable takes all incoming traffic on 443, shoots it to the listening port for either of those tools. And then by default, either of those tools create a certificate that’s not valid, right? I mean, unless you go get a valid certificate, but it’s not valid. And so what it responds back should throw an error to that client device.
Matt Brown(50:46.92)
Yeah.
Matt Brown(51:00.883)
Yeah.
Paul Asadoorian(51:08.09)
And I think what your documentation said, was they don’t always check that the SSL is valid. like, so just setting this up, putting a device behind it, running some traffic through it, helps you determine if there’s a finding there or not. I this is great.
Matt Brown(51:14.886)
yeah.
Matt Brown(51:23.283)
Yep. so that, yeah. then, so redirecting it to those two tools you mentioned, there’s another one I want to mention called cert-mitem. It’s, yeah, S-E-R-T-M-I-T-M. And that’s a tool that will do a very similar thing, but it will run multiple different tests. It will, you know, and you can even put like a let’s encrypt, a valid certificate that’s not the right name.
Vlad Babkin(51:24.053)
No.
Paul Asadoorian(51:31.379)
Mm-hmm.
Paul Asadoorian(51:50.302)
Mm-hmm. Yep. Yes.
Matt Brown(51:51.636)
because there are some weird edge cases where some devices will check if it’s signed by a trusted CA but they won’t check they won’t do the CN the common name match up check
Paul Asadoorian(52:04.328)
Is that certificate pinning? Or does certificate pinning help that not be exploitable?
Matt Brown(52:13.181)
There’s a couple different ways you can do certificate pinning. The way I describe it, certificate pinning is a subset of certificate validation. typical certificate validation is we verify that the LEAF certificate that is being presented to us is signed down a chain to a root that we trust. Certificate pinning is usually
Paul Asadoorian(52:16.906)
Mm-hmm.
Paul Asadoorian(52:38.899)
Right, right.
Matt Brown(52:41.827)
like providing a hash or providing the key and just matching against the leaf saying I don’t care if you have something that’s by a root that I trust I’m only gonna check I’m gonna basically do a hard-coded check against the leaf certificate the server certificate that’s being presented so what cert minimum does is it will run through its various
Paul Asadoorian(52:49.108)
Mm-hmm.
Paul Asadoorian(52:59.431)
Mm-hmm.
Paul Asadoorian(53:04.052)
Gotcha.
Matt Brown(53:08.327)
different kind of test variations. And then if they all fail, if the man in the middle attack fails, it will allow that, you know, server name and port combination to pass. It will allow them to pass by because what you’ll see sometimes on a device is it’ll be making multiple TLS communications. But let’s say there’s a device where it reaches out to server one.
Paul Asadoorian(53:22.697)
Mm-hmm.
Matt Brown(53:36.06)
and then it gets a response from server one and then only after that successful exchange, it will reach out to server two. Well, what if the request to server two is the vulnerable one? If you’re just running SSL strip or cert minimum, you’re never gonna get past to that point. so, or sorry, SSL strip or minimum proxy, but if you use cert minimum, it will allow you to keep going down to further flows.
Paul Asadoorian(53:38.802)
Mm-hmm.
Paul Asadoorian(53:42.195)
Yes.
Paul Asadoorian(53:48.808)
Right.
Paul Asadoorian(53:56.073)
Mm-hmm.
Paul Asadoorian(54:03.762)
that’s interesting. Because what I’ve done is, in my script, I can tell it not to run man in the middle proxy or SSL strip. So I get that initial. Because actually, I had a device do exactly that. It had to get a valid response back from, and it actually did validate the SSL certificate. It had to get that back before it reached out to, like in your case, server two. And server two is where the vulnerability potentially was. Yeah, that’s common.
Matt Brown(54:27.731)
Yeah, yeah, and SSL strip and man in the middle proxy, I believe they have like white listing and black listing options for domains, right? So you could, with those tools, just incrementally black list domains that are not passing, or that your man in middle tech aren’t working on. You could incrementally black list them out and just keep rerunning your testing, but CertMinim just kind of does all that for you, which is nice.
Paul Asadoorian(54:36.362)
Mm-hmm.
Paul Asadoorian(54:43.934)
Right.
Paul Asadoorian(54:51.866)
Mm-hmm. That’s awesome. Yeah. I also built in DNS spoofing into it. So DNS mask is run. And then actually, Claude created a separate scripting configuration file and made it really easy. So you can just add hostname and IP pairs, and it’ll update the local DNS server. And if it is configured to honor the DHCP requests and have the DNS server that you provide it,
Matt Brown(54:58.046)
nice.
Paul Asadoorian(55:18.064)
it’ll go to your DNS server and look it up and then you can send any hostname to any IP address that you want. Just using DNSmask without any extra DNS tooling, but I think the other thing I want to build into it is that extra DNS tooling because the article pointed out recently on the NanoKVMs that it will ignore DHCP, was it option 6 I think is the DNS server?
Matt Brown(55:44.412)
Yeah.
Paul Asadoorian(55:44.619)
It’ll ignore that, has its own hard-coded DNS server, which just happens to be in China. And I’m like, I don’t like that. And also if you want to control the internet, now you’ve got to do DNS interception, which I’m working on some tooling, trying to find something that does that really well. So, which shouldn’t be that hard. mean, we’ve been doing that kind of stuff for a long time.
Matt Brown(55:50.739)
yeah. Fun.
Matt Brown(56:05.127)
That’s really cool.
Matt Brown(56:09.659)
Yeah, that’s fun. and another tool that I like to redirect traffic to is NTP. And spoofing NTP, you’ll get some weird like off bypasses or stuff sometimes. I mean, I can’t name the device, but there was a smart lock where it had the ability of rules you could add. You could like give out an access code that only worked during a certain time or a date.
Paul Asadoorian(56:15.945)
Mmm.
Paul Asadoorian(56:22.14)
Really? Interesting. Yeah.
Paul Asadoorian(56:38.491)
Mm-hmm. I see.
Matt Brown(56:39.219)
and you could, but it was relying on NTP. And so if you could, if you could man in the middle and spoof NTP, you could get it to like shift the top, time it thought it was to allow like an access code to work when it shouldn’t work. Yeah, there’s a cool, there’s a tool out there called DeLorean. It’s like, it’s just a Python script that will, you can tell it to, you know, spoof a specific time. You can tell it just to do like,
Paul Asadoorian(56:49.96)
Yep. I’m assuming people have already people are written tools to do that, I would assume.
Paul Asadoorian(56:59.418)
Mm-hmm. Yeah.
Matt Brown(57:07.025)
just to increment the time back or forth or something like that as an offset.
Paul Asadoorian(57:13.118)
That’s awesome. It’s great stuff. So we’re getting close to the end of time. Matt, was there anything else you wanted to share with our audience that you’re working on? A sneak peek into some of stuff you got planned on your YouTube channel.
Matt Brown(57:27.731)
yeah, so the other thing I’ve been working on recently, I’ve put out a number of videos so far, is this Android set-top box called Superbox. It’s advertised as… Yeah, yeah.
Paul Asadoorian(57:36.699)
yeah, it was the Krebs article, man. you got me excited. I love this. I mean, I hate this, but I also find it fascinating, right?
Matt Brown(57:45.244)
Yeah, so I’ve done a number of videos kind of like tearing down, getting root, looking at some of the web traffic coming out of this box. And the thing I’m working on right now is reverse engineering these API calls. So the core app store that runs this device, not sketchy at all, like, it’s doing its web requests and responses in clear text.
Paul Asadoorian(57:50.824)
Mm-hmm.
Paul Asadoorian(58:03.882)
Mm-hmm.
Matt Brown(58:11.835)
But it’s like doing this custom obscure AES encrypting of the URL parameters, the payload basically of the request and response. So it’s like, so you can’t see inside of that to see what’s happening. I’m doing a video, I’m doing the research and gonna do a video soon where I talk about reversing all that.
Paul Asadoorian(58:12.074)
Mmm.
Paul Asadoorian(58:20.872)
Mm-hmm.
Paul Asadoorian(58:25.994)
Mm.
Paul Asadoorian(58:35.53)
It was interesting, know, I was talking about it last night on my Paul Security Weekly show that in the Krebs article, which is very lengthy, just keep reading, keep reading, because it gets better when they talk about Superbox first, which is that service, the thing you can buy, free streaming, they send you a device, it’s got malware and backdoors. Then they talk about Badbox 2.0, which is an Android TV streaming devices like the T95 and one of the X88 something.
Two devices I have extra wrote post about that a couple of years ago a hundred percent these devices have Android that has apps Android apps that have backdoors and malware and are doing malicious things and I validated that right I didn’t find anything new like I validated I went on Amazon ripped the devices down got the firmware I’m like, yep, this this is definitely a thing that’s continued right Google has filed a lawsuit FBI has issued a warning What I thought was interesting was
I think was Ashley from Census was doing some of the research and it was Badbox. They discovered that there was a proxy app being put on these devices and there’s a Chinese company selling like a VPN service that the back end is using all the residential gateways as proxy devices. And they’re using that for fraud and crime and also to as a crawler for AI because
How great is it to come from a bunch of residential IP addresses?
Matt Brown(01:00:01.263)
Yep, exactly.
Yep, it’s easy to get by the WAFs then. It’s lot easier than running your scanning infrastructure in EC2 or something. They’re going to detect that right away.
Paul Asadoorian(01:00:13.898)
Yeah. So like I never would have dreamed that that’s how people would monetize vulnerabilities in IoT devices, right? I think it’s super creative. Like 15 years ago, I gave a presentation at BrewCon and talked about some of this stuff that’s actually happening today. I actually kind of predicted some of it, but in my wildest dreams, I would have never thought that China would be using us as a proxy and charging money for it. That’s just crazy.
Matt Brown(01:00:40.669)
Yup.
Paul Asadoorian(01:00:42.686)
Well, Matt, as always, good seeing you, man. Thanks for coming on the show today. Vlad, thank you very much for coming on the show as well as great chatting with you guys. Love the conversation. I hope people learn stuff from some of the things that we’ve been doing and experimenting with. So that will conclude this edition of Below the Surface. Thanks, everyone, for listening and watching. We’ll see you next time.
Vlad Babkin(01:00:49.869)
Thank you.
