PODCASTS

Unveiling AI Infrastructure Vulnerabilities - BTS #85

Eclypsium principal security researcher Jesse Michael joins Paul Asadoorian, Chase Snyder, and Vlad Babkin to explain new research demonstrating how attackers can manipulate running AI models through the underlying hardware infrastructure, without modifying model files or executing malicious code on the host.

What happens when an AI model’s weights are intact on disk, its signatures are valid, and its application appears to be running normally, but the model has been silently modified in memory? In this episode of Below the Surface, Paul Asadoorian, Chase Snyder, and Vlad Babkin are joined by Eclypsium principal security researcher Jesse Michael to unpack more than a year of research into the hardware attack surface beneath AI workloads.

The investigation began with NVIDIA BlueField-3 data processing units (DPUs) and their ability to access host memory through direct memory access (DMA). Jesse explains how IOMMU configurations intended to support performance and inspection capabilities can also permit powerful read and write access to system memory. From there, the team explored how vulnerabilities in baseboard management controllers (BMCs), firmware, and other privileged components could provide a path to modifying AI models while they are running.

The researchers demonstrated attacks that locate model weights in CPU and GPU memory, replace tensors, swap fine-tuned adapters, and alter system prompts. In one demonstration, a model’s assessment of a geopolitical scenario changed dramatically after its weights were modified in memory. Other experiments showed how similar techniques could influence software development advice or introduce unwanted behavior into models connected to enterprise tools.

The conversation moves from the technical details of DMA and memory manipulation to a broader question: how can organizations trust AI systems when the infrastructure executing them may be compromised? The episode examines the limitations of boot-time attestation, the importance of runtime model integrity, and why securing firmware and underlying hardware is an essential part of protecting AI infrastructure.

Key Topics Covered

  • The origins of the AI infrastructure research: Jesse explains how experiments with NVIDIA BlueField-3 cards in a Grace Hopper research server led the team to investigate the NVIDIA DOCA SDK, its AppShield functionality, and the security implications of inspecting host memory from a DPU.
  • DPUs and privileged hardware access: How data processing units operate as independent computing environments within AI servers, with their own operating systems, management controllers, accelerators, and memory-access capabilities that create additional security boundaries.
  • IOMMU configuration and DMA security: Why setting the IOMMU to pass-through mode can allow devices to read and write host memory, how that expands the attack surface, and the unresolved performance-versus-security tradeoffs involved in enabling memory protections.
  • The layered attack surface inside AI servers: Modern AI systems contain multiple independent execution environments, including host processors, DPUs, BMCs, UEFI, SmartNIC firmware, and accelerator cores. Each introduces software, firmware, and configuration that operators must account for.
  • BMC vulnerabilities as an initial access path: How longstanding IPMI weaknesses, OpenBMC vulnerabilities, and privileged management interfaces can provide paths into DPUs. Jesse discusses how BMC administrative access and Redfish functionality could be used to modify DPU boot behavior and obtain code execution.
  • Finding AI models in physical memory: How the researchers used DMA to scan host and accelerator memory, locate Linux process structures and page tables, identify Llama server processes, and recover GGML tensors representing model weights.
  • Stealing and modifying model weights: The distinction between extracting proprietary model weights and changing a running model. The team demonstrated both targeted tensor replacement and broader changes to model weights, without injecting executable code into the host.
  • Silent file modification through the Linux page cache: An unexpected finding revealed that modifying cached file data through DMA could cause the Linux kernel to write altered contents back to disk. This created a path for changing files without conventional host-side file modification activity or corresponding audit events.
  • Manipulating model behavior and outputs: The research demonstrated how replacing model weights could change responses to geopolitical risk questions and software development security recommendations. The central concern is that manipulated responses may appear to be ordinary model behavior rather than evidence of an infrastructure compromise.
  • Fine-tuning, LoRA adapters, and model replacement: How tools such as Heretic and Unsloth were used to prepare modified model behavior offline, and how fine-tuned weights or adapters could subsequently be introduced into a live inference process through memory manipulation.
  • Runtime model integrity and attestation: Jesse describes building an experimental tool to inspect model tensors in memory and identify changes against a baseline. The discussion explores why validating model files on disk, checking executable code segments, or verifying the boot process does not establish that model weights remain unchanged during inference.
  • AI agents and the consequences of manipulated models: Vlad and Jesse consider the implications for AI systems that can invoke tools, access credentials, interact with external services, or execute actions. When a model has access to enterprise systems, changes to its behavior can produce consequences beyond incorrect answers.
  • Infrastructure security and NVIDIA collaboration: Chase discusses Eclypsium’s work with NVIDIA and the shared focus on verifying the hardware and firmware supporting AI infrastructure. The conversation connects the research to broader efforts around BlueField security, infrastructure assurance, and agentic AI.
  • Practical defenses for AI infrastructure operators: Why organizations should review IOMMU settings, restrict unnecessary DMA access, patch and isolate management controllers, reduce IPMI exposure, and consider runtime verification of model weights alongside conventional hardware, firmware, and software integrity controls.

Timestamps

The timestamps below are taken from the episode transcript.

  • 00:00 – Introduction to the new research on AI model manipulation and infrastructure security
  • 02:02 – How the research began with NVIDIA BlueField-3, DOCA, and AppShield
  • 04:11 – Inspecting host memory from a DPU and the limitations of AppShield
  • 05:32 – IOMMU pass-through, DMA access, and unrestricted memory writes
  • 07:23 – Performance implications of IOMMU security settings
  • 08:38 – The hardware-level tradeoff between security and performance
  • 10:16 – Why read-only memory inspection is difficult to enforce
  • 11:05 – How an attacker could obtain the access needed to manipulate a model
  • 11:53 – The multiple execution environments inside NVIDIA BlueField-3 hardware
  • 14:19 – SmartNIC mode, DPU mode, host isolation, and trust boundaries
  • 16:32 – BMC attack paths and longstanding IPMI vulnerabilities
  • 17:08 – OpenBMC vulnerabilities and privilege escalation
  • 18:57 – Using BMC access and Redfish functionality to obtain DPU code execution
  • 20:35 – Internet-exposed BMCs and the difficulty of measuring the full attack surface
  • 21:57 – UEFI, PXE boot, and other firmware-level vulnerabilities
  • 22:58 – Extending DMA access to arbitrary host memory regions
  • 25:59 – Locating Llama server tensors and extracting or replacing model weights
  • 27:30 – System prompt manipulation and unexpected effects on cached files
  • 30:27 – Silent file changes through Linux memory and filesystem caching
  • 31:06 – Demonstrating changes to model behavior through memory manipulation
  • 34:49 – Tools used to prepare modified models and fine-tuned adapters
  • 35:06 – Heretic, model obliteration, and refusal behavior
  • 37:15 – Unsloth, supervised fine-tuning, and applying changes to live models
  • 37:55 – Replacing an entire model rather than modifying individual weights
  • 38:18 – Experimental runtime attestation and identifying changes to model tensors
  • 40:23 – How to establish trust in running models
  • 40:37 – Defensive recommendations for IOMMU settings and runtime model verification
  • 42:50 – Agentic AI, tool execution, and enterprise-wide consequences
  • 43:20 – Malicious tool calls, credential exfiltration, and model sandboxing
  • 45:09 – Eclypsium and NVIDIA’s collaboration on AI infrastructure security
  • 46:30 – Comparing hardware-driven model manipulation with conventional model poisoning
  • 47:29 – Why trusting an AI model requires trusting the infrastructure running it
  • 48:47 – Secure Boot, SPDM, attestation, and their limitations
  • 49:54 – NVIDIA’s BMC security research and the broader AI infrastructure security ecosystem
  • 51:31 – Final observations and closing remarks

Core Research & Related Reading

Technical Concepts & Frameworks

  • NVIDIA BlueField-3 and Grace Hopper: The hardware environment used for the research, including a DPU with independent processing, management, and acceleration capabilities.
  • NVIDIA DOCA and AppShield: The SDK and memory-inspection functionality that helped initiate the team’s investigation.
  • DMA and IOMMU: Direct memory access and the hardware mechanism used to control device access to memory, central to the attack technique and its mitigations.
  • BMC, IPMI, Redfish, and OpenBMC: Server management technologies discussed as possible initial access paths or privileged control surfaces.
  • UEFI and Secure Boot: Firmware boot technologies and integrity mechanisms whose protections do not necessarily establish the integrity of a running AI model.
  • SPDM: A device security and attestation protocol discussed in the context of hardware trust.
  • GGML tensors and LoRA adapters: Model data structures and fine-tuning mechanisms used in the researchers’ experiments.
  • Runtime model attestation: The proposed practice of comparing model weights in memory with a trusted baseline to detect unauthorized changes during execution.

Tools & Platforms

  • NVIDIA DOCA — Development environment for BlueField DPU applications.
  • Ollama — Local model execution environment used in the research.
  • llama.cpp — Inference framework associated with Llama server and GGML model structures.
  • vLLM — Alternative inference engine discussed as another potential target.
  • Heretic — Model obliteration tooling used in the experiments.
  • Unsloth — Framework used for supervised fine-tuning.
  • Hugging Face — Model distribution ecosystem discussed in relation to model trust and sandboxing.
  • Claude Code — Agentic coding tool mentioned while discussing the security implications of models that can act on enterprise systems.

Transcript

Paul Asadoorian (00:26): Welcome to Below the Surface. This is episode number 85, being recorded on October 30, 2026. I’m Paul Asadoorian, joined by our regular hosts, Mr. Chase Snyder. Chase, welcome. Mr. Vlad Babkin. And of course, our very own Jesse Michael. Jesse’s your title principal security researcher here at Eclypsium. Outstanding. It’s wonderful to have you. We’re just gonna do a quick announcement and then we’ll dig into it. Below the surface listeners can learn more about Eclypsium and how we help organizations assure and secure the infrastructure their operations depend on by visiting eclipsium.com forward slash go. You’ll find our guide to AI data center security, our white paper on eradicating hidden threats on network edge devices, and a case study showing how a major cloud provider used Eclypsium to protect its infrastructure hardware during a merger IT consolidation. You can also request a demo, see how Eclypsium verifies device integrity, hardens against risk, and detects active exploits across enterprise infrastructure, from AI data centers to network edge devices to user endpoints. Find it all at eclipsium.com forward slash go. Jesse, it’s wonderful to have you on below the surface. I guess we would start at the most logical place. And actually, someone actually asked me this question, so I thought we’d answer it on air. How did we come up with the crazy idea to do this crazy amount of research that spanned over a year?

Jesse Michael (01:56): We started out looking at some BlueField-3 cards. We have a Grace Hopper server in our research lab that we do been doing some experiments with. We were looking at the BlueField-3, seeing what kind of capabilities it had and looking at things in the DPU or data processing unit, which is essentially this out of band acceleration processing environment where you can run your own code in it and you can install your own version of Ubuntu, there’s a bunch additional functionality built into the DPU, so it seemed like an interesting place to look. There’s also the DOCA SDK. DOCA is NVIDIA’s data center on a chip architecture, which the I guess DOCA you can pronounce more easily than DCA. Doesn’t really make sense based on the acronym, but essentially it has this entire SDK for building your own applications that run in the DPU in other parts of NVIDIA’s ecosystem related to this. But one of the features that we saw is there’s like a DMA library and functionality about how to use that we dove into some of the details there. Looked at how it worked and experimented with that a little bit and also saw there’s this AppShield of the DOCA SDK that basically is a library and instructions for how to build your own application that runs in the DPU to do interesting things like looking inside of the host memory to do things like find processes and check the integrity of code segments within those processes that are running inside the host.

Paul Asadoorian (03:53): Interesting. So the DPU is designed specifically to like monitor what’s happening on the rest of the system with your AI model and has facilities to interact with that.

Jesse Michael (04:06): Only reason, but that’s one of the one of the use cases is being able to do inspection of what’s running in the host. And it’s a it’s an interesting capability, but we were looking at it and the AppShield functionality only looked at code segments within a running process and it had a bunch of extra constraints like you needed to run a collection tool on the host first in order to make a map and send that over to the code that’s running in the DPU. And we wanted to look and see what we could do with that and if we could get around those limitations. So we didn’t actually d need to run anything on the host and do all our collection from the DPU and what we could actually do with that. So that kind of was how we first started st got started looking at this area in the first place. And one of the one of the key parts of this that ended up being a big part of the research is that the AppShield instructions in the d the DOCA SDK explicitly tell you to set the IOMMU into pass-through mode in order to enable this functionality. And if you’re setting yeah, if you’re setting the IOM MMU of the host into pass-through mode, That doesn’t just give you read access into the host. It also gives you write access into the host. So we wanted to look and see what we could do with that. So

Paul Asadoorian (05:33): IOMMU is really just a fancy term for like the gate that gates what can be read and written into memory directly through DMA.

Jesse Michael (05:43): It’s you might have heard the term of like a memory management unit. It’s essentially similar to what you would have for just the CPU going into the host. You can also have devices that are initiating their own DMA and then that’s going through the IOMMU as well. It kind of encompasses more things in the system, more different initiators or request initiators and things that it can send requests to. So it is this fundamental permission area within the system. And I if you leave it in the default configuration If you try and do DMA from the DPU or other devices, like maybe you stick a random PCI device into your server that has bad vulnerable firmware gets exploited and then that device starts doing DMA transactions, those get blocked by the IOMMU. But NVIDIA’s instructions for enabling AppShield explicitly said to set the IOMMU into pass through, and that seemed like a big red flag for us. So we w ran from there and I did just check and NVIDIA’s Grace performance tuning guide still says to that if you set the IOMMU to into pass through mode you can see some performance increases and gives instructions on how to set the IOMMU into pass through mode. So it doesn’t that was

Paul Asadoorian (07:19): Yeah, a question that I got, so I actually presented this live yesterday, right after we announced it and someone asked, have we done tests to see the performance impact of enabling or disabling IOMMU? I don’t know that we have done those tests or do we understand the impacts or

Jesse Michael (07:34): Talked about doing those tests but I haven’t actually done any s performance testing with it enabled or disabled. I’ve just done access control tests to see what happens if I try and do requests with it enabled, what if happens if I try and do requests with it in pass through mode versus not in pass through mode. Mm-hmm and what you can do there. I th I think it I if there is a performance impact, it is a like a legitimate strategy where some me people might not care about this particular threat model, but they should at least be aware of the implications of the setting. Because if you’re turning off this security setting, it should at least be like a You should know the ramifications of turning off a security setting like this, I essentially.

Chase Snyder (08:32): Performance versus security runs deep. Montagues and Capulets level, all the way down into the hardware.

Jesse Michael (08:43): No, I there’s the whole mindset of like move fast, break things and like everybody is in a rush to get things as fast as possible. So people won’t necessarily turn on legitimate security features that the say that slow things down.

Chase Snyder (09:04): I love the move fast and break stuff like meme because I think that was from Zuckerberg and Facebook in like and I think they changed it. It was like this is our internal mantra, move fast and break stuff. And I think they changed it in like 2014 to move fast with stable infrastructure, which is so funny because of how just like dry it is. But also like that the move fast and break things meme. Has done so much harm. Probably a lot of good too, but people built a lot of cool stuff, but dang, that thing was sticky.

Jesse Michael (09:44): I mean it there’s a lot of trade offs between like moving fast and hoping you don’t get caught by this type of security exposure, but some sometimes you do and suddenly there’s like a huge breach and a lot of bad things can happen if you don’t pay attention to some of these security implications.

Paul Asadoorian (10:10): Is it just is it just security? It also sounds like there’s functionality implications, right? That your DPU doesn’t can’t have all the functionality.

Jesse Michael (10:18): So well that was one of the things that I was looking at also. It’s like if you’re just using the DPU inspecting state of the device, there isn’t really a good way to just enable read access without enabling write access if you’re just setting it in a pass through mode. So if you want to if you want to enable just read access, then to be a little more fine grained and that might require like architectural changes to the IOMMU itself. I don’t know if that’s something that you could do without like revving a an entirely new architecture. But it seems like a useful functionality to have that of only read access.

Paul Asadoorian (11:05): Yeah, and I think it’s also I think we’ve described some of the attack, right? The attacker needs access to the DPU. These settings allow what we’ll talk about in terms of how to manipulate AI models. Some of the questions that I received, and I think rightfully so, are like how do attackers get into this position where they can manipulate the model? And I suppose we should probably start with what I like to call the multiple stacks of Russian dolls. AI infrastructure. There’s so many components. And so many computers. It reminds me of my Sun Solaris workstation in 2001 that had a card in it that was a full Windows PC so that I could run both Solaris and a Windows PC at the same time. DPU seems kind of similar to that.

Jesse Michael (11:47): Yeah, I mean like the a lot of people who are familiar with servers will know like you have the main host processor, you have a BMC, the baseboard management controller for out of management. You have various different devices like PCI cards and other things. There’s a bunch of other execution environments on the rest of the server that we don’t necessarily need to go into today, but just in the BlueField-3 card that’s in a PCI slot, that has its own essentially like a host processor with the DPU, where it can run a full Linux of Ubuntu with whatever you want to put in there. By default it’s running Kubernetes inside this PCI card in the Ubuntu installation. So there’s probably quite a bit of vulnerable code just running the default stack that’s running in this BlueField-3. It also has its own BMC in the PCI card. So like you have like the host main CPU, the platform BMC, and then just on this card you have the DPU ARMOS, the BlueField-3 BMC. It also has the these RISC-V accelerator it’s like the a the APU you can write applications I forget the specific term that they use for the code that you write that you write into the APU, but that’s like three different major execution environments for this. There’s also the SmartNIC code itself, which has its own separate firmware. There’s a

Paul Asadoorian (13:32): UEFI both in the DPU and on the host system, right as well.

Jesse Michael (13:37): So when the DPU operating system is booting up, that is also running its own UEFI and loads and runs the operating system, does the handover to the operating system that you have running. And it’s a very complex system that has a lot of places that you can code.

Paul Asadoorian (13:58): A lot of attacks that and the question’s Well, all of the neo clouds, all of the big frontier models, like would they succumb to these attacks? I’m Well, they could just because there’s so much attack surface to worry about as we just described.

Vlad Babkin (13:58): And to add something to this discussion also, there is like multiple modes of operation for this NIC. So like if you run it in SmartNIC mode, I believe it trusts the host, right? And if you run it in DPU mode, it doesn’t. Like I don’t remember the exact names for it.

Jesse Michael (14:31): Yeah, so there’s SmartNIC mode which doesn’t have the DPU up and running. And then there’s DPU mode. So you have the switch between SmartNIC mode and DPU mode. And in DPU when you have it in DPU mode, there’s something called host isolation or zero trust mode, where in that case interfaces between the host and the DPU are restricted, so when the host is in host privileged mode, there’s a device node called dev R ship. Like console where you can open a terminal and get like the console of the DPU. You can push firmware updates or push updates through this rshim driver from the host into the DPU if it’s in this host privileged mode. If it’s in host isolation mode or zero trust mode, the DPU is still up and running and still has a lot of these functions, but it doesn’t allow the host to do the same privilege requests to the DPU. And it’s a it is an interesting functionality. And if you do set it into host privileged mode, that does restrict the ways that the host can directly get code execution in the DPU. But it doesn’t close off things like being able to DMA from the DPU into into the host. So it’s basically Deprivileging the host, but not really depriv deprivileging the d the DPU. So it the DPU still has some interesting mechanisms for attacking the host.

Paul Asadoorian (16:26): Crazy so many attack paths. I don’t think we even explored all of them, right? But I think the primary one that people should be aware of is the BMC. There’s been a lot of research both from Vlad, a lot of research from Vlad specifically. A lot of research from HD Moore, who’s an amazing person, an amazing researcher, of course. And that’s becoming such a ripe attack sort of we can go back to 2013 to that vulnerability that really was never fixed in IPMI, that offers, pathways into that BMC.

Jesse Michael (17:01): Yeah, so there’s been a ton of research related to IPMI, including HD Moore’s Dan Farmer’s work, a lot of kind of groundbreaking showing that like the IPMI architecture itself had some design flaws where the way that the key exchange works with IPMI there is a essentially a way to dump hashes from the target system. Lot of systems that still support IPMI are still vulnerable to this because it’s a architectural issue. You mentioned HD Moore, he included some he published some new research just this year for different and different vulnerabilities that some of those included a couple of vulnerabilities against OpenBMC, including a authentication bypass where Unprivileged remote attacker can essentially bypass an authentication and get a valid session of a user on the system or a user on the BMC and a second one that can basically escalate that privilege to administrator access. So tying those two together, you can go from having no credentials on a remote attack surface. To getting administrator in the BMC and that is something that did apply to the BlueField-3. So and that would

Paul Asadoorian (18:37): From administrative user on the BMC to either I forget if I enabled or created a new user that lets you SSH in as root. So some of the things that administrative access to that BMC can give you just from the you know GUI perspective.

Jesse Michael (18:51): Yeah, in the BlueField, w one of the a lot of a lot of open BMC stuff says if you SSH in you’ll not get a full shell, but in the BlueField-3, if you SSH in to their BMC, you actually get a unrestricted shell in the BMC, which is it’s pretty powerful and it’s less stepped, but even if you just are before not even SSHing into the BMC. Just with administrator access to the BlueField-3 BMC and administrator credentials that you’ve stolen or exploited a vulnerability like this to get access to crack password IPMI password hashes, things like that, you can use functionality in the BMC to essentially get code execution and run your own code inside of the DPU.

Paul Asadoorian (19:45): Yeah, that’s awesome. That’s well, awesome for attackers. Not so great to kinda defend this infrastructure which has all of these attack paths that we’re uncovering for our listeners, which is great.

Jesse Michael (19:57): Yeah, I mean HD Moore’s research this year also pointed that I think at 70,000 or 80,000 devices were still listening on the internet and were vulnerable to this IPMI from 2013. So peop people still connect devices to the internet and that’s a terrible idea.

Vlad Babkin (20:19): Yeah the better question is if 70,000 really l really all of them like I’m not so sure that everyone runs a device on the default port like the devices running on non default ports might even make a lot more surface like for example imagine that you have default IPMI port but then one router can expose 1,000 BMCs behind it all on non default ports. Like I can totally see somebody doing this setup. And finding all of them is really difficult. So 70,000 is just what’s easily findable. This is what I would call

Paul Asadoorian (20:19): Those are just people that expose them to the internet too. Most BMCs are not exposed to the internet. And I think some of HD’s research included the ones that from telemetry from runZero, right? Where like yeah, there’s a lot more that exist not on the internet that are also vulnerable.

Vlad Babkin (20:19): Yep, a lot, a lot more exist in closed-off networks, which are might not be so closed off with all of the vulnerabilities we’re discussing. Like some of the vulnerabilities can be used to go back into the BMC.

Paul Asadoorian (20:19): All right. So there’s lots of attack paths to gain access to the DPU just in this case. And then with that access to the DPU, as commonly we’ll find, is that IOMMU protections aren’t enabled for all the reasons we mentioned. So Jesse, now what does an attacker do? This is where it gets really super interesting, right?

Jesse Michael (21:43): Yeah, so there w there were a couple of different things that we wanted to experiment with. I mentioned that we wanted to do things like can we steal a proprietary model? We wanted to see what we could modify. And one of the constraints that we ha that we wanted to have is we wanted to see what we could do just changing data. We didn’t want to inject code, we only wanted to do data changes because that seemed Stealthier and more interesting. So one of the things that we did is from the DPU, once we have this DMA privilege, I had mentioned that the intended way to use this was that you there was some negotiation where like the host sets up DMA buffer registers it and then a peer rec r in the D the DPU requests it, does requests like that, or the AppShield functionality is like getting some information from the host from the DPU using that information to correct set up maps to go into the host memory. One of the things that we found is that AppShield was using a private or internal library function that was being set up to set the range. So instead of just having a limited range of let’s look in this particular area, we set our range from like zero to top of memory and that works. So we were able to use that to just go look at any physical addresses within the host instead of just the ones that had already kind of negotiated ahead of time. Using that access to go look anywhere in physical memory, we could do things like scan the host memory or the host CPU shared memory. We can scan the GPU V RAM in the HBM memory to look for things like the model specifically. One thing we did was scan Linux kernel, find the task list, iterate through that Basically the task list to find page tables for a particular process, found things like the Llama server that’s running in memory. For our proof of concept, we were looking specifically at code running under like Ollama or Llama server. We could do a similar attack targeting like vLLM or other inference engines, things like that. But for our experiment, we were looking at Ollama and Llama server. So what

Paul Asadoorian (24:41): Where we have to know where in memory the model is loaded, basically. You have to

Jesse Michael (24:44): But we f we found the Llama server and then inside Llama server we had the page tables for Llama server. Then we could scan those pages pretty quickly and look for things that looked like GGML tensor structures, verified basically found a bunch found a bunch of different candidates and then narrowed that down through some heuristics so like these are legitimate la actual tensors or model weights. And then we could basically steal those and Exfiltrate those if we wanted to. I if all we want to do is just modify an existing model, we could do just a simple search and replace where it’s like we search for the beginning of each tensor and then replace the entire tensor, our own modified contents. And that search and replace is actually pretty quick. So I think we were getting about just a simple, very simple just search and replace was running at about 3 GB a second. So Scan scanning through the host memory is pretty quick. If we’re doing more complicated things like finding the Llama server, page tables and do that, it takes a little bit more time, like maybe five minutes to find all the stuff that we want to do. And then it’s just like what you want to do at that point, whether you’re modifying the model weight, maybe you want to swap in your own fine tune adapter or to adjust things separately. Like if you steal the model you can then adapter or train your own model based on those weights make your modifications live in memory and that all work. One of the other th interesting things we ran into is that if you just do like a naive like search and replace essentially a marker that’s like search for this value in memory and change it everywhere in host memory, including both CPU memory and GPU memory. Depending on what you’re searching for, you can find some interesting interesting things that you hit. Like we did some experiments with changing the system prompt. So not target not targeting the model weight itself, but changing the system prompt that’s being used to process this as additional instructions before it even goes into the inference. If you just do a search and replace for that system prompt, you’ll end up hitting it a bunch of places in addition to just in the Llama server process. Like Linux kernel file system buffers. And if you replace the contents in the Linux kernel file system buffer, it’ll actually silently write that value back out to the disk. I was doing some experiments where I was just doing a search and replace of the system prompt. It was also changing the system prompt in my Llama configuration on the disk itself. That and I had to go figure out what was going on and then I had to scope it down so I was avoiding the Linux kernel memory ranges when I was doing that search and replace to target the system. That’s another interesting capability because it doesn’t need to be it doesn’t need to be a file that’s open writable access. If it’s if the file is in the Linux kernel file system, even for read-only access, if you write to that bar via DM, the Linux kernel will notice that page has changed and write it back to the disk. So I have a funny experiment where There’s a sysfs node that you can write to drop caches. If you drop the caches in the Linux kernel, just cat a particular file to pull it into the file system buffer, even with only read access, do the search and replace, drop caches again, you’ll see and then read the file again, you’ll see that it has written back to the disk. So

Paul Asadoorian (29:12): Getting around all protection in the file system in Linux ’cause you can manipulate memory directly.

Jesse Michael (29:17): It’s something that doesn’t create a like there’s no like syslog event or anything or audit log event that shows up. Just silent silently changing this buffer in the file system cache and it gets written back to disk, which was kind of cool. And it seems like a an interesting capability if you want to be stealthy about modifications to things like audit logs and other things that are being written to the local file system and not like immediately shipped off via like a remote syslog service or something. So

Paul Asadoorian (29:51): We did more than just adjust the prompt, right? Did we lo basically side load a new model in that would change the answers or you can see it in our demo videos like we’re manipulating the model to is there an eminent attack that China’s gonna attack Taiwan? We can change that answer to whatever we want, basically.

Jesse Michael (30:14): The system prompt, but there’s really only so much you can do with modifying the system prompt unless you have like unlimited like a frontier model or something that the guardrails are implemented in the system prompt where if you like nuke the system prompt then you’re removing guardrails. So if you want to actually change the model behavior and change things that are baked into the model itself, you do need to target the weights of the model itself. So that is one thing that we did for the demo about like the national security applications. Basically ask the model what is what’s the likelihood of important happening. Like is China likely to attack Taiwan in the next 30 days without changes the models like no it’s unlikely don’t worry about it. And we did some manipulation to basically patch in our own model weights to the running live model and then it was like yeah there’s like an 80% chan chance within the next two weeks. So there were some other that was something that we did with it was modifying the model weights in memory. There were some other experiments that we did with other models. There were experiments we did with modifying the base model itself, the weights for the base model itself. We did some experiments with modifying a LoRA adapter and swapping in a separate fine tune like fine-tuned adapter. For our own biased fine tune adapter versus modifying the base model itself. So we did do experiments with all these things, including other scenarios, including Just say I’m a developer who’s building a financial banking app and asking about algorithms I should use for strong something is going to be running in a financial environment and giving it bad advice like it’s totally fine to use MD5 and other really old deprecated algorithms, things like that. So there’s a lot of different things that you can do as far as bias biasing a particular answer in a particular way and get getting it to respond differently. And then if you apply that in memory from a DMA attack, it’s relatively quick to do so. And then you could swap it back out and unapply that so that the behavior changes later and there’s not really any indicators to host side telemetry that this is happening because nobody is really looking at the model weight themselves in memory. Like people are doing things like Check checking the weights on the disc and checking signatures and things while it’s loading things, but really checking the weights themselves at runtime and

Paul Asadoorian (33:36): Was it I and I know I asked you through this question along the way, so for the audience benefit, did we develop there was some combination of tools that were available to manipulate models and tweaks that we or tools that we developed ourselves?

Jesse Michael (33:51): So we use some existing tools in order to build things like we did experiments with like heretic for obliterating models and building so with heretic you can basically take a base model Of models would in essentially encode refusals into a particular vector direction within the model. And Heretic or other obliteration tools would do a lot of experiments try to check basically have a fitness function with experiments where it sees with this adjustment to the model, it only refuses like 10%. Two percent of the requests were previously was refusing 100% of the of these considered bad requests. And essentially give you a LoRA that you can then apply to the model stop refusing to give bad requests that are considered out of policy by the trainers of the model. This is a an interesting mechanism that people have used to essentially uncensor or kind of jailbreak different models is to do this obliteration mechanism. Some other tools that we used were unsloth. We’re doing there’s the unsloth framework for doing supervised fine tuning where you basically have a particular model, you provide it a bunch of test prompts or sample prompts intended responses and it will go through and tr do an additional training adjust the model to provide the answers that you are providing as part of this fine tuning process.

Paul Asadoorian (35:51): So then can do that and slip your model weights in via memory and then you just basically changed how the model answers question.

Jesse Michael (36:00): So supervised fine tuning is a common approach to basically say I have a base model but it doesn’t quite do what I want. I want to adjust it a little bit. So that’s one of the mechanisms that we use. Like once we have a base model, we can then do a fine tuning based on that. And then we have the changes that we need to apply back to the base model in memory. That’s that fine tuning is like something you do on a different system offline and then you come back to the system and actually apply it. I guess you could theoretically on unsloth and do your own fine tuning in the DPU, but it would probably it would probably be faster to run it offline on your own system somewhere else.

Vlad Babkin (36:41): Even b even worse yet, nothing is really stopping you from just replacing the model with another one. Yeah, there is some there is some extra complexity around the code expecting a certain weight structure, but nothing stops you from doing the model in a similar family. And yeah, this just showcases a little bit less technical, but a little bit easier to understand. Like you can just do another model entirely.

Jesse Michael (37:05): Yeah, no, that’s one of the things that I was ex that I was looking at with so the experiment for stealing weights where we basically were parsing through the Llama server memory, finding these GGML tensor structures. A as part of this experiment I also did build an attestation tool where my tool that runs in the DPU can scan through the Llama server. Determine if any weights were added or changed or removed from the running process against the baseline. In order to be able to run this capability in the DPU, you do need to have IOMMU pass through set on set, which then makes you vulnerable to this. So it’s like I wish there was a way that we could enable read access to the host memory so that you could do things like do this attestation, but because we are singing through But basically scan it. Doing a page walk in order to follow pointer. Basically we’re in the host memory in the Lama Llama server process, the GGML tensor structure has virtual pointers within that address space. So we have to do like page translation. So we convert those virtual addresses into physical addresses to be able to do DMA and verify that we have the data that we think we have. There’s also page tables in the GPU memory regions also. So we do page walks both in the CPU pages and the GPU pages in order to find all the data structures. But with that capability, we can insert new tensors and new weights into the Llama server running process. We can remove weights also so we could totally entirely swap it out to be a different model entirely like Vlad was pointing out.

Paul Asadoorian (39:07): That’s amazing. So you how do we trust our models? What was the like we d we spent a lot of time in this research, right? And I think all of us share the concern of well, how do we move to a point where we can trust our models?

Jesse Michael (39:22): Yeah, I think one thing is actually turning off that IOMMU pass through setting, like verify that the IOMMU is not going to allow DMA access to any arbitrary device that’s plugged into the system. Even if you want to have read and security scanning from the DPU, in order to enable that, you’re basically allowing other devices to be able to do DMA if you follow just the high level instructions where it’s saying enable this. You can do some additional restrictions to say allow this specific device in an IOMMU group. So you can say have the host and the DPU in a specific IOMMU group and that would restrict it so only those two devices can talk to this and other DMA devices can’t. So that’s one thing to look out at if you really do need pass through enabled for something like inspection from the DPU. But if you’re not using that functionality, absolutely turn it off. I think it would be useful to have functionality that does scan the Llama server or vLLM in memory. Have there been modifications? Like don’t just look at code segments to look and see if you have code that’s running in the process. There’s a lot of like host side tooling to look for like process injection and other things like that, but if you’re doing this sort of inference, the models themselves are pretty interesting even if you are not changing the x86 or ARM64 code itself, because you do have computation this happens when this inference process is running through the model weights. So that is effectively a type of code that we do need to look at the data, not just what’s historically considered Code within the system.

Vlad Babkin (41:35): And the core part of this assumption is that inference or process de facto can call tools potentially all across your infrastructure. Imagine that like okay, there is like people like developers like using Claude Code or like your CEO using call Claude Code that has full access to his machine. This is literally code that like de facto executes across your entire organization. So it’s little code.

Jesse Michael (42:06): There was a nice write up recently just about a very simple fine tune to train to do tool calls to basically silently exfiltrate your credentials and showing how simple that was, where a lot of these things are tool calls where it’s yeah, I’m gonna allow web search or web requests, but a lot of times if you have like a tool call that’s doing a web request or things like that it can exfiltrate things and poison these tools. And if you’re downloading model weights from like Hugging Face or other places How much are you actually sandboxing them? There’s a lot of talk about running these as sandboxes. If we’re allowing tool calls to do things like run curl or other things like that, you could still have it exfiltrating inf interesting build from your environment.

Paul Asadoorian (43:07): Yeah, so do you can’t let your credentials get into your and Claude warns you about that a lot too, but it’s still inevitable that they’re gonna end up in there.

Vlad Babkin (43:07): Yep, and a sandbox in the model is especially hard. Like if your sandbox has curl with unrestricted DNS, the model can just automatically find the route to actually exfiltrate data through DNS. We have to observe this in OpenAI’s own reports. It’s not news.

Paul Asadoorian (43:07): It will find a way. That’s kind of scary.

Jesse Michael (43:39): Well, I there’s a lot of talk about like AI agents you like communicating over forum posts and stuff like that, which is

Vlad Babkin (43:46): Right. Sandboxes are escape rooms for a agent.

Paul Asadoorian (43:46): We also announced a partnership with NVIDIA. Chase, did you have more on what that partnership means? I know that was part of this pretty big announcement along with the research.

Chase Snyder (43:46): Yeah, sure. I mean they published a blog post on their site that was you know co-created with us that kind of describes the capabilities that Eclypsium as a company and as a product brings to securing AI infrastructure at this. Hardware layer. So they acknowledge that there’s a challenge there. There’s these privileged components inside of the AI hardware that have, as we have demonstrated this possibility to be targeted and lead to manipulating model outputs from the hardware up, as we said. So yeah, it was great to I think our leadership and their leadership work closely together to put out these Posts and we’re we’ve partnered with them in some other ways as well. Initially we were we’re in the inception partnership program. And then we’re also in the I’m gonna need to look at it to get the wording right of it, but the like open agent security initiative when we actually figure out what the actual wording of it is, it’s a bunch of words, but fundamentally we just have a lot of touch points with them around this. The security of AI infrastructure is really Fundamentally, it’s top of mind for them, it’s top of mind for us. And yeah, fundamentally they recommend paying attention. And even using Eclypsium for the sort of infrastructure hardware level security. We’ve talked a lot over the past year or so about how security in the AI realm often focuses on. Model level and application level sort of security. We talk about model poisoning, and that’s not exactly what this is. I don’t know, Jesse, how would you how would you make the comparison between what people mean when they say model poisoning, which is a widely used term versus what we’re talking about here? It’s not really maybe it’s similar but through a sort of different access path that goes through this privileged hardware that’s in all the

Jesse Michael (46:14): I mean part of the thing is you need to be you’re running in the first place. If you don’t if you don’t fully secure the hardware that the model is running, be confident that the model is even the model that you think you’re running. So I think that’s one thing to kind of pay attention to here because a lot of it is like talking about like the overall behavior of different models and what these capabilities are, but if you’re not even running the model that you think you’re running you don’t have the same expectations that you even are planning on. So like maybe you’ve done training to maybe you’ve done your own training to build a model to have certain policies baked into the model then your assumptions of when you’re interacting with the model and having it do things are based on that, but somebody else has manipulated or replaced the model without you realizing it, a lot of the assumption assumptions that you have are gonna be thrown out the window. Like for example,

Paul Asadoorian (47:34): Everything’s fine. And there’s also SPDM, there’s an attestation framework, but that doesn’t necessarily help you here either.

Jesse Michael (47:41): I mean like a lot of the attestation frameworks are about things like you running the code that you thought you were when the system booted. But if there are vulnerabilities in the system, it’s like I have been I have attested that I’m running the vulnerable version of the code, not that the code is bug free. So that’s something to kind of pay attention to also is like definitely update the system, don’t just stay on old versions because new things come out like HD Moore’s new research that exposed other vulnerabilities. If you are still using IPMI anywhere, you should probably block it any anywhere you can. I think IPMI is has a lot of danger that’s still lurking there. It’s still causing systems to be exploited years later.

Chase Snyder (48:38): When we were initially talking about the BMC aspect, I wanted to bring up that NVIDIA, I think in 2023 published this great white paper called Breaking BMC, the Forgotten Key to the Kingdom. And it describes a lot of the stuff that you’re talking about, going all the way back to that vulnerability that’s baked into the IPMI protocol. They cite Dan Farmer’s research, they cite Eclypsium research, and yeah, it was just a great rundown. I don’t know if we can include it in the When we post this episode, we can include the NVIDIA blog about the Eclypsium partnership and the this white paper that’s a great just sort of primer on the various challenges in BMCs as specifically relevant to NVIDIA hardware. I want to make sure that I state correctly the name of the various alliances and partnerships that we’re in with NVIDIA, though. So we’re in the Open Secure AI Alliance, which I believe is something that NVIDIA participated in, but then as Sort of also being led by the Linux Foundation. And then also the NVIDIA Open Agent Safety Platform. And then we’ve got this blog post that’s on our website and on their website called Building Trusted AI Infrastructure for Safe and Secure AI. And it leads with this statement that NVIDIA and cybersecurity partner Eclypsium are advancing infrastructure security for cloud computing and agentic AI with NVIDIA BlueField and Eclypsium Infrastructure Assurance. So it’s this yeah, hardware level infrastructure level security. Increasingly top of mind as folks like Jesse demonstrate the capacity for compromise at that layer

Vlad Babkin (48:38): By the way the things Nvidia listed in their Forgotten Key to the Kingdom white paper and article are I believe also in AMI MegaRAC mainly. So this literally translates to what we just discussed. Those can be used as initial access vector and then every single DMA attack Jesse described applies.

Paul Asadoorian (48:38): Well, that’s awesome. Jesse, thank you so much for appearing on the show today. Thank you, Chase and Vlad. You can again, we’ll put all the links to all the research, all the demo videos, everything in the show notes. So thanks everyone for listening and watching this edition of Below the Surface. We’ll see you next time.

Back to all Podcasts