BTS #51 - When Windows 10 Expires
In this episode, the hosts discuss the impending end of life for Windows 10 and the necessary preparations for upgrading to Windows 11. They explore the specific hardware requirements for Windows 11, including the importance of Secure Boot and TPM 2.0, and the challenges enterprises face in managing large-scale migrations. The conversation underscores the importance of meticulous planning to prevent costly failures and the influence of legacy systems on the upgrade process. In this conversation, the speakers discuss the implications of transitioning to Windows 11, focusing on the challenges posed by legacy systems, supply chain issues, and the importance of modern hardware for security. They delve into the Black Lotus UEFI boot kit and the necessary mitigations, emphasizing the need for organizations to validate their security controls and establish a robust trust framework. The discussion also highlights the growing importance of third-party risk management in cybersecurity, particularly in relation to supply chain security.
Subscribe
Transcript
Paul Asadoorian (01:32.418): Welcome to below the surface it’s episode number fifty one being recorded on Wednesday may twenty first twenty twenty five of course pause doryan joined by a more regular co-host here miss chase niner is here with us chase welcome
Wes Dobry (01:35.738): You
Chase Snyder (01:46.589): Hey guys, good to be here. Thanks, Paul.
Paul Asadoorian (01:49.046): Longtime listener first-time caller. Mr. Wes Dombrey is here with us. Wes, welcome.
Wes Dobry (01:53.956): Glad to be here. Thank you for having me.
Paul Asadoorian (01:56.45): Wonderful to have you, Wes. Describe for our audience your role at Eclypsium and a little bit about your background.
Wes Dobry (02:02.621): Yeah, absolutely. So my role here at Eclypsium is I run our sales or solutions engineering team globally. So I’m the geek in the room typically. So I get to go and have all the fun conversations with our customers on how we align our strategies and our technologies with the things that really matter the most to them. A little bit about me is I’m a 25 year plus practitioner, which is probably showing a bit too much of my age here. I actually started in IT migrating devices from Windows NT4 to Windows 2000. So the topic at hand today is actually very close to my heart where I’ve helped organizations around the world migrating between Windows versions, doing architecture all the way to heavy, heavy security architecture for data centers and data center build-outs.
Paul Asadoorian (02:37.23): Yeah.
Paul Asadoorian (02:53.048): Awesome. Wes is very much a nerd. It’s a love hanging out with Wes. You get nerdy with ESP32 gadgets and all that stuff too. So it’s awesome. Just a quick announcement before we dig into it below the surface listeners. You can learn more about Eclipseum by visiting eclipseum.com forward slash go there. You’ll find several resources, including the ultimate guide to supply chain security and other presentation called unraveling digital supply chain threats and risk.
Chase Snyder (02:57.723): With your people at West.
Paul Asadoorian (03:20.0): A paper on the relationship between ransomware and the supply chain and a customer case study with Digital Ocean. If you’re interested in seeing our product in action, you can sign up for a demo. All that at Eclipseum.com forward slash go. The topic today is a hot one. I didn’t anticipate this being a hot topic and I caught wind that, yeah, Windows 10 is eventually going to go end of life. I’m like, wow, that date’s coming up pretty soon. And then I was like, wow. The requirements for Windows 11 have been a thing. Obviously, Windows 11 has been around for a while now. And I was like, you need some interesting newer hardware to run this. And I was like, well, this could be the year of the Linux desktop. And then that topic kind of exploded. I’m speaking at RVA sec for our listeners, June 4th, Wednesday, I’ll be in Richmond, Virginia, speaking at RVA sec on using Linux on the desktop. And one of the reasons I did that talk was because of the topic we’re talking about today. Is that Windows 10 is going end of life. So do you throw away your hardware? Do you hack Windows 11 to install on it anyway? Do you pay Microsoft to continue Windows 10 support? Or do you Linux on your desktop? Those are the kinds of questions that people are asking. So Wes, I’ll turn it over to you to kind of, know, from a customer enterprise perspective, what’s going through people’s heads?
Wes Dobry (04:41.628): Yeah, yeah, so, you know, this is such an interesting challenge relating to Windows 10 to Windows 11 because as you pointed out that this is the first time that we’ve really had some foundational security components that are required to perform the upgrade, which is going to massively affect how organizations approach this one. You know, in the past, at least in the organizations I’ve been part of, we’ve done less of that discovery and planning and more of the execution side of things where it’s like, if you have this much CPU, you have this much RAM, you have this much hard disk space, we can blast out an in-place upgrade. For the ones that we can’t really do that, we’ll bring them in and either replace the hardware to upgrade it or as you put it, we’ll hack it to get the operating system on there. With Windows 11, we’re now in a completely different scenario.
Paul Asadoorian (05:29.814): Yeah.
Wes Dobry (05:35.132): The foundational requirements are now things that are baked into the systems themselves. And if you don’t have them, you’re not getting Windows 11. And at the very least, you’re not getting Windows 11 supported on it, which eliminates the opportunity for many organizations to do that.
Paul Asadoorian (05:41.697): Mm-hmm.
Paul Asadoorian (05:48.6): Right.
Paul Asadoorian (05:52.0): Yeah, yeah, I see. Yeah, because you’re right. I put Windows 11 on unsupported hardware, Microsoft isn’t going to give me support. I probably would still get security updates, but I’m not going to get support from Microsoft.
Chase Snyder (06:04.305): Just as a side note about that, I looked into it, did a little SEO research and noted that there is a lot of Google traffic right now for install Windows 11 on unsupported hardware. It’s hard to tell if that’s consumers that are like, I’m trying to update my laptop or if it’s like enterprise or like small media business. There’s a lot of people out there that might want to do that when they realize the implications of this update.
Wes Dobry (06:04.523): Absolutely.
Paul Asadoorian (06:13.868): Yes, well, because a lot of people that. I think it’s consumers.
Paul Asadoorian (06:23.789): Yeah.
Paul Asadoorian (06:29.196): It’s not really recommended though. Let me just talk about that for a moment, right? The articles that I’ve read that show you how to do it are like, yeah, but like don’t do that. Like it’s not, I think the supported hardware is there for a reason. And Windows 11 was coded for supporting that hardware. And now you’re going against that, which is, could be a bad day.
Chase Snyder (06:54.237): Planned obsolescence, but sometimes that’s good and sometimes it’s bad, maybe, I don’t know. It’s a challenging, know, the motivations behind the update requiring all these hardware, you know, having all these hardware requirements is fundamentally about security and trust and like the chain of trust within the device, which is obviously a huge priority, especially for businesses, but to get forced to upgrade your hardware because…
Wes Dobry (07:02.587): You
Chase Snyder (07:22.813): It’s a big cost, it’s a big burden, you know, trade-offs.
Paul Asadoorian (07:27.022): Well, yeah, because you go to enterprises that have, let’s say, 100,000 users. I mean, I remember working for the university, it reminded me of this, and talking to large vendor representatives who were there for networking gear. And they were behind on operating system revisions, whereas we were ahead. I was like, why are you guys still on Windows whatever it was? I had all the different versions, but they were behind. Dude, there’s like hundreds of thousands of people that work there. Like rollouts take time, especially 25 years ago, right? 20 years ago, it took a lot longer than it takes today. We have much better, would you agree Wes, right? Much better tooling to handle these upgrades.
Wes Dobry (08:10.443): Absolutely. Mean, tons better tooling and tons better capabilities. Mean, even back in the day when things like business desktop deployment, which turned into the zero touch installer within SCCM, even back in those days, you had tons of development that you had to do to just simply go and do something like an in-place upgrade. And best practices was you still should collect everything and do it through a deployment network.
Paul Asadoorian (08:22.563): Mm-hmm.
Wes Dobry (08:39.138): Segment and nowadays, know, with fast connected sites and even the ability to go and do things over the internet with Intune, you’re so much better off with the capabilities of doing in-place upgrades.
Paul Asadoorian (08:39.552): Mm-hmm.
Paul Asadoorian (08:48.438): Right, right, yeah.
Paul Asadoorian (08:54.28): Unless your hardware doesn’t support it, right? And so some of the things that you need off the top of my head, the big one was the TPM version 2.0. And the second biggest thing in my mind is at least an eighth gen Intel processor. Is there an AMD equivalent that they’re supporting?
Wes Dobry (09:11.287): Yeah.
Wes Dobry (09:14.707): I’m not certain. I mean, I saw a list of processors for the AMD side as well. So, I mean, there is a limitation on that. I don’t know what it is off the top of my head, but we can circle back to it once I look it up. But the other biggest thing is, and this is going to probably come as a surprise, but the fact that you have to be running UEFI Secure Boot.
Paul Asadoorian (09:18.062): Mm-hmm.
Paul Asadoorian (09:25.39): Mm. Sure.
Paul Asadoorian (09:38.318): Yes, right. So I see a lot of articles that talk about regular, what we call regular BIOS booting without UEFI. It’s not compatibility mode, legacy boot. Thank you. I know it was a brain fart. Legacy boot. And I see articles that talk about both. Like, if your computer doesn’t have UEFI today, I mean, you’re way out of date. I feel like that’s like a moot point at this point.
Wes Dobry (09:50.105): Legacy.
Paul Asadoorian (10:07.05): Most modern system. I mean, I have older systems here in the studio and they all have UEFI. I mean, now we’re going way back, right?
Wes Dobry (10:16.173): Well, I mean, you would think, but I think you would actually be quite surprised. I mean, a lot of enterprises will have replaced hardware, let’s say within the last five years. And I completely agree that those systems would have UEFI on them and likely at the very least TPM 1.2, but doesn’t hit that TPM 2.0 requirement that you mentioned. But a lot of organizations were not prepared. To do secure boot even as recent as a few years ago. So they’ve purposely disabled that because of the difficulties that it’s presented to them for rolling that out.
Paul Asadoorian (10:47.34): Mm-hmm.
Paul Asadoorian (10:52.642): Right. Well, now I’m thinking of you know, appliances, specialty purpose things. You get that old crusty Windows computer that’s running your historian for your PLC and ICS that’s running your MRI machine in a medical environment. Those systems I can see not enabling UEFI or Secure Boot, but also maybe not even having a UEFI BIOS, having legacy BIOS. Yeah.
Wes Dobry (11:07.043): Yeah.
Wes Dobry (11:17.121): Exactly. Yeah, I mean, even the implications of there’s applications out there which use DRM, which replace the bootloader, which requires you to turn off secure boot. People that use gaming applications with DRM that requires you to turn off secure boot for those things too. I mean, the implications here spread massively for the
Paul Asadoorian (11:30.018): Yep. Yep.
Paul Asadoorian (11:37.175): Mm-hmm.
Wes Dobry (11:43.3): Shocks that people are going to encounter with you know saying I’ll try to do the upgrade and all of a sudden why can’t I?
Paul Asadoorian (11:49.88): Right. Question, Windows 11 requires that secure boots enabled or requires that it’s available.
Wes Dobry (11:58.027): Yeah, I believe it requires that it’s enabled at time of the upgrade.
Paul Asadoorian (12:00.832): Interesting. Interest at time of up. I’m sure there’s bypass. I mean, for a lot of things with Windows, there’s bypasses, right? When I looked into it, the, it forces you to create a Microsoft account or have a Microsoft account when you’re installing it. There’s workarounds for that. I used the, what is it called? Knee something. There’s a installer. I meant to put it in the show notes, but there’s a, what is it called? No, it’s, someone’s like name or something like that.
Wes Dobry (12:24.224): NeoWin.
Paul Asadoorian (12:30.142): I’ll put it in the show notes for after the show, but you go to this website. I also think they have a GitHub if you want to do it locally and you fill out this massive form and it’s all the things that you can tune and customize in your Windows installation. Do you want a local account? Yes. Okay, give me your local account. You want to give it a wifi network so it has that? Yes. Do you want to disable, not install all these apps that people call bloatware and you can do that. I you could highly customize. Get rid of lot of the annoying things in Windows. I a lot of people just really want an operating system. I don’t need the weather and advertisements in my start menu. Like you can remove that stuff and then it gives you an XML file that you load on your bootable drive and it will install it with those settings. And I see a lot of maybe companies and individuals taking advantage of this, although enterprises probably have their own process to generate custom installations, right? There’s a lot of different ways to skin that cat.
Wes Dobry (12:59.942): Mm-hmm.
Wes Dobry (13:12.311): Mm-hmm.
Wes Dobry (13:24.534): Yeah, I mean, you’ll see every level of maturity for enterprises or even any organization to do it. And a lot of places that do not use centralized management of operating systems or SCCM, Intune, Autopilot, etc. Will use something like exactly what you mentioned. Or they’ll build their own, you know, Dism scripts to create the installer with the components already removed out of it. And I did check my
Paul Asadoorian (13:50.606): Right, Yeah, so it’s like, go ahead.
Wes Dobry (13:54.803): Sorry, I did check my notes and it says secure boot capable, not secure boot enabled at time of upgrade.
Paul Asadoorian (13:59.692): Okay. Gotcha. Yeah. But so I also think it’s, I’m all over the place. Go from topic to topic, but I think it’s interesting that Microsoft, this is a positive thing. They’re requiring that you have hardware that’s capable of implementing security controls, right? Like secure boot, like tons of things use the TPM. Are they requiring BitLocker or? I wonder if that’s a requirement or just… No, they’re not requiring it, right? Okay.
Wes Dobry (14:31.903): No, Yeah, BitLocker is nowhere in the requirements. But the good part is that if you’re secure boot capable and you have TPM, and especially TPM 2.0, yeah, it’s enabling BitLocker at that point is a no-brainer.
Paul Asadoorian (14:36.417): Okay.
Paul Asadoorian (14:41.208): TPM, BitLocker’s fine. Yeah.
Paul Asadoorian (14:48.032): Right. With the key or passcode or something, is the more secure mode for BitLocker.
Wes Dobry (14:55.191): Correct, yeah. And that’s an interesting conversation in itself about BitLocker best practices. I mean, with TPM 2.0, you’re almost always pretty good if you have upgraded firmware. Now, there were some TPM chips that had some security flaws for physical attack and things like that where you had susceptibility to extract BitLocker decryption keys from them. But I mean, it…
Paul Asadoorian (15:02.062): Right.
Paul Asadoorian (15:22.584): Yep. I saw a couple of those posts with the, I forget what it’s called, the leads that go on the bus, that clip onto the bus. There’s a name for that we talked about on the show, yeah.
Wes Dobry (15:27.51): Mm-hmm.
Wes Dobry (15:32.791): Yeah, like the you could use like things like pogo pins and such to do that. Yep, and And in those situations, I mean, you know I saw one where they used a Raspberry Pi as basically the decrypter of it and You know novel approach But in the grand scheme of things if you’re using, know modern hardware with modern configuration You don’t really have to worry about that, but I will always recommend a boot pin
Paul Asadoorian (15:37.602): Pogo pin. Yep.
Paul Asadoorian (15:45.142): Yes.
Paul Asadoorian (15:57.933): Mm.
Wes Dobry (16:01.258): That a user knows or an integration with some kind of a PAM solution so that you can do centralized authentication with Preboot Auth. That’s really the best way to go about it. Now,
Paul Asadoorian (16:07.574): Yes. Agreed.
Paul Asadoorian (16:13.198): But this is setting up organizations to have better security. I’m sure the decision to use an eighth gen Intel, for example, and I didn’t look into the details, I’m sure there’s security features in that processor that they require.
Wes Dobry (16:28.352): So there’s security features that are in there, but another big reason why they’re pushing for these newer processors is some of the acceleration for things like integration with Copilot and the AI integration. So when we look at the list of processors, there’s actually a separate list for what can actually allow you to use Copilot in Microsoft branded applications. And those have kind of a minimum requirement for
Paul Asadoorian (16:40.974): Okay, yep.
Paul Asadoorian (16:54.094): I got you.
Wes Dobry (16:58.207): Basically being either having specific amounts of processing capability within the graphics card or in the processor itself or the system on chip to support effective local model execution.
Paul Asadoorian (17:09.006): Mm-hmm.
Paul Asadoorian (17:14.926): So what tools exist today to help manage this migration? I know as an individual user, my Surface Pro that’s sitting here that I need to upgrade from 10 to 11, and the hardware does support it. So I’m like, cool, I can get more usage out of my laptop. But it tells me, it did the analysis all on its own and told me, hey, your hardware is good for Windows 11. But. If I’m an enterprise with hundreds of thousands of computers, how do I manage this upgrade?
Wes Dobry (17:48.572): Yeah, so that’s a really good question. I mean, the first stop is looking at your CMDB for some of the things that are within the device. Like, you know, what CPU and how much RAM and how much disk space do you have in it? But as you get into more of the particulars, like, does it, is it running on a GP, GPT based
Paul Asadoorian (18:00.782): Mm-hmm.
Wes Dobry (18:10.952): Disk format and is TPM 2.0 installed, configured and available? And of course, you know, is the system secure boot capable? Those are the questions where you’re not going to get those without doing some custom scripts and coding to get out of your CMDB, for example. So, you know, that’s where I’ll plug the branding that Chase is wearing over here, where, you know, that’s one of the key things that we do at Eclipseum is that
Paul Asadoorian (18:16.034): Mm-hmm.
Paul Asadoorian (18:26.782): Mm-hmm. Mm-hmm.
Paul Asadoorian (18:36.29): Right.
Wes Dobry (18:40.593): Actually understanding what the hardware is capable and does is one of the things that we help organizations with today.
Paul Asadoorian (18:46.732): Yeah, because otherwise, PowerShell can answer most of your questions from what I’ve seen. In Linux as well, we have utilities and scripting languages that we could script it to describe the hardware to a certain extent. It’s not going to give you the full mapping of UEFI that our product gives you, but you can script your way to doing some of it. But then you also, then you have to build some application around that.
Wes Dobry (18:53.0): Mm-hmm. Yep.
Wes Dobry (18:59.636): You
Paul Asadoorian (19:15.278): Like I pulled data from 100,000 systems, how do I then parse through that data and display it in a format where I can then start assigning things and taking action and controlling it? That’s where our platform really, that’s why I like using our platform is I can look at all my systems here in the studio. Can go, wait, which ones are this, which ones are that and filter and sort.
Wes Dobry (19:21.14): Mm-hmm. Mm-hmm.
Chase Snyder (19:36.039): So if what happens, let’s wargame it out, an organization does not do this mapping work ahead of time or doesn’t do it thoroughly enough to identify the systems, know, 100,000 is the number you threw out, totally reasonable for a large enterprise. Some percentage of those do not support or cannot be upgraded to Windows 11 and they don’t figure that out ahead of time. They try to start the Windows 11 rollout. And at that time they like run into a swath of their systems that can’t support the update. What happens, you know, from your experience in this realm, both of you, what’s that like organizationally? What’s the pain? How do you get over that once you reach that point where you’re in the middle of the rollout and you’re like, this whole area can’t take it?
Paul Asadoorian (20:29.986): Yeah, then you’re to go to your manager and ask for more money to pay Microsoft to continue support for Windows 10. And I think that some of the cautionary tale that we have in this show, in this episode is don’t be in that situation. Wes, go ahead.
Wes Dobry (20:42.312): Yeah, yeah, and you know, it ends up resulting in simply more failures. You you typically are going to assume you have X number of failures as part of the process, but if you, anything you don’t validate prior and already know about and have 100 % confidence that the devices that you have in that first round of in-place upgrades is going to go successfully, you end up with exceptions to the process. And those exceptions are costly. Because that means you’ve to roll a truck, you’ve got to get a desk-side tech to go out and get it. With the pervasiveness of working from home now, that’s okay, I need this employee to come into the office. In that case, you’re likely going to do an exchange of hardware. You’re not going to ask for them to sit there while you do it in place. You’re going to do a data migration or ask them to move everything to a network share. And then when they come into the office, you’re just going to go, here’s your new laptop, let me get your old one.
Paul Asadoorian (21:17.005): Yep.
Paul Asadoorian (21:32.994): Mm-hmm.
Paul Asadoorian (21:40.418): Mm-hmm.
Wes Dobry (21:41.283): And when you start talking about needing to physically swap hardware to make the process faster, your costs of the upgrade go through the roof. Because now you’re buying new hardware, you may have supply chain challenges which are causing you to fight with everyone else going through this exact same process before October as well. And now you’ve slowed things down. And to Paul’s point, If you’re not already midway through this process now, you’re not hitting October before you’re done and you now either go unsupported on Windows 10 with no patches or you’re paying Microsoft for custom support and extended support.
Paul Asadoorian (22:24.354): Wes, is there a way for Windows admins to query the Active Directory and say, hey, tell me about all the Windows 10 systems I have and then tell me which ones don’t support a Windows 11 upgrade? I would imagine, go ahead.
Chase Snyder (22:24.669): I didn’t even.
Wes Dobry (22:36.925): So not to my knowledge, nothing that’s going to be effective in that regard. I mean, you can get a list of Windows 10 systems out there and what patch level they’re on to an extent based upon versioning information. Anything beyond that is going to come from some kind of a CMDB like SCCM, BigFix and others.
Paul Asadoorian (22:55.63): Wow.
Paul Asadoorian (23:00.334): Interesting. For your Microsoft, you need to have SCCM to get better analytics about who supports what.
Wes Dobry (23:08.016): Yeah, absolutely. That’s solely, and I say SCCM, that’s also Intune in the same grain. So I come from the SMS world, so back from the SMS three days. So, you know, my Windows, yeah, yeah.
Paul Asadoorian (23:13.878): Mm-hmm.
Paul Asadoorian (23:20.834): Yeah, was systems management service or something like that. Simpsons management software or something like that, yeah. And then it was SCCM and now it’s in, Intune replaced it?
Wes Dobry (23:26.532): Yeah, way back in the day.
Wes Dobry (23:31.57): So you now have Microsoft Endpoint Manager, which is kind of your parallel to Intune. And Intune is kind of your cloud-based, mostly cloud-based management platform. And Endpoint Manager can be used mostly for on-prem type stuff. They kind of, the line blurs these days. And what you end up finding most organizations do is leveraging Intune plus
Paul Asadoorian (23:50.318): I see.
Wes Dobry (23:59.634): Plus autopilot scripts for a majority of their modern architecture. But a lot of the older stuff, like let’s say the Windows 7 that may still exist out there, they’re likely using SCCM or Microsoft Endpoint Manager.
Paul Asadoorian (24:02.498): Mm-hmm.
Paul Asadoorian (24:10.638): Mm-hmm.
Paul Asadoorian (24:14.05): Mm. Gotcha.
Chase Snyder (24:16.813): Yeah, I was gonna bring up the amount of Windows 7 that’s almost certainly still out there. In my past I’ve worked in cybersecurity for operational technology like industrial control systems where they’ve got, you know, a workstation out on like an oil rig or something. It’s like just sort of, they got all kinds of devices out there that are just physically really hard to get out there and replace and also are supporting all this legacy hardware and like, you know. Operational technology widgets moving parts that any sort of disruption or downtime to do some sort of OS update is extremely costly and so they just don’t you know don’t do it. Ditto government a lot of Windows 7 out there in the government there’s yeah countless countless tales of just
Wes Dobry (24:59.057): No.
Chase Snyder (25:10.909): Basically impossible to update sort of linchpin systems or devices where it’s like, if we update this thing, we don’t know what else is going to break and it’s just not worth it. And I think I read that Windows 7 still continued to get some sort of updates, like definitions updates or something up to up until like last year. So like 13 years or so past end of life for Windows 7.
Wes Dobry (25:30.661): Mm-hmm.
Paul Asadoorian (25:31.34): Yeah.
Chase Snyder (25:36.603): And there were still some amount of updates to it coming out of Microsoft.
Paul Asadoorian (25:42.22): Yeah, there’s going to be a lot of legacy. I mean, basically the end of life of Windows 10 instantly in October, there’s going to be a plethora of now true legacy systems out there.
Chase Snyder (25:44.988): Yeah.
Chase Snyder (25:53.499): Yeah, Wes, I didn’t even think of that situation that you brought up of the supply chain bottleneck, the potential supply chain bottleneck, as more and more organizations get close to this deadline and realize the pickle that they’re in. And so they’re like any sort of services that they might try to tap into to make this easier, any sort of hardware purchases that they need to make, especially given the current global supply chain dynamics for computer parts and manufacturing and all of that stuff. Some some organizations could end up in.
Wes Dobry (25:54.32): You
Paul Asadoorian (26:02.695): Yeah.
Wes Dobry (26:15.993): Mm-hmm.
Paul Asadoorian (26:16.642): Yeah.
Chase Snyder (26:23.047): Difficult backlogged state for getting their needs met.
Paul Asadoorian (26:26.574): But I like your point, Chase, that you’re not the only organization that’s going to have this problem. Several thousands of organizations may wait till the last minute and then go, I need to all buy hardware all at the same time. And that’s going to put even more stress on the supply chain.
Wes Dobry (26:45.328): Yeah, and let’s not even take into account the cost of potential political ramifications related to that too, where incoming hardware may have additional costs related to it as well.
Paul Asadoorian (26:57.068): Right, Yeah, so get your hardware. Mean, people should have already been planning for this and getting hardware, right?
Wes Dobry (27:04.752): You know. That that’s the fun part and fun not fun part of Windows migrations is as technologists were saying, oh, you should actually be early in the adoption cycle to alleviate a lot of these risks and organizations say, well, when do we have to do it by? And you’re like, oh, it’s October of next year. And then they’re like, oh, well, we can do it next year. And then you suddenly get to a point where and it happens everywhere.
Paul Asadoorian (27:24.93): Yeah.
Paul Asadoorian (27:28.3): Yeah, it’s a next year problem.
Wes Dobry (27:34.101): In every situation, you have to rush through it. And that’s the part where that whole 90-10 approach of 90 % plan and 10 % of it’s actually the execution of it comes into play, where you get about 10 % of the time to plan and 90 % to execute, but your costs are four times as much as what your initial estimates were.
Paul Asadoorian (27:54.946): Yeah. For me, mean, being a security practitioner, there’s a security driver here that it may be an unintended consequence. But if I think back before UEFI, back before eighth gen Intel processors, and then we think about the number of security features and enhancements that have been introduced since then, you’re in a much better security posture with modern hardware configured properly with UEFI, secure boot. BitLocker like that, TPM 2.0, all of that puts you in a much better security posture. In my head, I’m just thinking of all of the other protections that UEFI affords you, other than secure boot, right? You’ve got boot guard and all of the other various features that Intel and others have introduced that you can take advantage of by just upgrading your hardware.
Wes Dobry (28:30.292): Mm-hmm. Mm-hmm.
Wes Dobry (28:47.148): Absolutely. Yeah, I mean, even if you take it a step further and look at things like secure core PCs and some of those feature sets that come on those devices where, you know, you’re not even using the publicly available UEFI root of trust, you’re using a defined and custom root of trust on those devices, you end up much more secure because you’re actually ended up locking out
Paul Asadoorian (29:07.01): Mm-hmm.
Wes Dobry (29:13.519): In some regards locking out non-Microsoft operating systems. But if your organization doesn’t need compatibility with Red Hat and Ubuntu, for example, that makes a whole lot more sense because then that saves you from a lot of the threats like boot locker threats or excuse me, boot loader threats. Yeah, and we’ve talked about those before with things like Black Lotus and Boot Kitty, for example.
Paul Asadoorian (29:21.176): Right.
Paul Asadoorian (29:33.366): Yeah. Yeah. Bootkits. Yeah.
Paul Asadoorian (29:42.572): Yeah, I was just we were talking about this recently because we’ve got some enhancements coming up on this boot loaders and configuration for secure boot. And if your windows only, if you lock yourself into that ecosystem, that means someone if secure boots enabled can’t bring unless they have a bypass can’t bring another boot loader like a Linux boot like a vulnerable grub and introduce it into the system, which I think helps reduce your attack surface.
Wes Dobry (30:05.865): Mm-hmm.
Wes Dobry (30:09.967): Yeah.
Paul Asadoorian (30:12.278): If it’s not going to be dual boot and look a lot of enterprises probably don’t have dual booting Windows and Linux on there. I mean you shouldn’t allow that like take you know pick your poison when you want Windows or Linux on it and your Linux will support secure. I Ubuntu is I think it’s been great about supporting secure boot so. But yeah, you can reduce your attack surface.
Wes Dobry (30:31.343): And you know, actually said something there that was a little interesting is enterprises are likely not dual booting. And you’re absolutely correct because nowadays you’re not going to dual boot. You’re going to throw a virtual machine or throw some kind of virtualization and virtualization subsystem on that device, which I actually think having virtualization based
Paul Asadoorian (30:48.974): Mm-hmm.
Wes Dobry (30:56.75): Security or VBS enabled. Think that may also be a prerequisite to Windows 11, but I’m going to double check that while I think about it.
Paul Asadoorian (31:01.997): Yeah. Yeah, I think you might be right there.
Chase Snyder (31:08.049): Was I was glad you mentioned Black Lotus a minute ago because I feel there’s a related topic we talked about a little bit with this Windows 11 update, or not related, but sort of a similar situation where, okay, I guess background, Black Lotus was a UEFI boot kit tracked as CVE 2023-24932, so. UEFI boot kit from a couple years ago that made a pretty big splash I think at the time it was like a very serious serious issue and I thought I Started looking into it recently for some reason to realize that Microsoft has published an enterprise enterprise deployment guidance for the Black Lotus UEFI boot kit this year In February of 2025, they published this enterprise deployment guidance. And it basically says, we’re not going to roll this out to enterprises ourselves. We’re just publishing this guidance so you can control your deployment plan and your timing of deployments. And it specifically references that because there is a huge combination of device hardware and firmware, and Microsoft is unable to test all of those combinations, you need to test the representative devices in your environment before deploying them broadly. You know doing doing the mitigation means that you need to add the new Windows UEFI CA 2023 certificate and untrust the Microsoft Windows Production PCA 2011 certificate and it’s essentially gonna be an irreversible change. You’re not once once you do that. You can’t go back and and still use secure boot and I’ve been seeing discussion online on like reddit and spiceworks and stuff. That references an enforcement period that Microsoft is maybe gonna have, but they haven’t put a date on it yet, where essentially it’s like, if you haven’t deployed these black lotus mitigations yet, something bad happens. I don’t know, your devices get bricks. They blanket revoke the 2011 cert, and then anybody that still has that is SOL. I’m not exactly sure.
Paul Asadoorian (33:15.63): I still think it’s recoverable. What I think I’m hearing is that you need to swap out your Keck or your key exchange key. Yeah, that’s what they’re, yeah. That chain, that’s in the rest of right? That’s part of your chain of trust for secure boot, right? Got your PK, your platform key, got your key exchange key, and then you’ve got two variables, your DB, which is the allow list, and then you’ve got your DBX, which is a revocation list.
Wes Dobry (33:22.285): Yeah, I mean, that’s effectively exactly what it is, Paul.
Paul Asadoorian (33:43.52): And all that comes into play when there’s a piece of software that’s involved in booting your computer that has a vulnerability has to be revoked, right? Or if, I don’t know why they’re swapping out the CAC and not using the DB. That’s interesting. But there was a reason for that. Feel like Microsoft explained it. Do you remember what that was? Their certificate was expired. Was it their certificate expiring? I think that’s what it was.
Wes Dobry (34:01.965): Yeah, well, mean, part of the challenge. Yeah, it’s one, it’s the certificates expiring and two, is maneuvering towards the opportunity to add more things to DBX. So, correct. And part of the reasoning behind that is that there’s now so many flavors of Black Lotus that the DBX is getting rather large.
Paul Asadoorian (34:19.384): Do S to do SBAT.
Paul Asadoorian (34:29.966): Thank you.
Wes Dobry (34:32.224): You know, at this point, their typical next step is to revoke the key that they’re using above. And when you get to the point where they’re at, where they need to revoke, you’re now revoking legitimate bootloaders and boot components that are going to cause a lot of systems to be bricked. Now, I mean, to your point, yeah, is it recoverable? Sure. Is it recoverable by the common person? Absolutely not.
Paul Asadoorian (34:38.625): Mm-hmm.
Paul Asadoorian (34:46.719): Mm-hmm.
Paul Asadoorian (34:55.83): No, because if you bork it good enough, that’s a technical term, you have to go into your UEFI BIOS and manipulate the keys through the BIOS menu. My first thing would be how to recover it.
Wes Dobry (35:08.064): Yeah, you know what I would tell anybody affected by that? Reinstall. And that’s the only thing that’s going to really fix it for you.
Paul Asadoorian (35:11.832): Yeah. Well, it though? Would reinstallation update your keys?
Wes Dobry (35:20.106): Yeah, it would attempt to go and re-add.
Paul Asadoorian (35:22.42): And re- yeah, I gotcha. It’ll re-add the keg, right? Because the key exchange key signs the DB and the DBX, because this is a root of trust. So it’s safeguarding those allow lists and deny lists on your system, which are all stored as UEFI variables, right? They’re just data in a variable that’s stored on the spy flash on your computer in UEFI. But there’s protections, obviously. Around those, but there’s also facilities to update them. Like when we talk about you update them from your operating system. So those variables can be manipulated. There’s not a boot service, not a boot services variable. Those can’t be once the operating system is loaded, modified. It’s a, what do they call the ones that can’t be modified? West, I forget the term. It’s not boot services, user service. There’s another name for that type of variable.
Wes Dobry (36:09.355): I don’t know after that my head
Paul Asadoorian (36:16.066): Yeah, so Chase, this is why people have a lot of questions because it’s, Wes and I have been doing this for a long time and like studied it and our product is centered around this stuff. But if you haven’t done secure boot or updated all this stuff, can be very, it gets confusing. It’s very confusing to be quite honest.
Wes Dobry (36:16.171): Yeah.
Wes Dobry (36:34.188): Well, you know, it’s also part of this is a bit of a, I’ll call it a chicken and egg problem is, so what do you revoke? And, know, do you end up revoking the things that like black Lotus, for example, or do you revoke the key that there that is signing all the things that, that, you know, is the black Lotus is using and
Paul Asadoorian (36:53.048): Yeah.
Wes Dobry (36:57.75): You know, we actually, a few years back now, we actually found another major manufacturer that was including a utility that was, we’ll say, less than secure in the boot process. You know, was one of those things where it was found to be, effectively, think of it like a backdoor that was added in that was easily susceptible to do nefarious things within the operating system through this utility. But what ended up happening is you had this deployed on
Paul Asadoorian (37:15.754): Mm-hmm.
Wes Dobry (37:25.835): You know, let’s just say thousands of copies of firmware that existed out there for various, various models. And the question is, is do you revoke each one of those and fill dbx or do you revoke the key that’s also been used for legitimate things?
Paul Asadoorian (37:40.844): Mm-hmm.
Wes Dobry (37:44.147): And then now end up bricking a whole bunch of devices out there? Or do you go the route that Chase is mentioning where you have a set long duration where you’re going to go and do this and you’re going to do things more securely, but you have a period that you work up towards. And hopefully by that time they’ve upgraded to Windows 11, they’ve installed Windows updates, which has stepped them forward in this process.
Paul Asadoorian (37:47.406): Right.
Paul Asadoorian (37:59.246): Mm-hmm.
Wes Dobry (38:08.031): But there’s still going to be that point where you’ve got to flip that switch and there’s going to be that small percentage that’s affected. And, you know, these are also probably the same types of folks that are, you know, either not using secure boots or not prepared for windows upgrades. And, you know, all of this comes together to, if you’re doing things right, you’re probably going to have an easy job. If you’re, if you’re not doing things right,
Paul Asadoorian (38:14.188): Yep. Right.
Paul Asadoorian (38:35.406): Mm-hmm.
Wes Dobry (38:37.36): You’re in a world of pain that you just don’t even know about just yet.
Paul Asadoorian (38:41.326): Yeah, and our latest rounds of enhancements will help you with this in that I worked with research and engineering on many of these features and some of them are specific to Secure Boot in that it will now, our product will now tell you, hey, you’ve got a bootloader, let’s say, that is in a DBX update revocation list, which you haven’t applied. Right? That’s something you want to know about. Whereas maybe previously we would have said, well, your DBX is out of date, which is enough, but then it’s kind of you’re on your own to go, well, what bootloaders do I have? And will that update invalidate my bootloader, which means the system’s not going to with secure boot. Secure boot is going to stop it. So there’s all these interesting scenarios that we’ve enhanced to tell you, give you some more of this telemetry because it is complex and there’s a lot of just a lot of different scenarios that you should be aware of that some could be suspicious. Some are like, hey, you’ve got an EFI shell in your boot order. Like that’s kind of suspicious, right? Or you’re running Windows, it’s not dual boot. And one of your boot loaders is Grub, it’s a Linux boot loader. Why is that there? That in and of itself is suspicious. Could be completely legitimate, but more than likely suspicious. But going back to the upgrade, this is a great reason.
Chase Snyder (40:04.636): Yeah.
Paul Asadoorian (40:08.086): If your hardware is older and needs to be refreshed, sometimes it might be easy to just put new hardware. Got a brand new system. All the keys are there. Windows 11 is installed and you’re off to the races. Maybe resource wise, in a lot of cases, it could be better to just issue new hardware.
Chase Snyder (40:27.943): Start fresh. That’s really a step beyond turn it off, turn it on again. Turn it off, throw it away, start up a new one.
Paul Asadoorian (40:31.937): Yeah.
Wes Dobry (40:36.265): So, you know, it’s always interesting when you go down the path of how do you start fresh and, you know, this day and age, there’s so many opportunities where if you do end up starting fresh, where you can take advantage of a lot of newer technology too. You know, I mentioned autopilot a few times today and, you know, the organizations and the enterprises I talked to that are doing compelling work from home.
Paul Asadoorian (40:52.557): Yes. Mm-hmm.
Wes Dobry (41:04.249): Leverage things like autopilot where they’re drop shipping from a third party using autopilot to do initial provisioning of their operating system and features and their security tooling before they ever give access to any of their corporate data. So they basically sign in with an Azure AD and that’s the first check is the, okay, we’ve got your credentials. We know what your machine should look like. We’re going to start provisioning it.
Paul Asadoorian (41:32.812): Mm-hmm.
Wes Dobry (41:33.809): When they come out the other end of it, they’ve got all their applications, their data, their features and functionalities, and they’ve been doing posture checking and conditional access policies to actually do things securely as well.
Paul Asadoorian (41:46.146): Right.
Paul Asadoorian (41:49.518): I noticed that when I was rebuilding systems a couple of years ago, that during the rebuild, it reached out to the Azure AD domain and put certain restrictions and checks, which from a security perspective is great. Usability in that case wasn’t so great, but you know, there’s that.
Wes Dobry (42:08.553): Well, that’s actually one of the cool things that I’ve been doing with some of our more bleeding edge customers is a way that I would equate it to is we’re actually doing posture checking on the equipment, making sure that those foundational security controls are what you would expect prior to ever actually enabling that device for reaching any privileged data.
Paul Asadoorian (42:34.87): Mm-hmm.
Wes Dobry (42:35.845): I always refer to this as taking zero trust beyond the network. Let’s actually build trust in that equipment and do these validation steps before we ever let that device touch anything we don’t want getting into the wrong hands.
Paul Asadoorian (42:51.374): Yeah, and it’s great because it needs to go beyond just is secure boot enabled because secure boot can be enabled as we were talking about earlier, but not necessarily 100 % effective. Well, I guess it’s never 100 % effective, but is your secure boot vulnerable to known exploits and vulnerabilities, right? Because your revocation list isn’t up to date, that means an attacker can subvert secure boot. And there’s lots of other secure boot style bypasses that
Wes Dobry (43:10.29): Yeah.
Paul Asadoorian (43:18.946): Due to configuration, older software firmware could have vulnerabilities.
Wes Dobry (43:22.759): Yeah, same with BitLocker. Know, BitLocker is one of those fun things where you can have it in a conditional access policy that you have to have BitLocker enabled. And it’s funny because you can enable BitLocker and never encrypt anything. And there are so many tools out there that will go and see that BitLocker is enabled and never actually check that something is actually secured.
Paul Asadoorian (43:25.826): Yeah.
Paul Asadoorian (43:39.212): Interesting.
Paul Asadoorian (43:44.748): Mm-hmm.
Wes Dobry (43:49.477): By BitLocker. So that was one of the tricks I learned a few years back was you could simply do that, never encrypt anything, and you could easily pull data off that system. But all the MDM and MAM solutions out there were like, yeah, BitLocker’s on, BitLocker’s good. And it always cracked me up.
Paul Asadoorian (44:08.332): Yeah, it’s similar when I’m updating Linux systems and some distributions are like, hey, here’s a new version of grub fixes a whole bunch of vulnerabilities that happened in February, those were disclosed. And some are like, you have to do an extra step and install it. So as an administrator or user, if you’re not paying attention, yeah, you got the new grub. And a vulnerability scanner may look at all the packages and say, you got the latest grub package, like you’re good. But it hasn’t been installed on the EFI partition and it’s not active. You still have older bootloaders there. And that’s a condition that we’re working on checking as well.
Wes Dobry (44:46.728): Yeah, I I was just going to say, mean, that’s part of it is that when we’re looking at effective security solutions, it’s not just about checking for a version. It’s actually about checking to make sure the control is actually in place and functioning.
Chase Snyder (44:48.079): Are you? Go ahead.
Paul Asadoorian (45:02.84): Right.
Chase Snyder (45:04.774): Yeah, there’s sort of overlapping initiatives across third party risk management and cybersecurity posture management and vulnerability management. I feel like supply chain security, but being applied to your hardware and your vendors, not your own application development or CICD processes. And I feel like it’s gonna have to… Third party risk management right now is at this place where it’s like a lot of questionnaires and you just sort of let your vendors tell you their security posture and that is just not gonna cut it. There has to be sort of an accountability back to them where if you are A, checking stuff before it comes in. I’m curious too whether you see that becoming a more and more common requirement to do the kind of sort of acceptance testing that you’re talking about where you before ever connecting
Wes Dobry (45:37.159): Mm-hmm.
Paul Asadoorian (45:37.602): Yeah.
Chase Snyder (45:59.325): New gear, new devices into the environment and giving it access to sensitive data, you validate that the security controls are in place and that it has the versions and components that you expect. Is that becoming more and more common for folks to require that or is that still kind of high maturity organizations that are doing that? Because the 2025 DBR, the Data Breach Investigations Report, showed that the involvement, the number of breaches involving
Wes Dobry (46:20.775): Yeah.
Chase Snyder (46:29.135): A third party doubled since the last report. So, and that I think would include, I gotta read the fine print, I guess, but I think that would include stuff like bringing in devices with, you know, either a lack of security controls or just random vulnerabilities into your environment from some vendor that then that’s the avenue by which you get pwned. You would think that with that sort of a curve of the amount that third parties are the vector of a breach. You would start doing a little more of that validation and acceptance testing before you even plug it in.
Wes Dobry (47:04.198): Yeah, mean, it’s really all of this boils down to what Paul mentioned earlier around established route of trust and then validating every layer of that trust to ensure that all the controls that you have in place are actually effective. And most organizations ignore anything beyond, you know, like an EDR. They might throw their EDR on there. And their group policies coming from Active Directory and go, okay, we have all of our controls in place, everything’s golden and grand. As you get more mature, the more mature the organization, the more they validate these controls for effectiveness. And that’s where you start getting into things like zero trust based security segmentation, micro segmentation on the endpoint itself. You start getting into using things like software-defined networks to actually get these devices access to things over the internet or on corporate networks even. And you started to move down that stack a little bit. Where we’re still not really seeing organizations do is that that last piece that you were talking about is the validation of one, the OS security controls like secure boot, the boot loader and the boot sequence of the device into the hardware validation.
Paul Asadoorian (48:29.902): Mm.
Wes Dobry (48:30.659): You know, there’s actually an executive order that’s helping with this and I don’t know the EO number off the top of my head, but it was around building security in UEFI and firmware itself. And that’s helping the industry move towards all firmware being signed and providing provenance as part of that. We’re not really seeing manufacturers jump on board with this tremendously, but we’re starting to see the ship steer towards devices having the ability to, for you to validate the ATTA stations that they’re doing. And you know, today that’s largely within the TPM, but you can also see things out there like the NSA’s, HIMS and HIRS and PACOR for a generation of things like platform certificates.
Paul Asadoorian (49:12.6): Mm-hmm.
Wes Dobry (49:23.481): That allow you to independently validate what that system contains from a hardware and firmware perspective. And this is your traditional thinking of an SBOM, so like a Cyclone DX or SPDX, but for hardware and firmware. And one of the things that I evangelize heavily when I talk to resellers, VARs, enterprises, is that let’s add in that check.
Paul Asadoorian (49:39.35): Yeah.
Wes Dobry (49:50.638): Where you actually validate that what you purchased is actually what you’ve received, and there’s nothing surprising within it. So, you know, that means a step in that process of taking that measurement, not just trusting the TPM, because why should I trust a device that’s a black box doing a measurement and spitting out a hash? Like, why should I ever even trust that too? But doing an independent verification and validation of that.
Paul Asadoorian (49:55.598): Mm-hmm.
Paul Asadoorian (50:11.566): Mm.
Wes Dobry (50:19.134): And so to your point, Chase, you know, we will see more enterprises doing that. And what I would like to see is more OEMs coming to the table to say, here’s your device with a cryptographically verifiable platform certificate on it that you can use a third party verifier like Eclipseum to validate that against the hardware that you’ve received and say this is as built from the factory.
Paul Asadoorian (50:46.37): Yeah, that’s what we’re striving for. Think it’s interesting this whole attack surface before the operating system, I hope gets more attention, right? And that’s really the reason why we’re having this conversation and the real danger is attackers that will prey upon and live in the attack surface that exists before the operating system. And it’s hard to talk about that without going into technical details. Maybe it’s just me and my brain, right? But there’s… Whole bunch of software and firmware and hardware that is involved before your operating system loads. And I think that similar to how attackers are going after network and security appliances, because there’s low visibility, there’s vulnerabilities there, they’re going to go after this attack surface as well on your PC servers and laptops. Because as the operating system tightens down, although, you know, every day there’s a new bypass, for EDR or Microsoft Defender, right? But as that gets more difficult, they’re gonna move to this attack surface, much like they’ve moved to network and security appliances.
Wes Dobry (51:54.372): Yeah, you’re 100 % correct. And you know, it’s all about areas of opportunity as they move up in the supply chain. The rigor of us validating that everything is copacetic with what we’ve received becomes more important. You know, the opportunity of a bad actor getting into something like a major manufacturer and either embedding a vulnerability that only they know about, or even potentially all the way to the extent something like a hardware flaw.
Paul Asadoorian (52:09.452): Mm.
Wes Dobry (52:24.152): You know, there was a while back where Infineon 1.2, TPM 1.2s were around that actually had a hardware security flaw in them. You know, we actually received some servers just recently where they were outputting PCR zero as blank zero. And that TPM was functioning as expected to the operating system. And, you know, these are situations where
Paul Asadoorian (52:24.427): Mm-hmm.
Wes Dobry (52:52.927): If you’re a user and you’re not doing the validation of it, this would slide right through your process and everything would look perfect on that system. And the foundational security components are completely flawed, allowing an attacker to do something nefarious. I don’t know if you guys ever heard of the other thing that we discovered where we had a major server manufacturer that never was doing the Intel end of manufacturing process. They weren’t burning out the fuses at the end to turn on security controls. So
Paul Asadoorian (53:17.175): Yep.
Wes Dobry (53:22.048): You know, that device was sitting in the hands of users with, you know, SMM open and there were no controls in there to go from the OS into the hardware and the firmware itself. And that happened on a significant number of servers that were out in production users’ hands and they never knew about it.
Paul Asadoorian (53:28.046): Mm-hmm.
Paul Asadoorian (53:39.724): Yeah. And that’s not something you can fix on your own. You need a patch. There’s no work around for a lot of these in this attack surface, right? Which makes it, I think, even scarier is that you’re solely relying on your supply chain to fix that. When it comes to software, it’s more malleable, right? You can have alternatives like, my Chrome is vulnerable. I can use Firefox, right? You don’t have that when we’re talking about firmware, hardware, and fuses and things like that. Fuses are a one-time thing. Once it’s fused, that’s it.
Wes Dobry (53:46.37): Exactly.
Wes Dobry (54:10.342): Yeah, and I was just going to say, I mean, this goes back to Chase’s point where we’re starting to see a merging of things like third party risk management, supply chain risk management, or my favorite term, the C-SCRIM, cyber supply chain risk management, where the overall policies and controls that we’re putting in as an organization
Chase Snyder (54:10.661): And you can have like… Go ahead.
Wes Dobry (54:34.945): Are not actually effective in a lot of regards where we now need to have multiple avenues of it, like saying, I’m validating that my third party supplier is preparing patches on a set timeframe. The second part of that is that they’re actually applying the patches in a set timeframe and validating against it, for example. So we’re going to continue to see that maturation in organizations.
Paul Asadoorian (54:53.101): Mm.
Paul Asadoorian (55:03.982): Fantastic, I think it’s a great closing note. Thank you, Chase. Thank you, Wes, for joining me today. It a great discussion. Thanks everyone for listening and watching. That concludes this edition of Below the Surface. We’ll see you next time.
Chase Snyder (55:16.893): Thanks, Paul. Thanks, Wes.
