BTS #66 - Beyond the Label: The Truth About Hardware Trust
In this episode of Below the Surface, host Paul Asadoorian is joined by co-hosts Larry Pesci, Joshua Marpet, and Vlad Babkin to delve into the complexities of hardware supply chain security. The discussion is sparked by a presentation from Andrew ‘Bunny’ Wong at Black Hat Asia, which raised critical questions about how we can trust the silicon in our devices. The conversation explores the challenges of validating hardware components, the potential for backdoors in devices, and the implications of counterfeit components in the supply chain. The hosts share anecdotes and insights about their experiences with hardware security, emphasizing the need for independent testing and the importance of understanding the provenance of hardware components.
Subscribe
Transcript
Paul Asadoorian (02:15.944): Welcome to Below the Surface. It’s episode 66, recorded Friday, January 9th, 2025. I’m your host, Paul Asadoorian, joined by my coworker Vlad Babkin. Vlad, welcome.
Vlad (02:31.315): Hello.
Paul Asadoorian (02:33.337): Many Security Weekly listeners will recognize our next co-hosts. Mr. Larry Pesci is here with us. Larry, welcome.
Larry (02:42.333): Hey everybody, thanks for watching.
Paul Asadoorian (02:43.954): Mr. Josh Marpet is here with us.
Joshua Marpet (02:46.087): How are you doing?
Paul Asadoorian (02:48.084): Doing great. You guys have been on the show before, so no strangers here. Quick announcement before we get started—Below the Surface listeners can learn more about Eclypsium by visiting eclypsium.com/go. You’ll find the ultimate guide to supply chain security, an on-demand webinar, and a customer case study with DigitalOcean.
Joshua Marpet (02:50.241): Yes.
Larry (02:52.573): Just not necessarily at the same time.
Paul Asadoorian (03:17.51): And a customer case study with DigitalOcean. If you’re interested in seeing our product in action, you can sign up for a demo at Eclypsium.com/go. So what prompted this discussion was a while back, Andrew aka Bunny Wong gave a presentation at Black Hat Asia and I heard some other podcasters talking about it. The talk was basically about how do we validate and move towards being able to trust
Joshua Marpet (03:40.268): Thank you.
Paul Asadoorian (03:46.982): All of the hardware that’s in all of the devices that we use every single day in terms of computing. So I was unable to find a copy of that presentation somewhere. I really wanted to see this talk. He ended up giving it again at hardware.io in the Netherlands, I believe, and they posted it to YouTube. So I’ve been watching it and calling a lot of stuff from it and brought together
Joshua Marpet (03:53.643): Okay. Okay.
Paul Asadoorian (04:16.86): All of us today to kind of talk about the problem of how do we trust the chips beyond, you know, traditionally what I’m doing when I look at hardware mostly is identifying which chips are there so that I can conduct further security research. So I trust the chips by reading the label. However, I don’t know what’s inside those chips.
Joshua Marpet (04:38.392): Okay.
Paul Asadoorian (04:41.736): Which is kind of a gateway into what our topic today is.
Joshua Marpet (04:45.003): Wait, Paul, you’re saying that people lie on the internet and on labels? No.
Paul Asadoorian (04:51.348): Right. Well, and I have the same concerns with any kind of bill of materials. I’m like, what if someone’s lying? By getting a… go ahead, Larry.
Joshua Marpet (04:58.592): And that’s the thing. You’ve got software bill of materials, hardware bill of materials, cryptographic bill of materials, compliance bill, governance bill of materials. I don’t care what initial is in front of the bill of materials. People can lie on the suckers.
Paul Asadoorian (05:06.622): Firmware bill of materials.
Paul Asadoorian (05:11.538): Well, if it’s enough we can call it a bomb, so that’s kind of interesting too.
Larry (05:14.909): Yeah, that would be a firmware bomb, right? But yeah, Paul, you’re absolutely right. I think back to one of the classic examples about not trusting. Back when we were seeing FTDI chipset stuff for doing serial UART type of deal, FTDI had a company—
Joshua Marpet (05:16.683): Yes, that would be a firmware bomb.
Paul Asadoorian (05:34.066): Yeah. Yeah, yeah, yeah.
Larry (05:42.631): A bunch of companies cloned their chips. To the point that the drivers at one point would actually brick the cloned devices. Like they look identical, they have the same function for the most part because they were reverse engineered, but they weren’t the actual true product.
Paul Asadoorian (05:55.891): Right.
Larry (06:09.467): Like all appearances, it looks like the right thing, but how do you actually ensure that is the right thing that you’re purchasing?
Paul Asadoorian (06:18.364): It’s interesting to think about the attack vector there too. That chip inside of the USB to serial device—what if that was collecting your keystrokes or something for later?
Paul Asadoorian (06:18.364): It’s interesting to think about the attack vector there too, where that chip inside of the USB to serial device, right? Like what if that was collecting your keystrokes or something for later? I mean, they’d have to have some kind of network connectivity, but what if in the silicon they were enabling some kind of covert capture?
Joshua Marpet (06:34.835): So.
Paul Asadoorian (06:52.766): And those are the kinds of things that we worry about. It’s one thing to know what software you have. It’s one thing to know what firmware you have.
Joshua Marpet (07:08.235): You.
Paul Asadoorian (07:13.52): But then there’s what is inside the chip. There’s the surface of the chip, but what’s actually inside of it, and how do we determine that?
Joshua Marpet (07:14.663): Yeah.
Paul Asadoorian (07:23.335): And a lot of the talk is about ways to validate or attempt to validate it, which gets super hard.
Larry (07:14.663): Yeah, and—
Larry (07:23.335): Yeah. Yeah. I mean, back to your thinking about what’s inside the chip, Paul, maybe it’s not even doing covert capture. Maybe it’s doing covert monitoring and then performing actions later.
Joshua Marpet (07:38.377): Wait.
Joshua Marpet (07:42.915): Data manipulation in transit. Yeah, like there’s so much, Larry.
Paul Asadoorian (07:45.747): Yeah, yeah.
Larry (07:53.273): So we’re going to delay, and then we’re going to run a command that changes firewall settings. Or we receive a message on a wireless device and then cause the device to explode. We saw exploding pagers. We saw exploding radios.
Paul Asadoorian (08:05.94): Right.
Vlad (08:13.811): So it gets even more interesting.
Larry (08:16.657): Because there was no ability for us to validate what was inside those chips as part of the supply chain.
Paul Asadoorian (08:24.926): Yeah. Vlad?
Vlad (08:25.917): Yep. At one point I read an article where someone took an open source chip and added exactly one wire. Just one extra trace. That wire created a covert backdoor that could be triggered to get root access. One wire.
Paul Asadoorian (09:01.268): Yeah.
Vlad (09:01.8): And there’s no way you’re going to find that with software tooling.
Paul Asadoorian (09:26.729): Yeah, certainly not with software. Some of the things Bunny talks about would allow you to find it, but you have to know what you’re looking for.
Joshua Marpet (09:01.8): And that’s—
Joshua Marpet (09:26.729): I want to point out that there’s history here. Twenty years ago we caught the NSA intercepting Cisco shipments and adding firmware, then re-taping them with Cisco tape.
Paul Asadoorian (09:53.643): Okay. Okay.
Joshua Marpet (10:26.884): Travis Goodspeed talked about putting Linux on the PCB of a hard drive so commands could be triggered without anything being on the platters.
Paul Asadoorian (10:26.884): Right, that would be hilarious.
Larry (10:29.725): You know, one of my coworkers was at CES and saw a robotic coffee machine running Ubuntu on the front panel.
Joshua Marpet (11:00.026): Nice.
Joshua Marpet (11:17.16): So there’s history here, but we keep going deeper—from firmware to PCB to silicon straight from the factory.
Paul Asadoorian (11:17.16): Yeah.
Paul Asadoorian (11:58.414): Bunny talks about the supply chain for hardware itself. Chips are designed, fabbed, packaged, shipped—often in different countries.
Joshua Marpet (11:58.414): Tamper evident village would like to talk to you.
Paul Asadoorian (12:13.178): You could insert hardware before tamper seals ever go on.
Joshua Marpet (12:32.647): I’ll tell you a quick story. A buddy at a major hardware manufacturer said when they need NAND chips, they take whatever ship arrives first that meets spec.
Paul Asadoorian (12:51.667): Mm-hmm.
Joshua Marpet (13:01.552): So end-to-end hardware BOMs are really hard.
Paul Asadoorian (13:25.96): Yeah, and how do we know if we have a counterfeit component?
Joshua Marpet (13:29.785): Or malicious.
Paul Asadoorian (13:40.489): Yeah.
Paul Asadoorian (13:54.739): Not everyone has an X-ray machine. We happen to have one in the office, but operationalizing that at scale is hard.
Larry (13:46.266): Yeah.
Paul Asadoorian (14:15.923): How do you X-ray thousands of routers? How do you know what to look for?
Vlad (15:23.091): It gets worse. Without a reference image from the vendor, you don’t even know what “good” looks like.
Paul Asadoorian (16:13.756): Yeah, agreed.
Larry (16:59.101): Same stuff we were doing with the Linksys book—component changes between revisions.
Paul Asadoorian (17:18.492): Yep, because they swapped the component out.
Paul Asadoorian (17:24.916): Independent researchers can help, but it doesn’t scale.
Larry (18:01.905): Yes.
Joshua Marpet (18:16.452): Details.
Paul Asadoorian (18:18.919): Right.
Paul Asadoorian (18:42.715): That’s fairly common—to include hardware that firmware isn’t using yet, but may in the future.
Larry (18:42.715): And they did. They did.
Joshua Marpet (18:44.664): Yeah, but when it’s a camera and a microphone, that gets awkward.
Paul Asadoorian (18:49.713): When a camera and microphone are on a KVM—one of the Nano KVMs had a microphone inside it. The Linux firmware actually had full capability to use it.
Larry (18:50.417): Yeah.
Joshua Marpet (18:54.328): Yeah.
Joshua Marpet (19:11.193): Yep.
Larry (19:11.197): Now, if they were smart, that firmware wouldn’t have had those capabilities. Some argued it could be used for diagnostics, like hearing a fan fail, but still—not great.
Joshua Marpet (19:17.292): I’ll set up a camera for that. I don’t want it built into my KVM.
Paul Asadoorian (19:20.327): Right.
Paul Asadoorian (19:40.562): Independent security researchers can make an impact by publishing findings when undocumented components exist.
Larry (19:54.193): Yeah.
Paul Asadoorian (20:14.576): But this doesn’t scale. What if only some devices are backdoored? What if it’s an enterprise product researchers can’t easily access?
Joshua Marpet (20:18.839): Well, it does and it doesn’t scale.
Joshua Marpet (20:31.779): The problem is corroboration. You can publish a beautiful BOM, but how do I know you’re not lying?
Paul Asadoorian (21:10.483): Great.
Paul Asadoorian (21:20.708): We’ve seen people use bounties for security research and corroboration.
Larry (21:33.039): Yeah, we covered it recently on Security Weekly.
Joshua Marpet (21:33.122): I was thinking Zerodium, but no.
Paul Asadoorian (21:44.509): Yeah.
Vlad (21:46.043): Firmware BOMs make this even harder. You won’t detect firmware tampering with X-rays, and many devices give no firmware access at all.
Paul Asadoorian (22:28.989): Yeah.
Paul Asadoorian (22:52.637): Bunny talked about debug interfaces—hardware can lie to them if it’s been tampered with.
Larry (22:52.637): Yeah.
Paul Asadoorian (23:01.877): Once the attacker controls the hardware, it’s game over.
Larry (23:16.989): Yeah.
Paul Asadoorian (23:31.315): Decapping chips is dangerous and complicated. I don’t recommend it casually.
Larry (23:56.163): Pretty common though.
Paul Asadoorian (24:15.923): Some cheaper techniques exist, like infrared imaging. Have you done that?
Larry (24:39.011): No.
Joshua Marpet (24:40.702): Yeah, the cut filter.
Joshua Marpet (24:50.805): You’re talking about EPROMs—the chips with windows wiped by UV.
Larry (24:59.899): Yep.
Paul Asadoorian (25:18.801): Is that similar to how night vision cameras work?
Joshua Marpet (25:23.776): You remove the IR cut filter. It’s the projector that wipes chips, not the receiver.
Paul Asadoorian (25:50.792): Bunny describes shiny chips being good for infrared analysis using cheap USB microscopes or modified cameras.
Joshua Marpet (26:07.602): Yeah.
Paul Asadoorian (26:28.113): He used an older Sony mirrorless camera with a special lens to see under the chip cap.
Larry (26:42.801): Yeah, that’s fascinating.
Larry (27:06.557): But again, this doesn’t scale. You can’t tear apart every device without voiding warranties.
Paul Asadoorian (27:41.748): I want to try setting one of these up—it’s more affordable than I thought.
Joshua Marpet (27:54.619): Yeah.
Paul Asadoorian (28:06.64): There’s a site Bunny mentioned—SiliconPrawn—or SiliconPron with a zero.
Larry (28:11.921): Yep.
Larry (28:26.531): It’s siliconprawn.org—P-R-A-W-N.
Joshua Marpet (29:00.674): You can upload images if you get an account.
Paul Asadoorian (29:10.515): That’s awesome.
Larry (29:39.261): AI was understandably confused on that one.
Paul Asadoorian (29:44.818): Bunny referenced a lot of tools—I need to rewatch the talk.
Paul Asadoorian (30:10.393): But then he gets into some really fancy machinery. He had open source software and hardware you could build yourself that uses infrared technology to look at silicon. One setup spun light around the chip and took multiple images, which was pretty awesome.
Paul Asadoorian (30:39.261): He also talks about Moore’s Law and how around 2010, adding more transistors actually started getting more expensive instead of cheaper.
Larry (31:28.157): Yeah, he said it was around the 28 nanometer mark.
Joshua Marpet (31:31.484): Yeah, 28 nanometers.
Joshua Marpet (31:57.901): But Bunny’s kind of wrong there. Moore’s Law isn’t dead—it shifted. It’s cheaper to add more cores than cram more transistors into a single core.
Paul Asadoorian (32:55.032): Mm. Right.
Joshua Marpet (32:56.422): And Moore’s Law has shifted toward quantum computing.
Paul Asadoorian (33:08.113): Yeah. And how do we translate this to the quantum realm?
Joshua Marpet (33:11.848): Same problem—what is the silicon actually doing?
Vlad (33:22.195): Last time I checked, quantum computers only had around 100 qubits. That’s not enough for most tasks yet.
Joshua Marpet (33:40.989): Moore’s Law doesn’t care where you start, just that things double every 18 months.
Vlad (33:50.611): Quantum computing will likely exceed that rate for a while.
Vlad (34:21.853): But it won’t make sense for every device—it would be too expensive and overkill.
Joshua Marpet (34:32.889): Yeah, you can make it smaller, but it’s insanely expensive.
Vlad (34:39.123): And saving 10 milliwatts—why do I care?
Joshua Marpet (34:51.715): Ooh. Yeah. Exactly.
Larry (34:54.202): Yeah.
Joshua Marpet (35:00.955): My kid literally asked me, “Why do I care?”
Paul Asadoorian (35:06.579): Mm-hmm.
Vlad (35:16.915): I don’t even know that reference.
Joshua Marpet (35:21.532): That’s the funny part.
Larry (35:23.579): You should be more worried when they ask, “Did you die?”
Paul Asadoorian (35:25.242): We’ll send you links after the show.
Joshua Marpet (35:31.542): God.
Joshua Marpet (36:02.823): Cheap chips are where we plateau. Office workers don’t need more power.
Paul Asadoorian (36:02.823): Two things—memory is expensive now, and Windows 11 forces better hardware.
Paul Asadoorian (36:22.49): Hardware roots of trust are good, but it all comes back to trust.
Paul Asadoorian (36:56.061): Alan Friedman talks about hardware BOMs that include the chip’s full journey—design, fab, packaging, shipping.
Joshua Marpet (37:52.521): Funny you say that—this is exactly what I’m working on right now.
Joshua Marpet (38:04.44): Value chain risk is a nightmare. We don’t know where software, firmware, or hardware really comes from.
Paul Asadoorian (38:25.873): Yeah, right.
Joshua Marpet (38:27.45): Malware learned how to avoid sandboxes—hardware is doing the same thing.
Paul Asadoorian (38:56.061): Vitamins are the same problem. How do you know what’s in them?
Joshua Marpet (38:58.892): They’re barely FDA touched.
Larry (39:00.256): And tested—sometimes.
Paul Asadoorian (39:05.203): Where are they tested and how often? Same issue with hardware.
Larry (39:23.345): You need provenance—chain of evidence—for hardware, just like food or drugs.
Paul Asadoorian (39:56.455): Most vitamins aren’t FDA approved.
Joshua Marpet (39:59.629): Exactly.
Larry (40:29.617): Eventually we’ll need the same provenance for code and hardware.
Joshua Marpet (40:31.961): It’s all the same problem.
Larry (40:35.505): Yeah, like where did the code come from? It’s an open source project on GitHub contributed by people all over the world. You need provenance—where it originated and what went into it.
Joshua Marpet (40:48.185): All those damn Antarcticans.
Joshua Marpet (41:00.269): And how often is it checked? Point-in-time security isn’t sufficient. We need continuous assurance.
Larry (41:04.647): Hmm.
Joshua Marpet (41:29.824): If the risk is a connected vehicle and brakes lock once a year at random, that’s a quantifiable risk.
Paul Asadoorian (41:52.199): Yeah, but we don’t have facilities to test things like vitamins or hardware at home.
Joshua Marpet (41:58.136): Of course not.
Joshua Marpet (42:04.44): Protein powders, hot sauce—how do you know there aren’t heavy metals?
Paul Asadoorian (42:22.49): Right, and enterprises aren’t testing hardware either. We’re blindly trusting vendors.
Paul Asadoorian (42:46.993): Independent testing helps. We’ve talked about cyber UL for years, but it never really materialized.
Larry (43:00.636): Thank you.
Larry (43:02.855): Cyber UL arguably became the Cyber Trust Mark, which then stalled.
Paul Asadoorian (43:10.77): Mm-hmm.
Joshua Marpet (43:13.963): Has it fully gone?
Paul Asadoorian (43:19.952): Regulation is tricky. Too much stifles entry into the market.
Larry (44:11.549): Yep.
Paul Asadoorian (44:12.728): Think about cheap Amazon hardware—certification costs would kill margins.
Joshua Marpet (44:24.597): Ha ha ha.
Paul Asadoorian (44:42.066): Larry brought up the Haribo battery packs. Independent testing found poor-quality batteries and Amazon pulled them.
Larry (44:45.637): Yep—and dangerous.
Larry (44:55.581): Hack5 couldn’t source commodity routers anymore, so they built their own hardware.
Paul Asadoorian (45:18.236): Yep. Yep.
Larry (45:24.663): FCC certification required volume to make it viable, so products didn’t change for years.
Paul Asadoorian (45:42.394): Yeah, to make it worth it.
Joshua Marpet (45:56.514): Right.
Paul Asadoorian (45:56.56): That’s why those devices cost more—they went through certification.
Larry (45:58.397): And their pager couldn’t be sold until FCC approval finished.
Paul Asadoorian (46:15.93): Right. But FCC approval only covers radio communications.
Joshua Marpet (46:16.736): But I want to point out Dreagorn redesigned the firmware and didn’t need FCC recertification.
Paul Asadoorian (46:22.578): Yeah, it’s awesome.
Larry (46:41.181): Correct.
Paul Asadoorian (46:41.582): Because the radio didn’t change—that’s what FCC tests.
Larry (46:52.061): Yep, radio transmissions only.
Paul Asadoorian (47:04.794): I’ve bought devices on Amazon with no FCC ID that still transmit.
Larry (47:16.813): Completely illegal—and cheap.
Paul Asadoorian (47:21.234): That’s interesting.
Joshua Marpet (47:28.806): There are blatantly illegal devices on Amazon.
Paul Asadoorian (47:36.336): You can’t buy a Flipper Zero, but you can buy malware-filled Android TV boxes.
Joshua Marpet (47:54.26): You can buy jammers.
Paul Asadoorian (47:58.095): Interesting.
Larry (48:14.109): Illegal to manufacture or use—not necessarily to possess.
Paul Asadoorian (48:32.37): Okay.
Joshua Marpet (48:48.455): Operation, marketing, or sale—that’s where it’s illegal.
Paul Asadoorian (48:56.186): I thought there were research exemptions.
Joshua Marpet (48:59.07): Labs have to be certified.
Paul Asadoorian (49:05.956): Right.
Joshua Marpet (49:42.406): You can technically jam ISM bands if you’re wirelessly powering lights.
Paul Asadoorian (50:12.762): Yeah. Yeah.
Larry (50:13.085): Thomas’s story.
Paul Asadoorian (50:16.188): This brings us back to trusting hardware. Researchers often buy devices straight from China.
Joshua Marpet (50:46.974): Perish the thought.
Paul Asadoorian (50:57.794): What if security research hardware itself is backdoored?
Joshua Marpet (51:25.661): How many of us monitor our own network traffic?
Paul Asadoorian (51:30.736): There’s a lot of implicit trust.
Paul Asadoorian (51:57.964): Once a hardware root of trust is broken, it’s game over.
Joshua Marpet (52:57.964): That leads to the idea of high-assurance hardware.
Vlad (53:07.899): It gets even worse.
Vlad (53:14.757): A single person can now build covert transmitting devices at home.
Paul Asadoorian (53:52.443): Yeah.
Paul Asadoorian (54:12.678): Like the KGB backdoored typewriters.
Joshua Marpet (54:18.457): And the Great Seal.
Paul Asadoorian (54:21.714): Electronics hidden in embassy typewriters transmitting RF.
Larry (54:51.005): IBM Selectrics, if I recall.
Paul Asadoorian (55:11.141): Today’s hardware is far more complex.
Joshua Marpet (55:11.141): We should do another talk on high-assurance hardware.
Paul Asadoorian (55:31.079): Right.
Vlad (55:36.349): Keyboards now have dual-core ARM CPUs.
Joshua Marpet (55:51.153): Perfect.
Paul Asadoorian (55:58.494): Firmware updates change everything—benign can become malicious.
Vlad (56:58.439): Linux webcams can be updated with unsigned firmware.
Paul Asadoorian (57:12.592): Yeah, I have some of those too.
Vlad (57:22.435): They can present themselves as network cards or keyboards.
Paul Asadoorian (58:00.263): Yep.
Paul Asadoorian (58:04.688): You can even run Doom on them.
Paul Asadoorian (58:07.697): Unfortunately, we’re out of time. Thanks everyone for listening to Below the Surface.
Joshua Marpet (58:25.202): Thank you.
