Introduction
The IT supply chain is filled with software vulnerabilities, many resulting from significant code reuse across multiple vendors. The economic forces at play form a ‘race to the bottom’ competitive landscape, where feature velocity and low cost drive development practices. Often, basic Secure Systems Development Lifecycle (SSDLC) practices, such as baking in static code vulnerability analysis into the development and QA lifecycles, are eschewed. Furthermore, vendors often leverage and re-use OSS (Open Source Software) to gain efficiencies.
As we’ll see in the research below, it isn’t difficult for an attacker to abuse IT supply chain dynamics for their own advantage. Even well-meaning research, performed ethically and via proper CVD (Coordinated Vulnerability Disclosure) processes, can be leveraged to find the same vulnerabilities present in other manufacturers.
Background
In February, I came across a blog by OneKey about a pair of vulnerabilities they found in NetModule routers. These are industrial routers used by multiple industries, including public transit, geospatial and railway transportation. The vulnerabilities discovered were related to authenticated command injection and authenticated path traversal in the web administration interface of the device firmware.
Both vulnerabilities are simple and caused by a lack of proper sanitization of user input; this vulnerability is so common it is number three on the OWASP top 10 list. This is something that basic SSDLC practices would catch via static or dynamic analysis, but often in the development of IT supply chain devices, such practices are rare and routinely overlooked.
Vulnerability 1 (CVE-2023-0861): Authenticated command injection
The web administration page /home/www-data/admin/gnssAutoAlign.php contains this code:
<?php
require_once('config/config.php');
if (isset($c))
$device_id = $c;
else $device_id = $_REQUEST['device_id'];[TRUNCATED FOR BREVITY]
if (isset($_POST['toggleAlignment'])) {
if ($status == "disabled") {exec("/usr/local/sbin/www-scripts/various/doAutoAlignment " .
$device_id . " > /dev/null &"); $status = "starting";
}
else {
exec("kill $(cat ". PID_FILENAME . ")");
$status = "stopping";
}
}
By failing to sanitize $device_id, an attacker can inject malicious code into the system command called by the exec() function, which runs with administrator privileges.
Vulnerability 2 (CVE-2023-0862) : Authenticated path traversal
One of the web administration pages, /home/www-data/admin/sdkJobs.php, has an include statement to use the file /home/www-data/admin/include/sdkJobs.php – this file has a function to allow a user to upload a file, presumably a scripted job submitted via the SDK (for brevity, only the relevant code is shown):
$name = trim($_POST['scriptName']);
$uploadpath = UPLOAD_DIR . "/" . $name;It then calls move_uploaded_file() to move the uploaded file but fails to sanitize the user input before doing so:
if (!move_uploaded_file($_FILES["scriptUpload"]["tmp_name"], $uploadpath))Replicating the results
I’ve been spending a lot of time working with ChatGPT to see how valuable it can be in vulnerability research, inspired by a fantastic experiment my friend Juan Andres Guerrero Saade did at the Alperovitch Institute. The core idea is that ChatGPT, for all its flaws, is a powerful tool when it comes to analyzing code and can automate some of the mundane tasks in programming, reverse engineering and vulnerability research. With this in mind, I sent ChatGPT the PHP file gnssAutoAlign.php from Vulnerability 1, asking if it contained any insecure coding errors. The results were impressive:
Unfortunately, due to the token limitations of ChatGPT, the sdkJobs.php file from Vulnerability 2 was too large to send to ChatGPT, and it was unable to identify the vulnerability from small snippets of code. It is worth noting that ChatGPT responds better when asked about insecure coding practices than when asked to find a vulnerability, and it took a few attempts before it identified Vulnerability 1.
I’ve reviewed a significant number of automated IoT vulnerability scanners – both open source and commercial – and found they mostly perform the same tasks; they unpack the firmware image using binwalk, identify CWEs, software components, and their associated vulnerabilities based on version. They also look for binary protection methods, hard-coded credentials, and some perform basic Static Application Security Testing (SAST). I’ve found two open source tools, FACT and EMBA perform these tasks as well, many times better than commercial offerings – so much so that I gave a talk recently about using them to find vulnerabilities in data center control systems like door security panels, IP KVM, ethernet-to-serial adaptors and other devices.
EMBA in particular has proven extremely powerful due to its use of the open-source version of semgrep, a lightweight SAST tool that is remarkably similar to the vulnerability analysis seen in the OneKey blogs.
Lighting strikes thrice
Having taken an interest in the research OneKey has been doing, I saw a new blog from them in March about more vulnerabilities they discovered, this time in Phoenix Contact industrial routers. What was striking about this blog was the vulnerabilities were identical to the ones they found in NetModule; so much so that even the screenshots on the two blogs are the same. At this point, I realized the same web UI is used by at least 2 industrial router vendors, and it’s likely that others use the same framework. This is not vendor-unique code, this is a library or module of some sort meaning the vulnerability affects the supply chain of these devices, however prevalent they may be.
I was unable to find any reference to the two files containing the vulnerabilities on GitHub or via a Google search, nor was there any useful copyright information in the web UI directory structure that would indicate where they came from, who the developer was or what company licensed it, so I turned to another one of my favorite tools, Shodan, to see if I could find more devices with the same web panel. One of the more powerful ways to search Shodan is via HTTP information, and I extracted the index.html file from one of the firmware images:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html><head><title>Web Manager</title>
<meta http-equiv=Content-Type content="text/html; charset=windows-1252">
</head>
<body>
<a href="admin/index.php">Web Manager</a>
</body>
</html>A quick search for ‘http.title:”Web Manager”’ returned 1900+ hits, however, most of them were devices I knew weren’t running whatever this web framework was:
However, from analyzing the firmware through EMBA & FACT, I knew it ran Lighttpd, an open source web server, so I refined my search:
In the first few hits I saw NB2700; which I knew from this research was a NetModule router. The Netbox Web Manager device is also made by NetModule so I skipped it as I knew it had already been analyzed. Now I just had to find some other devices which OneKey hadn’t already researched. The first device I was able to connect to was a router called M!DGE by Racom:
Like Phoenix Contact and NetModule, Racom is an industrial router manufacturer so I felt there was a strong possibility this was in fact running the same web framework.
Fortunately, Racom allows anyone to download their firmware but as I looked at their site I noticed something interesting: their most recent firmware v4.6.40.103 was dated April 6th, 2023. Considering that NetModule released a fix for these vulnerabilities in November 2022 and Phoenix Contact released their fix in March 2023, the timing would indicate that Racom’s latest firmware release was in response to the same vulnerabilities. However, the release notes for v4.6.40.103 make no mention of any vulnerabilities being addressed in the update.
They also have an archive of older firmware, so I grabbed the previous version, v4.6.40.102 as well and turned to FACT & EMBA.
I generally use both tools, each for different purposes. FACT has a nice GUI, and makes it very easy to browse through the filesystem of a firmware image to view the files inside it. EMBA is more powerful when it comes to SAST & running the firmware in an emulator (QEMU) to extract version details, and it has better SBOM generation capabilities. So, I put the most recent version of the M!DGE2 firmware into FACT, the older version into EMBA & went to the gym.
When I got back, I was not disappointed. EMBA had correctly identified the same command injection vulnerability OneKey found!
Armed with this knowledge, I quickly checked FACT to see if I could file this as a vulnerability report with Racom however my suspicions about the recent release were confirmed: Racom had fixed this vulnerability in April by wrapping $device_id in escapeshellarg():
They had also fixed the authenticated path traversal vulnerability:
Vulnerable version:
Patched version:
There would be no CVE’s for me today, however the exercise was a success. The only unanswered questions were how Racom learned of the vulnerabilities, and why they silently fixed them. Interestingly enough, the vulnerable v4.6.40.102 release notes make mention of a security fix for a very similar vulnerability:
Authenticated Remote Code Execution and Privilege Escalation in PHP-CLI – A user with access to PHP-CLI could inject shellcode which was executed with administrative rights. The validation and escaping of command arguments was improved to prevent such an attack.
Takeaways
Supply chain vulnerabilities impact multiple vendors due to the reuse of code libraries and components. In this exercise, I spent around 6 hours total, pivoting from a known vulnerability in two vendors to discovering the same vulnerability present in others, simply because the same vulnerable libraries were re-used by the third vendor to perform a basic and common function on the device. The vast majority of that time was spent analyzing the NetModule & Phoenix Contact firmware images (automated tools go BRRRR for hours); finding the third vendor on Shodan and confirming the vulnerability took about 2 hours.
I also identified a fourth vendor using this framework: Leica Geosystems. They sell ComGate industrial routers and a few showed up on Shodan. While I wasn’t able to download firmware for these devices to analyze, the product page for these routers states they were created in collaboration with NetModule so I have high confidence that at least one version of their firmware is vulnerable. The models I found are ComGate10, a version they no longer sell which usually means firmware updates are infrequent, if ever. Both OneKey advisories reference the same two CVEs and the CVE details only mention NetModule so it’s quite possible NetModule is the top of the supply chain for this code.
For operational environments (as well as IT environments) to truly manage risks associated with the IT supply chain dynamic explored here, organizations must have visibility into their devices at the SBOM (Software Built of Materials) component level, in order to query for such ‘shared’ vulnerable components within their environment. The industrial routers explored here often perform critical functions and serve as an ideal attack surface from which attackers can persist, disrupt operations, or move laterally to further their objectives.
While patches have been issued, the unfortunate reality is that most operational environments do not perform routine security updates for these devices, nor do they have the requisite visibility into those devices in order to gauge the potential impact scenarios and associated risk to safety and/or uptime.
If a researcher like me can discover these types of vulnerabilities on a whim, imagine what determined and resourced adversaries can find and exploit. The only real-world mitigation is true IT supply chain component-level visibility and proactive patch management that this visibility affords.
