Recently there has been an increase in attacks that have compromised well known software companies’ supply chains, enabling attackers to gain access to customer systems by injecting their own malicious code or backdoor capabilities into third-party systems. These third-party systems (along with the malicious code) then get incorporated into software or other digital products.
These risks are combined with the ongoing challenge of knowing what software components developers are using as part of the products they are building, as well as impending legislation around the need to provide a Software Bill of Materials (Software BOM) when we ship or launch digital products.
We can take a step towards implementing better controls and visibility of our software delivery process by ensuring that we have an effective software supply chain.
What is the software supply chain?
The software supply chain includes everything that is necessary to deliver our applications into production. This includes everything that goes into the container, our application code, libraries, and third-party tools. It also includes the container host operating system and the container runtime, as well as any orchestration tools such as kubernetes.
The software supply chain has become a long, complex and incredibly tangled web of different software components that are all brought together to deliver the required functionality necessary for an application. This complex web enables developers to build applications faster and with lower effort by using pre-existing tools and libraries available from a myriad of sources.
What is the risk?
The security challenge comes from this necessary flexibility, enabling an attacker to potentially inject an infected library or tool into the process, either during the development process or later, when components are updated.
Key Risk Areas
Outside of the kernel, runtime, and orchestrator, the key sources of this risk are the components that are typically included in the containers. These components usually come from 3 possible sources:
- Application Code: This is usually inhouse-written code and libraries, but can also include upstream open source applications that have been adopted by the organization. The risk here is that this code is often considered trusted, and as such doesn’t get the level of scrutiny that it should.
- Vendor Libraries and tools: These include libraries and tools that have either shipped with the OS, runtimes or other vendor delivered software. This category is probably — and I stress probably — the lowest risk category as the code is usually scanned and verified through the vendor’s supply chain.
- Upstream Libraries and tools: These are tools and libraries that have been developed by the open source community and made available on github or other platforms. This category is our highest risk, as we have a tendency to trust open source community software because of the idea that many eyes should pickup up anything malicious. The reality is that there have been a number of direct attacks on these open source repositories, or more subtle attacks where code has been slowly injected through the normal contribution process and has gone undetected for a long time.
Whilst the risk has always been there in the traditional software supply chains, typically it has been mitigated by software companies performing detailed security checks and extensive testing on their software before shipping it out.
With the rise of more agile development processes and containerised applications that can be built, tested and productionised on the fly using technologies, such as Docker Compose or Helm to define complex application sets, dockerfile to quickly and easily define what is needed to automatically build each container, and CI/CD solutions to build and test and push applications to product much faster, we have changed (and increased) the places in which vulnerabilities can be introduced.
Secure Software Supply Chain Summary
In order to mitigate the risk of someone injecting unwanted code into our applications, we need to change the way we think about trusting our code. Essentially, we need to become more paranoid. We can no longer trust anyone in the supply chain, and need to be able to verify and scan everything at each stage of the process. In addition, once we have verified the code and built the images, we need to verify the images and securely store and sign them so we can be sure that the images we are storing are the ones we approved. Finally, we need a way to ensure that we can only run the images that we have approved, and that images that have not been through the approval process can not be executed on our production environment, or any environments we should choose.