Why the Software Supply Chain Needs More Security

Kim Lewandowski is co-founder and head of product at the security startup Chainguard. Before Chainguard, she was a product manager at Google, most recently focused on hardening the company’s development pipeline and the open source tools it relied on.

In this interview, she talks about what a secure software supply chain means and why the idea has gained such strong traction over the last several years. She also looks at the difficulty of balancing developer productivity with the need to review open source components, and at why large enterprises need to back the open source tools they use in the interest of security and project longevity.

---

What is a secure software supply chain?

You worked on a lot of projects during your time at Google, but one of the more relevant ones for the purpose of this discussion is Tekton. Can you explain Tekton briefly?

KIM LEWANDOWSKI: Tekton is an open source project. It’s a CI/CD framework and a highly pluggable system that helps developers with the source-to-deploy pipeline. While I was starting Tekton, that was when I heard many customers at many companies asking about security; it was always top of mind for everyone involved.

Then there was the chance to help protect Google from open source security threats. There was a very natural transition from doing the actual CI/CD implementation pipeline work and then examining it through a security lens. Not only the vulnerabilities piece, but a broader set of risks that companies take on when they begin depending more heavily on open source.

So, basically the idea of a secure software supply chain. How would you define that for someone who’s not up to speed on that discussion?

Some people hear the phrase and assume it refers to the physical supply chain — manufacturing, shipping, and so on — and in a way it does. Consider how products such as our laptops are made: there are so many separate steps, stages, and parts that go into the final result. When you think about the software supply chain and software such as Zoom, for example, which we’re using now, there are hundreds, if not thousands, of developers underneath it who contributed code that ended up in the finished product. And then it becomes even more complex, because the final product is built from so many other dependencies.

And there are other kinds of supply chain attacks, like developer impersonation or project takeover. It’s a bit more comprehensive than simply thinking about a vulnerability in the code or something.

When you begin to look at the whole picture — how the software moved from a developer writing code all the way to the end — there are weaknesses at every link and stage in that pipeline. All the dependencies, all the things used to build the software, and all the parts in between are what we’ve been calling the software supply chain.

Why now?

Why is this the subject of conversation now? What changed about the way apps are developed and the way systems are built and architected that is making people pay attention to the various pieces that make up their applications?

I think about it a bit like the pandemic that’s going on right now. There’s one major incident, and everyone scrambles and focuses on it and chips away at it a little, and then something else happens and it’s a similar story. So I don’t think the problem is new exactly, but we have seen a huge rise in attacks over the last few years. For supply chain attacks, the latest number I’ve seen is a 650% increase in 2021.

There are many different reasons. One reason I’ve heard is that we’ve gotten better at protecting other things in technology, so this is the next weakest link attackers have latched onto. And open source has become much more widely adopted and it introduces an entire set of different risk vectors that we haven’t done a great job securing against. And just the economics of it — with some of the attacks we’ve seen recently, like the Log4j exploit, attackers are getting a lot of bang for their buck.

But it’s not exactly a new idea: the one paper that’s always cited is Ken Thompson’s Reflections on Trusting Trust, from 1984, which explains how you could backdoor a compiler and then do lots of malicious things through it.

What is the role of open source in all of this? You’ve called it a security risk, but there is also an argument that open source is more secure, because you have all those extra eyes on it …

You can make the case either way, but I think the reality — and this is how it should be — is that open source makes up the vast majority of our software applications today. And what that does is let companies innovate and speed up production of what they’re trying to do. But along with that comes risk.

When we talk about open source security risks, there are lots of open source projects — many of them that companies depend on, or that a developer will simply find and use because it does what it needs to do — that are not very well supported or maintained. There isn’t always a large enterprise company behind these things. There’s an xkcd image that keeps getting shared around, and it’s all this modern infrastructure and architecture being held up by this tiny little block maintained by some lonely developer in Nebraska. That’s really the situation.

So, while open source is amazing and has moved us all further ahead in technology and everything, it comes with risk around unsupported projects, not knowing how they’re being maintained, what best practices they’re following, and that sort of thing.

Container security

How much of the software supply chain conversation really is about containers, and the now very common practice of packaging applications and tools as containers?

Well, that’s what we’re focused on from my company’s point of view, because that’s where we have the most expertise right now. And container development, in general, is simply where people are heading. Unfortunately, I’ve seen security pushed even further down the priority list as people move to containerized app development. If I’m a developer, I’m not a security developer — I need to ship a feature, I need to get my product built. If there’s some open source package that does image editing or video editing that I need, I might just take that instead of writing the code from scratch, without even thinking twice.

Aside from carefully vetting the container images you’re pulling, what’s the answer there? If you look at Docker Hub, for example, there are reports saying that some not-insignificant percentage of images have vulnerabilities. How do you vet them all?

It would be worth hearing how you define careful vetting, because I think that is where it begins. It is simply not the case that the people pulling in these packages, or containers, are doing a line-by-line review of the code. In Docker Hub, there is no way to even trace where the source code came from that was placed into the container.

Returning to the physical supply chain analogy, you have to be able to trace these things back to their origin, first of all: Who developed them? What engineers are putting code into them? How was it built? What packages and what dependencies went into it? Understanding these pipelines, and having evidence of relevant information along the way, will let you make better trust decisions than you can today.

That sounds like a case for standardization inside organizations, and for making sure different teams and different developers are not all building snowflakes or following different guidelines.

Exactly. And that is where we are seeing big enterprises struggle today, because we all know the SolarWinds incident was something that attacked the build system, specifically. So now companies are taking a more closely scrutinized look at how the code is being built even internally. This is not just open source risk that we are talking about — insider risk is a real thing that large companies need to worry about.

It is usually the Wild West, because companies do not want to get in the way of how their developers are getting their jobs done. So they say, “Hey, devs, go take any tool, go use any build system you need to get your job done.” But then from a security standpoint, that becomes a nightmare. There is one company I talked to that has something like 120 different build systems. Imagine manufacturing, say, a medicine in 120 different factories that you do not understand anything about.

Balancing productivity and security

How are individuals or companies supposed to square that circle, where on one hand, we say developer productivity is job number one, and on the other hand we say security is job number one?

I think the first thing is awareness. There is still not broad awareness of the risks that are out there and how people are thinking about these things, even at more executive level. I do not know if people are fully aware of the different threat models involved here, just as a starting point. I do think the answer here is making developer-friendly tools in different technologies that make this less difficult. Chainguard’s mission is to make the software supply chain secure by default, because we think the only way you can get developers to adopt more secure practices and more secure tooling is to make sure you are not getting in their way — make it super easy for them.

I just got off a call about an open source thing we are working on for developers to cryptographically sign Git commits that does not get in their way at all. You are able to sign your code commit with your credentials, and then people can verify on the other side, basically. That is one example of how a developer can start being more secure without lots of things getting in their way.

Now, this does put a little bit more onus on developers because you have to log in and do an identity flow, but the economics show that if you catch things earlier, you are not paying so much for them on the other side.

However, the way we have been looking at the problem is that it is not only shifting left, it is shifting all the way right, too. You have to understand what the heck you are running in your production systems and be able to trace it back. That was a big thing we saw with the fallout of the Log4j incident: lots of companies not knowing if they were impacted at all. So I think we do need things on both sides of the spectrum to help solve this problem.

Many organizations are afraid to update, as they think everything will break. The downside of this is that they leave vulnerable software running, and when they do update many versions later, it is a nightmare. We need to get organizations updating more often. Broadly, I think the key to all of this is the automation piece — to get out of developers’ way, and to shift left and right to make sure things are as automated as possible.

Shifting Left

Shifting left is the practice of doing testing, security checks, and other processes earlier in the software development lifecycle, in order to catch them early and avoid problems in the later stages.

If everything is so complex, why are so many people committed to microservices or cloud-native development or whatever you want to call it? Is there not an argument for dialing it back?

Maybe someone is going to make that argument, but I am not going to make that argument. In terms of what it has done for us as a society, and the technology coming out of it now, I think containers just give us a new way to reason about these problems and to be able to catch up. And I do not even know if it is “catch up” because, to be honest, I am not sure how good the tooling compares without the container stuff as we were doing with VMs and whatnot.

We think the problem is tractable, I guess, but it is just a lot of work, especially when you start talking about these open source projects that everyone depends on. Because they are not supported by large enterprises; large enterprises might use them but, often, they are not contributing back to the project or providing any sort of financial support.

Securing open source

Supporting open source projects is a big topic of discussion, but beyond better tooling, what can be done to improve the security aspect in particular?

It is huge. Overall, I think it is a number of things: I think it is resources for these projects. I think it is large enterprises just taking a better look at their inventory, the things they are relying on, and hopefully contributing back where they are taking dependencies on some of these projects. There are efforts to make more curated open source projects, where people know they are more hardened.

I also do think that there has been a lot of talk at the government level around the software supply chain, and introducing more regulation around it. So companies selling directly to the government will be required to do a certain number of things or meet certain compliance requirements. And then it just becomes another turtles all the way down problem, because those people selling to the government rely on their own vendors, who are using their own open source code. We are just going to see it trickle down all the way, I think, over the next 5 or 10 years.

Can this have a California emissions standard type of effect, where if you are building something to comply with one major market or buyer, everyone benefits?

This is an enormous problem area, and I genuinely think it is a positive thing for everybody that these large companies are all joining forces to improve the overall state of affairs. So any momentum inside open source communities, or in major open source efforts like Kubernetes, is just great to watch, and I only hope we add more fuel to that fire over the next few years or so. Because I do think it is not a competitive edge for many of these companies, but it does shield them and also helps raise everyone else up.

Who's the buyer and what's coming next?

Who are you speaking with, or what sort of person are you speaking with, inside possible customers?

We’re still quite early — we launched in October and only recently announced our first product — but many of the people reaching out to us come from regulated industries. Banks, healthcare, anyone in a regulated space — this is all very much on our radar. I talk with CISOs and try to understand what is keeping them up at night, and I ask how they plan to deal with software supply chain issues. And are they worried about new regulation? How are they going to satisfy that?

But in the end, you also have to sell these things to developers, because they are the ones actually using them day to day. I think that is where the champions matter — being able to demonstrate and highlight different capabilities to developers and security teams so they can understand how this will really function in their actual production environments. It is a bit like a sandwich model: the people up here have the money, and the people down here are the ones doing the work.

I do not know whether I would envy the CISOs at these companies. They are under a huge amount of pressure to make things more secure, but it is difficult for them to bring down the hammer across an entire organization and say, “You must start doing things this way!” It is about finding that balance between improving security and not making everyone angry.

What does the future look like for what we might call “legacy” security approaches, firewalls and antivirus programs and the like, in a world that is increasingly focused on software supply chains, zero-trust, and these more modern methods?

With firewalls, I think the truth is that we are not all operating behind closed walls. A lot of migration to the cloud is still happening, both for infrastructure and for SaaS, which means you need a framework like zero-trust to validate access to data, resources, or whatever at the user level instead of simply trusting everything on the network.

And let’s not forget that you are still going to run open source software inside all of these firewalled environments. There are probably very few systems in existence today that are fully air-gapped and do not use any open source or third-party code. If any of those components are vulnerable and get exploited, attackers are already inside your network and the call is coming from inside the house.

My only hope is that people begin taking this more seriously, start examining their systems — where their risks are — and start adopting the best practices and other things that are coming down. Because I think a lot of good things are happening, especially in the open source communities, to help make everyone’s lives a little better.