Kernel security now: Linux's unique method for securing code

The Linux Foundation's Greg Kroah-Hartman delivered a comprehensive talk this week on the current state and future challenges of Linux kernel security. Speaking at the Open Source Summit (OSS) Japan 2023, Kroah-Hartman -- Linux stable kernel maintainer and a prominent member of the Linux kernel security team -- shed light on the evolving landscape of open-source software security, regulatory challenges, and the Linux kernel's response to these issues.

Hardly a day goes by without a software security issue popping up, and governments are now trying to direct how companies and organizations should mitigate security issues. There's just one problem: Governments barely understand how to use software, much less how open-source developers create software.

Also: Linux might be your best bet for heightening your desktop computer security

For example, the European Union's proposed Cyber Resilience Act (CRA) is full of good intentions, but it's a bad fit for anyone who builds open-source software. While the most recent version is much better, as Kroah-Hartman pointed out, it still means that since "all the world sells their devices and products into the EU, this is going to define their security requirements."

Are we ready to deal with this new wave of regulation? No, we aren't. 

As for the Linux community, Kroah-Hartman has said that the Linux kernel security team is fundamentally reactive, contrary to other security teams that adopt a proactive stance. Since its formal inception in 2005, the team has operated informally, without corporate affiliations or contracts. This allows for neutrality and flexibility in addressing security issues. This approach has fostered trust among companies and effectively managed and triaged security problems.

"There are other groups, kernel security teams, and other projects," he added,  "that are proactive. But that's not what we do. We just react to problems."

And there are plenty of problems to go around. For instance, Kroah-Hartman highlighted the ongoing challenges with hardware security, particularly in the wake of vulnerabilities like Spectre and Meltdown. Indeed, as he pointed out, it's been more than three years since those serious CPU bugs appeared, and while "they keep trying to fix them in hardware, another one just got announced a few hours ago. So there's no end to this anytime soon." 

This underscores the complexities of dealing with hardware embargoes and the longer development cycles of hardware compared to software. Ideally, Kroah-Hartman wants the hardware companies to "get on the ball faster," a sentiment echoed by governments and regulatory bodies.

Kroah-Hartman also pointed out that, "a lot of people [today] don't realize that while the Linux commercial distribution model is not dead, it's not the majority anymore by far. 80% of the world's servers and systems run free and open source projects based on Debian, Fedora, or openSUSE" -- not Red Hat Enterprise Linux (RHEL) or SUSE Linux Enterprise Server (SLES). 

Also: Want a simple, stable, and secure Linux distribution? Then SpiralLinux is for you

That reality has complicated security challenges because, Korah-Hartman explained, "the communities that work with these open-source projects can't sign a non-disclosure agreement (NDA) because their community members live in other countries or work for different companies." 

Instead, the Linux kernel developers' security team is an "ad hoc informal group" without a  contract. Kroah-Hartman continued, "And that was the best thing that could ever happen to set the stage for us doing this in a company- and government-neutral way. It's saved us so many problems."

How it works: People send security reports to the group's members. There's not even an email list. There's just a small group, which doesn't represent any companies. Kroah-Hartman added, "It's all kept quiet, and since 2005, we've never had any leaks."

What they do, Kroah-Hartman continued, "is triage the reports, figure out what's wrong, and drag in the proper developers, if they're not on the list already, to create the fix as soon as possible. This patch is then included in the stable branch of Linux. That's it."

When they say "as soon as possible", they mean it. "Once we have a fix, the most we'll hold on to is seven days. This is," Kroah-Hartman continued, "after we have a fix. When we get a report, we start working on it as soon as possible. We have had some fixes to take over a year. We've had some networking issues. I think we went on 18 months before we fixed it properly. But once we fixed it, the fix goes in." 

Also: Ubuntu Linux 23.10 is adding an important new security feature

The group also does not make announcements of security fixes. " We don't announce anything. We don't say anything special. We just push it in so that it looks like a normal bug fix."

Yes, that does make people angry. But, Kroah-Hartman explained, "to people on the security team, a bug is a bug is a bug. There's nothing special about security fixes. And if we call out security fixes as being special, that implies that other fixes are not special."

That's a mistake because, according to Kroah-Hartman, "any bug has the potential of being a security issue at the kernel level." A small bug fix he'd made years ago to TTY, a minor subsystem in Linux today, turned out to have a killer security hole. It enabled anyone to get root on RHEL systems.  You never know where or when a security problem will crop up.

Kroah-Hartman also observed that while the "Linux kernel has about 30 million lines of code, you only use about two million lines in your server, 4 million in your phone, and one and a half million in your TV. But we don't know what you're using. Linux is everywhere, in your cars, in satellites, and it's in cow-milking machines. We don't know your use case. We don't know how you're using Linux. We don't know what the security model is." Therefore, everything and anything must be considered essential.  

Also: 7 things even new Linux users can do to better secure the OS

So, what can you do about it to protect yourself? Kroah-Hatman stressed that you should always use the latest long-term stable (LTS) kernel. 

Unfortunately, very few Linux distributions do that. He criticized companies that fail to update their kernels regularly. Outdated systems, from where he sits, are inherently insecure. 

This isn't his opinion alone. After years of study, Kroah-Hartman cited the Google Android security team, which found that stable Linux kernels had fixed every known recent security problem before they were reported. They have documented proof that taking stable kernels always works and that your systems will be secure. As a Google Linux kernel engineer, Kees Cook said, "So what is a vendor to do? The answer is simple, if painful: continuously update to the latest kernel release, either major or stable."