Linus Torvalds on the future of Linux kernel developers and development

Once more, at The Linux Foundation's virtual Open Source Summit,  VMware's Chief Open Source Officer, Dirk Hohndel, and Linux's creator, Linus Torvalds had a wide-ranging conversation about Linux development. 

The illustrious pair started with Hohndel asking about the large size of the recent Linux kernel 5.8 initial release. Hohndel wondered if it might have been so big because developers were staying home thanks to the coronavirus. Torvalds, who always worked at home, said, "I suspect 5.8 might be [so large]  because of people staying inside but it might also be, it's just happened that several different groups ended up coming at roughly the same time, with new features in 5.8."

While COVID-19 has slowed down many technologies, while speeding up other tech developments, it hasn't affected Linux development much at all. "None of my co-developers have been hugely impacted either. I was worried for a while because one of our developers was offline for a month or two. … [But,] it  turned out that it was just RSI [repetitive strain injury], and RSI is kind of an occupational hazard to deal with." He added. "One of the things that is so interesting about the Linux community is how much it has always been email-based and remote, how rarely we get together in person."

In any case, Torvalds trusts this new build. Indeed, he ran his end of the videoconference from his new developer machine running the first release candidate of 5.8. 

Thinking of his new AMD Threadripper 3970x-based processor-powered developer desktop, Torvalds later added that, although he had been concerned about its fan noise it actually works well for him. Torvalds moved to this new homebrew computer because he needed the speed. "My 'allmodconfig' test builds are now three times faster than they used to be." That's important because Torvalds "strives to do about 20 to 30 [pull requests] a day, which is above my limit, … [but] in order to do that, [he needs] a lot of computing power. 

Hohndel then turned the conversation to diversity in the Linux kernel developer community. Hohndel said: 

One of the things that I found striking when I look at our community and the CNCF [Cloud Native Computing Foundation], I see a significantly larger share of Black contributors and leaders, Kelsey Hightower and Bryan Liles. And I don't think I'm seeing that in Linux. Is it just me not seeing the right people, or are we, less racially diverse than some younger foundation simply as a function of the time when we all started almost 30 years ago? 

Torvalds replied, "Honestly, I don't know. I mean, I don't know on two different levels. One is I suspect that a lot of the people who are in low technology are the kind of people who started several decades ago. And that's how they got into the whole low level hardware interfaces and getting into operating systems. But the other reason I don't know is that I literally don't know."

Having been to many Linux Plumber summits, I can say the majority of Linux kernel developers are white, with a large number of Indian and Chinese developers. There are only a handful of Black kernel programmers. These annual get-togethers tend to host a few hundred of the most senior Linux developers. 

Torvalds added, the kernel developer crew is a "very homogeneous crowd. But I don't even know about all the new developers, we have every single release. We have over 1,000 new developers. And I do not know what the people are or if they are even people. There could be pull requests being sent by artificial intelligences."

He also speculated that one reason why the CNCF and other newer projects may have a more heterogeneous developer community is that, frankly, the cloud-related programs are more interesting. "I've told people if they're looking for a new exciting project that kernels are not the future. The basics of a modern operating system were set in the 60s."

Torvalds continued, a lot of kernel work is "literally very fundamental stuff, we're cleaning up, and fixing problems." In some ways, kernels are boring and should be taken for granted. As Hondhel reminded Torvalds, Torvalds has said "Linux is boring and that's how it should be."

But, if you're really interested in the low-level interaction between the hardware and the operating system, Torvalds said, "there is a lot of interesting technical work being done right now." 

This led to another natural question, which has come before: Is Linux's leadership graying out?  Hohndel said, "Many of us have a five at the beginning of our ages, while a couple are approaching their 60s. So at some point, we as a community need to start thinking about generational change."

Torvalds agreed. "For us, there is nothing more interesting than interacting at a low level with the hardware and really controlling everything that's going on. So don't get me wrong, kernels are not boring but it's definitely true that the core people have been around for decades. Yes, we are getting grey." 

He continued, many of the oldest developers "have moved into maintenance and management. I don't like the word management, because I don't think of myself as a manager, but realistically that's what I do." Indeed, Torvalds thinks of himself these days as a code manager and maintainer, not a developer. 

The new people are the ones who are often doing the [programming] work. We have managers and retainers who are old and starting to gray, that's, I think, a completely different issue. But we do we have a generation of people in their 30s who are moving up through the ranks of maintainers so that we have that next wave of people to take over eventually, I mean, look, we've been doing this for almost 30 years so we need to start thinking, the next 20 to 30 years. And so we need to have that next generation.

The problem is, Torvalds continued, "We don't have enough maintainers. It turns out it's really hard to find people who are maintainers. It's interesting and it's challenging, but one of the downsides of being a kernel maintainer is you have to be there all the time. Maybe it's not 24 hours a day, but every day you react to email, you have to be there."

Moreover, "it's not a very easy niche, to get into. And we do not have enough maintainers. We do have a lot of people who write code. We have hundreds of people who are beginners, and that's usually more than most projects have. But at the same time, the one gating issue we often have is it's our need for maintainers to look at other people's code and help."

Another problem with finding maintainers is that:

It takes time, it takes experience. You have to have done this for a while, as a low-level maintainer to slowly move up and then gain the trust of enough people including your trust. And I think the keyword is trust. No, it's not just trust from other maintainers, it's also trust from all the people writing the code. And, that just, it takes time. 

Still, there are maintainers moving up in the kernel developer system. Hohndel wondered if another generational problem for Linux developers is finding people who are experienced C programmers. He worried, "Is there a risk that we are becoming the COBOL programmers of the 2030s?"

Torvalds doesn't think so. "I think C is still one of the top 10 languages." That said, Torvalds continued, "people are actively looking at doing drivers and things that are not very central to the kernel, for example, in Rust. People have been looking at that for years now. I'm convinced it's going to happen one day."

The pair also talked about how Torvalds only tests builds on the x86 architecture. But Hohndel wonders, now that Apple is switching its Macs to ARM, if this " will change the landscape of the hierarchy of CPUs but in a few years." Torvalds thinks it might. "For ten years or so I'd be complaining about the fact that it's really, really hard to find ARM hardware that is usable for development. They exist, but they have certainly not been real competition for x86 so far."

Now, though with Apple moving Macs to ARM, this will help. Sure, on the cloud with AWS and its Graviton processors you have the needed power. But Torvalds thinks for "kernel developers, you want to have a machine in front of you, you want to feel like eating your own dog food. I refuse to basically develop for anything that I can't use as my desktop portal."

Hohndel replied, "Apple if you're listening, get Linus an early ARM laptop." 

OK, Apple, the ball's in your court now. 

Related Stories: