AlmaLinux discovers working with Red Hat isn't easy

When Red Hat announced that Red Hat Enterprise Linux's (RHEL) source code would no longer be easily available, it transformed how the RHEL clones like AlmaLinux, Oracle Linux, and Rocky Linux create their distros. While Oracle and Rocky plan on fighting, AlmaLinux opted for a more peaceful course. That hasn't worked out as well as it hoped.

AlmaLinux has stopped trying to be 100% source code compatible with RHEL. Instead, the AlmaLinux OS developers decided to be Application Binary Interface (ABI) compatible. For almost all practical use purposes, that's more than enough. 

Also: Elive 3.8.34: A thing of beauty that any old-school Linux user would love

So, the AlmaLinux Board voted unanimously to "continue to aim to produce an enterprise-grade, long-term distribution of Linux that is aligned and ABI compatible with RHEL in response to our community's needs, to the extent it is possible to do, such that software that runs on RHEL will run the same on AlmaLinux." 

As AlmaLinux chairperson benny Vasquez explained, the precise goal is "ABI compatibility [which] in our case means working to ensure that applications built to run on RHEL (or RHEL clones) can run without issue on AlmaLinux. Adjusting to this expectation removes our need to ensure that everything we release is an exact copy of the source code that you would get with RHEL."

To do that, AlmaLinux will use the CentOS Stream source code. In return, Vasquez added, "We'll continue to contribute upstream in Fedora and CentOS Stream and to the greater Enterprise Linux ecosystem, just as we have been doing since our inception, and we invite our community to do the same!"

Also: Linux Mint 21.2: Your new and improved Linux desktop for the next three years

Officially, Red Hat had nothing to say. But, I'm told by Red Hatters that this is exactly "the approach that we've suggested that RHEL-like distributions take - working with the broader community in CentOS Stream."

So, what's the problem? Well, KnownHost CTO and AlmaLinux Infrastructure Team Leader Jonathan Wright recently posted a CentOS Stream fix for CVE-2023-38403, a memory overflow problem in iperf3. Iperf3 is a popular open-source network performance test. This security hole is an important one, but not a huge problem. Still, it's better by far to fix it than let it linger and see it eventually used to crash a server.

That's what I and others felt anyway. But, then, a senior Red Hat software engineer replied, "Thanks for the contribution. At this time, we don't plan to address this in RHEL, but we will keep it open for evaluation based on customer feedback." 

That went over like a lead balloon. 

Also: The best Linux laptops

The GitLab conversation proceeded: 

AlmaLinux:  "Is customer demand really necessary to fix CVEs?" 

Red Hat: "We commit to addressing Red Hat defined Critical and Important security issues. Security vulnerabilities with Low or Moderate severity will be addressed on demand when [a] customer or other business requirements exist to do so."

AlmaLinux: "I can even understand that, but why reject the fix when the work is already done and just has to be merged?" 

At this point, Mike McGrath, Red Hat's VP of Core Platforms, AKA RHEL, stepped in. He explained, "We should probably create a 'what to expect when you're submitting' doc. Getting the code written is only the first step in what Red Hat does with it. We'd have to make sure there aren't regressions, QA, etc. … So thank you for the contribution, it looks like the Fedora side of it is going well, so it'll end up in RHEL at some point."

Things went downhill rapidly from there. 

Also: Linux has over 3% of the desktop market? It's more complicated than that

One user wrote, "You want customer demand? Here is customer demand. FIX IT, or I will NEVER touch RHEL EVER." While another, snarked, "Red Hat: We're going totally commercial because Alma never pushes fixes upstream! Also, Red Hat: We don't want your fixes, Alma!"

On Reddit, McGrath said, "I will admit that we did have a great opportunity for a good-faith gesture towards Alma here and fumbled." 

Finally, though the Red Hat Product Security team rated the CVE as "'Important,' the patch was merged.

So, the immediate problem has been fixed. Still, bad feelings have been left behind. As Wright wrote, "The worst part of this for me is feeling that I wasted my time by even submitting a PR [Pull Request] here." That's the last reaction you want from developers in an open-source community. 

Looking ahead, though, Vasquez is optimistic.  In an interview, she said, "This is uncharted territory for all of us, and they appear to be willing to make things better. If we go back to our true goal (improve the ecosystem for everyone), this interaction is a learning opportunity for everyone. They have processes and practices for accepting stuff from the SIGs [CentOS Stream Special Interest Groups] already, but I'm hoping they'll get better about accepting PRs outside of the SIGs."

We'll see.