‘Being serious about security is a must’ – Apache Software Foundation custodians on fulfilling its founding mission

Board members offer a behind-the-scenes look at the non-profit

The Apache Software Foundation (ASF) is fulfilling its founding mission – developing software that serves the public well – at colossal scale.

The non-profit, which is funded by donations and corporate sponsorships, ranks fourth on GitHub by star rating, a barometer of how many developers have bookmarked its projects – surpassed only by Microsoft, Google, and Facebook.

Established in 1999, the ASF has around 730 members and 7,000 committers maintaining more than 350 popular, free-to-use open source projects without payment. These include the foundation’s inaugural project, Apache HTTP Server, which powers one in three websites worldwide.

Speaking to The Daily Swig, senior figures at the organization gave a glimpse into its inner workings, listing its key strength as its process for handling security vulnerabilities.

‘Serious about security’

Offering a broad outline of the process, Sheng Wu, one of three recently elected board members, said: “ASF has its security group as a liaison and driver to coordinate with the security issue reporter and the project Project Management Committee (PMC). The security issue should be identified by the project’s PMC with a clear result – accepted or rejected.”

Bertrand Delacretaz, a returning director, hails the ASF’s “simple standardized process for handling security vulnerabilities”, citing Apache Sling as a typical exemplar of adherence.

“This helps emphasize the fact that being serious about security is a must for ASF projects.”

Read more of the latest open source security news

Continuous monitoring and consequences for violations, meanwhile, give the process teeth.

“Our security team keeps track of all vulnerability reports and provides a list of outstanding ones in their monthly report to the ASF’s Board of Directors,” explains Delacretaz, adding that “not being responsive to security reports can ultimately result in the Board moving a project to our ‘Attic’”.

Light touch

Mark J Cox, the foundation’s vice president of security, emphasizes that clarity and rigor should not come at the expense of agility.

“The ASF in general tries to ‘get out of the way’ of projects as much as possible, so when we do have policies that they must follow, like the security handling policy, it’s good to try to make that as easy and lightweight as possible,” he explains.

Delacretaz, who is also a PMC member of Apache Incubator, NetBeans, and OpenWhisk, adds: “I think we can be proud of having established such a simple and well-documented process, including guardrails to make sure it is applied. As a programmer who’s not a security specialist I have appreciated the guidance.”

Revisiting Equifax

Delacretaz says these processes were stress-tested successfully in relation to the 2017 Equifax security breach, despite the calamitous downstream delays.

The ASF demonstrated “that our process had worked and that the key vulnerability in that case had been patched and announced correctly from the Apache Struts side”, he says.

Anecdotally, The Daily Swig’s coverage of vulnerability research has revealed that open source projects are often as responsive in remediating vulnerabilities – or more so – than large organizations with significant resources.

Sheng Wu, an ASF co-founder along with Cox, points out that by definition, the foundation’s volunteer-driven model means project maintainers are motivated not by a salary, but to make a project “better, more stable, more powerful, and of course more secure”.

Delacretaz added that the open source paradigm’s intrinsic transparency creates a helpful incentive.

“I think the fact that your work is exposed publicly plays a key role,” he said. “Our open source activities are a living resume – we want that to look good, and in order to sleep well we need to provide great quality software to our users.”

‘Blind trust’ in the supply chain

That commitment to quality and security is being severely tested by the burgeoning threat of software supply chain attacks, with the open source ecosystem perilously vulnerable to dependency confusion attacks that can put millions of users at risk.

Bertrand Delacretaz worries that “blind trust” is being routinely invested in package developers.

“I am worried about how little effort people generally put to verify the integrity of binaries that they download for builds, container images, etc,” he explains, adding that it is everyone’s responsibility to alert colleagues and fellow engineers to the risks and ensure that defensive mechanisms are in place.

DON’T MISS Apache Pulsar bug allowed account takeovers in certain configurations

Wu says “robust” community models like the ASF’s are best placed to evaluate a project’s (direct at least) upstream dependencies status. “Oe ncevery part of the chain is being watched and secured, the whole open source supply chain is well protected,” says Wu.

Mark J Cox, who is also a software security engineer at Red Hat, says the Open Source Security Foundation (OpenSSF), of which he is a founding board member, is also doing some useful work on tacking the supply chain threat through its working groups.

Scoring security issues

The ASF is one of 168 organizations authorized to assign CVEs to security vulnerabilities.

“Our general guiding principle is to assign a CVE to anything that is an actual security vulnerability, independent of the severity or if the issue was found internally or externally,” says Cox, who has created several reports for Red Hat and Apache on how to gauge vulnerability risks.

“While it’s tempting to pay a lot of attention every time a vulnerability with a name, cute logo, or domain name [is discovered], branded issues are often not the ones you should be paying attention to.”

Echoing concerns recently expressed by Redscan, he also believes it unwise to triage vulnerabilities solely based on their CVSS score.

“We generally don’t recommend our projects use scoring systems like CVSS either, as that can create a situation where a user may ignore an issue due to a low score (or indeed have to take actions due to a high score) where that score doesn’t represent how exactly they are using the component and how they configured it.”

BACKGROUND CNAs and CVEs – Can allowing vendors to assign their own vulnerability IDs actually hinder security?

The ASF recently launched a web portal that facilitates interaction between Apache projects and the Apache Candidate Naming Authority’s (CNA) security team.

“We can use this portal to instantly allocate CVE names from the CVE Project using their new automation API,” says Cox, who says another API in the pipeline will automate the commitment of files to a project’s GitHub repository.

This tool truncates the process of publishing a CVE from a few days – or even weeks when errors are made – to a matter of hours.

Projects are also now obliged to create a public list of all product versions affected by a CVE.

Read more of the latest security vulnerability news

“A big part of the challenge of having inventories of components and dependencies in a supply chain is the naming and versioning details, so having machine-readable lists of project versions affected by a given CVE is useful for tool vendors as well as downstream projects and end users,” says Cox.

Another problem that needs to be tackled, he continues, is the use of security scanning tools that flag unfixed CVEs and out-of-date dependencies on Apache software, regardless of context.

“More often than not it’s the case where the dependency CVE has no impact on our project’s use of that dependency,” he explains.

“One way to stop these reports is just to rebase all your dependencies without looking at the underlying issues, but for any complex project with layers of dependencies you can easily get into a cycle of endless dependency updates.

“I expect in the coming years this will become more of a problem, and a burden, with the current focus on software bill-of-materials (SBOM).”

The Apache Way

Asked why he, like thousands of other open source contributors, is willing to volunteer his time to make the internet a better, safer place without financial reward, Delacretaz says: “I love learning so I think it’s how much I have learned from working with the kind of bright, open, and opinionated people that I am interacting with at the ASF.”

Cox, an ASF contributor for more than 25 years, says: “While the nature of my participation and time commitment has varied from year to year, I keep getting pulled back to the ASF as the mission, values, and goals of the organization resonate with me.

“Those values and the Apache Way are shared by the Apache community, making it a fun, rewarding, and challenging environment.”

Shedding more light on the ‘Apache Way’, Wu told The Daily Swig: “ASF delivers the most valuable and widely used open source projects in the world. It always welcomes new people to join and provides help generously.”

YOU MAY ALSO LIKE GitHub changes policy to welcome security researchers