The Linux Foundation's demands to the University of Minnesota for its bad Linux patches security project

To say that Linux kernel developers are livid about a pair of University of Minnesota (UMN) graduate students playing at inserting security vulnerabilities into the Linux kernel for the purposes of a research paper "On the Feasibility of Stealthily Introducing Vulnerabilities in Open-Source Software via Hypocrite Commits" is a gross understatement. 

Greg Kroah-Hartman, the Linux kernel maintainer for the stable branch, well-known for being the most generous and easygoing of the Linux kernel maintainers, exploded and banned UMN developers from working on the Linux kernel. That was because their patches had been "obviously submitted in bad faith with the intent to cause problems." 

The researchers, Qiushi Wu and Aditya Pakki, and their graduate advisor, Kangjie Lu, an assistant professor in the UMN Computer Science & Engineering Department of the UMN then apologized for their Linux kernel blunders. 

That's not enough. The Linux kernel developers and the Linux Foundation's Technical Advisory Board via the Linux Foundation have asked UMN to take specific actions before their people will be allowed to contribute to Linux again. We now know what these demands are.

The letter, from Mike Dolan, the Linux Foundation's senior VP and general manager of projects, begins:

It has come to our attention that some University of Minnesota (U of MN) researchers appear to have been experimenting on people, specifically the Linux kernel developers, without those developers' prior knowledge or consent. This was done by proposing known-vulnerable code into the widely-used Linux kernel as part of the work "On the Feasibility of Stealthily Introducing Vulnerabilities in Open-Source Software via Hypocrite Commits"; other papers and projects may be involved as well. It appears these experiments were performed without prior review or approval by an Institutional Review Board (IRB), which is not acceptable, and an after-the-fact IRB review approved this experimentation on those who did not consent.

This is correct. Wu and Lu opened their note to the UMN IRB by stating: "We recently finished a work that studies the patching process of OSS." They only asked the IRB's permission after they'd shared the paper's abstract on Twitter.  Then after they admitted the abstract's publication had caused "heated discussion and pushback," they removed the abstract and apologized to the IRB for causing "many confusions and misunderstandings." 

While the IRB appears to have approved this research after the fact, the Linux kernel community was not kept in the loop. The researchers claim that they spoke to people in the Linux community, but they are never identified. Hence, Kroah-Hartman's reaction when, once more, he was presented with "nonsense patches" and yet another attempt to waste the Linux kernel maintainers' time by "continuing to experiment on the kernel community developers."

Dolan continued:

We encourage and welcome research to improve security and security review processes. The Linux kernel development process takes steps to review code to prevent defects. However, we believe experiments on people without their consent is unethical, and likely involves many legal issues. People are an integral part of the software review and development process. The Linux kernel developers are not test subjects, and must not be treated as such.

This is a major point. The researchers first claim in their IRB FAQ that: "This is not considered human research. This project studies some issues with the patching process instead of individual behaviors, and we did not collect any personal information." 

In the next paragraph, though, the UMN researchers back off from this claim.

"Throughout the study, we honestly did not think this is human research, so we did not apply for an IRB approval in the beginning. We apologize for the raised concerns. This is an important lesson we learned -- Do not trust ourselves on determining human research; always refer to IRB whenever a study might be involving any human subjects in any form."

Dolan went on: 

This also wasted their valuable time and put at risk the billions of people around the world who depend on their results. While the U of MN researchers claimed to take steps to prevent inclusion of vulnerabilities in the final software, their failure to gain consent suggests a lack of care. There are also amplified consequences because Linux kernel changes are picked up by many other downstream projects that build off of the kernel codebase.

Then we get the heart of the matter. While Dolan said the UMN researchers' apology was promising, the Linux community needs more. Or, as Kroah-Harman bluntly stated:

As you know, the Linux Foundation and the Linux Foundation's Technical Advisory Board submitted a letter on Friday to your University outlining the specific actions which need to happen in order for your group, and your University, to be able to work to regain the trust of the Linux kernel community.

Until those actions are taken, we do not have anything further to discuss about this issue.

These "requests" are:

Please provide to the public, in an expedited manner, all information necessary to identify all proposals of known-vulnerable code from any U of MN experiment. The information should include the name of each targeted software, the commit information, purported name of the proposer, email address, date/time, subject, and/or code, so that all software developers can quickly identify such proposals and potentially take remedial action for such experiments.

Finding all this code is a real problem. Senior Linux kernel developer, Al Viro, who spotted the first April bogus patch, noted: "The lack of data is a part of what's blowing the whole thing out of proportion -- if they bothered to attach the list (or link to such) of SHA1 of commits that had come out of their experiment, or, better yet, maintained and provided the list of message-ids of all submissions, successful and not, this mess with blanket revert requests, etc. would've been far smaller (if happened at all)."

As it is, the Linux developers and committers are now burning time reviewing several hundred UMN Linux kernel patches. They are not amused.

Dolan moved on to ask that the paper be withdrawn "from formal publication and formal presentation all research work based on this or similar research where people appear to have been experimented on without their prior consent. Leaving archival information posted on the Internet is fine, as they are mostly already public, but there should be no research credit for such works."

Thanks to the paper's FAQ, we already know that it has been accepted for publication by the IEEE Symposium on Security and Privacy (IEEE S&P) 2021. This is a top forum for computer security researchers. The 2021 virtual meeting will be happening shortly between May 23 to May 27. The UMN has not said yet whether it will be withdrawn.

Dolan pressed to ensure further UMN experiments on people have IRB review prior to the experiment commencing. 

"Ensure that all future IRB reviews of proposed experiments on people will normally ensure the consent of those being experimented on, per usual research norms and laws," he said.

At this time, the UMN has not responded to our request for information on what the school plans to do.

The point of all this, Dolan said, is "to eliminate all potential and perception of damage from these activities, eliminate any perceived benefit from such activities, and prevent their recurrence. We would hope to see productive, appropriate open-source contributions in the future from your students and faculty as we have seen in prior years from your institution."

The Linux Foundation wants the school to respond to these requests as soon as possible. The Linux maintainers also want to know what's what with the UMN patches, so they can find them and move on. They would much rather be working on improving Linux than chasing down possible deliberately seeded errors.  

So far, they're not finding any. But when you're charged with maintaining the most important operating system on the planet, it's better to be safe than sorry.

Related Stories: