Linux Foundation backs high-performance and exascale computing

High-performance computing (HPC) uses parallel data processing to deliver the speediest possible computing performance. Whether it's supercomputers, such as the Exabyte fast Frontier HPE Cray supercomputer, or clusters of hundreds or thousands of servers, it's all about taking complex tasks and running them as fast as machinely possible.

Now, The Linux Foundation, a leading open-source innovation organization, has announced two new foundations for advancing HPC and storage solutions: The High-Performance Software Foundation (HPSF) and the DAOS Foundation. 

Also: How to choose the right Linux desktop distribution for you

The HPSF aims to build, promote, and advance a portable core software stack for high-performance computing. The stack consists of 10 different programs. These are:

Spack: the HPC package manager

Kokkos: a performance-portable programming model for writing modern C++ applications in a hardware-agnostic way.

AMReX: a performance-portable software framework designed to speed up solving partial differential equations on block-structured, adaptively refined meshes.

WarpX: a performance-portable Particle-in-Cell code with advanced algorithms that won the 2022 Gordon Bell Prize

Trilinos: a collection of reusable scientific software libraries, known in particular for linear, non-linear, and transient solvers, as well as optimization and uncertainty quantification.

Apptainer: a container system and image format specifically designed for secure, high-performance computing.

VTK-m: a toolkit of scientific visualization algorithms for accelerator architectures.

HPCToolkit: performance measurement and analysis tools for computers ranging from laptops to the world's largest GPU-accelerated supercomputers.

E4S: the Extreme-scale Scientific Software Stack

Charliecloud: HPC-tailored, lightweight, fully unprivileged container implementation.

What do these programs have in common? They're all open-source, which only makes sense. For years, Linux and open-source computing have powered all the TOP500 supercomputers 

They're also all designed to simplify life for high-performance software developers. With them, programmers can gain continuous integration, turnkey software stacks, and performance testing. This will make their lives much easier. 

This new foundation addresses the growing ubiquity of HPC in scientific computing, digital engineering, and AI applications. The foundation plans to leverage investments from the US Department of Energy's Exascale Computing Project, the EuroHPC Joint Undertaking, and other international projects to enhance performance across diverse computing architectures.

Also: Linux has over 3% of the desktop market? It's more complicated than that

Key supporters of HPSF include Amazon Web Services, Argonne National Laboratory, CEA, CIQ, Hewlett Packard Enterprise, Intel, and NVIDIA, among others. The foundation will establish a technical advisory committee to manage working groups on various HPC topics, following a governance model inspired by the Cloud Native Computing Foundation (CNCF).

In short, Rick Stevens, Argonne National Laboratory's associate laboratory director for computing, environment, and life sciences, explained, "HPSF will provide a collaborative environment and access to resources that will enable the community to collectively develop the software needed to drive current and future generation HPC systems, and Argonne is honored to join as a founding member."

The DAOS Foundation focuses on the governance and development of the Distributed Asynchronous Object Storage (DAOS) project. Pronounced to rhyme with "boss," DAOS is a high-performance open-source storage system that has revolutionized performance for HPC and AI workloads. It offers a fully distributed key-value storage architecture, overcoming the limitations of traditional POSIX-based HPC storage systems.

Also: AI pioneer Cerebras is having 'a monster year' in hybrid AI computing

DAOS supports high bandwidth, low latency, and high I/O operations per second (IOPS). Again, the name of the game is speed, speed, speed. This is a mature project that Intel started working on in 2012. Intel will donate the DAOS source code to the foundation and continue as a major contributor.

Like HPFS, DAOS comes with the support of a laundry list of HPC movers and shakers, such as Argonne National Laboratory, Enakta Labs, Google Cloud, Hewlett Packard Enterprise, and Intel.

Under the Linux Foundation umbrella, these two foundations signify a major leap in collaborative efforts toward advancing open-source solutions in high-performance computing and storage. They'll foster innovation, lower barriers to entry, and support the development of critical technologies in HPC and AI/ML.