As published in HPCwire
With the announcement this week
that storage maker Xyratex has acquired Oracle’s Lustre assets, the popular open source parallel file system is once again completely under the
control of HPC stakeholders. The deal gives Xyratex ownership of the Lustre trademark, logo, and lustre.org website, as well as related intellectual
property still under the Oracle banner.
Xyratex will also take on Oracle’s remaining Lustre support contracts and bring over an undetermined number of employees from the database giant that
were involved in that effort. The deal was announced on Tuesday, with neither company disclosing the terms of the transaction.
Although UK-based Xyratex has been providing more generic storage hardware to OEMs for some time, the company took a deep dive into the HPC storage
space in 2010 when it acquired ClusterStor, a Colorado-startup that built HPC cluster storage systems of the same name. Shortly thereafter Xyratex
brought Peter Braam and Peter Bojanic on board to lead the company’s Lustre development. Braam and Bojanic were two of the original Lustre architects
and have been key to keeping the technology on course as it’s made a circuitous route around the industry, from Carnegie-Mellon University (1999)
to Cluster File Systems (2001) to Sun Microsystems (2007) to Oracle (2010), and now to Xyratex (2013).
Thanks to its open source pedigree, the file system was able to survive the 14-year journey intact. In fact, as a result of its growing developer and
customer base, the Lustre software has become more reliable and robust as it’s aged. Its moment of crisis came under Oracle, a company that has
expressed little interest in the high performance computing market. The database maker developed a version for its own commercial offerings, but
had no obvious motivation to move the technology forward as an open source HPC solution.
When Lustre was treading water at Oracle, Whamcloud emerged as a key player in coalescing the community. The company helped stabilize the Lustre source
tree and put the technology back on an HPC-centric roadmap. When Intel bought Whamcloud in 2012, it added even more credibility to Lustre’s long-term
prospects.
Building on Whamcloud’s momentum was the Open Scalable File Systems (OpenSFS) consortium, which brought together all the major Lustre developers: Cray,
DataDirect Networks, Intel/Whamcloud, Lawrence Livermore National Lab, Oak Ridge National Lab, and Xyratex. OpenSFS is committed to the continuing
development of Lustre for HPC vendors and customers and using the technology as a basis for future exascale file systems.
Xyratex is also engaged with the European Open File System (EOFS) organization, another consortium committed to the adoption of open source parallel
file systems. Like OpenSFS, EOFS is interested in a supporting and adding new features to Lustre. The organization brings with it some notable
European HPC vendors and end users, including Bull, Eurotech, ParTec, CEA, the Max Planck Society, Forschungszentrum Jülich, and Leibniz-Rechenzentrum,
to name a few.
With some of the biggest users behind it, Lustre has come to dominate the top tier of HPC systems, inhabiting six of the fastest 10 supercomputers
and more than 60 of the speediest 100. Although less frequently encountered in smaller and medium-sized HPC clusters, Lustre is still the most
common parallel file system across high performance computing. In our 2012 survey, Intersect360 Research found that the most frequently mentioned
file management software (disregarding in-house or vendor-supplied tools) was Lustre, at 23 percent.
The technology continues to push the envelope in performance and scalability as systems enter the double-digit petabyte realm. For example, Lustre
is currently slinging files for Blue Waters, the newly minted multi-petaflop Cray supercomputer housed at the National Center for Supercomputing
Applications (NCSA). That machine’s 25 petabytes of Cray Sonexion storage, which is based on Xyratex ClusterStor gear and Lustre file management,
delivers a terabyte per second of I/O to more than 25 thousand compute nodes.
Besides Cray, Xyratex has also hooked up with Dell and HP, who will resell ClusterStor systems for HPC and “big data” installations. Now that it has
cut itself a bigger slice of the Lustre pie, Xyratex may be able to entice other OEMs into reseller partnerships.
For the rest of the Lustre community, it should be business as usual. The code repository will remain open source and be available to vendors, government
labs, and academics alike, whether they do business with Xyratex or not. OpenSFS’ mission to shepherd the technology for Lustre stakeholders remains
unchanged. In fact with one of its top tier members now in control of the trademark and website, it should be even easier for the organization
to promote Lustre adoption and keep the technology evolving into the exascale realm.