Addison Snell of Intersect360 Research presented the firm’s findings on HPC Software Environments in 2014 to the members of the HPC500, an elite user group
of organizations that represent a worldwide, diverse collection of established HPC professionals from a cross-section of academic, government, and
commercial organizations.
The presentation drew on findings from several different reports. Intersect360 Research recently released two HPC User Site Census reports, one on applications
and one on middleware. The reports examined the supplier, products, and primary usage of the application software and middleware reported at a number
of sites over the previous year. The presentation also included findings from the research firm’s annual HPC user budget map, which tracks spending
patterns and shows a percent of spending by category, as well as from a special study on “The Big Data Opportunity for HPC,” which surveyed both HPC
and non-HPC enterprises (through a partnership with Gabriel Consulting) on Big Data applications and infrastructures.
Intersect360 Research segmented their research this year into High Performance Business Computing (HPBC) and High Performance Technical Computing (HPTC)
super-segments. About three-fourths of applications surveyed were HPTC packages, such as structural mechanics, computational fluid dynamics, and weather/environmental
modeling. The remaining one-fourth were HPBC packages, including financial analysis, business intelligence, and MapReduce-based applications. More
than half of HPBC packages were business intelligence applications.
Eighty percent of the applications — a mix of in-house, packaged, and open-source applications — were mentioned only once in the HPC Applications
survey, indicating the diversity of application needs and the experimental nature of the HPC market. This trend also held true with middleware, where
62% of applications were only mentioned once. The amount of unique middleware packages tracked more than tripled over the previous year with the addition
of more than 340 packages, for a total of 496 packages.
Open Source emerged as an ever more important component of the HPC software landscape, especially among commercial HPC users. Open Source was the leading
source of applications with 38% of software mentions, its rise driven largely by pricing models and new categories.
The increasing dominance of Open Source presents a challenge for Independent Software Vendors (ISVs), who are already struggling with the inevitable growth
of multi-core and many core systems. These systems have caused stress on their business model due to the evolution of new processing paradigms and
the resulting changes in licensing models for ISVs. This is especially visible in the Computational Fluid Dynamics space, where Open Foam is the fastest
growing application in the HPC market, challenging the leading ISV package, Fluent by ANSYS.
Traditional market thinking is that software in commercial HPC is dominated by commercial ISVs, but it is instead split almost equally with one-third apiece
for in-house, commercial ISVs, and open source solutions. This differs between vertical markets, of course — for example, biosciences tends to
use more commercial applications, whereas energy slightly favors in-house applications. HPC usage in commercial markets overall is growing, especially
as compared to the public sector.
Big Data provides a breakout opportunity for HPC, and is currently comprised primarily of in-house applications. For non-HPC sites (businesses who don’t
think of themselves as using HPC) the use of in-house applications is diminishing as Open Source solutions are increasing, due to the growing acceptability
of Hadoop-based platforms.
Over time, more commercially available applications are expected, which will result in fewer in-house and open-source applications. This is a familiar
pattern also seen over the decade-long span of ERP adoption, which began as high-priced consulting for large enterprise and ended with any enterprise
organization being able to purchase and implement an existing package.
Last month Intersect360 Research was commissioned by the U.S. Council on Competitiveness to survey 100+ HPC-using
companies in industries such as manufacturing, energy, finance, pharmaceuticals and entertainment. The interviews formed the basis of Solve. The Exascale Effect: The Benefits of Supercomputing Investment for U.S. Industry,a
publication of the High Performance Computing (HPC) Initiative.
The findings in Solve that pertain to software focused on the barriers to HPC scalability. Many of those surveyed thought they could use 10x greater scalability,
and they cited scalability of software — especially in regards to new models of parallelism (multi-core and many core) — as the major limitation
to achieving this goal.
The presentation concluded with a lively Q&A session with HPC500 members. The HPC500 is comprised of the foremost people who bring high performance
computing technology to bear on challenging problems in science, engineering, and business. HPC500 members are leading organizations at the vanguard
of their areas of specialization, spanning geographies, budget sizes, and application areas.