What happens when you have 235 Cray XE6 cabinets and 30 Cray XK6 cabinets and you want to connect them together with datacenter class Ethernet? Well, if you are like NCSA, you go shopping and come home with an Extreme Networks BlackDiamond X8!
The raw numbers of the NCSA Blue Waters effort are astounding. 48,576 AMD CPUs, 3,072 NVIDIA GPUs, 1.5 Petabytes of high speed RAM with 25 PB of useable storage with 300 PB of RAIT-protected near-line storage all supporting aggregate IO performance of 1.1TB/s. One number you will not be seeing is anything related to the TOP500 list. The Blue Waters folks are smart and realize that optimizing a system to run specific benchmarks and optimizing a system to do real work are different things. While it is fairly simple to bolt together an arbitrarily large collection of high power GPUs and set up the system to run Linpack really fast, it is also very hard to program such systems to do useful work. To put things in automotive terms, some of these benchmarks are like doing 0-60 times on a family sedan, but only after removing the radio, the air conditioner, the back seat, gutting the interior, fitting racing tires and doing other things that would make for good numbers but a relatively poor car.
Beyond the numbers and of even greater interest would be the applications Blue Waters is being used for. Some of these include simulations of the development of the universe in the first billion years after the big bang and what happens in a Type 1a supernova when the dying star starts to fuse Carbon 12. Other efforts include modeling high temperature plasmas to better understand the impact of solar wind and solar flares, studying the disassembly of the protein capsid, a critical step in HIV infection, climate/weather simulations, simulations of eddy flows in hurricanes, studies of the helium convection shells in giant stars, turbulent fluid and combustion flows and a variety of other efforts requiring vast amounts of computational horsepower. The smallest of the small and the largest of the large, vanishingly brief events as well as massive processes spanning billions of years – these are all within the domain of Blue Waters and all of which have the potential to expand our scientific understanding of the universe.
While we are certainly proud to be a part of any effort with tens of thousands of CPUs and petabytes of storage, we also share the Blue Waters perspective that there is more to life than pure specs and that the real value of a system is what you do with it. With this in mind, considering the various scientific boundaries being pushed by Blue Waters, we are even more proud to be part of this project.
See the Extreme Networks Press Release for more on our involvement. Interested in learning more about the Cray hardware inside Blue Waters? You may find this video interesting.