SGI today announced that large-scale SGI Altix 3000 systems have once again generated breakthrough performance results on demanding scientific applications. Recent achievements include attaining record levels of sustained performance and scalability on a 256-processor global shared-memory SGI Altix 3000 system, the largest such system ever to run on the Linux operating system.The rest of the press release:
The breakthroughs were achieved at NASA Ames Research Center as part of an agency research program between Ames, Goddard Space Flight Center, and the Jet Propulsion Laboratory (JPL) to accelerate the solution of large-scale problems in the earth sciences domain. To demonstrate the scalability and performance of the Altix system, NASA selected the ECCO (Estimating the Circulation and Climate of the Ocean) application, the product of a collaborative effort between the Massachusetts Institute of Technology, NASA’s JPL, and the Scripps Institution of Oceanography. ECCO’s importance is in improving the understanding of large-scale ocean dynamics by assimilating observational data gathered from satellites and in situ hydrographic measurements.
Running on NASA Ames’ 256-processor single system image, the ECCO ocean circulation model ran a one-quarter-degree global problem at a rate of 1.4 simulated years per wall clock day-the fastest ECCO result ever achieved on any system. “Working with the JPL team, the Altix system has enabled us to complete computations in ocean modeling involving terabytes of output at
record speeds,” said James Taft, president of Sienna Software, Inc. and currently a consultant at NASA Ames.
“Fueled by an unquenchable thirst for discovery, our partners at NASA are once again
staking new ground in computational science,” said Dave Parry, senior vice president and
general manager, Server and Platform Group, SGI. “Our relationship with NASA Ames
spans more than two decades, and today’s achievement with SGI Altix 3000 proves that
this remarkable collaboration continues to show the world what is possible with computing
technology.”
As part of their testing and evaluation of the 256-processor Altix system, researchers at NASA said they have run more than 40 benchmarks used for system evaluations at the facility. The benchmarks include production computational fluid dynamics (CFD) codes, ocean and atmospheric circulation models, numerical analysis and other mission-relevant applications at NASA. The tests evaluated single processor performance, the parallel scaling of OpenMP, MPI, and MLP, as well as interference effects in large throughput scenarios. NASA reported that the SGI Altix system performed well on all tests, and yielded significant per-processor performance gains while maintaining scalability.
“The results we’ve been seeing are extraordinary for 256 processors,” said Taft. “For instance, the OVERFLOW CFD application has generated 200 GFLOPS of sustained performance with 256 processors, which is exactly twice the performance that NASA Ames team achieved with the first 128-processor Altix system – all without optimizing application code.”
NASA sought to implement a 256-processor shared-memory system because many key production codes have been optimized to run on such systems. Researchers reported that running their codes on a 256-processor shared-memory system has allowed the facility to see immediate scalability benefits with little or no porting or tuning. The Altix family of servers combines industry-standard 64-bit Linux with the Intel ® Itanium ® 2 processor family and SGI ® NUMAflex architecture to enable global shared memory systems from a few to hundreds of processors, which is a first for Linux OS-based computing. Powered by the third-generation NUMAflex supercomputing architecture, even the largest data sets can be handled and analyzed with ease and in record time.
Only the SGI Altix 3000 family of servers is designed around this scalable shared-memory architecture that analyzes data sets as whole entities, without breaking them up into smaller segments to be handled by individual processors. The Altix architecture has proven ideal both for complex shared-memory applications running on a large single system image, and for communication-intensive applications optimized for clustering.
“Clearly, the economic environment for large-scale systems continues to force designers to
amortize as much R&D as possible from broader horizontal markets. Now we have one
company integrating and creating the system memory interconnect, another providing the
processing elements, and the open-source community providing a substantial part of the
system software,” said Bob Ciotti, lead for the Tera-scale Application Group at NASA
Ames. “This type of arrangement is resulting in systems that are highly competitive in price
and performance for large-scale computation.”
While the achievement at NASA was driven to meet the facility’s own research objectives,
the announcement builds on the formal efforts of SGI and its customers to continue to
drive the scalability of the SGI Altix platform and of Linux itself. In August, the U.S. Naval
Research Laboratory (NRL) became the first SGI customer under its global beta program
to install a 128-processor single system image SGI Altix 3000 supercomputer. The
announcement came on the heels of the August launch of the global beta program, whose
participants, along with NRL and NASA Ames, include Oak Ridge National Laboratory in
Oak Ridge, Tenn., Pacific Northwest National Laboratory in Richland, Wash., the
University of Queensland in Brisbane, Australia, and The Computing Center at Johannes
Kepler University in Linz, Austria.
Availability
Scalable SGI Altix 3000 systems are available today in server configurations of 4 to 64
processors, and supercluster configurations of 4 to 256 processors. For customers
demanding even larger Altix superclusters, SGI plans to support configurations of 512
processors in October 2003 and 1,024 processors in May 2004. Additional Altix system
technical and availability information is posted on www.sgi.com/servers/altix.
Since its introduction in January, SGI Altix 3000 has been recognized as the first Linux
cluster that scales up to 64 processors within each node and the first cluster to allow global
shared-memory access across nodes. Inspired by the success of the SGI Altix family and
the powerful combination of standard Linux running on 64-bit Intel processors, more than
60 high-performance manufacturing, science, energy and environmental applications have
been ported by their commercial developers to the 64-bit Linux environment. Over two
thirds of these applications have certified and optimized their code for differentiated
performance on the Altix platform.
This news release contains forward-looking statements regarding SGI technologies and
third-party technologies that are subject to risks and uncertainties. These risks and
uncertainties could cause actual results to differ materially from those described in such
statements. The viewer is cautioned not to rely unduly on these forward-looking
statements, which are not a guarantee of future or current performance. Such risks and
uncertainties include long-term program commitments, the performance of third parties, the
sustained performance of current and future products, financing risks, the impact of
competitive markets, the ability to integrate and support a complex technology solution
involving multiple providers and users, the acceptance of applicable technologies by
markets and customers, and other risks detailed from time to time in the company’s most
recent SEC reports, including its reports on From 10-K and Form 10-Q.
Linux can’t scale like a real Unix (read – Solaris), ask Bascule! 😛
But seriously, thats very, very cool. 😀
Interesting certainly.
Of course judging by MS reports people may actually need an Altix 3000 to run Longhorn. I think perhaps a bigger desk is going to be required :>.
A SSI as well. Very kinky.
Tears of joy are falling down my face…
Welcome to a new world of knowledge buildging and sharing. My god, does Linux rock!
—-
But remember Sun, Linux has no place on the server! LOL
http://www.suse.com/us/company/press/press_releases/archive03/sgi.h…
Linux scales. Can’t dispute that anymore.
Seems like Sun and SGI both like SuSE a lot for some reason.
They both like SuSe because Redhat has their head up their ass in the licensing area. You can bet both companies tried to work with Redhat first, and decided it wasn’t worth it.
What kernel version are they running?
I just know this is going to get me flammed, but I have to ask….Is the scalability of this machine a function of Linux or more of a function of SGI’s NUMAflex architecture
Only the SGI Altix 3000 family of servers is designed around this scalable shared-memory architecture that analyzes data sets as whole entities, without breaking them up into smaller segments to be handled by individual processors. The Altix architecture has proven ideal both for complex shared-memory applications running on a large single system image, and for communication-intensive applications optimized for clustering.
Linux should certainly be appreciated for the inroads it has made in its ability to scale (NUMA support, NPTL, 0(1) scheduler, etc), but isn’t the scalability of a shared-memory SMP machine more of a function of how the underlying architecture manages/accesses memory to avoid cache coherence conflicts and lengthy execution of spin locks?
Don’t flame, Inform.
what an incredible machine…
maybe Santa Clause will bring me one for christmas…
> I just know this is going to get me flammed, but I have to >ask….Is the scalability of this machine a function of Linux >or more of a function of SGI’s NUMAflex architecture
The below would indicate this is NOT a NUMA box.
After all, NUMA is about breaking “data”/memory into segments to be handled by certain processors.
>>Only the SGI Altix 3000 family of servers is designed around
>>this scalable shared-memory architecture that analyzes data
>>sets as whole entities, without breaking them up into >>smaller segments to be handled by individual processors. The
>>Altix architecture has proven ideal both for complex >>shared-memory applications running on a large single system
>>image, and for communication-intensive applications >>optimized for clustering.
The article contains the following:
The Altix family of servers combines industry-standard 64-bit Linux with the Intel ® Itanium ® 2 processor family and SGI ® NUMAflex architecture to enable global shared memory systems from a few to hundreds of processors, which is a first for Linux OS-based computing. Powered by the third-generation NUMAflex supercomputing architecture, even the largest data sets can be handled and analyzed with ease and in record time.
Brought to you by SGI!
*Assuming SGI has bascially used the same arch as on the Origin Mips boxes*
The Origins have CPU modules, each of which have 2-4 processors and memory slots. Those are then interconnected with high-speed NUMA links. The OS generally tries to keep threads on the same group of processors so it isn’t send large amount of data over the NUMA links, but any processor can access any memory in the system.
Linux is an effective operating choice for this system because it fits the same role as a scientific computing cluster, which would most likely be running Linux as well.
When used for anything but scientific computing, Linux is not an ideal choice for these sorts of systems because it lacks resource partitioning and management features, such as Solaris’s Fair Share Scheduler which can be used to divy system resources into shares that are assigned to specific project groups, thus it becomes impossible for any single user/program on the system to monopolize system resources.For the purposes of scientific computing (which is what this system is designed for) these sorts of features aren’t really necessary, as the entire resources of the system will most likely be monopolized by a single scientific computing application which parallelizes across the entire system with shared memory MPI.
I see this system as filling an extremely small niche. Its only advantage over a cluster is bandwidth, meaning to actually leverage the full power of the hardware the application using the system must perform relatively minimal computations on a large amount of data, enough that when parallelized the program becomes I/O bound.
We run an MPI-enabled atmospheric modelling program here called RAMS (http://atmet.com/) which is soon to be released under the GPL. This program is very much CPU bound, to the point that using GigE instead of 100bT for interconnect between nodes of a cluster has a negligable effect on the performance of the model. Purchasing an Altix system for this purpose would be a horrific waste of money compared to a cluster with the same processors (although we always go x86 for cost reasons, although some are moving to Opteron). The performance of our scientific computing application on an Altix system or a cluster with 100bT interconnect would most likely be negligable, as RAMS would not benefit from the extra bandwidth afforded by an Altix system.
All of that said, keep in mind that the Altix should not be seen as a competator to some of Sun’s high end offerings. Sun’s systems fit roles of mission critical systems where Solaris’s resource management is needed. The resources of a system of this caliber need to be explicitly dedicated through Solaris mechanisms that Linux lacks, such as a Fair Share Scheduler.
I haven’t taken too much time looking through the Altix whitepaper, but I’m still not certain if Linux/Altix has been patched to include a number of facilities that are present on Solaris. For example, since Solaris 7 a scrubber has been available for Solaris, so that if a CPU malfunction is detected, provided it doesn’t bring the kernel down all processes on that CPU are moved off of it. The same functionality is provided for system RAM as well. Any of these components can then be hot swapped for good ones after a failure has been detected. In this way a mission critical system can continue running despite the failures of individual components.
The Altix does not seem designed for such mission critical roles, only for scientific computing purposes where it fits the same role as a cluster of low end systems. I don’t see it as being competative with clusters in any way, and to reiterate it simply fills the very small niche of highly parallelizable I/O bound scientific computing applications. In such a role the system scheduler isn’t taxed as each processor will be effectively used for a single process, so it’s possible for Linux to be used as a platform for these purposes as “high end” features like project groups aren’t necessary.
However, don’t expect a Linux-powered system to be filling the role of a Sun Fire 15k any time soon.
They used Craylink interconnects on the Origin series systems. I think what they are using today is a generation ahead of the Origin systems. Probably faster interconnects with optimized hardware and software for a more efficient and scalable system.
Arch: Origin 2000
CPU: 300Mhz MIPS R12k
RAM: 4GB/brick
Interconnect: 1.6 GB/s Craylink
Arch: Altix 3000
CPU: 1300Mhz Intel Itanium2
RAM: 32GB/brick
Interconnect: 6.4 GB/s Numalink
I think O2000 had more dynamic hardware configurations, but the Altix is probably at least an order of magnitude faster. I helped build an O2000 64 processor Reality Monster a few years ago. Very nice system to work with!
Christopher X (IP: —.ok.ok.cox.net)
Linux can’t scale like a real Unix (read – Solaris), ask Bascule! 😛
As I stated in my above post, Linux fits the role of providing a platform for MPI applications, as in this role it will manage a single process per processor. In terms of scalability for a general purpose server, it lacks the facilities needed to effectively manage the system, unless SGI has provided these through kernel patches and system-specific management tools.
Tim (IP: —.sympatico.ca)
Linux should certainly be appreciated for the inroads it has made in its ability to scale (NUMA support, NPTL, 0(1) scheduler, etc), but isn’t the scalability of a shared-memory SMP machine more of a function of how the underlying architecture manages/accesses memory to avoid cache coherence conflicts and lengthy execution of spin locks?
I think it’s much more a function of how the scientific computing application the system is being used for parallelizes itself. When using shared memory MPI as traditionally implemented by MPICH, features like the O(1) scheduler (that assuming they’re using a 2.6 kernel) and NPTL are irrelevant, as processes are used instead of threads and communicate using SVR4 shared memory (at least in the MPICH implementation). The system scheduler in such a setup manages little, as each processor will be monopolized by a single process.
I can only hope that SGI is providing their own MPI implementation which has been optimized specifically for the Altix, as a typical MPI implementation (such as MPICH) would not be able to utilize this system effectively.
From SGI’s four color glossies of the system, it does appear they have resource partitioning and high availability features:
System management tools: Partitioning, FailSafeTM (for high-availability environments), Comprehensive System Accounting, job containers
I’d greatly appreciate a link to the whitepaper on these features if someone is able to find one. I’d certainly like to see how it has been implemented and how actual management tasks are carried out.
It’d also be nice to know if the source code to these components has been released (as I don’t believe any of them are available in the Linux mainline)
It also appears that SGI does provide their own MPI implementation as part of their Message Passing Toolkit:
http://www.sgi.com/software/mpt/overview.html
(perhaps I should read a bit more before posting)
fyi, Oracle, SuSE and SGI are beginning to promote the Altix for DB server use.
fyi, Oracle, SuSE and SGI are beginning to promote the Altix for DB server use.
The only endorsement I was able to find for Oracle on the Altix comes from this SGI press release:
http://www.sgi.com/newsroom/press_releases/2003/august/suse.html
It contains this quote from Dave Dargo, vice president of Oracle’s Linux Program Office and the Performance Engineering team within the Platform Technologies Division at Oracle Corporation:
“Oracle welcomes SGI and SuSE’s alliance, because it will enable Oracle customers on that platform to avail themselves of Oracle’s support for Linux,” said Dave Dargo, vice president of Platform Partners, Linux and Performance, Oracle Corp. “SGI servers, coupled with solutions based on Oracle Database and SuSE Linux Enterprise Server, will give customers increased choice for their enterprise Linux applications.”
With that as the only endorsment for the Altix from Oracle, somehow I don’t see many people flocking to the Altix as an Oracle platform.
One of the big questions with the Altix is how do you determine who is competant to administrate it? Are the tools described in the four color glossies simply ports of the Irix tools? I would hope so, but there’s no way of knowing without reading the whitepaper.
I found this link to the Altix whitepaper but registration is required in order to view it:
http://www.sgi.com/servers/altix/whitepapers/index.html
I bet 10,000 quatloos that the Linux kernel running on that box isn’t something you can get your paws on…
bascule:
With that as the only endorsment for the Altix from Oracle, somehow I don’t see many people flocking to the Altix as an Oracle platform.
There was quite a bit of talk at Oracleworld. OTOH, Oracle have clearly indicated that they think the days of the megaboxen are limited and the way forward is RAC.
I’m inclined to agree with them. We’ll certainly be migrating some of our clients to RAC’s during the next upgrade cycle. The price performance ratio compared to single monolithic boxes (of whatever flavour) is killer in this scenario.
I bet 10,000 quatloos that the Linux kernel running on that box isn’t something you can get your paws on…
http://oss.sgi.com/projects/sgi_propack/
I’d like a tripple breasted whore with my quatloos please.
I bet 10,000 quatloos that the Linux kernel running on that box isn’t something you can get your paws on…
As Linux is licensed under the GPL, this would most likely be illegal. I don’t believe it would be possible for Linux to operate effectively/at all on this system without directly altering the kernel source.
More than likely SGI is either giving away or selling the source code (selling would be an interesting scenario, as they can only sell it for however much they’re factoring the cost of Linux itself into the Altix)
It seems that someone can’t handle an informed analysis of the posted article. What exactly was it about Bascule’s comments that was abusive? I’m a little curious.
As Linux is licensed under the GPL, this would most likely be illegal. I don’t believe it would be possible for Linux to operate effectively/at all on this system without directly altering the kernel source.
GPL states that if someone gives you the moidifed binary, they have to provide you the source. In this case, only NASA has right to demand the access to modified kernel code.
I agree, who’s doing that to Bascule? The guys opinionated, but he’s no troll.
GPL states that if someone gives you the moidifed binary, they have to provide you the source.
Correct
In this case, only NASA has right to demand the access to modified kernel code.
Careful, I don’t think you have enough information to draw that conclusion. You would have to know if SGI is charging anything for Linux on the Altix. If they are, then under the “equivalent access” clause of the GPL (http://www.gnu.org/licenses/gpl-faq.html#DoesTheGPLAllowDownloadFee) they must make the source available for a price equal to or less than what they’re charging for the Linux binaries.
http://www.gnu.org/licenses/gpl-faq.html#GPLRequireSourcePostedPubl…
————–8<————–
The GPL does not require you to release your modified version. You are free to make modifications and use them privately, without ever releasing them. This applies to organizations (including companies), too; an organization can make a modified version and use it internally without ever releasing it outside the organization.
But if you release the modified version to the public in some way, the GPL requires you to make the modified source code available to the program’s users, under the GPL.
Thus, the GPL gives permission to release the modified program in certain ways, and not in other ways; but the decision of whether to release it is up to you.
————–8<————–
Therefore, the only thing they have to do is to provide NASA with the source of their kernel version.
Interesting article i found there that appeard on linuxjournal back in jan. It’s by the “Linux engineering director” at sgi
http://www.linuxjournal.com/article.php?sid=6440
The GPL does not require you to release your modified version. You are free to make modifications and use them privately, without ever releasing them. This applies to organizations (including companies), too; an organization can make a modified version and use it internally without ever releasing it outside the organization.
But if you release the modified version to the public in some way, the GPL requires you to make the modified source code available to the program’s users, under the GPL.
Please click this link: http://www.gnu.org/licenses/gpl-faq.html#DoesTheGPLAllowDownloadFee
You will clearly see that if you sell binaries of a GPL program, you must make the source available for an equal or lesser price.
Therefore, the only thing they have to do is to provide NASA with the source of their kernel version.
If they are selling the kernel binaries, they must make the source available to anyone for an equal or lesser cost. See the link.
Of course, this is all moot given this: http://oss.sgi.com/projects/sgi_propack/
…although I don’t see the high availability tools there. How does one hot swap a defective CPU or RAM? Is this even possible on the Altix, or would a defective CPU or RAM bring down the entire kernel?