This is the second installment of the “Linux on the Opteron, are we ready?” article. Basically, it is a “where are we now?” article, noting that what once did work now does not, and others that did not work now do. The first article was published on OSNews almost three months ago. Since that time not too much has happened publicly in regards to the amd64 Linux situation, but a lot of people mailed to tell me that I should have checked out SuSE or the new Mandrake which was “about to be released” at that time. Also since that time I have upgraded the RAM and acquired a larger hard disk for the machine. I will give a brief rundown of the system as it stands now, what I tried to install on it, and what works.
The system now: Basically it is the same as before with some minor changes and a BIOS update:
Motherboard: Asus SK8N, BIOS upgraded to 1004
CPU: AMD Retail Opteron 240 (1.4GHz)
RAM: two 512MB sticks of Kingston (ASUS recommended 333MHz dual channel DDR, ECC registered)
DISKS: One Maxtor 7200rpm 80GB UltraATA drive, One Liteon CDR/RW drive.
Network: One $5 Linksys PCI 10/100 NIC
Video: ATI FireGL, AGP, 128MB ram.
Case: Lian-Li PC-7, with optional Coolermaster white cold cathode fan in the back (makes it easier to see in the case when working on it)
Power Supply: Enermax 460Watt, fan monitor controlled.
Recall from the previous article, that this is the same setup with a new case, new ram, bigger drive, different video and bios update.
Also the 3COM NIC is now in another machine.
So, first, a super quick rundown of my previous experience with this setup (mostly from the first article, and more observations from time spent with the machine about a week after the article came out):
Mandrake Beta2 for AMD64 worked, sluggish drive performance, KDE worked like a charm, intermittent lockups, but mostly only when writing large files to the drive (e.g. a Linux iso).
Red Hat Taroon beta for AMD64 worked also, X worked slowly and again, had intermittent lockups similar to the Mandrake install.
Gentoo for AMD64 worked great and fast, but no X windows. (you could run X in twm mode but no other window managers)
SuSe 8.2 for AMD64 worked, but again sluggish IDE performance and compilation of 32bit software would fail badly. KDE worked.
Red Hat 9.0 for i386 worked fully, but X acceleration was not supported.
Note that in all of the above setups with the exception of Mandrake
that compilation of 32bit applications failed. Precompiled binaries worked
as far as I could tell (with the limited number of binaries I loaded).
Keep in mind that all of the above was with the Nvidia Geforce4, which
has been replaced by the ATI FireGL. After the article was
said and done, I installed Red Hat 9.0 for i386 and used it until
I switched back to my old AthlonXP so I could have accelerated X windows.
Note that I did try the FireGL in the original Opteron setup also.
Having a 64 bit machine sitting around collecting dust is not
my idea of a good time, so I decided (since the semester is over 🙂
to have another go at installing again, and also to see what improvements
were made in distributions since some new releases have come out.
Specifically, I have installed (or tried to) the following (all of
which are the 64bit version, also known as amd64 or x86_64):
Gentoo LiveCD beta (10-24-2003)
Gentoo LiveCD beta (11-09-2003)
Mandrake RC1
Suse 8.2 beta2
Suse 9.0 (ftp install, no iso files)
Red Hat Taroon (again, to see on the FireGL)
I also installed Fedora core 1 to check it out just to compare
since I had installed Red Hat 9 on the original Opteron setup.
I shall present my problems and observations with the above
distributions in order, but first I would like to say two things:
1) YMMV. I have read through some of the problems others have had
and how they overcame them on various mailing lists. It seems
that some of the problems I had were specific to my setup and
others did not have them at all on other motherboards or with
other video cards.
2) I did not spend all year testing each distribution. I was
on a quest to get something on my Opteron that I can run my
analysis software on, and compile my OSS on. I wish to use
the Opteron due to its large cache and 64bit architecture.
Having said that, I did not just install and say “eh, it does
not work” and move on either. I at least tried to get each
one working and looked for ways to overcome problems, being
as I had already spent time putting the software on the machine.
Ok, here we go. Gentoo LiveCD 10-24-2003: this is the one
I actually used the least, as I could not get X working at all,
and wanted to move on. I will note however, that with the
first Gentoo amd64 release that I picked up (not the liveCD)
X worked, but no window managers would compile (less the built
in one). It is worth mentioning that everything else worked,
and worked well and fast. I would recommend this (actually
the 11-09-2003 CD) for servers.
The second Gentoo LiveCD (11-09-2003) was no different from
the first (IMHO). It seemed like everything works with the exception
of the X, _however_ X would not even compile with this distribution.
A little poking around the IRC channel and the bugzilla list
did not seem to point to anything in particular, and people on
the list said they had no idea. Others seemed to have it working
but could not get the Radeon driver to work, while even more
users said it worked fully. I received compile time errors
on Xfree all over the place, too many for me to give a good diagnosis
of. I gave up on this one and moved on. Again, other than that,
it is using the 2.6.x line of kernels, all of my hardware was
found (with the exception of the on board NIC, which no distribution
of Linux can “see”), and it is quick. I recommend this for a server
box.
Before I stuck in the Mandrake RC1 CD, I did have that funny smile
on my face, knowing that since Mandrake beta1 worked so nicely with
the exception of the “writing a large file to the drive lockup” that
this was going to be the distribution. This was going to work, and
work great. (sometimes I have a soft spot for Mandrake) This was not
the case. The install would go smoothly for the first two steps and
then tell me that there is a problem with my hard disk and that it
needs to be wiped, and would I like to proceed? Odd. Telling it to
move on and wipe the disk results in an error: “Can not format
disk, drive error”. I put in the Gentoo disk and fdisk the drive,
no problems. I then partition the drive into three partitions (/boot,
swap and /) so that Mandrake can have an easier time installing.
Again, when I go to install Mandrake it tells me that the drive
is corrupt/damaged/no space, and would I like to wipe/format it.
I tell it yes, and then it tells me that there is a drive error.
To make a long story short, I checked the drive, I even installed
Fedora core 1 on it, and Mandrake still would not install. I even
went so far as to try all possible hard disk access modes in the
BIOS with no luck. Mandrake was no go. I decided to fall back
on Mandrake beta1 and then see if I could upgrade. Mandrake
beta1 had the exact same error. My conclusion is that the updated
BIOS has some issue that Mandrake hates. Oddly, no other distribution
exhibited this problem.
Suse 8.2 beta2 went well during install, and seemed to not
have any problems. It did identify that there is an on board
network controller on the ASUS, but could not do anything with it.
After install I started to have the same problems that I had with
Mandrake etc on the old setup, random lockups when writing huge
files to/from the drive. Other than that problem, everything seemed
to work ok, even X and KDE. Having seen that 9.0 was up on the
ftp site, and having some time to install it, I decided to once
again scrap the drive and try out Suse 9.0…
Suse 9.0 (what I am running now, and typing this on) was a fairly long
install (due to my slow net access). The install took about two and a half
hours to fully complete from start to finish.
I only encountered two problems with Suse, one of which is also present
in some fashion in all other distributions with the exception of Gentoo:
ACPI. In Suse (again, on this ASUS motherboard) unless you turn ACPI
off (by going into failsafe on the install, or passing the acpi=off
kernel argument) the kernel will fail to load completely. Slightly
annoying. Secondly, during the install phase, it would not let me change
the resolution on the video card (it was defaulted to 1024×768) and
I wanted to change it before I “started the system”. It is not a
big deal, but made me wonder if anything else that I left default
before would also have this problem. I am not sure, and after
2 hours of installing via ftp, I did not want to go back and find out.
In Suse 9.0, KDE works, X works, and amazingly, OpenGL works.
I have not noted this before, but no other distribution (which
has had X working at least) has been able to do OpenGL on my
ATI card. Xfree under Suse is using the “radeon” driver, but so was
Mandrake and Red Hat 9.0 (original setup, with the FireGL installed).
I make a second notation here in that the Nvidia also would not
do OpenGL on any distribution (original setup). I was slightly
pleased to see this. The only questionable aspect of the Suse
distribution is the choice of kernel, which is 2.4.21. I know
that 2.6.x is beta for now, but it does seem (from the Gentoo
installs) that it is faster and able to play nice with the ACPI,
unlike 2.4.x on this motherboard.
Some notes that may be of interest, that I have made along the way:
Nvidia cards on these boards are very hard to setup, I suggest
something else, or wait. Nvidia has new 64 bit compiled binaries on
the Nvidia web site, but I have not tried them as my Nvidia card
is now stuck in another machine. When installing on the Opteron
which for now I think all motherboards use the Nforce3 chipset,
pass two kernel parameters just to be safe: acpi=off and idebus=50
I found out by reading the source that the idebus option will
not take a value greater than 50, but setting it to 50 is a
LOT faster than the default 33MHz mode. If you have an all SCSI
system, skip the idebus argument.
Final thoughts:
Benchmarking:
Again, I do not have any benchmarks on CPU performance to show
off. I did run the povray benchmark, but looking over the Povray benchmark
page I see P4 3.0GHz machines getting beat out by Athlon 2800’s and
vice-versa, so I have little faith in the results (or at least in
peoples postings to the site). Other sites on the net have run benchmarks
on the Opteron, but without a 64 bit OS, and without 64 bit compiled
benchmarks, I also have little faith in them. They should be used as
a “touchy-feely-sort-of-may-bee” mark at best, in my opinion.
Cost:
This is the major factor in my mind. The setup I have above is
actually cheap, when you think about it. For those bargain shoppers
out there, this may not seem like a cheap computer at all, but for
those of us who need 64 bit computing, its a deal. For example, the
configuration above costs (price from pricewatch as of
12-16-2003) $1196 with shipping, $300 of which is the video card.
Yes, this may sound like a huge amount for a computer, but consider
the alternatives. A Sun Blade 1500 with comparative features for instance.
The “large” model comes with 1GB ram and an 80GB IDE drive,
it costs $3995, and the Ultra-SPARC CPU is running at 1GHz.
While it is unclear of the performance differences between say
the Opteron and an Ultra-SPARC or a Power CPU, a large corporation
could just purchase one of each and perform their own internal
benchmarking, if they were considering purchasing any more
than a few machines.
“And now for something completely different”:
Again on cost, if you are doing normal day to day activities on the
machine and do not need 64 bit computing, the cost is large. I would
suggest a AMD Athlon XP Barton instead. In general, the performance
of loading and using a web browser, or Open Office, etc,
is about on par with a speedy AMD Athlon XP. I do not have any Intel
based machines to compare it to, so that is why they are left out
here. The cost difference is non trivial in the AMD line too, as a Athlon XP
based machine is going to be much less than an Opteron or FX/64.
I digress once again, having said that, and make the notation that
for a 1.4GHz machine, the Opteron is very speedy.
Rant on Suse 9.0:
In the past I have mostly run Red Hat. I have been using Linux since
1992 (tarballs and TSX-11 archives from MIT), and after slackware came
out I moved onto that, and then Red Hat. Today is a different story,
as we know of the recent changes at Red Hat. I have previously used
(or at least tried out for a week) SuSe 6.2, so I am not a complete
newcomer to SuSe. I find that SuSe 9.0 was very impressive for a
distribution that I think is still classified as beta. Everything
worked out of the box (application wise) and the Xwindows performance
was impressive (compared to the others). My suggestion thus far for
anyone building an Opteron/FX system who wishes to run Linux would
be SuSe 9.0, although the install could take an extreme amount of
time if broadband is not available. One option is to have someone
with broadband mirror the ftp site and then perform an NFS install
if you have a spare machine to mount the CD in. The 9.0 install
does not have ISO’s and can not be installed via CD. This is really
a shame as it is (thus far, IMHO) the best distribution out there
for the Opteron/FX in a desktop/workstation environment.
Conclusion:
The coming months (weeks?) should be interesting in that
Mandrake is set to release the AMD64 version any time now,
as they are taking pre orders for it in the Mandrake store.
Recall, it was one of the best (if not the best) in my first
review, and I blame the drive problems on the Asus BIOS update.
Gentoo is nearing (from what I read) a really stable working
system, and I have read repeatedly that others have it working
fully (as a workstation with X windows) on other motherboards,
so I again blame the Asus for my troubles with Gentoo. Red Hat is
another story, having dropped the desktop edition, the “workstation”
edition is well beyond my financial reach. A corporation may consider
purchasing a copy for evaluation, but I would be tempted to wait
on Mandrake or Suse.
FYI: costs as of 12-16-2003 for AMD64 Linux distributions:
Mandrake pre order $100
Mandrake corporate server $750 (standard support) $1500 (unlimited support)
Red Hat AMD64 workstation $792
Red Hat Advanced Server $1992
Suse Professional 9.0 $120 (distribution on DVDs, no CDs)
Suse Enterprise Server $767 (2 cpu) $1450 (4 cpu)
Looking at the above cost matrix and my experience, it is almost tempting to purchase SuSe just to have the DVDs (no CDs, strange). The enterprise/server editions seem to all be priced about the same, with no definitive mention of CPU capability from RedHat or Mandrake on the server editions. (I assume at least 2 CPU capability built into the kernel)
Final thoughts on hardware:
The Opteron system can be run with a 365 Watt power supply, which is what I was using in the first setup. I switched to a larger one
as more disks will be added to the system. One need not purchase a huge power supply for this system unless you need it, but a warning is in order: looking over the CPU specs, be sure to purchase a good power supply that can sink lots of current on the 3.3V line. Purchase good, high speed RAM. I switched from the 266MHz to the 333MHz _dual channel_ RAM and the difference was non trivial. As always, a good case is in order, not that the Opteron gets very hot in my Lian-Li, in fact after large compilations for hours the CPU heatsink is only warm to the touch, but the RAM can get hot, as well as the video card. Besides, you don’t want $600 worth of RAM/CPU/motherboard to be sweating it out in your case.
About the Author:
Robert Minvielle is a Ph.D. student at the Center for Advanced Computer Studies at UL at Lafayette. He has a Masters in Physics and a B.S. in Computer Engineering, and enjoys building hardware, embedded systems, and programming. He can be reached at [email protected].
i’m a fan of AMD. But I think I’ll wait on getting an opteron.
i don’t want to be a beta tester (again).
thanks for the info.
I have an asus sk8n opteron system. I’m running gentoo on it 32bit only and after a small struggle (nearly) everything is working. The only thing that doesn’t work is the onboard NIC. I’ve got x and dri opengl working with my firegl card. I’m not brave enough to try a 64bit install yet. One question, can someone expand on the mentioned idebus=50 kernel option? what does it do and is it safe?
One last thing, the driver (talking kernel 2.6 test 11) for the promise sata controller is still beta and causes my drives to dma-timeout sometimes.
Well, the idebus argument is passed to the kernel to force
the PCI IDE speed. Notice on your gentoo/opteron system that
if you check the dmesg output you will see that the drives
are falling back to PIO mode at 33MHz. The idebus=50 kernel
arg will up this to 50MHz… or you can take rac’s patch
for the nforce3 chipset and apply it to the kernel to make
full use of the Nforce3 chipset and possibly go UDMA.
Yes, I have seen the promise driver have problems… I would
wait on it.
I have also gotten the opengl/etc to work on the firegl
in 32bit land on the opteron, but not in 64 bit under
gentoo.
Cheers.
Why didn’t you try the FreeBSD port ? They list AMD64 as a tier 1 (fully supported) architecture. I’d try it myself but I sadly lack the funds for this type of system.
My guess is it sets the bus speed to 50 Mhz for the IDE controller, possibly fixing some timing problems which shouldn’t be there in the first place.
I have a motherboard with some ITE chipset IDE RAID and serial ATA, but I didn’t have any drivers for any of that in any of the Linux distros I had handy at the time so I threw in a PCI promise IDE controller and was able to quickly build a RAID this morning. I wish chipset manufacturers could offer some backwards compatibility or Linux drivers offerred more forward compatibility or something.
I agree with the post above, I don’t want to be a beta tester, again. I just want things to work out of the box. Is that so much to ask for? I mean we give everyone full access to the source code, including drivers for all their old hardware. What more do they need? And who’s fault is this? Ours? For running any OS other than Microsoft Windows?
I’d greatly like to see this myself. Unfortunately I don’t have an Opteron system at the present time…
solution in here ?
http://bugs.gentoo.org/show_bug.cgi?id=31457
I’d guess all the problems experienced are due to the ASUS motherboard and its shoddy BIOS/poor engineering.
Every ASUS motherboard I have owned (ASUS A7Pro, CUPLE-VM, A7N8X) has required BIOS upgrades to work correctly, onboard peripherals diabled, exhibited lockups and have just generally been a pretty poor experience ‘out of the box’
Sure, after you put in 20 hours work or so hunting for information on how to work around these issues and upgrading BIOSes multiple times, the boards can be made to run relatively stably.
But ASUS, as far as I am concerned, don’t produce quality linux-ready boards, so if you buy their rubbish, expect to have to waste hours fixing their problems.
My latest board, the A78NX ‘Deluxe’ gives me massive disk corruption under a 2.4 kernel and locks up at least once a day under either 2.4 or 2.6 if APIC/ACPI is enabled.
booting the kernel with ‘noapic nolapic acpi=off’ seems to give me stability, but why should I have to run with these features diabled – ASUS are clowns.
There is a Preview Release of Fedora Core 1 for AMD64 systems available now (actually since Dec 1). It certainly represents another option for a cost effective Linux Opteron solution. Check it out at: http://www.linuxtx.org/amd64faq.html
Hi,
We have been testing Linux (both Redhat and Suse) in Opteron chip and we have got really good performance boost when applications are re-compiled for Opteron. Avegare is about 30% faster than using 32-bit binaries. After testing Opteron with Linux, I never wanna go to Intel world.
Check out this info should help you out with issue
http://dev.gentoo.org/~brad_mssw/amd64-tech-notes.html
I’ve got 12 Opterons at work:
6x IBM e325 Dual 2.0ghz
6x Rioworks HDAMA Dual 1.4 & 1.8ghz Models (4xCCSI & 4x whitebox)
I’ve loaded MANY linux distro’s with the following notes:
1. RedHat (GinGin64)
Works perfectly on e325 (Using USB->PS2 Adapters as the e325 has no PS2 ports).
Works on HDAMA, but KDE doesn’t feel as “snappy” as Mandrake.
2. Mandrake (Corporate)
Works perfectly on HDAMA, but no promise SATA support (blame promise).
USB->PS2 Mouse problems (would not connect) on IBM e325. USB Mouse works fine, but tell that to your corp KVM switch…
3. Suse (Corporate)
Works beautifully on IBM e325 (IBM has fully supported drivers).
4. RedHat 8.0 (32bit)
Worked fine on HDAMA, but was slower than GINGIN64 Beta
5. Windows 2003 (32bit)
Loaded w/o problems on HDAMA (use current BIOS). Nice to see windows start in 18 seconds.
Does not load on e325. Had to load Windows 2000, and in place upgrade to 2003 (then worked fine). I assume some BIOS issue that IBM will need to address.
Is there a ‘home user’ 64 bit linux version out now. Or do you have to go the Enterprise route?
http://fedora.linux.duke.edu/fc1_x86_64/
Search the fedora-devel list archives for more info.
I would prefer to use 64bit Windows, because there will be applications, games and other software with functionality. If you want something else of no use, then just use linux.
We have a 150-node dual CPU Opteron cluster soon be processing data.
Although some of our applications are restricted to 32-bits because 64-bit builds of some 3rd party libraries are not available yet, we still gain a handsome 35% speed up when running our 32-bit apps on a 64-bit OS, like SuSE’s 64-bit Linux OS.
Preliminary tests show that we gain another 35% when our apps are 64-bit; for a total speed up of ~70% over pure 32-bit.
[quote]I’d guess all the problems experienced are due to the ASUS motherboard and its shoddy BIOS/poor engineering.
Every ASUS motherboard I have owned (ASUS A7Pro, CUPLE-VM, A7N8X) has required BIOS upgrades to work correctly, onboard peripherals diabled, exhibited lockups and have just generally been a pretty poor experience ‘out of the box’
Sure, after you put in 20 hours work or so hunting for information on how to work around these issues and upgrading BIOSes multiple times, the boards can be made to run relatively stably.
But ASUS, as far as I am concerned, don’t produce quality linux-ready boards, so if you buy their rubbish, expect to have to waste hours fixing their problems.
My latest board, the A78NX ‘Deluxe’ gives me massive disk corruption under a 2.4 kernel and locks up at least once a day under either 2.4 or 2.6 if APIC/ACPI is enabled.
booting the kernel with ‘noapic nolapic acpi=off’ seems to give me stability, but why should I have to run with these features diabled – ASUS are clowns. [/quote]
Sorry to hear you have issues with your boards, but I wish to point out that along with Tyan, Asus has and continues to produce the best consumer boards on the market. I strongly suggest you look in the mirror for the source of your problems.
Quoting the author if the article “Suse Professional 9.0 $120 (distribution on DVDs, no CDs)”
That is simply not true. Here are some quotes right off of SuSE’s website:
Scope of the Personal edition ($39.95):
3 CD-ROMs, 1 manual (User Guide), 60 days of installation support
Scope of the Professional edition ($79.95):
5 CD-ROMs, 1 double DVD, 2 manuals (Administration Guide, User Guide), 90 days of installation support
32-bit Suse worked perfectly on the Opteron/SK8N. I was able to find drivers for everything and they worked (I didn’t bother with the Promise card). I ran FFTW benchmarks. I then installed the 64-bit Suse 9.0. There is no comprehensive driver package. Following instructions at Suse’s website I was able to patch, compile, and install Nvidia’s 3d drivers. I think they are working but the performance is about halfway between the 32-bit 2D only drivers doing 3D emulation, and Nvidia’s 32-bit 3D drivers. The sound system worked without any extra help. However I cannot get the integrated network controller to work. I ended up throwing an old 3Com NIC in and that worked. I haven’t tried to configure the Promise card yet.
My SK8N is bios 1003. I cannot flash it to 1004 for some reason….
I compiled and ran the FFTW benchmarks under the 64-bit Suse and the results showed a solid 20% increase in speed. FFTW is Fast Fourier Transform code….
Both installations of Suse were done from retail DVD’s.
It’s curious you couldn’t get Suse 9.0 Pro on CD. I just bought a copy of it 2 weeks ago for $80 and it came with BOTH a DVD and 5 CDs.
Look in the mirror?
The system doesnt stay up for more than 2 hours with ACPI and APIC turned on, and its MY fault?
Turn APIC and ACPI off and the system runs solidily (5 days and counting), and this is not because of Asus’s poor BIOS and/or engineering?
It’s ME? Oh, of course! it’s ME who is to blame for the broken APIC/ACPI on the motherboard! of course!
Is it the reviewers fault his motherboard (Hey, what a coincidence, its an ASUS) exhibits very similar bugs and lockup problems under Linux?
Are you seriously claiming that all the threads on the LKML about NForce 2 hard lockups (mostly affecting the A7N8X) are all the fault of the customers of these boards?
Thats ludicrous, and you dont know what youre talking about.
I was wondering if anyone knows of any functioning workstations based on Tyan K8W (dual opteron monster board) running 64 bit Linux (any variety, but I’d be particularly interested in Gentoo specimens) with SMP enabled? Our lab is thinking of acquiring such a beast.
Tim:
Gentoo has free ISOs available for download, but no official release yet. It sounds like the same is true for Fedora.
[quote]I’d guess all the problems experienced are due to the ASUS motherboard and its shoddy BIOS/poor engineering.
Every ASUS motherboard I have owned (ASUS A7Pro, CUPLE-VM, A7N8X) has required BIOS upgrades to work correctly, onboard peripherals diabled, exhibited lockups and have just generally been a pretty poor experience ‘out of the box’
Sure, after you put in 20 hours work or so hunting for information on how to work around these issues and upgrading BIOSes multiple times, the boards can be made to run relatively stably.
But ASUS, as far as I am concerned, don’t produce quality linux-ready boards, so if you buy their rubbish, expect to have to waste hours fixing their problems.
My latest board, the A78NX ‘Deluxe’ gives me massive disk corruption under a 2.4 kernel and locks up at least once a day under either 2.4 or 2.6 if APIC/ACPI is enabled.
booting the kernel with ‘noapic nolapic acpi=off’ seems to give me stability, but why should I have to run with these features diabled – ASUS are clowns. [/quote]
Well, first of all, A78NX board are based on the nVidia chipset, which bloody sucks under Linux. I think 2.6 has support for it, but for 2.4, you have to use nVidia’s proprietary drivers.
As for the other two boards, well, all I have to say is that my A7V8X-X has worked just fine out of the box : no disk corruption, no lockups, and only a weird memory problem (wouldn’t take a DDR400 memory stick for some obscure reason, so I just exchanged it for a DDR333 stick and it works fine – considering the XP2500+ I’ve got in there is a FSB333 CPU, I don’t mind =))
Just to let everyone know that Turbolinux has had a AMD64 distro released since April. It’s one of the least expensive supported OSs available. http://www.turbolinux.com/products/tl8a/
I’ve got a quad processor HP rx8620 itanium2 box at work with (a measly) 16GB of RAM in it. It’s my new testbed system. Runs 64 bit Linux (debian distro, with some stuff re-compiled in 64 bit). The db is postgres (32 bit), so the big win is just flat memory addressing and those sexy big caches.
The windows weenie has a quad processor Xeon with 16GB RAM running Win2K DataCenter and MS SQL Enterprise that’s been kicking my little workstation install of postgres for a couple of months. That’s some hella expensive software and hardware. And he sure didn’t have any trouble reminding me of it.
Anyway, the db’s performance on linux64/postgres flat out blows the windows system out of the water. Seriously, I’m getting performance gains in excess of an order of magnitude. First we thought there was a mistake in the testing but… not that I’m rubbing it in or anything.
Of course this isn’t exactly the kind of gear you put on a desktop. It lives in a rack in the land-o-unpleasant air-conditioning. Shame really since it’s got some sexy blinkenlights.
I think that guy has given up with Gentoo too soon.
I have had same problems, but managed to overcome them.
Main headaches were:
‘I had to use stage3 tarball (instead of stage1), emerge the things I wanted and then reemerge everything again with “emerge -e world”
Only periphery that doesn’t work on my Dual Opteron is sound. Everything else works very well.
From the software perspective, many packages are still masked, but main things work (X, KDE, Gnome, some games, and yes-OpenGL works too). On non-working list are OpenOffice and AbiWord, for example.
I could probably make them work under 32-bit emulation, but I didn’t want to fudge with this, I will rather wait to see 64-bit version.
Opteron works very nicely under Win2000/Win XP, but with Gentoo in 64-bit mode it really flies.
I have bought two identical dual Opteron machines and I’m so pleased with performance that I’m buying me a third one.
One machine works on Win XP, one is dual boot Win2000/Gentoo and third will replace existing fileserver/firewall/pribteserver/etc.
I am using:
Opteron 240 (2 CPU)
Motherboard Tyan Thunder K8W S2885
2 Gb DDR PC 2700 ECC Reg. RAM (4 sticks x512Mb)
nVidia GF4 Ti4200
HDD 120 Gb WDC
All machines are on 1 Gbit Ethernet switch (i have found quite nice and still on the cheap side Linksys SD2008)
It’s fun to see hard disk as a bottleneck when copying files through the LAN ;o)
Also, PCI-X slots absolutely rock. Old PCI on my dual P3 board camo on the verge of saturation when trensferring files to/from disks. With a nice RAID card (I’m waiting for 3Ware 7506-12) this problem should be mnitigated for quite a few years to come…
Don’t necessarily blame the BIOS upgrade for the disk problems you experienced with Mandrake.
I have an old Compaq Deskpro 4000 which I upgraded the CPU on with an Evergreen 400MHz AMD 6 CPU upgrade kit and a Maxtor 30GB hard drive and the original Maxtor 2GB drive. I can install Red Hat 7.0 on it with absolutely no problems. When I tried to install Red Hat 7.3, Red hat 8.0 and Mandrake 9.0 on it, ALL THREE of them barfed on the Maxtor 30GB drive, claiming they could not read the partition table even after ALL THREE of them themselves created and initialized the partition table. Whether with fdisk or Partition Magic or their own tools, NONE of them could handle the drive.
This on the same machine and drive that Red Hat 7.0 blows right on with not one single complaint about the drive or anything else.
This tells me that sometime between RH 7.0 and 7.3 somebody fscked up the IDE kernel code so that certain drives simply cannot be handled. Either that or certain BIOS’s like the Compaq (which is an oddball BIOS anyway) cannot be handled. My understanding is that after the Linux loader is done, Linux does not use the BIOS for accessing the hard drive geometry, which seems to me to leave it as the fault of the kernel IDE code.
Somebody needs to look into this since who knows where and on what drives this problem will bite.
We have a K8W with 8 GB RAM, and 2 Opteron 242’s. We got it from Monarchcomputer.com. They were very responsive, and did a good job. It cost about $4,500 without the monitor (8x 1GB PC2700 doesn’t come cheap). This system has been completely stable.
I put Redhat 3.0 AS beta (Taroon) on it with only one hitch. It didn’t recognise the fancy dual-DVI Matrox video card. A plain text screen also is odd; It is stretched two lines too long and I couldn’t adjust the monitor enough to fix it. I downloaded the X source and compiled it, and X runs now, but with just a plain window manager, so I just dropped that. I have mostly been accessing the machine remotely with ssh and VNC, and that works just great for me.
We were able to obtain the source code for the gene fragment assembly software CAP3 (many thanks to the author Xiaoqiu Huang) and easily recompile it. The largest job so far took about 6.8 GB RAM, and took over 30 hours to run. This same job was impossible on our dual Athlon. We can finally do some really big stuff and are very happy so far.
We are still waiting for a good non-beta 64-bit os (something like WhiteBox), but can get by for now. I don’t intend to get _trapped_ into anything RedHat ever again. (Yes, I know they have an academic version. No, I don’t want Fedora64. I’m not bitter, see?).
Regards,
Steve Wanamaker
Department of Botany and Plant Sciences
University of California, Riverside
We’re discussing the 64 bit version of SuSE here. This is only available on DVD. The “ordinary” 32 bit version is confusing you.
From http://www.suse.com/us/private/products/suse_linux/i386/64bit.html
Scope of the AMD64 edition
1 double DVD, 2 manuals (Administration Guide, User Guide), 90 days of installation support
I did not want to buy an ASUS ( too much problems with them for the last fiew years) so I bought an MSI and, …..
NO PROBLEM, I plugged my AMD64 3200+ in it and started the install, no problem like acpi or anything else. Every thing is working like a charm
I’ve been very interested in finding out generally how well SuSE 9.0 runs on AMD 64 bit systems. My most major concern has been their implementation of a 64 bit architecture – looks great on paper, but I’m much more interested in the capabilities in practice over paper. Planning for the future is great – but with the majority of apps run on the AMD64 chips being 32 bit, the AMD64 is largely unproven on the grand scale in my eyes. Ever the skeptic:)
That said, I have a great deal of faith in AMD for producing a solid product. I guess it’s just that the 64 bit side of the new chips are something of a hocus-pocus black box sales pitch at this very moment… and I’m very, very curious to find out what advancements will come from 64 bit computing over 32 bit with regard to AMD’s 64 bit architexture. After all – according to paper anyway – they did much more than just extend the 32 bit instruction set. Can anyone comment on applications compiled in 64 bit for AMD on a somewhat technical level? Especially under SuSE 9.0? I’m curious about any latency issues – positive or negative, how well GCC takes advantage of the new registers, any issues with ram < 4gb, any heat/power issues in 64 bit vs. 32 bit mode, overall stability of the system, etc… In a nutshell – anything wrong with the chip itself, or unexpected or extreme boosts/slumps in performance.
Ahem:) Atm my focus is the chip. Motherboard/Driver issues I expect will get ironed out quickly enough – but anything inherently wrong with the processor itself, or even just not up to par with expectations (and thus no real incentive to migrate) is of great concern to me. That’s why so much interest in SuSE’s performance -as afaik it is the best running OS on AMD’s 64 bit architecture.
Anyway, sorry for rambling. Thanks in advance for any information you can provide, and thanks again for the article – which provided me with some very helpful information in itself.
I’ve heard a lot of people gripe about SuSE and their decision to not offer ISOs. I’m sure if you volunteered to pay SuSE’s bandwidth bills they’d jump to offer ISOs on their sight.
Regardless, I’ve had great success burning SuSE to DVD after downloading it from their FTP site and installing from there.
Why do people make it sound like it’s the end of the world that SuSE’s ftp version can’t be burned to CD when DVD burners are selling for less than $150 these days??
I’m guessing that your lockups and Mandrake problems stem from the same thing. Either your disk is actually faulty, or Linux’s driver for it is not very good at this point. If it weren’t for the fact that your hardware is rather new, I’d be very worried about your hardware. But in your case it may indeed just be driver issues, especially since a lot of them might not be well tested in 64 bit mode.
I just built a box with dual opty’s and I have to say I’m very impressed with the linux support that is out there. Here is the basic setup:
1 x Tyan Thunder K8W
2 x AMD Opteron Model 246 2.0GHz
2 x 512MB Mushkin PC4000 <– [edit] Replaced with dual 512MB ECC PC3200 Mushkin sticks
2 x Thermaltake A1744 Venus 12 HSF w/ 80mm fan
1 x Antec 550W EPS12V PSU
1 x PNY Quadro4 NVS200 64MB SDR AGP w/ Dual DVI out
1 x Hercules 7.1 Fortissimo II
1 x Adaptec SCSI Ultra 160
1 x Quantum Atlas 10K 36GB U160 HD
2 x Quantum Atlas 10K 9GB U160 HD (Soft RAID 0)
1 x Western Digital 120GB UATA HD
1 x Sony DRU500A DVD+-RW
1 x Pioneer DVD303 slot load
2 x Samsung 191T
First thing I did with my MSDN subscription was download the WXP 64-bit evaluation and take it for a test drive. It loaded right up with 64-bit drivers for everything except my Quatro and Fortissimo. NVidia has a wxp 64-bit driver for the Quatro which runs very nicely in dual-headed mode, but unfortunately there are no 7.1 drivers for my sound card so I was stuck with the onboard AMD sound. No *huge* deal but it would be nice to use my 5.1 speakers.
I quickly loaded up the Visual Studio .NET 2003 installation (the only reason to run windows to begin with) and I found that the setup MSI wrapper does not allow installations on “64-bit operating systems”. I know this is a load of bull because it will work fine with the dual Opty’s, but I they will probably fix this when Wxp64 becomes retail and a future patch to VS.NET 2k3 installer gets applied. With sole exception of VS.NET2k3 everything worked GREAT.
So after that fiasco I pulled Wxp64 off there and loaded up Wxp32 pro. NOTHING was recognized. I to used my wife’s computer to go get the 32-bit drivers for everything and entered the usual windows install/reboot loop for about the next 2 hours.
Once all the drivers were in place, I have a fully operational 32-bit workstation with VS.NET2K3 and all the trimmings of 5.1 sound, but that’s NOT why I bought dual Opterons. Even though the dual procs where faster than my old workstation, I needed to feel 64-bit day to day computing.
Enter Linux. I’ve been using Gentoo for quite some time now, and of course that’s the first distro I went to when I needed 64-bit linux. I found that the x86_64 stage’s were experimental and I wasn’t very excited about a crippled workstation. I tried to install gentoo64 on a single ext3 partition (from reading the docs they warned of Rieser and XFS problems) so I wanted to play it safe. Fast-forward about 3 days later and I had a completely non-functional system. Something happenend and even Windows was crapping all over the place.
So my experience with Gentoo64 wasn’t so sweet, but I didn’t give up. After my entire system was hosed I decided to rethink my partition setup. Ultimately I decided to dedicate the 36gb drive and a portion of the 120gb drive to windows, and do soft RAID-0 on the two 9gb’s for linux. Getting the MD’s to work under Gentoo64 was too much of a headache for even ME, so I decided maybe it was time to rethink my distro.
I switched to Suse 9.0. I’ve wanted to try out Suse for a long time now but I loved portage so much I never switched. Finding that Suse 9 reports a “production ready” x86_64 distribution I was thrilled at not being labeled “experimental” any more. The installation was awesome, it found everything in my system (including 64-bit drivers for my sound card and all my usb stuff!). This kicked Wxp64 out of the water.
Unfortunately when it came to actually copying the files I ran into some problems. Pretty much every file off the DVD’s were “corrupted” when they were copied to disk and failed MD5 checksumming. I found that if I clicked retry enough times (read: about 20) things would install. I experienced this through both the Sony on the IDE channel and the Pioneer on the SCSI channel. I ended up having one of the kids build me some Lego hackery to hold down the Alt+E key. I set a spool of blank dvd’s on top of it and left it “retrying” overnight on all the packages.
The next morning I awoke to a fully functional Suse 9.0 system. I just got Xinerma support working under KDE last night with the nVidia GLX and kernel modules (by default Suse didn’t see my dual-headed setup bitching about not having two video cards). Everything is working great, and I’m very happy with my new system. Suse 9.0 was the easiest installation I’ve ever used.
Special thanks to all the part-time hackers that gave me 64-bit desktop computing! Happy holidays!
[quote]
Enter Linux. I’ve been using Gentoo for quite some time now, and of course that’s the first distro I went to when I needed 64-bit linux. ….
From reading the docs they warned of Rieser and XFS problems) so I wanted to play it safe.
[/quote]
Here is the paragraph you are referencing.
(from http://dev.gentoo.org/~brad_mssw/amd64-tech-notes.html )
We strongly recommend sticking with ext2/3 for now. We have had random reported problems with reiserfs on amd64, and have heard of major problems with running JFS on 64bit systems (which definately seems odd considering it was designed for 64bit systems initially). Though we have had successful reports of XFS, and no negative reports at all.
–end gentoo AMD64 filesystem snippet—-
They explicitly state that they have heard only positive
reports from XFS users.
Had the same good experience with SuSE Linux Enterprise Server 8 (SLES8) and the YaST2 tool makes administration chores a simple enough that a junior admin can do it with little supervision. I work for a large Brokerage house on Wall Street and we were looking to explore the AMD processor for some of our work. Called SuSE since as the article stated they have supported 64 bit computing on the Opteron the longest. We were informed by SuSE that they had a business partner, Systems Solutions in the Downtown NYC area, when we contacted them it turned out they are a couple of blocks form our location. Upon arriving we signed the obligatory non-disclosure agreement, we informed the group from Systems Solutions that we were interested in the AMD version of SLES 8 and were particularly looking to run it in a clustered environment. They asked if we had decided on which hardware platform we wanted to run, which we hadn’t yet. They invited us into their server room where they had 3 IBM e325 with 2 GHz Opterons running and then offered to provide the lead developer in our group a workstation and desk to do his coding and use their 2 of their severs as a proof of concept. Naturally we were surprised by this offer as was the manager from the Line of Business and decided to accepted their offer.
Systems Solutions’ offer saved considerable time in the initial development and testing and while our developer was working at their site their staff came to our location to create the lab to house the future cluster. As it turns out they are an integrator with a vast array of resources at their disposal they even have an electrical firm that they are partnered with that specializes in Data-Center wiring and power and they are also dealer for enterprise level UPS and cooling. After a successful testing of the application we purchased and had Systems Solutions install a 24 node 48 CPU cluster of IBM e325 servers and the configuration has broken every benchmark we have thrown at it. We had the added pleasure of informing the senior management of a rival group within our company that our AMD cluster easily beat their HP Superdome which uses 64 875MHz PA-8700+ processors with 12 less processors and at a fraction of the cost.
While the group was celebrating at the project closing party the EVP from the Line of Business jokingly said to the Project Manager form Systems Solutions, if they could fly him and 5 of his direct reports to a meeting in Florida then they would truly be a full service integrator. Everyone from our company was laughing then the Systems Solutions Project Manager took out his wallet as presented the EVP with his FAA pilots’ license and informed him that he is also the Director for the Flight Department and could have one of the 6 corporate jets ready within an hour. Then the Systems people started laughing, just a scary group of people, and a few days latter the EVP used their Flight Department and said it was the best service he ever experienced.
It was just an all around great experience with a professional vendor and we are looking forward to working with them again.