Linux on the Opteron, Part III

Continuing with the Linux on the AMD64 series of articles, this installment is to be a summary of two new distributions, and the changes that have been made to Gentoo since the last installment. Here I review the installation of TurboLinux 8 (both with and without the update CD), the installation of Fedora Core for x86_64, and more news on Gentoo for the AMD64.

For those that missed the previous articles, my AMD Opteron system consists of the following: (The system is the same as in the previous articles with the exception of the power supply and NIC)

AMD Opteron 240 CPU, retail version with AMD heat sink and fan
ASUS SK8N motherboard, with latest bios update
1GB Micron DDR ECC Registered ram (333MHz, ASUS recommended)
ATI FireGL AGP video card with 128MB of ram
Maxtor 20GB UDMA drive, 7200rpm
Maxtor 80GB UDMA drive, 7200rpm
LG CDRW
D-Link DFE-538TX NIC
Enermax 420W PS
Lian-LI PC-60 case

A small aside:

I note that on some discussion boards a lot of AMD64 users are having better luck (performance?) with the VIA KT800 chipset based motherboard. I have not been up to speed much with the entire 64FX line from AMD, but it seems that some motherboard manufacturers are migrating to the KT800 for the FX line. My digression is really to point out that many have observed better support for the KT800, regardless of using an Opteron or a 64FX. Since I do not have a 64FX CPU, nor a KT800 motherboard I can not comment directly. YMMV.

TurboLinux

Installing TurboLinux was not as straightforward as Fedora or Suse (although Suse was an ftp install, it was without real problems). The first attempt caused a kernel OOps, and glancing over the documentation, I noted they state that if this happens, try to boot with the kernel option no1394 (to disable firewire). This worked, but then the installer would hang detecting the hard disks. This problem was overcome by using the kernel option noapic. OK, we are ready to go, almost. The installer comes up and everything seems to work, until I get to the drive partitioning. If I try to do anything to the disk other than add a new partition, it fails, tells me it can not write or read the drive and asks me to reboot.

This is _very_ similar to the Mandrake AMD64 beta install problem I had. Simple solution was to fdisk the drive with Gentoo, do not create partitions and then boot the TurboLinux installer. (note, this seemed to be fixed (mostly) in the update) Another two reboots and I am on my way. The installation was without problem (other than the aforementioned drive problems) from then on, only minor annoyances. First, for those interested in security, I was paying attention to the installer options when I noted this:
at the bottom of the “Installing boot loader / options” screen, a little check box which read: “Use root password for Grub password” was checked. From a security point of view, this is bad. Apart from that, Xwindows would not do 1280×960, which is my preferred setting on my monitor. Oddly, when I modified the XF86Config file by hand, it _still_ would not do 1280×960. This is very strange considering that four other distributions do it perfectly on the same hardware, running the same XFree server!

Since we are on the topic of installation, I must mention that I installed TurboLinux twice. The first install was straight from the TurboLinux for AMD64 CD’s. The second installation was done using the update CD. The kernel was updated (still in 2.4 however) as well as the XFree server and a few other minor packages. The install itself was identical with the exception of swapping the CD’s for the update CD and then back to the install CDs. I noticed no real performance increases or issues between them with the exception of the firewire being recognized. TurboLinux claims that the update CD fixes some Nvidia Nforce3 chipset issues. I suppose it did, but from an end user perspective
I did not notice anything new being supported (except firewire).

TurboLinux felt much like the Suse for AMD64 in the respect that it was clean and everything just worked. 32-bit binaries are supported and 64 compilation is there right out of the box. I did not run across any packages that did not work. TurboLinux is using KDE as the window manager, and it is fairly unmodified with the exception of icons and the background. Not a whole lot to say about the window system here. ext3 is the default filesystem (I note this as it has occurred to me that Suse defaults Riser, while Fedora, TurboLinux and Mandrake default to ext3), 2.4.20-1 is the default kernel on the default install and 2.4.21-2 is the default kernel on the upgraded system. One more thing to note about TurboLinux that some may find nice: Boot service selection. When TurboLinux boots, after grub loads the kernel and it starts to load, a message appears at the top of the screen informing the user that if they hit ‘S’ you can load services individually. I do not think this is anything new in operating systems as I think I remember people being able to do this in windows ninety-something or other and DOS, but this would be the first time I have seen this in a Linux distribution. I think it is not needed, as if I (or you) do not want a service to run we would just disable it in the initscripts or rc directories, but I suppose new users may find it useful in some way. On one last eye candy note, TurboLinux has no flashy splash screens at startup and does text only until the window manager loads (if you have it defaulted to initlevel 5). Suse, Mandrake and Fedora go into a graphics mode just after GRUB and stay there until X is fully loaded, showing the kernel output in a window (if you click “show details” or somethingof that nature in Mandrake and Fedora). I find this behavior disturbing if you want a server machine, but it can be turned off (choices, this is why we use Linux, no?)

Fedora Core for AMD64

Fedora Core for the AMD64 installation is exactly the same as the x86 installation (at least from my last install of Fedora Core 1 on my AMD XP machine). No problems were encountered during the install, and unlike TurboLinux and Suse (recall Suse needed the noapic option also) I did not need to pass any special kernel parameters to get it to boot into the install. Again, if you have installed Fedora Core for x86 machines, this installation is the same. Gnome and KDE were both available for installation, although I left it defaulted. I note that TurboLinux does KDE only, and Suse defaults to KDE also. After install was complete the machine rebooted and came up with out problems. As mentioned before, Fedora does a graphical loading (not only grub, but from them on out), but details are available if you hit the “details” button. Unlike TurboLinux, but just like all the other distributions, I set the display for 1280×960 and that resolution worked fine after logging out and then logging back in.

Like most other distributions, Fedora is so far doing the dual /lib and /lib64 setup so that software compiled on 32 bit machines is supported (and yes, Kernel support for 32 bit binaries is on out of the box). I did find a few small binaries from my 32 bit machine that would not run, but they needed a not so obscure library that was not installed. This is not a huge problem to overcome and in fact searching for a pre-built Fedora rpm turned up one that worked fine (in fact, one could copy the rpm from the 32 bit Fedora). All of the installed programs that I tried worked, and I would guess that some are still in 32 bit mode (as seen on Gentoo, etc). However, like Suse and TurboLinux, I did not run across any software installed out of the box that did not work. From an average desktop users point of view, there is no visible difference between Fedora Core for x86 and the AMD64 version, with the exception of the price of the hardware.

Gentoo for AMD64

Gentoo has made many strides since my last article, X has a few window managers to choose from, and in fact I have loaded KDE without a problem, after loading and using Enlightenment until KDE was done. There were several applications which I tried to install during the last article which were not ready for the AMD64 (masked in Gentoo portage speak). Of course one can try their luck and emerge them anyway, but most of the time I wait until the Gentoo team gives it the O.K. I will not rant on and on about Gentoo, but wanted to mention that they are moving ahead with many apps being ported, window managers and kernels. I would like to point out at this time that they are the only one I have tested that is (as of this writing) using the 2.6 series of kernels.

Which brings us to: The Kernel.

I do not wish to get into an argument about the kernel version used with these distributions, as there has been much discussion on the net about which one is better now, and which one should be used. The 2.4.x kernel series is of course the second to latest stable release, and is considered by many to be _the_ stable kernel, with 2.6.x only having been out a short while. The 2.6.x kernel, being more recent, does have many advantages over the 2.4.x series in terms of Opteron support. Specifically, it supports the Opteron directly, and it seems to overcome many of the problems associated with peripherals on Opteron motherboards (most probably due to increased Nvidia Nforce support which must be hacked into the 2.4 kernel to work properly). It is my opinion that if you use an Nforce based motherboard (or any for that matter) and you run the Opteron, the 2.6 series is the kernel of choice. It also seems to run faster in many situations on the Opteron (again, most likely due to the increases in performance that have been incorporated into the kernel).

To summarize:

TurboLinux claims to have had Opteron support the longest, and it does seem polished, but it does have a few oddities to it (disk install problems, etc) but again, most of these have been fixed with the update CD. Gentoo is moving right along with porting, they now have window managers (for those interested) and they are using the 2.6 kernel on the live CDs. Fedora Core is still beta, but it has never given me any problems (it is the desktop OS of choice on my Opteron) and everything works. I did do some small, highly debatable benchmarking on these different distributions, but I stronly recommend that if you want to use the Opteron for any CPU intensive task, benchmarking of the application to be used should be performed.

On a lighter note, Opteron owners now have plenty of choices and the price of the hardware is coming down to boot. The VLSI software[1] I use compiled on them fine and in fact the 32bit binaries copied from my AMD Athlon system worked directly (with the exception of Fedora which I had to add a few 32 bit libs for). For the price of a home built PC one can get a 64 bit CPU and have a 64 OS running on it.

Small performance comparisons:

OK, I think I can handle the flaming, so here goes… I decided to do a few very minor performance benchmarks on the machines…specifically Fedora and Gentoo (Turbo should be “like” Fedora). I tested Kernel compile time, and the apache benchmark (ab test, static 2k page)

Apache —
Gentoo: 79.62 req/s
Fedora: 54.59 req/s
RedHat9 on AMD XP 1.93GHz: 47.3 req/s

Apache, after playing with the server settings a little… see notes at end of this section!
Gentoo: 1141.59 req/s
Fedora: 54.65 req/s
RedHat9 on AMD XP 1.93GHz: 54.85 req/s

Kernel — make bzImage: See notes and comments at end of this section!

First Build times:
Gentoo: 4m27.485s
Fedora: 4m27.501s
RedHat9 on AMD XP 1.93GHz: 5m33.600s

Second Build times:
Gentoo: 0m16.880s
Fedora: 0m17.155s
RedHat9 on AMD XP 1.93GHz: 0m13.323s

Notes on apache: Gentoo, Fedora and the RedHat9 box are running apache 2.0.
All of the tests were done with: ab -n 1000 -c 1000 http://IPaddressofserver/index.html
All ab testing was done from a separate machine, across a 100Mb/s switched local LAN. Other machines on the LAN were taken offline in order to keep the results fair. Tweaking apache.conf for the second test set was comprised of setting the following values in apache.conf:

servers = 20
min spares = 10
max spares = 20
max clients = 250

I know there are better optimizations, but I was just playing around with it for a moment. Also, I am not sure why the Opteron numbers look really bad for Fedora, and why the RedHat9 numbers failed to increase at all either. Again, I do not know a huge amount about the Apache optimizations.

Notes on kernel compile: Gentoo was using 2.6.0, Fedora was using 2.4.22 , RedHat9 was using 2.4.20. The configuration was mostly stock, with the exception of the options required in Gentoo (devfs and the like). At the moment I have no hard explanation for the outcome of the second test run under RedHat9, other than the 500MHz clock advantage of the AMD XP (not Barton) machine. The freshly installed kernel build times do not support this theory, as can be seen from the above numbers. The kernel compile tests were actually performed several times to make sure this was the case.

AMD XP 32 bit machine hardware:

Abit KR7A motherboard
AMD Athlon XP 2400+ 256k cache
1GB 233MHz DDR ECC registered ram
80GB Maxtor UDMA 7200rpm drive
Nvidia 5200Ultra AGP video, 128Mb ram
3Com boomerang 10/100 NIC
Enermax 365watt power supply

Exit notes:

I find some reviews I see of the Opteron and AMD 64/64FX hardware a little disturbing in the sense that they complain that no 64 bit OS exists or that it does not have software support. I think that these reviewers are thinking in terms of MicroSoft (TM) (C) (R) desktop use (e.g. no Windows (TM) (C) (R) release, no Windows (TM) (C) (R) 64 bit apps, etc)…they do not realize that on UNIX, most software for servers was written for 64 bit hardware (IBM RS/6000, SUN UltraSparc, HP PA-RISC, etc) and it will not be long (IMHO) until they are ported over. As for the desktop side of things, I have been using it as a desktop for some time (although I admit I am probably not the average desktop user) so I find the argument (which has been on the net and in the news for some time) puzzling. The Opteron as a desktop is a good performer, and as a scientific or workstation machine, a great performer, although if one was not going to do any 64 bit computing, a bit pricey and overkill. These distributions for it are getting better by the day, and prices on the hardware have been falling (as they always do). Of course, one can not deny that at some point in the not so distant future, 64 bit computing will be omnipresent, and all sfotware will run on it (optimized for 64 bit, we hope).

[1] Magic, developed and written by John Ousterhout, in the 1980s at Berkeley. Yes the same John Ousterhout of Tcl fame. A great tool for VLSI layout which has stood the test of (computer) time.

Previous aricles in the series:
Linux on the Opteron, Part II
Linux on the Opteron, are we ready?

About the Author:
Robert Minvielle is a Ph.D. student at the Center for Advanced Computer Studies at UL at Lafayette. He has a Masters in Physics and a B.S. in Computer Engineering, and enjoys building hardware, embedded systems, and programming. He can be reached at [email protected].