Submitted by PopularMechanics has performed three types of benchmarks on Apple and ‘ordinary’ PC machines, with the former running Mac OS X Leopard, and the latter Windows Vista. The first type of benchmark consisted of users giving ratings to things like design, ergonomics, web browsing experience, and so on. The second benchmark focused on real-world performance (launching applications, boot/shutdown times, and so on), while the third and final benchmark consisted of Geekbench and Cinebench runs. PM concludes: “The results gave us a clear winner in the performance categories, but the big surprise was how little difference we found in user preferences. Turns out, both platforms are capable and easy to use, but only one was the victor.” Even before I got to the results, I noticed a whole set of problems with the benchmarks performed in this article, that would seriously skew the results.The biggest problem is that the machines are not at all similar in specifications. They used desktops and laptops, but in both cases, the Apple machines had far less memory to go around, but did have slightly faster processors with better bus speeds. The Gateway all-in-one machine had 2GB of RAM more than the iMac, but a 400Mhz slower processor. In the laptop category, the ASUS notebook had 1GB of RAM more than the MacBook (on page 1, that is – the results page say the MacBook had 1GB more), but a 2.2Ghz processor compared to the 2.4Ghz in the MacBook. In addition, the MacBook had an integrated Intel graphics chipset, where the ASUS notebook had an Ati one. This means that the ability to draw conclusions on the performance-related benchmarks is limited, at best.
The hardware differences will also skew the battery results in the laptop comparison. The ASUS has a more powerful graphics chipset, which also drains more battery power. The MacBook had a Penryn processor, which, according to AnandTech, has a significant increase in battery life compared to its predecessor in the ASUS. In addition, the ASUS has a much larger screen (15.4″ compared to 13.3″) which also affects battery life.
Another big issue is that the Vista computers did not seem to have Service Pack 1 installed, which is another thing that could have seriously skewed the results – this time in favour of the Macintosh. Service Pack 1 is said to fix a set of performance-related problems, and it has been released quite a while back already – it should have been installed. When you do these comparisons, you always use the latest versions, fully patched. Common practice.
The last major issue I have with the results are summed up in the article’s first page:
These things are largely matters of preference and style, but you can still make a reasonable attempt to quantify them, and we did. We tested two all-in-one desktops and two laptops – one Mac and one PC per category – and assembled a panel of testers with a range of experience and preference that ran the gamut from expert users to my wife’s stepfather, who, by his own account, had never actually turned on a computer. Our testers were asked to set up the computers right out of the box and explore the machines through everyday tasks such as Web surfing, document creation, uploading photos, downloading Adobe Acrobat files and playing music and movies through Media Center and Front Row (the entertainment software suites integrated into Vista and Leopard, respectively). Our testers were instructed to divorce themselves as much as possible from their previous technological preferences and rate their experiences with each computer’s software and hardware.
Seeing as no further details are given regarding the test subjects – no background information, no direct quotes, no anecdotes, no experience levels – the ‘usability test’, as PM calls it, is extremely anecdotal, and more or less useless – a common problem in usability testing. Usability testing is not a matter of putting grandma behind a computer and see how she gets around. It requires a scientific setup, reduction or control of external influences, larger test samples, detailed information regarding the subjects, before you can draw any conclusions from what you have observed.
As for the results of the benchmarks themselves – read the article. While many of the results do reflect my personal feelings and opinions, I do not take these specific results seriously. Even if results are in line with my own feelings and opinions, if they are based on faulty testing, they do nothing to strengthen those feelings and opinions.
The only benchmark I would need is how fast I can get my work done on the computer in front of me. If I’m spending all my time fussing with the OS — whether it’s fighting off spyware and viruses with Windows Vista or tweaking hundreds of config files and driver patches with Linux — then I have less time to write, to build web apps, to make music, or even just to read news sites and post comments. I’m a tinkerer at heart and I have had years of fun playing around in the inner workings of both Windows and Linux but when it comes down to it, I need an OS that gets out of my way. OS X does that for me and it’s the main reason I’ve not switched from my old, slow eMac to a faster but ultimately less useful Vista/Linux machine. As for gaming, well the only games I play are World of Warcraft and various console emulators, all of which run very well on this old thing.
As a bonus, when I do feel like mucking around in my computer’s inner workings, OS X is BSD and therefore just as fun as Linux.
I just booted up the recent rc release of Ubuntu 8.04 (aka Hardy Heron) for a test run. I’m running it right now as I post this, in fact … and it is running on this machine for the very first time.
My entire machine worked correctly on boot-up without having to tweak a single thing … right off the liveCD without editing a single configuration file at all.
Furthermore, a full suite of desktop applications … from simple text editor through calculator, system tray applets all the way through to Office suite … were all already available to me without having to install anything. Far, far more functional than an OSX or Vista install. There are literally many thousands more additional applications available for free download just a few simple clicks to install.
You can’t get any more “gets out of your way” than that. It has both Vista and OSX beat cold on that front.
This is especially remarkable since this is all achieved on a machine that was “designed for XP” and not Ubuntu at all.
Edited 2008-04-21 11:08 UTC
Congratulations on a flawless Ubuntu experience. I’ve had much worse luck than that, but I fault my limited choice of PC hardware and not the Ubuntu team for that. When it comes to Linux I absolutely love Ubuntu, and wouldn’t run anything else given the right hardware.
As for how well OS X works out of the box, well for me it already has most everything I need. I write in TextEdit, I chat in AOL (iChat), iLife takes care of my musical and video editing hobbies, and Mail, Address Book and iCal are the best of each that I’ve ever used. I will say that I don’t care for iTunes especially compared to Amarok, but it does the job and I can’t complain too much. As for what I do have to install beyond the core OS, there’s Galerie for web albums, Taco HTML Edit for web authoring, and Filezilla for FTP. All of those apps work simply, beautifully and don’t cause me headaches. I can’t ask for more than that.
I’m pleased that you enjoy Mac OSX … I don’t have any quibble with it really as OSX “tries much harder” to be cross-platform inter-operable and open standards complaint than Vista or XP does. Enjoy.
However, I do have a quibble with claims that OSX requires less configuration or “gets out of your way” more than Linux. That may have been true some time ago, but it is not so any longer, really.
I back this up with the observation that this machine has a widescreen 1440×900 LCD monitor driven by a relatively recent ATI video card … and many Linux distributions as recently as last year required some significant manual coaxing to get this combination going properly.
Works out of the box, now … correct resolution & everything. Not even XP (which it was originally designed for) or Vista (which was released after this hardware was already on the market) can claim that. Clear fonts without any fuss at all. The video card even now has an open-source native Linux driver.
Edited 2008-04-21 11:32 UTC
Bla blah Linux is great blah blah. Please stop hijacking every story and turn into a Linux advocacy group. It’s starting to really, really annoy me. I use Linux every day, but even I can see this is running out of hand.
This is a thread on a benchmark regarding Apple and ‘normal’ PC hardware, and OS X/Vista. I’m tempted to moderate this thread into oblivion. do not reply to this comment either – either say something on-topic or related to the original story, or don’t say anything at all.
Thank you.
Meh. Fair enough.
All I would ask in return is that people don’t turn every Mac or Vista article into a untruthful Linux bashing opportunity.
I do support the observation that Mac OSX is a fine product, and that any benchmark comparison between OSX and Vista should be performed on the identical hardware that can run both … a dual-boot arrangement using Parallels or something.
That is the only valid apples-with-apples comparison testing between these two. That is not what we have here.
Edited 2008-04-21 12:16 UTC
lol did you try a mac ?
I think everyone here knows how Linux comes with everything pre-installed, and thousands of dandy free apps available with a couple clicks.
The problem is many of the apps people are looking for aren’t available on Linux, only on OSX and Windows. Not Linux’s fault, but a problem for the end user nevertheless.
And when hardware doesn’t work in Linux, even with Ubuntu, have fun.
I just got in an HP XW6600 dual Xeon workstation, with an nVidia FX1700. Doesn’t work in 7.10, at all. Even with the restricted driver. A manual install of the driver doesn’t work either. So…..
It’s not always as easy as popping the CD in and answering a few questions.
EDIT and by the way, this test is ridiculous, as Thom noted. If you’re interested in benching OSes against each other, use the same identical hardware. Otherwise the test is meaningless.
Edited 2008-04-21 13:56 UTC
Likewise, try installing Vista on a machine not designed for Vista, or try installing OSX on anything other than a Mac … it just won’t go at all.
Precisely. Macs are therefore the best hardware to do this on, as that is the only hardware (AFAIK) that can boot all three OSes on the same hardware.
Even then, a Mac is designed to run both OSX and Vista … so once again it is a credit to Linux that it can run on this hardware also.
Anyway, only with such a triple-boot setup on the same machine can you then do a side-by-side apples-with-apples comparison of all three desktop OSes on the exact same hardware.
I’m fairly sure Vista would come third. OSX would probably win (since Mac hardware is designed to run OSX) … but I don’t believe that Linux would be all that far adrift in second place.
However … this is all just speculation on my part. I haven’t seen any report in wide-circulation media that even attempts such a comparison.
Edited 2008-04-21 14:11 UTC
If you want to be completely fair you can’t triple-boot. Hard disk speed isn’t the same across the entire disk, so the OS at the end of the disk as a disadvantage.
Correct. 3 iMacs each having a single instance of each of the 3 OSs would be more fair.
If you really want to be fair, then you’d test on a range of hardware that supports both rather than just one model.
(eg some operating systems perform better with faster CPUs where as others perform better with slower CPUs but more cores.)
He says he uses his machine to play WoW and to make music. Both of those things are difficult to do in linux, and when you do get them working, they don’t work as well as other platforms.
Also, by “Out of your way”, he is talking about the lack of configuration and learning curve you get on OSX. There still is some, but not as much as windows, and FAR less then linux.
Not saying linux sucks or anything, but it isn’t the best choice for every situation or every person.
No OS is the best choice for every situation or every person.
The only benchmark I would need is how fast I can get my work done on the computer in front of me. If I’m spending all my time fussing with the OS — whether it’s fighting off spyware and viruses with Windows Vista or tweaking hundreds of config files and driver patches with Linux — then I have less time to write, to build web apps, to make music, or even just to read news sites and post comments.
I hear this argument a lot but I have a different perspective. I prefer Linux because it is extremely configurable. I can set up my desktop exactly how I want it which saves me time in the end. If I had a Mac or if I used Windows I would have to endure their focus settings, and multiple desktop settings (if any), and their keybindings amongst other things. With Linux I can have my desktop exactly how I want it.
There’s absolutely nothing wrong with that. I think it all comes down to what works for you, the user. It just so happens that OS X was a short and easy learning curve for me, and I found it much easier to adapt myself to it than to adapt Linux and (horrors!) Windows to my needs. It’s not going to be that way for everyone and that’s just fine. I wonder sometimes what drives the fanboys out there to scream murder to any OS but their own choice, when they should be celebrating the fact that there is indeed choice in the market.
Just to clarify, I wasn’t calling you a fanboy at all, that was just a general observation about the inevitable flamewars associated with articles like this. I’m sure there is such a hatefest forthcoming.
There’s absolutely nothing wrong with that. I think it all comes down to what works for you, the user. It just so happens that OS X was a short and easy learning curve for me, and I found it much easier to adapt myself to it than to adapt Linux and (horrors!) Windows to my needs. It’s not going to be that way for everyone and that’s just fine. I wonder sometimes what drives the fanboys out there to scream murder to any OS but their own choice, when they should be celebrating the fact that there is indeed choice in the market.
I agree, that’s why I said I have a different perspective, not that you are wrong. It all comes down to preference. I have serious qualms with most of Microsoft’s products but for some they work well enough. They just never worked well enough for me. OSX is a very nice system but I’ve never owned a Mac because they just aren’t configurable (hardware and software) enough to suit me. I just need to be able to muck around with patchsets, scripts, and desktop layouts too much to settle on a monolithic system.
I really did want for the Mac to win out on this, apple make some nice hardware in my expirience. BUT how hard would it have been to compare similar systems? 2GB difference in RAM on the desktops!!!!????
That fact alone discredits the comparison, I belive this despite the Mac winning
The startup and shutdown times are very telling though. The Macs were up in <45s, and down in 4s, and Vista- well, you could have composed a short orchestra in the time that takes to move itself.
How is a <30 second difference telling? I guess by “compose a short orchestra” you mean “play a note”
Have you ever sat in front of a basic Vista laptop and waited for over three infuriating, never-ending minutes for it to shut down? Vista’s startup/shutdown times are worse than XP and unbearable on anything but the highest spec machine. You’re forgetting that the bulk of users out there don’t even have 2GB of RAM.
nope, never waited that long, and I don’t have a high end machine (2ghz core 2 duo, 2 gigs of ram), I have a mid range. the shutdown time is irritating, but again, we are talking about seconds here, not minutes.
And if you don’t have 2 gigs of ram, you definately shouldn’t be running vista (or the latest gnome/kde, or leopard). Just because its possible doesn’t make it a good idea. Ram is cheap and life is short.
I’ve grudgingly agreed with most of your comments in this thread, but I feel the need to nitpick this one. I haven’t used vista, leopard or KDE 4 (although I am strongly of the impression that the latter two will run okay with 1 GB) but I know for a fact that gnome works fine with rather less than 2 gigs. 512 MB should be fine; maybe even 256 if you use light apps.
You are right, although I would recommend a gig, not 512 megs for gnome/kde for what I would consider normal use (say OO.o, amarok, a few applets, pidgin, and firefox with a few tabs open). Leopard I would still say two gigs. I find multitasking in OSX is fantastic, but paging is brutal, and every time it starts doing that you are really going to feel it.
Latest GNOME and KDE run just fine in 512Mb.
Sorry but 2 or 3 years ago just before I switched to the mac os, the Apple hardware was very slow compared to the PC hardware.
But at this time I remember reading people telling that i does not make sens to compare speed : they had a better life with slow macs than with faster pcs.
Well, now that speeds are about equals who will find importance if one is just a small bit better ?
It makes you wonder.
I have a WinXP machine sitting next to my Mac OS X machine. The WinXP desktop has a 1.8 GHz Sempron 3100+ and an nVidia 7600GS graphics card while the 3+ year old PowerBook has a 1.33 GHz G4 and ATI Radeon Mobility 9700.
For sheer speed, the Windows machine is better, but the Mac OS X machine is ready more quickly once I see the desktop as the Windows machine takes time to load the anti-virus software, etc. at that point. It’s somewhat frustrating.
WinXP tries to help me sometimes, which I have to confirm that I don’t want the help, thank you very much. Those time wasters plus all the time for the virus scanning every boot. I still seem to get more work out of the PowerBook, but as it’s old, I can’t find a game for it anywhere.
These new machines are just a matter of which company you trust, what price you’re willing to pay, and style, just as it’s been in the overall Windows market for a long time.
It’s very much a common knowledge among geeks by now that, with all else being equal, Vista is sluggish when its comes to start-up and shutdown. But why are these people comparing PC system with 2.0 GHz Intel Core 2 Duo to Apple system with 2.4 GHz Intel Core 2 Duo? This alone is enough to dismiss all the numbers in comparison.
Edited 2008-04-21 13:51 UTC
I’ve got a 2.16GHz core 2 duo iMac with Leopard on it… I also have vista in bootcamp and Leopard is much more responsive and boots faster…
However, why is bootup a problem when most of us put our computers to sleep…
Frankly, I see nothing wrong with people comparing different OS on a thread specific to a Mac, Windows, Linux or BSD’s.
Actually, it should be stimulated. Debate is the best thing to happen and I’m against any arbitrary form used to suppress this from happening. It allow us to better understand the differences and have a better base to compare the features of each other.
If you want to control the OS fanboys troll freaks, fine, that is what moderation is for, but every other case should be considered part of the former form of constructive debate.
Thank you, acobar. This is one point of contention between me and Thom on this site. It is, after all, OS News and not Windows News or Linux Times or Mac Reporter. Granted, the subject matter of this article had nothing to do with Linux, but you simply cannot discuss two without the other of the top three desktop choices popping up for further comparison and discussion.
Thom, this is not a personal attack on you, far from it. I have a lot of respect for you. However, I’ve called you out for being heavy-handed with the censorship before and I won’t hesitate to do so again. Call me a rebel but free speech is in my blood and while this may be your website, you opened the doors to public discourse by allowing us to comment freely. I urge you to take the high road and allow us to self-censor using the more than adequate moderation system you put in place. Otherwise, what is the point of even having an open forum?
OS X leopard on PC and it’s very fast, boot time to desktop is about 18 seconds, all nice default apps and fast response over all.
I have tried leopard on a iMac lower spec than my PC(it was a 2.4/800fsb/1Gb ram) and it responded faster than my (2.66/1333fsb/2Gb ram) so I do think they Mac’s have a advantage in some way. Maybe it’s the loss of latency on PC’s with the hacked kexts, the vanilla kernel does respond quicker.
I think the “problem” lies within EFI’s emulation layer of hackintosh. You cant expect it to run exactly the same as a genuine mac does IMHO.
Either you compare operating systems/software and take the exact same hardware and run different software on it trying to do similar things, or you compare hardware and you take the exact same OS/programs and different hardware.
Easy, isn’t it?
If we are to be on topic for “OSNews” for a thread entitled ‘The Ultimate Lab Test’? … then what you would perhaps require to do such a test is to boot all three desktop OSes on the same hardware, and then run the same applications and/or benchmark programs on all three.
Candidates: OpenOffice, Firefox, Emacs/XEmacs, MySQL? gcc? GIMP? Mono? One would also probably try to run some performance benchmarking algorithms written in Python, Perl, PHP or Java or somesuch cross-platform language.
Edited 2008-04-21 14:47 UTC
I think that the important “take home” message of those tests is that Leopard is undoubtedly faster than Vista, and not with a small margin. Startup, shutdown, Open GL, etc, are significantly faster on Leopard.
Leopard is fast, and it does it with less memory requirement than Vista.
On my machine, a C2 Duo I have not noticed particularly bad startup and shutdown times with Vista.
However I have noticed it has ridiculously fast suspend to and resume from sleep times. Given its a desktop machine, it never gets shut down so this is what counts for me. In my experience suspend or resume are near instant.
Presumably Macs can also do this, but unfortunately for my hardware suspend/resume is broken in Ubuntu… The time saved by this functionality in Vista is enough for me to switch to the ‘dark’ side until linux catches up…
Edited 2008-04-21 15:02 UTC
One major flaw is that the all-in-one of Gateway is NOT a normal PC. Far from it. And far from a good one either.
They tested different computers with different hardware configurations.
For this test to have had credibility the configurations would have had to be virtually identical, with them also running the same operating systems.
The cost of the Gateway One is inflated because it isn’t a typical PC form factor. In general, normal ATX tower PCs can be bought much cheaper than Mac PCs. The Acer laptop depicted in the test had a larger screen, hard drive, and a much better graphics adapter for the same price. But the laptop did have a 200 MHz slower CPU and 1 GB less of RAM, which is the hardware that affects performance most.
Edited 2008-04-21 16:29 UTC
I don’t think I needed to read a review to find that Vista is slow.
Vista’s slow well thats a shocker
Come on people, get over it. I seriously doubt that there is one person in this forum that is a member of the target group of the article. We are not talking about Ars or Anand or wired. We are talking about Popular Mechanics.
The article is a good article for the intended readers and the perceived, yes I said perceived, bias is probably more just that than an actuality. I hate to be the one to break this to you, but you don’t think like the typical computer user does.
I found the article to be interesting and I also found the comments here to be very illuminating. We can all cite cases to show that one OS is superior to another. Well, the fact is that the best OS to use is the one that gets the job done quickly and with the least amount of hassle. Anyway, for most people the OS is a tool and not a passion. Use the best tool for the job.
Did anyone notice that when you add up the stars for the laptop section, the Asus won? Even though the article comes to a different conclusion? The macbook scored an average of 4.083 stars v.s. the Asus which scored 4.16 stars. Unless there is a precision error (We don’t know what earns the jump to .5 stars) If it is a precision error, then they must be very close scores, probably within the margin of error. At best (for him) they tie. That’s before we call the Mhz difference. (Which also has an effect on user experience.) Or the Penryn difference which caused the mac to win ‘time to install office,’ and the ‘stress launching test,’ as seen from their non-penryn clip from the magazine.
http://www.popularmechanics.com/content/imageview.cfm?img=http%…
(Also, why not remove RAM or over/underclock? Another interesting bit. The move to Penryn & 3GB helped OSX by 10% in the memory benchmark, but only helped Vista (in boot camp) by 3%. The ‘Scream’ score for Vista actually went down! Also, look at the laptop bootcamp score for graphics. Either bootcamp or their benchmark programs/methodology are $#!+.) They should leave computers and benchmarking to the pros.
..and this is not an excuse for Vista but we all know how Vista is notorious for trying to cache everything at the startup on its disk…right sort of what google desktop does whne you install it? It runs that process which indexes your hard drive? I remember when I had Vista unfortunately on a top of the line Dell XPS laptop I had ordered and it was reeeeeeeeally slow because of that. Once it had finished (which took a loooong time) the performance got significantly better but my point is did this review/benchmark take that time into consideration?
And no surprise there that Vista is so slow. Now if it was XP 64 bit that would be a different scenario altogether where both the systems would just be left in the dust….
Even before I got to the results, I noticed a whole set of problems with the benchmarks performed in this article, that would seriously skew the results.
Yeah, that set of problems consisting of performing a test in which one OS is running MAC and the other OS is running Vista.
I’m all for comparisons, but these articles are just stupid. “I just loaded the latest Ubuntu distro, Red Hat, Mac OS, and Windows 7 Alpha….now lets see which OS is better!!! Shockingly, the tests show that the PC is actually faster when it’s powered off!”
The test was not very scientific, but it was reasonable. To the average user, both machines look the same, have the same part “names” and cost about the same, that’s good enough.
To be specific though, the Macs tend to use higher bin processors that are not on the consumer retail shelf at the same price points. On the other hand, why should it matter, the machines cost the same dollars!! The PC has more raw speed but the important tasks: On, Off, and speed to GET to the programs give the Mac the edge.
I have a Vista Business PC at work and a Macbook at home, both have the same ram and Core2 T7000 series CPUs (2.0 & 2.16) The Vista PC is nice but it is frustrating due to the long start up times and bad access to core functions like wireless networking, network drives, & power on/off. The stuff that should be easy is harder than it should be and not thought out well compared to mac. As far as programs and such they run just fine on both but it’s the little things that make the Vista PC really hurt.
Coming soon: which is the best car?
http://myweb.liu.edu/sonic/viewtopic.php?p=267#267 ORDER XANAX ONLINE
http://myweb.liu.edu/sonic/viewtopic.php?p=268#268 buy valium from india
http://myweb.liu.edu/sonic/viewtopic.php?p=269#269 buy xanax valium online florida
http://myweb.liu.edu/sonic/viewtopic.php?p=270#270 buy oxycontin online without a prescription
http://www.cs.montana.edu/forum/viewtopic.php?p=1327#1327 cheap generic xanax