There’s a bit of a ruckus on the web about how Microsoft was supposedly cheating when it comes to Internet Explorer 9’s performance on benchmarks. Digitizor, as well as some enterprising readers over at HackerNews, came to the conclusion that Microsoft included code in IE9 specifically to ace the SunSpider benchmark. I was ready to write a scathing article about this, – until I loaded up the IEBlog. As it turns, it’s not cheating, it’s not a bug – it’s an actual piece of smart code optimisation other browsers don’t have yet.
A Mozilla engineer found out that Microsoft’s Internet Explorer 9 did particularly well on the math-cordic benchmark in the SunSpider suite. When people over at HackerNews (comments at HackerNews: good stuff. Recommended) dove into this matter, they concluded that Microsoft must’ve been cheating. Naturally, I was set to write a scathing item about this, because if there’s one thing I hate, it’s cheating.
Obviously, the first thing I did was try and see if Microsoft had already said something about this matter. As it turns out, they did, and what they have to say about this is kind of embarrassing for those that automatically assumed Microsoft was cheating (including myself) – Redmond is not cheating, and it’s not a bug either; they’ve implemented something called dead code elimination.
“Some of the optimizations we’ve made to the JavaScript interpreter/compiler in IE9 are of a type known in the compiler world as dead code elimination,” the IEBlog explains, “Dead code elimination optimizations look for code that has no effect on a running program, and removes the code from the program. This has a benefit of both reducing the size of the compiled program in memory and running the program faster.”
This explains why the math-cordic benchmark runs so well on IE9. “The benchmark runs an expensive loop, and then does nothing with the results; the benchmark is written exactly in a way that triggers this general optimization,” they add, “Of course, the benchmark could be rewritten to avoid triggering this optimization, which would bring our performance on this specific benchmark in line with other browsers.”
Of course, any sane person already knows that these silly benchmarks are, well, silly. I honestly don’t give a rat’s bum about how well my browser performs on these tests – I care about how fast it renders web pages, how fast it starts up, and how well-designed and responsive the interface is.
naaaahhh, its Microsoft they must have been cheating. it can not be any other way… /end_sarcasm
Being closed source and bugs being hidden behind the most hideous of login walls doesn’t help with transparency when it comes to this problem. They did the right thing in explaining clearly on the blog though.
Incredible how people are!
As if Microsoft was ever convicted for bad businesses practices, for breaking monopoly laws, or for screwing with “parteners”.
</sarcasm>
Edited 2010-11-18 00:08 UTC
Or for humans righs violations … oh ,sorry that was Nokia.
http://www.eff.org/deeplinks/2010/10/saharkhiz-v-nokia
as if they weren’t punished for such crimes and have changed such practices in the 15 years since committing them.
[/sarcasm]
So people should forget about 20 years of wrongdoing and just be happy … Yeah, as if you can tell an old dog new tricks …
Well it’s certainly not hard to understand how people could get that notion if they had ever written code.
Adding either of these two, a ‘true’ and a ‘return’ which both does nothing in the context of this code and thus would certainly be optimized away as dead code suddenly prevents this dead code elimination optimization that before either of these (again, doing nothing) changes was able to optimize away the entire function.
http://people.mozilla.com/~sayrer/2010/sunspider/diff1.html
http://people.mozilla.com/~sayrer/2010/sunspider/diff2.html
But even so I don’t think they are cheating, I believe this is due to the ‘frailty’ of their dead code eliminator, which as proven by this needs more work.
I’ve seen alot weirder things than this in my days as a programmer and I can’t see why the IE devs would cheat this way if they were interested in cheating (which I don’t think they are, I may have issues with Microsoft as a company but I have nothing but respect for their developers and no self-respecting dev would do something like hardcoding a dead code elimination, not even with Ballmer breathing down their necks).
Good news, I’m glad most common browsers have now a very fast javascript engine, it is time to eat all those javascript books getting dust on the shelf.
Yeah and also many not so common browsers use these engines.
Recently found Conkeror, which looks damn nice to me. Efficient (in the vi(m) way), fast, lightweight _very_ (easily) extensible and compatible with Firefox extensions. Oh and it’s a xulrunner application, which means it is easy to hack, since you start it isn’t used as a precompiled application.
Yeah, I know there is vimperator, but it’s different. Pretty nice concept IMO. Oh and I tried an web application (mix of HTML, JS an plugins), which didn’t work on any of the big browsers -didn’t try IE though – and it worked.
Still using Firefox, since I currently don’t have the time to readjust.
Sorry for the off topic. Just wanted to mention it, because it’s a nice alternative browser which I didn’t know and is different from other alternative browsers, like Arora or Midori.
Conk is awesome. I’ve been using it for a couple years now.
Interestingly enough, vimperator is a fork of Conkeror, from back when Conkeror was an FF extension. Probably little to no code in common any more, though they have gotten ideas from each other.
The biggest difference is, like the vim/emacs difference, one uses a config format, the other uses a full turing-complete language.
Edited 2010-11-18 16:02 UTC
It clearly isn’t cheating, but it does mean that the Sunspider benchmark isn’t terribly useful for trying to assess how fast IE9 is going to be for real-world javascript application, which can’t be simply optimised out (not run).
This is a particularly interesting point to note considering that, very shortly after Fireox 4.0b7 has been released containing the new Jaegermonkey engine, Microsoft have just rushed out a new beta of IE9.
http://arstechnica.com/microsoft/news/2010/11/internet-explorer-9-p…
OK then, we can all agree with that. So what do Microsoft do next? They proceed use the flawed Sunspider benchmark, and ONLY the Sunspider benchmark, to compare their new preview of IE9 against other browsers.
So where is the compariosn using Google’s v8bench benchmark? More interestingly, where is the comparison using Mozilla’s Kraken benchmark, which is the one benchmark amongst them all that actually claims that “the focus is on real-world website performance, not on popular but narrow benchmarks like the SunSpider JavaScript benchmark”?
http://blog.mozilla.com/blog/2010/09/14/release-the-kraken-2/
So, Microsoft, we are wondering … how does IE9 perform in comparison to other browsers when measured on a benchmark that actually tries to assess “real-world website performance”? Since there IS a benchmark which actually claims measuring this performance as its primary aim, and since Microsoft say that that is what they are aiming at, then why didn’t they use the Kraken benchmark?
Just saying.
Edited 2010-11-17 23:48 UTC
Well Kraken is actually quite new and SunSpider has been recognised as “the” benchmark to use by everyone for a while now. So I wouldn’t go blaming Microsoft because they are using the most popular benchmark.
There is nothing stopping you or any other person to run these tests in other benchmarking suites and post a review, of course.
Why not? Why shouldn’t I criticise Microsoft, using Microsoft’s own words?
http://blogs.msdn.com/b/ie/archive/2010/09/14/performance-what-comm…
From the horse’s mouth, so to speak.
As you say, Kraken is fairly new, but here is what Microsoft had to say (before Kraken was available) about its predecessor, Dromaeo.
What can I say, other than question why Microsoft are publishing results ONLY from Sunspider, when they are fully aware that it is nonsense to do so? Especially when you consider that in the exact same breath, Microsoft had just said that real world performance was what actually mattered?
Just saying.
Edited 2010-11-18 00:23 UTC
LOL, reminds me of the old days when CPU manufacturers swore up and down that megahertz didn’t matter, until/unless they had the chip with the highest number of mhz, at which time it became a marketing bullet point.
Look, you’re simply a BIGOTED HYPOCRITE. When it served your purposes to use SunSpider to bash Microsoft, it was the greatest thing since sliced bread. Now that Microsoft actually did work to optimize its code to improve its performance on SunSpider, suddenly the test isn’t useful to you anymore as a propaganda tool. LOL!
Quote please.
It wasn’t me who bashed the usefulness of Sunspider. I have had zero comment on it, I haven’t even looked at the code. This very sub-thread, which I started, even has the words “It isn’t cheating” in the title.
It was Microsoft engineers who bashed Sunspider. Here once again is what Microsoft had to say about it:
http://blogs.msdn.com/b/ie/archive/2010/09/14/performance-what-comm…
Let’s stick to the facts, please. We wouldn’t want anyone to get all hysterical and start sprouting unwarrented insults that are clearly wrong, or libelous even.
Oh, wait … too late.
Edited 2010-11-18 21:47 UTC
It was MS?
Aren’t you saying in this topic SunSpider is flawed? sure MS has bashed sunspider, and so do you.
You are so pathetic.
Oh and here is the quote you pathological hypocrite:
They proceed use the flawed Sunspider benchmark
Edited 2010-11-19 01:08 UTC
My my, testy, aren’t we? What an angry little critter.
I called it flawed because Microsft themselves did. I am quoting Microsoft.
I haven’t evaluated javascript benchmarks, but I have read a Microsoft analysis of them. Here it is, so that you too may read it:
http://blogs.msdn.com/b/ie/archive/2010/09/14/performance-what-comm…
Quote from Microsoft regarding Sunspider flaws: “This approach is not representative of real world scenarios and favors some JavaScript engine architectures over others.”
Edited 2010-11-19 01:48 UTC
Haha, try to fix it now.
To late.
Au contraire, I posted the exact same link soon after the post you quoted, which is the fourth post on this topic:
The post you quoted (4th post on this thread):
http://www.osnews.com/permalink?450385
The eleventh post on this thread:
http://www.osnews.com/permalink?450394
lemur2:
That is where I pointed out that I am quoting Microsoft’s own words when I talk about the Sunspider benchmark.
Ages OK.
Edited 2010-11-19 02:37 UTC
This is OT so I’ll keep it short ‘n’ sweet. I’ve been a long time lurker on osnews and never have I seen someone presume they have the right to talk to another human being like that.
Yes you have a difference of opinion, yes you have some good points – I presume (I don’t care to read anything else you write). But with that attitude you just lost the argument.
Please for the sake of a fantastic tech community and news resource, try to treat people with respect, or if that’s beyond you. Just go away.
…
“LOL!”
Thanks for the comment, lemur3, or lemur4, or lemur5, or whoever you are. I could care less.
tomcat cheers for Microsoft. When it happens that he goes off the rails like that, one can be sure that one has made a significant and convincing point that tomcat cannot debate with civility. Ad hominem attacks (a.k.a attack the messenger, not the message) in response are his only recourse.
Indeed one doesn’t have to read tomcats points (if indeed one can find any) to know that when he posts in that way, he has already lost the argument.
I would like to post a note of appreciation for your words, and also to let you know that you are not really OT in posting them. Some people are swayed by strong language and vitriol, thinking that there must be some reason behind such anger. There is, but the reason is simply an attempt to discredit that other party in the debate, it has nothing to do with the validity or otherwise of the actual points made in the debate. Anyway, it can’t but help to point these things out, so that perhaps people won’t be swayed by them.
Fixed it for you.
im glad you ask, I just made a benchmark with kraken of IE9 P7 vs Chrome and the winner was IE9 PV7.
Edited 2010-11-18 00:02 UTC
I completely disagree. Have you seen much real world JavaScript code??? I think Microsoft is just scratching the surface with dead code removal. Look out for the upcoming announcement of stupid code removal, which will revolutionize the internet by actively avoiding execution of about the 80% or so of JavaScript code which does nothing useful anyway.
You might well disagree with me, but Microsoft’s own engineers happen to agree with me. See for yourself:
http://blogs.msdn.com/b/ie/archive/2010/09/14/performance-what-comm…
http://www.osnews.com/permalink?450394
It is only the Microsoft PR people (who presumably are desparate to spread false impressions) who would put up results from Sunspider ONLY, after having just said in the very sentence before that what really mattered was real-world performance.
It is pretty funny, really.
Just saying.
Edited 2010-11-18 00:30 UTC
My post was a joke… Sorry if that wasn’t obvious.
That aside, if you want a serious response. I don’t care what Microsoft’s PR people say, just as I don’t generally care what any PR people say – their job is to present a version of the truth that suits their employers desired outcome.
Just to push the point, benchmarking a product in order to give developers feedback to improve it is a useful exercise. The minute you (as in the software developer) publish the benchmark you are in PR territory and caveat emptor to the reader.
I’m not saying that all PR in the software world is veiled deception, but it sure as hell can be, and it doesn’t matter who the company is. There isn’t much use in pointing fingers at them when they twist the truth, you may as well blame a skunk for stinking…
And no, I don’t think it is particularly funny or surprising that some actual developers at MS don’t agree with how PR characterizes their work – that is probably true of any software company. You may not like MS, but they employ quite a lot of intelligent developers, and most intelligent people who work hard at what they do don’t like their work being distilled down to sound bites and promoted like it was a soft drink – it cheapens what is otherwise a satisfying experience (i.e. trying to write better software). So I would expect any good software company to have a few developers that don’t tow the PR party line – it is a sign of health if you ask me.
Its the companies that manage to present a single unified message to the outside world without a single voice of dissent in sight which you should be frightened of…
Sorry, you are right, I didn’t pick up on the joke in your post. I read it too quickly. Pardon me.
Fair enough, yours is a pretty reasonable attitude actually. So too is the apparent attitude of the Microsoft developers.
One small part of my complaint is that this is not what ordinary people get to hear.
A larger part of my complaint is that websites such as Ars Technica should know better, and they should present material like this (performance comparisons) with a bit of technical balance to it, and do some investigations of their own, and not simply regurgitate what Microsoft marketing PR has to say. We all know how reliable Microsoft PR and marketing is liable to be … which is to say, not one bit reliable.
The most significant part of what I would say about all this is for people to take this as a lesson learned. There are a few useful “rules to live by” that can be gleaned from this:
(1) Do not believe what mainstream media is trying to tell you without examining it at least a little bit for yourself. “Follow the money” is a good rule of thumb to use to sort out what is really going on.
(2) Look at whatever Microsoft marketing say, and then examine what they do NOT say about a topic. The latter examination is most likely to uncover some actual truth.
(3) If someone makes a counter-claim about a talking point that Microsoft marketing are pushing, do not discount out of hand what the naysayers have to say. Microsoft marketing and PR most definitely has an agenda. “Follow the money” once again is a good rule of thumb.
The thing is, if a javascript engine is in the realm of V8, it is way more then fast enough. If it is 0.8x as fast, or 2x as fast is actually pretty irrelevant, because it will be more then fast enough for anything that a webpage will throw at it.
The major issue nowadays is dom manipulation (and style recalculation) speeds. Sun Spider numbers are for browser developers ego more then anything else when you are talking about modern browsers. Sun Spider like tests still make sense for the server side implementations though.
Reminds me of a story back in my days at DEC.
Another software engineer came over to my office all upset… I asked why and he said “I’ve written the same exact program in 4 languages, compiled it and ran it…”
“Yes, so?”
“The results make NO SENSE!”
So I went over to watch and he ran the test (just a simple loop/timer test). He wrote a program in C, BASIC, FORTRAN and COBOL. We made an assumption that since the compiler groups share technologies, that they should all pretty much compile down to the same machine code and the execution time should be similar. We expected BASIC and COBOL to take longer simply because they always load some extra memory management stuff ahead of the game (pre-allocate memory for string handling and such). But the results (and this is just shooting out of my butt because this was 20 years ago) were something like:
COBOL : 1.2 seconds
BASIC : 2.0 seconds
C : 0.5 seconds
FORTRAN: 0 seconds
He was upset because 0 was impossible. So he went into the machine code and found that the FORTRAN compiler was so good at optimizing looping and math code that it simply determined the code was unused and optimized it out. The program did nothing so the run time was zero.
It was pretty funny.
The point behind the story being large companies in the habit of producing compilers that other businesses depend on have dedicated engineers always finding ways to improve the performance. IBM, Intel, MS, DEC, etc. should be expected to implement optimizations like that.
I’m not a coder, but am interested: How important or valuable is “dead code elimination” in day-to-day use. Apparently it is not included in other browsers. I would guess that it is not that much of a speed booster since I cannot imagine that Microsoft coders are that much smarter that all other coders (a practical statement, not a slur on MS coders). You can see how this would indicate Microsoft’s intentions. Would someone help me understand, please?
All I know is that it is or at least WAS a common practice back in the day of mainframes. Compilers were written to be highly optimizing which included being able to identify unused or non-functional (code that doesn’t contribute to the data end result in any way) and not include it when translating to machine code.
This was a practice in place decades ago.
I do not see why it should indicate their intentions other than to add optimizations that could possibly speed things up. I liken it more to a developer sitting there one day and having light bulb go on in his head and saying “HEY! I’ve got an idea, why don’t we do THIS!”
But since we cannot know this, it could very well be that they analyzed the tests the competition uses and realized they could speed it up that way…
Also, I should note… the test code that was written in my above example was an increment algorithm.
The programs basically were:
var x = 0;
while (x < 1000000) do
x = x + 1
end
print x
…the FORTRAN compiler recognized the loop, actually performed the calculation and placed it in the code as:
print 1000000
That’s a nice optimization when it saves tons of CPU cycles in a program. Another thing to note is that DEC wrote one of the best FORTRAN compilers in existence and it was used in scientific and government circles for large calculations.
Given the fact that at least as far as I can see, this is strictly a compile time optimization (even if it being done by JIT), the compiler can’t really notice that:
for (count=0;count<x;count++); y+=count
… is dead code when x=0 (resulting in y=y), as long as x isn’t constant (read: calculated or given as input).
As such, dead code removal is more-or-less limited to bad-programmer-prevention – read: cleaning up after sloppy programmers.
I doubt that it has much effect in real life programming. (Though I’m a C programmer, so I can’t really comment about JS programmers…)
– Gilboa
Dead code can sometimes occur as a result of other optimizations. You could have every optimization that might produce dead code try to clean it up in some ad hoc way, but it’s a lot easier and cleaner to simply make a global dead code elimination pass.
Here is a (somewhat contrived) example for constant propagation:
x = 5;
y = x + z;
The compiler can propagate the constant and rewrite y to be:
y = 5 + z;
at which point the assignment to x becomes dead.
True, but such optimization will usually remove a single line. While it is possible that such an optimization will remove a big chunk of code (E.g. for/while/if/etc), my gut tells me that it’ll be very rare.
– Gilboa
Often dead code elimination results in a cascade of dead code, so some if statements might end up being eliminated. Loops will almost never be eliminated because doing so might change the behavior of the program.
while(true) { }
is pure dead code, but eliminating it will change the behavior of the program and make a non-terminating program terminate.
Actually while (true) must -not- be deleted by dead code removal.
At times, endless while look is actually quite useful. (E.g. waiting for debugger to connect).
– Gilboa
Well, it can notice that,
– given x <= 1, the loop is dead code (noop)
– given x > 1, the whole loop is equivalent to: y += x*(x-1)/2.
Hence, a really good compiler should probably replace the loop by a proper
if (x>1) y += x*(x-1)/2
Edit: what I’m trying to prove here is that, when different compiler optimizations start to work together, the results can be amazing and really difficult to predict. Put it in other way, you don’t need *bad* code nor a silly programmer to produce code that can be significantly optimized…
Edited 2010-11-22 00:14 UTC
Th sensible conclusion is that the benchmark’s test makes no sense. It is too trivial to be representative of any significant proportion of real-world execution scenarios (which cannnot be eliminated to zero cycles).
I don’t disagree.
If you look at the first comment on the hacker news post that you link to–or various other places around the web–you see that people have actually looked at the idea that it’s dead code analysis.
One of many things that have been done to test is just adding a “true;” statement into the function body that is being tested, (that’s also a noop, basically the epitome of dead code) with that “dead code” inserted the Chakra Javascript engine does the full test. (Similar other meaningless changes to the code also result in Chakra running the full test.)
What this means is that if IE is actually doing dead code analysis, it’s doing it incredibly narrowly. Which is what people have been saying since the beginning. It’s still probably cheating, even though Microsoft decided to talk about it.
To be fair, I would be careful of trying to characterize this as cheating. Since it is closed source, we can’t see what the engine is actually doing.
I’m just saying it is entirely plausible that adding a true statement is pushing some internal threshold for this optimization over some edge. The point being that since JavaScript is a dynamic language dead code elimination is quite a bit more complex than with static languages and their implementation may just be initially very simple-minded so as not to cause issues during runtime code changes.
I do think it would be possible to determine with some thorough testing whether or not they are actually cheating here. If they are proven to be cheating shame on them. I just don’t think that a few iterations of salting the code with some true or return statements is thorough.
Just saying, don’t attribute to malice which can be easily explained by ignorance…
I don’t think is cheating, I ran the kraken bentchmark and It was faster than Chrome.
Edited 2010-11-18 01:28 UTC
Again, to be fair, that may be true but it doesn’t in any way indicate that they were not cheating, as dead code elimination generally only optimizes poorly thought out code – most of the time it shouldn’t have any positive effect at all because the dead code usually indicates an error on the programmers part (properly written code generally doesn’t contain any dead code paths).
Some thorough testing will determine whether they were cheating easily enough… Since they seem to be sticking by their guns I expect they will be vindicated after further review. If not, like I said shame on them.
Is not cheating, is smart way of optimization. Bypassing useless code is a well known algoritm.
Edited 2010-11-18 01:58 UTC
Agreed. The question is whether the browser itself is optimizing out dead code – or whether it’s been written to recognize the specific example of dead code in this particular benchmark.
Being closed-source, there’s no way of knowing, but the claimed fragility in the face of minor changes to the test suggests cheating is possible. Of course, it just could be that the optimizer just isn’t that good yet – it just happens to work well in this case because the benchmark doubles as a test-case for the engine…
Are you taking a leaf out of Microsoft’s book here, and only talking about results where IE9 came out ahead?
I could do the same if you like. Here you go then, here is a nice graph to chew on:
http://blog.mozilla.com/dmandelin/files/2010/11/kraken-1103.png
This is the kind of graph that you WON’T see (I’d wager) in the mainstream media.
I trust the benthmark I can run, not the betchmarks a well known anti-ms troll like you put. And my bentchmar concurr with kraen and sunspider, IE9 P7 is faster than Chrome.
Edited 2010-11-18 02:02 UTC
Here is some help for running the Kraken benchmark for yourself:
http://krakenbenchmark.mozilla.com/
You have probably already found that link. OK, apart from IE9 preview 7, here is another link that you might need:
http://www.mozilla.com/en-US/firefox/all-beta.html
Let us know how it turns out.
PS: As far as trusting benchmarks one can run oneself, it transpires that IE9 P7 doesn’t run on my machine, it doesn’t install properly. In fact, the installer doesn’t even get as far as the first “next” button. I am running Arch Linux, which runs Firefox, Chrome and Opera really nicely. Do you have any suggestions?
Edited 2010-11-18 02:37 UTC
IE9 wins again.
I doubt it. I doubt it very much indeed.
That’s your problem, not mine.
Not if the dude who claimed IE9 wins all the possible benchmarks obviously lives in hallucination, and had made some comments like
Hey, he could be right! I’m sure everyone at Microsoft utterly believes that IE9 will completely blow the competition away, and that Windows Phone 7 is so fantastic it will sell like hotcakes and utterly remove Android from the marketplace.
After all, Microsoft marketing PR tells us so.
http://seattletimes.nwsource.com/html/microsoftpri0/2013442919_stev…
Follow the money.
Edited 2010-11-18 04:01 UTC
I never read your spam links, so don’t waste your time.
Edited 2010-11-18 04:11 UTC
Your loss … the link to the Seattle Times had a hot stock tip about a local company.
Edited 2010-11-18 04:19 UTC
Still, it comes from you, so It can’t be good.
Edited 2010-11-18 04:19 UTC
It doesn’t come from me, it comes from the Seattle Times. Unlike Microsoft PR benchmarking results, I’m sure it is accurate reporting, and it probably has significant portent for the future.
Edited 2010-11-18 04:24 UTC
Still, you put the link, so I won’t click.
What a good policy, like these guys I suppose.
http://en.wikipedia.org/wiki/File:Hear_speak_see_no_evil_Toshogu.jp…
But, just for you, free for today, I will try to save you from yourself.
http://seattletimes.nwsource.com/html/microsoftpri0/2013442919_stev…
Onya Stevie!
http://www.wildjunket.com/2009/02/03/the-aussie-slang-gday-mate/
Edited 2010-11-18 04:52 UTC
Pointless.
Maybe it is. I admit it is a bit off-topic, but of all people who had a notion of the future profitability of Microsoft, one would think that Ballmer world be it.
Ballmer appears to be facing a bit of a push from shareholders and financial powerborokers to break up the company.
http://www.techflash.com/seattle/2010/11/ballmer-and-gates-heres-wh…
Reading between the lines, Ballmer says he had a look at it, but he decided that if they break up the company now, they will get pulverized in phones, servers and search. The only thing holding it together is the semi lock-in they have in trying to create the impression that you need to “go with Microsoft” all the way.
You actaully don’t. You are actually better off dropping Microsoft in any or all of those areas. The cutting edge is elsewhere, Microsoft is all lock-in, expense and encumberances that you don’t need. They are fighting an uphill battle. Every blackhat and cracker on the planet is targetting their platform. They have to struggle and struggle to make even the most minor of changes. It is nearly impossible for them to innovate.
I think Ballmer can see the writing on the wall.
I think this is why Ballmer talks gobbledegook to the press.
I think this is why Microsoft marketing has to make up these little lies all of the time.
Edited 2010-11-18 05:17 UTC
And your point is?
Edited 2010-11-18 04:11 UTC
That hallucinating guys are not trustable?
Kraken on IE9 Platform Preview 1.9.8023.6000
=============================================
http://krakenbenchmark.mozilla.com/kraken-1.0/results.html?%7B~…
Kraken JavaScript Benchmark Results Content Version: kraken-1.0Run Again
(You can bookmark this results URL for later comparison.)
To compare to another run, paste a saved result URL in the text field below and press enter:
===============================================RESULTS (means and 95% confidence intervals)———————————————–Total: 14450.0ms +/- 0.4%———————————————– ai: 1079.2ms +/- 1.3% astar: 1079.2ms +/- 1.3% audio: 6502.2ms +/- 0.7% beat-detection: 1333.0ms +/- 0.5% dft: 2817.2ms +/- 1.2% fft: 1304.8ms +/- 0.9% oscillator: 1047.2ms +/- 0.7% imaging: 4843.6ms +/- 0.5% gaussian-blur: 2799.3ms +/- 0.4% darkroom: 1125.7ms +/- 0.8% desaturate: 918.6ms +/- 1.1% json: 226.4ms +/- 0.8% parse-financial: 100.8ms +/- 1.2% stringify-tinderbox: 125.6ms +/- 1.4% stanford: 1798.6ms +/- 0.3% crypto-aes: 353.1ms +/- 0.6% crypto-ccm: 263.5ms +/- 0.8% crypto-pbkdf2: 909.7ms +/- 0.4% crypto-sha256-iterative: 272.3ms +/- 0.7%
Kraken on Chromium 9.0.579.65687
================================
http://krakenbenchmark.mozilla.com/kraken-1.0/results.html?%7B~…
===============================================
RESULTS (means and 95% confidence intervals)
———————————————–
Total: 13804.0ms +/- 0.7%
———————————————–
ai: 914.7ms +/- 0.6%
astar: 914.7ms +/- 0.6%
audio: 5035.8ms +/- 0.9%
beat-detection: 1320.0ms +/- 0.6%
dft: 1881.6ms +/- 1.7%
fft: 1245.5ms +/- 1.1%
oscillator: 588.7ms +/- 3.0%
imaging: 5769.0ms +/- 0.5%
gaussian-blur: 2381.8ms +/- 0.4%
darkroom: 1679.9ms +/- 1.1%
desaturate: 1707.3ms +/- 0.8%
json: 469.9ms +/- 0.8%
parse-financial: 216.8ms +/- 1.3%
stringify-tinderbox: 253.1ms +/- 1.1%
stanford: 1614.6ms +/- 1.3%
crypto-aes: 291.7ms +/- 4.7%
crypto-ccm: 244.1ms +/- 1.3%
crypto-pbkdf2: 799.3ms +/- 0.8%
crypto-sha256-iterative: 279.5ms +/- 1.1%
The comparison of the two:
—————————-
TEST COMPARISON FROM TO DETAILS
====================================================================== ==============
** TOTAL **: 1.047x as fast 14450.0ms +/- 0.4% 13804.0ms +/- 0.7% significant
====================================================================== ==============
ai: 1.180x as fast 1079.2ms +/- 1.3% 914.7ms +/- 0.6% significant
astar: 1.180x as fast 1079.2ms +/- 1.3% 914.7ms +/- 0.6% significant
audio: 1.29x as fast 6502.2ms +/- 0.7% 5035.8ms +/- 0.9% significant
beat-detection: 1.010x as fast 1333.0ms +/- 0.5% 1320.0ms +/- 0.6% significant
dft: 1.50x as fast 2817.2ms +/- 1.2% 1881.6ms +/- 1.7% significant
fft: 1.048x as fast 1304.8ms +/- 0.9% 1245.5ms +/- 1.1% significant
oscillator: 1.78x as fast 1047.2ms +/- 0.7% 588.7ms +/- 3.0% significant
imaging: *1.191x as slow* 4843.6ms +/- 0.5% 5769.0ms +/- 0.5% significant
gaussian-blur: 1.175x as fast 2799.3ms +/- 0.4% 2381.8ms +/- 0.4% significant
darkroom: *1.49x as slow* 1125.7ms +/- 0.8% 1679.9ms +/- 1.1% significant
desaturate: *1.86x as slow* 918.6ms +/- 1.1% 1707.3ms +/- 0.8% significant
json: *2.08x as slow* 226.4ms +/- 0.8% 469.9ms +/- 0.8% significant
parse-financial: *2.15x as slow* 100.8ms +/- 1.2% 216.8ms +/- 1.3% significant
stringify-tinderbox: *2.02x as slow* 125.6ms +/- 1.4% 253.1ms +/- 1.1% significant
stanford: 1.114x as fast 1798.6ms +/- 0.3% 1614.6ms +/- 1.3% significant
crypto-aes: 1.21x as fast 353.1ms +/- 0.6% 291.7ms +/- 4.7% significant
crypto-ccm: 1.079x as fast 263.5ms +/- 0.8% 244.1ms +/- 1.3% significant
crypto-pbkdf2: 1.138x as fast 909.7ms +/- 0.4% 799.3ms +/- 0.8% significant
crypto-sha256-iterative: *1.026x as slow* 272.3ms +/- 0.7% 279.5ms +/- 1.1% significant
And HOLY MOLEY! Here is the Firefox 4 B7 results
=================================================
http://krakenbenchmark.mozilla.com/kraken-1.0/results.html?%7B~…
===============================================
RESULTS (means and 95% confidence intervals)
———————————————–
Total: 6969.8ms +/- 1.4%
———————————————–
ai: 1817.5ms +/- 4.0%
astar: 1817.5ms +/- 4.0%
audio: 2580.0ms +/- 0.8%
beat-detection: 716.6ms +/- 1.1%
dft: 552.3ms +/- 1.9%
fft: 572.4ms +/- 2.2%
oscillator: 738.7ms +/- 0.6%
imaging: 1534.9ms +/- 0.6%
gaussian-blur: 654.5ms +/- 0.3%
darkroom: 298.5ms +/- 0.7%
desaturate: 581.9ms +/- 1.3%
json: 222.5ms +/- 0.8%
parse-financial: 148.1ms +/- 1.1%
stringify-tinderbox: 74.4ms +/- 0.9%
stanford: 814.9ms +/- 1.7%
crypto-aes: 211.3ms +/- 0.8%
crypto-ccm: 140.9ms +/- 0.5%
crypto-pbkdf2: 335.6ms +/- 4.5%
crypto-sha256-iterative: 127.1ms +/- 1.8%
Since the back and forth on this were bugging me…
Just ran these, and I don’t much care what wins since I prefer Chrome mostly for reasons other than performance (I just like the UI).
13″ Macbook Pro/Windows 7 Ultimate – 64-bit
2.4Ghz Core2 Duo/4GB Memory
Kraken 1.0 Benchmark Results
Chrome 7.0.517.44
18887.1ms
IE9 B2 9.0.7930.16406
61780.8ms
IE9 P7 9.0.8023.6000
18594.9ms
Firefox 4 B7
9115.5ms
Note: FF doesn’t report a build number in the about box, 2.0.0.3960 is the file version as reported by the properties on firefox.exe, which may not mean anything useful. But it is B7 I just downloaded and installed it)
So yes, IE9 P7 does beat the production release of Chrome by a hair, however is gets smoked quite badly by FF B7. Granted results are going to probably differ some based on platform, but on a Core2 Duo IE9 gets it’s ass kicked severely in this benchmark by FF. To be fair it is Mozilla’s benchmark, so I’m not terribly surprised. But the fact that IE9 P7 is on the same level as Chrome (and have roughly tripled performance compared to their beta in a fairly short time) leads me to believe they are definitely on the right track.
That said, the UI in FF B7 feels sluggish compared to Chrome… Ive only used it for a few minutes, but I already don’t like it much. Hopefully that will improved over time.
Edited 2010-11-18 05:37 UTC
Yeah, my scores are posted above, with a lot of cruft, sorry. Basically the same results on a 6 Core AMD with 8 gigs of RAM…
Chrome still feels snappiest to me… FF B7 next followed by IE9. It’s probably all just some weird perception thing since page rendering times between the three cannot be vastly different.
I am using FF B7 to write this post.
But as an aside, I thought FF was going to use process isolation for tabs like Chrome in FF4? It isn’t, apparently. Comparatively speaking, FF B7 uses about 10000K more of RAM than Chrome 9…
Mate, you did a great job earlier by including uncertainties in your benchmark duration analyses and showing how in som cases the time differences are statistically insignificant. Now you’v just let yourself down worrying about an extra 10 MB memory usage on a machine you boast has 8 GB RAM. How sigificant is that? It’s pretty meaningless these days unless you are on an severely constrained embedded system (even they are starting to get large amounts of memory since it is so easy to work with and so cheap to manufacture). Keep up the good stats work though.
LOL!
True… but I come from a time when you used variable naming conventions that consisted of a max of 6 characters for a variety of reasons, and memory management had to be very aggressive. It still shocks me that mainframes could manage 60 people on 64 MB of ram; I have not only a super computer today, but a super-DUPER computer by the computing standards of the 70’s and 80’s.
And one final note, IE9 uses about half the RAM of both Chrome 9 and FF 4 (for same number of tabs, same sites).
HOWEVER… Font rendering on IE9 and FF5 is the same, and I prefer the rendering of Chrome, which is crisper.
Edited 2010-11-18 05:46 UTC
How’d you get IE9 P7 to use tabs? It only lets me open one site at a time… Or do you just use multiple open windows?
I noticed that too. I much prefer Chrome’s font rendering (at least on my laptop’s screen).
OH! Sorry… IE 9 Beta … the memory usage was with the beta, not the latest preview (altho’ I have done the “cheat” and set up a copy of IE9 Beta with the latest preview of the back end.)
Gotcha. Yeah, it looks about the same on my system.
That’s my problem with Chrome, the NIH mentality to do its own font rendering, when it should follow the system setting.
Anyway, I do like Chrome’s font rendering (on Linux), so what I did was to tweak the system setting such that all other apps render the font that way. However, if I happened to not like Chrome’s font rendering, I doubt there is a way to make it render the font like all other apps.
I don’t think it NIH was the reason why Chrome does its own font rendering, Google probably wanted Chrome looking and working exactly the same cross OS.
Why not answer to the point (e.g. by posting your benchmark results) instead of engaging in 3 y/o like personal attacks against the OP?
He posted results. You have contrary results? Post them!
– Gilboa
We need to not just benchmark for making good code fast.
We need to make benchmarks that test making terrible code at least functional. Sisyphean, but think of the rewards if it works!
Facebook?
It’s too bad geocities is offline. That was always a great place to find crap code, whether Javascript or HTML, for real-world-crap testing.
Ha ha! I was going to write that FIRST…
I figured I’d take the next best and still active thing.
There’s always Myspace.
It may not be cheating but its certainly dishonest. Microsoft knew that they weren’t running the full test they way it was designed to be run. Yet they tout their results without telling anyone that the test is now meaningless because they added a short cut to their code. Its like a marathon runner taking a taxi to the finish line. Nice and classy Microsoft.
Not at all – this isn’t a race, with rules about following the correct route. If their runtime can correctly recognise that a bunch of code has no effect, then skipping it is absolutely the right thing to do – in compiled languages, this kind of thing is normal.
The only concern with this is whether it skips the test because the runtime itself is smart enough to recognize the test code as unnecessary, or because a Microsoft engineer recognized the test code as unnecessary and wrote the runtime to skip it.
So basically it’s a bug in Sunspider
Basically. Looks like a ‘naive’ benchmark.
Maaaybe the “true;” caused the dead-code detection to fail, although I can’t see how.
But how on earth can adding “return;” to the end of a function change anything at all? It shouldn’t change the AST at all.
Still highly suspicious.
Rob Sayre has posted a new entry on his blog,
http://blog.mozilla.com/rob-sayre/2010/11/17/dead-code-elimination-…
Can anyone explain to me how the redundant “return” at the end of the function can cause the DCE to fail.
This is too funny
While I can buy that MS did put a dead-code eliminator in there, the changes that were introduced that broke IE9 should not have – the optimizer should have determined that it was still dead code that didn’t do anything, and thus it should have continued to run at the same level. Thereby, either Microsoft is cheating or their optimizer is broken if it exists at all.
Well, again I can see that one would be able to draw that conclusion from what we’ve seen, but guys over at hackerne.ws have been able to create other code snippets different from cordicsincos() in which IE9 optimizes away the entire function, and it fails to do so also in those cases when a true or return is added so logically there really must be a dead code elimination optimization in there, but it’s obviously very fragile or very specifically tuned.
But EITHER WAY, it doesn’t matter because it says absolutely nothing about real world performance because there’s no reason for a real world program to call a function that does nothing or returns nothing of interest, which means it won’t be able to optimize it away. The snippets shown on msdn are not real world examples, and while there are certainly often occurences of code that is never called in programs, for performance gains to be had they would actually have to be called, and for these gains to be of any potential it would have to be called very often.
So for this optimization to be of use in the ‘real world’ someone would have had to write a program that continously call an expensive function that in the end does nothing or returns nothing that will be used, which would then mark it as a candidate to be removed as ‘dead code’. In short, you won’t see this optimization yeilding any major gains outside synthetic benchmarks, which is probably why no other browser has yet bothered with implementing it.
Without examining it in detail, this seems to me a very good reason why Microsoft has been able to just slightly edge out other browsers (by the barest of margins) on the Sunspider benchmark, yet still manage to get soundly beaten on other benchmark tests which reportedly (and according to Microsoft’s own engineers) include more real-world-like javascript code.
Essentially, as I understand it, if the javascript code in question actually does something, the code won’t be dead code, and Microsoft’s optimization won’t be able to eliminate it.
Paraphrased: As far as I can see, this optimization is only useful for the purpose of obtaining a better score on weak benchmarks like Sunspider (as opined by Microsoft’s engineers).
Figuring out exactly how “obtaining a better score on weak benchmarks like Sunspider” helps users in any way, or is anything other than pure PR deception, is an exercise I will leave up to the readers.
Well, the code may do something, but what it does is useless since it’s not used in anyway by the program. It’s akin to someone standing in the corner and counting to a million for no reason other than having to do just that before being allowed to move on.
This happens in synthetic benchmarks, but not in real world programs, unless someone actually wants their code to run slower for some obscure reason.
Precisely. Spot on. Exactly so.
So, follow that observation through. Since dead code is typically found in (flawed) synthetic benchmarks but not so much in real world code, what exactly is the purpose of dead code optimization? Is it perhaps to be able to run synthetic benchmarks tolerably well, even when one’s browser doesn’t do so well for real world code? Maybe?
It is very normal for any compiler to perform dead code elimination, it is a very standard technique. It isn’t at all shocking that Microsoft has also implemented it.
It may not be cheating, but it is broken:
http://arstechnica.com/microsoft/news/2010/11/lies-damned-lies-and-…
Thom did you consider last update on
http://digitizor.com/2010/11/17/internet-explorer-9-caught-cheating…
? They re-responsed Microsoft.
from digitizor.com:
Microsoft has updated their blog post addressing this issue. (Read here.) They attribute this to dead code elimination. They did not include any explanation as to why the dead code elimination fails miserably on adding “true†or “return“, which changes nothing in the actual function, or even why it fails when the for statement is replaced by while as demonstrated in Hacker News. It could be a bug or them over-training the dead code elimination method for SunSpider though.