Khoi Vinh on why 24 hour or even weeklong reviews are dumb:
However I’ve come to believe that there’s at least one thing wrong with this whole notion of product reviews – and with smartphone revirews in particular – and that’s that by and large they’re only ever interested in these phones when they’re brand new.
When an iPhone debuts it’s literally at the very peak of its powers. All the software that it runs has been optimized for that particular model, and as a result everything seems to run incredibly smoothly.
As time goes on though, as newer versions of the operating system roll out, as there are more and more demands put on the phone, it inevitably gets slower and less performant. A case in point: I’m upgrading to this iPhone X from a three-year old iPhone 6 Plus and for at least the last year, and especially over the last three months, it has struggled mightily to perform simple tasks like launching the camera, fetching email, even basic typing. People who have recently had the misfortune of having to use my phone tell me almost instantly, “Your phone sucks.”
You could argue that three years is an unrealistically long time to expect a smartphone to be able to keep up with the rapidly changing – and almost exponentially increasing – demands that we as users put on these devices. Personally, I would argue the opposite, that these things should be built to last at least three years, if for no other reason than as a society we shouldn’t be throwing these devices away so quickly.
This is, of course, the reason behind the odd embargo strategy Apple employed regarding the iPhone X – if you only give people an hour or at best, 24 hours, to review a device, people will still be in the honeymoon phase of owning a product, where you’re still rationalising spending €1200 for a phone (or any other high price for any other product, for that matter). Choice-supportive bias is a real thing, and each and every one of us experiences it. During this period, initial flaws aren’t as apparent, and long-term flaws or flaws that only pop up in specific situations aren’t yet taken into account. It makes the product appear better than it really is.
This is why, back when I still did reviews for OSNews, I had my own rule of using a product for at least four weeks before publishing a review. This gave me enough time to get over this initial phase, and made sure I had a more levelheaded look at the whole thing. We don’t do many reviews anymore – I have to buy everything myself, and I’m not rich – so it’s not an issue at this point, but even if companies were to approach us today for reviews, I would still ask for that four week period, and if they were to object – sorry, but no review.
This is, of course, what the major publications should’ve done. Nobody forced The Verge or whomever else to publish a review within 24 hours. The initial embargo rush is important for the bottom-line, I get that, but it still feels rather suspicious. What can you really learn about a product in just 24 hours? Can you really declare something “the best damn product Apple ever made” after using it for less than a day? At what point does writing most of the review in advance before you even receive the product in the first place, peppering it with a few paragraphs inspired by the 24 hours, cross into utter dishonesty?
By reviewing products in a day or less, popular tech media is really doing readers and consumers a huge disservice, only further strengthening the idea that the tech press is often nothing but an extension of a company’s PR department. This erodes credibility, and in turn hurts those among the media who do take their time to properly review a product.
It’s okay to not rush writing a review to meet some asinine embargo. It’s okay to not ask “how high?” when a company tells you to jump. It’s okay to publish a review a week or even a month after an embargo has been lifted. It’s okay to not post unboxing videos of non-retail boxes.
It’s okay to, sometimes, just say no.
You probably don’t know this, but Australia has just had a nation-wide opinion poll paid for by the government regarding Marriage Equality.
And the “oppose” voice’s cactchline was “It’s OK to say ‘No'”
Made for an odd reading of your end-point from an Australian perspective :p
While I don’t disagree with Thom’s comments, they seem disconnected with what I took away from the article.
It is not just that reviewers only use the device for 24 hours. They could use it for 4 weeks and still be impressed.
The core point of the article, in my view, was that we should be writing product reviews of hardware that has already been in the market for a year or more. That is, after all, when a great deal of the market will first purchase the device. Reading an article about a device that was released 2 years ago will give an unrealistically positive impression versus the actual experience of owning that device today.
I am an iPhone 6s owner now. I agree with the article. Although the specs say my phone should still be decent it is, in practice, a terrible tool. It is unreliable, erratic, and performs poorly enough that it impacts my ability to use it at all in many day-to-day situations. If I wrote a review about my phone today, I could not recommend it. This is a device that was arguably the best in the market not all that long ago.
Edited 2017-11-09 00:38 UTC
Counter-point, I just recently got a 6s (to replace a 1.5-year old midrange Android which surely affects my perspective, not coming from top of te line flagsip), and I find the 6s performs about on-par as an iPhone 8 (I used an 8 for a week before going for the 6s).
Note that I don’t do social media, and got that free battery replacement from Apple due to unexpected shutdown issues (which I didn’t experience but previous owner did). Performance is great, typing is great, no complaints here.
(And this is why we need multiple reviews, too, because experiences vary!)
I didn’t read the article itself, but just from the lead quoted here, that is also what I thought the article was about. And it is a perfectly valid point, although I don’t see how year-long review periods would be feasible in today’s “I want it now” Zeitgeist. This might especially be true for Apple-fanbois.
Such reviews are not completely unheard of, however. At least with cars, one can find reviews that start with “I have had this car for X years…”. Unfortunately, it is not the norm with other appliances.
Edited 2017-11-09 10:05 UTC
It really depends not just on what’s being reviewed, but also on the expected product lifetime and how much the product is likely to change when it gets updated.
For most people, cars have an expected lifetime of at least five years, and potentially more than a decade. They also have a very consistent versioning cycle, and upgrades from version to version are usually small incremental changes that would be classified as practical only for a point release in most software. As a result, they tend to be given much longer periods of review prior to reviews being posted, especially because people who care about having the ‘newest’ car don’t care as much how it will perform long term (since they’re liable to just buy a new one in a year or so anyway).
You can see a similar tendency to car reviews in American reviews of firearms, which similarly to cars tend to have a long expected product lifetime and change relatively little from iteration to iteration. They tend to focus specifically on accuracy, and to a lesser degree, ergonomics, which are the two biggest factors for most people who are doing their research when trying to decide what gun to buy, and usually do the testing with a sufficient amount of ammunition (typically at least five hundred rounds, more often at least a thousand) to cover the typical wear on the gun from a owner who does a good job of properly maintaining it (which coincidentally tends to be the same type of person who does their research for something like this).
Smart phones, computers, and similar technology however, carry a general perception of a relatively short expected lifetime (often no more than six months to a year) among the people who tend to read reviews, and usually change very drastically from iteration to iteration (look for example at how much the iPod changed at each iteration, and how much the iPhone and Samsung Galaxy S phones change each year). Because of this, and the fact that most people consider ‘best’ to be the same as ‘newest’ when talking about technology (rightfully so to a certain degree), there’s a much greater immediacy to the review, although I agree that the immediacy doesn’t warrant only a day or two of usage.
My iPhone 5 works fine.
I’m also on a three-year old iPhone 6 (non Plus though) and my experience has been better than the authors. Heck, I’d say the 6 performed worst with iOS 8 (not a version iOS historians look back on fondly) that it shipped with. It improved considerably with iOS 9, and stayed on par throughout iOS 10. Now currently on iOS 11.1, and aside from the occasional stutter when typing (something that’s been there with every iOS version this phone has had), some apps taking a bit longer to launch, apps being forced to reload more often when multitasking (only 1GB RAM) and battery life that’s representative of the degradation from three years worth of charge cycles, it’s been pretty decent in daily use. I must be doing something wrong.
Other than that bit, I agree that most of these “early” reviews are dumb. Lets not even talk about that iMore reviewer, shall we. Apple could released bottled water tomorrow and the guy would say it’s the best water ever made, tasting more watery than any water that’s come before.
Edited 2017-11-09 01:52 UTC
I completely agree from a quality-content point of view, but commercial realities for publications are complicated…
There are probably principled publications that refused the 24-hour review, but they aren’t noticed for it. (They just have no review.)
And these short, all-hype reviews are probably more widely read (generating more ad revenue) than in-depth articles. As long as there’s money to be made in publishing that sort of content, it’s probably going to continue.
It’s annoying, and it’s worth complaining about (because backlash is one of the few counterbalances to how successful cheap content is)… I guess I don’t disagree, but also find it hard to blame the publications
I think another aspect missed in Thom’s listing, is that reviewers are experts in things. They care way too much about things most people simply do not.
Most people when looking at an image couldn’t tell you which phone it came from, and wouldn’t care if it were more saturated, or pixilated or any of the other photo enthusiast terms.
They also don’t care about ui consistency, as long as its obvious how to do a specific task.
Yeah Facebook is a different UI than Twitter. And Apples have to be eaten differently than oranges, And planes have different controls than bicycles. Sometimes different is better, or at least not impossible or annoying to most people.
I agree with the assessment. I’d take the iPhone 8 any day over the X. Same SOC with Touch ID.
To properly asses a product you’d need at least a month on all use cases and with daily and weekly observations.
First world problems?
Who cares?
It amazes me how we just nod and play the games these companies want us to…
This reminds me of the post-golden hifi review days. Several of those magazines, love them or hate them, would review equipment for at least a full month (this was the bare minimum) before writing a review. Several of them would describe the months they spent reviewing and comparing the item under review to their reference systems and other gear.
Of course, the pace of audio innovation was much slower than computers and phones are today.
“… at least the last year, and especially over the last three months, it has struggled mightily to perform simple tasks…”
That instantly reminded me of short animated movie called iDiots:
https://www.youtube.com/watch?v=InEnWwzCxcw
I can’t wait for this era of cheap credit we live in to end, so people will not be rushing to replace existing devices so often just to have the latest shiny, a shiny that is more often than not bought with debt (either credit card debt or a carrier “subsidy”), because it encourages a throwaway culture, and it can’t be good for the environment or for people who are not rich and choose to manage their money more responsibly.
It doesn’t happen only with electronic devices. People are getting rid of cars after 7 years. Gone are the days a car would be kept for 10-15 years or so.
Back on the front of electronic devices, the pinnacle of waste are costly electronic trinkets like smartwatches that barely last the day and smart home gadgetry that gets obsoleted by the parent company on a whim.
Just make it stop already. No more phones or other stuff that is designed to be thrown away after X number of single digit years.
Take cheap credit from people and they will start spending more responsibly, but you can’t do that without sparking a revolution.
Edited 2017-11-09 13:59 UTC
I use my phone more then any other device I own. I buy the highest end iPhone every year to make that experience as optimal as it can be and I consider it money well spent.
K
More power to you.
The thing is that most people don’t save near enough to be able to afford the latest shiny every year (latest iPhone, Galaxy S, you name it), and to also afford a new car every 7 years and to also deal with an emergency spend. They just pile the cost of these on their existing debt and then complain that their phone bills are high and that the bank is hounding them and that their car is sucking their wallet dry.
Throwaway culture will never be good for the majority of people, even if it’s good for you.
Edited 2017-11-10 10:59 UTC
“This is why, back when I still did reviews for OSNews, I had my own rule of using a product for at least four weeks before publishing a review. This gave me enough time to get over this initial phase, and made sure I had a more levelheaded look at the whole thing. We don’t do many reviews anymore – I have to buy everything myself, and I’m not rich – so it’s not an issue at this point, but even if companies were to approach us today for reviews, I would still ask for that four week period, and if they were to object – sorry, but no review. ”
Can I tempt you to do a review of your purchase of the Apple iPad Pro please? I can’t grasp why people spend so much money on hardware that seems to be unable to replace everything a cheaper priced laptop would do but I would love to change my opinion…
Not gonna happen. I’m not going to spend weeks writing a review only for people to yell at me how their 10 kg ThinkPad with racing stripes running MULTICS can do some arcane thing the iPad can’t, all the while discounting all the things the iPad does that their ThinkPad can’t.
Such a review would serve nobody. The iPad vs. traditional laptop debate is toxic AF right now, and I’m not going to waste weeks of work on it.
Sorry.
Grant, I’ll stick to my racing stripes, OOPS, I mean MULTICS, oops, Thinkpad, ok maybe not
So you know the iPad is crap, you’re admitting the iPad is crap, and you don’t want to make a review on it becuase it will reveal juts how much you spent on a device that doesn’t do all the tasks demanded of such a high-priced device
Thanks for proving my point.
So turn off comments under such review.
(a ~comparison with some decent Android tablet would be also nice…)
I have an iPhone X and I find that the reviews were generally accurate.
I do always find that some reviews are hyperbolic. Ultimately your buying a smart phone. Arguably it’s the best ever phone, but is that a surprise in an age of rapid technical innovaation?
I agree with Thom. The race to publish a ‘review’ means that the content is less accurate but it isn’t about the content but the clicks it generates.
For some bits of kit even a month of daily use isn’t enough. These bits of kit are so complex that there isn’t the time or opportunity to get to know the thing properly and thus form an opinion based upon experiences rather than a fact sheet.
I have a DSLR Camera that I bought more than a year ago. I’ve taken more than 5,000 shots with it yet there are parts of its functionality that I’ve yet to go near at the moment.
Many car magazines have long term reviews. These are far more value to me than those done from a few drives during the launch of the car in some nice part of the world.
More long term reviews please.
In my opinion information about how long journalist tested a device would be appreciated.
Here is a shocker: At the moment it is only possible to write a review based on limited experience. Every reviewer was specifically mentioning that they had limited experience with the device
A week, a month, and a year from now they can write another review, based on more experiences and they can update their previous review or at least link to it.
There is no reason not to write a review now if you are going to write another one next month.
However there is a huge reason (clicks/view/moneys) to write that review now, whether you are going to write another one later or not
I’m still rocking an iPhone 5S as my daily driver. It still feels fast to me, and I have it on the latest OS. Battery lasts about all day depending on my use, and it’s the original. The screen is starting to separate, and I could replace it for cheap.
I also use my nexus 4 often (at least 3x a week). In fact, it’s with me at work today for some serial stuff I need to do. I replaced the battery in this about 2 months ago. It’s not rocking Android official, since they stopped with new OSes for this long ago, so it’s running LineageOS N and it flies compared to the original OS release on this thing.
I seem to have no issue with older phones. There isn’t much the new phones offer me that I can justify spending a ton of money on (let alone drop $1k on). Heck, I built my beautiful motorcycle for less than $1k! There is seriously nothing wrong with these older phones. In fact, the ONLY thing right now that I’m considering a new phone for is the better cameras. Problem is, they are all so much larger than my iPhone 5S, they just feel huge.
As a side note, winter is approaching and that’s when I pull out my webOS gear to start toying with. I love my Veer and wish it had more horses and a better camera.
Everyone I know who used Windows Mobile switched to Android but the reviews of Windows Phone handsets were actually pretty good.
What was going on? The embargo effect, the fact that reviewers were sent an evaluation unit and the fact that almost all the reviewers were loyal iPhone users.
So you had a bunch of reviews including the phrase ‘buttery smooth UI’, which presumably came from a press release.
So you end up with a product that almost no one who used Windows Mobile actually bought because it didn’t run their old applications. Still the iPhone using Tech Journalist crowd obliged the PR people with rather good reviews. Windows Phone stumbled on for a few years until Microsoft pulled the plugs.