but James, in the professional world when benchmarking is done by professionals (like Toms Hardware) benchmarks can be extremely telling. why? because they do benchmarking scientifically with completely controlled environment/variables, give exact machine specs, use multiple machine comparisons, discuss results in 20-page explanations. i agree that the benchmarks shown on these amateur UMPC websites is absolutely laughable & accomplishes nothing but misinforming the public. they toss up the CPU type & a graphic bar and say “look, Intels better!”. but…

what are the full specs of the winning machine?
what are the specs of the other machines?
what OS?
what were the environment settings?
what other applications were running?
what is the power consumption?
on & on…

personally, i really get annoyed at these ghz-to-ghz CPU comparisons that DONT list power consumption. design choices make ghz comparisons almost useless. all that matters is actual performance & at how many watts.

What gets me is that every so often people grumble about benchmarks, but nobody ever seems to question these awful unboxing videos and photo sets that are so popular. At least a benchmark gives you SOME indication of a system’s relative performance. An unboxing is all about a person who has not used something documenting themselves not using it. That’s oh-so-better than any old benchmark and really helps me make informed purchases.

Philosophically I agree with you, but practically I’m desperate to see more benchmarking of different laptops, particularly wrt graphics performance.

One of the reasons we’re not seeing any Tablets or “Ultraportables” with decent graphics cards is no-one is reporting on a regular basis how underpowered the current options are. Benchmarking can at least help with this.

Should it matter? Possibly not, but my own experience is that Vista with one of those ageing Intel chipsets isn’t all fun. Now you are free to disagree, but at the very least we need the information to help us select a machine based on our needs.

http://translate.google.com/translate?u=http%3A%2F%2Fwww.eeepcnews.de%2F2008%2F04%2F18%2Fintel-atom-benchmarks-via-isaiah-vergleich%2F&langpair=de%7Cen&hl=en&ie=UTF8

The truth will out anyway.

http://www.liliputing.com/2008/04/via-isaiah-v-intel-atom-benchmarks.html

While benchmarks might not tell you how a computer will operate in a real world situation, they do offer an important view of how powerful the machine is. In a desktop world, that isn’t much of a problem because most modern desktop processors will handle any app that a home user would come across. However when people want to run Photoshop or play games on their UMPC, having an idea of how powerful the system is becomes very important.

I agree with the general opinion that James takes. But I will also say that I don’t frequent jkOnTheRun for benchmarks. There are other websites that do that and do that well. I come to jkOnTheRun to get a sense of the overall user experience. For the most part, James handles his equipment for an extended period of time. I look forward to the long term experiences that he talks about. These focal points can be far different from what we read in a one-shot review that looks at how A compares to B. Even knowing what James is looking at matters because it coincides so often with what I am looking at. Constructing a mobile life is not an easy task. Even the discussions on carry bags is useful.

But I’ll never disregard the value of benchmarks. They do matter. The best websites that detail benchmarks are consistent in their application of the benchmarks and they give a good general comparison. But the beauty of the Internet is that I’m not limited to one source of information. Long live JKOTR!

Just an idea:

Maybe if you feel like a particular benchmark is overstating or understating real-world performance in a given area, a quick video of the device performing some sample task is great to show how fast/slow things really are.

I remember when Tom’s Hardware posted a video of side-by-side P4 3.2 GHz and 3.0 GHz HT systems booting XP, starting Photoshop, then starting Word, then loading the same file. Seeing the applications take different amounts of time to launch was a great demonstration of the dramatic speed increase offered by hyperthreading technology, and was helpful as I was looking to purchase a desktop PC at the time.

While I mostly agree with you James, there is one area that I’m forced to disagree on: battery life. I fully agree that synthetic benchmarks rarely, if ever, provide any indication of how a system will perform in the real world.

That said, I have to disagree with you when it comes to battery life benchmarks. Why? It’s the same mantra that you guys repeat over and over around here: “everyone’s usage model is different.” What that means here is that your “real world” battery life for a given computer will be different from mine, because we use our computers differently. The advantage of something like a Battery Eater benchmark is that it allows you to say “I get __ hours of use in the real world, but regardless of how you use the system it won’t be worse than ___.” That means that when I’m considering a new computer, I can say that I’ll expect to get about the battery life that you do, because our usage model is probably pretty similar, but also that if I want to do heavy computing tasks (which I occasionally do while mobile) or I want to watch a movie on an airplane (if I didn’t have time to put it on the Zune), then I also know the battery won’t die halfway through my movie because I know the worst-case battery life. So, when I’m reading reviews, I find it greatly advantageous to have both the subjective “real world” battery life and the “worst case” Battery Eater benchmark so I can get an accurate picture of what to expect from a given system.

Performance benchmarks help to balance and support a reviewer’s opinions by providing quantifiable measures from standardized tests. I like to look at the individual component scores (CPU, graphics, memory, HDD) to understand a machine’s relative strengths and weaknesses. This provides quantifiable data to either support or spur discussion regarding the qualitative perceptions presented in a review.

The “grain of salt” philosophy should be applied to reviews in general — whether it’s computers, cars, restaurants, wine, etc. Consumers should first try to understand their own needs/preferences and then look to a variety of trusted sources to determine whether a product meets those requirements.

Aside: Boot time and standby recovery time are important benchmarks when using an ultra-portable device for PDA functions. I’ve had to wait up to 3 minutes for my mobile PC to wake up from standby mode just so I could check my calendar and book an appointment. IMO device makers and Microsoft could do a lot more to improve the ‘real world’ responsiveness of portable computers.

But I also think that a good performed and understated benchmark will be an addition to review. Especially because it is objective & repeatable. A writers review often reflects only the subjective view/experience of himself. In that case you can only agree or disagree with him. That also applies for the user.

Btw, improving systems is a bit hard without measuring something.

>>>That is the reason why I never run nor publish benchmarks in my system reviews, to the chagrin of some.

Bah! Just send the Mini Note to me and I’ll do the damned benchmarks and post them.

Of course, you can expect the Mini Note to get lost in transit on the way back. But you can’t have everything — even though you surely try! Heh-heh.

Bravo. Ultimately, what’s important is real-world performance. Lacking first hand experience, benchmarks may serve as a proxy for comparison purposes. However, we shouldn’t fool ourselves into thinking that benchmarks can replace first hand experience, or that benchmarks are, somehow, better than first hand experience.