[RANT] How to make sure your awards/reviews are totally meaningless

[Massman]

I came across one of these guys' review and decided to check out their website. Rather quickly I realised this site seems to hand out awards and high-point scores on a regular basis. In fact ... they do it in every single review.

 

I'm done with this website. Their point-scoring metholody is pretty much useless (giving almost everything 9/10 or higher) and so area their awards. I sincerely hope no hardware vendors are bragging with those awards ... they just give them away for free it seems. And although there supposed to be a system for the awards, they seem to be handed out almost randomly. Some products have the exact same score, but still receive different awards!

 

An overview. Just picked the last 5 reviews in the categories.

 

Processors:

 

- AMD Phenom II X6 1100T Black Edition (9/10, silver award)

- Intel Core i7-950 (10/10, gold award)

- Intel Core i7-2600K Processor (10/10, gold and performance award)

- Intel Core i5-2500K Processor (10/10, gold award)

- AMD Phenom II X4 975 Black Edition (9/10, silver award)

- Phenom 2 X4 980 Black Edition (9/10, recommended)

- AMD A6-3650 Accelerated Processor (8/10, value award)

 

Mainboards:

 

- Intel DP67BG Motherboard (9/10, performance award)

- Intel DH67BL Motherboard (9/10, silver award)

- ECS A890GXM-A2 Black Extreme (9/10, value award)

- Sapphire Pure Platinum A75 (9/10, recommended award)

- Sapphire Pure Platinum H67 (9/10, recommended award)

- ASRock Fatal1ty P67 Performance (9/10, performance award)

- ASRock Z68 Pro3 (9/10, value award)

- 990FXA-UD7 (10/10, gold award)

- Gigabyte A75-UD4H (9/10, recommended award)

- Maximus IV Gene-Z (10/10, gold award)

- Fatal1ty Z68 Professional Gen3 (10/10, gold and performance award)

 

Power Supplies:

 

- Silencer 760W (10/10, gold award)

- OCZ ZX Series 850W (9/10, silver award)

- Corsair GS800 (9/10, silver award)

- High Current Gamer 900W (9/10, value award)

- Corsair HX1050 (9/10, performance award)

 

Graphics cards:

 

- Sapphire 6950 DiRT 3 Edition (10/10, gold and value award)

- ASUS Matrix GTX 580 (10/10, diamond award)

- Powercolor Radeon 6950 PCS+ Vortex 2 (10/10, gold award)

- HIS Radeon 6970 IceQ Mix (9/10, recommended award)

- NVIDIA's GTX 580M (9/10, performance award)

 

Memory:

 

- Kingston HyperX DDR3-2400 (10/10, gold and performance award)

- Kingston HyperX DDR3-1600 LoVo (10/10, gold award)

- Corsair Vengeance 12GB (10/10, gold and value award)

- Corsair Vengeance DDR3 Kit (9/10, value award)

- Crucial Ballistix Smart Tracer DDR3 Kit (9/10, recommended award)

- G.Skill Ripjaws DDR3 Kit (9/10, recommended award)

- Kingston HyperX t1 DDR3 Kit (9/10, recommended award)

- Patriot Viper Xtreme DDR3 Kit (9/10, performane award)

 

I hate this kind of 'give-us-hardware-and-we-give-you-award' websites.

 

Just look at this: Fatality Z68 on the left and ROG Gene Z68 on the right.

 

 

Same scores, but the ASUS board does not get the performance award? Also, the Value score for both boards is the same although the Gene board is significantly cheaper!

 

Gigabyte 990FX - exactly the same score. That's three mainboard reviews in a row with the same evaluation numbers ...

 

 

Seems it's also allowed to round up or down in whatever way you want:

 

 

- 10+9+8+8 = 35 / 4 = 8.75 => Overall 8

- 10+10+8+10 = 38 / 4 = 9.5 => Overall 10

Edited July 19, 2013 by Massman

[Christian Ney]

, need to give them my benchtable and my xeon rotatory cooling system to review. Go for 10/10

[K404]

Send them Leeghoofd and they can do a review of him.... see if the 9/10, recommended awards continue

 

:D

[Leeghoofd]

Leeghoofd is new "leeghoofd biased" now. As I have to wait for boards to come through via some of these "A" class websites. So getting boards late, missing out on hits etc...

 

Honesty is not appreciated

 

Some vendors don't even want to mention my review on their site as it's got some negative pointers in it... sigh....

 

I must be a prick in the real and virtual world ( plz no comments from massman and Christian Ney on that plz )

Edited August 2, 2011 by Leeghoofd

[Christian Ney]

plz no comments from massman and Christian Ney on that plz

 

copy that!

Edited August 2, 2011 by Christian Ney

[TaPaKaH]

you think someone's rivalling your beloved madshrimps on the number of handed our awards (=badges =recommendations) ?

[Hondacity]

i never liked their website lol

 

most manufacturers love them though...

[xxbassplayerxx]

i never liked their website lol

 

most manufacturers love them though...

 

I can see why!

[Luebke]

'give-us-hardware-and-we-give-you-award'

 

i bet the vendor can decide wich award he wants to recive for the test

[Leeghoofd]

Why didn't they call the website AwardHeaven instead of HardwareHeaven

[Massman]

Hahaha!

 

http://twitter.com/#!/CorsairMemory/status/98435016673075200

[Monstru]

I sincerely hope no hardware vendors are bragging with those awards

 

I bet what you want that they do brag about these awards and they do put them in their catalogues and marketing material. I do not understand how this comes as a surprise to you. Awards are a very, very ugly marketing tool which most manufacturers are desperate about, that is why we never gave awards and we never will

[Leeghoofd]

I have no issue with an award if it's well argumented Tudor... but this particular site is the perfect example of how it should not be done... but the marketing guys don't give damn... high score and an award is what makes them tick !!

[Trouffman]

Sadly not new... and sadly not the only site doing it....

 

I hate reward like that... I love badges that fit on a specific rule...

[Maxi]

OK now look me in the eye and tell me you've never glossed any aspect of a review

[IanCutress]

This is why I refuse to give scores out in reviews, and I haven't given an award out this year. Awards should be for the premium product which is perfect in every aspect, AND beat all their competitors in every aspect as well - price, performance, warranty, extras, support, comparison to other products.

 

In my eyes, 5/10 should be average, where 7/10 is an awesome score, and very rarely should anything get above 7. The problem is that people tend to give scores in the 5-10 range - so a really crap product STILL gets 5/10. What does the manufacturer have to do to get worse? Take a massive dump on it?

 

It all comes back to Kane and Lynch on Gamespot. K&L publishers spend $million on Gamespot advertising - their reviewer gives it 6/10 and gets fired because the advertising was pulled. In reality, 6/10 should be a great game.

 

For most reviews now, I just look at the score out of 10 and then subtract five, then that's the score out of five.

[knopflerbruce]

I don't think 7/10 is that awesome. Average is 5, slightly above average is 6, and above average is 7. Good is 8, very good 9, and awesome (=flawless, or at least close to it): 10. Ditto below 5.

 

Another point.... should a product get a worse score because other products are better? I can't think of many boards that deserve less than 4... maybe the 6/7-series nvidia chipset boards, but not many others. A board must have some fairly severe issues if it's not worth 4/10 or more. Today I feel most mobos are at least average, so it's not so strange that we don't see really bad scores, although there aren't as many 9's and 10's as the reviews tell us I think we set the bar too high if we want mobos to have some really good features just to get a 6 or 7. A 5/10 board should do what it's supposed to do, and nothing more as I see it... add some overclocking functionality and on board power button/couple of similar features and you have a 6. The way I see it...

[Monstru]

OK now look me in the eye and tell me you've never glossed any aspect of a review

 

Nope, never. If the product it's to crappy, the best I can do is not write about it, but that's the biggest concesion I can make.

[_mat_]

I bet what you want that they do brag about these awards and they do put them in their catalogues and marketing material. I do not understand how this comes as a surprise to you. Awards are a very, very ugly marketing tool which most manufacturers are desperate about, that is why we never gave awards and we never will

... and I guess they keep asking you to give awards every once in a while and try to convice you, that awards will bring you so much more hits to your website, because they will backlink you on several sites.

 

Scores and too many different awards are useless in my opinion because hardware has very indivual use cases. What is perfect for one setup, can be a senseless upgrade for another. So instead of letting the reader be lazy and take scores and awards for granted, they should always have to read the conclusion. We - as writers or/and editors - should take care of that! As a wise man said: "With great power, comes great responsibilty". Hardwareheaven.net obviously did not watch Spiderman ...

[sin0822]

i noticed this as well, they even scored the asrock X58 extreme 6 higher than the UD9 b/c it performed better, they even said it oced better i believe or some outrageous claim like that, and i noticed all their stuff was all high rated. When i started doing forums reviews i did points, but now I don't, i just like to write about the pros and about the cons. i am building a blog site, and i want to be as object as i can while being true, so i am working with different review techniques. I can learn a lesson for this though, thanks for pointing it out massman.

Edited August 5, 2011 by sin0822

[DrSwizz]

Awards: completely useless measurement

 

Last year I was considering buying an Asus 890G motherboard (I forget which model). I did some googling, but found very little information about the motherboard as it was a relatively new product. On the official Asus page for that particular motherboard there was 2-3 awards given by some obscure hardware websites. I decide to take a look at those sites. It turns out that none of those websites had actually reviewed the motherboard in question. On one of the websites there was a review of one motherboard with similar sounding model name, but with different hardware. On the other site(s) I found no review whatsoever of any motherboard even similar to the one I was considering buying. I never bought that Asus motherboard

[Massman]

Hardware review whose awards are meaningless - part 2

 

KitGuru - http://www.kitguru.net

 

Again pulling their latest reviews:

 

Motherboards:

 

- M4A88T-M (8.5/10, worth buying)

- Sapphire Pure Platinum A75 (8.5/10, worth buying)

- ASRock Fatal1ty Z68 Professional Series Gen 3 (9/10, must have)

- Gigabyte 990FXA-UD7 (9/10, must have)

- Crosshair V Formula (9/10, must have)

- Asus Maximus IV Gene-Z motherboard (9/10, must have)

- Z68X-UD7-B3 (9/10, must have)

 

Graphics cards:

 

- Sapphire HD 6950 Toxic Edition (?/10, must have)

- XFX HD6870 Black Edition Crossfire (9/10, must have)

- HIS HD6970 IceQ Mix Edition (8/10, worth buying)

- VTX3D HD6870 X2 (8/10, worth buying)

- Asus Matrix Platinum GTX580 (9/10, must have)

 

Processors:

 

- AMD Vision A8-3850 APU (9,5/10, must have)

 

Memory:

 

- G.Skill 8GB DDR3 1600Mhz RipjawsX (9/10, must have)

- G.Skill Sniper DDR3 (8,5/10, worth buying)

- Kingston HyperX T1 12GB (9/10, must have)

- Kingston HyperX T1 12GB (9/10, must have)

 

Power Supplies:

 

- LEPA B850W Power Supply Review (9/10, must have)

- Cougar GX 1050W power supply (8/10, worth buying)

- Be Quiet! Power Pro BQT P9 550W (8/10, worth buying)

- Corsair CX430 V2 (9/10, must have)

- Strider Plus 1000W (9/10, must have)

 

Conclusion:

 

So, if I have a bit of spare money and I want to find a piece of hardware that would be a nice addition to my system, pretty muh all reviewed gear is worth buying. Moreover, no less than 15 hardware products are 'must have'!

 

On the plus side: at least the awards are given on a consistent basis. Consistently meaningless.

[Leeghoofd]

It could also mean that al hardware is top notch lately and you have been sleeping under a rock for ages...

[DaveJN86]

I wrote this a while ago, and it appears to have gotten buried somewhere at Overclockers.com:

 

http://www.overclockers.com/overclockerscom-rates-products/

 

We want people to read our reviews; our reviewers put a serious amount of work into each and every review they write. Please read what they’ve written – don’t rely on an often quite arbitrary points, percentages, or stars system. One of the most important parts of any given review is where the pro’s and con’s are compared, and this is where you will get a good feeling for whether this is a product you want to own or not.

 

EDIT: Oops, accidental thread necro. I saw this linked on Facebook

[hokiealumnus]

Dave, thou knowest not thy own site. It's linked from the approved/meh/fail image in every review as well as from the 'Welcome to Overclockers.com' page. TheOverclocker posted about this at KPC forums a while ago; this was my response (emphasis added):

 

You're absolutely right they have become accustomed to the high scores and almost none get low scores. Part of that stands to reason - even if the manufacturer makes crappy product along with their top tier stuff, who in their right mind would send the less-than-stellar product for review?

 

FWIW, after much debate we decided to stay away from scoring altogether at Overclockers. We have three ratings, "Approved", "meh" & "FAIL". Every "award" image is linked to this page, which explains them. A component has to pretty much not overclock at all or have issues to not be approved, as Approved means "The product performs well at stock and at overclocking, for modding, etc. where relevant. It isn’t necessarily the best of its type, but it performs well enough that we could recommend it with a clear conscience."

 

If it gets a "meh" it doesn't do something it should. If it gets a "FAIL", dayum it's bad.

 

Basically, this avoids the arbitrary numbering associated that type of system. It also gives incentive for people to READ the reviews, which is the whole point after all. No skipping to the end and seeing 7/10 or 9/10. People need to read the review to get the full picture; good, bad and ugly.

 

(Heh, then Tom's Hardware moved to an eerily similar system shortly after we did, literally using the same "Approved", so apparently we're a trendsetter, heh).

 

Heh, not a necro when it's linked from today's post. We're just posting in the wrong thread.