[RANT] How to make sure your awards/reviews are totally meaningless

[Massman]

Mr. Massman sniping around

 

Not sniping, they send this to my email ...

[IanCutress]

Are all the tests

?

Edited June 18, 2013 by borandi

[der8auer]

So once a memory kit is not DOA it gets the award. Seems legit.

[IanCutress]

Even if it is DOA, get another one in as clearly DOA isn't the norm.

[Massman]

I think Ian and me are on the same line here. Great to see the site give "Best of Computex" awards ...

[Christian Ney]

2 weeks before Computex starts

[I.M.O.G.]

Fwiw, I don't follow all of tweaktowns stuff, like I have never read a memory review there previously.

 

But I do read their storage stuff, and their content within SSD reviews is really good. Chris Ramseyer writes those articles, and he is an acquaintance/friend of mine. So for that reason, I just wantedbto put it out there that writing Tweaktown off across the board may leave you missing out on what they do have that is good.

[Massman]

Actually, yeah, you are right Matt! I came across a couple of memory reviews from Chris Ramseyer and was genuinely impressed. Sorry if that post caused him any trouble.

[knopflerbruce]

I just thought of something regarding memory... the differences are very small between comparable kits. Nothing really makes one particular part number so much better (for an average guy, that is) than the competition. Small performance difference, small price difference between comparable kits, and the heatspreader design is irrelevant as long as the sticks are not too tall. I'd like to give HH credit for one thing: they don't overdo the small differences just to show that there is SOMETHING that separates one kit from another. I mean... you can't say that one kit is worth 80% and another 90% just because one costs 5% more and "doesn't look as good" as the other

 

Maybe I'm a bitch, but if they lowered the scores by 5% (except the 85% one, which should be raised, if anything), it would look quite reasonable. If the sticks run cool, don't consume much power, and run stock nicely + have a little bit of headroom for overclocking, then they deserve to be recommended. What else can we ask for?

[xxbassplayerxx]

Most recent test is TT, not HH, but the point still stands. As long as a kit runs XMP specs and isn't a horrible sample for the included IC's and advertising, it should do well. Something that should score badly would be Corsair's 1866C9 Vengeance "overclocking" kit that has Nanya chips in it...

[Massman]

Are most of the kits "must have" or "good choice"? Since when is a kit that works (ie: XMP ratings) a must-have buy? Why is it oh-so great that a kit can do exactly what it says it can do?

 

It is impressive when a kit can do DDR3-3000, sure. But ... DDR3-1866C9? Really, we should be happy it can do the advertised spec?

[knopflerbruce]

Are most of the kits "must have" or "good choice"? Since when is a kit that works (ie: XMP ratings) a must-have buy? Why is it oh-so great that a kit can do exactly what it says it can do?

 

It is impressive when a kit can do DDR3-3000, sure. But ... DDR3-1866C9? Really, we should be happy it can do the advertised spec?

 

Well, is the overclocking potential what separates a 60% and a 90% score? Of course not. I'd say overclocking is worth 10%, 15% tops if you're really into it.

 

"Must have" is too much, but "good choice" - that's quite reasonable in these cases, I bet. A "good choice" product - IMO - is a product that does it's job and can do a bit more than the bare minimum we can expect. 1866C9 -> 2133C10 should be enough here. A kit that can go from 1866C9 to 2666 sounds like a contender for a must-have, it matches what you can expect to run on air/water, and surely it matches the specs of much more expensive kits, even if the specs are very value-like these days.

 

And a quick reminder: most P67/Z77 motherboards were barely overclockable, maybe those ratings were a notch or two too high. I mean... 100bclk... surely we should expect at least 150MHz, no? Maybe 175 to get the top ratings.

[Massman]

A good product is one that will not make you unhappy after purchase. Most memory kits are "good choices" because it simply doesn't matter what you put in your PC. At least not in terms of performance; perhaps it matters with looks. A "Must Have" is way more than that as it implies that you should take your money and go buy it right now. Must is the key word.

 

If you give a 90% and no award, I could live with that. Although you don't have much room for rating crazy good overclocking kits like Flare or GTX2 significantly higher. But none of those memory kits I linked to are "Must Have" and the readers of those articles are tricked into believing they are. Anyway, to each publication their own. Everyone has to make decisions on how to ensure future business ...

[knopflerbruce]

A good product is one that will not make you unhappy after purchase. Most memory kits are "good choices" because it simply doesn't matter what you put in your PC. At least not in terms of performance; perhaps it matters with looks. A "Must Have" is way more than that as it implies that you should take your money and go buy it right now. Must is the key word.

 

If you give a 90% and no award, I could live with that. Although you don't have much room for rating crazy good overclocking kits like Flare or GTX2 significantly higher. But none of those memory kits I linked to are "Must Have" and the readers of those articles are tricked into believing they are. Anyway, to each publication their own. Everyone has to make decisions on how to ensure future business ...

 

Agreed!

 

...except I'm not sure if I agree to the GTX2 being a "must have" given the lack of overclockability. They really cost alot, and were hard to push much more than 10% above spec on air for 24/7-stability, from what I recall.

[Leeghoofd]

Bad comparison there Knopf... GTX2 are/were awesome, even years later they made the Sandy platform shine... it's not always about increase in clock frequency as we tend to see lately. Still wondering how efficient them +3200Mhz results are ? How about tighter timings?

[M.Beier]

Im with Alby on this one Knut, GTX2 could do something.... Mine sure did more than 10% above on air, I just never used them for daily ram, because they were too overkill and unique, saved them for benching....

 

Also, Im on same page as Massman, feels really akward! But I sure have noticed tweaktown slacking on quality, great examples, no doubt, you can find same in graphics cards..... Its really sad

[knopflerbruce]

Bad comparison there Knopf... GTX2 are/were awesome, even years later they made the Sandy platform shine... it's not always about increase in clock frequency as we tend to see lately. Still wondering how efficient them +3200Mhz results are ? How about tighter timings?

 

But if you "must" include overclocking potential in the "score equation", it IS a drawback that they cannot be pushed a fair bit above stock, no? I'm fully aware of the quality of the GTX2's, which was exactly my point - if a huge part of the rating is based on a ~50% expected increase in memory performance then good kits are bound to get relatively low ratings.

 

I'm not doing memory reviews (maybe that's a good thing!), but I find it difficult to ask for insane clocks on low end sticks, and then lower the expectations for high end kits - effectively making it difficult to compare awards or ratings - even though it is unreasonable to expect the highest bins to exceed world records on stock air. The only reasonable way seems to be to lower the expectations for value/midrange kit overclockabilities.

[websmile]

A reviewer does not rate only the percentage of overclock as main criteria, he rates overall performance, stability, price, optic and so on. GTX2 were best of brand, apart from a few lucky draws at other manufacturers, there were no better overall performance Hypers than these. What makes me agree on Pieters rant is that you cannot give an average rating of 90% to all products you review, this leaves no room for seperating the extraordinary from the good, the good from mediocre and the mediocre from the bad and leaves no headroom for extraordinary performing hardware. If mediocre stuff already gets 90%+, where is the room for best of brand mems that give you outstanding performance? You can give different awards if you want to rate special features, budget award for example for lowend good priced mems that give reasonable performance and good bang per buck, or enthusiast award for extremely high priced mems meant to break benchmark records and therefore offering best overall clocks- but rating all kits that you review at 90% and more doesn´t help make decisions for people reading the reviews and it doesn´t help at all to make differences between kits clear

[Leeghoofd]

You have to keep in mind that some sticks will not go as high as you want them too... if you can reach them accepted speeds then it's a good OCing kit even IF the potential is just 10% from the high stock speeds...

 

If a low end kit can do the same speeds then it's an amasing Ocing one...

 

Secondly it's not all about raw MHz as you know, I rather run 2400C9 with tight subs then 2600C11 loose as hell

 

 

All stuff that HH and TT review get's highly rated... ow I spot a similarity in the abbreviations... double trouble There's no more bad hardware out there, which is a good thing not ?

Edited June 23, 2013 by Leeghoofd

[knopflerbruce]

A reviewer does not rate only the percentage of overclock as main criteria, he rates overall performance, stability, price, optic and so on. GTX2 were best of brand, apart from a few lucky draws at other manufacturers, there were no better overall performance Hypers than these. What makes me agree on Pieters rant is that you cannot give an average rating of 90% to all products you review, this leaves no room for seperating the extraordinary from the good, the good from mediocre and the mediocre from the bad and leaves no headroom for extraordinary performing hardware. If mediocre stuff already gets 90%+, where is the room for best of brand mems that give you outstanding performance? You can give different awards if you want to rate special features, budget award for example for lowend good priced mems that give reasonable performance and good bang per buck, or enthusiast award for extremely high priced mems meant to break benchmark records and therefore offering best overall clocks- but rating all kits that you review at 90% and more doesn´t help make decisions for people reading the reviews and it doesn´t help at all to make differences between kits clear

 

If we talk about motherboards and more "complicated" hardware, absolutely - a motherboard has many variables - BIOS, layout, choice of controllers, cooling etc. A set of sticks got speed/timings (= stock performance), design (looks are very subjective, and almost any heatspreader is adequate in terms of cooling today, but OK...), price and overclockability (I guess that's what you mean by stability?). GTX2 got great specs, very fast even when not overclocked. Though... what are the FPS gains in games compared to a much cheaper 2000 cl8 kit? 2-3% tops? Other real world applications? Basically, GTX2 was never a good buy for a 24/7 setup. A review meant for a regular consumer would normally conclude with the fact that the performance increase is so low that it's not worth the extra cost at all, and let the rating suffer from that. A review targeted to overclockers would probably look at things from a different perspective. Enthusiast award sounds good, as an enthusiast doesn't care that much about the cost.

 

A top SCORE product should give bang for the bucks, and not only on the spec sheet.

 

As for mediocre stuff... again, what are the real world performance differences between a mediocre kit and an "outstanding" kit? Doubtfully enough alone to use those words in a review targeted at regulars. I guess price is what could create some differences here, as a low bang for buck ratio makes a kit not that great of a purchase. For a superpi 32m addict... things are different.

 

You can put the "base line" wherever you want. You put it on 80% instead of 90, then you'll most likely never get above 90% 80% of the kits would be between 75% and 85% because of what I've written above - the performance differences are too small, and for most kits the cost difference is the same. Maybe the old 1600 7-8-7 kits with good chips could be good examples of 90+ kits Maybe the 6-8-6 kits, too, but I have a feeling they were a bit too pricey in the beginning.

[websmile]

There were times memory freq was important for overall overclock, seeing pre grandmother oc times with locked multipliers and fsb/HT having to be overclocked - today imho you simply rate the product itself, quality of craftmansship, price-performance ratio, undervolting potential and overclocking, overall max frequency and other options like warranty, goodies and so on. At reviews I made, I only rated the kits themselves on several criterias, and believe me that there was a wide range of overall ratings in the end.

If it makes sense for daily gamer to buy expensive highend stuff like 2933 kits today or GTX2 in the past was always discussed, seeing the gain at FPS you mentioned, better VGA or CPU accelerates more, so I agree with you. But the differences at memory kits itself it is what has to be rated at such reviews, and every other hardware product. The percentage rating is not all that counts but what people see first together with awards, because of this I like clear differences there and not 90% for every product.

This is no easy subject to discuss, reviews need experience and knowledge about the stuff you have to judge. I had 100s of mem kits I bought myself, from values up to highend, which I all tested and which gave me my "baseline". If you had for example several 37 Euro Samsung value kits that did 2666 cas10 or 11 and went up to 2800, it is not easy to give 2400 specced 90 euro kit that does the same or less 90%.... Hope I made my point clear, I tend to lose grip on discussions like these , of course every reviewer has his own style to measure things and his own experiences, but imho he has the obligation to show differences that are there, and also show these by making noticeable differences at rating - just my 2 cents

[Massman]

http://www.techpowerup.com/reviews/?category=Memory&manufacturer=&pp=25&order=date

 

I love reading TPU so I don't really want to go look at the awards. But the ratings are suspiciously high and clicking 8 random reviews yielded in 8 awards given to the reviewed product.

 

:( :(

[thebanik]

http://www.techpowerup.com/reviews/?category=Memory&manufacturer=&pp=25&order=date

 

I love reading TPU so I don't really want to go look at the awards. But the ratings are suspiciously high and clicking 8 random reviews yielded in 8 awards given to the reviewed product.

 

:( :(

 

Personally I do not mind the awards and do not overly place an importance on them while reading reviews. Being a reviewer myself (ofcourse of a more regional website) I feel if the Cons of a product have been highlighted properly then Rating/Award is useless (not so much for the reviewer, because he can remain in the good books of the manufacturer without cheating the readers). This is ofcourse for products which atleast manage to do what they set out to claim at a decent price.

 

There are exceptions ofcourse where I personally have sent the results to manufacturer, and there have been cases where they have opted out of getting the Review Published. This is like a courtesy which reviewer can give to the manufacturer for timely delivery of review samples.

[Massman]

Depends on what the award says. "Recommended", "must have", and all other buying recommendations shouldn't be given to every single product reviewed.

 

I like THG's "Smart Buy".

 

[sin0822]

Depends on what the award says. "Recommended", "must have", and all other buying recommendations shouldn't be given to every single product reviewed.

 

I like THG's "Smart Buy".

 

 

come on massman you and I both know when tom's and anand's and maybe TPU/TT/HH give an award it shows up on a box somewhere, it becomes good advertising for both the site and the manufacturer. Manufacturer says look they approve, and the site probably gets more ads. I mean even GC-Extreme has TPU review stating on the back! haha

 

We should just assume big sites are mostly BS, and sometimes the big sites writer's aren't in control of the award or the score, or maybe fully aware, maybe sometimes the editor puts up an award the writer didn't approve of, but is good for site business. But seriously the lack of knowledge is scary, If I goto Tom's motherboard review and ask him to point out which is the high-side MOSFET and which is the low side mosfet in the VRM I bet you $1000 he would get it wrong just because he has no idea how the VRM functions and what to look for. Id even give him a DMM and allow him to back trace the connections and I bet he'd get it wrong. Then the issue is if you can't figure out the components of the VRM, the limit parts of the VRM on a motherboard, what f*cking business do you have reviewing the product? what are you going to benchmark a f*cking motherboard? Really who gives a flying f*ck? A BIOS update can change EVERYTHING in terms of performance, the only thing a motherboard reviewer needs to report is if there is a performance deficiency that needs to be corrected, I could care less if you went over the hardware in 1 page, and then went over the benchmarks of the board in the next 11 pages like I saw one random site did, i was like WTF dude. Quality does show, and it doesn't show in 1 year, it shows in 3 yeas, we throw away or sell our boards by then(most of us) but most people don't.

 

Dave is a good motherboard reviewer, one of the few I actually read his reviews sometimes to see what he says. i don't like how he tests RMAA(he doesn't disable software effects and match bit rates like I do) but overall he does a decent job.

 

IMO you want to change the way the review industry is then you need to have the review sites and the companies be totally separate entities. Period, of curse that will never happen, so you have forums with normal people who buy the products giving feedback, but even that isn't good b/c most people only post when there are issues and it is in ratio with the amount of product sold.

Edited July 19, 2013 by sin0822