How do you rate a HW product?

[Massman]

Hey guys,

 

Very recently, my mind was blown when I realized a lot of people draw conclusions based on the overclocking results of one or two people. Just an example: reaching high spots in the 3DMark Vantage benchmark equals "very good mainboard". Reaching the highest clock in 3DMark03 means "the best videocard".

 

Obviously, things aren't as simple as that. The results of one person may be an indication of the overclocking capabilities, it doesn't mean that it's the best per definition. The question is now ... how do you compare hardware in terms of overclockability? What do you take into account for making a hardware ranking?

 

Once the questions are answered, we'll try to create some sort of 'hw ranking' based on the feedback. We've already been experimenting with this with the R5870 cards and it looks okay (apart from the chart size, sorry for that)

 

 

Since algoritms for mainboard, memory, videocard and cpu will be different, feel free to split up.

[Massman]

Do note: we are focussing on OVERCLOCKING and PERFORMANCE here. Not daily usage and power saving

[Antinomy]

The motherboard is the hardest thing.

It's obviously the highest Bus clock it can achieve, the overall frequency, the memory frequency and timings in Pi 32M/Pifast and efficiency in Pi 1M/32M. It's very tough to calculate it though.

The CPU is divided by 3D and 2D. 2D is divided at least by multithread and singlethread applications. 3D can be divided too. Both overclocking and frequency are important as well as CPU score for example. It is possible to make charts for each application were hardware can be compared. But an overall chart... too complex for the recent info given in submissions.

[Massman]

For the VGA chart, we've used the average clock of the best submission per user per card. So for instance if Hicookie got 1350, 1400 and 1500MHz submissions, only the 1500MHz submission will be taken into account. We've also not taken into account the multi-gpu submissions ... only the single-GPU. I think that's good enough to compare the overclocking capabilities of VGA overclocking series?

 

Maybe we also have to factor in the spread of the achieved clock frequencies.

 

For CPUs, I think a performance chart can be build from the raw results we have in our database already ... shouldn't be a big issue since there are no custom CPUs made.

 

For Memory it's a bit more tricky ... applying same concept as with the VGA charts should be possible, though.

 

For mainboards it's véry difficult. Still pondering on how to make a valuable chart for this part.

[Antinomy]

MultiGPU shouldn't be taken in account - there's a number of factors that have a greater influence than in single-GPU. For the overclocking - yes, I agree. But what about performance?

 

This is the main bitch for the MoBo - is the best MoBo selected by BCLK, it's memory overclocking capabilities (both timings and frequency) or the CPU frequency? Or is it CPU performance? Or memory performance? Not taking the PCI-E overclockability, for example and CB/CBB.

 

I suggest to start from the easiest - when glitches are wiped out, you'll have experience how to handle several factors between each other, it'll be more easy.

 

I'll think of this if I get what you want. ATM I don't see all of the picture - the usage and what's it aimed at or whom.

[TaPaKaH]

memory chart is totally unnecessary - what you'll get is just "how many volts do ppl put on average to product X"

motherboard rank is near impossible to judge - unless you can compare results from same user, same cpu, same cooling

[Massman]

Another approach, instead of looking at the highest possible scores, is looking at 'how easy it is to reach a decent result'.

 

For instance, for the memory: instead of using the memory benchmarks (MaxxMem and Memory Clock), we use the memory frequencies indicated in 2D and 3D applications. The underlying thought here is: the easier to reach X frequency/timing, the more likely it's going to be set in non-memory dependant benchmarks.

 

Example: GTX2 can reach 2000CL7-8-7 very easily, so I use this in 3D benching. Valueram can only reach 1600CL8 each, so I use this for 3D benching. Maybe the valueram can reach 1800CL8 with a lot of tweaking ... but I will not run it 3D because it's not so important.

 

Of course this can be split up in 2D/3D/Suicide or so.

[knopflerbruce]

I'd say high bclk/fsb/htt/whatever on a motherboard would be a good chart to display. Even though other things are important, in most cases (locked CPUs...), high bclk is what matters the most.

 

@Massman, I don't quite get the last post - what sort of ranks are you thinking about?

[Massman]

The idea is to have charts available that are a good representation of the overclocking and performance capabilities of hardware. My last post was an attempt to make a memory overclocking chart based on the principle: "the easier to reach X frequency/timing, the more likely it's going to be used".

 

- 2500CL7 is not easy to reach => not going to be used a lot

- 2000CL9 is very easy to reach => going to be used a lot

 

The chart would then be something like:

 

1. Adata XPG+ --- 2000MHz --- 10 users

2. Adata XPG --- 1900MHz --- 5 users

3. Corsair GTX2 --- 1800MHz --- 10 users

 

By using non-memory benchmarks, we can filter out the overclocking results that are virtually unstable (eg: maximum memory clock). It's not because one person ran 3V through the memory kit at -196°C and got super results that an 'overclocking capabilities' ranking should be ranking those 1st.

 

We could also restrict memory cooling to 'air' only, but then we might have to deal with faulty data (people forgot to enter LN2).

[Antinomy]

Interesting catch. Though not everybody writes what voltage he's running at. In case of TaPaKaH it's important, he's crazy on this

[Massman]

I quickly tried a query to show what I mean

 

Note: a lot of incorrect data at the moment. Needs to add some filters.

 

SELECT manufacturer.name, memoryproductgroup.label, AVG(result.memClock), COUNT(result_id)
FROM result 
JOIN memoryproductgroup ON (memoryproductgroup.memproductgroup_id = result.mem_productgroup_id) 
JOIN mem_productgroup_model ON (mem_productgroup_model.memproductgroup_id = memoryproductgroup.memproductgroup_id)
JOIN manufacturer ON (memoryproductgroup.manufacturer_id = manufacturer.manufacturer_id) 
WHERE (result.memcooling_id<4 AND memClock<1250 AND memClock>600 AND application_id!=19 AND application_id!=21 AND mem_productgroup_model.mem_id=1)
GROUP BY memoryproductgroup.memproductgroup_id 
HAVING COUNT(result_id)>50
ORDER BY AVG(result.memClock) DESC;

 

 

Or tCl=7 results

 

SELECT manufacturer.name, memoryproductgroup.label, AVG(result.memClock), COUNT(result_id)
FROM result 
JOIN memoryproductgroup ON (memoryproductgroup.memproductgroup_id = result.mem_productgroup_id) 
JOIN mem_productgroup_model ON (mem_productgroup_model.memproductgroup_id = memoryproductgroup.memproductgroup_id)
JOIN manufacturer ON (memoryproductgroup.manufacturer_id = manufacturer.manufacturer_id) 
WHERE (result.memcooling_id<4 AND memClock<1250 AND memClock>600 AND application_id!=19 AND application_id!=21 AND mem_productgroup_model.mem_id=1 AND result.memTCas=7 )
GROUP BY memoryproductgroup.memproductgroup_id 
HAVING COUNT(result_id)>50
ORDER BY AVG(result.memClock) DESC
LIMIT 25;

 

[TaPaKaH]

the thing about memory - people don't max it out very often. most 2d / 3d guys run at safe clocks (since it's not about the mems these days) so that memory doesn't cause any problems

[Massman]

the thing about memory - people don't max it out very often. most 2d / 3d guys run at safe clocks (since it's not about the mems these days) so that memory doesn't cause any problems

 

Yes. That's the principle I'm using to make these ranks. The easier it is to reach frequency X, the more likely people will use it as safe clock. The higher the safe clock, the better the product.

 

It's not perfect, but it's something to start from. I need more input on how you guys rate memory to make better charts. Besides price ...

 

Here's another attempt, taking into account the average Cas Latency used and average frequency to calculate latency time. Latency = (1000 x tCl) / MHz

 

SELECT manufacturer.name, memoryproductgroup.label, AVG(result.memClock), AVG(result.memTCas), COUNT(result_id)
FROM result 
JOIN memoryproductgroup ON (memoryproductgroup.memproductgroup_id = result.mem_productgroup_id) 
JOIN mem_productgroup_model ON (mem_productgroup_model.memproductgroup_id = memoryproductgroup.memproductgroup_id)
JOIN manufacturer ON (memoryproductgroup.manufacturer_id = manufacturer.manufacturer_id) 
WHERE (result.memcooling_id<4 AND memClock<1250 AND memClock>600 AND application_id!=19 AND application_id!=21 AND mem_productgroup_model.mem_id=1 AND result.memTCas>5)
GROUP BY memoryproductgroup.memproductgroup_id 
HAVING COUNT(result_id)>50
ORDER BY AVG(result.memClock) DESC
LIMIT 25;

 

[Massman]

Another attempt.

 

Some guys pointed out that, next to tCL, tRCD plays a huge role in the overclockability and therefore should also be taken into account in an overclockability chart. So, I made a new chart based on:

 

- Average frequency

- Average tCL

- Average tRCD

- Score = 1000 x (tCL + tRCD) / MHz

 

- Excluding the following benchmarks: MaxxMem, Memory Clock

- Upper memory clock = 1300

- Lower memory clock = 600

- tCL must exceed 5

 

- Need to add a bit more science into the equation. Since now the equation says that the importance of tCL = tRCD, which isn't the case. More testing is needed to get the right factors for tCL and tRCD.

- Also need to find out exactly how much data is skewed because people use the incorrect format to enter memClk data.

 

- Feedback?

 

SELECT manufacturer.name BRAND, memoryproductgroup.label PRODUCT, AVG(result.memClock) CLK, AVG(result.memTCas) tCL, AVG(result.memTRCD) tRCD, COUNT(result_id) RESULTS
FROM result 
JOIN memoryproductgroup ON (memoryproductgroup.memproductgroup_id = result.mem_productgroup_id) 
JOIN mem_productgroup_model ON (mem_productgroup_model.memproductgroup_id = memoryproductgroup.memproductgroup_id)
JOIN manufacturer ON (memoryproductgroup.manufacturer_id = manufacturer.manufacturer_id) 
WHERE (result.memcooling_id<4 AND memClock<1300 AND memClock>600 AND application_id!=19 AND application_id!=21 AND mem_productgroup_model.mem_id=1 AND result.memTCas>5)
GROUP BY result.mem_productgroup_id
HAVING COUNT(result_id)>50
ORDER BY CLK DESC
LIMIT 25;

 

[Massman]

 

SELECT manufacturer.name BRAND, memoryproductgroup.label PRODUCT, result.mem_productgroup_id, AVG(result.memClock) CLK, AVG(result.memTCas) tCL, AVG(result.memTRCD) tRCD, COUNT(result_id) RESULTS
FROM result 
JOIN memoryproductgroup ON (memoryproductgroup.memproductgroup_id = result.mem_productgroup_id) 
JOIN mem_productgroup_model ON (mem_productgroup_model.memproductgroup_id = memoryproductgroup.memproductgroup_id)
JOIN manufacturer ON (memoryproductgroup.manufacturer_id = manufacturer.manufacturer_id) 
WHERE (status_id<10 AND memClock<1350 AND memClock>500 AND application_id!=19 AND application_id!=21 AND mem_productgroup_model.mem_id=1 AND result.memTCas>5 AND result.memTrcd>4)
GROUP BY result.mem_productgroup_id
HAVING COUNT(result_id)>25
ORDER BY CLK DESC
LIMIT 50;

[TaPaKaH]

well okay, you've proven that GTX2 owners indeed run their mems a bit higher than others

is it dictated by memory quality? don't think so

[Massman]

Then please give input on how you define "quality" .

[Hondacity]

drats i should hae put my sst in the manuf info....

[Massman]

well okay, you've proven that GTX2 owners indeed run their mems a bit higher than others

is it dictated by memory quality? don't think so

 

Up.

 

I'm building the charts as we speak. I would REALLY want to know how I can extract 'memory quality' from raw data

[jmke]

collect RMA data from shops&manufacturers

[Massman]

RMA data doesn't tell you anything about the overclocking and performance capabilities of memory kits

[1Day]

To determine quality from a statistical base you first have to attempt a qualitative description of quality. Once you have reached agreement or accord rather on what quality is in memory then you can begin to define quantitative parameters that reflect that quality.

 

So begin with anecdotal evidence, what rams do people claim or say are quality rams. Take a sample of those rams and break their submitted data down into measurable's. Initially all measurable's I guess would be needed. Then compare and contrast that raw data against ram that is not considered quality. See what the differences are and you have a possible means of differentiation. Not perfect at all but it is a starting point.

 

Motherboard I suspect will be harder to quantify than rams.

[jmke]

RMA data doesn't tell you anything about the overclocking and performance capabilities of memory kits

 

but everything about the quality

[knopflerbruce]

I think that if the function of the chart is to show what memory brand/model is the best for BENCHING, then using CL7 is the best choice - as that's what mot people use for benching (or maybe even CL6?).

[Massman]

I'm wrapping up the article as we speak. It's a first version of the chart