Adjustment for Global Points - Work in progress

[Leeghoofd]

Well I want to see on the test server what these changes bring for most of us.. till now I have to agree with Sam... it iwll shake things up badly (unless I misunderstood the whole idea)

[gradus]

For hardware points old system norm, only need more points for hard categories. 2pounts for $hit category, more points for normal. bcs its really hard.

hard benching = more points. easy nab benching = 2pnts ok.

 

 

 

PS ans finnaly, can we have bonus on each socket/slot cpus. for example best score on socket 7, not only on p233mmx, but in all socket7? wr on s1150, or wr on775....its intresting competition.

[Massman]

First of all, let me once me re-iterate that this is not the final version of the points as it will be adjusted based on the feedback.

 

If I understand this properly... in short this proposed change would mean all submissions scoring less than 75%/50% of the best result in HW/GL category are basically worthless in terms of getting points?

 

For example Core 2 Duo E6600 PCMark 05 HW ranking:

http://hwbot.org/benchmark/pcmark_2005/rankings?hardwareTypeId=processor_873&cores=2#start=0#interval=20

http://hwbot.org/submission/2363595_havli_pcmark_2005_core_2_e6600_%282.4ghz%29_20228_marks

 

The top score is 36998 marks, my score is 20228 marks. At the moment my submission is ranked 27th out of 227 and getting 17.3 points. However in the new system it would mean 0.1 points... since is only 54.67% of the top score?

 

 

This new system seems to be working very similar to the current one in rankings with uniform score distribution. However rankings with few good scores and many average ones doesn't look very good.

Example - R11.5 32cores global http://hwbot.org/benchmark/cinebench_-_r11.5/rankings?cores=32#start=0#interval=20

 

Top score = 43.24... the lower treshold is 43.24*0.5 = 21.62. So it seems anything at current position 8 and below will get just 1 point.

 

The hardware points ranking would be like this

 

Rank	Points	Score
1	50.2	36998
2	46.2	36591
3	31.7	33853
4	26.4	33009
5	13.8	30363
6	12.5	30110
7	12.5	30109
8	11.9	29992
9	0.1	26881
10	0.1	26261
11	0.1	25862
12	0.1	25251
13	0.1	24801
14	0.1	24485
15	0.1	24292

 

I think for this category in particular, we have to take into account that PCMark05 is also a freak category. If you don't know the two or three key tweaks, you will have like 20,000 points less than everyone else. Also, bear in mind that supporting hardware like storage configurations have a large impact on the benchmark.

 

For the E6600 it looks like 30K would be a 'normal' score, in which case there would be points up to 50th place.

[Massman]

First of all, let me once me re-iterate that this is not the final version of the points as it will be adjusted based on the feedback.

 

In my opinion, the new distribution for hardware points is a really bad idea because the rewards will no longer be representative of the actual ability to bench legacy hardware.

 

Just some examples:

- Benching a totally obscure (or made-up) CPU/VGA at stock will now yield 10-20 hardware points.

- A semi-obscure category, like Pentium 4 630 SuperPI 32M will now be a 50-pointer. An argument "the categories will become more difficult once people discover and fill them up with scores" is not valid here since there will be thousands of 50-pointer categories and, realistically, there will not be enough active overclockers to fill them all up.

- Moving up a few spots in a really challenging category (like GeForce GTX780Ti Catzilla 720p) will yield zero-point-something points, unless you're talking about top 3 (getting where in the first place is an achievement of its own) in which case improving your score will give you a "generous" reward of 2-3 points.

 

This means that getting a decent coverage of hardware points (say, 700+) that count towards a league rank will reduce from "getting 2nd or better in 20 categories with 250+ participants each" to "getting 20 decent scores in non-obscure hardware categories", the latter of which can be achieved by skill-free scatter benching.

 

Also, for the HW masters league it will mean that two garbage scores will outweigh one "proper" score. So, the only way of becoming the HW king will now be by going through warehouse-like amounts of old and useless (non-resellable) hardware, benching which at any level (let alone properly) might take a decade of 100% free-time commitment. I can't see many people liking this since it is currently possible to become the HW king by benching family/wife-compatible amounts of relatively modern (and resellable) hardware and/or not benching 2D or 3D at all.

 

Took the time to plot the two rankings you mentioned:

 

 

Do I get your feedback correct if I read that you ask for:

• Reduce amount of 50-boint categories (increase threshold)
  
• Make it more rewarding to be at the top of the leaderboard (increase slope angle)
  

 

FYI, currently there are 313 hardware rankings where first place yields >=48 hardware points. The distribution per benchmark, in case you're wondering:

amount	name

38	3DMark06
32	XTU
28	3DMark05
27	SuperPi - 1M
27	3DMark03
25	CPU Frequency
23	3DMark2001 SE
22	3DMark Vantage - Performance
20	Aquamark
19	wPrime - 32m
12	SuperPi - 32M
10	PiFast
8	wPrime - 1024m
5	3DMark11 - Performance
3	Cinebench - R15
3	Cinebench - R11.5
3	HWBOT Prime
2	Unigine Heaven - Xtreme Preset
2	PCMark 2005
2	Memory Clock
1	3DMark - Fire Strike
1	MaxxMem Read Bandwidth

[TaPaKaH]

I don't think that you need to change anything with the current HW points. The distribution seems perfectly fair to me.

 

You shouldn't be getting more than 30 points per sub unless you are in Top 3 of a really popular category, of which there are currently plenty (~500, I guess) for anyone to choose from. If you extend the number of categories to 5K++ then maxing (or close-to-maxing) out HW points will be very easy, the value of the points will go down and so will the quality of scores since you will no longer need to bin and/or mod the stuff.

[Mr.Scott]

I tend to agree with Sam. HW points were never an issue. The issue was 3D points, but sight was lost over a bunch of other peoples wish lists.

 

I do also understand the an overhaul of algorithms in necessary for the future, but that should not be jumped into with the initial problem so quickly otherwise you'll just be doing it again shortly after. Think. Measure twice, cut once.

Edited January 4, 2016 by Mr.Scott

[Rasparthe]

I don't think that you need to change anything with the current HW points. The distribution seems perfectly fair to me.

 

You shouldn't be getting more than 30 points per sub unless you are in Top 3 of a really popular category, of which there are currently plenty (~500, I guess) for anyone to choose from. If you extend the number of categories to 5K++ then maxing (or close-to-maxing) out HW points will be very easy, the value of the points will go down and so will the quality of scores since you will no longer need to bin and/or mod the stuff.

 

I too don't have all that much issue with the hardware points are distributed currently, only with the potential maximum points an overclockerr is allowed in hardware points. I think it should match your potential max in globals.

 

I think you may be off on the assumption that the value of the points would go down and it will be nothing but easy points for everyone. It may start out that way where easy hardware points exist in some categories but eventually you will have to work at grabbing those easy points, those 'easy' scores get pushed down and lose value as each of these categories gain attention and get more competitive. The cap on points available ensures this (real example have a look at what it takes to get 0.2 HW points in e8600 cpuz category, 6070mhz almost no points for an overclock unattainable for most motherboards).

 

The problem is 3d categories don't see this type of competition because they aren't worth enough. Not because of a bias against 3d but mostly on the nature of adding more costs and added layer of knowledge needed. By artificially inflating the initial points you can only hope it will draw the interest to add more competitive scores.

 

I don't think it's really needed but I can see the reasoning behind such a change. Personally I just don't see this 'imbalance' of 3d that is in the OP, in fact, I would say 3d benchers have the distinct advantage of being able to gather 2d points as well. It doesn't work vice versa. Unless the idea is to reserve the top spots of each league for those with the biggest bank accounts that can afford several $1000 GPU.

Edited January 5, 2016 by Rasparthe
typing on a tablet is PITA

[K404]

I don't think that you need to change anything with the current HW points. The distribution seems perfectly fair to me.

 

You shouldn't be getting more than 30 points per sub unless you are in Top 3 of a really popular category, of which there are currently plenty (~500, I guess) for anyone to choose from. If you extend the number of categories to 5K++ then maxing (or close-to-maxing) out HW points will be very easy, the value of the points will go down and so will the quality of scores since you will no longer need to bin and/or mod the stuff.

 

In the interest of covering every angle... there was a time when hardware gold was max 15 points... and now we're used to 50. We've just changed the value of the reward for having the best score.... and maybe we will do so again soon.

 

I do believe that only the most popular categories should have the highest number of points available (whatever that number might be)

 

What if there were 2-4 categories for CPU and 2-4 for GPU that got the max and every other catgory was awarded points as a % of results compared to the most popular?

 

e.g.... 50,000 results for E8600 gets "X" and a CPU with 25,000 results gets X/2?

 

I don't think there should be a fixed saturation point. It should manage itself based on whatever the most popular categories are.

 

The big benefit of this is that it's FUTUREPROOF and doesn't need further interference!

 

Part of me is thinking that i've almost just suggested what's already being worked on :/

 

 

If points could be assigned as a % difference of the one beaten or to beat, that would be nice... with a multiplier towards the top because the top score should be tangibly better... because it's the top score.

 

E.G..... popular category: 50point gold. Take gold by 0.6%, multiplied by (e.g.) 10 ... Silver is worth either 6% less than 50 (47 points) and the leaderboard is calculated by working down from the top.

 

Big Downside: That would be a ****ing nightmare for the server to calculate.

 

 

The biggest benefit that I can think of with a higher number of points is the option of having points as an integer. A user gets 12 points. Not 11.8, not 12.1. They get 12. This is not diving or ballroom dancing Is it easier to explain to "an outsider" that they have 500 points, not 488.3? The obvious question to be asked from an outsider is "is the 0.3 important?"

 

 

Whatever is decided... I think it's important that the new algorithm is futureproof and can look after it's own scaling so it can be LEFT ALONE. EVERY TIME these is talk of a change, ther is a lot of talk and complaining and backlash. I know... sometimes I am a loud part of that, but would I be right in saying that lessons aren't being properly learned?

 

OR... is it important to benching to keep shaking things up? It only works when it works....

Edited January 5, 2016 by K404

[Massman]

Moved some posts from the technical discussion into the general discussion: http://forum.hwbot.org/showthread.php?t=150719

[speed.fastest]

How about Global Point for 3D is based on GPU Generation like HD 7000 and GTX 600 is in same generation? If based on generation is too hard maybe based on GPU Family. Make the Global Point not many, but based on Top 20 GPU Generation if not based on Top 10 GPU Family?

 

It's really hard to get Global Point for Low End 3D because there is no Global Low End 3D Category. Like @steponz said maybe IGP category can compete in Global IGP as CPU compete in core count like i3 Skylake?

[Kvekan_SWE]

Reading the comments and looking at the enthusiast people and how they might get screwed over by this new HW score system I got thinking. Why does it have to be linear? Could it be somewhat curved instead but go down to maybe 70 or even 65%, more people would get at least some points making more people submit scores while the ones that scores between like rank5 and 80% get slightly less points helping against this HW point inflation some are afraid of.

[Massman]

Small update: I've made a draft proposal for the new algorithm and sent it to Frederik to get technical feedback on how long it would take to implement, scalability and server load.

[Massman]

Another brief update.

 

The implementation is on track. A lot of time goes in to writing tests which are necessary to make sure the points don't break over time as we add more features. The general message from the devs is that the new method will be a huge stress release for the server. This should make calculation much quicker, maybe even instantaneous!

 

To give you an idea of the progress, we've recently completed a first calculation of Global points. And it went well!

 

Oh, we also added one more parameter in the algorithm design to determine the weight of a benchmark. In short, this parameter will allow us to arbitrarily limit points of certain benchmarks that are over valued. I'm looking at you, XTU.

[Rauf]

Ok, nice!

 

But what changes are you working on? There have been a lot of ideas in this thread.

[xxbassplayerxx]

Ok, nice!

 

But what changes are you working on? There have been a lot of ideas in this thread.

 

Seconded! I know we discussed a lot and plenty of options were thrown around. Did you decide to move forward with one? Will the community have input on the specifics? Perhaps a new thread to do so?

[Guest george.kokovinis]

It is more than obvious folks, that Pieter, besides anything else, is focused on XTU.

It would be nice and kind though, if the Administration, gave us a more

clear horizon to look at.

[Alan_Alberino]

What about XTU Hardware points? Some people has almost 20 hardware points with XTU on locked 4th gen CPUs and earn like 400 points without almost doing OC

[Leeghoofd]

It is more than obvious folks, that Pieter, besides anything else, is focused on XTU.

It would be nice and kind though, if the Administration, gave us a more

clear horizon to look at.

 

Focusses on limiting or keeping it as it is ?

 

In short, this parameter will allow us to arbitrarily limit points of certain benchmarks that are over valued. I'm looking at you, XTU.

[Guest george.kokovinis]

Focusses on limiting or keeping it as it is ?

 

In short, this parameter will allow us to arbitrarily limit points of certain benchmarks that are over valued. I'm looking at you, XTU.

 

 

Understandable.

Certainly overvalued benchmarks should be adjusted in line with the rest.

I was just asking if there is anything else we should know, since so many options have been discussed.

[Xtreme Addict]

What about XTU Hardware points? Some people has almost 20 hardware points with XTU on locked 4th gen CPUs and earn like 400 points without almost doing OC

 

To squeeze those extra 2-5 MHz BLCK you have to go LN2 (DICE is not enough) and go mems @ LN2. Also bin few cpus. Some guys might remember G.skill World Cup qualifier last year with Pentium G3240, how hard it was to get high BCLK cpu. That is not that easy and much harder than playing i3 6320 @ air with 103 MHz lock by Intel. Also HW points are counted for the amount of submissions by certain HW. Nothing to be pissed about it, those cpus are just popular.

 

In fact I don't see anything wrong in current point system. Instead of reducing points - award more points for 3D and everything will be fine.

Edited February 22, 2016 by Xtreme Addict

[Alan_Alberino]

To squeeze those extra 2-5 MHz BLCK you have to go LN2 (DICE is not enough) and go mems @ LN2. Also bin few cpus. Some guys might remember G.skill World Cup qualifier last year with Pentium G3240, how hard it was to get high BCLK cpu. That is not that easy and much harder than playing i3 6320 @ air with 103 MHz lock by Intel. Also HW points are counted for the amount of submissions by certain HW. Nothing to be pissed about it, those cpus are just popular.

 

In fact I don't see anything wrong in current point system. Instead of reducing points - award more points for 3D and everything will be fine.

 

I'll take for example i5-4590:

 

http://hwbot.org/benchmark/xtu/rankings?hardwareTypeId=processor_3635&cores=4#start=0#interval=20

 

Since 4th, they are all with normal cooling (air/aio/water), and all on that first page with +20pts, so you can gain +20pts without using sub-zero cooling.

[Xtreme Addict]

and what is the point? go subzero and grab more points/beat those guys (or me ). i5 4590 has more than 2000 entries, that's why it has big points. HW points are related to popularity of the model, that's all. Bin few cpus to get 110+ BCLK with around -120*C, push PSC on LN2 2800+ 6-10-6 supertight and grab high points

 

If you are whining that it's too easy, what about old GPUs? Most of the 50 hw points are done on GPUs @ air, just new generation cpu for legacy benchmarks on cold, sometimes even SS and rerunning old gpus all over again when new CPU like Haswell, Skylake appears.

Edited February 22, 2016 by Xtreme Addict

[nickolp1974]

Has something already been done with XTU global points?? Lost nearly 30pts other day (i3 6320) still in 2nd, which now awards 65 being 1st awards 166pts hope it does not stay like that, riduculous gap!

[Rauf]

Depends on how many nr1 there is...

[Massman]

Ok, nice!

 

But what changes are you working on? There have been a lot of ideas in this thread.

 

The main test is the transformation from rank-based algorithm to percentage-based algorithm. Here's the explanation of why it's necessary. Then it includes the rough version of my latest post. I received a message from the developer we're moving to further performance testing, which is basically ensuring the scaling with all the rankings works fine. Once this is completely I can go back to the theory and verify if the algorithm works as intended.

 

Once that's verified I will update the thread here again with more detailed information and the community can check UAT and feedback on the points (max, min, scaling, etc). Once we have a general consensus on the forum and among staff, we'll do a full write-up of the change and post it in public. If the general consensus is positive too, we can set a date of deployment for Rev 7. Still a long way .

 

As a preview, here is the 3DMark Vantage 1xGPU ranking on our test server. The main difference is the scaling of the points as they are determined not by which rank you have but by how far your score is off the top score. The scaling is very flat, but this is something we can address by tuning the various algorithm parameters

 

Points R7 Test (based on PCT)

 

Points R6 Test (based on Rank)

 

It is more than obvious folks, that Pieter, besides anything else, is focused on XTU

 

This is written 3 posts above your reply: "Oh, we also added one more parameter in the algorithm design to determine the weight of a benchmark. In short, this parameter will allow us to arbitrarily limit points of certain benchmarks that are over valued. I'm looking at you, XTU.".

 

I mean, I call XTU literally by its name ...