Adjustment for Global Points - Work in progress

[Massman]

As many of you know, recently there has been a couple of discussions about the distribution of global points. Below you can find the threads that reference this topic.

• Updating my feeling
  
• How about 3D
  
• ..
  

To keep a long story short, the core of the issue is that due to XTU attracting a lot of active members, it yields the highest amount of global points. Along with XTU, the 2D benchmarks are also more popular than the 3D benchmarks in contemporary overclocking. The technical side of the problem is that the point algorithm is based on the popularity of a certain ranking to determine how much points are given. This has as effect:

• Due to extreme high activity, XTU is maxed out in points
  
• Due to diminishing popularity, the legacy and 3D benchmarks yield less points than 2D benchmarks
   
  In other threads there's been plenty of discussion on how to address this problem. The main stream of thought is to add a "difficulty" parameter to the point algorithm. In this thread I want to explore a different path and propose a more technical solution to the current situation.
  To keep things fairly simple: the HWBOT point function or equation has 6 parameters which determine how much points a given result yields. By adjusting these parameters we can change things like the maximum of points for a #1 position, the difference between the #1 and the rest of the top-10, the maximum amount of points in a given ranking, and so on. I've worked on adjusting these parameters to achieve the following goals:
  
  1. More benchmarks will hit the maximum participation and thus maximum points
     
  2. the maximum points for a #1 score increases to 200 (now 167)
     
  3. less steep point slope for top scores in global ranking
     

  The adjustment is currently being tested on the UAT server (recalculation in progress) but is not final in design. The main discussion points are:

  1. How many global rankings should be at maximum point capacity?
     
  2. How steep should the point slope be for the global rankings?
     

  I will update the thread when the recalculation on our UAT test server has completed. Below a bit more information about the various aspects of the adjustment.

   

  1. Maximum Participation and Maximum Points

   

  The point algorithm is a natural logarithm. The equation expresses the points of a given ranking as function of participation and position. Participation is measure as amount of unique users who submitted in a ranking in the past 365 days. There is a participation threshold set at 2000 participants. The threshold prevents the points from growing infinitely. It is this threshold that caps the XTU 4xCPU global ranking with over 17,000 participants from being excessively more valued than HWBOT Prime 4xCPU with about 1,800 participants this year. The most active 3D benchmark is 3DMark Fire Strike 1xGPU with close to 1,000 participants.

   

  Note: take a moment to consider the algorithm is able to deal quite well with the XTU popularity.

   

  The parameter adjustment is to lower the threshold to 1,000 participants. This will increase the amount of global rankings that hit the maximum points from 2 to 5 and the amount of ranking that are over half capacity from 1.5% to 8.3%. The five rankings are: XTU 4xCPU, XTU 2xCPU, XTU 6xCPU, HWBOT Prime 4xCPU and 3DMark Fire Strike 1xGPU.

   

  If we would also enlarge the participation window from 1 year to 2 year, there would be 12 maxed out rankings and 14% of the rankings would be over half capacity.

   

  2. Maximum Points for First Place

   

  As you read in the previous section, the current maximum global points is 167pts at maximum participation. In the adjustment we will increase this to 200pts. This is mainly to avoid people losing points because of the adjustment.

   

  Note that this will affect the balance between global and hardware points in the Overclockers League!

   

  

   

  3. Point Slope in Global Ranking

   

  The global points of any submission is derived from the points of first place in the ranking using a natural logarithm, except for second place and third place which are a fixed fraction. The current implementation has a very steep slope to reward the first place in a global ranking. This is to reward being the best in the ranking in times when binning was not such a widespread practice yet.

   

  The adjument changes the slope quite drastically, increasing the points for second place from 75% to 95% of #1 and the points for third place from 56.25% to 92.5% of #1. From position 4 the points increase via updated parameters.

   

  In a maxed out ranking, this means that the the point distribution is:

  Pos.	Points
  
  #1 	199.8 pts, currently 166.4 pts
  #2 	189.8 pts, currently 124.8 pts
  #3 	184.9 pts, currently 93.6 pts
  #4 	179.9 pts, currently 64.8 pts
  #5 	171.9 pts, currently 61.8 pts
  #10 	147.0 pts, currently 52.6 pts
  #50 	89.1 pts, currently 31.1 pts
  #100	64.2 pts, currently 21.9 pts
  #250	31.2 pts, currently 9.7 pts
  #500	6.3 pts, currently 0.1 pts
  #1000	0.1 pts, currently 0.1 pts
  

  

[TaPaKaH]

So, a 300MHz overclock on a 6700K will score more points (~#175 in 4x XTU hence ~50 globals) than any hardware gold. Awesome!

[Massman]

"will" -> "would"

 

The adjustment is currently being tested on the UAT server (recalculation in progress) but is not final in design. The main discussion points are:

1. How many global rankings should be at maximum point capacity?
   
2. How steep should the point slope be for the global rankings?
   

[der8auer]

I don't really understand how this fixes the 2D vs 3D "issue". To me this looks like everyone will just have more points

 

(I still think there is nothing to fix. 2D vs 3D is a simple natural result)

[Massman]

I don't really understand how this fixes the 2D vs 3D "issue". To me this looks like everyone will just have more points

 

(I still think there is nothing to fix. 2D vs 3D is a simple natural result)

 

Lower threshold = more rankings closer to max participation, including the 3D benchmarks.

 

The parameter adjustment is to lower the threshold to 1,000 participants. This will increase the amount of global rankings that hit the maximum points from 2 to 5 and the amount of ranking that are over half capacity from 1.5% to 8.3%. The five rankings are: XTU 4xCPU, XTU 2xCPU, XTU 6xCPU, HWBOT Prime 4xCPU and 3DMark Fire Strike 1xGPU.

 

If we would also enlarge the participation window from 1 year to 2 year, there would be 12 maxed out rankings and 14% of the rankings would be over half capacity.

[Noxinite]

Changing the slope of the points based on position makes sense more for HW pts than Global pts to me.

 

But otherwise the current points system seems to work - not perfectly, but it works. (If anything I would say there are too many global points for some benchmarks *shrugs*. But I prefer hardware points, so...)

[der8auer]

Ah okay now I get it. It could be a solution but the question is: Is the "solution" the correct path?

 

In the end 3D mainly has a lower participation because of the price. A competitive rig for 3D is always at least double the price than a 2D rig.

Adjusting the points in this direction would also adjust it more into the "money-game-direction".

I mean it's not like everyone has 980Ti cards laying around but is just too lazy to bench them.

Edited December 19, 2015 by der8auer

[WhiteWulfe]

Will similar changes be occurring to hardware points as well, especially in regards to thresholds?

[aerotracks]

I mean it's not like everyone has 980Ti cards laying around but is just too lazy to bench them.

 

This.

 

Also, another problem was that mainstream graphics cards like GTX 970 aren't worth benching. This would be fixed to some degree I guess, but it will still be "get in line after 980Ti*".

*unless some idiot accidentally underclocks one

Fortunately with 2D globals it's not 5960X or GTFO.

Edited December 19, 2015 by aerotracks

[zeropluszero]

Frankly, I don't really care about the points system, I'll just see how I'm doing after the fact and keep benching what I want to/can afford.

 

What I want to see, is those who have an opinion really put that forth, and have their thoughts really listened to by Pieter and/or whoever else determines these things. Create a poll, ask people to vote for what they want, and then go with that poll.

[Rauf]

I think the idea is basically very good! But maybe not increase the points. Have to keep hw points relevant.

 

Also agree with adjusting the threshold, and maybe slope, on hwpoints, with every generation hw points gets a bigger and bigger problem.

 

Or maybe the lower threshold should be just for 3D?

 

Another thing I have been thinking about that would be massively beneficial for all parties would be to get hwbot-integration of the next 3Dmark.

[BenchBrothers.de]

In my oppinion the popularity of a benchmark here on hwbot depends on the amount of points you can "earn" with it. Like aerotracks said, 970 isn't really worth benching. Why? Because of the algorithm hwbot is contributing points to it. And thats the case for lots of other hardware.

 

Don't mix up popularity with the chance to get lots of points with a certain benchmark or hardware...

 

Back in the days where 3d-benchmarks were distinguish in single-card and multiple cards, nearly every 3d-bencher tried to bench a dual-gpu-card. After hwbot started to count gpus, dual-gpu-cards lost nearly all attractivity.

 

The same applies to multi-gpu-setups. Multi-gpu is often a tough challenge but nowadays you don't get many points so nearly no one is benching such a setup. And because nearly no one is benching such setups the algorithm of hwbot contributes less points.

 

All decisions made at hwbot relevant to the point algorithm have effects of the "popularity" of a benchmark. People choose to bench these setups where they get the most points. So again, don't mix up popularity with the chance to get lots of points...

 

 

 

By the way: For the current XTU-WR Dinos22 takes home a total of 429.4 points (UWP, UGP, UHP, GTPP, HTPP). 8 Pack with the current WR in 3DMark FireStrike Extreme gets 233.4 points - thats only 54.4 percent for a benchmark, that needs more attention. More hardware involved, more pots and ln2 involved and a much longer duration of the benchmark.

 

That relationship doesn't fit anymore and even a slight variation to this algorithm won't fix this...

[Massman]

The recalculation for certain global rankings finished. In the below screenshot you can see the effect of increasing the maximum points and less steep slope. Note that on the UAT server we are running an older database, so the results are a bit outdated. But you can easily compare position by position.

 

Current vs. UAT Test

 

[Massman]

Will similar changes be occurring to hardware points as well, especially in regards to thresholds?

 

Not on the planning. What are your main concerns with how hardware points are distributed?

 

What I want to see, is those who have an opinion really put that forth, and have their thoughts really listened to by Pieter and/or whoever else determines these things. Create a poll, ask people to vote for what they want, and then go with that poll.

 

Thanks Matt.

 

I would really appreciate specific feedback on the technical solution for the points. It's not easy to balance all the benchmarks, hardware and leagues, so it would be really great to see a general community consensus on the direction we should take.

[Mr.Scott]

Quote:

Originally Posted by zeropluszero

What I want to see, is those who have an opinion really put that forth, and have their thoughts really listened to by Pieter and/or whoever else determines these things. Create a poll, ask people to vote for what they want, and then go with that poll.

 

 

so it would be really great to see a general community consensus on the direction we should take.

I believe we have seen all this before, and then the opposite was done anyway.

There will never be a good answer that will please everybody. Just leave it alone.

[Doug2507]

Lowering the threshold seems to be a good idea. It will solve the issue of amount of 3d points gained compared to 2d and create a better balance between the two.

 

Having said that, due to the large amount of benchmarks being awarded points, it'll be a lot easier to score more and fill personal point boards running same same benches. (Setup for one, repeat 5 times).

 

Not sure increasing points awarded is all that great though. Smoother slope sure as it'll make everyone fight harder for placement and every point will count. As for guys loosing points, sour grapes. It applies to everyone.

 

It'll also take away value from hardware points, the majority of these usually acquired by 3d which would be counter productive to OP. Keeping/giving value to these is important, it encourages guys to bench older gear, learn to mod, and usually an affordable option for all. This should also encourage top tier guys to get their hw pts full. Guys at the top should have an all round skill set in benching, not just latest and greatest hw.

Edited December 19, 2015 by Doug2507

[Noxinite]

Not on the planning. What are your main concerns with how hardware points are distributed?

 

It would be better if the points slope was less steep.

[zeropluszero]

I've mentioned from the start I'd like to see points removed entirely from duplicate benchmarks. Points only for one cinebench, one fire strike etc.

Problem when you go removing them is when you have people that only enjoy one benchmark lose their points, they lose interest entirely. Look at pcmark05. Now its a flawed benchmarks, but there were guys oj the ocau them who enjoyed this a lot, and have disappeared since it was removed.

You cant please everyone, but listening to what they say and having the community vote is important.

[zeropluszero]

What sort of solution could we add for encouraging enthusiast league overclockers to keep going? How do people like the idea of a separate front page daily leader board for enthusiast league? Just below the current one?

I see we get plenty of new users through xtu, but do they submit a score and then test something else?

[WhiteWulfe]

Not on the planning. What are your main concerns with how hardware points are distributed?

 

I think it isn't really a concern on how they're distributed (although I will admit I really don't like how things drop dramatically between first and even fifth place), but more along the lines of how lowering the thresholds for the points to start going up would actually help increase the amount of participation with various gear. It would also help validate the effort people have put into older (or rarer) hardware. Sometimes it's somewhat disheartening to put several hours (or even days) into trying to get a really good score on a particular benchmark, place in the top third of submissions and then get.... Oooh, congratulations, you're 36/75, you get a whopping 0.3 hardware points!

 

Also, with a broader scaling for hardware points those who are competing on air and/or water would get a much greater feeling about their rankings since they'd have a higher chance at obtaining points. Sure, they won't reach the same levels as those using single stage, dry ice, or liquid nitrogen, but still. With how things currently are with the abrupt slope (assuming hardware point slopes are similar to those of global points, although looking at the more popular benchmarks with popular cards it seems to be similar) there seems to be a feeling that there isn't as much of a point in pushing whatever you have (or can afford) if you can't get up into the top 20% or so of scores.

 

Am I saying that hardware points in general need to be given out more freely? I suppose it could be taken that way, but what I think I'm trying to get at is can we have a more gradual slope with the points with a lower threshold for maximum points, and that I feel by doing such it could help improve the amount of people participating (because it would be lovely if they didn't just go "ugh, only five hundred other people have submitted benches on this hardware, is there really a point in even benching this?" and instead went "oh neat, five hundred other people have run this I wonder how well I can do against them, because there's a decent amount of points to be had").

 

Or something to that effect.

[WhiteWulfe]

I've mentioned from the start I'd like to see points removed entirely from duplicate benchmarks. Points only for one cinebench, one fire strike etc.

Problem when you go removing them is when you have people that only enjoy one benchmark lose their points, they lose interest entirely. Look at pcmark05. Now its a flawed benchmarks, but there were guys oj the ocau them who enjoyed this a lot, and have disappeared since it was removed.

You cant please everyone, but listening to what they say and having the community vote is important.

 

I personally feel that there is value in having multiples of the same benchmark receiving points because each of them not only requires different tuning but a completely different approach to how you're doing things. Additionally, it gives extra value to those who've supported the company that made the benchmark in the first place (since Firestrike Extreme and Ultra are only available legally if you pay) something else to do, or a new challenge to try out. Not everyone can afford to pay for a license (hey, a number of people have families, or other priorities) but they still want to bench... So in that case, if you were to remove points from the base version one could lose quite a few people from that benchmark, and then eventually... They might very well just say screw this overclocking hobby.

 

Besides, it's only your top 15 globals and top 20 hardware scores, so why does it matter if someone does it with four versions of everything versus someone doing it with just one or two pieces of software? Variety is the spice of life as they say

 

What sort of solution could we add for encouraging enthusiast league overclockers to keep going? How do people like the idea of a separate front page daily leader board for enthusiast league? Just below the current one?

I see we get plenty of new users through xtu, but do they submit a score and then test something else?

 

I personally think some sort of event that targets them directly would be great for engagement, like how the Rookie Rumble and Novice Nimbles are..... But keep the random prize draws. Ignoring all the various people on my team bugging me to actually sign up, the random prize draw for Rookie Rumble was the biggest draw for signing up in the first place. Might be hard to come up with something PG for the allliteration though

[speed.fastest]

I think there is no problem with current Global Point Distribution. The problem is i think Hardware Point is based on position, not based on how much leading score from the baseline score. For baseline is based on hardware lowest score (HWBot can adjust the baseline if automatic baseline is too low or too high). I know we cant be perfect but at least we tried. Keep pushing it guys! Sorry if you dont understand my english.

[newlife]

The changes put far too much focus on global points and does very little to address the problem at hand with 3d which is very largely based around cost and basically more focus around lower end cards could encourage more people to bench 3d because most people are scared bunnyextractionless by the idea of overclocking and killing a high price card

 

To give an idea on too much focus on global they triple but hardware points stay the same

[Rasparthe]

 

There will never be a good answer that will please everybody. Just leave it alone.

 

This ^^^ , the points system is already heavily skewed to globals, if the solution is increasing the amount of globals given out, please leave it alone.

 

I still have hopes that HWBOT will one days recognize time, knowledge, and skill in all generations equally and steer away from rewarding the biggest wallets.

[lanbonden]

I think the idea to increase the points for people that didnt place first is good and I think the idea to decrease the cap for how many submissions before the global reach the maximum is good BUT Id go the other way and reduce the maximum globabal possible to 100 points instead of increasing it too 200 points.

 

Then Id also add the new way of calculating points to HW points (stay with 50 as maximum), for really.. HW points are where it is, Globals shows that you spend alot on binning the latest CPU.

Just look at the 900 series for example, 980ti gets all the globals but so few bench it compared to for example 970 that it hardly gives any HW points atm.