R5: UGP/GTPP fix for exotic categories

[Massman]

As expected, the min_points=60 rule for UGP categories has made it possible to dominate rankings by simply running very obscure hardware.

 

http://hwbot.org/league/enthusiast

 

Dead Things is now #1 of the Enthusiast League mainly by hitting the #1 spots in multithreaded benchmarks with his stock clocked single, dual and quad 10-core cpus. That brings 60p x 3 (config) x 3 (benchmarks) for just having the hardware. In these categories, there's almost no competition.

 

I think we should have implemented the variable min_points from the start. This is suggested by a member and seems good.

 

- 10pts for a global with 10 or less submissions

- 20pts for 10-20 subs

- 30pts for 20-30 subs

- 40pts for 30-40

- 50pts for 40-50

- 60pts for 51+

 

The exact point scheme isn't fixed yet, but the scheme Borandi (again!) suggested seems pretty good. Currently the scheme is

 

1 - 60

2 - 45

3 - 33.6

4 - 26.2

5 - 22.4

6 - 21.7

7 - 21.1

8 - 20.5

9 - 19.3

 

If we follow the community suggestion from Borandi, if there are 9 participants in a ranking, the scheme would be:

 

1 - 10

2 - 7.5

3 - 5.6

4 - 4.4

5 - 3.7

6 - ...

...

 

Any comments?

[K404]

Sounds......alright? What are the requirements for >60 points for #1? Most of the 2x/3x/4x 3D leaderboards have 60 points for gold and there are hundreds of submissions..... thousands? 60 points for gold for 50 submissions or 500, it's a big zone for 60 points to cover

[sumonpathak]

Sounds ok to me...just because someone has access to exotics shouldn't be a criteria to get more points...afterall...there are no effort involved in those rankings...

no offense intended...

[knopflerbruce]

Sounds......alright? What are the requirements for >60 points for #1? Most of the 2x/3x/4x 3D leaderboards have 60 points for gold and there are hundreds of submissions..... thousands? 60 points for gold for 50 submissions or 500, it's a big zone for 60 points to cover

 

This COULD be because an additional "feature" got implemented - the engine doesn't count subs that are more than one year old. That's why singlecore wprime rankings, with an insane number of subs, get alot less than you'd think.

 

Get rid of this, pleeeeeeeeeease:D If it's there, maybe I'm just dreaming:o it doesn't really do any good at all, as time has got nothing to do with the quality of a result. A sempron 140 result from two years ago is just as good as a sempron 140 result obtained today, for example.

[knopflerbruce]

Sounds ok to me...just because someone has access to exotics shouldn't be a criteria to get more points...afterall...there are no effort involved in those rankings...

no offense intended...

 

This is not entirely true, unless the ranking is REALLY exotic (as in top spot ran at stock). You see the same thing in the popular rankings, although not near the top spots. Buy random highend HW, run full pot, and *bam* you get a bunch of points. Not exactly a similarity between points and skill there either:D usually top spots with more than a handful subs are hard to get, as people will still fight for the gold cups. Probably really easy to get top 5, but grabbing gold cups isn't as easy as you think just because there are less submissions:)

[sumonpathak]

^well..am not too well versed with sub zero...so couldn't comment there...but some of the submissions really sticks out like anything....

at times am just like..wtf? no tweaks..nothing..they just ran it and bam? 60 freaking points?

[dhenzjhen]

^well..am not too well versed with sub zero...so couldn't comment there...but some of the submissions really sticks out like anything....

at times am just like..wtf? no tweaks..nothing..they just ran it and bam? 60 freaking points?

 

 

the problem is that i have access to those stock hardwares and u don't

[sumonpathak]

expected u there....

see...i wouldn't mind u guys getting bucket loads of points..just make sure that the guys submitting pretty obscure hardware's gets the same amount of points too...after all....they have "access" to those hardware's..and we dont

[Bobnova]

Get rid of this, pleeeeeeeeeease:D If it's there, maybe I'm just dreaming:o it doesn't really do any good at all, as time has got nothing to do with the quality of a result. A sempron 140 result from two years ago is just as good as a sempron 140 result obtained today, for example.

 

I agree with this. Heavily.

[IanCutress]

This COULD be because an additional "feature" got implemented - the engine doesn't count subs that are more than one year old. That's why singlecore wprime rankings, with an insane number of subs, get alot less than you'd think.

 

Get rid of this, pleeeeeeeeeease:D If it's there, maybe I'm just dreaming:o it doesn't really do any good at all, as time has got nothing to do with the quality of a result. A sempron 140 result from two years ago is just as good as a sempron 140 result obtained today, for example.

 

I agree with this. Heavily.

 

QFT. I think if you're going to implement a feature like this, 12 months is too short - especially when the best hardware could still be the old stuff. Especially on single core wP et al.

 

We all know the basis behind this change was to drive people towards benching the stuff that interests the majority, but there's still just as much enthusiasm going for some of the old tough stuff as the newer stuff.

 

If there is a benchmark, we will run it. So to speak.

[Massman]

Interesting points - I'll bring them up at our next staff meeting (which we hold every day). I tend to agree that even though a time-based algorithm might be more elegant, it's not really bringing to the table. Just more frustration about dropping points.

[knopflerbruce]

And probably eating up a few CPU cycles as well, am I right?

[Massman]

The opposite actually as the amount of submissions that need to be taken into account to calculate the weight is lower.

[Yamunsa]

Get rid of this, pleeeeeeeeeease:D If it's there, maybe I'm just dreaming:o it doesn't really do any good at all, as time has got nothing to do with the quality of a result. A sempron 140 result from two years ago is just as good as a sempron 140 result obtained today, for example.

 

seconded here. many i know bench 2 or 3 gens behind purely for financial reasons. i could never have afforded a Q9770 setup when they came out but now you can find them cheap and have a go at the older scores with everyone rewarded for their efforts not just the recent sub.

[I.M.O.G.]

Sounds good.

[Massman]

Small update:

 

The update was planned for this sprint, but will be postponed until the next one. Biggest problem at the moment is figuring out how to not overload the server when recalculating the submissions when adjustment goes live. It takes about 4 days to completely recalculate all submissions.

[IanCutress]

Set the update all system to process in batches of 100, with a 60 second wait command between.

[Massman]

It's one of the options, yes. But recalculating will take over a week then, I think. A week in which new submissions have been made and the people who submitted those also want their points instantly.

[I.M.O.G.]

Small update:

 

The update was planned for this sprint, but will be postponed until the next one. Biggest problem at the moment is figuring out how to not overload the server when recalculating the submissions when adjustment goes live. It takes about 4 days to completely recalculate all submissions.

 

"The update" meaning the Rev 5 changes, or just this points adjustment in this thread?

[Massman]

The UGP/GTPP fix. It was planned for the sprint that ends tonight, but is postponed to the next development sprint which ends in two weeks.

 

Not really a big change ... hence the "small" update

[Massman]

Alright, update. So, it took a while before we could apply this patch on the production server. The main reason was to determine when we would hurt our little engine by requesting a large recalculation. We've decided to do it this development sprint. Now, before we do so, here are the now and improved algoritms. Because of the complexity of the HWBoint algoritm, we had to adjust more than just the global points for exotic categories to ensure future-proof behavior.

 

We decided to go with a more simple logarithmic distribution. The maximum number of points that can be earned is determined by the number of participants. The idea is that 1st place always gets 100% of this, and lower ranks get a certain percentage, which is based on another logarithmic distribution. Points given are largely the same as with the old system, the major changes are:

• No guaranteed 60 points for more exotic categories
  
• A lot more points can be gained from medium-popular categories (100 to 500 participants)
  
• Slightly more points for lower rankings
  
• There's a little hiccup for last place though, you actually gain points when more participants join for exotic categories. That's something that can't really be fixed but the impact is relatively minor we think.
  

 

We ran the new algorithm on a test server with a database snapshot of 23/02/2013. You can check the differences of the top 20 in each league below. Main thing to notice is that rankings largely stay the same, but Dead Things is completely gone (dropped to position 40 with 383 points).

 

Points for first and last place (global): Old versus New algoritm

 

 

XOC Ranking Difference (snapshot 23/02/2013)

 

 

EL Ranking Difference (snapshot 23/02/2013)

 

[TaPaKaH]

finally.

[zzolio]

why not just copy hwpoints?

[Teemto]

Is the database still in the progress of updating? I see some real weird things going on in the Xtreme OC League rankings.

Last week I was ranked at around position 150, over the weekend I jumped to position 96 and today I'm at 482??? Same thing is happening to my teammates Leeghoofd and FunSkilZZ.

[Rasparthe]

Sad day for HWbot, too bad core misunderstanding of what drives HWBOT still alive and well. Can't wait until all points are disabled unless you are using latest CPU (4770k) and latest GPU (Titan). Very disappointing, I'm sure policies that treat certain segments of the site like second class members will continue to grow and expand the membership.