R4.2.4/R4.3: HWBoints finetuning

[Massman]

As mentioned earlier in this subforum, we have been looking at the feedback we received from the transition to HWBOT R4 and the changes it brought to the point algoritm. We're currently in the process of finetuning the boints and will introduce all changes in one big package which will be included in either R4.2.4 (due 19/11) or R4.3 (due 01/12).

 

FYI - next to the updates mentioned below, we will also implement a fix for the existing problems with the Global Team Powerpoints as described here. The fix is currently planned for R4.3, but the development is R4.2.4 is going so incredibly smooth that we are considering moving the fix to the current development sprint. If we can't include the GTP fix in R4.2.4, we will postpone the fixes described below and package them in R4.3. Just a matter of keeping things coherent.

 

Changes:

 

Currently, we have installed two updates to the member hwboints algoritm for global rankings:

 

1) participation weight based on 12-month period instead of 6 months

2) introduce min_participation constant

 

Background information:

 

The current hwboint algoritm used for calculating the points awarded to global member rankings is, just like the very first hwboint revision, mainly based around the amount of competition there is within the ranking. In Rev4, we introduced a time limitation to calculated the level of competition within one ranking. From Rev4, competitivity is determined by the amount of different users who have submitted to a certain global ranking over the past 6 months.

 

Based on feedback such as "I'm losing points in Wprime if I submit more results", we found out that a six month period is quite short. We also found that the drop in points is quite big if, for some reason, a benchmark is suddenly not so popular anymore. Extreme scenarios can be found in Wprime where the global points for #1 position dropped from 133p (without 6-month period) to 13,1p (with 6 month period).

 

To resolve this situation we've prolonged the period for calculating the level of competition in a specific ranking as well as introduced a minimum participation level for every global ranking.

 

Practical consequences:

 

Most importantly, practically this means that pretty much everyone is gaining points. Hooray!

 

On a more serious note, these two updates do have some interesting consequences. For one, in case a benchmark is no longer popular, the points will drop at a much lower speed than they did now. Secondly, this also means that every global ranking has a minimum amount of points. For every #1 global rank, you will now receive a minimum of 59,8 points. This is quite interesting for:

 

- people who frequently run 3/4-way setups, but not subzero

- for enthusiast league members who score top scores in overall benchmark rankings

- 1xCPU multi-threaded CPU applications (where competitivity drops due to lack of newly released single core cpus)

- unpopular but challenging benchmarks (eg: pcmark)

 

In general, you can just think about this update as a guarantee that a global #1 will always receive a minimum of 59.8p. I also want to stress again that these fixes only affect the points awarded in global member rankings and that the formula for calculating the league ranking is not affected. In other words: the game is completely the same, only a few more ways to be competitive.

 

Test environment:

 

This weekend, we'll updated our test server environment with a new dump from the result database and open it up for you to check it out and see your rank. For now, here are just some examples.

 

Some examples:

 

- Booj' 3xGPU Heaven DX11: 5.1p -> 59.8p

- Ananerbe's 1xCPU Wprime 1024M record: 11.0p -> 59.8p

- **DP**'s 3xGPU Vantage record: 8.1p -> 59.8p

- Nacho_arroyo's 6xCPU Wprime1024M record: 9.6p -> 59.8p

- Moonman's 3xGPU 3DMark01 record: 6.3p -> 59.8p

 

Example of the updated leagues. Note: this is using an outdated dump of the production database. It's very likely that your latest submissions are not taken into account yet.

 

 

 

[Hondacity]

im confused x 10^2

[Mr.Scott]

Well....Faster should be happy now.

[TaPaKaH]

so now there's an even bigger disbalance between TPPG and TPPH as you started awarding more of the prior ?

[Massman]

so now there's an even bigger disbalance between TPPG and TPPH as you started awarding more of the prior ?

 

No, because this change only affects MEMBER points.

[K404]

Well I guess if the unpopular crap gets awarded a lot more points, more people will be motivated to bench and fewer categories will be unpopular.

 

Good for keeping a wide selection of hardware under some form of spotlight, bad (open to interpretation) because again points are the motivator to bench. Points bring binning. Binning starts to undermine the "little guy"

 

I know not everyone feels the same way, that's cool

 

 

 

Weirdly, getting a good score with.... e.g.... an AMD tri-core is worth more points because it has a global ranking than it does for being any kind of popular chip

Edited November 12, 2011 by K404

[Massman]

FYI, the point distribution comparison between "uncompetitive Wprime32M 3xCPU" (global points) and "competitive 8800GTX 3DM01 1xGPU" (hardware points).

 

Beating lots of people in a hardware class yields more points (in general) than beating few people in a global class.

 

 

[K404]

Personally, I think the exponent is too sharp. I know we've talked about that before, but the global tail off is even worse

[matose]

So, the competition points are still not implemented? The ideea is still on the "to do list"?

 

Ontopic: I don't think it's a good ideea to give so much global points to all the shitty overall's WR . Makes the WR's that really matters (single card 3D for example) less important...

[Massman]

Competition points are implemented, but not very transparent. R4.2.4 has some extra overview options for it.

 

FYI, the difference between "shitty overall wr" and a "single gpu wr" is over 100p: 59.8p (eg: 3x CPU wprime) versus 168p (eg: 1x GPU 3DM06).

[K404]

@ RB.... Either too many points for #1 or not enough for #10

 

Massman and me spoke about this once.... what if the points increase was smoother, but with "bumps" at.... eg.... 100, 50, 20, 10 and 5

 

 

@PJ.... at the moment, there is a difference. If there is enough of a reason "for the masses" to bench the category then the points gap will close, you know that

Edited November 12, 2011 by K404

[Massman]

@PJ.... at the moment, there is a difference. If there is enough of a reason "for the masses" to bench the category then the points gap will close, you know that

 

?

 

It's 'unreasonable' that the points gap closes when more people compete in a ranking?

[matose]

My profile says I got 12 points from competitions but I cannot find any details about that.

I'm reffering to MOA, GOOC, Country Cup, etc. Where can I see what points I have and how can I add the competitions I've won?

 

FYI, the difference between "shitty overall wr" and a "single gpu wr" is over 100p: 59.8p (eg: 3x CPU wprime) versus 168p (eg: 1x GPU 3DM06).

 

Yes, but 2 x "shitty overall wr" = "single gpu wr" and that's not exactly fair in my book. Single GPU WR takes much more work to obtain that 2 x 1-core WPrime WR's for example

Edited November 12, 2011 by matose

[K404]

?

 

It's 'unreasonable' that the points gap closes when more people compete in a ranking?

 

I honestly don't know.

 

 

Basing points purely on popularity....... if there were 10,000 results each of 3DM11 single card and 3xCPU wP32M.... are they equally impressive achievements that deserve equal points?

 

 

Should the least popular global leaderboard be worth 1/3-1/2 of the points of the most popular?

 

 

Not commenting, just giving ya food for thought

 

 

EDIT: Matose and me are thinking along the same lines. Quick, everyone take cover!

[Massman]

My profile says I got 12 points from competitions but I cannot find any details about that.

I'm reffering to MOA, GOOC, Country Cup, etc. Where can I see what points I have and how can I add the competitions I've won?

 

An overview is coming in R4.2.4 and will be visible under the points tab in your profile. Example:

 

 

For MOA and GOOC we need additional code to support competitions that are not based on the hwbot database. The code is planned for R4.3 release.

[Massman]

Yes, but 2 x "shitty overall wr" = "single gpu wr" and that's not exactly fair in my book. Single GPU WR takes much more work to obtain that 2 x 1-core WPrime WR's for example

 

Yes, but you only have 15 slots for Global points for your league ranking.

[matose]

You're missing the point Pieter .

If I have 6 x "shitty WR" and you have 3 x "single-card WR" we will have the same points, this makes me as good as you?!

[K404]

Why? If there's only a few results in the category.......

 

Because of kinda random categories like the (AMD) 3x CPUs, it's *almost* the same as the hardware ranking. That's not popular, so it only gets a few hardware points. Why should a category with 292 results be worth 60 global points? The PheII 555 only gets 5.8 hardware points

 

 

I have the Unigine DX9 world record and that category has 626 results.... zero ponts. It's a world record isn't it?

 

(I don't want points added for Unigine DX9, but I hope you see my point)

[TaPaKaH]

so, now we're coming back to CPU binning, aren't we?

found a good GT => benched 03/05/06/AM3 with 2/3/4way on air, got much higher globals than before

[Massman]

The points don't define how good someone is 100% accurately; it's merely an estimation using some algoritm that weighs in the amount of competition and the position in a certain category. The idea of "good" is very subjective and everyone has a different opinion about it. For you, "good" may mean having a Single-GPU record, for others "good" might mean scoring top positions on various platforms and architectures.

 

Let's take some examples of those so-called "shitty" records:

 

- Single core Wprime 1024M record (sempron over 6.1GHz)

- Kingpin's 12xCPU Wprime32M record (2 cpus at 5.6G+)

- SF3D's 3xGPU 3DMark11 score (3 GPUs over 1350MHz core)

- Mtech's 4xCPU PCMark05 record (overall record)

 

Each of these scores have some level of difficulty here, but are awarded too little points because there's no competition in the ranking. If someone would try to beat any of these scores, it would take a lot of time and effort to get there. Now, if someone were to be the holder of these 4 records, it would give that person ~ 240p.

 

Do you honestly believe that, for instance, Single GPU 3DMark06 + Single GPU 3DMark05 (together ~320p) is actually underrated in comparison to those 4 other records?

 

How much importance you attributed to a certain ranking depends on how you personally feel about benchmarks, how the benchmarks behave (cpu-limited 3D benchmarks), etc. We all know how the Sandy Bridge has affected the benchmarking scene or how Gulftown affected the scene.

[TaPaKaH]

people won't aim for those categories if they want points ... they'll just go for the multiCPU server access and Gulftown binning

[knopflerbruce]

You're missing the point Pieter .

If I have 6 x "shitty WR" and you have 3 x "single-card WR" we will have the same points, this makes me as good as you?!

 

Even the "shitty" WRs are hard to beat (unless we're talking about 24x wprime and stuff like that). It's not fair that you can buy some random gulftown and a gtx580, ln2 both, and get plenty of globals in all 3dmarks with no effort, whereas you get only a handful of boints when submitting a score that's actually really good - for example a wprime 3x gold cup. Being top 10 is quite easy in some of those rankings, but making it to top 3 requires alot of skill and effort, so it's a good thing that such submissions will receive an amount of boints that shows the quality of the result.

 

Edit: this shouldn't apply to ALL rankings, only where you can actually overclock - we don't have to give 60 boints to a server ran at stock, that's just silly. But anything up to and including 12x CPU - yes

Edited November 12, 2011 by knopflerbruce

[Hondacity]

question by example...for this thread.

 

e8600 wprime 2core 1st place, 60pts?

i5-670 wprime 2core 1st place, 60pts?

 

those are two different classes imo.

[knopflerbruce]

question by example...for this thread.

 

e8600 wprime 2core 1st place, 60pts?

i5-670 wprime 2core 1st place, 60pts?

 

those are two different classes imo.

 

I don't think this change is about the core/thread discussion, so I assume we still count cores, and nothing else => both are considered to be dual cores, and therefore 2x.

[Massman]

As we speak, we're re-calculating all the submissions on our test server with adjusted TPP algoritms. The updated algoritm has fixed the TPP Global bug and has some additional weight finetuning.