Single-Threaded benchmarks with part disabled CPUs

[mickulty]

Hi folks,

Since alder lake P-core categories have been added, and FX 1 core per CU categories have begun to be rolled out, I think clear guidance is needed on how to treat single-threaded CPU benchmarks (and validations, for that matter).  Conventionally it's very common to disable cores for lightly threaded and singlethreaded work, but we're now in a situation where disabling idle cores can change the hardware category even though the work is being done by the same cores.  Currently there seems to

There are two approaches I can think of that make sense;

1. Mandate that fully enabled scores go in the base hardware category, and scores with only P-cores enabled go in the P-core category.  This would make pretty much the same OC of the same hardware, likely getting pretty much the same score, eligible for multiple categories.  On the plus side, one could argue it would represent what the 1T capability of the hardware is in different configurations.
2. Mandate that validations and 1T benches only go in the base category.  This would probably be more enforcement effort in the short term, but in the long term the site could perhaps be written to handle it well (for example, removing P-core categories from the hardware list when uploading).  On the plus side, you'd avoid having multiple rankings for the same hardware.

Currently there seems to be some confusion, for example at least one bencher has what seems to be an 8P superpi 32M score in the base 12900K rankings while also having a score in the 8P rankings.

I don't have a strong opinion, but my general inclination is that people have always disabled cores for lightly threaded benchmarks and it would be silly for it to just now start effecting the hardware category (therefore option #2 would be better).  However I mostly just would like there to be a clear policy.

It would be presumptive to start a poll myself, but perhaps one is called for?

[denvys5]

24 minutes ago, mickulty said:

likely getting pretty much the same score

On the other hand, these scores may become more different over time. Currently, disabling e-cores allows for higher ring clocks and a bit better p-core clocks.

In some time, we may see 1T runs with, e.g., 1+1 configurations, for the best score in P+E category. I dont think that result would be directly comparable to 1P core result. Although, thats just a worse result than 1+0 in 1T by definition.

From another point of view, E cores maybe be used for score optimization. *Speculation* Like we already have 3-4 core runs in SPI/old 3dmarks, those cores may be E-cores with seperate frequency control, which may allow to clock higher single P-core (1P+4E)

But, I dont have any particular opinion on the solution for this right now.

[_mat_]

5 hours ago, denvys5 said:

On the other hand, these scores may become more different over time. Currently, disabling e-cores allows for higher ring clocks and a bit better p-core clocks

And enables AVX-512.

[yosarianilives]

8 minutes ago, _mat_ said:

And enables AVX-512.

Avx vnni is an interesting compromise for when the e cores are enabled, it's the avx512 instruction except 256 bit. Not sure if it's any faster than Avx2 or if it has more to do with allowing easier software compatibility. 

[_mat_]

2 hours ago, yosarianilives said:

2 hours ago, _mat_ said:

And enables AVX-512.

Avx vnni is an interesting compromise for when the e cores are enabled,

That's for deep learning mostly, so not really that much of use for general purpose benchmarks.

Anyways, just wanted to point out that disabling E cores has a substantial impact on CPU.

[Strunkenbold]

IMO all single core results have to go into the base category because the overclocker needs to choose which way is the fastest and we really dont want to give double points for same hardware. We should probably already start to moderate those categories and move all single thread results from the p cores categories.

Later this needs to be handled with some new code. Those double categories arent the best thing yet. We likely want to have only one category per CPU. Maybe its possible to have something similar like the unlocked K10 CPUs.

[I.nfraR.ed]

I agree single-threaded benchmarks should go into one single category. 32M is always ran on the best core anyway.

I though overclocking goes hand in hand with getting the best score/performance possible out of the the hardware in a specific benchmark/game. For single-threaded benchmarks it means you want to disable everything that affects the clocks and performance in a negative way. Leaving all 6 cores on a Thuban CPU hurts CPU-NB clocks compared to e.g. 2 cores enabled, does that mean we should add another category and assign points for both?

And I still don't get these P categories. For me it's all out on multi-threaded benchmarks for highest possible score, not artificially pushing the CPUs to categories with less cores, just because they are more competitive there (at least that's how it looks from "a distance"). I know it's not just black and white, but still can't get my head around that, sorry.

Edited November 18, 2021 by I.nfraR.ed

[Leeghoofd]

Mandating scores with E cores disabled thus to get higher core speed and better performance yet still have to go into the base category, because the OCer ran a single threaded benchmark seems counter productive to what has been decided upon.

If you didn't read it yet, it was mentioned several times already that there will be a solution for the double points and that this is all a trial. One has to analyze things before making new changes and we will have to sit it out and watch what upcoming CPU releases will bring.

Regarding submitting, besides  a few human errors,  subbing in the correct category is pretty smooth till now.

Poll no need, an open mindset might be better

[I.nfraR.ed]

Why mandate anything for the single-threaded benchmarks?

You want to run Pi 32M, it's your choice if you want to run it on P core, on E core, disable E cores, run single channel...whatever it works for you, the goal is to get best score possible with your system from the current bench session. Then sub it in the base (normal) category (no P, no E, no X category). IMO only one category should exist for benches like SuperPi, pifast, etc. If you want to run it with all cores enabled and get lower clocks, it's your choice, why we should care about that? That's what I fail to understand :p

I guess it is down to implementation, since they are listed as different processors and each processor has the full set of benchmarks?

I'm not that active anymore, so I've probably missed a lot form the ongoing discussions, so please excuse me if you read that same thing for the n-th time :d

[Leeghoofd]

Hwbot took the decision once you disable all the E cores you fall into eg 12900k(8p) category. Seems pretty easy to understand and logic.

The issue for many is that you keep on comparing disabling cores on old gen with this new hybrid design, it is not the same for those that run it here as we took into account future CPU releases. If you feel it is the same, fine for you. But I'm not gonna waste any more energy to convince people of what we try to do coz some saw their points take a plunge, even though we warned them it is due the current crap boint algorithm. 

HWBOT took this decision to go this direction. If it will be a dead end, then you can all have a giggle in approx a year from now. 

 

[Strunkenbold]

3 hours ago, Leeghoofd said:

Hwbot took the decision once you disable all the E cores you fall into eg 12900k(8p) category. Seems pretty easy to understand and logic.

Well with logic we have sometimes our problems here. Even I struggle sometimes, after all those years (ok this might be a problem I have in general).  

I always thought one of the unwritten rules was to don't create extra categories for the same named hardware even when some new revision brings new features like 64 bit or a die shrink. We always said its up to the overclocker benching with the best silicon available. Now having two categories for single thread results just because something is disabled or enabled pretty much breaks the mentioned logic. I always thought those new categories are just an intermediate step because of the limitations of the current code but if they keep existing in parallel I'm lost. Its okay to try something new with the hybrid stuff but please consider merging base and p cores categories when code is ready and let the user choose upon submission entering if he disabled e cores so the submission goes into e.g. 8 or 16 cores category just like unlocking a Phenom X2 goes into 3 or 4 cores hardware category.

[I.nfraR.ed]

@LeeghoofdThanks for answering and I believe it must be really annoying for you already.

Following the same logic, let's say I have an FX-8350 and want to run 32M. As I normally do, I will disable unneeded cores.

Does that mean I need to now submit it in the 4P catergory, as opposed to before that change where you submit in the base category regardless of disabled cores - it's up to you to tune your system to get the maximum out of it. Basically the same issue that @mickulty raised, but from the FX point of view.

PS: Or do I need to be careful what is disabled - am I running in "1 core per CU" or have I disabled one of the CUs and based on that submit in a different category?

Edited November 19, 2021 by I.nfraR.ed

[Leeghoofd]

1 hour ago, I.nfraR.ed said:

PS: Or do I need to be careful what is disabled - am I running in "1 core per CU" or have I disabled one of the CUs and based on that submit in a different category?

Ask @mickulty about it, he will explain in detail, it is his brainchild.

 

@Strunkenbold that is indeed Roman's plan, but before we will have that code readied we will be at least one CPU gen further, so we took this intermediate step and are analyzing the effects. Sorry guys I can't keep on explaining things over and over again because someone has an idea of what to do next. The state HWBot was in during the transfer to Roman and Eike and the state it is now is already day and nite. But each time Tim fixes something , usually something else will pop up. And he still hasn't touched anything for us in the Dbase to make our life easier.  For now we took this decision to split-up and that is how it will be till we can merge again or even see if it can work like this (with once points for the ST benches and not in the 2 categories) But we are busy on other stuff first than this.

[Strunkenbold]

If thats really the plan, then we don't need discuss this much further, as single core results will be in the long run in the same category anyway. There is no reason to make a difference between FX and AL as its the same logic. The base category should have all single core results and we should move results from the p cores.

[mickulty]

@I.nfraR.edAs far as I'm concerned it's the same principle, HOWEVER, there are existing scores in the base categories.  That complicates things.  Even if I made a full list, moving over all the relevant scores would still be a lot of work for some mod and I already owe leeg at least two beers.  Also I know leeg sees it as a "trial" so messing with the existing database might not be a good idea.

I think the practical answer for FX is to keep all 1T FX scores in the base category for now.  That's not based on any long-term principle, just the fact that splitting them with FX would create a lot of mode work that doesn't exist with ADL.  If the ultimate solution is something similar to phenom unlocking then it won't matter too much if there are 1c/CU results in the base rankings.

BTW you can tell if you're in 1c/CU by looking at L2 cache in CPU-Z.  Same L2 cache count as core count = 1 core per CU.

 

Edited November 19, 2021 by mickulty

[unityofsaints]

How will/would 1 core per CU be distinguished from subs that have a number of modules disabled such that the total core count is the same? Say what you will about the shortsighted decision to allow two different core counts to earn globals on Alder Lake, at least CPU-Z clearly shows the difference.

[yosarianilives]

2 hours ago, unityofsaints said:

How will/would 1 core per CU be distinguished from subs that have a number of modules disabled such that the total core count is the same? Say what you will about the shortsighted decision to allow two different core counts to earn globals on Alder Lake, at least CPU-Z clearly shows the difference.

You can easily tell by the l2 cache numbers