Possible upcoming y-cruncher v0.8.x Tweaks

[Mysticial]

Developer of y-cruncher here. Looks like a lot has changed since I've last visited.

2.5b is now a thing, but only under BenchMate presumably because I never gave Hwbot the decryption keys for it. (Yes, the program has decryption keys for all the benchmark sizes, but I only submitted the ones for 25m, 1b, and 10b into the system when I first added the benchmark years ago.)

Anything I can do here to make things compatible again? (or just just drop the submitter since BenchMate looks superior in every way)

Staff can DM or email me to take this offline.

--------

In any case, the reason why I'm here is that y-cruncher v0.8.1 is coming out soon (1-2 months?). Internally a bunch of stuff is getting rewritten so there will be large performance swings from v0.7.10.

One of the things I'm considering changing is to expose more of the program's tuning parameters. For example, each of the binaries have tuning tables that are currently hard-coded. Here is a (semi-contrived) example for cache management:

//         size,       sharing,  explicit/implicit, flags
{
    {(upL_t)512 << 10,      1,  true, LoadType::NORMAL, StoreType::NORMAL, PrefetchType::PREFETCH_T0},
    {(upL_t)1408*18 << 10,  36, true, LoadType::NORMAL, StoreType::NORMAL, PrefetchType::PREFETCH_T1},
    {(upL_t)-1,             0,  false, LoadType::NON_TEMPORAL_PREFETCH_NTA, StoreType::NON_TEMPORAL, PrefetchType::PREFETCH_T2}
};

Don't understand it? That's the point. The average person won't know what to do when thrown into a 747's cockpit.

Many of these (hard-coded) tables are extremely specific to the set of processors that I tune the binaries on. And even within the same processor class one set of settings which are optimal for one chip may be actively detrimental to another. For example, the above is for the Skylake X 7980XE. But it is actively detrimental to the lower core-count chips of the same line.

To summarize, the purpose of exposing more settings is to allow users who know what they're doing to improve performance. However, these parameters are difficult to touch if you don't know what you're doing. Thus it inherently gives an advantage to those who know what they're doing. You can think of this as a motherboard suddenly exposing secondary and tertiary DRAM timings for an OC competition. But instead of for the hardware, it's for the benchmark itself.

So what are people's thoughts about exposing these internal benchmark parameters? How much havoc would it wreck on competitive OC'ing?

Intuition tells me that these internal settings will be welcomed by the folks chasing Pi size records since those tend to run for many months at a time. (Yes, those many-trillion digit computations that require petabytes of storage.

 

 

[Jor3llBR/Elano]

Hello, I really appreciate the work you put in y-cruncher great benchmark (and shortcut for initial overall system stability test)!

I believe exposing settings for a selected few to tweak/code the benchmark thus changing the output of their scores is a terrible idea and will probably not be condoned by HWBot (they will just keep using the older version).

Every time one can 'tweak/code' the benchmark software itself to gain unfair advantage compared to regular benchers is a bad idea in my opinion. I see HWBot has come a long way changing the rules slowly to level the playing field. Sure there are still old benchmarks giving global points w/ these unpublished unfair tweaks (some of them even at the hardware level) but I believe the path forward will be to stay away from those. 

 

 

[Mysticial]

Some other mitigations to consider if such parameters were exposed.

• Require HWBOT submissions or per just within the scope of a competition to run with certain parameters locked. This forces a level playing field.
• If parameter changing is allowed, require that all submissions either show the parameters or have them downloadable for others to copy.

[Leeghoofd]

Initial thoughts.:

We will probably create a new benchmark for this new version if the performance difference is too big. From a programmers point better coding and thus generating a performance boost is one thing. For a database like HWBOT this is an utter nitemare. Plz provide asap a test version so we can see how it scales versus the previous y-cruncher. 

If you want to expose new settings plz make them available to everyone via a gui, thus not only for the software wiz kids so everybody can try them out. 

Any thoughts on other predefined bench settings like 2b, 6b and 12b? 

 

[Mysticial]

Quote

We will probably create a new benchmark for this new version if the performance difference is too big. From a programmers point better coding and thus generating a performance boost is one thing. For a database like HWBOT this is an utter nitemare. Plz provide asap a test version so we can see how it scales versus the previous y-cruncher. 

Yes it will be large. Potentially double-digit % in both directions depending on the processor. In particular I'm removing a lot of optimizations for things that are more than 10 years old because they are no longer relevant to the new stuff and are just a maintenance burden for development.

Will be a while before there's build ready as it's not done yet and there's still an tremendous amount of testing to do. Mostly likely, things will roll out incrementally over multiple releases of v0.8.x.

I can hold back HWBOT submitter support on v0.8.x until things or ready. Or retire the submitter and hand it off to BenchMate.

Quote

If you want to expose new settings plz make them available to everyone via a gui, thus not only for the software wiz kids so everybody can try them out.

Will probably be a config file for starters since there's no UI built into the main program. I'm currently 50/50 on whether to do this as it will depend on how much the parameters actually affect perf. It won't be in first release (v0.8.1) since there's already enough complexity to validate and test.

Quote

Any thoughts on other predefined bench settings like 2b, 6b and 12b? 

You can kinda already do this. BenchMate sends a command to the program. Instead of the "bench" command, you can send the "custom pi -dec:2b" command and it'll do it. I may need to add some extra lines to the spot check files if there aren't already entries for it. This won't work with the HWBOT submitter though.

[Mysticial]

To expand on your idea of splitting out separate versions for every major perf change, you could also keep an "anything goes" category where you're allowed to use *any* version of y-cruncher. That way old chips can use old versions that still have optimizations for them while there's no holding back to the latest either hardware or benchmark tweaks.

[Leeghoofd]

By sekecting/splitting the selectable benchmarks  you get no mixups btn versions. 

Due to the announced perf differences this version will 99% get it's own benchmark category. If you change the selected computation values it might become more challenging as you tend to focus on newer aka faster hardware. 

Keep us posted how it goes plz 

[Apfelkuchen]

Having a different version sounds like the most sane approach. I like having some options to play around with, but not a fan of having too much freedom of how the benchmarking software operates as that changes the focus on tweaking hardware settings to a more programming-focused approach (or just endless brute-forcing). Which isn't per se bad, but kind of a different hobby for many here I'd say.

[Domdtxdissar]

On 6/5/2023 at 6:27 AM, Leeghoofd said:

Initial thoughts.:

We will probably create a new benchmark for this new version if the performance difference is too big. From a programmers point better coding and thus generating a performance boost is one thing. For a database like HWBOT this is an utter nitemare. Plz provide asap a test version so we can see how it scales versus the previous y-cruncher. 

If you want to expose new settings plz make them available to everyone via a gui, thus not only for the software wiz kids so everybody can try them out. 

Any thoughts on other predefined bench settings like 2b, 6b and 12b? 

 

Pretty much agree, think best option is to make a new y-cruncher benchmark version.

Will also maybe suggest to drop the 25M ranking as its way too fast to complete on today's computers. And since minimum 32GB is the "norm" for regular systems sold today, maybe make a ranking for this highest "B" that fit into this buffer ? (dont think if that something like 6B or 8B)

 

[_mat_]

I will integrate into BM whatever the HWBOT community decides on. Separate benchmark categories is the best approach in my opinion to retain score comparability. I can add both y-cruncher version to BM at the same time. It might be best though to release the new version as a new major release to avoid confusion. Maybe it's time to bump to y-cruncher 1.x.

The current categories would be renamed to y-cruncher 0.x and the new ones to y-cruncher 1.x or something like that.

As for additional tuning options, I can also expose these in the y-cruncher wrapper as well as in the result dialog (for comparability, reproduction and fairness). This could be fun. It is basically the same thing as GPUPI's batch and reduction size. You can tune the benchmark so it can make better use of the potential of the hardware.

[Splave]

yay more benchmarks

[Leeghoofd]

2 hours ago, Splave said:

yay more benchmarks

yes, but let us remove some 3D legacy ones first.

[Mysticial]

Another thing to keep in mind is that not all the perf changes will be coming at once. It will *start* with v0.8.1. But, more optimizations may be coming in the following releases.

It's a very big rewrite and thus will not be done all at once. I've done enough to reach parity on the Bulldozer line (the oldest chip I still care about). So I'm drawing the line there for v0.8.1. There's still optimizations which have not been done. If and when I do them is TBD but will be after v0.8.1 (I estimate in the v0.8.3+ timeframe). For now, I'm shifting priority to finishing other parts of the rewrite like swap mode (which isn't relevant to HWBOT), then coming back later to do optimizations.

That said, I expect the big change to be v0.7.10 -> v0.8.1. I don't believe what's left to be more than single-digit % swings, but I have surprised myself before.

I expect v0.8.1 to be out within a month assuming nothing major shows up during testing. OTOH, I have no timeline for the 2nd round of optimizations. Though the goal for now is to coincide with Zen5. (which could be early again if AMD sends me one like last time)

[klopcha]

Leeghoofd: When will it become available to sumbit 2.5B y-cruncher  without BM ?

 

[Mysticial]

y-cruncher v0.8.1 got released yesterday. I've intentionally withheld HWBOT support until we decide what to do here.

[Leeghoofd]

It will be listed as a new Y-Cruncher benchmark category, eg Y-cruncher2

which versions do you think we need to run as I would go for 2.5, 5 and 10B

@Mysticial Your choice if cooperating with Mat from BenchMate and/or also allow straight submitting from your benchmark

[_mat_]

Ok. I need the exact names of the new categories and the submission keys as usual please.

Both benchmark versions will be added to BenchMate and automatically upload into these categories.

New BM 11 beta is btw ready. I can add this asap.

[Mysticial]

2 hours ago, _mat_ said:

Ok. I need the exact names of the new categories and the submission keys as usual please.

Both benchmark versions will be added to BenchMate and automatically upload into these categories.

New BM 11 beta is btw ready. I can add this asap.

I haven't actually picked those yet since they're normally set by the HWBOT submitter app. (which I haven't updated for v0.8.1)

Question: Do we want to continue with the dual HWBOT submitter + BenchMate support? Or do we want to move forward with only BenchMate?

If the latter, then you are free to pick your own submissions keys. (I assume you're parsing the validation file right?)

If the former, then I'll need to time to update the HWBOT submitter app with the new set of keys.

[_mat_]

It's done, y-cruncher 0.8.1 will be in the upcoming BenchMate 11 Beta.

[_mat_]

2 hours ago, Mysticial said:

If the latter, then you are free to pick your own submissions keys. (I assume you're parsing the validation file right?)

Normally @Leeghoofd needs to send the keys, because I think all benchmark developers lost access to the configuration of their benchmarks. The request for the keys was actually for him. Sorry, should have stated that cleary.

Congrats on the new release! It works well so far. ❤️

[Mysticial]

That was fast. ?

Judging by the poll in the other thread, it looks like things might be staying the same.

The HWBOT submitter has some formula that generates the submission keys for all the sizes. (obviously not hard coded) So it will let you try to submit any size. Except that the 2.5b size was added to HWBOT without coordinating with me so it uses a different key from the one I would have entered into the system. (as you mentioned, we no longer have access to the benchmark configs)

If we decide to extend the sizes on the same benchmark, I can send you and @Leeghoofdthe keys for all the sizes to use in the future. (everything from 25m all the way to 1t+)

[_mat_]

It was perfect timing. I am giving finishing touches to BenchMate 11, but new stuff always comes up.

I just rechecked my code and all y-cruncher keys are dynamically generated as well. Even the hardcoded ones are stored encrypted and are dynamically processed to at least avoid being recovered from pure static analysis. It's a lost game though, because the submission api was not designed well security-wise.

I personally would be thankful if you could add all new categories to the submitter (so at least 2.5b depending on the outcome of the poll). BenchMate needs somewhat modern hardware to be able to do the additional timer measurements for skew detection. It's fine starting with Bulldozer and Nehalem, but generations before that are very hard to support. There will also never be XP support. So it would help me a lot if there would be an additional way to submit scores.

[Mysticial]

How difficult would it be to change the 2.5b submission keys to keep everything consistent? /cc @Leeghoofd

I can give both of you the formula I used to generate the keys for all y-cruncher submissions - of all sizes.

[Mysticial]

Quote

Even the hardcoded ones are stored encrypted and are dynamically processed to at least avoid being recovered from pure static analysis. It's a lost game though, because the submission api was not designed well security-wise.

This is fundamentally not solvable. You would need something similar to what the Nintendo Switch does with a dedicated security chip that does the encryption/decryption. The chip itself has the key burned into it and all use of the key is inside the chip itself.

I'm not sure if TPM can be utilized to make this possible. The main problem here is getting the BenchMate key into the TPM in a way that cannot be eavesdropped by a listener. You could probably do it using public-key encryption, but you can't easily stop someone from impersonating the TPM and tricking BenchMate into sending it the encryption key which can then be decrypted and recovered.

 

 

 

[_mat_]

It can be mitigated at least with a server-side approach and user accounts. Both are available here.
Each user has an asymmetric key; one stored in the database, one is downloaded during authentication and is stored locally. The files are encrypted with your local key and decrypted on server-side with the asymmetric key. If your key gets lost or you do something bad, your user and all your results will be blocked.
It also solves the problem of sharing result files with other users. You can only uploaded the encrypted files into your own account.

We already implemented it for BenchMate, but I don't have the time to build a platform around it and probably never will. So I am unsure what to so with the securely uploaded results.