Math turns benchmark: y-cruncher meets HWBOT

[Mysticial]

I'd like to bring back the topic of HPET platform timers and open up a discussion about virtual machines (VMs).

 

y-cruncher submissions to HWBOT currently require either HPET or ACPI for Windows 8 and 10. But I've come to realize that there are other platform clocks out there. When you use bcdedit to force on the platform clock, you're not guaranteed to get either HPET or ACPI.

 

My very own laptop seems to use an unknown platform clock which I've been unable to identify. Since neither y-cruncher nor the HWBOT submitter is aware of this clock, it blocks me from submitting benchmarks from my laptop.

 

The broad problem is that only the TSC clock is affected by the clock skew on Windows 8+10. Does anyone mind if I change clock enforcement to blacklist TSC rather than whitelist HPET and ACPI? This should allow all other platform clocks.

 

The downside of this is that it also allows VM clocks. Some of the Linux benchmarks that I've gotten from people use the "xen" VM clock. But that seems to be mostly a Linux thing. VMs in general are difficult to tackle since they (by design) attempt to trick the software into thinking it's running natively. So everything from clocks to instructions can (theoretically) be faked. y-cruncher v0.7.1 will attempt to detect if it's in a VM. But it currently doesn't block VM benchmarks because:

1. They aren't a problem right now. I'm not aware of anyone using customized VMs to cheat benchmarks.
   
2. The VM detection is not 100% reliable since VMs intentionally make themselves difficult to detect.
   

 

Speaking of Linux, the submitter currently does not block any Linux submissions regardless of the reference clock. So you are (technically) allowed to run benchmarks in Linux, take a screenshot there, then transfer the files to Windows and submit to HWBOT. But given that there's no CPUz for Linux, this probably won't work for the majority of the competitions.

Edited September 18, 2016 by Mysticial

[_mat_]

The broad problem is that only the TSC clock is affected by the clock skew on Windows 8+10. Does anyone mind if I change clock enforcement to blacklist TSC rather than whitelist HPET and ACPI? This should allow all other platform clocks.

I've researched this topic for some days back when I was developing GPUPI and in my opinion the only manageable option for me was to ban TSC from 8 and 10 and only allow HPET there. Using ACPI as a timer if available is possible but depends on how it's done. I would not advise to use Windows' QPC functions for example. Edited September 18, 2016 by _mat_

[Mysticial]

I've researched this topic for some days back when I was developing GPUPI and in my opinion the only available option is to ban TSC from 8 and 10 and only allow HPET there. Using ACPI as a timer if available is possible but depends on how it's done. I would not advise to use Windows' QPC functions for example.

 

ACPI + QPC is vulnerable to clock skew? I admit that I have no way to test this since the only box I have that lets me disable HPET (to force ACPI) is my AMD box - which isn't affected IIRC.

 

y-cruncher uses more than just QPC. So even if QPC gets skewed, it theoretically should still be okay. I can PM you the details if you're interested. Either way, I have no way to test this.

[_mat_]

QPC falls back in various ways depending on the hardware and OS version, it can be HPET, ACPI or even RTC. The timer resolution of QPC differs greatly as well so it's difficult to find out which timing method is currently in use.

Edited September 19, 2016 by _mat_

[Mysticial]

QPC falls back in various ways depending on the hardware and OS version, it can be HPET, ACPI or even RTC. The timer resolution of QPC differs greatly as well so it's difficult to find out which timing method is currently in use. That's why I decided to avoid QPC, it's too unreliable for benchmark measurements. Read around the web, software like VirtualDub banned it as well.

 

I think we might be talking past each other. I'm fully aware that QPC varies depending on the clock. That's sorta the whole point of this discussion. It's easy (for me at least) to determine whether QPC is backed by HPET, ACPI, TSC, or none of those. But you implied in your previous post that QPC is vulnerable to skew when backed by ACPI.

[_mat_]

Oops, I meant that I am avoiding QPC when HPET is not enabled. Sorry, I have currently a lot on my plate.

 

I can't remember if it's precisely ACPI that's vulnerable, but on Windows 7 - which is not affected by the RTC bug - QPC gets skewed if HPET is disabled. My best guess is, that it falls back to ACPI, otherwise the fallback to RTC would not produce skewed results. See my results here: https://www.overclockers.at/articles/gpupi-2-1 ... I should have displayed the timer frequencies as well, hrmpf.

[Mysticial]

Btw, I've recently received some benchmarks on a 68-core (272 thread) Knights Landing Xeon Phi:

• 1 billion digits: 41.844 seconds
  
• 10 billion digits: 504.873 seconds
  

As far as I can tell, these are world records for single-socket processors. And despite running at a stock 1.4 GHz, it edges out all the LN2 systems.

It still falls short of the really big Haswell-E and Broadwell-E dualies. But the program is completely untuned for the Xeon Phi. So there's probably a lot more performance that can be squeezed out from just software alone.

 

I won't submit these to HWBOT since they were done in Linux using the unreleased AVX512-CD binary. The rules currently require Windows with a screenshot. And I prefer to keep benchmarks of unreleased versions separate from the released stuff.

 

The validation files for these with all the details are available from my website if you want to take a closer look. Just don't try to steal them and submit them to HWBOT. The submitter app can't distinguish a legit benchmark from one stolen from someone else. So it will actually allow such a submission.

[IvanCupa]

Hello, I've playing little bit for the benchmark. And i have some question.

 

1. For HWBot submission, It's allowed to run the benchmark via standalone binaries instead of HWBot submitter launcher?

And It's allowed to use Screenshot created by another apps for making the datafile?

Since both of them is possible for now, right?

In GPUPI / HWBot x265 saving the datafile will capture the screen & only the screenshot from datafile is valid.

 

2. The datafile is created by an TXT validation file + screenshot (or any images file ).

And created when clicked 'Submit to HWBot', deleted by the apps after uploading.

 

It's possible to directly make / save (multiple) datafile (With Screenshot included) after benchmarking?

I mean, after the benchmark finished, there's no TXT validation file appear first. And the datafile created when I click 'submit to HWBot'. And screenshot is captured & included in the datafile directly when I click 'submit to HWBot'.

I'm aware, if you implement this idea the benchmark must run from the HWBot Submitter. And my first question above is no longer valid.

 

or

 

I'ts possible to make multiple validation file (TXT) created by the binaries, to make datafile, after the benchmark running?

Which is easier?

My concern it's, we are talking on OCed system, that anything can happen suddenly. Corrupted TXT Validation file, problems when saving, crash, etc.

 

3. Why you make the datafile deleted after submitting to HWBot?

I think it's better to keep those file. Make good datafile naming mechanism / scheme. Such as (preset - time.HWBOT). It's related to my idea in 2nd question.

I still can keep it by copying the datafile when it was made though. But it's useless for now since manual submitting doesn't possible.

 

4. Make manual submit possible.

Many integrated HWBot benchmark could do this, HWBot Prime, GPUPI, HWBot x265 or even XTU, etc. With no issue i think.

It's funny (at least for me) if you willing to submit from another computer you must copy the entire folder which contains the application, and run the apps on another system just for submit. In fact in the end we had to open the browser also to fill in complete submission data form on the HWBot web to finish the submission processes

 

Many thanks, regards.

[Mysticial]

Hello, I've playing little bit for the benchmark. And i have some question.

 

1. For HWBot submission, It's allowed to run the benchmark via standalone binaries instead of HWBot submitter launcher?

 

Yes. The submitter already lets you do this, but you're also free to manually run them. You are also free to fully customize the settings from within the y-cruncher application itself. You aren't required to run the benchmark from the submitter.

 

And for that matter, you may need to do this anyway since the default settings (as chosen by the submitter) aren't always the best.

 

(I have a feeling that some people won't like this ability to over-customize the settings. Since it means it's not 100% about hardware anymore. Someone with a sufficient understanding of the program itself and how it works will be able to pick better settings for their hardware setup. But when the goal is to be as fast as possible, I consider that as part of the game.)

 

And It's allowed to use Screenshot created by another apps for making the datafile?

Since both of them is possible for now, right?

In GPUPI / HWBot x265 saving the datafile will capture the screen & only the screenshot from datafile is valid.

 

The official rules say you need to use the built-in screenshot. But that's impossible to enforce since the built-in screenshot is no different from any other screenshot. (The submitter doesn't watermark the screenshot.)

 

 

2. The datafile is created by an TXT validation file + screenshot (or any images file ).

And created when clicked 'Submit to HWBot', deleted by the apps after uploading.

 

It's possible to directly make / save (multiple) datafile (With Screenshot included) after benchmarking?

I mean, after the benchmark finished, there's no TXT validation file appear first. And the datafile created when I click 'submit to HWBot'. And screenshot is captured & included in the datafile directly when I click 'submit to HWBot'.

I'm aware, if you implement this idea the benchmark must run from the HWBot Submitter. And my first question above is no longer valid.

 

No it's not possible. The submitter is completely separate from the main y-cruncher application. This is by design and I have no intention to merge them.

 

The y-cruncher application is primarily used outside of HWBOT and it has its own validation system. The submitter is only responsible for integrating with HWBOT. I admit that the integration is less than ideal, but it's the best that can be done without being too intrusive to the main y-cruncher project and with the limited man-power that I have.

 

 

or

 

I'ts possible to make multiple validation file (TXT) created by the binaries, to make datafile, after the benchmark running?

Which is easier?

 

Can you rephrase that? I don't think I understand. The benchmark already makes a new validation file for every computation. And the datafile is made by the submitter - which is after the computation is done running.

 

My concern it's, we are talking on OCed system, that anything can happen suddenly. Corrupted TXT Validation file, problems when saving, crash, etc.

 

How is that different from any other benchmark? And isn't that just part of the game? Realistically speaking, if the overclock can pass a y-cruncher benchmark, it'll have a pretty decent chance of handling a screenshot as well.

 

3. Why you make the datafile deleted after submitting to HWBot?

I think it's better to keep those file. Make good datafile naming mechanism / scheme. Such as (preset - time.HWBOT). It's related to my idea in 2nd question.

I still can keep it by copying the datafile when it was made though. But it's useless for now since manual submitting doesn't possible.

 

4. Make manual submit possible.

Many integrated HWBot benchmark could do this, HWBot Prime, GPUPI, HWBot x265 or even XTU, etc. With no issue i think.

It's funny (at least for me) if you willing to submit from another computer you must copy the entire folder which contains the application, and run the apps on another system just for submit. In fact in the end we had to open the browser also to fill in complete submission data form on the HWBot web to finish the submission processes

 

Many thanks, regards.

 

Forget about the HWBOT datafile. You aren't supposed to know or care about it. If you're seeing it at all, that probably means that a submission failed. And when that happens, it won't delete it so that you can send it to me as a bug report.

 

Instead of the HWBOT datafile, you should care about the validation .txt file. With that (and a screenshot), you can submit from anywhere.

 

Right now, manual submission is disabled because:

1. You're not supposed to see the HWBOT datafile anyway.
   
2. The server doesn't check if the score you submitted is to the right benchmark. In other words, you can submit a 25m score into the 1b bracket.
   

Edited October 24, 2016 by Mysticial

[P5ych0]

Hi Guys....

 

 

i can´t submit any scores with hwbot submitter on Win 7 because the log Window don´t show me scores. .txt data file is still there

So whats wrong ?

[Mysticial]

Hi Guys....

 

 

i can´t submit any scores with hwbot submitter on Win 7 because the log Window don´t show me scores. .txt data file is still there

So whats wrong ?

 

Can you clarify the problem? If I'm understanding you correctly, the validation file is there (i.e. "Pi - 20160916-194531.txt" ). But it doesn't show up in the submitter?

Edited October 30, 2016 by Mysticial

[P5ych0]

Can you clarify the problem? If I'm understanding you correctly, the validation file is there (i.e. "Pi - 20160916-194531.txt" ). But it doesn't show up in the submitter?

Exactly.

Is there another way to submit without Hwbot submitter?

 

Cheers...

[Mysticial]

Exactly.

Is there another way to submit without Hwbot submitter?

 

Cheers...

 

No there isn't. The submitter is the only way right now.

 

Can you PM me the validation .txt file? I'll take a look.

[P5ych0]

No there isn't. The submitter is the only way right now.

 

Can you PM me the validation .txt file? I'll take a look.

 

PM ed you :-)

[Mysticial]

PM ed you :-)

 

I found the problem. The submitter is bailing out when trying to read the memory configuration.

 

   DIMM:              4.00 GiB -  - 
   DIMM:              4.00 GiB -  - 
   DIMM:              4.00 GiB -  - 
   DIMM:              4.00 GiB -  - 

 

It's supposed to look something like this:

 

   DIMM:              8.00 GiB - Undefined - CMK32GX4M4A2400C14 
   DIMM:              8.00 GiB - Undefined - CMK32GX4M4A2400C14 
   DIMM:              8.00 GiB - Undefined - CMK32GX4M4A2400C14 
   DIMM:              8.00 GiB - Undefined - CMK32GX4M4A2400C14 

 

Java's String.split() doesn't behave the way I expected it to when the fields are empty. The submitter skips any file that it has trouble reading so that it won't error things which are not validation files. Since it had trouble reading this file, it thinks it's not a validation file. Hence why it doesn't show up.

 

 

I'll roll out a fix in a bit.

[Mysticial]

Fixed rolled out. (HWBOT Submitter v0.9.5.107.jar)

 

In addition to fixing the memory parsing thing. I've also relaxed the reference clock restrictions. Rather than explicitly whitelisting only HPET and ACPI, it now only blacklists TSC. This solves a problem on my laptop which uses some platform clock other than HPET and ACPI. Hopefully this won't cause any problems.

[P5ych0]

Fixed rolled out. (HWBOT Submitter v0.9.5.107.jar)

 

In addition to fixing the memory parsing thing. I've also relaxed the reference clock restrictions. Rather than explicitly whitelisting only HPET and ACPI, it now only blacklists TSC. This solves a problem on my laptop which uses some platform clock other than HPET and ACPI. Hopefully this won't cause any problems.

 

thanks for your work man, i will check it.....

 

i also found something... i flashed back from Broadwell E to Haswell E Bios and it Works....

[PKBO]

How can i save data file for future submit ? drivers for lan doesnt exist under XP.

[Mysticial]

How can i save data file for future submit ? drivers for lan doesnt exist under XP.

 

The program won't run at all on Windows XP. Support for Windows XP was dropped more than 4 years ago.

 

You will need a minimum of Windows Vista to run the program at all.

You need Windows 7 SP1 to be at all competitive since that's the earliest version of Windows that supports AVX.

You'll want Windows 8 or 10 to get the best performance since the schedulers are much better.

 

 

But to answer your question directly, you can transfer the validation .txt file and the screenshot to a different computer and submit from there.

[PKBO]

Yup, thx for reply, my question was little bit faster than i cant research it myself offcourse afert a few hours left by installing useless XP here.

[DrWeez]

Can you clarify the problem? If I'm understanding you correctly, the validation file is there (i.e. "Pi - 20160916-194531.txt" ). But it doesn't show up in the submitter?

 

i seem to have run into this issue,

 

checking the file it does detect the memory config.

 

file attached

[Mysticial]

i seem to have run into this issue,

 

checking the file it does detect the memory config.

 

file attached

 

Thanks for the report. I'll take a look when I get home tonight.

[Mysticial]

i seem to have run into this issue,

 

checking the file it does detect the memory config.

 

file attached

 

That validation file works fine for me. The only thing unusual about it is that it's using an unrecognized platform clock. The newest version of the submitter will allow it, but older versions won't.

 

Are you using version 0.9.5.107?

[DrWeez]

That validation file works fine for me. The only thing unusual about it is that it's using an unrecognized platform clock. The newest version of the submitter will allow it, but older versions won't.

 

Are you using version 0.9.5.107?

 

yes 0.9.5.107

[Mysticial]

yes 0.9.5.107

 

Well. That validation file works fine for me. I'm able to actually submit your score. So I can't reproduce your problem. Can you elaborate on exactly what the issue is?