The official BenchMate support thread

[flanker]

5 hours ago, _mat_ said:

No, free version is enough but 0.7.1 had a problem with capturing the scores. It has been fixed already in 0.8.

Skew measurement is very different now on BenchMate, more precise and has less performance impact. I will go into details about it in the announcement post. Needs testing for sure.

PNG, noted and no problem at all. Might need to be combined with Gzip compression so the result uploading won't take forever on slow connections.

Mat, is 0.8 version ready to public now? Thanks

[_mat_]

3 hours ago, cbjaust said:

Re: Skew, I found that CINEBENCH R15 would get a skew error warning quite a bit.

That's exactly why I rewrote a huge chunk of code. CINEBENCH uses timerGetTime() which internally uses RTC. That's a very bad choice, especially nowadays where there are other, much more precise options. So what happens here is that timeGetTime() is sometimes up to 5 ms inprecise PER SECOND and that triggers the skew detection of this timer.

What I do normally for all well written benchmarks is, that I inject my hardware implementation of the HPET timer (Note: Does not have to be enabled in Windows, only via BIOS!) and completely emulate these timer calls. But that's sadly not possible with CINEBENCH because it calls this timer facility about a million times in 20 seconds. The driver emulation would result in a performance impact (small but it's there), because the HPET timestamp has to be queried via a driver IOCTL while timeGetTime() makes use of an OS syscall. Nothing is faster than that + RTC is faster than HPET as well.

To sum that up: Don't call ANY timer facility hundreds of thousands of times. There is no reason for that if the calculation code is independent from the GUI. Also NEVER use RTC ever. You will produce a slot machine where several of the same benchmark runs will end up with different results. It's horrible to tweak the last few milliseconds out of a bench like that.

Btw, even in CB R20 the timer code was not updated. It's kind of ridiculous that I have to fix the mistakes of other benchmark developers. :p

So what I do now in BenchMate 0.8 is that I inject QPC into the CINEBENCH (which should be the CPU's Timestamp Counter, TSC) and detect skewing on those timestamp. Much more reliable and no false positives so far!

[_mat_]

3 hours ago, flanker said:

Mat, is 0.8 version ready to public now? Thanks

Last breaths before it is. Please be patient!

[Bilko]

On 7/3/2019 at 7:19 PM, keeph8n said:

X399 Creation and 2950X

1903 W10 

11/14/18 drivers/bios for the mobo

Ive tried reinstalling, new downloads, etc. basically I have gotten the software to open once with this setup to do some testing. Same OS but with my 990FXA-UD3/9370 combo and it pulls up just fine. 

Basically what happens when I open the program is the cmd prompt looking box opens, does it thing, closes and then nothing. Task Manager shows BenchMate running but I get no box to pop up. I don’t show anything in the task manager keeping it from opening. 

I was having this issue as well on my TR4 system (X399 Gaming 7 + 2920X) and enabling the disabling HPET seemed to fix it but now it's back again.

Almost exactly the same process, Benchmate cmd windows pops up, loads then vanishes leaving benchmate client as a background task in task manager and that's where it ends.
About 15 minutes prior this was all working perfectly, Only thing that were changed was the desktop background and memory strap.
 

[Leeghoofd]

@_mat_ I just tried it on my two laptops, the Benchmate 0.8.0 result screen is too big (res 1920 x1080) Compare it with the above screenie (same width as  CPUZ. For 2D this is no problem, for 2D when more tabs needs to be open it might result in a lack of free space....

Maybe this is a Win8.1 issue , we had similar stuff in the simulator with some fonts and co, will try it on the desktop with Win10

 

Edited July 14, 2019 by Leeghoofd

[_mat_]

The result dialog is as big as it needs to be, it adjusts its size to the content. "CINEBENCH R11.5" is just huge as a heading.

I can make the font smaller of course. But we have to keep in mind, that the goal is to see what is going on as fast as possible.

Edit: @Leeghoofd I see it now, the font seems to be bigger on Windows 8! Ok, I will have a look at it!

Edited July 14, 2019 by _mat_

[_mat_]

Another thought on the topic as a whole: I think it's important that we keep an open mind here and try to find the best way to verify and moderate benchmark results. Not try to force the new way to adopt to the old way!

The need to see the whole benchmark window is a very good example. This is a requirement that could be easily removed by automation now. BenchMate could make a second "screenshot" of the benchmark window alone (without any overlapped windows of course) and upload it together with the full screenshot. So two screens would be visible on the result page by default and show exactly what's necessary.

BUT this would require a small update of HWBOT to allow uploading of multiple screens with one result file.

@richba5tard Do you think that's possible?

Edited July 14, 2019 by _mat_

[Leeghoofd]

The rendered scene rule has to stay, one screenshot is sufficient in my book, not going for two Matt.. Imagine a good score but the screenshot of the full rendered scene goes corrupt...

Also be aware not all OCers will opt for benchmate usage  Especially if benchmate can't support all of the benchmarks ( Wprime eg ) we can't enforce it either.  Meaning we have to enforce more rules for the benches it will support....

I see Benchmate as the ultimate tool for all RTC buggable platforms and Win8/10, Benchmate will allow users with older gear but a modern OS to submit a proper non bugged result.

 

Moderation if fine as it is, no need to redo something that works.... If changing the font makes it smaller than that is the easiest path to follow. Thanks for the effort!

 

 

 

[Splave]

Imo if it's an opt in sort of plan then pretty much scrap the idea completely. Why would I put extra effort into security so someone else can not.

[Leeghoofd]

Biggest showstopper for me is it might not support all of our current benchies Allen, Intels latest CPUs are not affected by the RTC bug, so no problem for you right?

If we want to include the AMDs, then we will enforce benchmate,usage. But they will only be able to bench what the Mate application will support on the chosen OS.

The Tool is looking great but it is still early days befor global usage

Edited July 14, 2019 by Leeghoofd

[_mat_]

To spoil it for everybody here in the community: Nearly every benchmark on HWBOT is easily cheatable by just downloading a single, publicly available tool. Apart from that there are a lot of other attack vectors (some very easy to find and abuse, some harder), that threaten the already non-existing credibility of screenshot benchmark results. I'd also like to clarify that GPUPI is not secure in any way in its current state. In retrospective it's a pile of shit, although sadly miles ahead of other benchmarks like wPrime or SuperPi.

Leaving aside the fact that benchmark security is in a horrible state, the same can be said for timer reliability. Yeah, there is an RTC and TSC bug for Windows 8+ on any CPU that is not Intel Skylake+. This bug has definitely an impact on some benchmarks while others might have fixed this if the code is bug free and future-proof. I assure you from my experience of supporting GPUPI for the last five years that neither is true. It's also a fact that even well-versed overclockers have a hard time to find the right choice of OS to produce valid results. And I completely get it, it's a nightmare.

Last but not least let's keep in mind that all of the above just describes our current situation. What about the future? When will the next timer produce bugged results? Do we really want to wait for the next undetectable cheat, incompatibility or bug to ruin a beloved benchmark? It's going to happen sooner or later, the question is if we want to do something about it.

So to get to the point: No, I don't think that the way HWBOT valiates results works at all.

Running easily cheatable benchmarks with possibly unreliable timers does not work.
Uploading an easily editable screenshot by hand that shows CPU-Z and GPU-Z windows that were opened after the actual run does not work.
Filling in submission data into a form by hand does not work.

I agree with you that it's too early for BenchMate to be used right now. It needs time and room for improvement. But I completely disagree that it should only be used for AMD for all the reasons stated above.

About wPrime, I think it's a bad idea to let such an important decision like the integration of BenchMate depend on an old benchmark, that's neither maintained nor bug-free or actually very special at all.

[GeorgeStorm]

Unless you've changed your mind about xp support then I don't see how benchmate would ever work as always needing to be used.

 

If anything if you're wanting to change a bunch of benches, change the way benches are run, uploaded etc, then sounds like you just want a new benchmark site

Edited July 14, 2019 by GeorgeStorm

[Mr.Scott]

56 minutes ago, GeorgeStorm said:

Unless you've changed your mind about xp support then I don't see how benchmate would ever work as always needing to be used.

 

If anything if you're wanting to change a bunch of benches, change the way benches are run, uploaded etc, then sounds like you just want a new benchmark site

+1

Thank you George.

[_mat_]

Missing XP support is not something I don't want to do, it's sadly impossible to go even half way of where BenchMate is now security-wise. It misses vital functionality like some very important kernel driver features that I need to secure benchmarks as well as BenchMate itself. Most functionality is Vista+, which was a big step in the right direction for Windows.

Windows 7 support alone was hard enough to achieve and took a lot of time. Sometimes I had to find very creative ways or code features myself so it's on par with the Windows 10 version.

Fun fact: If BenchMate would only support Windows 10 and onwards you could have started benchmarks from your desktop and they would have been safe. Not possible in any prior version of Windows though so I had to settle for integrated launches.

A big problem I see with BenchMate integration into HWBOT is, that HWBOT is not ready yet to go all the way. There are some important functions missing to bring the full concept to life. For example:

• Competitions can't be integrated into BenchMate (or anything else for that matter). According to @richba5tard on a github ticket the query API was written by a temp and never finished. At its current state it only returns how many competitions are found but no results. Same for users, benchmarks you name it.
• There needs to be a full file hash check of every important benchmark file on the server-side. It's an important part of validation. For now I have settled with showing hash numbers of some of the files that have to be compared manually. Something I really would like to get rid of.
• Important information of the result dialog is missing on HWBOT. Windows version, benchmark version, BenchMate driver and client version info, timer information, BIOS versions, enabled CPU cores and threads and so on. And I have much more things in mind here, I'm talking graphs other analysis tools that would help to understand scores better.
• The submission upload has many problems and is also missing important functionality. Authentication is done with plain text passwords, Facebook/Google authentication seems to be not supported (so whoever made an account with these, can't upload scores), auth tokens have to be inserted into the encrypted data file,  submission comments are missing, multiple screenshot upload would be necessary, ...
• BenchMate needs some version control feature so you can't upload with old clients if a bug is found.

That's just from the top of my head. So yeah, it's a good question if HWBOT can/is willing to go forward and be ready for the future. If you believe it or not, I actually don't want to develop another platform if it can be helped. But I also don't want to settle with things here just because "they always were that way and that's fine". I want to see competitive overclocking on the same level of credibility as Formula 1, skiing or pro gaming and I don't see that happening with things as they are here.

PS: i really don't want to talk bad about the moderators here. They are doing a fine job with the tools at hand. I just think that the right tools are not there yet. My very ambitious goal would be to have 99% of the results not needing any moderation, so the rest can be thoroughly examined manually with enough data to make the best decision.

Edited July 14, 2019 by _mat_

[Mr.Scott]

A lot of people would like to see that Mat. Unfortunately, a clean slate start is the only way to achieve that and I doubt you'll ever see that here.

[Leeghoofd]

As you state yourself Matt, no benchmarks is uncheatable, however  not everybody here wants to go that deep and try to break the code as you do. 

Your example of GPUPI is perfect to highlight that we need to go back in time were only a few OSses were required to have fun... and they worked for all that we threw at them.

And that's my point of view on Benchmate: it will allow users to have fun with one OS (being it Win10). If it is the most efficient one, that doesn't matter as long as they have fun and can submit and no asshole moderator pulls their scores due to e.g. some timer issue.  That is my simple perception on your great software, assuring solid modern OS compatibility with enhanced safety checks (if the benchmarks allows it)

Question: If the benchmark is unsafe (in your book)  can't Benchmate just verify e.g. the RTC bug ?  And no Wprime is not a show stopper, but to drop it, would be a shame.

You want to create the Ultimate safety and submission tool Even though applaud your hard work I fear this is utopia as software/hardware changes constantly. I have a lot of members in my team that can't get 0.71 nor 0.8 working on Win7, but all of them got it working on Win10. Why bother with 7, heck even XP? Everybody has trimmed down OSses, no idea if they want to drop those Lite versions and maybe loose efficiency on eg WinXP for SuperPI 32M

Maybe focus for now just on Win10 and try to get a few more 2D and 3D benches in. Just imagine we have to organise a Team CUP with only 6 2D benchmarks and 3 3D benchmarks in your approved benchmark suite... That would be a boring competition right? 

I can follow your perception on ultimate safe benching and automatic submitting. Now that would be my ultimate wet dream as no moderation would be required at all. But it is way too early days and to be honest it might never work here, Who hasn't seen weird stuff when benching under extreme conditions? Hard and software evolve to quick and debugging things would require it to become a full time job to keep everything up and running.

I think we both have our views on how to use the tool, mine is purely from a competitive and moderator point of view. The integration and other stuff I leave that up to you and Frederik.

 

Keep up the good work!

 

[_mat_]

The problem is that no code is necessary to break every legacy benchmark on here. A download and three clicks and you have an undetectable edge over everybody else here.

I am not living in a fairy tale where I think benchmarks or BenchMate can be unbreakable. That's especially true because benchmarking needs security with next to no impact on performance. Otherwise I would have implemented a hypervisor sandbox. In future versions of Windows there will be an Isolated Mode for applications, that will help to offload certain tasks in a secure container. But that can never be used for the benchmark itself, so we are kind of on our own here.

The thing is that we don't need to have a perfectly secure wrapper or benchmarks, even if it would be possible. We just need to raise the bar a bit higher so real, individual effort is needed to cheat.

That aside a unification for submission upload has many other benefits as I stated before. We would have a weapon against the uncertain future of hardware and software, as you said it yourself.

Regarding benchmark integration: Yeah, it is still early times for BenchMate and many other benchmarks can be integrated over next few months. Think of it like that: We don't need any HWBOT integration any more to add a bench, once it's BenchMate compatible, it works for the bot. So soon you will have more benchmarks to choose from than you ever had!

Also there are much more categories possible to use for competitions. This is the state today:

• CB R20, CB 15 + Single Core, CB 11.5 + Single Core
• 3DMark03, 3DMark05, 3DMark06
• Geekbench3 Single + Multi Core
• Geekbench4 Single + Multi Core + Compute
• SuperPi 1M + 32M (all sizes would be possible)
• GPUPI 3.3 1B + 32B GPU, 100M + 1B CPU
• GPUPI 3.2/2.3 (yes, GPUPI 2.3 can be used as well now!)

I'm counting 23 benchmark categories possible that are well-known. Even more if we would add GPUPI 500M or SuperPi 2M for example.

That's just for now!

Regarding Windows 7: As Win 10 was the primary dev OS, it was clear that other OSes will need some work. We will get there, I promise. I'm not going to half-ass this. Windows 7 will work soon and all bugs will be fixed. Just takes time and effort.

As to your question: Sure, it would be possible to integrate only a part of the features, but I see no reason for that now for Windows 7 to 10 on any bench. BenchMate could of course be only a light version wrapper for XP. We can discuss this at a later point. The main reason I would not like to go down this route is to focus on the future and not on the past. If I would get my hand on the SuperPi code for example, I might be able to do a SuperPi 2 that would work better on modern OSes and might help us out of the XP corner. That would take SuperPi truly to the next level to regain meaning in today's world. A step in the right direction!

Thanks for the kind words, Leeg. Very much appreciated!

[Mr.Scott]

Some legacy hardware will run on nothing but XP or older.

You would condemn this to death.

I suppose that this would be an acceptable loss for most on HWB, but not for a few of us. 

[Leeghoofd]

The past is part of the database HBOT has become, therefore we just can't leave our legacy benchmarks behind.  Does modern hardware have to bench them, maybe not , but that's part of the evolution of Hardware: better and faster. But some still love to compete with the old gear on maybe WinXP, they still have to have their share of fun!

Hence why I vote for benchmate to be used primarily on Win10 (security and Bug proof) and at alater stage also for 7 (Security)

Edited July 14, 2019 by Leeghoofd

[_mat_]

Very true, old hardware should be left out of this.

BenchMate has its limits as well. The common denominator is not only Windows 7 but HWiNFO needs to run as well. I think the next weeks will show very clearly where to draw the line. I just don't think it's AMD + Win 10 because BenchMate is so much more than a timer skew check.

Edited July 14, 2019 by _mat_

[Mr.Scott]

It is encouraging to see both of you at least thinking about the legacy hardware benchers. I thank you for that.

[Leeghoofd]

43 minutes ago, _mat_ said:

 I just don't think it's AMD + Win 10 because BenchMate is so much more than a timer skew check.

We all know this, but at the moment it's already near perfect to already handle that task, I just look on the ongoing stuff, with BenchMate the new Ryzen crowd can sub with Win10.

Benchmark safety and faster and easier submitting will enhance everybodys life. 

[macsbeach98]

2 hours ago, Leeghoofd said:

 

Hence why I vote for benchmate to be used primarily on Win10 (security and Bug proof) and at alater stage also for 7 (Security)

W7 works off the RTC regardless of platform why do we need Benchslow for W7.

I vote for W8/10 only for it

[_mat_]

Not true at all. The EVGA SR-2 skews RTC on Windows 7 as well. I've fixed this in GPUPI by detecting the mainboard and making HPET mandatory. God knows how many skewed SR-2 scores are on the bot because of that bug, because many other benches use RTC as well.

It's an assumption based on a few tests that RTC is safe on Windows 7. We don't know that for sure. In my tests I've figured out that RTC is very unstable and often skews more than 5 milliseconds per second. That's why BenchMate sometimes injects a TSC timer into a bechmark when HPET is not possible due to performance reasons. Then it tests the skew on every run that involves a TSC timestamp and adds the information to the result dialog in the screenshot.

It's not necessary to assume anything with BenchMate. We are hardware guys, let's measure and analyze.

Also I've stated many other reasons for using it in the posts above. It's not only about timer reliability!

Have you tested it? Isn't it great to upload a result in a matter of seconds not having to bother about screenshots or entering numbers manually into a form?

Edited July 14, 2019 by _mat_

[_mat_]

Btw, two small but nice features in BenchMate 0.8:

I changed SuperPi's message box at the end of every run so newbies don't ruin their score. And you can move around the SuperPi window even with the message box active. Always hated that I can't do that!

For Cinebench I've removed the nasty dialogs when quiting the bench. I don't know how many times I've had to click those. No, I don't want to save my score, man!

It's also those little things that we can fix now in legacy benches. If you have another fix like these in mind, just share them and I'm going to look into it.