The official BenchMate support thread

[_mat_]

Good idea.

It would also be great to let people know that BenchMate 0.9.3 should only be used on a separate OS, so it is not mixed with BenchMate 0.10. That causes a lot of frustration on all sides.

[unityofsaints]

On 7/7/2020 at 12:22 AM, Alex@ro said:

This software  is junk with capital letters.

Secondly, this made me consider retiring from benchmarking for the first time in 12 years, unbeliavable. Not even ryzen on cold, not even B-die benching, not even cracks when full-pot, nothing did the job except the amazing bench-banana-mate.

It makes absolutely no joy to bench anymore with this application.

Yeah, everything is nice and dandy, unicorns are pink and gooses are flying until this software will simply not work. 

In my experience over time i found that eventually you can find answers to most common issues with benchmarks. Google it, this forum, ask a fellow overclocker, eventually you know what to do. But with benchmate, you are left blind and dumb, both at the same time.

Thanks for speaking out. There are others around here who share your opinions but don't talk about it because they're afraid it would come across in the wrong way or because they've given up trying.

It's interesting that your post has drawn so much criticism for being too ranting, if anything it it wasn't ranting enough and left out some things. While it touches on the many bugs and frustrations with Benchmate, it leaves out the two other valid criticisms: single point of failure for support, and benchmarks scoring higher with Benchmate than without. Really, any of those three issues on their own are enough to disqualify Benchmate from serious use. Maybe the Google model of "perpetual Beta" applies here.

Please don't retire! You're a great bencher and many of us share your frustrations, even if we aren't necessarily posting about it. And as they say, don't feed the trolls ?

[_mat_]

Please specify the bugs that you are referring to. It would really help to come forward and offer feedback and bug reports.

There are currently a few limitations mainly because BenchMate currently works best with modern platforms. These are:

• Old or cheap CPUs won't work because they don't have the AES-NI instruction set. The benchmark will just close.
• Don't use a 12 year old NVIDIA card because the driver wants to inject stuff into the benchmark, which is a big no-no and BenchMate is right to disallow this.
• Windows 7 without SHA2 patch. BenchMate won't start. Installt the patch first.
• Don't mix BenchMate 0.9.3 and 0.10. If you want to bench Geekbench, you are on your own. Thank the Geekbench guy for that!

All of the above was named many times in this thread and in the Discord channels. And every single point is already fixed and will be released with the upcoming version.

I really don't know how to be more understanding, open and thankful for feedback, reported bugs and critique. I answer fast and always, nearly around the clock. If there is a problem, just let me know! I'm sure it can be fixed.

The only thing I currently need is your patience. We are working very late hours on BenchMate 1.0 and the new online validation platform to make it happen asap. We gunned for July but it seems like August will be more realistic.

In the meantime just bench without BM if you have a problem. The way I see it, HWBOT will always stay open for manual submission as long as the results can still be moderated. Especially old hardware will always need a fallback.

As for the improved speed with BenchMate: I looked into every single benchmark, saw the time meaurement mistakes and fixed them with an appropriate, more reliable method. That's what you get for doing stuff the right way. I'm not even sorry.

 

Edited July 20, 2020 by _mat_

[zeropluszero]

2 hours ago, _mat_ said:

As for the improved speed with BenchMate: I looked into every single benchmark, saw the time meaurement mistakes and fixed them with an appropriate, more reliable method. That's what you get for doing stuff the right way. I'm not even sorry.

 

what the actual fuck.

[ozzie]

3 hours ago, _mat_ said:

 

There are currently a few limitations mainly because BenchMate currently works best with modern platforms. These are:

• Old or cheap CPUs won't work because they don't have the AES-NI instruction set. The benchmark will just close.
• Don't use a 12 year old NVIDIA card because the driver wants to inject stuff into the benchmark, which is a big no-no and BenchMate is right to disallow this.

 

 

matt does this mean that it will affect the older hardwares used for eg the  hardware selected for this years teams cup ?

[_mat_]

I don't know what stages and hardware will be used with BenchMate. Can you list them?

[ozzie]

10 hours ago, _mat_ said:

I don't know what stages and hardware will be used with BenchMate. Can you list them?

it would be probably easier and quicker if you looked at the legacy stages in ddr, ddr2, ddr3 matt ?

[_mat_]

I only see a single stage for BM and that's wPrime 1024M with AM4. All good there.

[ozzie]

56 minutes ago, _mat_ said:

I only see a single stage for BM and that's wPrime 1024M with AM4. All good there.

ok matt, thankyou mate ?

[unityofsaints]

On 7/21/2020 at 8:17 AM, _mat_ said:

As for the improved speed with BenchMate: I looked into every single benchmark, saw the time meaurement mistakes and fixed them with an appropriate, more reliable method. That's what you get for doing stuff the right way. I'm not even sorry.

There's the problem right there. We're not scientists, no one is _actually_ interested in how quickly the scene is rendered in Cinebench 15. We are interested in scores that are comparable to each other in a competitive ranking.

Your efforts are like someone analysing all 3-point shots in the NBA since its introduction in 1979 and concluding that an outside shot is actually 55% harder than an inside shot instead of 50% harder and should therefore be awarded 3.1 instead of 3 points. Just imagine players in 2020 could get 0.1 more points as long as they wear a tiny bracelet around their wrist when they shoot. Some would forget. Some would refuse. But anyone making as many shots as Kobe during his famous 81-point game would end up beating his record. I don't think the fans or players would want that!

Change the rules of the game or make a new game but don't change how an existing game is scored, it messes too much with the integrity of the rankings.

[_mat_]

It's far more complicated than your analogy assumes. Using CINEBENCH on Windows 7 has a completely different timer implementation than running it on Windows 10. By changing OS settings you can additionally change the hardware being used for the meaurement. Actually you don't even need to change anything, Windows might select different time keeping hardware on its own depending on the available platform. And that's alright because benchmarks use WIN32 API functions to measure time and these functions exist to separate application code from platform-dependent code.

All of the above truly leads to comparing apples and oranges in line with your analogy. The worst part is that we have no data showing time measurement information to understand what is happening behind the scenes.

Now with BenchMate in control of the benchmark code we can do two things to fix this:

1) The same time measurement code, logic and hardware is used for all benchmarks.

2) Add timer statistics to benchmark runs for better analysis

BenchMate obviously does much more to ensure reliable results. Like using multiple timers to at least triple check the measured time period.

Of course it is true that there might be differences between BM runs and unprotected runs. The same goes for runs between different systems and OS, so there wasn't any integrity in the first place. But it's never too late to do things the right way.

[nick_name]

On 8/1/2020 at 7:34 PM, _mat_ said:

It's far more complicated than your analogy assumes. Using CINEBENCH on Windows 7 has a completely different timer implementation than running it on Windows 10. By changing OS settings you can additionally change the hardware being used for the meaurement. Actually you don't even need to change anything, Windows might select different time keeping hardware on its own depending on the available platform. And that's alright because benchmarks use WIN32 API functions to measure time and these functions exist to separate application code from platform-dependent code.

All of the above truly leads to comparing apples and oranges in line with your analogy. The worst part is that we have no data showing time measurement information to understand what is happening behind the scenes.

Now with BenchMate in control of the benchmark code we can do two things to fix this:

1) The same time measurement code, logic and hardware is used for all benchmarks.

2) Add timer statistics to benchmark runs for better analysis

BenchMate obviously does much more to ensure reliable results. Like using multiple timers to at least triple check the measured time period.

Of course it is true that there might be differences between BM runs and unprotected runs. The same goes for runs between different systems and OS, so there wasn't any integrity in the first place. But it's never too late to do things the right way.

Is this why benchmate keeps getting flagged by av software?  And then when I can get it installed av later comes and screws it up?  I tried using it a few months ago, but gave up.  I'm on a team now though and would like to participate when benchmate is required.  

I'm sorry if this was discussed already.  

[_mat_]

Yes, some av software recognizes some of the patterns necessary to secure the processes. Which is actually very close to the techniques that the av software is using itself, but also rootkits and the likes.

You can either exclude the BenchMate path from realtime detection or disable AV altogether for the time you are benching. It will only hurt your performance.

[jab383]

Offline benching needs no anti-virus nor firewall or any of those other resource leaches.  That's tough when using your 24/7 rig for benching, though.

[_mat_]

In the latest (unreleased) dev versions this is fixed btw. I tested the previously problematic executables with virustotal.com and it didn't trigger a single detection engine, even the ml/dl ones, which are well known for false positives.

[nick_name]

21 minutes ago, _mat_ said:

In the latest (unreleased) dev versions this is fixed btw. I tested the previously problematic executables with virustotal.com and it didn't trigger a single detection engine, even the ml/dl ones, which are well known for false positives.

Would an idiot be helpful in the testing?  Someone that has no experience with benchmate at all?  Because I'm that idiot.  

[_mat_]

Thanks for offering. We are currently rewriting the client (no stone will be left unturned), so we aren't quite in the testing phase yet.

[Fragmeister]

I am wondering why when I use Benchmate with Cinebench R-15 I don't receive personal hardware points just Team hardware points, if I submit without Benchmate I do receive personal hardware points but they won't count as I'm on a AM4 platform.

 

https://hwbot.org/submission/4532261_

 

Tia

[_mat_]

The "with BenchMate" categories were always meant to be temporary until HWBOT can integrate the submission API into its legacy benchmarks. Therefore they are beta categories that don't give any points except team points (which is a bug).

You can still submit manually with the screenshot in BenchMate's result folder. The easiest way to get there is within BenchMate itself. Right click the result entry in BenchMate's result list and select "Show in Explorer" or "View Screenshot".

I know, it sucks and I'm also not happy about the situation. The only possible upload directly into the HWBOT database is with GPUPI.

[Leeghoofd]

34 minutes ago, _mat_ said:

I know, it sucks and I'm also not happy about the situation. The only possible upload directly into the HWBOT database is with GPUPI.

lobbying to get the datafile implementation when benchmarks are merged Matt, but at moment nothing moves till the code will get a first midlife update....

[_mat_]

Thanks. Please keep me updated and let me know if can be of any help.

[Broken APEX49]

WindowsXPはもう

使用不可能にすればいい

[_mat_]

Quote

 

Windows XP is already

It should be disabled

 

I have no idea what that could possibly mean. Wrong thread maybe?

[nick_name]

I disabled a ton of services for my SuperPi 32M run and broke BenchMate.  Can you tell me what I broke to end up with this error?  If this screenshot isn't enough please tell me what you need.  Thank you.

 

 

[Mr.Scott]

Can't have 2 versions of BM on the same drive.