Benchmate 10.6.8 is out!! NO MORE SUPPORT FOR GEEKBENCHES!!

[keeph8n]

Realistically it almost sounds like WE as XOC people should have a letter written up explaining that we want the parties to be able to work together moving forward, and ALL of us submit emails.

 

There is no reason for them to not work WITH the people that use their benchmarks consistently. They are missing a large opportunity here imo.

[MrGenius]

The Cinebench fiasco all over again.

"I'll sue you if you freely distribute my software that I'm freely distributing. Because...reasons(even though I'm going to change my mind about that when I realize how stupid and pointless the idea is)."

Just a bunch of pointless stupidity for the sake of pointless stupidity. Fact is...other than causing unnecessary aggravation...it doesn't change anything. If we want to use GB with BM, that's what we're going to do. Where we got GB is totally and completely irrelevant and inconsequential. Makes no difference whatsoever.

[keeph8n]

I mean I think its honestly a bit more than that, as @_mat_ is the one on the line here.

[TerraRaptor]

I guess they don't realize that if GB5 won't be secured by Benchmate it is likely miss global points in future. No global points - no need to buy it for most of us.

[MrGenius]

Hmmm...am I missing something? 

• BM dev freely distributes free to use version(s) of GB.
• GB dev says "Stop! Or I'll sue".
•  BM dev stops. And no one is sued.
•  Community finds out about it and expresses much disdain for such behavior on the part of the GB dev.

Which is pretty much the same situation that happened with the recent release of CB 20. Except for the part where the "complainant" dev eventually sees the error of their ways, and drops the ridiculous shenanigans.

Listen...I'm not trying to trivialize it any more than it already is. The bottom line is we're pissed at the GB dev for being unreasonable for no good reason. I think we're all in agreement with that. . I have been enlightened..

Side note: What happened with CB 10? Why did that one get skipped? Is there any reason we shouldn't be doing that one too?

Edited September 5, 2019 by Leeghoofd

[Cautilus]

@MrGenius Well to quote mat:

19 hours ago, _mat_ said:

To clarify what is happening here with Geekbench: The developer of Geekbench threatened to sue me for integrating his benchmarks into BenchMate.

...

Last but not least removing Geekbench was not enough, Primate Labs wants me to remove the integration or "if you continue to modify Geekbench in violation of the EULA we reserve the right to pursue additional remedies.".

I think this wording is saying "stop messing with Geekbench with your program or we'll sue". Mat already agreed to stop distributing the benchmark, but now the devs want him to stop BenchMate from doing anything at all to Geekbench.

If they want mat to completely withdraw any functionality that BenchMate has for Geekbench, then I think it should be slated for removal of points if/when HWBOT moves to needing BenchMate for everything.

Edited September 5, 2019 by Cautilus

[jpoole]

I'm John Poole, the founder of Primate Labs, the company that develops Geekbench.  I'd like to clarify our position on Benchmate.

I approached Matthias on Wednesday and asked him to remove Geekbench from the Benchmate bundle -- the way in which Benchmate included Geekbench created legal and logistical issues for us.  This request was straightforward yet took significant back-and-forth to resolve.

I also expressed my concern over how Benchmate integrates with Geekbench.  Benchmate does more than change the timers Geekbench uses to measure performance.  Benchmate "snoops" into Geekbench to fetch results and submit them directly to HWBot, and appears to be taking steps towards subverting Geekbench's licensing code.  Matthias disagreed with our assessment and refused to remove the integration. We are concerned about an application built on top of ours without our permission because of methodological, statistical, business, and legal reasons.

I'm disappointed in how this incident has unfolded, and I'm surprised to see comments that we're not taking these issues seriously, or that we're not supporting the community.  I believe we can take these issues seriously while still disagreeing about whether Benchmate and the techniques it uses are the correct approaches to resolving these issues.

[keeph8n]

@jpoole so in interest of a third party looking in here, is there a way that both parties can work together to resolve this in a way that makes everyone happy?

[Splave]

Hi John welcome to hwbot, its unfortunate it took you this long to make an account and be represented on hwbot especially if you care about our community

Bench-mate does not subvert the licensing to geekbench at all, you run geekbench via the launcher (benchmate) and include your key if you want to use the 64 bit version. (trial is limited to 32bit). 

I can see why you may not have liked the software being bundled. (make take some clicks away from your site etc.) But it also makes it seamless and distributing what amounts to shareware will just lead back to people buying the key for 64 bit off your site no? 

How can this be fixed and moved forward in your mind? Thanks again.

 

EDIT: I apologize for anyone calling you names. Surely that is childish and will be fixed by the mods. Most of us are grown ups here. 

Edited September 5, 2019 by Splave

[mickulty]

24 minutes ago, jpoole said:

I'm disappointed in how this incident has unfolded, and I'm surprised to see comments that we're not taking these issues seriously, or that we're not supporting the community.  I believe we can take these issues seriously while still disagreeing about whether Benchmate and the techniques it uses are the correct approaches to resolving these issues.

What evidence would you offer to show that you are taking these issues seriously?  As someone who doesn't follow every minute detail of every ongoing piece of software development, all I know is that geekbench 3 and 4 are not approved for use on windows 8 and 10, even with a result validation link.  Do hwbot mods have that wrong?

I would love to be able to get legitimate results running geekbench on later operating systems *without* a wrapper, in fact I'm sure everyone would love that if possible.

[jpoole]

@Splave Unfortunately I'm locked out of my account from 2008!

I'm not sure how we move forward from here.  I'm sure everyone's feelings are a bit raw right now.  I know mine were after I read Matthias' initial comments about Geekbench 5 and the development team here at Primate Labs.

Regarding the timer issues, I was only vaguely aware of them prior to this week, and assumed that since Geekbench was included in the HWBot leaderboards the issue had been resolved to HWBot's satisfaction.  I know that's not the case now, and we're looking into ways to solve this inside Geekbench across all platforms we support.

Regarding the 16 million timer calls, we've already fixed that issue in Geekbench 5.0.1.  The underlying issue was that a third-party library used QueryPerformanceCounter() to profile individual function calls within the library.  Disabling that profiling brings the number of timer calls back in line with Geekbench 4.

[GeorgeStorm]

Nice to see you here.

Hopefully we can work things out, definitely agreed things were a little heated, and now we can have a discussion and come to a positive arrangement for all parties

[Leeghoofd]

Yes , best to have a sleep over it, tomorrow we can all work together as it is in the best interest for everybody that the experts on the matter join forces  Thus improving the things that need to be adressed.

Regarding most benchmarks used by the OC ommunity: Biggest issue for our community has been the release of Win8-10 which created a whole new aspect of buggability, plus one CPU manufacturer still not addressing something his competitor managed to achieve like 3 CPU gens ago....

 

Good to see you responding Mr Poole (which is a clear indication that you do care bout your product and the present community)

I can just vouch for Mat's skills and he has some great ideas. Downside thanks to his young enthousiasm he can get a bit carried away  ( but he has the best intentions ) Also don't forget sometimes a small traduction error can lead to misunderstandings, aggrevation and elevated blood pressure.... We are not all native English native speakers here.

 

Edited September 5, 2019 by Leeghoofd

[Splave]

how will me move forward ban all old geek bench results? require a new build version and ban all the old results? sigh

[mickulty]

39 minutes ago, jpoole said:

Regarding the timer issues, I was only vaguely aware of them prior to this week, and assumed that since Geekbench was included in the HWBot leaderboards the issue had been resolved to HWBot's satisfaction.  I know that's not the case now, and we're looking into ways to solve this inside Geekbench across all platforms we support.

This makes sense.  Sorting it out like this would be a perfect solution and would make the question of benchmate a non-issue - if geekbench is reasonably secure then there's no need for a launcher/wrapper/whatever.

I can see how it's easy to confuse "hasn't magically realised something needs doing" with "doesn't care".  I'm sorry for my lack of faith - clearly you do care.

[Noxinite]

While we're on the topic, is there any risks of this happening with any of the other bundled benchmarks?

Although, I would have thought that altering and redistributing another entities code probably should have been questioned at the starting point.

But otherwise I agree with Leeghoofd's statement above.

[mickulty]

Just now, avalanche said:

Bugging Geek as you know was super easy ... sigh alright.  Here we go again 

Lots of members hard work & good results + some dodgy flushed down the toilet again. We needed Mat's Benchmate like 10 years ago to keep results safe

 

33 minutes ago, Splave said:

how will me move forward ban all old geek bench results? require a new build version and ban all the old results? sigh

No need for this, surely?  Just require a new version going forwards?  The community is pretty good, it's not like there's huge numbers of suspect results to worry about.

[jfpoole]

Thanks to @GeorgeStorm for getting me back into my old account!

Thanks, @mickulty, I appreciate the kind words.  We've spent a lot of energy dealing with mobile vendors who manipulate Geekbench scores (see, e.g., https://www.geekbench.com/blog/2018/09/huawei-benchmark-boost/) but this has been a blind spot for us.  I feel really dumb that we just assumed everything was ok.  The team's been discussing strategies on how we can address this issue and I hope we'll have something to share in the coming weeks.

[bigblock990]

I'm glad you are here now, I really hope a solution can be found by all the parties working together. Even though I initially called to remove geekbench from hwbot, I would rather see it stay. I also really like benchmate, and want to see geekbench remain supported there.

However I do feel the need to point something out. You have 5 posts total between both your new account and 11yr old account. To me this clearly indicates that you actually weren't a part of this community. If you had been prior to today, you wouldn't have been surprised by the comments at all. Us benchers are small and very passionate community. Saying that @_mat_ is an active and valuable member here is a huge understatement. So it didn't surprise me one bit to see the reaction this thread has caused.

Again I really hope you decide to stick around, and that in the future this "incident" can be considered water under the bridge.

Edited September 5, 2019 by Leeghoofd

[_mat_]

Sorry for the late reply, I was busy breaking my hand and visiting the hospital. ?

@jfpoole

I think it's great that you are reaching out to the community now and taking these problems seriously. I don't want any troubles, I'm simply trying to solve the issues at hand for competitive benchmarking.

15 hours ago, jpoole said:

I approached Matthias on Wednesday and asked him to remove Geekbench from the Benchmate bundle -- the way in which Benchmate included Geekbench created legal and logistical issues for us.  This request was straightforward yet took significant back-and-forth to resolve.

You wrote four mails, the last two contained two sentences each. Don't forget that I had to go from "Hey, a dev checking in, hopefully we can work together now to better integrate Geekbench for competitive benchmarking" to being threatened to be sued in the end. I was never reluctant to remove Geekbench from my bundle (I stated that already in my second reply), but I am challenging wheter it is your right to remove the support for Geekbench from BenchMate.

16 hours ago, jpoole said:

I also expressed my concern over how Benchmate integrates with Geekbench.  Benchmate does more than change the timers Geekbench uses to measure performance.  Benchmate "snoops" into Geekbench to fetch results and submit them directly to HWBot, and appears to be taking steps towards subverting Geekbench's licensing code.  Matthias disagreed with our assessment and refused to remove the integration. We are concerned about an application built on top of ours without our permission because of methodological, statistical, business, and legal reasons.

You didn't want to know anything about the integration and refused to have a meeting with me to get the full details. You also repeatedly stated that BenchMate doesn't uphold your license requirements, but failed to answer any requests to specify what BenchMate does to circumvent any of those. I ensure you that BenchMate doesn't do anything, the user executes the exact same code that the Tryout version provides. It's my opinion that this integration is in no way harming your profits, but rather expands the target group of possible buyers.

 

Lastly, I want to remind the whole community that there is a bigger picture to this. Let's say Primate Labs fixes the timer issues with Windows 8 and 10 for all its benchmarks (which I can only welcome). That doesn't change the fact, that we have to rely on screenshots (which would need version proof to fix the above) and manual data entry to submit results to HWBOT. Then there is also the fact, that Geekbench can be fooled easily as I have done in a matter of minutes without a single line of code and with tools available on the internet right now.

The goal was to make benchmarking and overclocking more accessible. To reduce the number of rules, make benchmarks future-proof and regain trust on results, especially inside our community, but also vendor scores for example. If you think this to the end, we will need a unified benchmark standard and some launcher/wrapper to enforce this.

If the community doesn't want this, then please tell me and we can stop wasting anyone's time here.

[mickulty]

1 hour ago, _mat_ said:

That doesn't change the fact, that we have to rely on screenshots (which would need version proof to fix the above) and manual data entry to submit results to HWBOT. Then there is also the fact, that Geekbench can be fooled easily as I have done in a matter of minutes without a single line of code and with tools available on the internet right now.

The goal was to make benchmarking and overclocking more accessible. To reduce the number of rules, make benchmarks future-proof and regain trust on results, especially inside our community, but also vendor scores for example. If you think this to the end, we will need a unified benchmark standard and some launcher/wrapper to enforce this.

If the community doesn't want this, then please tell me and we can stop wasting anyone's time here.

Screenshot-based validation in general is not ideal and depends on how much trust you want to have, but geekbench already has a mechanism for online validation as well, the same as UL benchmarks do.

Screenshot of result+cpu-z cpu+cpu-z mem(+gpu-z gpu) alongside a datafile or validation link seems to me a very consistent, accessible way to do things.  GPUPI and x265 do this already, UL benchmarks effectively do this for high ranked or windows 8/10 scores, benchmate does this too.  Geekbench could certainly do the same without having to be run from benchmate.

Benchmate is fantastic as a way to make sure the legacy benchmarks people love can carry on - benchmarks like wprime where there's literally no validation at all other than one number in the screenshot.  Hopefully there's no need for it with a living, actively maintained benchmark like Geekbench and instead there can be co-operation towards a situation where Geekbench works without a wrapper.

[Splave]

what do we do about the special older version of geek 3 that scores higher if we are required to run some new one that has more security and scores in line with the latest versions. Asking for you guys because it just locks my scores in so why do I care? 

[l0ud_sil3nc3]

On 9/5/2019 at 1:29 PM, jpoole said:

I'm John Poole, the founder of Primate Labs, the company that develops Geekbench.  I'd like to clarify our position on Benchmate.

I approached Matthias on Wednesday and asked him to remove Geekbench from the Benchmate bundle -- the way in which Benchmate included Geekbench created legal and logistical issues for us.  This request was straightforward yet took significant back-and-forth to resolve.

I also expressed my concern over how Benchmate integrates with Geekbench.  Benchmate does more than change the timers Geekbench uses to measure performance.  Benchmate "snoops" into Geekbench to fetch results and submit them directly to HWBot, and appears to be taking steps towards subverting Geekbench's licensing code.  Matthias disagreed with our assessment and refused to remove the integration. We are concerned about an application built on top of ours without our permission because of methodological, statistical, business, and legal reasons.

I'm disappointed in how this incident has unfolded, and I'm surprised to see comments that we're not taking these issues seriously, or that we're not supporting the community.  I believe we can take these issues seriously while still disagreeing about whether Benchmate and the techniques it uses are the correct approaches to resolving these issues.

Just wow.

One guy fixes a problem your whole team could/would not fix, for free nonetheless and you threaten him with litigation? Should maybe think about scrapping your team and hiring Mat since he actually cares about benchmark integrity and is trying to actively do something for the community (with no ulterior motives btw) which no one else has done to date. You still need a license to use any version of Geekbench whether you're on using it standalone or integrated through Benchmate, so saying that Benchmate is "taking steps towards subverting Geekbench's licensing code." seems like fake news.

Seeing this makes me never want to benchmark any Geekbench ever again, and I for sure will not be purchasing any more licenses simply because of how you have handled this issue. Welcome back to hwbot

 

On 9/5/2019 at 1:42 PM, Splave said:

EDIT: I apologize for anyone calling you names. Surely that is childish and will be fixed by the mods. Most of us are grown ups here. 

 

Edited September 6, 2019 by l0ud_sil3nc3

[Guest]

12 hours ago, _mat_ said:

Sorry for the late reply, I was busy breaking my hand and visiting the hospital. ?

______________________________________________________________________________

The goal was to make benchmarking and overclocking more accessible. To reduce the number of rules, make benchmarks future-proof and regain trust on results, especially inside our community, but also vendor scores for example. If you think this to the end, we will need a unified benchmark standard and some launcher/wrapper to enforce this.

If the community doesn't want this, then please tell me and we can stop wasting anyone's time here.

How is the hand man? Think everyone missed this point. The precious code typing hands, like a Pianist take care of your most important tools.

I'd bet this previous week of upset, you'd be smashing them keyboard keys pretty hard 

________________________________________

If one of us write we support you on your endeavours ... then there are hundreds of members feel/ think the same as I/ we do. 

Benchmate was needed 10 years ago. Keep going  

[Guest]

... anyways guys as funny as that post is above mine ... let's stop being mean to jpoole. He did come in HWbot & say hello.

Which is good, we caught his attention. 

_________

I would hope Mr Poole & Mat can work something out & slip Geek in with some other small programs in Benchmate.

This is the way going forward for all of us. 1x download super easy. New members to HWbot will love this benchmate

 

My first use of it was painless on W10. It worked great ... open a bench /  run / upload score. BOOM done.  Saves time

Big fan of benchmate on my first use. So while Mat & Hwbot have some kinks to iron out ... get onto supporting them with your wallet.