_mat_
-
Posts
1003 -
Joined
-
Last visited
-
Days Won
41
Content Type
Profiles
Forums
Events
Blogs
Posts posted by _mat_
-
-
Seems to be a time conversion bug in HWBOT from seconds to days/hours/...
-
28 minutes ago, Noxinite said:
So it is unlikely to have bugged. It would have had to bugged the timer pretty crazily to get such a big boost.
The timer used in this score is HPET btw.
-
1
-
-
The result was done with BenchMate 0.8, which didn't have child process tracking of timer calls yet. So there is a small chance for time skewing, which could go by undetected.
That said there are lots of Win 10/Zen 2 scores on the bot that are not using BenchMate, which is against the rules and even more prone to undetected skewing. So I would give this one the benefit of the doubt as long as nobody else achieves nearly 6G on a 3900X in Geekbench 4.
-
4 minutes ago, Mysticial said:
As someone who loves mischievous thought experiments, an alternate (but childish) approach is to compromise all the big name benchmarks. For example, publicly release a hack application that DLL-injects the timers to compromise these big-name benchmarks. (IOW, make it very easy to cheat them.) Then have many people (botnet?) massively submit cheated benchmarks into the respective databases. Make them indistinguishable from real results. This will force a hand, but it obviously won't make you too many friends.
Have at it! You can already do this with Cheat Engine, but to crack 3DMark for example you need my TimerBench (menu -> Measure Process...).
I have reported this to Intel and UL/Futuremark 2 years ago and it changed nothing. That's where BenchMate actually started.
-
1
-
-
30 minutes ago, jfpoole said:
FWIW removing the profiling code (which wasn't controlled via a setting) didn't change Rigid Body Physics performance on any of our ~150 test systems.
Try HPET enabled for the OS (useplatformclock) on Kaby Lake X, Skylake X and above as well as Ryzen and Threadripper. It will have a huge impact.
34 minutes ago, jfpoole said:It's trivial for us to filter out results that are potentially affected by the LAPIC timer bug -- we can exclude results gathered from Windows 10, or from systems that have a non-standard BCLK, or any one of the number of system metrics we collect
This can be a marker for some scores, but in the end it's undetectable. Also what's a non-standard bclock? It can vary by itself due to fluctuations by mainboards and on several generations even some XMP settings can alter it. So you would get false negatives and false positives. Please don't downplay the issue, this is also about the sanctity of your result database and the trust in your (Windows) results.
40 minutes ago, jfpoole said:What we can't do is filter out Benchmate results because there's no way for us to identify those results!
That can be easily arranged. But it needs cooperation.
-
Thank you, @Mysticial.
Your input on the integration is also very much appreciated, which is why I want to spend more time on y-cruncher, working on this together! I think there is a lot of unused potential in your benchmark and frankly it's one of the best, most optimized benchmark code out there.
It's time to get your benchmark points.
-
6
-
-
Thanks for the feedback (and the flowers), @Dancop
I've learned from all this troubles, that I am not going to add another benchmark without official approval. The Geekbench integration took many days to do it perfectly right, although not all is lost, these techniques can be used for other benchmarks now (child process protection and tracking for example).
Last week I've contacted the devs of pifast, wPrime and CINEBENCH. Only the pifast dev has answered, all others did not. I'm not surprised that Maxon (CINEBENCH) is not taking me seriously. There is no business in this, so I'm low priority at best.
-
1
-
-
2 hours ago, jfpoole said:
Benchmate integration introduces a number of issues for us, both technical and legal. On the technical side, Benchmate has introduced issues that have prevented Geekbench from working properly, or have prevented Geekbench from accurately measuring performance. On the legal size, what if Benchmate's modifications to Geekbench cause Geekbench to delete data from a user's system and we get sued because of it?
We may be shooting ourselves in the foot here, but I fear this is very much a damned if we do, damned if we don't decision in which case I'd rather make the decision that gives us the most control over our destiny.
It's your decision, sure. But your accusations are baseless. There was a small contention bug in the driver for 32 bit HPETs only because of your 14-million-timer-calls bug. That has been fixed on the first day of the release of Geekbench 5 and the BenchMate support!
Also I'd like to sum up what BenchMate has done for you and your benchmark, just to make this perfectly clear:
Because of my work you were made aware ...
- that you left some profiling setting on that called QPC for 14 million times in a subtest, which influenced the result depending on the QPC's configured timer.
- that neither Geekbench 3 or 4, neither your latest version 5 has ever mitigated the "LAPIC timer bug" on nearly all AMD platforms and Intel Pre-Skylake on Windows 8 and 10.
You have fixed the first bug thanks to my report. So basically I am resposible that Geekbench 5 is measuring performance more accurately in one of the subtests.
As for the second bug, if you give this a little thought, all scores in your database with systems affected by the "LAPIC timer bug" that were not done on Windows 7 are invalid.
So if you really want to blame someone for inaccurate scores, blame yourself, not BenchMate.
-
1
-
1
-
Every version of BenchMate before 0.9 could only handle a licensed Geekbench. That was a "bug", that actually led to some people buying the full license during the BenchMate testing competition. I have a licensed GB version myself and therefore did not anticipate the differences. BenchMate 0.9 fixed that and makes Tryout versions available as well.
I also think that BenchMate (in any version) does not in any way subvert any license requirements of Geekbench, neither by intent nor by mistake. All Tryout limitations are honored, BenchMate just fixes the timer issues by checking if the timer functions used are skewing in between the run time (not on every timestamp, just before and after the run). The capturing of results is done by tracking and parsing the standard output return value of the inner geekbench executable. You can do this yourself if you just call geekbench_x86_64.exe for example with the parameters "--cpu --backend". This results in a JSON after the run has finished. You can use programs like Process Monitor or API Monitor to get insights like this into applications. There is nothing devious about being curious about what is running on your system.
@jfpoole Any chance you have reconsidered your claim to pursue legal action for supporting Geekbench? I'm not talking about distribution, that's off the table. But if you give your official approval here for the support, I think we can bury the hatchet and leave things as they are.
If I can't get official approval, I will remove the Geekbench's from BenchMate in the upcoming version sometime in the next few days. Subsequently we should go forward with the community decision of letting Geekbench either stay or go. So the future is in your own hands.
About the future of BenchMate: I'm working on the integration of pifast right now. It will have its own command line interface wrapper, which is kind of cool.This will enable BenchMate to run all different kinds of command line benchmarks securely!
Having learnt from the past, I'm also in contact with the developer of pifast and he has given his preliminary approval for integration. I'm expecting final confirmation after I've sent him the first screens of the new wrapper.
-
2
-
-
Impressive achievement!
-
Very nice efficiency. Congrats!
-
1
-
-
Yes, that's true according to the rules of HWBOT. CB uses timeGetTime, which is in direct relation to any reference clock change. Sorry man, great score though!
-
Sorry for the late reply, I was busy breaking my hand and visiting the hospital. ?
I think it's great that you are reaching out to the community now and taking these problems seriously. I don't want any troubles, I'm simply trying to solve the issues at hand for competitive benchmarking.
15 hours ago, jpoole said:I approached Matthias on Wednesday and asked him to remove Geekbench from the Benchmate bundle -- the way in which Benchmate included Geekbench created legal and logistical issues for us. This request was straightforward yet took significant back-and-forth to resolve.
You wrote four mails, the last two contained two sentences each. Don't forget that I had to go from "Hey, a dev checking in, hopefully we can work together now to better integrate Geekbench for competitive benchmarking" to being threatened to be sued in the end. I was never reluctant to remove Geekbench from my bundle (I stated that already in my second reply), but I am challenging wheter it is your right to remove the support for Geekbench from BenchMate.
16 hours ago, jpoole said:I also expressed my concern over how Benchmate integrates with Geekbench. Benchmate does more than change the timers Geekbench uses to measure performance. Benchmate "snoops" into Geekbench to fetch results and submit them directly to HWBot, and appears to be taking steps towards subverting Geekbench's licensing code. Matthias disagreed with our assessment and refused to remove the integration. We are concerned about an application built on top of ours without our permission because of methodological, statistical, business, and legal reasons.
You didn't want to know anything about the integration and refused to have a meeting with me to get the full details. You also repeatedly stated that BenchMate doesn't uphold your license requirements, but failed to answer any requests to specify what BenchMate does to circumvent any of those. I ensure you that BenchMate doesn't do anything, the user executes the exact same code that the Tryout version provides. It's my opinion that this integration is in no way harming your profits, but rather expands the target group of possible buyers.
Lastly, I want to remind the whole community that there is a bigger picture to this. Let's say Primate Labs fixes the timer issues with Windows 8 and 10 for all its benchmarks (which I can only welcome). That doesn't change the fact, that we have to rely on screenshots (which would need version proof to fix the above) and manual data entry to submit results to HWBOT. Then there is also the fact, that Geekbench can be fooled easily as I have done in a matter of minutes without a single line of code and with tools available on the internet right now.
The goal was to make benchmarking and overclocking more accessible. To reduce the number of rules, make benchmarks future-proof and regain trust on results, especially inside our community, but also vendor scores for example. If you think this to the end, we will need a unified benchmark standard and some launcher/wrapper to enforce this.
If the community doesn't want this, then please tell me and we can stop wasting anyone's time here.
-
7
-
-
Sadly he wants the whole integration to be gone. I would be okay with not including it in the Big Bundle and would still support it if bugs were to be found. But he doesn't want it at all, and that's why he threatened with legal action.
As my money is on the line here, I'm going to remove it of course. But I didn't want to do anything rash, so I'm postponing that for a little while and wait until our community has discussed this properly.
-
The funny thing is that with the new Geekbench 5 integration I didn't even touch the timers. I can't do that with 14 million timer calls, HPET is too slow for that and there would be an impact of at least 1000 points on the score. So I'm merely tracking the calls and checking the timers for skewing.
-
5
-
-
They don't know. In fact any application like Cheat Engine can intrude Geekbench and change its timers to whatever and it wouldn't have a clue.
New world record incoming ...
-
1
-
4
-
1
-
-
I have no personal vendetta against the Geekbench guy. I've always liked the Geekbenchs, that's why I integrated them in the first place. I am also a customer of his and have bought GB 3/4 and used it regularly for my reviews on overclockers.at.
I'm concerned though where this is going. Geekbench has no mitigations for Windows 10, but still allows these scores to be uploaded to his database, even if they can possibly - with or without intention - be skewed. That's his right, but says a lot about the quality of his database too.
But the bot has different, stricter rules than he applies. That forces the bot to do extra moderation based on uploaded screenshots, that can't rely completely on the validation links to the Geekbench browser. That's a very hard task, and time-consuming as well.
Now BenchMate fixes these things and makes his scores compliant to the bot's standards, but he doesn't want it. That leaves us with a BenchMate that can't wrap all the benches we have. BenchMate is less useful for overclockers now, without any fault of its own. This lessens the chances that it will ever get fully adopted here because it is not allowed to support 4 point-giving benchmark categories.
I'm talking about the many days it took me to integrate Geekbench correctly. I'm talking about decisions like these that are out of our hands but threating our future of being more accessible to newcomers and deliver more trustworthy results. That sucks.
I also think that the community should decide on the next steps. Please just don't forget that this is not only about Geekbench.
-
1 hour ago, cbjaust said:
I got the email from John Poole of Primate Labs with the GB5 upgrade key and after reading the above I replied to that email suggesting he reconsider his position given that I for one wouldn't have bought GB3 if not for hwbot, nor would I have bought GB4 if not for Benchmate. I won't be giving Primate Labs any money in future unless their position changes. I also suggested that Primate Labs should be supportive of hwbot and BenchMate for driving sales of geekbench. Heh
You sir, are a hero!
Thanks for the support, guys! You are my motivation!
34 minutes ago, Splave said:I'm fine with dumping geek bench. It's become bug bench anyways, maybe Matt can secure 2 other benches to replace them that are open source. Black hole time!
I'm rethinking my stance on wPrime. Let's put in the effort and integrate it into BenchMate instead of Geekbench. I will add pifast as well, x265 is on my list and y-cruncher as well.
Any requests?
-
2
-
-
19 minutes ago, Mythical tech said:
@_mat_thank you for all your hard work for this community , I say we ditch geekbench and never give them another penny. I would love to see benchmate receive points when you and the devs of the bot think it is ready.
Thanks!
We are working on the integration into HWBOT. BenchMate will be an optional tool to submit results into the categories of supported benchmarks.
-
To give an update here as well: BenchMate 0.9.3 is out.
The new "feature" is that it removes Geekbench from the Big Bundle, because the dev of Geekbench rather wants to sue me than making his benchmark valid for competitive overclocking.
Full details here:
-
To clarify what is happening here with Geekbench: The developer of Geekbench threatened to sue me for integrating his benchmarks into BenchMate.
Apparently I'm benefiting from his work by fixing his benchmark (for free) to make Geekbench results valid for competitive benchmarking. This is even more important for Geekbench 5, which is only available for Windows 10 but doesn't mitigate the "RTC bug" and will therefore have time drifting as a new "feature" for all AMD CPUs and Intel Pre-Skylake.?
I tried to be polite, even apologized for my harsh words on the bot forums for calling the usage of timer functions 14 million times (!) during a single run as "incredibly stupid and obviously unnecessary". I know that I can be a straight-forward guy sometimes and a pain in the ass. Anyways, he couldn't forgive me for my "attacks" (as he calls them) and doesn't want to start fresh as I have proposed after the apology. So I was forced to remove all Geekbench versions from the Big Bundle of BenchMate, you will sadly have to install them manually now.
This makes me really sad. The benchmark community is small and I think we should hold together and overcome stuff like that for a better future.
Last but not least removing Geekbench was not enough, Primate Labs wants me to remove the integration or "if you continue to modify Geekbench in violation of the EULA we reserve the right to pursue additional remedies.".
So my fellow overclockers, what should we do? Anybody here that still wants to have fun with Geekbench? The dev seriously doesn't give a fuck about the community, so why should we?
-
2
-
2
-
-
That's true, these API calls can be faked. But I can track that the call comes from inside the executable and BenchMate additionally does various debugging checks and avoids access the processes's memory. And to check that the executable itself has not been changed, a file integrity check is done and the hash part of the result dialog.
14 minutes ago, Mysticial said:Oh, I was just answering the part about being able to transfer data between the two components in a way that cannot be read, cannot be altered, and cannot be faked.
An impossible task so to speak. But you are right, it can be hardened to make it very difficult to reverse and fake. From experience I try not to store any kind of key in the executable, it can always be found/reversed.
These are both viable ways, my first proposal will be easier for you, but harder for me to secure. The second way will take some time for us to figure it out and much more intrusive. Don't forget that we can't do that from the Java wrapper, it has to be inside y-cruncher.
-
What data would you need to transfer that needs encryption?
-
17 hours ago, Mysticial said:
Now that I think about it, it would only detect it if the clock skew happens mid-bench. It tries to reconcile rdtsc with multiple wall clocks and will flag any benchmark where they get out-of-sync. So if the wall clocks are coming from BenchMate, but rdtsc gets skewed mid-bench, it will block the score. While rdtsc can be skewed, it's not easy to fake unless the system is being virtualized (which it heuristically detects, but the information isn't used atm). While you can play with the rdtsc offsets (which also requires kernel mode), I can't think of a useful manner to exploit it without doing backwards jumps which is easy to detect from the benchmark. (though I currently don't try to detect it) In any case, there are bigger problems if the attacker has kernel-level access.
(Very) old versions of y-cruncher would try to rec the measured rdtsc frequency with what the OS measured it at boot time. This blocked a number of hacks back in the Win7 days, but it also produced too many false positives. So I disabled that check.
I have written a hypervisor during my research that does exactly that. Catch RDTSC(P) instructions, count them and skew it as necessary. It is a lot of effort though and with proper hypervisor detection inside the benchmark/BenchMate, it's much worse of course. That's why I also have a hypervisor detection code in there.
There are two main problems with RDTSC as I see it and that's why I'm not using it. First of all it skews with the bclock/reference clock on Pre-Skylake and AMD CPUs, so it's not reliable at all. Secondly, it's hard to get the correct TSC frequency on AMD. There is no CPUID register that reports the correct frequency as far as I'm aware - at least on Ryzen - (and even on Intel it's not that easy), so you have to get the frequency yourself. I've seen really bad implementations to do that out there and they are far from valid in my opinion. And even if they would be correct, you have to assume that reference clock changes will occur and make RDTSC invalid.
I can't recommend comparing RDTSC against other timers on Windows 8/10 as well because nearly every timing method will either depend on RDTSC or the LAPIC timer and both have the same time source, that will skew with the reference clock. The only exception is HPET, but that needs to be enabled for the whole system, which can have a severe performance impact on CPUs with high core count and especially on Kaby Lake X and Skylake X due to a very slow HPET implementation. I've written thorougly about this here if you are interested.
Most of the things I've written above are also valid for Linux and MacOS btw, but it's much harder there to change your bclock/reference clock in the OS and that's why it's not considered a thing there.
17 hours ago, Mysticial said:Is there a way to detect dependency injection and verify that it's trusted?
There is not authentication currently. As the favored integration I outlined above is a one-way communication of the benchmark to signal certain kind of actions (beginrun, endrun, result) with information that is publicly shown as well (the result for example), I don't think we need it for now. It's different if we want to transfer something secretly, but that would need a lot more effort. To be honest I haven't found a way yet, that can't be reverse engineered and faked easily. Shared Memory File Mapping is horrible in that aspect for example.
17 hours ago, Mysticial said:The timing aspect can be made fundamentally secure under the assumption that the benchmark binary hasn't been compromised. Do the timing server-side on HWBOT. Then use public-key encryption to match the start/end messages. Just make sure the benchmark runs long enough that network delay has negligible impact.
You could theoretically write a fundamentally secure crypto benchmark by means of a crypto time capsule. (https://en.wikipedia.org/wiki/LCS35) The server sends the problem parameters. The benchmark is to solve the problem and send it back to the server. The timing is done server side.
It would be the ultimate solution to move as much code as possible to the server-side, but that would need a system to be always online. That's a compromise that might be a little too much for the time being. I think that BenchMate as it is brings a lot of changes with it and that can be difficult to adapt to for overclockers on here. I guess there will be a time for "always online" at some point in the future, but for now it's important to find some middle-ground to get this new standard for benchmarks going.
Benchmate 10.6.8 is out!! NO MORE SUPPORT FOR GEEKBENCHES!!
in HWBOT News
Posted · Edited by _mat_
No, Benchmate runs fine on 32 bit as well and Windows 7. I've tried a Core 2 Duo when testing 0.9 and it worked fine too.
Oh, I just noticed that I have no clue what you are talking about.