Jump to content
HWBOT Community Forums

_mat_

Members
 View Profile  See their activity

  • Posts

    1003

  • Joined

  • Last visited

  • Days Won

    41

 Content Type 

Posts posted by _mat_

    BenchMate Integration

    in Benchmark software

    Posted

    18 hours ago, havli said:

    Elapsed time is measured directly by the wrapper, using Java nanoTime() function. And remaining time is prediction based on the amount of frames finished and actual framerate.

    The x265 encoder executable also measures time (internally, but reports it as fps). However this is the "wall clock time"... so less accurate and vulnerable to system time manipulation. In earlier versions of the benchmark, the score was taken directly from the encoder. But after some cheated scores were discovered I changed the time measuring to use nanoTime() and calculate the fps myself.

    That could pose a challenge, because the I guess your fps calculation and the internal one of the x265 encoder can be different (even without skewing), right?

    The BenchMate integration would fix the timing inside the x265 encoder, no doubt about that. So the value in the encoder would be the correct one afterwards. Any chance we can find a solution together to change that behaviour when the integration is active? Details about that might be better via private message.

    Some of you guys/girls/transgenders are all talk.

    in HWBOT Software and Apps

    Posted · Edited by _mat_

    4 hours ago, ozzie said:

    theres always going to be " cheats " that want to try beat the system, to try get that advantage over whomever in whatever benchmark

    But they will have to try A LOT harder with BenchMate. We can't eradicate the problem, but we can raise the bar to a level where it would be less effort to just make the real score.

    Right now you can download tool "X" and beat nearly every benchmark out there (except the latest 3DMarks). They have raised the bar high enough so you need at least programming skills.

    Geekbench 5

    in Benchmark software

    Posted

    GB itself makes the timer calls, the BenchMate integration just counts the calls and checks that they are not skewed.

    Intel has nothing to do with that. Maybe they sponsored the deep learning test though, because with the new instructions in Cascade Lake-X they will rock it. ;)

    • Like 1

    The official BenchMate support thread

    in Benchmark software

    Posted

    1 hour ago, ajc9988 said:

    Please check out the behavior BEFORE activating GB5 in the new benchmate. So far, it was primarily GB5 that said something about the timer skew. After the notifications stopped popping up about the timer skew on GB5 (after I had installed the stand alone), it did show a timer skew one time in benchmate.

    Rest of the scores come up fine, except the program never pops open the browser to upload the results (which may be because the program is still reading some form of timer error).

    I haven't fully run down what is going on yet, but have posted here and at GB about it. But I'd like to rule out it being just my rig by having others play with it related to Ryzen.

    Thank you!

    BenchMate 0.9.2 should fix any timer errors. If they still occur, please file a bug report with the screenshot showing the run dialog of GB and the error dialog.

    If something else is not working as expected with BenchMate, but working WITHOUT it, please file a bug report within BenchMate right after the run fails and I will have a look at it.

    • Like 1

    Geekbench 5

    in Benchmark software

    Posted

    And here is the update: BenchMate 0.9.2

    Download | Change Log | Donate

    The Geekbench 5 scores are also much better now, more than 1000 points for multicore score on my 9900K@5G. That's because I'm not emulating 14 million QPC calls through the driver with HPET. So QPC uses what it is configured to use, but is regularly checked for skewing. That's the same method that is used in CINEBENCH as well as 3DMark03, 05 and 06.

    On a side note: Don't ever try to enable HPET on modern CPUs and run Geekbench 5. QPC will be awefully slow and your score will tell you. You will effectively bench your HPET timer latency. :D

    • Thanks 2

    Geekbench 5

    in Benchmark software

    Posted

    I've just encountered a strange synchronization bug on Ryzen with the new Geekbench 5 integration. As I've looked further I realized that instead of using the QueryPerformanceCounter() function about 2000 to 3000 times like in Geekbench 3 and 4, it's used about 14 million times during the same duration. I really don't know what to say other than that's incredibly stupid and obviously completely unnecessary.

    So there is a contention bug currently showing only on AMD CPUs in BenchMate 0.9.1 + Geekbench 5. I've already fixed it and will upload an update in a few minutes or so.

    BenchMate Integration

    in Benchmark software

    Posted

    Thanks for taking the time to reply here, havli.

    I will look into it. If it can work in a reliable way, I will use the DLL injection method to hijack everything around the x265 executable.

    A small question: Is the timing shown in the wrapper taken from the executable or does the wrapper do the timing itself (so the times differ)?

    Some of you guys/girls/transgenders are all talk.

    in HWBOT Software and Apps

    Posted

    Interesting thread title for sure. I was pretty surprised when I clicked on the thread and suddenly read my name. :D
    Thanks to your post (and the new update) BenchMate has two new patrons! Thank you, guys!

    Financially BenchMate is still in a difficult position. I've just paid another yearly license cost of 500 Euros (~ 550 USD) and that's straight out of my pocket. I myself have thought that something like this is dearly needed and that we would at least got the license cost by now. That's sadly not the case. :(

    I will be very open about donations, so here is the current standing:

    One-Time Donations: 108.37 Euro (excl. PayPal fees)
    Patreon Donations: 43.38 Euro (incl. Patreon fees, excl. payout fees)

    So we are currently at about 150 Euro (~164 USD).

    • Thanks 1

    BenchMate Integration

    in Benchmark software

    Posted · Edited by _mat_

    @Mysticial

    Sorry for the delay. I was very busy finishing BenchMate 0.9(.1).

    On 8/28/2019 at 11:27 PM, Mysticial said:

    I don't know enough about DLL injection to fully comment on it. But I'm guessing this is aimed at all the "frozen" benchmarks that are no longer maintained?

    Yes, either frozen or currently "out of reach". It's still very early for BenchMate, so approaching well-known developers like Futuremark/UL or Geekbench with the new concept would be very risky (also for them).

    On 8/28/2019 at 11:27 PM, Mysticial said:

    The main problem with this approach is that this essentially subverts the timers for the benchmark. So if BenchMate can do it, then any other program (including a cheat tool) can do it as well. If the validation and HWBOT submission is still handled by the benchmark, then this approach is very vulnerable. You can just DLL inject your own timers and completely fool the program into sending a bad score to HWBOT.

    (Translation: Everything we have now is completely broken. Yes we already knew that.)

    Exactly! Everything we have can be easily broken and I've done this during research for BenchMate. For example all versions of 3DMark and PCMark can be hijacked to run with unnoticed settings like LOD, different resolution and much more. I dug deep into Intel's XTU as well and written a full analysis on its security vulernabilities. For XTU they made some really horrible choices.

    On 8/28/2019 at 11:27 PM, Mysticial said:

    Thus, it seems that the only source that can be trusted to submit a score to HWBOT is BenchMate itself. But BenchMate doesn't know when the benchmark started or ended unless it tries to parse program output.

    For y-cruncher, this will work conceptually since it has a parse-able output that can be trusted (the validation file). But if not done correctly, it will conflict with y-cruncher's own internal protections. My guess is that if you run y-cruncher with BenchMate DLL-injection, then you change the base clock in a way that skews the timers, BenchMate will report the correct score. But y-cruncher itself will detect a clock skew and block the score.

    This is probably fixable if we pull the right strings.

    BenchMate uses different low-level timer facilities like HPET, that are accessed via its own kernel driver (which does much more btw). These low-level timers are then either injected or tracked with BenchMate and correctly verified by multiple time sources (not API functions!). In the end BenchMate does not use any timers that are linked to the bclock/ref clock.

    How do you detect clock skews in y-cruncher? Which timer functions are you using?

    On 8/28/2019 at 11:27 PM, Mysticial said:

    So it would need to either be done in a wrapper, or some "official" DLL side-load into the main binaries that is officially supported by the program. Both approaches are messy. In either case, I would need to mock out the relevant timers.

    For the wrapper solution, the "trusted" timers are only available in the wrapper (where BenchMate lives). Every important timer call will require a secure RPC over either SHM mapping or TCP. y-cruncher currently has a TCP stack, but it's for a different purpose and it's not secure.

    The wrapper is written in Java, so it's easily reversable and debuggable. I had a quick look just yet and although it's minimized, it's still very clear what is happening. Sadly, it would be easy to get the encryption key as well. But not as easy as it was with HWBOT Prime and the x265 wrapper - that was a matter of minutes.

    What I'm trying to say is, that passing data through the wrapper isn't very secure. It would be best to directly grab it from the inner benchmark executable. This way it doesn't matter what language the wrapper is written in, it's just a nice way to configure y-cruncher and display insecure, but eye-catching results. To do this I suggest some kind of one-way protocol to transfer readonly data to BenchMate. That could be done with accesses to files, that don't exist but will be caught by my hooks into the process. As for the timing functions, if you don't want to include BenchMate as a dependency to y-cruncher, they need to be "injected" in some way.

    Well, maybe DLL injection isn't the worst way to do this and it wouldn't break anything on your side as long as these API calls are happening (any call to QPC for example, and the CreateFile/fopen invocations for beginrun, endrun and result).

    On 8/28/2019 at 11:27 PM, Mysticial said:

    Other Thoughts:

    From a scalability perspective, it might be worth considering going one step further with BenchMate and put the HWBOT integration into it. Each benchmark then gives a score (and relevant metadata) to BenchMate and it will handle everything from there. This will also allow BenchMate to append any additional metadata that the benchmark doesn't track.

    That way everybody doesn't have to reimplement the same thing. Likewise, everybody doesn't have to update it when the HWBOT API/protocols change. And validation bugs and vulnerabilities only need to be fixed in one place. (the flip side being that any vulnerability that is found will likely apply to all benchmarks)

    Exactly! The wrapper is too transparent for this kind of tasks anyway and I guess it's best if only one of us has to struggle with the peculiarities of the HWBOT submission API. :D

    Geekbench 5

    in Benchmark software

    Posted · Edited by _mat_

    Geekbench 5 has landed today. It officially requires Windows 10 (64 bit) for the first time.

    I've looked into it from a technical standpoint and it doesn't fix anything related to timers or security. It uses QPC and GetSystemTime(), both highly unreliable and easily attackable, and both will skew with bclock/reference clock on Pre-Skylake and all AMD CPUs on Windows 10. The negligence seriously amazes me.

    BenchMate to the rescue! I've already integrated Geekbench 5 into BenchMate and released a new version: BenchMate 0.9.1
    Geekbench 5 is also bundled with the "Big Package", no need to install it.

    218319615_BenchMate-Geekbench5.thumb.jpg.41f4d3b8f82a6e9fe507f42f9ef84c19.jpg

    I've also taken the liberty to add Geekbench 5 as a beta benchmark here on HWBOT:

    Geekbench5 - Single Core: https://hwbot.org/benchmark/geekbench5_-_single_core/
    Geekbench5 - Multi Core: https://hwbot.org/benchmark/geekbench5_-_multi_core/
    Geekbench5 - Compute: https://hwbot.org/benchmark/geekbench5_-_compute/

    You can directly upload into these categories inside BenchMate or by uploading the .hwbot-file manually.

    • Like 3
    • Thanks 5

    The official BenchMate support thread

    in Benchmark software

    Posted · Edited by _mat_

    BenchMate 0.9 is here!

    After six weeks of (absolutely necessary) legwork I'm finally releasing the next major version of my Benchmark Verfication Software. All reported bugs have been fixed, a good part of the benchmark integration rewritten. I also put more effort into supporting Windows 7 and have extensively tested it on multiple platforms.

    If you like where this is going, spread the word and bench your stuff with BenchMate whenever possible. This is the most secure, reliable and trustworthy option to present your overclocking achievements. ?

    Download | Change Log | Donate

    • Like 6
    • Thanks 1

    BenchMate Integration

    in Benchmark software

    Posted · Edited by _mat_

    I'd like to invite all active benchmark developers to discuss integration into BenchMate. If you have any technical questions about BenchMate or the integration, please feel free to ask here.

    In general there are two ways of integration:

    1) External Integration

    A DLL is injected into the benchmark's process and all of its child processes. Several WIN32 and CRT functions are hooked as needed to inject accurate timing and file integrity hashing as well as result capturing.

    2) SDK integration

    One of the next releases will bring native BenchMate integration. The SDK will be available for C/C++ first as a static library. There will be several classes available to help you use BenchMate's features in your benchmark. Reliable timer functions, secure string classes, installation verification, file integrity hashing and much more. Additionally, the SDK integration lets BenchMate know when a run is started, ended or canceled and securely transfer results to the client for validation. No DLL injection is necessary here.

    Let's get this going! :)

    @Mysticial @havli

    HWBOT Y-Cruncher Pi-25m Benchmark Validation Regulations

    in Old HWBOT Rules (pre 2021)

    Posted

    I hear you and you are completely right. If you think it to the end, and that's what we devs usually have to do, a screenshot is for viewing pleasure only and should have nothing to do with validation.

    Did you have a look at my latest project, BenchMate? It would provide your benchmark with the necessary hardware detection like frequency, voltage, temperatures and so on. It also adds another protection layer to the benchmark process (and child processes) - specifically DLL injection for Timer API hijacking. Last but not least you would not have to bother with timer reliability issues, there are functions available by a low-level driver that give access to HPET without enabling it for the whole OS (as System Clock Source).

    What language is your wrapper written in? I am aiming to provide a SDK for benchmark developers to add BenchMate's features directly into it. It will still take some time before that's finished, but there is the possibility to add integration externally (by hijacking API functions).

    Let me know if you are interested.

    The official BenchMate support thread

    in Benchmark software

    Posted

    3 hours ago, bigblock990 said:

    This is not gpupi problem, its nvidia driver "feature". Cuda is compute so the driver sets p2 instead of p0, which results in full core clocks but lower mem clocks.

    Exactly.

    There is also a GPUPI support thread, which would have been a better place to ask this. You're sadly not using BenchMate judging by your screen.

    • Thanks 1
×
×
  • Create New...