FM_Jarnis

Just a heads up; Futuremark is following this development with great interest and on the most basic level, anything that improves real world performance in games is obviously a good thing and as long as benchmark score improvement and real world gaming experience improvement match, benchmark workloads themselves are valid. On the other hand, benchmark scores are useful only if all factors that contribute to the score are known - if you can't tell apart runs from existing normal system and hardware-identical another system with this software in place (except that the second case would have considerably higher number as a score), it would render benchmark scores somewhat less useful. We're investigating and I would imagine that the first priority is to have an ability to detect something like this and flag the scores accordingly. Edit: Also it should be noted that a lot depends on what effect this has on image quality. Tessellation tweaking isn't comparable because it changed image quality.

Just FYI, Catalyst 12.1 is now approved.

No, this is purely due to no combined AMD driver available. We do not approve beta drivers. 12.1 will be approved this week. It doesn't support 7xxx series, so... Edit: Also note that GTX 560Ti experienced exact same issue.

03 has built-in SystemInfo which cannot be updated (and no patch is planned, it is no longer supported) - it can go boom on new hardware that didn't exist when it was released. For 06 (or later), install latest SystemInfo, it most likely fixes the issue.

Yes, 1.0.3 requirement for competitions/HoF is not yet live because we want to give people time to ugprade their benchmarks. Well, that and the fact that we don't want to push major 3dmark.com code updates right before the holidays. Our programmers dislike the idea that they may have to spend the holidays fixing stuff at the office... 3Dmark.com / ORB validation and anti-cheat stuff has been improved in 1.0.3 as stated in the patch notes. If you know still a way to get a bogus result through with 3DMark 11 1.0.3, by all means let us know and we'll see what we can do. No amount of automation can catch *everything* so contest related stuff needs to be manually validated in case of suspicions anyway and really it is completely up to you what standards you enforce in your contests and toplists etc. Futuremark requires valid 3dmark.com submission for Hall of Fame and for FM-run contests. They will require 3DMark 11 version 1.0.3 soon. Anything beyond that is up to the individuals managing each contest/site that uses 3DMark results. FM recommends that only the latest version of each benchmark is used.

We can't. A lot of the validation is done on the 3dmark.com site. It is the only way to make sure the validation is not hackable. Any indicator on the GUI that would be based on what 3dmark.com says would be complex to implement and still hackable. The only real way to ensure that the result is not tampered with is to check the result on 3dmark.com. I know you guys use screenshots and all that, but you really should also require a 3dmark.com result link which indicates that the result is valid. On 3dmark.com, we may do that on a later date. We don't want to do it instantly because there are millions of 3DMark 11 installations out there and people would get very very pissed if they run the benchmark and then as they go to submit (3dmark.com view is the only one on basic edition) it would tell that they would have to patch and rerun before getting a result. Basically once large enough percentage of all submits are 1.0.3, we may disallow previous versions completely. If I recall right, at the moment 1.0.1 is the minimum for getting a result (1.0 would just tell you to upgrade). But again, this will influence HoF sooner. The only reason this will take a bit is because it means a change in HoF code and we are already rewriting the whole page, so the change will come with the HoF page update (it will also have new features!)

Correct. We plan to change that in January.

You will be able to submit to 3dmark.com and get a link even on score that is not officially valid. But it won't appear on FM.com hall of fame page unless it is deemed valid. The 3dmark.com result page will also have this box that tells how and why the score is not valid for Hall of Fame purposes - stuff like unapproved drivers, tessellation tweaking on AMD drivers or if the hardware isn't properly detected etc.

This one is in the tracking database as a confirmed issue but I do not know if/when PCMark Vantage will receive a patch. There is also some uncertainty if the bug is actually in Microsoft code (naturally it could be worked around in our code, but that may require full revalidation and testing cycle for all benchmark results as such a workaround would affect the run on all platforms). So at this point the answer is "I don't know". I'll poke the PCMark lead programmer when he gets back from the Christmas holiday and check if there is any news.

1.02 at least. Not sure about 1.01. But it won't count for very long, we are going to change it.

Will add that to the feature suggestion list internally and we'll take a look in January.

PCMark 04 is so old and unsupported, this will not be fixed. Sorry. As for PCMark 7, I'm not sure what the problem you are referring to is? We do have PCMark 7 patch in development for (tentatively) early 2012 launch and there are some issues that are getting fixed but I'm not certain which one you refer to.

For 3dmark.com, you can see what benchmark version was used for each result. Regarding FM Hall of Fame, our web team is in process of completely revamping that page and a new version should appear in Jan/Feb - included in that is enforcement that HoF only shows results from the latest version of each benchmark so for Hall of Fame purposes, old scores will be kinda-invalidated at that point. Up to you how you deal with this in hwbot.org or in any other toplists etc. - the version used (Benchmark and SystemInfo) is visible always on 3dmark.com so it can be verified for any benchmark result that is submitted to the 3dmark.com service. If you have further feedback on this, just toss me an email at info [at] futuremark.com and I'll happily forward it to the people who make decisions related to this at FM. All feedback is always welcome and will be considered. This is especially true for matters that relate to our benchmarks and any competitive benchmarking; we are making the product for our users and naturally want it to be relevant and fair. For example, we may consider adding a line to the result validity warning box on 3dmark.com (the one that includes warnings for unapproved drivers / tessellation controls etc) for results obtained with older versions. Would that help?

Sorry, no idea. It would appear that for some reason the GPU isn't fully stressed on GT4 on some runs. As every run asks the driver to do exact same things, I can only summarize that there is something funky going on with the drivers. We have not seen this kind of effect in-house and considering the thousands of runs the benchmark went through during pre-launch verification testing, this kind of massive score disparity randomly on some runs would have stood out like a big flashing beacon. So I'm fairly certain that this is not a 3DMark 11 bug, but an issue with the rest of the system doing what 3DMark 11 asks it to do during the benchmark run.

Physics? Or Combined? That number-of-GPUs-switch / SI enabled should not affect any other test except Combined Test. In combined test if the switch is set to something other than the actual GPU count (or auto, as per what SI detects), yes the score changes because the test workload is different. GPU count determines how many copies of the GPU physics simulation is spawned. Normally there should be a copy of the simulation running on each individual GPU (to eliminate cross-GPU data traffic on physics simulation state). The whole "GPU physics using DirectCompute" is fairly complicated issue on multi-GPU setups. In any case, AMD just released new drivers and we're going to investigate in-house if anything has changed with either of them (11.12 or the 12.1 preview drivers) and will bug AMD again if there are still issues. I've also suggested internally that FM should make a full ("lots of big technical words") disclosure on this specific issue - we'll see if it can be made happen. Could explain exactly what is going on.

Ask AMD? Currently FM has no solution for this problem and the issue is within the video drivers. We have told AMD this (several times) and can only wait for them to correct the issue. All Single GPU configurations work 2 GPU Crossfire works SLI with 2, 3 and 4 GPUs works

Disabled Tessellation invalidates the score on any configuration. You are effectively telling the driver to skip work that the benchmark requests it to do. You can do so, but the score is meaningless. Naturally up to you under what "rules" you benchmark. I'm talking about FM hall of fame and the result being flagged in 3dmark.com result browser. Just disabling SI affects the score only on multi-GPU configurations (and only with Combined Test).

Not too shocking. PCMark series has always been part hardware benchmark, part Windows OS benchmark - it runs a lot of Windows component code for workloads. Tamper with Windows and yes, the results can change a lot. It is really up to you what you think is "okay tweaking" and what is "cheating" in this context. (ultimate PCMark 05 tuning: Write your own Windows API compatible OS optimized for running PCMark 05! )

Just a heads up... 3DMark 11 version 1.0.3 release notes * Bullet physics library updated to 2.79 in order to improve compatibility with current and future CPUs and GPUs * Manual GPU Selection option now available on the Help-tab * Updated SystemInfo module to version 4.6 with improved compatibility with current and future hardware * Improved error handling and messages * A new “More” tab with information about new Futuremark benchmarks * Professional Edition activation now requires online key verification SystemInfo Version 4.6 Release Notes * Updated CPUID SDK * Added new security measures against benchmark result file tampering * Updated system scanning components to resolve reported compatibility issues with existing and future hardware http://www.3dmark.com/3dmark11/download/update/ SystemInfo update that is included in this patch also applies to other benchmarks. It is available separately here http://www.3dmark.com/support/systeminfo-updates/ ...and due to the second line of SystemInfo update notes, we strongly recommend that for any contests and toplist purposes, runs only with 3DMark 11 1.0.3 and SI 4.6 (or later) should be approved, effective immediately. 3DMark.com will also start rejecting submits from older versions in the near future (probably around mid-January, to give plenty of time for everyone to update). Hall of Fame will get picky sooner. The update also influences scores slightly but differences should be within margin of error (we're talking 1% differences) and the cause is the new Bullet physics library. Old scores are not invalidated but future contests should require the use of latest version to avoid issues related to this. Also those who may have access to pre-release hardware; Yes, this update fixes the issues 3DMark 11 had with Ivy Bridge integrated GPU (that were actually problems with the Bullet physics library, hence the update).

Be advised that any run with SystemInfo disabled produces incorrect results from Combined Test with multi-GPU setups. The benchmark needs to know the number of GPUs for that test. The issues you are seeing with 3+ Radeon GPUs in this test are due to AMD driver issues. We've told them, they have so far disregarded the issue. Sorry, out of our hands. The result from SI-disabled, while the score seems great, it is meaningless. The benchmark doesn't do the same workload that it would do with SI enabled (or with SI enabled SLI run).

It is a complex issue and nothing has been decided on that yet, but obviously we are aware that it is a key aspect that has to make sense...

Indeed. You may also probably do some 1+1 math and figure out what would be required from a benchmark to do so, across all that hardware space... can't go into all the tech details yet (mostly because all of that isn't yet set in stone, heck a lot about Windows 8 isn't set in stone yet), but it won't be a "3DMark 11 Second Edition" - it will be a complete new benchmark.

Just a FYI... http://www.futuremark.com/pressroom/pressreleases/60054/ Not much yet, but "it's coming". Key bits off the PR: Edit: Oh, and there is one concept art image (*not* a screen capture)... http://www.futuremark.com/images/products/3DMarkWindows8_ConceptArt.jpg

We have deployed a new update to 3DMark.com which allows the approval of new ATI/AMD Catalyst drivers for 3DMark 11. 3DMark.com can now verify new benchmark submits for the Tessellation Control settings used. All submits that have been run with SystemInfo 4.2 (or later) and with default Tessellation Control settings ("AMD Optimized"), without touching any of the tessellation controls should now show as approved. Old submits (prior to SystemInfo 4.2) unfortunately cannot be verified, so while they may stop complaining about unapproved drivers, they are still flagged with a problem. There are now two new possible error messages in 3DMark.com for 3DMark 11 submits with ATI/AMD hardware. Quoting from the FAQ: Catalyst 11.9 drivers are now approved for 3DMark 06, 3DMark Vantage and 3DMark 11. Also all previous Catalyst versions between 11.1 and 11.8 that were approved for 06 and Vantage are now approved for 3DMark 11 as well. If you have any issues related to this update or ATI/AMD driver approval, please let us know. Happy benchmarking.

This is normal. Bullet library was updated between patches and the scores of the Physics and Combined tests changed slightly. What you are seeing is exactly this. The difference was considered small enough so that we did not invalidate results from older versions but for any contests etc. there should always be a rule to use the latest available version at the start of the competition or this kind of issue may crop up...