PCMark05 , keep or remove from hwbot.

[Massman]

We have different opinions and monitoring this thread. This discussion is interesting!

[chispy]

We have different opinions and monitoring this thread. This discussion is interesting!

 

Thank you Pieter , always your opinion and saying its appreciate it.

 

Kind Regards: Angelo.

[Massman]

We built a wrapper around heaven right? Couldn't we build a wrapper around pcm05 and for example - have it report memory mapping to make sure ramdisk is not used?

 

I see we're getting predictable .

 

About a week ago, we spend a little bit (read: couple of minutes) trying to figure out if we could make some application for the benchmark. We can easily extract the subtest results from the memory mapping and put it in a separate window. All subtests could then be visible.

 

I'm sure we could also figure out what harddisk has been used, but that would require more time to be invested.

 

Also, I have no idea what Futuremark would think of this. Since the software is still commercially available, I guess it's possible they won't like us hacking the software to extract information

[pro]

Also, I have no idea what Futuremark would think of this.

 

dr dre says "who kare"

 

[El Gappo]

Ahahhaha

[chispy]

Hahaha... Poor Andre.

[Gautam]

dr Dre Says "who Kare"

 

Lol

[Maxi]

The 220 cap was actually hwbot's idea, as was the 300 cap that was debated and ultimately unused. There were good reason for doing that which apparently everyone has forgotten. I don't care about this benchmark so whatever the community decides is fine w/me. Removing all caps would be the easiest from a mod perspective, but what's the point if there is zero useful comparison info from the benchmark between platforms? That is why benchmarks exist lol.

[Gautam]

The proper alternative to the cap is active moderation. The cap was just a lazy alternative. If it was hwbot's idea then I guess it's one more thing we can blame on masbo . No different from how there used to be a cap at 40k 01 way back in the day when OPP broke it and needed to get it lifted. These days its actually MUCH easier to break 220 with a legitimate setup than it is to stay under it.

[SteveRo]

These days its actually MUCH easier to break 220 with a legitimate setup than it is to stay under it.

 

This is very true - seems like just two SSDs in R0 using the sandy bridge controller is all it takes to break this these days.

[Massman]

So, if we'd keep the bench and update the rules, which are the rules we should include?

[topdog]

Keep the same rules and add :

No Powertoy tweaks, if you allow this it means that all previous submissions will be at a disadvantage

Keep XP startup at 220 for the same reason as above

Two screenshots that show the full results of the subtests

A valid Futuremark orb link for all submissions

[Massman]

Powertoy should be allowed, I think. This is just a GUI for registry adjustments.

[chispy]

So, if we'd keep the bench and update the rules, which are the rules we should include?

 

1. allow powertoy or any other encoder , as it is right now you cannot accurate tell if powertoy it is used or any other encoder has been used or not.

2. No 220 cap for xp start up , it just makes sense as the hardware in the market its more than capable of that since 2007 and we are in 2011, Please remove completely that cap and will be a fair level playing field + we can easily spot softwareRam Disk Drives and report them on sight , and those shall not be allowed never as well as software MFT

3.IE change for another browser or to use any IE and any tweaking ,optimization and manipulation goes , as it is right now you cannot accurately tell whats going on with that subtest.

4. Mandatory full screensshot of all subtest , memory cpu-z screenie, cpu cpu-z screenie , motherboard cpu-z screenie, GPU-z screenie ,Very important will be the screenshot of HDD used for subtest as target HDD from device manager , Raid Card used , ICH10R used , I-Rams and how many used , Acards and how many used , SSDs and how many used , Raid0 or what type and how many Raid0 arrays used ,Normal HD used and how many. This will determine and will help if the score numbers its legal and no software Ram disk its used.

5. Always a Futuremark Link included.

6. Everything and anything goes except Software Ram Disk and MFT Software , everything else allowed.

7.If i think of something else i shall express my opinion later.

 

 

Much easier and less pain to moderate that way , any mod will be able to moderate PCMark05 accurately.

Edited July 27, 2011 by chispy

[S_A_V]

I list here all that I know can be used for better pcmark05 result. If you consider some of the following as cheat (or forbidden tweak that must not be allowed) - please add it to the new edition of pcmark05 submission rules:

1. Internal windows tools (regedit.exe, taskmgr.exe, msconfig.exe, services.msc)

2. Aero Tuner - small third-party application to tweak Windows Aero settings.

3. Intel RST driver and RAID options like stripe size and write-back cache.

4. Moving the mouse during the test

5. Changing desktop resolution (800x600, 1024x768, etc).

6. PowerToy (yes, it's nothing than front-end GUI for WMV-encoder registry keys): all allowed, all disallowed, or all allowed except "Encoding Width/Height" options. Only "Encoding Width/Height" options gives huge boost to Video Encoding subtest (8000-9000), all other options gives very small increase of result.

7. Easy accessible options from internet browser's menu (disable/remove plugins, pictures, html-formatting, scripts, etc).

8. Windows Shell (explorer.exe) and Internet browser (iexplore.exe) - advanced settings in registry:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\

9. Easily accessible video driver setting (Catalyst Control Panel, NVIDIA ForceWare Control Panel).

10. Advanced video driver setting (RivaTuner's Power User settings, video driver registry keys, etc.)

11. Usung GPU-accelerated rendering/encoding/decoding/etc (OpenCL, DirectCompute, CUDA, etc.). Unsure if it helps to the current versions of codec and browser, but this situation can be changed in any time - with new versions/generation of software/drivers/hardware.

12. Unreleased OS, alpha/beta builds leaked to public (Windows 8).

13. Changing PCMark storage partition on-the-fly during the test (subst, diskpart, self-made scripts and/or applications). Using different storages for different hdd subtest. For example, general usage on ACARDs with discrete controller (600+ MB/s) and in the same pcmark run - virus scan on SSD with integrated controller (1300+ MB/s virus scan).

14. Using third-party video/audio codecs and internet browsers (I mean anything not developed by Microsoft).

[Gluvocio]

if you allow this it means that all previous submissions will be at a disadvantage

 

I like this thinking.

 

Also S_A_V said interesting things. I believe that what he's wrote could be the new rules.

[Guest subzero]

i agree

[Hondacity]

remove is only 24%

 

it won't be removed

 

stay happy

[knopflerbruce]

If we ever want to remove the xp startup cap, now is the right moment. We're still changing the rules, so older submissions will already be at a disadvantage to new ones.

[S_A_V]

Keep PCMark05 and the Rules as it is.

This is not the case. Current rules not compatible with current real moderation of this benchmark.

Scores done with powertoy (wmv codec registry settings) blocked all the time, but powertoy still not added to the submission rules as forbidden tweak. Please add this to the rule or stop blocking such scores. Otherwise current rules looks confusing.

The same with new browser tweak found by Gluvocio and his blocked score. I can confirm such high web page rendering can be done with only browser settings. There is no info in the rules that browser settings can not be changed.

 

Keep PCMark05 but make new , improved up to date Rules, whats allowed and whats not.

Yes, rules must be updated with list what is not allowed.

 

Keep PCMark05 update the Rules as anything and everything goes and remove the 220xp start up cap.

220 MB/s XP Startup cap was stupid idea from the beginning. But if we remove it now - a lot of old hard earned scores will be easily beatable. Such action may looks disrespectful for "storage guys" who benched PCMark05 for ages with a lot of different hardware.

Personally, I don't have so much PCMark05 scores, so removing the cap is ok for me. May be we need start separated vote about this topic - keep or remove cap?

But if we decide to remove - it must be removed completely. Not changed to the new cap (300-400-500-1000 MB/s, etc).

I got over 500 MB/s XP Startup with current generation SSD in RIAD0. Something like RevoDrive X3 or next gen SSD will raise it even higher. Software RamDisk gives over 3000 MB/s in every HDD-subtest, this is way more than current hardware storage systems, so it can be clearly visible in results. Of course, this numbers can be done with current generation hardware, not something ancient. On older platforms results will be proportionally smaller.

 

Remove PCMark05 from hwbot.

PCMark05 still support validation system (ORB). HIGHLY tweakable, same as SuperPI 32M and 3DMark01.

It's not need 4xGTX580 for Top20 score. Two SATA3 SSD in R0 on integrated controller is enough for 40K+ score (I prove it already).

It measures overall system performance, not only CPU or RAM or GPU.

For me PCMark05 is still best version of all PCMarks.

PCMark04 is buggy (grammar check), discontinued (no more official patches) and lack of support multicore CPUs.

PCMark Vantage way too long (more LN2 waster than Unigine!), way less tweakable than PCMark05 and needs to install OS to storage that it measures.

PCMark7 light versions of PCMark Vantage, almost the same. Less depends on storage subsystem performance than both PCMark05 and PCMark Vantage.

Edited July 27, 2011 by S_A_V

[pro]

4. Mandatory full screensshot of all subtest , memory cpu-z screenie, cpu cpu-z screenie , motherboard cpu-z screenie, GPU-z screenie ,Very important will be the screenshot of HDD used for subtest as target HDD from device manager , Raid Card used , ICH10R used , I-Rams and how many used , Acards and how many used , SSDs and how many used , Raid0 or what type and how many Raid0 arrays used ,Normal HD used and how many. This will determine and will help if the score numbers its legal and no software Ram disk its used.

 

 

i certainly agree with the proposed rules so far from Chispy and SAV,

 

this part of showing device manager, how many raid cards used, how many devices, we need to think about that... i know i certainly dont have software installed on my OS that displays my whole raid configuration and how many devices are making up my raid.. and i am sure when i am loading 4 windows of cpuz + gpuz + the extra tests + device manager + some areca raid web page controller + possibly and ICH based controller program i will probably crash with my gpu at 1450 and my chip at -150 or whatever its at,

 

there is a point of taking that too far... but you could consider just asking people to open device manager and expand the "Storage controllers" tab, past that its all a bit grey i think...

 

 

6. Everything and anything goes except Software Ram Disk and MFT Software , everything else allowed.

 

so everything allowed? you sure you want to make that a rule? im happy with it but i dont want to hear any complains when..

 

 

thats all day every day

[S_A_V]

Storage system may looks not so informational in software. By default it may be something like "Array_001", "Array_002"...

 

Besides of risk to crash extreme overclocked rig while run all this applications and windows for the screen, if we will require a lot of things include in screen - more people will make screens in huge desktop resolution, they will not fit in 300kb limit, so we will see a lot more highly downsized hardly-readable crap...

 

I think better way is include a rule to always attach real picture of the system with storage used for the score. But not for all, only for those who scores high enough in HDD subtest (over 220 MB/s in XP Startup, for example). Or as alternative - picture only for top20, no matter how much in hdd subtests.

Edited July 28, 2011 by S_A_V

[chispy]

Might be better like James proposed " device manager and expand the "Storage controllers" tab " I fully agree, But it is very important and nesesary to write down all the details of every Storage set up used on that run (theres a space named disk details when you submit to hwbot for that , please people use it.) to avoid confusion and to help determine if software Ram disk was used or not , As S_A_V proposed pictures of the Storage set up its also a great idea. I keep seen submisions with almost 0 , zero ,nada information, sometimes not even VGA clocks used or Type of array used and Raid0 how many drives. Please make mandatory this and FM link for all PCMark05 submisions. We need the user to give full details.

 

keep the ideas coming guys , me thinks we are heading the right direction in order to keep alive and save the old PCMark05. Many ideas here will be so much helpful for the Mods and much easier to moderate.

Edited July 28, 2011 by chispy

[Massman]

I don't mind users not giving the full details on their configuration. If someone finds a neat trick, they should be allowed to keep it to themselves to some extend, right? Of course, there is a difference between finding a trick that does x2 and one that does x10.

 

In addition to the change of the rules, I'd also like to change a little policy regarding the usage of the word 'cheater'. In my opinion, this word has been flying around too much especially in the PCMark05 discussions. I already figured we could censor the word, but I don't like censorship. Anyone has any ideas?

[pro]

i have an idea id be happy to abide by,

 

and don't get it twisted, i have certainly called people out in the past, and it really is unacceptable behavior without proper proof of wrong doing,

 

my suggestion would be for reporting of submissions to be anonymous, so the person whos submission is reported doesnt see who reported them, let hwbot do the investigation and respect their findings, and all this name calling and cheat calling in public should result in a 1 week ban from hwbot..