PCMark05 , keep or remove from hwbot.

[Hondacity]

yes agree...

 

very fair ruling! wow you grew balls in minutes lol

[Massman]

Fixed.

 

Okay, let's go with:

 

- No ramdisk, ramcache or MFT software allowed

- 220MB/s XP startup cap

- Benchmark must run on default web browser (Internet Explorer, any version) and default audio/video codec

- Browser enhancement tweaks such as disabling/removing features, plugins, make-up etc are allowed

- Audio and video codec tweaks are allowed. Powertoy is, as GUI for the registry, allowed

 

Also, let's make an agreement that we come back to this thread in 3 months (9th of November) and evaluate if these new rules work. If turns out the benchmark is pretty much 'unsaveable', we can still remove it after a couple of months. Given the outcome of the POLL, I'd say these are the two only options we have.

 

And yeah ... http://hwbot.org/forum/showthread.php?t=34852

[knopflerbruce]

So 4.5 months after we change the rules we want to remove the boints? Pffft. We had a chance to remove the benchmark, and we didn't - this is too early to start talking about that sort of stuff if you ask me. Plus, I don't see any real issues - ramcache shouldn't be allowed if we look at the current rules (you can't boot from ram cache, I guess)

[ZMBKLN]

GREAT JOB MASSMAN!

 

THANK YOU!

 

It's nice to see that submission removed, and the new rule put into place so quickly.

 

There is absolutely nothing wrong with this benchmark, other than the enforcement of the rules.

 

I myself would hate to see it go, because it truly is one of the more "Fun" benchmarks to run and get good scores out of.

 

I just wish people could get into this whole... "Spirit of the Rules" thing, and quit trying to cheat the system.

 

Anyway... Thanks for dealing with this so swiftly.

 

-David

[SteveRo]

I think yes - add no ramcache to the rules but it's probably a temporary fix.

 

Longer term - either a wrapper or replace.

 

Anvil (XS) would be best to describe what needs to be in the wrapper - or maybe just the latest version of anvil's Storage Benchmark in the screenshot - it provides a lot of info about the storage used.

 

Replace is tough - pcmark7 is awful as it currently is, pcmv is way skewed to storage - much more than pcm05. But skewed towards storage sounds great to me!

[Hondacity]

its a storage benchmark. caching bypasss the storage hardware.

 

I'm with SteveRo

[SteveRo]

So 4.5 months after we change the rules we want to remove the boints? Pffft. We had a chance to remove the benchmark, and we didn't - this is too early to start talking about that sort of stuff if you ask me. Plus, I don't see any real issues - ramcache shouldn't be allowed if we look at the current rules (you can't boot from ram cache, I guess)

 

Yes, you can boot from a drive assisted by ramcache - that's the problem.

 

edit - actually only part of the problem <rolls eyes>.

[Bobnova]

Woot. Thanks massman!

 

I figure whatever comes up next can be dealt with when it comes up.

[knopflerbruce]

Yes, you can boot from a drive assisted by ramcache - that's the problem.

 

edit - actually only part of the problem <rolls eyes>.

 

I wonder how that works - I was thinking ramcache was something you messed with after entering the OS:p

 

maybe the HDD rule should be that you should be able to install windows on the drives used. You can't install windows on a setup with only RAM available as storage. (and if that is possible at a later time, I say we allow it - then the RAM would work exactly like a regular harddrive).

[SteveRo]

I wonder how that works - I was thinking ramcache was something you messed with after entering the OS:p

 

maybe the HDD rule should be that you should be able to install windows on the drives used. You can't install windows on a setup with only RAM available as storage. (and if that is possible at a later time, I say we allow it - then the RAM would work exactly like a regular harddrive).

 

Intel Rapid Storage Technology is a basic form of disk cache that uses system dram for write combining. Many SSDs have there own dram inside. Raid controllers have controller dram, some have a lot. Some of the new pcie direct connect drives have this capability built into the storage driver (ability to use system ram as cache). The amount of system dram that can be used by these drivers will go nowhere but up. Good for system speed but a dilema for benchmarking.

[Dead Things]

It's true. I use RAM caching in my day-to-day computing. It's non-volatile like ramdisk since all data is stored on a physical drive. But the RAM cache learns over time what data is accessed most often and caches it in RAM for faster access thus improving your overall storage experience no matter how large or small the cache (obviously, the bigger the cache the better for day-to-day stuff).

 

It's a bit of a philosophical dilemma, I fear. RAM Disks are ruled out for PC Mark 05 since they do not reflect the real-world computing performance that PC Mark is meant to measure. RAM Caches, on the other hand, do reflect real-world computing. And yet the way PC Mark perceives a RAM cache is somewhat exaggerated since basically all test data can be cached in even with only 2GB installed.

[Dead Things]

Example results:

 

Without RAM cache (average over three runs):

XP Startup = 84.938

General Use = 62.592

Virus Scan = 218.089

 

First run with 1.5GB RAM cache enabled:

XP Startup = 297.482

General Use = 149.934

Virus Scan = 239.430

 

With RAM cache enabled (average of next three runs):

XP Startup = 1251.457

General Use = 1008.222

Virus Scan = 1516.974

 

...one might argue since most people tend to use their computers for similar tasks on a day-to-day basis and thus can take full advantage of a RAM cache if they were so inclined, that these results are actually a reasonable approximation of real-world performance.

[knopflerbruce]

This is slightly OT, but is ramcache just a pcmark05 issue, or is the same thing a problem in 04, vantage and 7 as well?

 

If we decide something isn't allowed for 05, does that mean it's not allowed for ANY benchmark?

[Christian Ney]

there is no impact on PCMark04.

 

Only PCMark05, Vantage and 7 have hdd tests

[Christian Ney]

Add "ramcache" to the disallow list?

 

I'm fine with that.

 

So first you say it s allowed and now it's no more ?

 

Christian 2 - 0 Massman

 

(For those who doesnt remember I did the same back in the 3dmark99/00 days) I proved that the "trick" was bs.

 

But isnt the point of pcmark to "show" daily use performance ?

I mean I know guys using ramcache everydays.

 

Even retail hardware that Seagate hybridrive use "caching" and algorithms to boost performance, ramcaching is just a "tweaked"/improvement of it.

 

Is Seagate hybdridrive also disallowed ?

 

Is ram caching disallowed for pcmark vantage and pcmark 7 as well ?

 

ho btw, there is a rule that says when a rule is made after score been submited we are not allowed to remove/block it no ?(score without screenshots before xx/xx/xxxx, score without cpuz links before xx/xx/xxxx, score using a different version of the software) ?

 

My score was valid when I submited it, even massman approved it.

 

 

Rules rework, I ask for it every days, nothing done

Edited January 8, 2012 by Christian Ney

[Mr.Scott]

PCM05 has turned into a hackers bench. It's no longer about hardware at all, it's about what rule you can exploit next.

Dump it.

[Sweet]

PCM is softwareclocking no overclocking, i dont like this, if it continues, at last is cheaper software, hardware is more expensive

 

I have the software license (SC)

[KONAKONA]

Why does no one seem to listen when I suggest that photo verification should be used? I mean, you can still give PC05 the smack down with storage speed if you are willing to spend the money on some serious raid cards and SSDs or ramdrives or whatever. Everyone in this thread seems to be butt blistered because people are supposedly using software ram drives and such, if you have to show a picture of it then it should be very obvious if your score is in the ballpark or not.

 

Here are the rest of my opinions on this mess:

 

*) Regardless of what PC05 is supposed to show performance of, the rules have always been more or less that if you are using hardware to do better, then it's allowed and if it's software then it's not. Even if a software ram cache is something you might use in a daily situation, it doesn't fall within those rules. hwbot has never been about using the benchmarks for what they were originally intended for, or we wouldn't have benches like 01 and 03 where it's more of a CPU benchmark than a GPU at the end of the day. hwbot is about taking a benchmark and seeing how good of a score you can get without "cheating".

 

*) I'd still like someone to explain to me what the XP startup limit is doing anymore. I mean, PC05 is still very much a storage benchmark. It limits people that have 20 SSDs in raid just as much as it does someone with a software ramdrive, and again picture verification would be able to solve the whole software issue. Unless it's there to try and "even out" the bench, so to speak. I don't have enough experience with PC05 to know if it is or not.

 

*) PC05 has a lot of tweaks. All of the sudden, people hate tweaks and want a benchmark to be all about how much hardware speed they can throw at it. Why don't you go do heaven or 3d11 or something like that then? Benchmarking is just as much about tweaking the bench as it is tweaking your hardware. Again, if it wasn't, then we wouldn't have 3d01 on the site. Personally, I love 3d01 because of it's tweaking, and I don't think many people would disagree on that point. The only reason to complain about tweaking is because you don't know how to do the tweaks. I don't get my panties in a jumble when I'm doing 3d01 and don't know all of kingpin's tweaks.

 

That's my wall of text for the day. I'm hungry.

[Gautam]

Nice post, I particularly like the last point. However, I'm starting to feel that the time has come to drop PCMark05. It's true that we could use photos and other forms of verification in an effort to enforce the set rules, but there is a fundamental reason why PCMark05 always finds itself in the center of controversy and benchmarks like 01 and 32m do not. The benchmark itself is full of holes that the Hwbot rules simply make an effort to patch up. We can keep trying to add and enforce rules, but I feel it's going to be a never ending battle.

[SteveRo]

Konakona - +1 on the picture verification

 

Mr.Scott and Sweet - In pcm05 raw horsepower still trumps tweaks -

 

http://hwbot.org/submission/2232179_mtech_pcmark_2005_core_2_e8600_(3.33ghz)_39435_marks

 

 

http://hwbot.org/submission/2238915_s_a_v_pcmark_2005_phenom_ii_x2_560_be_37905_marks

[Sweet]

Konakona - +1 on the picture verification

 

Mr.Scott and Sweet - In pcm05 raw horsepower still trumps tweaks -

 

http://hwbot.org/submission/2232179_mtech_pcmark_2005_core_2_e8600_(3.33ghz)_39435_marks

 

 

http://hwbot.org/submission/2238915_s_a_v_pcmark_2005_phenom_ii_x2_560_be_37905_marks

 

Yes Steve, for other reasons, yesterday I saw this, I find very dubious, link first, second no, second link i think is correct ( maybe ?)

 

Ramdisk in combination ramcache, with other tweaks in windows OS (storage and virtual disks in raid), makes it very difficult to evaluate the benchmark (perhaps this...was a secret)

 

 

 

I want to clarify something, I used PCMark'05 much, i love this benchmark, but I saw with gimmicks is degraded, some valid and some not, and now as a moderator I to do with what has become the benchmark, is sad.

 

You can see in my profile that I've used this benchmark much

Edited January 8, 2012 by Sweet

[I.M.O.G.]

+1 Konakona, that was pretty well said.

 

Nice post, I particularly like the last point. However, I'm starting to feel that the time has come to drop PCMark05. It's true that we could use photos and other forms of verification in an effort to enforce the set rules, but there is a fundamental reason why PCMark05 always finds itself in the center of controversy and benchmarks like 01 and 32m do not. The benchmark itself is full of holes that the Hwbot rules simply make an effort to patch up. We can keep trying to add and enforce rules, but I feel it's going to be a never ending battle.

 

I agree on his last point. We could all run stock settings, or only permit changing frequencies - but then that isn't a competition as we'd all post relatively the same scores if we just match the other guy's settings. HWBot by nature is founded on tweaking/familiarity with benchmarks - that is where we compete. Every benchmark has tweaks, even the simplest like sp1m, pifast, or wp32.

 

In my view, PCmark05 has roughly the same amount of holes in comparison to 3D01 - the main difference is 3D01 was popular because it became a classic 3D benchmark, and PCMark05 is not as popular because it is a bit unusual in that it isn't ruled by processor/GPU speed. Both have some rather considerable tweaks and there is a lot of planning and strategy that goes into maximizing a run. Good clean fun for those who honestly want to compete and not cheat.

 

I think the ability of a benchmark to be run effectively should be judged primarily by the top 20 submissions in that benchmark. If you have a bunch of people in the top 20 that are obviously gaming the benchmark, the benchmark is broken... I DO NOT SEE THAT on pcmark05. So I'm not sure why this discussion has gone as far as it has.

 

The current PCM05 rules, in light of Kona's comments, are working and rankings are not gamed heavily in the top 20 - maybe some lower rankings go unreported or unresolved. The top places in pcmark05 are mostly split by mtech and stevero currently however, using raid ssds or raid hardware ramdrives, and it goes to show the power of strong storage in the benchmark.

 

I think that is the reason the vote clearly shows PCM05 should be kept.

 

Full Disclosure: I just invested nearly 2K in a storage setup to compete in PCM05, so maybe I'm bias, however I've looked at the rankings extensively and my conclusion is that top scores can be taken with public knowledge tweaks using current generation CPU, GPU, and storage. You don't have to know every hocus pocus tweak to compete well in the benchmark, if you use the right hardware, and can max its performance, you will do well! (Watch for my submissions when I return home from CES)

 

I think another big part of why people perceive a problem with PCM05 is because it is so storage driven, and storage is one of the few technologies that has exploded in its advancement in the past couple years in the presence of SSDs and improved caching techniques... Some seem to think this is in some way unfair.

Edited January 8, 2012 by I.M.O.G.

[Hondacity]

+1x10~inf I.M.O.G

[ZMBKLN]

+1 Konakona, That Was Pretty Well Said.

 

 

 

I Agree On His Last Point. We Could All Run Stock Settings, Or Only Permit Changing Frequencies - But Then That Isn't A Competition As We'd All Post Relatively The Same Scores If We Just Match The Other Guy's Settings. Hwbot By Nature Is Founded On Tweaking/familiarity With Benchmarks - That Is Where We Compete. Every Benchmark Has Tweaks, Even The Simplest Like Sp1m, Pifast, Or Wp32.

 

In My View, Pcmark05 Has Roughly The Same Amount Of Holes In Comparison To 3d01 - The Main Difference Is 3d01 Was Popular Because It Became A Classic 3d Benchmark, And Pcmark05 Is Not As Popular Because It Is A Bit Unusual In That It Isn't Ruled By Processor/gpu Speed. Both Have Some Rather Considerable Tweaks And There Is A Lot Of Planning And Strategy That Goes Into Maximizing A Run. Good Clean Fun For Those Who Honestly Want To Compete And Not Cheat.

 

I Think The Ability Of A Benchmark To Be Run Effectively Should Be Judged Primarily By The Top 20 Submissions In That Benchmark. If You Have A Bunch Of People In The Top 20 That Are Obviously Gaming The Benchmark, The Benchmark Is Broken... I Do Not See That On Pcmark05. So I'm Not Sure Why This Discussion Has Gone As Far As It Has.

 

The Current Pcm05 Rules, In Light Of Kona's Comments, Are Working And Rankings Are Not Gamed Heavily In The Top 20 - Maybe Some Lower Rankings Go Unreported Or Unresolved. The Top Places In Pcmark05 Are Mostly Split By Mtech And Stevero Currently However, Using Raid Ssds Or Raid Hardware Ramdrives, And It Goes To Show The Power Of Strong Storage In The Benchmark.

 

I Think That Is The Reason The Vote Clearly Shows Pcm05 Should Be Kept.

 

Full Disclosure: I Just Invested Nearly 2k In A Storage Setup To Compete In Pcm05, So Maybe I'm Bias, However I've Looked At The Rankings Extensively And My Conclusion Is That Top Scores Can Be Taken With Public Knowledge Tweaks Using Current Generation Cpu, Gpu, And Storage. You Don't Have To Know Every Hocus Pocus Tweak To Compete Well In The Benchmark, If You Use The Right Hardware, And Can Max Its Performance, You Will Do Well! (watch For My Submissions When I Return Home From Ces)

 

I Think Another Big Part Of Why People Perceive A Problem With Pcm05 Is Because It Is So Storage Driven, And Storage Is One Of The Few Technologies That Has Exploded In Its Advancement In The Past Couple Years In The Presence Of Ssds And Improved Caching Techniques... Some Seem To Think This Is In Some Way Unfair.

 

Qft.

[xXSebaSXx]

I'm in for option 2.

PCM05 is my favorite benchmark because its "tweakability". Even before the "new tweaks" were allowed; I used to spend hours upon hours just testing out different settings on IE, different LOD numbers, etc. I found some tweaks that had never been mentioned before, I ran into obscure tweaks on sites in languages I could not read... And at the end of the day; when I applied all those tweaks, I was able to improve upon my previous scores. Things got even better when the newly approved tweaks came about.

 

Here's my take on this...

The rule about SW RAM Disks has been in place since I joined and it was a generally agreed upon rule. So what has changed from then to today that there is such an uprising with RAM Cacheing? I may not know much about that technology, but it sounds to me that it is just fancy words for a SW RAM Disk and that's not allowed.

 

So PCM05 is more tweakable than other benches, live with it... Just because some people don't have the means to be able to get the kind of HW that is needed to compete in a benchmark doesn't mean that those that can should not be allowed to play. If that's the case; every 3D benchmark that benefits from "modded" BIOS files and "franken-TiNd" GPUs should be thrown out too. I know I could never compete with Vince in Vantage (simply because I don't have access to his resources and knowledge), but that doesn't mean that the fact that he runs 4 x 580s that have been under the TiN treatment should be reason for me to bitch about his "inflated" scores.

I simply nod and Ohhh and Ahhh when I see them posted and wonder about how nice it would be to be able to play with all his toys.

 

I guess what I mean is... "Stop stomping on the poor dead horse"... PCM05 is there for those that are willing to make the investment in HW to max it out and for those willing to spend countless hours scouring over registry files looking for a way to make IE load faster. Just because some of you can't afford the HW to compete, or don't have the time to search for tweaks doesn't mean that the rest of us shouldn't get to play the game.