Democritise HWBoints

[knopflerbruce]

I would not replace benchmarks just to replace them. Currently we have a good base of benchmarks which is working nice. Why touch a system which is running fine?

 

If there is a new one coming out we can think about replacing and older benchmark with it. As long as there are no good alternatives there is no need to change anything.

 

For example 3DMark12 is ready and tested. We can publish a poll and ask the community about the replacement. If you find a decent 2D benchmark we can for example take away the global points from PiFast.

 

 

2D needs a special treatment imo. We have single and multi threaded benchmarks. Thus a single threaded benchmark should replace a single threaded benchmark to keep a variety of benches.

 

Apart from the PCMark series (which are not really THAT multithreaded anyway), we've got 3 singlethreaded CPU benchmarks - pifast, superpi 1m and superpi 32m - plus CPUZ. The multithreaded ones are just wprime 32m and 1024m, of the ones currently receiving global points. wp32m has also reached a "ceiling" where more cores won't help. IMO we could replace one singlethreaded one with one that's multithreaded, with scaling to a very high number of cores. Cinebench, or something similar in order to actually have some nice multithreaded benchmarks for multi socket systems. UCBench is decent, but 64 cores aren't enough anymore Plus, I'm not sure if it's good enough to get globals. Doesn't feel like very secure benchmark...

[knopflerbruce]

Quality over quantity

 

I really think if anything there are already too many benchmarks, I think in hardware leagues more benchmarks make sense

 

In pro/oc league I think having the highest quality results possible is important, sometimes a single result can take two weeks to do properly, personally I'd rather see less benchmarks but higher quality results and competition

 

Bruce I think lots of people would be very sad if 01/32 ever disappeared, myself included

 

If "lots of people" care, then they won't. That's my point. If few people care, then they go. I just don't want there to be a handful of nostalgic people who can rule over the masses if they don't want them here. It's not worse to lose 32m than pifast if you ask me.

[GENiEBEN]

If "lots of people" care, then they won't.

 

There is always a minority voting for the majority. Just watch any poll.

 

Plus, I'm not sure if it's good enough to get globals. Doesn't feel like very secure benchmark...

 

I know 2 hack methods for this (including verification file / site validation), but if Massman gives it globals I will 'wrap it

Heck, if you want I can write a similar one with GPU acceleration and replace it altogether lol.

[knopflerbruce]

The text file seems rather unsafe, but not if that part is vital.

[Sweet]

The text file seems rather unsafe, but not if that part is vital.

 

I know 2 hack methods for this (including verification file / site validation),

 

I know that too.

 

Think, if "Sweet" know how to hack this, everyone knows

 

 

 

Talking about Pcmark:

 

You need a sys bench, and 2004/Vantage/7 are not popular, nor will ever be.

 

 

Uffff...anyway PcMark'05, is older, highly insecure, today very few use and also highly hacked, and does not measure system capability, hack only measures.

 

But, if you wish ? so, then continue.

Edited December 1, 2012 by Sweet

[Massman]

How about "freezing" the points fo benchmarks that are removed? So when a benchmark is "not active", the global points for the submissions that were submitted will be frozen and not lost. You can still submit results for non-active benchmarks, but you will not get points for it.

 

This way, voting a benchmark out will not make you lose the points you worked for in the last couple of years.

[GENiEBEN]

How about "freezing" the points fo benchmarks that are removed? So when a benchmark is "not active", the global points for the submissions that were submitted will be frozen and not lost. You can still submit results for non-active benchmarks, but you will not get points for it.

 

This way, voting a benchmark out will not make you lose the points you worked for in the last couple of years.

 

As long as the replacement gets points the moment is added.

Btw, before adding them let me know so I can check their 'integrity' lol.

[der8auer]

How about "freezing" the points fo benchmarks that are removed? So when a benchmark is "not active", the global points for the submissions that were submitted will be frozen and not lost. You can still submit results for non-active benchmarks, but you will not get points for it.

 

This way, voting a benchmark out will not make you lose the points you worked for in the last couple of years.

 

That's a very interesting idea

 

So only freeze the global points but you can still submit and recieve hardware points?

[Christian Ney]

How about "freezing" the points fo benchmarks that are removed? So when a benchmark is "not active", the global points for the submissions that were submitted will be frozen and not lost. You can still submit results for non-active benchmarks, but you will not get points for it.

 

This way, voting a benchmark out will not make you lose the points you worked for in the last couple of years.

 

This means free safe points

 

You talked about Globals, what about hardware points ?

 

It will be a mess this way anyway, because points will be frozen but not rankings. This means after a while TOP 10 have no points an the 11th has 60 Global points ?

[Massman]

Shall I move this discussion topic to the public forums so we can hear more opinions? Obviously, they will be able to read all the comments we wrote in this thread already.

[GENiEBEN]

Shall I move this discussion topic to the public forums so we can hear more opinions? Obviously, they will be able to read all the comments we wrote in this thread already.

 

Definitely, nothing to hide

[Massman]

Ok, moved to public

[I.M.O.G.]

How about "freezing" the points fo benchmarks that are removed? So when a benchmark is "not active", the global points for the submissions that were submitted will be frozen and not lost. You can still submit results for non-active benchmarks, but you will not get points for it.

 

This way, voting a benchmark out will not make you lose the points you worked for in the last couple of years.

 

Not this. "Safe" or "unbeatable" points don't exist currently. If a bench isn't active, those would be the most valuable rankings to have because those are the only points that don't decay as others beat you.

 

That would not give the intended affect I don't think. Pcm05 is almost like this currently. Very low competition, easy points that almost don't go away when you get them because no one but a few want to run it.

Edited January 22, 2013 by I.M.O.G.

[K404]

This goes against the idea of keeping things simple and easy to explain for newcomers.

 

I'm happy with the number of benchmarks, I don't mind if there's more. There is a practical limit though, yes.

 

People will vote to remove the benchmark that their main rival is benefiting most from.

 

Instead of adding/removing benches, what about increasing or decreasing the hardware and global point weighting? (more emphasis on newer 3D benches...or good new 2D benchmarks that come along.) We really need a better wPrime (ie.....fixed length of time to "future-proof" it)

 

A lot of people won't like this, but i'm gonna say it.....

 

3DMark01: at global level, a bencher watches "999" for 6 tests and Chase High..... then hopes for the best at the end when the score comes up. It's not interesting to watch, it's 12 years old, newcomers won't understand why there's so much emphasis on such an old bench......that only runs well on an OS that was replaced 5 years ago. It's hard to bench in front of an audience because there's nothing to watch and there's every chance they'll look at the LOD and think "cheat." It's CPU-dependent and ORB verification has been dropped. A .jpg image is all that sets the noobs apart from the pros. IMO.... and I know a LOT of people will disagree..... at practical level, it is time for this benchmark to very slowly start moving out of the light.

Edited January 22, 2013 by K404

[der8auer]

The question is always: Are we talking about hardware or global points?

 

Like I already said earlier in this thread it would be very bad for our hardware masters to disable hardware points for a benchmark which used to recieve points in the past. Global points - fine. But don't ruin the work of guys who spent months and years with it

[TaPaKaH]

In my opinion, the whole purpose of hardware points is that you can repeatedly earn them, whether it's 5 benchmarks on 10 cpus or 10 benchmarks on 5 cpus, so I don't mind having more, or even 'way too many' benchmarks with hardware points as there is already 'way too many' hardware you can earn them with.

 

As for globals, I agree with Kenny. We can't keep '3D' tests that only scale with CPU frequency forever.

[NoMS]

K404 idea looks reasonable to me.

People usually tends to run the benchmark which will give more points, just rebalance global points to make shiny new benchmarks more "profitable" than older ones making them more interesting to people bench.

Edited January 22, 2013 by NoMS

[Bobnova]

I don't like the idea of frozen points.

 

I like the idea of replacing some benches for globals especially. 3d05 and pc05 for example. 3d05 because it's almost entirely a CPU benchmark, PC05 because it's almost entirely a "how many things that haven't been branded cheats yet can I get away with, without getting them branded as cheats?" disaster. If HWBot needs a system benchmark (I like the concept), PC05 is not doing the job right now, and will only get worse. There has to be a more secure system benchmark somewhere, maybe one that can use more than 4 threads, ideally one that isn't full of holes.

 

Given the current limitation on the number of results that count, adding or removing benchmarks doesn't seem likely to matter all that much for score totals.

 

The benches I personally would remove to make room for new stuff (or on general principle) are the following:

 

PC05: It's broken. I think we all know it's broken. It's a laughable "full system" benchmark anyway. I would pull this from global and HW points.

 

PiFast: So incredibly easy to cheat it's a joke. Reference John Lam. No checksum = lol. I would pull this from both as well.

 

WPrime: Not secure either, SS is fairly easy to create if a person is so inclined. Easier than PiFast really, as there's only one calculation number.

 

3D05: So incredibly CPU bound it's almost funny. I would pull this from globals, and leave it in place for HW. There's plenty of older HW that it's a 3d bench on. Modern HW? Not so much.

3D01: I personally love the bench, but it's a CPU bench and an ugly one (LOD). I'd pull globals, after all the best card for it will be two generations old shortly, and before that the best card was from the dark ages of 55/65nm!

 

3D06: See 3D05.

 

Aquamark: Hey look, another CPU bench! This one is almost entirely a CPU bench for the vast majority of GPUs, there's something to be said for yanking it completely. I'd definitely take globals away. At least yank WR points, the best scores are all single card, no reason to reward the extra spending for lower performance.

 

 

I think I would give memory clock globals, I think it's something the masses understand and are interested in. Not spectacular live, but at least it gives a lot of frost.

 

That leaves plenty of room for new benches, a few GPU benches, 1-3 2d benches, and a system bench.

 

I'd like to save WPrime, but it really needs more security. Getting the checksum validation setup back (it was before my time, but I know it existed) would be nice. Adding one to PiFast would be nice too.

Edited January 22, 2013 by Bobnova

[K404]

"3D06: See PC05."

 

Did you mean "See 3D05?" Other than being CPU bound with a good card, I don't see a problem with it.

 

 

 

 

Something else to mention that I think is important: every year, GPUs get us ~~40% higher scores (between architecture and MHz) (assuming no CPU bottleneck... maybe I should use GFlop increase as a marker)

 

CPUs increase by.....10-15%. 3D Benchmark coders are NOT coding long-term benchmarks- the difference in progress between CPU and GPU is "too damned high"

[GENiEBEN]

Adding one to PiFast would be nice too.

 

That is being taken care of, I have it on hold for the time being.

 

Wprime: Agree :banana:

 

PC05: Any good alternatives? And no PC7 does not qualify, it takes forever to complete it and on some systems it stucks in an endless loop.

 

 

Aquamark: It's fast, perfect for lazy people. :wink:

 

3D05/06: Agree, you could put the focus on 03 and Vantage.

 

3D Benchmark coders are NOT coding long-term benchmarks- the difference in progress between CPU and GPU is "too damned high"

 

That's because it's easier to write for stuff that you can actually TEST yourself now, not in the future. In a matter of years people will laugh at 'octa-core' optimized software for their 64 core systems.

Edited January 22, 2013 by GENiEBEN

[TaPaKaH]

CineBench seems a decent alternative for a CPU benchmark provided that is has or can be made to have a simple verification mechanism. At least it's rather popular outside the benching community so it should bring more people in.

[Splave]

CineBench seems a decent alternative for a CPU benchmark provided that is has or can be made to have a simple verification mechanism. At least it's rather popular outside the benching community so it should bring more people in.

agreed

[Bobnova]

Cinebench needs a wrapper, there are some files you can edit to set your score to anything you want, or to (seriously) muck with the timer that calculates said score. That was as of cinebench 11.something (I think) about a year and a half ago, haven't looked since.

 

Very nice to see pifast getting a solid wapper, I like the bench (though I'm terrible at it), so I'm glad it'll be secured and sticking around.

 

I did indeed mean 3d05 not pc05, thanks!

 

I don't know of any good, secure, system benches unfortunately.

 

 

I think that if I had supreme control over everything, I'd first spend some time securing the benches that we want to keep, before doing almost anything else. Once that was done, I'd shift focus to adding/removing benches/points/etc.

It's hard to have a meaningful competition when MSPaint can be king, you know?

I personally bench because benching is fun, benching and beating other people is more fun, but I enjoy just the benching. That said, I'm far enough down the rankings that there's no incentive other than beating more people (down in the rankings...) to cheat.

At the top level, the pro level, there is a far stronger incentive to win at all costs by any means. This is far from unique to OCing of course! A quick read through a vaguely honest history of NASCAR (or F1, or almost any other racing series) will turn up all sorts of entertainment...

Anyway, securing the benchmarks the pros are going to be using in the new league would be #1 for me. Then securing the benchmarks that give globals, and then securing those that give HW points.

[GENiEBEN]

Cinebench needs a wrapper, there are some files you can edit to set your score to anything you want, or to (seriously) muck with the timer that calculates said score. That was as of cinebench 11.something (I think) about a year and a half ago, haven't looked since.

 

I was just about to look over it, if you can share details by PM it will be greatly appreciated. Im guessing you are referring to the feature that allows you to load last score, which is probably saved on hard disk / registry and it's easy to edit.

[GENiEBEN]

I was just about to look over it, if you can share details by PM it will be greatly appreciated. Im guessing you are referring to the feature that allows you to load last score, which is probably saved on hard disk / registry and it's easy to edit.

 

Quoting myself lol, but WTFBBQ