The official HWBOT Team Cup 2012 - SC2:

[gugle]

why 9000 series isn't included?

[teurorist]

qx9705 skulltrail

[Planet]

Qx9775*

[nedernakker]

so I guess it's time to discuss the manner of how to run the benchmark. since there seems to be posted scores witch have an out of proportion high score for their clockspeeds. this due to changing the benchmarks command line..

 

is this allowed? so yes then I'll rerun my benchmarks too..

 

rules state then benchmark must be run at it's standard settings..

[knopflerbruce]

Which results are you talking about? I tried to run with all tests selected, and then only the one that produces the highest scores, and I got roughly the same scores in both cases.

[Luebke]

ive tried, too and between default settings and only the best test are more than 100 points difference.

i guess only default-settings should be allowed.

 

I tried to run with all tests selected,

you have to use default settings for a valid result, don´t you?

[I.nfraR.ed]

There's no difference when running only fastest or all instructions in my testing. At least for all cpu's I've tested.

Can you show us an example with such a difference? Maybe throttling, because of the increased time to have the score?

It was allowed, because noone saw it affects the final score like e.g. changing resolution in 3dmark. CPU still cracks the password, it just uses the fastest method which it takes the score from, even if you run all tests.

 

Ran it on my office machine, SSSE3 being the best here.

 

Edited July 3, 2012 by I.nfraR.ed

[nedernakker]

I'm not talking about leaving instruction sets out, I do that myself too, since SSE3 seems to give the highest score there is no need to run the others.

 

During the country cup 2011 Massman already approved scores with such settings.

 

I'm am talking about these two scores bij MR.Paco:

 

http://www.hwbot.org/submission/2290924

http://www.hwbot.org/submission/2290963

 

where he manually adjusted the affinity settings which clearly gives him a gain to beat scores which are with 500~600 MHz faster cpu's

Edited July 3, 2012 by nedernakker

[I.nfraR.ed]

Setting threads count is allowed as well, I think. It's like setting 4 threads in wprime, even if your cpu is dual core only.

However the boost there is not the same, if you get any.

 

In any case, everyone is running like this currently.

It was discussed before, I think, but don't remember where and what was the exact outcome. If it's still allowed or forbidden, then rules should be actualized.

Official rules state "default UCBench2011 settings" and that's why I was wondering as well if I'm allowed to do that, but seeing others do it and scores are not removed, lead me to the conclusion I can do it too.

Plus leaving instructions out is already messing with the default settings. Yes, it doesn't change the score, but reduces the load on the cpu, sometimes allowing you get slightly higher clocks.

 

Overall, I think rules should be updated.

Edited July 3, 2012 by I.nfraR.ed

[nedernakker]

in wprime setting more threads then the cpu has gives you a lower score (in my experience)

 

I just want to have a clear awnser or a link to the awnser from a mod if it has allready been given to make things clear.

 

if it is allowed the rules for the benchmark need to be changed because running like this has nothing to do with using the benchmark at standard settings

.

[knopflerbruce]

I wouldn't be surprised if there is a bug somewhere, it's totally anti-intuitive that 64 threads is faster than 2 on these CPUs, more threads to fight over the same resources. However, it's an easy thing to dettect, so IMO we could just allow it until it's proven to be a bug.

[GENiEBEN]

I wouldn't be surprised if there is a bug somewhere, it's totally anti-intuitive that 64 threads is faster than 2 on these CPUs, more threads to fight over the same resources. However, it's an easy thing to dettect, so IMO we could just allow it until it's proven to be a bug.

 

If I spill the beans app gets banned from hwbot

[nedernakker]

@ knoflerbruce

 

I think that is not the solution, a lot of people have put a lot of effort in producing score, some of them even using ln2. this means they have to redo al those benches without even knowing if the results will be accepted as legit.

 

HWbot should come with an awnser to make things clear....

[Massman]

If I get this right, MrPaco is basically enabling more threads and it gives a higher score? Interesting.

 

As far as I can see, the calculation the benchmark is doing is not changed by this little trick/tweak.

[Rasparthe]

If the scores were legit at the beginning of the competition changing it half way through seems unfair as well. There are always tweaks that some people know about and others don't. Changing the cmd line is changing the default settings so that first rule is already out the window. That was approved at Country Cup so arguing about cmd line tweaks now seems backwards.

[nedernakker]

but there is obviously a big difference between altering the tests being done and altering the way the test is done..

 

disabling all other tests except SSSE3 gives no change in the scores outcome. changing the affinity settings does.

 

don't know how these calculations are managed by the program, but is seems from the scores that each next core continues where the previous left of. since dual core scores consist of two parts where the first part is about half the points of the total score.

 

this means that if u change the affinity to 64 eacht core will have 32 tries to improve the score. although it has less cores the score does seem to improve due to the fact the cpu has more then one chance to do the calculations...

[knopflerbruce]

Well, default settings for wprime is 1 thread. Just saying...

[knopflerbruce]

but there is obviously a big difference between altering the tests being done and altering the way the test is done..

 

disabling all other tests except SSSE3 gives no change in the scores outcome. changing the affinity settings does.

 

don't know how these calculations are managed by the program, but is seems from the scores that each next core continues where the previous left of. since dual core scores consist of two parts where the first part is about half the points of the total score.

 

this means that if u change the affinity to 64 eacht core will have 32 tries to improve the score. although it has less cores the score does seem to improve due to the fact the cpu has more then one chance to do the calculations...

 

OK, so choosing just one SSEx is fine - we agree on that. I guess messing with the cores in such a way that you run the stock thread count only (like 12 for gulftown) is also fine for everyone, that's no advantage except saving some Ln2.

[nedernakker]

Well, default settings for wprime is 1 thread. Just saying...

 

That depends on which os you are using, and is more due to the age of the hwbot version and detection of the type of cpu.

 

But that has nothing to do with ucbench.

[knopflerbruce]

That depends on which os you are using, and is more due to the age of the hwbot version and detection of the type of cpu.

 

But that has nothing to do with ucbench.

 

If you discuss "default settings" it's relevant. You change the thread number in wprime, just like UCBench. Running wprime at 64 threads on a dual core is allowed, so why not UCBench?

 

The REAL question is if UCBench is bugged or not at high thread count compared to core count. There is no such gain in wprime, some people run 2x core count and get a minimal boost, not 20% like in this case.

[knopflerbruce]

Anyway, we're investigating this... if the conclusion is that it's a bug, then these scores will be blocked. IMO the best way to get a win win situation now is to run both 2 (or 4 for quads) threads, as well as 64.

 

Sucks that we didn't detect this earlier, but nobody notified us - and then it's not much we can do:)

[nedernakker]

I only found out about this yesterday browsing through the scores and noticing the difference.

 

i'm awaiting the outcome. for future events the rules should get more clear to prevent this from happening.

 

I only checked the submission in the top of the competition, found that some do manually set the affinity but do it according to the type of cpu they are benching, this doesn't affect the scores. the only two 'bugged' scores are those by Mr. Paco

[knopflerbruce]

I only found out about this yesterday browsing through the scores and noticing the difference.

 

i'm awaiting the outcome. for future events the rules should get more clear to prevent this from happening.

 

I only checked the submission in the top of the competition, found that some do manually set the affinity but do it according to the type of cpu they are benching, this doesn't affect the scores. the only two 'bugged' scores are those by Mr. Paco

 

If this is indeed bugged, the rule will most likely be that you have to select the number of threads matching the specifications of the CPU(s), or use the default selection.

 

It's impossible to take everything into account when making a set of rules, the rules are improved when issues appear (like this). The problem is that the people who bench it (including myself) needed a LOT of time to discover this option. If we (mods) are not aware of it, then how can we add a note to the rules? And if thousands of benchmarkers needed more than a year to figure this out it's unreasonable to expect the staff to have it all sorted before a launch.

[nedernakker]

i'm not trying to offend you guys. just saying this for 'the greater good'

 

the first E5300 submission 19 days a go got flagged too:

 

http://www.hwbot.org/submission/2290963

 

not by me but for what i can see the command line adjustment got mistaken and therefore no action was taken. just bringing it to light so we all know how to bench.. I have a lot of cpu's to rebench if this gets allowed.. it's only 11 days left and i have a life too

[Rasparthe]

The door was opened by allowing command line adjustments to start. Changing the rules on it mid competition seems unfair. Sure make the rules clearer but it should be done after the competition, especially since its obvious Mr. Paco didn't hide the fact in any way, shape or form. The command line adjustments shouldn't have been allowed to start (in my opinion) but once allowed changing it mid-competition isn't all that reasonable. It seems more like those that didn't know about the tweaks complain about how unfair it is, instead of what they really mean, "It is unfair that I didn't know about it"