Why GPU benchmarks are lame

[Bartmasta]

I know that for benching you should try to have the best hardware and cooling.

 

But I hate it how most benchmarks are extremely Intel biased. I'm a big AMD fanboy which why I didn't get Intel but I just got 5770 CF and somebody gets 5000 more 3dmark06 points than me with no GPU overclock because then have an i7.

[knopflerbruce]

In some ways I agree, GPU benchmarks shouldn't be about the CPU - just as CPU benchmarks are independant of GPUs. Unfortunately, we don't have a "GPU only" benchmark yet, but folding@home doesn't benefit much from the CPU, so there's hope:)

 

Current 3d benches are more system benches. They're fun to play with, even if the GPU dependancy isn't always the best:p

[r1ch]

03 is about as close as you get to a 3d benchmark now.

 

With rev 3, people now focus on one card so there is less of a cpu bottleneck which is a good thing that many don't realise.

[Massman]

Isn't Core i7 just a faster gaming platform than Phenom II?

[packet]

Isn't Core i7 just a faster gaming platform than Phenom II?

 

i second that lol

 

Gaming Systems rely on CPU

 

not that you need an i7 to game of course but the fact is the faster the platform the better the gpu will perform on any 3D benchmark 3dmark Crysis far cry etc

 

on the other hand if people develop a dedicated gpu render then im in lol

 

maybe the new Evga tool coming idk

 

peace

[Bartmasta]

Well if you look at game benchmarks at say 3.8 GHz FPS is very similar. It's just synthetic benchmarks such as 3dmark and aquamark that makes Intel look a lot better.

[Massman]

I always figured Phenom II is CPC similar to Yorkfield, but slower than Nehalem?

[Hyperhorn]

I wonder why people always try to highlight their favorite company/products and then hwbot should change something to fix the imbalance.

Should I complain about 3dfx, Matrox, Cyrix or Via? I guess not, but in case Intel is better for benching nowadays, it doesn`t mean this will never change. It`s the same lame idea about splitting the ranking into AMD and Intel. Do I get a Cyrix CPU ranking then? What about Transmeta, IDT and so on? Splitting is not the right way to go, rather sub-categories which affect the final score (e.g. extra points for the fastest Phenom II and so on) without splitting.

So it`s basically not the question which CPU manufacturer is ahead speed-wise, but the problem, that 3D depends on your - let`s say - 2D-unit AND your graphics cards obviously, while for 2D benchmarks you don`t have to care about your graphics card overclocking abilities. So you have some 2D benchmarks, some 2.5D benchmarks and sometimes 3D benchmarks.

What happened to "Super Pi for GPUs", which was mentioned @ XS long time ago? Cuda-Z sounds good, too. I think within the next 6-12 months we will see some progress when it comes to raw GPU calculation benchmarks. Fermi will help here for sure, but we need something which is suitable for graphics cards, which fulfill certain key data to be able to run the benchmark (like DX 10 for Vantage) instead of the right manufacturer logo.

[Tharamis]

hi,

 

What happened to "Super Pi for GPUs", which was mentioned @ XS long time ago? Cuda-Z sounds good, too. I think within the next 6-12 months we will see some progress when it comes to raw GPU calculation benchmarks. Fermi will help here for sure, but we need something which is suitable for graphics cards, which fulfill certain key data to be able to run the benchmark (like DX 10 for Vantage) instead of the right manufacturer logo.

 

well, this "Super Pi" for GPUs will misslead graphiccards performance. this means:

a card which will be able to calculate PI as the *fastest one,

does *not mean that its the fastest while playing or doing other graphical things.

be carefull.

 

so i think there must be some benchs, which meansure (for example) Vertex performance

via opengl. so why opengl? because identical interface on all systmes, Dx independent.

 

cu

[Hyperhorn]

Benchmarks at hwbot are about competition, reliability and scalability regarding overclocking and do not necessarily represent real world/gaming/folding/whatever performance.

The reason why I mentioned these programs is, that we are on the way to a new era of how to use graphics cards. Once there`s a solid, uniform programmability given, I would like to see graphics cards benchmarks which don`t focus on the speed of image representation, but the pure speed of calculation. To create decent 3D benchmarks with top-notch technique is very expensive and time consuming. While the origins of the 2D benchmarks used by hwbot are totally different, 3D is dominated by Futuremark. Without critizing Futuremark I think hwbot should try to stay as much independent as possible. Just imagine Futuremark go bankrupt - no new benchmarks, no patches, no driver approval, no Hall of Fame, no validation, just nothing. It would heavily affect hwbot even if hwbot would have done everything right regarding their own obligations.

To use the programmability of (up-coming) graphics cards might bring back a certain independence to hwbot and the whole overclocking community, might reduce the importance of superior CPUs as parts of a 3D setup and helps hwbot to stay diversified. The best example at the moment is MaxxPi in my opinion - done by a single, hard-working guy. I`ve never seen anything close to this at the classic 3D section from a private individual. I hope this will change soon.

Edited January 25, 2010 by Hyperhorn

[westsider]

In some ways I agree, GPU benchmarks shouldn't be about the CPU - just as CPU benchmarks are independant of GPUs. Unfortunately, we don't have a "GPU only" benchmark yet, but folding@home doesn't benefit much from the CPU, so there's hope:)

 

Current 3d benches are more system benches. They're fun to play with, even if the GPU dependancy isn't always the best:p

 

3Dmark Vantage scales GPU only

as far as CPu doesnt make a bottleneck

in mostcases CPU doesnt have bottleneck but it affects the whole score

but if you look at GPU score it scales only with GPU

 

i mean with i7 and for example 4850

i get the same gpu score with cpu@3.8 and cpu@4.8

just cpu score are different

 

 

i think we should accept gpu score only for rankigns and it would be pure GPU based benchmark

[westsider]

Isn't Core i7 just a faster gaming platform than Phenom II?

 

yes it is

and if somebody is a fanboy its not peoples problem

i will buy AMD if its good to me for each time period

i am not a fanboy from intel

i am a fanboy from good performance/good price

[westsider]

I always figured Phenom II is CPC similar to Yorkfield, but slower than Nehalem?

 

yes right

phenom II is yorkfield but it clocks over 6Ghz and it overruns yorkfield so

 

but to near nehalem performance phenom II needs 6.5Ghz+ and not even then

[K404]

 

i think we should accept gpu score only for rankigns and it would be pure GPU based benchmark

 

I've thought about this from time to time too... have a leaderboard for only certain tests in a benchmark.... like 3DV GPU tests... 3DM06 CPU test, 3DM01 Nature etc.... maybe just for a laugh to start with

[westsider]

no

what i mean is only vantage gpu score

its pure GPU Calculation

you can try different cpu speeds

it doesnt change anything

maybe 0.001%

[Massman]

0.001%? It must be brand biased!

[westsider]

0.001%? It must be brand biased!

 

do you want me to try for example

4850 with 700/1100

and i7 W3540@ 3.6,3.8,4.0,4.2,4.4 ans so on

and let us look at GPU score only

it will be to 99% the same and 1% difference is not because of performance

there is always some little difference between two bench runs even with same configuration

[K404]

Well.. we all know the effect of the CPU depends on how much GPU power there is

 

With a 9800GTX class, adding 4K to the CPU score adds 100 points to the final score

[Snoro]

just go look a little bit at the hd5770 single gpu vantage performance ranking, i have about the second highest clock on my hd5770 but because i dont have a i7 970 or a xeon, i am about top 10 even if i beat many people ranked higher than me for the gpu score. And still, everything is done on air with my card.

[IanCutress]

I'm currently working on a GPU calculation bench which is CPU independant, but it's currently in single GPU mode for nVidia CUDA cards only.

 

It runs a 3D Finite Difference problem on the GPU for 30 secs, and during that time monitors the best peak speed the card achieves.

 

Still to go are:

Checksum at the end

MultiGPU setups

OpenCL

 

So for example my Quadro FX 580 does:

 

Edited March 16, 2010 by borandi

[knopflerbruce]

Doesn't look too bad. Is it possible to show more "data"? Like steps etc. That makes it more difficult to use photoshop.

[IanCutress]

Doesn't look too bad. Is it possible to show more "data"? Like steps etc. That makes it more difficult to use photoshop.

 

Yeah, changed it a bit in the past 10 mins - picture updated. Now does multiple nVidia GPUs, but one at a time. Though the checksum implementation would be best to stop photoshop (which is relatively easy with console programs like this and PiFast) - I've looked online at solutions, but just ended up scratching my head further...

 

I've no experience of OpenCL though, which may be the main hurdle.

[richba5tard]

Very cool, but like you mention yourself this would be much more interesting if it were written in OpenCL. Otherwise you'll have ATI fanboys burning down your house if we ever give this points.

[IanCutress]

Here's the files if anyone wants to test. It's worked so far on a variety of setups, including 9800GX2, and 260M, on XP and win7. One person had an issue with GTX295 and win7 64-bit that I need to try some things to get it to work. I'd duly make a new thread here for it when I've an ATi version working, otherwise discussion is here and http://www.benchtec.co.uk/forums/showthread.php?t=4947

 

CUDA enabled card required, plus 185.25+ drivers (I think they're the lowest possible)

Edited March 16, 2010 by borandi

[snappy1]

^ Agreed, seems to be the fairest method