What the others try to explain is that your graphics score is also influenced by your CPU, aswell is the CPU score somewhat influenced by your GPU. the later is not an issue in most cases but the first is very typical for vantage. For reference watch Buildzoids video about his try to reach the #1 wr in vantage with a 7940X and 4 Furys. It was all down to CPU power at his point, as the benchmark could utilize 16 cores and he "just" had 14 under LN2.
The score is influenced so heavily by the CPU because it needs to feed the data fast enough to the gpus for them to max out their performance. The older the benchmark, the fewer the power is needed for the gpu to render a frame and so the faster it needs the new data to keep pushing. If you want a CPU indipendent benchmark, take Unigine Superposition, that has been about not CPU limited at all in my past. You can also use time spy, that is more CPU depending but as its newer, its not that extreme as vantage.