I.M.O.G. Posted March 4, 2013 Share Posted March 4, 2013 (edited) In data mining for the most valuable GPUs for hardware points, I came across something that surprised me. Coming in last among all teams yields pretty good HTPP - I thought it would yield nearly zero. The most valuable GPU for 3DM05 HTPP is the 8800 GT 512MB: http://url.hwbot.org/YR42Zs That didn't seem surprising. However the worst submission for the 8800 GT 512MB is worth 7.9HTPP. That is the lowest team high score ever submitted to hwbot for 3DM05 on the 8800 GT 512MB (ranked 165th out of 165 team submissions), and its worth 8 HTPP. Compare that to the 82nd out of 165, and that's worth 13.8 points. Only 6 extra points for beating half the team submissions, compared to the team who submitted the worst score on record. Same thing with Radeon 4870 in 3D06. 7.4 HTPP for 183 out of 183 teams. Compare that to an obscure card, and for being best out of 6 team submissions, 1st place gets you 6.1HTPP. Bottom line: Shouldn't the HTPP range start at 0 or something? The rankings are supposed to kind of reflect level of difficulty... And it might be kind of hard to come in last out of hundreds of submissions, I don't think this is what was intended. Edited March 4, 2013 by I.M.O.G. Quote Link to comment Share on other sites More sharing options...
Massman Posted March 5, 2013 Share Posted March 5, 2013 The range shouldn't really start at 0; the UHP also start at 0.1pts. 8 HTPP would be the equivalent of the 0.1 UHP. Using the word "beating" to indicate 82th out of 165 teams is maybe not the right terminology because a lot of these scores are done with not-so competitive hardware. I should look deeper into the matter, but getting 82th place shouldn't be that difficult, right? Are there any major complaints regarding the HTPP? Or more of a general note? Quote Link to comment Share on other sites More sharing options...
I.M.O.G. Posted March 5, 2013 Author Share Posted March 5, 2013 Nope, no major complaints. I don't really think its a problem, I mean the algorithm works. I guess this is just an edge case where it looks sort of weird and made me wonder how HTPP worked. Mainly for instance, in popular categories the points for last place seems to vary but its usually more than than first place in an unpopular category. Just seems weird and I hadn't noticed before. It makes sense though when you say 82nd out of 165 isn't really hard, so the reward makes sense when I think about it a little more in that situation. Quote Link to comment Share on other sites More sharing options...
Bobnova Posted March 5, 2013 Share Posted March 5, 2013 Seems kind of odd to me. On less popular HW last place HTPP is 1.8 points. Quote Link to comment Share on other sites More sharing options...
knopflerbruce Posted March 5, 2013 Share Posted March 5, 2013 Yeah, if anything it should be the other way around (5th out of 5 is "better" than 165th out of 165). Quote Link to comment Share on other sites More sharing options...
Massman Posted March 5, 2013 Share Posted March 5, 2013 Why is 5/5 better than 165/165 ? Quote Link to comment Share on other sites More sharing options...
Mr.Scott Posted March 5, 2013 Share Posted March 5, 2013 Why is 5/5 better than 165/165 ? Because less people finished ahead of you. Quote Link to comment Share on other sites More sharing options...
Massman Posted March 5, 2013 Share Posted March 5, 2013 That doesn't sound right ... Quote Link to comment Share on other sites More sharing options...
I.M.O.G. Posted March 5, 2013 Author Share Posted March 5, 2013 Why is 5/5 better than 165/165? Because the 5/5 submission has 160 chances to not become 165/165... Ya, it could actually be the worst score anyone could ever submit. But probability dictates that if 160 more submissions were made, at least one of them would likely be worse. I am not a betting man, but I would take that bet. A 5/5 score has the potential to be a pretty good result, beaten by 4 better results. It could also just suck, but it could be pretty good. A 165/165 score is terrible, either due to the choice of hardware, skills, or other problems with the rig. It cannot be a good result, because its been confirmed that everyone else who bothered to run it was able to run it better. Quote Link to comment Share on other sites More sharing options...
GENiEBEN Posted March 5, 2013 Share Posted March 5, 2013 ^^ What he said. Quote Link to comment Share on other sites More sharing options...
Mr.Scott Posted March 5, 2013 Share Posted March 5, 2013 Same thing I said, just more complex. Quote Link to comment Share on other sites More sharing options...
Massman Posted March 6, 2013 Share Posted March 6, 2013 The result that is ranked 5th out of 5 can not be compared to a result that is ranked 165th out of 165. The position of the one result does not say anything about the other result. You can be 165th on LN2 and 5th using stock clocks. You can be 165th because you used a Pentium 3 at 3GHz which is not enough to beat any 3770K and 5th because you didn't overclock your 3960X. I can agree if you assume that the position indicate a ranking at HWBOT and those rankings have a normal spread in results. But without further information on the characteristics of the rankings and the individual results, you cannot make any exact statement about which is the better result. Not ever. Of course, you can talk in terms of probability and statistics, but we all know that statistics are useful for large sets of data, not individual datapoints. Quote Link to comment Share on other sites More sharing options...
I.M.O.G. Posted March 6, 2013 Author Share Posted March 6, 2013 Whoa, that seems over complicated. To be last out of over 100 submissions, you have to do something wrong. What can you do wrong to get last place? -Bad platform choice -bad OS -didn't run the benchmark right -et cetera Pick any benchmark in the database where there are more than 100 submissions, look at the last place submission, and you can state exactly what was done wrong. Regardless of the reason when there is over 100 submissions and you get last place, it isn't open for debate - there is sufficient data to confidently say last place is a non-competitive submission. With very few submissions like <10, last place may be competitive, but there is not sufficient data for the algorithm to be sure if last place sucks. That said... Any way you want to slice the pie, when you have >100 submissions in a certain ranking, last place never deserves more points compared to when you have <10 submissions and last place. One is definitely not competitive, one could still be competitive but the algorithm can't be sure... If they received equal points, that might be logical. If the confirmed non-competitive submission receives more points than the potentially competitive submission, that isn't logical. As I mentioned before, I don't know that its a problem that needs to be fixed really. What difference does it make. I do think the algorithm isn't treating these cases logically however. Quote Link to comment Share on other sites More sharing options...
knopflerbruce Posted March 6, 2013 Share Posted March 6, 2013 Why is 5/5 better than 165/165 ? Counterexample: if 5/5 equals 165/165, then 1/1 (or 2/2 if you like) equals 5/5, which again equals 165/165. Though, the opposite must also be true: 1/5 equals 1/165, but... which one gets the most points? By that logic I should get 50 points for all my 1/x scores, where perhaps x must be different from 1 (those would get 50 and 0.1 at the same time, so perhaps 25 is reasonable?). Feel free to adjust this in the next update If you think about it: say you're 5/5 a day or two after a new product hit the shelves. If another 160 people submit their scores your score has to be REALLY bad if all those 160 folks beat you, right? It's reasonable to assume at least a few of those would be even worse, unless your score is... garbage. Quote Link to comment Share on other sites More sharing options...
GENiEBEN Posted March 6, 2013 Share Posted March 6, 2013 (edited) A couple more arguments in this thread and we will find the meaning of life. Edited March 6, 2013 by GENiEBEN Quote Link to comment Share on other sites More sharing options...
Bobnova Posted March 6, 2013 Share Posted March 6, 2013 Nah it'll take another stage in the official decision making process for that. This is HWBot! The meaning of life is arguing with massman! Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.