R5: Improving HWBOT API

[Massman]

Following Borandi's request, R5 will have improved API support. Currently, we're looking into adding API support for:

 

1) RELEASED: Benchmark Ranking API

 

The new API will allow benchmark developers to easily create benchmark rankings at HWBOT similar to the ones we already have.

 

2) RELEASED: Data-mining API

 

Mainly focussing on opening up the data regarding submissions, points and user/team details.

 

Not everything is set in stone yet, so feel free to give suggestions

[IanCutress]

As per my previous request,

 

Output in XML or JSON is preferred.

If enough people use the API badly, it could cause unnecessary strain. It might be worth setting an internal limit to only update the API data cache (on HWBot side) once per hour and only when requested. When a request is made, if it is past the hour it still shows the old cache but the request causes the system to update the system side cache. That saves a lot of headache from all sides.

 

E.g. I request the api.xml?user=borandi from HWBot

HWBot sends me the current contents in the HWBot cache

if it has been an hour, an internal request is made so the HWBot cache is updated (or put in a queue to be updated) and my initial request gets an 'inqueue' response along with the older cache data

I then need to re-request the data from the API to get the latest data.

The system detects it has been less than an hour since the last request for latest data, and therefore doesn't put in a request for an update to the cache, and my request gets an 'up-to-date' response along with the latest cache data.

 

Same goes for submissions or rankings.

[Massman]

Small update.

 

This development sprint, the team is working on the Benchmark Ranking API. As mentioned in the opening post, this will allow benchmark developers to easily get a ranking up and running for testing benchmarks software. Additionally to the API itself, we're also updating the benchmark application structure of the HWBOT database to allow more flexibility in the benchmarks we host.

[Massman]

Heads up: the benchmark API is in the finalisation stage (fixing couple of bugs and making it more user-friendly). Should be available very soon

[Massman]

HWBOT Benchmark Engine API

 

As announced before, we have been working on a benchmark ranking API to help you build an easy testing environment for your own benchmark. The whole process is simplified to simply following a wizard, so it should not difficult to add it!

 

You can find the link to the wizard on the bottom of the HWBOT site, or go directly to: http://hwbot.org/administration/application/new. Note: you need to be logged in to be able to add a benchmark application.

 

 

So, once you logged in and browsed to the page, you'll get the first of 3 wizard pages. To make it more understandible, I'll use the Hyper PI benchmark as an example application to show you how it works and highlight some information. Alright, here we go.

 

• 
   
  
• Step 1: General Settings (1/3)
   
  
  • 
     
    
  • Name: what's the name of your benchmark (please, no obscene stuff ...)
    
  • Website: if you have a page with more information (or even validation links), you can enter that url here
    
  • Download link (direct or landing page): where can we download your benchmark application
    
  • Accept User Agreement: agree or you can't continue
     
    

   

  [*]Step 2: Score Settings (2/3)

   

  • 
     
    
  • Score unit: what is the measurement unit of your benchmark (usually either seconds or marks)
    
  • Hardware component that is stressed the most with your benchmark: indicate how your benchmark needs to be ranked; selection: processor, videocard, memory, disk or motherboard
    
  • Names of subtests: if you benchmark has several subtests you can enter the subtest names here
    
  • Lower is better: yes or no (checkbox) - defines how the benchmark scores will be ranked
    
  • Multithreaded: yes or no (checkbox) - defines whether there will be multiple rankings divided on core basis
    
  • Users are allowed to manually submit scores: yes or no (checkbox) - manually submitting means prove your benchmark result with validation screenshots. If you select no, you can have your benchmark send a data package to our engine which includes all submission information
     
    

   

  [*]Step 3: Security Settings (3/3)

   

  • 
     
    
  • Encryption used for datafiles: specify the encryption type for your benchmark datafiles; options:
    • AES/CBC/PKCS5Padding
      
    • AES/ECB/PKCS5Padding
      
    • DESede/CBC/PKCS5Padding
      
    • DESede/EBC/PKCS5Padding

    [*]Checksum hash used for datafiles: specify the checksum has used for datafiles; options:

    • SHA-1
      
    • SHA-256
      
    • MD5

    [*]Compressed datafiles: yes or no (checkbox) - is the datafile compressed or not

  [*]Step 4: Prepare you benchmark!

  [*]Step 5: Locate your benchmark under your profile account settings

  [*]Step 6: Update your benchmark settings (mostly the same settings)

  • Version: indicate what is the benchmark version
    
  • Status: the benchmark can have different statuses
    
    • Alpha (hidden): you have created the benchmark at hwbot, but are still checking if the benchmark submission is going correctly. The benchmark is hidden for the public. This status is for debugging purposes.
      
    • Beta (public): the benchmark will be public, so people can make submissions to the rankings. However, there will be no points awarded to the benchmark
      
    • Released: the benchmark is accepted by HWBOT and can now possibly generate HWBoints for the community members.
      

  [*]Step 7: Find your benchmark amongst the others

  [*]Step 8: Watch overclockers set new records!

 

For more information, you can go through the API document: http://static.hwbot.org/api/Generic_API_1.1.pdf

 

 

The next step for introducing benchmarks to the public would be to figure out a way for the community to decide which benchmark they like and which they don't. This falls under Planned Feature #11, but is currently still being discussed by the staff.

[Massman]

Data Mining API available here: http://static.hwbot.org/api/Query_API_1.0.pdf

[IanCutress]

OK Data Mining is what I really wanted! (sorry on bailing on the Benchmark API testing, things got rough)

 

Tried the link in the PDF, http://hwbot.org/api/submissions?regionCode=europe&league=enthusiast&application=3dmark11_-

_performance&gpuCoolingId=3&pictureAttached=true&gpuFamily=geforce_600&gpuCoreFreq=1200&gpuCoreFreqOperator=less_or_equal&limit=1, got an error.

 

Regarding "The table below clarifies each API. API requests can be made with regular HTTP GET or POST requests, with different GET paremeters / POST data. The response format can be XML or JSON, depending on the request header.", my requests will be done via PHP, meaning we need an ?option=XML tag in there as there isn't a request header.

 

Will start looking into it a bit more after some work that needs doing But nicely done !

[Devroush]

Fixed

The default response should be in XML.

[IanCutress]

A few suggestions..

 

(a) Put in the help file that you can call something like http://hwbot.org/api/benchmarks to get all the benchmarks. Some benchmarkIds have no benchmark associated, and there's no easy xml list of them to iterate through.

 

(b) Simple PHP code like '$benchmarks = simplexml_load_file('http://hwbot.org/api/benchmarks');' fails, because it's returning JSON Though when I put the address into an address bar, it's returning XML. Is there some form of redirect going on?

 

The code above gives the following error, indicating JSON:

 

Warning: simplexml_load_file(): {"errors":null,"results":[{"name":"3DMark2001 SE","id":"1","type":"GPU","country in C:\Program Files (x86)\EasyPHP-12.1\www\HWBot API - main benchmarks\index.php on line 7

 

I tried with an ?option=XML, still nothing.

 

Going to use

$benchmarks = json_decode(file_get_contents('http://hwbot.org/api/benchmarks'));

instead, which parses the JSON into an object/array hybrid :thumb:

 

© Could we also get to use the tag ?applicationId=1 on the benchmarks api?

Edited January 31, 2013 by borandi

[IanCutress]

Another small thing, which occurs in the main search as well.

 

If I parse a command for the best single GPU result for 3DMV, it includes all the dual GPU card results. This is because the numberOfVideocards and something relating to the database. Could you include a numberOfGPUs as well, which is numberOfVideocards * GPUs per card?

[IanCutress]

Another request, an hardware API at /api/hardware. I essentially want a list of all the CPU/GPU hardware options, similar to the search/?q= query used in the search box on the main site. I know there's a text file with all the hardware somewhere on the server that was referenced in the source of the front page, but now it's obfuscated behind a link redirect That text file would be enough

[Devroush]

A few suggestions..

 

(b) Simple PHP code like '$benchmarks = simplexml_load_file('http://hwbot.org/api/benchmarks'>http://hwbot.org/api/benchmarks');' fails, because it's returning JSON Though when I put the address into an address bar, it's returning XML. Is there some form of redirect going on?

 

The code above gives the following error, indicating JSON:

 

Warning: simplexml_load_file(): {"errors":null,"results":[{"name":"3DMark2001 SE","id":"1","type":"GPU","country in C:\Program Files (x86)\EasyPHP-12.1\www\HWBot API - main benchmarks\index.php on line 7

 

I tried with an ?option=XML, still nothing.

 

Going to use

$benchmarks = json_decode(file_get_contents('http://hwbot.org/api/benchmarks'));

instead, which parses the JSON into an object/array hybrid :thumb:

 

Ah I know what's causing this. We check the accept header of the request to determine if we should return xml or json. Apparently if the request has Accept: */*, it will return json.

curl -H "Accept: application/xml" http://hwbot.org/api/benchmarks -> xml

curl -H "Accept: application/json" http://hwbot.org/api/benchmarks -> json

curl -H "Accept: */*" http://hwbot.org/api/benchmarks> json

Don't know if php has a way of setting accept headers but if you can, set Accept: application/xml .

 

I'll look for a solution though.

Edited February 1, 2013 by Devroush

[IanCutress]

Doesn't matter either way for me at least, my line of code works wonders with what comes out by default.

[Devroush]

Well it's fixed anyway, next deploy to production will have the fix .

[I.M.O.G.]

Borandi, can you share some pages of yours where you are using the api in their rendering? I'm interested in seeing working examples of what could be done with the API.

[IanCutress]

I've not got anything solid yet, haven't had the time. Actually building something that shows data like this, and coding it up, takes about half a week - mainly to do with organising queries, requests, and dealing with objects/arrays.

 

But take this one for example (in pseudo code)

 

 

1) Query API for list of benchmarks

2) Loop through the benchmarks, query number 1 score for that benchmark (Y benchmarks of API calls) and store in array

2a) If CPU benchmark, iterate through number of cores, query number 1 score for X cores, store in array

2b) If GPU benchmark, iterate through number of GPUs, query number 1 score for X GPUs, store in array

2c) If memory benchmark, iterate through memory types (may require another API call)

2d) If SSD benchmark, etc.

3) Loop through the benchmarks again, query number 1 score for the country, store in new array

4) Display table of benchmarks, no.1 score vs. best country score (comparing and outputting arrays), and number of points each score gets. Also can display the kit used, CPU MHz, and everything else

 

There is a 30 second limit on most PHP servers, so dealing with too many API calls in one script will cause the script to end - I find about 50 API calls to individual submissions before you hit the limit.

 

I have plenty of ideas, just need time. Or money, that works too.

[IanCutress]

Two additional suggestions Devroush:

 

(a) can you add the output to include the current server time/date in each API call as well? This would help with locally caching some data to make local loading faster. It would also help with a simple API call to just return the server time/date. Then I can do a local if(current-cache > 24hr) {update();} routine.

(b) another request for an API of all current hardware in the database, that can be queried by OEM, manufacturer, chipset, socket et al. Just to return the hardware strings for the searches. Or even better, just a link to the the output of the hardware search function when you put in '?q=' - as I mentioned, the old HWBot layout source code gave us access to this, but now it's hidden behind some wizardry

 

Many thanks!

[IanCutress]

If anyone's dealing with header issues and cURL, here's how to do it:

 

<?php
   // retrieve data via cURL
   $ch = curl_init();

   curl_setopt($ch, CURLOPT_URL, "http://hwbot.org/api/benchmarks");
   curl_setopt($ch, CURLOPT_HEADER, 0);
   curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
   $xml_content = curl_exec($ch);
   curl_close($ch);

   // now you can process the XML content (i.e. via a xml to array function)
   echo $xml_content;
?>

 

Also @IMOG, here's some things I have done with another APIs:

 

http://www.borandi.co.uk/BC2XS

http://www.borandi.co.uk/wpe

as well as a couple of things for academia

[GENiEBEN]

Can we have support for PKCS7 or no padding?