Memory Benchmark Wishlist

[jfpoole]

Over the past few months we've been toying around with the idea of creating a separate memory benchmark.  Our initial thought would be to use the Geekbench 3 and 4 memory tests as a starting point.  The memory test work and are well-liked, and moving them into a separate benchmark would give us the opportunity to fix the outstanding issues with them (especially on modern hardware). 

If we were to create something that focused on testing memory performance, what would people want to see?  What is your wishlist for a memory benchmark?
 

[TerraRaptor]

Should have MMX only option for oldest platforms (sdram/ddr).

Run time of around 50-60 seconds maximum no matter what speed is (so it is not "how much time it took to finish job" but "how much of the job was done"). Probably a separate workload for different DRAM technologies - to be able to address specific platform issues more easily? SDRAM+DDR (MMX only available)/DDR2+RDRAM (SSE2)/DDR3 (SSE3)/DDR4 (AVX+)/DDR5/DDR6.

 

Anyway, the most valuable thing should be consistency, repeatability and coverage of most hardware. 

[SparkysAdventure]

Something I consider important is tweakability, being responsive to OS tuning and specific subtiming tightening, not just higher frequency.

I am not sure how easy or difficult this would be to do, but OS and hardware compatibility would be a big interest for me. Being able to run on much older (early ddr3 and ddr2 systems) as well as more recent (ddr4 and ddr5) systems.

SuperPi (specifically 32M) is a good example benchmark, as it achieves both of these goals. It works on OSs way older than XP and still works with Windows 11, and is the same benchmark regardless if you use 32 or 64 bit operating systems. It does not use many if any "new" instruction sets, and it remains a relevant benchmark because it can compare between generations. The problem with it is that it takes a while to run, and many users don't know how to interpret their score or how to improve it most effectively. Another issue with it is that it's still a CPU benchmark.

Edited August 4, 2021 by Sparky's__Adventure

[Leeghoofd]

On top of the previous:

As we see more and more CPUs with bigger caches it should have an autodetect function so the benchmark can't be run only in the cache and provide weird outputs as what we see happening with low maxmem settings... old Geeks don't look good on the upcoming gens

 

[exaberries]

To add to what leeghoofd said with 3d stacking new cpus might start to have enough cache for entire benchmarks to run in. Though autodetecting the amount of cache and changing the benchmark is a good idea you have to be careful, with new releases it might just start scoring better diminishing the effect actual ram oc. I generally think that having a benchmark which significantly changes the workload it runs based on what hardware its running on can lead to wierd results for global ranking not really a problem for hardware ranking. IMO ideally that platform that the benchmark is run on should have little effect on the score but I know that's not really possible due to the effects of architecture like infinity fabric, etc...

I don't think a ram focused benchmark that uses a lot of ram (20GB+) / requires a lot of stability would be interesting since it would limit pushing things the edge. Maybe something that uses about 1GB so maxmem is still a thing but cache hopefully matters less. 

[jfpoole]

5 hours ago, Leeghoofd said:

As we see more and more CPUs with bigger caches it should have an autodetect function so the benchmark can't be run only in the cache and provide weird outputs as what we see happening with low maxmem settings... old Geeks don't look good on the upcoming gens

I worry about runtime checks that make decisions about how to run the benchmark.

Geekbench 3 and Geekbench 4 include runtime checks that modify the benchmark to accommodate older, less-capable hardware.  These checks are what folks are using to "tweak" their Geekbench scores.

My preference would be to have a "one size fits all" benchmark that works the same regardless of the underlying hardware.  It may limit the range of hardware it supports, but I think it's the best, most robust approach to having a sane benchmark that is resistant to hacks.

[TerraRaptor]

6 hours ago, jfpoole said:

to accommodate older, less-capable hardware

Why not to make a thing that acts like AIDA64 (measures read/write/copy/latency) with 3dmark01 like formula for final score? I was running aida on very different setups (from mendocino to haswell-e, diffferent chipsets (via/sis/intel/nvidia/amd) and it was quite informative. Problem about aida64 is that diffferent versions have different optimisations - still, you can now make a version that will cover all existing memory types from EDO to DDR4 (I think we won't see anything newer than alder lake with ddr4?). So that will be one "basic" part - which won't be updated with newer algorithms (just bugfixes). Call it "legacy mode". And all newer hardware will have different mode that you will be able to update - and changes in that part can be then covered with hwbot rules.

[exaberries]

41 minutes ago, TerraRaptor said:

Why not to make a thing that acts like AIDA64 (measures read/write/copy/latency) with 3dmark01 like formula for final score? I was running aida on very different setups (from mendocino to haswell-e, diffferent chipsets (via/sis/intel/nvidia/amd) and it was quite informative. Problem about aida64 is that diffferent versions have different optimisations - still, you can now make a version that will cover all existing memory types from EDO to DDR4 (I think we won't see anything newer than alder lake with ddr4?). So that will be one "basic" part - which won't be updated with newer algorithms (just bugfixes). Call it "legacy mode". And all newer hardware will have different mode that you will be able to update - and changes in that part can be then covered with hwbot rules.

From what I know you are already basically describing the geekbench memory test. I think it would be better to have two seperate benchmarks rather than a legacy mode. Updating is both kinda necessary to deal with changes in architecure but also a massive problem for rules/scoring. My preference would be having algorithms change with major versions and each major version being a seperate benchmark à la geekbench.