A major shakeup happened in the world of benchmarking software. AMD officially announced yesterday that they are resigning from the industry group BAPCo. As a part of the decision, AMD will no longer endorse the use of SYSmark 2012 for benchmarking or making purchasing decisions. This official announcement is only the tip of the iceberg though, with a lot more going on under the hood. Also, bear in mind that AMD is not the first member to leave the organization: CNET, Computer Shopper, VNU and ZDNet all left over the course of years.

Who is BAPCo?
At this point we should briefly explain some internals of BAPCo, that might not be known. BAPCo or Benchmark Application Performance Corporation is a consortium where member companies can submit proposals, that are then voted for. According to our sources at all companies involved, the problem is that Intel is the driving force of BAPCo and their developers are the ones that do majority of work on BAPCO benchmarks. You could argue the company has a sort of veto position in this committee. So if they don’t put up with a proposal, it won’t get through. Thus even if other companies such as AMD place their opinion and wishes for change there, basically nothing happens if it doesn’t get the Intel stamp of approval.

Why did AMD Quit?
According to a blog written by Nigel Dessau, Senior Vice President and Chief Marketing Officer of AMD, the problem with SYSmark is, that it runs a lot of demanding workloads that don’t correspond with typical usage of most users:

  • "While SM2012 is marketed as rating performance using 18 applications and 390 measurements, the reality is that only 7 applications and less than 10 percent of the total measurements dominate the overall score.  So a small class of operations across the entire benchmark influences the overall score.
  • In fact, a relatively large proportion of the SM2012 score is based on system performance rated during optical character recognition (OCR) and file compression activities – things an average user will rarely if ever do.
  • And SM2012 doesn?t represent the evolution of computer processing and how that evolution is influencing average users? experience. SM2012 focuses only on the serial processing performance of the CPU, and virtually ignores the parallel processing performance of the GPU. In particular, SM2012 scores do not take into account GPU-accelerated applications that are widely used in today?s business environments."

Personally I’d agree with the assessment, that OCR is not something most people do very often, however I’d beg to differ regarding compression. Speaking for myself I do it frequently, though your mileage may vary. AMDs main gripe is that GPU performance is totally neglected, which is simply denying the realities of modern computing. Also, BAPCo came out with rebuttal, claiming that by disclosing the way how BAPCo generates the final score, AMD broke the NDA with BAPCo. How can you break an NDA with the organization you left? By disclosing something other benchmark companies gladly disclose, and that is how benchmark actually calculates its score:

"BAPCo also notes for the record that, contrary to the false assertion by AMD, BAPCo never threatened AMD with expulsion from the consortium, despite previous violations of its obligations to BAPCo under the consortium member agreement.


"BAPCo is disappointed that a former member of the consortium has chosen once more to violate the confidentiality agreement they signed, in an attempt to dissuade customers from using SYSmark to assess the performance of their systems.  BAPCo believes the performance measured in each of the six scenarios in SYSmark 2012, which is based on the research of its membership, fairly reflects the performance that users will see when fully utilizing the included applications."

Reality Distortion Field: What is the Problem with SYSmark 2012?
BAPCo SYSmark 2012 consists out of 14 applications, out of which 8 are GPU accelerated. Their impact on score is ZERO.
BAPCo SYSmark 2012 consists out of 14 applications, out of which 8 are GPU accelerated. Their impact on final score is ZERO – according to a blog by AMD’s Senior VP and CMO 

Biggest problem that BAPCo now faces is the loss of perception that SYSmark 2012 provides valid application performance. After AMD’s disclosure, we decided to check what SYSmark 2012 features and the list was quite surprising. As you can see on the image above, majority of featured SYSmark 2012 applications is GPU accelerated! With Firefox 3.6.8, it can go either way, as WebGL was available in beta since 2009, but users watching video and animations in Firefox 3.6.8 will experience GPU acceleration as Flash 10.1 is GPU accelerated.

On top of that, the benchmarks are weighed in a way that supposedly favor Intel’s offerings. The new SYSmark 2012 suite is said to contain 390 individual measurements, but only about 10 percent of those make a big impact on the overall score, that is then used to compare products.

In a recent briefing concerning Fusion products, AMD repeatedly made some remarks about how "our competitors products might be good at running SYSmark, but we doubt that’s what the average user does." That perfectly lines up with the criticism explained by Mr. Dessau. With their Fusion line of products, AMD claims to better balance the compute resources between x86 cores and GPU cores in comparison to Intel’s current CPU products, that dominate in x86 performance but deliver abysmal GPU performance.

NVIDIA Quits First
However, even though AMD raised a lot of fuss, AMD wasn’t the first company to quit BAPCo over SYSmark 2012. NVIDIA left BAPCo prior to AMD, as the company was not mentioned in  the SYSmark 2012 press release dated June 7, 2011. Additionaly, we have received word from Ujesh Desai, Vice President Product Marketing at NVIDIA: "We just resigned recently, that is all I can say at this time." This should clear up some rumors claiming NVIDIA has left the committee as early as 2009 that sparked around the web, expanding on an initial story at the internet site SemiAccurate.
 
VIA Quits Last
But there is more. Richard Brown, VP Marketing at VIA Technologies just came back to us with the following statement:

"VIA today confirmed reports that we have tendered our resignation to BAPCo. We strongly believe that the benchmarking applications tests developed for SYSmark 2012 and EEcoMark 2.0 do not accurately reflect real world PC usage scenarios and workloads and therefore feel we can no longer remain as a member of the organization.

We hope that the industry can adopt a much more open and transparent process for developing fair and objective benchmarks that accurately measure real world PC performance and are committed to working with companies that share our vision."

 

There is one more aspect to consider, that was already covered by Mr. Dessau in a blog over a year ago. SYSmark is a benchmark that has major influence on purchasing decision of government and educational institutions, as it was one of sales tactics which is now prohibited by the Intel-FTC and Intel-AMD Settlements. With the uncompetitive claims set out by AMD, it becomes clear that it fosters not only waste of tax dollars, but also moves them into other hands on the basis of a questionable benchmark.

What to use for benchmarking?
Given the limitations the benchmark has, including the stupendous requirement to use U.S. English keyboard (mandatory in previous iterations), we at BSN* do not recommend to use SYSmark, or at least giving the user a proper disclaimer of what the benchmark actually does. Any review that should be considered thorough should use more than one single benchmark anyway, to stand on a justifiable basis.

Our stand has and will be that benchmarks used have real-world connection or if they are synthetic, that they completely disclose the manner in which scores are generated, such as:

  • ElcomSoft: Phone Password Breaker, Wireless Security Analyzer
  • FinalWire: AIDA64
  • Futuremark: 3DMark, PC Mark, Peacekeeper
  • Google: V8 JavaScript Engine
  • Maxon: Cinebench
  • Rightware: BaseMark, BrowserMark, SimulationMark
  • SiSoftware: SiSoft Sandra

The situation is far from over but then again, with BAPCo being reduced to a single silicon supplier – the question remains who will develop benchmarks and claim neutrality. In the words of our Editor-in-Chief: "As far as benchmarks go, I prefer Futuremark. AMD, Intel and NVIDIA are constantly pissed off at them and that’s the only mark of true neutrality."