Many people have been hearing about Qualcomm's Vellamo benchmark which was originally designed to benchmark HTML5 performance. This benchmark initially began as an internal benchmark and eventually became big enough that the company decided to make it publicly available. Over the first year or so that Vellamo has been out, Qualcomm has taken in a lot of advice and criticism from different reviewers and users alike. They've taken all of this feedback and integrated much of it into their new benchmark.
Vellamo is now a two part benchmark that focuses on HTML5 for the first 'chapter' and then CPU performance in the second 'chapter'. We believe that Qualcomm wanted to show the importance of the CPU in the benchmarking process as well as show off some of their new Krait CPU improvements that they've made with their new S4 class processors. The main additions to Vellamo are the addition of a WebGL Jellyfish benchmark, an inline video benchmark, and a Load and Reload benchmark that simulates 3G and 4G connections using a built-in web server.
The second chapter, known as 'Metal', is a series of different CPU benchmarks including a Dhrystone test, a LINPACK test, a Branch-K test, a stream 5.9 test, a RAMJam test and a Storage test. The first three tests are self explanatory as they are generally standard tests, with the Branch-K test testing the benching ability of the CPU with some memory bandwidth calls. The stream 5.9 test is a memory bandwidth test mostly for testing the memory bandwidth of the system based upon the controller and memory installed, which may vary from phone to phone. The RAMJam is also mostly self explanatory explaining what the peak performance of the memory controller is rather than what the onboard memory itself is capable of. The storage benchmark is also an important one to us, perhaps much less relevant to the CPU, but still very important to the whole system's performance as it tests the I/O operations and the actual read and write speeds of the storage which can determine load times, etc.
Overall, having tested this benchmark on our two latest Android devices, the HTC One S and HTC One X, we've found some interesting results. Even though this is a Qualcomm made benchmark, the HTC One X running NVIDIA Tegra 3 processor handily beats the One S with an MSM8960 (Snapdragon S4) in the HTML5 test. In the Metal test, however, the One S does overpower the One X by quite a bit as well. The HTC One X with Nvidia's Tegra 3 scored a 1592 in the HTML5 benchmark and 472 in the Metal benchmark. The HTC One S, on the other hand, scored 1523 in the HTML5 benchmark and 613 in the Metal bechmark. We can clearly see here that Nvidia's Tegra 3 quad-core is faster in the HTML5 bechmark, while the Qualcomm MSM8960 dual-core is faster in the Metal benchmark. In our short testing, this indicates a tie with each device and processor winning one of the benchmarks.
What is interesting, though, is that Qualcomm has ported this benchmark to x86 code as well as they've been able to test the Orange San Diego running Intel's Atom Z series SoCs. The Orange San Diego, according to Qualcomm's scores is faster than the Tegra 3 in the Metal benchmark but slower than the MSM8960. In the HTML5 benchmark, though, it is slower than both by a margin of over 30%.
See the HTC One S with Qualcomm MSM8960 benchmarks below...
Overall, Vellamo 2 made quite the big stride forward and there's no doubt that it will establish itself as one of benchmarks used to test mobile devices. The question of impartiality will always linger (unless Qualcomm decides to spins it off), but the fact that Tegra 3 managed to beat Snapdragon S4 in some tests clearly shows the company is trying to remain impartial to this development. If you are interested in the application and are running a phone with Google Android operating system, you can download it here.