[Qemu-devel] Benchmarking linux-user performance

[Emilio G. Cota]

Hi all,

Inspired by SimBench[1], I have written a set of scripts ("DBT-bench")
to easily obtain and plot performance numbers for linux-user.

The (Perl) scripts are available here:
  https://github.com/cota/dbt-bench
[ It's better to clone with --recursive because the benchmarks
(NBench) are pulled as a submodule. ]

I'm using NBench because (1) it's just a few files and they take
very little time to run (~5min per QEMU version, if performance
on the host machine is stable), (2) AFAICT its sources are in the
public domain (whereas SPEC's sources cannot be redistributed),
and (3) with NBench I get results similar to SPEC's.

Here are linux-user performance numbers from v1.0 to v2.8 (higher
is better):

                        x86_64 NBench Integer Performance
                 Host: Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz                
                                                                               
  36 +-+-+---+---+---+--+---+---+---+---+---+---+---+---+--+---+---+---+-+-+   
     |   +   +   +   +  +   +   +   +   +   +   +   +   +  +   +   +  ***  |   
  34 +-+                                                             #*A*+-+   
     |                                                            *A*      |   
  32 +-+                                                          #      +-+   
  30 +-+                                                          #      +-+   
     |                                                           #         |   
  28 +-+                                                        #        +-+   
     |                                 *A*#*A*#*A*#*A*#*A*#     #          |   
  26 +-+                   *A*#*A*#***#    ***         ******#*A*        +-+   
     |                     #       *A*                    *A* ***          |   
  24 +-+                  #                                              +-+   
  22 +-+                 #                                               +-+   
     |             #*A**A*                                                 |   
  20 +-+       #*A*                                                      +-+   
     |  *A*#*A*  +   +  +   +   +   +   +   +   +   +   +  +   +   +   +   |   
  18 +-+-+---+---+---+--+---+---+---+---+---+---+---+---+--+---+---+---+-+-+   
       v1.v1.1v1.2v1.v1.4v1.5v1.6v1.7v2.0v2.1v2.2v2.3v2.v2.5v2.6v2.7v2.8.0     
                                  QEMU version                                 


                     x86_64 NBench Floating Point Performance                  
                  Host: Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz               
                                                                               
  1.88 +-+-+---+--+---+---+---+--+---+---+---+---+--+---+---+---+--+---+-+-+   
       |   +   +  +  *A*#*A*  +  +   +   +   +   +  +   +   +   +  +   +   |   
  1.86 +-+           *** ***                                             +-+   
       |            #       #   *A*#***                                    |   
       |      *A*# #         # ##   *A*                                    |   
  1.84 +-+    #  *A*         *A*      #                                  +-+   
       |      #                        #                              *A*  |   
  1.82 +-+   #                          #                            ##  +-+   
       |     #                          *A*#                        #      |   
   1.8 +-+  #                               #  #*A*               *A*    +-+   
       |    #                               *A*   #                #       |   
  1.78 +-+*A*                                      #       *A*    #      +-+   
       |                                           #   ***#  #    #        |   
       |                                           *A*#*A*    #  #         |   
  1.76 +-+                                         ***         # #       +-+   
       |   +   +  +   +   +   +  +   +   +   +   +  +   +   +  *A* +   +   |   
  1.74 +-+-+---+--+---+---+---+--+---+---+---+---+--+---+---+---+--+---+-+-+   
         v1.v1.v1.2v1.3v1.4v1.v1.6v1.7v2.0v2.1v2.v2.3v2.4v2.5v2.v2.7v2.8.0     
                                   QEMU version                                

Same plots, in PNG: http://imgur.com/a/nF7Ls

These plots are obtained simply by running
        $ QEMU_PATH=path/to/qemu QEMU_ARCH=x86_64 make -j
from dbt-bench, although note that some user intervention was needed
to compile old QEMU versions.

I think having some well-defined, easy-to-run benchmarks (even
if far from perfect, like these) to aid development is better
than not having any. My hope is that having these will encourage
future performance improvements to the emulation loop and TCG -- or
at least serve as a warning when performance regresses excessively :-)

Let me know if you find this work useful.

Thanks,

                Emilio

[1] https://bitbucket.org/simbench/simbench
Simbench's authors have a paper on it, although it is not publicly
available yet (will be presented at the ISPASS'17 conference in April).
The abstract can be accessed here though: http://tinyurl.com/hahb4yj