[Qemu-discuss] problems achieving "fullspeed" system emulation (TCG, icount, sleep off, rtc is vm)

[Artem Pisarenko]

I'm trying to integrate linux/x86 system emulation into simulation model of
large system using QEmu running with options "-icount 1,sleep=off -rtc
clock=vm" (non-KVM mode, of course) modified with patch "qqq: module for
synchronizing with a simulation" by James Nutaro. Target system expected to
be isolated from host environment as it has to be deterministic,
particulary, it runs independently of real time on the host. The behavior
I'm expecting with given options is that QEmu will "drive" virtual clock as
fast as it can relative to realtime clock.

I will use term 'speed' to describe virtual clock to realtime clock average
ratio observed while running target. Speed expected to be limited only by
host performance.

As far as I understand, very(!) simplified model of how QEmu should work
may be described with following single-threaded pseudo-code:

while (1) {
  if ((virtualclock % icount) == 0)
    execute_vcpu_instruction();
  if ((virtualclock % VIRTUAL_TO_REAL_TIME_RATIO) == 0)
    increment_rtc_time_seen_by_guest();
  execute_all_other_hardware_activity(virtualclock);
  process_events(); // all other stuff, such as monitor commands, UI, etc..
  virtualclock++;
}

In other words, when it runs linux target system which ideally does nothing
but servicing console commands (i.e. being idle), each time I type
'hwclock' command to console, I should see huge advancement of shown time
relative to real world time being passed between typing, i.e. speed should
strive for infinity (given that host benchmark 'score' strives for infinity
and target OS strives for sleep cpu all time).

But actually I'm observing very and very strange behavior, described in
following scenarios:
1. Both target and host are idle. Target time runs only ~5 times faster
than real time. Host system monitor shows almost no cpu usage by QEmu
process (so it explains such poor speed-up).
2. Qemu options contains "-nodefaults". If I'm not adding either "-display
none -device qxl-vga" or "-display sdl", I''m getting target time much more
slower, even slower than real time. Target system doesn't even access
display hardware at all. Option combinations are exactly as I provided
(i.e. replacing sdl with gtk gives different results). Qemu process still
doesn't eat much cpu resources.
3. And finally the most shocking fact. When I'm loading host cpu in
parallel (e.g. by running any benchmark/stress test on all cores), QEmu
process activity increases to 2-4% and target time speeds up to ~100 times
!!! Less cores loaded - less speed-up.

These results after several days of investigating makes my brain screaming:
"WTF!?"

Something throttles virtual clock advancement, either intentionally or
indirectly, or by mistake... Looks like pseudo-code above contains calls
doing some synchronization with host system clock/timer/events/etc as a
result of:
- some emulated hardware misbehavior;
- poor overall design of internal event/timers processing;
- some wrong optimization technique not accounting such usage scenario;
- some legacy piece of code doing tricky sabotage;
- ...

Or maybe my expectations are wrong and QEmu just isn't intended to be used
in such way ?

QEmu: 2.12.0 (built with 'configure --target-list=x86_64-softmmu
--enable-debug'), patched with https://patchwork.kernel.org/patch/10124685/
Target system hardware configuration as minimal as possible.
Host system: 64-bit, Ubuntu 16.04, Intel Core i7-7700 CPU.
-- 

С уважением,
  Артем Писаренко