Re: [Qemu-discuss] problems achieving "fullspeed" system emulation (TCG, icount, sleep off, rtc is vm)

[Artem Pisarenko]

Hi James,

Looks like I introduced some confusion when referenced your patch...
Actually, the problem I described, has nothing to do with this patch. I
referenced it on order just to draw your attention and make explanation of
the task I'm trying to solve to be more clear for others.

Your patch is a key element of whole design decision I've made, i.e. it's a
only reason why I've choosen QEmu at all (because I don't need full system
emulation, it's just an extra overhead I pay for avoiding development of my
own emulator). Since I found a framework developer who solves very similar
task with modified Qemu, I expected that not only this patch will apply in
my case (and everything seems to work from a functional point of view), but
also I'll get "emulation at the maximum possible speed" as said here:
https://wiki.qemu.org/ChangeLog/2.4#TCG . The problem is that it just
doesn't look like "maximum possible" for me. I'll admit it as such only
when I observe almost 100% host resources usage by qemu process.

And yes, my pseudo-code missing a change, introduced by patch. I understand
it exactly as you described. Anyway, neither the patch himself nor
coordination with simulation environment haven't any sense in context of
solving this problem, because it exists regardless of using "-external_sim"
option. It's just a pure QEmu problem.


ср, 29 авг. 2018 г., 20:01 Nutaro, James J. <address@hidden>:

> Hi Artem,
>
>
> I've had to update the patch several times to follow the evolution of
> QEMU. If you are working with QEMU-2.12, try building with the patch
> available here:
>
>
> https://web.ornl.gov/~nutarojj/adevs/
>
>
> The adevs simulation package also contains two examples that exercise the
> QEMU synchronization feature.
>
>
> Its not entirely clear from your post how you are coordinating your
> simulation clock with the QEMU clock. The intended use is for QEMU to
> participate in an overarching discrete event simulation. To summarize how
> the simulator/QEMU interaction is meant to work (with the new patch):
>
>
> while(1)
>
> {
>
>    QEMU posts a time advance request
>
>    QEMU waits for a time advance grant
>
>    On receiving the grant, QEMU runs until its virtual clock equals the
> grant time
>
> }
>
>
> The request is issued to the overarching simulator, which then replies
> with the grant with the simulation clock has reached to requested time or
> some other event relevant to QEMU occurs in the simulation.
>
>
> This may result in a simulation that is faster or slower than real-time,
> depending on the workload of QEMU and the other activities taking place in
> your simulation.
>
>
> Let me know if the examples and new patch answer your question/solve the
> problem or, if not, what parts of the problem persist.
>
>
> Jim
>
>
> ------------------------------
> *From:* Artem Pisarenko <address@hidden>
> *Sent:* Wednesday, August 29, 2018 5:45 AM
> *To:* address@hidden
> *Cc:* Nutaro, James J.
> *Subject:* problems achieving "fullspeed" system emulation (TCG, icount,
> sleep off, rtc is vm)
>
> I'm trying to integrate linux/x86 system emulation into simulation model
> of large system using QEmu running with options "-icount 1,sleep=off -rtc
> clock=vm" (non-KVM mode, of course) modified with patch "qqq: module for
> synchronizing with a simulation" by James Nutaro. Target system expected to
> be isolated from host environment as it has to be deterministic,
> particulary, it runs independently of real time on the host. The behavior
> I'm expecting with given options is that QEmu will "drive" virtual clock as
> fast as it can relative to realtime clock.
>
> I will use term 'speed' to describe virtual clock to realtime clock
> average ratio observed while running target. Speed expected to be limited
> only by host performance.
>
> As far as I understand, very(!) simplified model of how QEmu should work
> may be described with following single-threaded pseudo-code:
>
> while (1) {
>   if ((virtualclock % icount) == 0)
>     execute_vcpu_instruction();
>   if ((virtualclock % VIRTUAL_TO_REAL_TIME_RATIO) == 0)
>     increment_rtc_time_seen_by_guest();
>   execute_all_other_hardware_activity(virtualclock);
>   process_events(); // all other stuff, such as monitor commands, UI, etc..
>   virtualclock++;
> }
>
> In other words, when it runs linux target system which ideally does
> nothing but servicing console commands (i.e. being idle), each time I type
> 'hwclock' command to console, I should see huge advancement of shown time
> relative to real world time being passed between typing, i.e. speed should
> strive for infinity (given that host benchmark 'score' strives for infinity
> and target OS strives for sleep cpu all time).
>
> But actually I'm observing very and very strange behavior, described in
> following scenarios:
> 1. Both target and host are idle. Target time runs only ~5 times faster
> than real time. Host system monitor shows almost no cpu usage by QEmu
> process (so it explains such poor speed-up).
> 2. Qemu options contains "-nodefaults". If I'm not adding either "-display
> none -device qxl-vga" or "-display sdl", I''m getting target time much more
> slower, even slower than real time. Target system doesn't even access
> display hardware at all. Option combinations are exactly as I provided
> (i.e. replacing sdl with gtk gives different results). Qemu process still
> doesn't eat much cpu resources.
> 3. And finally the most shocking fact. When I'm loading host cpu in
> parallel (e.g. by running any benchmark/stress test on all cores), QEmu
> process activity increases to 2-4% and target time speeds up to ~100 times
> !!! Less cores loaded - less speed-up.
>
> These results after several days of investigating makes my brain
> screaming: "WTF!?"
>
> Something throttles virtual clock advancement, either intentionally or
> indirectly, or by mistake... Looks like pseudo-code above contains calls
> doing some synchronization with host system clock/timer/events/etc as a
> result of:
> - some emulated hardware misbehavior;
> - poor overall design of internal event/timers processing;
> - some wrong optimization technique not accounting such usage scenario;
> - some legacy piece of code doing tricky sabotage;
> - ...
>
> Or maybe my expectations are wrong and QEmu just isn't intended to be used
> in such way ?
>
> QEmu: 2.12.0 (built with 'configure --target-list=x86_64-softmmu
> --enable-debug'), patched with
> https://patchwork.kernel.org/patch/10124685/
> Target system hardware configuration as minimal as possible.
> Host system: 64-bit, Ubuntu 16.04, Intel Core i7-7700 CPU.
> --
>
> С уважением,
>   Артем Писаренко
>
-- 

С уважением,
  Артем Писаренко