Re: [Qemu-devel] Modularizing QEMU RFC

[Fam Zheng]

On Fri, 07/31 17:45, Marc Marí wrote:
> Hi everyone
> 
> I propose improving the current modular driver system for QEMU so it
> can benefit everybody in speed and flexibility. I'm looking for other
> ideas, comments, critics, etc.
> 
> - Background -
> In order to speed up QEMU, I'm looking at the high number of libraries
> and dependencies that it loads. I've generated a QEMU image that needs
> 145 shared libraries to start, and there were still some options
> disabled. This is a lot, and this means that it is slow.
> 
> So I've been looking at actual module system. Yes, QEMU does have a
> module system, but disabled by default. The problem is, the modules get
> loaded always during startup. This means, booting with modules enabled
> is even slower, because loading at runtime is slower that letting the
> linker do all the work at the start. At this point, I doubt of the
> benefits of the actual modular system.
> 
> But, if disabling the actual block modules (iscsi, curl, rbd, gluster,
> ssh and dmg) gives 40 ms of speedup, I think is worth an effort of
> improving modules, and modularizing new parts
> 
> - Current module flow -
> The current module system is based on shared libraries. Each of these
> libraries has a constructor that registers the BlockDriver structure to
> the global bdrv_drivers list. This list is later searched by the type,
> the protocol, etc.
> 
> - Proposals -
> I have in mind two ideas, which are not mutually exclusive:
> 
> -- A --
> Having a huge lookup table with the names of the drivers that are in
> modules, and the name of the modules where it can be found. When
> some type is not found in the driver lists, this table can be traversed
> to find the library and load it.
> 
> This requires an effort by the driver developer to fill the tables. But
> the refactoring work can stay localized in the drivers and the modules,
> it is (possibly) not necessary to touch any other part of the QEMU
> system.
> 
> I don't specially like this huge manual-edited table.

Yeah, the way to fill the tables is an (implementation) question, but there are
still more questions if we go this way...

bdrv_find_protocol is easy, the protocol name is extracted from image uri or
blockdev options, we can load the module by the name.

bdrv_probe_all is harder. If we modularize a format driver, its .bdrv_probe
code will be in the module. If we want to do the format detection, we need to
load all format drivers. This means if the command line has an unspecified
format, we'll still need to load all drivers at starting phase. (I wish all
formats are probed according to magic bytes at offset 0, so we can simplify the
.bdrv_probe logic and do it with data matching in block.c like the protocol
case, but that's not true for VMDK :( )

I believe other sub systems have this kind of paradox as well.

> 
> -- B --
> The same --enable-X that is done at compile time, but directly on the
> command line when running QEMU or even in hot at the monitor.
> 
> This solution requires work on the monitor, the command line
> processing, the modules and the drivers system. It is also less
> transparent to the final user.

I'm afraid this breaks backward compatibility. :(

> 
> - My comments on my proposals -
> Ideally, I'd like a mixed solution. The user can specify what wants to
> load, but also, when something is not found, it is automatically
> searched.
> 
> In both options, the current module system has to be partly rewritten,
> and some of the current drivers with module capability might need to be
> modified to adapt to the new specifications.
> 
> And, a part from improving the current modular interface, there are a
> lot of other devices that might benefit from it, not just the block
> devices.
> 
> I still haven't looked at the memory footprint of QEMU, but I'm sure
> that the QEMU binary will lose a lot of weight with this addition.
> 
> - Closing -
> This is just a brief draft. These proposals can be improved a lot, or
> there might be some other solutions that I haven't thought of.
> 
> So, I'd like to ask for ideas, comments, critics, improvements, etc.
> And also ask for contributors to this endeavour, because it will be a
> huge amount of work.
> 
> Thanks for reading and have a nice weekend
> Marc
>