Re: [Qemu-devel] Modularizing QEMU RFC

[Marc Marí]

On Mon, 3 Aug 2015 11:09:06 +0800
Fam Zheng <address@hidden> wrote:

> On Fri, 07/31 17:45, Marc Marí wrote:
> > Hi everyone
> > 
> > I propose improving the current modular driver system for QEMU so it
> > can benefit everybody in speed and flexibility. I'm looking for
> > other ideas, comments, critics, etc.
> > 
> > - Background -
> > In order to speed up QEMU, I'm looking at the high number of
> > libraries and dependencies that it loads. I've generated a QEMU
> > image that needs 145 shared libraries to start, and there were
> > still some options disabled. This is a lot, and this means that it
> > is slow.
> > 
> > So I've been looking at actual module system. Yes, QEMU does have a
> > module system, but disabled by default. The problem is, the modules
> > get loaded always during startup. This means, booting with modules
> > enabled is even slower, because loading at runtime is slower that
> > letting the linker do all the work at the start. At this point, I
> > doubt of the benefits of the actual modular system.
> > 
> > But, if disabling the actual block modules (iscsi, curl, rbd,
> > gluster, ssh and dmg) gives 40 ms of speedup, I think is worth an
> > effort of improving modules, and modularizing new parts
> > 
> > - Current module flow -
> > The current module system is based on shared libraries. Each of
> > these libraries has a constructor that registers the BlockDriver
> > structure to the global bdrv_drivers list. This list is later
> > searched by the type, the protocol, etc.
> > 
> > - Proposals -
> > I have in mind two ideas, which are not mutually exclusive:
> > 
> > -- A --
> > Having a huge lookup table with the names of the drivers that are in
> > modules, and the name of the modules where it can be found. When
> > some type is not found in the driver lists, this table can be
> > traversed to find the library and load it.
> > 
> > This requires an effort by the driver developer to fill the tables.
> > But the refactoring work can stay localized in the drivers and the
> > modules, it is (possibly) not necessary to touch any other part of
> > the QEMU system.
> > 
> > I don't specially like this huge manual-edited table.
> 
> Yeah, the way to fill the tables is an (implementation) question, but
> there are still more questions if we go this way...
> 
> bdrv_find_protocol is easy, the protocol name is extracted from image
> uri or blockdev options, we can load the module by the name.
> 
> bdrv_probe_all is harder. If we modularize a format driver,
> its .bdrv_probe code will be in the module. If we want to do the
> format detection, we need to load all format drivers. This means if
> the command line has an unspecified format, we'll still need to load
> all drivers at starting phase. (I wish all formats are probed
> according to magic bytes at offset 0, so we can simplify
> the .bdrv_probe logic and do it with data matching in block.c like
> the protocol case, but that's not true for VMDK :( )
> 
> I believe other sub systems have this kind of paradox as well.

I managed to overlook that...

If the user doesn't specify the type of the block device, then, all
block drivers will have to be tested. I see this is based on scores. If
there is a score that means "this is my type for sure" (which I don't
know), the probe could be stopped there.

But the user can also specify the type for its block device (if I
remember correctly). So, if he wants more speed, he could just specify
the type of the block device.
 
> > 
> > -- B --
> > The same --enable-X that is done at compile time, but directly on
> > the command line when running QEMU or even in hot at the monitor.
> > 
> > This solution requires work on the monitor, the command line
> > processing, the modules and the drivers system. It is also less
> > transparent to the final user.
> 
> I'm afraid this breaks backward compatibility. :(

:(

Maybe my ideas are a bit too ideal without rewriting half QEMU. I
should have left my ideas to cool down and rest before writting them
down. So any other ideas to reduce the library overhead are appreciated.

Thanks
Marc