qemu-devel
[Top][All Lists]
Advanced
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] [RFC] qed: Add QEMU Enhanced Disk format

From: Anthony Liguori
Subject: Re: [Qemu-devel] [RFC] qed: Add QEMU Enhanced Disk format
Date: Sun, 12 Sep 2010 12:09:34 -0500
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.12) Gecko/20100826 Lightning/1.0b1 Thunderbird/3.0.7 
On 09/12/2010 10:56 AM, Avi Kivity wrote:
No, the worst case is 0.003% allocated disk, with the allocated clusters distributed uniformly. That means all your L2s are allocated, but almost none of your clusters are.
But in this case, you're so sparse that your metadata is pretty much co-located which means seek performance won't matter much.
But since you have to boot before you can run any serious test, if it takes 5 seconds to do an fsck(), it's highly likely that it's not even noticeable. 

What if it takes 300 seconds?
That means for a 1TB disk you're taking 500ms per L2 entry, you're fully allocated and yet still doing an fsck. That seems awfully unlikely. 


   if l2.committed:
       if l2.dirty
           l2.write()
           l2.dirty = False
       l2.mutex.unlock()
    else:
       l2.mutex.lock()
       l2cache[l2.pos] = l2
       l2.mutex.unlock()
The in-memory L2 is created by defaultdict(). I did omit linking L2 into L1, by that's a function call. With a state machine, it's a new string of states and calls.
But you have to write the L2 to disk first before you link it so it's not purely in memory.
It's far easier to just avoid internal snapshots altogether and this is exactly the thought process that led to QED. Once you drop support for internal snapshots, you can dramatically simplify.
The amount of metadata is O(nb_L2 * nb_snapshots). For qed, nb_snapshots = 1 but nb_L2 can be still quite large. If fsck is too long for one, it is too long for the other.
nb_L2 is very small. It's exactly n / 2GB + 1 where n is image size. Since image size is typically < 100GB, practically speaking it's less than 50.
OTOH, nb_snapshots in qcow2 can be very large. In fact, it's not unrealistic for nb_snapshots to be >> 50. What that means is that instead of metadata being O(n) as it is today, it's at least O(n^2).
Why is in n^2? It's still n*m. If your image is 4TB instead of 100GB, the time increases by a factor of 40 for both.
It's n*m but either n ~= m in which case it's n^2 or m << n, in which case, it's just n, or m >> n in which case, it's just O(m). 

This is where asymptotic complexity ends up not being terribly helpful :-)
Let me put this another way though, if you support internal snapshots, what's a reasonable number of snapshots to expect reasonable performance with? 10? 100? 1000? 10000?
Not doing qed-on-lvm is definitely a limitation. The one use case I've heard is qcow2 on top of clustered LVM as clustered LVM is simpler than a clustered filesystem. I don't know the space well enough so I need to think more about it.
I don't either. If this use case survives, and if qed isn't changed to accomodate it, it means that that's another place where qed can't supplant qcow2.
I'm okay with that. An image file should require a file system. If I was going to design an image file to be used on top of raw storage, I would take an entirely different approach. 

That spreads our efforts further.
No. I don't think we should be in the business of designing on top of raw storage. Either assume fixed partitions, LVM, or a file system. We shouldn't reinvent the wheel at every opportunity (just the carefully chosen opportunities).
Refcount table. See above discussion for my thoughts on refcount table.
Ok. It boils down to "is fsck on startup acceptable". Without a freelist, you need fsck for both unclean shutdown and for UNMAP. 

To rebuild the free list on unclean shutdown.


If you have an on-disk compact freelist, you don't need that fsck.
"If you have an on-disk compact [consistent] freelist, you don't need that fsck."
Consistency is the key point. We go out of our way to avoid a consistent freelist in QED because it's the path to best performance. The key goal for a file format should be to have exactly as much consistency as required and not one bit more as consistency always means worse performance.
On the other hand, allocating a cluster in qcow2 as it is now requires scanning the refcount table. Not very pretty. Kevin, how does that perform?
(an aside: with cache!=none we're bouncing in the kernel as well; we really need to make it work for cache=none, perhaps use O_DIRECT for data and writeback for metadata and shared backing images).
QED achieves zero-copy with cache=none today. In fact, our performance testing that we'll publish RSN is exclusively with cache=none.
In this case, preallocation should really be cheap, since there isn't a ton of dirty data that needs to be flushed. You issue an extra flush once in a while so your truncate (or physical image size in the header) gets to disk, but that doesn't block new writes.
It makes qed/lvm work, and it replaces the need to fsck for the next allocation with the need for a background scrubber to reclaim storage (you need that anyway for UNMAP). It makes the whole thing a lot more attractive IMO.
For a 1PB disk image with qcow2, the reference count table is 128GB. For a 1TB image, the reference count table is 128MB. For a 128GB image, the reference table is 16MB which is why we get away with it today.
Anytime you grow the freelist with qcow2, you have to write a brand new freelist table and update the metadata synchronously to point to a new version of it. That means for a 1TB image, you're potentially writing out 128MB of data just to allocate a new cluster.
s/freelist/refcount table/ to translate to current qcow2 nomenclature. This is certainly not fast. You can add a bunch of free blocks each time you mitigate the growth but I can't of many circumstances where a 128MB write isn't going to be noticeable. And it only gets worse as time moves on because 1TB disk images are already in use today.
NB, with a 64-bit refcount table, the size of the refcount table is almost exactly the same size as the L1/L2 table in QED. IOW, the cost of transversing the refcount table to allocate a cluster is exactly the cost of transversing all of the L1/L2 metadata to build a freelist. IOW, you're doing the equivalent of an fsck everytime you open a qcow2 file today.
It's very easy to neglect the details in something like qcow2. We've been talking like the refcount table is basically free to read and write but it's absolutely not. With large disk images, you're caching an awful lot of metadata to read the refcount table in fully.
If you reduce the reference count table to exactly two bits, you can store that within the L1/L2 metadata since we have an extra 12 bits worth of storage space. Since you need the L1/L2 metadata anyway, we might as well just use that space as the authoritative source of the free list information.
The only difference between qcow2 and qed is that since we use an on-demand table for L1/L2, our free list may be non-contiguous. Since we store virtual -> physical instead of physical->virtual, you have to do a full transversal with QED whereas with qcow2 you may get lucky. However, the fact that the reference count table is contiguous in qcow2 is a design flaw IMHO because it makes growth extremely painful with large images to the point where I'll claim that qcow2 is probably unusable by design with > 1TB disk images.
We can optimize qed by having a contiguous freelist mapping physical->virtual (that's just a bitmap, and therefore considerably smaller) but making the freelist not authoritative. That makes it much faster because we don't add another sync and let's us fallback to the L1/L2 table for authoritative information if we had an unclean shutdown.
It's a good compromise for performance and it validates the qed philosophy. By starting with a correct and performant approach that scales to large disk images, we can add features (like unmap) without sacrificing either. 

Regards,

Anthony Liguori
Yes, you'll want to have that regardless. But adding new things to qcow2 has all the problems of introducing a new image format.
Just some of them. On mount, rewrite the image format as qcow3. On clean shutdown, write it back to qcow2. So now there's no risk of data corruption (but there is reduced usability).
It means on unclean shutdown, you can't move images to older versions. That means a management tool can't rely on the mobility of images which means it's a new format for all practical purposes.
QED started it's life as qcow3. You start with qcow3, remove the features that are poorly thought out and make correctness hard, add some future proofing, and you're left with QED.
We're fully backwards compatible with qcow2 (by virtue that qcow2 is still in tree) but new images require new versions of QEMU. That said, we have a conversion tool to convert new images to the old format if mobility is truly required.
So it's the same story that you're telling above from an end-user perspective.
It's not exactly the same story (you can enable it selectively, or you can run fsck before moving) but I agree it isn't a good thing.
They are once you copy the image. And power loss is the same thing as unexpected exit because you're not simply talking about delaying a sync, you're talking staging future I/O operations purely within QEMU.
qed is susceptible to the same problem. If you have a 100MB write and qemu exits before it updates L2s, then those 100MB are leaked. You could alleviate the problem by writing L2 at intermediate points, but even then, a power loss can leak those 100MB.
qed trades off the freelist for the file size (anything beyond the file size is free), it doesn't eliminate it completely. So you still have some of its problems, but you don't get its benefits.
I think you've just established that qcow2 and qed both require an fsck. I don't disagree :-)
There's a difference between a background scrubber and a foreground fsck.
The difference between qcow2 and qed is that qed relies on the file size and qcow2 uses a bitmap.
The bitmap grows synchronously whereas in qed, we're not relying on synchronous file growth. If we did, there would be no need for an fsck.
If you attempt to grow the refcount table in qcow2 without doing a sync(), then you're going to have to have an fsync to avoid corruption.
qcow2 doesn't have an advantage, it's just not trying to be as sophisticated as qed is.
The difference is between preallocation and leaking, on one hand, and uncommitted allocation and later rebuilds, on the other. It isn't a difference between formats, but between implementations.
reply via email to
[Prev in Thread] Current Thread [Next in Thread]