Re: [Qemu-devel] Re: Strategic decision: COW format

[Markus Armbruster]

Anthony Liguori <address@hidden> writes:

> On 02/18/2011 03:57 AM, Kevin Wolf wrote:
>> Am 18.02.2011 10:12, schrieb Markus Armbruster:
>>    
>>> Kevin Wolf<address@hidden>  writes:
>>>
>>>      
>>>> Am 15.02.2011 20:45, schrieb Chunqiang Tang:
>>>>        
>>>>>> Chunqiang Tang/Watson/IBM wrote on 01/28/2011 05:13:27 PM:
>>>>>> As you requested, I set up a wiki page for FVD at
>>>>>>            
>>>>> http://wiki.qemu.org/Features/FVD
>>>>>          
>>>>>> . It includes a summary of FVD, a detailed specification of FVD, and a
>>>>>> comparison of the design and performance of FVD and QED.
>>>>>>            
>>>>>          
>>>>>> See the figure at http://wiki.qemu.org/Features/FVD/Compare . This
>>>>>>            
>>>>> figure
>>>>>          
>>>>>> shows that the file creation throughput of NetApp's PostMark benchmark
>>>>>>            
>>>>> under
>>>>>          
>>>>>> FVD is 74.9% to 215% higher than that under QED.
>>>>>>            
>>>>> Hi Anthony,
>>>>>
>>>>> Please let me know if more information is needed. I would appreciate your
>>>>> feedback and advice on the best way to proceed with FVD.
>>>>>          
>>>> Yet another file format with yet another implementation is definitely
>>>> not what we need. We should probably take some of the ideas in FVD and
>>>> consider them for qcow3.
>>>>        
>>> Got an assumption there: that the one COW format we need must be qcow3,
>>> i.e. an evolution of qcow2.  Needs to be justified.  If that discussion
>>> has happened on the list already, I missed it.  If not, it's overdue,
>>> and then we better start it right away.
>>>      
>> Right. I probably wasn't very clear about what I mean with qcow3 either,
>> so let me try to summarize my reasoning.
>>
>>
>> The first point is an assumption that you made, too: That we want to
>> have only one format. I hope it's easy to agree on this, duplication is
>> bad and every additional format creates new maintenance burden,
>> especially if we're taking it serious. Until now, there were exactly two
>> formats for which we managed to do this, raw and qcow2. raw is more or
>> less for free, so with the introduction of another format, we basically
>> double the supported block driver code overnight (while not doubling the
>> number of developers).
>>    
>
> Not sure what project you're following, but we've had an awful lot of
> formats before qcow2 :-)
>
> And qcow2 was never all that special, it just was dropped in the code
> base one day.  You've put a lot of work into qcow2, but there are
> other folks that are contributing additional formats and that means
> more developers.
>
>> The consequence of having only one file format is that it must be able
>> to obsolete the existing ones, most notably qcow2. We can only neglect
>> qcow1 today because we can tell users to use qcow2. It supports
>> everything that qcow1 supports and more. We couldn't have done this if
>> qcow2 lacked features compared to qcow1.
>>
>> So the one really essential requirement that I see is that we provide a
>> way forward for _all_ users by maintaining all of qcow2's features. This
>> is the only way of getting people to not stay with qcow2.
>>
>>
>> Of course, you could invent another format that implements the same
>> features, but I think just carefully extending qcow2 has some real
>> advantages.
>>
>> The first is that conversion of existing images would be really easy.
>> Basically increment the version number in the header file and you're
>> done. Structures would be compatible.
>
> qemu-img convert is a reasonable path for conversion.
>
>>   If you compare it to file systems,
>> I rarely ever change the file system on a non-empty partition. Even if I
>> wanted, it's usually just too painful. Except when I was able to use
>> "tune2fs -j" to make ext3 out of ext2, that was really easy. We can
>> provide the same for qcow2 to qcow3 conversion, but not with a
>> completely new format.
>>
>> Also, while obsoleting a file format means that we need not put much
>> effort in its maintenance, we still need to keep the code around for
>> reading old images. With an extension of qcow2, it would be the same
>> code that is used for both versions.
>>
>> Third, qcow2 already exists, is used in practice and we have put quite
>> some effort into QA. At least initially confidence would be higher than
>> in a completely new, yet untested format. Remember that with qcow3 I'm
>> not talking about rewriting everything, it's a careful evolution, mostly
>> with optional additions here and there.
>>    
>
> My requirements for a new format are as followed:
>
> 1) documented, thought-out specification that is covered under and
> open license with a clear process for extension.
>
> 2) ability to add both compatible and incompatible features in a
> graceful way
>
> 3) ability to achieve performance that's close to raw.  I want our new
> format to be able to be used universally both for servers and
> desktops.

I'd like to add

4) minimize complexity and maximize maintainability of the code.  I'd
gladly sacrifice nice-to-have features for that.

> I think qcow2 has some misfeatures like compression and internal
> snapshots.  I think preserving those misfeatures is a mistake because
> I don't think we can satisfy the above while trying to preserve those
> features.  If the image format degrades when those features are
> enabled, then it decreases confidence in the format.

I'm inclined to agree.  There's one way to prove us wrong: implement the
misfeatures without compromising the requirements.

> I think QED satisfies all of these today.