Re: [Qemu-devel] Re: Strategic decision: COW format

[Markus Armbruster]

Kevin Wolf <address@hidden> writes:

> Am 22.02.2011 09:37, schrieb Markus Armbruster:
>> Anthony Liguori <address@hidden> writes:
>> 
>>> On 02/18/2011 03:57 AM, Kevin Wolf wrote:
>>>> Am 18.02.2011 10:12, schrieb Markus Armbruster:
>>>>    
>>>>> Kevin Wolf<address@hidden>  writes:
>>>>>
>>>>>      
>>>>>> Am 15.02.2011 20:45, schrieb Chunqiang Tang:
>>>>>>        
>>>>>>>> Chunqiang Tang/Watson/IBM wrote on 01/28/2011 05:13:27 PM:
>>>>>>>> As you requested, I set up a wiki page for FVD at
>>>>>>>>            
>>>>>>> http://wiki.qemu.org/Features/FVD
>>>>>>>          
>>>>>>>> . It includes a summary of FVD, a detailed specification of FVD, and a
>>>>>>>> comparison of the design and performance of FVD and QED.
>>>>>>>>            
>>>>>>>          
>>>>>>>> See the figure at http://wiki.qemu.org/Features/FVD/Compare . This
>>>>>>>>            
>>>>>>> figure
>>>>>>>          
>>>>>>>> shows that the file creation throughput of NetApp's PostMark benchmark
>>>>>>>>            
>>>>>>> under
>>>>>>>          
>>>>>>>> FVD is 74.9% to 215% higher than that under QED.
>>>>>>>>            
>>>>>>> Hi Anthony,
>>>>>>>
>>>>>>> Please let me know if more information is needed. I would appreciate 
>>>>>>> your
>>>>>>> feedback and advice on the best way to proceed with FVD.
>>>>>>>          
>>>>>> Yet another file format with yet another implementation is definitely
>>>>>> not what we need. We should probably take some of the ideas in FVD and
>>>>>> consider them for qcow3.
>>>>>>        
>>>>> Got an assumption there: that the one COW format we need must be qcow3,
>>>>> i.e. an evolution of qcow2.  Needs to be justified.  If that discussion
>>>>> has happened on the list already, I missed it.  If not, it's overdue,
>>>>> and then we better start it right away.
>>>>>      
>>>> Right. I probably wasn't very clear about what I mean with qcow3 either,
>>>> so let me try to summarize my reasoning.
>>>>
>>>>
>>>> The first point is an assumption that you made, too: That we want to
>>>> have only one format. I hope it's easy to agree on this, duplication is
>>>> bad and every additional format creates new maintenance burden,
>>>> especially if we're taking it serious. Until now, there were exactly two
>>>> formats for which we managed to do this, raw and qcow2. raw is more or
>>>> less for free, so with the introduction of another format, we basically
>>>> double the supported block driver code overnight (while not doubling the
>>>> number of developers).
>>>>    
>>>
>>> Not sure what project you're following, but we've had an awful lot of
>>> formats before qcow2 :-)
>>>
>>> And qcow2 was never all that special, it just was dropped in the code
>>> base one day.  You've put a lot of work into qcow2, but there are
>>> other folks that are contributing additional formats and that means
>>> more developers.
>>>
>>>> The consequence of having only one file format is that it must be able
>>>> to obsolete the existing ones, most notably qcow2. We can only neglect
>>>> qcow1 today because we can tell users to use qcow2. It supports
>>>> everything that qcow1 supports and more. We couldn't have done this if
>>>> qcow2 lacked features compared to qcow1.
>>>>
>>>> So the one really essential requirement that I see is that we provide a
>>>> way forward for _all_ users by maintaining all of qcow2's features. This
>>>> is the only way of getting people to not stay with qcow2.
>>>>
>>>>
>>>> Of course, you could invent another format that implements the same
>>>> features, but I think just carefully extending qcow2 has some real
>>>> advantages.
>>>>
>>>> The first is that conversion of existing images would be really easy.
>>>> Basically increment the version number in the header file and you're
>>>> done. Structures would be compatible.
>>>
>>> qemu-img convert is a reasonable path for conversion.
>>>
>>>>   If you compare it to file systems,
>>>> I rarely ever change the file system on a non-empty partition. Even if I
>>>> wanted, it's usually just too painful. Except when I was able to use
>>>> "tune2fs -j" to make ext3 out of ext2, that was really easy. We can
>>>> provide the same for qcow2 to qcow3 conversion, but not with a
>>>> completely new format.
>>>>
>>>> Also, while obsoleting a file format means that we need not put much
>>>> effort in its maintenance, we still need to keep the code around for
>>>> reading old images. With an extension of qcow2, it would be the same
>>>> code that is used for both versions.
>>>>
>>>> Third, qcow2 already exists, is used in practice and we have put quite
>>>> some effort into QA. At least initially confidence would be higher than
>>>> in a completely new, yet untested format. Remember that with qcow3 I'm
>>>> not talking about rewriting everything, it's a careful evolution, mostly
>>>> with optional additions here and there.
>>>>    
>>>
>>> My requirements for a new format are as followed:
>>>
>>> 1) documented, thought-out specification that is covered under and
>>> open license with a clear process for extension.
>>>
>>> 2) ability to add both compatible and incompatible features in a
>>> graceful way
>>>
>>> 3) ability to achieve performance that's close to raw.  I want our new
>>> format to be able to be used universally both for servers and
>>> desktops.
>> 
>> I'd like to add
>> 
>> 4) minimize complexity and maximize maintainability of the code.  I'd
>> gladly sacrifice nice-to-have features for that.
>
> Especially if they are features that only other users use, right?
>
> What's the "Sankt-Florians-Prinzip" called in English?

NIMBY (not in my backyard)

Separating must-have from nice-to-have is basic requirements analysis.
Calling anything anybody would ever want a requirement isn't.

>>> I think qcow2 has some misfeatures like compression and internal
>>> snapshots.  I think preserving those misfeatures is a mistake because
>>> I don't think we can satisfy the above while trying to preserve those
>>> features.  If the image format degrades when those features are
>>> enabled, then it decreases confidence in the format.
>> 
>> I'm inclined to agree.  There's one way to prove us wrong: implement the
>> misfeatures without compromising the requirements.
>
> *sigh*
>
> It starts to get annoying, but if you really insist, I can repeat it
> once more: These features that you don't need (this is the correct
> description for what you call "misfeatures") _are_ implemented in a way
> that they don't impact the "normal" case. And they are it today.

Then you're on track to proving us wrong.  No need to get annoyed, just
finish the proof by showing us code that satisfies the nice-to-have
requirements in addition to the must-have requirements :)