Re: [Qemu-devel] disk image: self-organized format or raw file

[吴兴博]

Hi Fam,

  It's glad to hear you,

It is said in this post that "All files systems that support inodes (ext2/3/4, xfs, btfs, etc) support files with holes while creating the files..."

[http://serverfault.com/questions/558761/best-linux-filesystem-for-sparse-files]

I also heard this claim from other sources, and the only "popular" filesystems who don't support holes in real world are just the old FAT32 and other FAT*.

Note that holes appear in filesystems when creating a sparse file in inode-filesystems. While "punching holes" does remove the existent contents from the file, and it was  newly added to only xfs/ext4 in newer linux kernel.

In qemu's disk image, a hole delivers clear message---the corresponding sectors/blocks/clusters are never written. So it's up to the guest whether to initialize the sectors to zero or just ignore them (filesystems never confuse with a uninitialized sector right?). Filesystems should ignore uninitialized data just because it's meaningless. Once written, the data would be ever meaningful to the guest.

"punching holes" would add support for "DISCARD" for a image which could behave like a SSD. Otherwise the image behaves like a magnetic disk.

The message in below would not be accurate:

* cp has --sparse option to support read and create sparse files. 

* Sadly scp doesn't support sparse files.

* rsync also has a -S --sparse option to properly handle sparse files.

Not until recently did I realize that the hole is just widely supported in *almost* all filesystems. That's why I have come up this idea.

I understand your concern about the support of hole. If this just because the "hole" is never standardized as POSIX or something else?

So now I get one clear reason: hole is not guaranteed by standardized filesystems (I guess a POSIX would be enough).

Is their something else? If it's the only reason of not using a sparse raw file as image, and the only impediment is no-one-should-ever-use FAT32 or say the POSIX, we may be very close to  move one step forward.

Cheers!
       吴兴博  Wu, Xingbo <address@hidden>

On Mon, Aug 11, 2014 at 8:52 PM, Fam Zheng <address@hidden> wrote:

On Mon, 08/11 19:38, 吴兴博 wrote:
> Hello,
>
>   The introduction in the wiki page present several advantages of qcow2
> [1]. But I'm a little confused. I really appreciate if any one can give me
> some help on this :).
>
>  (1) Currently the raw format doesn't support COW. In other words, a raw
> image cannot have a backing file. COW depends on the mapping table on which
> we it knows whether each block/cluster is present (has been modified) in
> the current image file. Modern file-systems like xfs/ext4/etc. provide
> extent/block allocation information to user-level. Like what 'filefrag'
> does with ioctl 'FIBMAP' and 'FIEMAP'. I guess the raw file driver (maybe
> block/raw-posix.c) may obtain correct 'present information about blocks.
> However this information may be limited to be aligned with file allocation
> unit size. Maybe it's just because a raw file has no space to store the
> "backing file name"? I don't think this could hinder the useful feature.
>
>  (2) As most popular filesystems support delay-allocation/on-demand
> allocation/holes, whatever, a raw image is also thin provisioned as other
> formats. It doesn't consume much disk space by storing useless zeros.
> However, I don't know if there is any concern on whether fragmented extents
> would become a burden of the host filesystem.
>
>  (3) For compression and encryption, I'm not an export on these topics at
> all but I think these features may not be vital to a image format as both
> guest/host's filesystem can also provide similar functionality.
>
>  (4) I don't have too much understanding on how snapshot works but I think
> theoretically it would be using the techniques no more than that used in
> COW and backing file.
>
> After all these thoughts, I still found no reason to not using a 'raw' file
> image (engineering efforts in Qemu should not count as we don't ask  for
> more features from outside world).
> I would be very sorry if my ignorance wasted your time.

Hi! I think what you described is theoretically possible, but I'm not so
positive about this feature. What would be the advantages, compared to qcow2?

My major concern is that the file system hole's transparency, meaning that the
users normally can't tell if a "hole" is really zeroes or unallocated, would
cause data loss more easily: the user may expect scp (1) or cp (1) to work on
an image file, just as always, but these tools can legitimately fill the whole
with actual zeroes, if the target is filesystem does not supporting hole.
That's too dangerous but totally out of control of QEMU.

Fam