Re: [Qemu-devel] [PATCH] No change in userland tools after resizing qcow2 image

[김태하]

> On 07/19/2015 05:24 AM, Taeha Kim wrote:
> > Hello,
> >
> >
> > There is no change in userland tools after resizing qcow2 image except
> > file utility.
> >
> > For example when resize qcow2 image, the "file" utility is detectable
> > increased size.
> > However, the "ls", “stat”, and “du” utility still don't know how many
> > change size of image is changed.
> > The following patch enables to let userland tools - ls, du, stat -
> > know and apply changed size after resizing qcow2 image created by the
> > qemu-img tool.
> > Currently, “file” utility can only know without this patch.
> >
> > In addtion, "file" utility sometimes don't know qcow2 image format's
> > version as follows.
> >
> > $ file 100G.qcow2
> > 100G.qcow2: QEMU QCOW Image (Unknown)
> >
> >
> > So, user can't have any information about qcow2 image size.
> >
> > ======================
> > Signed-off-by: Taeha Kim <address@hidden> diff --git a/block.c
> > b/block.c index d088ee0..93427f8 100644
> > --- a/block.c
> > +++ b/block.c
> > @@ -2529,6 +2529,9 @@ int bdrv_truncate(BlockDriverState *bs, int64_t
> offset)
> >         if (bs->blk) {
> >             blk_dev_resize_cb(bs->blk);
> >         }
> > +       if (ret == 0) {
> > +           ret = truncate(bs->filename, offset);
> > +       }
> >     }
> >     return ret;
> > }
> > =====================
> >
> >
> > $ ./qemu/qemu-img --version
> > qemu-img version 2.3.91, Copyright (c) 2004-2008 Fabrice Bellard
> >
> > The how-to-reproduce is as follows with three reproduction.
> >
> > First let’s say that we create a qcow2 image using qemu-img tool as
> > follows. There is still no problem.
> >
> > $ ./qemu/qemu-img create -f qcow2 -o preallocation=metadata 100G.qcow2
> > 100G Formatting '100G.qcow2', fmt=qcow2 size=107374182400
> > encryption=off
> > cluster_size=65536 preallocation='metadata' lazy_refcounts=off
> > refcount_bits=16
> >
> > $ ./qemu/qemu-img check 100G.qcow2
> > No errors were found on the image.
> > 1638400/1638400 = 100.00% allocated, 0.00% fragmented, 0.00%
> > compressed clusters Image end offset: 107390828544
> >
> > $ ls -l 100G.qcow2
> > -rw-r--r--.   1 devjames devjames 107390828544 Jul 15 09:55 100G.qcow2
> >
> > $ stat 100G.qcow2
> >   File: ‘100G.qcow2’
> >   Size: 107390828544    Blocks: 32408          IO Block: 4096   regular file
> > Device: fd00h/64768d    Inode: 5778245         Links: 1
> > Access: (0644/-rw-r--r--)  Uid: ( 1000/devjames)   Gid: ( 1000/devjames)
> > Context: unconfined_u:object_r:user_home_t:s0
> > Access: 2015-07-15 09:55:17.269620129 +0900
> > Modify: 2015-07-15 09:55:17.269620129 +0900
> > Change: 2015-07-15 09:55:17.269620129 +0900
> >  Birth: -
> >
> > $ du -bb 100G.qcow2
> > 107390828544    100G.qcow2
> >
> > $ file 100G.qcow2
> > 100G.qcow2: QEMU QCOW Image (v3), 107374182400 bytes
> >
> > But, from now on there is a problem.
> >
> > Second let’s say we resize the qcow2 image size from 100G to 200G
> > using current qemu-img without the patch.
> >
> > $ ./qemu/qemu-img resize 100G.qcow2 200G Image resized.
> >
> > $ ./qemu/qemu-img check 100G.qcow2
> > No errors were found on the image.
> > 1638400/3276800 = 50.00% allocated, 0.00% fragmented, 0.00% compressed
> > clusters Image end offset: 107390894080
> >
> > $ ls -l 100G.qcow2
> > -rw-r--r--. 1 devjames devjames 107390832128 Jul 15 10:02 100G.qcow2
> >
> > $ stat 100G.qcow2
> >   File: ‘100G.qcow2’
> >   Size: 107390832128    Blocks: 32416          IO Block: 4096   regular file
> > Device: fd00h/64768d    Inode: 5778245         Links: 1
> > Access: (0644/-rw-r--r--)  Uid: ( 1000/devjames)   Gid: ( 1000/devjames)
> > Context: unconfined_u:object_r:user_home_t:s0
> > Access: 2015-07-15 10:02:04.842425479 +0900
> > Modify: 2015-07-15 10:02:04.854425682 +0900
> > Change: 2015-07-15 10:02:04.854425682 +0900
> >  Birth: -
> >
> > $ du -bb 100G.qcow2
> > 107390832128    100G.qcow2
> >
> > $ file 100G.qcow2
> > 100G.qcow2: QEMU QCOW Image (v3), 214748364800 bytes
> >
> > We can see that “ls”, “stat”, and “du” utilities don’t know change
> > size of qcow2 image except “file” one.
> >
> > Third let’s say we resize the qcow2 image size from 100G to 200G using
> > qemu-img with the patch.
> >
> > $ ./qemu/qemu-img resize 100G.qcow2 200G Image resized.
> >
> > $ ./qemu/qemu-img check 100G.qcow2
> > No errors were found on the image.
> > 1638400/3276800 = 50.00% allocated, 0.00% fragmented, 0.00% compressed
> > clusters Image end offset: 107390894080
> >
> > $ ls -l 100G.qcow2
> > -rw-r--r--. 1 devjames devjames 214748364800 Jul 15 10:08 100G.qcow2
> >
> > $ stat 100G.qcow2
> >   File: ‘100G.qcow2’
> >   Size: 214748364800    Blocks: 32416          IO Block: 4096   regular file
> > Device: fd00h/64768d    Inode: 5778245         Links: 1
> > Access: (0644/-rw-r--r--)  Uid: ( 1000/devjames)   Gid: ( 1000/devjames)
> > Context: unconfined_u:object_r:user_home_t:s0
> > Access: 2015-07-15 10:04:37.595018501 +0900
> > Modify: 2015-07-15 10:08:01.622484709 +0900
> > Change: 2015-07-15 10:08:01.622484709 +0900
> >  Birth: -
> >
> > $ du -bb 100G.qcow2
> > 14748364800    100G.qcow2
> >
> > $ file 100G.qcow2
> > 100G.qcow2: QEMU QCOW Image (v3), 214748364800 bytes
> >
> > Now we can see above all four utilities know change size of qcow2 image.
> > So I made the patch above for the "qemu-img" tool so that "ls",
> > “stat”, and “du” utility can be applied increased size of the qcow2
> > image file.
> >
> 

Hello,

Sorry for late response. At last, I found how to add ">" when the email is 
replied in outlook. :)

> It seems to me that the real issue here is that the resize command you are
> issuing doesn't respect your original pre-allocation desires.
> 

You're right. At first I thought that the change of resizing with 
pre-allocation needs to be applied in user space.
And the "preallocation" option doesn't have full, for example, 
"preallocation=full".
I expected that this will enable the qcow2 image to fill increased size fully 
when preallocated image is resized.

> When changing the size of the qcow2 file, it generally just updates the
> headers and doesn't pre-allocate those storage blocks. The actual size of
> the .qcow2 image doesn't take up disk space for empty blocks. That's part
> of its design. The reason 'file' can tell the "virtual" size of the disk
> is because it is reading the header to do so.
> 

If it updates only headers, I think the "preallocation=metadata" is right. But 
user still cannot know the exact size of the qcow2 image without the qemu-img 
tool.
If people who has only qcow2 images will guess or need many time to know the 
image size. Because currently qemu-img doesn't support shrinking, perhaps the 
user will have slack space unintentionally.

> ls and friends are telling you the size of the actual file, as you've
> noticed.
> 
> Changing the resize mechanisms to automatically truncate to achieve pre-
> allocation is wrong, I'm afraid.

I agree with you. Could you tell me your opinion about adding the resize 
mechanisms iff use "preallocation=full" or "preallocation=data"?
Of course, new patch will have other approach with internal functions related 
to qcow2 structure, not just simple userspace API.

> 
> I think if anything, we'd want to allow the qemu-img tool to take options
> for the resize, concerning pre-allocation strategy.

I didn't understand what you were saying. Could you please explain to me one 
more time?
What is exactly pre-allocation strategy? You mean that current resize function 
cannot change any more, or have to keep "as-is"?

> 
> --js
> 

Thank you very much.

Sincerely,
Taeha


> > I have created a VM using qcow2 image after patching using this patch.
> > The 100G.qcow2 has resized 200G is detected and available.
> > $ qemu-kvm -smp 4 -m 4096 100G.qcow2 -cdrom
> > ../iso/Fedora-Live-Workstation-x86_64-21-5.iso
> >
> > Thanks.
> >
> >
> >
> > Sincerely yours,
> > Taeha Kim
> > Senior Engineer, Network Functions Virtualization Part, Samsung
> > Electronics
> >