[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-block] [Qemu-devel] [PATCH for-2.12 0/4] qmp dirty bitmap API
|
From: |
Vladimir Sementsov-Ogievskiy |
|
Subject: |
Re: [Qemu-block] [Qemu-devel] [PATCH for-2.12 0/4] qmp dirty bitmap API |
|
Date: |
Thu, 7 Dec 2017 12:39:16 +0300 |
|
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 |
07.12.2017 03:38, John Snow wrote:
On 11/30/2017 07:10 AM, Vladimir Sementsov-Ogievskiy wrote:
18.11.2017 00:35, John Snow wrote:
On 11/17/2017 03:22 AM, Vladimir Sementsov-Ogievskiy wrote:
17.11.2017 06:10, John Snow wrote:
On 11/16/2017 03:17 AM, Vladimir Sementsov-Ogievskiy wrote:
16.11.2017 00:20, John Snow wrote:
On 11/13/2017 11:20 AM, Vladimir Sementsov-Ogievskiy wrote:
Hi all.
There are three qmp commands, needed to implement external backup
API.
Using these three commands, client may do all needed bitmap
management by
hand:
on backup start we need to do a transaction:
{disable old bitmap, create new bitmap}
on backup success:
drop old bitmap
on backup fail:
enable old bitmap
merge new bitmap to old bitmap
drop new bitmap
Can you give me an example of how you expect these commands to be
used,
and why they're required?
I'm a little weary about how error-prone these commands might be
and the
potential for incorrect usage seems... high. Why do we require them?
It is needed for incremental backup. It looks like bad idea to export
abdicate/reclaim functionality, it is simpler
and clearer to allow user to merge/enable/disable bitmaps by hand.
usage is like this:
1. we have dirty bitmap bitmap0 for incremental backup.
2. prepare image fleecing (create temporary image with
backing=our_disk)
3. in qmp transaction:
- disable bitmap0
- create bitmap1
- start image fleecing (backup sync=none our_disk -> temp_disk)
This could probably just be its own command, though:
block-job-fleece node=foobar bitmap=bitmap0 etc=etera etc=etera
Could handle forking the bitmap. I'm not sure what the arguments would
look like, but we could name the NBD export here, too. (Assuming the
server is already started and we just need to create the share.)
Then, we can basically do what mirror does:
(1) Cancel
(2) Complete
Cancel would instruct QEMU to keep the bitmap changes (i.e. roll back),
and Complete would instruct QEMU to discard the changes.
This way we don't need to expose commands like split or merge that will
almost always be dangerous to use over QMP.
In fact, a fleecing job would be really convenient even without a
bitmap, because it'd still be nice to have a convenience command for
it.
Using an existing infrastructure and understood paradigm is just a
bonus.
1. If I understand correctly, Kevin and Max said in their report in
Prague about new block-job approach,
using filter nodes, so I'm not sure that this is a good Idea to
introduce now new old-style block-job, where we can
do without it.
We could do without it, but it might be a lot better to have everything
wrapped up in a command that's easy to digest instead of releasing 10
smaller commands that have to be executed in a very specific way in
order to work correctly.
I'm thinking about the complexity of error checking here with all the
smaller commands, versus error checking on a larger workflow we
understand and can quality test better.
I'm not sure that filter nodes becoming the new normal for block jobs
precludes our ability to use the job-management API as a handle for
managing the lifetime of a long-running task like fleecing, but I'll
check with Max and Kevin about this.
2. there is the following scenario: customers needs a possibility to
create a backup of data changed since some
point in time. So, maintaining several static and one (or several) activ
bitmaps with a possiblity of merge some of them
and create a backup using this merged bitmap may be convenient.
I think the ability to copy bitmaps and issue differential backups would
be sufficient in all cases I could think of...
so, instead of keeping several static bitmaps with ability to merge them,
you propose to keep several active bitmaps and copy them to make a backup?
so, instead of new qmp command for merge, add new qmp command for copy?
in case of static bitmaps, we can implement saving/loading them to the
image to free RAM space,
so it is better.
or what do you propose for [2] ?
I'm sorry, I don't think I understand.
"customers needs a possibility to create a backup of data changed since
some point in time."
Is that not the existing case for a simple incremental backup? Granted,
the point in time was decided when we created the bitmap or when we made
the last backup, but it is "since some point in time."
If you mean to say an arbitrary point in time after-the-fact, I don't
see how the API presented here helps enable that functionality.
(by "arbitrary point in time after-the-fact I mean for example: Say a
user installs a malicious application in a VM on Thursday, but the
bitmap was created on Monday. The user wants to go back to Wednesday
evening, but we have no record of that point in time, so we cannot go
back to it.)
Can you elaborate on what you're trying to accomplish so I make sure I'm
considering you carefully?
Yes, point in time is when we create a dirty bitmap. But we want to
maintain several points in time.
For example it may be last 10 incremental backups.
User wants an ability to create incremental backup which will contain
changes from selected point in
time to current moment. It is needed for example if backup was deleted
(to save disk space) and now user
wants to recreate it.
In current scheme for incremental backup, after successful backup we
actually lose previous point in time,
and have only the last one.
With ability to merge bitmap we can do the following:
1. create bitmap1
2. disable bitmap1, do external backup by bitmap1, create bitmap2
2.1 on backup fail: merge bitmap2 to bitmap1, enable bitmap1, delete bitmap2
2.2 on backup success: do nothing
3. disable bitmap2, do external backup by bitmap2, create bitmap3
3.1 on backup fail: merge bitmap3 to bitmap2, enable bitmap2, delete bitmap3
3.2 on backup success: do nothing
...
so, we have disabled bitmaps bitmap_i, corresponding to difference
between points in time i and i+1
and if user wants to get data, changed from point in time number j, he
just merges bitmaps from
bitmap_j to the last(active) one to the new bitmap bitmap_temp and
create corresponding backup.
instead of storing several disabled bitmaps we can store enabled
bitmaps, in this case we will not need
merge operation but instead we can implement copy operation. However
maintaining disabled bitmaps
is better, as we can then implement storing them to the qcow2 image, to
save RAM space.
I also have another idea:
implement new object: point-in-time or checkpoint. The should have
names, and the simple add/remove API.
And they will be backed by dirty bitmaps. so checkpoint deletion is
bitmap merge (and delete one of them),
checkpoint creation is disabling of active-checkpoint-bitmap and
starting new active-checkpoint-bitmap.
Then we can implement merging of several bitmaps (from one of
checkpoints to current moment) in
NBD meta-context-query handling.
--
Best regards,
Vladimir