[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-block] [PATCH v13 00/10] Block replication for continuous chec
From: |
Dr. David Alan Gilbert |
Subject: |
Re: [Qemu-block] [PATCH v13 00/10] Block replication for continuous checkpoints |
Date: |
Mon, 1 Feb 2016 10:18:41 +0000 |
User-agent: |
Mutt/1.5.24 (2015-08-30) |
* Wen Congyang (address@hidden) wrote:
> On 01/29/2016 06:47 PM, Dr. David Alan Gilbert wrote:
> > * Wen Congyang (address@hidden) wrote:
> >> On 01/29/2016 06:07 PM, Dr. David Alan Gilbert wrote:
> >>> * Wen Congyang (address@hidden) wrote:
> >>>> On 01/27/2016 07:03 PM, Dr. David Alan Gilbert wrote:
> >>>>> Hi,
> >>>>> I've got a block error if I kill the secondary.
> >>>>>
> >>>>> Start both primary & secondary
> >>>>> kill -9 secondary qemu
> >>>>> x_colo_lost_heartbeat on primary
> >>>>>
> >>>>> The guest sees a block error and the ext4 root switches to read-only.
> >>>>>
> >>>>> I gdb'd the primary with a breakpoint on quorum_report_bad; see
> >>>>> backtrace below.
> >>>>> (This is based on colo-v2.4-periodic-mode of the framework
> >>>>> code with the block and network proxy merged in; so it could be my
> >>>>> merging but I don't think so ?)
> >>>>>
> >>>>>
> >>>>> (gdb) where
> >>>>> #0 quorum_report_bad (node_name=0x7f2946a0892c "node0", ret=-5,
> >>>>> acb=0x7f2946cb3910, acb=0x7f2946cb3910)
> >>>>> at /root/colo/jan-2016/qemu/block/quorum.c:222
> >>>>> #1 0x00007f2943b23058 in quorum_aio_cb (opaque=<optimized out>,
> >>>>> ret=<optimized out>)
> >>>>> at /root/colo/jan-2016/qemu/block/quorum.c:315
> >>>>> #2 0x00007f2943b311be in bdrv_co_complete (acb=0x7f2946cb3f60) at
> >>>>> /root/colo/jan-2016/qemu/block/io.c:2122
> >>>>> #3 0x00007f2943ae777d in aio_bh_call (bh=<optimized out>) at
> >>>>> /root/colo/jan-2016/qemu/async.c:64
> >>>>> #4 aio_bh_poll (address@hidden) at /root/colo/jan-2016/qemu/async.c:92
> >>>>> #5 0x00007f2943af5090 in aio_dispatch (ctx=0x7f2945b771d0) at
> >>>>> /root/colo/jan-2016/qemu/aio-posix.c:305
> >>>>> #6 0x00007f2943ae756e in aio_ctx_dispatch (source=<optimized out>,
> >>>>> callback=<optimized out>,
> >>>>> user_data=<optimized out>) at /root/colo/jan-2016/qemu/async.c:231
> >>>>> #7 0x00007f293b84a79a in g_main_context_dispatch () from
> >>>>> /lib64/libglib-2.0.so.0
> >>>>> #8 0x00007f2943af3a00 in glib_pollfds_poll () at
> >>>>> /root/colo/jan-2016/qemu/main-loop.c:211
> >>>>> #9 os_host_main_loop_wait (timeout=<optimized out>) at
> >>>>> /root/colo/jan-2016/qemu/main-loop.c:256
> >>>>> #10 main_loop_wait (nonblocking=<optimized out>) at
> >>>>> /root/colo/jan-2016/qemu/main-loop.c:504
> >>>>> #11 0x00007f29438529ee in main_loop () at
> >>>>> /root/colo/jan-2016/qemu/vl.c:1945
> >>>>> #12 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized
> >>>>> out>) at /root/colo/jan-2016/qemu/vl.c:4707
> >>>>>
> >>>>> (gdb) p s->num_children
> >>>>> $1 = 2
> >>>>> (gdb) p acb->success_count
> >>>>> $2 = 0
> >>>>> (gdb) p acb->is_read
> >>>>> $5 = false
> >>>>
> >>>> Sorry for the late reply.
> >>>
> >>> No problem.
> >>>
> >>>> What it the value of acb->count?
> >>>
> >>> (gdb) p acb->count
> >>> $1 = 1
> >>
> >> Note, the count is 1, not 2. Writing to children.0 is in flight. If
> >> writing to children.0 successes,
> >> the guest doesn't know this error.
> >>>> If secondary host is down, you should remove quorum's children.1.
> >>>> Otherwise, you will get
> >>>> I/O error event.
> >>>
> >>> Is that safe? If the secondary fails, do you always have time to issue
> >>> the command to
> >>> remove the children.1 before the guest sees the error?
> >>
> >> We will write to two children, and expect that writing to children.0 will
> >> success. If so,
> >> the guest doesn't know this error. You just get the I/O error event.
> >
> > I think children.0 is the disk, and that should be OK - so only the
> > children.1/replication should
> > be failing - so in that case why do I see the error?
>
> I don't know, and I will check the codes.
>
> > The 'node0' in the backtrace above is the name of the replication, so it
> > does look like the error
> > is coming from the replication.
>
> No, the backtrace is just report an I/O error events to the management
> application.
OK, but the guest did see the error as well, so I'd assumed it was that path.
> >>> Anyway, I tried removing children.1 but it segfaults now, I guess the
> >>> replication is unhappy:
> >>>
> >>> (qemu) x_block_change colo-disk0 -d children.1
> >>> (qemu) x_colo_lost_heartbeat
> >>
> >> Hmm, you should not remove the child before failover. I will check it how
> >> to avoid it in the codes.
> >
> > But you said 'If secondary host is down, you should remove quorum's
> > children.1' - is that not
> > what you meant?
>
> Yes, you should excute 'x_colo_lost_heartbeat' fist, and then excute
> 'x_block_change ... -d ...'.
OK, but then that's quite separate from the problem with teh guest seeing it.
> >>> 12973 Segmentation fault (core dumped)
> >>> ./try/x86_64-softmmu/qemu-system-x86_64 -enable-kvm $console_param -S
> >>> -boot c -m 4080 -smp 4 -machine pc-i440fx-2.5,accel=kvm -name
> >>> debug-threads=on -trace events=trace-file -device virtio-rng-pci
> >>> $block_param $net_param
> >>>
> >>> #0 0x00007f0a398a864c in bdrv_stop_replication (bs=0x7f0a3b0a8430,
> >>> failover=true, errp=0x7fff6a5c3420)
> >>> at /root/colo/jan-2016/qemu/block.c:4426
> >>>
> >>> (gdb) p drv
> >>> $1 = (BlockDriver *) 0x5d2a
> >>>
> >>> it looks like the whole of bs is bogus.
> >>>
> >>> #1 0x00007f0a398d87f6 in quorum_stop_replication (bs=<optimized out>,
> >>> failover=<optimized out>,
> >>> errp=<optimized out>) at /root/colo/jan-2016/qemu/block/quorum.c:1213
> >>>
> >>> (gdb) p s->replication_index
> >>> $3 = 1
> >>>
> >>> I guess quorum_del_child needs to stop replication before it removes the
> >>> child?
> >>
> >> Yes, but in the newest version, quorum doesn't know the block replication,
> >> and I think
> >> we shoud add an reference to the bs when starting block replication.
> >
> > Do you have a new version ready to test? I'm interested to try it (and
> > also interested
> > to try the latest version of the colo-proxy)
>
> I think we can post the newest version this week.
Thanks, that would be great.
Dave
> Thanks
> Wen Congyang
>
> >
> > Dave
> >
> >> Thanks
> >> Wen Congyang
> >>
> >>> (although it would have to be careful not to block on the dead nbd).
> >>>
> >>> #2 0x00007f0a398a8901 in bdrv_stop_replication_all (address@hidden,
> >>> address@hidden)
> >>> at /root/colo/jan-2016/qemu/block.c:4504
> >>> #3 0x00007f0a3984b0af in primary_vm_do_failover () at
> >>> /root/colo/jan-2016/qemu/migration/colo.c:144
> >>> #4 colo_do_failover (s=<optimized out>) at
> >>> /root/colo/jan-2016/qemu/migration/colo.c:162
> >>> #5 0x00007f0a3989d7fd in aio_bh_call (bh=<optimized out>) at
> >>> /root/colo/jan-2016/qemu/async.c:64
> >>> #6 aio_bh_poll (address@hidden) at /root/colo/jan-2016/qemu/async.c:92
> >>> #7 0x00007f0a398ab110 in aio_dispatch (ctx=0x7f0a3a6c21d0) at
> >>> /root/colo/jan-2016/qemu/aio-posix.c:305
> >>> #8 0x00007f0a3989d5ee in aio_ctx_dispatch (source=<optimized out>,
> >>> callback=<optimized out>,
> >>> user_data=<optimized out>) at /root/colo/jan-2016/qemu/async.c:231
> >>> #9 0x00007f0a3160079a in g_main_context_dispatch () from
> >>> /lib64/libglib-2.0.so.0
> >>> #10 0x00007f0a398a9a80 in glib_pollfds_poll () at
> >>> /root/colo/jan-2016/qemu/main-loop.c:211
> >>> #11 os_host_main_loop_wait (timeout=<optimized out>) at
> >>> /root/colo/jan-2016/qemu/main-loop.c:256
> >>> #12 main_loop_wait (nonblocking=<optimized out>) at
> >>> /root/colo/jan-2016/qemu/main-loop.c:504
> >>> #13 0x00007f0a396089ee in main_loop () at
> >>> /root/colo/jan-2016/qemu/vl.c:1945
> >>> #14 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized
> >>> out>) at /root/colo/jan-2016/qemu/vl.c:4707
> >>>
> >>> Dave
> >>>
> >>>> Thanks
> >>>> Wen Congyang
> >>>>
> >>>>>
> >>>>> (qemu) info block
> >>>>> colo-disk0 (#block080): json:{"children": [{"driver": "raw", "file":
> >>>>> {"driver": "file", "filename": "/root/colo/bugzilla.raw"}}, {"driver":
> >>>>> "replication", "mode": "primary", "file": {"port": "8889", "host":
> >>>>> "ibpair", "driver": "nbd", "export": "colo-disk0"}}], "driver":
> >>>>> "quorum", "blkverify": false, "rewrite-corrupted": false,
> >>>>> "vote-threshold": 1} (quorum)
> >>>>> Cache mode: writeback, direct
> >>>>>
> >>>>> Dave
> >>>>>
> >>>>> * Changlong Xie (address@hidden) wrote:
> >>>>>> Block replication is a very important feature which is used for
> >>>>>> continuous checkpoints(for example: COLO).
> >>>>>>
> >>>>>> You can get the detailed information about block replication from here:
> >>>>>> http://wiki.qemu.org/Features/BlockReplication
> >>>>>>
> >>>>>> Usage:
> >>>>>> Please refer to docs/block-replication.txt
> >>>>>>
> >>>>>> This patch series is based on the following patch series:
> >>>>>> 1.
> >>>>>> http://lists.nongnu.org/archive/html/qemu-devel/2015-12/msg04570.html
> >>>>>>
> >>>>>> You can get the patch here:
> >>>>>> https://github.com/Pating/qemu/tree/changlox/block-replication-v13
> >>>>>>
> >>>>>> You can get the patch with framework here:
> >>>>>> https://github.com/Pating/qemu/tree/changlox/colo_framework_v12
> >>>>>>
> >>>>>> TODO:
> >>>>>> 1. Continuous block replication. It will be started after basic
> >>>>>> functions
> >>>>>> are accepted.
> >>>>>>
> >>>>>> Changs Log:
> >>>>>> V13:
> >>>>>> 1. Rebase to the newest codes
> >>>>>> 2. Remove redundant marcos and semicolon in replication.c
> >>>>>> 3. Fix typos in block-replication.txt
> >>>>>> V12:
> >>>>>> 1. Rebase to the newest codes
> >>>>>> 2. Use backing reference to replcace 'allow-write-backing-file'
> >>>>>> V11:
> >>>>>> 1. Reopen the backing file when starting blcok replication if it is not
> >>>>>> opened in R/W mode
> >>>>>> 2. Unblock BLOCK_OP_TYPE_BACKUP_SOURCE and BLOCK_OP_TYPE_BACKUP_TARGET
> >>>>>> when opening backing file
> >>>>>> 3. Block the top BDS so there is only one block job for the top BDS and
> >>>>>> its backing chain.
> >>>>>> V10:
> >>>>>> 1. Use blockdev-remove-medium and blockdev-insert-medium to replace
> >>>>>> backing
> >>>>>> reference.
> >>>>>> 2. Address the comments from Eric Blake
> >>>>>> V9:
> >>>>>> 1. Update the error messages
> >>>>>> 2. Rebase to the newest qemu
> >>>>>> 3. Split child add/delete support. These patches are sent in another
> >>>>>> patchset.
> >>>>>> V8:
> >>>>>> 1. Address Alberto Garcia's comments
> >>>>>> V7:
> >>>>>> 1. Implement adding/removing quorum child. Remove the option
> >>>>>> non-connect.
> >>>>>> 2. Simplify the backing refrence option according to Stefan Hajnoczi's
> >>>>>> suggestion
> >>>>>> V6:
> >>>>>> 1. Rebase to the newest qemu.
> >>>>>> V5:
> >>>>>> 1. Address the comments from Gong Lei
> >>>>>> 2. Speed the failover up. The secondary vm can take over very quickly
> >>>>>> even
> >>>>>> if there are too many I/O requests.
> >>>>>> V4:
> >>>>>> 1. Introduce a new driver replication to avoid touch nbd and qcow2.
> >>>>>> V3:
> >>>>>> 1: use error_setg() instead of error_set()
> >>>>>> 2. Add a new block job API
> >>>>>> 3. Active disk, hidden disk and nbd target uses the same AioContext
> >>>>>> 4. Add a testcase to test new hbitmap API
> >>>>>> V2:
> >>>>>> 1. Redesign the secondary qemu(use image-fleecing)
> >>>>>> 2. Use Error objects to return error message
> >>>>>> 3. Address the comments from Max Reitz and Eric Blake
> >>>>>>
> >>>>>> Wen Congyang (10):
> >>>>>> unblock backup operations in backing file
> >>>>>> Store parent BDS in BdrvChild
> >>>>>> Backup: clear all bitmap when doing block checkpoint
> >>>>>> Allow creating backup jobs when opening BDS
> >>>>>> docs: block replication's description
> >>>>>> Add new block driver interfaces to control block replication
> >>>>>> quorum: implement block driver interfaces for block replication
> >>>>>> Implement new driver for block replication
> >>>>>> support replication driver in blockdev-add
> >>>>>> Add a new API to start/stop replication, do checkpoint to all BDSes
> >>>>>>
> >>>>>> block.c | 145 ++++++++++++
> >>>>>> block/Makefile.objs | 3 +-
> >>>>>> block/backup.c | 14 ++
> >>>>>> block/quorum.c | 78 +++++++
> >>>>>> block/replication.c | 545
> >>>>>> +++++++++++++++++++++++++++++++++++++++++++++
> >>>>>> blockjob.c | 11 +
> >>>>>> docs/block-replication.txt | 227 +++++++++++++++++++
> >>>>>> include/block/block.h | 9 +
> >>>>>> include/block/block_int.h | 15 ++
> >>>>>> include/block/blockjob.h | 12 +
> >>>>>> qapi/block-core.json | 33 ++-
> >>>>>> 11 files changed, 1089 insertions(+), 3 deletions(-)
> >>>>>> create mode 100644 block/replication.c
> >>>>>> create mode 100644 docs/block-replication.txt
> >>>>>>
> >>>>>> --
> >>>>>> 1.9.3
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>> --
> >>>>> Dr. David Alan Gilbert / address@hidden / Manchester, UK
> >>>>>
> >>>>>
> >>>>> .
> >>>>>
> >>>>
> >>>>
> >>>>
> >>> --
> >>> Dr. David Alan Gilbert / address@hidden / Manchester, UK
> >>>
> >>>
> >>> .
> >>>
> >>
> >>
> >>
> > --
> > Dr. David Alan Gilbert / address@hidden / Manchester, UK
> >
> >
> > .
> >
>
>
>
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK
- Re: [Qemu-block] [PATCH v13 00/10] Block replication for continuous checkpoints,
Dr. David Alan Gilbert <=