|
From: | Walid Nouri |
Subject: | Re: [Qemu-devel] Microcheckpointing: Memory-VCPU / Disk State consistency |
Date: | Wed, 13 Aug 2014 16:03:18 +0200 |
User-agent: | Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 |
Yes... Time is a problem, and it‘s currently running out... ;-)I think the first step is to reason about possible approaches and how they can be implemented in QEMU. The implementation can follow later :-)
Thank you for the hint with the drive-mirror feature. I will take a look at it and surely come back with new questions :-) I also think that disc replication ist the most pressing issue for MC.While looking to find some ideas for approaches to replicating block devices I have read the paper about the Remus implementation. I think MC can take a similar approach for local disk.
Here are the main facts that I have understood: Local disk contents is viewed as internal state the primary and secondary.In the explanation they describe that for keeping disc semantics of the primary and to allow the primary to run speculatively all disc state changes are directly written to the disk. In parrallel and asynchronously send to the secondary. The secondary keeps the pending writing requests in two disk buffers. A speculation-disk-buffer and a write-out-buffer.
After the reception of the next checkpoint the secondary copies the speculation buffer to the write out buffer, commits the checkpoint and applies the write out buffer to its local disk.
When the primary fails the secondary must wait until write-out-buffer has been completely written to disk before before changing the execution mode to run as primary. In this case (failure of primary) the secondary discards pending disk writes in its speculation buffer. This protocol keeps the disc state consistent with the last checkpoint.
Remus uses the XEN specific blktap driver. As far as I know this can’t be used with QEMU (KVM).
I must see how drive-mirror can be used for this kind of protocol. Am 11.08.2014 22:15, schrieb Michael R. Hines:
Excellent question: QEMU does have a feature called "drive-mirror" in block/mirror.c that was introduced a couple of years ago. I'm not sure what the adoption rate of the feature is, but I would start with that one. There is also a second fault tolerance implementation that works a little differently called "COLO" - you may have seen those emails on the list too, but their method does not require a disk replication solution, if I recall correctly.
I have taken a look at COLO. They have also published a good paper about their approach. The paper is about the XEN implementation of COLO. It’s an interesting approach to use a coarse grained lockspepping combined with checkpointing. From what i have understood the possible show stopper for general application is the depency of COLO to custom changes of the tcp stack to make it more deterministic.
IMHO there are two points. Custom changes of the TCP-Stack are a no-go for proprietary operating systems like Windows. It makes COLO application agnostic but not operating system agnostic. The other point is that with I/O intensive workloads COLO will tend to behave like MC. This is my point of view but i didn’t invest much time to understand everything in detail.
I know the time pressure that comes during a thesis, though =), so there's no pressure to work on it - but that is the most pressing issue in the implementation today. (Lack of disk replication in micro-checkpointing.) The MC implementation also needs to be re-based against the latest master - I just haven't had a chance to do it yet because some of my hardware has been taken away from me the last few months - will see if I can find some reasonable hardware soon. - Michael On 08/12/2014 01:22 AM, Walid Nouri wrote:Hi, I will do my best to make a contribution :-) Are there alternative ways of replicating local storage other than DRBD that are possibly feasible? Some that are directly build into Qemu? Walid Am 09.08.2014 14:25, schrieb Michael R. Hines:On Sat, 2014-08-09 at 14:08 +0200, Walid Nouri wrote:Hi Michael, how is the weather in Bejing? :-)It's terrible. Lots of pollution =(May I ask you some questions to your MC implementation? Currently i'm trying to understand the general working of the MC protokoll and possible problems that can occur so that I can discuss it in my thesis. As far as i have understand MC relies on a shared disk. Output of the primary vm are directly written, network output is buffered until the corresponding checkpoint is acknowledged. One problem that comes into my mind is: What happens when the primary vm writes to the disk and crashes before sending a corresponding checkpoint?The MC implementation itself is incomplete, today. (I need help). The Xen Remus implementation uses the DRBD system to "mirror" all disk writes to the source and destination before completing each checkpoint. The KVM (mc) implementation needs exactly the same support, but it is missing today. Until that happens, we are *required* to use root-over-iSCSI or root-over-NFS (meaning that the guest filesystem is mounted directly inside the virtual machine without the host knowing about it. This has the effect of translating all disk I/O into network I/O, and since network I/O is already buffered, then we are safe.Here an example: The Primary state is in the actual epoch epoch (n), secondary state is in epoch (n-1). The primary writes to disk and crashes before or while sending the checkpoint n. In this case the secondary memory state is still at epoch (n-1) and the state of the shared Disk corresponds to the primary state of epoch (n). How does MC guaranty that the Disk state of the backup vm is consistent with its Memory state?As I mentioned above, we need the equivalent of the Xen solution, but I just haven't had the time to write it (or incorporate someone else's implementation). Patch is welcome =)Is Memory-VCPU / Disk State consistency necessary under all circumstances? Or can this be neglected because the secondary will (after a fail over) repeat the same instructions and finally write to disk the same (as the primary before) data for a second time? Could this lead to fatal inconsistencies? Walid- Michael
[Prev in Thread] | Current Thread | [Next in Thread] |