Re: [Qemu-devel] [RFC PATCH v3 00/11] qemu-ga: fsfreeze on Windows using VSS

[Laszlo Ersek]

Sekiyama-san,

On 05/21/13 17:33, Tomoki Sekiyama wrote:

> * About errors in Windows 7 with patch v2
>   VSS requires to write to snapshot volumes just before making them read-only
>   at final commit phase. This feature is called `auto-recovery'
>   (See 
> http://msdn.microsoft.com/en-us/library/windows/desktop/aa384651(v=vs.85).aspx#base.vssgloss_auto_recoved_shadow_copy
>  for details).
> 
>   Since qemu and libvirt don't have feature to handle writable snapshots,
>   this patchset just disables auto-recovery by specifying
>   VSS_VOLSNAP_ATTR_NO_AUTORECOVERY flag to SetContext.
>   Unfortunately, this flag seems unsupported in Windows 7 or earlier.
>   It tries to look up the snapshot volumes to write and fails in
>   VSS_E_OBJECT_NOT_FOUND.
> 
>   For fundamental resolution we need a framework for guests to query snapshots
>   and to mount them as writable snapshots, but we just ignore the error in
>   this patchset.

I've been trying to understand VSS as documented in

<http://technet.microsoft.com/en-us/library/ee923636%28v=ws.10%29.aspx>

I'll be referring to the steps detailed there as "VSS steps".

Currently I'm under the impression that the "guest-fsfreeze-freeze" and
"guest-fsfreeze-thaw" commands do not match VSS from a "who has control
when" perspective.

VSS steps 1 to 5 correspond to "guest-fsfreeze-freeze". In the POSIX
flavor of qemu-ga, including fsfreeze hook scripts, at this point
qemu-ga gets back and returns control to the management stack, and
backup can be done.

VSS step 6 corresponds to the management stack preparing the snapshot(s)
or other means of backup.

VSS steps 7 to 8 correspond to "guest-fsfreeze-thaw". The POSIX flavor
of qemu-ga thaws filesystems and reactivates applications / enables them
to dirty their own data stores. This action is initiated by the
management stack which has full control over its timing and conditions etc.

I believe to discover the following (orthogonal) mismatches between VSS
and our current freeze/thaw APIs:

(i) Once VSS starts running at step 1, it doesn't relinquish control
after step 5, and it doesn't allow an external controller to relaunch it
from step 7. In other words, VSS step 6 is a callout / callback
initiated by Windows logic that doesn't match our current control ownership.

In theory VSS step 6 should issue a QMP event (?) that allows libvirt to
make the snapshot, and until libvirt is done with that, VSS step 6
should block.


(ii) Somewhat similar is the control over backup storage:

An fsfreeze hook script causes the corresponding application to clean up
its live data store. The application doesn't encounter the notion of a
separate / detached copy, and no additional storage is visible (needs to
be visible) in the guest. The management stack exploits this guarantee
when it (basically blindly) snapshots the disk image.

Under VSS, in-guest components (VSS providers, basically drivers) are
responsible for incremental backup (copy-on-write etc.) features. I'm
not sure how this can be combined with qcow2 features managed by
libvirt. The VSS provider could target a separate disk, update it
incrementally to mirror the current live data, and libvirt could keep
snapshots of this backup disk.


(iii) VSS step 10 exists because in fact I cheated in (i) and VSS steps
1 to 5 do not correspond to guest-fsfreeze-freeze; the VSS design has
aspects that are completely absent from guest-fsfreeze-freeze.

guest-fsfreeze-freeze, including the hook scripts, gives as much time to
applications to quiesce as they need. When the main hook script returns
control to qemu-qa, and when qemu-ga returns control to libvirt, the
layer taking back control is assured that the lower layer has completed
all of its freezing responsibilities.

This is not so with VSS steps 1-5, and the root cause is performance /
avoidance of live service interruption. VSS puts a 60 second limit on
application data store quiescing, and a 10 second limit on actual
snapshotting / backup (VSS step 6). Live service should resume in no
more than 70 seconds apparently.

VSS realizes that many applications won't be able to satisfy this
requirement. Therefore the application level quiescing is split in a
synchronous handler and an asynchronous "bottom half" (which is VSS step
10).
- Up to and including VSS step 5, an application prepares but a separate
dirty context (in 60 seconds).
- In VSS step 6, this context ("shadow copy") is detached (move to other
storage, probably).
- In VSS steps 7 and 8, the application resumes live service. Also,
- VSS step 10, the "bottom half" of the application (a subprocess or a
dedicated thread, maybe), cleans up the now detached dirty context, *in
parallel* to the live service of the application.

If we disable VSS step 10 (VSS_VOLSNAP_ATTR_NO_AUTORECOVERY), I'm still
confused / unsure about:

- the control ownership problem in (i) and the storage problem in (ii)
-- I believe your series must already solve this; can you please
describe it in a few words?

- the time limits for VSS steps 5 and 6 probably remain in place, but
without a bottom half (VSS step 10) applications may not be able to
satisfy them. Does VSS_VOLSNAP_ATTR_NO_AUTORECOVERY make applications
exempt from the time limits?


Interestingly, like at least
<http://wiki.iaso.com/index.php/Manual:MSSQL_VSS_Auto_Recovery> suggests
for MSSQL, VSS step 10 seems to depend on both the VSS requestor and the
application:

  !VSS_VOLSNAP_ATTR_NO_AUTORECOVERY && MSSQL.AllowVssAutoRecover

If this condition is not satisfied, then (I believe from the linked
article) MSSQL will not choose

  "breach time limits but create pristine backup",

instead it will opt for

  "prepare dirty backup and delay recovery until restore time".

This is probably not optimal and VSS step 10 could be a requirement if
quick restore is important.


Please point out any mistakes in the above reasoning (although I'm sure
it's easier to point out the few places where I'm *not* mistaken) -- I
need to understand clearly the VSS procedure and its mapping (steps 1 to
9) to our APIs for any half-useful review (I know nothing about
Windows). Apologies if this has been discussed before, can you please
provide pointers in that case?

Thank you,
Laszlo