Re: [Qemu-devel] [Patch] VNC: Fix crash with non-resizing clients

[Daniel P. Berrange]

On Wed, Oct 03, 2007 at 09:29:59PM +0200, Eduardo Felipe wrote:
> Hi,
> 
> 2007/9/25, GUERRAZ Francois <address@hidden>:
> >
> >
> > About your VNC problems : I have had problems w/ vnc too (see
> > http://qemu-forum.ipi.fi/viewtopic.php?p=10468) but had no answer as
> > well...
> 
> 
> This problem happens when the VNC client doesn't support the DesktopSize
> pseudo-encoding. Qemu crashes when the guest resizes down its display and
> the VNC client sends a SetPixelFormat afterwards.
> 
> Attached patch should fix this. It also forces a full buffer update after
> resize.

The memset calls in that patch are bogus & not correctly fixing the buffer
update problem. You're merely setting the 'old data' to have pixel value
42 - if the guest OS framebuffer happens to also have aras with pixel value
 of 42 too, the frame buffer will still not correctly update. The root
problem is overly-aggressive update minimization logic in vnc_update_client.
This is in turn flawed beause the dirty_row aray is trying to encode two
separate concepts - areas which are dirty, and areas which need to be 
sent to the client. The latter are a superset of the former, but the code
in vnc_update_client minimizes based on dirtiness, so updates will get
missed out. Setting the old data to 42 merely changes which areas will get
missed updates.

The QEMU code in Xen has added a update_row field, separate from the dirty_row
field. Thus after a resize it can update the entire framebuffer, regardless
of whether QEMU's copy of the framebuffer is dirty wrt to the guest copy.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|