Re: [PATCH v3] block/nbd: use non-blocking connect: fix vm hang on connect()

[Vladimir Sementsov-Ogievskiy]

19.08.2020 20:52, Eric Blake wrote:
> On 8/12/20 9:52 AM, Vladimir Sementsov-Ogievskiy wrote:
> > This make nbd connection_co to yield during reconnects, so that
> > reconnect doesn't hang up the main thread. This is very important in
> > case of unavailable nbd server host: connect() call may take a long
> > time, blocking the main thread (and due to reconnect, it will hang
> > again and again with small gaps of working time during pauses between
> > connection attempts).
>
> > How to reproduce the bug, fixed with this commit:
> >
> > 1. Create an image on node1:
> >     qemu-img create -f qcow2 xx 100M
> >
> > 2. Start NBD server on node1:
> >     qemu-nbd xx
> >
> > 3. Start vm with second nbd disk on node2, like this:
> >
> >    ./x86_64-softmmu/qemu-system-x86_64 -nodefaults -drive \
> >      file=/work/images/cent7.qcow2 -drive file=nbd+tcp://192.168.100.2 \
> >      -vnc :0 -qmp stdio -m 2G -enable-kvm -vga std
>
> Where is the configuration to set up retry on the nbd connection?  I wonder if 
> you have a non-upstream patch that turns it on by default in your builds; for 
> upstream, I would have expected something more along the lines of -blockdev 
> driver=nbd,reconnect-delay=20,server.type=inet,server.data.hostname=192.168.100.2,server.data.port=10809
>  (typing off the top of my head, rather than actually tested).

No, it's not necessary: reconnect is enabled always. reconnect-delay just says 
what to do with guest requests when connection is down. By default, they just 
fails immediately. But even with reconnect-delay=0 reconnect code works and 
tries to reestablish the connection.

> > 4. Access the vm through vnc (or some other way?), and check that NBD
> >     drive works:
> >
> >     dd if=/dev/sdb of=/dev/null bs=1M count=10
> >
> >     - the command should succeed.
> >
> > 5. Now, let's trigger nbd-reconnect loop in Qemu process. For this:
> >
> > 5.1 Kill NBD server on node1
> >
> > 5.2 run "dd if=/dev/sdb of=/dev/null bs=1M count=10" in the guest
> >      again. The command should fail and a lot of error messages about
> >      failing disk may appear as well.
>
> Why does the guest access fail when the server goes away?  Shouldn't the 
> pending guest requests merely be queued for retry (where the guest has not seen 
> a failure yet, but may do so if timeouts are reached), rather than being 
> instant errors?

And that's exactly how it should work when reconnect-delay is 0. If you set 
reconnect-delay to be >0, then in this period of time after detection of 
connection failure all the requests will be queued.

> >      Now NBD client driver in Qemu tries to reconnect.
> >      Still, VM works well.
> >
> > 6. Make node1 unavailable on NBD port, so connect() from node2 will
> >     last for a long time:
> >
> >     On node1 (Note, that 10809 is just a default NBD port):
> >
> >     sudo iptables -A INPUT -p tcp --dport 10809 -j DROP
> >
> >     After some time the guest hangs, and you may check in gdb that Qemu
> >     hangs in connect() call, issued from the main thread. This is the
> >     BUG.
> >
> > 7. Don't forget to drop iptables rule from your node1:
> >
> >     sudo iptables -D INPUT -p tcp --dport 10809 -j DROP


--
Best regards,
Vladimir