Re: [Qemu-discuss] qemu qmp unix sockets stop working with 'connection refused' randomly

[Brad Barnett]

So, here's the skinny.

Lots of background, but I have a series of scripts.  One starts/stops a
single qemu instance, and another calls that to start/stop all VMs.

The "all start/stop VMs" script is used on boot or shutdown, called by an
init script, and the single script does a variety of things -- like send
ACPI shutdown, waiting, etc, etc.

Anyhow when scripting these things, I noticed this behaviour in qemu if I
attempt to start a running qemu:

$ kvm <lots of command line args plus..> -pid pidfile.pid
Could not acquire pid file: Resource temporarily unavailable

So, I skipped past the "check if qemu" is running phase, and left the job
of not starting a running VM to qemu.  It should, after all, be best at
this.

(This was my thought process.)

However whenever qemu starts it opens / overwrites the socket, THEN sees
the PID file is locked, then closes the socket + exits.

And, you are then left with a running qemu which has no usable socket any
more.

The bare metal layer servers we're using, have both production and
development VM instances.  And, those development instances do get
stopped and started much more often.

Meaning, it is often convenient to stop individual dev VMs one at a time,
but start them all back up at once.

And then the next day, or next week I'd notice the qmp socket was MIA.

So, there's my story.  I hope you enjoyed it.  I'll be at the concierge's
office while the police arrive.




On Sun, 30 Oct 2016 10:05:39 +0100
Christian Theune <address@hidden> wrote:

> Hi,
> 
> I’ve seen a similar behaviour when connecting to the telnet monitor
> port. I changed out scripts to use QMP over unix socket as that
> appeared more stable (and is obviously (to me) the better choice than
> to “expect” on the telnet port). I did get a first report about a
> broken socket in the last week, so your mail makes me stick my head out
> for a potentially more widespread issue that may be similar between the
> telnet monitor port and the QMP unix socket.
> 
> Christian
> 
> > On 28 Oct 2016, at 01:49, Brad Barnett <address@hidden> wrote:
> > 
> > 
> > Hey all,
> > 
> > So, I've been unable to find anything about this via Google searches,
> > mailing list searches, IRC, you name it.  Nor, does the documentation
> > seem to indicate I'm doing anything wrong.
> > 
> > I'm starting qemu with various options, including this:
> > 
> > -qmp unix:/path_to_sock_dir/box.uniquename.sock,server,nowait
> > 
> > I'm using socat to connect:
> > 
> > socat UNIX-CONNECT:/path_to_sock_dir/box.uniquename.sock STDIO
> > 
> > This works fine, for the most part.  I can connect, issue commands:
> > 
> > {"QMP": {"version": {"qemu": {"micro": 2, "minor": 1, "major": 2},
> > "package": " (Debian 1:2.1+dfsg-12+deb8u6)"}, "capabilities": []}}
> > { "execute": "qmp_capabilities" }
> > {"return": {}}
> > 
> > And as you see, get responses.  When it works, it works perfectly.
> > 
> > Qemu version is the one shippied with Debian Jessie:
> > 
> > # qemu-system-x86_64 --version
> > QEMU emulator version 2.1.2 (Debian 1:2.1+dfsg-12+deb8u6), Copyright
> > (c) 2003-2008 Fabrice Bellard
> > 
> > What happens is that randomly, socat can no longer connect to the
> > socket in question.  This has happened after successful uses of that
> > very socket, and also if I've never used that socket before.
> > 
> > EG, I start qemu, later connect to socket, do things, exit.  I can
> > even connect a random amount of times, do things, etc.  Or, not
> > connect at all after start!
> > 
> > Randomly, I'll end up with the socket in a state where socat states
> > "connection refused' when trying to connect.
> > 
> > This was the only bug I could find, that seemed even remotely close:
> > 
> > https://bugs.launchpad.net/qemu/+bug/1156632
> > 
> > However, my issue also happen if I have never before connected to the
> > socket in question.
> > 
> > This has happened on multiple servers with varying configurations --
> > but all servers are the above qemu version, Debian Jessie, and
> > similar qemu command line configs.
> > 
> > ps shows qemu running as:
> > 
> > qvmuser-system-x86_64 -enable-kvm -net nic,macaddr=<mac address here>
> > -net vde,sock=/var/run/vde2/mytap.ctl -m 3G -smp 2 -name boxname
> > -drive file=/home/vmuser/images/boxname.raw,if=virtio,boot=on
> > -D /home/vmuser/log/boxname.logfile -pidfile /run/vmuser/boxname.pid
> > -monitor stdio -display vnc=127.0.0.1:14,password -vga qxl -qmp
> > unix:/home/vmuser/sock/boxname.sock,server,nowait
> > 
> > Anyone have any ideas?  Is there something I missed with qmp?
> > 
> > I do have some ability to test / debug on a local server farm, with
> > VMs, VM images, etc that are identical to production.
> > 
> > 
> > 
> > Thanks!
> > 
> > 
> 
> --
> Christian Theune · address@hidden · +49 345 219401 0
> Flying Circus Internet Operations GmbH · http://flyingcircus.io
> Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
> HR Stendal HRB 21169 · Geschäftsführer: Christian. Theune, Christian.
> Zagrodnick
>