Re: qmp-shell for GSoC/Outreachy?

[Kevin Wolf]

Am 06.02.2020 um 19:26 hat Dr. David Alan Gilbert geschrieben:
> * Kevin Wolf (address@hidden) wrote:
> > Am 06.02.2020 um 10:40 hat Markus Armbruster geschrieben:
> > > >> On 2/5/20 8:09 AM, Kevin Wolf wrote:
> > > >> > Am 28.01.2020 um 11:59 hat Kevin Wolf geschrieben:
> > > >> >>>> The other part that it needs to solve is how to be available by 
> > > >> >>>> default
> > > >> >>>> without specifying anything on the command line. Basically, if I 
> > > >> >>>> press
> > > >> >>>> Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
> > > >> >>>> implemented internally or by an external Python process, I don't 
> > > >> >>>> mind.
> > > >> >>>
> > > >> >>> That is a harder part. (I rarely use Ctrl-Alt-2 actually; I mostly
> > > >> >>> use HMP on stdin).
> > > >> >>
> > > >> >> I don't think it would be that hard, actually.
> > > >> >>
> > > >> >> If you have a -qmp-shell option that takes a chardev and defaults 
> > > >> >> to vc,
> > > >> >> you've solved the part with both stdio and Ctrl-Alt-2. Now all you 
> > > >> >> need
> > > >> >> to do is launch the Python child process, pass it a pair of pipes 
> > > >> >> for
> > > >> >> communication and forward everything between the pipes and the 
> > > >> >> chardev.
> > > >> >>
> > > >> >> (That's the theory anyway.)
> > > >> > 
> > > >> > If someone is interested, I did a quick proof-of-concept hack:
> > > >> > 
> > > >> >     https://repo.or.cz/qemu/kevin.git/shortlog/refs/heads/qmp-shell
> > > >> > 
> > > >> > It doesn't clean up anything properly (including the qmp-shell 
> > > >> > processes
> > > >> > it starts), but it spawns a usable qmp-shell on a user-specified
> > > >> > character device. stdio seems to work, though without readline
> > > >> > functionality (I suppose I still have line-buffering somewhere), vc
> > > >> > doesn't really work at all yet.
> > > >> > 
> > > >> > Try it out like this:
> > > >> > 
> > > >> >     $ ./qemu-storage-daemon --chardev stdio,id=m --monitor 
> > > >> > m,mode=qmp-shell
> > > >> >     monitor_qmp_event: 1
> > > >> >     Welcome to the QMP low-level shell!
> > > >> >     Connected to QEMU 4.2.50
> > > >> > 
> > > >> >     (QEMU) query-version
> > > >> >     {"return": {"qemu": {"micro": 50, "major": 4, "minor": 2}, 
> > > >> > "package": "v4.2.0-1188-gd95a3885a9"}}
> > > >> >     (QEMU) quit
> > > >> > 
> > > >> > (Or use x86_64-softmmu/qemu-system-x86_64, but it's based on the
> > > >> > refactorings in the storage daemon branch, so why not try both at 
> > > >> > once?)
> > > >> > 
> > > >> > Polishing this to make it mergable would still require substantial 
> > > >> > work,
> > > >> > so at the moment I'm not planning to do this. But if someone wants to
> > > >> > pick it up, feel free (just let us know).
> > > >> > 
> > > >> > Hm, in fact... A qmp-shell GSoC project?
> > > >> > 
> > > >> 
> > > >> That would be great. I worry that we should have a clear vision for the
> > > >> syntax before we give this project to an intern, though. With a clear
> > > >> vision and an outline for deliverables, it's an incredibly appropriate
> > > >> project.
> > > >> 
> > > >> Some things I think we want to define before we start:
> > > >> 
> > > >> 1. What are we trying to achieve with a standalone shell?
> > > 
> > > Projects without a clear goal rarely succeed.  Success within three
> > > months is even rarer.
> > >
> > > >> 2. What syntax should it use?
> > > 
> > > Leaving that to a GSoC student amounts to setting up for failure.
> > 
> > I think this subthread shows that we actually have many separate
> > projects that people wish to have someone work on. Each of them is
> > probably a bit too small for a whole GSoC, but all of them together are
> > probably too much. So I'll guess the student would pick maybe two of
> > them, and if time is left at the end, more can be added as a bonus.
> > 
> > 1. Something like --monitor mode=qmp-shell that just spawns an external
> >    Python script and passes it a QMP socket. This is the fundamental
> >    building block for having any kind of external monitor script
> >    actually integrated in QEMU, so I think just running the existing
> >    qmp-shell this way (with proper support for at least stdio and vc
> >    chardevs) would make sense as a first milestone.
> 
> I was originally going to suggest that should be sugar for a
> -chardev filter  that takes an in/out chardev - but I don't know how
> you'd handle tty stuff like formatting and tab completion etc.

Originally I thought the same, but it's actually not enough. The
external process needs not only stdio connected to the chardev, but also
another file descriptor for a QMP socket.

I guess tab completion should be possible if we can turn off line
buffering. Not sure what's the problem with formatting.

> > 2. Rewriting qmp-shell to use a better syntax for nested data
> >    structures. This would have to be defined before the project starts.
> > 
> > 3. Improving qmp-shell UI-wise, e.g. by having better autocompletion,
> >    support for counting brackets, or whatever else was mentioned. We
> >    have a few ideas, and there's room for the student to add their own
> >    ideas, too.
> > 
> > 4. Something HMP-like. This isn't QMP any more, so it could as well be a
> >    separate script (hmp-shell?). But it could also be integrated in
> >    qmp-shell in the form of additional commands that are implemented
> >    client-side. Or maybe have a single shell, but have a QMP mode and an
> >    HMP mode and the user can switch between these modes.
> > 
> >    The syntax for the HMP shell/mode could be the same or different from
> >    the QMP syntax. This would have to be defined beforehand, too.
> 
> separate script sharing the qmp interface code?

I would certainly expect that any QMP/HMP shell code uses the existing
QEMUMonitorProtocol Python class. qmp-shell already does.

Maybe they could share a bit more (command parsing?) if we want to have
separate scripts for HMP and QMP. Though actually I think I like the
idea of a combined QMP/HMP shell better at the moment.

Kevin