[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: qmp-shell for GSoC/Outreachy?
From: |
Dr. David Alan Gilbert |
Subject: |
Re: qmp-shell for GSoC/Outreachy? |
Date: |
Thu, 6 Feb 2020 18:26:39 +0000 |
User-agent: |
Mutt/1.13.3 (2020-01-12) |
* Kevin Wolf (address@hidden) wrote:
> Am 06.02.2020 um 10:40 hat Markus Armbruster geschrieben:
> > >> On 2/5/20 8:09 AM, Kevin Wolf wrote:
> > >> > Am 28.01.2020 um 11:59 hat Kevin Wolf geschrieben:
> > >> >>>> The other part that it needs to solve is how to be available by
> > >> >>>> default
> > >> >>>> without specifying anything on the command line. Basically, if I
> > >> >>>> press
> > >> >>>> Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
> > >> >>>> implemented internally or by an external Python process, I don't
> > >> >>>> mind.
> > >> >>>
> > >> >>> That is a harder part. (I rarely use Ctrl-Alt-2 actually; I mostly
> > >> >>> use HMP on stdin).
> > >> >>
> > >> >> I don't think it would be that hard, actually.
> > >> >>
> > >> >> If you have a -qmp-shell option that takes a chardev and defaults to
> > >> >> vc,
> > >> >> you've solved the part with both stdio and Ctrl-Alt-2. Now all you
> > >> >> need
> > >> >> to do is launch the Python child process, pass it a pair of pipes for
> > >> >> communication and forward everything between the pipes and the
> > >> >> chardev.
> > >> >>
> > >> >> (That's the theory anyway.)
> > >> >
> > >> > If someone is interested, I did a quick proof-of-concept hack:
> > >> >
> > >> > https://repo.or.cz/qemu/kevin.git/shortlog/refs/heads/qmp-shell
> > >> >
> > >> > It doesn't clean up anything properly (including the qmp-shell
> > >> > processes
> > >> > it starts), but it spawns a usable qmp-shell on a user-specified
> > >> > character device. stdio seems to work, though without readline
> > >> > functionality (I suppose I still have line-buffering somewhere), vc
> > >> > doesn't really work at all yet.
> > >> >
> > >> > Try it out like this:
> > >> >
> > >> > $ ./qemu-storage-daemon --chardev stdio,id=m --monitor
> > >> > m,mode=qmp-shell
> > >> > monitor_qmp_event: 1
> > >> > Welcome to the QMP low-level shell!
> > >> > Connected to QEMU 4.2.50
> > >> >
> > >> > (QEMU) query-version
> > >> > {"return": {"qemu": {"micro": 50, "major": 4, "minor": 2},
> > >> > "package": "v4.2.0-1188-gd95a3885a9"}}
> > >> > (QEMU) quit
> > >> >
> > >> > (Or use x86_64-softmmu/qemu-system-x86_64, but it's based on the
> > >> > refactorings in the storage daemon branch, so why not try both at
> > >> > once?)
> > >> >
> > >> > Polishing this to make it mergable would still require substantial
> > >> > work,
> > >> > so at the moment I'm not planning to do this. But if someone wants to
> > >> > pick it up, feel free (just let us know).
> > >> >
> > >> > Hm, in fact... A qmp-shell GSoC project?
> > >> >
> > >>
> > >> That would be great. I worry that we should have a clear vision for the
> > >> syntax before we give this project to an intern, though. With a clear
> > >> vision and an outline for deliverables, it's an incredibly appropriate
> > >> project.
> > >>
> > >> Some things I think we want to define before we start:
> > >>
> > >> 1. What are we trying to achieve with a standalone shell?
> >
> > Projects without a clear goal rarely succeed. Success within three
> > months is even rarer.
> >
> > >> 2. What syntax should it use?
> >
> > Leaving that to a GSoC student amounts to setting up for failure.
>
> I think this subthread shows that we actually have many separate
> projects that people wish to have someone work on. Each of them is
> probably a bit too small for a whole GSoC, but all of them together are
> probably too much. So I'll guess the student would pick maybe two of
> them, and if time is left at the end, more can be added as a bonus.
>
> 1. Something like --monitor mode=qmp-shell that just spawns an external
> Python script and passes it a QMP socket. This is the fundamental
> building block for having any kind of external monitor script
> actually integrated in QEMU, so I think just running the existing
> qmp-shell this way (with proper support for at least stdio and vc
> chardevs) would make sense as a first milestone.
I was originally going to suggest that should be sugar for a
-chardev filter that takes an in/out chardev - but I don't know how
you'd handle tty stuff like formatting and tab completion etc.
> 2. Rewriting qmp-shell to use a better syntax for nested data
> structures. This would have to be defined before the project starts.
>
> 3. Improving qmp-shell UI-wise, e.g. by having better autocompletion,
> support for counting brackets, or whatever else was mentioned. We
> have a few ideas, and there's room for the student to add their own
> ideas, too.
>
> 4. Something HMP-like. This isn't QMP any more, so it could as well be a
> separate script (hmp-shell?). But it could also be integrated in
> qmp-shell in the form of additional commands that are implemented
> client-side. Or maybe have a single shell, but have a QMP mode and an
> HMP mode and the user can switch between these modes.
>
> The syntax for the HMP shell/mode could be the same or different from
> the QMP syntax. This would have to be defined beforehand, too.
separate script sharing the qmp interface code?
Dave
> 5. Probably more that I just forgot now.
>
> Suggesting the exact goals is part of the student application process,
> but for fundamental things like the syntax we should probably already
> know what we want.
>
> > >> I think those are the hardest parts.
> > >>
> > >> Below, some musings:
> > >>
> > >> - An integrated QMP shell would be a great usability boost to users of
> > >> bare QEMU.
> > >>
> > >> - It is undesirable in general to support two interfaces. Feature
> > >> disparity is a problem, as is needing to document and test two separate
> > >> interfaces. The quality disparity between the two is also an issue.
> > >>
> > >> - Offering HMP via the GTK interface but not QMP is a discoverability
> > >> problem. Unfamiliar users might assume that HMP is our flagship
> > >> interface. It is not.
> > >>
> > >> - We are unlikely to re-expand HMP to cover everything QMP does; writing
> > >> a QMP shell that makes QMP easy to interface with is a better solution
> > >> for removing redundancy and complexity.
>
> I'm not entirely convinced of this because QMP is often too low-level to
> actually address the practical high-level needs of users.
>
> But these HMP-ish things are probably easier to maintain as scripts
> outside of the QEMU binary, so I think some kind of "QMP with
> extensions" for human could be the solution.
>
> Once it's an external script, it will also be easy to exchange the shell
> for another one depending on user preference, or to hack in whatever
> functionality they are missing.
>
> > >> - I suspect that the target audience for users of naked QEMU are:
> > >> - QEMU developers
> > >> - Upper-layer developers (RHV, oVirt, KubeVirt, libvirt, kata, et al)
> > >> researching, testing, and debugging integration.
> > >> - Devops professionals testing, implementing and debugging
> > >> configuration & infrastructure
> > >> - Security/infosec researchers
> > >> - Embedded platform developers
> > >> - Academic researchers
>
> Maybe kernel developers should be mentioned separately, but yes, this
> list looks plausible to me.
>
> > >> So please correct me if I am off the mark;
> > >>
> > >> Design Goals:
> > >> - The removal of HMP
> > >> - An easy-to-use interface that remains reasonably "close" to the
> > >> machine API such that it provides a smooth transition to scripting QEMU.
> > >> - Integration with our GTK interface for discoverability and
> > >> convenience
>
> As I listed above, I think these are actually three separate projects,
> rather than goals for a single big projects.
>
> > >> Syntax:
> > >> - TBD? Do we agree that the current syntax in qmp-shell is "bad" and
> > >> should be replaced? If yes, what should it look like?
> > >
> > > I believe it should be a python shell with added commands.
> > >
> > > Simple things should be simple.
> > > e.g. adding a disk from a local file should be trivial.
> > >
> > > Complex things can be complex - but it would be better if they were
> > > simple.
> > >
> > > It's OK if the worst case of a blockdev is a bit hairy, but
> > > watch out for cases where the hairyness creeps in unnecessarily.
> >
> > Designing interfaces to complex machinery is hard. Experience tells
> > that we do okay when we focus on the building blocks first. That's
> > -blockdev. When we start with trying to make simple things simple, we
> > end in swamps. That's -drive.
> >
> > Focus on building blocks is of course no excuse for unnecessary
> > hairiness.
> >
> > It's also no reason not to build more convenient things on top of the
> > building blocks. I doubt they should go into QMP, though.
>
> Right, they should be implemented in that external script, which would
> use the lower-level QMP building blocks to provide the functionality. I
> also think it's a good idea to keep QMP accessible for more exotic use
> cases when the simple thing just doesn't cut it any more.
>
> > > If the user screwsup, it should give an error that prompts the user
> > > to the parameter they got wrong.
> > >
> > > Output from commands should normally be pretty formatted (with an option
> > > to display raw json for those needing it).
> > > e.g. that 'query-version' should give either just the package
> > > version (as info version currently does) or:
> > > 4.2.50 Package: v4.2.0-1188-gd95a3885a9
> > >
> > > We shouldn't lose any HMP commands that some people find useful
> > > Ditching HMP isn't an option until we've got almost all of it
> > > covered.
> >
> > In particular, we currently use HMP for debugging and monitoring
> > purposes, where we don't need or want QMP's rigor, neither its rigorous
> > interface stability, nor its structured I/O. We want the "whipuptitude"
> > we get from monitor_printf(). This is actually a point David has made
> > several times.
> >
> > To have a qmp-shell replace HMP, I think it needs to be able to
> >
> > * Go beyond 1:1
> >
> > We tried a 1:1 mapping between HMP and QMP commands, and it didn't
> > work out. HMP's replacement should let us build convenient commands
> > from QMP building blocks.
> >
> > We tried a 1:1 mapping between HMP and QMP command arguments, guided
> > by @args_type. Worked out for simple cases, but was too constricting.
>
> We need to go beyond 1:1, but we probably want to be able to offer 1:1
> as a subset of commands accepted in that shell.
>
> Offering only 1:1 in a good way might already be a step forward.
>
> > * Preserve "whipuptitude" [David]
> >
> > I figure that means allowing some in QMP. Without compromising its
> > core mission, of course.
>
> As long as we confine it to x- commands, I think this is okay.
>
> > * As discoverable as HMP is now [Kevin]
> >
> > * Help, completion and such at least on par with what HMP provides now
>
> Will we want to add new annotations in the schema for this?
>
> For example, HMP has completion support for block device names. In the
> QAPI schema, these are simply 'str'. We could bake the knowledge that
> in command 'foo' the parameter 'bar' is a block device name, but that
> would be a hack and would probably rarely be consistent with what QEMU
> actually does. It's really something that schema introspection should be
> able to tell us.
>
> Kevin
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK
- qmp-shell for GSoC/Outreachy? (Was: Re: Making QEMU easier for management tools and applications), (continued)
- qmp-shell for GSoC/Outreachy? (Was: Re: Making QEMU easier for management tools and applications), John Snow, 2020/02/05
- Re: qmp-shell for GSoC/Outreachy? (Was: Re: Making QEMU easier for management tools and applications), Dr. David Alan Gilbert, 2020/02/05
- Re: qmp-shell for GSoC/Outreachy?, Markus Armbruster, 2020/02/06
- Re: qmp-shell for GSoC/Outreachy?, Daniel P . Berrangé, 2020/02/06
- Re: qmp-shell for GSoC/Outreachy?, Markus Armbruster, 2020/02/06
- Re: qmp-shell for GSoC/Outreachy?, Daniel P . Berrangé, 2020/02/06
- Re: qmp-shell for GSoC/Outreachy?, Dr. David Alan Gilbert, 2020/02/06
- Re: qmp-shell for GSoC/Outreachy?, John Snow, 2020/02/07
- Re: qmp-shell for GSoC/Outreachy?, Markus Armbruster, 2020/02/08
- Re: qmp-shell for GSoC/Outreachy?, Kevin Wolf, 2020/02/06
- Re: qmp-shell for GSoC/Outreachy?,
Dr. David Alan Gilbert <=
- Re: qmp-shell for GSoC/Outreachy?, Kevin Wolf, 2020/02/07
- Re: qmp-shell for GSoC/Outreachy?, John Snow, 2020/02/07
- Re: qmp-shell for GSoC/Outreachy?, Markus Armbruster, 2020/02/08
- Re: qmp-shell for GSoC/Outreachy?, Kevin Wolf, 2020/02/10
- Re: qmp-shell for GSoC/Outreachy?, Kevin Wolf, 2020/02/10
- Re: qmp-shell for GSoC/Outreachy?, Dr. David Alan Gilbert, 2020/02/06
- Re: qmp-shell for GSoC/Outreachy?, Markus Armbruster, 2020/02/07
- Re: qmp-shell for GSoC/Outreachy?, Eric Blake, 2020/02/07
- Re: qmp-shell for GSoC/Outreachy?, Markus Armbruster, 2020/02/08
- Re: qmp-shell for GSoC/Outreachy?, John Snow, 2020/02/07