On Mon, Jan 22, 2024 at 4:16 AM Andrew Hyatt <ahyatt@gmail.com> wrote:
>
>
> Hi everyone,
Hi Andrew,
I have some ideas to share, though keep in mind this is mainly
thinking out loud and I'm largely an LLM newbie.
> Question 1: Does the llm-flows.el file really belong in the llm
> package?
Maybe, but keep the functions isolated. I'd be interested in
a diff-mode flow which is different from this ediff-one you
demo. So it should be possible to build both.
The diff-mode flow I'm thinking of would be similar to the
diff option of LSP-proposed edits if your code, btw. See the
variable eglot-confirm-server-edits for an idea of the interface.
Great call-out, thanks. I'll check it out. I'm starting to get a better idea of how this might all work out in my mind.
> Question 3: How should we deal with context? The code that has the
> text corrector doesn't include surrounding context (the text
> before and after the text to rewrite), but it usually is helpful.
> How much context should we add?
Karthik of gptel.el explained to me that this is one of
the biggest challenges of working with LLMs, and that GitHub
Copilot and other code-assistance tools work by sending
not only the region you're interested in having the LLM help you
with but also some auxiliary functions and context discovered
heuristically. This is potentially complex, and likely doesn't
belong in the your base llm.el but it should be possible to do
somehow with an application build on top of llm.el (Karthik
suggests tree-sitter or LSP's reference finding abilities to
discover what's nearest in terms of context).
Interesting idea - yes, this should be customizable, and it will be quite complicated in some cases.
In case noone mentinoed this already, i think a good logging
facility is essential. This could go in the base llm.el library.
I'm obviously biased towards my own jsonrpc.el logging facilities,
where a separate easy-to-find buffer for each JSON-RPC connection
lists all the JSON transport-level conversation details in a
consistent format. jsonrpc.el clients can also use those logging
facilities to output application-level details.
In an LLM library, I suppose the equivalent to JSON transport-level
details are the specific API calls to each provider, how it gathers
context, prompts, etc. Those would be distinct for each LLM.
A provider-agnosntic application built on top of llm.el's abstraction
could log in a much more consistent way.
So my main point regarding logging is that is should live in a
readable log buffer, so it's easy to piece together what happened
and debug. Representing JSON as pretty-printed plists is often
very practical in my experience (though a bit slow if loads of text
is to be printed).
Good feedback, mainly for what I've already released in the llm package so far. JSON is useful for the initial request, but there's a lot of streaming that happens,which isn't really valid JSON, although sometimes it contains valid JSON. Sometimes it is just JSON, streaming a chunk at a time. So in general you have to deal with that stuff as just pure text.
So far, setting url-debug to non-nil is sufficient for basic debugging, but making a more standard and better logging facility would be very nice. I'll work on it.
Maybe these logging transcripts could even be used to produce
automated tests, in case there's a way to achieve any kind of
determinism with LLMs (not sure if there is).
Probably not, unfortunately, even if you can remove the randomness, little changes are always happening with newer version of the model or the processing around it. I do have included a fake LLM that can be used to test whatever flow, though.
Similarly to logging, it would be good to have some kind
of visual feedback of what context is being sent in each
LLM request. Like momentarily highlighting the regions
to be sent alongside the prompt. Sometimes that is
not feasible. So it could make sense to summarize that extra
context in a few lines shown in the minibuffer perhaps. Like
"lines 2..10 from foo.cpp\nlines42-420 from bar.cpp"
I like the idea of logging to the context. I think it might make sense to just add that to the debug buffer instead of the minibuffer, though. Hopefully things just work, and so the minibuffer would just show something like it does in the demo, just saying it's sending things to whatever LLM (perhaps a reference to the debug buffer would be nice, though).
So just my 200c,
Good luck,
João