[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[elpa] externals/llm 782b892de2 1/5: Add function calling, and introduce
|
From: |
ELPA Syncer |
|
Subject: |
[elpa] externals/llm 782b892de2 1/5: Add function calling, and introduce llm-capabilities |
|
Date: |
Sat, 2 Mar 2024 15:58:22 -0500 (EST) |
branch: externals/llm
commit 782b892de23423d4f2653dd02ed2f2aa10714794
Author: Andrew Hyatt <ahyatt@gmail.com>
Commit: Andrew Hyatt <ahyatt@gmail.com>
Add function calling, and introduce llm-capabilities
This also has some changes and cleanups to Gemini code, which needed to be
refactored somewhat to handle function calling.
---
NEWS.org | 3 +
README.org | 49 +++++++-
llm-fake.el | 3 +
llm-gemini.el | 67 ++++++++---
llm-llamacpp.el | 4 +
llm-ollama.el | 3 +
llm-openai.el | 226 ++++++++++++++++++++++++++----------
llm-provider-utils.el | 158 +++++++++++++++++++++++++
llm-request.el | 12 +-
llm-tester.el | 91 +++++++++++++++
llm-vertex.el | 173 +++++++++++++++++++++------
llm.el | 108 ++++++++++++++++-
utilities/elisp-to-function-call.el | 186 +++++++++++++++++++++++++++++
13 files changed, 951 insertions(+), 132 deletions(-)
diff --git a/NEWS.org b/NEWS.org
index 8a663e9299..e8496f5a0e 100644
--- a/NEWS.org
+++ b/NEWS.org
@@ -1,3 +1,6 @@
+* Version 0.11.0
+- Introduce function calling, now available only in Open AI and Gemini.
+- Introduce =llm-capabilities=, which returns a list of extra capabilities for
each backend.
* Version 0.10.0
- Introduce llm logging (for help with developing against =llm=), set
~llm-log~ to non-nil to enable logging of all interactions with the =llm=
package.
- Change the default interaction with ollama to one more suited for
converesations (thanks to Thomas Allen).
diff --git a/README.org b/README.org
index 9f0a98a4e7..44306b784d 100644
--- a/README.org
+++ b/README.org
@@ -3,11 +3,9 @@
* Introduction
This library provides an interface for interacting with Large Language Models
(LLMs). It allows elisp code to use LLMs while also giving end-users the choice
to select their preferred LLM. This is particularly beneficial when working
with LLMs since various high-quality models exist, some of which have paid API
access, while others are locally installed and free but offer medium quality.
Applications using LLMs can utilize this library to ensure compatibility
regardless of whether the us [...]
-LMMs exhibit varying functionalities and APIs. This library aims to abstract
functionality to a higher level, as some high-level concepts might be supported
by an API while others require more low-level implementations. An example of
such a concept is "examples," where the client offers example interactions to
demonstrate a pattern for the LLM. While the GCloud Vertex API has an explicit
API for examples, OpenAI's API requires specifying examples by modifying the
system prompt. OpenAI al [...]
+LLMs exhibit varying functionalities and APIs. This library aims to abstract
functionality to a higher level, as some high-level concepts might be supported
by an API while others require more low-level implementations. An example of
such a concept is "examples," where the client offers example interactions to
demonstrate a pattern for the LLM. While the GCloud Vertex API has an explicit
API for examples, OpenAI's API requires specifying examples by modifying the
system prompt. OpenAI al [...]
Certain functionalities might not be available in some LLMs. Any such
unsupported functionality will raise a ~'not-implemented~ signal.
-
-This package is still in its early stages but will continue to develop as LLMs
and functionality are introduced.
* Setting up providers
Users of an application that uses this package should not need to install it
themselves. The llm package should be installed as a dependency when you
install the package that uses it. However, you do need to require the llm
module and set up the provider you will be using. Typically, applications will
have a variable you can set. For example, let's say there's a package called
"llm-refactoring", which has a variable ~llm-refactoring-provider~. You would
set it up like so:
@@ -124,5 +122,50 @@ Conversations can take place by repeatedly calling
~llm-chat~ and its variants.
#+end_src
** Caution about ~llm-chat-prompt-interactions~
The interactions in a prompt may be modified by conversation or by the
conversion of the context and examples to what the LLM understands. Different
providers require different things from the interactions. Some can handle
system prompts, some cannot. Some may have richer APIs for examples and
context, some not. Do not attempt to read or manipulate
~llm-chat-prompt-interactions~ after initially setting it up for the first
time, because you are likely to make changes that only work fo [...]
+** Function calling
+*Note: function calling functionality is currently alpha quality. If you want
to use function calling, please watch the =llm=
[discussion](https://github.com/ahyatt/llm/discussions) section for any
announcements about changes.*
+
+Function calling is a way to give the LLM a list of functions it can call, and
have it call the functions for you. The standard interaction has the following
steps:
+1. The client sends the LLM a prompt with functions it can call.
+2. The LLM may return which functions to execute, and with what arguments, or
text as normal.
+3. If the LLM has decided to call one or more functions, those functions
should be called, and their results sent back to the LLM.
+4. The LLM will return with a text response based on the initial prompt and
the results of the function calling.
+5. The client can now can continue the conversation.
+
+This basic structure is useful because it can guarantee a well-structured
output
+(if the LLM does decide to call the function). *Not every LLM can handle
function
+calling, and those that do not will ignore the functions entirely*. The
function
+=llm-capabilities= will return a list with =function-calls= in it if the LLM
+supports function calls. Right now only Gemini, Vertex and Open AI support
+function calling. Ollama should get function calling soon. However, even for
+LLMs that handle function calling, there is a fair bit of difference in the
+capabilities. Right now, it is possible to write function calls that succeed in
+Open AI but cause errors in Gemini, because Gemini does not appear to handle
+functions that have types that contain other types. So client programs are
+advised for right now to keep function to simple types.
+
+The way to call functions is to attach a list of functions to the
+=llm-function-call= slot in the prompt. This is a list of =llm-function-call=
+structs, which takes a function, a name, a description, and a list of
+=llm-function-arg= structs. The docstrings give an explanation of the format.
+
+The various chat APIs will execute the functions defined in =llm-function-call=
+with the arguments supplied by the LLM. Instead of returning (or passing to a
+callback) a string, instead an alist will be returned of function names and
+return values.
+
+The client must then send this back to the LLM, to get a textual response from
+the LLM based on the results of the function call. These have already been
added
+to the prompt, so the client only has to call the LLM again. Gemini and Vertex
+require this extra call to the LLM, but Open AI does not.
+
+Be aware that there is no gaurantee that the function will be called correctly.
+While the LLMs mostly get this right, they are trained on Javascript functions,
+so imitating Javascript names is recommended. So, "write_email" is a better
name
+for a function than "write-email".
+
+Examples can be found in =llm-tester=. There is also a function call to
generate
+function calls from existing elisp functions in
+=utilities/elisp-to-function-call.el=.
* Contributions
If you are interested in creating a provider, please send a pull request, or
open a bug. This library is part of GNU ELPA, so any major provider that we
include in this module needs to be written by someone with FSF papers.
However, you can always write a module and put it on a different package
archive, such as MELPA.
diff --git a/llm-fake.el b/llm-fake.el
index 7cadba0a9e..69d0cccb48 100644
--- a/llm-fake.el
+++ b/llm-fake.el
@@ -118,5 +118,8 @@ message cons. If nil, the response will be a simple vector."
(cl-defmethod llm-name ((_ llm-fake))
"Fake")
+(cl-defmethod llm-capabilities ((_ llm-fake))
+ (list 'streaming 'embeddings))
+
(provide 'llm-fake)
;;; llm-fake.el ends here
diff --git a/llm-gemini.el b/llm-gemini.el
index 21bd1ff04b..444528159e 100644
--- a/llm-gemini.el
+++ b/llm-gemini.el
@@ -30,6 +30,7 @@
(require 'llm)
(require 'llm-request)
(require 'llm-vertex)
+(require 'llm-provider-utils)
(require 'json)
(cl-defstruct llm-gemini
@@ -81,33 +82,57 @@ If STREAMING-P is non-nil, use the streaming endpoint."
(if streaming-p "streamGenerateContent" "generateContent")
(llm-gemini-key provider)))
-(defun llm-gemini--get-chat-response (response)
- "Get the chat response from RESPONSE."
- ;; Response is a series of the form "text: <some text>\n", which we will
concatenate.
- (mapconcat (lambda (x) (read (substring-no-properties (string-trim x) 8)))
(split-string response "\n" t "\\s*") ""))
+(cl-defmethod llm-provider-utils-populate-function-calls ((_ llm-gemini)
prompt calls)
+ (llm-provider-utils-append-to-prompt
+ prompt
+ ;; For Vertex there is just going to be one call
+ (mapcar (lambda (fc)
+ `((functionCall
+ .
+ ((name . ,(llm-provider-utils-function-call-name fc))
+ (args . ,(llm-provider-utils-function-call-args fc))))))
+ calls)))
+
+(defun llm-gemini--chat-request (prompt)
+ "Return the chat request for PROMPT."
+ (mapcar (lambda (c) (if (eq (car c) 'generation_config)
+ (cons 'generationConfig (cdr c))
+ c))
+ (llm-vertex--chat-request prompt)))
(cl-defmethod llm-chat ((provider llm-gemini) prompt)
- (let ((response (llm-vertex--get-chat-response-streaming
- (llm-request-sync (llm-gemini--chat-url provider nil)
- :data (llm-vertex--chat-request-streaming
prompt)))))
- (setf (llm-chat-prompt-interactions prompt)
- (append (llm-chat-prompt-interactions prompt)
- (list (make-llm-chat-prompt-interaction :role 'assistant
:content response))))
- response))
+ (llm-vertex--process-and-return
+ provider prompt
+ (llm-request-sync (llm-gemini--chat-url provider nil)
+ :data (llm-gemini--chat-request prompt))))
+
+(cl-defmethod llm-chat-async ((provider llm-gemini) prompt response-callback
error-callback)
+ (let ((buf (current-buffer)))
+ (llm-request-async (llm-gemini--chat-url provider nil)
+ :data (llm-gemini--chat-request prompt)
+ :on-success (lambda (data)
+ (llm-request-callback-in-buffer
+ buf response-callback
+ (llm-vertex--process-and-return
+ provider prompt
+ data)))
+ :on-error (lambda (_ data)
+ (llm-request-callback-in-buffer buf
error-callback 'error
+
(llm-vertex--error-message data))))))
(cl-defmethod llm-chat-streaming ((provider llm-gemini) prompt
partial-callback response-callback error-callback)
(let ((buf (current-buffer)))
(llm-request-async (llm-gemini--chat-url provider t)
- :data (llm-vertex--chat-request-streaming prompt)
+ :data (llm-gemini--chat-request prompt)
:on-partial (lambda (partial)
(when-let ((response
(llm-vertex--get-partial-chat-response partial)))
- (llm-request-callback-in-buffer buf
partial-callback response)))
+ (when (> (length response) 0)
+ (llm-request-callback-in-buffer buf
partial-callback response))))
:on-success (lambda (data)
- (let ((response
(llm-vertex--get-chat-response-streaming data)))
- (setf (llm-chat-prompt-interactions
prompt)
- (append
(llm-chat-prompt-interactions prompt)
- (list
(make-llm-chat-prompt-interaction :role 'assistant :content response))))
- (llm-request-callback-in-buffer buf
response-callback response)))
+ (llm-request-callback-in-buffer
+ buf response-callback
+ (llm-vertex--process-and-return
+ provider prompt data)))
:on-error (lambda (_ data)
(llm-request-callback-in-buffer buf
error-callback 'error
(llm-vertex--error-message data))))))
@@ -121,7 +146,8 @@ If STREAMING-P is non-nil, use the streaming endpoint."
(cl-defmethod llm-count-tokens ((provider llm-gemini) string)
(llm-vertex--handle-response
(llm-request-sync (llm-gemini--count-token-url provider)
- :data (llm-vertex--to-count-token-request
(llm-vertex--chat-request-streaming (llm-make-simple-chat-prompt string))))
+ :data (llm-vertex--to-count-token-request
+ (llm-vertex--chat-request
(llm-make-simple-chat-prompt string))))
#'llm-vertex--count-tokens-extract-response))
(cl-defmethod llm-name ((_ llm-gemini))
@@ -132,6 +158,9 @@ If STREAMING-P is non-nil, use the streaming endpoint."
(cl-defmethod llm-chat-token-limit ((provider llm-gemini))
(llm-vertex--chat-token-limit (llm-gemini-chat-model provider)))
+(cl-defmethod llm-capabilities ((_ llm-gemini))
+ (list 'streaming 'embeddings 'function-calls))
+
(provide 'llm-gemini)
;;; llm-gemini.el ends here
diff --git a/llm-llamacpp.el b/llm-llamacpp.el
index 1863fef67d..a1952b12ba 100644
--- a/llm-llamacpp.el
+++ b/llm-llamacpp.el
@@ -187,5 +187,9 @@ them from 1 to however many are sent.")
;; CPP itself.
"Llama CPP")
+(cl-defmethod llm-capabilities ((_ llm-llamacpp))
+ "Return the capabilities of llama.cpp."
+ (list 'streaming 'embeddings))
+
(provide 'llm-llamacpp)
;;; llm-llamacpp.el ends here
diff --git a/llm-ollama.el b/llm-ollama.el
index de30e04158..0ebe0dd2b6 100644
--- a/llm-ollama.el
+++ b/llm-ollama.el
@@ -215,6 +215,9 @@ PROVIDER is the llm-ollama provider to use."
(cl-defmethod llm-chat-token-limit ((provider llm-ollama))
(llm-provider-utils-model-token-limit (llm-ollama-chat-model provider)))
+(cl-defmethod llm-capabilities ((_ llm-ollama))
+ (list 'streaming 'embeddings))
+
(provide 'llm-ollama)
;;; llm-ollama.el ends here
diff --git a/llm-openai.el b/llm-openai.el
index 15e84f9223..5927bf60b4 100644
--- a/llm-openai.el
+++ b/llm-openai.el
@@ -134,40 +134,94 @@ This is just the key, if it exists."
:data (llm-openai--embedding-request
(llm-openai-embedding-model provider) string))
#'llm-openai--embedding-extract-response))
-(defun llm-openai--chat-request (model prompt &optional return-json-spec
streaming)
+(defun llm-openai--chat-request (model prompt &optional streaming)
"From PROMPT, create the chat request data to send.
MODEL is the model name to use.
-RETURN-JSON-SPEC is the optional specification for the JSON to return.
+FUNCTIONS is a list of functions to call, or nil if none.
STREAMING if non-nil, turn on response streaming."
(let (request-alist)
(llm-provider-utils-combine-to-system-prompt prompt
llm-openai-example-prelude)
(when streaming (push `("stream" . ,t) request-alist))
- (push `("messages" . ,(mapcar (lambda (p)
- `(("role" . ,(pcase
(llm-chat-prompt-interaction-role p)
- ('user "user")
- ('system "system")
- ('assistant "assistant")))
- ("content" . ,(string-trim
(llm-chat-prompt-interaction-content p)))))
- (llm-chat-prompt-interactions prompt)))
+ (push `("messages" .
+ ,(mapcar (lambda (p)
+ (append
+ `(("role" . ,(llm-chat-prompt-interaction-role p))
+ ("content" . ,(let ((content
+
(llm-chat-prompt-interaction-content p)))
+ (if (stringp content) content
+ (json-encode content)))))
+ (when-let ((fc
(llm-chat-prompt-interaction-function-call-result p)))
+ (append
+ (when (llm-chat-prompt-function-call-result-call-id
fc)
+ `(("tool_call_id" .
+ ,(llm-chat-prompt-function-call-result-call-id
fc))))
+ `(("name" .
,(llm-chat-prompt-function-call-result-function-name fc)))))))
+ (llm-chat-prompt-interactions prompt)))
request-alist)
(push `("model" . ,(or model "gpt-3.5-turbo-0613")) request-alist)
(when (llm-chat-prompt-temperature prompt)
(push `("temperature" . ,(/ (llm-chat-prompt-temperature prompt) 2.0))
request-alist))
(when (llm-chat-prompt-max-tokens prompt)
(push `("max_tokens" . ,(llm-chat-prompt-max-tokens prompt))
request-alist))
- (when return-json-spec
- (push `("functions" . ((("name" . "output")
- ("parameters" . ,return-json-spec))))
- request-alist)
- (push '("function_call" . (("name" . "output"))) request-alist))
+ (when (llm-chat-prompt-functions prompt)
+ (push `("tools" . ,(mapcar #'llm-provider-utils-openai-function-spec
+ (llm-chat-prompt-functions prompt)))
+ request-alist))
request-alist))
(defun llm-openai--extract-chat-response (response)
"Return chat response from server RESPONSE."
- (let ((result (cdr (assoc 'content (cdr (assoc 'message (aref (cdr (assoc
'choices response)) 0))))))
- (func-result (cdr (assoc 'arguments (cdr (assoc 'function_call (cdr
(assoc 'message (aref (cdr (assoc 'choices response)) 0)))))))))
+ (let ((result (cdr (assoc 'content
+ (cdr (assoc
+ 'message
+ (aref (cdr (assoc 'choices response)) 0))))))
+ (func-result (assoc-default
+ 'tool_calls
+ (assoc-default 'message
+ (aref (assoc-default 'choices response)
0)))))
(or func-result result)))
+(cl-defmethod llm-provider-utils-populate-function-calls ((_ llm-openai)
prompt calls)
+ (llm-provider-utils-append-to-prompt
+ prompt
+ (mapcar (lambda (call)
+ `((id . ,(llm-provider-utils-function-call-id call))
+ (function (name . ,(llm-provider-utils-function-call-name call))
+ (arguments . ,(json-encode
+ (llm-provider-utils-function-call-args
call))))))
+ calls)))
+
+(defun llm-openai--normalize-function-calls (response)
+ "Transform RESPONSE from what Open AI returns to our neutral format."
+ (if (vectorp response)
+ (mapcar (lambda (call)
+ (let ((function (cl-third call)))
+ (make-llm-provider-utils-function-call
+ :id (assoc-default 'id call)
+ :name (assoc-default 'name function)
+ :args (json-read-from-string (assoc-default 'arguments
function)))))
+ response)
+ response))
+
+(defun llm-openai--process-and-return (provider prompt response &optional
error-callback)
+ "Process RESPONSE from the PROVIDER.
+
+This function adds the response to the prompt, executes any
+functions, and returns the value that the client should get back.
+
+PROMPT is the prompt that needs to be updated with the response."
+ (if (and (consp response) (cdr (assoc 'error response)))
+ (progn
+ (when error-callback
+ (funcall error-callback 'error (llm-openai--error-message response)))
+ response)
+ ;; When it isn't an error
+ (llm-provider-utils-process-result
+ provider prompt
+ (llm-openai--normalize-function-calls
+ (if (consp response) (llm-openai--extract-chat-response response)
+ (llm-openai--get-partial-chat-response response))))))
+
(cl-defmethod llm-chat-async ((provider llm-openai) prompt response-callback
error-callback)
(llm-openai--check-key provider)
(let ((buf (current-buffer)))
@@ -175,11 +229,10 @@ STREAMING if non-nil, turn on response streaming."
:headers (llm-openai--headers provider)
:data (llm-openai--chat-request (llm-openai-chat-model provider) prompt)
:on-success (lambda (data)
- (let ((response (llm-openai--extract-chat-response data)))
- (setf (llm-chat-prompt-interactions prompt)
- (append (llm-chat-prompt-interactions prompt)
- (list (make-llm-chat-prompt-interaction
:role 'assistant :content response))))
- (llm-request-callback-in-buffer buf response-callback
response)))
+ (llm-request-callback-in-buffer
+ buf response-callback
+ (llm-openai--process-and-return
+ provider prompt data error-callback)))
:on-error (lambda (_ data)
(let ((errdata (cdr (assoc 'error data))))
(llm-request-callback-in-buffer buf error-callback 'error
@@ -189,16 +242,13 @@ STREAMING if non-nil, turn on response streaming."
(cl-defmethod llm-chat ((provider llm-openai) prompt)
(llm-openai--check-key provider)
- (let ((response (llm-openai--handle-response
- (llm-request-sync (llm-openai--url provider
"chat/completions")
- :headers (llm-openai--headers provider)
- :data (llm-openai--chat-request
(llm-openai-chat-model provider)
- prompt))
- #'llm-openai--extract-chat-response)))
- (setf (llm-chat-prompt-interactions prompt)
- (append (llm-chat-prompt-interactions prompt)
- (list (make-llm-chat-prompt-interaction :role 'assistant
:content response))))
- response))
+ (llm-openai--process-and-return
+ provider prompt
+ (llm-request-sync
+ (llm-openai--url provider "chat/completions")
+ :headers (llm-openai--headers provider)
+ :data (llm-openai--chat-request (llm-openai-chat-model provider)
+ prompt))))
(defvar-local llm-openai-current-response ""
"The response so far from the server.")
@@ -222,48 +272,96 @@ them from 1 to however many are sent.")
nil t)
(line-end-position)))))
(when end-pos
- (let ((all-lines (seq-filter
- (lambda (line) (string-match-p complete-rx line))
- (split-string (buffer-substring-no-properties 1
end-pos) "\n"))))
- (setq current-response
- (concat current-response
- (mapconcat (lambda (line)
- (assoc-default 'content
- (assoc-default
- 'delta
- (aref (assoc-default
- 'choices
-
(json-read-from-string
-
(replace-regexp-in-string "data: " "" line)))
- 0))))
- (seq-subseq all-lines last-response) "")))
+ (let* ((all-lines (seq-filter
+ (lambda (line) (string-match-p complete-rx line))
+ (split-string (buffer-substring-no-properties 1
end-pos) "\n")))
+ (processed-lines
+ (mapcar (lambda (line)
+ (let ((delta (assoc-default
+ 'delta
+ (aref (assoc-default
+ 'choices
+ (json-read-from-string
+ (replace-regexp-in-string
"data: " "" line)))
+ 0))))
+ (or (assoc-default 'content delta)
+ (assoc-default 'tool_calls delta))))
+ (seq-subseq all-lines last-response))))
+ (if (stringp (car processed-lines))
+ ;; The data is a string - a normal response, which we just
+ ;; append to current-response (assuming it's also a string,
+ ;; which it should be).
+ (setq current-response
+ (concat current-response (string-join processed-lines
"")))
+ ;; If this is a streaming function call, current-response will be
+ ;; a vector of function plists, containing the function name and
the arguments
+ ;; as JSON.
+ (when (equal "" current-response)
+ (setq current-response (make-vector (length (car
processed-lines))
+ nil)))
+ (cl-loop for calls in processed-lines do
+ (cl-loop for call in (append calls nil) do
+ (let* ((index (assoc-default 'index call))
+ (plist (aref current-response index))
+ (function (assoc-default 'function
call))
+ (name (assoc-default 'name function))
+ (id (assoc-default 'id call))
+ (arguments (assoc-default 'arguments
function)))
+ (when name (setq plist (plist-put plist
:name name)))
+ (when id (setq plist (plist-put plist :id
id)))
+ (setq plist (plist-put plist :arguments
+ (concat (plist-get
plist :arguments)
+ arguments)))
+ (aset current-response index plist)))))
+
(setq last-response (length all-lines))))))
- (when (> (length current-response) (length llm-openai-current-response))
+ ;; Has to be >= because when we store plists the length doesn't change, but
+ ;; we still want to store the new response. For text, it should indeed be
+ ;; ever-growing (but sometimes it shrinks and we don't want to store that).
+ (when (>= (length current-response) (length llm-openai-current-response))
(setq llm-openai-current-response current-response)
(setq llm-openai-last-response last-response))
- current-response))
-
-(cl-defmethod llm-chat-streaming ((provider llm-openai) prompt
partial-callback response-callback error-callback)
+ ;; If we are dealing with function calling, massage it to look like the
+ ;; normal function calling output.
+ (if (vectorp current-response)
+ (apply #'vector
+ (mapcar (lambda (plist)
+ `((id . ,(plist-get plist :id))
+ (type . function)
+ (function
+ .
+ ((name . ,(plist-get plist :name))
+ (arguments . ,(plist-get plist :arguments))))))
+ current-response))
+ current-response)))
+
+(cl-defmethod llm-chat-streaming ((provider llm-openai) prompt partial-callback
+ response-callback error-callback)
(llm-openai--check-key provider)
(let ((buf (current-buffer)))
(llm-request-async (llm-openai--url provider "chat/completions")
:headers (llm-openai--headers provider)
- :data (llm-openai--chat-request (llm-openai-chat-model
provider) prompt nil t)
+ :data (llm-openai--chat-request (llm-openai-chat-model
provider) prompt t)
:on-error (lambda (_ data)
(let ((errdata (cdr (assoc 'error data))))
- (llm-request-callback-in-buffer buf
error-callback 'error
- (format "Problem calling Open
AI: %s message: %s"
- (cdr (assoc 'type
errdata))
- (cdr (assoc 'message
errdata))))))
+ (llm-request-callback-in-buffer
+ buf error-callback 'error
+ (format "Problem calling Open AI: %s
message: %s"
+ (cdr (assoc 'type errdata))
+ (cdr (assoc 'message
errdata))))))
:on-partial (lambda (data)
(when-let ((response
(llm-openai--get-partial-chat-response data)))
- (llm-request-callback-in-buffer buf
partial-callback response)))
+ ;; We only send partial text updates,
not
+ ;; updates related to function calls.
+ (when (stringp response)
+ (llm-request-callback-in-buffer buf
partial-callback response))))
:on-success-raw (lambda (data)
- (let ((response
(llm-openai--get-partial-chat-response data)))
- (setf (llm-chat-prompt-interactions
prompt)
- (append
(llm-chat-prompt-interactions prompt)
- (list
(make-llm-chat-prompt-interaction :role 'assistant :content response))))
- (llm-request-callback-in-buffer buf
response-callback response))))))
+ (llm-request-callback-in-buffer
+ buf
+ response-callback
+ (llm-openai--process-and-return
+ provider prompt
+ data error-callback))))))
(cl-defmethod llm-name ((_ llm-openai))
"Open AI")
@@ -289,6 +387,12 @@ them from 1 to however many are sent.")
4096)
(t 4096))))
+(cl-defmethod llm-capabilities ((_ llm-openai))
+ (list 'streaming 'embeddings 'function-calls))
+
+(cl-defmethod llm-capabilities ((_ llm-openai-compatible))
+ (list 'streaming 'embeddings))
+
(provide 'llm-openai)
;;; llm-openai.el ends here
diff --git a/llm-provider-utils.el b/llm-provider-utils.el
index 82ab4bfb83..4eb6009d2f 100644
--- a/llm-provider-utils.el
+++ b/llm-provider-utils.el
@@ -115,5 +115,163 @@ things. Providers should probably issue a warning when
using this."
((string-match-p "llama" model) 2048)
((string-match-p "starcoder" model) 8192))))
+(defun llm-provider-utils-openai-arguments (args)
+ "Convert ARGS to the Open AI function calling spec.
+ARGS is a list of `llm-function-arg' structs."
+ (let ((required (mapcar
+ #'llm-function-arg-name
+ (seq-filter #'llm-function-arg-required args))))
+ (append
+ `((type . object)
+ (properties
+ .
+ ,(mapcar (lambda (arg)
+ `(,(llm-function-arg-name arg) .
+ ,(if (and (listp (llm-function-arg-type arg))
+ (llm-function-arg-p (car (llm-function-arg-type
arg))))
+ (llm-provider-utils-openai-arguments
(llm-function-arg-type arg))
+ (append
+ `((type .
+ ,(pcase (llm-function-arg-type arg)
+ ('string 'string)
+ ('integer 'integer)
+ ('float 'number)
+ ('boolean 'boolean)
+ ((cl-type cons)
+ (pcase (car (llm-function-arg-type arg))
+ ('or (cdr (llm-function-arg-type arg)))
+ ('list 'array)
+ ('enum 'string)))
+ (_ (error "Unknown argument type: %s"
(llm-function-arg-type arg))))))
+ (when (llm-function-arg-description arg)
+ `((description
+ .
+ ,(llm-function-arg-description arg))))
+ (when (and (eq 'cons
+ (type-of (llm-function-arg-type arg))))
+ (pcase (car (llm-function-arg-type arg))
+ ('enum `((enum
+ .
+ ,(cdr (llm-function-arg-type arg)))))
+ ('list
+ `((items .
+ ,(if (llm-function-arg-p
+ (cadr (llm-function-arg-type
arg)))
+
(llm-provider-utils-openai-arguments
+ (cdr (llm-function-arg-type arg)))
+ `((type . ,(cadr
(llm-function-arg-type arg))))))))))))))
+ args)))
+ (when required
+ `((required . ,required))))))
+
+;; The Open AI function calling spec follows the JSON schema spec.
+;; See https://json-schema.org/understanding-json-schema.
+(defun llm-provider-utils-openai-function-spec (call)
+ "Convert `llm-function-call' CALL to an Open AI function spec.
+Open AI's function spec is a standard way to do this, and will be
+applicable to many endpoints.
+
+This returns a JSON object (a list that can be converted to JSON)."
+ `((type . function)
+ (function
+ .
+ ,(append
+ `((name . ,(llm-function-call-name call))
+ (description . ,(llm-function-call-description call)))
+ (when (llm-function-call-args call)
+ `((parameters
+ .
+ ,(llm-provider-utils-openai-arguments (llm-function-call-args
call)))))))))
+
+(defun llm-provider-utils-append-to-prompt (prompt output &optional
func-results role)
+ "Append OUTPUT to PROMPT as an assistant interaction.
+
+OUTPUT can be a string or a structure in the case of function calls.
+
+ROLE will be `assistant' by default, but can be passed in for other roles."
+ (setf (llm-chat-prompt-interactions prompt)
+ (append (llm-chat-prompt-interactions prompt)
+ (list (make-llm-chat-prompt-interaction
+ :role (if func-results
+ 'function
+ (or role 'assistant))
+ :content output
+ :function-call-result func-results)))))
+
+(cl-defstruct llm-provider-utils-function-call
+ "A struct to hold information about a function call.
+ID is a call ID, which is optional.
+NAME is the function name.
+ARG is an alist of arguments to values."
+ id name args)
+
+(cl-defgeneric llm-provider-utils-populate-function-calls (provider prompt
calls)
+ "For PROVIDER, in PROMPT, record that function CALLS were received.
+This is the recording before the calls were executed.
+CALLS are a list of `llm-provider-utils-function-call'."
+ (ignore provider prompt calls)
+ (signal 'not-implemented nil))
+
+(defun llm-provider-utils-populate-function-results (prompt func result)
+ "Append the RESULT of FUNC to PROMPT.
+FUNC is a `llm-provider-utils-function-call' struct."
+ (llm-provider-utils-append-to-prompt
+ prompt result (make-llm-chat-prompt-function-call-result
+ :call-id (llm-provider-utils-function-call-id func)
+ :function-name (llm-provider-utils-function-call-name func)
+ :result result)))
+
+(defun llm-provider-utils-process-result (provider prompt response)
+ "From RESPONSE, execute function call.
+
+RESPONSE is either a string or list of
+`llm-provider-utils-function-calls'.
+
+This should be called with any response that might have function
+calls. If the response is a string, nothing will happen, but in
+either case, the response suitable for returning to the client
+will be returned.
+
+PROVIDER is the provider that supplied the response.
+
+PROMPT was the prompt given to the provider, which will get
+updated with the response from the LLM, and if there is a
+function call, the result.
+
+This returns the response suitable for output to the client; a
+cons of functions called and their output."
+ (if (consp response)
+ (progn
+ ;; Then this must be a function call, return the cons of a the funcion
+ ;; called and the result.
+ (llm-provider-utils-populate-function-calls provider prompt response)
+ (cl-loop for func in response collect
+ (let* ((name (llm-provider-utils-function-call-name
func))
+ (arguments
(llm-provider-utils-function-call-args func))
+ (function (seq-find
+ (lambda (f) (equal name
(llm-function-call-name f)))
+ (llm-chat-prompt-functions prompt))))
+ (cons name
+ (let* ((args (cl-loop for arg in
(llm-function-call-args function)
+ collect (cdr (seq-find
(lambda (a)
+
(eq (intern
+
(llm-function-arg-name arg))
+
(car a)))
+
arguments))))
+ (result (apply
(llm-function-call-function function) args)))
+ (llm-provider-utils-populate-function-results
+ prompt func result)
+ (llm--log
+ 'api-funcall
+ :provider provider
+ :msg (format "%s --> %s"
+ (format "%S"
+ (cons
(llm-function-call-name function)
+ args))
+ (format "%s" result)))
+ result)))))
+ (llm-provider-utils-append-to-prompt prompt response)
+ response))
+
(provide 'llm-provider-utils)
;;; llm-provider-utils.el ends here
diff --git a/llm-request.el b/llm-request.el
index fdb471a853..949e91e0a9 100644
--- a/llm-request.el
+++ b/llm-request.el
@@ -25,7 +25,7 @@
(require 'url-http)
(require 'rx)
-(defcustom llm-request-timeout 20
+(defcustom llm-request-timeout 60
"The number of seconds to wait for a response from a HTTP server.
Request timings are depending on the request. Requests that need
@@ -170,10 +170,12 @@ the buffer is turned into JSON and passed to ON-SUCCESS."
;; to make callbacks.
(defun llm-request-callback-in-buffer (buf f &rest args)
"Run F with ARSG in the context of BUF.
-But if BUF has been killed, use a temporary buffer instead."
- (if (buffer-live-p buf)
- (with-current-buffer buf (apply f args))
- (with-temp-buffer (apply f args))))
+But if BUF has been killed, use a temporary buffer instead.
+If F is nil, nothing is done."
+ (when f
+ (if (buffer-live-p buf)
+ (with-current-buffer buf (apply f args))
+ (with-temp-buffer (apply f args)))))
(provide 'llm-request)
;;; llm-request.el ends here
diff --git a/llm-tester.el b/llm-tester.el
index b895f89b3a..a66e883f01 100644
--- a/llm-tester.el
+++ b/llm-tester.el
@@ -213,6 +213,97 @@
(message "SUCCESS: Provider %s provided a conversation with
responses %s" (type-of provider) (buffer-string))
(kill-buffer buf))))))))))
+(defun llm-tester-create-test-function-prompt ()
+ "Create a function to test function calling with."
+ (make-llm-chat-prompt
+ :context "The user will describe an emacs lisp function they
are looking
+for, and you need to provide the most likely function you know
+of by calling the `describe_function' function."
+ :interactions (list (make-llm-chat-prompt-interaction
+ :role 'user
+ :content "I'm looking for a function
that will return the current buffer's file name."))
+ :temperature 0.1
+ :functions
+ (list (make-llm-function-call
+ :function (lambda (f) f)
+ :name "describe_function"
+ :description "Takes an elisp function name and shows
the user the functions and their descriptions."
+ :args (list (make-llm-function-arg
+ :name "function_name"
+ :description "A function name to
describe."
+ :type 'string
+ :required t))))))
+
+(defun llm-tester-function-calling-sync (provider)
+ "Test that PROVIDER can call functions."
+ (let ((prompt (llm-tester-create-test-function-prompt)))
+ (message "SUCCESS: Provider %s called a function and got result %s"
+ (type-of provider)
+ (llm-chat provider prompt))))
+
+(defun llm-tester-function-calling-conversation-sync (provider)
+ "Test that PROVIDER can call functions in a conversation."
+ (let ((prompt (llm-tester-create-test-function-prompt))
+ (responses nil))
+ (push (llm-chat provider prompt) responses)
+ ;; The expectation (a requirement for Gemini) is we call back into the LLM
+ ;; with the results of the previous call to get a text response based on
the
+ ;; function call results.
+ (push (llm-chat provider prompt) responses)
+ (llm-chat-prompt-append-response prompt "I'm now looking for a function
that will return the directory of a filename")
+ (push (llm-chat provider prompt) responses)
+ (push (llm-chat provider prompt) responses)
+ (message "SUCCESS: Provider %s had a function conversation and got results
%s"
+ (type-of provider)
+ (nreverse responses))))
+
+(defun llm-tester-function-calling-async (provider)
+ "Test that PROVIDER can call functions asynchronously."
+ (let ((prompt (llm-tester-create-test-function-prompt)))
+ (llm-chat-async provider prompt
+ (lambda (result)
+ (message "SUCCESS: Provider %s called a function and got
a result of %s"
+ (type-of provider) result))
+ (lambda (type message)
+ (message "ERROR: Provider %s returned an error of type
%s with message %s"
+ (type-of provider) type message)))))
+
+(defun llm-tester-function-calling-conversation-async (provider)
+ "Test that PROVIDER can call functions in a conversation."
+ (let* ((prompt (llm-tester-create-test-function-prompt))
+ (responses nil)
+ (error-callback (lambda (type msg) (message "FAILURE: async function
calling conversation for %s, error of type %s received: %s" (type-of provider)
type msg)))
+ (last-callback (lambda (result)
+ (push result responses)
+ (message "SUCCESS: Provider %s had an async function
calling conversation, and got results %s"
+ (type-of provider)
+ (nreverse responses))))
+ (third-callback (lambda (result) (push result responses)
+ (llm-chat-async provider prompt last-callback
error-callback)))
+ (second-callback (lambda (result) (push result responses)
+ (llm-chat-prompt-append-response prompt "I'm now
looking for a function that will return the directory of a filename.")
+ (llm-chat-async provider prompt third-callback
error-callback)))
+ (first-callback (lambda (result) (push result responses)
+ (llm-chat-async provider prompt second-callback
error-callback))))
+ (llm-chat-async provider prompt first-callback error-callback)))
+
+(defun llm-tester-function-calling-streaming (provider)
+ "Test that PROVIDER can call functions with the streaming API."
+ (let ((partial-counts 0))
+ (llm-chat-streaming
+ provider
+ (llm-tester-create-test-function-prompt)
+ (lambda (_)
+ (cl-incf partial-counts))
+ (lambda (text)
+ (message "SUCCESS: Provider %s called a function and got a final result
of %s"
+ (type-of provider) text)
+ (unless (= 0 partial-counts)
+ (message "WARNING: Provider %s returned partial updates, but it
shouldn't for function calling" (type-of provider))))
+ (lambda (type message)
+ (message "ERROR: Provider %s returned an error of type %s with message
%s"
+ (type-of provider) type message)))))
+
(defun llm-tester-cancel (provider)
"Test that PROVIDER can do async calls which can be cancelled."
(message "Testing provider %s for cancellation" (type-of provider))
diff --git a/llm-vertex.el b/llm-vertex.el
index 45857e150d..8c9d6c0575 100644
--- a/llm-vertex.el
+++ b/llm-vertex.el
@@ -112,13 +112,13 @@ KEY-GENTIME keeps track of when the key was generated,
because the key must be r
(defun llm-vertex--error-message (err-response)
"Return a user-visible error message from ERR-RESPONSE."
- (let ((err (assoc-default 'error (aref err-response 0))))
+ (let ((err (assoc-default 'error err-response)))
(format "Problem calling GCloud Vertex AI: status: %s message: %s"
(assoc-default 'code err)
(assoc-default 'message err))))
(defun llm-vertex--handle-response (response extractor)
- "If RESPONSE is an error, throw it, else call EXTRACTOR."
+ "If RESPONSE is an errorp, throw it, else call EXTRACTOR."
(if (assoc 'error response)
(error (llm-vertex--error-message response))
(funcall extractor response)))
@@ -145,19 +145,28 @@ KEY-GENTIME keeps track of when the key was generated,
because the key must be r
:data `(("instances" . [(("content" . ,string))])))
#'llm-vertex--embedding-extract-response))
-(defun llm-vertex--get-chat-response-streaming (response)
+(defun llm-vertex--get-chat-response (response)
"Return the actual response from the RESPONSE struct returned.
This handles different kinds of models."
(pcase (type-of response)
- ('vector (mapconcat #'llm-vertex--get-chat-response-streaming
- response ""))
+ ('vector (when (> (length response) 0)
+ (let ((parts (mapcar #'llm-vertex--get-chat-response response)))
+ (if (stringp (car parts))
+ (mapconcat #'identity parts "")
+ (car parts)))))
('cons (if (assoc-default 'candidates response)
(let ((parts (assoc-default
'parts
(assoc-default 'content
(aref (assoc-default 'candidates
response) 0)))))
(if parts
- (assoc-default 'text (aref parts 0))
+ (or (assoc-default 'text (aref parts 0))
+ ;; Change function calling from almost Open AI's
+ ;; standard format to exactly the format.
+ (mapcar (lambda (call)
+ `(function . ,(mapcar (lambda (c) (if (eq
(car c) 'args) (cons 'arguments (cdr c)) c))
+ (cdar call))))
+ parts))
""))
"NOTE: No response was sent back by the LLM, the prompt may have
violated safety checks."))))
@@ -174,22 +183,49 @@ This handles different kinds of models."
(setq result (concat result (json-read-from-string (match-string
1))))))
result)))
-(defun llm-vertex--chat-request-streaming (prompt)
+(defun llm-vertex--chat-request (prompt)
"Return an alist with chat input for the streaming API.
PROMPT contains the input to the call to the chat API."
(llm-provider-utils-combine-to-user-prompt prompt llm-vertex-example-prelude)
(append
- `((contents
- .
- ,(mapcar (lambda (interaction)
- `((role . ,(pcase (llm-chat-prompt-interaction-role
interaction)
- ('user "user")
- ('assistant "model")))
- (parts .
- ((text . ,(llm-chat-prompt-interaction-content
- interaction))))))
- (llm-chat-prompt-interactions prompt))))
- (llm-vertex--chat-parameters prompt)))
+ `((contents
+ .
+ ,(mapcar (lambda (interaction)
+ `((role . ,(pcase (llm-chat-prompt-interaction-role
interaction)
+ ('user "user")
+ ('assistant "model")
+ ('function "function")))
+ (parts .
+ ,(if (and (not (equal
(llm-chat-prompt-interaction-role interaction)
+ 'function))
+ (stringp
(llm-chat-prompt-interaction-content interaction)))
+ `(((text . ,(llm-chat-prompt-interaction-content
+ interaction))))
+ (if (eq 'function
+ (llm-chat-prompt-interaction-role
interaction))
+ (let ((fc
(llm-chat-prompt-interaction-function-call-result interaction)))
+ `(((functionResponse
+ .
+ ((name .
,(llm-chat-prompt-function-call-result-function-name fc))
+ (response
+ .
+ ((name .
,(llm-chat-prompt-function-call-result-function-name fc))
+ (content .
,(llm-chat-prompt-function-call-result-result fc)))))))))
+ (llm-chat-prompt-interaction-content
interaction))))))
+ (llm-chat-prompt-interactions prompt))))
+ (when (llm-chat-prompt-functions prompt)
+ ;; Although Gemini claims to be compatible with Open AI's function
declaration,
+ ;; it's only somewhat compatible.
+ `(("tools" .
+ ,(mapcar (lambda (tool)
+ `((function_declarations . (((name .
,(llm-function-call-name tool))
+ (description .
,(llm-function-call-description tool))
+ (parameters
+ .
+
,(llm-provider-utils-openai-arguments
+ (llm-function-call-args
tool))))))))
+ (llm-chat-prompt-functions prompt)))))
+ (llm-vertex--chat-parameters prompt)))
(defun llm-vertex--chat-parameters (prompt)
"From PROMPT, create the parameters section.
@@ -204,44 +240,102 @@ nothing to add, in which case it is nil."
(when params-alist
`((generation_config . ,params-alist)))))
-(defun llm-vertex--chat-url (provider)
+(defun llm-vertex--normalize-function-calls (response)
+ "If RESPONSE has function calls, transform them to our common format."
+ (if (consp response)
+ (mapcar (lambda (f)
+ (make-llm-provider-utils-function-call
+ :name (assoc-default 'name (cdr f))
+ :args (assoc-default 'arguments (cdr f))))
+ response)
+ response))
+
+(cl-defmethod llm-provider-utils-populate-function-calls ((_ llm-vertex)
prompt calls)
+ (llm-provider-utils-append-to-prompt
+ prompt
+ ;; For Vertex there is just going to be one call
+ (mapcar (lambda (fc)
+ `((functionCall
+ .
+ ((name . ,(llm-provider-utils-function-call-name fc))
+ (args . ,(llm-provider-utils-function-call-args fc))))))
+ calls)))
+
+(defun llm-vertex--process-and-return (provider prompt response &optional
error-callback)
+ "Process RESPONSE from the PROVIDER.
+
+This returns the response to be given to the client.
+
+Any functions will be executed.
+
+The response will be added to PROMPT.
+
+Provider is the llm provider, for logging purposes.
+
+ERROR-CALLBACK is called when an error is detected."
+ (if (and (consp response)
+ (assoc-default 'error response))
+ (progn
+ (when error-callback
+ (funcall error-callback 'error (llm-vertex--error-message response)))
+ response))
+ (let ((return-val
+ (llm-provider-utils-process-result
+ provider prompt
+ (llm-vertex--normalize-function-calls
+ (llm-vertex--get-chat-response response)))))
+ return-val))
+
+(defun llm-vertex--chat-url (provider &optional streaming)
"Return the correct url to use for PROVIDER.
If STREAMING is non-nil, use the URL for the streaming API."
- (format
"https://%s-aiplatform.googleapis.com/v1/projects/%s/locations/%s/publishers/google/models/%s:streamGenerateContent";
+ (format
"https://%s-aiplatform.googleapis.com/v1/projects/%s/locations/%s/publishers/google/models/%s:%s";
llm-vertex-gcloud-region
(llm-vertex-project provider)
llm-vertex-gcloud-region
- (llm-vertex-chat-model provider)))
+ (llm-vertex-chat-model provider)
+ (if streaming "streamGenerateContent" "generateContent")))
;; API reference:
https://cloud.google.com/vertex-ai/docs/generative-ai/multimodal/send-chat-prompts-gemini#gemini-chat-samples-drest
-
(cl-defmethod llm-chat ((provider llm-vertex) prompt)
;; Gemini just has a streaming response, but we can just call it
synchronously.
(llm-vertex-refresh-key provider)
- (let ((response (llm-vertex--get-chat-response-streaming
- (llm-request-sync (llm-vertex--chat-url provider)
- :headers `(("Authorization" . ,(format
"Bearer %s" (llm-vertex-key provider))))
- :data (llm-vertex--chat-request-streaming
prompt)))))
- (setf (llm-chat-prompt-interactions prompt)
- (append (llm-chat-prompt-interactions prompt)
- (list (make-llm-chat-prompt-interaction :role 'assistant
:content response))))
- response))
+ (llm-vertex--process-and-return
+ provider prompt
+ (llm-request-sync (llm-vertex--chat-url provider)
+ :headers `(("Authorization" . ,(format "Bearer %s"
(llm-vertex-key provider))))
+ :data (llm-vertex--chat-request prompt))))
+
+(cl-defmethod llm-chat-async ((provider llm-vertex) prompt response-callback
error-callback)
+ (llm-vertex-refresh-key provider)
+ (let ((buf (current-buffer)))
+ (llm-request-async (llm-vertex--chat-url provider)
+ :headers `(("Authorization" . ,(format "Bearer %s"
(llm-vertex-key provider))))
+ :data (llm-vertex--chat-request prompt)
+ :on-success (lambda (data)
+ (llm-request-callback-in-buffer
+ buf response-callback
+ (llm-vertex--process-and-return
+ provider prompt data)))
+ :on-error (lambda (_ data)
+ (llm-request-callback-in-buffer buf
error-callback 'error
+
(llm-vertex--error-message data))))))
(cl-defmethod llm-chat-streaming ((provider llm-vertex) prompt
partial-callback response-callback error-callback)
(llm-vertex-refresh-key provider)
(let ((buf (current-buffer)))
(llm-request-async (llm-vertex--chat-url provider)
:headers `(("Authorization" . ,(format "Bearer %s"
(llm-vertex-key provider))))
- :data (llm-vertex--chat-request-streaming prompt)
+ :data (llm-vertex--chat-request prompt)
:on-partial (lambda (partial)
(when-let ((response
(llm-vertex--get-partial-chat-response partial)))
- (llm-request-callback-in-buffer buf
partial-callback response)))
+ (when (> (length response) 0)
+ (llm-request-callback-in-buffer buf
partial-callback response))))
:on-success (lambda (data)
- (let ((response
(llm-vertex--get-chat-response-streaming data)))
- (setf (llm-chat-prompt-interactions
prompt)
- (append
(llm-chat-prompt-interactions prompt)
- (list
(make-llm-chat-prompt-interaction :role 'assistant :content response))))
- (llm-request-callback-in-buffer buf
response-callback response)))
+ (llm-request-callback-in-buffer
+ buf response-callback
+ (llm-vertex--process-and-return
+ provider prompt data)))
:on-error (lambda (_ data)
(llm-request-callback-in-buffer buf
error-callback 'error
(llm-vertex--error-message data))))))
@@ -274,7 +368,7 @@ MODEL "
(llm-request-sync (llm-vertex--count-token-url provider)
:headers `(("Authorization" . ,(format "Bearer %s"
(llm-vertex-key provider))))
:data (llm-vertex--to-count-token-request
- (llm-vertex--chat-request-streaming
+ (llm-vertex--chat-request
(llm-make-simple-chat-prompt string))))
#'llm-vertex--count-tokens-extract-response))
@@ -293,6 +387,9 @@ MODEL "
(cl-defmethod llm-chat-token-limit ((provider llm-vertex))
(llm-vertex--chat-token-limit (llm-vertex-chat-model provider)))
+(cl-defmethod llm-capabilities ((_ llm-vertex))
+ (list 'streaming 'embeddings 'function-calls))
+
(provide 'llm-vertex)
;;; llm-vertex.el ends here
diff --git a/llm.el b/llm.el
index 11fc6c6b14..7b265e15d9 100644
--- a/llm.el
+++ b/llm.el
@@ -91,18 +91,74 @@ function, for continuing chats, the whole prompt MUST be a
variable passed in to the chat function. INTERACTIONS is
required.
+FUNCTIONS is a list of `llm-function-call' structs. These may be
+called IF the LLM supports them. If the LLM does not support
+them, a `not-implemented' signal will be thrown. This is
+optional. When this is given, the LLM will either call the
+function or return text as normal, depending on what the LLM
+decides.
+
TEMPERATURE is a floating point number with a minimum of 0, and
maximum of 1, which controls how predictable the result is, with
0 being the most predicatable, and 1 being the most creative.
This is not required.
MAX-TOKENS is the maximum number of tokens to generate. This is optional."
- context examples interactions temperature max-tokens)
+ context examples interactions functions temperature max-tokens)
(cl-defstruct llm-chat-prompt-interaction
"This defines a single interaction given as part of a chat prompt.
-ROLE can a symbol, of either `user' or `assistant'."
- role content)
+ROLE can a symbol, of either `user', `assistant', or `function'.
+
+FUNCTION-CALL-RESULTS is a struct of type
+`llm-chat-prompt-function-call-results', which is only populated
+if `role' is `function'. It stores the results of just one
+function call."
+ role content function-call-result)
+
+(cl-defstruct llm-chat-prompt-function-call-result
+ "This defines the result from a function call.
+
+CALL-ID is an ID for this function call, if available.
+
+FUNCTION-NAME is the name of the function. This is required.
+
+RESULT is the result of the function call. This is required."
+ call-id function-name result)
+
+(cl-defstruct llm-function-call
+ "This is a struct to represent a function call the LLM can make.
+
+FUNCTION is a function to call.
+
+NAME is a human readable name of the function.
+
+DESCRIPTION is a human readable description of the function.
+
+ARGS is a list of `llm-function-arg' structs. "
+ function
+ name
+ description
+ args)
+
+(cl-defstruct llm-function-arg
+ "An argument to an `llm-function-call'.
+
+NAME is the name of the argument.
+
+DESCRIPTION is a human readable description of the argument. It
+can be nil for enums.
+
+TYPE is the type of the argument. It can be one of `string',
+`integer', `float', `boolean' or the special lists, `(or <type1>
+<type2> ... <typen>)', `(enum <string1> <string2> ...
+<stringn>)', or `(list <type>)'.
+
+REQUIRED is whether this is required or not."
+ name
+ description
+ type
+ required)
(cl-defun llm--log (type &key provider prompt msg)
"Log a MSG of TYPE, given PROVIDER, PROMPT, and MSG.
@@ -130,6 +186,7 @@ this variable in this library. TYPE can be one of
`api-send',
(format "[%s --> Emacs]: %s"
(llm-name provider) msg))
('api-error "[Error]: %s" msg)
+ ('api-funcall "[%s execution] %s" msg)
('prompt-append (format "[Append to conversation]:
%s"
msg)))))))))
@@ -161,7 +218,12 @@ need to override it."
(cl-defgeneric llm-chat (provider prompt)
"Return a response to PROMPT from PROVIDER.
-PROMPT is a `llm-chat-prompt'. The response is a string response by the LLM.
+PROMPT is a `llm-chat-prompt'.
+
+The response is a string response by the LLM when functions are
+not called. If functions are called, the response is a list of
+conses of the function named called (as a symbol), and the
+corresponding result from calling it.
The prompt's interactions list will be updated to encode the
conversation so far."
@@ -185,11 +247,18 @@ conversation so far."
(let* ((llm-log nil)
(result (cl-call-next-method))
(llm-log t))
- (llm--log 'api-receive :provider provider :msg result)
+ (when (stringp result)
+ (llm--log 'api-receive :provider provider :msg result))
result))
(cl-defgeneric llm-chat-async (provider prompt response-callback
error-callback)
- "Return a response to PROMPT from PROVIDER.
+ "Call RESPONSE-CALLBACK with a response to PROMPT from PROVIDER.
+
+The response is a string response by the LLM when functions are
+not called. If functions are called, the response is a list of
+conses of the function named called (as a symbol), and the
+corresponding result from calling it.
+
PROMPT is a `llm-chat-prompt'.
RESPONSE-CALLBACK receives the final text.
@@ -230,10 +299,18 @@ be passed to `llm-cancel-request'."
result))
+(cl-defmethod llm-chat-function-call ((_ (eql nil)) _ _ _)
+ (error "LLM provider was nil. Please set the provider in the application
you are using."))
+
(cl-defgeneric llm-chat-streaming (provider prompt partial-callback
response-callback error-callback)
"Stream a response to PROMPT from PROVIDER.
PROMPT is a `llm-chat-prompt'.
+The response is a string response by the LLM when functions are
+not called. If functions are called, the response is a list of
+conses of the function named called (as a symbol), and the
+corresponding result from calling it.
+
PARTIAL-CALLBACK is called with the output of the string response
as it is built up. The callback is called with the entire
response that has been received, as it is streamed back. It is
@@ -326,6 +403,25 @@ be passed to `llm-cancel-request'."
(when-let (info (llm-nonfree-message-info provider))
(llm--warn-on-nonfree (car info) (cdr info))))
+(cl-defgeneric llm-capabilities (provider)
+ "Return a list of the capabilities of PROVIDER.
+
+This possible values are only those things that are not the bare
+minimum of functionality to be included in this package, which is
+non-streaming chat:
+
+`streaming': the LLM can actually stream responses in the
+ streaming call. Calls to `llm-chat-streaming' will work
+ regardless even if the LLM doesn't support streaming, it just
+ won't have any partial responses, so basically just operates
+ like `llm-chat-async'.
+
+`embeddings': the LLM can return vector embeddings of text.
+
+`function-calls': the LLM can call functions."
+ (ignore provider)
+ nil)
+
(cl-defgeneric llm-chat-token-limit (provider)
"Return max number of tokens that can be sent to the LLM.
For many models we know this number, but for some we don't have
diff --git a/utilities/elisp-to-function-call.el
b/utilities/elisp-to-function-call.el
new file mode 100644
index 0000000000..361f178a6b
--- /dev/null
+++ b/utilities/elisp-to-function-call.el
@@ -0,0 +1,186 @@
+;;; elisp-to-function-call --- Utility for converting elisp to function call
-*- lexical-binding: t; -*-
+
+;; Copyright (c) 2024 Free Software Foundation, Inc.
+
+;; Author: Andrew Hyatt <ahyatt@gmail.com>
+;; SPDX-License-Identifier: GPL-3.0-or-later
+;;
+;; This program is free software; you can redistribute it and/or
+;; modify it under the terms of the GNU General Public License as
+;; published by the Free Software Foundation; either version 3 of the
+;; License, or (at your option) any later version.
+;;
+;; This program is distributed in the hope that it will be useful, but
+;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+;; General Public License for more details.
+;;
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>.
+
+;;; Commentary:
+;; This is a utility class for clients of the llm library. It is used to make
+;; writing function calls for existing elisp code automated, through the use of
+;; function calling. We use a function call to take in a elisp function,
+;; something with documentation, and we add a function call to where the
+;; function was called from.
+
+(require 'llm)
+(require 'rx)
+(require 'cl-extra)
+
+(defvar elisp-to-function-call-provider nil
+ "The LLM provider to use for this. Must support function calls.")
+
+;; An example of the output - you can remove the function call definition and
+;; call `elisp-to-function-call-insert' to see this in action.
+(defconst elisp-to-function-call-switch-to-buffer
+ (make-llm-function-call :function
+ 'switch-to-buffer
+ :name
+ "switch_to_buffer"
+ :args
+ (list (make-llm-function-arg :name
+ "buffer_or_name"
+ :type
+ 'string
+ :description
+ "A buffer, a string
(buffer name), or nil."
+ :required
+ t)
+ (make-llm-function-arg :name
+ "norecord"
+ :type
+ 'boolean
+ :description
+ "If non-nil, do not put
the buffer at the front of the buffer list and do not make the window
displaying it the most recently selected one."
+ :required
+ t)
+ (make-llm-function-arg :name
+ "force_same_window"
+ :type
+ 'boolean
+ :description
+ "If non-nil, the buffer
must be displayed in the selected window when called non-interactively; if that
is impossible, signal an error rather than calling ‘pop_to_buffer’."
+ :required
+ t))
+ :description
+ "Display buffer BUFFER_OR_NAME in the selected
window. If the selected window cannot display the specified buffer because it
is a minibuffer window or strongly dedicated to another buffer, call
‘pop_to_buffer’ to select the buffer in another window. If called
interactively, read the buffer name using ‘read_buffer’. The variable
‘confirm_nonexistent_file_or_buffer’ determines whether to request confirmation
before creating a new buffer. See ‘read_buffer’ for feat [...]
+
+ )
+
+;; A demonstration of the resulting function call in action.
+(defun elisp-to-function-call-llm-switch-buffer (instructions)
+ "Send INSTRUCTIONS to the LLM so that it siwtches the buffer.
+It will call `elisp-to-function-call-provider.', and will pass
+the available buffers in the prompt."
+ (interactive "sInstructions: ")
+ (llm-chat-async elisp-to-function-call-provider
+ (make-llm-chat-prompt
+ :context (format "The user wishes to switch to a buffer.
The available buffers to switch to are: %s. Please call the switch_to_buffer
function and make your best guess at what which of the buffers the user wants,
or a new buffer if that is appropriate."
+ (json-encode
+ (seq-filter (lambda (s) (not
(string-match "^\s" s)))
+ (mapcar #'buffer-name
(buffer-list)))))
+ :interactions (list (make-llm-chat-prompt-interaction
+ :role 'user
+ :content instructions))
+ :functions (list elisp-to-function-call-switch-to-buffer))
+ (lambda (_)) ;; Nothing to do, the switch already happened.
+ (lambda (_ msg) (error msg))))
+
+(defun elisp-to-function-call-el-to-js-name (name)
+ "Convert NAME to a JavaScript name."
+ (replace-regexp-in-string (rx (seq (group-n 1 alpha) ?- (group-n 2 alpha)))
+ "\\1_\\2" name))
+
+(defun elisp-to-function-call-insert (f)
+ "For non-anonymous function F, insert a function spec for LLMs.
+The definition is for a `llm-function-call'.
+
+What comes out should be close to correct, but it may need some
+manual intervention.
+
+The function spec here makes Gemini error out, perhaps because it
+uses more nested function specs. This may go away eventually as
+Gemini improves."
+ (interactive "aFunction: ")
+ (let ((marker (point-marker))
+ (arglist (help-function-arglist f)))
+ (llm-chat-async elisp-to-function-call-provider
+ (make-llm-chat-prompt
+ :context "The user wants to get the data to transform an
emacs lisp
+function to a function usable in a OpenAI-compatible function call. The user
will
+provide the function name and documentation. Break that down into the
documentation
+of the function, and the argument types and descriptions for those arguments.
+
+Use lowercase for all argument names even if you see it in uppercase in the
documentation.
+Documentation strings should start with uppercase and end with a period."
+ :interactions (list (make-llm-chat-prompt-interaction
+ :role 'user
+ :content (format "Function:
%s\nArguments: %s\nDescription: %s"
+
(elisp-to-function-call-el-to-js-name (symbol-name f))
+ (if arglist
+ (format "%s"
arglist)
+ "No arguments")
+
(elisp-to-function-call-el-to-js-name
+ (documentation
f)))))
+ :functions
+ (list
+ (make-llm-function-call
+ :function (lambda (args description)
+ (with-current-buffer (marker-buffer marker)
+ (save-excursion
+ (goto-char marker)
+ (cl-prettyprint
+ `(make-llm-function-call
+ :function ,(list 'quote f)
+ :name
,(elisp-to-function-call-el-to-js-name (symbol-name f))
+ :args (list
+ ,@(mapcar (lambda (arg)
+
`(make-llm-function-arg
+ ,@(append
+ (list
+ :name
(downcase (elisp-to-function-call-el-to-js-name
+
(assoc-default 'name arg)))
+ :type (list
'quote (read (assoc-default 'type arg)))
+
:description (assoc-default 'description arg))
+ (if
(assoc-default 'required arg)
+ (list
:required t)))))
+ args))
+ :description ,description)))))
+ :name "elisp-to-function-info"
+ :description "The function to create a
OpenAI-compatible function call spec, given the arguments and their
documentation. Some of the aspects of the function call can be automatically
retrieved, so this function is supplying the parts that cannot be automatically
retrieved."
+ :args (list
+ (make-llm-function-arg
+ :name "args"
+ :type `(list
+ ,(make-llm-function-arg
+ :name "name"
+ :type 'string
+ :description "The name of the
argument"
+ :required t)
+ ,(make-llm-function-arg
+ :name "type"
+ :type '(enum string number integer
boolean "(list string)" "(list integer)" "(list number)")
+ :description "The type of the
argument. It could be 'string', 'number', 'integer', 'boolean', or the more
special forms.
+(list string) is for a list of strings, (list integer), etc."
+ :required t)
+ ,(make-llm-function-arg
+ :name "description"
+ :type 'string
+ :description "The description of the
argument"
+ :required t)
+ ,(make-llm-function-arg
+ :name "required"
+ :type 'boolean
+ :description "Whether the argument is
required or not"
+ :required t))
+ :description "The arguments of the function to
transform, in order.")
+ (make-llm-function-arg
+ :name "description"
+ :type 'string
+ :description "The documentation of the function
to transform.")))))
+ (lambda (result) (message "Result: %S" result))
+ (lambda (_ msg) (error msg)))))
+
+(provide 'elisp-to-function-call)
- [elpa] externals/llm updated (7b112f0b42 -> 9c33eb4a91), ELPA Syncer, 2024/03/02
- [elpa] externals/llm 782b892de2 1/5: Add function calling, and introduce llm-capabilities,
ELPA Syncer <=
- [elpa] externals/llm 4abca6c3f1 2/5: Fix inadvertent logging issue, ELPA Syncer, 2024/03/02
- [elpa] externals/llm 9c33eb4a91 5/5: Bump version to 0.10.0, ELPA Syncer, 2024/03/02
- [elpa] externals/llm 4b53eda1fd 4/5: In llm-tester, don't error on not streaming if it isn't supported, ELPA Syncer, 2024/03/02
- [elpa] externals/llm 041de98b3f 3/5: Improvements to llm-tester, dedicated output, respect capabilities, ELPA Syncer, 2024/03/02