Re: [Chicken-users] Optional sqlite3 egg update

[Thomas Chust]

On Tue, 7 Mar 2006, JörgF. Wittenberger wrote:

> [...]
> Would you mind to share your experiences about inference with user level
> threading?
>
> Background: for Askemos (www.askemos.org a kind of p2p web server with
> byzantine shared state of web apps) I wrote a sqlite3 binding to
> rscheme.  Now rscheme and chicken are quite simillar wrt. threading.
> Therefore we've been very careful never to call any procedure of unbound
> execution time from the scheme thread. [...]
>
> The sqlite3 binding of mine (which is rough in comparision the your egg)
> and the egg share this problem: calling sqlite3_step can potentially
> take for ever.  I'm not yet sure, but I feel that since I'm using
> sqlite3 with askemos, it's sometimes suddenly unbearable unresponsive
> and this seems to corellate with sqlite3 calls (probably autocommit).

Hello,

actually I must admit that I haven't used the SQLite3 egg for a real 
multithreaded application so far. Nevertheless I have done some testing, 
which suggests that everything works fine and the program stays responsive 
under normal circumstances because the SQLite3 logic is indeed very fast 
and in case a user defined function runs very long, the CHICKEN scheduler 
does switch to other threads in the meantime.

In order to avoid memory leaks and messing up of the internal SQLite3 
state, I do not allow user defined functions and collation sequences to 
leave their original scope by invocation of continuations or signalling of 
exceptions. This restriction seemed sensible to me and it is documented.

The use of SQL functions and collation sequences writen in Scheme of 
course incurs a large performance penalty in comparison to ones written in 
C, because the C<->Scheme boundary with all the wrapping and unwrapping of 
data and the table lookups of function definitions and seed values have to 
be gone through often.

> Therefore I consider to throw much more C code in and associate a
> pthread per sqlite3 struct, which would perform the actual calls and
> call back to Scheme, when sqlite3 is done.
>
> Do you think this is approach is useful and might be something to do
> with the chicken egg too?  Or would your experience suggest otherwise,
> e.g. sqlite beeing so fast, we would rarely notice and I better go back
> and find out where exactly those sudden delays come from.

Unfortunately I know nothing about the internals of RScheme, so I can't 
really judge whether this is a good idea. I can imagine that it would make 
several optimizations possible, but I can also imagine that in the end the 
interthread communication and synchronization compensates fully for other 
optimizations one could have made.

I definitely think that the scheme would be hard to implement for CHICKEN, 
whose C API is not reentrant. As I am currently not too concerned about 
speed problems with the sqlite3 egg, I will for the moment save myself the 
effort of looking into this possibility.

As to the source of your sudden delays, some research where exactly they 
come from may be useful...

> [...]
> --- sqlite3.scm~        2006-03-06 22:33:56.000000000 +0100
> +++ sqlite3.scm 2006-03-07 10:21:28.000000000 +0100
> @@ -524,7 +524,7 @@
>
> ;; check whether the database is in autocommit mode
> (define (sqlite3:auto-committing? db)
> -  (check-type 'sqlite3:auto-committin? db <sqlite3:database>)
> +  (check-type 'sqlite3:auto-committing? db <sqlite3:database>)
>   ((foreign-lambda bool "sqlite3_get_autocommit" sqlite3:database) db))
>
> ;; get the number of changes made to the database

Thank you for the patch, I applied it to the Subversion repository and 
updated the egg on my homepage.

cu,
Thomas