Re: The future: accessing vectors, arrays, etc from C

[Neil Jerram]

Marius Vollmer wrote:
> The proposed leasing interface doesn't try to solve the problem that I
> am adressing now ... [it] addresses the situation where you want to access
> a Scheme vector as a vector of doubles, and want to avoid to making a
> copy of the elements in the case the Scheme vector is a uniform
> numeric vector already.

Yes, I see that now - thanks for explaining.  (There is a minor 
similarity, in that both leases and your handles protect the underlying 
storage from GC, but that's about it.)

> A change in representation of an array might happen when the copy for
> a copy-on-write array takes place, or when the first 16-bit Unicode
> character is stored in a 8-bit string.

Just to check understanding: is it correct that:

- all such changes have to be driven from Scheme code?

- such changes do not affect the arrays obtained by C code through the 
scm_*_array_handle_*_elements functions?

>  In general, when we can come
> up with a C API to arrays that is convenient and allows such changes
> to happen, we should install it.  Once we can change represetation, we
> can implement a lot of cool things (whether they are a good idea or
> not), like growing arrays in place.

Agreed, subject to general concerns about API change which we discuss below.

> It will be more tedious to use than the old one mainly because it is
> more general, something which has nothing to do with allowing
> representations to change: When using the new API, your code must be
> able to deal with non-contiguously stored vectors, while previously
> you could just assume that all vectors are stored contiguously.

This worried me when I first read it, but in fact I think the only 
increase in tediousness is that the increment from one element to the 
next may not always be 1.  But it will always be some constant.  Is that 
right, or am I missing some other kinds of tediousness? :-)

I think this delta is OK, personally.  And in practice many applications 
 will be able to get away with asserting that the increment is 1, by 
simplying avoiding array mapping in more than one dimension.

> > Actually, to be precise, I'm not sure I'm bothered about the level
> > of change per se, as I believe the current reality is that a Guile
> > application writer has no choice but to select an available major
> > version of Guile (e.g. 1.6) and to target their application at
> > specifically that version; and fortunately the bugs in Guile are few
> > enough that it is feasible to stick with an older version.
>
>
> Yes, you will be missing all the cool new features, tho.

Yes, of course.  Also, in my view, the whole point of Guile is to 
minimize your C code and maximize your Scheme code; for anyone who 
agrees with this, the cost of upgrading to a new version (which is 
roughly proportional to the amount of C code) should be dwarfed by the 
benefit (roughly proportional to the amount of Scheme code).

Even so, I think we could usefully make a public statement (perhaps by 
adding it to the manual) reflecting the apparent consensus from this 
discussion, namely...

1. We do _not_ regard it as an overriding aim to maintain C API 
compatibility between Guile releases (that is, between versions w.x and 
y.z where ((w != y) || (x != z))); we believe it is more important to 
improve Guile's overall utility, which sometimes conflicts with 
retaining API compatibility.

2. Consequently, the developer of a non-trivial Guile application must 
in practice choose a particular Guile release and write their C code 
according to that release's C API, and should expect _some_ work to be 
required when enhancing their application to target a new Guile release.

3. On the other hand, we will obviously not pointlessly change the C API 
between releases, for at least four pragmatic reasons.

   - We don't want to annoy our users for no reason!

   - Any API change itself requires work by the core Guile developers.

   - The core Guile code uses its own APIs, so any unnecessary changes 
would require unnecessary work in Guile's own internals.

   - Most of the core Guile developers also write Guile applications, 
so any unnecessary changes would require unnecessary work in those 
applications as well.

4. Consequently the upgrade work mentioned in (2), while non-zero, 
should always be as small as it could reasonably be.

5. Where possible, we will support the old C API, for the next one or 
more releases, as a "deprecated" interface.  Guile's deprecation 
infrastructure both highlights to1	 the developer that they are using a 
deprecated interface, and gives them a one-line hint as to what they 
should be doing instead.

6. We will aim to document API changes completely.  The first point of 
reference for this is usually the NEWS file in each new release.  Such 
documentation may be imperfect in practice, but only for the same 
reasons that make documentation in general imperfect.

> What I am worried about is that I have not done the deprecation of the
> old interfaces as good as it could be done.
>
> For example, instead of the old SCM_VECTORP, SCM_VECTOR_REF and
> SCM_VECTOR_SET, you now should use scm_is_simple_vector,
> SCM_SIMPLE_VECTOR_REF, and SCM_SIMPLE_VECTOR_SET.  That might appear
> to be arbitrary, since only names have changed.
>
> On the other hand, this name change makes it clear that only the
> special case of simple vectors is being dealt with, and people might
> be motivated to look up in the manual how to deal with the general
> case.

Sounds good and sufficiently motivated to me.  I presume SCM_VECTORP, 
SCM_VECTOR_REF and SCM_VECTOR_SET will remain available as deprecated 
macros, won't they?

Regards,
        Neil