Re: [Gzz] A vob system based on structured graphics?

[Tuomas Lukka]

On Sun, Mar 30, 2003 at 02:35:46PM +0200, Benja Fallenstein wrote:
> 
> after a bit of discussion of IRC which only confused the others, here's 
> a first writeup of a possible re-write of the vob system I've been 
> intently thinking about for the last three days. This is inspired by 
> Fresco, which manages to be quite simple yet really powerful by relying 
> on separating functionality in structured graphics/widgets into simple, 
> orthogonal components.

After reading the rest, my overall impression is that what you describe
would be best implemented as a layer above the vob system, with some small
api support changes.

> This is not completely thought-out; it's just a start. I hope to spark 
> discussion. *If* this system is viable, though, I hope that it can 
> simplify the vob system very much, both for its users and its developers.

What I'm afraid is that the simplicity will force far greater
performance hits than the current system has. You'll need to convince
me it doesn't ;)

> In this system, a vob is a graphic of something, like a character, a 
> raster image, a window with background and border, a multiply nested 
> structured graphic, etc. We don't have vob scenes any more; rather, we 
> have different top-level vobs.

This is still similar to the vobscene recursion stuff I suggested.

However, this already seems to me to give away some flexibility: one
of the nicest features of the vob system with OpenGL is that you can
set the rendering state:

        vs.map.put(GLCache.getCallList("""
                PushAttrib ...
                ... OpenGL state commans
        """))

        ... render a bunch of stuff

        vs.map.put(GLCache.getCallList("""
                PopAttrib ...
        """))

How do you plan to achieve this?

> Vobs can give information like their size, position relative to a 
> baseline, etc. 

As shown above, this is hardly appropriate for *all* vobs.

> (They can of course be scaled by putting them into an 
> appropriate transform.)

What about nonlinear transformations?

> There are primitive vobs like rectangle or filled oval, there are vobs 
> that take one other vob and modify it somehow, and there are composite 
> vobs which take more than one vob and make these vobs into a single vob 
> somehow. 

This I don't disagree with - this I think is an idea definitely
worth looking into

> Vobs are transformed by modifier vobs, like Translate, Rotate 
> or Fisheye vobs. Composite vobs include HBox and VBox (as in TeX) and 
> Overlay, which takes a list of vobs and renders them above each other.

The problem here is that the current system gives you a lot of flexibility
by 

        1) allowing vobs to have more than one coordinate system; for instance,
           irregular quad has the contents and the edge described separately
        2) allowing coordinate systems to be defined by several others - 
           e.g. halfway between two points in different coordsyses you 
           could have a label.

You seem to be going back to a tree model which I don't like.

> Positioning a number of objects in different places on the screen, as 
> our structure views do, can be handled by nesting each object in a 
> Translate vob and putting the Translate vobs into an Overlay. 

What's an Overlay?

> This is 
> equivalent to some functionality VobScene has now.

Except that you need a java object for each coordsys, which can be too much.

> However, this would only be used in places where a view really places 
> objects at x/y coordinates on the screen. For example, when an object is 
> placed into a box-- with a background and a border around it-- we're not 
> really talking about placing things at x/y coordinates, though the 
> existing vob system would handle it that way (we would place the border, 
> then we'd place the content at a translation of (5,5) inside the border, 
> for example).
> 
> In the new system, we could use: a WrapperVob, which renders three vobs 
> (background, content, border); and a MarginVob, which puts a margin 
> around another vob. Then we could write something like,
> 
>     v = WrapperVob(bg, MarginVob(content, 5), border);
> 
> This is already a more structured approach, but we could make this into 
> something along the lines of,
> 
>     box = WrapperVob(bg, MarginVob(null, 5), border)
>     v = box.cloneContentReplaced(content)
> 
> I.e., using the prototype design pattern like for ColoredVobs, we could 
> have different types of frames as first-order objects.
> 
> The point is that this is MUCH closer to what the view actually wants to 
> think about.

Yes, *this* part of the idea I like. However, it still requires too many
Java objects: creating a new wrappervob for each use does not sound good.
Of course, most of the wrappervobs would probably just contain
two fields: the real wrapping vob and the wrapped vob... That might 
be the angle to start from.

I wonder if we could come up with a system that would work both ways...


> Vobs are more or less immutable. They are immutable in that you cannot 
> change the structure of a vob-- you cannot add or remove vobs inside it. 
> On the other hand, you may modify the parameters of a transformation 
> vob, to accomodate for continuous user actions like dragging.

URRRRRRRRGH! 

This is really bad, OTOH.

Here you get all the disadvantages of both mutable and immutable objects.
You can't change them as you like but you can't trust them to remain the
same either.

> Each vob can be seen as being made of a tree vob sub-vobs (for primitive 
> vobs, the tree contains only that vob). Each vob in this tree has a 
> number. That is, when you have an Overlay vob containing Transform vobs 
> which contain RectVobs, then each of these vobs has an index inside the 
> Overlay vob. The Overlay vob itself has index 0.
>
> - Leaf vobs contain just themselves, at index 0.
> - Modificator vobs contain themselves, at index 0, and then the vob they 
> modify and the vobs contained by that vob, at indices 1+.
> - Container vobs like Overlay contain themselves, then the first vob in 
> them with all the vobs contained by it, then the second vob with all the 
> vobs contained in it, and so on.
> 
> Simple arithmetic can be used: If vob U has index i in vob V, and vob V 
> has index j in vob W, then vob U has index i+j in vob W.
> 
> These indices can also be seen as a 'path' into a vob. Vobs support 
> getVob(index) and getTransform(index) methods, where getTransform(index) 
> gives the cumulative transformation from the root vob down to that vob 
> in the tree. For example, if the vob is wrapped in a Translation which 
> is wrapped in a Rotation, getTransform() would return a transform 
> concatenating those other two transforms. [getTransform() does not 
> return a vob, but a Transform object.]

Interesting idea, which forces the immutability...

> Each vob (except the root) has one parent vob and inherits one parent 
> transform from this vob. Given that we see each vob as a graphic, this 
> makes sense. (Note: A vob doesn't know its parent; rather, the parent 
> transform is passed to the render() method of the child vob. This allows 
> us to use flyweighting, i.e. use the same vob in different places.)

But we can't use it if, as you said above, the transforms are not fixed
inside a vob.

> Now, consider a connection. What's its parent, from a structured 
> graphics perspective? Surely not the things it connects, since it is not 
> inside them. From this perspective, it makes more sense if a view places 
> the connections into the same Overlay object as the RDF nodes it 
> connects. A connection can then be subjected to the usual structured 
> graphics transformations-- color, rotation, scaling, whatever.

This does make sense. This is something we don't have right now.

> The points it connects can be seen as *parametrizing* the connection-- 
> i.e., we create a line, but instead of telling it, "you are from (10,70) 
> to (44, 58)," we tell it, "you are from the right side of this vob to 
> the left side of that vob."

This we do have already.

> The problem with this is that the connection doesn't know its parent, 
> and thus if it is placed inside the same Overlay as the vobs it 
> connects, it has no way to actually look at that vob and see what it is 
> interpolated to and compute the current interpolation. One possibility 
> would be to wrap the Overlay inside the connection vob: Then we could 
> give the connection vob the indices of the vobs it connects, inside the 
> Overlay vob. Then the connection vob would know about the vobs it 
> connects and could retrieve their interpolation. Unfortunately, this 
> approach is rather ugly.
> 
> This area of the system is what I'm currently chewing on.

Do tell us when you swallow ;)  

> One important benefit from this system is caching. Since vobs are 
> essentially immutable and do not know about their parents, you can put 
> any vob into a cache.

Umm, they aren't as you said above.

> We may need one efficiency hack here. For the simple structural RDF 
> views, we would like to avoid having to create a new object per node by 
> putting the node vob into a cache. However, with the approach above, we 
> would still need to create at least a new Translation vob for every 
> node-- except if we place the node in exactly the same place as before 
> (not very useful).
> 
> To avoid this, as an efficiency hack, we may have a vob that is a mix of 
> Overlay and Translation: it would render a set of vobs and store a 
> translation (or even linear transform) for each. This wouldn't be as 
> clean as separating the concerns into two different vobs, but the 
> efficiency gains should justify it. (We can still have arbitrary 
> transforms by using the appropriate vobs, e.g. place a node into a 
> FisheyeVob).

Unkh... no. This favors one type of transformation way too much. I want
this either for all transformations or none.

> If we use caching, we could have a PrerenderVob that uses idle processor 
> time to prerender its contents. (If it hasn't had time to prerender 
> itself, during rendering it would simply render its children the normal 
> way.)

This works only for certain vobs. Some vobs (e.g. text) will be using
better methods to render themselves w.r.t. the pixels.

> Interpolation is handled by two modifier vobs: KeyVob and KeyPlane. 
> KeyVob assigns a key to its content. KeyPlane interprets the KeyVobs it 
> contains directly or indirectly. For a simple structural RDF view, you'd
> 
> - put each node into a KeyVob
> - put the KeyVobs into an Overlay, transformed
> - put the Overlay into a KeyPlane
> 
> When a KeyPlane is to be interpolated to another KeyPlane, it matches 
> the KeyVobs it contains against the KeyVobs the other KeyPlane contains. 
> The KeyVobs can be arbitrarily deep inside the KeyPlane's tree. The 
> KeyPlane takes into account all KeyVobs it contains directly or 
> indirectly, except that it doesn't take into account KeyVobs contained 
> in another KeyPlane inside the first KeyPlane. Thus we can e.g. use a 
> KeyPlane for the windows on a screen, each of which has its own KeyVob, 
> and use another KeyPlane in each window for matching the individual RDF 
> nodes.

More objects? 

> During interpolation, all vobs are rendered where they would be rendered 
> outside interpolaion, except KeyVobs and the vobs contained in them. 
> (KeyVobs which have no matching equivalent in the target KeyPlane are 
> omitted from the interpolation, as usual, but vobs not wrapped in 
> KeyVobs are shown.) This means for example that we do not need to assign 
> keys to the contents of an RDF node, just to the node as a whole; the 
> contents of the node will just "jump" when we switch vob 'scenes' if 
> they are not assigned keys, following the rule above. Also, things like 
> the 'current view' indicator we've been placing in the upper right 
> corner of each window doesn't need a key; if it's not in the target vob 
> 'scene,' it will simply disappear when we "jump."
> 
> This gets us rid of a LOT of keys-- all the infamous 'role keys.' We 
> only assign keys to things that actually correspond to objects in the 
> underlying model.

This we could do with the current system, I think also.

But also has problems: when scrolling, new nodes would appear too early
at their final locations - not good.

> Vobs like IrregularFrame would be done through parametrization, like 
> connections: IrregularFrame would (in its constructor) get two 
> transformations (Transform objects) 

Uhh, but this goes the wrong way around -- weren't you supposed to construct
the transformations *AROUND* the transformed objects? That's what
the above stuff was mostly based on...

        TUomas