Re: [Gzz] A vob system based on structured graphics?

[Benja Fallenstein]

Tuomas Lukka wrote:
> On Sun, Mar 30, 2003 at 02:35:46PM +0200, Benja Fallenstein wrote:
> > This is not completely thought-out; it's just a start. I hope to spark 
> > discussion. *If* this system is viable, though, I hope that it can 
> > simplify the vob system very much, both for its users and its developers.
>
> What I'm afraid is that the simplicity will force far greater
> performance hits than the current system has. You'll need to convince
> me it doesn't ;)

I can implement it for AWT. If someone else hooks it up with GL, we can 
benchmark.

> However, this already seems to me to give away some flexibility: one
> of the nicest features of the vob system with OpenGL is that you can
> set the rendering state:
>
>         vs.map.put(GLCache.getCallList("""
>                 PushAttrib ...
>                 ... OpenGL state commans
>         """))
>
>         ... render a bunch of stuff
>
>         vs.map.put(GLCache.getCallList("""
>                 PopAttrib ...
>         """))
>
> How do you plan to achieve this?

With a modifier vob.

public class FooAttribVob extends ModifierVob {
    public int putIntoListGL(...) {
        i = GLCache.getCallList(...push...).putIntoListGL(...);
        i = content.putIntoListGL(...);
        i = GLCache.getCallList(...pop...).putIntoListGL(...);
    }
}

This is not really different from a vob setting e.g. the color of 
another vob.

> > Vobs can give information like their size, position relative to a 
> > baseline, etc. 
>
> As shown above, this is hardly appropriate for *all* vobs.

Modifier vobs like the above inherit the size etc. of the vob they wrap.

> > (They can of course be scaled by putting them into an 
> > appropriate transform.)
>
> What about nonlinear transformations?

This includes nonlinear transforms.

> > Vobs are transformed by modifier vobs, like Translate, Rotate 
> > or Fisheye vobs. Composite vobs include HBox and VBox (as in TeX) and 
> > Overlay, which takes a list of vobs and renders them above each other.
>
> The problem here is that the current system gives you a lot of flexibility
> by 
>
>         1) allowing vobs to have more than one coordinate system; for instance,
>            irregular quad has the contents and the edge described separately
> 	2) allowing coordinate systems to be defined by several others - 
> 	   e.g. halfway between two points in different coordsyses you 
> 	   could have a label.
>
> You seem to be going back to a tree model which I don't like.

Any vob can be *parametrized* by an arbitrary number of transforms. The 
tree is for containment-- a vob in two places in the tree is shown in 
two places on the screen. I explaned below,

>>Vobs like IrregularFrame would be done through parametrization, like
>>connections: IrregularFrame would (in its constructor) get two
>>transformations (Transform objects)
>
> Uhh, but this goes the wrong way around -- weren't you supposed to 
construct
> the transformations *AROUND* the transformed objects? That's what
> the above stuff was mostly based on...

If the object is *graphically transformed*, yes. I.e., the object is a 
graphic, and if you put it into a rotation it is rotated, if you put it 
into a translation it is translated, and so on.

If an object takes two transforms, that already indicates these aren't 
graphical transforms, but parameters to the object. Like with 
connections: when put into two Translation transformations, the 
connection isn't translated-- its endpoints are, but the connection as a 
whole changes its shape in a more complex way. That's why this is a 
parametrization, not putting the connection into two transformations.

> > Positioning a number of objects in different places on the screen, as 
> > our structure views do, can be handled by nesting each object in a 
> > Translate vob and putting the Translate vobs into an Overlay. 
>
> What's an Overlay?

Explained in the previous paragraph:
>>Composite vobs include HBox and VBox (as in TeX) and
>>Overlay, which takes a list of vobs and renders them above each other.

(snip)
> > The point is that this is MUCH closer to what the view actually wants to 
> > think about.
>
>
> Yes, *this* part of the idea I like.

Good.

> However, it still requires too many
> Java objects: creating a new wrappervob for each use does not sound good.
> Of course, most of the wrappervobs would probably just contain
> two fields: the real wrapping vob and the wrapped vob... That might 
> be the angle to start from.
>
> I wonder if we could come up with a system that would work both ways...

Maybe, but I'm not yet convinced that this system will be too slow by 
itself.

> > Vobs are more or less immutable. They are immutable in that you cannot 
> > change the structure of a vob-- you cannot add or remove vobs inside it. 
> > On the other hand, you may modify the parameters of a transformation 
> > vob, to accomodate for continuous user actions like dragging.
>
> URRRRRRRRGH! 
>
> This is really bad, OTOH.
>
> Here you get all the disadvantages of both mutable and immutable objects.
> You can't change them as you like but you can't trust them to remain the
> same either.

You can trust them to remain the same in defined ways (structure) and 
allow to be changed in defined ways (linear parameters).

> > Each vob can be seen as being made of a tree vob sub-vobs (for primitive 
> > vobs, the tree contains only that vob). Each vob in this tree has a 
> > number. That is, when you have an Overlay vob containing Transform vobs 
> > which contain RectVobs, then each of these vobs has an index inside the 
> > Overlay vob. The Overlay vob itself has index 0.
> >
> > - Leaf vobs contain just themselves, at index 0.
> > - Modificator vobs contain themselves, at index 0, and then the vob they 
> > modify and the vobs contained by that vob, at indices 1+.
> > - Container vobs like Overlay contain themselves, then the first vob in 
> > them with all the vobs contained by it, then the second vob with all the 
> > vobs contained in it, and so on.
> >
> > Simple arithmetic can be used: If vob U has index i in vob V, and vob V 
> > has index j in vob W, then vob U has index i+j in vob W.
> >
> > These indices can also be seen as a 'path' into a vob. Vobs support 
> > getVob(index) and getTransform(index) methods, where getTransform(index) 
> > gives the cumulative transformation from the root vob down to that vob 
> > in the tree. For example, if the vob is wrapped in a Translation which 
> > is wrapped in a Rotation, getTransform() would return a transform 
> > concatenating those other two transforms. [getTransform() does not 
> > return a vob, but a Transform object.]
>
> Interesting idea, which forces the immutability...

Of structure. Linear parameter modification is not a problem.

> > Each vob (except the root) has one parent vob and inherits one parent 
> > transform from this vob. Given that we see each vob as a graphic, this 
> > makes sense. (Note: A vob doesn't know its parent; rather, the parent 
> > transform is passed to the render() method of the child vob. This allows 
> > us to use flyweighting, i.e. use the same vob in different places.)
>
> But we can't use it if, as you said above, the transforms are not fixed
> inside a vob.

You can only use linear parameter modification in vobs you rendered 
yourself, obviously.

On the other hand, we could also consider a copy-to-root approach to 
linear modification. Maybe.

> > The points it connects can be seen as *parametrizing* the connection-- 
> > i.e., we create a line, but instead of telling it, "you are from (10,70) 
> > to (44, 58)," we tell it, "you are from the right side of this vob to 
> > the left side of that vob."
>
> This we do have already.

Yes. The difference is that we handle this through containment, not 
parametrizing (passing a parameter to the constructor).

The point is that containment would be purely graphical.

> > The problem with this is that the connection doesn't know its parent, 
> > and thus if it is placed inside the same Overlay as the vobs it 
> > connects, it has no way to actually look at that vob and see what it is 
> > interpolated to and compute the current interpolation. One possibility 
> > would be to wrap the Overlay inside the connection vob: Then we could 
> > give the connection vob the indices of the vobs it connects, inside the 
> > Overlay vob. Then the connection vob would know about the vobs it 
> > connects and could retrieve their interpolation. Unfortunately, this 
> > approach is rather ugly.
> >
> > This area of the system is what I'm currently chewing on.
>
> Do tell us when you swallow ;)  

;-)

> > We may need one efficiency hack here. For the simple structural RDF 
> > views, we would like to avoid having to create a new object per node by 
> > putting the node vob into a cache. However, with the approach above, we 
> > would still need to create at least a new Translation vob for every 
> > node-- except if we place the node in exactly the same place as before 
> > (not very useful).
> >
> > To avoid this, as an efficiency hack, we may have a vob that is a mix of 
> > Overlay and Translation: it would render a set of vobs and store a 
> > translation (or even linear transform) for each. This wouldn't be as 
> > clean as separating the concerns into two different vobs, but the 
> > efficiency gains should justify it. (We can still have arbitrary 
> > transforms by using the appropriate vobs, e.g. place a node into a 
> > FisheyeVob).
>
> Unkh... no. This favors one type of transformation way too much. I want
> this either for all transformations or none.

I don't see why it favors one transformation too much. It is an 
efficiency hack and makes the common case more efficient. Efficiency 
hacks for unusual cases are evil.

> > If we use caching, we could have a PrerenderVob that uses idle processor 
> > time to prerender its contents. (If it hasn't had time to prerender 
> > itself, during rendering it would simply render its children the normal 
> > way.)
>
> This works only for certain vobs. Some vobs (e.g. text) will be using
> better methods to render themselves w.r.t. the pixels.

Not a problem.

> > Interpolation is handled by two modifier vobs: KeyVob and KeyPlane. 
> > KeyVob assigns a key to its content. KeyPlane interprets the KeyVobs it 
> > contains directly or indirectly. For a simple structural RDF view, you'd
> >
> > - put each node into a KeyVob
> > - put the KeyVobs into an Overlay, transformed
> > - put the Overlay into a KeyPlane
> >
> > When a KeyPlane is to be interpolated to another KeyPlane, it matches 
> > the KeyVobs it contains against the KeyVobs the other KeyPlane contains. 
> > The KeyVobs can be arbitrarily deep inside the KeyPlane's tree. The 
> > KeyPlane takes into account all KeyVobs it contains directly or 
> > indirectly, except that it doesn't take into account KeyVobs contained 
> > in another KeyPlane inside the first KeyPlane. Thus we can e.g. use a 
> > KeyPlane for the windows on a screen, each of which has its own KeyVob, 
> > and use another KeyPlane in each window for matching the individual RDF 
> > nodes.
>
> More objects? 

More cachable object, yes.

> > During interpolation, all vobs are rendered where they would be rendered 
> > outside interpolaion, except KeyVobs and the vobs contained in them. 
> > (KeyVobs which have no matching equivalent in the target KeyPlane are 
> > omitted from the interpolation, as usual, but vobs not wrapped in 
> > KeyVobs are shown.) This means for example that we do not need to assign 
> > keys to the contents of an RDF node, just to the node as a whole; the 
> > contents of the node will just "jump" when we switch vob 'scenes' if 
> > they are not assigned keys, following the rule above. Also, things like 
> > the 'current view' indicator we've been placing in the upper right 
> > corner of each window doesn't need a key; if it's not in the target vob 
> > 'scene,' it will simply disappear when we "jump."
> >
> > This gets us rid of a LOT of keys-- all the infamous 'role keys.' We 
> > only assign keys to things that actually correspond to objects in the 
> > underlying model.
>
> This we could do with the current system, I think also.
>
> But also has problems: when scrolling, new nodes would appear too early
> at their final locations - not good.

Not if they are keyed.

>>(KeyVobs which have no matching equivalent in the target KeyPlane are
>>omitted from the interpolation, as usual, but vobs not wrapped in
>>KeyVobs are shown.)

- Benja