[Fenfire-dev] Re: [vos-d] a 3D interface vision

[Toni Alatalo]

On Monday, October 20, 2003, at 08:55 AM, Peter Amstutz wrote:

we've discussed this a bit on irc (both #vos on freenode and #fenfire 
on ircnet, actually, so i'll cc the latter too) and i promised to 
return on the matter here, so here's a try:

> I've started talking to a professor at my university about the 
> possibility
> of applying for a grant to work on VOS.  This would mean that I could 
> be

great to hear - i'm working (also, besides this company) at a 
university too and so are the people in the Fenfire project I've 
mentioned (and will discuss a bit in this post). i think open source 
programming and academic research can go very well together, even 
somewhat practical programming that produces useful tools quickly 
enough :)

> paid to work on VOS full time, and could hire several other programmers
> and artists to help!  One possible angle for funding is research into

that would be quite something. btw we have a game developing research 
group at our university too, and they are a partner in the FirstAndLast 
-project I've mentioned, they just finished an update for their ut2003 
mod, http://ludocraft.oulu.fi/airbuccaneers/ , last week with quite 
nice art (i did ack your comment that Ter'Angreal (the 3d vos client 
made with Crystal Space that has the gecko rendering now, for fenfire 
people to know) aims at being no Quake3 though :) . and Kyperjokki 
(also cc'd) is a group of (sometimes starving :) artists of course .. 
we are also seeking funding for these things.

> human-computer interaction, specifically 3D interfaces, so I wrote up 
> the
> following "vision" document, talking about where I would like to go 
> with
> VOS...

HCI research and using 3d hardware for innovative interfaces has been 
very much in the focus of Fenfire lately (for the past 2 years?), so 
i'm trying to make this connection.. so to the matter (i don't quote in 
full length so ones who have not seen the original should refer to 
http://www.interreality.org/pipermail/vos-d/2003-October/000398.html ):

> We live in a three-dimensional world.  We think in terms of spacial
> relations: that one thing is next to another, what room something is
> in, or how far apart two places are.  For example, there is a mnemonic

interesting that you go to spac-/spatiality (i'm more used to the 
latter spelling, as it is used in the term spatial hypermedia etc. .. 
though they might be also spacial hypermedia, of course), 'cause it has 
been topical re: Fenfire in several ways, of which these come to mind:

1. FenPDF has currently spatial organization on canvases (e.g. 
http://an.org/gradu2/related_work.jpg) (how does it go with 
structlinks?)

2. Sense of space and direction (related to orientation) has been 
discussed when making views to information structures.

The latter is partly related to how ZigZag is locally euclidian and 
perhaps hence creates such sense of space, but RDF (which Fenfire uses) 
is different. this elaborated a bit later here.

(what else?-)

Oh, a general (philosophical?) note about the beginning sentence: you 
may say that we live in a 3d world, but don't we also live in social 
and mental (conceptual) worlds that have quite different structures 
than being points in a xyz coordinate system (although they may map to 
such in the 'wetware implementation', i.e. the brain, but that's 
another discussion i guess :o :) . And as a concrete manifestation of 
the physical 3d world already exists, isn't it perhaps more interesting 
to explore the abstract/conceptual worlds with other kinds of relations 
etc? And of course that's what much of use of computing (e.g. e-mailing 
:) is about.

> trick called "Memory Palace", where practitioner envisions some large
> building in the mind's eye, filling each room of the structure with
> items that remind of one of important things one wants to remember.
> To recall something, one mentally walks back through the palace
> reviewing the contents of each room.  This technique engages our
> visual and spacial cognition; one can often remember the location of
> something based on visual recollections and spacial sense about what
> is near it.  We have a highly developed sense of the placement and
> arrangement of things.

here i was first worried about what could i say or if the building of 
my argument was lost in this convincing example, but luckily the word 
'near' popped out and reminded of the important thing:

i (and 'we' referring to people sharing the Fenfire vision, or anyone 
with common sense perhaps :) agree that something being near another 
thing is very relevant. but, unlike in the physical 3d, nearness need 
not be the distance of two objects in a 3d space, but can be defined 
and dealt with by using other kind of relations, too. i try to approach 
the whole point by discussing this in the following:

first about ZigZag, 'cause it shares the vocabulary of dimensions with 
the 3d vision. ZZ structure can be described as a n-dimensional linked 
list (they say that is for people who already know what it is, so i try 
to think something for people who don't know it already, next :). hm. 
spreadsheet with no empty cells? not good in this context. hm. well, it 
is an n-dimensional space, where in each dimension an object may have 
exactly one other object connected it. this makes it locally euclidean, 
i.e. similar to physical 3d and the simulations of it in the sense that 
for every object there may be one object next to it in one direction. 
but the point is that there may be any other dimensions, whereby any 
other object can be next to the one, too. so, in effect, it is a 
non-euclidean space where objects may be in many places. for example:

let's have dimensions 1, 2 and 3, and axes x, y and z accordinly, and 
objects a,b,c, .. in that space, with relations like this meaning that 
b is next to a in dimension 1 (visualized on x axis): a-b

let's add the placement of c in dimension 2 (on y axis) like this:

a-b
|
c

so far, this is normal euclidean / 2d spatial stuff. but in zigzag, it 
may also be it so that also b and c are next to each other, like this: 
b-c , which results in the following overall structure (hopefully you 
are using a monospaced font):

  a-b-c
  |
b-c

here b and c are suddenly in two places (zigzag clients use 
focus+context views where the placement depends on the current focus), 
as illustrated by this screenshot with two windows with different views 
to the same data/structure: 
http://an.org/gzz/screenshot-gzigzag-abc.png (having the same focus 
here actually)

so the point tries to be that in a virtual/unreal/information/whatever 
world nearness need not be only what it is in the physical/3d worlds.

this did not even demo the n-dimensionality but perhaps was enough for 
now anyhow :)
more info and examples in http://xanadu.com/zigzag/

then on to Fenfire, which uses RDF. unlike ZigZag, RDF allows any 
statements about anything, so even this is possible:

a-b-c

a-c-b

hm. so? i guess the thing is that i don't know what happens with 
spatiality.. will need to let this 'stew' a bit, on to other things:

> Computers today barely take advantage of our spacial sense.  The

(do see spatial hypermedia for exceptions)

> so-called "desktop metaphor" used by modern graphical user interfaces
> (GUIs) breaks down more often than it applies.  The desktop is a

agreed :)

> transient space; applications are started and ended, icons are added,
> shuffled around, and taken away. There is little the user can do to
> create long-lasting information organization using spacial proximity
> (in the literal sense of being graphically closer) or visual 
> prominence.

one technique utilized in Fenfire Libvob (the UI / graphics toolkit) is 
interpolation, i.e. animating the changes in scenes. this is derived 
from what the same group prototyped already with gzigzag, see e.g. the 
previous example animated as 
http://an.org/gzz/screencap-gzigzag-abc.anim.gif and 
http://www.nongnu.org/fenfire/shots.html for videos made with (some 
time ago) with the current system.

>  Hierarchical filesystems are the prevailing method for
> storing and organizing information, but they are not particularly
> intuitive and describing nested directories as a
> folder-within-a-folder-within-a-folder stretches the metaphor quite

very interesting that you go in here, as also in Fenfire -- along with 
much of the common Desktop metafor (sometimes called PUI for PARC UI, 
same as WIMP i guess) -- the filesystem is discarded. the term 
item-based system has been used to describe having 'things people care 
about' directly at hand, and as we hopefully already saw, rdf is used 
to describe the relations in much more flexible ways than the 
folder-withing-folder you mention (more about this is in an unpublished 
(submitted) paper that's in the cvs)

for example, in FenPDF which is about reading research articles and 
making notes about them, there are no directories with named files nor 
directories, nor icons representing anything etc. -- just the articles 
and paper notes (canvases) floating around, structured by the 
connections the user makes. there are menus for actions though (both at 
the top and contextual popups) (the data is of course in the end stored 
in the filesystem, but the user normally never sees the files)

> thin indeed.  In the real world, users frequently spread their work
> out around them and create piles of things with a variety of
> classifications -- the current task, tools used in that task,
> reference material, completed work, and so forth.  People think in
> terms of managing the space around them, not in terms of creating and
> maintaining strict hierarchical relations between concepts and
> categories.

and in the virtual/artificial/computer-based worlds we can go much 
beyond that, enabling several simultanous classifications and different 
views for same things for different purposes, different views for them 
and tools for working with the structures that the classifications form.

people may e.g. define their own relation types. in zigzag this is 
about making new dimensions, for example 'childen' and 'marriage' in 
Nelson's genealogy demo (http://xanadu.com/zigzag/HolmFamilyDemo.html 
-- and unlike that outdated page says, it was made with gzigzag later 
too as Nelson himself demoed in HT01, i should have a copy  of that 
space somewhere), and with rdf there are vocabularies.

> Suppose you have a plain, flat desk, and over the course of your work
> you find you have so covered it with piles of paper it becomes
> cluttered.  If you can't get rid of anything, what do you do?  One
> solution is obvious: expand the available area of your desk into the
> third dimension: add shelves!  Similarly, a two-dimensional computer
> screen is easily cluttered, with too many windows, menus, and icons
> competing for the user's attention.  Can we too add a new dimension to
> our computer interface, expand into three dimensions, and use this
> exploit our understanding of spacial metaphors to improve the
> experience of working with our computer?

(i seem to be quoting in length anyhow..)

hm, i don't quite see how the shelves would work - would it be like 
hiding windows/applications that are not worked on right now, to reduce 
clutter? (i often do that by pressing apple-h on macs)

the note about the clutter is interesting also because it is related to 
the counter-arguments against 3d interfaces that i should be coming at, 
of which one is about placement on screen. but before that, we should 
note that (like you indirectly partly refer in the following, when 
mention the "Hackers" scene etc.) some things are certainly known to be 
applicable, like having remote things fade/shrink away.
that some sort of pseudo-3d was prototyped also in gzigzag with 
VanishingViews, see 
http://an.org/gzz/screenshot-gzigzag-stretchvanishing-10.png for a 
small screenshot demonstrating that (that cell there that is being 
edited actually sets the depth used, 10 is impractical (produces 
clutter) but seemed good for showing the effect)

in FenPDF, if there are a lot of links to/from one place, it may result 
in clutter but that is very fluently solved by zooming in closer to the 
current focus, which makes more remote connections go out of sight.

> There have been many depictions of three dimensional computer
> interfaces in popular media, such as the 1995 movie "Hackers", which
> depicted the file system of a corporate network as towers of names

didn't they use an actual 3d file browser in one of those films, that 
had been made (perhaps used to some extent, even?) on SGI (IRIX)? there 
is this quote from it when some of the characters said, seeing the 3d 
infoscape: 'I know this, this is UNIX!" and started working on it :)

btw there is also a 3d Finder (the mac file manager) 'replacement' 
called 3D OS X, see http://www.acm.uiuc.edu/macwarriors/projects/3dosx/

those cyclic things actually resemble what Asko made earlier with 
gzigzag (http://an.org/gzz/MindSundew.mpeg) and what they have now 
continued in Fenfire Loom (rdf browser, becoming editor) and MMDemo 
(mindmapping, a screenshot is 
http://an.org/FirstAndLast/FenMM-better.png)

> Despite these visions, the use of interactive 3D graphics on the
> desktop (aside from scientific visualization and computer-aided
> design) has so far been confined almost entirely to entertainment.

that is still the case -- hopefully changing quickly now!

besides the research projects mentioned earlier, do note that the 
macosx desktop is  currently entirely rendered using opengl, and the 
new release (upcoming 10.3) now uses 3d for the first time (before 
there was only alpha-channel and animation stuff) in this funny cube 
they have for swithing users:  
http://www.apple.com/macosx/features/fastuserswitching/ (see the 
bottom, each desktop is a texture on a cube that rotates when switching 
the active desktop (default one/user i think but can surely be hacked))

> Consider a virtual office.  It has many advantages over a normal

all that is fine, but are there benefits in having the office as a 3d 
space? i mean, for the user interface to resemble a physical space? us 
being vague about the definition of 3d is probably a problem in this 
discussion, hopefully the points get across anyhow.

> office: if it is too small, it can be easily made larger without
> inconveniencing the next door neighbor.  The user can use whatever ad
(...)
> proximity ("I know I left that file near the potted plant...").

here are again two central things that have been discussed: growing 
larger in space and being near. the point about zigzag was made that 
there, with n dimensions, there is in a way infinite amount of space 
everywhere, and this is (even more) so in the case of rdf.  this means 
that even though there are already many things in a place, or near some 
object, there need not be basically any limit to the amount of 
additional things that can be there, too. and there also does not need 
to be limits to the amount of places a single thing can be in 
simultaneously, etc.

structure and limits, filtering etc. are of course needed in the user 
interface (e.g. visualisations, potential actions etc.).

> Going further, one could abandon the office metaphor and create more
> abstract information representations.  In the "Otherland" series of
> novels by Tad Williams, one character keeps a virtual garden.  In this
> garden, plants represent people, places and things; if news is good
> then the plant grows and flourishes, if news is bad then the plant
> becomes sickly and dies.

oh, great that you mention this 'cause i didn't know about that book 
but had actually come up with a similar idea and suggested it to a 
project recently, where they are developing a sort of a 'virtual 
editorial office' for a magazine, where i thought the state of upcoming 
articles might be nicely represented as plants (ideas for writing as 
seeds, then they need to grow, finally might produce fruit to be 
published :) in a virtual garden.

we have also planned a PlantWorld (with 
http://studio.kyperjokki.fi/FirstAndLast/PlanT s) and would like to 
implement interactive lsystems in some game engine (possibly 
Ter'Angreal if that is used for FirstAndLast, otherwise would think 
Soya 'cause it's really nice to program in Python)

> So what do we need to get from here to there, to realize these ideas
> of a new way of using our computers?  The single greatest problem is

good question :) but it seems much is already underway and there are 
pieces around..

> step.  What is needed is a common platform for representing,
> manipulating and interacting with 3D objects on a level which is
> intuitive to humans.

Tuomas (Fenfire lead developer) argues that that is exactly the problem 
with 3D UI:s: they are difficult to interact with. In games it's ok 
'cause there things are supposed to be challenging, but for other 
things they are clumsy. He thinks that perhaps with actual 3d displays, 
very accurate ones, and sensitive&responsive data gloves etc. they 
*might* be usable, but not with the current (nor foreseeable in the 
near future?) technologies.

So again, why 3d objects, why in 3d space? It can be seen as too 
restrictive:

In the semi-3d / 2,5d / .. libvob the objects not being in a single 3d 
coordinate system enables placing them basically in any way necessary, 
having related objects right at hand and more remote further away, and 
changing the focus with very simple interactions etc. It works so that 
there are several coordinate systems (they are working on making them 
recursive (nested?)) where the items may be.

in physical-like 3d, objects are easily hidden behind others in vain 
etc. of course you can take that as a challenge, but i don't see it 
working fluently (e.g. there is the important object by the flower pot, 
and you happen to stand somewhere else in a room looking at something 
related - what if the pot happens to block the line of sight .. would 
you make the program move the pot away, semitransparent or even 
invisible, the object that's behind it move in sight (what if someone 
else was looking at it from another angle and it then becomes hidden 
for her, would you then clone it?) or what?) - departing from 
simulating the physical world and having more abstract structures on 
screen seems to be a way to avoid these issues.

yet the lessons learned from spacial-/spatiality etc. you mention 
should be simultaneously kept in mind.

> In the modern internetworked world, users may spend more time working
> with data hosted on remote systems than on their local terminal.  The
> desktop metaphor was created long before the Internet became popular,
> and so the ability to access remote resources as easily as local ones
> is unevenly supported, varying by type of resource, applications which
> use that resource, and protocols used to access that resource.  The

fun to see that you come at this, too, 'cause it's been one of the 
fields where also Fenfire has been working and actually the area where 
i was participating last spring. hopefully you'll find our paper about 
it relevant (the long version is outdated but a new one is coming..):

---from: http://www.ht03.org/papers/ ---

Storm: Using P2P to make the desktop part of the Web
Benja Fallenstein, Tuomas J. Lukka, Hermanni Hyytiälä, Toni Alatalo

Short paper: dangling links, peer-to-peer, location-independent 
identifiers, content addressable networks

We present Storm, a storage system which unifies the desktop and the 
public network, making Web links between desktop documents more 
practical. Storm assigns each document a permanent unique URI when it 
is created. Using peer-to-peer technology, we can locate documents even 
though our URIs do not include location information. Links continue to 
work unchanged when documents are emailed or published on the network. 
We have extended KDE to understand Storm URIs. Other systems such as 
GNU Emacs are able to use Storm through an HTTP gateway.

---end---

FenPDF uses Storm to store the articles, but you can use it for any 
other project as well. That is why i asked about using URN pointers 
with VOS.

> implications for the design of future applications.  With the
> network-transparent layer, any application could easily access and
> manipulate remote resources, or serve such resources itself with far
> less effort on the part of the programmer and user than is presently

this area is where VOS seems strong to me and is also something that 
Fenfire is not addressing (currently), real-time interactions over the 
net like in multiplayer games i mean.

> In summary:
> * Humans have strong spacial and visual thinking

agreed. one innovation related to visual thinking in Fenfire BTW is 
generating unique textures based on content-id's (for the articles and 
note-papers to be better identifyable, I will always remember the 
unpublished VOS-paper as this greenish thing, 
http://an.org/FirstAndLast/vos.jpg :) you can try the current 
implementation of the unique textures in 
http://himalia.it.jyu.fi/paperbot/paper

> * Modern user interfaces don't take advantage of this

yes they commonly suck

> * A 3D virtual environment would allow us to use spacial relations
> in our interface

.. but there are also even more allowing models

> * Using a 3D workspace would be highly intuitive, but would also allow
> automation not possible in the real world

it might also be clumsy. oh the point about automation reminded of that 
making macros / scripting like activities should be supported

> * Architecture should erase as much as possible the difference between
> local and network resources

"but not more" (or how did the saying go), yes, and in some cases 
location independent identifiers might be a good solution for that 
(well, Fenfire uses them for basically everything :o) - perhaps not so 
in real-time interaction though were vop-URLs may be fine.

> * Such architecture would easily support interaction with other users,
> visiting other virtual environments

.. that's what we might use VOS for

> * Potential to fundamentally change how information is represented and
> applications are designed

that would be welcome :)

there is the term applitude, which is another thing from Nelson's 
heritage fostered (m-w.com: "sharing nurture or parental care though 
not related by blood or legal ties" which fits the situation with 
patents and trademarks ;) in Fenfire. the idea it carries is that not 
only how applications are designed, but the whole concept of an 
application should be fundamentally changed. the problem with 
applications is that they tend to lock their data in so that no other 
application can refer to it. for example, there are e-mails discussing 
articles and 'word processing' applications for writing articles, but 
often they don't work together, e.g. there can't be interconnections 
between parts of the data. as said, Fenfire uses RDF for this.

> alsso discusses our goals.  It talks up the multiuser/social aspects 
> rather
> than the user-interface aspects, but I think that ultimately they are 
> all tightly connected:

that sounds like something to keep in mind..

ok, hopefully this mess brought something up and didn't cause too bad 
misunderstandings -- looking forward to revisions on all sides and, 
well, funding for the different projects and potential collaboration 
between them..

~Toni