Re: [Gzz] Raw pools?

[Benja Fallenstein]

Hi Tuomas,

Tuomas Lukka wrote:

> On Sun, Nov 17, 2002 at 10:46:17PM +0100, Benja Fallenstein wrote:
>  
>
> > Tuomas Lukka wrote:
> >
> > > Between pools?? 
> > Yes, so? (Whenever a block moves, it moves between pools? Except if a 
> > pool moves physically, such as moving a disk to a different place, which 
> > is not interesting to consider.)
> >    
>
> Oh, my definition of pool was "all mediaserver repositories anywhere with
> the same pool ID" -- more virtual.

Hmm, right, that was the original definition... I'm getting all confused 
thinking about this, right now it's not clear to me at all what the 
boundary of a pool in that sense should be and how to rightly decide in 
which pool to put a block etc. -- I considered pool to be one storage 
location, that could be addressed through a StormPool object. I guess I 
then thought that pools'd have names (ids) and names could be shared 
between pools... I.e., for me, mediaserver repository = storm pool.

Can someone make use cases for multiple pools on the same computer (when 
we want to use them, and in which pools the user would expect specific 
blocks to go)? I'm confuzzled right now.

> So you mean moving between physical locations?

Yes, where 'physical location' means Storm pool ;-P

No, really, it includes moving data from a CD to the hard disk or to a 
DHT as well as moving data from the 'cache' to the 'permanent' pool 
where it won't be deleted automatically.

> > > > Quite a suite. Would greatly help people trying to understand what we're 
> > > > up to, though.
> > > >  
> > > >
> > > >        
> > > How about starting with
> > >
> > > High-level
> > >
> > > 1. Storm concept and rationale
> > >         - eternal blocks, moving easily
> > >
> > >      
> > Should also have the commitment PEG at the same time.
> >    
>
> Well, that *could* also be later, as making this more exact.

Ok.

> > > 2. Storm and crypto: signatures, identities
> > >      
> > What's this?
> >    
>
> The overall intent on how Storm is supposed to work with crypto.

Hm, ok.

> > > Implementation
> > >
> > > 1. Storm block format
> > > 2. Storm pointers layer 1: no crypto, nothing except branching versioning
> > >
> > >      
> > Should also have the pointer concept PEG then. Describing the formats 
> > without the rationale doesn't make much sense.
> >    
>
> I didn't say formats here. This *would* be the concept. ;)

Then it's not implementation. :-)

Note that pointers are to Storm like TCP is to IP-- TCP is *one* 
application of IP, though a very important one. So it's not like 
pointers are a way of implementing a part of the Storm core; they 
implement a service on top of the Storm core which may be useful for may 
applications (but not all: xanalogical content blocks are specifically 
*not* versioned).

> > Also, 'implementation' is a misnomer: 'Formats' would be better.
> >    
>
> Well, then the above pointers thing shouldn't be here.

*gg*

Ok, it's just words ;-)

> Ok. Canonical blocks are bad, because
>
>         1) they don't solve the problem of getting the unadulterated data's 
> SHA-1
>            except through new features which just make things more complicated
>            (everyone has to support those mappings etc!)

Yes, but not *in the core*. It is an application, so latter implementors 
of the Storm formats won't have to support it. It might be nice if 
*someone* still implemented that application, but you could implement 
the Storm core without implementing these applications, too. (Which 
*may* be unnecessary because it may be either possible to retrieve the 
block itself, or alternately impossible to retrieve the article alone, 
because it dropped off the web.)

>         2) they have a part of the normal header but not all; everyone needs to 
> support
>            that as well.

You are saying that Storm blocks are required to have a Date: header 
etc.? That's *really bad*, we need canonicality for many applications 
(for example the email client, which takes care that equal body parts of 
the same email received through different paths get the same xanalogical 
id).

Also, I think this is very true for pdf/ps also: If I insert a PDF and 
make links to it, and someone else inserts the *same* PDF and makes 
links to it, the copies should have the same xanalogical id (thus this 
needs canonical headers even if header and body are separated). There's 
no problem with this here: any article PDF will contain enough data that 
it is impossible that someone else entered the same data by chance.

> I think they get us *more* complexity than the header-block separation would,
> so it might be good to ditch them *now*.
>  

I agree that 1) above is additional complexity; but I don't think that 
we actually need to support it eternally, as the header-block separation 
(because e.g. an external mapping would be *external to Storm*; if at 
some point we have xu links between blocks, or are able to distribute 
the Storm blocks or whatever, we can ditch it *without breaking any 
Storm blocks*, just one specific method to create these blocks by 
downloading them by SHA-1). I don't agree that 2) is additional 
complexity; IMO we need this anyway.

- Benja