Re: [Gzz] Raw pools?

[Tuomas Lukka]

On Mon, Nov 11, 2002 at 08:45:23PM +0100, Benja Fallenstein wrote:
> Tuomas Lukka wrote:
> 
> >On Sun, Nov 10, 2002 at 08:00:26PM +0100, Benja Fallenstein wrote:
> > 
> >
> >>As one of the important principles of Storm, I see what I call the 
> >>"persistency commitment" (which I should peg ;-) ):
> >>   
> >>
> >
> >We should probably PEG the basic ideology of Storm as a whole:
> >
> 
> Can we s/ideology/philosophy/? :-)

Absolutely. 

> >     - persistent blocks: operations
> >             - get a block's bits EXACTLY or "sorry, can't get them" 
> >             -answer
> >             - store a block, get an ID
> >     - pointers 
> >
> >     - ...
> >
> 
> I think we should have several related PEGs for these.

Please propose a set of PEGs.

> >I didn't mean that raw pools should be equal to normal storm pools.
> >The raw blocks would *NOT* have storm ids and would not be addressed in
> >that way. 
> >
> 
> Ok, I guess we can see my proposal as a way to implement yours, then :-)
> 
> (Let's not call these 'blocks,' then, though. 'Chunks' or something else 
> that hasn't a meaning in Storm yet would be more appropriate.)

Fine for me.

> >>>E.g. for xupdf, this would be vital for other people to be able
> >>>to use the demo.
> >>>
> >>>     
> >>>
> >>We have the canonical blocks (with just the Content-Type header); since 
> >>you have to call a program to put something inside Storm anyway (unless 
> >>you're going to calculate the SHA-1 hash yourself), I don't see the 
> >>difference it would make at this point in time.
> >>   
> >>
> >
> >It *does* make a big difference: the SHA-1 is not the same that someone
> >just obtaining the file would calculate. That's a big issue because
> >most SHA-1 -content-based-retrieval systems will *NOT* have the
> >same Content-TYpe header.
> >
> 
> I still don't agree that this makes a big difference *at this point in 
> time*. So far, I have *never* entered a SHA-1 in a 
> content-based-retrieval system. Besides, if the papers used in xupdf 
> were available in one, that would be illegal, while they're also legally 
> available from the respective web sites *we* got them from-- except if 
> the copyright holders would make them available there, which isn't 
> paricularly likely.
>
> I'm saying this to explain why I'm still not convinced we should change 
> Storm at this time. Can you explain why you feel this is so important 
> right now?

Yes.

I want to be able to *now* make a *permanent, eternal* link which uniquely
identifies a file, which I can't unfortunately redistribute, 

So that if someone enters all the files on their disk to a SHA-1 index,
they'd get a hit.

So that anyone who *does* have access to the file can verify that it *is* the 
same
file that I made the link to.

The permanence is the point.

Canonical blocks (with trivial header) feel like a very unclean solution;
I expect that sha-1 of files will see much more use in a few years.

> Would it, btw, be legal to provide a download script for the papers, 
> like Debian provided for Microsoft's web fonts?

Probably. But some of them are of course only available e.g. at universities
with deals with ACM.

> >>Let me put it like this: It has broad enough applicability that I'm 
> >>thinking this *might* just be good enough to warrant the burden on 
> >>future implementations.
> >>   
> >>
> >
> >Great, let's PEG it.
> >
> 
> Ok. Who's going to?

For the chunk thing, you're probably the best.

        Tuomas