Re: [Gzz] Raw pools?

[Tuomas Lukka]

On Sun, Nov 10, 2002 at 02:39:58PM +0100, Benja Fallenstein wrote:
> Tuomas Lukka wrote:
> 
> >An idea, related to the "canonical blocks":
> >Maybe we should label some pools "raw", i.e. no header, just  a block of
> >binary data. That way, we could be compatible with other content-based 
> >systems
> >for externally obtained data.
> > 
> >
> 
> That means we cannot move blocks from there to non-raw pools. Also, we'd 
> have to guess the type.

Yes. But I think that's acceptable for the intended purpose:
allowing us to use material that we can't distribute.

E.g. for xupdf, this would be vital for other people to be able
to use the demo.

How vehemently opposed are you?

> Here's another idea I had lately. It would be nice to have support for 
> tree hashing at the Storm level-- even though it's unfortunate that the 
> chunk granularity will be determined by the block creator, this way you 
> could check the id whenever a new chunk arrives, so you could e.g. show 
> parts of an image while it is loading (like in a Browser-- you only have 
> to wait until you have a complete chunk, then you can check the hash and 
> pass it upward). Also, we could more easily download a file from 
> multiple sources.
> 
> Now, how about we implement this in the following way:
> 
> - Have a new id type (02 ids).
> - Ids of this type contain a hash of the *header* of the block.
> - The header would have a "Storm-Body-Parts" field that would list the 
> hashes of all chunks in the body, plus the chunks' sizes.
> - (We could still store the block in a single file on the file system, 
> since the header is clearly limited and the body chunks are limited by 
> their size, so we could still check the hashes correctly when reading 
> the block in, since we know where one chunk ends and another one starts.)
> 
> With this, we would solve four things at once:
> - We would only have to read the header, then we could read the body 
> chunk-by-chunk (to stream or to download from multiple sources).
> - We could securely read the header without having to read the body as well.
> - If there's an existing file on a p2p network we'd like to refer to, we 
> can simply create a header for it that has as its sole body chunk that 
> file. Since body chunks are refered to by an SHA-1 hash, the body would 
> be the existing file, for the purposes of the p2p network.
> - We could even securely distribute the header without violating any 
> copyright by distributing the body.
> 
> (Note that the body chunks will *not* be Storm blocks, even though 
> they're refered to by an SHA-1 hash. It is therefore illegal to refer to 
> a body chunk through a Storm URI.)

So these would, essentially, be blocks in raw pools?

I think we can do this...

> On the downside, there would be more computation in id checking, since 
> we'd also need to parse the header.

Very small price to pay.

        Tuomas