Re: [Gzz] Raw pools?

[Benja Fallenstein]

Tuomas Lukka wrote:

> An idea, related to the "canonical blocks":
> Maybe we should label some pools "raw", i.e. no header, just  a block of
> binary data. That way, we could be compatible with other content-based systems
> for externally obtained data.
>  

That means we cannot move blocks from there to non-raw pools. Also, we'd 
have to guess the type.

Here's another idea I had lately. It would be nice to have support for 
tree hashing at the Storm level-- even though it's unfortunate that the 
chunk granularity will be determined by the block creator, this way you 
could check the id whenever a new chunk arrives, so you could e.g. show 
parts of an image while it is loading (like in a Browser-- you only have 
to wait until you have a complete chunk, then you can check the hash and 
pass it upward). Also, we could more easily download a file from 
multiple sources.

Now, how about we implement this in the following way:

- Have a new id type (02 ids).
- Ids of this type contain a hash of the *header* of the block.
- The header would have a "Storm-Body-Parts" field that would list the 
hashes of all chunks in the body, plus the chunks' sizes.
- (We could still store the block in a single file on the file system, 
since the header is clearly limited and the body chunks are limited by 
their size, so we could still check the hashes correctly when reading 
the block in, since we know where one chunk ends and another one starts.)

With this, we would solve four things at once:
- We would only have to read the header, then we could read the body 
chunk-by-chunk (to stream or to download from multiple sources).
- We could securely read the header without having to read the body as well.
- If there's an existing file on a p2p network we'd like to refer to, we 
can simply create a header for it that has as its sole body chunk that 
file. Since body chunks are refered to by an SHA-1 hash, the body would 
be the existing file, for the purposes of the p2p network.
- We could even securely distribute the header without violating any 
copyright by distributing the body.

(Note that the body chunks will *not* be Storm blocks, even though 
they're refered to by an SHA-1 hash. It is therefore illegal to refer to 
a body chunk through a Storm URI.)

On the downside, there would be more computation in id checking, since 
we'd also need to parse the header.
- Benja