Re: [Help-gnunet] INDIRECTION_TABLE_SIZE and download speed

[Christian Grothoff]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Saturday 07 September 2002 07:55 am, Igor Wronsky wrote:
> On Fri, 6 Sep 2002, James Blackwell wrote:
> > The problem is that GNUNet is by design causing data to get stuck on the
> > originating hosts by allowing that new data being published to be
> > concurrantly requested, causing all of the files of value to get stuck in
> > the tiny little straw of a pipeline most people have.
>
> I see your point. I just don't see a neat technical solution.
> A straightforward way would be for the node the keep track of
> which blocks are indexed locally and which blocks belong to
> which file. Once some block is accessed, prevent the downloading
> of blocks belonging to other locally indexed files for a certain
> period of time. The extension to multiple-files-simultaneously
> is straightforward.

Bad idea. When would you decide that content is now popular enough to just 
serve it at all times? Also, by not answering the request, all that happens 
is that you get it later again (after another 50000 nodes have been queried), 
so you just wasted bandwidth that you tried to conserve. If you get a request 
and you can answer (bandwidth limits, etc), do answer. Only then the other 
nodes get the chance to replicate the content. Locking away certain files 
(even for a limited time) is always just a bad idea, it's wasting storage 
space. 

> > For the most part this is not true for p2p networks, though there is one
> > vital exception: Getting the original data off of the originating node.
> > That's because there is only *one* place that has this data set.
>
> On freenet that doesn't hold. Inserted content is distributed
> to TTL hosts on insert. I think gnunet should push content out
> as well, but I haven't been able to convince CG of how that'd
> fit the economy model - because I don't know a good answer
> myself. Content pushing would solve the issue partly though,
> because the load would be instantly distributed, and we
> are inserting just one file at a time.

What if somebody inserts content while being off-line? What I am really not 
convinced of, is that the scenario 'new popular content inserted, everybody 
wants it, node can not serve' is the real problem (or anywhere near it). 
Because the popular content would be send to the first requester and 
*instantly* replicated on the way (similar to freenet). So this is definitely 
not the problem. The problem is, that if the person initially providing the 
content goes off-line (say forever) after half of the file has been 
downloaded. If the content is also popular, many people will try to download 
the file, at first even get decent speeds, but after 1/2 of the file has been 
retrieved (which is useless due to the random order of the blocks), the 
download speed drops to 0 because the rest is *not available*. At all, ever. 
And there is no way to prevent this from happening since you can not force 
the originator to go back on-line. 
The only solution is to only "allow" a download after n replicas of the entire 
file exist in the network and we can guarantee with reasonable probability 
that not all of them can disappear. But this is still not a good solution 
since malicious parties could still insert half a file and claim that it's 
the whole thing. Anonymity constraints prevent us from tracking such 
malicious parties. Signing individual content with 'I guarantee that it'll be 
there' does not help either. *Directories* which collections of good content 
will do a better job (the person creating the directory 'vouches' for content 
availability, if you got 3 files from the directory and they were all ok, the 
other 300 will probably also be ok). Note that we would not even need 
pseudonyms or anything else for these directories (the current file-encoding 
will suffice for integrity). 

> For this particular problem, I don't know if CG wants
> to add NO_GO_NOW -replies to the protocol, but it'd seem
> to require that, or otherwise the users won't know
> what gives and think the net even worse as now (when
> they can get that 200cps trickle).

Definitely not. What we rather need is some cleverness in the download-code to 
tell the user that 'the content is not available with probability X%'. And 
there are some improvements that I want to do to the routing code to find 
available content faster (fewer hops), but routing issues seem to be beyond 
the point of this discussion.

Christian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE9emeH9tNtMeXQLkIRAoxdAJ4t9oUOj/kMgZnaOidbUuWvvu/f+wCghvC8
gWdSylb74dJ5Jp0/myd8SP8=
=Lj6l
-----END PGP SIGNATURE-----