Re: [Help-gnunet] Inserting large amounts of content?

[Christian Grothoff]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Monday 21 April 2003 07:14 am, Per Kreuger wrote:
> Ok, I've tried the cvs version of (friday 18) and adjusted only the Disk
> quota to 5G.
>
> Still I failed after inserting about 3.5G in 946 files. At this point
> the size of /var/lib/GNUnet/data was about 2.5G i.e. half of the allowed
> disc quota.
>
> The error reported at failure was:
>
>       WARNING readTCPResult failed server, closed connection
>       WARNING server did not send confirmation of insertion
>       WARNING child->insert failed on level 1, pos 0, aborting
>       WARNING child->insert failed on level 2, pos 0, aborting
>       WARNING child->insert failed on level 3, pos 0, aborting
> Error insering the file (-1)

The cause for this is known. Currently, GNUnet can not do any exact 
predictions on how big the database will be, so we put in some values from 
experiments conducted by Eric -- and made them even a bit more conservative. 
Yes, that should be fixed, but you can certainly work around it.

> What is probably worse is very distinct degradation of insertion speed.
> I enclose a postscript file with the insertions-rate listed and graphed
> over the several hours it took to insert the 946 files.

Some degradation is to be expected with any database. You got a larger 
degradation due to the uneven distribution of content to buckets (see below).

> Since I failed inserting the last file I tried a gnunet-check -a on the
> resulting database. Somewhere in the index chek part it started to
> report failure
>
>       "PIdx database corrupt (could not unlink ...) from low DB ..."

Hmpf. That should not happen. I'll try to reproduce it and fix if possible.

> After some 40 hours I interupted this process and tried to restart
> gnunetd. It appears to come up fine and I can find and download at least
> some of the files I've inserted. I don't see any other host and I get a
> timeout from ojmv http when (I guess) downloading the nodelist but this
> could have other reasons.

If you did it last night, Purdue (where ovmj.org is hosted) was potentially 
off-line (I could not connect from within), so that may be trivial to 
explain. Could you access ovmj.org with your browser at that point?

> Trying to insert additional files after restarting gnunetd gives the
> same error as above but also fills the logfile with "PIdx database
> corrupt..." messages and then crashes gnunetd.

Can you be more specific about 'crashes gnunetd' (see FAQ on bug 
reporting...).

> It would be interesting to know why the insertion speed degrades. I
> noted that the size of gnunetd grew to about 60M during the insertion
> process and that the distribution of sizes of the DB-buckets was quite
> uneven all through the process. In the end, there are 20 content buckets
> (bucket.5120.0-19) and their size vary from 27M to 444M. The index
> buckets are all of approximately the same size (3-4M).

I experienced the same problem here and fixed it *after* the CVS version you 
are saying you're using. Igor now reports that for his 2 buckets,
"they stay nicely put with ratio like 1.00267". Note that if you insert many 
files of the same mime-type or with the same keyword, you may still get some 
spikes in the buckets for this keyword or mime-type.

> Why are there content buckets at all? I had content migration disabled
> in the config file.

Well, there is still some content (IBlocks - as in 'index summaries' and 
ContentIndex entries - as in 'index') that needs to be stored whenever you 
index a file with gnunet-insert. And even if the index information is small 
(say, about 114 bytes per kilobyte in the original file), the database may  
require a bit more space to actually store it.

Christian

>       Hope this helps
>
>       piak


> Christian Grothoff wrote:
> > On Thursday 17 April 2003 02:21 am, Per Kreuger wrote:
> >>I'm experimenting with inserting large amounts of content (about 20G)
> >>but fail (as axpected) at aboout 8G. The config file mentions changing
> >>parameters in src/include/config.h but I found no such parameter in that
> >>file.
> >>
> >>What to do?
> >
> > If you are using current CVS, you only have to edit the diskquota option
> > in gnunet.conf. All other places have been 'resolved'. Note that if you
> > change the quota (or the database), you need to run gnunet-convert to
> > convert the database over (to the new size or type).
> >
> > Christian
>
> _______________________________________________
> Help-gnunet mailing list
> address@hidden
> http://mail.gnu.org/mailman/listinfo/help-gnunet
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE+pCwd9tNtMeXQLkIRAqRVAKCh7tgJwyTP9gOFrBAU8xZ7vacmgwCfYt6r
xaaJUMC4H/LtkjLERuSTs/g=
=5DN8
-----END PGP SIGNATURE-----