Re: [Help-gnunet] seperate disk quota for migrated content

[Benjamin Paul Kay]

On Monday 06 October 2003 13:28, you wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Saturday 04 October 2003 12:49 pm, you wrote:
> > On Friday 03 October 2003 19:16, you wrote:
> > > -----BEGIN PGP SIGNED MESSAGE-----
> > > Hash: SHA1
> > >
> > > On Friday 03 October 2003 06:00 pm, Benjamin Paul Kay wrote:
> > > > I'd like to allow content migration on my node. After running freenet
> > > > for some time, I appreciate how content migration can benefit the
> > > > entire network.
> > > >
> > > > The problem:
> > > > gnunet.conf only lets me specify one disk quota. It claims that
> > > > content that is indexed only (which mine is) does not take up disk
> > > > quota, but I quickly learned that to continue to index files I must
> > > > make my disk quota bigger. Migrated content also uses the disk quota.
> > > > So if I leave content migration on, I can no longer index files. If I
> > > > leave content migration on long enough to fill the quota, turn it
> > > > off, and then increase the quota the benefits of content migration
> > > > are largely negated. Is there any way to specify one quota for
> > > > content I insert/index and another for migrated content?
> > > >
> > > > Thanks!
> > >
> > > Indexed content does take away some storage space, I would guess
> > > roughly about 10% of the size of the indexed files, but that's just an
> > > approximation and it also depends on the type of database that you are
> > > using.  The documentation was supposed to indicate that it uses much
> > > less space than full insertion. If you can point me to the ambiguous
> > > passage, I'll certainly try to improve that.
> > >
> > > I don't see that it would make too much sense to have 2 quota options.
> > > Especially since gnunet-insert does *remove* "worthless" migrated
> > > content in favour of indexed content.  You should only hit the quota if
> > > your indexing surpasses the quota disregarding the migrated content
> > > (assuming that the priority given for locally indexed content is
> > > sufficiently high, the default of 50 should be fine).
> > >
> > > Aside of that I consequently don't understand the need for such an
> > > option, it would be quite difficult to implement this type of
> > > segregation of DB quotas efficiently (without ending up with 2 DB
> > > lookups per query instead of one).
> > >
> > > Christian
> > > -----BEGIN PGP SIGNATURE-----
> > > Version: GnuPG v1.0.7 (GNU/Linux)
> > >
> > > iD8DBQE/fgNY9tNtMeXQLkIRAheYAJ9Is5/KbrkGT6vJC9ND1KYpLmAV/gCfSQ0O
> > > uy7QwUVb8W1hEIZZiAatXEE=
> > > =yWds
> > > -----END PGP SIGNATURE-----
> >
> > Thanks for the clarification! I just migrated my DB from gdbm, which was
> > very unstable, to mysql, which is tolerably unstable.
>
> Could you elaborate on "unstable"? It works fine for me...
>
> > Thanks to an xfs file
> > system, I can get away with an insanely huge DB without messing with
> > RAID. I will try re-enabling content migration and see what happens. Just
> > to clarify: 1. Indexed files take up about 10% of their size in the
> > database. So a 700MB file will take up roughly 70,000 blocks in the DB.
>
> Right.
>
> > 2. If content migration is enabled, content will be automatically added
> > to the DB until the DB is full. If I then choose to index a file,
> > migrated content will be deleted to make room for the new file.
>
> Right.
>
> > By the way - when gnunetd crashes while indexing, the DB blocks used by
> > the indexed file do not appear to be automatically freed. Will
> > 'gnunet-check -a' reclaim those DB blocks?
>
> No, but you can reclaim them using "gnunet-delete".
>
> Christian
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.0.7 (GNU/Linux)
>
> iD8DBQE/gaYx9tNtMeXQLkIRAg62AKCJizOIhOHlfAfKgW4RwcQZkiBmnACfS9/b
> jcYJ3IoGkVex/GDJHeC+oPY=
> =9wIz
> -----END PGP SIGNATURE-----

gnu-delete could work, but deleting data takes about as long as inserting it. 
some sort of database editing tool would be helpful here!

More to the point, gnunet is slow and unstable when in comes to indexing large 
files. Indexing a 700MB file takes about 12 hours and fails 3 out of 4 times, 
so a single index can take 48 hours! Using gnunet-delete before indexing 
files to make sure there are no duplicate indexes would only make this take 
longer. I'm hoping 0.6.0 will remedy this problem. Or maybe this is a problem 
unique to my machine - anyone else have trouble indexing large content?

It occured to me that it might be possible to fake the index of a large file 
(add it to the bloomfilters but don't actually have gnunet-insert index it) 
and then run gnunet-check. Operating on the assumption that gnunet-check is 
more stable than gnunetd/gnunet-insert, this would allow me to index large 
files effectively at the cost of taking my node offline during the process. 
The questions involved here are: 1) is this plausible 2) is gnunet-check 
stable 3) how would I go about doing it and 4) will this strategy work in 
harmony with content migration

If any one has any other suggestions that don't involve splitting up the large 
files, please let me know!