Re: [Help-gnunet] seperate disk quota for migrated content

[Christian Grothoff]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wednesday 08 October 2003 02:32 pm, you wrote:
> On Monday 06 October 2003 13:28, you wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > On Saturday 04 October 2003 12:49 pm, you wrote:
> > > On Friday 03 October 2003 19:16, you wrote:
> > > > -----BEGIN PGP SIGNED MESSAGE-----
> > > > Hash: SHA1
> > > >
> > > > On Friday 03 October 2003 06:00 pm, Benjamin Paul Kay wrote:
> > > > > I'd like to allow content migration on my node. After running
> > > > > freenet for some time, I appreciate how content migration can
> > > > > benefit the entire network.
> > > > >
> > > > > The problem:
> > > > > gnunet.conf only lets me specify one disk quota. It claims that
> > > > > content that is indexed only (which mine is) does not take up disk
> > > > > quota, but I quickly learned that to continue to index files I must
> > > > > make my disk quota bigger. Migrated content also uses the disk
> > > > > quota. So if I leave content migration on, I can no longer index
> > > > > files. If I leave content migration on long enough to fill the
> > > > > quota, turn it off, and then increase the quota the benefits of
> > > > > content migration are largely negated. Is there any way to specify
> > > > > one quota for content I insert/index and another for migrated
> > > > > content?
> > > > >
> > > > > Thanks!
> > > >
> > > > Indexed content does take away some storage space, I would guess
> > > > roughly about 10% of the size of the indexed files, but that's just
> > > > an approximation and it also depends on the type of database that you
> > > > are using.  The documentation was supposed to indicate that it uses
> > > > much less space than full insertion. If you can point me to the
> > > > ambiguous passage, I'll certainly try to improve that.
> > > >
> > > > I don't see that it would make too much sense to have 2 quota
> > > > options. Especially since gnunet-insert does *remove* "worthless"
> > > > migrated content in favour of indexed content.  You should only hit
> > > > the quota if your indexing surpasses the quota disregarding the
> > > > migrated content (assuming that the priority given for locally
> > > > indexed content is sufficiently high, the default of 50 should be
> > > > fine).
> > > >
> > > > Aside of that I consequently don't understand the need for such an
> > > > option, it would be quite difficult to implement this type of
> > > > segregation of DB quotas efficiently (without ending up with 2 DB
> > > > lookups per query instead of one).
> > > >
> > > > Christian
> > > > -----BEGIN PGP SIGNATURE-----
> > > > Version: GnuPG v1.0.7 (GNU/Linux)
> > > >
> > > > iD8DBQE/fgNY9tNtMeXQLkIRAheYAJ9Is5/KbrkGT6vJC9ND1KYpLmAV/gCfSQ0O
> > > > uy7QwUVb8W1hEIZZiAatXEE=
> > > > =yWds
> > > > -----END PGP SIGNATURE-----
> > >
> > > Thanks for the clarification! I just migrated my DB from gdbm, which
> > > was very unstable, to mysql, which is tolerably unstable.
> >
> > Could you elaborate on "unstable"? It works fine for me...
> >
> > > Thanks to an xfs file
> > > system, I can get away with an insanely huge DB without messing with
> > > RAID. I will try re-enabling content migration and see what happens.
> > > Just to clarify: 1. Indexed files take up about 10% of their size in
> > > the database. So a 700MB file will take up roughly 70,000 blocks in the
> > > DB.
> >
> > Right.
> >
> > > 2. If content migration is enabled, content will be automatically added
> > > to the DB until the DB is full. If I then choose to index a file,
> > > migrated content will be deleted to make room for the new file.
> >
> > Right.
> >
> > > By the way - when gnunetd crashes while indexing, the DB blocks used by
> > > the indexed file do not appear to be automatically freed. Will
> > > 'gnunet-check -a' reclaim those DB blocks?
> >
> > No, but you can reclaim them using "gnunet-delete".
> >
> > Christian
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.0.7 (GNU/Linux)
> >
> > iD8DBQE/gaYx9tNtMeXQLkIRAg62AKCJizOIhOHlfAfKgW4RwcQZkiBmnACfS9/b
> > jcYJ3IoGkVex/GDJHeC+oPY=
> > =9wIz
> > -----END PGP SIGNATURE-----
>
> gnu-delete could work, but deleting data takes about as long as inserting
> it. some sort of database editing tool would be helpful here!
>
> More to the point, gnunet is slow and unstable when in comes to indexing
> large files. Indexing a 700MB file takes about 12 hours and fails 3 out of
> 4 times, so a single index can take 48 hours! 

It fails!? Could you be a bit more specific about what you mean by 'fails'? 
Also, indexing performance largely depends on what type of database you are 
using (see benchmarks in the documentation).

>Using gnunet-delete before
> indexing files to make sure there are no duplicate indexes would only make
> this take longer.

You don't have to do gnunet-delete if you want to merely re-do the 
gnunet-insert run.  Just run gnunet-insert again, the system will 
automatically detect that a block already exists and will not use additional 
space. gnunet-delete should only be used if you want to remove the file for 
good.

> I'm hoping 0.6.0 will remedy this problem. Or maybe this
> is a problem unique to my machine - anyone else have trouble indexing large
> content?

I didn't have any reports so far, and your problem description is quite 
unspecific (other than "slow").

> It occured to me that it might be possible to fake the index of a large
> file (add it to the bloomfilters but don't actually have gnunet-insert
> index it) and then run gnunet-check. 

gnunet-check is much slower than gnunet-insert.

> Operating on the assumption that
> gnunet-check is more stable than gnunetd/gnunet-insert, this would allow me
> to index large files effectively at the cost of taking my node offline
> during the process.

I have not seen gnunetd/gnunet-insert crash in a really long time.  Elaborate 
on what you mean by unstable.  Also, gnunet-check typically is more 
problematic that any of the other tools since it is much more complex.

> The questions involved here are: 1) is this plausible
> 2) is gnunet-check stable 3) how would I go about doing it and 4) will this
> strategy work in harmony with content migration

I would not worry about content migration.  Explain better what you think is 
wrong with gnunet-insert (run with higher log-levels, read FAQ on reporting 
bugs, give me some output from the console, and we can see what is going 
on.).
Oh, and it is typically best if you use CVS when trying to report bugs -- just 
to make sure we're not chasing a dead horse.  And CVS has had some bugs fixed 
in gnunet-check/convert that are not in 0.6.0.

> If any one has any other suggestions that don't involve splitting up the
> large files, please let me know!

I don't think splitting would help much.  What DB are you using anyway (I 
sincerely hope it is not directories... :-)

Christian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)

iD8DBQE/hHbq9tNtMeXQLkIRAkakAJwJ8XOM96nouZOFYR8bjt14HxqDRgCbBp2V
uc9IHUhGStEbPFQroGqlD0c=
=iyXr
-----END PGP SIGNATURE-----