Re[2]: [GNUnet-developers] Some small patches

[Hendrik Pagenhardt]

>> [proposal of delayed inserts]

> Looking at 4.0.17 reference manual, I'm not so sure if this is a good
> idea. Particularly, I suspect that errors might go unnoticed or that
> the database size might be incorrectly reported when delayed inserts
> are used. The manual also says that insert delayed should only be used
> if you're really sure you need it.

I'm not sure how many more uncertainties delayed inserts would
introduce. I tested it on my machine and it worked really well. I even
ran gnunet-check -a after a big insertion session (not really to check
this, but because I suspected inconsistencies from a not gnunetd related
crash of my machine) without any errors showing up. So I think the use
of delayed inserts could be made an option for the more dangerous living
folks among us...

> All in all, I don't think gnunet is generally so faultless that we
> could afford adding any potential troublemakers.

You're probably right at this stage of development.

I thought a bit about the topic, and I think a good way to increase
insertion throughput might be the bundling of inserts within gnunet.
This probably could even be more efficient than the delayed inserts. The
abysmal performance of inserts is IMHO closely related to the sequential
nature of the insertion process (correct me if I'm wrong). And it's not
helping that we can't profit from the potentially parallel select and
insert capabilities of the database, because every bucket is locked with
a semaphore when a request is in progress. BTW which threads can run in
parallel when gnunetd is running? I would hope that at least one thread
for each connection (local or remote) is used?

For mysql this might be improved by collecting the inserts per bucket in
a separate thread and when a threshold number or a timeout is reached
from the last insert then a REPLACE statement with multiple value tuples
can be created and sent to the mysql server. Of course this would skew
the table quotas and of course this is more difficult to handle if
errors occur, but I think it might be worth the hassle because one of
the more annoying "features" of GNUnet is the slow insertion/indexing
process. Which of course might lead to less acceptance among users and
hindering the willingness to publish content...

Ciao,
     Hendrik