Re: [Dazuko-devel] Dazuko driver takes lots of cpu time, bottleneck discovered

[John Ogness]

Tikka, Sami wrote:
> When I have dazuko driver loaded and our on-access-scanner running, I
> frequently see my system go into a state where the system takes 50-90% of all
> cpu time (reported by 'top'). 
>
> This usually happens after accessing a lot of files, like hundreds of
> thousands. (Compile XFree86 and glibc)
>
> When I kill the on-access-scanner process, the system cpu usage goes down to
> almost nothing, i.e. the normal situation.
>
> During such a system-load situation I ran oprofile and this is what it
> reports:
>
> samples  %        app name                 symbol name
> 1606     56.0950  dazuko.o                 xp_compare_file
> 523      18.2676  dazuko.o                 dazuko_mark_hash_dirty
> 93        3.2483  dazuko.o                 dazuko_get_hash
> [deleted the rest]

All of these functions have to do with the temporary file pointers that 
are stored at ON_OPEN so that information can be returned at ON_CLOSE. 
This is a linked list structure which was slightly optimized. However, 
if many files are open simultaneously, it would still be very 
inefficient. I would like to change the structure to be an n-ary tree. 
These are simple to implement and are very fast.

However, I am more concerned that there is a "file" leak occurring. If 
ON_OPEN events occur without a corresponding ON_CLOSE event, the local 
file copy will remain allocated forever. This causes a gradual memory 
leak as well as damaging the hash-lookup performance.

If you compile Dazuko with debug, Dazuko will print out all currently 
allocated "open" files when Dazuko is being removed. If you compile 
XFree86, glibc, mozilla, openoffice, etc and then afterwards kill the 
scanner and unload Dazuko, do you see a huge list of "open" files?

John Ogness

--
Dazuko Maintainer