Re: [Dazuko-devel] Dazuko driver takes lots of cpu time, bottleneck discovered

[Sami Tikka]

On Mon, Oct 04, 2004 at 08:42:16PM +0200, John Ogness wrote:
> All of these functions have to do with the temporary file pointers that 
> are stored at ON_OPEN so that information can be returned at ON_CLOSE. 
> This is a linked list structure which was slightly optimized. However, 
> if many files are open simultaneously, it would still be very 
> inefficient. I would like to change the structure to be an n-ary tree. 
> These are simple to implement and are very fast.

What is the reason for keeping this list? Is it used only for
determining if a file is 'dirty', i.e. has been written to after open()?

If that's the case, then how about abandoning the list altogether and
using brute force: send DAZUKO_CLOSE_MODIFIED event for every close() if
the file was opened with flags O_WRONLY|O_RDWR and do not send the event
if the file was opened O_RDONLY. I believe the Linux 'struct file'
carries the open flags in its f_flags field.

> However, I am more concerned that there is a "file" leak occurring. If 
> ON_OPEN events occur without a corresponding ON_CLOSE event, the local 
> file copy will remain allocated forever. This causes a gradual memory 
> leak as well as damaging the hash-lookup performance.

Yes... I guess there could be situations where a process forgets to call
close, either on purpose or otherwise...

> If you compile Dazuko with debug, Dazuko will print out all currently 
> allocated "open" files when Dazuko is being removed. If you compile 
> XFree86, glibc, mozilla, openoffice, etc and then afterwards kill the 
> scanner and unload Dazuko, do you see a huge list of "open" files?

Wow... Thanks for the tip. I hadn't noticed that before. I have been
running with dazuko compiled with --enable-debug and when I looked into
my /var/log/messages, I found this:

(Lots of references to /var/run/utmp. I believe it is the 'top' utility
that is responsible for those.)


Oct  4 10:50:08 localhost kernel: <ko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/sis_driver.c
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/sis_opt.c
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/sis_setup.c
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/sis_shadow.c
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/sis_vb.c
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/sis_vga.c
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/sis_video.c
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/300vtbl.h
Oct  4 10:50:08 localhost kernel: dazuko: removing hash /var/run/utmp
Oct  4 10:50:08 localhost last message repeated 32 times
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/310vtbl.h
Oct  4 10:50:08 localhost kernel: dazuko: removing hash /var/run/utmp
Oct  4 10:50:08 localhost last message repeated 8 times
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/initdef.h
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/init.h
Oct  4 10:50:08 localhost kernel: dazuko: removing hash /var/run/utmp
Oct  4 10:50:08 localhost kernel: dazuko: removing hash /var/run/utmp
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/oem300.h
Oct  4 10:50:08 localhost kernel: dazuko: removing hash /var/run/utmp
Oct  4 10:50:08 localhost last message repeated 23 times
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/oem310.h
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/osdef.h
Oct  4 10:50:08 localhost kernel: dazuko: removing hash /var/run/utmp
Oct  4 10:50:08 localhost last message repeated 35 times
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/sis300_accel.h
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/sis310_accel.h
Oct  4 10:50:08 localhost kernel: dazuko: removing hash /var/run/utmp
Oct  4 10:50:08 localhost last message repeated 30 times
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/sis_accel.h
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/sis_cursor.h
Oct  4 10:50:08 localhost kernel: dazuko: removing hash /var/run/utmp
Oct  4 10:50:08 localhost last message repeated 9 times
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/sis_dri.h
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/sis_driver.h
Oct  4 10:50:08 localhost kernel: dazuko: removing hash /var/run/utmp
Oct  4 10:50:08 localhost last message repeated 5 times
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/sis.h
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/sis_regs.h
Oct  4 10:50:08 localhost kernel: dazuko: removing hash /var/run/utmp
Oct  4 10:50:08 localhost last message repeated 9 times
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/sis_shadow.h
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/sis_vb.h
Oct  4 10:50:08 localhost kernel: dazuko: removing hash /var/run/utmp
Oct  4 10:50:08 localhost last message repeated 44 times
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/vgatypes.h
Oct  4 10:50:08 localhost kernel: dazuko: removing hash 
/var/tmp/portage/xfree-4.3.0-r7/work/xc/programs/Xserver/hw/xfree86/drivers/sis/vstruct.h
Oct  4 10:50:08 localhost kernel: dazuko: removing hash /var/run/utmp
Oct  4 10:50:08 localhost last message repeated 135 times
Oct  4 10:50:08 localhost kernel: dazuko: No READY slot found, 
slotlist=0xdazuko: dazuko_device_release() [28428]
Oct  4 10:50:08 localhost kernel: dazuko: dazuko_unregister_daemon() [-1]

-- 
Sami Tikka                          tel: +358 9 2520 5115
Senior Software Engineer            fax: +358 9 2520 5013
F-Secure Corporation                http://www.f-secure.com/
BE SURE