[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Nano-devel] reducing system calls for file io
From: |
Jay Carlson |
Subject: |
Re: [Nano-devel] reducing system calls for file io |
Date: |
Sun, 21 Apr 2002 14:40:52 -0400 |
On Sunday, April 21, 2002, at 12:10 PM, Adam Rogoyski wrote:
On Sat, 20 Apr 2002, Jay Carlson wrote:
Because these kinds of problems affect many applications, there are
libraries that can reduce this overhead. One of them is called libc,
and the subsystem implementing this is available through #include
The correct thing would be to buffer the writes the same way it
buffers
the reads, which your patch removes. If you are really interested in
reducing the number of system calls, you want to use a larger buffer
that
what libc does. I think when I last looked at this libc tended to use
only half of BUFSIZ.
Why write more code?
On Linux, glibc looks up the preferred block size of the file/filesystem
(via stat.st_blksize) and uses that if available, rather than BUFSIZ.
See libio/filedoalloc.c for the code. For read/write files, this is
reasonable behavior. So usually stdio ends up with 4k buffers.
For stuff we're just reading, a larger buffer size might be
appropriate. Fortunately, stdio allows tuning of this behavior. After
call to fdopen in files.c:open_file(), add:
setvbuf(f, my_buffer, _IOFBF, sizeof(my_buffer));
Passing in an 8k buffer restores the previous system call count on
Linux, and this can be set arbitrarily high there. However, some
systems do have limits on setvbuf size (win32's is 32k) and on others,
some amount of accounting data gets skimmed off the top. My feeling is
that monkeying with this is just not worth it. My quick survey of
non-buffer-gap editors shows most of them making 4k reads on Linux.
Hell, /bin/cat uses 4k blocks.
If you *really* wanna cut system calls above all else, just mmap() the
damn file. I forget if any libc implementations out there implement
readonly streams that way, but it's always a possibility.
But libc usually has reasonable defaults for this kind of thing. I
guess my point isn't so much that arbitrarily cutting the number of
system calls is an unqualified good; it's that there are already
mechanisms that other people are maintaining that will solve the problem
well. If files.c used them, it would have done decimal orders of
magnitude fewer write() calls.
Since reading files is done much more that writing files, and writing
to devices it typically much slower than reading, no one probably
noticed
it being slow and no one bothered to change it. Only when reading a
file
are you actively waiting to do something. When writing a file,
typically
your task is done and an extra second doesn't usually bother people.
I agree that you should optimize for common tasks and for those that are
sensitive to the user. But stdio does reasonably well at these kinds of
IO tasks, it's been written already, it's maintained for you, and
because so many apps use stdio its performance will be scrutinized by
many people. Getting rid of code in your apps is a good thing.
Jay