[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: stat() order performance issues
From: |
Philip Rowlands |
Subject: |
Re: stat() order performance issues |
Date: |
Fri, 26 Jan 2007 02:54:19 +0000 (GMT) |
On Thu, 25 Jan 2007, Phillip Susi wrote:
I have noticed that performing commands such as ls ( even with -U )
and du in a Maildir with many thousands of small files takes ages to
complete. I have investigated and believe this is due to the order in
which the files are stat()ed. I believe that these utilities are
simply stat()ing the files in the order that they are returned by
readdir(), and this causes a lot of random disk reads to fetch the
inodes from disk out of order.
My initial testing indicates that sorting the files into inode order
and calling stat() on them in order is around an order of magnitude
faster, so I would suggest that utilities be modified to behave this
way.
If I shared the same filesystem type, kernel, and I/O subsystem then I
would likely be able to reproduce these results, but...
It's not really possible for the kernel to help by re-ordering the
requests to disk, as the system calls are issued serially.
Although there may be cases where "stat in inode order" doesn't help,
I'm struggling to see how it could be worse than "stat in directory
order", especially for high-churn directories. Are there filesystems
which re-order directories to optimize access patterns? At a quick
glance, I see nothing in POSIX which disallows this.
Cheers,
Phil