Re: [Mldonkey-users] Feature requst - download cache

[Goswin Brederlow]

Brett Dikeman <address@hidden> writes:

> At 12:48 PM +0100 3/14/03, address@hidden wrote:
> >  > I realy get fed up by all the fragmentation and thus slowdown of the
> >>  fs mldonkey generates.
> 
> This isn't mldonkey.  It's a lack of disk space on your part.

Usually 1-10 G free on a 20G partition.
 
> >  Without defragmenting now and then it reaches a
> >  > critical point where the disk speed is below the download speed.
> 
> This is simply not possible- you must be getting swapping, which is
> entirely different.  Check your swap rate using vmstat during this
> disk activity, I can practically guarantee it's swap.

Its not swap and its not mldonkey. Even without mldonkey running and
just plain cp I get that speed. Its the fragmentation. The read ahead
features of linux and lvm are actually working against one in this
case it seems.

> If you have an IDE drive, have youenabled DMA, unmasked IRQs, 32bit
> IO, and the fastest IDE mode your drive/controller supports?  These
> can cause -significant- improvements in disk IO on some systems.  A
> Dell system I used to use at work improved almost 30x in disk
> benchmarks for sequential reads.

Top speed is 60 GB/s write and 40GB/s reads. Its not the disks.

> You can also enable write caching, if your system is rock solid
> stable; I strongly recommend you at least switch to ext3, however, to
> reduce chances of filesystem corruption if the system does crash/get
> powered down accidentally(note, it is possible to jump between ext2
> and 3, you do not need to reformat- google for something like "ext2
> ext3 tunefs".)

Its ext3.

> Also, I'm not sure of what you mean by "without degramenting now and
> then"- the ext2 defrag utility is pretty much dead.  Is there some new
> utility out?

Yes, ext2 defrag in its new version. Some picked it up and updated it
to the current ext2 state. No ext3 support yet but one can allways
disable the kernel during defragmentation.

> >or use a filesystem without such drawbacks.
> 
> Uh, no.  Regardless of what operating system you use, if you've got
> next to no disk space available, you're going to get fragmentation,
> and there's nothing you can do about it.  Yes, FAT and NTFS are much

yes you can, defragment. Some filesystems do that semself.

> worse than others, but the unix and MacOS filesystems are much better,
> and ext2 doesn't suffer the same problems as FAT/NTFS, so comparing
> them is absurd.

Actually FAT does better due to its larger block size. It has at least
64K continous all the time because thats one block on such sized fs.

> Fact of life; statistically, the less free space, the less of a chance
> the OS will be able to find contiguous disk space.  Some filesystems
> are more clever than others, but the differences are not astounding,
> certainly not the "30x" claimed by one lister- that was due to the
> reformat(which, after you've copied all your data back, results in a
> completely unfragmented filesystem), -NOT- the filesystem type switch!
> 
> 
> Brett
> PS:most journaling filesystems are at least slightly slower than
> non-journaling systems.  We're not talking a drastic or even
> noticeable difference, however, so don't loose sleep over it.

The problem is that downloading 50 files with 500 connects produces
writes at 500 different places in the files. Every 4K a new block will
be allocated per connection. Even with linux allocating different
files in different block groups there are just not enough places to
put the files to avoid realy heavy defragmentation.

Its this completly random filling of spare files that kills the IO
performance.

MfG
        Goswin