Re: [Bug-wget] performace issues when saving a http iptv stream on a SD car=

[Bykov Aleksey]

Greetings, Hrvoje Niksic

Yes, at least my Windows is misconfigured (but  write buffer is surely
enabled).
Yes, this problem exist with all programs, that tries to write on flash
drive without themself buffering (at least on Win98, WinXP, Win Vista).

Can You please help with advise?

Results of checking at Win Vista SP2 x86 and Win XP SP3 x86 with three
different flash drives (16 Gb with 8 Kb cluster, 128 Mb and 2 Gb with 4 kb
cluster) is almost same. Sorry, no *nix system in access.
Test file size - 200 Mb
With fflush - 20-40 kb/s. Write it by variable size chunks (1,5 - 12 kb)
(checked by procmon). First Mb at full speed, then quick drop.
Without fflush - 300-450 kb/s. Write it by 4 kb chunks
Without fflush with additional 8Mb buffer - 600-650 kb/s. Write it by 256
kb chunk.
As default in MinGW BUFSIZ = 512.

Current variants, that i see (at current data set. Need more data from all
system...):
0. Leave it as is.
1. Simple add Windows to exception (to not fflush) (as currently VMS).
2a. Add user option to disable flush. As default - current behavior. In
documentation notate, that possible data loss and delays.
2b. Add user options to determine buffer size. 8Mb buffer - is overkill,
it's enough 8-64 Kb.
Side effect of second variant - converting links during breaking recursive
downloads by Ctrl-C ;-)

What more variants to solve this inconvenience is exist?
What variant brings more advantages?

Thanks in advance.

--
Best regars, Alex

> > I am really curious what the rationale behind the fflush was.
>
> The rationale was the data is not of much use to anyone while lingering
> in Wget's stdio buffer.  No one else sees it, and if you only press ^C,
> you lose it permanently.  Wget is not asking for the data to be
> immediately written out to disk, it is merely handing it off to the
> operating system, so other programs (be it tail -f or a locally running
> browser) can see it and put it to good use.  fflush != fsync.
>
> Plus, typical stdio buffer is smaller than Wget's network read buffer,
> so most of the data would have been flushed anyhow.  The fflush() only
> makes sure that by the time Wget announces that it has downloaded (say)
> 123,456 bytes of data, your file contains no *less* than 123,456 bytes.
>
> If some misconfigured system were really neglecting to cache filesystem
> writes, then it would have problems with most programs out there, not
> just Wget.  After all, BUFSIZ on a typical system is only 4-8K.