bug-wget
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Bug-wget] Special -i- to read one URL at a time


From: Joel Nothman
Subject: [Bug-wget] Special -i- to read one URL at a time
Date: Thu, 01 Nov 2012 17:09:10 +1100
User-agent: Opera Mail/11.61 (MacIntel)

wgetting a list of URLs on STDIN requires reading the entire file into memory before extracting a linked list of URLs into memory. I.e. the STDIN needs to have EOF before twice the input's memory is spent to store its URLs.

Among other things, this means wget can't deal with infinite URL lists:

$ yes http://example.com | wget -i-

I've patched wget to special-case reading from STDIN in retrieve_from_file.

Unfortunately my C's a bit rusty, and I wasn't sure how to create an appropriate set of tests.

See https://github.com/jnothman/wget (73e9da29)

Cheers, and thanks for a great util!

- Joel



reply via email to

[Prev in Thread] Current Thread [Next in Thread]