[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Bug-wget] --page-requisites and robot exclusion issue
From: |
Giuseppe Scrivano |
Subject: |
Re: [Bug-wget] --page-requisites and robot exclusion issue |
Date: |
Sun, 04 Dec 2011 22:33:36 +0100 |
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/24.0.92 (gnu/linux) |
address@hidden writes:
> But in cases where you *are* recursively downloading and using
> --page-requisites, it would be polite to otherwise obey the robots
5B> exclusion standard by default. Which you can't do if you have to use -e
> robots=off to ensure all requisites are downloaded.
it seems a good idea to handle -r and --page-requisites in this case,
wget shouldn't obbey the robots exclusion directives.
Thanks,
Giuseppe