[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Bug-wget] does wget honor robots meta tag?
From: |
Allan Spiegel |
Subject: |
[Bug-wget] does wget honor robots meta tag? |
Date: |
Fri, 14 Nov 2008 17:00:34 -0500 |
User-agent: |
Thunderbird 2.0.0.17 (Windows/20080914) |
I have some pages with
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
in the <head> section and when I crawl my site to make sure I have this
tag in all the right pages, wget gets these pages. does wget support
this tag?
here's the command I used:
wget --delete-after --cache=off --no-verbose --recursive --level=8
--no-directories --no-host-directories --no-check-certificate
--no-parent --page-requisites -U "Safari/41-wget_link_testing"
http://my.domain.com/folder/
wget -V
GNU Wget 1.11.4
If wget doesn't support this can someone suggest another way to verify
these tags are working properly?
Thanks
Allan
- [Bug-wget] does wget honor robots meta tag?,
Allan Spiegel <=