Re: [Wget-dev] wget2 | Using --robots=off / --no-robots downloads the robots.txt file and scans it for sitemaps (!454)

[Tim Rühsen]

Tim Rühsen commented:


Please amend the documentation for --robots in docs/wget2.md. Otherwise LGTM 
:-).

> This merge request addresses #456

If you change this into `This merge request closes #456`, then the issue 
becomes auto-closed when we merge this MR (Gitlab feature).

> I have done a clean install after making these changes.

You don't have to. That's what we have the CI runners for. A `make clean && 
make check` is fine as long as you don't see compiler warnings or failing 
tests. For development it is good to turn our manywarnings feature on with 
`touch .manywarnings && ./configure`.

`./bootstrap` and `./configure` is only needed when you touch configure.ac, 
Makefile.am, bootstrap.conf (and possibly some other files).

BTW, I didn't make a clean install since years.

> It appears to me that this was a very quick fix, there might be better ways 
> to do the same.

Yes, some fixes are straight forward. Others needs refactoring and many hours 
of work (I am just at one which should then close several issues).

Anyways, there might always be corner cases that we don't think about now. E.g. 
how robots.txt works with different combinations of options. We might see 
issues in the future, but then we'll fix them and add that kind of things to 
the test suite.

-- 
Reply to this email directly or view it on GitLab: 
https://gitlab.com/gnuwget/wget2/merge_requests/454#note_233725421
You're receiving this email because of your account on gitlab.com.