April 27, 2010 at 12:50 pm
filed under Uncategorized
Wget obeys robots.txt files. And that’s why I’ve spend the last 45 minutes trying to figure out why it didn’t download an Apache directory of images. It says so in the first paragraph of the man page. One day I will learn.
wget -e robots=off -r http://foo...
Just saying.
Comments are closed.
no comments
RSS /