Block Archive.org in nginxI just learned that Archive.org does not honor robots.txt, so I blocked them in nginx.conf as follows:
123 if ($http_user_agent ~ (ia_archiver|archive\.org)) { return 403;}
I just learned that Archive.org does not honor robots.txt, so I blocked them in nginx.conf as follows:
1 2 3 | if ($http_user_agent ~ (ia_archiver|archive\.org)) { return 403; } |