diff options
| author | Jan Nieuwenhuizen <janneke@gnu.org> | 2013-08-08 08:34:12 +0200 |
|---|---|---|
| committer | Phil Holmes <mail@philholmes.net> | 2013-08-08 11:07:44 +0100 |
| commit | ca4e2659497e5268abf9b17c8fd9756ee8a3347a (patch) | |
| tree | e7038b084b03e86a85d103a8379f9ad10ef36235 | |
| parent | 6e8698dcb9a9b9a98d8b1a644c84fcb737f99bdc (diff) | |
Deny rogue crawler AhrefsBot. Fixes web load.
The AhrefsBot is checking crawling files every second, including
all binaries. This increases the load in an unacceptable way.
| -rw-r--r-- | Documentation/web/server/lilypond.org.htaccess | 4 |
1 files changed, 4 insertions, 0 deletions
diff --git a/Documentation/web/server/lilypond.org.htaccess b/Documentation/web/server/lilypond.org.htaccess index b4780a9414..5e7dfae3d9 100644 --- a/Documentation/web/server/lilypond.org.htaccess +++ b/Documentation/web/server/lilypond.org.htaccess @@ -23,6 +23,10 @@ RewriteEngine On RewriteCond %{HTTP_USER_AGENT} httrack [NC] RewriteRule ^.*/source/.*$ /please-respect-robots.txt.html [L] +# Deny rogue crawler +RewriteCond %{HTTP_USER_AGENT} ^(.*)AhrefsBot(.*) [NC] +RewriteRule .* - [F,L] + # Permanent top level entry points -- ./doc RedirectMatch ^/music-glossary /glossary RedirectMatch ^/tutorial /learning |