It seems that some one is after one of the site i manage. I started to get huge amount of resources eaten up by a bot. As per awestats coming with the hosting, it shows as:
Unknown robot (identified by empty user agent string)
I can see the one for google, yahoo or alexa which is fine. I also see 2 as follow:
Unknown robot (identified by 'bot*')
Unknown robot (identified by 'crawl')
But these last two are not crawling too much.
I suppose i can just add 'crawl' and 'bot*' to the list of bots to block in the htaccess maker. But how can i add identified by empty user agent?
I started blocking the IP it was from, but obviously they switch IPs. I then started blocking an IP range. But they changed ISP.
I can not continue blocking everything or i will loose too much "real" traffic since they are coming from the same country as my target audience.
Is there some tricks or tips to harden bot crawling the site? And how can i block bots identified by empty user agent?
Thank you in advance.
- Which documentation pages did you read?
- most of the suggested results
- Which troubleshooter articles did you read?
- Have you searched the tickets before posting?
- Joomla! version (in x.y.z format)
- PHP version (in x.y.z format)
- MySQL/database version
- Host (who is hosting your site, not your domain)
- Admin Tools version (x.y.z format)