Last week, after the site upgrade we were hit pretty hard by bots, crawlers and spiders while they tried to re-index all of the site content. Luckily things seemed to have calmed down a lot since then.
As a means for keeping tabs on what bots are online when and if we get hit hard again I have created a simple module to log and display visits by robots. If you are logged on and there are some bots on line you will see a 'bots online' box on the left hand side. It provides a simple visual of how many nodes (pages) are being visited by bots over the previous 15 minutes.
The module only detects a few of the popular bots and is not meant to be 100% accurate. In time I can use the information logged to block out some bad bots or refine the module.
For now it's just interesting to see what bots are visiting.
#CSSCreator .com {
# Unknowen bots
This is a great idea.
ubo
Maybe I should have called them UBO unidentified bot objects.
I have set the time delay to 20 seconds between page visits for bots that follow that directive and filtered out some of the bad bots in htaccess.
Over the last week we have had at least 17691 page views by bots.
Ah... that's possibly why I
Ah... that's possibly why I was having a hard time accessing a lot of the forum. It was just taking forever!
cool module
Nicely done! Would you be willing to provide the source to the Drupal community?
Best,
Jason.
module
Hi Jason,
I will happily make it available to the drupal community.
When time becomes my friend
I like this bots thing
I am from 2-3 days member on this site but i have already learned some interesting things.
This bots things i have learned after i saw the online status and i clicked on the google bot and learned how to give access or deny access for bots.
Currently i am working on a portal which will be written in css, css/html compliant template, php code, mysql database based.
This site's link definitely will be on my portal!