I noticed last night that I was showing 6 guests on the forums. I know we have almost 29,000 posts for them to read, but this morning they still appear to be browsing. Wierd thing is that the 'last click' log shows them as active very recently, hmmmm - do we really have a large number of guests browsing all the time?
TL.
At the moment for example:
Online Users
Member Name Location Time
TeaLeaf ( XX.XX.XX.XX) Viewing Online List Sep 22 2003, 07:30 AM
Sadako ( XX.XX.XX.XX) Viewing Topic: Tactical Counter-strike Sep 22 2003, 07:29 AM
Guest ( 66.196.72.87 ) Viewing Board Index Sep 22 2003, 07:29 AM
Guest ( 66.196.72.86 ) Viewing Board Index Sep 22 2003, 07:29 AM
Guest ( 66.196.72.75 ) Viewing Board Index Sep 22 2003, 07:28 AM
Guest ( 66.196.72.30 ) Viewing Board Index Sep 22 2003, 07:28 AM
Guest ( 66.196.72.90 ) Viewing Board Index Sep 22 2003, 07:26 AM
Guest ( 66.196.72.11 ) Viewing Board Index Sep 22 2003, 07:26 AM
Guest ( 66.196.72.33 ) Viewing Board Index Sep 22 2003, 07:25 AM
btw, I put the XXs in Sadako's and my IP. But it looks to me like we have a spurious result here from the logs. Wassup?
TL.
I did a tracert and found this with one of the IPs
j3101.inktomisearch.com [66.196.72.11]
inktomisearch.com doesn't exist as a homepage though. :unsure:
Inktomi is a search spider. You can exclude spiders from the forums by using a "robots.txt" file - loads of info about this on Google..
DM
Think I saw something in the forum configs to do with spiders. I'll check it out later.
QuoteOriginally posted by DogMeat@Sep 22 2003, 09:01 AM
Inktomi is a search spider.
and are search spiders bad?
TL.
If they are, I can't find anything on Google about it... :lmfao:
DM
QuoteWhat are Search Engine Spiders?
Search engines such as Google 'spider' the web by using special programs to find and add links to the search engine database.
Invision Power can take advantage of separating the bots from the real users and you can then ensure that they are getting adequate information to help with search engine ranking.
Warning!
Invision Power Board recognises the search engine spiders and crawlers by their user_agent. Please keep in mind that it's not impossible to fake this and a malicious user could pose as a harmless search engine spider.
This is not a problem as by default the spider will have guest permissions, but it's something that you should keep in mind when allowing permissions.
Also keep in mind that most search engines do NOT search dynamically generated website for fear of crashing the server with the constant reading of topics.
I've made some changes which *should* mean that the spiders will be identified.