Web Crawler FAQ

User Agent :

Our User Agent is Webtendencies.com/1.0 -StatsSpider- (+http://www.webtendencies.com/bot.php)

What is StatsSpider ?

StatsSpider is an indexing crawler for our statistic database. The bot tries to categorize and index websites.

Why doesn't StatsSpider visit all pages on my site ?

StatsSpider scan your index page and save informations about all related pages of your website. The crawler will go deeper into your website to find more information by analyzing links and their surrounding text later. The crawler read also your sitemap file, if it exists. Later, the bot will come again to scan others page.

What informations are readed and saved ?

Only structure of the website, title tags and meta tags are saved and indexed in our database. Links and relations between pages are also saved but no content.

What about robots.txt ?

robots.txt is a standard document that can tell crawlers not to index some or all information from your web server as well as a crawl delay of how often a crawler may pull pages. StatsSpider doesn't obey to robots.txt files at this time. It 'll do it later.

How do i stop StatsSpider from indexing my website ?

At this time, you can not, but later, add this to your robots.txt file :

Who can i contact with questions or concerns about StatsSpider ?

Send us an e-mail at info*at*webtendencies.com