|
If you’re reading this, chances are you’ve seen the Contextured spider visiting your site while looking through your server logs. Our software obeys robots.txt files and robot META tags in HTML. These are the standard mechanisms for webmasters to tell web robots which portions of a site a robot is welcome to access
The reason Contextured has crawled your site is to provide you with business optimization suggestions. You can find these by logging into http://www.contextured.com
Our spider obeys the robots.txt exclusion standard, described at http://www.robotstxt.org/wc/exclusion.html#robotstxt. If you do not require any optimization assistance you can prevent our spider indexing your site by placing the following in your robots.txt file: User-agent: Contextured Disallow: /
Webmasters/Robots META If you do not have permission to edit the /robots.txt file on your server, you can still tell robots not to index your pages or follow your links. The standard mechanism for this is the robots META tag, as described at http://www.robotstxt.org/wc/meta-user.html
|
