1. The Companybook Crawler
Our crawler visits company websites to download information about the companies’ business. We make the information available to our end users in two ways:
- Company pages showing a description of the business and contact information for key people.
- Business search index for our end users to find prospects and potential suppliers.
The complete User-Agent of our crawler is:
3.2 robots.txt directives
The crawler follows robots.txt directives for Companybook-Crawler or if not present default (*) directives.
More information about robots.txt is available at http://www.robotstxt.org/orig.html.
Our crawler does not support subdomain specific robots.txt files.
If a.yourdomain.example has a different robots.txt file from yourdomain.example; our crawler will use the one at yourdomain.example for both for the main domain and all subdomains including a.yourdomain.example.
4. Contact information