CompanyBook Support

Search our database for all the information you need on Companybook

Companybook Crawler

1.   The Companybook Crawler

2.   Introduction

Our crawler visits company websites to download information about the companies’ business. We make the information available to our end users in two ways:
  • Company pages showing a description of the business and contact information for key people.
  • Business search index for our end users to find prospects and potential suppliers.
3.   Crawler identification

3.1   User-Agent

The complete User-Agent of our crawler is:
Companybook-Crawler (+ 

3.2   robots.txt directives  

The crawler follows robots.txt directives for Companybook-Crawler or if not present default (*) directives.

More information about robots.txt is available at

3.3 Subdomains

Our crawler does not support subdomain specific robots.txt files. 

If a.yourdomain.example has a different robots.txt file from yourdomain.example; our crawler will use the one at yourdomain.example for both for the main domain and all subdomains including a.yourdomain.example.

4. Contact information

For questions regarding our crawling; please contact

Feedback and Knowledge Base