Submit Your Requirement
Scroll down to discover

Q. Can I crawl any website?

April 13, 2020

We as a crawling company respect robots.txt and crawl a site only if bots are allowed in robots.txt file. If crawling is disallowed in robots.txt, even though crawling might be feasible technically, it involves legal issues for us as well as our clients. Also in cases where bots are allowed and we give data to clients, it is up to clients to conform to the Terms of Service for the usage of that data.

Get The Latest Updates

© Promptcloud 2009-2020 / All rights reserved.
To top