Submit Your Requirement

Download Web Data Acquisition Framework

Did you know that there are 12 factors to be considered while acquiring data from the web? If no, fret not! Download our free guide on web data acquisition to get started!



Questions on pricing, ETAs, SLAs and more.

Scroll down to discover

Q. What is the maximum frequency you can crawl data at?

Frequency would depend on your specific requirements. We can extract the data at a frequency ranging from a few minutes to once in a month.

Q. Can we get contextual information from the web page?

It would depend from site to site. However, we should generally be able to provide you with the preceding URL from where we discovered the final page URL.

Q. How do you maintain your code in order to deal with website structural changes?

While setting up crawlers, we setup automated check points to monitor structural changes. In case a site changes its structure, we would be notified and shall fix them accordingly.

Q. How do we access data on our side?

The data can be delivered in XML, JSON or CSV format. The default mechanism for delivering data is via our RESTful API. We can also push the data to one of your file sharing servers (FTP, SFTP, Amazon S3, Dropbox, Gdrive, Box or MS Azure). If you’re not very technically inclined, you can simply use the one-click data download option on CrawlBoard.

Q. Do you have an IP rotation service?

Yes, our platform, by default, handles IP rotation and mechanism to handle other common blocking issues.

Q. What kind of infrastructure do you offer?

As a client, you’d have access to our portal – CrawlBoard. This would be your centralized portal for technical support, billing and keeping a tab on the crawler activities and stats. You’d also be able to schedule ad-hoc crawls for the future. Error handling happens via our ticketing system.

Q. Do you offer volume discounts?

Yes! In fact, our offerings are also classified based on volumes. We will also be happy to work out attractive discounts in case your monthly data volumes are expected to be in millions.

Q. Is the multiple delivery option free of cost?

No, for customized delivery mechanisms (FTP, Amazon S3, Dropbox or Box), there will be an additional fee of $30 per month. Our default delivery mechanism is via PromptCloud Data API, which is free.

Q. What are your clauses for discontinuing an agreement?

In most cases, we expect you to notify at least a month in advance to release your project-specific resources. Each contract has a specific term with termination and renewal clauses.

Q. Can I crawl any website?

We as a crawling company respect robots.txt and crawl a site only if bots are allowed in robots.txt file. If crawling is disallowed in robots.txt, even though crawling might be feasible technically, it involves legal issues for us as well as our clients. Also in cases where bots are allowed and we give data to clients, it is up to clients to conform to the Terms of Service for the usage of that data.

Q. Can you crawl sites that block your IP?

Yes, we have system components in place to overcome IP blocking.

Q. Can you crawl sites that disallow bots?

No. We respect the robots.txt and crawl a site only if bots are allowed in robots.txt file.

Q. Is crawling legal?

Yes. For a more convincing answer, read our blog post here.

Q. What’s the billing frequency?


Q. Do you have any referral program?

We have a generous referral program for all of our existing customers. Get up to $100 credit for every friend you successfully refer and use that to pay for our data solutions.

Q. How is the monthly bill calculated based on volumes?

Monthly bill is calculated based on the crawling frequency and data volume. Eg: If we are crawling a site on weekly basis and deliver, say, 50,000 records in a month, cost for this site on the monthly bill would be: $15 (for volume fee) + $79 (towards site maintenance & monitoring fee) = $94 for the month. Note – this is assuming volume fee to be $5 per 10k records (prorated) and monthly site maintenance & monitoring fee as $79/site.

Q. What does the monthly site maintenance & monitoring fee cover?

The monthly maintenance and monitoring fee covers technical support, overheads in maintaining the data pipeline and related infrastructure as well as fixing the crawlers in case a target site undergoes structural changes.

Q. Does the frequency of crawls change pricing?

Yes, our pricing plan is based on crawling frequency. Volume charges may increase based on the number of records we deliver, which may be directly related to the crawl frequency.

Q. Can I dynamically add or remove sites?

Yes, new sites can be added any time. Just that the newly added sites will incur their own setup fee and time for setting up crawlers.

Q. What payment modes do you use?

We accept all the major credit cards.

Q. What are your different pricing schemes?

Our default pricing structure can be found here: However, we may be able to group similar websites and setup crawlers. Hence, there is a possibility of offering a different pricing model for you.

Q. Can you give us a demo?

Ours is a custom solution and do not have a specific software that can be demonstrated. The final deliverable would be data files in a format that you may specify. The best we could do is to share sample data from past projects that are similar in nature.

Q. Can we run a proof of concept to evaluate your offerings?

In order to provide a proof of concept, we’ll have to setup the crawlers in its entirety, which is a key step in the whole process. Hence, this will be a paid engagement. We provide 30-days paid PoC for a maximum of up to 2 sites.

Q. How do you differ from other providers?

We basically deal with large-scale data and operate on Data as a Service (DaaS). So you do not have to be involved in any of the set-up or monitoring and we take care of end-to-end data delivery. Our solution has been quite useful for clients who wanted to scale with data and had issues both crawling at that scale and then converting unstructured to structured. Other than that, we’ve a pretty low turnaround time and our set-up is capable of uploading data every few minutes from a site that’s more active.

Ready to share your requirements?

Request Information
© Promptcloud 2009-2020 / All rights reserved.
To top