# FAQ's

FREQUENTLY ASKED QUESTIONS

# Web scraping questions, answered​

 How managed web scraping works, what it costs, where the legal lines sit, and how the data arrives on your side. - [How it works](#how-scraping-works)
- [Managed vs. DIY](#managed-vs-diy)
- [AI and LLM data](#ai-llm-data)
- [Legal](#legal-ethical)
- [Pricing](#pricing-billing)

## What is web scraping, and how it works

The difference between extraction and crawling, and how the managed pipeline runs end to end.

###    [What is web scraping, and how is it different from web crawling?](#) 

Web scraping is the process of extracting specific data, like prices, listings, or reviews, from web pages and turning it into a structured format such as JSON or CSV. It is also referred to as data extraction or data harvesting, different names for the same underlying process. Web crawling is the step before that: discovering and navigating pages across a site or the web. In practice, most projects need both. A crawler finds the pages, and a scraper pulls the data out of them.

###    [How does PromptCloud's web scraping service work?](#) 

PromptCloud handles the entire pipeline for you: crawler setup, ongoing monitoring, data cleaning, and delivery, so your team never touches infrastructure or code. You tell us what data you need and how often, and we deliver it in the format and location you choose. [See how the managed service works.](https://www.promptcloud.com/how-it-works/)

###    [What is the maximum frequency you can crawl data at?](#) 

Crawl frequency ranges from once every few minutes to once a month, depending on your requirements. Higher frequency is common for pricing and inventory data, while monthly crawls suit slower-moving data like company directories.

###    [How do you handle it when a website changes its structure?](#) 

We set up automated checkpoints on every crawler that detect structural changes on the target site. When a site changes its layout, our team is notified and fixes the crawler, so your data delivery isn’t interrupted.

###    [What kind of infrastructure and support do I get as a client?](#) 

You get access to CrawlBoard, our client portal for technical support, billing, and monitoring crawler activity and stats. You can also schedule ad-hoc crawls and submit support tickets directly through the portal.

###    [How do we access the data on our side?](#) 

Data is delivered in XML, JSON, or CSV format, most commonly through our RESTful API. We can also push data directly to your preferred storage, including FTP, SFTP, Amazon S3, Dropbox, Google Drive, Box, or Microsoft Azure. If you’d rather not deal with any of that, CrawlBoard has a one-click download option.

###    [Do you have an IP rotation service?](#) 

Yes. IP rotation is built into the platform by default, along with handling for other common blocking mechanisms.

###    [Can you crawl sites that actively try to block your IP?](#) 

Yes. We have infrastructure in place specifically to work around IP blocking as it happens, so a single block doesn’t interrupt your data delivery.

## Managed service vs. DIY tools and APIs

 Where a managed pipeline differs from a scraping toolkit, and what your team still owns either way. ###    [What's the difference between a managed web scraping service and a DIY scraping tool or API?](#) 

A DIY tool or API gives you the building blocks, proxies, a scraping engine, maybe some templates, but your team still has to build, monitor, and fix scrapers as sites change. A managed service like PromptCloud takes on that entire workload, so you receive finished, structured data instead of a toolkit. We’ve laid out the full cost comparison in [In-House vs. Outsourcing Web Scraping](https://www.promptcloud.com/blog/in-house-vs-outsourcing-web-scraping-a-cost-benefit-analysis-by-promptcloud-experts/).

###    [Do I need any technical or coding expertise to use PromptCloud?](#) 

No. PromptCloud is a fully managed service, so there’s no infrastructure to set up and no code to write on your end. You define your data requirements, and our team builds and runs the crawlers.

###    [Can we run a proof of concept before committing?](#) 

Yes. A proof of concept requires us to build the crawlers in full, since that’s the core of the work, so it runs as a paid 30-day engagement covering up to two sites. This lets you evaluate real output before a longer commitment.

###    [Can you give us a demo?](#) 

Since PromptCloud delivers a custom data solution rather than off-the-shelf software, there’s no generic demo to walk through. What we can do is share sample data from past projects similar to yours, so you can see the actual output quality.

###    [How is PromptCloud different from other web scraping providers?](#) 

PromptCloud operates on a Data-as-a-Service model, meaning you’re not involved in setup or monitoring at any stage, we handle end-to-end delivery. This has proven especially valuable for businesses that need to scale data collection and have struggled with both the crawling and the unstructured-to-structured conversion at that scale, backed by fast turnaround times and near-continuous upload capability for high-activity sites.

###    [Do you offer enterprise-level plans or dedicated support?](#) 

Yes. Enterprise engagements come with a dedicated account manager and support built around the scale and structural complexity that our larger clients need. If your data requirements span many sites, multiple regions, or tighter compliance needs, this is typically the right starting point rather than a standard plan.

## Web scraping for AI and LLM-ready data

 How delivered data is structured for model training, grounding, and downstream analytics. ###    [Can PromptCloud data be used to train AI models or power LLM applications?](#) 

Yes. Clean, structured web data is one of the most common inputs for training and grounding AI and machine learning models, and our delivery pipeline is built to produce exactly that: deduplicated, normalized, and formatted for direct use. See how this plays out in practice in [Perplexity AI vs. Web Scraping Services](https://www.promptcloud.com/blog/perplexity-ai-vs-web-scraping-services/).

###    [How do you make sure the data is clean enough for analytics or machine learning use?](#) 

Every dataset goes through automated quality checks plus human QA before delivery, so you’re not receiving a raw dump that needs further cleaning on your end. Duplicates are removed, formats are normalized, and the data is structured according to your specific requirements before it reaches you.

## Legal, ethical, and robots.txt questions

 What we will and won't crawl, and where responsibility sits once the data is delivered. ###    [Is web scraping legal?](#) 

Yes, scraping publicly available data is generally legal. The specifics depend on what data you’re collecting and how, so read our full breakdown in [Is Web Scraping Legal?](https://www.promptcloud.com/blog/is-web-scraping-legal-in-us-a-complete-guide/)

###    [What is robots.txt, and why does it matter for web scraping?](#) 

robots.txt is a text file a website publishes to tell automated crawlers which parts of the site they’re allowed to access. It’s the first thing any responsible crawler checks before touching a page, and it’s the practical line between compliant scraping and scraping that creates legal exposure, which is why we check it on every project before writing a single line of crawler code.

###    [Is web scraping ethical?](#) 

Yes, when it’s done the way we do it: collecting only publicly available data, respecting robots.txt, and not overloading the target site with requests. Ethics and legality are related but separate questions, a scrape can be technically legal but still strain a site’s servers or ignore its stated access rules, so we treat both as requirements rather than treating legal compliance as the whole answer.

###    [Can you crawl any website?](#) 

No, not if the site’s robots.txt file disallows it. We respect robots.txt on every project, since crawling against those instructions creates legal exposure for both us and our clients, even where it might be technically possible.

###    [Can you crawl sites that disallow bots in their robots.txt file?](#) 

No. If a site’s robots.txt disallows crawling, we don’t crawl it, regardless of technical feasibility.

###    [Who's responsible for how the delivered data is used?](#) 

Once data is delivered, using it in line with the source website’s terms of service is the client’s responsibility. We only crawl sites where bots are permitted under robots.txt, but how you subsequently use that data falls under your organization’s compliance obligations.

###    [What are the terms for discontinuing an agreement?](#) 

Each contract has its own specific term with defined termination and renewal clauses. In most cases, we ask for at least a month’s notice so project-specific resources can be released properly.

## Pricing and billing

 How invoices are calculated, what the maintenance fee covers, and what adding a site costs. ###    [What's the billing frequency?](#) 

Billing is monthly, based on the crawl frequency and data volume set up for each site, so your invoice reflects actual usage rather than a flat subscription fee.

###    [How is the monthly bill calculated?](#) 

Your monthly bill combines a volume fee based on records delivered and a flat site maintenance fee. For example, a site crawled weekly that delivers 50,000 records in a month would cost $25 in volume fees (at $5 per 10,000 records) plus a $79 monthly maintenance and monitoring fee, for $104 total.

###    [What does the site maintenance and monitoring fee cover?](#) 

This fee covers technical support, data pipeline infrastructure, and fixing crawlers when a target site changes its structure. It applies per site, regardless of how much data that site delivers in a given month.

###    [Does crawl frequency affect pricing?](#) 

Yes. Pricing is tied to crawl frequency, and since more frequent crawls typically deliver more records, volume charges can rise accordingly.

###    [Can I add or remove sites at any time?](#) 

Yes, new sites can be added at any time, though each new site carries its own setup fee and setup time.

###    [Is there an extra cost for custom delivery methods like FTP or S3?](#) 

Yes, custom delivery mechanisms such as FTP, Amazon S3, Dropbox, or Box carry an additional $30 per month. Delivery through our default PromptCloud Data API is free.

###    [Do you offer volume discounts?](#) 

Yes. Pricing is tiered by volume, and we offer discounts for clients expecting monthly data volumes in the millions.

###    [What payment methods do you accept?](#) 

We accept all major credit cards. If your organization needs a different arrangement, that can be worked out as part of your contract.

###    [Do you have a referral program?](#) 

Yes. Existing customers get up to $100 in credit for every successful referral, which can be applied toward your data solutions.