The client shared the detailed requirements including the target sites, crawling frequency, their preferred data delivery format and the data points they wanted to extract from these sites. This use case comes under our site crawling service since the websites in the list had different structuring and design. The client needed the extracted data in JSON format and was ready to use the PromptCloud API to access the extracted data at their end.
As per their instructions, the different target sites had to be crawled at different frequencies, including twice a day, fortnightly and daily. Our team completed the web crawler set up for the three target sites in just five days and the initial set of data files was delivered to the client. About 2.5 million records were delivered to the client during the first webcrawl, solving their hotels price match issue.