The client wanted to extract credit card offers and other promotional information from bank websites to fuel their comparison engine. Since the target sites had dynamic and complex coding elements, the crawling project demanded an extensive infrastructure with high-end resources. The client lacked the technical know-how to go about this and wanted a fully-managed service that can take end-to-end ownership of the process. The data was to be extracted on a weekly basis and delivered in a clean and ready-to-use format.
The client shared the specifics of their requirements such as the target sites, crawling frequency, preferred data delivery format and the data fields they wanted to crawl from the sites. This use case comes under our site-specific crawl offering since the websites in the list had different structuring and design. The client wanted the extracted data delivered to their Dropbox account in JSON format.
We set up the crawlers for the target sites in just 3 days and the initial set of data files were delivered to the client. About 10,000 records were delivered to the client during the first crawl.