Scraping Product Prices and Reviews

Client’s Background

One of the leading residential furniture manufacturers in Europe, in ready-to-assemble category reached out to PromptCloud for scraping product prices and reviews across customer websites.

E-mail: sales@promptcloud.com

Data Requirement

Being one of the popular furniture manufacturers, our client sells a majority of products online via various eCommerce retailers. Since their products are being sold across different portals, it became imperative that they track how various customer sites are pricing the products, along with the opinions of users/customers.

Proposed Solution: Scraping Product Prices and Reviews

Issue One of the biggest hurdles was the fact that our client didn’t have any URLs of their products on various eCommerce portals but had only the SKU IDs as well as the product names.

Proposed Solution : After requesting for a few sample SKU IDs and Product Names, our team identified that it would be possible to search for exact products by using the site-wise SKU IDs which is most cases returned the desired results.

Since the client wanted control over the input information, we programmed the crawler to scrape product prices and read this data from a shared location (Google Sheets, in this case). Once the product pages were identified, extracting both product related data as well as reviews data was relatively straightforward, irrespective of the fact that some of these websites had a few blocking techniques in place.

Implementation Timeline

As soon as the crawl specifications were finalized, we started with setting up the crawlers and were delivering the data in no time. For 20 websites, which is equal to 40 site setups (20 for products and 20 for reviews), we had the entire setup, up and running in about 7 working days.

Data Consumption

As soon as a particular day’s crawl is completed, the data gets merged into one file per site and is pushed to the client’s FTP folder from where they import the data into their internal systems which automatically tracks the price discrepancies, if any, across the multiple vendors.Our client has also developed sentiment analysis systems to gauge how the users are perceiving their products from the reviews being collected. They find this useful in understanding what is working for them and what needs to be changed.

Project Completion

By using our site crawling service, client made sure that there is no manual layer involved in the data acquisition process, thereby saving up on a lot of man hours, server costs, human resource costs of having a dedicated team and maintenance of the crawler had it been developed in-house.Now, all that they have to do was to consume the data that we deliver as per their specifications.