Being one of the popular furniture manufacturers, our client sells a majority of products online via various eCommerce retailers. Since their products are being sold across different portals, it became imperative that they track how various customer sites are pricing the products, along with the opinions of users/customers.
Issue One of the biggest hurdles was the fact that our client didn’t have any URLs of their products on various eCommerce portals but had only the SKU IDs as well as the product names.
Proposed Solution : After requesting for a few sample SKU IDs and Product Names, our team identified that it would be possible to search for exact products by using the site-wise SKU IDs which is most cases returned the desired results.
Since the client wanted control over the input information, we programmed the crawler to scrape product prices and read this data from a shared location (Google Sheets, in this case). Once the product pages were identified, extracting both product related data as well as reviews data was relatively straightforward, irrespective of the fact that some of these websites had a few blocking techniques in place.
As soon as the crawl specifications were finalized, we started with setting up the crawlers and were delivering the data in no time. For 20 websites, which is equal to 40 site setups (20 for products and 20 for reviews), we had the entire setup, up and running in about 7 working days.