Client background One of the leading residential furniture manufacturers in the United States, who sells ready to assemble products that are popular across the country. Ready to assemble furniture is a popular category amongst many households because of the multiple styles and benefits that are available in everyone’s budget.
Requirements background Being one of the popular furniture manufacturers, our client sells a large chunk of products online via various eCommerce retailers. Since their products are being sold across different vendors, it became imperative that they track how various vendors are pricing the products along with the opinions of users/customers.
Customizations & Proposed Solution One of the biggest hurdles was the fact that our client didn’t have any URLs of their products on various eCommerce portals but had only the SKU ids as well as the product names. After requesting for a few sample SKU Ids and Product Names, our team identified that it would be possible to search for exact products by using the site-wise SKU IDs which is most cases returned the desired results. Since the client wanted control over the input information, we programmed the crawler to read this data from a shared location (Google Sheets, in this case). Once the product pages were identified, extracting both product related data as well as reviews data was relatively straightforward, irrespective of the fact that some of these websites had a few blocking techniques in place.
Implementation timeline As soon as the crawl specifications were finalized, we started with setting up the crawlers and were delivering the data in no time. For 20 websites, which is equal to 40 site setups (20 for products and 20 for reviews), we had the entire setup, up and running in about 7 working days.
Data consumption As soon as a particular day’s crawl is completed, the data gets merged into 1 file per site and is pushed to the client's FTP folder from where they import the data into their internal systems which automatically tracks the price discrepancies, if any, across the multiple vendors. Our client has also developed sentiment analysis systems to gauge how the users are perceiving their products from the reviews being collected. They find this useful in understanding what is working for them and what needs to be changed.
Completion By using our site-specific crawls, this client made sure that there is no manual layer involved in the data acquisition process, thereby saving up on a lot of man hours, server costs, human resource costs of having a dedicated team and maintenance of the crawler had it been developed in-house. Now, all that they have to do was to consume the data that we deliver as per their specifications.