Location based Data Mining for Cataloging

Being an e-commerce platform, the client sought complete product and pricing data from its outlets spread across the United States. Before looking to employ web scraping services, the client used to gather relevant data manually to perform the analysis.

The requirements included extracting product data from their e-commerce platform and retail store outlets across the nation, filtered based on zip codes of store locations. Taking the above-mentioned location based data mining, a similar data collection process had to be performed on competitor e-commerce sites to extract price and product data.

This data was used for further analysis in product strategy and price benchmarking for a complete product catalog. The client wanted to deploy PromptCloud’s web scraping service to automate the entire data extraction process based on location, zip codes in particular.

Site-specific crawls were employed in this case, which focused on the client’s website. The solution extracted pre-defined data points from the client’s website; important data fields were the unique serial identifier of a product, product name, category, URL link, crawling timestamp, store location, price, and inventory stock availability.

Considering the client’s interest in pricing benchmarking, web scrapers were also created for the competitors’ sites. Crawlers collected data from fields such as the unique identifier of a product, URL link, product name, category, crawl timestamp, store, location, price, stock availability in the inventory.

The collected data from the above two executions were then classified by zip codes for location, and was used by the client for further analysis. The dataset was delivered to the client in JSON format via PromptCloud’s REST API