Submit Your Requirement
Scroll down to discover

Web Crawling – The Backbone of E-commerce Applications

January 22, 2015Category : Blog
Web Crawling – The Backbone of E-commerce Applications

Last Updated on by Consultant

From e-commerce sites crawling their competitors’ or suppliers’ prices, product descriptions and images to site-specific web crawling done by price/product comparison sites and other e-commerce based web apps – we try to explore how online shopping sites exploit data crawling to their advantage.

Ecommerce Crawling bots for price comparison Service

The challenge of running an e-commerce site is further amplified due to the vast number of SKUs that are involved. With millions of products, the associated content such as pricing information, product descriptions, images and reviews can easily go into hundreds of millions for some of the largest e-tailers. Though most sites predominantly use their own data, clicking their own images and filling out the other associated details in-house, some data is obtained by crawling sites of the product sellers or suppliers.

Here are a few use cases where web crawling is useful in e-commerce

Real-Time Monitoring of Competitors’ Prices

Selling products at competitive prices all the time is a really crucial aspect of e-commerce. Some of these sites actually crawl their competitors’ sites to ensure that they’re either able to match the prices being offered over there or offer better prices. It has indeed been proved that pricing optimization techniques can improve gross profit margins by around 10%. Web crawling is also used by travel e-commerce companies to extract prices from airlines’ websites in (near) real time since a long time.

Price Vs Competitor

Price Comparison Shopping Engines

Price Comparison Shopping Engines

For e-commerce companies, customer loyalty is almost non-existent. Today, customers are willing to switch retailers even if the product that they’re looking for is available just 5% cheaper elsewhere. Price comparison sites such as Nextag, PriceGrabber and ShopZilla crawl data from millions of product pages across major e-commerce sites. These comparison shopping engines extract product feeds including all associated details regarding the product (title, description, ratings, images etc.) from multiple sites and juxtapose the products for enabling easy comparison.

Price Comparison

Product Performance Intelligence

In order to stay on top of their direct competitors, nowadays e-commerce sites closely monitor their counterparts. For example, say Amazon would want to know how their products are performing against BestBuy or Walmart, and whether their product coverage is complete. Towards this end, they would want to crawl product catalogs from these two sites to find the gaps in their catalog. They’d also want to stay updated about whether they’re running any promotions on any of the products or categories. This helps in gaining actionable insights that can be implemented in their own pricing decisions. Apart from promotions, sites are also interested in finding out details such as shipping times, number of sellers, availability, similar products (recommendations) etc. for identical products.

For requirements where the number of products involved is rather large; we also perform incremental crawls, wherein sources are crawled relative to how frequently they’re updated. For instance, through our Machine Learning algorithm, we can determine if a product gets a review daily, weekly or monthly (on an average) and the crawler prioritizes the fetching process accordingly.

Order Fulfilment Integration

Lately, some e-commerce sites have started providing package tracking information inside the customer area of their websites, along with the order details (as opposed to providing a third party tracking code). Though some courier companies might have an API for obtaining latest information, for others – their sites are crawled for product transit status frequently, and that data is integrated within the user interface itself or e-mailed to the customer.

Any sentiment analysis done at scale is a result of focused web crawls running in the background. Companies want to find out how their products are performing in the market, and whether their promotional efforts are having a direct impact on their sales. Traditionally, doing this would have incurred significant costs as it would involve asking customers face-to-face through a survey or by conducting a focused group discussion. With the advent of e-commerce, companies are now able to retrieve similar, unsolicited feedback from their customers in the form of reviews and ratings. It really separates the wheat from the chaff for vendors and companies looking for first hand opinions of customers on their products – information that they can incorporate in their product development and R&D almost instantaneously.

Order Fulfilment Integration

Through web crawling, relevant data is delivered to companies by crawling the customer reviews and ratings on a site. Apart from this, many forums are also treasure troves of sentiment data that can be extracted. Sometimes, prominent sellers on marketplace sites are interested in extracting reviews and ratings of their own products listed on those sites. One of our clients was a furniture company having an online sales channel on sites like and The client wanted to receive reviews posted on their products, and PromptCloud.

Fetching Product Information From Suppliers

Marketplace e-commerce websites have thousands of suppliers. Necessary information such as product descriptions, pricing, applying a markup %, images, and specifications needs to be pulled from the suppliers’ websites. The supplier might have feeds or an API, but mostly it is not the case. This is a gap that web scraping fills – by crawling the suppliers’ websites for specific product URLs or SKUs and extracting all the associated fields of a product listing. Doing this manually would cost several man-hours and isn’t really scalable.

A lot of web apps are changing the way images are searched. In order to find a similar product, the user just needs to upload an image to a visual product search engine like this one and it’ll do the rest. At the backend of these sites, there is a database of high-resolution images of products listed on all major e-commerce sites that are used for matching the image uploaded by a user.

Mobile Shopping and Product Delivery check

E-commerce apps are evolving daily, and the application of web crawling and data extraction to e-commerce is also becoming more widespread. Uses that hadn’t been thought of before are emerging constantly and it needs to be seen how much interdependent the two disciplines become over time.

Subscribe to our:


2 thoughts on “Web Crawling – The Backbone of E-commerce Applications
  • BF

    Why should I crawl the web for product data , I mean networks like CJ and ShareASale provide high quality product feed from all th amjor merchants.

    When you crawl the web how do you get the unique identifiers for the products ?

    This approach is not reliable as would be to use product feeds.

  • Vijay

    @BF Every vendor comes with a new set of requirements based on their business needs. PromptCloud offers a customized data crawls based on the client requirements. You can decide your fields to be extracted, frequency, Output format, & Re-crawl schedules.

Leave a Reply

Your email address will not be published. Required fields are marked *

Get The Latest Updates

© Promptcloud 2009-2020 / All rights reserved.
To top