Challenge:
Crawling 10-20 sites per month and furnish data to help build database of items/services with relevant information. Also includes database building for car-sharing services. The level of information here is macro: name, photo, owner name, location (Lat. & Long.), price etc. Specific geographical targeting demanded.
The Solution:
Part of the regular site-specific crawling, we ran slow crawls in sites to get the data according to fields. Slow crawl helped get data for this case smoothly.
- Building a crawl and extraction process setup a continuous stream of data
- Helped aggregate content from many sites to enhance client site
- Zero data processing efforts at client’s end
- Scalable infrastructure reduced client’s costs