A popular Brazilian media website needed web content extraction across blogs, news sites, forums, and bookmarking platforms, delivering more than 300,000 fresh records every day to power its news portal launch.
Client Overview
A popular media website in Brazil needed content extracted on a continuous basis from Brazilian news sites to power its own news portal, and the source list went well beyond a handful of major outlets. Blogs, news sites, forums, and content bookmarking sites all fed into what the portal needed, which meant web content extraction had to work reliably across a genuinely diverse mix of site types, not just a uniform set of traditional news publishers.
News content also loses value fast. A story worth aggregating today is often stale by tomorrow, so the media website needed this running on a daily cadence rather than a periodic pull, with fresh data arriving every single day to keep the news portal current. That combination, source diversity plus daily freshness, is what brought the media website to PromptCloud.
Client Requirements
The media website’s brief to PromptCloud specified both the sources and the exact fields needed:
- Continuous extraction from Brazilian blogs, news sites, forums, and content bookmarking sites
- Date of publishing, author name, title, main text content, and tags for every piece
- Daily crawl frequency, since fresh data sets were needed every single day
- Delivery in XML format, uploaded directly to the media website’s Dropbox servers
- A setup capable of handling more than 300,000 records a day
Challenges
Extracting from blogs, news sites, forums, and content bookmarking sites all at once meant no single template was going to work across the whole source list. Each site had its own structure and design, and web content extraction at this scale meant building extraction logic tuned to each source individually rather than assuming one generic approach would hold up everywhere.
The daily frequency raised the stakes further. News data ages quickly, and a crawl that fell behind even by a day would have handed the media website’s readers stale content next to competitors publishing current stories. Getting full text content, not just headlines or links, out of that many differently structured sources on a daily basis was the real technical demand behind this project.
Solutions
PromptCloud treated this as a site specific crawl and extraction project, building a dedicated approach for each source rather than forcing a uniform template across blogs, news sites, forums, and bookmarking sites alike.
Site Specific Crawls Across Every Source Type
Since each site in the list had a different structure and design, PromptCloud built site specific crawl and extraction logic for every source rather than one generic scraper stretched across blogs, news sites, forums, and content bookmarking sites alike. That approach is core to PromptCloud’s broader web scraping services, since content this varied in format and structure rarely yields to a single template no matter how well built. Getting the source-by-source approach right up front is what made daily extraction across this many site types actually reliable.
Five Fields Captured From Every Piece of Content
Every extracted piece carried date of publishing, author name, title, main text content, and tags, giving the media website’s news portal enough structure to organize and display content properly rather than a bare link and headline. Capturing the full main text, not just a summary or excerpt, mattered directly to a news portal built to be a genuine content destination rather than a thin aggregator sending readers elsewhere for the actual article.
Daily Delivery at More Than 300,000 Records
Once the crawlers were set up, data started flowing in, cleaned, formatted, and uploaded to the media website’s Dropbox servers in XML format every day. The number of records delivered per day ran above 300,000, a volume that reflected just how many blogs, news sites, forums, and bookmarking sites were being tracked simultaneously across the full source list.
Live in Three Days
The initial setup was completed in just 3 days, after which the supply of data stayed consistent day after day. That speed meant the media website could move from source list to a live, functioning content pipeline almost immediately, giving it what it needed to launch the news portal on short notice rather than waiting weeks for a slower buildout.
News Content Pipeline, Before and After PromptCloud
| Area | Before | After |
| Source handling | Risk of one template across very different site types | Site specific crawl and extraction per source |
| Content depth | Risk of headlines or links only | Full main text plus date, author, title, and tags |
| Frequency | Not established | Daily, with fresh data every day |
| Volume | No existing pipeline | 300,000+ records delivered daily |
Benefits to the Client
The media website launched its news portal backed by a content pipeline that never touched its own team on the technical side, every part of the crawling process, from site specific extraction to daily delivery, sat with PromptCloud. Monitoring set up for each site kept quality and consistency high even across a source list mixing blogs, news sites, forums, and bookmarking platforms.
Handling more than 300,000 records a day meant the news portal never ran short of fresh content, and the dynamic coding practices used by some source sites never slowed delivery down. Running this as a managed engagement cost well below what an equivalent in-house crawling setup would have required, letting the media website launch on short notice rather than a longer internal build.
Web Content Extraction That Holds Up Across Blogs, News Sites, and Forums Alike
A news portal pulling from blogs, news sites, forums, and bookmarking sites cannot run on one generic template applied everywhere. Web content extraction at this scale needs a site specific approach for every source, full text captured alongside the metadata that actually matters, and delivery fast enough to keep pace with news that ages within a day.
That is what turned a source list this varied into a dependable daily feed of more than 300,000 records, live within three days and ready for a news portal that needed to launch on short notice.
