Scrapinghub
Scrapinghub Accelerates Next-Generation Web Scraping Service with Confluent Cloud
Each day thousands of companies and more than a million developers rely on Scrapinghub tools and services to extract the data they need from the web. To strengthen its position as a market leader, Scrapinghub recently launched a new product, AutoExtract, that provides customers with AI-enabled, automated web data extraction at scale. Scrapinghub built AutoExtract on Confluent Cloud running on Google Cloud Platform (GCP), with an Apache Kafka®-based, event-streaming backbone for its service architecture. These technologies were chosen to shorten time to market, and to ensure reliability and scalability.
Challenge
Accelerate the delivery of a next-generation web scraping service, capable of handling growing customer demand with no downtime.
Solution
Use Confluent Cloud and Apache Kafka to implement a reliable, scalable event-streaming backbone that links web crawlers with AI-enabled data extraction components.
Results
- Deployment time halved
- Initial setup completed in minutes
- 100% uptime post-launch
- Latencies minimized with no cloud vendor lock-in