D4Vinci / Scrapling
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Why this job: says: web scraping; topics: web-scraping, playwright, crawler
What it does
- Scrapy-like Spider API. Define spiders with start_urls, async parse callbacks, and Request/Response objects.
- Concurrent Crawling. Configurable concurrency limits, per-domain throttling, and download delays.
- Multi-Session Support. Unified interface for HTTP requests, and stealthy headless browsers in a single spider - route requests to different sessions by ID.
- Pause & Resume. Save pending requests and seen URLs with crawldir. Press Ctrl+C to stop gracefully and restart with the same directory.
- Streaming Mode. Stream scraped items as they arrive via async for item in spider.stream() with real-time stats - ideal for UI, pipelines, and long-running crawls.
Get started
pip install scrapling- Maintenance
- Active
- Last commit yesterday
- Stars
- 100,613
- +518 this week
- Licence
- BSD-3-Clause
- Allows business use
- Downloads for
- None listed
- No release files
- Latest version
- v0.5.1
- Released 6 days ago
- Built with
- Python
- 100% of the code
Is it looked after?
- Contributors
- 16
- Last 90 days
- Issues
- 98% closed
- 3 still open
- Pull requests
- 150 merged
- 17 waiting
- Releases
- 55
- 24 in the last year
- Contributing guide (opens in new tab)
- No security policy
- Code of conduct (opens in new tab)
Similar repos
| Repo | Stars | Last commit | Licence | Downloads for |
|---|---|---|---|---|
| firecrawl / firecrawl | 200,182 | today | AGPL-3.0 | No downloads |
| ScrapeGraphAI / Scrapegraph-ai | 32,668 | 35 days ago | MIT | No downloads |
| alirezamika / autoscraper | 8,080 | 3 months ago | MIT | No downloads |
More: languages, topics, project record and preview image
Languages
- Python 100%
- HTML <1%
- Dockerfile <1%
Topics
- crawler
- crawling
- crawling-python
- playwright
- python
- scraping
- selectors
- stealth
- web-scraper
- web-scraping
- web-scraping-python
- webscraping
Project record
- JobAutomation and scraping
- StatusListed
- Commits1,825
- Forks10,513
- Watching350
- Stars in 28 days+6,755
- CreatedOct 13, 2024
- First seen by usNov 7, 2026
- CheckedNov 10, 2026