unclecode / crawl4ai
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Why this job: says: web scraping; topics: web-scraping, playwright, crawler, scraper, web-crawler
What it does
- Clean Markdown. headings, lists, tables and code blocks, in a structure an LLM reads well.
- Fit Markdown. filters remove menus, footers and boilerplate: PruningContentFilterLXML, BM25ContentFilter (for a query) and LLMContentFilter.
- Citations. page links become a numbered reference list.
- Your own strategy. plug in a custom Markdown generator.
- CSS and XPath schemas. fast extraction with no LLM (JsonCssExtractionStrategy, JsonXPathExtractionStrategy, RegexExtractionStrategy).
Get started
No single install command. Read the steps in the README (opens in new tab)
- Maintenance
- Active
- Last commit 5 days ago
- Stars
- 87,122
- +346 this week
- Licence
- Apache-2.0
- Allows business use
- Downloads for
- None listed
- No release files
- Latest version
- v0.9.5
- Released 26 days ago
- Built with
- Python
- 99% of the code
Is it looked after?
- Contributors
- 11
- Last 90 days
- Issues
- 95% closed
- 57 still open
- Pull requests
- 294 merged
- 264 waiting
- Releases
- 23
- 13 in the last year
Similar repos
| Repo | Stars | Last commit | Licence | Downloads for |
|---|---|---|---|---|
| firecrawl / firecrawl | 199,902 | today | AGPL-3.0 | No downloads |
| ScrapeGraphAI / Scrapegraph-ai | 32,645 | 34 days ago | MIT | No downloads |
| fastcrw / crw | 1,237 | today | AGPL-3.0 | Mac, Linux |
More: languages, topics, project record and preview image
Languages
- Python 99%
- JavaScript 1%
- Shell <1%
- Dockerfile <1%
Topics
- ai
- ai-agents
- crawler
- data-extraction
- llm
- markdown
- mcp
- open-source
- playwright
- python
- rag
- scraper
Project record
- JobAutomation and scraping
- StatusFeatured
- Commits1,792
- Forks9,081
- Watching423
- Stars in 28 days+1,858
- CreatedMay 9, 2024
- First seen by usNov 7, 2026
- CheckedNov 9, 2026