# Crawl4AI

Canonical: https://cqbase.net/en/entry/crawl4ai
Chinese: https://cqbase.net/zh/entry/crawl4ai
English: https://cqbase.net/en/entry/crawl4ai

> An Apache-2.0 crawler and extractor for AI agents, which you can self-host.

unclecode/crawl4ai is Apache-2.0. It is a crawler and extractor aimed at AI agents. The repository describes self-hosting the crawler and also mentions a hosted cloud with a free allowance. This page is about the open-source repository, not a cloud plan.

Run one extraction on a site you are allowed to fetch before wiring it into an agent workflow. Notice which parts of the output are verbatim source text, and do not republish a whole page. Keep keys and quotas in environment variables.

Mentioned by:

- [@0x_rody](https://x.com/0x_rody/status/2121328656094244982) mentioned this project on 14 November 2026 (26 likes and 0 reposts, captured 15 November 2026).

## What problem it solves

[unclecode/crawl4ai](https://github.com/unclecode/crawl4ai) is an Apache-2.0 crawler and extractor for AI agents and data pipelines. It turns pages into Markdown that models can read cleanly, and it can also pull structured JSON with CSS, XPath, or an LLM strategy. This catalog entry is about the open-source repository and the self-hosted path, not a hosted cloud plan.

The same README also describes Crawl4AI Cloud at [crawl4ai.com](https://crawl4ai.com/). The cloud and the library share a crawling engine, but they differ in how you deploy, pay, and call the API. This page mentions the cloud for contrast. The steps below follow the local Python library and the official Docker example.

## Who it is for

It is for people who want to run the browser themselves and turn public pages into context for RAG or an agent. You can crawl inside a Python process, try a page with the `crwl` CLI, or start a token-protected local server with the README's Docker command. If you would rather not operate browsers and proxies, look at the official cloud API instead.

Run one extraction on a site you are allowed to fetch before you wire it into a workflow. The output often contains verbatim source text. Treat it as source material. Do not republish a whole page as your own article. Keep cloud keys and self-hosted tokens in environment variables.

## How to install it

Verified on 2026-11-16 (Asia/Shanghai). The commands come from the repository [README](https://github.com/unclecode/crawl4ai/blob/main/README.md). On that day, PyPI listed [crawl4ai](https://pypi.org/project/crawl4ai/)  v0.9.5 as the stable release (the release notes are dated 14 October 2026). It requires Python 3.10 or newer. `pyproject.toml` sets `requires-python = ">=3.10"`, and the classifiers go through 3.13.

Install the library and set up the browser once:

```bash
pip install -U crawl4ai
crawl4ai-setup
crawl4ai-doctor
```

`crawl4ai-setup` installs browser dependencies. If that step fails, the README's manual fallback is:

```bash
python -m playwright install --with-deps chromium
```

Pre-release builds are available with `pip install crawl4ai --pre`. To print one page as Markdown from the official CLI:

```bash
crwl https://docs.crawl4ai.com -o markdown
```

**Self-hosting with Docker**

The README requires a token. Without one, the server answers only inside its container. The official example publishes port `11235`:

```bash
export CRAWL4AI_API_TOKEN="$(openssl rand -hex 32)"
docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g \
  -e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN" \
  unclecode/crawl4ai:latest
```

Send `Authorization: Bearer` on probe requests. The dashboard is at `http://localhost:11235/dashboard`. Fuller settings are in the [self-hosting guide](https://docs.crawl4ai.com/core/self-hosting/).

There is a documentation split to be aware of. The [Installation](https://docs.crawl4ai.com/core/installation/) page still described Docker as experimental on this review date, and it still carried older wording such as "2023 Edition" and a planned image overhaul in Q1 2025. That does not match the current README example that uses `unclecode/crawl4ai:latest` with a token. This page does not invent a single official story. Follow the README for install commands, and re-read the Installation page for the version you actually checked out.

## A worked example

This example fetches the official documentation home page with the local library, as a smoke test. Use only a public page you are allowed to fetch. The script was not run here, so this page does not invent the Markdown.

1. Confirm Python is 3.10 or newer, and that you already ran `pip install` and `crawl4ai-setup`.
2. Save the following script, for example as `try_crawl4ai.py`:

```python
import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://docs.crawl4ai.com")
        markdown = result.markdown or ""
        print("ok" if markdown else "empty")
        print(len(markdown))

asyncio.run(main())
```

3. Run `python try_crawl4ai.py`. The expected effect is that a browser starts and the script prints a non-zero Markdown length. Open the original page and check the headings yourself. Do not republish the output.
4. If the script reports a missing browser, rerun `crawl4ai-setup` or the Playwright command in the previous section, then use `crawl4ai-doctor`.

## Common pitfalls

- Treating this entry as a cloud-plan guide. The catalog entry and this page describe the open-source repository. Cloud allowances, per-minute limits, and paid plans starting at $10 are whatever [crawl4ai.com](https://crawl4ai.com/) shows on the day you check. Those figures can change.
- Running only `pip install` and skipping `crawl4ai-setup`. The first `arun` often fails if the browser is missing.
- Ignoring the split between the Installation page and the README on Docker. If you follow older docs and pull `unclecode/crawl4ai:basic`, or treat Docker as experimental in every case, you may not match the current README.
- Putting keys in a script or in chat. The cloud key is `CRAWL4AI_KEY`. The self-hosted token is `CRAWL4AI_API_TOKEN`. Both belong in the environment or in a request header.
- Skipping the README's License & Attribution section. On top of Apache-2.0, the project asks you to add a specified badge or a text credit. Follow that section.
- Fetching a site you are not allowed to crawl, or one whose robots or terms disallow automated collection. Verbatim output is still the original work.

## FAQ

### How is the open-source library different from Crawl4AI Cloud?

The library runs the browser in your Python process or in a self-hosted Docker server. It is Apache-2.0, and you pay for the infrastructure. The cloud hosts the browsers and proxies, and you call `https://api.crawl4ai.com` with `CRAWL4AI_KEY`. On the review date, the README stated a free allowance of 1,000 credits a month and paid plans starting at $10. Search-style endpoints (`/search`, `/serp`, `/answer`) are described as cloud-side capabilities. Use the official site for current prices.

### Do I have to use Docker?

No. A local smoke test only needs `pip install -U crawl4ai` and `AsyncWebCrawler`. Docker is useful when you want a token-protected local HTTP service that an agent can call over REST or MCP.

### Why do some official docs still call Docker experimental?

The [Installation](https://docs.crawl4ai.com/core/installation/) page still used "experimental" and older 2025 roadmap wording on 16 November 2026. The repository README instead shows a current `docker run` example with `CRAWL4AI_API_TOKEN`, and it points to the self-hosting guide. This page does not reconcile those two texts. Follow the README and the revision you checked out.

### What is the minimum Python version?

On this review date, PyPI and `pyproject.toml` required Python 3.10 or newer. Classifiers list versions through 3.13. If you are on a newer Python, check the release notes before you assume support.

### Do I have to provide attribution?

The README's License & Attribution section says you should add a specified badge or a text credit when you use the project. The license itself is Apache-2.0. The exact obligation is in [LICENSE](https://github.com/unclecode/crawl4ai/blob/main/LICENSE) and that section. This page does not add a legal interpretation.

## Sources and license

[unclecode/crawl4ai](https://github.com/unclecode/crawl4ai) is licensed under Apache-2.0. This page is a paraphrase, for CQBase — Context & Query Base, of the README, `pyproject.toml`, [docs.crawl4ai.com](https://docs.crawl4ai.com), and the PyPI page, checked on 16 November 2026. The most recent repository push on that day was at 21:32 Asia/Shanghai on 13 November 2026. If install commands, Docker wording, or cloud prices change, the official repository and site win.

## Catalog record

- Author: unclecode
- License: Apache-2.0
- Source: https://github.com/unclecode/crawl4ai
- Added: 2026-11-15
- GitHub stars: 87378 (captured 2026-11-15)

