GcrawlAI

Turn any website into clean, LLM-ready data. Open-source web crawler with stealth mode, distributed crawling, real-time WebSocket progress & Markdown output. Power your AI apps with GcrawlAI. Maintained by GramosoftAI on GitHub, where it has earned 32 stars from the community.

It's actively developed around ai, celery, data-pipeline, and is a solid reference for anyone building with these tools.