A Web Crawler based on LLMs implemented with Ray and Huggingface. The embeddings are saved into a vector database for fast clustering and retrieval. Use it for your RAG. Maintained by Aavache on GitHub, where it has earned 104 stars from the community.
