Deepcrawl
Free and open-source agentic toolkit that makes any website data AI ready
npm i deepcrawl
No headless browser tax or virtual DOM overhead required for simple HTML parsing! Edge-native Workers on V8 engine returns response in milliseconds with specially optimized Deepcrawl's parsers and tuned HTML cleaners, and smart dynamic caching controls.
Multi-purposes Features And Endpoints
Not only does it include enhanced commonly used tools, but there are more requested features and endpoints are regularly added!
Headless Browser-rendering and asynchronous durable crawling are coming soon!
A full platform included, not just API endpoints
Not only does it include enhanced commonly used tools, but there are also many more planned features coming soon!
Worker with Hono.js and oRPC
REST + oRPC APIs powering the backend workflow.
Auth Worker with Better Auth
Better Auth, OAuth, and passkeys out of the box.
Next.js Dashboard On Vercel
Optimized Next.js dashboard, monitoring, playground, and key management interface.
FAQs
Frequently asked questions about Deepcrawl.