CogniScrape is an intelligent web scraping library with LLMs — a TypeScript package combining traditional scraping with LLMs for structured data extraction.
GitHub: github.com/Riddhish1/CogniScrape · npm: npmjs.com/package/cogniscrape
Overview
CogniScrape bridges the gap between unstructured HTML and strongly typed, validated data schemas using local and cloud-based LLMs.
Highlights
- Dual LLM Support: Works seamlessly with Ollama for local execution and Google Gemini for cloud reasoning.
- Graph-based Node Pipelines: Composable nodes for reusable scraping workflows and transformations.
- Production-Ready Features: Built-in caching, automatic retries, rate limiting, and proxy rotation.
- Schema Validation & Multi-Format Exports: Schema enforcement via Zod with export options for JSON, CSV, XML, and PDF.
- Browser Automation & RAG: Playwright integration for dynamic single-page applications and RAG for enhanced extraction precision.