crawl
Crawl and extract website content with configurable depth, regex filtering, and semantic chunking for LLM context.
Install
npx skills add https://github.com/tavily-ai/skills --skill crawlStats
| Total installs | 3,548 |
| Weekly installs | 3.5K |
| GitHub stars | 143 |
| First seen | Jan 25, 2026 |
| Source | @tavily-ai/skills |
Summary
- Supports configurable crawl depth (1-5 levels), breadth limits, and page caps to balance coverage against performance
- Includes path filtering via regex patterns to focus on specific sections and exclude irrelevant content
- Offers two modes: full-page extraction for data collection, or semantic chunking with natural language instructions for feeding results into LLM context
- Provides a companion Map API for URL discovery without content extraction, useful for understanding site structure before full crawls
- Authenticates via OAuth (Tavily account required) or API key; saves crawled pages as individual markdown files when output directory is specified
Tags
Related skills
| Skill | Installs | vs crawl |
|---|---|---|
| pci-compliance | 3,304 | -244 |
| deep-agents-memory | 3,259 | -289 |
| attack-tree-construction | 3,333 | -215 |
| market-sizing-analysis | 3,673 | +125 |
| vue-pinia-best-practices | 4,809 | +1,261 |
FAQ
- How many installs does crawl have?
- crawl has 3,548 total installs and 3.5K installs this week.
- Where is crawl hosted?
- crawl is published by @tavily-ai/skills at https://github.com/tavily-ai/skills.
- How many GitHub stars does crawl have?
- crawl has 143 GitHub stars.
- When was crawl first indexed?
- OrangeBot.AI first indexed crawl on Jan 25, 2026.
- How do I install crawl?
- Run: npx skills add https://github.com/tavily-ai/skills --skill crawl