Introduction
# AnyCrawl Skill
AnyCrawl API integration for OpenClaw - Scrape, Crawl, and Search web content with high-performance multi-threaded crawling.
## Setup
### Method 1: Environment variable (Recommended)
```bash export ANYCRAWL_API_KEY="your-api-key" ```
Make it permanent by adding to `~/.bashrc` or `~/.zshrc`: ```bash echo 'export ANYCRAWL_API_KEY="your-api-key"' >> ~/.bashrc source ~/.bashrc ```
Get your API key at: https://anycrawl.dev
### Method 2: OpenClaw gateway config
```bash openclaw config.patch --set ANYCRAWL_API_KEY="your-api-key" ```
## Functions
### 1. anycrawl_scrape
Scrape a single URL and convert to LLM-ready structured data.
**Parameters:** - `url` (string, required): URL to scrape - `engine` (string, optional): Scraping engine - `"cheerio"` (default), `"playwright"`, `"puppeteer"` - `formats` (array, optional): Output formats - `["markdown"]`, `["html"]`, `["text"]`, `["json"]`, `["screenshot"]` - `timeout` (number, optional): Timeout in milliseconds (default: 30000) - `wait_for` (number, optional): Delay before extraction in ms (browser engines only) - `wait_for_selector` (string/object/array, optional): Wait for CSS selectors - `include_tags` (array, optional): Include only these HTML tags (e.g., `["h1", "p", "article"]`) - `exclude_tags` (array, optional): Exclude these HTML tags - `proxy` (string, optional): Proxy URL (e.g., `"http://proxy:port"`) - `json_options` (object, optional): JSON extraction with schema/prompt - `extract_source` (string, optional): `"markdown"` (default) or `"html"`
**Examples:**
```javascript // Basic scrape with default cheerio anycrawl_scrape({ url: "https://example.com" })
// Scrape SPA with Playwright anycrawl_scrape({ url: "https://spa-example.com", engine: "playwright", formats: ["markdown", "screenshot"] })
// Extract structured JSON anycrawl_scrape({ url: "https://product-page.com", engine: "cheerio", json_options: { schema: { type: "object", properties: { product_name: { type: "string" }, price: { type: "number" }, description: { type: "string" } }, required: ["product_name", "price"] }, user_prompt: "Extract product details from this page" } }) ```
### 2. anycrawl_search
Search Google and return structured results.
**Parameters:** - `query` (string, required): Search query - `engine` (string, optional): Search engine - `"google"` (default) - `limit` (number, optional): Max results per page (default: 10) - `offset` (number, optional): Number of results to skip (default: 0) - `pages` (number, optional): Number of pages to retrieve (default: 1, max: 20) - `lang` (string, optional): Language locale (e.g., `"en"`, `"zh"`, `"vi"`) - `safe_search` (number, optional): 0 (off), 1 (medium), 2 (high) - `scrape_options` (object, optional): Scrape each result URL with these options
**Examples:**
```javascript // Basic search anycrawl_search({ query: "OpenAI ChatGPT" })
// Multi-page search in Vietnamese anycrawl_search({ query: "hướng dẫn Node.js", pages: 3, lang: "vi" })
// Search and auto-scrape results anycrawl_search({ query: "best AI tools 2026", limit: 5, scrape_options: { engine: "cheerio", formats: ["markdown"] } }) ```
### 3. anycrawl_crawl_start
Start crawling an entire website (async job).
**Parameters:** - `url` (string, required): Seed URL to start crawling - `engine` (string, optional): `"cheerio"` (default), `"playwright"`, `"puppeteer"` - `strategy` (string, optional): `"all"`, `"same-domain"` (default), `"same-hostname"`, `"same-origin"` - `max_depth` (number, optional): Max depth from seed URL (default: 10) - `limit` (number, optional): Max pages to crawl (default: 100) - `include_paths` (array, optional): Path patterns to include (e.g., `["/blog/*"]`) - `exclude_paths` (array, optional): Path patterns to exclude (e.g., `["/admin/*"]`) - `scrape_paths` (array, optional): Only scrape URLs matching these patterns - `scrape_options` (object, optional): Per-page scrape options
**Examples:**
```javascript // Crawl entire website anycrawl_crawl_start({ url: "https://docs.example.com", engine: "cheerio", max_depth: 5, limit: 50 })
// Crawl only blog posts anycrawl_crawl_start({ url: "https://example.com", strategy: "same-domain", include_paths: ["/blog/*"], exclude_paths: ["/blog/tags/*"], scrape_options: { formats: ["markdown"] } })
// Crawl product pages only anycrawl_crawl_start({ url: "https://shop.example.com", strategy: "same-domain", scrape_paths: ["/products/*"], limit: 200 }) ```
### 4. anycrawl_crawl_status
Check crawl job status.
**Parameters:** - `job_id` (string, required): Crawl job ID
**Example:** ```javascript anycrawl_crawl_status({ job_id: "7a2e165d-8f81-4be6-9ef7-23222330a396" }) ```
### 5. anycrawl_crawl_results
Get crawl results (paginated).
**Parameters:** - `job_id` (string, required): Crawl job ID - `skip` (number, optional): Number of results to skip (default: 0)
**Example:** ```javascript // Get first 100 results anycrawl_crawl_results({ job_id: "xxx", skip: 0 })
// Get next 100 results anycrawl_crawl_results({ job_id: "xxx", skip: 100 }) ```
### 6. anycrawl_crawl_cancel
Cancel a running crawl job.
**Parameters:** - `job_id` (string, required): Crawl job ID
### 7. anycrawl_search_and_scrape
Quick helper: Search Google then scrape top results.
**Parameters:** - `query` (string, required): Search query - `max_results` (number, optional): Max results to scrape (default: 3) - `scrape_engine` (string, optional): Engine for scraping (default: `"cheerio"`) - `formats` (array, optional): Output formats (default: `["markdown"]`) - `lang` (string, optional): Search language
**Example:** ```javascript anycrawl_search_and_scrape({ query: "latest AI news", max_results: 5, formats: ["markdown"] }) ```
## Engine Selection Guide
| Engine | Best For | Speed | JS Rendering | |--------|----------|-------|--------------| | `cheerio` | Static HTML, news, blogs | ⚡ Fastest | ❌ No | | `playwright` | SPAs, complex web apps | 🐢 Slower | ✅ Yes | | `puppeteer` | Chrome-specific, metrics | 🐢 Slower | ✅ Yes |
## Response Format
All responses follow this structure:
```json { "success": true, "data": { ... }, "message": "Optional message" } ```
Error response: ```json { "success": false, "error": "Error type", "message": "Human-readable message" } ```
## Common Error Codes
- `400` - Bad Request (validation errors) - `401` - Unauthorized (invalid API key) - `402` - Payment Required (insufficient credits) - `404` - Not Found - `429` - Rate limit exceeded - `500` - Internal server error
## API Limits
- Rate limits apply based on your plan - Crawl jobs expire after 24 hours - Max crawl limit: depends on credits
## Links
- API Docs: https://docs.anycrawl.dev - Website: https://anycrawl.dev - Playground: https://anycrawl.dev/playground