Introduction
# Stealth Browser Skill š„·
Anti-bot browser automation that bypasses Cloudflare Turnstile, Datadome, and aggressive fingerprinting.
## When to Use
- Standard Playwright/Selenium gets blocked - Site shows Cloudflare challenge or "checking your browser" - Need to scrape Airbnb, Yelp, or similar protected sites - `playwright-stealth` isn't working anymore
## Tool Selection
| Target Difficulty | Tool | When to Use | |------------------|------|-------------| | **Browser** | Camoufox | All protected sites - Cloudflare, Datadome, Yelp, Airbnb | | **API Only** | curl_cffi | No browser needed, just TLS spoofing |
## Quick Start
All scripts run in `pybox` distrobox for isolation.
ā ļø **Use `python3.14` explicitly** - pybox may have multiple Python versions with different packages installed.
### 1. Setup (First Time)
```bash # Install tools in pybox (use python3.14) distrobox-enter pybox -- python3.14 -m pip install camoufox curl_cffi
# Camoufox browser downloads automatically on first run (~700MB Firefox fork) ```
### 2. Fetch a Protected Page
**Browser (Camoufox):** ```bash distrobox-enter pybox -- python3.14 scripts/camoufox-fetch.py "https://example.com" --headless ```
**API only (curl_cffi):** ```bash distrobox-enter pybox -- python3.14 scripts/curl-api.py "https://api.example.com/endpoint" ```
## Architecture
``` āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā OpenClaw Agent ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā distrobox-enter pybox -- python3.14 scripts/xxx.py ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā pybox Container ā ā āāāāāāāāāāāāāāā āāāāāāāāāāāāāāā ā ā ā Camoufox ā ā curl_cffi ā ā ā ā (Firefox) ā ā (TLS spoof)ā ā ā āāāāāāāāāāāāāāā āāāāāāāāāāāāāāā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ```
## Tool Details
### Camoufox - **What:** Custom Firefox build with C++ level stealth patches - **Pros:** Best fingerprint evasion, passes Turnstile automatically - **Cons:** ~700MB download, Firefox-based - **Best for:** All protected sites - Cloudflare, Datadome, Yelp, Airbnb
### curl_cffi - **What:** Python HTTP client with browser TLS fingerprint spoofing - **Pros:** No browser overhead, very fast - **Cons:** No JS execution, API endpoints only - **Best for:** Known API endpoints, mobile app reverse engineering
## Critical: Proxy Requirements
**Datacenter IPs (AWS, DigitalOcean) = INSTANT BLOCK on Airbnb/Yelp**
You MUST use residential or mobile proxies:
```python # Example proxy config proxy = "http://user:[email protected]:8080" ```
See **[references/proxy-setup.md](references/proxy-setup.md)** for proxy configuration.
## Behavioral Tips
Sites like Airbnb/Yelp use behavioral analysis. To avoid detection:
1. **Warm up:** Don't hit target URL directly. Visit homepage first, scroll, click around. 2. **Mouse movements:** Inject random mouse movements (Camoufox handles this). 3. **Timing:** Add random delays (2-5s between actions), not fixed intervals. 4. **Session stickiness:** Use same proxy IP for 10-30 min sessions, don't rotate every request.
## Headless Mode Warning
ā ļø Old `--headless` flag is DETECTED. Options:
1. **New Headless:** Use `headless="new"` (Chrome 109+) 2. **Xvfb:** Run headed browser in virtual display 3. **Headed:** Just run headed if you can (most reliable)
```bash # Xvfb approach (Linux) Xvfb :99 -screen 0 1920x1080x24 & export DISPLAY=:99 python scripts/camoufox-fetch.py "https://example.com" ```
## Troubleshooting
| Problem | Solution | |---------|----------| | "Access Denied" immediately | Use residential proxy | | Cloudflare challenge loops | Try Camoufox instead of Nodriver | | Browser crashes in pybox | Install missing deps: `sudo dnf install gtk3 libXt` | | TLS fingerprint blocked | Use curl_cffi with `impersonate="chrome120"` | | Turnstile checkbox appears | Add mouse movement, increase wait time | | `ModuleNotFoundError: camoufox` | Use `python3.14` not `python` or `python3` | | `greenlet` segfault (exit 139) | Python version mismatch - use `python3.14` explicitly | | `libstdc++.so.6` errors | NixOS lib path issue - use `python3.14` in pybox |
### Python Version Issues (NixOS/pybox)
The `pybox` container may have multiple Python versions with separate site-packages:
```bash # Check which Python has camoufox distrobox-enter pybox -- python3.14 -c "import camoufox; print('OK')"
# Wrong (may use different Python) distrobox-enter pybox -- python3.14 scripts/camoufox-session.py ...
# Correct (explicit version) distrobox-enter pybox -- python3.14 scripts/camoufox-session.py ... ```
If you get segfaults or import errors, always use `python3.14` explicitly.
## Examples
### Scrape Airbnb Listing
```bash distrobox-enter pybox -- python3.14 scripts/camoufox-fetch.py \ "https://www.airbnb.com/rooms/12345" \ --headless --wait 10 \ --screenshot airbnb.png ```
### Scrape Yelp Business
```bash distrobox-enter pybox -- python3.14 scripts/camoufox-fetch.py \ "https://www.yelp.com/biz/some-restaurant" \ --headless --wait 8 \ --output yelp.html ```
### API Scraping with TLS Spoofing
```bash distrobox-enter pybox -- python3.14 scripts/curl-api.py \ "https://api.yelp.com/v3/businesses/search?term=coffee&location=SF" \ --headers '{"Authorization": "Bearer xxx"}' ```
## Session Management
Persistent sessions allow reusing authenticated state across runs without re-logging in.
### Quick Start
```bash # 1. Login interactively (headed browser opens) distrobox-enter pybox -- python3.14 scripts/camoufox-session.py \ --profile airbnb --login "https://www.airbnb.com/account-settings"
# Complete login in browser, then press Enter to save session
# 2. Reuse session in headless mode distrobox-enter pybox -- python3.14 scripts/camoufox-session.py \ --profile airbnb --headless "https://www.airbnb.com/trips"
# 3. Check session status distrobox-enter pybox -- python3.14 scripts/camoufox-session.py \ --profile airbnb --status "https://www.airbnb.com" ```
### Flags
| Flag | Description | |------|-------------| | `--profile NAME` | Named profile for session storage (required) | | `--login` | Interactive login mode - opens headed browser | | `--headless` | Use saved session in headless mode | | `--status` | Check if session appears valid | | `--export-cookies FILE` | Export cookies to JSON for backup | | `--import-cookies FILE` | Import cookies from JSON file |
### Storage
- **Location:** `~/.stealth-browser/profiles/<name>/` - **Permissions:** Directory `700`, files `600` - **Profile names:** Letters, numbers, `_`, `-` only (1-63 chars)
### Cookie Handling
- **Save:** All cookies from all domains stored in browser profile - **Restore:** Only cookies matching target URL domain are used - **SSO:** If redirected to Google/auth domain, re-authenticate once and profile updates
### Login Wall Detection
The script detects session expiry using multiple signals:
1. **HTTP status:** 401, 403 2. **URL patterns:** `/login`, `/signin`, `/auth` 3. **Title patterns:** "login", "sign in", etc. 4. **Content keywords:** "captcha", "verify", "authenticate" 5. **Form detection:** Password input fields
If detected during `--headless` mode, you'll see: ``` š Login wall signals: url-path, password-form ```
Re-run with `--login` to refresh the session.
### Remote Login (SSH)
Since `--login` requires a visible browser, you need display forwarding:
**X11 Forwarding (Preferred):** ```bash # Connect with X11 forwarding ssh -X user@server
# Run login (opens browser on your local machine) distrobox-enter pybox -- python3.14 scripts/camoufox-session.py \ --profile mysite --login "https://example.com" ```
**VNC Alternative:** ```bash # On server: start VNC session vncserver :1
# On client: connect to VNC vncviewer server:1
# In VNC session: run login distrobox-enter pybox -- python3.14 scripts/camoufox-session.py \ --profile mysite --login "https://example.com" ```
### Security Notes
ā ļø **Cookies are credentials.** Treat profile directories like passwords: - Profile dirs have `chmod 700` (owner only) - Cookie exports have `chmod 600` - Don't share profiles or exported cookies over insecure channels - Consider encrypting backups
### Limitations
| Limitation | Reason | |------------|--------| | localStorage/sessionStorage not exported | Use browser profile instead (handles automatically) | | IndexedDB not portable | Stored in browser profile, not cookie export | | No parallel profile access | No file locking in v1; use one process per profile |
## References
- [references/proxy-setup.md](references/proxy-setup.md) ā Proxy configuration guide - [references/fingerprint-checks.md](references/fingerprint-checks.md) ā What anti-bot systems check