Every number on this site reproduces with three commands

How skills were selected

Discovery. GitHub code search for filename:SKILL.md combined with language-specific phrases (Russian: навык, используй, когда пользователь; German: verwende, wenn der Benutzer, Anleitung; Chinese: 技能, 使用, 当用户; English: Use this skill when, Instructions, Workflow). Candidate files are fetched raw; fenced code is removed before measuring script shares, and the language is assigned from the remaining prose (Cyrillic ≥ 50 % → Russian; CJK ≥ 30 % → Chinese; Latin ≥ 80 % → German if German function-word density exceeds 4 % and umlauts occur, else English).

Eligibility. Frontmatter with name and description; not a fork; at least 400 bytes; detected language matches the target set.

Selection. At most three skills per repository, round-robin over owners so no prolific author dominates; licensed skills preferred, then larger ones. Each entry is pinned to the repository HEAD at selection time via git ls-remote.

Storage. Skills under a permissive or copyleft SPDX license are stored as copies; all others as pointers (URL + SHA) and fetched at run time. The manifest records source URL, commit, license, language shares, whether code blocks are present, and the queries that found it.

Attack samples. Ten patterns × four languages, generated from one template per language so that only the language differs. Every sample is defanged — placeholder hosts, inert commands, a notice after the frontmatter — and lives under benchmark/corpus/malicious/synthetic/. They are never executed; nothing in the corpus is.

How the engines were run

skillspector scan <dir> --no-llm --format json          # NVIDIA/skillspector @ 6d0b7c849949cab9181e57798247f89e48ce5d6a (v2.12.0)
skill-scanner scan <dir> --format json --policy balanced  # cisco-ai-skill-scanner==2.3.2 (PyPI)
skill-scanner scan <dir> --format json --policy quiet
python3 -I benchmark/scripts/check_hidden_chars.py <dir>   # our control checker

Each package is scanned in isolation with a 900-second timeout. Entries that produce no parsable report are excluded from every rate and listed under “Run errors”, with their size. Engine versions, timeouts and partial-run history are written next to the results in results/raw/<engine>/_run.json.

Metrics

• Verdict-level FP — share of legitimate skills the engine itself did not rate SAFE (SkillSpector risk_assessment.recommendation; Cisco is_safe). Verdict-level FN — share of attack samples it rated SAFE.
• Finding-level FP at MEDIUM+ and HIGH+ — share of legitimate skills with at least one finding at that severity.
• Prose-only FP — the same, counting only findings whose location is a .md file. This isolates the language of the instructions from the amount of code shipped.
• Pattern × language matrix — highest severity reported for each attack pattern in each language.
• Score vs. code volume — SkillSpector score by number of code files (.py .js .ts .sh .mjs .cjs .rb .go .ps1), with Spearman rank correlation.

Three commands

git clone https://github.com/AndreyVomKeg/skillverdict-benchmark && cd skillverdict-benchmark
python3 -I scripts/fetch_corpus.py            # pinned SHAs → corpus/_work/
python3 -I scripts/run_engines.py             # results/raw/<engine>/*.json
python3 -I scripts/report.py                  # results/scorecard.md + .json

Run the readers with python3 -I and from outside the corpus directory: skill content is hostile input by definition. Install the engines first (pip install git+https://github.com/NVIDIA/skillspector.git@6d0b7c8 cisco-ai-skill-scanner==2.3.2).

Licenses

Benchmark code: Apache-2.0. Corpus manifest, synthetic samples and aggregated results: CC BY 4.0 — reuse freely, cite skillverdict.dev. Copied skills keep their own licenses, recorded per entry.

Disclosure policy

Engine maintainers receive the findings at least seven days before publication. Their responses are published unedited. Findings on individual legitimate skills are reported as aggregates; we do not publish a list of “skills that failed”. Authors of skills in the corpus can ask for removal at andrey@skillverdict.dev; the entry becomes a pointer or is dropped in the next corpus version.