The static score tracks package size, not intent

§1 The setup

An agent skill is a SKILL.md with instructions for a coding agent, usually with scripts and reference files next to it. People — and, more and more, the agents themselves — install skills straight from GitHub. Two open-source scanners offer a verdict before installation: SkillSpector (NVIDIA; static patterns, AST, YARA, OSV, optional LLM pass) and skill-scanner (Cisco AI Defense; static analyzers, YARA, optional LLM pass).

We asked a narrow question: what do these verdicts look like on ordinary, published skills — and does the answer change with the language the skill is written in?

Corpus. 146 legitimate skills found through GitHub code search by language-specific phrases, deduplicated (at most three per repository, authors rotated), pinned to a commit SHA, with stated licenses: 39 English, 38 Russian, 30 German, 39 Chinese. Plus 40 synthetic attack samples: ten patterns (prompt override in prose, hidden HTML-comment instruction, exfiltration, curl | bash, env-var harvesting, fake installer, base64 shell, zero-width characters, scope creep, persistence) written identically in all four languages and defanged. Legitimate here means published with a license and reviewed provenance; we did not audit them.

Engines. SkillSpector 2.12.0 with --no-llm; Cisco skill-scanner 2.3.2 with --policy balanced and --policy quiet, no LLM. Versions are pinned in engines.lock.yaml. LLM-assisted modes were not run; they are the next step and will be reported separately. As a control we added a 60-line hidden-character checker of our own, which is in the tables because it also produced false positives and we see no reason to hide that.

All of it — manifest, scripts, results — is in the benchmark repository and reproduces with three commands.

§2 The static score tracks package size

SkillSpector reduces its findings to a score from 0 to 100 and a recommendation: SAFE up to 20, CAUTION to 50, DO NOT INSTALL above. The strongest predictor of that score in our corpus is not what the skill does but how many code files sit next to SKILL.md.

None of the 43 legitimate skills with six or more code files were rated SAFE; 38 of them were rated DO NOT INSTALL. The 40 attack samples — one SKILL.md and at most one script each — scored a median of 29 and a maximum of 51. Only one of the ten patterns reached DO NOT INSTALL in any language. A legitimate English skill with a scripts/ folder is rated worse than a one-file sample that reads credential files and posts them to a URL.

Cisco’s output points the same way with smaller magnitude. Its most frequent MEDIUM-and-above findings on legitimate skills — DATA_EXFIL_NETWORK_REQUESTS, DATA_EXFIL_JS_NETWORK, UNANALYZABLE_BINARY, COMMAND_INJECTION_* — are all raised by bundled code, not by the instructions.

This matters for everything that follows: the four language sets differ a lot in how much code they ship. Russian and German skills in the wild are mostly Markdown-only; English and Chinese skills carry scripts, sometimes whole applications. Any per-language number has to be read with that in mind, which is why §4 looks only at findings located in Markdown.

§3 Verdicts by language

What a user sees is the verdict, so that is what we count first: the share of legitimate skills the engine itself did not rate SAFE (SkillSpector: recommendation; Cisco: is_safe), and the share of attack samples it did.

Read together with §2: SkillSpector’s 94 % on English is mostly the code-volume effect; its 37 % on Russian, where almost nothing ships code, is mostly one rule (§4). Cisco’s 3 % on Russian against 46 % on English is the same volume effect seen through a different scorer. Neither engine’s static mode separates the attack samples from the legitimate set: SkillSpector rates 40 % of them SAFE, Cisco 85 %.

§4 The rules behind it

Counting only findings whose location is a Markdown file removes most of the code-volume effect and leaves the language of the prose:

Four rules account for most of the language-specific behaviour:

• SkillSpector AE4, “Suspicious Unicode normalization or mixed-script content.” 207 findings on Russian Markdown, 3 on English, 2 on Chinese, 0 on German. Any Russian SKILL.md that contains a code block, a product name or a command mixes Cyrillic and Latin script and triggers it. It is MEDIUM, so on its own it moves a Markdown-only Russian skill from SAFE to CAUTION in roughly a third of cases. (Our own skill, written in Russian, collects four of them.)
• SkillSpector RP1, “npx commands without a version suffix.” 346 findings on English Markdown — the largest single source of MEDIUM findings on English prose. Documentation that says npx something is treated as a supply-chain risk even when it is an example, not an install step.
• Cisco YARA_prompt_injection_unicode_steganography. 112 findings on Chinese packages, 24 on German, 0 on Russian — mostly in code and data files, not in SKILL.md, and the main reason for Cisco’s higher Chinese rate.
• Cisco YARA_code_execution_generic. 31 findings on English Markdown, 0 on the other three languages. Its patterns match English words; prose in other languages cannot trigger it, and attacks written in those languages cannot either.

Rules firing on legitimate Markdown, MEDIUM and above

SkillSpector

Cisco skill-scanner, balanced

§5 What nobody caught

The ten attack patterns were written to be recognisable, not stealthy: the point was to see whether recognition survives translation.

• P-01, an instruction override in plain prose (“ignore the previous instructions and …”). Both engines flag the English sample; neither flags the Russian, German or Chinese translation of the same sentence.
• P-09, overly broad trigger and scope creep (“use this skill for any request”). Not flagged by anyone in any language. It is a prose-only pattern and arguably the most common real-world problem.
• P-06, a fake “prerequisites installer” download. Cisco flags it in English only; SkillSpector not at all.
• SkillSpector catches the structural patterns — HTML-comment instruction, curl | bash, env-var harvesting, base64 shell, zero-width characters, persistence — in all four languages. Cisco’s static mode catches env-var harvesting and base64 in all four, nothing else outside English.

Language-independent detection exists exactly where the pattern is in code. Where the attack is in the instructions, detection is English-only or absent — which is the inverse of where the risk now sits, since the agent follows the instructions and only sometimes runs the code.

§6 Operational notes

• SkillSpector did not finish within 15 minutes on 8 of 186 packages (16–73 MB, whole repositories with a root SKILL.md, one with 6,342 JavaScript files). Cisco finished each of them in under a minute. The 8 are excluded from every rate above and listed here: de-059, en-160, en-165, en-174, en-190, zh-112, zh-122, zh-142.
• Our own hidden-character checker, written for this benchmark, flagged legitimate skills until it learned that emoji sequences use U+200D and that soft hyphens are normal in German. It still flags 7 of 146 legitimate packages, 2 of them in Markdown. We report it for the same reason we report the others.
• Cisco rejects a skill whose file is named Skill.md or _template-skill.md; two such entries were dropped from the corpus rather than renamed.

§7 What we would ask of a scanner

None of this is an argument against scanning. It is an argument for calibrating the verdict, and the fixes are concrete:

1. Separate volume from risk. Cap the contribution of repeated low-information findings (one npx rule should not add up to 100) and say “large package, partially analysed” instead of DO NOT INSTALL when the honest answer is “we did not look”.
2. Make Unicode rules script-aware. Cyrillic-plus-Latin inside a Russian document is the baseline, not an anomaly; the anomaly is a Cyrillic letter inside a Latin word. The same applies to CJK and to German soft hyphens.
3. Test prose rules in more than one language. A rule that matches “ignore previous instructions” should be shipped with its translations, or should be an LLM check — and the test suite should say which.
4. Publish a false-positive corpus. Every scanner ships malicious test fixtures; almost none ship a set of ordinary skills that must pass. The corpus here is CC BY 4.0 and can be that set.

§8 Caveats

• Static modes only. Both engines have LLM-assisted modes that may change the picture, especially for prose patterns. Running them is the next step.
• The attack samples are short and single-purpose. They test whether a pattern is recognised in a given language, not how a real attacker would package it.
• “Legitimate” is a provenance judgement, not an audit. Some findings on the legitimate set may be right.
• One corpus, one date. The scorecard will be re-run on each engine release; the first number that matters is the next one.

§9 Vendor responses

Written by Andrey Trukhachev. Corrections and disputes: andrey@skillverdict.dev. Severity words on this page (SAFE, CAUTION, DO NOT INSTALL, HIGH, MEDIUM) quote the engines’ own output and are not our judgement of any skill.