Why it matters
Adversa tested eight open-source AI skill scanners with paired unobfuscated and obfuscated malicious skills, finding that every scanner passed an attack through either a true bypass, a blind spot, or an injectable model judge. The study covers encoding, Unicode, command reconstruction, truncation, allowlists, bundled files, paraphrase, and remote stages; its 4,000-skill benign set also found no scanner beat an always-block baseline on F1. Most tools ran offline without optional model triage, and some were reconstructed from retained artifacts.
My takeaway: Treat a clean skill scan as advisory, not authorization to install. Evaluate scanners with control pairs and adaptive attacks; decode, normalize, canonicalize, and rescan content; inspect every bundled file and language; disclose skipped or truncated coverage; and keep a deterministic severity floor beneath model judges. Pair scanning with curated sources, pinned versions, sandboxing, restricted egress, and secrets kept outside the agent runtime.