SkillTrustBench Leaderboard

This leaderboard reports the loose non-normal screening track for SkillTrustBench.

In this track, ground-truth malicious and suspicious cases are merged into the positive class, while normal cases are treated as the negative class.

The public view is split into two tables:

Tool Comparison compares tools and agent skills; each row documents its backing model and scoring mapping where applicable.
Model Comparison compares backing models. All runs use Tencent AI-Infra-Guard as the fixed analysis tool.

Best Agent Skill

Best Tool

Best Model

Lowest Tool-Table FPR

How scores are computed. Raw outputs are first normalized to normal (safe), suspicious, or malicious; the leaderboard then uses loose binary scoring, where malicious + suspicious are risk-positive and normal is risk-negative. Precision, recall, Loose F1, and FPR are computed from that collapsed view; see Evaluation Protocol for tool-specific mappings, including when actual_safe is used instead of severity buckets.

Evaluation scope. Current public scores are computed on a fixed 10% evaluation subset of SkillTrustBench v1.0 for cost-controlled tool and model comparison. The subset contains 556 cases: 285 malicious, 105 suspicious, and 166 normal. Exact case IDs are published as data/evaluation_subset_10pct.jsonl; full-corpus scores should be reported separately and should not be directly compared with this subset leaderboard.

Tool Comparison

Tool and agent-skill comparison. Rows may use scanner-native models or tool-specific scoring mappings; see result notes for details.

Tool comparison

Rank	System	Role	Official Link	Commercial Product	Loose F1	Precision	Recall	FPR
1	BraneMatrix SkillsScanner	agent skill	https://github.com/cisco-ai-defense/skill-scanner	Yes	0.9809	0.9747	0.9872	0.0602

Model Comparison

Backing-model comparison. All runs use Tencent AI-Infra-Guard as the fixed analysis tool.

Model comparison