silent-failure-detection

SKILL.md

Detect when LLM agents fail silently - appearing to work while producing incorrect results.

| Detection priority | Tool skipping > Gibberish > Loops > Anomalies | | Quality check | LLM-as-judge with heuristic pre-filter | | Loop threshold | 10 iterations or 3x baseline tokens | | Anomaly threshold | Z-score > 3.0 (99.7% confidence) | | Alert strategy | Alert on silent failure, not just errors |

| Tool Skipping | Expected vs actual tool calls | Critical | | Gibberish Output | LLM-as-judge + heuristics | High | | Infinite Loop | Iteration count + token spike | Critical | | Quality Degradation | Score < baseline | Medium | | Latency Spike | p99 > threshold | Medium |

原文を見る