A systematic workflow to move AI products beyond "vibe checks" by identifying specific failure modes and building automated LLM judges. Use this when your AI outputs feel "janky," when you need a feedback signal for prompt engineering, or when monitoring production performance at scale.
Quickly install ai-error-analysis-and-eval-design AI skill to your development environment via command line
Source: samarv/shanon.
To build great AI products, you must transition from subjective "vibe checks" to systematic measurement. This process identifies exactly where an LLM is failing and creates a feedback loop for continuous improvement.
Phase 1: Open Coding (The "Benevolent Dictator" Phase) Before automating, you must manually ground yourself in the data. Appoint one "Benevolent Dictator"—typically the Product Manager or domain expert—to define "good" taste.
Phase 2: Axial Coding (Categorization) Synthesize your mess of notes into actionable categories using an LLM.
A systematic workflow to move AI products beyond "vibe checks" by identifying specific failure modes and building automated LLM judges. Use this when your AI outputs feel "janky," when you need a feedback signal for prompt engineering, or when monitoring production performance at scale. Source: samarv/shanon.
Stable fields and commands for AI/search citations.
npx skills add https://github.com/samarv/shanon --skill ai-error-analysis-and-eval-designA systematic workflow to move AI products beyond "vibe checks" by identifying specific failure modes and building automated LLM judges. Use this when your AI outputs feel "janky," when you need a feedback signal for prompt engineering, or when monitoring production performance at scale. Source: samarv/shanon.
Open your terminal or command line tool (Terminal, iTerm, Windows Terminal, etc.) Copy and run this command: npx skills add https://github.com/samarv/shanon --skill ai-error-analysis-and-eval-design Once installed, the skill will be automatically configured in your AI coding environment and ready to use in Claude Code, Cursor, or OpenClaw
https://github.com/samarv/shanon