nemo-evaluator
✓Use when evaluating LLMs, running benchmarks like MMLU/HumanEval/GSM8K, setting up evaluation pipelines, or asking about "NeMo Evaluator", "LLM benchmarking", "model evaluation", "MMLU", "HumanEval", "GSM8K", "benchmark harnesses"
Installation
SKILL.md
NeMo Evaluator SDK evaluates LLMs across 100+ benchmarks from 18+ harnesses using containerized, reproducible evaluation with multi-backend execution (local Docker, Slurm HPC, Lepton cloud).
| lm-evaluation-harness | 60+ | MMLU, GSM8K, HellaSwag, ARC | | simple-evals | 20+ | GPQA, MATH, AIME | | bigcode-evaluation-harness | 25+ | HumanEval, MBPP, MultiPL-E | | safety-harness | 3 | Aegis, WildGuard | | vlmevalkit | 6+ | OCRBench, ChartQA, MMMU | | bfcl | 6 | Function calling v2/v3 |
| run | Execute evaluation with config | | status | Check job status | | ls tasks | List available benchmarks | | ls runs | List all invocations | | export | Export results (mlflow/wandb/local) | | kill | Terminate running job |
Facts (cite-ready)
Stable fields and commands for AI/search citations.
- Install command
npx skills add https://github.com/eyadsibai/ltk --skill nemo-evaluator- Source
- eyadsibai/ltk
- Category
- </>Dev Tools
- Verified
- ✓
- First Seen
- 2026-02-17
- Updated
- 2026-02-18
Quick answers
What is nemo-evaluator?
Use when evaluating LLMs, running benchmarks like MMLU/HumanEval/GSM8K, setting up evaluation pipelines, or asking about "NeMo Evaluator", "LLM benchmarking", "model evaluation", "MMLU", "HumanEval", "GSM8K", "benchmark harnesses" Source: eyadsibai/ltk.
How do I install nemo-evaluator?
Open your terminal or command line tool (Terminal, iTerm, Windows Terminal, etc.) Copy and run this command: npx skills add https://github.com/eyadsibai/ltk --skill nemo-evaluator Once installed, the skill will be automatically configured in your AI coding environment and ready to use in Claude Code or Cursor
Where is the source repository?
https://github.com/eyadsibai/ltk
Details
- Category
- </>Dev Tools
- Source
- skills.sh
- First Seen
- 2026-02-17