nemo-evaluator-sdk
✓Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.
Installation
SKILL.md
NeMo Evaluator SDK evaluates LLMs across 100+ benchmarks from 18+ harnesses using containerized, reproducible evaluation with multi-backend execution (local Docker, Slurm HPC, Lepton cloud).
Run core academic benchmarks (MMLU, GSM8K, IFEval) on any OpenAI-compatible endpoint.
| lm-evaluation-harness | 60+ | MMLU, GSM8K, HellaSwag, ARC | | simple-evals | 20+ | GPQA, MATH, AIME | | bigcode-evaluation-harness | 25+ | HumanEval, MBPP, MultiPL-E | | safety-harness | 3 | Aegis, WildGuard | | garak | 1 | Security probing | | vlmevalkit | 6+ | OCRBench, ChartQA, MMMU | | bfcl | 6 | Function calling v2/v3 |
Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking. Source: orchestra-research/ai-research-skills.
Facts (cite-ready)
Stable fields and commands for AI/search citations.
- Install command
npx skills add https://github.com/orchestra-research/ai-research-skills --skill nemo-evaluator-sdk- Category
- </>Dev Tools
- Verified
- ✓
- First Seen
- 2026-02-11
- Updated
- 2026-02-18
Quick answers
What is nemo-evaluator-sdk?
Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking. Source: orchestra-research/ai-research-skills.
How do I install nemo-evaluator-sdk?
Open your terminal or command line tool (Terminal, iTerm, Windows Terminal, etc.) Copy and run this command: npx skills add https://github.com/orchestra-research/ai-research-skills --skill nemo-evaluator-sdk Once installed, the skill will be automatically configured in your AI coding environment and ready to use in Claude Code or Cursor
Where is the source repository?
https://github.com/orchestra-research/ai-research-skills
Details
- Category
- </>Dev Tools
- Source
- skills.sh
- First Seen
- 2026-02-11