modelscope / evalscope

What it does

• Comprehensive Evaluation Benchmarks. Built-in multiple industry-recognized evaluation benchmarks including MMLU, C-Eval, GSM8K, and more.
• Multi-modal and Multi-domain Support. Supports evaluation of various model types including Large Language Models (LLM), Vision Language Models (VLM), Embedding, Reranker, AIGC, and more.
• System One Evaluation. Evaluate text Choice decision models via --eval-type systemone_api, with task instructions, few-shot examples and structured probabilities/confidence. See the usage guide.
• Multi-backend Integration. Seamlessly integrates multiple evaluation backends including OpenCompass, VLMEvalKit, RAGEval to meet different evaluation needs.
• Agent Evaluation Mode. Drives benchmarks (e.g. GSM8K, AIME, SWE-bench Agentic) inside a controlled multi-turn AgentLoop with pluggable strategies, tools and Docker sandbox; full per-sample Agent Trace is recorded and visualizable.