SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents
Kuangshi Ai (University of Notre Dame), Haichao Miao (Lawrence Livermore National Laboratory), Kaiyuan Tang (University of Notre Dame), Nathaniel Gorski (University of Utah), Jianxin Sun (University of Nebraska-Lincoln), Guoxi Liu (The Ohio State University), Helgi Ingolfsson (Lawrence Livermore National Laboratory), David Lenz (Argonne National Laboratory), Hanqi Guo (The Ohio State University), Hongfeng Yu (University of Nebraska-Lincoln), Teja Leburu (Anthropic PBC), Michael Molash (Anthropic, PBC), Bei Wang (University of Utah), Tom Peterka (Argonne National Laboratory), Chaoli Wang (University of Notre Dame), Shusen Liu (Lawrence Livermore National Laboratory )
Jump to Discussion
Presentation
- Session
- Figuring out how to do good research
- Time
- Tuesday, Nov 10, 16:00 – 16:12 (US/Eastern) · session 15:00 – 16:30
- Room
- Hall America center
- Presenting from
- Boston
Links
Sign in to access the Paper PDF.
- Preprint PDF(opens in new tab)
- Supplemental Material(opens in new tab)
Abstract
Recent advances in large language models (LLMs) have enabled agentic systems to translate natural-language intent into executable scientific visualization (SciVis) tasks. Despite rapid progress, the community lacks a principled and reproducible benchmark for evaluating these emerging SciVis agents in realistic, multi-step analysis settings. We present SciVisAgentBench, a comprehensive and extensible benchmark for evaluating scientific data analysis and visualization agents. Our benchmark is grounded in a structured taxonomy spanning four dimensions: application domain, data type, complexity level, and visualization operation. It currently comprises 108 expert-crafted cases covering diverse SciVis scenarios. To enable reliable assessment, we introduce a multimodal outcome-centric evaluation pipeline that combines LLM-based judging with deterministic evaluators, including image-based metrics, code checkers, rule-based verifiers, and case-specific evaluators. We also conduct a validity study with 12 SciVis experts to examine the agreement between human and LLM judges. Using this framework, we evaluate representative SciVis agents and general-purpose coding agents to establish initial baselines and reveal capability gaps. SciVisAgentBench is designed as a living benchmark to support systematic comparison, diagnose failure modes, and drive progress in agentic SciVis. The benchmark is available at https://scivisagentbench.github.io/.
For Practitioners
Scientific visualization practitioners, computational scientists, and developers of AI agents may find this paper useful. They can use the benchmark and evaluation framework to assess, compare, and improve agents for complex scientific data analysis and visualization tasks.