Multimodal Benchmark · Vision-Language Models · Radiology
Spatial Perception & Anatomical Reasoning in Clinical Radiology
SPARC-Rad is a manually curated benchmark dataset and evaluation pipeline with 300 image-question pairs from healthy-control TCIA studies, spanning CT, MRI, and radiography across five anatomical categories. It tests whether vision-language models can reason about radiologic anatomy as a spatial system — not just classify disease.
Tripathi S., Seker M.E., Quevada K., Enwerem E.D., Khandelwal P., Meltem E., Koca B., Faghani S., Arnold J., Daye D., Cook T. SPARC-Rad: A Multimodal Benchmark Dataset and Evaluation Pipeline for Spatial and Anatomical Reasoning in Radiology Vision-Language Models. arXiv:2608.00100.
Results
Leaderboard
Overall accuracy on 300 image-question pairs, graded by LLM-as-judge with human quality review. Expand any row to see modality and organ-category breakdowns.
| Rank | Model | Overall Accuracy | CT | MRI | X-Ray |
|---|
Accuracy is reported with reference to the 300 image-question pairs; 95% confidence intervals are recommended for interpreting small differences in rank. See the paper for details.
Breakdowns
Top Models by Modality & Organ Category
Subgroup accuracy reveals that strong aggregate performance can mask region- or modality-specific weaknesses.
Dataset
300 Manually Curated Image-Question Pairs
Images are drawn from healthy-control studies in The Cancer Imaging Archive (TCIA). Radiology trainees designed image-grounded questions targeting anatomical identification, localization, laterality, regional recognition, device identification, counting, and inter-structure spatial relationships.
Distribution by Modality
Distribution by Anatomical Category
Representative Benchmark Instance
How It Works
Evaluation Pipeline
Every image-question pair is run through a reproducible pipeline: standardized prompting, raw-output collection, response normalization, LLM-as-judge grading, and human review.
Prompt
“You are evaluating a radiology image. Answer the following question using only the provided image. Provide a concise answer. Question: [question]”
Collect & Normalize
Raw outputs are stored for audit, then normalized — lowercasing, punctuation removal, abbreviation and synonym expansion — while preserving clinically meaningful distinctions such as left vs. right.
LLM-as-Judge
A separate judging model receives the question, reference answer, accepted synonyms, and the model response to determine anatomical and spatial equivalence.
Human Review
Radiology reviewers adjudicate ambiguous answers, especially for laterality, device localization, and relational questions. Each item receives a binary correctness label.
Reasoning Types Evaluated
- Anatomical identification
- Localization
- Laterality (left vs. right)
- Regional classification
- Inter-structure spatial relationships
- Counting / device recognition
Error Types Analyzed
- Laterality error (right instead of left)
- Wrong structure identified
- Wrong body region
- Overly vague response
- Relational error (medial/lateral, superior/inferior)
- Modality misunderstanding
- Hallucinated finding
Evaluation Workflow
Get on the Leaderboard
Submit Your Model
We run candidate vision-language models through the SPARC-Rad evaluation pipeline on the held question set and add verified results to the leaderboard. Submissions are graded with LLM-as-judge plus human radiology review before publication.