🏥 MedVidBench Leaderboard

Current Rankings

The leaderboard displays all submitted models ranked by their performance across 10 metrics on 8 medical video understanding tasks.

Note: Models with all caption metrics (DVC_llm, VS_llm, RC_llm) at 0.0 can be re-evaluated with LLM judge using the section below.

🤖 Run LLM Judge Evaluation

If a model was submitted with --skip-llm-judge (caption metrics are 0.0), you can run LLM judge evaluation here. This will compute DVC_llm, VS_llm, and RC_llm scores using GPT-4.1/Gemini.

✅ Background Execution: The evaluation runs in the background - you can close the browser and come back later!

Note: This feature is only available when ALL three caption metrics (DVC_llm, VS_llm, RC_llm) are 0.0.

Submit Your Model Results

Upload your model's predictions only on the MedVidBench test set (6,245 samples) to be added to the leaderboard.

📋 Requirements

1. Run inference on the full test set (download from HuggingFace)
2. Upload predictions JSON in the format below (NO ground truth needed)
3. Provide model info (name, organization)

📄 Expected File Format

Your predictions JSON should contain 6,245 samples with this structure:

[
  {
    "id": "video_id&&start&&end&&fps",
    "qa_type": "tal",
    "prediction": "Your model's answer here"
  },
  {
    "id": "another_video&&0&&10&&1.0",
    "qa_type": "video_summary",
    "prediction": "The surgeon performs..."
  }
]

Required fields:

• id: Sample identifier (matches test data from HuggingFace dataset)
• qa_type: Task type (tal/stg/next_action/dense_captioning/video_summary/region_caption/skill_assessment/cvs_assessment)
• prediction: Your model's answer (text output)

Important:

• ✅ Submit predictions only (no ground truth needed)
• ✅ Must include all 6,245 test samples
• ✅ Format can be list or dict (dict values will be extracted)
• ❌ Do NOT include ground truth fields (server handles this securely)

⚙️ Evaluation Process

After upload, the system will:

1. Validate your predictions file format
2. Merge your predictions with server-side ground truth (private)
3. Run evaluation for all 8 tasks across 10 metrics
4. Add to leaderboard if successful

Evaluation takes: ~5-10 minutes (includes LLM judge for caption quality assessment)

Security: Ground truth data is stored privately and never exposed to users.

MedVidBench Benchmark Tasks

The benchmark evaluates models across 8 diverse tasks spanning video, segment, and frame-level understanding:

Evaluation Metrics

• TAL (Temporal Action Localization): mAP@0.5 - mean Average Precision at IoU threshold 0.5
• STG (Spatiotemporal Grounding): mIoU - mean Intersection over Union (spatial + temporal)
• Next Action: Accuracy - Classification accuracy
• DVC (Dense Video Captioning): LLM Judge - GPT-4.1/Gemini scoring (average of top-5 aspects)
• VS (Video Summary): LLM Judge - GPT-4.1/Gemini scoring (average of top-5 aspects)
• RC (Region Caption): LLM Judge - GPT-4.1/Gemini scoring (average of top-5 aspects)
• Skill Assessment: Accuracy - Surgical skill level classification (JIGSAWS)
• CVS Assessment: Accuracy - Clinical variable scoring

LLM Judge Details

Caption tasks (DVC, VS, RC) use GPT-4.1 or Gemini-Pro with rubric-based scoring (1-5 scale) across 5 key aspects:

• R2: Relevance & Medical Terminology
• R4: Actionable Surgical Actions
• R5: Comprehensive Detail Level
• R7: Anatomical & Instrument Precision
• R8: Clinical Context & Coherence

The final score is the average across these 5 aspects.

Test Set Statistics

• Total samples: 6,245
• Source datasets: 8 (AVOS, CholecT50, CholecTrack20, Cholec80_CVS, CoPESD, EgoSurgery, NurViD, JIGSAWS)
• Video frames: ~103,742
• Task distribution:
  • TAL: ~800 samples
  • STG: ~900 samples
  • Next Action: ~700 samples
  • DVC: ~800 samples
  • VS: ~900 samples
  • RC: ~1000 samples
  • Skill Assessment: ~600 samples
  • CVS Assessment: ~545 samples

About MedVidBench

MedVidBench is a comprehensive benchmark for evaluating Video-Language Models on medical and surgical video understanding. It was introduced in the MedGRPO paper (Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understanding).

Key Features

• 8 diverse tasks covering multiple levels of video understanding
• 8 source datasets from various surgical procedures
• 6,245 test samples with high-quality annotations
• Automatic evaluation with standardized metrics
• LLM-based judging for caption quality assessment

Paper

@article{su2024medgrpo,
  title={MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understanding},
  author={Su, Yuhao and Choudhuri, Anwesa and Gao, Zhongpai and Planche, Benjamin and Nguyen, Van Nguyen and Zheng, Meng and Shen, Yuhan and Innanje, Arun and Chen, Terrence and Elhamifar, Ehsan and Wu, Ziyan},
  journal={arXiv preprint arXiv:2512.06581},
  year={2025}
}

Links

• 📄 Paper: https://arxiv.org/abs/2512.06581
• 🌐 Project Page: https://yuhaosu.github.io/MedGRPO/
• 💾 Dataset: https://huggingface.co/datasets/UIIAmerica/MedVidBench
• 💻 GitHub: https://github.com/YuhaoSu/MedGRPO
• 🏆 Leaderboard: https://huggingface.co/spaces/UIIAmerica/MedVidBench-Leaderboard

Dataset

The MedVidBench benchmark includes:

• 21,060 training samples
• 6,245 test samples
• Multi-modal annotations (video, text, temporal spans, bounding boxes)
• 8 source datasets covering various medical procedures

License

• Dataset: CC BY-NC-SA 4.0 (Non-commercial, Share-alike)
• Leaderboard Code: Apache 2.0
• Evaluation Scripts: MIT

Contact

For questions or issues:

• Open an issue on GitHub
• Visit the project page
• Email: Contact via GitHub

Admin Panel

This panel allows administrators to manage leaderboard submissions.

Features:

• View all submissions
• Delete individual models
• Cleanup test/dummy data

Note: Admin password is set via ADMIN_PASSWORD environment variable in HuggingFace Spaces settings.