VFS-Bench 1.0 Leaderboard

Each model runs every task on both interfaces. Compare its success rate, tokens and cost with VFS and with MCP.

Blog View the tasks

VFS vs MCP

Success Rate vs Cost

Up and to the left is better

Success Rate

PASS@1Higher is better

Claude Opus 5.5+4%

VFS86%

MCP83%

GPT-6.1 Sol0%

VFS75%

MCP75%

GPT-5.6 Luna+20%

VFS73%

MCP61%

Claude Haiku 5.5+6%

VFS74%

MCP70%

Gemini 3.8 Flash+4%

VFS52%

MCP50%

Average Tokens per Task

Lower is betterToken Reduction Rate

Claude Opus 5.573%

VFS166k

MCP617k

GPT-6.1 Sol72%

VFS193k

MCP683k

GPT-5.6 Luna60%

VFS1.56M

MCP3.91M

Claude Haiku 5.510%

VFS2.54M

MCP2.82M

Gemini 3.8 Flash25%

VFS5.07M

MCP6.72M

By task

Runs passed and mean tokens per run for Claude Opus 5.5 on each Toolathlon task, VFS vs MCP
Task	VFS	MCP	VFS Tokens	MCP Tokens	Token Reduction Rate
A/B testing	1/1	1/1	80.8k	315k	74%
Academic PDF report	1/1	1/1	383k	613k	37%
Academic warning	1/1	1/1	87.4k	328k	73%
Add BibTeX	1/1	1/1	180k	474k	62%
Apply PhD email	1/1	1/1	54.0k	238k	77%
Arrange workspace	0/1	0/1	50.1k	348k	86%
Canvas exam schedule	1/1	1/1	206k	533k	61%
Canvas course setup	1/1	1/1	520k	2.65M	80%
Canvas art history quiz	1/1	1/1	86.3k	451k	81%
Unfinished Canvas quizzes	1/1	1/1	376k	1.22M	69%
Canvas Python homework grading	1/1	1/1	342k	570k	40%
Canvas new student notification	1/1	1/1	187k	283k	34%
Canvas late work submission	1/1	1/1	135k	304k	56%
Cooking guidance	1/1	1/1	97.0k	453k	79%
Course assistant	1/1	1/1	38.9k	152k	74%
Course schedule	1/1	0/1	55.1k	147k	62%
Courses TA homework	1/1	1/1	121k	246k	51%
CVPR postdoc search	1/1	1/1	122k	780k	84%
Dataset license issue	1/1	1/1	127k	424k	70%
Detect revised terms	0/1	0/1	190k	211k	10%
Dietary health	1/1	1/1	65.8k	124k	47%
Email paper homepage	1/1	1/1	255k	703k	64%
Excel data transformation	1/1	1/1	63.5k	131k	52%
Excel market research	1/1	1/1	31.3k	79.9k	61%
Experiment recordings	0/1	0/1	419k	4.37M	90%
Fill out online forms	1/1	1/1	102k	1.11M	91%
Filter low-selling products	1/1	1/1	205k	2.62M	92%
Find the Alita paper	1/1	1/1	117k	169k	31%
Flagged transactions	1/1	1/1	86.2k	261k	67%
Game statistics	1/1	1/1	68.5k	149k	54%
GDP CR5 analysis	1/1	1/1	140k	268k	48%
Git bug hunt	1/1	1/1	17.5k	73.3k	76%
Git milestone	1/1	1/1	16.1k	31.9k	50%
Git repo	1/1	1/1	33.5k	267k	87%
Hong Kong top conference papers	0/1	0/1	269k	401k	33%
Hugging Face upload	1/1	1/1	238k	267k	11%
Identify all songs	1/1	0/1	58.9k	409k	86%
ImageNet survey table	1/1	1/1	39.8k	68.9k	42%
Inter final performance analysis	1/1	1/1	763k	908k	16%
Interview report	1/1	1/1	205k	7.29M	97%
Inventory sync	1/1	1/1	318k	2.86M	89%
Investment decision analysis	1/1	1/1	195k	544k	64%
Invoice org	1/1	1/1	51.3k	112k	54%
Kubernetes deployment cleanup	1/1	1/1	94.7k	238k	60%
Kubernetes MySQL debugging	1/1	1/1	61.2k	165k	63%
Kubernetes PR preview testing	1/1	1/1	158k	435k	64%
Kubernetes Redis Helm upgrade	0/1	1/1	159k	455k	65%
Kubernetes security audit	0/1	0/1	129k	789k	84%
Landing task reminder	1/1	1/1	229k	364k	37%
LaTeX prompt box	1/1	1/1	323k	1.21M	73%
Live transactions	0/1	0/1	108k	267k	59%
LLM training dataset	1/1	1/1	107k	346k	69%
Logical datasets collection	1/1	1/1	10.2k	33.4k	69%
Machine operating	1/1	1/1	115k	905k	87%
Meeting assignment	1/1	1/1	68.6k	179k	62%
Music analysis	0/1	1/1	211k	2.32M	91%
NHL back-to-back analysis	1/1	1/1	51.2k	498k	90%
Notion job search	1/1	1/1	90.9k	203k	55%
Notion HR record	1/1	1/1	219k	482k	55%
Notion movie tracker	1/1	1/1	98.1k	212k	54%
Notion personal website	1/1	1/1	224k	240k	7%
NVIDIA institutional ownership	0/1	0/1	963k	877k	−10%
NVIDIA stock analysis	1/1	1/1	149k	960k	85%
Oil price	1/1	1/1	164k	430k	62%
Paper checker	0/1	0/1	277k	438k	37%
Payable invoice checker	1/1	1/1	693k	1.46M	53%
Personal website construct	1/1	1/1	280k	866k	68%
PPT analysis	1/1	1/1	87.7k	188k	53%
Price comparison	1/1	1/1	60.7k	136k	55%
Privacy desensitization	0/1	0/1	76.5k	87.8k	13%
Online profile update	1/1	1/1	109k	283k	61%
Quantitative financial analysis	1/1	1/1	131k	588k	78%
Reimbursement form filler	1/1	1/1	56.2k	211k	73%
Sales accounting	1/1	1/1	53.7k	308k	83%
California school search	1/1	1/1	78.7k	132k	40%
Set the camera-ready deadline reminder	1/1	1/1	54.7k	74.8k	27%
Shopping helper	1/1	1/1	49.8k	270k	82%
SLA timeout monitor	1/1	1/1	101k	253k	60%
Stock build position	1/1	1/1	64.7k	130k	50%
Student interview scheduling	1/1	1/1	96.4k	207k	54%
Singapore subway planning	1/1	0/1	20.4k	94.0k	78%
Sync TODO to README	1/1	1/1	132k	481k	72%
Train ticket plan	1/1	1/1	45.6k	84.0k	46%
Travel exchange	1/1	1/1	148k	298k	50%
Travel expense reimbursement	0/1	0/1	452k	816k	45%
Tokyo to Kamakura trip	1/1	1/1	76.2k	165k	54%
Paris trip itinerary	1/1	1/1	76.5k	183k	58%
Update material inventory	1/1	1/1	98.8k	148k	33%
UPenn campus route	1/1	1/1	63.4k	118k	46%
Verl dataset	1/1	1/1	51.0k	119k	57%
VLM history completion	1/1	1/1	357k	363k	2%
W&B best score	1/1	1/1	130k	182k	29%
W&B shortest length	1/1	1/1	134k	283k	53%
WooCommerce customer survey	1/1	1/1	106k	649k	84%
WooCommerce new product	1/1	1/1	110k	1.73M	94%
WooCommerce new customer welcome	1/1	0/1	435k	937k	54%
WooCommerce product recall	0/1	0/1	169k	980k	83%
WooCommerce stock alert	1/1	0/1	143k	396k	64%
WooCommerce product cover update	1/1	1/1	82.5k	788k	90%
YouTube ML repositories	0/1	0/1	592k	1.65M	64%