The arms race to build smarter AI models has a measurement problem: the tests used to rank them are becoming obsolete almost as quickly as the models improve. On Monday, Artificial Analysis, an independent AI benchmarking organization whose rankings are closely watched by developers and enterprise buyers, released a major overhaul to its Intelligence Index that fundamentally changes how the industry measures AI progress.The new Intelligence Index v4.0 incorporates 10 evaluations spanning agents, coding, scientific reasoning, and general knowledge. But the changes go far deeper than shuffling test names. The organization removed three staple benchmarks — MMLU-Pro, AIME 2025, and LiveCodeBench — that have long been cited by AI companies in their marketing materials. In their place, the new index introduces evaluations designed to measure whether AI systems can complete the kind of work that people actually get paid to do.type: embedded-entry-inline id: 1bCmRrroGCdUb07IuaHysL”This index shift reflects a broader transition: intelligence is being measured less by recall and more by economically useful action,” observed Aravind Sundar, a researcher who responded to the announcement on X (formerly Twitter).Why AI benchmarks are breaking: The problem with tests that top models have already masteredThe benchmark overhaul addresses a growing crisis in AI evaluation: the leading models have become so capable that traditional tests can no longer meaningfully differentiate between them. The new index deliberately makes the curve harder to climb. According to Artificial Analysis, top models now score 50 or below on the new v4.0 scale, compared to 73 on the previous version — a recalibration designed to restore headroom for future improvement.This saturation problem has plagued the industry for months. When every frontier model scores in the 90th percentile on a given test, the test loses its usefulness as a decision-making tool for enterprises trying to choose which AI system to deploy. The new methodology attempts to solve this by weighting four categories equally — Agents, Coding, Scientific Reasoning, and Genera l— while introducing evaluations where even the most advanced systems still struggle.The results under the new f …
