The public conversation about artificial intelligence swings between two claims: that it will solve humanity’s problems and that it is about to make large numbers of workers redundant. A benchmark built from real freelance jobs challenges the second claim, at least for now. Its results are striking and, for some, uncomfortable.
How the test works
The Remote Labor Index, or RLI, was developed by Scale AI and the Center for AI Safety. Rather than quizzing models with exam-style questions, it gives AI agents 240 real, completed professional projects taken from the freelancing platform Upwork, spanning design, video, software, data work and writing. Human evaluators then compare the AI’s deliverable against the work the paid freelancer originally produced. The aim is to find out whether current systems can deliver what a client actually paid a professional for.
The results
When the benchmark launched in late 2025, Scale AI reported that the best-performing agent completed only 2.5 percent of the projects to an acceptable standard. On the current leaderboard the top model is Anthropic’s Claude Opus 4.5 with a success rate of 3.75 percent, meaning it fell short of the professional’s work on more than 96 percent of tasks. Google’s Gemini models scored lower. Newer versions have improved slightly, but the overall picture has not changed much.
The researchers identified four main reasons for failure. Agents sometimes produced corrupt or unusable files. Many deliverables were incomplete, such as an eight-second clip in place of an eight-minute video. Quality was often below standard, and some designs and 3D models were internally inconsistent. The finding is that reliable AI performance in open-ended professional settings remains very limited.
Where AI does better
The benchmark also shows where current systems hold up. Agents performed relatively well on tasks such as generating creative concepts, advertising copy, simple coding, data scraping and report drafting. That pattern helps explain why Microsoft and other large companies are building AI into their products, even as recent software problems illustrate the risk of depending on it fully.
Prominent researchers have raised related concerns. Yann LeCun, the Turing Award winner and former chief AI scientist at Meta, has argued repeatedly that scaling up data and computing power alone will not deliver human-level intelligence, and that today’s large language models imitate human language rather than understand the world.
Why the gap matters
The study is a reminder that AI can be a powerful assistant without being a full replacement for skilled people. It also carries an economic warning. Hundreds of billions of dollars are being invested on the assumption that AI will soon substitute for human labor at scale. If that assumption proves wrong on the timeline investors expect, the financial consequences could be severe.
The technology may well live up to its promise eventually. At the current stage, the evidence argues for a realistic view. The history of technology shows that every major invention arrives wrapped in expectations that take years to be tested against what the tools can actually do.
