Here is a link to the StressingLLMs OpenCode benchmark results. The process consisted of replicating my StressingLLMs tests using models available through OpenCode. I evaluated 11 models, most of which were free: Muse Spark 1.3, Muse Spark 1.2, Big Pickle, Nemotron 3 Ultra, Ling 3.0 Flash, MiMo v2.5, and Nemotron 3.5 Lightning. I also tested four paid models: GPT 6 Astra, Claude Fable 5.1, GLM, and Kimi K3. Altogether, I spent about $100. I don’t understand how people afford to benchmark models regularly. Claude and GPT 6 could chew through $20 in less than an hour.
The results are fun to explore. Most of the free models performed on par with the local models. The free model that surprised me most was Muse Spark 1.3. It was able to write a decryptor for an executable with 57 rounds, which is basically a cryptographic hashing workload, but failed at 58 rounds. GLM 5.2 made it to 67 rounds before failing. GPT 6 Astra and Claude Fable 5.1 both completed 68 rounds. Shortly after reaching those numbers, I stopped testing because my curiosity wasn’t worth the cost.
Based on the runs I completed, I estimated the cost, token usage, and time required for a single run. These figures are available in the “Estimated Single-Attempt Resources” table. According to the estimate, a single 10,000-round run using Claude Fable 5.1 would cost $239.14, take 4.7 hours, and use 118.9 million tokens. There’s no way to know whether this estimate is accurate without actually running the test, but it’s still fun to explore.
That highlights an important point about the page: I created it as a way to explore the data, and its results should not be treated as absolute.
One section I recommend exploring is the “Roadmap” table. Clicking the checkboxes displays the corresponding results and submission. It’s fascinating to see how different models approached the same problem.
No comments:
Post a Comment