We finally got a benchmark that actually matches reality. Thank you Browserbase for sponsoring! Check them out at: https://soydev.link/browserbase SOURCES: https://deepswe.datacurve.ai/ https://x.com/theo/status/2059352130289651925 Want to sponsor a video? Learn more here: https://soydev.link/sponsor-me Check out my Twitch, Twitter, Discord more at https://t3.gg S/O @Ph4seon3 for the awesome edit 🙏
ADVERTISEMENT
best ai benchmark video counter: 1
Best model ever -> I was wrong cycle continues
Obsidian would never let the cheating slide
That actually IS a good video. We all suspected it, kudos to the creators of this benchmark
would be interesting to see composer-2.5 result on this bench
finally a benchmark that agrees with me
Gemini models never fail to un-suprise me.
I would like to see more languages there. Idiomatic modern C++ (at least C++20) and maybe also C. These are way more common than rust or Go, so not sure why they aren't there. I found that some models do way worse in some languages than others.
Have to agree this is the only benchmark that actully show what I felt
Glm5.1 is months old tough, and it is still doing OK
Great to have this. Lines up much better with the experience most of us have been having with these models
The important point you are missing is even if all tests are green, it does not mean that it is good code that follows the standards and implied rules of the repo. LLMs rarely suggest refactoring, which is an important part of real projects. They do tasks like a stranger that has no idea what the "folklore" behind the repo is. Over time, consistency degrades and maintainability decreases.
DeepSeek V4 Pro aside (which, as I understand it, really needs MAX thinking enabled to perform at its best), I've used both Kimi K2.6 and GLM 5/5.1, and in practice I've found both to be better than Claude Sonnet 4.6. That makes me wonder whether the results are being influenced by the specific languages included in the evaluation. It would have been useful to see more backend-oriented languages represented, such as Java, C#, PHP, and C++. As it stands, the overall score appears to be heavily weighted toward Go, Python, and TypeScript, leaving much of the broader software development landscape untested. Because of that, I think it's premature to conclude that the observed separation is simply due to the benchmark being more representative of real-world development tasks.
Sure, my business has an internal benchmark for stuff we care about.
Damn it I really want to know how composer 2.5 performs on this. Maybe one day
Google gonna have to pull one of them code reds again lol
14:40 never wrote a prompt like that
I'm more shocked that the GLM Kimi and DeepSeek costed that much, wasn't the entire point that they were cheap? Idk that's suspicious
If deepswe is good, are we saying gemini flash 3.5 is actually better than pro 3.1? Gemini Flash 3.5 > Opus 4.6? Opus 4.7 >>> 4.6? Sonnet 4.6 > Opus 4.6?
I really want to see a benchmark that compiles tests of different kinds (feature implementation, project from scratch, different kind of difficult logic) & different prompting styles (plan first, one-shot, back and forth, lots of threads) and see what models do well in that matrix.