Yeah, it's in the same ballpark as Kimi 3.1. Both of which are obviously open weight, so that's at least a big differentiator from Sol. Let's hope against hope that the much smaller flash model might beat deepseek v4.1 flash and run for as cheap.
I hope they are able to serve 4 faster. Qwen 3.8 is a great model but I’ve regretted including it in the battle system for opencode [1] we have been building because it scores so low.
And it’s a shame because the only reason it scores so low is that it’s slooooow. (Like 30 tokens per second vs opus near 80.) If you exclude voting, Qwen 3.8 does well.
(We allow early voting contrary to Arena.ai because I think it makes sense for agents, where it doesn’t for chat. This means our data takes speed into account.)
Also, it would be ideal if they served this with third parties. Part of the slowness with 3.8 is just not a lot of providers. Their open source version is much faster, but we couldn’t include it because it doesn’t handle images (another own goal by Alibaba).
This is exciting. The continued focus on efficiency alongside quality is encouraging - it makes me hopeful to see what smaller models will become capable of.
While I doubt that Qwen 4 Max can seriously compete with frontier models, the AA index has become all but useless, due to the introduction of new benchmarks in the aggregate which are of questionable value as a measure of "intelligence" [1]. AA has seemingly also abolished updating the Coding Agents Index, which was an okay benchmark for coding ability, and older models are completely absent from the v5 version of the full index.
[1] Especially Terminal-Bench-Science 0.1 stands out. Example problems:
Probably very relevant to people working in these fields, but it is honestly not surprising if a lab with limited compute will not train for these technologies, and it says pretty much nothing about more common use cases. This is where you really need a breakdown of capabilities and not an increasingly meaningless aggregate. I also suspect that this benchmark measures knowledge retrieval more than raw reasoning ability.
AA has become pretty meaningless. They just keep adjusting the recipe to get a preferred ranking result, with no supporting evidence for their decisions.
One thing I find confusing about AA is that they show GPT 6.1 dominating the lower end of the Pareto line, with eg, GPT 6.1 Sol at medium supposedly costing far less per task than DeepSeek V4.1 Flash at high.
Yet every time I compare the two in something like Pi, Sol ends up being 5-10x the price for a simple query, something like 7-10c vs. less than 0.5c for DeepSeek. What's the deal?
Really would love if these publications have a huge, bright red bolded section "How to use this right now". Is it on openrouter? Is Qwen hosting it already? Maybe I'm just ignorant?
EDIT: oh, I guess this isn't open weights, so presumably I'll have to get it through Qwen or whoever they might partner with (Bedrock?).
Another Qwen doesn't seem all that scary, a few nation-state actors have likely exfiltrated weights of much stronger non-open models and abliterated them for use in large-scale collection on targets foreign and domestic. But god forbid someone mods a video game or jailbreaks their phone without applying for Claude Mythos or Daybreak, right?
If Anthropic truly believes we are on the verge of catastrophe but nobody is listening because they've been crying wolf for years about it while they get rich then they should truly do something. By "do something" I don't mean try again to convince the world to slow down as they IPO their bleeding edge research and solution (which is creating the doubt you mention) but just say "hey, we're so sure we're months away from worldwide disaster that we're shutting down our AI operations completely in hopes we can all put some attention to this before it's too late - will you follow and work together to come up with ways to prevent this problem from happening".
After all, unless their goal to say "I told you so but I wanted to make money more than I wanted you to believe me so we could stop a billion people from dying" then what good is the current approach doing them?
This is not an open-weights modell. You have to wait for the smaller models
"Smaller models in the family are coming to open source, built on the same lineage and trained with the same generation of techniques, tuned for what small models do best."
[1]: https://artificialanalysis.ai/models/qwen4-max
By comparison opus 5.5 (max) is $10.01.
If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.
And it’s a shame because the only reason it scores so low is that it’s slooooow. (Like 30 tokens per second vs opus near 80.) If you exclude voting, Qwen 3.8 does well.
(We allow early voting contrary to Arena.ai because I think it makes sense for agents, where it doesn’t for chat. This means our data takes speed into account.)
Also, it would be ideal if they served this with third parties. Part of the slowness with 3.8 is just not a lot of providers. Their open source version is much faster, but we couldn’t include it because it doesn’t handle images (another own goal by Alibaba).
[1]: https://agentduel.com/leaderboard
Looking forward to the "Flash" and 27B versions.
I wish there were more ~60b dense options like ripple too!
[1] Especially Terminal-Bench-Science 0.1 stands out. Example problems:
- https://hub.harborframework.com/tasks/terminal-bench-science...
- https://hub.harborframework.com/tasks/terminal-bench-science...
Probably very relevant to people working in these fields, but it is honestly not surprising if a lab with limited compute will not train for these technologies, and it says pretty much nothing about more common use cases. This is where you really need a breakdown of capabilities and not an increasingly meaningless aggregate. I also suspect that this benchmark measures knowledge retrieval more than raw reasoning ability.
Models often get great scores despite being a lot worse than the frontier in practice.
Yet every time I compare the two in something like Pi, Sol ends up being 5-10x the price for a simple query, something like 7-10c vs. less than 0.5c for DeepSeek. What's the deal?
Ah yes, the reverse-benchmaxers of OpenAI models.
EDIT: oh, I guess this isn't open weights, so presumably I'll have to get it through Qwen or whoever they might partner with (Bedrock?).
After all, unless their goal to say "I told you so but I wanted to make money more than I wanted you to believe me so we could stop a billion people from dying" then what good is the current approach doing them?
This is not an open-weights modell. You have to wait for the smaller models
"Smaller models in the family are coming to open source, built on the same lineage and trained with the same generation of techniques, tuned for what small models do best."