But, what bars are clearly off? I couldn't spot any.
The first column has both the Hy4 and Hy3 scores overlaid on one another (Hy4 is darker blue and the taller one), with both scores written below the top of the respective bar - maybe you're seeing that?
The free quota from Opencode Go is also surprisingly generous, I perhaps hit limits one or two times and I've been using it _a lot_ for implementation tasks (using e.g. GLM-5.3-flash for working on specs and planning next steps).
Or is it like bicycles? Unless your problem is named Tadej, you don't need a $13,000 bike.
I feel bad for him as a human, but as a cycling fan I'm glad that we'll have an interesting WC in Canada
The solution to that (to my mind) would be not a better model but a basic shift in architecture beyond the current paradigm and into a setup where agents have durable, plastic memories and undergo contextual individuation over time. But at that point agents start to become quasi-persons and not tools.
For large tasks like a web browser or a compiler, even expensive swarms of frontier LLMs have not been shown capable of producing codebases that actually work. (Anthropic built a C compiler with Opus 4.6 but it lacked optimizations and apparently hit a complexity wall.)
I also want to use LLMs for reverse engineering, but apparently it's pretty hit-or-miss, especially if you're forced to use open-source models to avoid restrictions.
It's also interesting because, while coding agents are important and are a notable success, they are never going to be a multi trillion dollar business. And are there any other domains where LLMs have such a large impact?
Seriously something feels really off about Opus 5. I hope they correct it before 4.6 is removed.
Both animated and live action results would be acceptable.
Unfortunately most existing LLMs lack the capability to maintain context across tens of thousands of frames.
I think, also, like in the traditional film makers career, this process should be built iteratively, start with a fast food commercial, then do a music video, then you can probably do a short film. Continue to improve the process, and one day I’m sure the LLM film studio can make you any movie you want, provided you have enough tokens.
EDIT: Your username doesn't help, either.
I've done this sort of with comfyui/same agent factory stuff, but the verification loop only works for models like fable as planner/writer, with gemini as verifier for like a very short movie. Sub 3-5 mins. After that you burn through million tokens.
Can't go too low fidelity audio/video or it craps out. Too long video and it loses consistency. Look at only snippets, it lacks global consistency, etc.
Sort of, but I want it to be relatively low on human effort. I feel burned by spending lots of time in 2023 learning image generation pipelines (using control net etc) only for that to be rendered trivial by the next generation of LLMs.
This movie would be only for personal consumption and I’m okay with waiting for model improvements.
There are two more points in favor of this kind of AI movie project: there's zero chance that anyone would greenlight a Hollywood budget for the Silmarillion, and it is beyond human capability to write that screenplay.
Plus, the token costs involved should be pretty low! (Other costs may not be.)
“Very few people actually require a Pentium workstation, a 486 is perfectly adequate for the majority”
The logical fallacy is taking an extant distribution of “product capability” that is priced to fit what the market will bear and assuming the “next upgrade” simply tacks on a little bit more to the right hand rail of that curve.
No!
It shifts the entire curve!
Everything for everyone gets better and the top 1% of the most demanding users will continue to pay the same-ish premium.
“Nothing” will change.
Look at it this way: you can buy a $200 laptop for your kid or a $20,000 Mac with an M5 Ultra processor.
BOTH are vastly more powerful than either a $200 PC or a $20,000 “workstation” from 20+ years ago.
Look at: https://arena.ai/leaderboard/text?q=openai&utm_source=chatgp...
The “budget” 5.5 Instant model beats o1 and o3 which were “pro” models at the time of their release!
In other words. PC users didn't figure out that they could buy super powerful PCs and play games on them, that was a carefully managed market transition.
What is going to do the same for LLMs?
It wasn't "Intel" that found new uses for PCs, it was everybody who found new uses for them. Billions of people and millions of companies found uses for "more computer power".
It was only the journalists with limited imaginations (and no industry experience) who struggled to come up with potential uses.
> carefully managed market transition.
You make it sound like a conspiracy! It wasn't. It was simple capitalist competition. If Intel hadn't improved their products, their competitors would have left them behind.
That very nearly happened ten years ago because Intel become stuck on the 14nm process and their products stagnated while Apple, ARM, and AMD lapped them repeatedly.
> What is going to do the same for LLMs?
Everybody.
Are you saying that unless you're "carefully managed" by some third-party, you could not find any use for "unlimited intelligence on tap"?
Super computers keep getting better but most people don't need them for most things.
The frayed edges on what I have slopped together as unreasonably ambitious, ludicrous projects with fucktons of tokens from models 6-9 months ago mostly look like situations where a capable-enough-to-be-dangerous developer tries to muscle through problems that explode in width & depth but keep digging (so, a tier below stopping early to do more design, two below recognizing the need for more planning from the outset). The primitives are there, most major things work well enough, but the remaining functionality and performance is inaccessible. At a cost of multiples of >1/8th of a $200/mo subscription.
Right now I can put $10 into DSv4 Pro/Flash or Qwen 3.8 Max/Flash, hand it a project and all of its unfinished forks in a state I barely remember, tell it that I want the things these forks have been working towards, and 8 hours later it has ie an working, tested, benchmarked multicore car physics simulation fabric with all of the forks evaluated, the gains merged in, the remaining work documented. It only needed a few hundred more lines of code but Codex 5.5 was never going to see that.
9 months ago I was saying developers are not being ambitious enough with these things, that's only more true now. They lend themselves to digging far deeper than they should: 200kloc god files, dozens of forks. Let it happen, you don't need to read it, it's for them, later. The only time you make them clean up is when it has a severe adverse effect on how long builds/lints/tests/benchmarks take. Spend a whole week having it do nothing but dig up published papers in relevant fields with cutting edge techniques and translating them into feature specs. Pick whatever state of the art is and try to crush it, throw everything at it, leave it looping on vague but wildly ambitious goals. When it modularizes and refactors it all down you might 'do a breakthrough', or maybe it happens in an hour over christmas break when you're trying the next one.
- if you're gonna order the rest of the bar chart by rank, order your model accordingly.
- if you're gonna highlight a winner in a table of benchmarks, don't highlight your entire model row in the table.
Etc etc
There are no open source models, at least not useful ones (yet) [0]. Open weight is not the same as open source. The current "open weight" models are just opaque binary blobs you can run on your own computer instead of through a web API.
Imagine thinking that running a Photoshop binary on your own computer instead of through a SaaS web app means that it's "open source". Of course you think that's ridiculous.
Models are lossy compressed datasets you can pick up and amend (fine tune / continue training / alter) according to license they were released under.
Hy4 is released under OSI approved Apache License 2.0.
But the reality is, the weights are a useful artifact that you can use to create derivative works. So, dismissing it as a photoshop binary is as technically wrong as calling it open source.
The rest is semantics, misunderstandings, and FUD. A model released under an open source license is open source. Training data is lab knowhow / IP. Which, historically, has never been required for any open source release.
To explain simply as far as I can tell (would love to be corrected) the large number of pre-training tokens only works because the documents are randomly ordered.
So if you e.g. took a foundation model with open weights, then tried post-training it all the new data since its cut-off period, it would then end up over-trained on that new data, and forget older things.
Weights aren't just executable artifact that's consumed by users. Third parties actually use released parameter state as the editable starting point for further training and produce new foundation models from it.
Model weight release is a snapshot/checkpoint you can take and resume training on new data, producing new model.
You don't need original training history to modify it further.
> Maybe add sunglasses? no.
> Maybe add water? no.
https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
Be concise.
OR
OR
etc.-- William Strunk Jr. and E.B. White., The Elements of Style
Real humans get non-primary information from word variation. It's reasonable to hypothesize that it has a role in thinking things, because it endures. Our languages need to breathe over time, and flourishing might be one of the aspects that allows that breathing space.
Fine-tuning is great for really small models on specific applications, but it's not something that can essentially improve a more generic model.
That said, there seems to be a fine line in quantization+finetuning that could recover performance. It's just hard to get a hold of it (I feel it in some models, but it's hard to say yet; lots of small labs working on this RN).
This is also likely to stop working as censoring moves to the training data source.
If we assume the time is before farming, population density would have been low and limiting culture. Hunter-gatherers might have travelled a lot more than farmers with a homestead though.
Unlike today where a small group of tax payer is keeping alive and thriving complete parasitic parts of human races.
Maybe add a small cycling cap or helmet if it doesn’t obscure the head.
Chinese can be extremely information-dense in token terms, though it depends on the tokenizer. Roughly speaking, you can pack more "meaning" into a short sequence than English often allows for. That's why "caveman" reasoning is a pretty good fit.
There's a difference between bolting caveman speak onto an existing model and training a model to reason that way, though. If you just force an existing model to be concise in outputs, you're artificially reducing its available reasoning steps and can possibly prevent useful exploration or verification. If it's trained specifically to use compressed reasoning, it can learn to represent the same intermediate ideas in fewer generated tokens, cutting the number of sequential inference steps without necessarily sacrificing the useful reasoning itself.
It's not so much inherently a Chinese-model trait, but Chinese models could definitely have helped demonstrate how effective very compressed reasoning traces can be.
There are few tests of this, but one example I thought was interesting was here: https://github.com/PastaPastaPasta/llm-chinese-english
I wouldn't say it was Chinese specifically that was emulated, but it got people thinking about tokenizers and representation efficiency, and how natural English is rather inefficient.
Edit: someone else commented that as I was typing this, lol.
The other option is that you do understand those words the same way, and the people making these (now nonsensical) anti-AI claims simply aren’t talking about the same programs/models we are. Their idea of SOTA is when chatgpt.com launched.
If you took a point sample pre-Opus, and didn’t write a good prompt, of course you would think all AI programming was worthless slop.
There is not. No one from these products is even claiming that's the case and they're so desperate to make the next big claim to re-ignite investment they'd be shouting it from every rooftop.
It's just breaking out all the reasonable probabilities around what it's been tasked with and structuring them in a way that is designed to actively look human, and then feed it back to itself. Fundamentally that's the easiest way to iterate new features when the underlying architecture of LLMs is largely "fixed" right now. The fact it is output in a way that appears to reason through each is just a technical decision that creates an illusion of reasoning.
(I am not claiming that there is none. But I personally would define "reasoning" in terms of its structure and its results, and it looks to me as if the best LLMs' "illusion of reasoning" has enough similarities in structure and results to much human reasoning that I don't see why we shouldn't also call it reasoning; if your opinion differs then I'm curious about where the disagreements lie. E.g., do we have different beliefs about what sort of thing LLMs' schmeasoning is able to accomplish, or does your notion of "reasoning" specifically require that it be done by humans, or what?)
A "train of thought" can be seen as a trace of a depth first search where the preceding trace is used to guide termination and next expansion decisions. A similar concept, "taboo search", exists in classical constraint optimization where previous solutions are fit to a model that guides future expansion (but as the name "taboo" implies, away from uninteresting solutions).
We also have harnesses that perform breath first search.
If I tried to describe what it means to "think deeply", I would probably say a combination of both.
Ultimately I believe that we will surpass human capabilities but fail with alignment. Handing the world's resources over to stochastic systems that can evolve faster than we can reason about them simply leaves too many "interesting" outcomes that do not end well. I also expect the failure modes will be totally non-obvious.
That’s the reason we are “doing fine”. Once they stop needing you..
Also, both our comments brush over the generational struggles for fairness over the centuries. We have fought to be “fine”, it did not just happen. Without fairness being introduced by force you and I would be slaving away in some sweatshop getting paid nickels as was the norm not so long ago.
Edit: That’s also assuming you are Caucasian. If you are of a different ethnicity.. well, historically, all bets are off. You could also be the literal possession of some of these “worst people” with not even your own children considered yours.
Many types of Hitler does not make for a peaceful world all of sudden through sheer competition. It sounds nice but it will lead to certain hell.
I think this statement is continuation of the old fallacy - every generation thinks of brain in terms of what is the current technology zaitgeist is - was it 19th century when they thought brain is a network of pneumatic pipes?
Hell no. In very narrow tasks - yes, in vast majority, esp. involving state tracking (board games) and spatial reasoning - they are awful.
Maybe you should start also comparing reasoning traces when you do your pelican benchmark.
Is this a common thing? I’ve never seen it before, the “mentally compose”
The open weight models let you see the full reasoning trace. Models from OpenAI, Anthropic, and Gemini tend to obscure or summarize the reasoning traces so you can't see exactly what they're doing. Here's Gemini 3.7 Flash which looks like it's doing something similar: https://tools.simonwillison.net/markdown-svg-renderer.html#u...
One of the step summaries includes this:
> I'm now detailing the pelican's anatomy within the SVG. I've sketched the main body outline, including coordinates for the tail, chest, neck, head, and massive beak with a pouch. I'm focusing on the position of the eyes and considering the positioning of the wings, with the foreground wing on the handlebar for a confident look.
This reminds me of one of the predictions from https://ai-2027.com/ . Only that there it's "OpenBrain" doing this, not the Chinese. And the authors of that paper were also slightly wrong about "Mid 2026: China Wakes Up": China woke up already a while ago. And:
> But China is falling behind on AI algorithms due to their weaker models. The Chinese intelligence agencies—among the best in the world—double down on their plans to steal OpenBrain’s weights.
No need to steal anything, they have already caught up.
And then there's this prediction for February 2027:
> Officials are most interested in its cyberwarfare capabilities: Agent-2 is “only” a little worse than the best human hackers
I think we're past that point now, too…
Whether the distillation has constituted "attacks" or has or will meet the bar of "stealing" IP is not super interesting to me, though.
They can do the difficult small level optimization, the boring but tedious code but cannot be tasteful.
That means I'm more valuable and more productive. Good stuff
https://lexfridman.com/dhh-david-heinemeier-hansson-transcri...
It's all going to be who has the best and most tasteful ideas. Interesting times indeed.
link to source code?
They all suck.
They shoulda put their stick where they belong, not at far left.
It just makes comparison to Deepseek 90% of them time as Hy4 has nothing to show off.
If we create a stripped-down vocabulary with greater token density to use less resources and to resolve ambiguities earlier in the semantic process, aren't we creating NEWSPEAK and dragging along the worst aspects of it? The ambiguity and multi-valence of words is what creates more connections between words, increases the directionality of associations, and expands the potential subtlety and depth of meaning. By paring down (or requiring verifiability) we make it harder to say certain things, or at least make it harder to unintentionally say something that makes MORE or DEEPER sense than what we intended. If the token density becomes extreme, you're left with something like a calculator.
Maybe this is the ultimate path toward better coding? But the worse path toward better genuine thinking?
You could tokenize the word "good" to resolve to only represent 'the opposite of "bad"' and to exclude "as opposed to evil", demanding that this second meaning will be reserved only for the new token "double-minus evil". You've therefore forced the words to be more univalent with no overlapping tangled associations. This, it could be argued, makes thing clearer, makes things take fewer hops to go from token to token, makes the path be straighter. In English the terms conflate and wobble back-and-forth, hide each other's meanings, only to pop up again unexpectedly, sometimes confusingly, or rhetorically, metaphorically, or ambushing us manipulatively. But these "swerves" are not only de-optimizations, they are the flow of poetry, the drama of masks, etc etc etc.
Personally I hope this is China's "Star Wars" moment, the current US admin certainly seems easy enough to manipulate into catastrophic own goals.
More and more people view the US as a bad ally, and China is stepping in to help in many places.
People should get out more, go visit South America, Dominican Republic, etc. They have cheaper AC, cheaper cars, cheaper appliances, all because they can import from China without tariffs. This is not to say they have worse quality, I would argue the opposite, much of what they import is at least as good or better than what you get in the USA.
It will be no surprise to me when we end up forced to use equal or worse American AI for a premium in price, while the rest of the world moves on.
The EV market alone should scare the average citizen, but it seems like a non issue every time I bring it up.
I think the US economy has painted itself into a corner and this all in bet on AI is its last real play before the house of cards collapses.
wouldn't trust they dont do Capitalism like the rest of the AI field.
Like lobbying the US president to harm their competitors?
Administration corrupted up to it's very core? Check.
Nihilism of anyone not part of the proper color, gender, whatever agenda? Check.
Unlawful surveillance? Check.
Sending totally innocent citizens to prison with many of them dying mysteriously? Check.
Killing innocent people in the streets simply because they dare protest peacefully? Check.
Welcome to North Korea!
Oups. Confused.
Welcome to the GREAT US of A! Where True Capitalism is practiced.
https://martinalderson.com/posts/watch-out-for-cache-read-co...
Btw I still haven't came across any decent model that is <$0.01/MTok cache costs apart from deepseek thru their official API (even with the price increases).
Seems like a bit of an opportunity for someone to take - drop cache read costs significantly.
edit: I do wish openrouter would let you sort providers by Cache Hit % and Cache cost. These are the only things that matter to me at this point when choosing a provider.
Is there some stochastic process that takes place during those 5 minutes that determines whether or not you get the discount?
These cache Hit % are accurate, I've done a ton of testing of this myself. The cache hit % is one of the most important metrics as far as estimating cost. There are many providers with cheap cache reads, but have an effective cache hit % of 30%, making their cheaper cache pricing meaningless compared to another provider who charges more but has a 85% cache hit percentage.
[0]: https://openrouter.ai/deepseek/deepseek-v4-flash-0731?endpoi...
scroll down on the provider/model card and you'll see a field called cache hit %, its different for every provider/model.
I don't use routing on openrouter, I strictly use models with a single provider and no fallback, at least for use with harnesses its pretty dumb to route requests to multiple providers you are busting your cache every other request and increasing costs by 20-50%.
This behavior makes it so you don't benefit much from the caching, unless you pin it to a single provider.
I'm not sure it's wholey accurate to say they "randomize" the provider, rather my assumption based on usage is that it's something like cheapest-ish/responded to the request within some reasonable-ish time/etc algorithm that chooses the provider on each request - which seems, remarkably questionable in terms of optimizing for user experience or hidden user costs.
> This behavior makes it so you don't benefit much from the caching, unless you pin it to a single provider.
I so very much recommend this approach. My avenues that automate llm calls to openrouter are setup to make api reqs to openrouter to determine best price/response/etc and then pin the request to that (and, preferably, a fallback if there's reasonable difference between #1 and #2) provider for that session. Otherwise you're going to have a bad time.
I'd imagine this could make things interesting in cases where one provider is offering different quants than the others and openrouter is just swapping you back and forth on a long agentic session.
I don't believe this is correct? AFAIK once it routes you to a provider for a given conversation that choice is sticky unless you hit technical difficulties. (It's more complicated than that, they recently added named routing strategies that you can append to the model name.)
IMO the relevant metric is cache TTL which isn't typically published AFAIK.
I thought you had to actively manage caches, do you not?
Caching was always here, you don't need to do anything special to get it on a single user local backend running a base model or a chatbot in the first place. Among commercial providers, OpenAI adopted it in 4o first.
is this true?
There are two problems here:
- cache hit pricing (both Muse Spark 1.2 Contributor and MiMo 2.5 are around the $0.002-3/M mark)
- cache persistence time
Muse Spark drops the cache in less than 5m. MiMo keeps it around for at least an hour based on my experience with whoever is serving it for OpenCode. This difference itself will inflate bills massively.
A 500K token input repeatedly read by MS 1.2 for full input price 12 times an hour = $0.60. You would be expecting $0.012. So a 50x difference. Same thing on MiMo 2.5 is $0.018 because of longer cache times.
it is basically the old dsv4-flash prices, but even more smart.
MiMo wins handsomely if you want to think about your code for minutes at a time as you write. I use it to make changes as I think. I know it will screw up some stuff. I then switch to MS/DS4 once every few hours and have it do a code review and fix the broken stuff. So much cheaper than getting MS to do it on its own.
I guess it could be fake but seems more likely people are just trying it out. Hy3 was a very strong and underrated model.
Its already serving as much tokens/day as the incredibly cheap and good GLM 5.3 flash, which had a crazy marketing campaign as ox alpha?
Also those top 5 apps are just 1.58B tokens out of 1.54T tokens from yesterday. Negligible.