If we cant' build the hardware fast enough. Can someone explain to me why there's apparently 50% of the gpu and memory that hyperscalers buy sitting in warehouses due to lack of data centers? The hyperscalers are hoarding tech and can't even use it. It's ridiculous.
That depends on how long it’s sitting there, given they are two different things to manage it makes sense that you’d either have gpus sitting unused while a dc gets built or a dc sitting empty while you wait for gpus.
Because slowing down compute purchase is collapsing the hype train which is the same as death. Vibes are a wave that must be maintained or crash down onto reality.
That's pretty innovative. But a brand new tokenizer and training method need a lot of effort to do the retraining. Whether it will be widely implemented depends on real frontier performance it can reach.
Exactly. I used to think new tokenizers for domain-specific models was the way (makes intuitive sense, better representations etc), but the added overhead of essentially training from scratch made it much more expensive than a FT.
This looks very interesting. Their new language modelling objective shows some large improvements over masked diffusion modeling.
However, they hardly compare against the conventional next-token-prediction objective, which suggests they don't beat it, except for highly non-linear tasks like sudoku. Unfortunately, machine learning papers often tend to focus exclusively on the positives without mentioning bad aspects.
I hope Apple Silicon will one day be competitive with GPUs. It dominates much of the CPU market, and it has a GPU, but it's still not even remotely comparable to Nvidia for inference (factor of >5 in some cases, especially in prefill) or even for gaming ("edge computing" in Nvidia terms).
I feel like their GPU architecture is going to need a significant redesign. There are limits to this tile-based mobile architecture.
I keep getting clickbait youtube videos about how "NCM" obsoletes "LLMs" and "Nvidia memory chips" (yes, it's that bad).
This makes me wonder what these people think an "LLM" even is. The name itself implies that it is large and about language. It does not imply that an "LLM" is inherently autoregressive or that it has any particular architecture even though most LLMs are transformers. Anything can be a large language model as long as it is large and models language.
NCM is basically a hierarchical diffusion transformer where the further layers take one big token and then generate two smaller tokens. So it is more like an evaluation strategy. You could apply this to any model architecture.
I personally think the name NCM itself is terrible because it contains the word "next" which inherently implies an ordering and somehow we are supposed to think that "LLM" which contains no mention of ordering, is constrained to ordered token production.
I'm not sure why but using X instead of the word "times" annoys me. Probably because one of the first things I got taught at university is that mathematical notation is not prose and shouldn't be used as such.
Different cultures have different norms and rules. I think you're saying that, in your native language, they taught you not to use mathematical notation in prose.
They didn't teach you that about English, though. It's quite common to write "21x" for "21 times" in English and is pretty much universally understood and accepted by native speakers.
The Oxford comma is the comma before the conjunction in a list of three or more terms, e.g. "one, two, and three". I think your sentence actually didn't need a comma anway.
However, they hardly compare against the conventional next-token-prediction objective, which suggests they don't beat it, except for highly non-linear tasks like sudoku. Unfortunately, machine learning papers often tend to focus exclusively on the positives without mentioning bad aspects.
I feel like their GPU architecture is going to need a significant redesign. There are limits to this tile-based mobile architecture.
This makes me wonder what these people think an "LLM" even is. The name itself implies that it is large and about language. It does not imply that an "LLM" is inherently autoregressive or that it has any particular architecture even though most LLMs are transformers. Anything can be a large language model as long as it is large and models language.
NCM is basically a hierarchical diffusion transformer where the further layers take one big token and then generate two smaller tokens. So it is more like an evaluation strategy. You could apply this to any model architecture.
I personally think the name NCM itself is terrible because it contains the word "next" which inherently implies an ordering and somehow we are supposed to think that "LLM" which contains no mention of ordering, is constrained to ordered token production.
And I don't think I've ever used the Oxford comma.
There was probably some rule about not beginning sentences with Or and And but you probably know better than I.
They didn't teach you that about English, though. It's quite common to write "21x" for "21 times" in English and is pretty much universally understood and accepted by native speakers.