I have a tiny suspicion that a new technique is going to emerge soon that will cause some frustration for Nvidia's investors. For instance, observing how little quantization error seems to matter makes me believe there are probably some other massive wins lurking throughout. Before I'd invest, I'd consider the "doomsday" scenario wherein someone develops a technique for running an OpenAI, GPT-4-scale model on a MacBook or iPhone.
I know it probably seems impossible to many on HN that there could be another bucket of 3-4 orders of magnitude sitting on the table, but progress over the last ~6 months seems to provide a compelling argument against that point of view.
There are also radically-different architectures that have seen zero serious effort put towards them. Mostly things that are CPU-bound. In my view, GPU is starting to cause more harm than good with regard to development of new concepts for problem solving. There are neural network architectures that simply don't work well across the PCIe bus. Eventually, someone is going to start playing around with these ideas as GPU scarcity rages onwards.
> I'd consider the "doomsday" scenario wherein someone develops a technique for running an OpenAI, GPT-4-scale model on a MacBook or iPhone.
That's not a doomsday scenario for Nvidia. There is essentially infinite demand for better AI, only limited by what can be provided at acceptable cost. If you can run GPT-4 on a macbook, you can do even better with a more massive model. If you can run a good image model on a macbook, then the next frontier is running a good video model, etc.
The real doomsday scenario for Nvidia is that there seems to very little differentiating their hardware. Their lead in the space exists because they developed good software support early, which lead to everyone standardizing on them. But AI is not like graphics where the problem domain is complex and the APIs are very ill-defined, and you can do all these tricks to make it faster and better. Instead, AI is almost entirely doing just a handful of very simple operations. Other vendors should be capable of providing good software support eventually, and at that point, what is Nvidia's moat? What justifies their margin?
Yes, you can run a better model if you have more hardware, but does a better model translate to a more compelling user experience?
Will people be willing to pay for slightly better AI images of giant cats using the Empire State Building as a scratching post if good-enough cats can be rendered locally.
Yeah but the challenge lately has been that efficiency gains have also made the capabilities of large compute clusters even more powerful. Transformers were introduced as an efficient alternative to RNNs and conv nets, then it turned out that however much they supercharged a single GPU, they were dramatically more powerful deployed at datacenter scale.
I thought the whole deal with machine learning was that inference is easy, but training is hard.
So creating the models from the data is what takes the billions of transistors on many cards.
running the models doesn't need so much hardware.
also... with respect to powerful processing - I haven't seen graphics cards tapering off. There's a ravenous demand for better graphics hardware each year. For every technique that is commoditized, the next year there's a new way of doing things that is better. I remember things like lighting, or realistic hair, or physics or whatever making a new graphics card better. Why wouldn't AI stuff be any different?
That's often true for classification models (like image recognition), but it the generation models (SD, Llama and GPT-4) everyone is excited about use a lot of compute for inference.
OP here. That’s exactly right. In the article, I describe CUDA and the rest of the software suite as their competitive moat to hardware competition from AMD or AI accelerators.
Yes, CUDA is their moat. And it's a very deep moat, filled with sharks, alligators, and mines. It will take a truly massive effort to unseat CUDA and NVIDIA as the dominant software/hardware tools for deep learning.
I'm curiously watching Intel and their OneAPI platform; but it just feels like they're starting from so far behind. And Intel hasn't exactly had a stellar few years on top of that.
I don’t think its CUDA. If it was CUDA people would use AMDs Radeon Open Compute and save tons of money on cards with more memory.
The issue is that the Radeon platform isn’t well developed, and AMD is charging as much as they can get for their cards just like Nvidia. if they had working competitive compute they’d also charge as much as Nvidia, and make more money on fewer cards sold.
I think there’s just something with card architecture that makes it not as good as Nvidia for this purpose. Just like mining bitcoins or VR, sometimes card architecture makes a difference.
Me too. My hope is that Intel, AMD, Google, MS and others eventually catch up with a common standard and the software doesn't require special customization as an end user (on paper something like DirectML has potential for this).
Another thing that could happen with so much money in the AI hype train right now, I wonder if we will see something similar to bitcoin, with special ASIC chips optimized for specific models like ChatGPT, CLIP or Stable Diffusion, or some shift to FPGAs.
Eh its more so that the entire ecosystem is built on CUDA. I don't think most developers would care or even notice if you swapped out CUDA for something that'll work seamlessly with the existing Python libraries. Sadly such an alternative doesn't exist and probably won't exist.
There wasn't a strong incentive to get either ML training or inference working on non-nvidia devices before. You can sure bet there is now though.
To expect that NVDA will be the only player for years to come is quite likely to be proven false. Google already has their foot in this game too via their TPU. The financial incentives have recently gotten an order of magnitude stronger
HIP is their effort to do that. Not sure they're putting enough dev and optimization effort to extract actual max perf from their hardware. So you're putting one (or 16) 300W board in your system with lower efficiency, risk of no support, bad long term support. And for (easier, more efficient) training you need large memory and hug interconnect. Only NVIDIA has this combination. I'll wait until I see XeLink bridged/switched servers actually in prod, benchmarkable by any noob, before I touch Intel GPUs for training. And if you were thinking H100s were unobtainium, try buying Intel Gaudi stuff...
Is there an open software standard that different AI-specific accelerators can all use?
Is there any investment in consumer-focused AI expansion cards or devices?
What about research into memristors and things? You would think if it were feasible at all that now there would be a huge research push to make radical compute-in-memory paradigms that into real products.
I know it probably seems impossible to many on HN that there could be another bucket of 3-4 orders of magnitude sitting on the table, but progress over the last ~6 months seems to provide a compelling argument against that point of view.
There are also radically-different architectures that have seen zero serious effort put towards them. Mostly things that are CPU-bound. In my view, GPU is starting to cause more harm than good with regard to development of new concepts for problem solving. There are neural network architectures that simply don't work well across the PCIe bus. Eventually, someone is going to start playing around with these ideas as GPU scarcity rages onwards.
That's not a doomsday scenario for Nvidia. There is essentially infinite demand for better AI, only limited by what can be provided at acceptable cost. If you can run GPT-4 on a macbook, you can do even better with a more massive model. If you can run a good image model on a macbook, then the next frontier is running a good video model, etc.
The real doomsday scenario for Nvidia is that there seems to very little differentiating their hardware. Their lead in the space exists because they developed good software support early, which lead to everyone standardizing on them. But AI is not like graphics where the problem domain is complex and the APIs are very ill-defined, and you can do all these tricks to make it faster and better. Instead, AI is almost entirely doing just a handful of very simple operations. Other vendors should be capable of providing good software support eventually, and at that point, what is Nvidia's moat? What justifies their margin?
Yes, you can run a better model if you have more hardware, but does a better model translate to a more compelling user experience?
Will people be willing to pay for slightly better AI images of giant cats using the Empire State Building as a scratching post if good-enough cats can be rendered locally.
New paradigm stuff. Infinite demand. It will be able to create a perpetual motion machine easily.
So creating the models from the data is what takes the billions of transistors on many cards.
running the models doesn't need so much hardware.
also... with respect to powerful processing - I haven't seen graphics cards tapering off. There's a ravenous demand for better graphics hardware each year. For every technique that is commoditized, the next year there's a new way of doing things that is better. I remember things like lighting, or realistic hair, or physics or whatever making a new graphics card better. Why wouldn't AI stuff be any different?
2. If AI gets more capable/efficient, the demand will increase not decrease.
Did weapons manufactures go out of business when machine guns were invented?
Some certainly did.
That alone proves your first point wrong.
There are brand agnostic alternatives, but they are less popular and usually slower.
I'm curiously watching Intel and their OneAPI platform; but it just feels like they're starting from so far behind. And Intel hasn't exactly had a stellar few years on top of that.
https://www.modular.com/engine
The issue is that the Radeon platform isn’t well developed, and AMD is charging as much as they can get for their cards just like Nvidia. if they had working competitive compute they’d also charge as much as Nvidia, and make more money on fewer cards sold.
I think there’s just something with card architecture that makes it not as good as Nvidia for this purpose. Just like mining bitcoins or VR, sometimes card architecture makes a difference.
Another thing that could happen with so much money in the AI hype train right now, I wonder if we will see something similar to bitcoin, with special ASIC chips optimized for specific models like ChatGPT, CLIP or Stable Diffusion, or some shift to FPGAs.
To expect that NVDA will be the only player for years to come is quite likely to be proven false. Google already has their foot in this game too via their TPU. The financial incentives have recently gotten an order of magnitude stronger
It may be hard but no reason why it can’t be done.
Except AMD doesn’t seem very good at doing software.
Thought maybe an open source effort could do it.
Is there any investment in consumer-focused AI expansion cards or devices?
What about research into memristors and things? You would think if it were feasible at all that now there would be a huge research push to make radical compute-in-memory paradigms that into real products.