Maybe if you restrict it similarly to the Deepseek paper to "Gemini uses TPU for the final successful training run and for scaled inference" you might be correct, but there's no way that GPUs aren't involved for at minimum comparability and more rapid iteration reasons during the extremely buggy and error prone point of getting to the final training run. Certainly the theoretical and algorithmic innovations that are often being done at Google and do make their way into Gemini also sometimes using Nvidia GPUs.

GCP has a lot of, likely on the order of at least 1 million GPUs in their fleet today (I'm likely underestimating). Some of that is used internally and is made available to their engineering staff. What constitutes "deep learning for gemini" is very up to interpretation.

It’s never come even close to penciling out in practice.

For small models there are people working on this implemented in flash memory eg Mythic.

analog neural network hardware

physical neural network hardware

Put "this paper" after each one to get academic research. Try it with and without that phrase. Also, add "survey" to the next iteration.

The papers that pop up will have the internal jargon the researchers use to describe their work. You can further search with it.

The "this paper," "survey," and internal jargon in various combinations are how I find most CompSci things I share.

Arguably that's a GPU? Other than (currently) exotic ways to run LLMs like photonics or giant SRAM tiles there isn't a device that's better at inference than GPUs and they have the benefit that they can be used for training as well. You need the same amount of memory and the same ability to do math as fast as possible whether its inference or training.

Not really and for the same reason Chinese players like Biren are leapfrogging - much of the workload profile in AI/ML is "embarrassingly parallel", thus reducing the need for individual ASICs to be bleeding edge performant.

If you are able to negotiate competitive fabrication and energy supply deals, you can mass produce your way into providing "good enough" performance.

Finally, the persona who cares about hardware performance in training isn't in the market for cloud offered services.

Is anyone else getting crypto flashbacks?

I don't know. The transformer architecture uses only a limited number of primitives. Once you have ported those to your new architecture, you're good to go.

Also, Google has been using TPUs for a long time now, and __they__ never hit a brick wall for a lack of CUDA.

You do not need most of CUDA, or most of the GPU functionality, so dedicated chips make sense. It was great to see this theory put to the test in the original llama.cpp stack which showed just what you needed, the tiny llama.c that really shows how little was actually needed and more recently how a small team of engineers at Apple put together MLX.

Yes, in the sense that this is at least partially inspired by Apple's vertical integration playbook, which has now been extended to their own data centers based on custom Apple Silicon¹ and a built-for-purpose, hardened edition of Darwin².

¹ https://security.apple.com/blog/private-cloud-compute/ ² https://en.wikipedia.org/wiki/Darwin_(operating_system)