Y Combinator

Creator perspective · Money

Zero-order training doesn't scale: gradient-estimate error grows linearly with model size

A key limitation of the SPSA-style solver: the error in the gradient estimate scales worse — linearly less accurate — as the model size grows. Because of this, you can't train big models with this zero-order optimizer: for a 10-billion-parameter model, the number of perturbations required would be extremely large and infeasible — certainly more flops than a forward pass plus a backward pass — so it gets really compute inefficient.

Y Combinator · What If We Stopped Using GPUs? | YC Paper Club

Claim from Y Combinator host

host's analysis

Report · Money

SPSA — a little-known finite-difference update rule — worked best among zero-order methods

The learning procedure that worked best is SPSA, which pretty much no one knows about except James Spall and his students who invented it. It works like Calc 1 finite differences: take two forward passes of a theta model, perturbing it in a random direction with radius epsilon (plus and minus), compute a finite difference, and scale the update in that random direction by the finite difference. Repeat for a number of perturbations: if the finite difference is zero, don't scale that direction at all; if huge, scale that direction a lot.

Y Combinator · What If We Stopped Using GPUs? | YC Paper Club

Claim from Y Combinator host

host's experimental report; provenance flagged possible numeric error on one atom

Principle · Money

Sharding caps gradient noise: ~64 perturbations per step suffice, and more sharding keeps improving

Sharding a model caps the gradient noise to an amount determined by the size of each expert, making perturbation-based training viable: only about 64 perturbations per training step (or fewer) are needed and it works completely fine. Scaling laws exist in three dimensions — number of perturbations, batch size, and degree of model sharding. Increasing batch size reduces gradient estimate noise, and sharding the model more and more continually improves performance — a previously unknown scaling dimension.

Y Combinator · What If We Stopped Using GPUs? | YC Paper Club

Claim from Y Combinator host

host's claims with approximate numbers

Creator perspective · Money

Backprop wouldn't work on alternative substrates — storing bits in light is very expensive and hard

The host argues that if you tried to do backprop in any alternative substrate — optics, neural, neuromorphics — it wouldn't work, because storing bits in light is very expensive and very hard. In optical computing, cost per gigaflop is way low (computation in light is like free; you can do lots of it), but you just can't easily store stuff. There are also non-differentiability and activation issues, and VRAM cost is very expensive. Bandwidth with light is pretty much instantaneous; neural compute could work over plain copper wire; each substrate will have different properties.

Y Combinator · What If We Stopped Using GPUs? | YC Paper Club

Claim from Y Combinator host

host's argument

Creator perspective · Money

Sharded perturbation training still loses to backprop because forward passes remain expensive — unless substrates make them nearly free

The reason sharded perturbation training loses to backprop is that the forward pass cost remains very expensive. But if forward passes were extremely cheap — optical (~free), neuromorphic (~20 watts), or analog — perturbation-based training without backprop could become competitive. Forward passes in optical computing are essentially free (just beams of light doing computation), and in neuromorphic computing consume about 20 watts.

Y Combinator · What If We Stopped Using GPUs? | YC Paper Club

Claim from Y Combinator host

hypothetical reasoning; provenance flagged possible numeric error on one atom

Method · Money

Training a brain culture to play a game via synchronous and asynchronous stimulation

The team trains a living neuron culture to play a game by sending synchronous stimulation when the culture does something good and asynchronous stimulation when it does something bad. Positive and negative feedback are defined relative to the game: asynchronous feedback reinforces that a specific action is unwanted. The culture is not optimizing for feeling good or bad; it optimizes for reducing entropy in the culture itself, and would self-organize to reduce the rate of asynchronous (high-entropy) stimulation it receives.

Y Combinator · What If We Stopped Using GPUs? | YC Paper Club

Method described by Y Combinator Paper Club speakers

Method preserved for inspection; not a verified protocol.

The setup is described as a reinforcement-learning-style analogy; the culture's optimization is characterized as entropy reduction, not reward seeking.

Report · Money

Post-2020, transformers put AI on NVIDIA's radar; Ampere began the memory-focused era

After 2020, GPT-2/GPT-3-era transformers demanded more memory capacity and bandwidth than ever before, putting AI on NVIDIA's radar. NVIDIA started caring much less about compute efficiency and much more about memory capacity and bandwidth, launching Ampere as the first chip of this memory-focused ilk.

Y Combinator · What If We Stopped Using GPUs? | YC Paper Club

Claim from Y Combinator host (about NVIDIA)

host's account; provenance flagged possible numeric error on some atoms

Creator perspective · Money

Surprise-based feedback scaling using a silicon critic and TD error

The team had to predict surprise: they use the critic in the silicon decoder to figure out how far an action is from the silicon critic's expectation and how surprising the action is. Feedback is scaled by surprise using the TD error, applied to both the frequency and amplitude of the feedback sent to the cells, not just the reward sign. This modulates feedback in a more equal way so they don't keep stimulating with negative feedback, allowing the cells to build connections over time.

Y Combinator · What If We Stopped Using GPUs? | YC Paper Club

Claim from Y Combinator Paper Club speakers

Described as the team's own implementation choice; the claimed benefit (more equal feedback modulation enabling connection growth) is their interpretation.

Principle · Money

Cost per gigaflop must drop significantly to reach human-level intelligence; the brain runs on ~20 watts

The host argues the cost per gigaflop has to come down by significant amounts to reach human-level intelligence, contrasting with the human brain, which draws about 20 watts — like a light bulb — while doing insane amounts of things.

Y Combinator · What If We Stopped Using GPUs? | YC Paper Club

Claim from Y Combinator host

host's argument

Reported position · Money

Jensen Huang reportedly didn't care about AI until 2020; NVIDIA was focused on flops-per-joule

According to the host's understanding, Jensen Huang really didn't care about AI at all until 2020. Before then, NVIDIA was mostly focused on getting more compute per joule (flops per joule), a focus the host links to CNNs (not memory- or bandwidth-bound) and, with uncertainty, to Ethereum crypto mining.

Y Combinator · What If We Stopped Using GPUs? | YC Paper Club

Position reported by Y Combinator host (about Jensen Huang / NVIDIA)

explicitly the speaker's understanding; crypto link flagged uncertain and asks to be corrected

Creator perspective · Money

Memory demand grows exponentially because attention is n-squared; compute efficiency has petered out

The host attributes the exponential growth curve of memory capacity and bandwidth to attention being n-squared. Meanwhile, GFLOP-per-joule improvements have petered out — in the last two years there has been little improvement in compute efficiency, so something else is needed. He quips that as long as a chip produces the flops, it will 'burn up all the oceans in the world' — energy cost is being ignored.

Y Combinator · What If We Stopped Using GPUs? | YC Paper Club

Claim from Y Combinator host

host's interpretation with hyperbole

Creator perspective · Money

Optics is probably the right answer for compute: conversions burn ~99.99% of joules

The host argues optics is probably the right answer for compute: we communicate to the data center in light, then hit a switch and convert back to electrons, then back to light through DACs and ADCs, burning 99.99% of the joules in the conversions. Why not do the whole computation in light? It would be instantaneous and draw no watts — and that's what the aliens are probably doing. He adds he's open to a lot of different ideas.

Y Combinator · What If We Stopped Using GPUs? | YC Paper Club

Claim from Y Combinator host

host's argument; provenance flagged possible numeric error

Creator perspective · Money

LSTMs and transformers were designed for backprop — maybe the wrong architectures for gradient-free training

The host notes LSTMs were designed to allow gradients to flow through them, so since his approach does only forward passes with no gradient flowing anywhere, maybe LSTMs aren't the right architecture. Similarly, the transformer was really built for backpropagation — its parallelization during training is beautiful, and it was designed with backprop and GPUs in mind.

Y Combinator · What If We Stopped Using GPUs? | YC Paper Club

Claim from Y Combinator host

host's interpretation with hedging ('maybe')

Creator perspective · Money

The field is deep in a rabbit hole of hardware, architecture, and optimizer co-adaptation

The host argues that deep learning is so far down a rabbit hole of hardware, software, and optimizer co-adaptation that the whole stack is entangled with one another, which motivates questioning the current substrate.

Y Combinator · What If We Stopped Using GPUs? | YC Paper Club

Claim from Y Combinator host

host's own framing

Creator perspective · Money

The brain is massively feed-forward, so backprop can't be happening in it

The host argues the brain is massively feed-forward: backpropagation would require a neuron to fire forward, then the brain to turn off and fire backwards to update synapses — and the brain certainly doesn't do that.

Y Combinator · What If We Stopped Using GPUs? | YC Paper Club

Claim from Y Combinator host

host's argument

Principle · Money

Perturbation-based training does not scale to big models

The speakers describe why perturbation-based optimizers lose to backprop: the error in the gradient estimate scales worse than linearly, becoming less accurate as model size grows, so you cannot train big models with it. Training a 10 billion parameter model would require an extremely large and infeasible number of perturbations, and the cost of forward pass remains really expensive. Sharding a model caps gradient noise to the size of each expert, requiring only about 64 perturbations per step for training to work fine.

Y Combinator · What If We Stopped Using GPUs? | YC Paper Club

Claim from Y Combinator Paper Club

scaling claims are the speakers' analysis

Creator perspective · Money

Entropy penalties instead of entropy bonuses in PPO

Traditionally in reinforcement learning you want a bit of an entropy bonus so you can start exploring the space around it. In their code they implemented entropy penalties, which is very abnormal for PPO. They actively penalized entropy because they found it was helpful and the cells provided sufficient entropy for exploration.

Y Combinator · What If We Stopped Using GPUs? | YC Paper Club

Claim from Y Combinator Paper Club speakers

The claim that penalizing entropy was helpful is the team's own finding, not an externally validated result.

Report · Money

SOMA: cluster Common Crawl with TF-IDF/SVD, ship one parquet per GPU, train small experts, route at test time

SOMA works by taking the Common Crawl dataset, clustering it with TF-IDF and some SVD, then shipping one parquet file per GPU to N GPUs throughout the world with N clusters, and training a small expert (a small LCM, like 41K) on each. At test time you pull all the experts together and route with a little router — like the benefit of MoE, except it's a mixture of models, not just one layer of feed-forward layers. The premise is asking: how far can this go?

Y Combinator · What If We Stopped Using GPUs? | YC Paper Club

Claim from Y Combinator host

host's description; provenance flagged possible numeric error

Principle · Money

SPSA: reproducible noise makes optimization learnable

The host explains that with SPSA, if you can make the noise reproducible from the same seed, the optimization becomes completely learnable. He notes SPSA requires two forward evaluations for forward difference (one to perturb, one to scale the gradient) and three for central difference. A Stanford neuroscientist, Shraddha Druckman, is cited on the main obstacle to zero-order methods being biologically plausible: you cannot justify how you get the same seed to a vector twice.

Y Combinator · What If We Stopped Using GPUs? | YC Paper Club

Claim from Y Combinator Paper Club host

evaluation-count claim flagged as possible numeric error in the source

Principle · Money

Co-design models with the physics of new hardware

The speakers argue you should build models around the specific physical behavior of new hardware systems to co-design something with an interesting result, working within the system's strengths and weaknesses rather than blindly applying an architecture. When starting from a unique model architecture instead, the advice is to pick the right physical manifestation to bring it to life. Whether to iterate model-first or hardware-first depends on where your angle comes from.

Y Combinator · What If We Stopped Using GPUs? | YC Paper Club

Claim from Y Combinator Paper Club

presented as endorsed heuristics