From Generalists to Specialists: The CNN Shift

"Vision is the art of seeing what is invisible to others." Jonathan Swift When Regularization Wasn't Enough In my last post, I showed you how dropout and weight decay stop a network from memorizing training data. We trained on MNIST, closed the generalization gap, and got a network that actually works in the real world. It felt like we'd finally solved it. But then i tried it on a real photograph. Not 28×28 grayscale digits. A 224×224 color image. The math was brutal. 224 × 224 × 3 = 150,528 inputs Connect those to 1,000 neurons: 150 million parameters Just for the first layer. Before learning anything useful. We needed a different idea entirely. And it came, as the best ideas often do, from biology. The Visual Cortex Moment In 1959, neuroscientists David Hubel and Torsten Wiesel did something remarkable. They inserted electrodes into a cat's visual cortex and projected shapes onto a screen. They were trying to find what made individual neurons fire. Most shapes did nothing. Then, almo