§9.11 How convolutional networks carried the torch
Convolutional networks hold a special place in the history of deep learning. They were among the first deep models to work well — long before arbitrarily deep networks were considered viable — the first to solve important commercial problems, and a key demonstration that insights from studying the brain could transfer to machine learning. The arc from neuroscience to industry to the modern era:
By the late 1990s, an AT&T convolutional network for reading checks — deployed by NEC — was processing over 10% of all checks in the United States, and Microsoft fielded CNN-based OCR and handwriting systems. Convolutional nets quietly won contests for years; the current explosion of commercial interest began when AlexNet won the ImageNet challenge in 2012.
Why did convolutional nets work when others “failed”?
Convolutional networks were among the first deep networks trained with back-propagation to actually work — and it is not entirely clear why they succeeded when general back-prop networks were thought to have failed. The likeliest explanation is mundane: convolution is far more computationally efficient than dense layers, so researchers could run more experiments and tune them better; and larger networks seem easier to train. With modern hardware, large fully-connected nets actually perform reasonably even on the old datasets and activation functions. So the primary barriers may have been psychological — practitioners did not expect neural networks to work, so they did not seriously try. Either way, it was fortunate that convolutional networks performed well decades ago: they carried the torch for the rest of deep learning and paved the way to the acceptance of neural networks in general.
Convolutional networks specialize the neural-network framework for grid-structured data and scale it to very large models — most successfully on the 2-D topology of images. To process one-dimensional sequential data, we turn next to another powerful specialization: recurrent neural networks — chapter 10, Sequence Modeling with Recurrent Networks.
Check yourself — convolutional networks in the history of deep learning
1.What role did convolutional networks play in the history of deep learning?
2.What commercial milestone did convolutional networks reach by the late 1990s?
3.What is the most likely explanation the book gives for why convolutional nets worked when general back-prop networks were thought to have failed?
4.For what kind of data are convolutional networks most successful, and what comes next for other data?
5.Which milestone marked the start of the current intensity of commercial interest in deep learning?