§11.3Going Beyond Text: Vision & Tables

Text has taken transformers a long way, but training only on text has limits: reporting bias (text frequencies ≠ real frequencies), missing common sense (rarely written down), unreliable facts, and — the root issue — no other modality (audio, vision, tables). Solving modality could address the rest. This section (and the next) show transformers branching out.

Vision

Convolutional nets (CNNs) long owned vision, but transformers have caught up.

iGPT (“image GPT”) applies the GPT recipe to pixels: treat an image as a sequence of pixels and autoregressively predict the next one. Pretrained on large image sets, it can “autocomplete” a partial image, and adding a classification head makes it a solid classifier.

ViT

Vision Transformer (ViT) is the BERT-style counterpart. The trick is turning an image into a sequence of tokens:

Each patch embedding patch embedding The linear projection of a fixed-size image patch into a vector — ViT's analogue of a token embedding. defined in ch. 11 — open in glossary is the linear projection of a patch — ViT’s analogue of a token embedding. After that, it’s the same encoder we built in §3.4:

Pretraining masks some patches and predicts their average color (a visual masked-language-modeling). ViT doesn’t beat CNNs on small data (ImageNet) but scales better on large datasets. Using it is a one-liner — the image-classification pipeline classifies a photo of a dog:

from transformers import pipeline
image_classifier = pipeline("image-classification")
preds = image_classifier(image)   # a photo of a husky

A natural next step is video — models like TimeSformer add a temporal attention dimension on top of the spatial one.

Tables

Lots of data lives in tables, not prose. TAPAS (“Table Parser”) does question answering over tables by feeding the flattened table and the query into a transformer. Take this table of contents:

TAPAS answers natural-language questions about it — and crucially predicts an aggregator (NONE / SUM / AVERAGE / COUNT) when the answer needs computation:

The magic is letting a non-programmer ask “how many chapters have more than 20 pages?” instead of writing Pandas. TAPAS is arguably multimodal multimodal Describing models that combine more than one input modality, such as text with images or text with audio. defined in ch. 11 — open in glossary — but the table is still treated as text. Genuinely combining two modalities is the subject of speech-to-text and vision-plus-text.