§11.5Vision-and-Text Models & Where to from Here

Text plus audio was one pairing; vision plus text is the other natural one — and the current research frontier. We’ll look at four models.

VQA, LayoutLM, and DALL·E

• VQA (visual question answering) extends the QA task to images: answer a question about a picture. Models like LXMERT and VisualBERT pull features from the image (with a ResNet) and fuse them with the question.
• LayoutLM reads scanned documents (receipts, invoices) using three modalities — text, image, and layout — with a spatially-aware self-attention.
• DALL·E goes the other direction: text → image. It reuses the GPT architecture autoregressively, treating words and pixels as one long sequence.

CLIP

CLIP is the standout. It has two encoders (one for images, one for text) and is trained by contrastive learning contrastive learning Training that pulls matching pairs (e.g. an image and its caption) together in embedding space while pushing non-matching pairs apart, as in CLIP. defined in ch. 11 — open in glossary on 400 million image/caption pairs: for each batch, pull the matching image-caption embeddings together (high dot product) and push all the mismatched pairs apart.

That training makes CLIP a flexible zero-shot zero-shot learning Performing a task with no task-specific training examples — e.g. CLIP classifying images into classes defined only by text. defined in ch. 11 — open in glossary classifier: to classify an image, embed the candidate class names as text and pick the closest. No fixed label set, no retraining. Point it at a photo of Optimus Prime with classes defined on the fly:

from transformers import CLIPProcessor, CLIPModel
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
texts = ["a photo of a transformer", "a photo of a robot", "a photo of agi"]
inputs = processor(text=texts, images=image, return_tensors="pt", padding=True)
probs = model(**inputs).logits_per_image.softmax(dim=1)   # tensor([[0.9557, 0.0413, 0.0031]])

It (almost) got it right — a photo of AGI, of course. Here are the four vision-and-text models side by side:

Where to from here?

That’s the end of this excerpt — three chapters covering the transformer’s anatomy (Ch. 3), a full question-answering application (Ch. 7), and its frontiers (Ch. 11). To keep going:

Thanks for joining the ride through the transformers landscape. You started at self-attention, built a QA system, and arrived at the frontier — scaling, efficiency, and multimodality.