§11.4Multimodal Transformers: Speech-to-Text

So far we’ve extended transformers to one new modality at a time. True multimodal multimodal Describing models that combine more than one input modality, such as text with images or text with audio. defined in ch. 11 — open in glossary models combine two at once. We start with the most natural human interface after text: speech.

Speech-to-text

Talking is easier than typing for most people — and essential for accessibility — which is why assistants like Siri and Alexa keep spreading. The core task is automatic speech recognition automatic speech recognition (ASR) Converting spoken audio into text; wav2vec 2.0 is a transformer-based approach using unlabeled pretraining. defined in ch. 11 — open in glossary (ASR): turning spoken audio into text.

The wav2vec 2.0 family is a landmark. It pairs a CNN (to digest the raw waveform) with a transformer (to contextualize the audio), and — crucially — pretrains on huge amounts of unlabeled audio, so it needs only a few minutes of labeled speech to adapt to a new language:

Using it is the same familiar pipeline. Feeding it a clip from the SUPERB benchmark transcribes it perfectly:

from datasets import load_dataset
asr = pipeline("automatic-speech-recognition")
ds = load_dataset("superb", "asr", split="validation[:1]")
pred = asr(ds[0]["speech"])
# {'text': 'MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE
#           GLAD TO WELCOME HIS GOSPEL'}

Transformers can now “read” text and “hear” audio. Can they also see? Yes — the final frontier of this excerpt is vision-and-text models.