§7.2Extracting Answers from Text

To answer “Is it waterproof?” from the review “This watch is waterproof at 30m depth”, our model must output the span “waterproof at 30m”. Doing that requires three things: framing the learning problem, tokenizing for QA, and handling contexts longer than the model can read.

Span classification

The standard framing is span classification span classification Framing extractive QA as predicting the start-token and end-token of the answer span. defined in ch. 7 — open in glossary : the model predicts, for every token, a start logit and an end logit; the answer is bounded by the argmax of each. The QA “head” is just a linear layer on top of the encoder’s hidden states — QA is a form of token classification.

Since SubjQA has only 1,295 training examples, we don’t train the head from scratch — we start from a model already fine-tuned on SQuAD squad The Stanford Question Answering Dataset, the canonical extractive-QA benchmark; v2 adds unanswerable questions. defined in ch. 7 — open in glossary (the label structure is identical across QA datasets). A few good baselines:

We’ll use MiniLM (deepset/minilm-uncased-squad2) for fast iteration.

Tokenizing text for QA

QA inputs are (question, context) pairs. The tokenizer packs them as [CLS] question [SEP] context [SEP] and adds a token_type_ids token_type_ids A per-token tensor marking which input tokens belong to the question (0) versus the context (1) in a QA pair. defined in ch. 7 — open in glossary tensor marking question tokens (0) from context tokens (1):

question = "How much music can this hold?"
context = "An MP3 is about 1 MB/minute, so about 6000 hours depending on file size."
inputs = tokenizer(question, context, return_tensors="pt")
print(tokenizer.decode(inputs["input_ids"][0]))
# [CLS] how much music can this hold? [SEP] an mp3 is about 1 mb / minute, so
# about 6000 hours depending on file size. [SEP]

Predicting the answer span

Running the QA model returns a start_logits and an end_logits tensor — one logit per token. Hover the bars: the start peaks on the quantities, the end on their units, and the two argmaxes bound the answer.

from transformers import AutoModelForQuestionAnswering
model = AutoModelForQuestionAnswering.from_pretrained(model_ckpt)
outputs = model(**inputs)
start_idx = torch.argmax(outputs.start_logits)
end_idx   = torch.argmax(outputs.end_logits) + 1
answer = tokenizer.decode(inputs["input_ids"][0][start_idx:end_idx])
# 'answer': '6000 hours'

Follow the shapes — QA is just two per-token logit vectors reduced to one span:

In practice you use the question-answering pipeline, which softmaxes the logits into a score, can return the topk answers, and — with handle_impossible_answer=True — maps “no answer” to a high score on the [CLS] token:

pipe = pipeline("question-answering", model=model, tokenizer=tokenizer)
pipe(question=question, context=context, topk=3)
# [{'score': 0.27, 'answer': '6000 hours'}, ...]

Dealing with long passages

A review can have more tokens than the model’s 512-token limit. Truncating (fine for classification, where the [CLS] token summarizes) is dangerous for QA — the answer might sit near the end and get cut off. The fix is a sliding window sliding window Splitting a long context into overlapping token windows (via stride) so no window exceeds the model's maximum length. defined in ch. 7 — open in glossary (return_overflowing_tokens=True, sized by max_seq_length, overlapped by doc_stride), which turns one long example into several overlapping (question, context-window) pairs:

tokenized_example = tokenizer(example["question"], example["context"],
                              return_overflowing_tokens=True, max_length=100, stride=25)
for i, w in enumerate(tokenized_example["input_ids"]):
    print(f"Window #{i} has {len(w)} tokens")
# Window #0 has 100 tokens
# Window #1 has 88 tokens

With answer extraction understood, the next step is retrieving the right reviews to read in the first place — the retriever-reader pipeline.