§7.5Evaluating the Reader & Domain Adaptation

With the retriever measured, we turn to the reader reader The QA component (a reading-comprehension model) that extracts an answer span from retrieved documents. defined in ch. 7 — open in glossary . How do we score an extracted answer against the ground truth?

Exact Match and F1

Extractive QA uses two metrics:

• Exact Match exact match A strict QA metric (EM): 1 if the predicted answer exactly equals the ground truth after normalization, else 0. defined in ch. 7 — open in glossary (EM) — a strict binary: 1 only if the prediction equals the gold answer exactly (after normalization), else 0.
• F1 f1-score The harmonic mean of precision and recall; for QA, computed over the token overlap between predicted and gold answers. defined in ch. 7 — open in glossary — the harmonic mean of token-level precision and recall, so partial overlaps still get partial credit.

Both first normalize (lowercase, strip punctuation, fix whitespace) and compare as a bag of words. Edit the prediction below to feel the difference — one extra token zeroes EM, and F1 can be generous with wrong-but-overlapping answers:

For pred = "about 6000 hours" vs gold = "6000 hours", EM = 0 (the extra “about”) but F1 = 0.8. Swap in "about 6000 dollars" and F1 falls to 0.4 — so tracking both metrics is important: EM underestimates, F1 overestimates.

from farm.evaluation.squad_evaluation import compute_f1, compute_exact
compute_exact("6000 hours", "about 6000 hours")   # 0
compute_f1("6000 hours", "about 6000 hours")      # 0.8

Since a question can have several valid answers, the score is the best over all gold answers, averaged across all question-answer pairs. Haystack’s EvalAnswers node reports top_1_em and top_1_f1.

The domain gap

Running our SQuAD-fine-tuned MiniLM on SubjQA, the scores are much worse than its SQuAD 2.0 numbers (EM 76.1, F1 79.5). Two reasons: customer reviews are informal and unlike Wikipedia, and SubjQA’s questions and answers are subjective. Models are also known to overfit to SQuAD specifically.

Domain adaptation

The fix is domain adaptation domain adaptation Further fine-tuning a model on the target domain's data to close the gap left by training on a different domain. defined in ch. 7 — open in glossary : fine-tune the reader further on SubjQA. FARMReader.train() expects the SQuAD JSON format, which groups question-answer pairs under each product’s reviews:

# create_paragraphs() builds, per product, an array of {context, qas}:
[{"qas": [{"question": "How is the bass?", "id": "...",
           "is_impossible": False,
           "answers": [{"text": "The only fault in the sound is the bass",
                        "answer_start": 650}]}],
  "context": "I have had many sub-$100 headphones ..."}]

reader.train(data_dir=".", use_gpu=True, n_epochs=1, batch_size=16,
             train_filename="electronics-train.json",
             dev_filename="electronics-validation.json")

The payoff is large — domain adaptation increased EM by ~6× and more than doubled F1:

Why not fine-tune on SubjQA alone? With only 1,295 examples the model overfits; the SQuAD pretraining supplies broad reading comprehension that SubjQA then specializes.

Both components are now tuned. Next we measure the whole pipeline together and look beyond extractive QA.