§7.6The Whole Pipeline, Beyond Extractive QA & Conclusion

Evaluating the whole pipeline

We measured retriever and reader separately; now chain them. Fixing the retriever at k = 10, the reader receives multiple contexts per query (unlike the clean SQuAD-style single-context setup), so end-to-end EM/F1 drop:

Going beyond extractive QA

So far the answer is always a literal span. But sometimes the answer is scattered across passages and needs synthesizing. Generative (abstractive) QA generative qa Abstractive QA where a model generates a free-form answer rather than extracting a span from the text. defined in ch. 7 — open in glossary generates the answer instead of extracting it. The state of the art is retrieval-augmented generation retrieval-augmented generation Generative QA (RAG) that feeds DPR-retrieved documents to a seq2seq generator (T5/BART), trainable end-to-end. defined in ch. 7 — open in glossary (RAG): keep the retriever, but swap the extractive extractive qa Question answering where the answer is a contiguous span of text located within a document. defined in ch. 7 — open in glossary reader for a generator — a seq2seq transformer (T5 or BART) — fed by DPR dense passage retrieval A dense retriever (DPR) using two BERT encoders — a bi-encoder — to embed the question and the passage and score them by similarity. defined in ch. 7 — open in glossary .

Because DPR and the generator are both differentiable, the whole thing fine-tunes end-to-end (the context encoder stays frozen). There are two variants:

from haystack.generator.transformers import RAGenerator
from haystack.pipeline import GenerativeQAPipeline

generator = RAGenerator(model_name_or_path="facebook/rag-token-nq",
                        embed_title=False, num_beams=5)
pipe = GenerativeQAPipeline(generator=generator, retriever=dpr_retriever)
# "What is the main drawback?" -> "the price" / "no flash support" / "the cost"

RAG produces sensible, phrased answers (“the price”, “no flash support”), though subjective questions can still confuse it.

Conclusion

We built an end-to-end QA system: an extractive retriever-reader pipeline over customer reviews, compared BM25 vs DPR retrievers, boosted the reader with domain adaptation, measured everything with recall / EM / F1, and previewed generative QA with RAG.

A useful way to prioritize QA work is the hierarchy of needs — nail search first, add extraction next, and reach for generation only once the other two are exhausted:

The reader is useful beyond on-demand queries too — teams have used it to auto-extract product pros/cons and even do zero-shot NER via questions like “What kind of camera?”. Looking ahead, exciting frontiers include multimodal QA (over text, tables, and images — see §11.3) and QA over knowledge graphs.

That completes our tour of question answering. Next, we turn to where transformers are headed: scaling, efficiency, and new modalities.