§7.3Using Haystack to Build a QA Pipeline

In §7.2 we handed the model both the question and the context. But real users only give a question — so the system must find the relevant reviews itself. Concatenating all of a product’s reviews into one giant context is too slow (30 reviews × 100 ms ≈ 3 s per query). The answer is the retriever-reader architecture retriever-reader architecture The two-stage QA design: a retriever fetches relevant documents, then a reader extracts the answer from them. defined in ch. 7 — open in glossary .

Retriever and reader

• The retriever retriever The QA component that fetches the documents most relevant to a query. defined in ch. 7 — open in glossary fetches the documents relevant to a query. It comes in two flavors:
  • Sparse sparse retriever A retriever that represents query and document as sparse word-frequency vectors (e.g. TF-IDF, BM25). defined in ch. 7 — open in glossary retrievers represent query and document as sparse word-frequency vectors ( TF-IDF tf-idf Term frequency–inverse document frequency: a classic scheme weighting words by how often they occur in a document versus the corpus. defined in ch. 7 — open in glossary , BM25 bm25 'Best Match 25', an improved TF-IDF that saturates term frequency and normalizes for document length; the default sparse retriever. defined in ch. 7 — open in glossary ) and score by inner product — fast, exact-match.
  • Dense dense retriever A retriever that represents query and document as dense contextualized embeddings from a transformer (e.g. DPR). defined in ch. 7 — open in glossary retrievers use a transformer to embed query and document as dense contextualized vectors, capturing semantic meaning even when the words don’t match.
• The reader reader The QA component (a reading-comprehension model) that extracts an answer span from retrieved documents. defined in ch. 7 — open in glossary extracts the answer from the retrieved documents — our span-classification model from §7.2.

Building it with Haystack

We use Haystack (by deepset), built around this architecture. Beyond the retriever and reader it adds two components: a document store document store A document-oriented database that holds the documents and metadata served to the retriever at query time. defined in ch. 7 — open in glossary (a document database queried at runtime) and a pipeline (wires the components into a query flow).

The document store

Each store supports a different set of retrievers. We pick Elasticsearch because it supports both sparse and dense retrievers:

Elasticsearch (built on Lucene) indexes our reviews; we load them as {text, meta} dicts so we can later filter by product and split:

from haystack.document_store.elasticsearch import ElasticsearchDocumentStore
document_store = ElasticsearchDocumentStore(return_embedding=True)

for split, df in dfs.items():
    docs = [{"text": row["context"],
             "meta": {"item_id": row["title"], "question_id": row["id"], "split": split}}
            for _, row in df.drop_duplicates(subset="context").iterrows()]
    document_store.write_documents(docs, index="document")
# Loaded 1615 documents

The retriever

We start sparse, with BM25 (“Best Match 25”) — an improved TF-IDF tf-idf Term frequency–inverse document frequency: a classic scheme weighting words by how often they occur in a document versus the corpus. defined in ch. 7 — open in glossary that saturates term frequency quickly and normalizes for document length (favoring short, focused reviews):

from haystack.retriever.sparse import ElasticsearchRetriever
es_retriever = ElasticsearchRetriever(document_store=document_store)

item_id = "B0074BW614"          # an Amazon Fire tablet
retrieved_docs = es_retriever.retrieve(
    query="Is it good for reading?", top_k=3,
    filters={"item_id": [item_id], "split": ["train"]})
# each doc carries a relevance `score` (higher = better match)

The reader

Haystack offers two readers: FARMReader (deepset’s FARM framework — can fine-tune, doesn’t normalize logits across passages so scores compare across documents) and TransformersReader (inference-only, softmaxes per passage). Since we’ll fine-tune later, we use FARMReader with our MiniLM checkpoint:

from haystack.reader.farm import FARMReader
model_ckpt = "deepset/minilm-uncased-squad2"
reader = FARMReader(model_name_or_path=model_ckpt, progress_bar=False,
                    max_seq_len=384, doc_stride=128, return_no_answer=True)

The max_seq_len and doc_stride are the sliding-window controls from §7.2.

Putting it together

An ExtractiveQAPipeline chains one retriever and one reader:

from haystack.pipeline import ExtractiveQAPipeline
pipe = ExtractiveQAPipeline(reader, es_retriever)

preds = pipe.run(query="Is it good for reading?", top_k_retriever=3, top_k_reader=3,
                 filters={"item_id": [item_id], "split": ["train"]})
# Answer 1: I mainly use it for book reading
# Answer 2: the larger screen compared to the Kindle makes for easier reading
# Answer 3: it is great for reading books when no light is available

We have an end-to-end QA system! But the answers aren’t perfectly ranked — the best ones are second and third. To do better we need to measure each component, starting with evaluating the retriever.