§7.4Improving Our Pipeline: Evaluating the Retriever

Our prototype works, but the answers weren’t perfectly ranked. Since the retriever retriever The QA component that fetches the documents most relevant to a query. defined in ch. 7 — open in glossary sets an upper bound on the whole system — the reader reader The QA component (a reading-comprehension model) that extracts an answer span from retrieved documents. defined in ch. 7 — open in glossary can only extract from what it’s given — the first thing to measure and improve is retrieval.

Recall

The key retriever metric is recall recall In retrieval, the fraction of all relevant documents that appear in the retriever's top-k results. defined in ch. 7 — open in glossary : the fraction of questions for which a relevant document (one containing the answer) appears in the top-k results. Slide k below — recall climbs as we retrieve more documents:

At k = 3, BM25 already reaches 0.95 recall; there’s an inflection around k = 5, and recall is near-perfect from k = 10. But higher k means more documents for the reader to chew on, so we want good recall at small k.

To measure this we build a small evaluation Pipeline — each node has a run() method and named inputs — that pairs the retriever with an EvalDocuments node, then sweep k over the test set:

from haystack.pipeline import Pipeline
from haystack.eval import EvalDocuments

class EvalRetrieverPipeline:
    def __init__(self, retriever):
        self.retriever = retriever
        self.eval_retriever = EvalDocuments()
        pipe = Pipeline()
        pipe.add_node(component=self.retriever, name="ESRetriever", inputs=["Query"])
        pipe.add_node(component=self.eval_retriever, name="EvalRetriever",
                      inputs=["ESRetriever"])
        self.pipeline = pipe

# sweep k and record recall over the whole test set
def evaluate_retriever(retriever, topk_values=[1, 3, 5, 10, 20]):
    ...
    return pd.DataFrame.from_dict(topk_results, orient="index")
# Recall@3: 0.95

Dense Passage Retrieval

BM25 has a well-known weakness: it matches on exact terms, so a question phrased differently from the review can miss. The state-of-the-art alternative is Dense Passage Retrieval dense passage retrieval A dense retriever (DPR) using two BERT encoders — a bi-encoder — to embed the question and the passage and score them by similarity. defined in ch. 7 — open in glossary (DPR), a dense retriever dense retriever A retriever that represents query and document as dense contextualized embeddings from a transformer (e.g. DPR). defined in ch. 7 — open in glossary built from two BERT encoders — a bi-encoder: one embeds the question, one embeds the passage, each into a d-dimensional [CLS] vector, and their dot product is the relevance score.

The encoders are trained on (question, positive passage, negative passage) triples to make relevant pairs score higher. In Haystack, DPR plugs in like any retriever, then we re-embed every indexed document:

from haystack.retriever.dense import DensePassageRetriever
dpr_retriever = DensePassageRetriever(document_store=document_store,
    query_embedding_model="facebook/dpr-question_encoder-single-nq-base",
    passage_embedding_model="facebook/dpr-ctx_encoder-single-nq-base",
    embed_title=False)
document_store.update_embeddings(retriever=dpr_retriever)

Comparing recall head-to-head, though, DPR gives no boost here — it edges ahead at k = 1 but saturates around k = 3:

The retriever is in good shape. Now for the other half of the system — the reader, and how well it actually answers.