Ever needed to wade through oceans of documents for one fact? Search engines increasingly just tell you: ask Google “When did Marie Curie win her first Nobel Prize?” and you get 1903 in a snippet, no clicking required. The technology behind this is question answering question answering Finding the answer to a natural-language question, often by searching and reading a corpus of documents. defined in ch. 7 — open in glossary (QA), and this chapter builds a working QA system from scratch.
The QA snippet, conceptually
Google first retrieves ~319,000 relevant documents, then extracts the answer span from the best passage:The most common flavor is extractive QA extractive qa Question answering where the answer is a contiguous span of text located within a document. defined in ch. 7 — open in glossary : the answer is a span of text inside a document (a web page, contract, or — for us — a product review). The two-stage retrieve-then-extract process underpins semantic search, intelligent assistants, and more; we’ll build it up over the next few sections.
Other flavors of QA
Community QA matches your question to user-written answers on forums like Stack Overflow. Long-form QA generates paragraph answers to open questions (“Why is the sky blue?”). You can even do QA over tables — models like TAPAS (see §11.3) can aggregate to produce an answer.The use case: answering questions from reviews
Popular products have hundreds of reviews, so finding the one that answers “Does this guitar come with a strap?” is a drag. Instead of waiting days for a community answer, let’s extract it instantly from the reviews themselves.
The SubjQA dataset
We’ll use SubjQA: 10,000+ English customer reviews across six domains (TripAdvisor, Restaurants, Movies, Books, Electronics, Grocery). Each review is paired with a question answerable from one or more of its sentences. We focus on Electronics. What makes it hard — and realistic — is that most questions and answers are subjective (they depend on personal experience), and some are deliberately unanswerable.
Does the keyboard lightweight?
review (context)
I really like this keyboard. I give it 4 stars because it doesn't have a CAPS LOCK key so I never know if my caps are on. But for the price, it really suffices as a wireless keyboard. I have very large hands and this keyboard is compact, but I have no complaints.
- “this keyboard is compact”chars 212–236
The answer is a literal span in the review. Note the question is not grammatical — common in e-commerce FAQs.
Note — closed- vs open-domain QA
Closed-domain QA closed-domain qa QA over a narrow topic or small document set, such as a single product category. defined in ch. 7 — open in glossary answers questions about a narrow topic (one product category) and searches few documents. Open-domain QA open-domain qa QA over a very large, general corpus, such as an entire product catalog or all of Wikipedia. defined in ch. 7 — open in glossary answers questions about almost anything (a whole catalog, all of Wikipedia). Ours is closed-domain.Loading and exploring
SubjQA lives on the Hugging Face Hub. We pick the electronics subset and flatten
the nested answers dictionary into a DataFrame:
from datasets import get_dataset_config_names, load_dataset
import pandas as pd
get_dataset_config_names("subjqa")
# ['books', 'electronics', 'grocery', 'movies', 'restaurants', 'tripadvisor']
subjqa = load_dataset("subjqa", name="electronics")
dfs = {split: dset.to_pandas() for split, dset in subjqa.flatten().items()}
for split, df in dfs.items():
print(f"Number of questions in {split}: {df['id'].nunique()}")
# train: 1295 test: 358 validation: 255Only 1,908 examples in total — small, because getting domain experts to label QA data is expensive (the CUAD legal-QA dataset is valued at ~$2 million!). The columns that matter:
| Description | |
|---|---|
| title | The ASIN (Amazon product id) of the product |
| question | The question |
| answers.text | The span of text in the review labeled by the annotator |
| answers.answer_start | The start character index of the answer span |
| context | The customer review |
An answer span answer span The substring of the context marked as the answer, given by a start index and the answer text. defined in ch. 7 — open in glossary is stored as its text plus a start character index, so you can slice it straight out of the review:
start_idx = sample_df["answers.answer_start"].iloc[0][0] # e.g. 215
end_idx = start_idx + len(sample_df["answers.text"].iloc[0][0])
sample_df["context"].iloc[0][start_idx:end_idx]
# 'this keyboard is compact'An empty answers.text marks an unanswerable question. And glancing at how
questions begin, “How”, “What”, and “Is” dominate:
| Example question | |
|---|---|
| How | “How is the camera?” · “How fast is the charger?” |
| What | “What is the quality of the construction of the bag?” |
| Is | “Is sound clear?” · “Is it a wireless keyboard?” |
| Does / Do | “Does the keyboard lightweight?” |
The Stanford Question Answering Dataset (SQuAD)
SubjQA’s (question, review, [answer]) format was pioneered by SQuAD squad The Stanford Question Answering Dataset, the canonical extractive-QA benchmark; v2 adds unanswerable questions. defined in ch. 7 — open in glossary — Wikipedia paragraphs with crowd-sourced Q&A. In SQuAD 1.1 every answer existed in the passage; models soon beat humans. SQuAD 2.0 added adversarial unanswerable questions to raise the bar. Google’s Natural Questions pushes further with real search queries and longer answers. We’ll fine-tune on SQuAD 2.0 next.Now that we know our data, let’s see how a transformer actually extracts an answer span from text.
Check yourself: the QA task and dataset
1.In extractive QA, what form does the answer take?
2.How is an answer span stored in SubjQA, so it can be sliced from the review?
3.What does an empty answers.text entry mean in SubjQA?
4.Our review-based system searches reviews for one product at a time. Which QA setting is this?
5.What did SQuAD 2.0 add over SQuAD 1.1, and why?