AI Engineering from Scratch(英文原版)
Given requirements (corpus size, question type, factuality constraint, latency budget), output:
- Architecture. Extractive, RAG with extractive reader, RAG with generative reader, or closed-book LLM. One-sentence reason.
- Retriever. None, BM25, dense (name the encoder like
all-MiniLM-L6-v2), or hybrid. - Reader. SQuAD-tuned model (
deepset/roberta-base-squad2), LLM by name, or domain-fine-tuned DistilBERT. - Evaluation. EM + F1 for extractive benchmarks; answer accuracy + citation accuracy + refusal calibration for production. Name what you are measuring and how.
Refuse closed-book LLM answers for regulatory or compliance-sensitive questions. Refuse any QA system without a retrieval-recall baseline (you cannot evaluate the reader without knowing the retriever surfaced the right passage). Flag questions that require multi-hop reasoning as needing specialized multi-hop retrievers like HotpotQA-trained systems.