8. Retrieval & Knowledge

In the preceding vision and understanding layers, models primarily rely on "knowledge learned within their own parameters" to understand and generate content. In real business settings, however, many problems cannot be solved by "memory" alone: internal company policies change daily, regulations and industry standards are continuously updated, and a particular customer's history exists only in internal databases. In these cases, relying solely on what the model has "memorized" is far from sufficient — what matters more is whether the model can efficiently retrieve and reason over external knowledge bases, structured data, and knowledge graphs.

Think of this layer as adding an "external brain that can look up references and query databases" on top of the model's capabilities. When a user poses a question, the system no longer generates an answer directly; instead, it first goes to the appropriate data sources to "look things up": document repositories, databases, search engines, knowledge graphs, logs, and business systems... and then has the model produce an answer or decision based on the genuinely retrieved content. This not only significantly improves accuracy and timeliness but also substantially enhances explainability and compliance (for example, citing sources and preserving executed SQL records).

Within this layer, common capabilities broadly fall into two directions: one is Retrieval-Augmented Generation (RAG), primarily oriented toward "natural language Q&A + document/knowledge base retrieval"; the other is Structured Data & Knowledge Graphs (Structured Data & KG), responsible for more precise, controllable access and reasoning over databases, graph databases, and domain knowledge platforms. These are detailed below.

8.1 Retrieval-Augmented Generation (RAG)

RAG (Retrieval‑Augmented Generation) can be thought of as an "LLM that knows how to look things up." Unlike relying purely on a model's internal parameters, RAG, before answering each question, first retrieves from an external knowledge base, finds the most relevant document fragments (chunks), and then feeds those retrieved pieces as "context" to the LLM so that it generates an answer based on "having seen the material." For enterprise knowledge base Q&A, industry report search, legal/medical/financial professional Q&A, internal document search bots, and similar scenarios, RAG has become the default paradigm.

Architecturally, a typical RAG system can be broken down into three layers: the indexing layer, the retrieval layer, and the generation layer. The first two are primarily about "retrieving accurately," while the last is responsible for "articulating clearly." The following sections expand on these three layers, with further refinement of core design and practices in the subsections.

• Scenarios
  • Enterprise internal knowledge Q&A: employees ask questions in natural language about policies and procedures, technical documentation, or project materials; the system retrieves relevant content from internal documents and wikis, then the LLM generates clear answers with citations.
  • Industry report and research search: searching across large volumes of PDFs, reports, and papers for content related to an industry question (e.g., "changes in new energy vehicle subsidy policies"), and automatically summarizing, comparing, and listing sources.
  • Legal / medical / financial domain Q&A: retrieval augmentation based on authoritative materials such as legal provisions, court judgments, clinical guidelines, and product manuals, reducing the risk of "fabrication."
  • Internal document / ticket search bots: helping operations, customer support, and R&D quickly locate answers in knowledge bases, tickets, and change logs, then summarizing the results in natural language.
• Principles The core idea of RAG is "store knowledge externally, delegate reasoning to the model":
  • Break unstructured documents (PDFs, web pages, Word documents, technical docs, etc.) into chunks suitable for retrieval, map them into vector space using an Embedding model, and build vector indexes (e.g., FAISS, Milvus, PGVector).
  • At query time, leverage both semantic vector retrieval and keyword retrieval (Hybrid Search) to find the document chunks most relevant to the question, then re-rank them (Re‑ranking) based on relevance and coverage.
  • Feed the retrieved context, the user's question, and any necessary system instructions / format constraints into the LLM; the model answers under the constraint of "visible evidence" and includes source citations in its output to improve explainability and auditability.
• Models A typical RAG system is often a model composition architecture:
  • Embedding model: encodes queries and document chunks into the same semantic space and is key to vector retrieval quality (includes general-purpose embeddings and domain‑specific embeddings).
  • Retrieval and re‑ranking models: Hybrid Search (e.g., BM25 + Vector) handles the first round of recall; a Cross‑Encoder Re‑ranker or the LLM itself performs finer-grained re‑ranking on the recalled results.
  • Generation model: the LLM answers given the retrieved context; in more complex RAG / HyDE / ReAct + RAG setups, the LLM also participates in processes such as "pseudo-document generation," "multi‑turn tool calling," and "thought + retrieval alternation" to improve recall, reduce forgetting, and enhance reasoning ability.

8.1.1 Index Construction & Knowledge Asset Curation

In any RAG system, index construction is foundational. Without high‑quality indexing, even the most powerful LLM downstream is like "a skilled cook without ingredients." The goal of index construction is to transform messy document resources into "retrievable, maintainable, and scalable knowledge assets."

From a process perspective, typical index construction involves the following key steps:

1. Document chunking and preprocessing Documents are often long PDFs, PPTs, Word files, or web pages. Vectorizing an entire document at once both tends to cause "dilution" (one document spans multiple topics) and hinders efficient retrieval. Therefore, you need to:
   1. Chunk by paragraph, heading, page number, and chapter structure, balancing "semantic completeness" and "chunk size";
   2. Handle formatting issues (tables, formulas, OCR for text in images), denoise (headers, footers, tables of contents, copyright notices, etc.);
   3. Generate "context labels" for each chunk (e.g., source document, chapter title, page number) to prepare for subsequent interpretation and citation.
2. Embedding and vector index Based on the chunks, generate semantic vectors for each document chunk:
   1. Select an appropriate Embedding model (e.g., general‑purpose semantic embeddings, domain‑finetuned models) to ensure strong representation capability for the target language and domain terminology;
   2. Use FAISS, Milvus, PGVector, or similar tools to build high‑dimensional vector indexes that support approximate nearest‑neighbor search at scale;
   3. Handle multi‑version and incremental updates: when documents are updated, support incremental index rebuilds, version tracking, and stale‑version cleanup strategies.
3. Meta‑information indexing and filtering Semantic vectors alone cannot meet complex filtering needs; a meta‑information index is usually also required:
   1. Supplement each document chunk with metadata such as time, author, source, document type, line of business, and sensitivity level;
   2. Support pre‑filtering based on metadata during retrieval (e.g., time range, department, permission level) to reduce irrelevant results;
   3. Lay the groundwork for access control and auditing, preventing RAG from leaking content that the user is not authorized to view.

8.1.2 Retrieval & Re‑ranking: From "Recalling Relevant" to "Finding the Most Suitable Evidence"

Once index construction is complete, when a user submits a query, the system enters the retrieval and re‑ranking phase. The key here is not just "finding some relevant documents," but finding an evidence set that is both relevant, sufficiently comprehensive, and supportive of reasoning.

1. Hybrid retrieval: the complementarity of vector + keyword Pure vector retrieval excels at capturing semantic similarity, but for exact terminology, codes, table fields, etc., keyword retrieval (e.g., BM25) is often more robust. Therefore, Hybrid Search is widely adopted in engineering practice:
   1. First perform vector retrieval and keyword retrieval on the query separately, obtaining two sets of candidate document chunks;
   2. Merge the two candidate sets using weighted scoring or a learned fusion strategy;
   3. In some scenarios, dynamically adjust the weights of vector and keyword retrieval based on the query type (FAQ‑style Q&A vs. legal provision lookup).
2. Re‑ranking: more fine‑grained selection of the "evidence set" Initial retrieval results often include many "marginally relevant" or "redundant" document chunks, requiring re‑ranking to improve the quality of the final Top‑K:
   1. Use a Cross‑Encoder to perform bidirectional encoding and relevance scoring on each "query–document chunk" pair; it is more accurate than a dual‑tower Embedding model but more expensive, making it suitable as a second‑stage re‑ranker;
   2. When performance allows, introduce the LLM for lightweight re‑ranking, letting the model judge which chunks are truly "useful" based on richer semantic and contextual information;
   3. Simultaneously consider coverage and diversity to avoid all retrieved chunks concentrating on the same document or paragraph, which would narrow the answer's perspective.
3. Retrieval‑generation closed‑loop optimization In more advanced practice, retrieval and generation are no longer a one‑way pipeline but form a closed loop:
   1. Use the LLM to analyze the "usage" of retrieval results (which chunks are cited, which are consistently ignored) and feed that back to guide optimization of indexing and chunking strategies;
   2. Leverage "follow‑up / correction" signals from conversation logs to annotate and retrain on recall failures and false‑positive retrievals, improving the system's robustness for ambiguous queries and long‑tail problems.

8.1.3 Generation & Citation: Answering Questions "Constrained by Evidence"

The final link is the generation layer, which directly determines the user experience. The goal here is not to let the model "freely improvise," but to have it produce clear, bounded, and well‑cited answers under the constraints of the retrieved evidence.

1. Controlled generation based on retrieval context In a RAG architecture, the LLM receives not only the user's question but also multiple retrieved document chunks and system instructions. The system typically:
   1. Constrains the model via prompts to "answer only based on the given documents" and "clearly state if the answer cannot be found in the documents";
   2. Structurally organizes the retrieval context (segmented, numbered, source‑annotated) so that the model can easily understand and cite it;
   3. Controls the output format (lists, tables, bullet‑point explanations, etc.) to accommodate downstream systems or frontend display.
2. Citation and explainability (Source Citation) To facilitate auditing and traceability — especially in high‑risk domains like legal, medical, financial, and internal corporate policy — answers often need to include explicit citations:
   1. Annotate citations in the output, e.g., "[Document A, Chapter 3, Section 2]" or "[Regulation X, Article 12]";
   2. Support one‑click navigation to the original text location in the frontend, making it easy for users to verify and read further;
   3. Persist the complete "question → retrieval results → cited chunks → final answer" chain log in the backend, providing data for subsequent risk control and model improvement.
3. Advanced RAG variants: HyDE / ReAct + RAG, etc. To further improve results in challenging scenarios, more complex RAG variants are used in practice:
   1. HyDE: the LLM first generates a "hypothetical answer document" based on the question, then uses that document's vector to retrieve real documents, thereby improving recall quality;
   2. ReAct + RAG: the LLM operates in a "Reasoning + Action" mode, calling the retrieval tool multiple times during reasoning to progressively refine the question and supplement evidence — akin to "thinking while looking things up";
   3. Multi‑turn RAG: during a conversation, historical retrieval results and answers are retained, forming a context‑aware, long‑term knowledge session rather than merely "one question, one retrieval."

8.2 Structured Data & Knowledge Graphs (Structured Data & KG)

If RAG primarily addresses "how to look up information in large‑scale unstructured documents," then the structured data and knowledge graph layer is more about "how to elegantly leverage structured knowledge in databases, reporting systems, and graph databases."

In enterprise environments, the truly critical business data — orders, customers, contracts, inventory, behavioral logs — often resides in relational databases, data warehouses, OLAP engines, or graph databases. These systems are already very mature in terms of query capability, computational efficiency, and auditing, but for business users, writing SQL or DSL directly still has a fairly high barrier to entry. Text‑to‑SQL / Text‑to‑DSL and knowledge graph Q&A and reasoning aim to let the LLM serve as a "natural language interface" and "reasoning collaboration partner" without disrupting the stability of these systems.

• Scenarios
  • BI intelligent Q&A and self‑service analytics: business users ask questions in natural language (e.g., "Show me the repurchase rate trend for new customers in East China over the last 3 months"); the system automatically generates SQL, queries the data warehouse, and returns results in natural language with visualizations.
  • Operations / sales analysis assistant: operations staff can explore data conversationally ("Why did this campaign's conversion rate drop?" "Which channels contributed the most high‑value users?"), progressively refining conditions and dimensions over multiple turns.
  • Domain knowledge platform: organize entities, concepts, rules, and cases into a knowledge graph, supporting upstream‑downstream relationship exploration and compliance checks around specific entities.
  • Graph database Q&A and reasoning systems: in scenarios such as risk control, anti‑money laundering, and supply chain analysis, use graph databases together with LLMs to answer and explain questions involving "relationship chains" and "multi‑hop reasoning."
• Principles The core of this layer is transforming the LLM from "someone who directly gives answers" into "an assistant that can query databases and graph databases":
  • In database Q&A, the model needs to understand the user's natural language intent, combine it with the database schema (table structures, field meanings, constraints, etc.), generate correct SQL / GraphQL / internal DSL, and then explain and visualize the execution results.
  • In knowledge graph scenarios, the system first needs to extract entities and relationships from documents and logs to build a structured graph; then, during Q&A, the LLM is responsible for translating natural language questions into graph queries (e.g., Cypher) and performing multi‑hop reasoning and explanation based on the query results.
  • Unlike RAG, the emphasis here is on precise access to structured data and graph structures — ensuring semantic correctness and syntactic rigor on one hand, while controlling side‑channel attacks, sensitive data exposure, and expensive queries on the other.
• Models Typical solutions are usually "LLM + specialized components" multi‑module architectures:
  • Text‑to‑SQL model: models pretrained or finetuned on large‑scale SQL corpora (e.g., PICARD, DIN‑SQL), emphasizing syntactic correctness and schema alignment, sometimes paired with execution feedback for self‑correction.
  • Information extraction and graph construction pipeline: through modules such as named entity recognition (NER), relation extraction, and event extraction, build and update knowledge graphs from text and logs; the LLM can assist with hard‑case extraction and borderline ambiguous relationships.
  • LLM + graph database joint Q&A: the LLM handles question parsing, query generation, and result explanation, while the graph database (e.g., Neo4j) handles efficient execution and multi‑hop relationship search; the two interface via tool‑calling protocols or an intermediate DSL.

8.2.1 Database Q&A (Text‑to‑SQL / DSL) in Practice

The goal of database Q&A is to let business users "ask data questions in natural language" while the system automatically handles query generation, execution, and explanation behind the scenes. Doing this well requires balancing semantic accuracy, syntactic correctness, and execution safety.

1. Natural language to SQL / DSL conversion In the most basic pipeline, the system needs to:
   1. Parse user intent: identify the query target (e.g., "new customers in East China"), filter conditions (time, region, channel), aggregation methods (count, average, YoY/QoQ), and display requirements (trend, ranking, Top‑N);
   2. Combine with the database schema: understand which tables and fields can express the above concepts, and how to join, group by, and sort;
   3. Generate executable SQL / GraphQL / internal DSL, and ensure structural legality through a syntax validator or a specialized Text2SQL model (PICARD, DIN‑SQL, etc.).
2. Natural language explanation and visualization of execution results After query execution, the system must also turn "a cold result set" into "understandable insights":
   1. Provide textual explanations for simple results, e.g., "Over the last 3 months, the repurchase rate of new customers in East China has shown an overall upward trend, rising from 15% to 21%";
   2. Select appropriate visualization forms for complex results (line chart, bar chart, pie chart, distribution chart, etc.) and provide brief analysis;
   3. Support users asking follow‑up questions based on current results (e.g., "Which channels mainly drove this growth?"), automatically constructing new queries based on historical SQL and context.
3. Security and control: preventing "reckless queries" and "unauthorized access" Because LLM‑generated SQL is highly flexible, a layer of security and governance mechanisms is essential:
   1. Strictly limit the queryable databases, tables, fields, and time ranges based on user roles and permissions;
   2. Equip model‑generated SQL with static/dynamic review rules that filter out dangerous operations (e.g., large‑scale scans, high‑cost joins, cross‑tenant queries);
   3. Fully log "natural language question → generated SQL → execution result → final answer" for auditing and anomaly analysis.

8.2.2 Knowledge Graph Construction & Querying

Knowledge graphs attempt to organize knowledge scattered across text, tables, and logs into a structured network of "entities–relationships–attributes–events," thereby better supporting relationship exploration, multi‑hop reasoning, and complex Q&A. In this direction, LLMs complement traditional information extraction and graph databases well.

1. Extracting entities and relationships from documents to build graphs Constructing a knowledge graph typically uses a multi‑stage pipeline:
   1. Information extraction: use models such as NER, relation extraction, and event extraction to identify entities (people, organizations, products, places, concepts, etc.), their relationships (subordination, collaboration, dependency, causality), and key events (transactions, risks, changes) from text;
   2. Normalization and alignment: normalize different expressions of the same entity (abbreviations, aliases, spelling variants) and align them to a unified ID;
   3. Graph updates and version management: support incremental updates, conflict resolution, and error correction, ensuring the graph maintains quality and consistency over long‑term evolution. LLMs can assist traditional algorithms in disambiguation, relationship type refinement, and rule induction.
2. LLM + graph database (Neo4j, etc.) querying and reasoning Once the graph is built, the graph database handles efficient storage and retrieval, while the LLM can play the role of "natural language entry point + reasoning controller":
   1. Question parsing and graph query generation: translate natural language questions into graph query statements (e.g., Neo4j's Cypher), including determining the starting entity, relationship types, path length, and filter conditions;
   2. Multi‑hop reasoning: use paths and local subgraphs obtained from graph queries, then have the LLM explain and summarize, e.g., "Customer A is indirectly connected to high‑risk entity B through three intermediary companies";
   3. Result visualization and explainability: present graph query results as a visual network, while the LLM provides a verbal explanation to help users understand complex relationship structures.
3. Domain knowledge platform and unified services In larger‑scale enterprise or industry‑level applications, knowledge graphs often serve as a "domain knowledge platform":
   1. Provide a unified entity and relationship view for upstream business systems (risk control, compliance, customer 360 view, supply chain analysis, etc.);
   2. Together with RAG and database Q&A, form a unified knowledge service layer, where a unified LLM orchestration logic decides whether the current question should access the document index, relational database, or graph database;
   3. Under security and compliance requirements, use graph‑level access control and masking strategies to further reduce the risk of sensitive information leakage.

The shared goal of this layer is to upgrade "the model can talk" to "the model can both talk and truly connect to the enterprise's real data and knowledge assets." When RAG, Text‑to‑SQL, knowledge graphs, and traditional data infrastructure are effectively integrated, AI systems can maintain both intelligence and flexibility in complex business environments while also possessing controllability, explainability, and long‑term evolution capability.