9. Safety, Alignment & Evaluation

In previous chapters, we focused more on "what the model can do": interpreting images, writing code, conversing with users. But in real-world large model systems, mere "capability" is far from enough: how do we prove these capabilities are stable, reliable, and controllable? How do we ensure outputs align with values and compliance requirements? How do we continuously monitor, iterate, and regress during long-term operations? This layer is concerned with: capability evaluation & benchmarks, value alignment & training, content safety & compliance, and robustness & hallucination control — together forming a sustainable "infrastructure layer" for large model operations.

From a product perspective, these capabilities span the entire model lifecycle: models require standard benchmarks and professional evaluation during the lab phase; they must pass alignment training and safety reviews before launch; post-launch they rely on content safety gateways, log auditing, and A/B testing for continuous monitoring; and when facing new scenarios and threats, they return to the evaluation and alignment stages for retraining and re-validation. Below, we explore four dimensions: capability evaluation & benchmarks, value alignment & training, content safety & compliance, and robustness & hallucination control.

9.1 Capability Evaluation & Benchmarks

In the process of developing and deploying large models, capability evaluation & benchmarks is the critical link that transforms "model capability" into "observable signals": it must answer both "how good is this model overall" and "how does it perform in a specific domain or real business scenario." On one hand, we use standardized benchmark suites and automated evaluation systems to measure model performance on general dimensions such as language understanding & generation, reasoning & math, and knowledge & factuality. On the other hand, we need to build specialized evaluations for domains like healthcare, law, finance, and education, and continuously validate and refine them through real user conversations, A/B testing, and business metrics (Task Success Rate, CSAT, ticket closure rate, etc.). Overall, this layer ultimately crystallizes into an internal capability evaluation platform and an external-facing "capability specification," providing a unified decision-making basis for model selection across multiple versions, tenants, and scenarios. Below, we elaborate from three perspectives: scenarios, principles, and models.

• Scenarios
  • General Capability Evaluation Scenarios: When releasing a base model or major version update, we need to systematically evaluate its performance on language understanding & generation tasks such as reading comprehension, summarization, translation, and dialogue quality, as well as its ability in reasoning & math tasks like arithmetic, multi-step reasoning, and code/logic problems. At the same time, we measure its knowledge & factuality level through fact-based QA, open-domain QA, and knowledge coverage tasks, to determine "whether the new model represents an overall improvement."
  • Domain-Specific Evaluation Scenarios: For specialized fields such as healthcare, law, finance, and education, we need to design professional Q&A and decision-making simulations — for example, disease Q&A and triage recommendations, legal text comprehension and case classification, investment analysis and risk control judgments, teaching assistance and homework tutoring — and test the model's consistency and stability in multilingual and multicultural environments, confirming whether it can "say the right thing, and say it appropriately" in high-stakes settings.
  • Real-World & Business Metric Evaluation Scenarios: During product launch and ongoing operations, through user conversation log replay, online A/B testing, etc., we map model performance to business metrics such as Task Success Rate, Customer Satisfaction (CSAT), and ticket closure rate. At this point, the evaluation target is effectively the entire "model + policy + product flow" system, used to guide version rollbacks, policy tuning, and new feature rollout.
• Principles The capability evaluation system can be viewed as a layered "measurement systems engineering" effort, with core principles including:
  • Standard Benchmark Suites: Common Yardsticks & Reproducible Experiments
    • Language / Reasoning: Use comprehensive tasks like MMLU and BIG-Bench, combined with math and logic problems such as GSM8K and MATH, to establish a unified yardstick for language understanding, knowledge mastery, and multi-step reasoning.
    • Coding: Through HumanEval, MBPP, and Codeforces problem sets, quantify code generation, program repair, and problem-solving capabilities.
    • Multimodal: Leverage benchmarks like VQA, MMBench, ScienceQA, and MathVista to evaluate image-text understanding, visual question answering, and mathematical reasoning in images. These benchmarks emphasize standardization, reproducibility, and comparability, facilitating cross-model and cross-institution horizontal comparisons and external disclosure.
  • Automated Evaluation: Scaling & Continuous Regression
    • LLM-as-a-Judge: Use a stronger or specially trained model to score/rank responses, evaluating correctness, completeness, style, and safety, enabling large-scale automated subjective evaluation.
    • Rule-based Metrics: Such as BLEU / ROUGE / BERTScore for measuring text similarity, Pass@k for measuring code problem pass rates, etc., allowing rapid comparison of differences across versions on fixed datasets. The key to automated evaluation lies in stability and consistency — even if imperfect, as long as "the bias is consistent," it can reliably reflect relative model changes in continuous integration (CI).
  • Human Evaluation: Aligning with Human Perception & Business Objectives
    • Pairwise Comparison & Scoring Annotation: Annotators perform pairwise selection or multi-dimensional scoring (helpful / honest / harmless, etc.) on A/B model responses — an important data source for training RLHF / RLAIF reward models.
    • Online User Experiments: Conduct A/B testing in deployment scenarios such as conversational assistants, search/recommendation, directly observing the impact of different models/policies on metrics like user satisfaction and conversion rates. Human evaluation serves both to calibrate automated evaluation and as an important basis for externally "explaining model behavior."
• Models In engineering practice, capability evaluation crystallizes into a relatively complete "platform + pipeline + metrics system":
  • Internal Capability Evaluation Platform & CI Pipeline: Unified management of various benchmark suites, evaluation scripts, LLM-as-a-Judge configurations, and human annotation tools, supporting one-click benchmark regression upon new model or policy submission; automatically aggregating metric changes across different tasks and dimensions, providing visual dashboards and regression alerts.
  • External "Capability Specification" & Model Profile: Organizing internal evaluation results into an externally consumable "capability specification," including representative benchmark scores, recommended applicable scenarios (e.g., general conversation, code assistance, multimodal understanding), known limitations, and unsuitable scenarios — helping customers form correct expectations and providing a basis for compliance and liability delineation.
  • Multi-Tenant / Multi-Version Unified Model Evaluation & Selection Tool: Under a single evaluation framework, uniformly compare models of different sizes, alignment strategies, or architectures, supporting weight configuration by industry, region, and SLA requirements, automatically generating a "performance–cost–latency" composite score to help product and business teams make model selection and canary release decisions.

9.1.1 General & Domain-Specific Capability Evaluation: From Benchmarks to Scenario Validation

General and domain-specific capability evaluation is the "first layer of the foundation" for the entire evaluation system, with the key focus being: first measuring the model's foundational capabilities on a unified yardstick, then validating its usability and risk in specialized scenarios.

In general capability evaluation, tasks are typically broken down into three dimensions: language understanding & generation, reasoning & math, and knowledge & factuality. The first examines whether the model can accurately understand context, control style, and output coherent text through reading comprehension, summarization, translation, and dialogue quality tasks; the second evaluates the model's ability in complex reasoning chains and program structure through arithmetic, multi-step reasoning, and code/logic problems; the third measures knowledge coverage and factuality through fact-based QA and open-domain QA. In domain-specific evaluation, industry experts need to be involved in data design: for example, in medical Q&A, setting contexts such as medical history and lab results, requiring the model to provide risk warnings and boundaries for medical advice in its responses; in legal tasks, designing statute retrieval, case comparison, and legal applicability analysis; in finance and education, focusing on compliance disclosures and instructional guidance. This layer of evaluation often combines standard benchmark suites with custom-built datasets, pursuing both comparability and business relevance.

9.1.2 Automated Evaluation & LLM-as-a-Judge: Making Evaluation Scalable

When the scale of tasks and the number of model versions grow rapidly, relying solely on human evaluation can no longer meet evaluation needs — at this point, an automated evaluation system is required to achieve scalability and high-frequency regression.

One approach uses traditional rule-based metrics: for translation and summarization tasks, comparing against reference answers using BLEU / ROUGE / BERTScore; for coding tasks, using Pass@k to test whether at least one of multiple generated samples passes unit tests. These metrics are simple to implement and highly automatable, but are insensitive to answer diversity and stylistic nuances. Another more representative approach is LLM-as-a-Judge: using a stronger or specially trained model as a "scoring referee," performing dimensional scoring or pairwise ranking of the evaluated model's outputs based on predefined scoring rubrics. This enables efficient automated evaluation even in open-ended QA and dialogue tasks where there are no standard answers and responses are diverse. In practical engineering, the scoring criteria and prompts for LLM-as-a-Judge need to be calibrated and iterated against human-annotated data to ensure consistency with human judges.

9.1.3 Human Evaluation & Business Metrics: Closing the Loop to Real User Experience

No matter how comprehensive offline metrics are, they can only approximate real user experience. To close the capability evaluation loop to the business, two types of methods are needed: human evaluation and online experiments.

On the human evaluation side, the most common approach is Pairwise Comparison: annotators, without knowing the model identity, make preference selections or score A/B responses based on dimensions such as helpful / honest / harmless, producing high-quality preference data. This data is used both for direct evaluation and for training reward models in RLHF / RLAIF. On the business side, online A/B testing compares the impact of different models, prompts, and policy configuration versions on key metrics such as task completion rate, customer satisfaction (CSAT), and ticket closure rate, supplemented by user conversation log replay and manual spot checks, to continuously monitor the model's real-world performance post-launch. The output of this layer of evaluation in turn feeds back to guide the focus and weight adjustments of the capability evaluation platform, forming a closed loop of "offline metrics — human evaluation — online metrics."

9.2 Value Alignment & Training

After acquiring powerful foundational capabilities, for a large model to become a "safe, reliable, and controllable" product, it must also undergo value alignment & training. This layer is no longer concerned with whether the model "can answer," but with "whether the answer is helpful, honest, and harmless" and "how it should speak in different roles and industries." From an engineering perspective, the alignment process roughly involves three steps: first, clearly defining alignment goal definitions (What to Align) through documentation and specifications, decomposing Helpful, Honest, and Harmless into annotatable and trainable standards; second, constructing broad-coverage instruction data and safety data, covering normal tasks, gray-area cases, and inappropriate responses; and finally, through methods such as SFT, RLHF / RLAIF, and refusal/redirection policy modeling, embedding these preferences and rules into model behavior, supplemented by upstream dialogue management and policy engines to achieve end-to-end safety alignment. Below, we again elaborate from three perspectives: scenarios, principles, and models.

• Scenarios
  • General Consumer-Facing Assistant Scenarios: For chat assistants and information retrieval assistants targeting the general public, they need to remain "friendly, helpful, and within bounds" across a broad range of topics: providing professional, task-focused answers while honestly expressing limitations when uncertain, and refusing or gently redirecting clearly inappropriate requests.
  • Professional Industry Assistant Scenarios: In fields such as healthcare, law, finance, and education, beyond basic safety, industry-specific regulations must be layered on: for example, medical assistants need to repeatedly emphasize "non-diagnostic nature + risk warnings + recommend consulting a doctor," legal assistants should avoid providing advice on circumventing the law, financial assistants must comply with investment compliance disclosure requirements, and education assistants need to consider minor protection and age-appropriate content.
  • Enterprise Configurable Alignment Layer Scenarios: Enterprises often wish to further embed their own industry requirements, brand tone, and internal policies on top of a general safety baseline, thus requiring a configurable alignment layer that allows customers to self-configure safety thresholds, sensitive categories, and phrasing styles without retraining the underlying large model.
• Principles Value alignment can be understood as "constraining the model's behavioral space with human and organizational values," with core principles including:
  • Alignment Goal Definition (What to Align)
    • Helpful: Responses should be high-quality, professional, well-structured, and focused on the task objective, without excessive divergence or idle chatter.
    • Honest: Avoid fabrication as much as possible; when knowledge is missing or understanding is unclear, proactively acknowledge uncertainty, provide estimated ranges, or suggest verification channels.
    • Harmless: Comply with laws and platform policies, avoid generating content involving hate, discrimination, self-harm encouragement, illegal activity guidance, etc., and respect user dignity and boundaries. These goals are written into annotation guidelines and policy documents, serving as unified standards for subsequent data construction, reward modeling, and evaluation.
  • Alignment Training Data Construction
    • Instruction Data: Design a broad range of task instructions and ideal responses covering Q&A, writing, summarization, coding, planning, and other scenarios, teaching the model optimal behavior for "normal requests."
    • Safety Data: Construct "good response vs. inappropriate response" comparison samples, with special attention to gray zones, such as educational information vs. specific operational instructions, emotional support vs. self-harm encouragement, legitimate debate vs. hate incitement — providing the model with fine-grained boundary examples.
  • Alignment Training Methods
    • SFT (Supervised Fine-Tuning): Conduct supervised fine-tuning on high-quality dialogue/instruction data — the first step in shaping the model's baseline behavior and tone.
    • RLHF / RLAIF: Build preference data through human or model scoring, train a reward model, then perform policy optimization so that the model tends to generate "preferred" responses (more helpful, safer, more honest) during generation.
    • Refusal / Redirection Policy Modeling: For high-risk or inappropriate requests, train the model not only to refuse but also to provide reasonable explanations and guide users to safe alternatives (e.g., offering help resources, encouraging consultation with professionals, etc.).
• Models In system design, value alignment typically manifests as a combination of "bottom-layer alignment training + upper-layer policy guardrails":
  • SFT + RLHF / RLAIF Aligned Models: The SFT stage teaches the model the basic patterns of ideal responses; the RLHF / RLAIF stage further "tightens" behavior through preference learning, making it more aligned with human preferences and safety standards. In the safety dimension, separate reward heads or classifiers can be built for harmfulness, used to apply penalties during policy optimization.
  • Constitutional AI / Policy-based Alignment: By first drafting a set of "Constitution" or policy documents, then having the model self-criticize and rewrite according to these rules, generating large amounts of "self-supervised revision data," strengthening the model's internalization of rules while reducing human labor costs.
  • Dialogue Management & Intent Detection Collaboration: In the product pipeline, move safety/alignment logic partially upstream to the dialogue management layer, using intent recognition, slot filling, and task routing to determine whether a request should be passed to the large model, whether additional safety filtering or templated responses are needed. This creates a dual safeguard of "model alignment + policy guardrails."
  • Internal Alignment Platform & Role Configuration: Build an internal alignment platform providing annotation/scoring tools, policy version management, and training pipelines; simultaneously support configuring differentiated alignment goals and phrasing styles for different roles (customer service, medical advice, educational tutoring, etc.), enabling the same base model to exhibit distinctly different yet controllably consistent personalities across different products.

9.2.1 Alignment Goals & Training Data: Turning Values into Learnable Signals

The first step in value alignment is translating "abstract values" into signals that the model can learn — and this depends on alignment goal definition and training data construction.

At the alignment goal level, teams typically produce a detailed set of behavioral specification documents, decomposing Helpful / Honest / Harmless into specific provisions, such as: prohibiting specific step-by-step instructions for certain high-risk operations, requiring disclaimers and risk warnings for medical/legal advice, maintaining neutrality and multi-perspective presentation on controversial topics, etc. Then, in the instruction data phase, diverse tasks and ideal responses are built around these indicators, covering scenarios like chatting, writing, coding, and Q&A, incorporating multilingual and multicultural contexts. In the safety data phase, paired "good/bad response" examples are constructed targeting harmful content, high-risk domains, and gray zones, providing training material for subsequent preference learning and safety classifiers. Through this approach, value objectives are "translated" into actual data distributions, becoming signals that model training can directly perceive.

9.2.2 SFT, RLHF / RLAIF & Refusal Strategies: Shaping Model Behavior

With alignment goals and data in place, the next step is to write these goals into model behavior through a multi-stage training process.

In the SFT stage, the model undergoes supervised fine-tuning on high-quality human demonstration data — akin to "textbook-style learning": it determines the model's tone, structure, and standard problem-solving paradigms for the vast majority of normal requests. Subsequently, RLHF / RLAIF is used for preference optimization: first, a reward model is trained using preference labels produced by human annotation or a larger LLM, then policy optimization algorithms (such as PPO) are used to adjust the model so that it tends to achieve higher rewards during generation. In this way, the model not only "knows what the correct answer looks like" but also knows "which type of answer better aligns with human preferences and safety requirements." On top of this, various refusal & redirection policies are specifically modeled: for questions that are clearly illegal, extremely high-risk, or unsuitable for AI to answer, the model should learn to give clear refusals with explanations and provide safe alternative paths (such as helplines, professional consultation, etc.), rather than simply remaining silent or casually deflecting.

9.2.3 Policy Layer & Alignment Platform: Making Alignment Configurable and Evolvable

Even when the underlying model has undergone sufficient alignment training, a policy layer and alignment platform are still needed in real-world systems to achieve finer-grained controllability and evolvability.

The policy layer typically includes intent recognition, risk assessment, and routing logic: when user input reaches the system, a lightweight model first determines its intent, domain, and risk level, then decides whether to directly invoke the large model, whether additional safety filtering is needed, or whether it should fall into a templated response or human handoff channel. For different industries and customers, the policy layer can load different policy configurations, enabling customization of sensitive categories, refusal styles, and brand tone. Meanwhile, the internal alignment platform manages all alignment-related assets: annotation/scoring tools, reward model versions, policy change records, online A/B results, etc., allowing teams to rapidly iterate and canary-release alignment policies without frequently retraining the base model, thereby maintaining continuous control over model behavior.

9.3 Content Safety & Compliance

As large models become embedded in search, dialogue, content creation, social platforms, and even enterprise internal systems, content safety & compliance has shifted from an "add-on feature" to a "barrier to entry." This layer is concerned with: whether the model generates illegal or harmful content when producing text, images, audio, and video; whether the system complies with the laws and regulations of the countries/regions and industries in which it operates when processing user data; and whether it can provide clear, traceable evidence chains when facing audits and regulatory oversight. To this end, we need to build a complete technical and governance system covering multimodal content moderation, regional & industry compliance, and local privacy & data protection, and package it into product forms such as SaaS content safety services, enterprise compliance middle platforms, and industry security gateways. Below, we again elaborate from three perspectives: scenarios, principles, and models.

• Scenarios
  • Multimodal Content Moderation & Filtering Scenarios: In conversational products, UGC platforms, communities, and social applications, large models generate or receive large volumes of text, images, audio, and video content, requiring unified multimodal moderation capabilities to identify and block high-risk outputs involving personal privacy, illegal activity guidance, hate incitement, extreme violence, pornography, and inappropriate content involving minors in real time.
  • Compliance Constraints & Localization Scenarios: Different countries/regions have different legal and regulatory requirements for data protection, minor protection, and content regulation; different industries (healthcare, finance, education, advertising, etc.) also have detailed compliance norms. Therefore, the system must support loading different policy templates by region and industry to comply with local regulatory requirements.
  • User Privacy & Data Protection Scenarios: During model training and online services, large volumes of user conversations and business data need to be processed. How to achieve data anonymization, desensitization, and minimal collection, while protecting privacy through technical and institutional measures during both training and inference, is another pillar of the content safety & compliance system — especially critical in highly sensitive industries such as finance and healthcare.
• Principles The underlying principles of content safety & compliance can be divided into three layers: policy, filtering, and privacy:
  • Safety Policy System (Policy Engine)
    • Formalize laws, regulations, platform rules, and industry norms into executable policies, using a rule engine combined with model scoring to classify content by risk level (safe / gray zone / high-risk).
    • Support selecting different policy templates by scenario and customer, for example configuring different sensitive categories and thresholds for teen products, professional communities, or multinational enterprises.
  • Multi-Level Content Filtering: Pre-Event — In-Event — Post-Event
    • Pre-Event: Intercept and rewrite user prompts (Prompt Shielding), blocking clearly illegal or highly sensitive intents before they reach the large model, or guiding them toward safer expressions.
    • In-Event: During model output generation, use safety classification models and rules for real-time content review (Real-time Safety Filter), truncating, replacing, masking, or triggering refusal for high-risk content.
    • Post-Event: Conduct sampling audits and human review of conversation and generation logs, perform root cause analysis on discovered issues, then update policies and models, and provide traceable records for external regulatory oversight.
  • Privacy Protection Technologies & Data Governance
    • Before data storage and training, perform anonymization and desensitization on user conversation data, removing or replacing sensitive fields such as names, ID numbers, phone numbers, and addresses, following the principle of minimal collection to retain only necessary information.
    • In certain scenarios, adopt Differential Privacy (DP) to limit the influence of individual samples on model parameters, or use Federated Learning (FL) to keep training within local data domains, avoiding raw data from going to the cloud.
    • Use access control mechanisms such as RBAC / ABAC to strictly limit who can access what level of logs and sensitive data, complemented by audit logs to ensure access paths are traceable.
• Models From a product and system design perspective, content safety & compliance ultimately evolves into a series of reusable "safety services and middle platforms":
  • SaaS Content Safety Service: Package text/image/audio/video moderation capabilities as a unified API, connecting to upstream applications; input content, output risk type, classification, and handling recommendations (allow, block, human review), helping developers quickly integrate safety modules.
  • Enterprise Internal Compliance Middle Platform: Provide large enterprises with centralized compliance policy configuration, audit reporting, and risk alerting capabilities, connecting to internal business systems and human review teams, enabling each business line to execute customized rules under unified policies while meeting external regulatory reporting requirements.
  • High-Risk Industry-Specific Security Gateways & Log Auditing Systems: In high-risk industries such as finance and healthcare, proxy all large model calls through dedicated security gateways, performing real-time traffic inspection and desensitization, retaining critical logs locally or in compliant regions, and providing detailed access auditing and incident traceability capabilities to meet stringent regulatory requirements.

9.3.1 Multimodal Moderation & Policy Engine: Turning Rules into "Executable Code"

A real content safety system must first be able to "understand" content from different channels and modalities, and then enforce policies on every request and response.

In multimodal moderation, the system typically builds multiple detection models for text, images, video, etc.: text-side models identify sensitive keywords, contextual semantics, and veiled expressions; image and video-side models detect violence, pornography, minors, hate symbols, illegal items, and other content, combining OCR, ASR, and visual features for joint judgment where necessary. The policy engine binds these model outputs to regulatory requirements: for example, if a region has stricter restrictions on gambling or political content, the sensitivity of related detection categories can be raised in the corresponding policy template, or content matching these categories can be forced into human review. By transforming abstract rules into rule chains, thresholds, and actions (allow/block/human review/mask), the Policy Engine makes compliance requirements truly "operational."

9.3.2 Multi-Level Filtering & Log Auditing: Building an End-to-End Safety Closed Loop

Single-point interception can hardly cover all risks, so content safety systems generally adopt a pre-event — in-event — post-event three-layer defense design.

In the pre-event stage, the system performs rapid detection on user input, directly rejecting or rewriting clearly violating or highly sensitive prompts, guiding users to ask questions in a safe manner; for borderline attempts and ambiguous requests, it can also proactively add disclaimers and risk warnings. In the in-event stage, model outputs pass through a real-time safety filtering component: this component uses text classification and rule matching to clip, replace, or trigger refusal flows for potentially high-risk outputs, ensuring that the content ultimately presented to the user falls within an acceptable range. In the post-event stage, through log auditing and sampling mechanisms, safety teams or trusted automated systems periodically replay and review conversations, analyze false positives, false negatives, and emerging risk patterns, and accordingly update policies, training data, and detection models. This forms a continuously evolving safety closed loop rather than a "one-time configuration."

9.3.3 Privacy Protection & Industry Security Gateways: Making Data Security "Provable"

In highly sensitive industries, "not outputting harmful content" is far from enough — one must also prove that "internal use of user data is equally safe, compliant, and traceable."

Privacy protection begins the moment data enters the system: anonymization and desensitization are performed as early as the collection and storage stages, ensuring that even if logs are leaked, they are difficult to directly associate with specific individuals. During training, differential privacy, sampling strategies, or federated learning reduce the impact and leakage risk of individual user data on the final model. For model inference traffic, a security gateway provides unified access control: all requests and responses pass through the gateway's content inspection, permission verification, and audit logging, with different access policies and data views applied as needed based on business line and user role. Ultimately, these logs and policy change records crystallize into an "evidence chain" viewable by internal auditors and external regulators, enabling the enterprise to not only be compliant in fact but also "provably compliant" in form.