模型评估

一个模型的好坏，取决于你衡量它的方式。

类型： Build 语言： Python 前置要求： 阶段 1（概率与分布、ML 统计学）、阶段 2 第 1-8 课 预计时间： ~90 分钟

学习目标

从零实现 K 折和分层 K 折交叉验证，并解释为什么分层对不平衡数据很重要
从零计算精确率、召回率、F1、AUC-ROC 和回归指标（MSE、RMSE、MAE、R 方）
解读学习曲线，诊断模型是高偏差还是高方差
识别常见的评估错误，包括数据泄漏、指标选错和测试集污染

问题背景

你训练了一个模型。它在你的数据上拿到 95% 准确率。它好吗？

也许。也许不。如果你 95% 的数据属于同一个类，一个永远预测那个类的模型也能拿 95% 准确率，却完全没用。如果你在训练用的同一批数据上评估，这 95% 就毫无意义，因为模型只是把答案背了下来。如果你的数据有时间维度，而你在划分前随机打乱了，那你的模型可能在用未来的数据预测过去。

模型评估是大多数 ML 项目翻车的地方。错的指标让差模型看起来好。错的划分让模型作弊。错的对比让你选了更差的模型。把评估做对不是可选项。它决定了一个模型是在生产里能用，还是一看到真实数据就崩。

核心概念

训练、验证、测试

mermaid

flowchart LR
    A[完整数据集] --> B[训练集 60-70%]
    A --> C[验证集 15-20%]
    A --> D[测试集 15-20%]
    B --> E[拟合模型]
    E --> C
    C --> F[调超参数]
    F --> E
    F --> G[最终模型]
    G --> D
    D --> H[报告性能]

三种划分，三个用途：

训练集：模型从这部分数据学习。它在训练时看到这些样本。
验证集：用来调超参数、在模型之间做选择。模型从不在这部分数据上训练，但你的决策受它影响。
测试集：只在最后碰一次，用来报告最终性能。如果你看了测试性能再回头改模型，它就不再是测试集了，它变成了第二个验证集。

测试集是你的留出担保，确保报告的性能反映模型在真正未见过的数据上的表现。

K 折交叉验证

数据集小的时候，单次训练/验证划分既浪费数据又给出噪声大的估计。K 折交叉验证把所有数据都用于训练和验证：

mermaid

flowchart TB
    subgraph Fold1["第 1 折"]
        direction LR
        V1["验证"] --- T1a["训练"] --- T1b["训练"] --- T1c["训练"] --- T1d["训练"]
    end
    subgraph Fold2["第 2 折"]
        direction LR
        T2a["训练"] --- V2["验证"] --- T2b["训练"] --- T2c["训练"] --- T2d["训练"]
    end
    subgraph Fold3["第 3 折"]
        direction LR
        T3a["训练"] --- T3b["训练"] --- V3["验证"] --- T3c["训练"] --- T3d["训练"]
    end
    subgraph Fold4["第 4 折"]
        direction LR
        T4a["训练"] --- T4b["训练"] --- T4c["训练"] --- V4["验证"] --- T4d["训练"]
    end
    subgraph Fold5["第 5 折"]
        direction LR
        T5a["训练"] --- T5b["训练"] --- T5c["训练"] --- T5d["训练"] --- V5["验证"]
    end
    Fold1 --> R["平均各分数"]
    Fold2 --> R
    Fold3 --> R
    Fold4 --> R
    Fold5 --> R

把数据分成 K 个等大的折
对每一折，在 K-1 折上训练，在剩下那折上验证
把 K 个验证分数平均

K=5 或 K=10 是标准选择。每个数据点恰好被用于验证一次。平均分数比任何单次划分都更稳定。

分层 K 折：在每一折里保持类别分布。如果你的数据集 70% 是类 A、30% 是类 B，那每一折都大致保持这个比例。这对不平衡数据集很重要，因为随机划分可能把所有少数类样本都塞进一折里。

分类指标

混淆矩阵：基础。对于二分类：

	预测为正	预测为负
实际为正	真正例 (TP)	假负例 (FN)
实际为负	假正例 (FP)	真负例 (TN)

所有其他指标都从这个矩阵推出：

准确率 = (TP + TN) / (TP + TN + FP + FN)。预测正确的比例。类别不平衡时会误导人。
精确率 = TP / (TP + FP)。在所有被预测为正的里，有多少真的是正？当假正例代价高时用它（比如垃圾邮件过滤器把真邮件标成垃圾）。
召回率（灵敏度）= TP / (TP + FN)。在所有真正为正的里，我们抓到了多少？当假负例代价高时用它（比如癌症筛查漏掉肿瘤）。
F1 分数 = 2 * precision * recall / (precision + recall)。精确率和召回率的调和平均。当两者都不明显占优时平衡两者。
AUC-ROC：ROC 曲线下面积。在各种分类阈值下画真正例率对假正例率。AUC = 0.5 意味着随机猜，AUC = 1.0 意味着完美分开。它与阈值无关：衡量的是模型把正样本排在负样本之上的能力，不管你挑哪个截断点。

回归指标

MSE（均方误差）= mean((y_true - y_pred)^2)。对大误差二次惩罚。对离群点敏感。
RMSE（均方根误差）= sqrt(MSE)。和目标变量同单位。比 MSE 更好解读。
MAE（平均绝对误差）= mean(|y_true - y_pred|)。线性对待所有误差。比 MSE 更抗离群点。
R 方 = 1 - SS_res / SS_tot，其中 SS_res = sum((y_true - y_pred)^2)，SS_tot = sum((y_true - y_mean)^2)。模型解释的方差比例。R^2 = 1.0 是完美。R^2 = 0.0 意味着模型不比永远预测均值更好。如果模型比均值还差，R^2 可以为负。

学习曲线

把训练分数和验证分数作为训练集大小的函数画出来：

高偏差（欠拟合）：两条曲线都收敛到一个低分。加更多数据没用。你需要一个更复杂的模型。
高方差（过拟合）：训练分数高，但验证分数低得多。两者之间差距大。加更多数据应该有帮助。

验证曲线

把训练分数和验证分数作为某个超参数的函数画出来：

低复杂度时：两个分数都低（欠拟合）
合适复杂度时：两个分数都高且靠得近
高复杂度时：训练分数仍高，但验证分数下降（过拟合）

最优超参数值在验证分数达到峰值的地方。

常见评估错误

数据泄漏：测试集的信息渗进了训练。例子：在划分前对整个数据集拟合缩放器、在时间序列预测里混入未来数据、用一个由目标派生出的特征。永远先划分，再预处理。

类别不平衡：99% 的交易是合法的，1% 是欺诈。一个永远预测"合法"的模型拿 99% 准确率。改用精确率、召回率、F1 或 AUC-ROC。

指标选错：本该优化召回率（医疗诊断）却优化了准确率，或者数据有大量离群点时优化了 RMSE（该用 MAE）。

不用分层划分：对不平衡数据，随机划分可能让验证折里的少数类样本极少，给出不稳定的估计。

测试太频繁：每次你看测试性能再调整，就在过拟合测试集。测试集是一次性的。

figure

precision-recall-threshold

动手构建

第 1 步：训练/验证/测试划分

python

import random
import math

def train_val_test_split(X, y, train_ratio=0.6, val_ratio=0.2, seed=42):
    random.seed(seed)
    n = len(X)
    indices = list(range(n))
    random.shuffle(indices)

    train_end = int(n * train_ratio)
    val_end = int(n * (train_ratio + val_ratio))

    train_idx = indices[:train_end]
    val_idx = indices[train_end:val_end]
    test_idx = indices[val_end:]

    X_train = [X[i] for i in train_idx]
    y_train = [y[i] for i in train_idx]
    X_val = [X[i] for i in val_idx]
    y_val = [y[i] for i in val_idx]
    X_test = [X[i] for i in test_idx]
    y_test = [y[i] for i in test_idx]

    return X_train, y_train, X_val, y_val, X_test, y_test

第 2 步：K 折和分层 K 折交叉验证

python

def kfold_split(n, k=5, seed=42):
    random.seed(seed)
    indices = list(range(n))
    random.shuffle(indices)

    fold_size = n // k
    folds = []

    for i in range(k):
        start = i * fold_size
        end = start + fold_size if i < k - 1 else n
        val_idx = indices[start:end]
        train_idx = indices[:start] + indices[end:]
        folds.append((train_idx, val_idx))

    return folds

def stratified_kfold_split(y, k=5, seed=42):
    random.seed(seed)

    class_indices = {}
    for i, label in enumerate(y):
        class_indices.setdefault(label, []).append(i)

    for label in class_indices:
        random.shuffle(class_indices[label])

    folds = [{"train": [], "val": []} for _ in range(k)]

    for label, indices in class_indices.items():
        fold_size = len(indices) // k
        for i in range(k):
            start = i * fold_size
            end = start + fold_size if i < k - 1 else len(indices)
            val_part = indices[start:end]
            train_part = indices[:start] + indices[end:]
            folds[i]["val"].extend(val_part)
            folds[i]["train"].extend(train_part)

    return [(f["train"], f["val"]) for f in folds]

def cross_validate(X, y, model_fn, k=5, metric_fn=None, stratified=False):
    n = len(X)

    if stratified:
        folds = stratified_kfold_split(y, k)
    else:
        folds = kfold_split(n, k)

    scores = []
    for train_idx, val_idx in folds:
        X_train = [X[i] for i in train_idx]
        y_train = [y[i] for i in train_idx]
        X_val = [X[i] for i in val_idx]
        y_val = [y[i] for i in val_idx]

        model = model_fn()
        model.fit(X_train, y_train)
        predictions = [model.predict(x) for x in X_val]

        if metric_fn:
            score = metric_fn(y_val, predictions)
        else:
            score = sum(1 for yt, yp in zip(y_val, predictions) if yt == yp) / len(y_val)
        scores.append(score)

    return scores

第 3 步：混淆矩阵和分类指标

python

def confusion_matrix(y_true, y_pred):
    tp = sum(1 for yt, yp in zip(y_true, y_pred) if yt == 1 and yp == 1)
    tn = sum(1 for yt, yp in zip(y_true, y_pred) if yt == 0 and yp == 0)
    fp = sum(1 for yt, yp in zip(y_true, y_pred) if yt == 0 and yp == 1)
    fn = sum(1 for yt, yp in zip(y_true, y_pred) if yt == 1 and yp == 0)
    return tp, tn, fp, fn

def accuracy(y_true, y_pred):
    tp, tn, fp, fn = confusion_matrix(y_true, y_pred)
    total = tp + tn + fp + fn
    return (tp + tn) / total if total > 0 else 0.0

def precision(y_true, y_pred):
    tp, tn, fp, fn = confusion_matrix(y_true, y_pred)
    return tp / (tp + fp) if (tp + fp) > 0 else 0.0

def recall(y_true, y_pred):
    tp, tn, fp, fn = confusion_matrix(y_true, y_pred)
    return tp / (tp + fn) if (tp + fn) > 0 else 0.0

def f1_score(y_true, y_pred):
    p = precision(y_true, y_pred)
    r = recall(y_true, y_pred)
    return 2 * p * r / (p + r) if (p + r) > 0 else 0.0

def roc_curve(y_true, y_scores):
    thresholds = sorted(set(y_scores), reverse=True)
    tpr_list = []
    fpr_list = []

    total_positives = sum(y_true)
    total_negatives = len(y_true) - total_positives

    for threshold in thresholds:
        y_pred = [1 if s >= threshold else 0 for s in y_scores]
        tp = sum(1 for yt, yp in zip(y_true, y_pred) if yt == 1 and yp == 1)
        fp = sum(1 for yt, yp in zip(y_true, y_pred) if yt == 0 and yp == 1)

        tpr = tp / total_positives if total_positives > 0 else 0.0
        fpr = fp / total_negatives if total_negatives > 0 else 0.0

        tpr_list.append(tpr)
        fpr_list.append(fpr)

    return fpr_list, tpr_list, thresholds

def auc_roc(y_true, y_scores):
    fpr_list, tpr_list, _ = roc_curve(y_true, y_scores)

    pairs = sorted(zip(fpr_list, tpr_list))
    fpr_sorted = [p[0] for p in pairs]
    tpr_sorted = [p[1] for p in pairs]

    area = 0.0
    for i in range(1, len(fpr_sorted)):
        width = fpr_sorted[i] - fpr_sorted[i - 1]
        height = (tpr_sorted[i] + tpr_sorted[i - 1]) / 2
        area += width * height

    return area

第 4 步：回归指标

python

def mse(y_true, y_pred):
    n = len(y_true)
    return sum((yt - yp) ** 2 for yt, yp in zip(y_true, y_pred)) / n

def rmse(y_true, y_pred):
    return math.sqrt(mse(y_true, y_pred))

def mae(y_true, y_pred):
    n = len(y_true)
    return sum(abs(yt - yp) for yt, yp in zip(y_true, y_pred)) / n

def r_squared(y_true, y_pred):
    mean_y = sum(y_true) / len(y_true)
    ss_res = sum((yt - yp) ** 2 for yt, yp in zip(y_true, y_pred))
    ss_tot = sum((yt - mean_y) ** 2 for yt in y_true)
    if ss_tot == 0:
        return 0.0
    return 1.0 - ss_res / ss_tot

第 5 步：学习曲线

python

def learning_curve(X, y, model_fn, metric_fn, train_sizes=None, val_ratio=0.2, seed=42):
    random.seed(seed)
    n = len(X)
    indices = list(range(n))
    random.shuffle(indices)

    val_size = int(n * val_ratio)
    val_idx = indices[:val_size]
    pool_idx = indices[val_size:]

    X_val = [X[i] for i in val_idx]
    y_val = [y[i] for i in val_idx]

    if train_sizes is None:
        train_sizes = [int(len(pool_idx) * r) for r in [0.1, 0.2, 0.4, 0.6, 0.8, 1.0]]

    train_scores = []
    val_scores = []

    for size in train_sizes:
        subset = pool_idx[:size]
        X_train = [X[i] for i in subset]
        y_train = [y[i] for i in subset]

        model = model_fn()
        model.fit(X_train, y_train)

        train_pred = [model.predict(x) for x in X_train]
        val_pred = [model.predict(x) for x in X_val]

        train_scores.append(metric_fn(y_train, train_pred))
        val_scores.append(metric_fn(y_val, val_pred))

    return train_sizes, train_scores, val_scores

第 6 步：一个用于测试的简单分类器，加上完整演示

python

class SimpleLogistic:
    def __init__(self, lr=0.1, epochs=100):
        self.lr = lr
        self.epochs = epochs
        self.weights = None
        self.bias = 0.0

    def sigmoid(self, z):
        z = max(-500, min(500, z))
        return 1.0 / (1.0 + math.exp(-z))

    def fit(self, X, y):
        n_features = len(X[0])
        self.weights = [0.0] * n_features
        self.bias = 0.0

        for _ in range(self.epochs):
            for xi, yi in zip(X, y):
                z = sum(w * x for w, x in zip(self.weights, xi)) + self.bias
                pred = self.sigmoid(z)
                error = yi - pred
                for j in range(n_features):
                    self.weights[j] += self.lr * error * xi[j]
                self.bias += self.lr * error

    def predict_proba(self, x):
        z = sum(w * xi for w, xi in zip(self.weights, x)) + self.bias
        return self.sigmoid(z)

    def predict(self, x):
        return 1 if self.predict_proba(x) >= 0.5 else 0

class SimpleLinearRegression:
    def __init__(self, lr=0.001, epochs=200):
        self.lr = lr
        self.epochs = epochs
        self.weights = None
        self.bias = 0.0

    def fit(self, X, y):
        n_features = len(X[0])
        self.weights = [0.0] * n_features
        self.bias = 0.0
        n = len(X)

        for _ in range(self.epochs):
            for xi, yi in zip(X, y):
                pred = sum(w * x for w, x in zip(self.weights, xi)) + self.bias
                error = yi - pred
                for j in range(n_features):
                    self.weights[j] += self.lr * error * xi[j] / n
                self.bias += self.lr * error / n

    def predict(self, x):
        return sum(w * xi for w, xi in zip(self.weights, x)) + self.bias

def standardize(values):
    n = len(values)
    mean = sum(values) / n
    var = sum((v - mean) ** 2 for v in values) / n
    std = math.sqrt(var) if var > 0 else 1.0
    return [(v - mean) / std for v in values], mean, std

def make_classification_data(n=300, seed=42):
    random.seed(seed)
    X = []
    y = []
    for _ in range(n):
        x1 = random.gauss(0, 1)
        x2 = random.gauss(0, 1)
        label = 1 if (x1 + x2 + random.gauss(0, 0.5)) > 0 else 0
        X.append([x1, x2])
        y.append(label)
    return X, y

def make_regression_data(n=200, seed=42):
    random.seed(seed)
    X = []
    y = []
    for _ in range(n):
        x1 = random.uniform(0, 10)
        x2 = random.uniform(0, 5)
        target = 3 * x1 + 2 * x2 + random.gauss(0, 2)
        X.append([x1, x2])
        y.append(target)
    return X, y

def make_imbalanced_data(n=300, minority_ratio=0.05, seed=42):
    random.seed(seed)
    X = []
    y = []
    for _ in range(n):
        if random.random() < minority_ratio:
            x1 = random.gauss(3, 0.5)
            x2 = random.gauss(3, 0.5)
            label = 1
        else:
            x1 = random.gauss(0, 1)
            x2 = random.gauss(0, 1)
            label = 0
        X.append([x1, x2])
        y.append(label)
    return X, y

if __name__ == "__main__":
    X_clf, y_clf = make_classification_data(300)

    print("=== Train/Validation/Test Split ===")
    X_train, y_train, X_val, y_val, X_test, y_test = train_val_test_split(X_clf, y_clf)
    print(f"  Train: {len(X_train)}, Val: {len(X_val)}, Test: {len(X_test)}")
    print(f"  Train class distribution: {sum(y_train)}/{len(y_train)} positive")
    print(f"  Val class distribution: {sum(y_val)}/{len(y_val)} positive")

    model = SimpleLogistic(lr=0.1, epochs=200)
    model.fit(X_train, y_train)

    print("\n=== Classification Metrics ===")
    y_pred = [model.predict(x) for x in X_test]
    tp, tn, fp, fn = confusion_matrix(y_test, y_pred)
    print(f"  Confusion matrix: TP={tp}, TN={tn}, FP={fp}, FN={fn}")
    print(f"  Accuracy:  {accuracy(y_test, y_pred):.4f}")
    print(f"  Precision: {precision(y_test, y_pred):.4f}")
    print(f"  Recall:    {recall(y_test, y_pred):.4f}")
    print(f"  F1 Score:  {f1_score(y_test, y_pred):.4f}")

    y_scores = [model.predict_proba(x) for x in X_test]
    auc = auc_roc(y_test, y_scores)
    print(f"  AUC-ROC:   {auc:.4f}")

    print("\n=== K-Fold Cross-Validation (K=5) ===")
    cv_scores = cross_validate(
        X_clf, y_clf,
        model_fn=lambda: SimpleLogistic(lr=0.1, epochs=200),
        k=5,
        metric_fn=accuracy,
    )
    mean_cv = sum(cv_scores) / len(cv_scores)
    std_cv = math.sqrt(sum((s - mean_cv) ** 2 for s in cv_scores) / len(cv_scores))
    print(f"  Fold scores: {[round(s, 4) for s in cv_scores]}")
    print(f"  Mean: {mean_cv:.4f} (+/- {std_cv:.4f})")

    print("\n=== Stratified K-Fold Cross-Validation (K=5) ===")
    strat_scores = cross_validate(
        X_clf, y_clf,
        model_fn=lambda: SimpleLogistic(lr=0.1, epochs=200),
        k=5,
        metric_fn=accuracy,
        stratified=True,
    )
    strat_mean = sum(strat_scores) / len(strat_scores)
    strat_std = math.sqrt(sum((s - strat_mean) ** 2 for s in strat_scores) / len(strat_scores))
    print(f"  Fold scores: {[round(s, 4) for s in strat_scores]}")
    print(f"  Mean: {strat_mean:.4f} (+/- {strat_std:.4f})")

    print("\n=== Imbalanced Data: Why Accuracy Lies ===")
    X_imb, y_imb = make_imbalanced_data(300, minority_ratio=0.05)
    positives = sum(y_imb)
    print(f"  Class distribution: {positives} positive, {len(y_imb) - positives} negative ({positives/len(y_imb)*100:.1f}% positive)")

    always_negative = [0] * len(y_imb)
    print(f"  Always-negative baseline:")
    print(f"    Accuracy:  {accuracy(y_imb, always_negative):.4f}")
    print(f"    Precision: {precision(y_imb, always_negative):.4f}")
    print(f"    Recall:    {recall(y_imb, always_negative):.4f}")
    print(f"    F1 Score:  {f1_score(y_imb, always_negative):.4f}")

    X_tr_i, y_tr_i, X_v_i, y_v_i, X_te_i, y_te_i = train_val_test_split(X_imb, y_imb)
    model_imb = SimpleLogistic(lr=0.5, epochs=500)
    model_imb.fit(X_tr_i, y_tr_i)
    y_pred_imb = [model_imb.predict(x) for x in X_te_i]
    print(f"\n  Trained model on imbalanced data:")
    print(f"    Accuracy:  {accuracy(y_te_i, y_pred_imb):.4f}")
    print(f"    Precision: {precision(y_te_i, y_pred_imb):.4f}")
    print(f"    Recall:    {recall(y_te_i, y_pred_imb):.4f}")
    print(f"    F1 Score:  {f1_score(y_te_i, y_pred_imb):.4f}")

    print("\n=== Regression Metrics ===")
    X_reg, y_reg = make_regression_data(200)

    col0 = [x[0] for x in X_reg]
    col1 = [x[1] for x in X_reg]
    col0_s, m0, s0 = standardize(col0)
    col1_s, m1, s1 = standardize(col1)
    X_reg_scaled = [[col0_s[i], col1_s[i]] for i in range(len(X_reg))]

    X_tr_r, y_tr_r, X_v_r, y_v_r, X_te_r, y_te_r = train_val_test_split(X_reg_scaled, y_reg)
    reg_model = SimpleLinearRegression(lr=0.01, epochs=500)
    reg_model.fit(X_tr_r, y_tr_r)
    y_pred_r = [reg_model.predict(x) for x in X_te_r]

    print(f"  MSE:       {mse(y_te_r, y_pred_r):.4f}")
    print(f"  RMSE:      {rmse(y_te_r, y_pred_r):.4f}")
    print(f"  MAE:       {mae(y_te_r, y_pred_r):.4f}")
    print(f"  R-squared: {r_squared(y_te_r, y_pred_r):.4f}")

    mean_baseline = [sum(y_tr_r) / len(y_tr_r)] * len(y_te_r)
    print(f"\n  Mean baseline:")
    print(f"    MSE:       {mse(y_te_r, mean_baseline):.4f}")
    print(f"    R-squared: {r_squared(y_te_r, mean_baseline):.4f}")

    print("\n=== Learning Curve ===")
    sizes, train_sc, val_sc = learning_curve(
        X_clf, y_clf,
        model_fn=lambda: SimpleLogistic(lr=0.1, epochs=200),
        metric_fn=accuracy,
    )
    print(f"  {'Size':>6} {'Train':>8} {'Val':>8}")
    for s, tr, va in zip(sizes, train_sc, val_sc):
        print(f"  {s:>6} {tr:>8.4f} {va:>8.4f}")

    print("\n=== Statistical Model Comparison ===")
    model_a_scores = cross_validate(
        X_clf, y_clf,
        model_fn=lambda: SimpleLogistic(lr=0.1, epochs=100),
        k=5, metric_fn=accuracy,
    )
    model_b_scores = cross_validate(
        X_clf, y_clf,
        model_fn=lambda: SimpleLogistic(lr=0.1, epochs=500),
        k=5, metric_fn=accuracy,
    )
    diffs = [a - b for a, b in zip(model_a_scores, model_b_scores)]
    mean_diff = sum(diffs) / len(diffs)
    std_diff = math.sqrt(sum((d - mean_diff) ** 2 for d in diffs) / len(diffs))
    t_stat = mean_diff / (std_diff / math.sqrt(len(diffs))) if std_diff > 0 else 0.0
    print(f"  Model A (100 epochs) mean: {sum(model_a_scores)/len(model_a_scores):.4f}")
    print(f"  Model B (500 epochs) mean: {sum(model_b_scores)/len(model_b_scores):.4f}")
    print(f"  Mean difference: {mean_diff:.4f}")
    print(f"  Paired t-statistic: {t_stat:.4f}")
    print(f"  (|t| > 2.78 for significance at p<0.05 with df=4)")

实际使用

用 scikit-learn，评估内置在工作流里：

python

from sklearn.model_selection import cross_val_score, StratifiedKFold, learning_curve
from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score,
    roc_auc_score, confusion_matrix, mean_squared_error, r2_score,
)
from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
scores = cross_val_score(model, X, y, cv=StratifiedKFold(5), scoring="f1")

从零版本让你看清交叉验证到底干了什么（没魔法，就是 for 循环加索引跟踪）、每个指标怎么算出来（就是数 TP/FP/TN/FN）、以及为什么分层重要（在每折里保持类别比例）。库版本加了并行、更多评分选项，以及和流水线的集成。

拿去用

本节课产出：

outputs/skill-evaluation.md - 一个涵盖分类和回归模型评估策略的 skill

练习

实现精确率-召回率曲线：在不同阈值下画精确率对召回率。计算平均精确率（PR 曲线下面积）。在一个不平衡数据集上把 PR 曲线和 ROC 曲线对比，解释各自何时更有信息量。
构建一个嵌套交叉验证循环：外层评估模型性能，内层调超参数。用它公平地对比两个模型，而不把验证数据泄漏进评估。
实现一个用于模型对比的置换检验：打乱标签、重新训练、衡量性能。重复 100 次构建零分布。计算观测到的模型性能相对这个分布的 p 值。

关键术语

术语	大家怎么说	它实际是什么
过拟合	"把训练数据背下来"	模型抓住了训练数据里的噪声，在训练上表现好但在未见数据上差
交叉验证	"在不同子集上测试"	系统性地轮换用哪部分数据做验证，把所有轮换的结果平均
精确率	"预测为正的里有多少对"	TP / (TP + FP)：正预测里真正为正的比例
召回率	"真正为正的我们找到了多少"	TP / (TP + FN)：真正为正里被正确识别的比例
AUC-ROC	"模型把类分得有多开"	跨所有阈值的真正例率对假正例率的曲线下面积，从 0.5（随机）到 1.0（完美）
R 方	"解释了多少方差"	1 - (残差平方和 / 总平方和)：模型捕获的目标方差比例
数据泄漏	"模型作弊了"	训练时用了预测时拿不到的信息，导致乐观的评估
学习曲线	"性能随数据增多怎么变"	训练分数和验证分数对训练集大小的图，揭示欠拟合或过拟合
分层划分	"保持类别比例平衡"	划分数据使每个子集的各类占比和完整数据集相同

Claude 认证课程

lessons

00-certification-strategy

01-claude-product-and-model-landscape

02-model-selection-and-token-economics

03-prompting-and-task-decomposition

04-context-knowledge-memory-and-caching

05-output-evaluation-and-validation

06-governance-safety-and-responsible-use

07-workflow-design-and-human-handoffs

08-messages-api-and-application-lifecycle

09-structured-output-and-defensive-parsing

10-tool-use-and-agentic-loops

11-mcp-server-design-and-integration

12-claude-agent-sdk-and-hooks

13-application-security-and-secrets

14-evals-testing-debugging-and-observability

15-claude-code-for-development-teams

16-multi-agent-orchestration-and-delegation

17-agent-sdk-sessions-subagents-and-context

18-tool-contracts-errors-and-progressive-discovery

19-claude-code-memory-rules-skills-and-ci

20-reliable-extraction-batch-and-reviewers

21-long-context-reliability-provenance-and-escalation

22-business-discovery-requirements-and-slas

23-end-to-end-architecture-and-value-tradeoffs

24-rag-retrieval-and-data-pipelines

25-integration-protocols-identity-and-least-privilege

26-production-observability-latency-and-cost

27-enterprise-governance-compliance-and-hitl

28-stakeholder-communication-adrs-and-lifecycle

29-associate-workflow-capstone

30-developer-application-capstone

31-architect-foundations-scenario-capstone

32-architect-professional-system-capstone

references

research

阶段 0：环境与工具

01-dev-environment

outputs

04-apis-and-keys

outputs

05-jupyter-notebooks

outputs

09-data-management

outputs

12-debugging-and-profiling

outputs

阶段 1：数学基础

01-linear-algebra-intuition

outputs

02-vectors-matrices-operations

outputs

03-matrix-transformations

outputs

04-calculus-for-ml

outputs

05-chain-rule-and-autodiff

outputs

06-probability-and-distributions

outputs

07-bayes-theorem

outputs

08-optimization

outputs

09-information-theory

outputs

10-dimensionality-reduction

outputs

11-singular-value-decomposition

outputs

12-tensor-operations

outputs

13-numerical-stability

outputs

14-norms-and-distances

outputs

15-statistics-for-ml

outputs

16-sampling-methods