线性回归

线性回归在你的数据里画出最佳直线。它是机器学习的 "hello world"。

类型： Build 语言： Python 前置要求： 阶段 1（线性代数、微积分、优化）、阶段 2 第 1 课 预计时间： ~90 分钟

学习目标

推导均方误差的梯度下降更新规则，并从零实现线性回归
从计算复杂度和适用场景两方面对比梯度下降和正规方程
构建一个带特征标准化的多元线性回归模型，并解读学到的权重
解释 Ridge 回归（L2 正则化）如何通过惩罚大权重来防止过拟合

问题背景

你手上有数据：一批房子的面积和它们的成交价。你想根据一套新房子的面积预测它的价格。你可以在散点图上靠眼睛估，但你需要一个公式。你需要一条最贴合数据的直线，这样代入任意面积就能得到价格预测。

线性回归给你这条线。更重要的是，它引入了整套 ML 训练循环：定义模型、定义代价函数、优化参数。每个 ML 算法都遵循这同样的模式。在这里用最简单的情形把它吃透，你以后到处都能认出它。

这不只是用来解决简单问题的。线性回归在生产系统里被用于需求预测、A/B 测试分析、金融建模，以及作为每个回归任务的基线。

核心概念

模型

线性回归假设输入（x）和输出（y）之间是线性关系：

y = wx + b

w（权重/斜率）：x 每增加 1，y 变化多少
b（偏置/截距）：x = 0 时 y 的值

对于多个输入（特征），它扩展为：

y = w1*x1 + w2*x2 + ... + wn*xn + b

或写成向量形式：y = w^T * x + b

目标是：找到一组 w 和 b，让在所有训练样本上预测的 y 尽可能接近真实的 y。

代价函数（均方误差）

你怎么衡量"尽可能接近"？你需要一个单一数字来刻画你的预测错得有多离谱。最常见的选择是均方误差（MSE）：

MSE = (1/n) * sum((y_predicted - y_actual)^2)

为什么要平方？两个原因。第一，它对大误差的惩罚比小误差重得多（误差 10 比误差 1 糟糕 100 倍，而不是 10 倍）。第二，平方函数处处光滑可微，这让优化变得直接。

代价函数构成一个曲面。对于单个权重 w 和偏置 b，MSE 曲面长得像一个碗（凸抛物面）。碗底就是 MSE 最小的地方。训练就是找到那个碗底。

梯度下降

梯度下降通过一步步往下走来找到碗底。

mermaid

flowchart TD
    A[随机初始化 w 和 b] --> B[计算预测：y_hat = wx + b]
    B --> C[计算代价：MSE]
    C --> D[计算梯度：dMSE/dw, dMSE/db]
    D --> E[更新参数]
    E --> F{代价够低了吗？}
    F -->|否| B
    F -->|是| G[完成：找到最优 w 和 b]

梯度告诉你两件事：每个参数该往哪个方向移动，以及移动多少。

对于 y_hat = wx + b 的 MSE：

dMSE/dw = (2/n) * sum((y_hat - y) * x)
dMSE/db = (2/n) * sum(y_hat - y)

更新规则：

w = w - learning_rate * dMSE/dw
b = b - learning_rate * dMSE/db

学习率控制步长。太大：你会越过最小值然后发散。太小：训练慢得没完。常见的起始值：0.01、0.001 或 0.0001。

正规方程（闭式解）

专门针对线性回归，有一个直接公式能不经任何迭代就给出最优权重：

w = (X^T * X)^(-1) * X^T * y

它通过求逆矩阵一步解出 w。在小数据集上完美好用。对于大数据集（数百万行或数千个特征），更推荐梯度下降，因为矩阵求逆关于特征数是 O(n^3) 的复杂度。

多元线性回归

有多个特征时，模型变成：

y = w1*x1 + w2*x2 + ... + wn*xn + b

其余一切照旧：MSE 是代价函数，梯度下降同时更新所有权重。唯一的区别是你拟合的是一个超平面，而不是一条线。

特征缩放在这里很关键。如果一个特征取值范围是 0 到 1，另一个是 0 到 1000000，梯度下降会很吃力，因为代价曲面会被拉得很扁。训练前先标准化特征（减均值、除以标准差）。

多项式回归

如果关系不是线性的怎么办？你仍然可以用线性回归，办法是构造多项式特征：

y = w1*x + w2*x^2 + w3*x^3 + b

这仍然是"线性"回归，因为模型关于权重（w1、w2、w3）是线性的。你只是用了 x 的非线性特征。

更高次的多项式能拟合更复杂的曲线，但有过拟合风险。一个 10 次多项式会穿过 10 点数据集里的每一个点，却在新数据上预测得很差。

R 方分数

MSE 告诉你错得多离谱，但这个数值依赖 y 的尺度。R 方（R^2）给出一个与尺度无关的度量：

R^2 = 1 - (残差平方和) / (相对均值的偏差平方和)
    = 1 - SS_res / SS_tot

R^2 = 1.0：完美预测
R^2 = 0.0：模型不比每次都预测均值更好
R^2 < 0.0：模型比预测均值还差

正则化预告（Ridge 回归）

当你有很多特征时，模型可能通过赋予很大的权重来过拟合。Ridge 回归（L2 正则化）加了一个惩罚项：

Cost = MSE + lambda * sum(w_i^2)

惩罚项抑制大权重。超参数 lambda 控制这个权衡：lambda 越大，权重越小、正则化越强。这会在后面的课里深入讲。现在你只需知道它存在，以及它为什么有用。

figure

linear-regression-fit

动手构建

第 1 步：生成示例数据

python

import random
import math

random.seed(42)

TRUE_W = 3.0
TRUE_B = 7.0
N_SAMPLES = 100

X = [random.uniform(0, 10) for _ in range(N_SAMPLES)]
y = [TRUE_W * x + TRUE_B + random.gauss(0, 2.0) for x in X]

print(f"Generated {N_SAMPLES} samples")
print(f"True relationship: y = {TRUE_W}x + {TRUE_B} (+ noise)")
print(f"First 5 points: {[(round(X[i], 2), round(y[i], 2)) for i in range(5)]}")

第 2 步：用梯度下降从零实现线性回归

python

class LinearRegression:
    def __init__(self, learning_rate=0.01):
        self.w = 0.0
        self.b = 0.0
        self.lr = learning_rate
        self.cost_history = []

    def predict(self, X):
        return [self.w * x + self.b for x in X]

    def compute_cost(self, X, y):
        predictions = self.predict(X)
        n = len(y)
        cost = sum((pred - actual) ** 2 for pred, actual in zip(predictions, y)) / n
        return cost

    def compute_gradients(self, X, y):
        predictions = self.predict(X)
        n = len(y)
        dw = (2 / n) * sum((pred - actual) * x for pred, actual, x in zip(predictions, y, X))
        db = (2 / n) * sum(pred - actual for pred, actual in zip(predictions, y))
        return dw, db

    def fit(self, X, y, epochs=1000, print_every=200):
        for epoch in range(epochs):
            dw, db = self.compute_gradients(X, y)
            self.w -= self.lr * dw
            self.b -= self.lr * db
            cost = self.compute_cost(X, y)
            self.cost_history.append(cost)
            if epoch % print_every == 0:
                print(f"  Epoch {epoch:4d} | Cost: {cost:.4f} | w: {self.w:.4f} | b: {self.b:.4f}")
        return self

    def r_squared(self, X, y):
        predictions = self.predict(X)
        y_mean = sum(y) / len(y)
        ss_res = sum((actual - pred) ** 2 for actual, pred in zip(y, predictions))
        ss_tot = sum((actual - y_mean) ** 2 for actual in y)
        return 1 - (ss_res / ss_tot)

print("=== Training Linear Regression (Gradient Descent) ===")
model = LinearRegression(learning_rate=0.005)
model.fit(X, y, epochs=1000, print_every=200)
print(f"\nLearned: y = {model.w:.4f}x + {model.b:.4f}")
print(f"True:    y = {TRUE_W}x + {TRUE_B}")
print(f"R-squared: {model.r_squared(X, y):.4f}")

第 3 步：正规方程（闭式解）

python

class LinearRegressionNormal:
    def __init__(self):
        self.w = 0.0
        self.b = 0.0

    def fit(self, X, y):
        n = len(X)
        x_mean = sum(X) / n
        y_mean = sum(y) / n
        numerator = sum((X[i] - x_mean) * (y[i] - y_mean) for i in range(n))
        denominator = sum((X[i] - x_mean) ** 2 for i in range(n))
        self.w = numerator / denominator
        self.b = y_mean - self.w * x_mean
        return self

    def predict(self, X):
        return [self.w * x + self.b for x in X]

    def r_squared(self, X, y):
        predictions = self.predict(X)
        y_mean = sum(y) / len(y)
        ss_res = sum((actual - pred) ** 2 for actual, pred in zip(y, predictions))
        ss_tot = sum((actual - y_mean) ** 2 for actual in y)
        return 1 - (ss_res / ss_tot)

print("\n=== Normal Equation (Closed-Form) ===")
model_normal = LinearRegressionNormal()
model_normal.fit(X, y)
print(f"Learned: y = {model_normal.w:.4f}x + {model_normal.b:.4f}")
print(f"R-squared: {model_normal.r_squared(X, y):.4f}")

第 4 步：多元线性回归

python

class MultipleLinearRegression:
    def __init__(self, n_features, learning_rate=0.01):
        self.weights = [0.0] * n_features
        self.bias = 0.0
        self.lr = learning_rate
        self.cost_history = []

    def predict_single(self, x):
        return sum(w * xi for w, xi in zip(self.weights, x)) + self.bias

    def predict(self, X):
        return [self.predict_single(x) for x in X]

    def compute_cost(self, X, y):
        predictions = self.predict(X)
        n = len(y)
        return sum((pred - actual) ** 2 for pred, actual in zip(predictions, y)) / n

    def fit(self, X, y, epochs=1000, print_every=200):
        n = len(y)
        n_features = len(X[0])
        for epoch in range(epochs):
            predictions = self.predict(X)
            errors = [pred - actual for pred, actual in zip(predictions, y)]
            for j in range(n_features):
                grad = (2 / n) * sum(errors[i] * X[i][j] for i in range(n))
                self.weights[j] -= self.lr * grad
            grad_b = (2 / n) * sum(errors)
            self.bias -= self.lr * grad_b
            cost = self.compute_cost(X, y)
            self.cost_history.append(cost)
            if epoch % print_every == 0:
                print(f"  Epoch {epoch:4d} | Cost: {cost:.4f}")
        return self

    def r_squared(self, X, y):
        predictions = self.predict(X)
        y_mean = sum(y) / len(y)
        ss_res = sum((actual - pred) ** 2 for actual, pred in zip(y, predictions))
        ss_tot = sum((actual - y_mean) ** 2 for actual in y)
        return 1 - (ss_res / ss_tot)

random.seed(42)
N = 100
X_multi = []
y_multi = []
for _ in range(N):
    size = random.uniform(500, 3000)
    bedrooms = random.randint(1, 5)
    age = random.uniform(0, 50)
    price = 50 * size + 10000 * bedrooms - 1000 * age + 50000 + random.gauss(0, 20000)
    X_multi.append([size, bedrooms, age])
    y_multi.append(price)

def standardize(X):
    n_features = len(X[0])
    means = [sum(X[i][j] for i in range(len(X))) / len(X) for j in range(n_features)]
    stds = []
    for j in range(n_features):
        variance = sum((X[i][j] - means[j]) ** 2 for i in range(len(X))) / len(X)
        stds.append(variance ** 0.5)
    X_scaled = []
    for i in range(len(X)):
        row = [(X[i][j] - means[j]) / stds[j] if stds[j] > 0 else 0 for j in range(n_features)]
        X_scaled.append(row)
    return X_scaled, means, stds

y_mean_val = sum(y_multi) / len(y_multi)
y_std_val = (sum((yi - y_mean_val) ** 2 for yi in y_multi) / len(y_multi)) ** 0.5
y_scaled = [(yi - y_mean_val) / y_std_val for yi in y_multi]

X_scaled, x_means, x_stds = standardize(X_multi)

print("\n=== Multiple Linear Regression (3 features) ===")
print("Features: house size, bedrooms, age")
multi_model = MultipleLinearRegression(n_features=3, learning_rate=0.01)
multi_model.fit(X_scaled, y_scaled, epochs=1000, print_every=200)

print(f"\nWeights (standardized): {[round(w, 4) for w in multi_model.weights]}")
print(f"Bias (standardized): {multi_model.bias:.4f}")
print(f"R-squared: {multi_model.r_squared(X_scaled, y_scaled):.4f}")

第 5 步：多项式回归

python

class PolynomialRegression:
    def __init__(self, degree, learning_rate=0.01):
        self.degree = degree
        self.weights = [0.0] * degree
        self.bias = 0.0
        self.lr = learning_rate

    def make_features(self, X):
        return [[x ** (d + 1) for d in range(self.degree)] for x in X]

    def predict(self, X):
        features = self.make_features(X)
        return [sum(w * f for w, f in zip(self.weights, row)) + self.bias for row in features]

    def fit(self, X, y, epochs=1000, print_every=200):
        features = self.make_features(X)
        n = len(y)
        for epoch in range(epochs):
            predictions = [sum(w * f for w, f in zip(self.weights, row)) + self.bias for row in features]
            errors = [pred - actual for pred, actual in zip(predictions, y)]
            for j in range(self.degree):
                grad = (2 / n) * sum(errors[i] * features[i][j] for i in range(n))
                self.weights[j] -= self.lr * grad
            grad_b = (2 / n) * sum(errors)
            self.bias -= self.lr * grad_b
            if epoch % print_every == 0:
                cost = sum(e ** 2 for e in errors) / n
                print(f"  Epoch {epoch:4d} | Cost: {cost:.6f}")
        return self

    def r_squared(self, X, y):
        predictions = self.predict(X)
        y_mean = sum(y) / len(y)
        ss_res = sum((actual - pred) ** 2 for actual, pred in zip(y, predictions))
        ss_tot = sum((actual - y_mean) ** 2 for actual in y)
        return 1 - (ss_res / ss_tot)

random.seed(42)
X_poly = [x / 10.0 for x in range(0, 50)]
y_poly = [0.5 * x ** 2 - 2 * x + 3 + random.gauss(0, 1.0) for x in X_poly]

x_max = max(abs(x) for x in X_poly)
X_poly_norm = [x / x_max for x in X_poly]
y_poly_mean = sum(y_poly) / len(y_poly)
y_poly_std = (sum((yi - y_poly_mean) ** 2 for yi in y_poly) / len(y_poly)) ** 0.5
y_poly_norm = [(yi - y_poly_mean) / y_poly_std for yi in y_poly]

print("\n=== Polynomial Regression (degree 2 vs degree 5) ===")
print("True relationship: y = 0.5x^2 - 2x + 3")

print("\nDegree 2:")
poly2 = PolynomialRegression(degree=2, learning_rate=0.1)
poly2.fit(X_poly_norm, y_poly_norm, epochs=2000, print_every=500)
print(f"  R-squared: {poly2.r_squared(X_poly_norm, y_poly_norm):.4f}")

print("\nDegree 5:")
poly5 = PolynomialRegression(degree=5, learning_rate=0.1)
poly5.fit(X_poly_norm, y_poly_norm, epochs=2000, print_every=500)
print(f"  R-squared: {poly5.r_squared(X_poly_norm, y_poly_norm):.4f}")

print("\nDegree 2 fits the true curve well. Degree 5 fits training data slightly better")
print("but risks overfitting on new data.")

第 6 步：Ridge 回归（L2 正则化）

python

class RidgeRegression:
    def __init__(self, n_features, learning_rate=0.01, alpha=1.0):
        self.weights = [0.0] * n_features
        self.bias = 0.0
        self.lr = learning_rate
        self.alpha = alpha

    def predict_single(self, x):
        return sum(w * xi for w, xi in zip(self.weights, x)) + self.bias

    def predict(self, X):
        return [self.predict_single(x) for x in X]

    def fit(self, X, y, epochs=1000, print_every=200):
        n = len(y)
        n_features = len(X[0])
        for epoch in range(epochs):
            predictions = self.predict(X)
            errors = [pred - actual for pred, actual in zip(predictions, y)]
            mse = sum(e ** 2 for e in errors) / n
            reg_term = self.alpha * sum(w ** 2 for w in self.weights)
            cost = mse + reg_term
            for j in range(n_features):
                grad = (2 / n) * sum(errors[i] * X[i][j] for i in range(n))
                grad += 2 * self.alpha * self.weights[j]
                self.weights[j] -= self.lr * grad
            grad_b = (2 / n) * sum(errors)
            self.bias -= self.lr * grad_b
            if epoch % print_every == 0:
                print(f"  Epoch {epoch:4d} | Cost: {cost:.4f} | L2 penalty: {reg_term:.4f}")
        return self

print("\n=== Ridge Regression (L2 Regularization) ===")
print("Same data as multiple regression, with alpha=0.1")
ridge = RidgeRegression(n_features=3, learning_rate=0.01, alpha=0.1)
ridge.fit(X_scaled, y_scaled, epochs=1000, print_every=200)
print(f"\nRidge weights: {[round(w, 4) for w in ridge.weights]}")
print(f"Plain weights: {[round(w, 4) for w in multi_model.weights]}")
print("Ridge weights are smaller (shrunk toward zero) due to the L2 penalty.")

实际使用

现在用 scikit-learn 做同样的事，这才是你在生产里真正会用的。

python

from sklearn.linear_model import LinearRegression as SklearnLR
from sklearn.linear_model import Ridge
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np

np.random.seed(42)
X_sk = np.random.uniform(0, 10, (100, 1))
y_sk = 3.0 * X_sk.squeeze() + 7.0 + np.random.normal(0, 2.0, 100)

X_train, X_test, y_train, y_test = train_test_split(X_sk, y_sk, test_size=0.2, random_state=42)

lr = SklearnLR()
lr.fit(X_train, y_train)
y_pred = lr.predict(X_test)

print("=== Scikit-learn Linear Regression ===")
print(f"Coefficient (w): {lr.coef_[0]:.4f}")
print(f"Intercept (b): {lr.intercept_:.4f}")
print(f"R-squared (test): {r2_score(y_test, y_pred):.4f}")
print(f"MSE (test): {mean_squared_error(y_test, y_pred):.4f}")

poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly_sk = poly.fit_transform(X_train)
X_poly_test = poly.transform(X_test)

lr_poly = SklearnLR()
lr_poly.fit(X_poly_sk, y_train)
print(f"\nPolynomial degree 2 R-squared: {r2_score(y_test, lr_poly.predict(X_poly_test)):.4f}")

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

ridge = Ridge(alpha=1.0)
ridge.fit(X_train_scaled, y_train)
print(f"Ridge R-squared: {r2_score(y_test, ridge.predict(X_test_scaled)):.4f}")
print(f"Ridge coefficient: {ridge.coef_[0]:.4f}")

你的从零实现和 scikit-learn 产出相同的结果。区别在于：scikit-learn 处理了边界情况、数值稳定性和性能优化。生产用库，从零版用来理解底层发生了什么。

拿去用

本节课产出：

outputs/skill-regression.md - 一个根据问题选择合适回归方法的 skill

练习

实现批量梯度下降、随机梯度下降（SGD）和小批量梯度下降。在同一数据集上对比收敛速度。哪个收敛最快？哪个代价曲线最平滑？
从一个三次函数（y = ax^3 + bx^2 + cx + d + 噪声）生成数据。拟合 1 次、3 次和 10 次多项式。对比训练 R^2 和测试 R^2。到几次时过拟合变得明显？
实现 Lasso 回归（L1 正则化：penalty = alpha * sum(|w_i|)）。在多特征房价数据上训练。对比哪些权重归零，和 Ridge 比一比。为什么 L1 产生稀疏解而 L2 不会？

关键术语

术语	大家怎么说	它实际是什么
线性回归	"在数据里画条线"	找到权重 w 和偏置 b，使 wx+b 与真实 y 之间的平方差之和最小
代价函数	"模型有多糟"	把模型参数映射到一个衡量预测误差的单一数字的函数，优化就是去最小化它
均方误差	"误差平方的平均"	(1/n) * sum((预测 - 真实)^2)，对大误差的惩罚不成比例地重
梯度下降	"往下走"	用偏导数沿着减小代价函数的方向迭代调整参数
学习率	"步长"	控制每步梯度下降参数变化幅度的标量
正规方程	"直接解出来"	闭式解 w = (X^T X)^-1 X^T y，不经迭代就给出最优权重
R 方	"拟合得有多好"	模型解释了 y 中多大比例的方差，取值从负无穷到 1.0
特征缩放	"让特征可比"	把特征变换到相近范围（如零均值、单位方差），让梯度下降收敛更快
正则化	"惩罚复杂度"	给代价函数加一项来缩小权重，防止过拟合
Ridge 回归	"L2 正则化"	在 MSE 上加 lambda * sum(w_i^2) 惩罚项的线性回归
多项式回归	"用线性数学拟合曲线"	在多项式特征（x, x^2, x^3, ...）上做线性回归，关于权重仍是线性的
过拟合	"把训练数据背下来"	用一个复杂到连训练数据噪声都拟合的模型，结果在新数据上失败

Claude 认证课程

lessons

00-certification-strategy

01-claude-product-and-model-landscape

02-model-selection-and-token-economics

03-prompting-and-task-decomposition

04-context-knowledge-memory-and-caching

05-output-evaluation-and-validation

06-governance-safety-and-responsible-use

07-workflow-design-and-human-handoffs

08-messages-api-and-application-lifecycle

09-structured-output-and-defensive-parsing

10-tool-use-and-agentic-loops

11-mcp-server-design-and-integration

12-claude-agent-sdk-and-hooks

13-application-security-and-secrets

14-evals-testing-debugging-and-observability

15-claude-code-for-development-teams

16-multi-agent-orchestration-and-delegation

17-agent-sdk-sessions-subagents-and-context

18-tool-contracts-errors-and-progressive-discovery

19-claude-code-memory-rules-skills-and-ci

20-reliable-extraction-batch-and-reviewers

21-long-context-reliability-provenance-and-escalation

22-business-discovery-requirements-and-slas

23-end-to-end-architecture-and-value-tradeoffs

24-rag-retrieval-and-data-pipelines

25-integration-protocols-identity-and-least-privilege

26-production-observability-latency-and-cost

27-enterprise-governance-compliance-and-hitl

28-stakeholder-communication-adrs-and-lifecycle

29-associate-workflow-capstone

30-developer-application-capstone

31-architect-foundations-scenario-capstone

32-architect-professional-system-capstone

references

research

阶段 0：环境与工具

01-dev-environment

outputs

04-apis-and-keys

outputs

05-jupyter-notebooks

outputs

09-data-management

outputs

12-debugging-and-profiling

outputs

阶段 1：数学基础

01-linear-algebra-intuition

outputs

02-vectors-matrices-operations

outputs

03-matrix-transformations

outputs

04-calculus-for-ml

outputs

05-chain-rule-and-autodiff

outputs

06-probability-and-distributions

outputs

07-bayes-theorem

outputs

08-optimization

outputs

09-information-theory

outputs

10-dimensionality-reduction

outputs

11-singular-value-decomposition

outputs

12-tensor-operations

outputs

13-numerical-stability

outputs

14-norms-and-distances

outputs

15-statistics-for-ml

outputs

16-sampling-methods