[{"data":1,"prerenderedAt":3859},["ShallowReactive",2],{"post-\u002Fposts\u002Fdl-framework-autograd-mini":3,"all-posts-nav":3588},{"id":4,"title":5,"body":6,"categories":3571,"date":3573,"description":3574,"draft":3575,"extension":3576,"hidden":3575,"meta":3577,"navigation":852,"path":3578,"published":3575,"seo":3579,"stem":3580,"tags":3581,"__hash__":3587},"posts\u002Fposts\u002Fdl-framework-autograd-mini.md","Week 1：DL 框架与 Autograd——从计算图、反向传播到 Mini Autograd 实现",{"type":7,"value":8,"toc":3538},"minimark",[9,27,44,51,56,59,69,76,127,130,174,177,181,184,187,226,229,235,238,273,287,294,298,301,371,374,391,409,415,436,440,443,446,452,455,461,464,470,476,482,489,492,496,499,559,562,565,579,582,585,588,592,595,601,604,610,613,619,634,637,639,645,647,653,664,667,669,675,678,728,735,738,744,751,757,761,771,777,784,790,793,799,809,815,822,826,829,2313,2316,2322,2342,2348,2354,2415,2418,2438,2441,2447,2457,2463,2477,2483,2489,2504,2507,2513,2527,2530,2555,2562,2568,2574,2589,2592,2642,2654,2659,2664,2678,2681,2685,2690,2703,2706,2729,2736,2741,2744,2765,2768,2771,2774,3134,3137,3143,3146,3150,3153,3156,3162,3165,3182,3185,3189,3192,3224,3227,3231,3237,3243,3246,3252,3255,3261,3264,3300,3303,3306,3338,3341,3355,3359,3362,3390,3393,3416,3419,3425,3429,3432,3459,3462,3466,3469,3505,3508,3511,3514,3526,3534],[10,11,12,13,17,18,22,23,26],"p",{},"如果只用一句话概括 PyTorch \u002F TensorFlow 的本质：",[14,15,16],"strong",{},"它们是在张量计算之上，自动构建计算图，并用链式法则自动求梯度的系统","。训练神经网络看起来是调用 ",[19,20,21],"code",{},"loss.backward()"," 和 ",[19,24,25],{},"optimizer.step()","，但底层真正发生的是：前向阶段记录依赖关系，反向阶段沿图逆序传播梯度，同时在显存、计算量和调度开销之间做工程权衡。",[10,28,29,30,33,34,33,37,33,40,43],{},"这篇文章对应 Week 1 的学习目标，只聚焦 CMU 10-414 中最核心的四块：Computation Graph、Backpropagation、Automatic Differentiation、Memory Optimization。最后我会写一个只支持 ",[19,31,32],{},"matmul","、",[19,35,36],{},"relu",[19,38,39],{},"softmax",[19,41,42],{},"cross_entropy"," 的 mini autograd，用最少代码把深度学习框架的骨架讲清楚。",[10,45,46],{},[47,48],"img",{"alt":49,"src":50},"Forward and backward graph","\u002Fimages\u002Fposts\u002Fmini-autograd\u002Fgraph.svg",[52,53,55],"h2",{"id":54},"为什么先学-autograd","为什么先学 Autograd",[10,57,58],{},"深度学习框架表面上提供了很多能力：GPU 张量、神经网络层、优化器、数据加载、分布式训练、模型导出。但如果把这些能力拆到底，核心闭环只有四步：",[60,61,67],"pre",{"className":62,"code":64,"language":65,"meta":66},[63],"language-text","参数 W 初始化\n  -> forward 计算预测值\n  -> loss 衡量预测和标签差距\n  -> backward 计算每个参数的梯度\n  -> optimizer 用梯度更新参数\n","text","",[19,68,64],{"__ignoreMap":66},[10,70,71,72,75],{},"其中最重要的是 ",[19,73,74],{},"backward","。因为模型训练不是手写每个参数的求导公式，而是让框架自动求导。例如一个两层 MLP：",[60,77,81],{"className":78,"code":79,"language":80,"meta":66,"style":66},"language-python shiki shiki-themes github-dark-dimmed github-light","logits = relu(x @ w1) @ w2\nloss = cross_entropy(logits, y)\nloss.backward()\n","python",[19,82,83,110,121],{"__ignoreMap":66},[84,85,88,92,96,99,102,105,107],"span",{"class":86,"line":87},"line",1,[84,89,91],{"class":90},"ssh_m","logits ",[84,93,95],{"class":94},"s6PUj","=",[84,97,98],{"class":90}," relu(x ",[84,100,101],{"class":94},"@",[84,103,104],{"class":90}," w1) ",[84,106,101],{"class":94},[84,108,109],{"class":90}," w2\n",[84,111,113,116,118],{"class":86,"line":112},2,[84,114,115],{"class":90},"loss ",[84,117,95],{"class":94},[84,119,120],{"class":90}," cross_entropy(logits, y)\n",[84,122,124],{"class":86,"line":123},3,[84,125,126],{"class":90},"loss.backward()\n",[10,128,129],{},"框架需要知道：",[131,132,133,150,159,168,171],"ul",{},[134,135,136,139,140,33,143,33,146,149],"li",{},[19,137,138],{},"logits"," 是怎么由 ",[19,141,142],{},"x",[19,144,145],{},"w1",[19,147,148],{},"w2"," 算出来的；",[134,151,152,155,156,158],{},[19,153,154],{},"loss"," 对 ",[19,157,138],{}," 的梯度是多少；",[134,160,161,33,163,33,165,167],{},[19,162,36],{},[19,164,32],{},[19,166,42],{}," 各自的局部导数是什么；",[134,169,170],{},"多条路径汇合时，梯度如何累加；",[134,172,173],{},"哪些中间激活要保存，哪些可以反向时重新计算。",[10,175,176],{},"所以学 Autograd，本质是在学 PyTorch \u002F TensorFlow 的“训练引擎”。",[52,178,180],{"id":179},"computation-graph计算图是什么","Computation Graph：计算图是什么",[10,182,183],{},"计算图是一个有向无环图。节点代表数据或操作，边代表依赖关系。",[10,185,186],{},"以这段代码为例：",[60,188,190],{"className":78,"code":189,"language":80,"meta":66,"style":66},"z = x @ w\nh = relu(z)\nloss = cross_entropy(h, y)\n",[19,191,192,207,217],{"__ignoreMap":66},[84,193,194,197,199,202,204],{"class":86,"line":87},[84,195,196],{"class":90},"z ",[84,198,95],{"class":94},[84,200,201],{"class":90}," x ",[84,203,101],{"class":94},[84,205,206],{"class":90}," w\n",[84,208,209,212,214],{"class":86,"line":112},[84,210,211],{"class":90},"h ",[84,213,95],{"class":94},[84,215,216],{"class":90}," relu(z)\n",[84,218,219,221,223],{"class":86,"line":123},[84,220,115],{"class":90},[84,222,95],{"class":94},[84,224,225],{"class":90}," cross_entropy(h, y)\n",[10,227,228],{},"可以拆成：",[60,230,233],{"className":231,"code":232,"language":65,"meta":66},[63],"x ----\\\n       matmul -> z -> relu -> h -> cross_entropy -> loss\nw ----\u002F                                      ^\n                                            y\n",[19,234,232],{"__ignoreMap":66},[10,236,237],{},"这里有两类对象：",[239,240,241,261],"ol",{},[134,242,243,246,247,33,249,33,252,33,255,33,258,260],{},[14,244,245],{},"Tensor \u002F Value 节点","：保存真实数据，例如 ",[19,248,142],{},[19,250,251],{},"w",[19,253,254],{},"z",[19,256,257],{},"h",[19,259,154],{},"。",[134,262,263,266,267,33,269,33,271,260],{},[14,264,265],{},"Op \u002F Function 节点","：保存操作规则，例如 ",[19,268,32],{},[19,270,36],{},[19,272,42],{},[10,274,275,276,279,280,33,282,284,285,260],{},"动态图框架 PyTorch 通常在 Python 执行前向代码时即时建图。你写 ",[19,277,278],{},"z = x @ w","，框架马上生成一个新 Tensor，并让这个 Tensor 记住：它来自 ",[19,281,142],{},[19,283,251],{},"，创建它的操作是 ",[19,286,32],{},[10,288,289,290,293],{},"静态图框架早期 TensorFlow 1.x 则先定义图，再统一编译执行。静态图更利于全局优化，动态图更符合 Python 调试习惯。今天 PyTorch 2.x 通过 ",[19,291,292],{},"torch.compile"," 又把动态图捕获为可优化图，本质上是在易用性和编译优化之间折中。",[52,295,297],{"id":296},"forward-graph前向图保存什么","Forward Graph：前向图保存什么",[10,299,300],{},"前向阶段不只是算数值，还要为反向传播留线索。一个 Tensor 通常至少需要这些字段：",[60,302,304],{"className":78,"code":303,"language":80,"meta":66,"style":66},"class Tensor:\n    data        # 真实数值，例如 numpy.ndarray\n    grad        # loss 对当前 Tensor 的梯度\n    requires_grad # 是否需要追踪梯度\n    parents     # 这个 Tensor 依赖哪些父 Tensor\n    op          # 是哪个操作创建了它\n    backward_fn # 给定上游梯度，如何算父节点梯度\n",[19,305,306,318,327,335,344,353,362],{"__ignoreMap":66},[84,307,308,311,315],{"class":86,"line":87},[84,309,310],{"class":94},"class",[84,312,314],{"class":313},"sqRhv"," Tensor",[84,316,317],{"class":90},":\n",[84,319,320,323],{"class":86,"line":112},[84,321,322],{"class":90},"    data        ",[84,324,326],{"class":325},"sgHix","# 真实数值，例如 numpy.ndarray\n",[84,328,329,332],{"class":86,"line":123},[84,330,331],{"class":90},"    grad        ",[84,333,334],{"class":325},"# loss 对当前 Tensor 的梯度\n",[84,336,338,341],{"class":86,"line":337},4,[84,339,340],{"class":90},"    requires_grad ",[84,342,343],{"class":325},"# 是否需要追踪梯度\n",[84,345,347,350],{"class":86,"line":346},5,[84,348,349],{"class":90},"    parents     ",[84,351,352],{"class":325},"# 这个 Tensor 依赖哪些父 Tensor\n",[84,354,356,359],{"class":86,"line":355},6,[84,357,358],{"class":90},"    op          ",[84,360,361],{"class":325},"# 是哪个操作创建了它\n",[84,363,365,368],{"class":86,"line":364},7,[84,366,367],{"class":90},"    backward_fn ",[84,369,370],{"class":325},"# 给定上游梯度，如何算父节点梯度\n",[10,372,373],{},"例如：",[60,375,377],{"className":78,"code":376,"language":80,"meta":66,"style":66},"z = x @ w\n",[19,378,379],{"__ignoreMap":66},[84,380,381,383,385,387,389],{"class":86,"line":87},[84,382,196],{"class":90},[84,384,95],{"class":94},[84,386,201],{"class":90},[84,388,101],{"class":94},[84,390,206],{"class":90},[10,392,393,394,396,397,400,401,404,405,408],{},"会创建一个新 Tensor ",[19,395,254],{},"。它的 ",[19,398,399],{},"parents"," 是 ",[19,402,403],{},"(x, w)","，它的 ",[19,406,407],{},"backward_fn"," 会记录矩阵乘法的导数规则：",[60,410,413],{"className":411,"code":412,"language":65,"meta":66},[63],"z = x @ w\n如果上游梯度是 dz，则：\ndx = dz @ w.T\ndw = x.T @ dz\n",[19,414,412],{"__ignoreMap":66},[10,416,417,418,155,420,422,423,426,427,429,430,22,433,260],{},"这里的“上游梯度”就是 ",[19,419,154],{},[19,421,254],{}," 的梯度，通常写作 ",[19,424,425],{},"dL\u002Fdz","。反向函数负责把 ",[19,428,425],{}," 变成 ",[19,431,432],{},"dL\u002Fdx",[19,434,435],{},"dL\u002Fdw",[52,437,439],{"id":438},"backpropagation反向传播的本质","Backpropagation：反向传播的本质",[10,441,442],{},"反向传播不是神秘算法，它就是链式法则在计算图上的系统化应用。",[10,444,445],{},"假设：",[60,447,450],{"className":448,"code":449,"language":65,"meta":66},[63],"x -> z -> h -> loss\n",[19,451,449],{"__ignoreMap":66},[10,453,454],{},"那么：",[60,456,459],{"className":457,"code":458,"language":65,"meta":66},[63],"dloss\u002Fdx = dloss\u002Fdh * dh\u002Fdz * dz\u002Fdx\n",[19,460,458],{"__ignoreMap":66},[10,462,463],{},"如果图里有分叉和汇合，例如：",[60,465,468],{"className":466,"code":467,"language":65,"meta":66},[63],"      -> a ->\nx           + -> loss\n      -> b ->\n",[19,469,467],{"__ignoreMap":66},[10,471,472,473,475],{},"那么 ",[19,474,142],{}," 的梯度需要把所有路径的贡献加起来：",[60,477,480],{"className":478,"code":479,"language":65,"meta":66},[63],"dloss\u002Fdx = dloss\u002Fda * da\u002Fdx + dloss\u002Fdb * db\u002Fdx\n",[19,481,479],{"__ignoreMap":66},[10,483,484,485,488],{},"这就是 Autograd 里 ",[19,486,487],{},"grad += contribution"," 的原因。一个参数可能被多次使用，或者一个中间结果可能影响多个后续节点，梯度天然需要累加。",[10,490,491],{},"反向传播的执行顺序必须是拓扑逆序：先算离 loss 最近的节点，再算更早的节点。因为某个节点只有收齐所有下游贡献，才能继续把梯度传给它的父节点。",[52,493,495],{"id":494},"automatic-differentiation自动微分不是数值微分","Automatic Differentiation：自动微分不是数值微分",[10,497,498],{},"常见的求导方法有三种：",[500,501,502,518],"table",{},[503,504,505],"thead",{},[506,507,508,512,515],"tr",{},[509,510,511],"th",{},"方法",[509,513,514],{},"思路",[509,516,517],{},"问题",[519,520,521,533,548],"tbody",{},[506,522,523,527,530],{},[524,525,526],"td",{},"手动求导",[524,528,529],{},"人写每层公式",[524,531,532],{},"模型一复杂就不可维护",[506,534,535,538,545],{},[524,536,537],{},"数值微分",[524,539,540,541,544],{},"用 ",[19,542,543],{},"(f(x+eps)-f(x))\u002Feps"," 近似",[524,546,547],{},"慢且有数值误差",[506,549,550,553,556],{},[524,551,552],{},"自动微分",[524,554,555],{},"把程序拆成基本算子并套链式法则",[524,557,558],{},"框架主流方案",[10,560,561],{},"自动微分不是符号求导。它不会把整个程序化简成一个数学表达式，而是在程序实际执行时记录每一步基本操作，然后对每个基本操作调用已知的局部反向规则。",[10,563,564],{},"自动微分主要有两种模式：",[239,566,567,573],{},[134,568,569,572],{},[14,570,571],{},"Forward-mode AD","：从输入往输出推导导数，适合输入维度小、输出维度大的场景。",[134,574,575,578],{},[14,576,577],{},"Reverse-mode AD","：从输出往输入反传梯度，适合深度学习，因为 loss 通常是一个标量，而参数量巨大。",[10,580,581],{},"神经网络训练几乎都用 reverse-mode AD。一次 backward 就能得到所有参数对同一个标量 loss 的梯度。",[52,583,584],{"id":584},"四个核心算子的反向公式",[10,586,587],{},"下面是 mini autograd 要支持的四个算子。",[589,590,591],"h3",{"id":32},"MatMul",[10,593,594],{},"前向：",[60,596,599],{"className":597,"code":598,"language":65,"meta":66},[63],"C = A @ B\n",[19,600,598],{"__ignoreMap":66},[10,602,603],{},"形状：",[60,605,608],{"className":606,"code":607,"language":65,"meta":66},[63],"A: [N, D]\nB: [D, M]\nC: [N, M]\n",[19,609,607],{"__ignoreMap":66},[10,611,612],{},"反向：",[60,614,617],{"className":615,"code":616,"language":65,"meta":66},[63],"dA = dC @ B.T\ndB = A.T @ dC\n",[19,618,616],{"__ignoreMap":66},[10,620,621,622,625,626,629,630,633],{},"直觉：矩阵乘法把 ",[19,623,624],{},"A"," 的每一行和 ",[19,627,628],{},"B"," 的每一列做内积。上游梯度 ",[19,631,632],{},"dC"," 告诉我们每个输出元素对 loss 的影响，再乘回另一个输入，就得到当前输入的梯度。",[589,635,636],{"id":36},"ReLU",[10,638,594],{},[60,640,643],{"className":641,"code":642,"language":65,"meta":66},[63],"relu(x) = max(x, 0)\n",[19,644,642],{"__ignoreMap":66},[10,646,612],{},[60,648,651],{"className":649,"code":650,"language":65,"meta":66},[63],"dx = dout * (x > 0)\n",[19,652,650],{"__ignoreMap":66},[10,654,655,656,659,660,663],{},"如果前向时 ",[19,657,658],{},"x \u003C= 0","，ReLU 输出被截断为 0，局部导数为 0；如果 ",[19,661,662],{},"x > 0","，ReLU 是恒等映射，局部导数为 1。",[589,665,666],{"id":39},"Softmax",[10,668,594],{},[60,670,673],{"className":671,"code":672,"language":65,"meta":66},[63],"softmax(x_i) = exp(x_i) \u002F sum_j exp(x_j)\n",[19,674,672],{"__ignoreMap":66},[10,676,677],{},"实际实现必须做数值稳定：",[60,679,681],{"className":78,"code":680,"language":80,"meta":66,"style":66},"shifted = x - max(x)\nexp = np.exp(shifted)\nprob = exp \u002F exp.sum()\n",[19,682,683,702,712],{"__ignoreMap":66},[84,684,685,688,690,692,695,699],{"class":86,"line":87},[84,686,687],{"class":90},"shifted ",[84,689,95],{"class":94},[84,691,201],{"class":90},[84,693,694],{"class":94},"-",[84,696,698],{"class":697},"swcJU"," max",[84,700,701],{"class":90},"(x)\n",[84,703,704,707,709],{"class":86,"line":112},[84,705,706],{"class":90},"exp ",[84,708,95],{"class":94},[84,710,711],{"class":90}," np.exp(shifted)\n",[84,713,714,717,719,722,725],{"class":86,"line":123},[84,715,716],{"class":90},"prob ",[84,718,95],{"class":94},[84,720,721],{"class":90}," exp ",[84,723,724],{"class":94},"\u002F",[84,726,727],{"class":90}," exp.sum()\n",[10,729,730,731,734],{},"如果不减最大值，",[19,732,733],{},"exp(1000)"," 很容易溢出。",[10,736,737],{},"Softmax 的完整 Jacobian 是：",[60,739,742],{"className":740,"code":741,"language":65,"meta":66},[63],"∂s_i\u002F∂x_j = s_i * (1(i=j) - s_j)\n",[19,743,741],{"__ignoreMap":66},[10,745,746,747,750],{},"在代码里我们通常不显式构造 ",[19,748,749],{},"[C, C]"," 的 Jacobian，而是直接写向量-Jacobian 乘积：",[60,752,755],{"className":753,"code":754,"language":65,"meta":66},[63],"dx = s * (dout - sum(dout * s))\n",[19,756,754],{"__ignoreMap":66},[589,758,760],{"id":759},"cross-entropy","Cross Entropy",[10,762,763,764,767,768,770],{},"对于分类任务，标签 ",[19,765,766],{},"y"," 是类别 id，预测 ",[19,769,10],{}," 是 softmax 概率：",[60,772,775],{"className":773,"code":774,"language":65,"meta":66},[63],"loss = -log(p_y)\n",[19,776,774],{"__ignoreMap":66},[10,778,779,780,783],{},"如果 batch size 是 ",[19,781,782],{},"N","：",[60,785,788],{"className":786,"code":787,"language":65,"meta":66},[63],"loss = mean_i -log(p[i, y_i])\n",[19,789,787],{"__ignoreMap":66},[10,791,792],{},"对 softmax 概率的梯度：",[60,794,797],{"className":795,"code":796,"language":65,"meta":66},[63],"dp[i, y_i] = -1 \u002F p[i, y_i] \u002F N\n",[19,798,796],{"__ignoreMap":66},[10,800,801,802,805,806,783],{},"工程里通常会把 ",[19,803,804],{},"softmax + cross_entropy"," 融合成一个更稳定的 ",[19,807,808],{},"cross_entropy_with_logits",[60,810,813],{"className":811,"code":812,"language":65,"meta":66},[63],"dlogits = (softmax(logits) - one_hot(y)) \u002F N\n",[19,814,812],{"__ignoreMap":66},[10,816,817,818,821],{},"这也是 PyTorch 中 ",[19,819,820],{},"torch.nn.CrossEntropyLoss"," 接收 logits 而不是 softmax 后概率的原因。",[52,823,825],{"id":824},"mini-autograd-完整代码","Mini Autograd 完整代码",[10,827,828],{},"这个实现只依赖 NumPy，代码目标不是功能完整，而是把 autograd 的核心结构讲清楚。",[60,830,832],{"className":78,"code":831,"language":80,"meta":66,"style":66},"import numpy as np\n\n\ndef ensure_tensor(x):\n    if isinstance(x, Tensor):\n        return x\n    return Tensor(x)\n\n\nclass Tensor:\n    def __init__(self, data, requires_grad=False, parents=(), op=\"\"):\n        self.data = np.asarray(data, dtype=np.float64)\n        self.requires_grad = requires_grad\n        self.grad = None\n        self.parents = tuple(parents)\n        self.op = op\n        self._backward = lambda: None\n\n    def __repr__(self):\n        return f\"Tensor(data={self.data}, grad={self.grad}, op={self.op!r})\"\n\n    def __matmul__(self, other):\n        other = ensure_tensor(other)\n        out = Tensor(\n            self.data @ other.data,\n            requires_grad=self.requires_grad or other.requires_grad,\n            parents=(self, other),\n            op=\"matmul\",\n        )\n\n        def _backward():\n            if self.requires_grad:\n                self._accumulate_grad(out.grad @ other.data.T)\n            if other.requires_grad:\n                other._accumulate_grad(self.data.T @ out.grad)\n\n        out._backward = _backward\n        return out\n\n    def relu(self):\n        out = Tensor(\n            np.maximum(self.data, 0.0),\n            requires_grad=self.requires_grad,\n            parents=(self,),\n            op=\"relu\",\n        )\n\n        def _backward():\n            if self.requires_grad:\n                self._accumulate_grad(out.grad * (self.data > 0))\n\n        out._backward = _backward\n        return out\n\n    def softmax(self, axis=-1):\n        shifted = self.data - np.max(self.data, axis=axis, keepdims=True)\n        exp = np.exp(shifted)\n        probs = exp \u002F np.sum(exp, axis=axis, keepdims=True)\n        out = Tensor(\n            probs,\n            requires_grad=self.requires_grad,\n            parents=(self,),\n            op=\"softmax\",\n        )\n\n        def _backward():\n            if self.requires_grad:\n                dot = np.sum(out.grad * probs, axis=axis, keepdims=True)\n                self._accumulate_grad(probs * (out.grad - dot))\n\n        out._backward = _backward\n        return out\n\n    def cross_entropy(self, target):\n        target = np.asarray(target, dtype=np.int64)\n        probs = np.clip(self.data, 1e-12, 1.0)\n        batch_indices = np.arange(target.shape[0])\n        loss_value = -np.log(probs[batch_indices, target]).mean()\n        out = Tensor(\n            loss_value,\n            requires_grad=self.requires_grad,\n            parents=(self,),\n            op=\"cross_entropy\",\n        )\n\n        def _backward():\n            if self.requires_grad:\n                grad = np.zeros_like(self.data)\n                grad[batch_indices, target] = -1.0 \u002F probs[batch_indices, target]\n                grad \u002F= target.shape[0]\n                self._accumulate_grad(out.grad * grad)\n\n        out._backward = _backward\n        return out\n\n    def backward(self, grad=None):\n        if grad is None:\n            if self.data.shape != ():\n                raise RuntimeError(\"grad must be specified for non-scalar tensors\")\n            grad = np.ones_like(self.data)\n\n        topo = []\n        visited = set()\n\n        def build_topo(tensor):\n            if id(tensor) in visited:\n                return\n            visited.add(id(tensor))\n            for parent in tensor.parents:\n                build_topo(parent)\n            topo.append(tensor)\n\n        build_topo(self)\n        self.grad = grad\n\n        for tensor in reversed(topo):\n            tensor._backward()\n\n    def zero_grad(self):\n        self.grad = None\n\n    def _accumulate_grad(self, grad):\n        if self.grad is None:\n            self.grad = grad\n        else:\n            self.grad = self.grad + grad\n",[19,833,834,848,854,858,870,881,889,897,902,907,916,950,973,986,999,1015,1028,1047,1052,1063,1117,1122,1133,1144,1155,1168,1186,1202,1216,1222,1227,1239,1251,1265,1273,1289,1294,1305,1313,1318,1328,1337,1354,1366,1380,1392,1397,1402,1411,1420,1446,1451,1460,1467,1472,1491,1531,1541,1570,1579,1585,1596,1609,1621,1626,1631,1640,1649,1679,1697,1702,1711,1718,1723,1734,1752,1777,1794,1808,1817,1823,1834,1847,1859,1864,1869,1878,1887,1903,1921,1937,1949,1954,1963,1970,1975,1993,2010,2026,2042,2057,2062,2073,2087,2092,2103,2120,2126,2138,2152,2158,2164,2169,2179,2191,2196,2213,2219,2224,2234,2245,2250,2261,2276,2287,2295],{"__ignoreMap":66},[84,835,836,839,842,845],{"class":86,"line":87},[84,837,838],{"class":94},"import",[84,840,841],{"class":90}," numpy ",[84,843,844],{"class":94},"as",[84,846,847],{"class":90}," np\n",[84,849,850],{"class":86,"line":112},[84,851,853],{"emptyLinePlaceholder":852},true,"\n",[84,855,856],{"class":86,"line":123},[84,857,853],{"emptyLinePlaceholder":852},[84,859,860,863,867],{"class":86,"line":337},[84,861,862],{"class":94},"def",[84,864,866],{"class":865},"saVmf"," ensure_tensor",[84,868,869],{"class":90},"(x):\n",[84,871,872,875,878],{"class":86,"line":346},[84,873,874],{"class":94},"    if",[84,876,877],{"class":697}," isinstance",[84,879,880],{"class":90},"(x, Tensor):\n",[84,882,883,886],{"class":86,"line":355},[84,884,885],{"class":94},"        return",[84,887,888],{"class":90}," x\n",[84,890,891,894],{"class":86,"line":364},[84,892,893],{"class":94},"    return",[84,895,896],{"class":90}," Tensor(x)\n",[84,898,900],{"class":86,"line":899},8,[84,901,853],{"emptyLinePlaceholder":852},[84,903,905],{"class":86,"line":904},9,[84,906,853],{"emptyLinePlaceholder":852},[84,908,910,912,914],{"class":86,"line":909},10,[84,911,310],{"class":94},[84,913,314],{"class":313},[84,915,317],{"class":90},[84,917,919,922,925,928,930,933,936,938,941,943,947],{"class":86,"line":918},11,[84,920,921],{"class":94},"    def",[84,923,924],{"class":697}," __init__",[84,926,927],{"class":90},"(self, data, requires_grad",[84,929,95],{"class":94},[84,931,932],{"class":697},"False",[84,934,935],{"class":90},", parents",[84,937,95],{"class":94},[84,939,940],{"class":90},"(), op",[84,942,95],{"class":94},[84,944,946],{"class":945},"sXfbr","\"\"",[84,948,949],{"class":90},"):\n",[84,951,953,956,959,961,964,968,970],{"class":86,"line":952},12,[84,954,955],{"class":697},"        self",[84,957,958],{"class":90},".data ",[84,960,95],{"class":94},[84,962,963],{"class":90}," np.asarray(data, ",[84,965,967],{"class":966},"sNjOc","dtype",[84,969,95],{"class":94},[84,971,972],{"class":90},"np.float64)\n",[84,974,976,978,981,983],{"class":86,"line":975},13,[84,977,955],{"class":697},[84,979,980],{"class":90},".requires_grad ",[84,982,95],{"class":94},[84,984,985],{"class":90}," requires_grad\n",[84,987,989,991,994,996],{"class":86,"line":988},14,[84,990,955],{"class":697},[84,992,993],{"class":90},".grad ",[84,995,95],{"class":94},[84,997,998],{"class":697}," None\n",[84,1000,1002,1004,1007,1009,1012],{"class":86,"line":1001},15,[84,1003,955],{"class":697},[84,1005,1006],{"class":90},".parents ",[84,1008,95],{"class":94},[84,1010,1011],{"class":697}," tuple",[84,1013,1014],{"class":90},"(parents)\n",[84,1016,1018,1020,1023,1025],{"class":86,"line":1017},16,[84,1019,955],{"class":697},[84,1021,1022],{"class":90},".op ",[84,1024,95],{"class":94},[84,1026,1027],{"class":90}," op\n",[84,1029,1031,1033,1036,1038,1041,1044],{"class":86,"line":1030},17,[84,1032,955],{"class":697},[84,1034,1035],{"class":90},"._backward ",[84,1037,95],{"class":94},[84,1039,1040],{"class":94}," lambda",[84,1042,1043],{"class":90},": ",[84,1045,1046],{"class":697},"None\n",[84,1048,1050],{"class":86,"line":1049},18,[84,1051,853],{"emptyLinePlaceholder":852},[84,1053,1055,1057,1060],{"class":86,"line":1054},19,[84,1056,921],{"class":94},[84,1058,1059],{"class":697}," __repr__",[84,1061,1062],{"class":90},"(self):\n",[84,1064,1066,1068,1071,1074,1078,1081,1084,1087,1090,1092,1094,1097,1099,1102,1104,1106,1109,1112,1114],{"class":86,"line":1065},20,[84,1067,885],{"class":94},[84,1069,1070],{"class":94}," f",[84,1072,1073],{"class":945},"\"Tensor(data=",[84,1075,1077],{"class":1076},"sxsTv","{",[84,1079,1080],{"class":697},"self",[84,1082,1083],{"class":90},".data",[84,1085,1086],{"class":1076},"}",[84,1088,1089],{"class":945},", grad=",[84,1091,1077],{"class":1076},[84,1093,1080],{"class":697},[84,1095,1096],{"class":90},".grad",[84,1098,1086],{"class":1076},[84,1100,1101],{"class":945},", op=",[84,1103,1077],{"class":1076},[84,1105,1080],{"class":697},[84,1107,1108],{"class":90},".op",[84,1110,1111],{"class":94},"!r",[84,1113,1086],{"class":1076},[84,1115,1116],{"class":945},")\"\n",[84,1118,1120],{"class":86,"line":1119},21,[84,1121,853],{"emptyLinePlaceholder":852},[84,1123,1125,1127,1130],{"class":86,"line":1124},22,[84,1126,921],{"class":94},[84,1128,1129],{"class":697}," __matmul__",[84,1131,1132],{"class":90},"(self, other):\n",[84,1134,1136,1139,1141],{"class":86,"line":1135},23,[84,1137,1138],{"class":90},"        other ",[84,1140,95],{"class":94},[84,1142,1143],{"class":90}," ensure_tensor(other)\n",[84,1145,1147,1150,1152],{"class":86,"line":1146},24,[84,1148,1149],{"class":90},"        out ",[84,1151,95],{"class":94},[84,1153,1154],{"class":90}," Tensor(\n",[84,1156,1158,1161,1163,1165],{"class":86,"line":1157},25,[84,1159,1160],{"class":697},"            self",[84,1162,958],{"class":90},[84,1164,101],{"class":94},[84,1166,1167],{"class":90}," other.data,\n",[84,1169,1171,1174,1176,1178,1180,1183],{"class":86,"line":1170},26,[84,1172,1173],{"class":966},"            requires_grad",[84,1175,95],{"class":94},[84,1177,1080],{"class":697},[84,1179,980],{"class":90},[84,1181,1182],{"class":94},"or",[84,1184,1185],{"class":90}," other.requires_grad,\n",[84,1187,1189,1192,1194,1197,1199],{"class":86,"line":1188},27,[84,1190,1191],{"class":966},"            parents",[84,1193,95],{"class":94},[84,1195,1196],{"class":90},"(",[84,1198,1080],{"class":697},[84,1200,1201],{"class":90},", other),\n",[84,1203,1205,1208,1210,1213],{"class":86,"line":1204},28,[84,1206,1207],{"class":966},"            op",[84,1209,95],{"class":94},[84,1211,1212],{"class":945},"\"matmul\"",[84,1214,1215],{"class":90},",\n",[84,1217,1219],{"class":86,"line":1218},29,[84,1220,1221],{"class":90},"        )\n",[84,1223,1225],{"class":86,"line":1224},30,[84,1226,853],{"emptyLinePlaceholder":852},[84,1228,1230,1233,1236],{"class":86,"line":1229},31,[84,1231,1232],{"class":94},"        def",[84,1234,1235],{"class":865}," _backward",[84,1237,1238],{"class":90},"():\n",[84,1240,1242,1245,1248],{"class":86,"line":1241},32,[84,1243,1244],{"class":94},"            if",[84,1246,1247],{"class":697}," self",[84,1249,1250],{"class":90},".requires_grad:\n",[84,1252,1254,1257,1260,1262],{"class":86,"line":1253},33,[84,1255,1256],{"class":697},"                self",[84,1258,1259],{"class":90},"._accumulate_grad(out.grad ",[84,1261,101],{"class":94},[84,1263,1264],{"class":90}," other.data.T)\n",[84,1266,1268,1270],{"class":86,"line":1267},34,[84,1269,1244],{"class":94},[84,1271,1272],{"class":90}," other.requires_grad:\n",[84,1274,1276,1279,1281,1284,1286],{"class":86,"line":1275},35,[84,1277,1278],{"class":90},"                other._accumulate_grad(",[84,1280,1080],{"class":697},[84,1282,1283],{"class":90},".data.T ",[84,1285,101],{"class":94},[84,1287,1288],{"class":90}," out.grad)\n",[84,1290,1292],{"class":86,"line":1291},36,[84,1293,853],{"emptyLinePlaceholder":852},[84,1295,1297,1300,1302],{"class":86,"line":1296},37,[84,1298,1299],{"class":90},"        out._backward ",[84,1301,95],{"class":94},[84,1303,1304],{"class":90}," _backward\n",[84,1306,1308,1310],{"class":86,"line":1307},38,[84,1309,885],{"class":94},[84,1311,1312],{"class":90}," out\n",[84,1314,1316],{"class":86,"line":1315},39,[84,1317,853],{"emptyLinePlaceholder":852},[84,1319,1321,1323,1326],{"class":86,"line":1320},40,[84,1322,921],{"class":94},[84,1324,1325],{"class":865}," relu",[84,1327,1062],{"class":90},[84,1329,1331,1333,1335],{"class":86,"line":1330},41,[84,1332,1149],{"class":90},[84,1334,95],{"class":94},[84,1336,1154],{"class":90},[84,1338,1340,1343,1345,1348,1351],{"class":86,"line":1339},42,[84,1341,1342],{"class":90},"            np.maximum(",[84,1344,1080],{"class":697},[84,1346,1347],{"class":90},".data, ",[84,1349,1350],{"class":697},"0.0",[84,1352,1353],{"class":90},"),\n",[84,1355,1357,1359,1361,1363],{"class":86,"line":1356},43,[84,1358,1173],{"class":966},[84,1360,95],{"class":94},[84,1362,1080],{"class":697},[84,1364,1365],{"class":90},".requires_grad,\n",[84,1367,1369,1371,1373,1375,1377],{"class":86,"line":1368},44,[84,1370,1191],{"class":966},[84,1372,95],{"class":94},[84,1374,1196],{"class":90},[84,1376,1080],{"class":697},[84,1378,1379],{"class":90},",),\n",[84,1381,1383,1385,1387,1390],{"class":86,"line":1382},45,[84,1384,1207],{"class":966},[84,1386,95],{"class":94},[84,1388,1389],{"class":945},"\"relu\"",[84,1391,1215],{"class":90},[84,1393,1395],{"class":86,"line":1394},46,[84,1396,1221],{"class":90},[84,1398,1400],{"class":86,"line":1399},47,[84,1401,853],{"emptyLinePlaceholder":852},[84,1403,1405,1407,1409],{"class":86,"line":1404},48,[84,1406,1232],{"class":94},[84,1408,1235],{"class":865},[84,1410,1238],{"class":90},[84,1412,1414,1416,1418],{"class":86,"line":1413},49,[84,1415,1244],{"class":94},[84,1417,1247],{"class":697},[84,1419,1250],{"class":90},[84,1421,1423,1425,1427,1430,1433,1435,1437,1440,1443],{"class":86,"line":1422},50,[84,1424,1256],{"class":697},[84,1426,1259],{"class":90},[84,1428,1429],{"class":94},"*",[84,1431,1432],{"class":90}," (",[84,1434,1080],{"class":697},[84,1436,958],{"class":90},[84,1438,1439],{"class":94},">",[84,1441,1442],{"class":697}," 0",[84,1444,1445],{"class":90},"))\n",[84,1447,1449],{"class":86,"line":1448},51,[84,1450,853],{"emptyLinePlaceholder":852},[84,1452,1454,1456,1458],{"class":86,"line":1453},52,[84,1455,1299],{"class":90},[84,1457,95],{"class":94},[84,1459,1304],{"class":90},[84,1461,1463,1465],{"class":86,"line":1462},53,[84,1464,885],{"class":94},[84,1466,1312],{"class":90},[84,1468,1470],{"class":86,"line":1469},54,[84,1471,853],{"emptyLinePlaceholder":852},[84,1473,1475,1477,1480,1483,1486,1489],{"class":86,"line":1474},55,[84,1476,921],{"class":94},[84,1478,1479],{"class":865}," softmax",[84,1481,1482],{"class":90},"(self, axis",[84,1484,1485],{"class":94},"=-",[84,1487,1488],{"class":697},"1",[84,1490,949],{"class":90},[84,1492,1494,1497,1499,1501,1503,1505,1508,1510,1512,1515,1517,1520,1523,1525,1528],{"class":86,"line":1493},56,[84,1495,1496],{"class":90},"        shifted ",[84,1498,95],{"class":94},[84,1500,1247],{"class":697},[84,1502,958],{"class":90},[84,1504,694],{"class":94},[84,1506,1507],{"class":90}," np.max(",[84,1509,1080],{"class":697},[84,1511,1347],{"class":90},[84,1513,1514],{"class":966},"axis",[84,1516,95],{"class":94},[84,1518,1519],{"class":90},"axis, ",[84,1521,1522],{"class":966},"keepdims",[84,1524,95],{"class":94},[84,1526,1527],{"class":697},"True",[84,1529,1530],{"class":90},")\n",[84,1532,1534,1537,1539],{"class":86,"line":1533},57,[84,1535,1536],{"class":90},"        exp ",[84,1538,95],{"class":94},[84,1540,711],{"class":90},[84,1542,1544,1547,1549,1551,1553,1556,1558,1560,1562,1564,1566,1568],{"class":86,"line":1543},58,[84,1545,1546],{"class":90},"        probs ",[84,1548,95],{"class":94},[84,1550,721],{"class":90},[84,1552,724],{"class":94},[84,1554,1555],{"class":90}," np.sum(exp, ",[84,1557,1514],{"class":966},[84,1559,95],{"class":94},[84,1561,1519],{"class":90},[84,1563,1522],{"class":966},[84,1565,95],{"class":94},[84,1567,1527],{"class":697},[84,1569,1530],{"class":90},[84,1571,1573,1575,1577],{"class":86,"line":1572},59,[84,1574,1149],{"class":90},[84,1576,95],{"class":94},[84,1578,1154],{"class":90},[84,1580,1582],{"class":86,"line":1581},60,[84,1583,1584],{"class":90},"            probs,\n",[84,1586,1588,1590,1592,1594],{"class":86,"line":1587},61,[84,1589,1173],{"class":966},[84,1591,95],{"class":94},[84,1593,1080],{"class":697},[84,1595,1365],{"class":90},[84,1597,1599,1601,1603,1605,1607],{"class":86,"line":1598},62,[84,1600,1191],{"class":966},[84,1602,95],{"class":94},[84,1604,1196],{"class":90},[84,1606,1080],{"class":697},[84,1608,1379],{"class":90},[84,1610,1612,1614,1616,1619],{"class":86,"line":1611},63,[84,1613,1207],{"class":966},[84,1615,95],{"class":94},[84,1617,1618],{"class":945},"\"softmax\"",[84,1620,1215],{"class":90},[84,1622,1624],{"class":86,"line":1623},64,[84,1625,1221],{"class":90},[84,1627,1629],{"class":86,"line":1628},65,[84,1630,853],{"emptyLinePlaceholder":852},[84,1632,1634,1636,1638],{"class":86,"line":1633},66,[84,1635,1232],{"class":94},[84,1637,1235],{"class":865},[84,1639,1238],{"class":90},[84,1641,1643,1645,1647],{"class":86,"line":1642},67,[84,1644,1244],{"class":94},[84,1646,1247],{"class":697},[84,1648,1250],{"class":90},[84,1650,1652,1655,1657,1660,1662,1665,1667,1669,1671,1673,1675,1677],{"class":86,"line":1651},68,[84,1653,1654],{"class":90},"                dot ",[84,1656,95],{"class":94},[84,1658,1659],{"class":90}," np.sum(out.grad ",[84,1661,1429],{"class":94},[84,1663,1664],{"class":90}," probs, ",[84,1666,1514],{"class":966},[84,1668,95],{"class":94},[84,1670,1519],{"class":90},[84,1672,1522],{"class":966},[84,1674,95],{"class":94},[84,1676,1527],{"class":697},[84,1678,1530],{"class":90},[84,1680,1682,1684,1687,1689,1692,1694],{"class":86,"line":1681},69,[84,1683,1256],{"class":697},[84,1685,1686],{"class":90},"._accumulate_grad(probs ",[84,1688,1429],{"class":94},[84,1690,1691],{"class":90}," (out.grad ",[84,1693,694],{"class":94},[84,1695,1696],{"class":90}," dot))\n",[84,1698,1700],{"class":86,"line":1699},70,[84,1701,853],{"emptyLinePlaceholder":852},[84,1703,1705,1707,1709],{"class":86,"line":1704},71,[84,1706,1299],{"class":90},[84,1708,95],{"class":94},[84,1710,1304],{"class":90},[84,1712,1714,1716],{"class":86,"line":1713},72,[84,1715,885],{"class":94},[84,1717,1312],{"class":90},[84,1719,1721],{"class":86,"line":1720},73,[84,1722,853],{"emptyLinePlaceholder":852},[84,1724,1726,1728,1731],{"class":86,"line":1725},74,[84,1727,921],{"class":94},[84,1729,1730],{"class":865}," cross_entropy",[84,1732,1733],{"class":90},"(self, target):\n",[84,1735,1737,1740,1742,1745,1747,1749],{"class":86,"line":1736},75,[84,1738,1739],{"class":90},"        target ",[84,1741,95],{"class":94},[84,1743,1744],{"class":90}," np.asarray(target, ",[84,1746,967],{"class":966},[84,1748,95],{"class":94},[84,1750,1751],{"class":90},"np.int64)\n",[84,1753,1755,1757,1759,1762,1764,1766,1769,1772,1775],{"class":86,"line":1754},76,[84,1756,1546],{"class":90},[84,1758,95],{"class":94},[84,1760,1761],{"class":90}," np.clip(",[84,1763,1080],{"class":697},[84,1765,1347],{"class":90},[84,1767,1768],{"class":697},"1e-12",[84,1770,1771],{"class":90},", ",[84,1773,1774],{"class":697},"1.0",[84,1776,1530],{"class":90},[84,1778,1780,1783,1785,1788,1791],{"class":86,"line":1779},77,[84,1781,1782],{"class":90},"        batch_indices ",[84,1784,95],{"class":94},[84,1786,1787],{"class":90}," np.arange(target.shape[",[84,1789,1790],{"class":697},"0",[84,1792,1793],{"class":90},"])\n",[84,1795,1797,1800,1802,1805],{"class":86,"line":1796},78,[84,1798,1799],{"class":90},"        loss_value ",[84,1801,95],{"class":94},[84,1803,1804],{"class":94}," -",[84,1806,1807],{"class":90},"np.log(probs[batch_indices, target]).mean()\n",[84,1809,1811,1813,1815],{"class":86,"line":1810},79,[84,1812,1149],{"class":90},[84,1814,95],{"class":94},[84,1816,1154],{"class":90},[84,1818,1820],{"class":86,"line":1819},80,[84,1821,1822],{"class":90},"            loss_value,\n",[84,1824,1826,1828,1830,1832],{"class":86,"line":1825},81,[84,1827,1173],{"class":966},[84,1829,95],{"class":94},[84,1831,1080],{"class":697},[84,1833,1365],{"class":90},[84,1835,1837,1839,1841,1843,1845],{"class":86,"line":1836},82,[84,1838,1191],{"class":966},[84,1840,95],{"class":94},[84,1842,1196],{"class":90},[84,1844,1080],{"class":697},[84,1846,1379],{"class":90},[84,1848,1850,1852,1854,1857],{"class":86,"line":1849},83,[84,1851,1207],{"class":966},[84,1853,95],{"class":94},[84,1855,1856],{"class":945},"\"cross_entropy\"",[84,1858,1215],{"class":90},[84,1860,1862],{"class":86,"line":1861},84,[84,1863,1221],{"class":90},[84,1865,1867],{"class":86,"line":1866},85,[84,1868,853],{"emptyLinePlaceholder":852},[84,1870,1872,1874,1876],{"class":86,"line":1871},86,[84,1873,1232],{"class":94},[84,1875,1235],{"class":865},[84,1877,1238],{"class":90},[84,1879,1881,1883,1885],{"class":86,"line":1880},87,[84,1882,1244],{"class":94},[84,1884,1247],{"class":697},[84,1886,1250],{"class":90},[84,1888,1890,1893,1895,1898,1900],{"class":86,"line":1889},88,[84,1891,1892],{"class":90},"                grad ",[84,1894,95],{"class":94},[84,1896,1897],{"class":90}," np.zeros_like(",[84,1899,1080],{"class":697},[84,1901,1902],{"class":90},".data)\n",[84,1904,1906,1909,1911,1913,1915,1918],{"class":86,"line":1905},89,[84,1907,1908],{"class":90},"                grad[batch_indices, target] ",[84,1910,95],{"class":94},[84,1912,1804],{"class":94},[84,1914,1774],{"class":697},[84,1916,1917],{"class":94}," \u002F",[84,1919,1920],{"class":90}," probs[batch_indices, target]\n",[84,1922,1924,1926,1929,1932,1934],{"class":86,"line":1923},90,[84,1925,1892],{"class":90},[84,1927,1928],{"class":94},"\u002F=",[84,1930,1931],{"class":90}," target.shape[",[84,1933,1790],{"class":697},[84,1935,1936],{"class":90},"]\n",[84,1938,1940,1942,1944,1946],{"class":86,"line":1939},91,[84,1941,1256],{"class":697},[84,1943,1259],{"class":90},[84,1945,1429],{"class":94},[84,1947,1948],{"class":90}," grad)\n",[84,1950,1952],{"class":86,"line":1951},92,[84,1953,853],{"emptyLinePlaceholder":852},[84,1955,1957,1959,1961],{"class":86,"line":1956},93,[84,1958,1299],{"class":90},[84,1960,95],{"class":94},[84,1962,1304],{"class":90},[84,1964,1966,1968],{"class":86,"line":1965},94,[84,1967,885],{"class":94},[84,1969,1312],{"class":90},[84,1971,1973],{"class":86,"line":1972},95,[84,1974,853],{"emptyLinePlaceholder":852},[84,1976,1978,1980,1983,1986,1988,1991],{"class":86,"line":1977},96,[84,1979,921],{"class":94},[84,1981,1982],{"class":865}," backward",[84,1984,1985],{"class":90},"(self, grad",[84,1987,95],{"class":94},[84,1989,1990],{"class":697},"None",[84,1992,949],{"class":90},[84,1994,1996,1999,2002,2005,2008],{"class":86,"line":1995},97,[84,1997,1998],{"class":94},"        if",[84,2000,2001],{"class":90}," grad ",[84,2003,2004],{"class":94},"is",[84,2006,2007],{"class":697}," None",[84,2009,317],{"class":90},[84,2011,2013,2015,2017,2020,2023],{"class":86,"line":2012},98,[84,2014,1244],{"class":94},[84,2016,1247],{"class":697},[84,2018,2019],{"class":90},".data.shape ",[84,2021,2022],{"class":94},"!=",[84,2024,2025],{"class":90}," ():\n",[84,2027,2029,2032,2035,2037,2040],{"class":86,"line":2028},99,[84,2030,2031],{"class":94},"                raise",[84,2033,2034],{"class":697}," RuntimeError",[84,2036,1196],{"class":90},[84,2038,2039],{"class":945},"\"grad must be specified for non-scalar tensors\"",[84,2041,1530],{"class":90},[84,2043,2045,2048,2050,2053,2055],{"class":86,"line":2044},100,[84,2046,2047],{"class":90},"            grad ",[84,2049,95],{"class":94},[84,2051,2052],{"class":90}," np.ones_like(",[84,2054,1080],{"class":697},[84,2056,1902],{"class":90},[84,2058,2060],{"class":86,"line":2059},101,[84,2061,853],{"emptyLinePlaceholder":852},[84,2063,2065,2068,2070],{"class":86,"line":2064},102,[84,2066,2067],{"class":90},"        topo ",[84,2069,95],{"class":94},[84,2071,2072],{"class":90}," []\n",[84,2074,2076,2079,2081,2084],{"class":86,"line":2075},103,[84,2077,2078],{"class":90},"        visited ",[84,2080,95],{"class":94},[84,2082,2083],{"class":697}," set",[84,2085,2086],{"class":90},"()\n",[84,2088,2090],{"class":86,"line":2089},104,[84,2091,853],{"emptyLinePlaceholder":852},[84,2093,2095,2097,2100],{"class":86,"line":2094},105,[84,2096,1232],{"class":94},[84,2098,2099],{"class":865}," build_topo",[84,2101,2102],{"class":90},"(tensor):\n",[84,2104,2106,2108,2111,2114,2117],{"class":86,"line":2105},106,[84,2107,1244],{"class":94},[84,2109,2110],{"class":697}," id",[84,2112,2113],{"class":90},"(tensor) ",[84,2115,2116],{"class":94},"in",[84,2118,2119],{"class":90}," visited:\n",[84,2121,2123],{"class":86,"line":2122},107,[84,2124,2125],{"class":94},"                return\n",[84,2127,2129,2132,2135],{"class":86,"line":2128},108,[84,2130,2131],{"class":90},"            visited.add(",[84,2133,2134],{"class":697},"id",[84,2136,2137],{"class":90},"(tensor))\n",[84,2139,2141,2144,2147,2149],{"class":86,"line":2140},109,[84,2142,2143],{"class":94},"            for",[84,2145,2146],{"class":90}," parent ",[84,2148,2116],{"class":94},[84,2150,2151],{"class":90}," tensor.parents:\n",[84,2153,2155],{"class":86,"line":2154},110,[84,2156,2157],{"class":90},"                build_topo(parent)\n",[84,2159,2161],{"class":86,"line":2160},111,[84,2162,2163],{"class":90},"            topo.append(tensor)\n",[84,2165,2167],{"class":86,"line":2166},112,[84,2168,853],{"emptyLinePlaceholder":852},[84,2170,2172,2175,2177],{"class":86,"line":2171},113,[84,2173,2174],{"class":90},"        build_topo(",[84,2176,1080],{"class":697},[84,2178,1530],{"class":90},[84,2180,2182,2184,2186,2188],{"class":86,"line":2181},114,[84,2183,955],{"class":697},[84,2185,993],{"class":90},[84,2187,95],{"class":94},[84,2189,2190],{"class":90}," grad\n",[84,2192,2194],{"class":86,"line":2193},115,[84,2195,853],{"emptyLinePlaceholder":852},[84,2197,2199,2202,2205,2207,2210],{"class":86,"line":2198},116,[84,2200,2201],{"class":94},"        for",[84,2203,2204],{"class":90}," tensor ",[84,2206,2116],{"class":94},[84,2208,2209],{"class":697}," reversed",[84,2211,2212],{"class":90},"(topo):\n",[84,2214,2216],{"class":86,"line":2215},117,[84,2217,2218],{"class":90},"            tensor._backward()\n",[84,2220,2222],{"class":86,"line":2221},118,[84,2223,853],{"emptyLinePlaceholder":852},[84,2225,2227,2229,2232],{"class":86,"line":2226},119,[84,2228,921],{"class":94},[84,2230,2231],{"class":865}," zero_grad",[84,2233,1062],{"class":90},[84,2235,2237,2239,2241,2243],{"class":86,"line":2236},120,[84,2238,955],{"class":697},[84,2240,993],{"class":90},[84,2242,95],{"class":94},[84,2244,998],{"class":697},[84,2246,2248],{"class":86,"line":2247},121,[84,2249,853],{"emptyLinePlaceholder":852},[84,2251,2253,2255,2258],{"class":86,"line":2252},122,[84,2254,921],{"class":94},[84,2256,2257],{"class":865}," _accumulate_grad",[84,2259,2260],{"class":90},"(self, grad):\n",[84,2262,2264,2266,2268,2270,2272,2274],{"class":86,"line":2263},123,[84,2265,1998],{"class":94},[84,2267,1247],{"class":697},[84,2269,993],{"class":90},[84,2271,2004],{"class":94},[84,2273,2007],{"class":697},[84,2275,317],{"class":90},[84,2277,2279,2281,2283,2285],{"class":86,"line":2278},124,[84,2280,1160],{"class":697},[84,2282,993],{"class":90},[84,2284,95],{"class":94},[84,2286,2190],{"class":90},[84,2288,2290,2293],{"class":86,"line":2289},125,[84,2291,2292],{"class":94},"        else",[84,2294,317],{"class":90},[84,2296,2298,2300,2302,2304,2306,2308,2311],{"class":86,"line":2297},126,[84,2299,1160],{"class":697},[84,2301,993],{"class":90},[84,2303,95],{"class":94},[84,2305,1247],{"class":697},[84,2307,993],{"class":90},[84,2309,2310],{"class":94},"+",[84,2312,2190],{"class":90},[52,2314,2315],{"id":2315},"每个结构体和函数细讲",[589,2317,2319],{"id":2318},"tensordata",[19,2320,2321],{},"Tensor.data",[10,2323,2324,2327,2328,2331,2332,33,2335,33,2338,2341],{},[19,2325,2326],{},"data"," 是真实数值。这里用 ",[19,2329,2330],{},"np.ndarray"," 保存。真实框架里，Tensor 不只保存数据，还要保存设备、dtype、stride、storage、layout 等信息。例如 PyTorch Tensor 可能在 CPU 或 CUDA 上，可能是 ",[19,2333,2334],{},"float32",[19,2336,2337],{},"float16",[19,2339,2340],{},"bfloat16","，也可能是非连续内存视图。",[589,2343,2345],{"id":2344},"tensorrequires_grad",[19,2346,2347],{},"Tensor.requires_grad",[10,2349,2350,2353],{},[19,2351,2352],{},"requires_grad"," 表示是否需要追踪梯度。输入数据通常不需要梯度，模型参数需要梯度。",[60,2355,2357],{"className":78,"code":2356,"language":80,"meta":66,"style":66},"x = Tensor([[1, 2]], requires_grad=False)\nw = Tensor([[0.1], [0.2]], requires_grad=True)\n",[19,2358,2359,2387],{"__ignoreMap":66},[84,2360,2361,2364,2366,2369,2371,2373,2376,2379,2381,2383,2385],{"class":86,"line":87},[84,2362,2363],{"class":90},"x ",[84,2365,95],{"class":94},[84,2367,2368],{"class":90}," Tensor([[",[84,2370,1488],{"class":697},[84,2372,1771],{"class":90},[84,2374,2375],{"class":697},"2",[84,2377,2378],{"class":90},"]], ",[84,2380,2352],{"class":966},[84,2382,95],{"class":94},[84,2384,932],{"class":697},[84,2386,1530],{"class":90},[84,2388,2389,2392,2394,2396,2399,2402,2405,2407,2409,2411,2413],{"class":86,"line":112},[84,2390,2391],{"class":90},"w ",[84,2393,95],{"class":94},[84,2395,2368],{"class":90},[84,2397,2398],{"class":697},"0.1",[84,2400,2401],{"class":90},"], [",[84,2403,2404],{"class":697},"0.2",[84,2406,2378],{"class":90},[84,2408,2352],{"class":966},[84,2410,95],{"class":94},[84,2412,1527],{"class":697},[84,2414,1530],{"class":90},[10,2416,2417],{},"如果一个操作的任意父节点需要梯度，那么输出也需要梯度：",[60,2419,2421],{"className":78,"code":2420,"language":80,"meta":66,"style":66},"requires_grad=self.requires_grad or other.requires_grad\n",[19,2422,2423],{"__ignoreMap":66},[84,2424,2425,2427,2429,2431,2433,2435],{"class":86,"line":87},[84,2426,2352],{"class":90},[84,2428,95],{"class":94},[84,2430,1080],{"class":697},[84,2432,980],{"class":90},[84,2434,1182],{"class":94},[84,2436,2437],{"class":90}," other.requires_grad\n",[10,2439,2440],{},"这就是梯度追踪在图上传播的方式。",[589,2442,2444],{"id":2443},"tensorgrad",[19,2445,2446],{},"Tensor.grad",[10,2448,2449,2452,2453,2456],{},[19,2450,2451],{},"grad"," 保存当前 Tensor 的梯度，也就是 ",[19,2454,2455],{},"dLoss\u002FdTensor","。注意它不是局部导数，而是最终 loss 对这个 Tensor 的总导数。",[10,2458,2459,2460,2462],{},"为什么初始是 ",[19,2461,1990],{}," 而不是 0？因为这样可以区分“还没算过梯度”和“梯度确实为 0”。真实训练中每个 step 前都要清空梯度，否则梯度会跨 batch 累加。",[60,2464,2466],{"className":78,"code":2465,"language":80,"meta":66,"style":66},"w.zero_grad()\nloss.backward()\n",[19,2467,2468,2473],{"__ignoreMap":66},[84,2469,2470],{"class":86,"line":87},[84,2471,2472],{"class":90},"w.zero_grad()\n",[84,2474,2475],{"class":86,"line":112},[84,2476,126],{"class":90},[10,2478,2479,2480,260],{},"PyTorch 也是默认累加梯度，所以训练循环里必须写 ",[19,2481,2482],{},"optimizer.zero_grad()",[589,2484,2486],{"id":2485},"tensorparents",[19,2487,2488],{},"Tensor.parents",[10,2490,2491,2493,2494,2496,2497,2500,2501,2503],{},[19,2492,399],{}," 保存当前 Tensor 的输入依赖。例如 ",[19,2495,278],{},"，那么 ",[19,2498,2499],{},"z.parents = (x, w)","。反向传播构建拓扑排序时要沿着 ",[19,2502,399],{}," 一直追溯到叶子节点。",[10,2505,2506],{},"叶子节点一般是用户直接创建的 Tensor，例如输入和参数。中间节点是由操作产生的 Tensor。",[589,2508,2510],{"id":2509},"tensorop",[19,2511,2512],{},"Tensor.op",[10,2514,2515,2518,2519,2522,2523,2526],{},[19,2516,2517],{},"op"," 只是为了调试展示。真实框架里会有更复杂的 ",[19,2520,2521],{},"grad_fn"," 或 ",[19,2524,2525],{},"Function"," 对象，里面保存算子类型、反向规则、上下文缓存等。",[10,2528,2529],{},"例如 PyTorch 里：",[60,2531,2533],{"className":78,"code":2532,"language":80,"meta":66,"style":66},"z = x @ w\nprint(z.grad_fn)\n",[19,2534,2535,2547],{"__ignoreMap":66},[84,2536,2537,2539,2541,2543,2545],{"class":86,"line":87},[84,2538,196],{"class":90},[84,2540,95],{"class":94},[84,2542,201],{"class":90},[84,2544,101],{"class":94},[84,2546,206],{"class":90},[84,2548,2549,2552],{"class":86,"line":112},[84,2550,2551],{"class":697},"print",[84,2553,2554],{"class":90},"(z.grad_fn)\n",[10,2556,2557,2558,2561],{},"你会看到类似 ",[19,2559,2560],{},"MmBackward"," 的对象。",[589,2563,2565],{"id":2564},"tensor_backward",[19,2566,2567],{},"Tensor._backward",[10,2569,2570,2573],{},[19,2571,2572],{},"_backward"," 是 mini autograd 的核心。每个操作在前向时创建输出 Tensor，并给输出 Tensor 塞一个闭包函数。这个闭包知道：",[131,2575,2576,2579,2582],{},[134,2577,2578],{},"当前操作的输入是谁；",[134,2580,2581],{},"当前操作的输出是谁；",[134,2583,2584,2585,2588],{},"输出的梯度 ",[19,2586,2587],{},"out.grad"," 如何转成输入的梯度。",[10,2590,2591],{},"例如 matmul：",[60,2593,2595],{"className":78,"code":2594,"language":80,"meta":66,"style":66},"def _backward():\n    if self.requires_grad:\n        self._accumulate_grad(out.grad @ other.data.T)\n    if other.requires_grad:\n        other._accumulate_grad(self.data.T @ out.grad)\n",[19,2596,2597,2605,2613,2623,2629],{"__ignoreMap":66},[84,2598,2599,2601,2603],{"class":86,"line":87},[84,2600,862],{"class":94},[84,2602,1235],{"class":865},[84,2604,1238],{"class":90},[84,2606,2607,2609,2611],{"class":86,"line":112},[84,2608,874],{"class":94},[84,2610,1247],{"class":697},[84,2612,1250],{"class":90},[84,2614,2615,2617,2619,2621],{"class":86,"line":123},[84,2616,955],{"class":697},[84,2618,1259],{"class":90},[84,2620,101],{"class":94},[84,2622,1264],{"class":90},[84,2624,2625,2627],{"class":86,"line":337},[84,2626,874],{"class":94},[84,2628,1272],{"class":90},[84,2630,2631,2634,2636,2638,2640],{"class":86,"line":346},[84,2632,2633],{"class":90},"        other._accumulate_grad(",[84,2635,1080],{"class":697},[84,2637,1283],{"class":90},[84,2639,101],{"class":94},[84,2641,1288],{"class":90},[10,2643,2644,2645,33,2647,33,2650,2653],{},"闭包会捕获 ",[19,2646,1080],{},[19,2648,2649],{},"other",[19,2651,2652],{},"out","。这就是动态图框架非常自然的地方：Python 执行到哪里，反向函数就记录到哪里。",[589,2655,2657],{"id":2656},"ensure_tensor",[19,2658,2656],{},[10,2660,2661,2663],{},[19,2662,2656],{}," 负责把普通数字或数组包装成 Tensor。这样以后可以支持：",[60,2665,2667],{"className":78,"code":2666,"language":80,"meta":66,"style":66},"x @ np_array\n",[19,2668,2669],{"__ignoreMap":66},[84,2670,2671,2673,2675],{"class":86,"line":87},[84,2672,2363],{"class":90},[84,2674,101],{"class":94},[84,2676,2677],{"class":90}," np_array\n",[10,2679,2680],{},"真实框架中类似逻辑会更复杂，因为要处理 dtype promotion、device 对齐、广播规则等。",[589,2682,2683],{"id":74},[19,2684,74],{},[10,2686,2687,2689],{},[19,2688,74],{}," 做三件事：",[239,2691,2692,2695,2698],{},[134,2693,2694],{},"如果当前 Tensor 是标量 loss，则默认上游梯度为 1；",[134,2696,2697],{},"从 loss 出发 DFS 构建拓扑序；",[134,2699,2700,2701,260],{},"逆拓扑序调用每个 Tensor 的 ",[19,2702,2572],{},[10,2704,2705],{},"关键代码：",[60,2707,2709],{"className":78,"code":2708,"language":80,"meta":66,"style":66},"for tensor in reversed(topo):\n    tensor._backward()\n",[19,2710,2711,2724],{"__ignoreMap":66},[84,2712,2713,2716,2718,2720,2722],{"class":86,"line":87},[84,2714,2715],{"class":94},"for",[84,2717,2204],{"class":90},[84,2719,2116],{"class":94},[84,2721,2209],{"class":697},[84,2723,2212],{"class":90},[84,2725,2726],{"class":86,"line":112},[84,2727,2728],{"class":90},"    tensor._backward()\n",[10,2730,2731,2732,2735],{},"为什么要反过来？因为 ",[19,2733,2734],{},"topo"," 是从叶子到 loss 的顺序，反向传播必须从 loss 回到叶子。",[589,2737,2739],{"id":2738},"_accumulate_grad",[19,2740,2738],{},[10,2742,2743],{},"梯度必须累加：",[60,2745,2747],{"className":78,"code":2746,"language":80,"meta":66,"style":66},"self.grad = self.grad + grad\n",[19,2748,2749],{"__ignoreMap":66},[84,2750,2751,2753,2755,2757,2759,2761,2763],{"class":86,"line":87},[84,2752,1080],{"class":697},[84,2754,993],{"class":90},[84,2756,95],{"class":94},[84,2758,1247],{"class":697},[84,2760,993],{"class":90},[84,2762,2310],{"class":94},[84,2764,2190],{"class":90},[10,2766,2767],{},"如果一个 Tensor 影响 loss 的路径不止一条，每条路径都会贡献一部分梯度。Autograd 的正确性依赖累加，而不是覆盖。",[52,2769,2770],{"id":2770},"跑一个最小训练例子",[10,2772,2773],{},"下面构造一个两层分类模型：",[60,2775,2777],{"className":78,"code":2776,"language":80,"meta":66,"style":66},"np.random.seed(0)\n\nx = Tensor(np.array([\n    [1.0, 2.0, 1.0],\n    [2.0, 0.0, 1.0],\n    [0.0, 1.0, 2.0],\n]), requires_grad=False)\n\ny = np.array([0, 1, 1])\n\nw1 = Tensor(np.random.randn(3, 4) * 0.1, requires_grad=True)\nw2 = Tensor(np.random.randn(4, 2) * 0.1, requires_grad=True)\n\nfor step in range(50):\n    for p in (w1, w2):\n        p.zero_grad()\n\n    logits = (x @ w1).relu() @ w2\n    probs = logits.softmax(axis=1)\n    loss = probs.cross_entropy(y)\n    loss.backward()\n\n    lr = 0.5\n    w1.data -= lr * w1.grad\n    w2.data -= lr * w2.grad\n\n    if step % 10 == 0:\n        print(step, loss.data)\n",[19,2778,2779,2788,2792,2801,2820,2836,2852,2865,2869,2891,2895,2931,2962,2966,2985,2998,3003,3007,3026,3044,3054,3059,3063,3073,3089,3103,3107,3126],{"__ignoreMap":66},[84,2780,2781,2784,2786],{"class":86,"line":87},[84,2782,2783],{"class":90},"np.random.seed(",[84,2785,1790],{"class":697},[84,2787,1530],{"class":90},[84,2789,2790],{"class":86,"line":112},[84,2791,853],{"emptyLinePlaceholder":852},[84,2793,2794,2796,2798],{"class":86,"line":123},[84,2795,2363],{"class":90},[84,2797,95],{"class":94},[84,2799,2800],{"class":90}," Tensor(np.array([\n",[84,2802,2803,2806,2808,2810,2813,2815,2817],{"class":86,"line":337},[84,2804,2805],{"class":90},"    [",[84,2807,1774],{"class":697},[84,2809,1771],{"class":90},[84,2811,2812],{"class":697},"2.0",[84,2814,1771],{"class":90},[84,2816,1774],{"class":697},[84,2818,2819],{"class":90},"],\n",[84,2821,2822,2824,2826,2828,2830,2832,2834],{"class":86,"line":346},[84,2823,2805],{"class":90},[84,2825,2812],{"class":697},[84,2827,1771],{"class":90},[84,2829,1350],{"class":697},[84,2831,1771],{"class":90},[84,2833,1774],{"class":697},[84,2835,2819],{"class":90},[84,2837,2838,2840,2842,2844,2846,2848,2850],{"class":86,"line":355},[84,2839,2805],{"class":90},[84,2841,1350],{"class":697},[84,2843,1771],{"class":90},[84,2845,1774],{"class":697},[84,2847,1771],{"class":90},[84,2849,2812],{"class":697},[84,2851,2819],{"class":90},[84,2853,2854,2857,2859,2861,2863],{"class":86,"line":364},[84,2855,2856],{"class":90},"]), ",[84,2858,2352],{"class":966},[84,2860,95],{"class":94},[84,2862,932],{"class":697},[84,2864,1530],{"class":90},[84,2866,2867],{"class":86,"line":899},[84,2868,853],{"emptyLinePlaceholder":852},[84,2870,2871,2874,2876,2879,2881,2883,2885,2887,2889],{"class":86,"line":904},[84,2872,2873],{"class":90},"y ",[84,2875,95],{"class":94},[84,2877,2878],{"class":90}," np.array([",[84,2880,1790],{"class":697},[84,2882,1771],{"class":90},[84,2884,1488],{"class":697},[84,2886,1771],{"class":90},[84,2888,1488],{"class":697},[84,2890,1793],{"class":90},[84,2892,2893],{"class":86,"line":909},[84,2894,853],{"emptyLinePlaceholder":852},[84,2896,2897,2900,2902,2905,2908,2910,2913,2916,2918,2921,2923,2925,2927,2929],{"class":86,"line":918},[84,2898,2899],{"class":90},"w1 ",[84,2901,95],{"class":94},[84,2903,2904],{"class":90}," Tensor(np.random.randn(",[84,2906,2907],{"class":697},"3",[84,2909,1771],{"class":90},[84,2911,2912],{"class":697},"4",[84,2914,2915],{"class":90},") ",[84,2917,1429],{"class":94},[84,2919,2920],{"class":697}," 0.1",[84,2922,1771],{"class":90},[84,2924,2352],{"class":966},[84,2926,95],{"class":94},[84,2928,1527],{"class":697},[84,2930,1530],{"class":90},[84,2932,2933,2936,2938,2940,2942,2944,2946,2948,2950,2952,2954,2956,2958,2960],{"class":86,"line":952},[84,2934,2935],{"class":90},"w2 ",[84,2937,95],{"class":94},[84,2939,2904],{"class":90},[84,2941,2912],{"class":697},[84,2943,1771],{"class":90},[84,2945,2375],{"class":697},[84,2947,2915],{"class":90},[84,2949,1429],{"class":94},[84,2951,2920],{"class":697},[84,2953,1771],{"class":90},[84,2955,2352],{"class":966},[84,2957,95],{"class":94},[84,2959,1527],{"class":697},[84,2961,1530],{"class":90},[84,2963,2964],{"class":86,"line":975},[84,2965,853],{"emptyLinePlaceholder":852},[84,2967,2968,2970,2973,2975,2978,2980,2983],{"class":86,"line":988},[84,2969,2715],{"class":94},[84,2971,2972],{"class":90}," step ",[84,2974,2116],{"class":94},[84,2976,2977],{"class":697}," range",[84,2979,1196],{"class":90},[84,2981,2982],{"class":697},"50",[84,2984,949],{"class":90},[84,2986,2987,2990,2993,2995],{"class":86,"line":1001},[84,2988,2989],{"class":94},"    for",[84,2991,2992],{"class":90}," p ",[84,2994,2116],{"class":94},[84,2996,2997],{"class":90}," (w1, w2):\n",[84,2999,3000],{"class":86,"line":1017},[84,3001,3002],{"class":90},"        p.zero_grad()\n",[84,3004,3005],{"class":86,"line":1030},[84,3006,853],{"emptyLinePlaceholder":852},[84,3008,3009,3012,3014,3017,3019,3022,3024],{"class":86,"line":1049},[84,3010,3011],{"class":90},"    logits ",[84,3013,95],{"class":94},[84,3015,3016],{"class":90}," (x ",[84,3018,101],{"class":94},[84,3020,3021],{"class":90}," w1).relu() ",[84,3023,101],{"class":94},[84,3025,109],{"class":90},[84,3027,3028,3031,3033,3036,3038,3040,3042],{"class":86,"line":1054},[84,3029,3030],{"class":90},"    probs ",[84,3032,95],{"class":94},[84,3034,3035],{"class":90}," logits.softmax(",[84,3037,1514],{"class":966},[84,3039,95],{"class":94},[84,3041,1488],{"class":697},[84,3043,1530],{"class":90},[84,3045,3046,3049,3051],{"class":86,"line":1065},[84,3047,3048],{"class":90},"    loss ",[84,3050,95],{"class":94},[84,3052,3053],{"class":90}," probs.cross_entropy(y)\n",[84,3055,3056],{"class":86,"line":1119},[84,3057,3058],{"class":90},"    loss.backward()\n",[84,3060,3061],{"class":86,"line":1124},[84,3062,853],{"emptyLinePlaceholder":852},[84,3064,3065,3068,3070],{"class":86,"line":1135},[84,3066,3067],{"class":90},"    lr ",[84,3069,95],{"class":94},[84,3071,3072],{"class":697}," 0.5\n",[84,3074,3075,3078,3081,3084,3086],{"class":86,"line":1146},[84,3076,3077],{"class":90},"    w1.data ",[84,3079,3080],{"class":94},"-=",[84,3082,3083],{"class":90}," lr ",[84,3085,1429],{"class":94},[84,3087,3088],{"class":90}," w1.grad\n",[84,3090,3091,3094,3096,3098,3100],{"class":86,"line":1157},[84,3092,3093],{"class":90},"    w2.data ",[84,3095,3080],{"class":94},[84,3097,3083],{"class":90},[84,3099,1429],{"class":94},[84,3101,3102],{"class":90}," w2.grad\n",[84,3104,3105],{"class":86,"line":1170},[84,3106,853],{"emptyLinePlaceholder":852},[84,3108,3109,3111,3113,3116,3119,3122,3124],{"class":86,"line":1188},[84,3110,874],{"class":94},[84,3112,2972],{"class":90},[84,3114,3115],{"class":94},"%",[84,3117,3118],{"class":697}," 10",[84,3120,3121],{"class":94}," ==",[84,3123,1442],{"class":697},[84,3125,317],{"class":90},[84,3127,3128,3131],{"class":86,"line":1204},[84,3129,3130],{"class":697},"        print",[84,3132,3133],{"class":90},"(step, loss.data)\n",[10,3135,3136],{},"完整过程是：",[60,3138,3141],{"className":3139,"code":3140,"language":65,"meta":66},[63],"x @ w1\n  -> relu\n  -> @ w2\n  -> softmax\n  -> cross_entropy\n  -> backward\n  -> w1.grad \u002F w2.grad\n  -> SGD update\n",[19,3142,3140],{"__ignoreMap":66},[10,3144,3145],{},"这个例子虽然小，但已经包含了深度学习训练最重要的机制。",[52,3147,3149],{"id":3148},"梯度流为什么会消失或爆炸","梯度流：为什么会消失或爆炸",[10,3151,3152],{},"梯度流指梯度从 loss 往前层传播的过程。每经过一个操作，梯度都会乘上局部导数。",[10,3154,3155],{},"如果很多局部导数小于 1，梯度会越来越小，形成梯度消失；如果很多局部导数大于 1，梯度会越来越大，形成梯度爆炸。",[60,3157,3160],{"className":3158,"code":3159,"language":65,"meta":66},[63],"dL\u002Fdx = dL\u002Fdh_n * dh_n\u002Fdh_{n-1} * ... * dh_1\u002Fdx\n",[19,3161,3159],{"__ignoreMap":66},[10,3163,3164],{},"这解释了很多网络设计：",[131,3166,3167,3170,3173,3176,3179],{},[134,3168,3169],{},"ReLU 比 sigmoid 更常用，因为正区间导数是 1，更利于梯度通过；",[134,3171,3172],{},"ResNet 用残差连接，让梯度可以沿 identity path 直接传播；",[134,3174,3175],{},"LayerNorm \u002F BatchNorm 缓解激活分布漂移；",[134,3177,3178],{},"合理初始化让前向激活和反向梯度保持稳定尺度；",[134,3180,3181],{},"梯度裁剪可以防止 RNN \u002F Transformer 中的梯度爆炸。",[10,3183,3184],{},"理解梯度流之后，很多训练技巧不再是经验魔法，而是为了让链式法则的乘积更稳定。",[52,3186,3188],{"id":3187},"memory-optimization显存到底花在哪里","Memory Optimization：显存到底花在哪里",[10,3190,3191],{},"训练时显存主要来自：",[239,3193,3194,3200,3206,3212,3218],{},[134,3195,3196,3199],{},[14,3197,3198],{},"参数","：模型权重；",[134,3201,3202,3205],{},[14,3203,3204],{},"梯度","：每个参数对应一份梯度；",[134,3207,3208,3211],{},[14,3209,3210],{},"优化器状态","：Adam 会保存一阶、二阶动量，通常是参数量的 2 倍；",[134,3213,3214,3217],{},[14,3215,3216],{},"激活值","：前向中间结果，反向需要用；",[134,3219,3220,3223],{},[14,3221,3222],{},"临时 buffer","：算子执行过程中的 workspace。",[10,3225,3226],{},"很多人以为显存主要被参数占用，但在大 batch、长序列、深网络里，激活值经常非常可观。因为反向传播需要前向时的中间结果。例如 ReLU backward 要知道前向输入是否大于 0，matmul backward 要知道另一个输入矩阵。",[52,3228,3230],{"id":3229},"activation-checkpoint用计算换显存","Activation Checkpoint：用计算换显存",[10,3232,3233],{},[47,3234],{"alt":3235,"src":3236},"Activation checkpoint","\u002Fimages\u002Fposts\u002Fmini-autograd\u002Fcheckpoint.svg",[10,3238,3239,3240,260],{},"Activation checkpoint 的核心思想：",[14,3241,3242],{},"不要保存所有中间激活，只保存少量 checkpoint；反向传播时，把缺失的中间激活重新算一遍",[10,3244,3245],{},"普通训练：",[60,3247,3250],{"className":3248,"code":3249,"language":65,"meta":66},[63],"forward: 保存 a1, a2, a3, a4\nbackward: 直接使用 a1, a2, a3, a4\n",[19,3251,3249],{"__ignoreMap":66},[10,3253,3254],{},"Checkpoint：",[60,3256,3259],{"className":3257,"code":3258,"language":65,"meta":66},[63],"forward: 只保存 a0, a4\nbackward: 从 a0 重新计算 a1, a2, a3，再做局部 backward\n",[19,3260,3258],{"__ignoreMap":66},[10,3262,3263],{},"代价很清楚：",[500,3265,3266,3279],{},[503,3267,3268],{},[506,3269,3270,3273,3276],{},[509,3271,3272],{},"方案",[509,3274,3275],{},"显存",[509,3277,3278],{},"计算",[519,3280,3281,3292],{},[506,3282,3283,3286,3289],{},[524,3284,3285],{},"保存所有激活",[524,3287,3288],{},"高",[524,3290,3291],{},"低",[506,3293,3294,3296,3298],{},[524,3295,3235],{},[524,3297,3291],{},[524,3299,3288],{},[10,3301,3302],{},"这就是 memory vs compute tradeoff。训练大模型时，如果显存是瓶颈，宁愿多算一点，也要把 batch size、sequence length 或模型规模撑起来。",[10,3304,3305],{},"PyTorch 中常用：",[60,3307,3309],{"className":78,"code":3308,"language":80,"meta":66,"style":66},"from torch.utils.checkpoint import checkpoint\n\nout = checkpoint(block, x)\n",[19,3310,3311,3324,3328],{"__ignoreMap":66},[84,3312,3313,3316,3319,3321],{"class":86,"line":87},[84,3314,3315],{"class":94},"from",[84,3317,3318],{"class":90}," torch.utils.checkpoint ",[84,3320,838],{"class":94},[84,3322,3323],{"class":90}," checkpoint\n",[84,3325,3326],{"class":86,"line":112},[84,3327,853],{"emptyLinePlaceholder":852},[84,3329,3330,3333,3335],{"class":86,"line":123},[84,3331,3332],{"class":90},"out ",[84,3334,95],{"class":94},[84,3336,3337],{"class":90}," checkpoint(block, x)\n",[10,3339,3340],{},"使用 checkpoint 时要注意：",[131,3342,3343,3346,3349,3352],{},[134,3344,3345],{},"被 checkpoint 的函数最好是纯函数，不要依赖会变化的外部状态；",[134,3347,3348],{},"dropout 等随机操作需要正确处理 RNG 状态；",[134,3350,3351],{},"不是所有层都值得 checkpoint，通常选显存占用大的 block；",[134,3353,3354],{},"checkpoint 会增加训练时间，不是免费优化。",[52,3356,3358],{"id":3357},"pytorch-和-tensorflow-本质上做了什么","PyTorch 和 TensorFlow 本质上做了什么",[10,3360,3361],{},"当你写 PyTorch：",[60,3363,3365],{"className":78,"code":3364,"language":80,"meta":66,"style":66},"loss = model(x).softmax(dim=-1).log().mean()\nloss.backward()\n",[19,3366,3367,3386],{"__ignoreMap":66},[84,3368,3369,3371,3373,3376,3379,3381,3383],{"class":86,"line":87},[84,3370,115],{"class":90},[84,3372,95],{"class":94},[84,3374,3375],{"class":90}," model(x).softmax(",[84,3377,3378],{"class":966},"dim",[84,3380,1485],{"class":94},[84,3382,1488],{"class":697},[84,3384,3385],{"class":90},").log().mean()\n",[84,3387,3388],{"class":86,"line":112},[84,3389,126],{"class":90},[10,3391,3392],{},"框架做的事情和 mini autograd 没有本质区别，只是工程复杂度高很多：",[131,3394,3395,3398,3401,3404,3407,3410,3413],{},[134,3396,3397],{},"Tensor 支持 CPU \u002F GPU \u002F 多种 dtype；",[134,3399,3400],{},"算子调用高性能 kernel，例如 cuBLAS、cuDNN、Triton；",[134,3402,3403],{},"Autograd engine 处理拓扑调度、并行反传、线程安全；",[134,3405,3406],{},"View \u002F inplace \u002F broadcasting 有复杂的梯度语义；",[134,3408,3409],{},"编译器会做图捕获、融合、常量折叠、内存规划；",[134,3411,3412],{},"分布式训练会插入通信算子，例如 all-reduce；",[134,3414,3415],{},"混合精度会处理 loss scaling 和 fp16\u002Fbf16 数值稳定。",[10,3417,3418],{},"但心智模型仍然是：",[60,3420,3423],{"className":3421,"code":3422,"language":65,"meta":66},[63],"Tensor + Op + Graph + Chain Rule + Scheduler + Memory Planner\n",[19,3424,3422],{"__ignoreMap":66},[52,3426,3428],{"id":3427},"mini-autograd-的限制","Mini Autograd 的限制",[10,3430,3431],{},"这个 mini 版本故意省略了很多真实框架能力：",[131,3433,3434,3437,3440,3443,3446,3449,3452],{},[134,3435,3436],{},"不支持 broadcasting 的梯度反推；",[134,3438,3439],{},"不支持 view、reshape、transpose 的复杂共享内存语义；",[134,3441,3442],{},"不支持 inplace 修改检测；",[134,3444,3445],{},"不支持 GPU；",[134,3447,3448],{},"不支持高阶梯度；",[134,3450,3451],{},"不支持动态图释放和显存复用；",[134,3453,3454,22,3456,3458],{},[19,3455,39],{},[19,3457,42],{}," 分开实现，数值稳定性不如融合版。",[10,3460,3461],{},"但它已经完整覆盖了深度学习框架最核心的训练闭环。",[52,3463,3465],{"id":3464},"week-1-学完应该掌握什么","Week 1 学完应该掌握什么",[10,3467,3468],{},"学完这一周，不要求背公式，而是要能解释清楚这些问题：",[239,3470,3471,3474,3479,3482,3493,3496,3499,3502],{},[134,3472,3473],{},"前向计算时，框架为什么要记录计算图？",[134,3475,3476,3478],{},[19,3477,21],{}," 为什么要按拓扑逆序执行？",[134,3480,3481],{},"多条路径指向同一个 Tensor 时，梯度为什么要累加？",[134,3483,3484,33,3486,33,3488,33,3490,3492],{},[19,3485,32],{},[19,3487,36],{},[19,3489,39],{},[19,3491,42],{}," 的反向公式是什么？",[134,3494,3495],{},"为什么深度学习训练主要用 reverse-mode AD？",[134,3497,3498],{},"为什么训练比推理更耗显存？",[134,3500,3501],{},"Activation checkpoint 为什么能省显存，代价是什么？",[134,3503,3504],{},"PyTorch 的动态图和 TensorFlow 的静态图各有什么取舍？",[10,3506,3507],{},"如果这些问题都能讲顺，基本就理解了深度学习框架的本质。",[52,3509,3510],{"id":3510},"最后总结",[10,3512,3513],{},"Autograd 的核心非常朴素：前向时记录图，反向时套链式法则。复杂的是工程化：如何让它在 GPU 上快、在大模型上省显存、在动态图里易调试、在编译图里可优化。",[10,3515,3516,3517,3520,3521,22,3523,3525],{},"Mini autograd 的意义不是替代 PyTorch，而是帮我们建立底层直觉。只要理解了 ",[19,3518,3519],{},"Tensor"," 如何保存 ",[19,3522,399],{},[19,3524,2572],{},"，理解了拓扑逆序和梯度累加，再看 PyTorch、TensorFlow、JAX 的设计，就不会只停留在 API 层，而能真正理解它们为什么这样工作。",[10,3527,3528,3529,3533],{},"补充：完整可运行代码已放在 ",[3530,3531,3532],"a",{"href":3532},"\u002Fdownloads\u002Fcode\u002Fmini_autograd.py","，可以直接下载运行。",[3535,3536,3537],"style",{},"html pre.shiki code .ssh_m, html code.shiki .ssh_m{--shiki-default:#ADBAC7;--shiki-light:#24292E}html pre.shiki code .s6PUj, html code.shiki .s6PUj{--shiki-default:#F47067;--shiki-light:#D73A49}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html pre.shiki code .sqRhv, html code.shiki .sqRhv{--shiki-default:#F69D50;--shiki-light:#6F42C1}html pre.shiki code .sgHix, html code.shiki .sgHix{--shiki-default:#768390;--shiki-light:#6A737D}html pre.shiki code .swcJU, html code.shiki .swcJU{--shiki-default:#6CB6FF;--shiki-light:#005CC5}html pre.shiki code .saVmf, html code.shiki .saVmf{--shiki-default:#DCBDFB;--shiki-light:#6F42C1}html pre.shiki code .sXfbr, html code.shiki .sXfbr{--shiki-default:#96D0FF;--shiki-light:#032F62}html pre.shiki code .sNjOc, html code.shiki .sNjOc{--shiki-default:#F69D50;--shiki-light:#E36209}html pre.shiki code .sxsTv, html code.shiki .sxsTv{--shiki-default:#F47067;--shiki-light:#005CC5}",{"title":66,"searchDepth":112,"depth":123,"links":3539},[3540,3541,3542,3543,3544,3545,3551,3552,3563,3564,3565,3566,3567,3568,3569,3570],{"id":54,"depth":112,"text":55},{"id":179,"depth":112,"text":180},{"id":296,"depth":112,"text":297},{"id":438,"depth":112,"text":439},{"id":494,"depth":112,"text":495},{"id":584,"depth":112,"text":584,"children":3546},[3547,3548,3549,3550],{"id":32,"depth":123,"text":591},{"id":36,"depth":123,"text":636},{"id":39,"depth":123,"text":666},{"id":759,"depth":123,"text":760},{"id":824,"depth":112,"text":825},{"id":2315,"depth":112,"text":2315,"children":3553},[3554,3555,3556,3557,3558,3559,3560,3561,3562],{"id":2318,"depth":123,"text":2321},{"id":2344,"depth":123,"text":2347},{"id":2443,"depth":123,"text":2446},{"id":2485,"depth":123,"text":2488},{"id":2509,"depth":123,"text":2512},{"id":2564,"depth":123,"text":2567},{"id":2656,"depth":123,"text":2656},{"id":74,"depth":123,"text":74},{"id":2738,"depth":123,"text":2738},{"id":2770,"depth":112,"text":2770},{"id":3148,"depth":112,"text":3149},{"id":3187,"depth":112,"text":3188},{"id":3229,"depth":112,"text":3230},{"id":3357,"depth":112,"text":3358},{"id":3427,"depth":112,"text":3428},{"id":3464,"depth":112,"text":3465},{"id":3510,"depth":112,"text":3510},[3572],"技术","2026-05-05 09:00:00","如果只用一句话概括 PyTorch \u002F TensorFlow 的本质：它们是在张量计算之上，自动构建计算图，并用链式法则自动求梯度的系统。训练神经网络看起来是调用 loss.backward() 和 optimizer.step()，但底层真正发生的是：前向阶段记录依赖关系，反向阶段沿图逆序传播梯度，同时在显存、计算量和调度开销之间做工程权衡。",false,"md",{},"\u002Fposts\u002Fdl-framework-autograd-mini",{"title":5,"description":3574},"posts\u002Fdl-framework-autograd-mini",[3582,3583,3584,3585,3586],"深度学习","Autograd","PyTorch","CMU 10-414","Mini Framework","VJjkv-67X-S0bc20yZyOWyPyNlD53n-IWjc6ggV5RlA",[3589,3602,3614,3620,3632,3641,3650,3660,3670,3679,3688,3698,3710,3722,3730,3739,3751,3762,3771,3774,3785,3791,3797,3803,3811,3820,3828,3834,3842,3850],{"slug":3590,"path":3591,"title":3592,"date":3593,"tags":3594,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":904},"multimodal-rag-from-scratch","\u002Fposts\u002Fmultimodal-rag-from-scratch","从零实现多模态 RAG：BM25、Dense 检索、RRF 融合、MMR 重排全部手写","2026-06-30 18:00:00",[3595,3596,3597,3598,3599,3600,3601],"RAG","多模态","AI Infra","BM25","向量检索","混合检索","实习求职",{"slug":3603,"path":3604,"title":3605,"date":3606,"tags":3607,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":918},"mini-llm-engine-deep-dive","\u002Fposts\u002Fmini-llm-engine-deep-dive","讲透 mini-llm-engine：从显存碎片到六大推理优化","2026-06-30 14:00:00",[3608,3597,3609,3610,3611,3612,3613,3601],"LLM","vLLM","PagedAttention","KV Cache","推理优化","投机解码",{"slug":3615,"path":3616,"title":3617,"date":3618,"tags":3619,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":909},"mini-llm-engine-from-scratch","\u002Fposts\u002Fmini-llm-engine-from-scratch","从零实现 LLM 推理引擎：深挖 vLLM 的六大核心优化","2026-06-30 10:00:00",[3608,3597,3609,3610,3612,3613,3601],{"slug":3621,"path":3622,"title":3623,"date":3624,"tags":3625,"description":3631,"draft":3575,"hidden":3575,"published":852,"readingTime":1049},"bytedance-recommendation-architecture-intern-interview","\u002Fposts\u002Fbytedance-recommendation-architecture-intern-interview","字节推荐架构实习生 Data 面试准备：推荐系统、实时特征与高并发八股","2026-06-09",[3626,3627,3628,3629,3630],"面试","推荐系统","后端架构","实时计算","字节跳动","面向字节跳动推荐架构团队实习岗位的八股准备清单，覆盖推荐系统链路、实时数据、特征服务、高并发后端、分布式系统与项目包装。",{"slug":3633,"path":3634,"title":3635,"date":3624,"tags":3636,"description":3640,"draft":3575,"hidden":3575,"published":852,"readingTime":952},"high-concurrency-rate-limiting-algorithms","\u002Fposts\u002Fhigh-concurrency-rate-limiting-algorithms","高并发限流算法：固定窗口、滑动窗口、漏桶与令牌桶",[3637,3638,3628,3639,3626],"高并发","限流","系统设计","面试和工程都常见的限流算法总结，讲清楚固定窗口、滑动窗口、漏桶、令牌桶、并发数限制以及分布式限流如何落地。",{"slug":3642,"path":3643,"title":3644,"date":3624,"tags":3645,"description":3649,"draft":3575,"hidden":3575,"published":852,"readingTime":899},"kafka-producer-broker-consumer","\u002Fposts\u002Fkafka-producer-broker-consumer","Kafka 入门：生产者、Broker、消费者和“消费”到底是什么意思",[3646,3647,3648,3626,3627],"Kafka","消息队列","后端","用推荐系统里的用户行为日志为例，讲清楚 Kafka 的作用、Producer、Broker、Consumer、Topic、Partition、Offset 和消费语义。",{"slug":3651,"path":3652,"title":3653,"date":3624,"tags":3654,"description":3659,"draft":3575,"hidden":3575,"published":852,"readingTime":355},"leetcode-lru-merge-k-reverse-list","\u002Fposts\u002Fleetcode-lru-merge-k-reverse-list","链表与缓存高频题：LRU Cache、合并 K 个有序链表、反转链表",[3655,3656,3657,3658,3626],"算法","链表","LRU","LeetCode","面试高频算法题速记，整理 LRU Cache、合并 K 个有序链表、反转链表的核心思路、复杂度和 C++ 代码。",{"slug":3661,"path":3662,"title":3663,"date":3664,"tags":3665,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":1017},"meddococr-interpreter-source-analysis","\u002Fposts\u002Fmeddococr-interpreter-source-analysis","MedDocOCR-Interpreter 源码导读：医疗文档 OCR、结构化抽取与报告解读原型","2026-05-22 10:30:00",[3666,3596,3667,3595,3668,3669],"OCR","医疗 AI","Python","源码分析",{"slug":3671,"path":3672,"title":3673,"date":3674,"tags":3675,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":909},"cet6-writing-model-essays","\u002Fposts\u002Fcet6-writing-model-essays","六级写作范文背诵包：10 个高频话题","2026-05-20 09:00:00",[3676,3677,3678],"English","CET6","Writing",{"slug":3680,"path":3681,"title":3682,"date":3683,"tags":3684,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":1188},"claude-code-context-management","\u002Fposts\u002Fclaude-code-context-management","Claude Code 上下文管理机制：从 Microcompact 到 Auto Compact","2026-05-19 10:00:00",[3685,3686,3608,3597,3687],"Claude Code","Agent","上下文工程",{"slug":3689,"path":3690,"title":3691,"date":3692,"tags":3693,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":1054},"nanotron-llm-pretraining-framework-analysis","\u002Fposts\u002Fnanotron-llm-pretraining-framework-analysis","Nanotron 项目详解：Hugging Face 的大模型预训练框架怎么做分布式训练","2026-05-10 12:10:00",[3608,3694,3695,3696,3697,3597,3584],"大模型训练","分布式训练","Nanotron","Hugging Face",{"slug":3699,"path":3700,"title":3701,"date":3702,"tags":3703,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":1157},"gofoundry-go-backend-foundation-framework","\u002Fposts\u002Fgofoundry-go-backend-foundation-framework","GoFoundry 项目详解：基于 Go 的后端基础框架套件设计","2026-05-10 11:20:00",[3704,3705,3706,3707,3708,3647,3709],"Go","后端框架","ORM","分布式缓存","分布式锁","项目架构",{"slug":3711,"path":3712,"title":3713,"date":3714,"tags":3715,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":1188},"cloudvault-go-cloud-storage-system","\u002Fposts\u002Fcloudvault-go-cloud-storage-system","CloudVault 项目详解：基于 Go 的云端存储与网盘系统架构设计","2026-05-10 10:30:00",[3704,3716,3717,3718,3709,3719,3720,3721],"云存储","网盘系统","分布式系统","Redis","RabbitMQ","Elasticsearch",{"slug":3723,"path":3724,"title":3725,"date":3726,"tags":3727,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":1065},"openclaw-source-code-analysis","\u002Fposts\u002Fopenclaw-source-code-analysis","OpenClaw 源码导读：个人 AI 助手的网关、通道、插件与运行时架构","2026-05-08 16:30:00",[3728,3686,3597,3729,3669],"OpenClaw","TypeScript",{"slug":3731,"path":3732,"title":3733,"date":3734,"tags":3735,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":909},"flow-matching-generative-models","\u002Fposts\u002Fflow-matching-generative-models","Flow Matching：从噪声到数据的连续流生成模型","2026-05-07 00:00:00",[3736,3737,3738,3582],"生成模型","Diffusion","Flow Matching",{"slug":3740,"path":3741,"title":3742,"date":3743,"tags":3744,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":1001},"database-ai-week4","\u002Fposts\u002Fdatabase-ai-week4","Week 4：数据库速成——从 Storage、Index、Query Optimization 到 Vector DB 与 RAG","2026-05-05 12:00:00",[3745,3746,3747,3595,3748,3749,3750],"数据库","CMU 15-445","Vector DB","LLM Memory","Query Optimization","Caching",{"slug":3752,"path":3753,"title":3754,"date":3755,"tags":3756,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":1017},"distributed-systems-week3","\u002Fposts\u002Fdistributed-systems-week3","Week 3：分布式系统速成——MapReduce、Raft、容错与 Distributed KV Store","2026-05-05 11:00:00",[3718,3757,3758,3759,3760,3761,3686],"MIT 6.824","MapReduce","Raft","KV Store","Ray",{"slug":3763,"path":3764,"title":3765,"date":3766,"tags":3767,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":1001},"gpu-inference-acceleration-week2","\u002Fposts\u002Fgpu-inference-acceleration-week2","Week 2：GPU 与推理加速——从 Kernel、算子融合到 LLM Serving","2026-05-05 10:00:00",[3582,3768,3769,3608,3585,3609,3770],"GPU","推理加速","TensorRT",{"slug":3772,"path":3578,"title":5,"date":3573,"tags":3773,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":1049},"dl-framework-autograd-mini",[3582,3583,3584,3585,3586],{"slug":3775,"path":3776,"title":3777,"date":3778,"tags":3779,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":952},"lock-free-concurrency-notes","\u002Fposts\u002Flock-free-concurrency-notes","无锁并发入门：从 CAS 到 Atomic Ring Buffer","2026-04-25",[3780,3781,3782,3783,3784],"C++","并发","无锁编程","性能优化","量化开发",{"slug":3786,"path":3787,"title":3788,"date":3789,"tags":3790,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":346},"agent-memory","\u002Fposts\u002Fagent-memory","Agent 对话记忆化：从原理到实现","2026-04-24",[3608,3686,3595,3626],{"slug":3792,"path":3793,"title":3794,"date":3789,"tags":3795,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":355},"llm-context-compression","\u002Fposts\u002Fllm-context-compression","LLM 上下文五层压缩机制详解",[3608,3686,3796,3626],"上下文压缩",{"slug":3798,"path":3799,"title":3800,"date":3801,"tags":3802,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":904},"cpp-concurrency-basics","\u002Fposts\u002Fcpp-concurrency-basics","C++ 并发编程入门：从数据竞争到线程池","2026-04-15",[3780,3781,3626,3784],{"slug":3804,"path":3805,"title":3806,"date":3807,"tags":3808,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":87},"travel-in-shenzhen","\u002Fposts\u002Ftravel-in-shenzhen","XCPC 深圳游记","2026-04-13",[3809,3780,3810],"XCPC","比赛",{"slug":3812,"path":3813,"title":3814,"date":3815,"tags":3816,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":364},"backend-stack-deep-dive","\u002Fposts\u002Fbackend-stack-deep-dive","后端五件套：FastAPI \u002F Node.js \u002F SQLAlchemy async \u002F PostgreSQL \u002F Docker 面试速通","2026-04-07",[3648,3817,3818,3819,3626],"FastAPI","PostgreSQL","Docker",{"slug":3821,"path":3822,"title":3823,"date":3824,"tags":3825,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":355},"deepscientist-tech-stack","\u002Fposts\u002Fdeepscientist-tech-stack","DeepScientist 技术栈全解析：一个 AI 科研平台的架构设计","2026-04-06",[3826,3817,3827,3818,3626],"全栈","Next.js",{"slug":3829,"path":3830,"title":3831,"date":3824,"tags":3832,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":899},"minicode-source-analysis","\u002Fposts\u002Fminicode-source-analysis","MiniCode 源码解析：用 5000 行 TypeScript 实现一个 AI 编程助手",[3729,3833,3608,3669,3626],"CLI",{"slug":3835,"path":3836,"title":3837,"date":3824,"tags":3838,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":346},"nova-theme-implementation","\u002Fposts\u002Fnova-theme-implementation","我是怎么从零实现 Nova 主题的",[3839,3840,3841],"Hexo","前端","开源",{"slug":3843,"path":3844,"title":3845,"date":3846,"tags":3847,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":112},"git-cheatsheet","\u002Fposts\u002Fgit-cheatsheet","Git 常用操作备忘","2026-04-05 14:00:00",[3848,3849],"Git","工具",{"slug":3851,"path":3852,"title":3853,"date":3854,"tags":3855,"description":66,"draft":3575,"hidden":3575,"published":852,"readingTime":123},"github-actions-intro","\u002Fposts\u002Fgithub-actions-intro","GitHub Actions 入门：自动化你的工作流","2026-04-04 09:00:00",[3856,3857,3858],"GitHub Actions","CI\u002FCD","自动化",1782796012101]