[{"data":1,"prerenderedAt":2635},["ShallowReactive",2],{"post-\u002Fposts\u002Fcuda-fused-rmsnorm-kernel":3,"archive-public-posts":2125},{"id":4,"title":5,"body":6,"categories":2104,"cover":2106,"coverAlt":2107,"date":2108,"description":2109,"draft":2110,"extension":2111,"haloPublished":2112,"haloSync":2112,"hidden":2110,"legacy":2110,"meta":2113,"navigation":2112,"path":2114,"published":2110,"readingTime":2115,"seo":2116,"series":2117,"seriesOrder":593,"stem":2118,"tags":2119,"__hash__":2124},"posts\u002Fposts\u002Fcuda-fused-rmsnorm-kernel.md","从 PyTorch 到 CUDA Kernel：融合 RMSNorm 在 RTX 4090D 上加速 4.27 倍",{"type":7,"value":8,"toc":2079},"minimark",[9,13,16,48,68,88,91,98,104,109,112,117,120,128,135,139,142,221,224,230,237,241,244,247,253,256,262,265,268,304,307,311,314,317,323,349,352,356,359,434,437,441,444,450,457,463,467,470,473,476,480,518,521,525,528,1225,1288,1291,1305,1308,1312,1315,1341,1348,1351,1357,1363,1367,1374,1380,1383,1387,1390,1463,1466,1470,1473,1516,1519,1529,1533,1536,1539,1566,1569,1576,1583,1587,1590,1613,1620,1624,1627,1630,1667,1670,1673,1700,1704,1717,1880,1886,1893,1896,1908,1912,1915,1939,1945,2008,2011,2018,2022,2025,2042,2045,2049,2052,2072,2075],[10,11,12],"p",{},"为了准备 AI Infra 岗位，我做了一个尽量小而完整的 CUDA 项目：不追求堆很多算子，而是选一个真实的 LLM 推理路径，把实现、正确性、benchmark 和性能解释全部做扎实。",[10,14,15],{},"项目实现了两个 PyTorch CUDA 算子：",[17,18,23],"pre",{"className":19,"code":20,"language":21,"meta":22,"style":22},"language-python shiki shiki-themes github-dark-dimmed github-light houston","rms_norm(x, weight)\nfused_add_rms_norm(x, residual, weight)\n","python","",[24,25,26,39],"code",{"__ignoreMap":22},[27,28,31,35],"span",{"class":29,"line":30},"line",1,[27,32,34],{"class":33},"sq3c-","rms_norm",[27,36,38],{"class":37},"sWURj","(x, weight)\n",[27,40,42,45],{"class":29,"line":41},2,[27,43,44],{"class":33},"fused_add_rms_norm",[27,46,47],{"class":37},"(x, residual, weight)\n",[10,49,50,51,54,55,58,59,63,64,67],{},"最终在 RTX 4090D 的 ",[24,52,53],{},"256 × 4096 FP16"," workload 上，融合算子相对 ",[24,56,57],{},"torch.compile(..., fullgraph=True)"," 获得 ",[60,61,62],"strong",{},"4.27 倍 hot-cache 加速","和 ",[60,65,66],{},"3.11 倍 cold-cache 加速","。代码、测试和原始 CSV 都已公开：",[69,70,71,81],"ul",{},[72,73,74],"li",{},[75,76,80],"a",{"href":77,"rel":78},"https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Ffast-llm-kernels",[79],"nofollow","GitHub：fast-llm-kernels",[72,82,83],{},[75,84,87],{"href":85,"rel":86},"https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Ffast-llm-kernels\u002Ftree\u002Fmain\u002Fresults",[79],"完整 benchmark 结果",[10,89,90],{},"这篇文章不只展示结果，更重要的是解释这些数字为什么成立，以及它们不能说明什么。",[10,92,93],{},[94,95],"img",{"alt":96,"src":97},"Fused RMSNorm kernel data flow","\u002Fimages\u002Fposts\u002Fcuda-fused-rmsnorm\u002Fkernel-pipeline.svg",[10,99,100],{},[101,102,103],"em",{},"图：本文原创绘制。一个 CUDA block 负责一个 token row，先完成 residual add 与平方和归约，再进行归一化和缩放。",[105,106,108],"h2",{"id":107},"_0-阅读前的前置知识","0. 阅读前的前置知识",[10,110,111],{},"如果已经熟悉混合精度和 CUDA 编程，可以直接跳到下一节。第一次接触 GPU kernel，则建议先建立下面这些概念之间的关系。",[113,114,116],"h3",{"id":115},"_01-tokenhidden-state-和-hidden-size","0.1 Token、hidden state 和 hidden size",[10,118,119],{},"模型不会直接处理文字，而是先把文字切分成 token。每个 token 在 Transformer 内部对应一个浮点数向量，称为 hidden state：",[17,121,126],{"className":122,"code":124,"language":125,"meta":22},[123],"language-text","一个 token -> [0.12, -0.53, 0.87, ...]\n","text",[24,127,124],{"__ignoreMap":22},[10,129,130,131,134],{},"向量中元素的数量就是 hidden size。例如输入 shape 为 ",[24,132,133],{},"[256, 4096]","，表示有 256 个 token，每个 token 用 4096 个浮点数表示。RMSNorm 会分别处理这 256 行，每一行之间没有数据依赖。",[113,136,138],{"id":137},"_02-fp32fp16-和-bf16","0.2 FP32、FP16 和 BF16",[10,140,141],{},"FP 表示 floating point，即浮点数。后面的数字表示一个数占用的 bit 数：",[143,144,145,168],"table",{},[146,147,148],"thead",{},[149,150,151,155,159,162,165],"tr",{},[152,153,154],"th",{},"格式",[152,156,158],{"align":157},"right","每个元素",[152,160,161],{"align":157},"指数位",[152,163,164],{"align":157},"尾数位",[152,166,167],{},"主要特点",[169,170,171,189,206],"tbody",{},[149,172,173,177,180,183,186],{},[174,175,176],"td",{},"FP32",[174,178,179],{"align":157},"4B",[174,181,182],{"align":157},"8",[174,184,185],{"align":157},"23",[174,187,188],{},"精度和范围高，但显存与带宽开销大",[149,190,191,194,197,200,203],{},[174,192,193],{},"FP16",[174,195,196],{"align":157},"2B",[174,198,199],{"align":157},"5",[174,201,202],{"align":157},"10",[174,204,205],{},"精度较高，表示范围比 FP32 小",[149,207,208,211,213,215,218],{},[174,209,210],{},"BF16",[174,212,196],{"align":157},[174,214,182],{"align":157},[174,216,217],{"align":157},"7",[174,219,220],{},"范围接近 FP32，小数精度低于 FP16",[10,222,223],{},"大模型推理通常使用 FP16 或 BF16，以减少显存占用和数据搬运。但 RMSNorm 需要累加几千个平方值，直接用低精度累加容易产生明显误差。因此本项目采用 mixed precision：",[17,225,228],{"className":226,"code":227,"language":125,"meta":22},[123],"FP16\u002FBF16 输入\n    -> 转成 FP32 计算平方和\n    -> 得到归一化系数\n    -> 转回 FP16\u002FBF16 输出\n",[24,229,227],{"__ignoreMap":22},[10,231,232,233,236],{},"这就是文中 ",[60,234,235],{},"FP32 accumulation"," 的含义。它不一定表示输入输出都是 FP32，而是指关键的 reduction 使用 FP32。",[113,238,240],{"id":239},"_03-rmsnormlayernorm-和-residual","0.3 RMSNorm、LayerNorm 和 residual",[10,242,243],{},"Normalization 的目标是控制 hidden state 的数值尺度，避免深层网络中的激活越来越大或越来越小。",[10,245,246],{},"LayerNorm 会先减去均值，再除以标准差：",[17,248,251],{"className":249,"code":250,"language":125,"meta":22},[123],"(x - mean(x)) \u002F sqrt(variance(x) + eps)\n",[24,252,250],{"__ignoreMap":22},[10,254,255],{},"RMSNorm 不计算和减去均值，只根据平方均值缩放：",[17,257,260],{"className":258,"code":259,"language":125,"meta":22},[123],"x \u002F sqrt(mean(x²) + eps)\n",[24,261,259],{"__ignoreMap":22},[10,263,264],{},"因此 RMSNorm 的计算更简单，被 Llama、Qwen、Mistral 等模型广泛使用。",[10,266,267],{},"Residual 是残差连接。Transformer 不直接丢弃上一层的输入，而是把新结果加回原信息：",[17,269,271],{"className":19,"code":270,"language":21,"meta":22,"style":22},"residual_out = x + residual\noutput = rms_norm(residual_out, weight)\n",[24,272,273,291],{"__ignoreMap":22},[27,274,275,278,282,285,288],{"class":29,"line":30},[27,276,277],{"class":37},"residual_out ",[27,279,281],{"class":280},"sf2cM","=",[27,283,284],{"class":37}," x ",[27,286,287],{"class":280},"+",[27,289,290],{"class":37}," residual\n",[27,292,293,296,298,301],{"class":29,"line":41},[27,294,295],{"class":37},"output ",[27,297,281],{"class":280},[27,299,300],{"class":33}," rms_norm",[27,302,303],{"class":37},"(residual_out, weight)\n",[10,305,306],{},"这样既能保留旧信息，也有利于深层网络训练。本文优化的正是这条 residual-add + RMSNorm 路径。",[113,308,310],{"id":309},"_04-kernelthreadwarpblock-和-sm","0.4 Kernel、thread、warp、block 和 SM",[10,312,313],{},"CUDA kernel 是运行在 GPU 上的函数。CPU 负责发起 kernel launch，GPU 再让大量线程执行它。",[10,315,316],{},"NVIDIA GPU 的执行层级可以简化为：",[17,318,321],{"className":319,"code":320,"language":125,"meta":22},[123],"GPU\n  -> 多个 SM\n      -> 多个 block\n          -> 多个 warp\n              -> 32 个 thread\n",[24,322,320],{"__ignoreMap":22},[69,324,325,331,337,343],{},[72,326,327,330],{},[60,328,329],{},"Thread","：最小执行单元，每个线程处理一部分元素；",[72,332,333,336],{},[60,334,335],{},"Warp","：GPU 的基本调度单位，NVIDIA GPU 上通常包含 32 个线程；",[72,338,339,342],{},[60,340,341],{},"Block","：一组可以同步、共享 shared memory 的线程；",[72,344,345,348],{},[60,346,347],{},"SM","：真正执行 warp 和 block 的硬件单元。",[10,350,351],{},"本项目采用一个 block 处理一个 token row，每个 block 启动 256 个线程，也就是 8 个 warp。",[113,353,355],{"id":354},"_05-registershared-memoryl2-和显存","0.5 Register、shared memory、L2 和显存",[10,357,358],{},"GPU 存储层级的速度与容量差异很大：",[143,360,361,377],{},[146,362,363],{},[149,364,365,368,371,374],{},[152,366,367],{},"层级",[152,369,370],{},"速度",[152,372,373],{},"可见范围",[152,375,376],{},"本项目中的用途",[169,378,379,393,407,421],{},[149,380,381,384,387,390],{},[174,382,383],{},"Register",[174,385,386],{},"最快",[174,388,389],{},"单个线程",[174,391,392],{},"保存局部平方和",[149,394,395,398,401,404],{},[174,396,397],{},"Shared memory",[174,399,400],{},"很快",[174,402,403],{},"一个 block",[174,405,406],{},"保存 8 个 warp 的归约结果",[149,408,409,412,415,418],{},[174,410,411],{},"L2 cache",[174,413,414],{},"较快",[174,416,417],{},"整个 GPU",[174,419,420],{},"缓存重复读取的输入和 weight",[149,422,423,426,429,431],{},[174,424,425],{},"VRAM \u002F GDDR \u002F HBM",[174,427,428],{},"较慢、容量大",[174,430,417],{},[174,432,433],{},"存放 tensor 数据",[10,435,436],{},"GPU kernel 优化经常不是减少数学运算，而是减少数据在这些存储层级之间的搬运。",[113,438,440],{"id":439},"_06-reductionwarp-shuffle-和同步","0.6 Reduction、warp shuffle 和同步",[10,442,443],{},"Reduction 是把一组数据合并成一个值，例如：",[17,445,448],{"className":446,"code":447,"language":125,"meta":22},[123],"[1, 2, 3, 4] -> sum -> 10\n",[24,449,447],{"__ignoreMap":22},[10,451,452,453,456],{},"RMSNorm 需要将一行中所有 ",[24,454,455],{},"x[i]²"," 合并成平方和。每个线程先计算自己的局部结果，再通过 warp shuffle 在 warp 内交换 register 数据。8 个 warp 的结果写入 shared memory，最后合并成整个 block 的平方和。",[10,458,459,462],{},[24,460,461],{},"__syncthreads()"," 是 block 内同步屏障：所有线程到达这里后才能继续，避免某些线程还没写完 shared memory，其他线程就提前读取。",[113,464,466],{"id":465},"_07-memory-boundfusion-和向量化","0.7 Memory-bound、fusion 和向量化",[10,468,469],{},"如果算子主要受计算单元吞吐限制，称为 compute-bound；如果大量时间花在等待数据，称为 memory-bound。",[10,471,472],{},"RMSNorm 每读取一个元素只进行少量平方、加法和乘法，因此通常是 memory-bound。优化重点是减少显存访问和 kernel launch，而不是使用更多计算单元。",[10,474,475],{},"Kernel fusion 是把多个操作合进一次 kernel launch。Pack-4 向量化则让每个线程一次读取四个连续元素，减少访存指令。二者分别优化“启动与中间结果”和“单次访存效率”。",[113,477,479],{"id":478},"_08-latencyspeeduphot-cache-和-cold-cache","0.8 Latency、speedup、hot cache 和 cold cache",[69,481,482,488,498,504,510],{},[72,483,484,487],{},[60,485,486],{},"Latency","：一次算子执行需要多少时间，本文使用微秒（us）；",[72,489,490,493,494,497],{},[60,491,492],{},"Speedup","：baseline latency 除以 CUDA latency，例如 ",[24,495,496],{},"40 \u002F 10 = 4x","；",[72,499,500,503],{},[60,501,502],{},"Hot cache","：重复使用同一组输入，数据很可能已经位于 L2；",[72,505,506,509],{},[60,507,508],{},"Cold cache","：计时前主动驱逐 L2，让输入更多地从显存读取；",[72,511,512,517],{},[60,513,514],{},[24,515,516],{},"torch.compile","：PyTorch 的图编译功能，会尝试融合操作并生成优化 kernel，是比 eager PyTorch 更强的 baseline。",[10,519,520],{},"真实推理一般位于 hot 和 cold 两种极端之间，因此后文会同时报告两组数据，而不是只选择最好看的结果。",[105,522,524],{"id":523},"_1-为什么选择-rmsnorm","1. 为什么选择 RMSNorm",[10,526,527],{},"RMSNorm 广泛用于 Llama、Mistral、Qwen 等 decoder-only 模型。对一行 hidden state，它计算：",[27,529,532],{"className":530},[531],"katex-display",[27,533,536,652],{"className":534},[535],"katex",[27,537,540],{"className":538},[539],"katex-mathml",[541,542,545],"math",{"xmlns":543,"display":544},"http:\u002F\u002Fwww.w3.org\u002F1998\u002FMath\u002FMathML","block",[546,547,548,647],"semantics",{},[549,550,551,556,560,564,573,576,578,584,587,638,640],"mrow",{},[552,553,555],"mi",{"mathvariant":554},"normal","RMSNorm",[557,558,559],"mo",{},"⁡",[557,561,563],{"stretchy":562},"false","(",[565,566,567,570],"msub",{},[552,568,569],{},"x",[552,571,572],{},"i",[557,574,575],{"stretchy":562},")",[557,577,281],{},[565,579,580,582],{},[552,581,569],{},[552,583,572],{},[557,585,586],{},"⋅",[588,589,590,594],"mfrac",{},[591,592,593],"mn",{},"1",[595,596,597],"msqrt",{},[549,598,599,606,623,633,635],{},[588,600,601,603],{},[591,602,593],{},[552,604,605],{},"H",[607,608,609,612,621],"munderover",{},[557,610,611],{},"∑",[549,613,614,617,619],{},[552,615,616],{},"j",[557,618,281],{},[591,620,593],{},[552,622,605],{},[624,625,626,628,630],"msubsup",{},[552,627,569],{},[552,629,616],{},[591,631,632],{},"2",[557,634,287],{},[552,636,637],{},"ϵ",[557,639,586],{},[565,641,642,645],{},[552,643,644],{},"w",[552,646,572],{},[648,649,651],"annotation",{"encoding":650},"application\u002Fx-tex","\\operatorname{RMSNorm}(x_i) = x_i \\cdot \\frac{1}{\\sqrt{\\frac{1}{H}\\sum_{j=1}^{H}x_j^2 + \\epsilon}} \\cdot w_i",[27,653,657,752,810,1176],{"className":654,"ariaHidden":656},[655],"katex-html","true",[27,658,661,666,675,679,736,740,745,749],{"className":659},[660],"base",[27,662],{"className":663,"style":665},[664],"strut","height:1em;vertical-align:-0.25em;",[27,667,670],{"className":668},[669],"mop",[27,671,555],{"className":672},[673,674],"mord","mathrm",[27,676,563],{"className":677},[678],"mopen",[27,680,682,686],{"className":681},[673],[27,683,569],{"className":684},[673,685],"mathnormal",[27,687,690],{"className":688},[689],"msupsub",[27,691,695,727],{"className":692},[693,694],"vlist-t","vlist-t2",[27,696,699,722],{"className":697},[698],"vlist-r",[27,700,704],{"className":701,"style":703},[702],"vlist","height:0.3117em;",[27,705,707,712],{"style":706},"top:-2.55em;margin-left:0em;margin-right:0.05em;",[27,708],{"className":709,"style":711},[710],"pstrut","height:2.7em;",[27,713,719],{"className":714},[715,716,717,718],"sizing","reset-size6","size3","mtight",[27,720,572],{"className":721},[673,685,718],[27,723,726],{"className":724},[725],"vlist-s","​",[27,728,730],{"className":729},[698],[27,731,734],{"className":732,"style":733},[702],"height:0.15em;",[27,735],{},[27,737,575],{"className":738},[739],"mclose",[27,741],{"className":742,"style":744},[743],"mspace","margin-right:0.2778em;",[27,746,281],{"className":747},[748],"mrel",[27,750],{"className":751,"style":744},[743],[27,753,755,759,799,803,807],{"className":754},[660],[27,756],{"className":757,"style":758},[664],"height:0.5945em;vertical-align:-0.15em;",[27,760,762,765],{"className":761},[673],[27,763,569],{"className":764},[673,685],[27,766,768],{"className":767},[689],[27,769,771,791],{"className":770},[693,694],[27,772,774,788],{"className":773},[698],[27,775,777],{"className":776,"style":703},[702],[27,778,779,782],{"style":706},[27,780],{"className":781,"style":711},[710],[27,783,785],{"className":784},[715,716,717,718],[27,786,572],{"className":787},[673,685,718],[27,789,726],{"className":790},[725],[27,792,794],{"className":793},[698],[27,795,797],{"className":796,"style":733},[702],[27,798],{},[27,800],{"className":801,"style":802},[743],"margin-right:0.2222em;",[27,804,586],{"className":805},[806],"mbin",[27,808],{"className":809,"style":802},[743],[27,811,813,817,1167,1170,1173],{"className":812},[660],[27,814],{"className":815,"style":816},[664],"height:3.0514em;vertical-align:-1.73em;",[27,818,820,824,1164],{"className":819},[673],[27,821],{"className":822},[678,823],"nulldelimiter",[27,825,827],{"className":826},[588],[27,828,830,1155],{"className":829},[693,694],[27,831,833,1152],{"className":832},[698],[27,834,837,1131,1140],{"className":835,"style":836},[702],"height:1.3214em;",[27,838,840,844],{"style":839},"top:-2.11em;",[27,841],{"className":842,"style":843},[710],"height:3.2577em;",[27,845,847],{"className":846},[673],[27,848,851],{"className":849},[673,850],"sqrt",[27,852,854,1122],{"className":853},[693,694],[27,855,857,1119],{"className":856},[698],[27,858,861,1096],{"className":859,"style":860},[702],"height:1.2577em;",[27,862,866,870],{"className":863,"style":865},[864],"svg-align","top:-3.8em;",[27,867],{"className":868,"style":869},[710],"height:3.8em;",[27,871,874,951,955,1026,1029,1084,1087,1090,1093],{"className":872,"style":873},[673],"padding-left:1em;",[27,875,877,880,948],{"className":876},[673],[27,878],{"className":879},[678,823],[27,881,883],{"className":882},[588],[27,884,886,939],{"className":885},[693,694],[27,887,889,936],{"className":888},[698],[27,890,893,910,921],{"className":891,"style":892},[702],"height:0.8451em;",[27,894,896,900],{"style":895},"top:-2.655em;",[27,897],{"className":898,"style":899},[710],"height:3em;",[27,901,903],{"className":902},[715,716,717,718],[27,904,906],{"className":905},[673,718],[27,907,605],{"className":908,"style":909},[673,685,718],"margin-right:0.0813em;",[27,911,913,916],{"style":912},"top:-3.23em;",[27,914],{"className":915,"style":899},[710],[27,917],{"className":918,"style":920},[919],"frac-line","border-bottom-width:0.04em;",[27,922,924,927],{"style":923},"top:-3.394em;",[27,925],{"className":926,"style":899},[710],[27,928,930],{"className":929},[715,716,717,718],[27,931,933],{"className":932},[673,718],[27,934,593],{"className":935},[673,718],[27,937,726],{"className":938},[725],[27,940,942],{"className":941},[698],[27,943,946],{"className":944,"style":945},[702],"height:0.345em;",[27,947],{},[27,949],{"className":950},[739,823],[27,952],{"className":953,"style":954},[743],"margin-right:0.1667em;",[27,956,958,964],{"className":957},[669],[27,959,611],{"className":960,"style":963},[669,961,962],"op-symbol","small-op","position:relative;top:0em;",[27,965,967],{"className":966},[689],[27,968,970,1017],{"className":969},[693,694],[27,971,973,1014],{"className":972},[698],[27,974,977,999],{"className":975,"style":976},[702],"height:0.9812em;",[27,978,980,983],{"style":979},"top:-2.4003em;margin-left:0em;margin-right:0.05em;",[27,981],{"className":982,"style":711},[710],[27,984,986],{"className":985},[715,716,717,718],[27,987,989,993,996],{"className":988},[673,718],[27,990,616],{"className":991,"style":992},[673,685,718],"margin-right:0.0572em;",[27,994,281],{"className":995},[748,718],[27,997,593],{"className":998},[673,718],[27,1000,1002,1005],{"style":1001},"top:-3.2029em;margin-right:0.05em;",[27,1003],{"className":1004,"style":711},[710],[27,1006,1008],{"className":1007},[715,716,717,718],[27,1009,1011],{"className":1010},[673,718],[27,1012,605],{"className":1013,"style":909},[673,685,718],[27,1015,726],{"className":1016},[725],[27,1018,1020],{"className":1019},[698],[27,1021,1024],{"className":1022,"style":1023},[702],"height:0.4358em;",[27,1025],{},[27,1027],{"className":1028,"style":954},[743],[27,1030,1032,1035],{"className":1031},[673],[27,1033,569],{"className":1034},[673,685],[27,1036,1038],{"className":1037},[689],[27,1039,1041,1075],{"className":1040},[693,694],[27,1042,1044,1072],{"className":1043},[698],[27,1045,1048,1060],{"className":1046,"style":1047},[702],"height:0.7959em;",[27,1049,1051,1054],{"style":1050},"top:-2.4231em;margin-left:0em;margin-right:0.05em;",[27,1052],{"className":1053,"style":711},[710],[27,1055,1057],{"className":1056},[715,716,717,718],[27,1058,616],{"className":1059,"style":992},[673,685,718],[27,1061,1063,1066],{"style":1062},"top:-3.0448em;margin-right:0.05em;",[27,1064],{"className":1065,"style":711},[710],[27,1067,1069],{"className":1068},[715,716,717,718],[27,1070,632],{"className":1071},[673,718],[27,1073,726],{"className":1074},[725],[27,1076,1078],{"className":1077},[698],[27,1079,1082],{"className":1080,"style":1081},[702],"height:0.413em;",[27,1083],{},[27,1085],{"className":1086,"style":802},[743],[27,1088,287],{"className":1089},[806],[27,1091],{"className":1092,"style":802},[743],[27,1094,637],{"className":1095},[673,685],[27,1097,1099,1102],{"style":1098},"top:-3.2177em;",[27,1100],{"className":1101,"style":869},[710],[27,1103,1107],{"className":1104,"style":1106},[1105],"hide-tail","min-width:1.02em;height:1.88em;",[1108,1109,1115],"svg",{"xmlns":1110,"width":1111,"height":1112,"viewBox":1113,"preserveAspectRatio":1114},"http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","400em","1.88em","0 0 400000 1944","xMinYMin slice",[1116,1117],"path",{"d":1118},"M983 90\nl0 -0\nc4,-6.7,10,-10,18,-10 H400000v40\nH1013.1s-83.4,268,-264.1,840c-180.7,572,-277,876.3,-289,913c-4.7,4.7,-12.7,7,-24,7\ns-12,0,-12,0c-1.3,-3.3,-3.7,-11.7,-7,-25c-35.3,-125.3,-106.7,-373.3,-214,-744\nc-10,12,-21,25,-33,39s-32,39,-32,39c-6,-5.3,-15,-14,-27,-26s25,-30,25,-30\nc26.7,-32.7,52,-63,76,-91s52,-60,52,-60s208,722,208,722\nc56,-175.3,126.3,-397.3,211,-666c84.7,-268.7,153.8,-488.2,207.5,-658.5\nc53.7,-170.3,84.5,-266.8,92.5,-289.5z\nM1001 80h400000v40h-400000z",[27,1120,726],{"className":1121},[725],[27,1123,1125],{"className":1124},[698],[27,1126,1129],{"className":1127,"style":1128},[702],"height:0.5823em;",[27,1130],{},[27,1132,1134,1137],{"style":1133},"top:-3.4877em;",[27,1135],{"className":1136,"style":843},[710],[27,1138],{"className":1139,"style":920},[919],[27,1141,1143,1146],{"style":1142},"top:-3.9347em;",[27,1144],{"className":1145,"style":843},[710],[27,1147,1149],{"className":1148},[673],[27,1150,593],{"className":1151},[673],[27,1153,726],{"className":1154},[725],[27,1156,1158],{"className":1157},[698],[27,1159,1162],{"className":1160,"style":1161},[702],"height:1.73em;",[27,1163],{},[27,1165],{"className":1166},[739,823],[27,1168],{"className":1169,"style":802},[743],[27,1171,586],{"className":1172},[806],[27,1174],{"className":1175,"style":802},[743],[27,1177,1179,1183],{"className":1178},[660],[27,1180],{"className":1181,"style":1182},[664],"height:0.5806em;vertical-align:-0.15em;",[27,1184,1186,1190],{"className":1185},[673],[27,1187,644],{"className":1188,"style":1189},[673,685],"margin-right:0.0269em;",[27,1191,1193],{"className":1192},[689],[27,1194,1196,1217],{"className":1195},[693,694],[27,1197,1199,1214],{"className":1198},[698],[27,1200,1202],{"className":1201,"style":703},[702],[27,1203,1205,1208],{"style":1204},"top:-2.55em;margin-left:-0.0269em;margin-right:0.05em;",[27,1206],{"className":1207,"style":711},[710],[27,1209,1211],{"className":1210},[715,716,717,718],[27,1212,572],{"className":1213},[673,685,718],[27,1215,726],{"className":1216},[725],[27,1218,1220],{"className":1219},[698],[27,1221,1223],{"className":1222,"style":733},[702],[27,1224],{},[10,1226,1227,1228,1257,1258,1287],{},"其中 ",[27,1229,1231,1244],{"className":1230},[535],[27,1232,1234],{"className":1233},[539],[541,1235,1236],{"xmlns":543},[546,1237,1238,1242],{},[549,1239,1240],{},[552,1241,605],{},[648,1243,605],{"encoding":650},[27,1245,1247],{"className":1246,"ariaHidden":656},[655],[27,1248,1250,1254],{"className":1249},[660],[27,1251],{"className":1252,"style":1253},[664],"height:0.6833em;",[27,1255,605],{"className":1256,"style":909},[673,685]," 是 hidden size，",[27,1259,1261,1274],{"className":1260},[535],[27,1262,1264],{"className":1263},[539],[541,1265,1266],{"xmlns":543},[546,1267,1268,1272],{},[549,1269,1270],{},[552,1271,644],{},[648,1273,644],{"encoding":650},[27,1275,1277],{"className":1276,"ariaHidden":656},[655],[27,1278,1280,1284],{"className":1279},[660],[27,1281],{"className":1282,"style":1283},[664],"height:0.4306em;",[27,1285,644],{"className":1286,"style":1189},[673,685]," 是可学习缩放参数。",[10,1289,1290],{},"这个算子的 FLOPs 很少，却需要读取整行输入、计算 reduction，再写出结果。它的 arithmetic intensity 很低，通常不是算力不够，而是受以下因素限制：",[69,1292,1293,1296,1299,1302],{},[72,1294,1295],{},"global memory traffic；",[72,1297,1298],{},"kernel launch overhead；",[72,1300,1301],{},"reduction 的同步开销；",[72,1303,1304],{},"小 batch 下不足的并行度。",[10,1306,1307],{},"因此 RMSNorm 很适合用来练习 AI Infra 中真正重要的能力：判断瓶颈、设计 fusion 边界、处理混合精度，并建立可信的性能实验。",[105,1309,1311],{"id":1310},"_2-为什么融合-residual-add","2. 为什么融合 residual add",[10,1313,1314],{},"Transformer 的 pre-norm 路径通常包含：",[17,1316,1317],{"className":19,"code":270,"language":21,"meta":22,"style":22},[24,1318,1319,1331],{"__ignoreMap":22},[27,1320,1321,1323,1325,1327,1329],{"class":29,"line":30},[27,1322,277],{"class":37},[27,1324,281],{"class":280},[27,1326,284],{"class":37},[27,1328,287],{"class":280},[27,1330,290],{"class":37},[27,1332,1333,1335,1337,1339],{"class":29,"line":41},[27,1334,295],{"class":37},[27,1336,281],{"class":280},[27,1338,300],{"class":33},[27,1340,303],{"class":37},[10,1342,1343,1344,1347],{},"如果分开执行，至少需要一次 add kernel 和一次 normalization 路径。中间的 ",[24,1345,1346],{},"residual_out"," 需要写回显存，再被后续 kernel 读取。",[10,1349,1350],{},"本项目将 residual add、平方和归约、归一化与 weight scaling 放进一次 kernel launch：",[17,1352,1355],{"className":1353,"code":1354,"language":125,"meta":22},[123],"x + residual\n    -> FP32 square accumulation\n    -> block reduction\n    -> rsqrt(mean + eps)\n    -> normalize * weight\n",[24,1356,1354],{"__ignoreMap":22},[10,1358,1359,1360,1362],{},"融合后仍然返回 ",[24,1361,1346],{},"，因为下一个 decoder block 还需要这条 residual stream。这个细节很重要：kernel fusion 不能只追求理论上的最少写入，还必须保持模型调用方需要的语义。",[105,1364,1366],{"id":1365},"_3-kernel-映射一行一个-block","3. Kernel 映射：一行一个 block",[10,1368,1369,1370,1373],{},"把输入展平为 ",[24,1371,1372],{},"[tokens, hidden_size]"," 后，每一行的 RMSNorm 彼此独立，因此使用：",[17,1375,1378],{"className":1376,"code":1377,"language":125,"meta":22},[123],"1 CUDA block \u003C-> 1 token row\n256 threads   \u003C-> 并行遍历 hidden dimension\n",[24,1379,1377],{"__ignoreMap":22},[10,1381,1382],{},"每个线程处理若干元素并累积局部平方和。典型 hidden size 是 4096 或 8192，256 个线程能让每个线程处理多个连续元素，同时保持足够的并发。",[113,1384,1386],{"id":1385},"_31-pack-4-向量化访存","3.1 Pack-4 向量化访存",[10,1388,1389],{},"当 hidden size 能被 4 整除时，kernel 把连续四个元素包装成一个对齐 pack：",[17,1391,1395],{"className":1392,"code":1393,"language":1394,"meta":22,"style":22},"language-cpp shiki shiki-themes github-dark-dimmed github-light houston","template \u003Ctypename T, int N>\nstruct alignas(sizeof(T) * N) Pack {\n  T value[N];\n};\n","cpp",[24,1396,1397,1425,1444,1457],{"__ignoreMap":22},[27,1398,1399,1403,1406,1409,1413,1416,1419,1422],{"class":29,"line":30},[27,1400,1402],{"class":1401},"sST-V","template",[27,1404,1405],{"class":37}," \u003C",[27,1407,1408],{"class":1401},"typename",[27,1410,1412],{"class":1411},"sBBmv"," T",[27,1414,1415],{"class":37},", ",[27,1417,1418],{"class":1401},"int",[27,1420,1421],{"class":1411}," N",[27,1423,1424],{"class":37},">\n",[27,1426,1427,1430,1434,1438,1441],{"class":29,"line":41},[27,1428,1429],{"class":1401},"struct",[27,1431,1433],{"class":1432},"svkvy"," alignas(",[27,1435,1437],{"class":1436},"s7VMM","sizeof",[27,1439,1440],{"class":1432},"(T)",[27,1442,1443],{"class":37}," * N) Pack {\n",[27,1445,1447,1450,1454],{"class":29,"line":1446},3,[27,1448,1449],{"class":37},"  T ",[27,1451,1453],{"class":1452},"s0K4T","value",[27,1455,1456],{"class":37},"[N];\n",[27,1458,1460],{"class":29,"line":1459},4,[27,1461,1462],{"class":37},"};\n",[10,1464,1465],{},"这能减少 load\u002Fstore 指令数量并提高 memory transaction 利用率。对于 255 这类非对齐维度，代码会自动切换到标量 fallback，而不是假设所有模型 shape 都是整齐的。",[113,1467,1469],{"id":1468},"_32-fp32-累加","3.2 FP32 累加",[10,1471,1472],{},"即使输入是 FP16 或 BF16，平方和也使用 FP32：",[17,1474,1476],{"className":1392,"code":1475,"language":1394,"meta":22,"style":22},"float square_sum = 0.0f;\nsquare_sum += value * value;\n",[24,1477,1478,1499],{"__ignoreMap":22},[27,1479,1480,1483,1486,1488,1492,1496],{"class":29,"line":30},[27,1481,1482],{"class":1401},"float",[27,1484,1485],{"class":37}," square_sum ",[27,1487,281],{"class":1401},[27,1489,1491],{"class":1490},"sgENW"," 0.0",[27,1493,1495],{"class":1494},"s5AKZ","f",[27,1497,1498],{"class":37},";\n",[27,1500,1501,1504,1507,1510,1513],{"class":29,"line":41},[27,1502,1503],{"class":37},"square_sum ",[27,1505,1506],{"class":1401},"+=",[27,1508,1509],{"class":37}," value ",[27,1511,1512],{"class":280},"*",[27,1514,1515],{"class":37}," value;\n",[10,1517,1518],{},"hidden size 往往有几千维，直接用低精度累加会放大舍入误差，甚至产生溢出。输出仍然转换回原始 dtype，因此这是数值稳定性与吞吐之间的常见折中。",[10,1520,1521,1522,1525,1526,1528],{},"fused path 还有一个容易忽略的语义：",[24,1523,1524],{},"x + residual"," 必须先舍入到目标 dtype，再用这个实际写入 residual stream 的值计算平方和。否则 BF16 下归一化使用的值和返回的 ",[24,1527,1346],{}," 会不一致。",[105,1530,1532],{"id":1531},"_4-warp-shuffle-reduction","4. Warp shuffle reduction",[10,1534,1535],{},"每个线程得到局部平方和后，需要在 block 内合并。",[10,1537,1538],{},"第一阶段在每个 warp 内使用：",[17,1540,1542],{"className":1392,"code":1541,"language":1394,"meta":22,"style":22},"value += __shfl_down_sync(0xffffffff, value, offset);\n",[24,1543,1544],{"__ignoreMap":22},[27,1545,1546,1549,1551,1555,1557,1560,1563],{"class":29,"line":30},[27,1547,1548],{"class":37},"value ",[27,1550,1506],{"class":1401},[27,1552,1554],{"class":1553},"sS_0i"," __shfl_down_sync",[27,1556,563],{"class":37},[27,1558,1559],{"class":1494},"0x",[27,1561,1562],{"class":1490},"ffffffff",[27,1564,1565],{"class":37},", value, offset);\n",[10,1567,1568],{},"32 个线程的数据直接通过寄存器交换完成，不需要把每一步都写入 shared memory。",[10,1570,1571,1572,1575],{},"第二阶段只把 8 个 warp 的结果写入 shared memory，再由第一个 warp 完成最终归约。256-thread kernel 因此只使用 ",[60,1573,1574],{},"32B shared memory","。",[10,1577,1578,1579,1582],{},"得到整行平方和后计算 ",[24,1580,1581],{},"inv_rms","，然后执行第二遍读取并写出归一化结果。为什么不把整行输入一直放在寄存器中？因为保存 4096 或 8192 个值会显著增加 register pressure，降低 occupancy。这里选择第二遍读取，是经过资源约束后的工程取舍。",[105,1584,1586],{"id":1585},"_5-pytorch-扩展边界","5. PyTorch 扩展边界",[10,1588,1589],{},"Kernel 快不等于扩展可用。项目还处理了：",[69,1591,1592,1595,1598,1604,1607],{},[72,1593,1594],{},"CUDA、contiguous、shape 和 dtype 校验；",[72,1596,1597],{},"FP32、FP16、BF16 类型 dispatch；",[72,1599,1600,1603],{},[24,1601,1602],{},"CUDAGuard"," 多卡设备保护；",[72,1605,1606],{},"当前 PyTorch CUDA stream；",[72,1608,1609,1612],{},[24,1610,1611],{},"C10_CUDA_KERNEL_LAUNCH_CHECK"," 异步启动错误。",[10,1614,1615,1616,1619],{},"这使它可以像普通 PyTorch 算子一样调用，而不是一个只能独立运行的 ",[24,1617,1618],{},".cu"," 示例。",[105,1621,1623],{"id":1622},"_6-hot-cache-和-cold-cache","6. Hot cache 和 cold cache",[10,1625,1626],{},"只测一组重复输入，GPU 很可能从 L2 cache 读取数据，得到非常漂亮的结果，但这不能完全代表显存压力较大的真实 workload。",[10,1628,1629],{},"因此 benchmark 同时报告两种模式：",[143,1631,1632,1645],{},[146,1633,1634],{},[149,1635,1636,1639,1642],{},[152,1637,1638],{},"模式",[152,1640,1641],{},"测量方式",[152,1643,1644],{},"主要观察内容",[169,1646,1647,1657],{},[149,1648,1649,1651,1654],{},[174,1650,502],{},[174,1652,1653],{},"连续使用同一组 tensor",[174,1655,1656],{},"最低延迟、launch 与计算效率",[149,1658,1659,1661,1664],{},[174,1660,508],{},[174,1662,1663],{},"每次计时前覆盖并同步 256MB GPU buffer",[174,1665,1666],{},"更接近 DRAM-bound 的工作集",[10,1668,1669],{},"Cold cache 也不等于“完全没有缓存”。同一 kernel 内第一次和第二次读取之间仍可能发生 L2 复用。因此本文给出的 GB\u002Fs 是逻辑 tensor traffic 除以延迟，不是 DRAM 硬件计数器。",[10,1671,1672],{},"计时方法包括：",[69,1674,1675,1678,1681,1684,1687,1694,1697],{},[72,1676,1677],{},"CUDA Event；",[72,1679,1680],{},"50 次 warmup；",[72,1682,1683],{},"50 次测量；",[72,1685,1686],{},"5 轮重复并取中位数；",[72,1688,1689,1690,1693],{},"eager PyTorch 与 ",[24,1691,1692],{},"torch.compile(fullgraph=True)"," 两种 baseline；",[72,1695,1696],{},"FP16 与 BF16；",[72,1698,1699],{},"1、32、256、1024 tokens 以及 4096\u002F8192 hidden size。",[105,1701,1703],{"id":1702},"_7-实测结果","7. 实测结果",[10,1705,1706,1707,1710,1711,1713,1714,1575],{},"测试环境：RTX 4090D 24GB、CUDA 12.8、PyTorch 2.8.0、",[24,1708,1709],{},"sm_89","。下表使用更强的 ",[24,1712,516],{}," baseline，shape 为 ",[24,1715,1716],{},"256 × 4096",[143,1718,1719,1741],{},[146,1720,1721],{},[149,1722,1723,1726,1729,1732,1735,1738],{},[152,1724,1725],{},"dtype",[152,1727,1728],{},"cache",[152,1730,1731],{},"operation",[152,1733,1734],{"align":157},"PyTorch",[152,1736,1737],{"align":157},"CUDA",[152,1739,1740],{"align":157},"speedup",[169,1742,1743,1761,1779,1797,1814,1831,1847,1863],{},[149,1744,1745,1747,1750,1752,1755,1758],{},[174,1746,193],{},[174,1748,1749],{},"hot",[174,1751,555],{},[174,1753,1754],{"align":157},"35.122 us",[174,1756,1757],{"align":157},"6.861 us",[174,1759,1760],{"align":157},"5.12x",[149,1762,1763,1765,1767,1770,1773,1776],{},[174,1764,193],{},[174,1766,1749],{},[174,1768,1769],{},"add + RMSNorm",[174,1771,1772],{"align":157},"39.670 us",[174,1774,1775],{"align":157},"9.298 us",[174,1777,1778],{"align":157},"4.27x",[149,1780,1781,1783,1786,1788,1791,1794],{},[174,1782,193],{},[174,1784,1785],{},"cold",[174,1787,555],{},[174,1789,1790],{"align":157},"57.344 us",[174,1792,1793],{"align":157},"16.384 us",[174,1795,1796],{"align":157},"3.50x",[149,1798,1799,1801,1803,1805,1808,1811],{},[174,1800,193],{},[174,1802,1785],{},[174,1804,1769],{},[174,1806,1807],{"align":157},"63.648 us",[174,1809,1810],{"align":157},"20.480 us",[174,1812,1813],{"align":157},"3.11x",[149,1815,1816,1818,1820,1822,1825,1828],{},[174,1817,210],{},[174,1819,1749],{},[174,1821,555],{},[174,1823,1824],{"align":157},"36.415 us",[174,1826,1827],{"align":157},"6.984 us",[174,1829,1830],{"align":157},"5.21x",[149,1832,1833,1835,1837,1839,1842,1844],{},[174,1834,210],{},[174,1836,1749],{},[174,1838,1769],{},[174,1840,1841],{"align":157},"40.572 us",[174,1843,1775],{"align":157},[174,1845,1846],{"align":157},"4.36x",[149,1848,1849,1851,1853,1855,1858,1860],{},[174,1850,210],{},[174,1852,1785],{},[174,1854,555],{},[174,1856,1857],{"align":157},"62.720 us",[174,1859,1793],{"align":157},[174,1861,1862],{"align":157},"3.83x",[149,1864,1865,1867,1869,1871,1874,1877],{},[174,1866,210],{},[174,1868,1785],{},[174,1870,1769],{},[174,1872,1873],{"align":157},"69.648 us",[174,1875,1876],{"align":157},"20.592 us",[174,1878,1879],{"align":157},"3.38x",[10,1881,1882],{},[94,1883],{"alt":1884,"src":1885},"FP16 benchmark comparison","\u002Fimages\u002Fposts\u002Fcuda-fused-rmsnorm\u002Ffp16-benchmark.svg",[10,1887,1888],{},[101,1889,1890,1891,1575],{},"图：本文基于公开 CSV 原创绘制。横轴为单次延迟，越短越好；baseline 为 ",[24,1892,1692],{},[10,1894,1895],{},"结果显示两件事：",[1897,1898,1899,1905],"ol",{},[72,1900,1901,1902,1904],{},"小而专用的 CUDA kernel 即使面对 ",[24,1903,516],{},"，仍能显著减少通用路径和 reduction 的开销；",[72,1906,1907],{},"cold-cache 的加速比低于 hot-cache，说明这个算子确实受到内存层级影响，只报告 hot-cache 会高估收益。",[105,1909,1911],{"id":1910},"_8-正确性和资源占用","8. 正确性和资源占用",[10,1913,1914],{},"项目在真实 GPU 上通过 23 项测试，覆盖：",[69,1916,1917,1920,1923,1926,1929,1932],{},[72,1918,1919],{},"FP32、FP16、BF16；",[72,1921,1922],{},"255 维 scalar fallback；",[72,1924,1925],{},"256、4096、8192 维 vectorized path；",[72,1927,1928],{},"二维和三维输入；",[72,1930,1931],{},"fused residual output；",[72,1933,1934,1935,1938],{},"PyTorch ",[24,1936,1937],{},"F.rms_norm"," 参考结果。",[10,1940,1941,1944],{},[24,1942,1943],{},"cuobjdump --dump-resource-usage"," 给出的 vectorized fused kernel 资源如下：",[143,1946,1947,1965],{},[146,1948,1949],{},[149,1950,1951,1953,1956,1959,1962],{},[152,1952,1725],{},[152,1954,1955],{"align":157},"registers\u002Fthread",[152,1957,1958],{"align":157},"shared",[152,1960,1961],{"align":157},"local",[152,1963,1964],{"align":157},"stack",[169,1966,1967,1982,1995],{},[149,1968,1969,1971,1974,1977,1980],{},[174,1970,193],{},[174,1972,1973],{"align":157},"30",[174,1975,1976],{"align":157},"32B",[174,1978,1979],{"align":157},"0B",[174,1981,1979],{"align":157},[149,1983,1984,1986,1989,1991,1993],{},[174,1985,210],{},[174,1987,1988],{"align":157},"26",[174,1990,1976],{"align":157},[174,1992,1979],{"align":157},[174,1994,1979],{"align":157},[149,1996,1997,1999,2002,2004,2006],{},[174,1998,176],{},[174,2000,2001],{"align":157},"29",[174,2003,1976],{"align":157},[174,2005,1979],{"align":157},[174,2007,1979],{"align":157},[10,2009,2010],{},"没有 local memory 和 stack 使用，说明当前版本没有发生寄存器溢出。",[10,2012,2013,2014,2017],{},"我也尝试使用 Nsight Compute 获取 DRAM throughput、L2 hit rate 和 warp stall reason，但 AutoDL 容器禁止访问 GPU performance counters，返回 ",[24,2015,2016],{},"ERR_NVGPUCTRPERM","。这里选择明确记录限制，而不是用逻辑带宽冒充硬件计数器。",[105,2019,2021],{"id":2020},"_9-为什么这个项目还不能叫生产级","9. 为什么这个项目还不能叫“生产级”",[10,2023,2024],{},"当前版本有清晰的边界：",[69,2026,2027,2030,2033,2036,2039],{},[72,2028,2029],{},"只实现 forward，适合推理，没有 backward；",[72,2031,2032],{},"block size 固定为 256，没有按 GPU 架构和 hidden size autotune；",[72,2034,2035],{},"只测了单算子，没有接入完整 decoder block；",[72,2037,2038],{},"只有 RTX 4090D 数据，缺少 A100\u002FH100 等数据中心 GPU 对比；",[72,2040,2041],{},"还没有加入 Apex 和独立 Triton baseline。",[10,2043,2044],{},"下一阶段最值得做的不是继续堆零散 kernel，而是加入 block-size dispatch、Triton 对照实现，并测量融合算子对端到端 time per output token 的影响。",[105,2046,2048],{"id":2047},"_10-总结","10. 总结",[10,2050,2051],{},"这个项目让我把“算子融合可以减少访存”这句话落到了可验证的工程细节上：",[69,2053,2054,2057,2060,2063,2066,2069],{},[72,2055,2056],{},"先从模型调用语义确定 fusion 边界；",[72,2058,2059],{},"用 roofline 思维判断 RMSNorm 是 memory-bound；",[72,2061,2062],{},"用 pack-4 和 warp shuffle 实现具体优化；",[72,2064,2065],{},"用 FP32 accumulation 和 dtype 舍入保证数值正确；",[72,2067,2068],{},"用 hot\u002Fcold cache 与强 baseline 设计可信实验；",[72,2070,2071],{},"最后如实报告硬件、原始数据和测量限制。",[10,2073,2074],{},"对 AI Infra 项目而言，真正有价值的不是一个最大的加速数字，而是能解释每一项优化为什么成立、在哪些条件下成立，以及下一步该验证什么。",[2076,2077,2078],"style",{},"html pre.shiki code .sq3c-, html code.shiki .sq3c-{--shiki-default:#ADBAC7;--shiki-light:#24292E;--shiki-cyber:#00DAEF}html pre.shiki code .sWURj, html code.shiki .sWURj{--shiki-default:#ADBAC7;--shiki-light:#24292E;--shiki-cyber:#EEF0F9}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .cyber .shiki span {color: var(--shiki-cyber);background: var(--shiki-cyber-bg);font-style: var(--shiki-cyber-font-style);font-weight: var(--shiki-cyber-font-weight);text-decoration: var(--shiki-cyber-text-decoration);}html.cyber .shiki span {color: var(--shiki-cyber);background: var(--shiki-cyber-bg);font-style: var(--shiki-cyber-font-style);font-weight: var(--shiki-cyber-font-weight);text-decoration: var(--shiki-cyber-text-decoration);}html pre.shiki code .sf2cM, html code.shiki .sf2cM{--shiki-default:#F47067;--shiki-light:#D73A49;--shiki-cyber:#EEF0F9}html pre.shiki code .sST-V, html code.shiki .sST-V{--shiki-default:#F47067;--shiki-light:#D73A49;--shiki-cyber:#54B9FF}html pre.shiki code .sBBmv, html code.shiki .sBBmv{--shiki-default:#F69D50;--shiki-light:#6F42C1;--shiki-cyber:#ACAFFF}html pre.shiki code .svkvy, html code.shiki .svkvy{--shiki-default:#6CB6FF;--shiki-light:#005CC5;--shiki-cyber:#EEF0F9}html pre.shiki code .s7VMM, html code.shiki .s7VMM{--shiki-default:#6CB6FF;--shiki-light:#6F42C1;--shiki-cyber:#EEF0F9}html pre.shiki code .s0K4T, html code.shiki .s0K4T{--shiki-default:#ADBAC7;--shiki-light:#24292E;--shiki-cyber:#4BF3C8}html pre.shiki code .sgENW, html code.shiki .sgENW{--shiki-default:#6CB6FF;--shiki-light:#005CC5;--shiki-cyber:#FFD493}html pre.shiki code .s5AKZ, html code.shiki .s5AKZ{--shiki-default:#F47067;--shiki-light:#D73A49;--shiki-cyber:#4BF3C8}html pre.shiki code .sS_0i, html code.shiki .sS_0i{--shiki-default:#DCBDFB;--shiki-light:#6F42C1;--shiki-cyber:#00DAEF}",{"title":22,"searchDepth":41,"depth":1446,"links":2080},[2081,2091,2092,2093,2097,2098,2099,2100,2101,2102,2103],{"id":107,"depth":41,"text":108,"children":2082},[2083,2084,2085,2086,2087,2088,2089,2090],{"id":115,"depth":1446,"text":116},{"id":137,"depth":1446,"text":138},{"id":239,"depth":1446,"text":240},{"id":309,"depth":1446,"text":310},{"id":354,"depth":1446,"text":355},{"id":439,"depth":1446,"text":440},{"id":465,"depth":1446,"text":466},{"id":478,"depth":1446,"text":479},{"id":523,"depth":41,"text":524},{"id":1310,"depth":41,"text":1311},{"id":1365,"depth":41,"text":1366,"children":2094},[2095,2096],{"id":1385,"depth":1446,"text":1386},{"id":1468,"depth":1446,"text":1469},{"id":1531,"depth":41,"text":1532},{"id":1585,"depth":41,"text":1586},{"id":1622,"depth":41,"text":1623},{"id":1702,"depth":41,"text":1703},{"id":1910,"depth":41,"text":1911},{"id":2020,"depth":41,"text":2021},{"id":2047,"depth":41,"text":2048},[2105],"技术","\u002Fimages\u002Fposts\u002Fcuda-fused-rmsnorm-kernel\u002Fcuda-rmsnorm-cover-20261018.webp","融合残差加法与向量化 float4 访存经 Warp Shuffle 蝴蝶树归约输出归一化张量的等距硬件架构插画","2026-07-22 18:16:50","从零实现 PyTorch CUDA 扩展，拆解 RMSNorm 的访存瓶颈、warp 归约、向量化加载、算子融合与冷热缓存 benchmark。",false,"md",true,{},"\u002Fposts\u002Fcuda-fused-rmsnorm-kernel",null,{"title":5,"description":2109},"AI Infra 实战","posts\u002Fcuda-fused-rmsnorm-kernel",[1737,2120,2121,1734,2122,2123],"AI Infra","GPU","LLM 推理","Kernel 优化","11nzlF1baArACE63vK5XNQxtNLggkENXGSoNMgm4OBU",[2126,2144,2158,2173,2187,2202,2217,2232,2246,2259,2273,2285,2299,2312,2324,2338,2355,2370,2381,2393,2408,2421,2435,2447,2460,2472,2487,2501,2515,2526,2539,2556,2571,2584,2588,2600,2613,2626],{"slug":2127,"path":2128,"title":2129,"date":2130,"tags":2131,"description":2136,"cover":2137,"coverAlt":2138,"excerpt":2139,"draft":2110,"hidden":2110,"published":2112,"readingTime":2140,"series":2141,"seriesOrder":2140,"codeUrl":2142,"attachmentUrl":2143},"cs336-cross-entropy-adamw-training-loop","\u002Fposts\u002Fcs336-cross-entropy-adamw-training-loop","Stanford CS336 学习记录 08：数值稳定交叉熵、AdamW、余弦调度与完整训练闭环","2026-10-27",[2132,1734,2133,2134,2135],"CS336","AdamW","CrossEntropy","训练闭环","用 LogSumExp 消去交叉熵中的指数溢出，手写解耦权重衰减的 AdamW 优化器、全局 L2 梯度裁剪、Warmup 余弦退火调度、memmap 数据采样、Checkpoint 序列化与 Top-p 解码，完成 CS336 Assignment 1 全部 48 项测试。","\u002Fimages\u002Fposts\u002Fcs336-cross-entropy-adamw-training-loop\u002Fcs336-cross-entropy-adamw-training-loop-cover.webp","滑动窗口采样、余弦退火曲线、双动量优化轨迹与检查点归档构成的训练闭环 AI 辅助概念插画","作为 Stanford CS336 Assignment 1 系列的收官篇，我们已经拥有了分词器（bpe.py、tokenizer.py）和完整的语言模型架构（transformer.py）。要让模型在语料上真正训练并生成文本，还需",8,"Stanford CS336","https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fcs336-labs\u002Ftree\u002Fb5c93ad0cc73178b2b2a62e518a0900da47a0fb7","\u002Fmaterials\u002Fcs336\u002Fexperiments\u002Fcs336-08-adamw-training-loop-20261027.zip",{"slug":2145,"path":2146,"title":2147,"date":2148,"tags":2149,"description":2154,"cover":2155,"coverAlt":2156,"excerpt":2157,"draft":2110,"hidden":2110,"published":2112,"readingTime":2140},"editcache-video-dit-inference-acceleration","\u002Fposts\u002Feditcache-video-dit-inference-acceleration","EditCache：视频 Diffusion Transformer (DiT) 局部编辑的跨步特征复用与区域重算加速","2026-10-25",[2150,2151,2120,2152,2121,2153],"扩散模型","视频生成","推理加速","论文解读","拆解视频扩散模型（Wan2.1-VACE）局部编辑中的三重算力冗余，通过可重入 Latent、跨步 Block 隐状态缓存复用、带 Halo 边界的局部区域重算与自适应推理起点，在单卡 H100 上实现 2.46× 模型计算加速与 1.49× 严格同步端到端加速。","\u002Fimages\u002Fposts\u002Feditcache-video-dit-inference-acceleration\u002Feditcache-dit-cover-20261025.webp","视频扩散 Transformer 时空三维体素网格中冻结背景缓存与暖橙色局部编辑重算通道的等距技术插画","在基于 Diffusion Transformer（DiT）的视频编辑与局部重绘（Video Inpainting \u002F Local Editing）任务中，用户的典型工作流并不是一次性生成最终成片，而是针对同一段源视频反复调整掩码区",{"slug":2159,"path":2160,"title":2161,"date":2148,"tags":2162,"description":2167,"cover":2168,"coverAlt":2169,"excerpt":2170,"draft":2110,"hidden":2110,"published":2112,"readingTime":2140,"series":2141,"seriesOrder":2171,"codeUrl":2142,"attachmentUrl":2172},"cs336-bpe-tokenizer-training-encoding","\u002Fposts\u002Fcs336-bpe-tokenizer-training-encoding","Stanford CS336 学习记录 07：Byte-Level BPE 分词器训练、倒排索引加速与流式编码",[2132,2163,2164,2165,2166],"Tokenizer","BPE","Tiktoken","算法优化","实现 GPT-2 正则预分词、特殊 token 硬边界切分、基于 pair-to-word 倒排索引的增量 BPE 合并算法（\u003C 0.25s 通过速度基准），以及与 tiktoken 完全对齐的 Tokenizer。","\u002Fimages\u002Fposts\u002Fcs336-bpe-tokenizer-training-encoding\u002Fcs336-bpe-tokenizer-training-encoding-cover.webp","UTF-8 字节块经过正则切分闸口后沿合并树聚合成子词单元的 AI 辅助概念插画","前面六篇完成了 Transformer 语言模型本身的全部神经网络组件。要让模型真正处理自然语言文本，还需要打通入口处的 ByteLevel BPE（BytePair Encoding）分词器。这一篇记录 cs336basics\u002Fb",7,"\u002Fmaterials\u002Fcs336\u002Fexperiments\u002Fcs336-07-bpe-tokenizer-20261025.zip",{"slug":2174,"path":2175,"title":2176,"date":2177,"tags":2178,"description":2183,"cover":2184,"coverAlt":2185,"excerpt":2186,"draft":2110,"hidden":2110,"published":2112,"readingTime":2140},"in-situ-autoguidance-editable-scientific-illustration","\u002Fposts\u002Fin-situ-autoguidance-editable-scientific-illustration","从原位自校正引导（In-situ Autoguidance）到可编辑科研插图生成（AutoFigure-Edit）","2026-10-24",[2150,2179,2180,2181,2182,2153],"多模态","Agent","ICML","ACL","串联两项生成式模型与多模态智能体研究：如何通过推理期局部随机微扰在单模型内构造免训练扩散自引导向量（ICML 2025 Workshop 第一作者，arXiv:2510.17136），以及如何通过五阶段视觉-符号混合流水线生成原生可编辑 SVG 科研插图（ACL 2026 Demo，FigureBench 83.0% 胜率）。","\u002Fimages\u002Fposts\u002Fin-situ-autoguidance-editable-scientific-illustration\u002Fautoguidance-autofigure-cover-20261024.webp","左侧扩散流形自校正轨迹矢量场与右侧五阶段可编辑 SVG 科研插图分层网格相衔接的等距技术插画","生成式视觉模型正在经历从“生成高保真像素（Pixel Synthesis）”向“可控结构化生成（Controllable & Editable Generation）”的演进。在这一进程中，我作为第一作者或核心贡献者参与了两项分别处",{"slug":2188,"path":2189,"title":2190,"date":2191,"tags":2192,"description":2196,"cover":2197,"coverAlt":2198,"excerpt":2199,"draft":2110,"hidden":2110,"published":2112,"readingTime":2140,"series":2141,"seriesOrder":2200,"codeUrl":2142,"attachmentUrl":2201},"cs336-pre-norm-transformer-block-lm","\u002Fposts\u002Fcs336-pre-norm-transformer-block-lm","Stanford CS336 学习记录 06：Pre-Norm Transformer Block 与完整语言模型装配","2026-10-23",[2132,1734,2193,2194,2195],"Transformer","Pre-Norm","FLOPs","将 RMSNorm、带 RoPE 的因果多头自注意力和 SwiGLU 组装成 Pre-Norm Transformer Block，搭建完整 TransformerLM，并核算 GPT-2 XL 的参数量与前向 FLOPs。","\u002Fimages\u002Fposts\u002Fcs336-pre-norm-transformer-block-lm\u002Fcs336-pre-norm-transformer-block-lm-cover.webp","干净残差流主干穿过多个 Pre-Norm 子层并汇聚到词表投影层的 AI 辅助概念插画","前五篇把 Linear、Embedding、RMSNorm、SwiGLU、RotaryPositionalEmbedding 和因果 MultiHeadSelfAttention 全部单独打通了。这一篇进入 Assignment 1",6,"\u002Fmaterials\u002Fcs336\u002Fexperiments\u002Fcs336-06-transformer-block-lm-20261023.zip",{"slug":2203,"path":2204,"title":2205,"date":2191,"tags":2206,"description":2212,"cover":2213,"coverAlt":2214,"excerpt":2215,"draft":2110,"hidden":2110,"published":2112,"readingTime":2216},"xgrammar-concurrent-cache-generation-fix","\u002Fposts\u002Fxgrammar-concurrent-cache-generation-fix","XGrammar 结构化解码缓存剖析：Unlimited 模式并发 Clear 字节记账竞态与代际修复（PR #1011）",[2120,2207,2208,2209,2210,2211],"C++","XGrammar","结构化解码","并发编程","源码分析","拆解 XGrammar GrammarCompiler 的双层缓存架构与 ThreadSafeLRUCache 锁分离实现，定位 Unlimited 模式下 Clear() 字节残留与在途编译（In-flight Computation）跨代污染根因，给出代际号 + 读写锁原子修复与零 sleep 确定性并发 GoogleTest 验证。","\u002Fimages\u002Fposts\u002Fxgrammar-concurrent-cache-generation-fix\u002Fxgrammar-cache-fix-cover-20261022.webp","双层磨砂玻璃流道经暖橙色代际闸门过滤后汇入晶格缓存矩阵的等距技术插画","在大语言模型服务（如 SGLang、vLLM）的结构化输出场景中，JSON Schema 或 EBNF 语法需要先编译为词表级掩码缓存（CompiledGrammar），才能在每个 Decode 步以微秒级开销生成合法 token ",14,{"slug":2218,"path":2219,"title":2220,"date":2221,"tags":2222,"description":2228,"cover":2229,"coverAlt":2230,"excerpt":2231,"draft":2110,"hidden":2110,"published":2112,"readingTime":2216},"multi-constraint-agent-routing-dag-eval","\u002Fposts\u002Fmulti-constraint-agent-routing-dag-eval","多约束生产级 Agent 系统实践：两阶段工具路由、DAG 局部重规划与轨迹级评测闭环","2026-10-22",[2180,2223,2224,2225,2226,2227],"LLM","工具调用","上下文工程","DPO","强化学习","围绕跨境酒旅等强耦合多约束业务场景，拆解向量 Top-K 召回 + LLM 精排 + 跨语言实体链接与 Schema 门控（96%+ 调用合法率）、基于状态机的 DAG 局部子图重规划、Observation 冗余裁剪（节省 40% Token），以及从线上失败恢复轨迹构造 SFT\u002FDPO 偏好对的评测与训练闭环。","\u002Fimages\u002Fposts\u002Fmulti-constraint-agent-routing-dag-eval\u002Fagent-dag-routing-cover-20261020.webp","候选模块经漏斗滤网进入双分支有向无环图轨道并在受阻节点触发暖橙色局部旁路重连的等距技术插画","在玩具级 Demo 里，几个工具加上一段 ReAct 循环就能跑通大多数单步问答。但当 Agent 进入跨境酒旅、复杂交易或研发沙箱等真实业务生产环境时，单靠线性 ReAct 循环很快会撞上四堵墙： 1. 工具空间与参数耦合爆炸：候",{"slug":2233,"path":2234,"title":2235,"date":2236,"tags":2237,"description":2241,"cover":2242,"coverAlt":2243,"excerpt":2244,"draft":2110,"hidden":2110,"published":2112,"readingTime":2245},"llm-mrd-distillation-auditable-review-agents","\u002Fposts\u002Fllm-mrd-distillation-auditable-review-agents","多视角推理蒸馏（LLM-MRD）与可审计生成式评审系统（DeepReviewer 2.0）深度拆解","2026-10-21",[2238,2179,2239,2180,2240,2153],"知识蒸馏","Qwen2.5-VL","Post-training","对比大模型落地的两条互补路径：如何用 Qwen2.5-VL 离线多视角分析结合 MLP 残差校准与温度缩放 KL 蒸馏，将推理能力压入 358M 学生模型实现 17.2 ms 独立推理（LLM-MRD，DASFAA 2026 Oral）；以及如何通过主张-证据-风险台账、PASA 可比性门控与导出预算校验构建可审计的同行评审 Agent（DeepReviewer 2.0）。","\u002Fimages\u002Fposts\u002Fllm-mrd-distillation-auditable-review-agents\u002Fmrd-review-distill-cover-20261016.webp","三块半透明几何棱镜发出平行光束穿过中央校准环汇聚至右侧高密度晶体模块的等距技术插画","把大语言模型或视觉语言模型（VLM）的深度推理能力用到实际系统中，通常会面临一个两难选择： 如果场景受限于毫秒级在线延迟（如高吞吐多模态内容审核与虚假信息检测）：直接在线调用 7B～72B 的多模态大模型做长链条思维链（CoT）生成",18,{"slug":2247,"path":2248,"title":2249,"date":2236,"tags":2250,"description":2253,"cover":2254,"coverAlt":2255,"excerpt":2256,"draft":2110,"hidden":2110,"published":2112,"readingTime":2171,"series":2141,"seriesOrder":2257,"codeUrl":2142,"attachmentUrl":2258},"cs336-rope-rotary-position-embedding","\u002Fposts\u002Fcs336-rope-rotary-position-embedding","Stanford CS336 学习记录 05：RoPE 旋转位置编码与带位置的多头注意力",[2132,1734,2193,2251,2252],"RoPE","Attention","从二维平面正交旋转推导 RoPE 的奇偶特征配对公式，理解为什么内积只依赖相对位置差 (i - j)，并用 register_buffer 预计算 cos\u002Fsin 缓存接入因果多头自注意力。","\u002Fimages\u002Fposts\u002Fcs336-rope-rotary-position-embedding\u002Fcs336-rope-rotary-position-embedding-cover.webp","二维特征平面中的旋转向量对与预计算余弦正弦网格的 AI 辅助概念插画","上一篇完成了不含位置编码的因果多头自注意力。如果只看那一套矩阵乘法，把输入序列前两个 token 交换顺序（并在因果 mask 允许的范围内看同一个后缀 query 对它们的点积），点积本身并不知道哪个 key 离当前 query ",5,"\u002Fmaterials\u002Fcs336\u002Fexperiments\u002Fcs336-05-rope-mha-20261021.zip",{"slug":2260,"path":2261,"title":2262,"date":2263,"tags":2264,"description":2269,"cover":2270,"coverAlt":2271,"excerpt":2272,"draft":2110,"hidden":2110,"published":2112,"readingTime":2140},"llm-gateway-semantic-cache-dynamic-batching","\u002Fposts\u002Fllm-gateway-semantic-cache-dynamic-batching","高并发 LLM 推理网关架构实践：两级精确\u002F语义缓存、50ms 有界动态批处理与尾部延迟治理","2026-10-19",[2120,2122,2265,2266,2267,2268],"分布式系统","Redis","FastAPI","后端","深度拆解生产级大模型推理网关 LLM-Gateway 的核心系统设计：基于 SHA-256 与 Redis 向量余弦相似度的两级精确\u002F语义缓存、50ms 有界等待动态批处理调度器（Dynamic Batching）、令牌桶多租户限流与 Prometheus 全链路可观测性，将命中请求 P99 延迟从 5,307 ms 压降至 17 ms。","\u002Fimages\u002Fposts\u002Fllm-gateway-semantic-cache-dynamic-batching\u002Fllm-gateway-cover-20261019.webp","高并发请求流依次穿过精确哈希滤网、向量球面语义缓存层与 50ms 动态批处理时间窗拼装器的等距系统架构插画","在将大语言模型接入高并发业务系统时，如果让上游微服务直接裸连底层推理引擎（vLLM \u002F SGLang 或外部模型 API），系统很快会暴露出三个典型的生产痛点： 1. 高频重复与近义查询消耗巨量算力：大量用户请求在语义上高度等价（例",{"slug":2274,"path":2275,"title":2276,"date":2263,"tags":2277,"description":2279,"cover":2280,"coverAlt":2281,"excerpt":2282,"draft":2110,"hidden":2110,"published":2112,"readingTime":2216,"series":2141,"seriesOrder":1459,"codeUrl":2283,"attachmentUrl":2284},"cs336-multihead-attention-shapes","\u002Fposts\u002Fcs336-multihead-attention-shapes","Stanford CS336 学习记录 04：多头注意力怎样拆头、计算和合头",[2132,1734,2193,2252,2278],"张量形状","沿着 (2, 5, 64) 的输入追踪 QKV 投影、reshape、transpose、共享因果 mask 和合头，理解每个头仍处理完整序列，以及最后为什么还需要输出 Linear。","\u002Fimages\u002Fposts\u002Fcs336-multihead-attention-shapes\u002Fcs336-multihead-attention-shapes-cover.webp","宽向量束分成多条并行特征轨道后重新合并的 AI 辅助概念插画","上一篇完成了缩放点积注意力。那个函数接收已经准备好的 Q、K、V，计算匹配分数，应用 mask，再用权重汇总 V。这一篇继续把它放进一个带参数的多头自注意力模块。 这次最需要看清的是“拆头”。代码里只是几行 reshape 和 tr","https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fcs336-labs\u002Ftree\u002F31363223498fafd1528c550a51efbe0aed231736","\u002Fmaterials\u002Fcs336\u002Fexperiments\u002Fcs336-04-multihead-attention-20261019.zip",{"slug":2286,"path":2287,"title":2288,"date":2289,"tags":2290,"description":2294,"cover":2295,"coverAlt":2296,"excerpt":2297,"draft":2110,"hidden":2110,"published":2112,"readingTime":2298},"qwen25-sft-dpo-alignment-tax-ablation","\u002Fposts\u002Fqwen25-sft-dpo-alignment-tax-ablation","大模型后训练偏好对齐实录：Qwen2.5 SFT 多维消融、动态参考 DPO 与对齐税（Alignment Tax）量化分析","2026-10-18",[2240,2226,2291,2292,2227,2293],"SFT","LoRA","Qwen2.5","基于 Qwen2.5 完整复现 Instruct SFT → Bradley-Terry DPO 两阶段后训练管线：通过 3 随机种子消融 LoRA Rank、学习率与数据规模（GSM8K +12.73pp），实现免双倍权重的动态 Adapter 切换 DPO 训练，并定量揭示偏好对齐对确定性数学推理的对齐税效应。","\u002Fimages\u002Fposts\u002Fqwen25-sft-dpo-alignment-tax-ablation\u002Fqwen25-sft-dpo-cover-20261018.webp","左侧监督微调轨迹流经中央 Bradley-Terry 偏好分流天平分离出正负样本通道并连接右侧对齐税监控仪表盘的等距技术插画","在大语言模型后训练（PostTraining）管线中，监督微调（SFT）负责教会模型任务格式与领域推理模式，而直接偏好优化（DPO, Direct Preference Optimization）则利用成对偏好数据 $(x, yw,",9,{"slug":2300,"path":2301,"title":2302,"date":2289,"tags":2303,"description":2305,"cover":2306,"coverAlt":2307,"excerpt":2308,"draft":2110,"hidden":2110,"published":2112,"readingTime":2309,"series":2141,"seriesOrder":1446,"codeUrl":2310,"attachmentUrl":2311},"cs336-softmax-attention-mask","\u002Fposts\u002Fcs336-softmax-attention-mask","Stanford CS336 学习记录 03：Softmax 的数值稳定与 Attention Mask",[2132,1734,2193,2304,2252],"Softmax","从减去最大值为何不改变 Softmax 开始，理清函数与有参数层的区别，再沿着 Q、K、V 的形状理解缩放点积注意力，以及 mask 为什么必须填负无穷。","\u002Fimages\u002Fposts\u002Fcs336-softmax-attention-mask\u002Fcs336-softmax-attention-mask-cover.webp","注意力分数、权重与因果遮罩的 AI 辅助概念插画","上一篇写到 SwiGLU，接下来进入 Attention。真正开始相乘之前，先做了 Softmax：它把一组分数变成总和为 1 的权重，后面用这些权重汇总不同 token 提供的信息。 这段学习里，有两个很具体的问题值得记下来。一个",11,"https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fcs336-labs\u002Ftree\u002Fa39eb684710817f7f389626423fcb26ca0f6d629","\u002Fmaterials\u002Fcs336\u002Fexperiments\u002Fcs336-03-softmax-sdpa-20261018.zip",{"slug":2313,"path":2314,"title":2315,"date":2316,"tags":2317,"description":2319,"cover":2320,"coverAlt":2321,"excerpt":2322,"draft":2110,"hidden":2110,"published":2112,"readingTime":2298,"series":2141,"seriesOrder":41,"codeUrl":2310,"attachmentUrl":2323},"cs336-rmsnorm-swiglu","\u002Fposts\u002Fcs336-rmsnorm-swiglu","Stanford CS336 学习记录 02：RMSNorm 的广播与 SwiGLU 的门控","2026-10-16",[2132,1734,2193,555,2318],"SwiGLU","用一个 token 向量理解均方根归一化，拆开 dim=-1 与 keepdim=True 的作用，再沿着两条分支读懂 SiLU、逐元素乘法和 SwiGLU。","\u002Fimages\u002Fposts\u002Fcs336-rmsnorm-swiglu\u002Fcs336-rmsnorm-swiglu-cover.webp","特征尺度整理和双分支门控的 AI 辅助概念插画","上一篇把 Linear 和 Embedding 的参数形状理顺了。继续往下，最先遇到的是 RMSNorm：公式不算长，但把它落到一批 token 上时，dim=1、keepdim=True 和广播必须一起理解。 然后是 SiLU 和","\u002Fmaterials\u002Fcs336\u002Fexperiments\u002Fcs336-02-rmsnorm-swiglu-20261016.zip",{"slug":2325,"path":2326,"title":2327,"date":2328,"tags":2329,"description":2332,"cover":2333,"coverAlt":2334,"excerpt":2335,"draft":2110,"hidden":2110,"published":2112,"readingTime":2336,"series":2141,"seriesOrder":30,"codeUrl":2310,"attachmentUrl":2337},"cs336-linear-embedding-shapes","\u002Fposts\u002Fcs336-linear-embedding-shapes","Stanford CS336 学习记录 01：从 Linear、Embedding 搞清张量形状","2026-10-14",[2132,1734,2193,2330,2331],"Linear","Embedding","从 PyTorch 类、参数与 forward 的关系开始，理解 Linear 为什么保存两行三列的权重、计算时为什么要转置，再区分 Embedding 查表与线性变换。","\u002Fimages\u002Fposts\u002Fcs336-linear-embedding-shapes\u002Fcs336-linear-embedding-shapes-cover.webp","token 方块、向量表与矩阵变换的 AI 辅助概念插画","开始沿着 Stanford CS336 的 Assignment 1 一步步实现。BPE 之前已经接触过，这轮先跳过分词器，从 Transformer 的基础层开始。这里记录的是当时写代码时真正卡住的问题，以及对应测试能确认到的范围",10,"\u002Fmaterials\u002Fcs336\u002Fexperiments\u002Fcs336-01-linear-embedding-20261014.zip",{"slug":2339,"path":2340,"title":2341,"date":2342,"tags":2343,"description":2348,"cover":2349,"coverAlt":2350,"excerpt":2351,"draft":2110,"hidden":2110,"published":2112,"readingTime":2352,"series":2344,"seriesOrder":2216,"codeUrl":2353,"attachmentUrl":2354},"cmu-15-779-kv-cache-granularity-study","\u002Fposts\u002Fcmu-15-779-kv-cache-granularity-study","CMU 15-779 学习记录 14：KV 页大小、共享边界与 CPU 实现成本","2026-10-12",[2344,2345,2346,2347],"CMU 15-779","KV cache","PagedAttention","系统实验","用 210 组 CPU 配置比较连续与分页 KV cache，观察 31\u002F32-token 共享前缀的页边界，分别记录活跃槽位、池容量、复制计数和完整调用耗时。","\u002Fimages\u002Fposts\u002Fcmu-15-779-kv-serving-speculative-sampling\u002Fcmu-serving-cover-20261019.webp","浅色手绘分页存储抽屉与共享前缀卡片，旁边连接不同长度的请求队列","个人课程项目中的 CPU 实验记录，延续记录 11 的分页 KV 实现。已完成 210 组配置与 1470 次计时，输出均通过连续存储参考对照；本篇不代表正式作业提交、整门课程或项目全部交付完成。 同样是共享 31 个 token，",12,"https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fcmu-15-779-labs\u002Ftree\u002F200faf569eccf94fb125e65110bb5d14094d454a","\u002Fmaterials\u002Fcmu-15-779\u002Fexperiments\u002Fkv-cache-granularity-20261019.zip",{"slug":2356,"path":2357,"title":2358,"date":2359,"tags":2360,"description":2363,"cover":2364,"coverAlt":2365,"excerpt":2366,"draft":2110,"hidden":2110,"published":2112,"readingTime":2257,"series":2344,"seriesOrder":2367,"codeUrl":2368,"attachmentUrl":2369},"cmu-15-779-moe-routing-capacity","\u002Fposts\u002Fcmu-15-779-moe-routing-capacity","CMU 15-779 学习记录 13：MoE 路由、稀疏分发与容量溢出","2026-10-09",[2344,2361,2362],"MoE","Routing","用 CPU top-2 MoE 检查稀疏分发的输出与梯度，统计 expert-token 行数，并保留容量丢弃会改变输出的反例。","\u002Fimages\u002Fposts\u002Fcmu-15-779-moe-routing-capacity\u002Fcmu-moe-cover-20261019.webp","浅色手绘分拣器将小珠分配到四个专家工作屋，队列呈现负载不均","第 18 讲的个人配套实验。36 号脚本串行运行 CPU 专家，检查 topk 路由后的分发与合并；没有训练 router，也没有执行 GPU grouped GEMM 或跨设备 alltoall。 上一节记录低秩适配，这次转向每个",13,"https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fcmu-15-779-labs\u002Ftree\u002Fc4851e0cedadf13af7c421867e83c8ff57244fa0","\u002Fmaterials\u002Fcmu-15-779\u002Fexperiments\u002Fcpu-parallel-serving-lora-moe-20261019.zip",{"slug":2371,"path":2372,"title":2373,"date":2374,"tags":2375,"description":2377,"cover":2378,"coverAlt":2379,"excerpt":2380,"draft":2110,"hidden":2110,"published":2112,"readingTime":2257,"series":2344,"seriesOrder":2352,"codeUrl":2368,"attachmentUrl":2369},"cmu-15-779-lora-uniform-quantization","\u002Fposts\u002Fcmu-15-779-lora-uniform-quantization","CMU 15-779 学习记录 12：LoRA 首步梯度、权重合并与均匀 4-bit 打包","2026-10-06",[2344,2292,2376],"Quantization","用一次 CPU SGD 更新检查 LoRA 初始化、冻结基座与合并等价，再分别记录均匀 4-bit 存储字节数和量化误差。","\u002Fimages\u002Fposts\u002Fcmu-15-779-lora-uniform-quantization\u002Fcmu-lora-cover-20261019.webp","浅色手绘的大计算板、两块细长适配板和成对打包的小方块","第 17 讲的个人配套实验，35 号脚本实际执行一次 CPU LoRA 更新，并实现均匀 4bit 打包。实验检查机制与数值误差，不评估微调任务效果，也没有实现完整 QLoRA。 这次把两个问题分开记录：低秩分支是否能合并进原权重，",{"slug":2382,"path":2383,"title":2384,"date":2385,"tags":2386,"description":2390,"cover":2349,"coverAlt":2391,"excerpt":2392,"draft":2110,"hidden":2110,"published":2112,"readingTime":2171,"series":2344,"seriesOrder":2309,"codeUrl":2368,"attachmentUrl":2369},"cmu-15-779-kv-serving-speculative-sampling","\u002Fposts\u002Fcmu-15-779-kv-serving-speculative-sampling","CMU 15-779 学习记录 11：分页 KV、连续批处理与推测采样","2026-10-03",[2344,2387,2388,2389],"LLM Serving","KV Cache","Speculative Decoding","在 CPU 上验证 KV 页共享和写时复制，用单位成本模型检查连续批处理，再用条件概率小模型核对推测采样的接受与补采样。","浅色手绘的共享卡片抽屉与不断补入小珠子的请求托盘","第 15、16 讲的个人配套实验。33 号实现 CPU 分页 KV 存储，37 号模拟请求准入，34 号运行小概率模型的推测采样。它们分别验证存储、调度和分布问题，没有测量真实 LLM 服务吞吐或 GPU 延迟。 记录 08已经对照",{"slug":2394,"path":2395,"title":2396,"date":2397,"tags":2398,"description":2403,"cover":2404,"coverAlt":2405,"excerpt":2406,"draft":2110,"hidden":2110,"published":2112,"readingTime":2407},"qwen3-mid-post-train-mlx","\u002Fposts\u002Fqwen3-mid-post-train-mlx","在 M2 Pro 上完成 Qwen3 两阶段训练：Mid Train、SFT 与可复核评估","2026-09-30 11:19:10",[2223,2399,2400,2292,2401,2291,2402],"Qwen3","MLX","持续预训练","Apple Silicon","用 Qwen3 0.6B Base 和 MLX 完成领域持续训练与监督微调，记录数据划分、聊天模板、损失掩码、数值稳定性修复和独立测试结果。","\u002Fimages\u002Fposts\u002Fqwen3-mid-post-train-mlx\u002Fqwen3-mlx-cover-20261018.webp","Apple Silicon 统一内存芯片底座支撑两阶段持续预训练与带精确 Loss 掩码滤网的监督微调桥接插画","在一台 M2 Pro、16 GB 内存的 Mac 上，能否把一个 Base 语言模型接续训练成输出 JSON 的运维事件分级助手？这次项目完成了领域持续训练、监督微调、检查点选择、重新加载和独立测试，并保留了权重、日志和数据哈希。 ",16,{"slug":2409,"path":2410,"title":2411,"date":2412,"tags":2413,"description":2417,"cover":2418,"coverAlt":2419,"excerpt":2420,"draft":2110,"hidden":2110,"published":2112,"readingTime":2200,"series":2344,"seriesOrder":2336,"codeUrl":2368,"attachmentUrl":2369},"cmu-15-779-cpu-parallelism-pipeline","\u002Fposts\u002Fcmu-15-779-cpu-parallelism-pipeline","CMU 15-779 学习记录 10：加权梯度、状态分片与同步流水线","2026-09-30",[2344,2414,2415,2416],"Parallel Training","Gloo","Pipeline","用双 CPU 进程 Gloo 检查不等数据分片的梯度，推导持久状态字节数，再用依赖模型比较 GPipe 与同步 1F1B 的激活存活。","\u002Fimages\u002Fposts\u002Fcmu-15-779-cpu-parallelism-pipeline\u002Fcmu-parallel-cover-20261019.webp","浅色手绘的四阶段桌面流水线与按数量汇总的托盘","第 12 至 14 讲的个人配套实验，脚本编号 31、32 不对应官方作业。Gloo 实际运行两个 CPU 进程；张量切分、状态字节数和流水线分别按下文的数学或成本模型解释。真实 GPU 实测按学习安排跳过。 上一节检查单个计算图的",{"slug":2422,"path":2423,"title":2424,"date":2425,"tags":2426,"description":2429,"cover":2430,"coverAlt":2431,"excerpt":2432,"draft":2110,"hidden":2110,"published":2112,"readingTime":2171,"series":2344,"seriesOrder":2298,"codeUrl":2433,"attachmentUrl":2434},"cmu-15-779-cpu-compiler-optimizations","\u002Fposts\u002Fcmu-15-779-cpu-compiler-optimizations","CMU 15-779 学习记录 09：CPU 编译优化、合法重写与候选搜索","2026-09-27",[2344,2427,2428,1734],"ML Compiler","TorchInductor","用 CPU 实验验证 BatchNorm 折叠、RMSNorm 重排与非线性反例，测量分块候选，并运行 TorchInductor，分别记录首调用与预热后的时间。","\u002Fimages\u002Fposts\u002Fcmu-15-779-cpu-compiler-optimizations\u002Fcmu-cpu-compiler-cover-20261019.webp","淡紫计算积木经过薄荷绿折叠模具变成浅蓝积木，前方摆放四种候选托盘的浅色手绘插画","编译章节的个人配套实验，编号 30 不对应官方作业。图重写、分块候选和 TorchInductor 均在 CPU 上运行；真实 GPU 实验按学习安排跳过。 上一节记录了分块 Attention 和 KV cache。这次转到编译优","https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fcmu-15-779-labs\u002Ftree\u002F9261bea6aedf31cdccbbab2541ca693fc97b3743","\u002Fmaterials\u002Fcmu-15-779\u002Fexperiments\u002Fcpu-compiler-optimizations-20261019.zip",{"slug":2436,"path":2437,"title":2438,"date":2439,"tags":2440,"description":2442,"cover":2443,"coverAlt":2444,"excerpt":2445,"draft":2110,"hidden":2110,"published":2112,"readingTime":2352,"series":2344,"seriesOrder":2140,"codeUrl":2353,"attachmentUrl":2446},"cmu-15-779-attention-online-softmax","\u002Fposts\u002Fcmu-15-779-attention-online-softmax","CMU 15-779 学习记录 08：分块 Attention、在线 Softmax 与 KV cache","2026-09-24",[2344,2252,2304,2441,1734],"FlashAttention","在 CPU 上对照完整 Attention、在线 Softmax 分块实现与 PyTorch SDPA，检查因果掩码、浮点误差和分数块大小，理解 FlashAttention 的一部分算法基础。","\u002Fimages\u002Fposts\u002Fcmu-15-779-attention-online-softmax\u002Fcmu-online-attention-cover-20261016.webp","浅紫色卡片分批进入绿色计算托盘，旁边三个状态容器保存累积量的浅色手绘插画","第 5 讲 Transformers & Attention 的个人配套实验。28\u002F29 号是 CPU 单头前向与 KV cache；后续补入 38 号 CUDA 多头合成张量的显式 SDPA MATH\u002FFLASH 对照。没有手写 ","\u002Fmaterials\u002Fcmu-15-779\u002Fexperiments\u002Fcuda-triton-sdpa-20261019.zip",{"slug":2448,"path":2449,"title":2450,"date":2451,"tags":2452,"description":2456,"cover":2457,"coverAlt":2458,"excerpt":2459,"draft":2110,"hidden":2110,"published":2112,"readingTime":2336,"series":2344,"seriesOrder":2171,"codeUrl":2353,"attachmentUrl":2446},"cmu-15-779-matmul-tiling-tree-reduction","\u002Fposts\u002Fcmu-15-779-matmul-tiling-tree-reduction","CMU 15-779 学习记录 07：矩阵分块、数据复用与树形归约","2026-09-21",[2344,1737,2453,2454,2455],"矩阵乘法","数据复用","归约","用 CPU 模型统计矩阵分块的输入读取次数，再比较串行求和和树形归约的依赖深度，记录补零、同步与浮点加法顺序的边界。","\u002Fimages\u002Fposts\u002Fcmu-15-779-matmul-tiling-tree-reduction\u002Fcmu-matmul-reduction-cover-20261016.webp","矩阵积木在共享托盘中复用，旁边的小珠沿树形管道逐层合并的浅色手绘插画","GPU Architecture & CUDA Programming 的个人配套实验。首次记录用 CPU 模型检查数据复用与依赖深度；后续补入单卡 GPU 原生 CUDA 矩阵验证，以及 Triton 两阶段归约和矩阵乘法实测。 ",{"slug":2461,"path":2462,"title":2463,"date":2464,"tags":2465,"description":2468,"cover":2469,"coverAlt":2470,"excerpt":2471,"draft":2110,"hidden":2110,"published":2112,"readingTime":2336,"series":2344,"seriesOrder":2200,"codeUrl":2353,"attachmentUrl":2446},"cmu-15-779-cuda-thread-indexing","\u002Fposts\u002Fcmu-15-779-cuda-thread-indexing","CMU 15-779 学习记录 06：CUDA 线程索引与边界检查","2026-09-18",[2344,1737,2121,2466,2467],"线程索引","边界检查","用 CPU 演示 CUDA 的 block、thread 与全局下标：10 个元素如何分给 3 个线程块，为什么要跳过越界线程，以及这些验证能说明什么。","\u002Fimages\u002Fposts\u002Fcmu-15-779-cuda-thread-indexing\u002Fcmu-cuda-indexing-cover-20261015.webp","分组的浅紫色计算方块与末端边界挡板组成线程分配教具的浅色插画","GPU Architecture & CUDA Programming 的个人配套练习。首次记录为 CPU 索引演示；后续在单卡 GPU 环境补入原生 CUDA 编译、运行与 memcheck，以及 Triton 向量加法实测。 上",{"slug":2473,"path":2474,"title":2475,"date":2476,"tags":2477,"description":2481,"cover":2482,"coverAlt":2483,"excerpt":2484,"draft":2110,"hidden":2110,"published":2112,"readingTime":2336,"series":2344,"seriesOrder":2257,"codeUrl":2485,"attachmentUrl":2486},"cmu-15-779-profiler-operator-batching","\u002Fposts\u002Fcmu-15-779-profiler-operator-batching","CMU 15-779 学习记录 05：Profiler、算子调用与批量推理","2026-09-15",[2344,1734,2478,2479,2480],"Profiler","批量推理","性能分析","对照 ML Systems 基础，用 Profiler 拆开两层网络的执行过程，再固定 1000 个样本扫描 batch size，观察算子调用次数、尾批和 CPU 耗时。","\u002Fimages\u002Fposts\u002Fcmu-15-779-profiler-operator-batching\u002Fcmu-pytorch-profiling-cover-20261015.webp","放大镜观察线性层与 ReLU 模块，旁边面板呈现执行片段的浅色技术教具插画","本篇对照课程官网的 Introduction to ML Systems 整理，是个人配套 CPU 实验，不是课程官方作业。脚本编号是练习顺序，不对应课程讲次。 前几篇一直在补 PyTorch 基础。到了这一步，我在学习聊天里问：现","https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fcmu-15-779-labs\u002Ftree\u002Fcecf5110d8a95161bb5763eadfca94cd7a0cc50f","\u002Fmaterials\u002Fcmu-15-779\u002Fexperiments\u002Fpytorch-profiler-batching-20261015.zip",{"slug":2488,"path":2489,"title":2490,"date":2491,"tags":2492,"description":2495,"cover":2496,"coverAlt":2497,"excerpt":2498,"draft":2110,"hidden":2110,"published":2112,"readingTime":2298,"series":2344,"seriesOrder":1459,"codeUrl":2499,"attachmentUrl":2500},"cmu-15-779-pytorch-linear-relu-absolute-value","\u002Fposts\u002Fcmu-15-779-pytorch-linear-relu-absolute-value","CMU 15-779 学习记录 04：多输出线性层、ReLU 与绝对值网络","2026-09-12",[2344,1734,2453,2493,2494],"ReLU","神经网络","把 nn.Linear 对应到矩阵乘法，弄清多个输出怎样计算，再逐层拆开一个绝对值网络，比较手动设置参数与训练得到的结果。","\u002Fimages\u002Fposts\u002Fcmu-15-779-pytorch-linear-relu-absolute-value\u002Fcmu-pytorch-absolute-cover-20261015.webp","双分支经过折角模块后汇入紫色 V 形曲线面板的浅色技术教具插画","本篇整理的是学习 CMU 15779 时补充的 PyTorch 预备练习。脚本编号不对应课程讲次，也不是课程官方作业。 接着前一篇的批量推理与矩阵乘法，我继续把 model(x) 拆开看：一层里有多个输出时，每个数是怎么算出来的？两","https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fcmu-15-779-labs\u002Ftree\u002F4cc63904a3a583da4a2122c246a14b43aac38528","\u002Fmaterials\u002Fcmu-15-779\u002Fexperiments\u002Fpytorch-linear-relu-abs-20261015.zip",{"slug":2502,"path":2503,"title":2504,"date":2505,"tags":2506,"description":2509,"cover":2510,"coverAlt":2511,"excerpt":2512,"draft":2110,"hidden":2110,"published":2112,"readingTime":2298,"series":2344,"seriesOrder":1446,"codeUrl":2513,"attachmentUrl":2514},"cmu-15-779-pytorch-dataloader-checkpoints-batching","\u002Fposts\u002Fcmu-15-779-pytorch-dataloader-checkpoints-batching","CMU 15-779 学习记录 03：DataLoader、模型保存与批量推理","2026-09-09",[2344,1734,2507,2479,2508],"DataLoader","模型保存","用 DataLoader 分批训练平方函数，保存并加载模型参数，再比较 1000 个样本与 1 个样本的推理耗时，最后把批量计算对应到矩阵乘法。","\u002Fimages\u002Fposts\u002Fcmu-15-779-pytorch-dataloader-checkpoints-batching\u002Fcmu-pytorch-batching-cover-20261015.webp","样本卡片按批次进入计算模块，旁边配有参数存储盒和输出矩阵面板的技术插画","本篇整理的是学习 CMU 15779 时补充的 PyTorch 预备练习。脚本编号不对应课程讲次，也不是课程官方作业。 接着前一篇的平方函数实验，这一节继续练习分批训练、保存参数和独立预测。最后做了一次计时：同一个小网络，逐个处理 ","https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fcmu-15-779-labs","\u002Fmaterials\u002Fcmu-15-779\u002Fexperiments\u002Fpytorch-batching-20261015.zip",{"slug":2516,"path":2517,"title":2518,"date":2519,"tags":2520,"description":2521,"cover":2522,"coverAlt":2523,"excerpt":2524,"draft":2110,"hidden":2110,"published":2112,"readingTime":2298,"series":2344,"seriesOrder":41,"codeUrl":2513,"attachmentUrl":2525},"cmu-15-779-pytorch-linear-module-relu","\u002Fposts\u002Fcmu-15-779-pytorch-linear-module-relu","CMU 15-779 学习记录 02：nn.Linear、张量形状与 ReLU","2026-09-06",[2344,1734,2494,2493],"从带偏置的线性回归开始，弄清 nn.Linear 和 nn.Module 的分工，再用平方函数观察线性模型的限制、ReLU 的作用和测试点的选择。","\u002Fimages\u002Fposts\u002Fcmu-15-779-pytorch-linear-module-relu\u002Fcmu-pytorch-relu-cover-20261015.webp","输入经过两层神经网络和 ReLU 折角模块，输出曲线拟合结果的技术插画","本篇整理的是学习 CMU 15779 时补充的 PyTorch 预备练习。脚本编号不对应课程讲次，也不是课程官方作业。 上一篇里，我用一个参数拟合了 $y=2x$。这一篇继续顺着往下写，先加上偏置，再把手写参数换成 nn.Linea","\u002Fmaterials\u002Fcmu-15-779\u002Fexperiments\u002Fpytorch-models-20261015.zip",{"slug":2527,"path":2528,"title":2529,"date":2530,"tags":2531,"description":2534,"cover":2535,"coverAlt":2536,"excerpt":2537,"draft":2110,"hidden":2110,"published":2112,"readingTime":2336,"series":2344,"seriesOrder":30,"codeUrl":2513,"attachmentUrl":2538},"cmu-15-779-pytorch-autograd-linear-regression","\u002Fposts\u002Fcmu-15-779-pytorch-autograd-linear-regression","CMU 15-779 学习记录 01：PyTorch 自动求导与线性回归","2026-09-03",[2344,1734,2532,2533],"自动求导","线性回归","从自动求导和手动梯度下降开始，比较学习率，再用 SGD 训练线性回归。记录 w * 4 写错后停在 1.25 的原因和修正结果。","\u002Fimages\u002Fposts\u002Fcmu-15-779-pytorch-autograd-linear-regression\u002Fcmu-pytorch-autograd-cover-20261015.webp","梯度曲面上的小球沿路径下降，前侧面板展示散点与拟合直线的技术插画","本篇整理的是学习 CMU 15779 时补充的 PyTorch 预备练习。脚本编号不对应课程讲次，也不是课程官方作业。 开始学习 CMU 15779 时，我会 Python，但还没用过 PyTorch。这次先在 Mac 上用 CPU","\u002Fmaterials\u002Fcmu-15-779\u002Fexperiments\u002Fpytorch-basics-20261013.zip",{"slug":2540,"path":2541,"title":2542,"date":2543,"tags":2544,"description":2550,"cover":2551,"coverAlt":2552,"excerpt":2553,"draft":2110,"hidden":2110,"published":2112,"readingTime":2367,"codeUrl":2554,"attachmentUrl":2555},"embodied-pickplace-lab-robustness","\u002Fposts\u002Fembodied-pickplace-lab-robustness","Pickplace Lab 项目复盘：抓取成功率之外的时延与失败分析","2026-08-29",[2545,2546,2547,1734,2548,2549],"具身智能","模仿学习","MuJoCo","机器人","鲁棒性","从 MuJoCo 抓取搬运闭环出发，记录 22,100 个评测回合。动作平滑在自建任务的 50 ms 延迟下有效，换到 Panda Lift 却没有支持同样的收益。","\u002Fimages\u002Fposts\u002Fembodied-pickplace-lab-robustness\u002Fpickplace-grasp-cover-20261016.webp","MuJoCo 实际渲染的笛卡尔夹爪抓取方块场景，桌面上标出放置目标","Pickplace Lab 从一个简单任务开始：让夹爪抓起方块，搬到指定位置，放下，再退开。先把仿真、示范采集、训练和闭环评测接起来，随后再问：干净条件下成功的策略，换一种执行条件，还能不能完成任务？ 最值得讨论的结果出现在指令延迟","https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fembodied-pickplace-lab\u002Ftree\u002Fv0.1.0","https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fembodied-pickplace-lab\u002Freleases\u002Fdownload\u002Fv0.1.0\u002Fpickplace-lab-research-v1.zip",{"slug":2557,"path":2558,"title":2559,"date":2560,"tags":2561,"description":2566,"cover":2567,"coverAlt":2568,"excerpt":2569,"draft":2110,"hidden":2110,"published":2112,"readingTime":2570},"qwen25-vl-sglang-verl-single-gpu-lab","\u002Fposts\u002Fqwen25-vl-sglang-verl-single-gpu-lab","从 Mac 到单卡 GPU：Qwen2.5-VL 部署、GRPO 与计数 LoRA 实践","2026-08-19",[2239,2179,2562,2563,2564,2565,2400],"SGLang","verl","GRPO","推理优化","在 M2 Pro 和云端单卡上部署 Qwen2.5-VL-3B，实测连续批处理、前缀缓存与 SGLang 参数调优，再用 verl 完成多模态 GRPO 更新和同条件前后评测，记录有效优化、兼容性问题与未观察到提升的结果。","\u002Fimages\u002Fposts\u002Fqwen25-vl-sglang-verl-single-gpu-lab\u002Fqvl-lab-cover-20261010.webp","笔记本与单卡计算模块通过图像和 token 光轨连接的实验台插画","这次实验从一个很具体的问题开始：M2 Pro、16 GB 统一内存，能不能在本机跑 Qwen2.5VL？跑起来之后，我又想把图文问答、推理性能测量和强化学习训练接成一条完整流程。 本机的 MLX 4bit 服务和云端的 SGLang",29,{"slug":2572,"path":2573,"title":2574,"date":2575,"tags":2576,"description":2580,"cover":2581,"coverAlt":2582,"excerpt":2583,"draft":2110,"hidden":2110,"published":2112,"readingTime":2367},"cycleresearcher-post-training","\u002Fposts\u002Fcycleresearcher-post-training","从 DeepReviewer 2.0 到 CycleResearcher：科研 Agent 如何真正做 Post-training","2026-08-07 17:33:05",[2223,2240,2180,2577,2578,2579],"SimPO","Reward Model","ICLR","区分推理时 Agent 编排与模型后训练，并拆解 CycleResearcher 的领域 SFT、生成式 Reward Model、Iterative SimPO 和评测风险。","\u002Fimages\u002Fposts\u002Fcycleresearcher-post-training\u002Fcycleresearcher-cover-20261018.webp","策略生成模块产出多份候选文稿经多维评审棱镜排序后通过偏好梯度反馈弧闭环优化的等距技术插画","看到一个系统会检索、规划、反思和调用工具，不能据此判断它做过 Posttraining。Agent 描述的是模型在运行时如何完成任务；Posttraining 描述的是预训练之后，是否继续用监督数据、偏好数据或奖励信号更新模型参数。",{"slug":2585,"path":2114,"title":5,"date":2108,"tags":2586,"description":2109,"cover":2106,"coverAlt":2107,"excerpt":2587,"draft":2110,"hidden":2110,"published":2112,"readingTime":2216,"series":2117,"seriesOrder":30},"cuda-fused-rmsnorm-kernel",[1737,2120,2121,1734,2122,2123],"为了准备 AI Infra 岗位，我做了一个尽量小而完整的 CUDA 项目：不追求堆很多算子，而是选一个真实的 LLM 推理路径，把实现、正确性、benchmark 和性能解释全部做扎实。 项目实现了两个 PyTorch CUDA ",{"slug":2589,"path":2590,"title":2591,"date":2592,"tags":2593,"description":2595,"cover":2596,"coverAlt":2597,"excerpt":2598,"draft":2110,"hidden":2110,"published":2112,"readingTime":2140,"series":2599,"seriesOrder":30},"agent-memory","\u002Fposts\u002Fagent-memory","Agent 上下文工程与分层记忆架构：从 Observation 剪枝、滚动摘要到向量召回闭环","2026-04-24",[2223,2180,2594,2120],"RAG","结合生产级多约束 Agent 与超 100K Token 科研智能体实践，系统拆解长程会话中的上下文膨胀与注意力稀释问题：涵盖 Observation 结构化剪枝（降低约 40% Token 开销）、滚动工作摘要、向量检索长期记忆及 Prefix Cache 友好的上下文组装。","\u002Fimages\u002Fposts\u002Fagent-memory\u002Fagent-memory-cover-20261018.webp","底层高噪数据经剪枝滤网进入中期滑动窗口与金色状态台账并由顶层向量星座按需召回的三层记忆架构插画","在构建多步工具调用（ToolUse）与长周期自主规划（Autonomous Planning）的 Agent 系统时，上下文管理（Context Engineering）往往比单纯改写 System Prompt 更决定线上稳定性。","Agent 上下文工程",{"slug":2601,"path":2602,"title":2603,"date":2604,"tags":2605,"description":2608,"cover":2609,"coverAlt":2610,"excerpt":2611,"draft":2110,"hidden":2110,"published":2112,"readingTime":2257,"series":2612,"seriesOrder":41},"backend-stack-deep-dive","\u002Fposts\u002Fbackend-stack-deep-dive","高并发 AI 平台后端基础设施实践：FastAPI 异步网关、SQLAlchemy 2.0 Async、PostgreSQL 与 Docker 沙箱编排","2026-04-07",[2268,2265,2267,2606,2607,2120],"PostgreSQL","Docker","结合 DeepScientist 生产级 AI 科研平台实践，拆解 FastAPI ASGI 异步事件循环、SQLAlchemy 2.0 + AsyncPG 连接池调优、PostgreSQL + MinIO 存储分层与 Docker SDK 动态沙箱容器编排的职责边界与核心设计。","\u002Fimages\u002Fposts\u002Fbackend-stack-deep-dive\u002Fbackend-stack-cover-20261018.webp","ASGI 异步事件循环涡轮将非阻塞协程流分发至 PostgreSQL 连接池、MinIO 存储矩阵与动态 Docker 沙箱的等距架构插画","结合 DeepScientist 生产级 AI 科研平台（服务 7,000+ 用户）的实际工程落地，本文系统拆解支撑高并发流式推理与容器沙箱调度的后端基础设施栈——FastAPI（ASGI 异步网关 + SSE 流式推送）、SQLA","DeepScientist 工程复盘",{"slug":2614,"path":2615,"title":2616,"date":2617,"tags":2618,"description":2621,"cover":2622,"coverAlt":2623,"excerpt":2624,"draft":2110,"hidden":2110,"published":2112,"readingTime":2309,"series":2625,"seriesOrder":30},"minicode-source-analysis","\u002Fposts\u002Fminicode-source-analysis","MiniCode 架构与工具评测深度拆解：开源终端 Agentic Coding 引擎（1k+ Stars）的 ReAct、MCP 与 30 项基准验证","2026-04-06",[2180,2619,2620,2211,2120],"TypeScript","MCP","深度剖析开源终端 AI 编程智能体 MiniCode（1k+ GitHub Stars）的核心架构：从零重型依赖的 ReAct 规划循环、声明式工具注册表与权限拦截层、MCP 协议帧自动协商，到 30 项可执行 Tool-Use 评测基准（86.7% 完成率、100% 错误恢复率）的失败分布归因与闭环优化。","\u002Fimages\u002Fposts\u002Fminicode-source-analysis\u002Fminicode-cover-20261018.webp","终端编程智能体控制台被 ReAct 状态环、Unified Diff 补丁预览门与 MCP 协议插槽环绕的等距技术插画","MiniCode 是我作为核心开发者参与构建的开源终端 Agentic Coding 引擎（目前已在 GitHub 获得 1,000+ Stars）。项目的核心设计目标是：在不引入庞大运行时框架的前提下，用约 5,000 行高内聚的","AI 项目源码导读",{"slug":2627,"path":2628,"title":2629,"date":2617,"tags":2630,"description":2631,"cover":2632,"coverAlt":2633,"excerpt":2634,"draft":2110,"hidden":2110,"published":2112,"readingTime":2336,"series":2612,"seriesOrder":30},"deepscientist-tech-stack","\u002Fposts\u002Fdeepscientist-tech-stack","DeepScientist 架构与运行时全解析：超 100K 上下文科研智能体、MCP 沙箱与高并发流式系统",[2180,2120,2620,2607,2267,2265],"深度拆解服务 7,000+ 用户的科研自主智能体平台 DeepScientist：从一题一仓库（Quest）持久化状态机、MCP 工具栈（memory \u002F artifact \u002F bash_exec）、Docker SDK 动态容器编排，到 SSE + WebSocket 双流通信与 PostgreSQL \u002F Redis \u002F Prometheus 生产级基础设施。","\u002Fimages\u002Fposts\u002Fdeepscientist-tech-stack\u002Fdeepscientist-cover-20261018.webp","科研智能体编排核心通过 memory、artifact 与隔离容器沙箱三座协议桥连接双流工作台的等距架构插画","DeepScientist（deepscientist.cc）是我们在西湖大学自然语言处理实验室研发的自主科学发现与科研管理平台，目前累计服务 7,000+ 科研用户。与单轮对话或纯文本文献问答工具不同，科研智能体需要在数小时乃至数",1795088943943]