[{"data":1,"prerenderedAt":1760},["ShallowReactive",2],{"post-\u002Fposts\u002Fcs336-bpe-tokenizer-training-encoding":3,"archive-public-posts":1260},{"id":4,"title":5,"body":6,"categories":1237,"cover":1239,"coverAlt":1240,"date":1241,"description":1242,"draft":1243,"extension":1244,"haloPublished":1243,"haloSync":1243,"hidden":1243,"legacy":1243,"meta":1245,"navigation":1246,"path":1247,"published":1243,"readingTime":1248,"seo":1249,"series":1250,"seriesOrder":1251,"stem":1252,"tags":1253,"__hash__":1259},"posts\u002Fposts\u002Fcs336-bpe-tokenizer-training-encoding.md","Stanford CS336 学习记录 07：Byte-Level BPE 分词器训练、倒排索引加速与流式编码",{"type":7,"value":8,"toc":1223},"minimark",[9,27,41,54,61,151,182,186,197,200,205,212,229,341,355,363,370,497,527,531,546,564,580,583,623,820,842,849,956,974,984,1067,1148,1151,1162,1170,1180,1192,1195,1219],[10,11,12,13,17,18,22,23,26],"p",{},"前面六篇完成了 Transformer 语言模型本身的全部神经网络组件。要让模型真正处理自然语言文本，还需要打通入口处的 ",[14,15,16],"strong",{},"Byte-Level BPE（Byte-Pair Encoding）分词器","。这一篇记录 ",[19,20,21],"code",{},"cs336_basics\u002Fbpe.py","（BPE 训练）与 ",[19,24,25],{},"cs336_basics\u002Ftokenizer.py","（编码、解码与流式迭代）的完整实现与性能优化。",[10,28,29,36,37],{},[30,31,35],"a",{"href":32,"rel":33},"https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fcs336-labs\u002Ftree\u002Fb5c93ad0cc73178b2b2a62e518a0900da47a0fb7",[34],"nofollow","查看代码仓库"," · ",[30,38,40],{"href":39},"\u002Fmaterials\u002Fcs336\u002Fexperiments\u002Fcs336-07-bpe-tokenizer-20261025.zip","下载本节代码与测试 ZIP",[42,43,45,46,49,50,53],"h2",{"id":44},"为什么选择-utf-8-byte-level-bpeproblem-unicode1-unicode2","为什么选择 UTF-8 Byte-Level BPE（Problem ",[19,47,48],{},"unicode1"," & ",[19,51,52],{},"unicode2","）",[10,55,56,57,60],{},"直接按 Unicode 码点建词表会遇到两个致命问题：Unicode 17.0 定义了近 16 万个字符，初始词表过大且长尾字符极度稀疏；一旦遇到训练未见的字符就会退化为 ",[19,58,59],{},"\u003Cunk>","（Out-of-Vocabulary）。",[10,62,63,64,67,68,142,143,146,147,150],{},"而将文本编码为 ",[14,65,66],{},"UTF-8 字节序列","后，任何的字符串都可以表示为 ",[69,70,73,103],"span",{"className":71},[72],"katex",[69,74,77],{"className":75},[76],"katex-mathml",[78,79,81],"math",{"xmlns":80},"http:\u002F\u002Fwww.w3.org\u002F1998\u002FMath\u002FMathML",[82,83,84,98],"semantics",{},[85,86,87,91,95],"mrow",{},[88,89,90],"mn",{},"0",[92,93,94],"mo",{},"∼",[88,96,97],{},"255",[99,100,102],"annotation",{"encoding":101},"application\u002Fx-tex","0 \\sim 255",[69,104,108,133],{"className":105,"ariaHidden":107},[106],"katex-html","true",[69,109,112,117,121,126,130],{"className":110},[111],"base",[69,113],{"className":114,"style":116},[115],"strut","height:0.6444em;",[69,118,90],{"className":119},[120],"mord",[69,122],{"className":123,"style":125},[124],"mspace","margin-right:0.2778em;",[69,127,94],{"className":128},[129],"mrel",[69,131],{"className":132,"style":125},[124],[69,134,136,139],{"className":135},[111],[69,137],{"className":138,"style":116},[115],[69,140,97],{"className":141},[120]," 之间的整数序列，基础词表大小固定为 256，天然实现 ",[14,144,145],{},"零 OOV","。相比 UTF-16 或 UTF-32，UTF-8 对英文和常见 ASCII 符号只需 1 个字节，不会引入大量无意义的零字节（",[19,148,149],{},"\\x00","），字节流更加紧凑。",[10,152,153,154,157,158,161,162,165,166,169,170,173,174,177,178,181],{},"需要特别注意解码时的多字节边界问题：在 UTF-8 中，一个汉字或 Emoji 往往由 2～4 个字节组成（例如 ",[19,155,156],{},"🙃"," 对应 4 个字节 ",[19,159,160],{},"\\xf0\\x9f\\x99\\x83","）。如果像讲义反例那样逐字节调用 ",[19,163,164],{},"bytes([b]).decode(\"utf-8\")","，任何非 ASCII 多字节字符都会立即抛出 ",[19,167,168],{},"UnicodeDecodeError","。正确做法是先把所有 token 对应的 ",[19,171,172],{},"bytes"," 拼接成完整字节串 ",[19,175,176],{},"b\"\".join(...)","，最后一次性调用 ",[19,179,180],{},".decode(\"utf-8\", errors=\"replace\")","。",[42,183,185],{"id":184},"预分词pre-tokenization与特殊-token-硬边界隔离","预分词（Pre-tokenization）与特殊 Token 硬边界隔离",[10,187,188,189,192,193,196],{},"如果在整篇语料上直接统计相邻字节对，BPE 会把跨词边界的标点与单词合并（例如把 ",[19,190,191],{},"dog."," 和 ",[19,194,195],{},"dog!"," 合并成完全不同的 token ID），而且每次合并都要扫描整篇文档。",[10,198,199],{},"因此训练和编码的第一步都是先做两级切分：",[201,202,204],"h3",{"id":203},"_1-先按-special-tokens-切分防止跨文档合并","1. 先按 Special Tokens 切分，防止跨文档合并",[10,206,207,208,211],{},"像 ",[19,209,210],{},"\u003C|endoftext|>"," 这样的特殊 token 用来标记文档边界。训练时，它们既不能被拆成普通字节参与频次统计，也不能让前一篇文档的末尾字节与后一篇文档的开头字节跨边界合并。",[10,213,214,215,192,217,220,221,224,225,228],{},"而在分词编码时，如果用户注册了互相包含的重叠特殊 token（例如同时注册 ",[19,216,210],{},[19,218,219],{},"\u003C|endoftext|>\u003C|endoftext|>","，对应官方测试 ",[19,222,223],{},"test_overlapping_special_tokens","），正则切分必须遵循",[14,226,227],{},"最长匹配优先","：",[230,231,236],"pre",{"className":232,"code":233,"language":234,"meta":235,"style":235},"language-python shiki shiki-themes github-dark-dimmed github-light houston","sorted_specials = sorted(self.special_tokens, key=len, reverse=True)\npattern_str = \"(\" + \"|\".join(re.escape(tok) for tok in sorted_specials) + \")\"\n","python","",[19,237,238,288],{"__ignoreMap":235},[69,239,242,246,250,254,257,261,264,268,270,273,276,279,281,285],{"class":240,"line":241},"line",1,[69,243,245],{"class":244},"sWURj","sorted_specials ",[69,247,249],{"class":248},"sf2cM","=",[69,251,253],{"class":252},"svkvy"," sorted",[69,255,256],{"class":244},"(",[69,258,260],{"class":259},"sRGqv","self",[69,262,263],{"class":244},".special_tokens, ",[69,265,267],{"class":266},"sA9aF","key",[69,269,249],{"class":248},[69,271,272],{"class":252},"len",[69,274,275],{"class":244},", ",[69,277,278],{"class":266},"reverse",[69,280,249],{"class":248},[69,282,284],{"class":283},"sgENW","True",[69,286,287],{"class":244},")\n",[69,289,291,294,296,300,303,306,309,313,316,319,322,326,329,332,335,338],{"class":240,"line":290},2,[69,292,293],{"class":244},"pattern_str ",[69,295,249],{"class":248},[69,297,299],{"class":298},"sWqY2"," \"(\"",[69,301,302],{"class":248}," +",[69,304,305],{"class":298}," \"|\"",[69,307,308],{"class":244},".",[69,310,312],{"class":311},"sq3c-","join",[69,314,315],{"class":244},"(re.",[69,317,318],{"class":311},"escape",[69,320,321],{"class":244},"(tok) ",[69,323,325],{"class":324},"slEfO","for",[69,327,328],{"class":244}," tok ",[69,330,331],{"class":324},"in",[69,333,334],{"class":244}," sorted_specials) ",[69,336,337],{"class":248},"+",[69,339,340],{"class":298}," \")\"\n",[10,342,343,344,347,348,351,352,354],{},"先按长度降序排列并用 ",[19,345,346],{},"re.escape"," 转义 ",[19,349,350],{},"|"," 等元字符，才能保证连续的两个 ",[19,353,219],{}," 被优先识别为单个长特殊 token，而不会被短 token 提前截断。",[201,356,358,359,362],{"id":357},"_2-再用-gpt-2-正则-refinditer-提取预分词频次表","2. 再用 GPT-2 正则 ",[19,360,361],{},"re.finditer"," 提取预分词频次表",[10,364,365,366,369],{},"在每个普通文本片段内部，使用 ",[19,367,368],{},"regex"," 库的 GPT-2 预分词模式：",[230,371,373],{"className":232,"code":372,"language":234,"meta":235,"style":235},"PAT = r\"\"\"'(?:[sdmt]|ll|ve|re)| ?\\p{L}+| ?\\p{N}+| ?[^\\s\\p{L}\\p{N}]+|\\s+(?!\\S)|\\s+\"\"\"\n",[19,374,375],{"__ignoreMap":235},[69,376,377,380,383,387,390,394,397,400,404,407,409,412,414,417,419,422,425,427,431,435,438,440,442,444,446,449,451,453,455,457,460,463,465,467,469,471,473,475,477,479,481,484,487,490,492,494],{"class":240,"line":241},[69,378,379],{"class":283},"PAT",[69,381,382],{"class":248}," =",[69,384,386],{"class":385},"sST-V"," r",[69,388,389],{"class":298},"\"\"\"",[69,391,393],{"class":392},"sflY6","'",[69,395,396],{"class":252},"(?:",[69,398,399],{"class":283},"[",[69,401,403],{"class":402},"s0kv9","sdmt",[69,405,406],{"class":283},"]",[69,408,350],{"class":248},[69,410,411],{"class":392},"ll",[69,413,350],{"class":248},[69,415,416],{"class":392},"ve",[69,418,350],{"class":248},[69,420,421],{"class":392},"re",[69,423,424],{"class":252},")",[69,426,350],{"class":248},[69,428,430],{"class":429},"shgEQ"," ?",[69,432,434],{"class":433},"s-Nz2","\\p",[69,436,437],{"class":392},"{L}",[69,439,337],{"class":429},[69,441,350],{"class":248},[69,443,430],{"class":429},[69,445,434],{"class":433},[69,447,448],{"class":392},"{N}",[69,450,337],{"class":429},[69,452,350],{"class":248},[69,454,430],{"class":429},[69,456,399],{"class":283},[69,458,459],{"class":248},"^",[69,461,462],{"class":252},"\\s",[69,464,434],{"class":433},[69,466,437],{"class":402},[69,468,434],{"class":433},[69,470,448],{"class":402},[69,472,406],{"class":283},[69,474,337],{"class":429},[69,476,350],{"class":248},[69,478,462],{"class":252},[69,480,337],{"class":429},[69,482,483],{"class":248},"(?!",[69,485,486],{"class":252},"\\S",[69,488,489],{"class":248},")|",[69,491,462],{"class":252},[69,493,337],{"class":429},[69,495,496],{"class":298},"\"\"\"\n",[10,498,499,500,503,504,503,507,510,511,514,515,518,519,522,523,526],{},"这个正则把英文缩写后缀（",[19,501,502],{},"'s","、",[19,505,506],{},"'ll",[19,508,509],{},"'ve","）、带前导空格的字母串、数字串、标点串以及连续空白切分成独立的 pre-token。注意末尾的 ",[19,512,513],{},"\\s+(?!\\S)|\\s+","：当遇到多行换行或连续空格后跟非空白字符时，它会把最后一个前导空格留给后面的单词（如 ",[19,516,517],{},"\" how\"","），其余前导空白单独成块，这正是与 ",[19,520,521],{},"tiktoken"," 在双换行与尾随换行测试（",[19,524,525],{},"test_encode_special_token_trailing_newlines","）上保持 100% 一致的关键。",[42,528,530],{"id":529},"训练加速从-3-秒到-019-秒的倒排索引增量更新","训练加速：从 3 秒到 0.19 秒的倒排索引增量更新",[10,532,533,534,537,538,541,542,545],{},"官方 ",[19,535,536],{},"test_train_bpe_speed"," 要求在 ",[19,539,540],{},"corpus.en"," 上完成 500 词表训练的耗时必须小于 ",[14,543,544],{},"1.5 秒","。如果每次合并都遍历所有唯一单词重新统计字节对频次，耗时通常在 3 秒左右。",[10,547,548,549,560,561,563],{},"观察发现：",[14,550,551,552,555,556,559],{},"一次合并 ",[19,553,554],{},"(p0, p1) -> new_token","，只会改变那些原本就包含 ",[19,557,558],{},"(p0, p1)"," 的单词的内部字节对计数","，其余 99% 的单词根本不受影响。因此我们在 ",[19,562,21],{}," 中维护两张全局表：",[565,566,567,574],"ol",{},[568,569,570,573],"li",{},[19,571,572],{},"pair_counts: dict[tuple[bytes, bytes], int]","：每个相邻字节对的当前总频次；",[568,575,576,579],{},[19,577,578],{},"pair_to_words: dict[tuple[bytes, bytes], set[int]]","：包含该字节对的唯一单词下标集合（倒排索引）。",[10,581,582],{},"每轮迭代时：",[565,584,585,596,606],{},[568,586,587,588,591,592,595],{},"按频次最高、平局时字典序更大（",[19,589,590],{},"(count, pair)"," 最大）选出 ",[19,593,594],{},"best_pair = (p0, p1)","；",[568,597,598,599,602,603,595],{},"仅从 ",[19,600,601],{},"pair_to_words.pop(best_pair)"," 取出受影响的单词下标 ",[19,604,605],{},"w_idx",[568,607,608,609,612,613,615,616,619,620,181],{},"对每个受影响单词，先将其旧相邻对的计数减去 ",[19,610,611],{},"freq","，再执行 ",[19,614,554],{}," 合并，最后将新单词的相邻对计数加回 ",[19,617,618],{},"pair_counts"," 并更新 ",[19,621,622],{},"pair_to_words",[230,624,626],{"className":232,"code":625,"language":234,"meta":235,"style":235},"for w_idx in affected_word_indices:\n    word = words[w_idx]\n    freq = freqs[w_idx]\n    # 1) 减去该词原有的非 best_pair 相邻对频次\n    for i in range(len(word) - 1):\n        old_pair = (word[i], word[i + 1])\n        if old_pair != best_pair:\n            ...\n    # 2) 在该词内部合并 (p0, p1) -> new_token\n    ...\n    # 3) 将合并后新词的相邻对频次加回全局表\n    for i in range(len(new_word) - 1):\n        new_pair = (new_word[i], new_word[i + 1])\n        pair_counts[new_pair] += freq\n        pair_to_words[new_pair].add(w_idx)\n",[19,627,628,640,650,661,668,698,716,731,737,743,749,755,779,796,808],{"__ignoreMap":235},[69,629,630,632,635,637],{"class":240,"line":241},[69,631,325],{"class":324},[69,633,634],{"class":244}," w_idx ",[69,636,331],{"class":324},[69,638,639],{"class":244}," affected_word_indices:\n",[69,641,642,645,647],{"class":240,"line":290},[69,643,644],{"class":244},"    word ",[69,646,249],{"class":248},[69,648,649],{"class":244}," words[w_idx]\n",[69,651,653,656,658],{"class":240,"line":652},3,[69,654,655],{"class":244},"    freq ",[69,657,249],{"class":248},[69,659,660],{"class":244}," freqs[w_idx]\n",[69,662,664],{"class":240,"line":663},4,[69,665,667],{"class":666},"sCxhR","    # 1) 减去该词原有的非 best_pair 相邻对频次\n",[69,669,671,674,677,679,682,684,686,689,692,695],{"class":240,"line":670},5,[69,672,673],{"class":324},"    for",[69,675,676],{"class":244}," i ",[69,678,331],{"class":324},[69,680,681],{"class":252}," range",[69,683,256],{"class":244},[69,685,272],{"class":252},[69,687,688],{"class":244},"(word) ",[69,690,691],{"class":248},"-",[69,693,694],{"class":283}," 1",[69,696,697],{"class":244},"):\n",[69,699,701,704,706,709,711,713],{"class":240,"line":700},6,[69,702,703],{"class":244},"        old_pair ",[69,705,249],{"class":248},[69,707,708],{"class":244}," (word[i], word[i ",[69,710,337],{"class":248},[69,712,694],{"class":283},[69,714,715],{"class":244},"])\n",[69,717,719,722,725,728],{"class":240,"line":718},7,[69,720,721],{"class":324},"        if",[69,723,724],{"class":244}," old_pair ",[69,726,727],{"class":248},"!=",[69,729,730],{"class":244}," best_pair:\n",[69,732,734],{"class":240,"line":733},8,[69,735,736],{"class":283},"            ...\n",[69,738,740],{"class":240,"line":739},9,[69,741,742],{"class":666},"    # 2) 在该词内部合并 (p0, p1) -> new_token\n",[69,744,746],{"class":240,"line":745},10,[69,747,748],{"class":283},"    ...\n",[69,750,752],{"class":240,"line":751},11,[69,753,754],{"class":666},"    # 3) 将合并后新词的相邻对频次加回全局表\n",[69,756,758,760,762,764,766,768,770,773,775,777],{"class":240,"line":757},12,[69,759,673],{"class":324},[69,761,676],{"class":244},[69,763,331],{"class":324},[69,765,681],{"class":252},[69,767,256],{"class":244},[69,769,272],{"class":252},[69,771,772],{"class":244},"(new_word) ",[69,774,691],{"class":248},[69,776,694],{"class":283},[69,778,697],{"class":244},[69,780,782,785,787,790,792,794],{"class":240,"line":781},13,[69,783,784],{"class":244},"        new_pair ",[69,786,249],{"class":248},[69,788,789],{"class":244}," (new_word[i], new_word[i ",[69,791,337],{"class":248},[69,793,694],{"class":283},[69,795,715],{"class":244},[69,797,799,802,805],{"class":240,"line":798},14,[69,800,801],{"class":244},"        pair_counts[new_pair] ",[69,803,804],{"class":248},"+=",[69,806,807],{"class":244}," freq\n",[69,809,811,814,817],{"class":240,"line":810},15,[69,812,813],{"class":244},"        pair_to_words[new_pair].",[69,815,816],{"class":311},"add",[69,818,819],{"class":244},"(w_idx)\n",[10,821,822,823,826,827,830,831,833,834,837,838,841],{},"需要特别注意平局决胜规则（Tie-breaking）：当多个字节对频次并列第一时，讲义要求选择",[14,824,825],{},"字典序更大","的字节对 ",[19,828,829],{},"max(pairs)","。加上倒排索引后，",[19,832,536],{}," 实测仅需约 ",[14,835,836],{},"0.18s","（远低于 1.5s 上限），生成的词表和 merges 列表与官方 ",[19,839,840],{},"train-bpe-reference-merges.txt"," 逐项完全一致。",[42,843,845,846],{"id":844},"分词器编码按-merge-rank-顺序合并与流式-encode_iterable","分词器编码：按 Merge Rank 顺序合并与流式 ",[19,847,848],{},"encode_iterable",[10,850,851,852,855,856,859,860,952,953,181],{},"在 ",[19,853,854],{},"Tokenizer.encode"," 中，拿到一个 pre-token 的初始单字节序列 ",[19,857,858],{},"[b_0, b_1, ...]"," 后，不能遍历全部 5 万条 merges 规则逐条检查（那会是 ",[69,861,863,899],{"className":862},[72],[69,864,866],{"className":865},[76],[78,867,868],{"xmlns":80},[82,869,870,896],{},[85,871,872,876,879,883,886,888,891,894],{},[873,874,875],"mi",{},"O",[92,877,256],{"stretchy":878},"false",[873,880,882],{"mathvariant":881},"normal","∣",[873,884,885],{},"M",[873,887,882],{"mathvariant":881},[92,889,890],{},"⋅",[873,892,893],{},"L",[92,895,424],{"stretchy":878},[99,897,898],{"encoding":101},"O(|M| \\cdot L)",[69,900,902,939],{"className":901,"ariaHidden":107},[106],[69,903,905,909,914,918,921,925,928,932,936],{"className":904},[111],[69,906],{"className":907,"style":908},[115],"height:1em;vertical-align:-0.25em;",[69,910,875],{"className":911,"style":913},[120,912],"mathnormal","margin-right:0.0278em;",[69,915,256],{"className":916},[917],"mopen",[69,919,882],{"className":920},[120],[69,922,885],{"className":923,"style":924},[120,912],"margin-right:0.109em;",[69,926,882],{"className":927},[120],[69,929],{"className":930,"style":931},[124],"margin-right:0.2222em;",[69,933,890],{"className":934},[935],"mbin",[69,937],{"className":938,"style":931},[124],[69,940,942,945,948],{"className":941},[111],[69,943],{"className":944,"style":908},[115],[69,946,893],{"className":947},[120,912],[69,949,424],{"className":950},[951],"mclose","），而应当预先构建哈希表 ",[19,954,955],{},"merge_ranks = {pair: rank}",[10,957,958,959,962,963,966,967,969,970,973],{},"每次在当前字节序列的相邻对中，找出在 ",[19,960,961],{},"merge_ranks"," 里 ",[14,964,965],{},"rank 最小（即训练时最早生成）"," 的那一对比邻块进行合并，循环直到没有任何相邻对出现在 ",[19,968,961],{}," 中，再配上高频词结果缓存 ",[19,971,972],{},"_word_cache","，就能在瞬间完成整篇文档的编码。",[10,975,976,977,979,980,983],{},"针对超大文件無法一次性塞入内存的场景（对应 ",[19,978,848],{}," 与 1MB 内存上限测试），",[19,981,982],{},"Tokenizer.encode_iterable(self, iterable: Iterable[str]) -> Iterator[int]"," 使用生成器按行\u002F按块惰性产出 token ID：",[230,985,987],{"className":232,"code":986,"language":234,"meta":235,"style":235},"def encode_iterable(self, iterable: Iterable[str]) -> Iterator[int]:\n    for chunk in iterable:\n        if not chunk:\n            continue\n        yield from self.encode(chunk)\n",[19,988,989,1024,1036,1046,1051],{"__ignoreMap":235},[69,990,991,994,998,1000,1003,1005,1009,1012,1015,1018,1021],{"class":240,"line":241},[69,992,993],{"class":385},"def",[69,995,997],{"class":996},"sS_0i"," encode_iterable",[69,999,256],{"class":244},[69,1001,260],{"class":1002},"sbU-9",[69,1004,275],{"class":244},[69,1006,1008],{"class":1007},"s9NhK","iterable",[69,1010,1011],{"class":244},": Iterable[",[69,1013,1014],{"class":252},"str",[69,1016,1017],{"class":244},"]) -> Iterator[",[69,1019,1020],{"class":252},"int",[69,1022,1023],{"class":244},"]:\n",[69,1025,1026,1028,1031,1033],{"class":240,"line":290},[69,1027,673],{"class":324},[69,1029,1030],{"class":244}," chunk ",[69,1032,331],{"class":324},[69,1034,1035],{"class":244}," iterable:\n",[69,1037,1038,1040,1043],{"class":240,"line":652},[69,1039,721],{"class":324},[69,1041,1042],{"class":385}," not",[69,1044,1045],{"class":244}," chunk:\n",[69,1047,1048],{"class":240,"line":663},[69,1049,1050],{"class":324},"            continue\n",[69,1052,1053,1056,1059,1061,1064],{"class":240,"line":670},[69,1054,1055],{"class":324},"        yield from",[69,1057,1058],{"class":259}," self",[69,1060,308],{"class":244},[69,1062,1063],{"class":311},"encode",[69,1065,1066],{"class":244},"(chunk)\n",[10,1068,1069,1070,1073,1074,1140,1141,1143,1144,1147],{},"对于离线语料预处理，为什么讲义推荐将 token ID 序列化为 ",[19,1071,1072],{},"np.uint16","？因为 16 位无符号整数的表示范围是 ",[69,1075,1077,1096],{"className":1076},[72],[69,1078,1080],{"className":1079},[76],[78,1081,1082],{"xmlns":80},[82,1083,1084,1093],{},[85,1085,1086,1088,1090],{},[88,1087,90],{},[92,1089,94],{},[88,1091,1092],{},"65,535",[99,1094,1095],{"encoding":101},"0 \\sim 65{,}535",[69,1097,1099,1117],{"className":1098,"ariaHidden":107},[106],[69,1100,1102,1105,1108,1111,1114],{"className":1101},[111],[69,1103],{"className":1104,"style":116},[115],[69,1106,90],{"className":1107},[120],[69,1109],{"className":1110,"style":125},[124],[69,1112,94],{"className":1113},[129],[69,1115],{"className":1116,"style":125},[124],[69,1118,1120,1124,1128,1136],{"className":1119},[111],[69,1121],{"className":1122,"style":1123},[115],"height:0.8389em;vertical-align:-0.1944em;",[69,1125,1127],{"className":1126},[120],"65",[69,1129,1131],{"className":1130},[120],[69,1132,1135],{"className":1133},[1134],"mpunct",",",[69,1137,1139],{"className":1138},[120],"535","，刚好能无损容纳 TinyStories（10K 词表）和 OpenWebText（32K 词表）的所有 token ID，同时相比 Python ",[19,1142,1020],{}," 或 ",[19,1145,1146],{},"int64"," 节省 75% 的磁盘与内存占用。",[42,1149,1150],{"id":1150},"官方测试验证结果",[10,1152,1153,1154,1157,1158,1161],{},"运行 ",[19,1155,1156],{},"test_train_bpe.py"," 与 ",[19,1159,1160],{},"test_tokenizer.py"," 全套测试：",[230,1163,1168],{"className":1164,"code":1166,"language":1167,"meta":235},[1165],"language-text","tests\u002Ftest_train_bpe.py::test_train_bpe_speed PASSED\ntests\u002Ftest_train_bpe.py::test_train_bpe PASSED\ntests\u002Ftest_train_bpe.py::test_train_bpe_special_tokens PASSED\ntests\u002Ftest_tokenizer.py (23 passed, 2 skipped on macOS)\n","text",[19,1169,1166],{"__ignoreMap":235},[10,1171,1172,1173,1175,1176,181],{},"所有 ASCII、多语言 Unicode（德语、Emoji）、特殊 token 边界以及与 OpenAI ",[19,1174,521],{}," GPT-2 编码器的逐 token 对照测试全部通过。最后一篇将继续完成 ",[30,1177,1179],{"href":1178},"\u002Fposts\u002Fcs336-cross-entropy-adamw-training-loop\u002F","Cross-Entropy、AdamW 优化器、余弦退火调度与完整训练闭环",[10,1181,1182],{},[1183,1184,1185,1186,1191],"em",{},"封面使用 ",[30,1187,1190],{"href":1188,"rel":1189},"https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Faigc-image-generator",[34],"aigc-image-generator"," 辅助生成，属于概念插画；准确的算法细节与代码实现以正文为准。",[42,1193,1194],{"id":1194},"参考资料",[1196,1197,1198,1205,1212],"ul",{},[568,1199,1200],{},[30,1201,1204],{"href":1202,"rel":1203},"https:\u002F\u002Farxiv.org\u002Fabs\u002F1508.07909",[34],"Sennrich et al., 2016: Neural Machine Translation of Rare Words with Subword Units",[568,1206,1207],{},[30,1208,1211],{"href":1209,"rel":1210},"https:\u002F\u002Fgithub.com\u002Fopenai\u002Ftiktoken",[34],"OpenAI tiktoken GPT-2 regex pattern",[568,1213,1214],{},[30,1215,1218],{"href":1216,"rel":1217},"https:\u002F\u002Fgithub.com\u002Fstanford-cs336\u002Fassignment1-basics",[34],"Stanford CS336 Assignment 1 官方仓库",[1220,1221,1222],"style",{},"html pre.shiki code .sWURj, html code.shiki .sWURj{--shiki-default:#ADBAC7;--shiki-light:#24292E;--shiki-cyber:#EEF0F9}html pre.shiki code .sf2cM, html code.shiki .sf2cM{--shiki-default:#F47067;--shiki-light:#D73A49;--shiki-cyber:#EEF0F9}html pre.shiki code .svkvy, html code.shiki .svkvy{--shiki-default:#6CB6FF;--shiki-light:#005CC5;--shiki-cyber:#EEF0F9}html pre.shiki code .sRGqv, html code.shiki .sRGqv{--shiki-default:#6CB6FF;--shiki-light:#005CC5;--shiki-cyber:#ACAFFF}html pre.shiki code .sA9aF, html code.shiki .sA9aF{--shiki-default:#F69D50;--shiki-default-font-style:inherit;--shiki-light:#E36209;--shiki-light-font-style:inherit;--shiki-cyber:#4BF3C8;--shiki-cyber-font-style:italic}html pre.shiki code .sgENW, html code.shiki .sgENW{--shiki-default:#6CB6FF;--shiki-light:#005CC5;--shiki-cyber:#FFD493}html pre.shiki code .sWqY2, html code.shiki .sWqY2{--shiki-default:#96D0FF;--shiki-light:#032F62;--shiki-cyber:#FFD493}html pre.shiki code .sq3c-, html code.shiki .sq3c-{--shiki-default:#ADBAC7;--shiki-light:#24292E;--shiki-cyber:#00DAEF}html pre.shiki code .slEfO, html code.shiki .slEfO{--shiki-default:#F47067;--shiki-default-font-style:inherit;--shiki-light:#D73A49;--shiki-light-font-style:inherit;--shiki-cyber:#54B9FF;--shiki-cyber-font-style:italic}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .cyber .shiki span {color: var(--shiki-cyber);background: var(--shiki-cyber-bg);font-style: var(--shiki-cyber-font-style);font-weight: var(--shiki-cyber-font-weight);text-decoration: var(--shiki-cyber-text-decoration);}html.cyber .shiki span {color: var(--shiki-cyber);background: var(--shiki-cyber-bg);font-style: var(--shiki-cyber-font-style);font-weight: var(--shiki-cyber-font-weight);text-decoration: var(--shiki-cyber-text-decoration);}html pre.shiki code .sST-V, html code.shiki .sST-V{--shiki-default:#F47067;--shiki-light:#D73A49;--shiki-cyber:#54B9FF}html pre.shiki code .sflY6, html code.shiki .sflY6{--shiki-default:#96D0FF;--shiki-light:#032F62;--shiki-cyber:#EEF0F9}html pre.shiki code .s0kv9, html code.shiki .s0kv9{--shiki-default:#F47067;--shiki-light:#005CC5;--shiki-cyber:#FFD493}html pre.shiki code .shgEQ, html code.shiki .shgEQ{--shiki-default:#F47067;--shiki-light:#D73A49;--shiki-cyber:#FFD493}html pre.shiki code .s-Nz2, html code.shiki .s-Nz2{--shiki-default:#8DDB8C;--shiki-default-font-weight:bold;--shiki-light:#22863A;--shiki-light-font-weight:bold;--shiki-cyber:#EEF0F9;--shiki-cyber-font-weight:inherit}html pre.shiki code .sCxhR, html code.shiki .sCxhR{--shiki-default:#768390;--shiki-default-font-style:inherit;--shiki-light:#6A737D;--shiki-light-font-style:inherit;--shiki-cyber:#EEF0F98F;--shiki-cyber-font-style:italic}html pre.shiki code .sS_0i, html code.shiki .sS_0i{--shiki-default:#DCBDFB;--shiki-light:#6F42C1;--shiki-cyber:#00DAEF}html pre.shiki code .sbU-9, html code.shiki .sbU-9{--shiki-default:#ADBAC7;--shiki-default-font-style:inherit;--shiki-light:#24292E;--shiki-light-font-style:inherit;--shiki-cyber:#ACAFFF;--shiki-cyber-font-style:italic}html pre.shiki code .s9NhK, html code.shiki .s9NhK{--shiki-default:#ADBAC7;--shiki-default-font-style:inherit;--shiki-light:#24292E;--shiki-light-font-style:inherit;--shiki-cyber:#FFD493;--shiki-cyber-font-style:italic}",{"title":235,"searchDepth":290,"depth":652,"links":1224},[1225,1227,1232,1233,1235,1236],{"id":44,"depth":290,"text":1226},"为什么选择 UTF-8 Byte-Level BPE（Problem unicode1 & unicode2）",{"id":184,"depth":290,"text":185,"children":1228},[1229,1230],{"id":203,"depth":652,"text":204},{"id":357,"depth":652,"text":1231},"2. 再用 GPT-2 正则 re.finditer 提取预分词频次表",{"id":529,"depth":290,"text":530},{"id":844,"depth":290,"text":1234},"分词器编码：按 Merge Rank 顺序合并与流式 encode_iterable",{"id":1150,"depth":290,"text":1150},{"id":1194,"depth":290,"text":1194},[1238],"学习笔记","\u002Fimages\u002Fposts\u002Fcs336-bpe-tokenizer-training-encoding\u002Fcs336-bpe-tokenizer-training-encoding-cover.webp","UTF-8 字节块经过正则切分闸口后沿合并树聚合成子词单元的 AI 辅助概念插画","2026-10-25","实现 GPT-2 正则预分词、特殊 token 硬边界切分、基于 pair-to-word 倒排索引的增量 BPE 合并算法（\u003C 0.25s 通过速度基准），以及与 tiktoken 完全对齐的 Tokenizer。",false,"md",{"codeUrl":32,"attachmentUrl":39},true,"\u002Fposts\u002Fcs336-bpe-tokenizer-training-encoding",null,{"title":5,"description":1242},"Stanford CS336","7","posts\u002Fcs336-bpe-tokenizer-training-encoding",[1254,1255,1256,1257,1258],"CS336","Tokenizer","BPE","Tiktoken","算法优化","RdUrhmhDFi0RoTkslPcpPg6iVv0mmr2vkTmNX3PjPrI",[1261,1276,1291,1295,1309,1323,1337,1352,1366,1378,1393,1405,1418,1430,1443,1456,1472,1486,1497,1509,1524,1537,1551,1563,1577,1589,1604,1618,1632,1643,1656,1673,1688,1701,1713,1725,1738,1751],{"slug":1262,"path":1263,"title":1264,"date":1265,"tags":1266,"description":1271,"cover":1272,"coverAlt":1273,"excerpt":1274,"draft":1243,"hidden":1243,"published":1246,"readingTime":733,"series":1250,"seriesOrder":733,"codeUrl":32,"attachmentUrl":1275},"cs336-cross-entropy-adamw-training-loop","\u002Fposts\u002Fcs336-cross-entropy-adamw-training-loop","Stanford CS336 学习记录 08：数值稳定交叉熵、AdamW、余弦调度与完整训练闭环","2026-10-27",[1254,1267,1268,1269,1270],"PyTorch","AdamW","CrossEntropy","训练闭环","用 LogSumExp 消去交叉熵中的指数溢出，手写解耦权重衰减的 AdamW 优化器、全局 L2 梯度裁剪、Warmup 余弦退火调度、memmap 数据采样、Checkpoint 序列化与 Top-p 解码，完成 CS336 Assignment 1 全部 48 项测试。","\u002Fimages\u002Fposts\u002Fcs336-cross-entropy-adamw-training-loop\u002Fcs336-cross-entropy-adamw-training-loop-cover.webp","滑动窗口采样、余弦退火曲线、双动量优化轨迹与检查点归档构成的训练闭环 AI 辅助概念插画","作为 Stanford CS336 Assignment 1 系列的收官篇，我们已经拥有了分词器（bpe.py、tokenizer.py）和完整的语言模型架构（transformer.py）。要让模型在语料上真正训练并生成文本，还需","\u002Fmaterials\u002Fcs336\u002Fexperiments\u002Fcs336-08-adamw-training-loop-20261027.zip",{"slug":1277,"path":1278,"title":1279,"date":1241,"tags":1280,"description":1287,"cover":1288,"coverAlt":1289,"excerpt":1290,"draft":1243,"hidden":1243,"published":1246,"readingTime":733},"editcache-video-dit-inference-acceleration","\u002Fposts\u002Feditcache-video-dit-inference-acceleration","EditCache：视频 Diffusion Transformer (DiT) 局部编辑的跨步特征复用与区域重算加速",[1281,1282,1283,1284,1285,1286],"扩散模型","视频生成","AI Infra","推理加速","GPU","论文解读","拆解视频扩散模型（Wan2.1-VACE）局部编辑中的三重算力冗余，通过可重入 Latent、跨步 Block 隐状态缓存复用、带 Halo 边界的局部区域重算与自适应推理起点，在单卡 H100 上实现 2.46× 模型计算加速与 1.49× 严格同步端到端加速。","\u002Fimages\u002Fposts\u002Feditcache-video-dit-inference-acceleration\u002Feditcache-dit-cover-20261025.webp","视频扩散 Transformer 时空三维体素网格中冻结背景缓存与暖橙色局部编辑重算通道的等距技术插画","在基于 Diffusion Transformer（DiT）的视频编辑与局部重绘（Video Inpainting \u002F Local Editing）任务中，用户的典型工作流并不是一次性生成最终成片，而是针对同一段源视频反复调整掩码区",{"slug":1292,"path":1247,"title":5,"date":1241,"tags":1293,"description":1242,"cover":1239,"coverAlt":1240,"excerpt":1294,"draft":1243,"hidden":1243,"published":1246,"readingTime":733,"series":1250,"seriesOrder":718,"codeUrl":32,"attachmentUrl":39},"cs336-bpe-tokenizer-training-encoding",[1254,1255,1256,1257,1258],"前面六篇完成了 Transformer 语言模型本身的全部神经网络组件。要让模型真正处理自然语言文本，还需要打通入口处的 ByteLevel BPE（BytePair Encoding）分词器。这一篇记录 cs336basics\u002Fb",{"slug":1296,"path":1297,"title":1298,"date":1299,"tags":1300,"description":1305,"cover":1306,"coverAlt":1307,"excerpt":1308,"draft":1243,"hidden":1243,"published":1246,"readingTime":733},"in-situ-autoguidance-editable-scientific-illustration","\u002Fposts\u002Fin-situ-autoguidance-editable-scientific-illustration","从原位自校正引导（In-situ Autoguidance）到可编辑科研插图生成（AutoFigure-Edit）","2026-10-24",[1281,1301,1302,1303,1304,1286],"多模态","Agent","ICML","ACL","串联两项生成式模型与多模态智能体研究：如何通过推理期局部随机微扰在单模型内构造免训练扩散自引导向量（ICML 2025 Workshop 第一作者，arXiv:2510.17136），以及如何通过五阶段视觉-符号混合流水线生成原生可编辑 SVG 科研插图（ACL 2026 Demo，FigureBench 83.0% 胜率）。","\u002Fimages\u002Fposts\u002Fin-situ-autoguidance-editable-scientific-illustration\u002Fautoguidance-autofigure-cover-20261024.webp","左侧扩散流形自校正轨迹矢量场与右侧五阶段可编辑 SVG 科研插图分层网格相衔接的等距技术插画","生成式视觉模型正在经历从“生成高保真像素（Pixel Synthesis）”向“可控结构化生成（Controllable & Editable Generation）”的演进。在这一进程中，我作为第一作者或核心贡献者参与了两项分别处",{"slug":1310,"path":1311,"title":1312,"date":1313,"tags":1314,"description":1318,"cover":1319,"coverAlt":1320,"excerpt":1321,"draft":1243,"hidden":1243,"published":1246,"readingTime":733,"series":1250,"seriesOrder":700,"codeUrl":32,"attachmentUrl":1322},"cs336-pre-norm-transformer-block-lm","\u002Fposts\u002Fcs336-pre-norm-transformer-block-lm","Stanford CS336 学习记录 06：Pre-Norm Transformer Block 与完整语言模型装配","2026-10-23",[1254,1267,1315,1316,1317],"Transformer","Pre-Norm","FLOPs","将 RMSNorm、带 RoPE 的因果多头自注意力和 SwiGLU 组装成 Pre-Norm Transformer Block，搭建完整 TransformerLM，并核算 GPT-2 XL 的参数量与前向 FLOPs。","\u002Fimages\u002Fposts\u002Fcs336-pre-norm-transformer-block-lm\u002Fcs336-pre-norm-transformer-block-lm-cover.webp","干净残差流主干穿过多个 Pre-Norm 子层并汇聚到词表投影层的 AI 辅助概念插画","前五篇把 Linear、Embedding、RMSNorm、SwiGLU、RotaryPositionalEmbedding 和因果 MultiHeadSelfAttention 全部单独打通了。这一篇进入 Assignment 1","\u002Fmaterials\u002Fcs336\u002Fexperiments\u002Fcs336-06-transformer-block-lm-20261023.zip",{"slug":1324,"path":1325,"title":1326,"date":1313,"tags":1327,"description":1333,"cover":1334,"coverAlt":1335,"excerpt":1336,"draft":1243,"hidden":1243,"published":1246,"readingTime":798},"xgrammar-concurrent-cache-generation-fix","\u002Fposts\u002Fxgrammar-concurrent-cache-generation-fix","XGrammar 结构化解码缓存剖析：Unlimited 模式并发 Clear 字节记账竞态与代际修复（PR #1011）",[1283,1328,1329,1330,1331,1332],"C++","XGrammar","结构化解码","并发编程","源码分析","拆解 XGrammar GrammarCompiler 的双层缓存架构与 ThreadSafeLRUCache 锁分离实现，定位 Unlimited 模式下 Clear() 字节残留与在途编译（In-flight Computation）跨代污染根因，给出代际号 + 读写锁原子修复与零 sleep 确定性并发 GoogleTest 验证。","\u002Fimages\u002Fposts\u002Fxgrammar-concurrent-cache-generation-fix\u002Fxgrammar-cache-fix-cover-20261022.webp","双层磨砂玻璃流道经暖橙色代际闸门过滤后汇入晶格缓存矩阵的等距技术插画","在大语言模型服务（如 SGLang、vLLM）的结构化输出场景中，JSON Schema 或 EBNF 语法需要先编译为词表级掩码缓存（CompiledGrammar），才能在每个 Decode 步以微秒级开销生成合法 token ",{"slug":1338,"path":1339,"title":1340,"date":1341,"tags":1342,"description":1348,"cover":1349,"coverAlt":1350,"excerpt":1351,"draft":1243,"hidden":1243,"published":1246,"readingTime":798},"multi-constraint-agent-routing-dag-eval","\u002Fposts\u002Fmulti-constraint-agent-routing-dag-eval","多约束生产级 Agent 系统实践：两阶段工具路由、DAG 局部重规划与轨迹级评测闭环","2026-10-22",[1302,1343,1344,1345,1346,1347],"LLM","工具调用","上下文工程","DPO","强化学习","围绕跨境酒旅等强耦合多约束业务场景，拆解向量 Top-K 召回 + LLM 精排 + 跨语言实体链接与 Schema 门控（96%+ 调用合法率）、基于状态机的 DAG 局部子图重规划、Observation 冗余裁剪（节省 40% Token），以及从线上失败恢复轨迹构造 SFT\u002FDPO 偏好对的评测与训练闭环。","\u002Fimages\u002Fposts\u002Fmulti-constraint-agent-routing-dag-eval\u002Fagent-dag-routing-cover-20261020.webp","候选模块经漏斗滤网进入双分支有向无环图轨道并在受阻节点触发暖橙色局部旁路重连的等距技术插画","在玩具级 Demo 里，几个工具加上一段 ReAct 循环就能跑通大多数单步问答。但当 Agent 进入跨境酒旅、复杂交易或研发沙箱等真实业务生产环境时，单靠线性 ReAct 循环很快会撞上四堵墙： 1. 工具空间与参数耦合爆炸：候",{"slug":1353,"path":1354,"title":1355,"date":1356,"tags":1357,"description":1361,"cover":1362,"coverAlt":1363,"excerpt":1364,"draft":1243,"hidden":1243,"published":1246,"readingTime":1365},"llm-mrd-distillation-auditable-review-agents","\u002Fposts\u002Fllm-mrd-distillation-auditable-review-agents","多视角推理蒸馏（LLM-MRD）与可审计生成式评审系统（DeepReviewer 2.0）深度拆解","2026-10-21",[1358,1301,1359,1302,1360,1286],"知识蒸馏","Qwen2.5-VL","Post-training","对比大模型落地的两条互补路径：如何用 Qwen2.5-VL 离线多视角分析结合 MLP 残差校准与温度缩放 KL 蒸馏，将推理能力压入 358M 学生模型实现 17.2 ms 独立推理（LLM-MRD，DASFAA 2026 Oral）；以及如何通过主张-证据-风险台账、PASA 可比性门控与导出预算校验构建可审计的同行评审 Agent（DeepReviewer 2.0）。","\u002Fimages\u002Fposts\u002Fllm-mrd-distillation-auditable-review-agents\u002Fmrd-review-distill-cover-20261016.webp","三块半透明几何棱镜发出平行光束穿过中央校准环汇聚至右侧高密度晶体模块的等距技术插画","把大语言模型或视觉语言模型（VLM）的深度推理能力用到实际系统中，通常会面临一个两难选择： 如果场景受限于毫秒级在线延迟（如高吞吐多模态内容审核与虚假信息检测）：直接在线调用 7B～72B 的多模态大模型做长链条思维链（CoT）生成",18,{"slug":1367,"path":1368,"title":1369,"date":1356,"tags":1370,"description":1373,"cover":1374,"coverAlt":1375,"excerpt":1376,"draft":1243,"hidden":1243,"published":1246,"readingTime":718,"series":1250,"seriesOrder":670,"codeUrl":32,"attachmentUrl":1377},"cs336-rope-rotary-position-embedding","\u002Fposts\u002Fcs336-rope-rotary-position-embedding","Stanford CS336 学习记录 05：RoPE 旋转位置编码与带位置的多头注意力",[1254,1267,1315,1371,1372],"RoPE","Attention","从二维平面正交旋转推导 RoPE 的奇偶特征配对公式，理解为什么内积只依赖相对位置差 (i - j)，并用 register_buffer 预计算 cos\u002Fsin 缓存接入因果多头自注意力。","\u002Fimages\u002Fposts\u002Fcs336-rope-rotary-position-embedding\u002Fcs336-rope-rotary-position-embedding-cover.webp","二维特征平面中的旋转向量对与预计算余弦正弦网格的 AI 辅助概念插画","上一篇完成了不含位置编码的因果多头自注意力。如果只看那一套矩阵乘法，把输入序列前两个 token 交换顺序（并在因果 mask 允许的范围内看同一个后缀 query 对它们的点积），点积本身并不知道哪个 key 离当前 query ","\u002Fmaterials\u002Fcs336\u002Fexperiments\u002Fcs336-05-rope-mha-20261021.zip",{"slug":1379,"path":1380,"title":1381,"date":1382,"tags":1383,"description":1389,"cover":1390,"coverAlt":1391,"excerpt":1392,"draft":1243,"hidden":1243,"published":1246,"readingTime":733},"llm-gateway-semantic-cache-dynamic-batching","\u002Fposts\u002Fllm-gateway-semantic-cache-dynamic-batching","高并发 LLM 推理网关架构实践：两级精确\u002F语义缓存、50ms 有界动态批处理与尾部延迟治理","2026-10-19",[1283,1384,1385,1386,1387,1388],"LLM 推理","分布式系统","Redis","FastAPI","后端","深度拆解生产级大模型推理网关 LLM-Gateway 的核心系统设计：基于 SHA-256 与 Redis 向量余弦相似度的两级精确\u002F语义缓存、50ms 有界等待动态批处理调度器（Dynamic Batching）、令牌桶多租户限流与 Prometheus 全链路可观测性，将命中请求 P99 延迟从 5,307 ms 压降至 17 ms。","\u002Fimages\u002Fposts\u002Fllm-gateway-semantic-cache-dynamic-batching\u002Fllm-gateway-cover-20261019.webp","高并发请求流依次穿过精确哈希滤网、向量球面语义缓存层与 50ms 动态批处理时间窗拼装器的等距系统架构插画","在将大语言模型接入高并发业务系统时，如果让上游微服务直接裸连底层推理引擎（vLLM \u002F SGLang 或外部模型 API），系统很快会暴露出三个典型的生产痛点： 1. 高频重复与近义查询消耗巨量算力：大量用户请求在语义上高度等价（例",{"slug":1394,"path":1395,"title":1396,"date":1382,"tags":1397,"description":1399,"cover":1400,"coverAlt":1401,"excerpt":1402,"draft":1243,"hidden":1243,"published":1246,"readingTime":798,"series":1250,"seriesOrder":663,"codeUrl":1403,"attachmentUrl":1404},"cs336-multihead-attention-shapes","\u002Fposts\u002Fcs336-multihead-attention-shapes","Stanford CS336 学习记录 04：多头注意力怎样拆头、计算和合头",[1254,1267,1315,1372,1398],"张量形状","沿着 (2, 5, 64) 的输入追踪 QKV 投影、reshape、transpose、共享因果 mask 和合头，理解每个头仍处理完整序列，以及最后为什么还需要输出 Linear。","\u002Fimages\u002Fposts\u002Fcs336-multihead-attention-shapes\u002Fcs336-multihead-attention-shapes-cover.webp","宽向量束分成多条并行特征轨道后重新合并的 AI 辅助概念插画","上一篇完成了缩放点积注意力。那个函数接收已经准备好的 Q、K、V，计算匹配分数，应用 mask，再用权重汇总 V。这一篇继续把它放进一个带参数的多头自注意力模块。 这次最需要看清的是“拆头”。代码里只是几行 reshape 和 tr","https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fcs336-labs\u002Ftree\u002F31363223498fafd1528c550a51efbe0aed231736","\u002Fmaterials\u002Fcs336\u002Fexperiments\u002Fcs336-04-multihead-attention-20261019.zip",{"slug":1406,"path":1407,"title":1408,"date":1409,"tags":1410,"description":1414,"cover":1415,"coverAlt":1416,"excerpt":1417,"draft":1243,"hidden":1243,"published":1246,"readingTime":739},"qwen25-sft-dpo-alignment-tax-ablation","\u002Fposts\u002Fqwen25-sft-dpo-alignment-tax-ablation","大模型后训练偏好对齐实录：Qwen2.5 SFT 多维消融、动态参考 DPO 与对齐税（Alignment Tax）量化分析","2026-10-18",[1360,1346,1411,1412,1347,1413],"SFT","LoRA","Qwen2.5","基于 Qwen2.5 完整复现 Instruct SFT → Bradley-Terry DPO 两阶段后训练管线：通过 3 随机种子消融 LoRA Rank、学习率与数据规模（GSM8K +12.73pp），实现免双倍权重的动态 Adapter 切换 DPO 训练，并定量揭示偏好对齐对确定性数学推理的对齐税效应。","\u002Fimages\u002Fposts\u002Fqwen25-sft-dpo-alignment-tax-ablation\u002Fqwen25-sft-dpo-cover-20261018.webp","左侧监督微调轨迹流经中央 Bradley-Terry 偏好分流天平分离出正负样本通道并连接右侧对齐税监控仪表盘的等距技术插画","在大语言模型后训练（PostTraining）管线中，监督微调（SFT）负责教会模型任务格式与领域推理模式，而直接偏好优化（DPO, Direct Preference Optimization）则利用成对偏好数据 $(x, yw,",{"slug":1419,"path":1420,"title":1421,"date":1409,"tags":1422,"description":1424,"cover":1425,"coverAlt":1426,"excerpt":1427,"draft":1243,"hidden":1243,"published":1246,"readingTime":751,"series":1250,"seriesOrder":652,"codeUrl":1428,"attachmentUrl":1429},"cs336-softmax-attention-mask","\u002Fposts\u002Fcs336-softmax-attention-mask","Stanford CS336 学习记录 03：Softmax 的数值稳定与 Attention Mask",[1254,1267,1315,1423,1372],"Softmax","从减去最大值为何不改变 Softmax 开始，理清函数与有参数层的区别，再沿着 Q、K、V 的形状理解缩放点积注意力，以及 mask 为什么必须填负无穷。","\u002Fimages\u002Fposts\u002Fcs336-softmax-attention-mask\u002Fcs336-softmax-attention-mask-cover.webp","注意力分数、权重与因果遮罩的 AI 辅助概念插画","上一篇写到 SwiGLU，接下来进入 Attention。真正开始相乘之前，先做了 Softmax：它把一组分数变成总和为 1 的权重，后面用这些权重汇总不同 token 提供的信息。 这段学习里，有两个很具体的问题值得记下来。一个","https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fcs336-labs\u002Ftree\u002Fa39eb684710817f7f389626423fcb26ca0f6d629","\u002Fmaterials\u002Fcs336\u002Fexperiments\u002Fcs336-03-softmax-sdpa-20261018.zip",{"slug":1431,"path":1432,"title":1433,"date":1434,"tags":1435,"description":1438,"cover":1439,"coverAlt":1440,"excerpt":1441,"draft":1243,"hidden":1243,"published":1246,"readingTime":739,"series":1250,"seriesOrder":290,"codeUrl":1428,"attachmentUrl":1442},"cs336-rmsnorm-swiglu","\u002Fposts\u002Fcs336-rmsnorm-swiglu","Stanford CS336 学习记录 02：RMSNorm 的广播与 SwiGLU 的门控","2026-10-16",[1254,1267,1315,1436,1437],"RMSNorm","SwiGLU","用一个 token 向量理解均方根归一化，拆开 dim=-1 与 keepdim=True 的作用，再沿着两条分支读懂 SiLU、逐元素乘法和 SwiGLU。","\u002Fimages\u002Fposts\u002Fcs336-rmsnorm-swiglu\u002Fcs336-rmsnorm-swiglu-cover.webp","特征尺度整理和双分支门控的 AI 辅助概念插画","上一篇把 Linear 和 Embedding 的参数形状理顺了。继续往下，最先遇到的是 RMSNorm：公式不算长，但把它落到一批 token 上时，dim=1、keepdim=True 和广播必须一起理解。 然后是 SiLU 和","\u002Fmaterials\u002Fcs336\u002Fexperiments\u002Fcs336-02-rmsnorm-swiglu-20261016.zip",{"slug":1444,"path":1445,"title":1446,"date":1447,"tags":1448,"description":1451,"cover":1452,"coverAlt":1453,"excerpt":1454,"draft":1243,"hidden":1243,"published":1246,"readingTime":745,"series":1250,"seriesOrder":241,"codeUrl":1428,"attachmentUrl":1455},"cs336-linear-embedding-shapes","\u002Fposts\u002Fcs336-linear-embedding-shapes","Stanford CS336 学习记录 01：从 Linear、Embedding 搞清张量形状","2026-10-14",[1254,1267,1315,1449,1450],"Linear","Embedding","从 PyTorch 类、参数与 forward 的关系开始，理解 Linear 为什么保存两行三列的权重、计算时为什么要转置，再区分 Embedding 查表与线性变换。","\u002Fimages\u002Fposts\u002Fcs336-linear-embedding-shapes\u002Fcs336-linear-embedding-shapes-cover.webp","token 方块、向量表与矩阵变换的 AI 辅助概念插画","开始沿着 Stanford CS336 的 Assignment 1 一步步实现。BPE 之前已经接触过，这轮先跳过分词器，从 Transformer 的基础层开始。这里记录的是当时写代码时真正卡住的问题，以及对应测试能确认到的范围","\u002Fmaterials\u002Fcs336\u002Fexperiments\u002Fcs336-01-linear-embedding-20261014.zip",{"slug":1457,"path":1458,"title":1459,"date":1460,"tags":1461,"description":1466,"cover":1467,"coverAlt":1468,"excerpt":1469,"draft":1243,"hidden":1243,"published":1246,"readingTime":757,"series":1462,"seriesOrder":798,"codeUrl":1470,"attachmentUrl":1471},"cmu-15-779-kv-cache-granularity-study","\u002Fposts\u002Fcmu-15-779-kv-cache-granularity-study","CMU 15-779 学习记录 14：KV 页大小、共享边界与 CPU 实现成本","2026-10-12",[1462,1463,1464,1465],"CMU 15-779","KV cache","PagedAttention","系统实验","用 210 组 CPU 配置比较连续与分页 KV cache，观察 31\u002F32-token 共享前缀的页边界，分别记录活跃槽位、池容量、复制计数和完整调用耗时。","\u002Fimages\u002Fposts\u002Fcmu-15-779-kv-serving-speculative-sampling\u002Fcmu-serving-cover-20261019.webp","浅色手绘分页存储抽屉与共享前缀卡片，旁边连接不同长度的请求队列","个人课程项目中的 CPU 实验记录，延续记录 11 的分页 KV 实现。已完成 210 组配置与 1470 次计时，输出均通过连续存储参考对照；本篇不代表正式作业提交、整门课程或项目全部交付完成。 同样是共享 31 个 token，","https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fcmu-15-779-labs\u002Ftree\u002F200faf569eccf94fb125e65110bb5d14094d454a","\u002Fmaterials\u002Fcmu-15-779\u002Fexperiments\u002Fkv-cache-granularity-20261019.zip",{"slug":1473,"path":1474,"title":1475,"date":1476,"tags":1477,"description":1480,"cover":1481,"coverAlt":1482,"excerpt":1483,"draft":1243,"hidden":1243,"published":1246,"readingTime":670,"series":1462,"seriesOrder":781,"codeUrl":1484,"attachmentUrl":1485},"cmu-15-779-moe-routing-capacity","\u002Fposts\u002Fcmu-15-779-moe-routing-capacity","CMU 15-779 学习记录 13：MoE 路由、稀疏分发与容量溢出","2026-10-09",[1462,1478,1479],"MoE","Routing","用 CPU top-2 MoE 检查稀疏分发的输出与梯度，统计 expert-token 行数，并保留容量丢弃会改变输出的反例。","\u002Fimages\u002Fposts\u002Fcmu-15-779-moe-routing-capacity\u002Fcmu-moe-cover-20261019.webp","浅色手绘分拣器将小珠分配到四个专家工作屋，队列呈现负载不均","第 18 讲的个人配套实验。36 号脚本串行运行 CPU 专家，检查 topk 路由后的分发与合并；没有训练 router，也没有执行 GPU grouped GEMM 或跨设备 alltoall。 上一节记录低秩适配，这次转向每个","https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fcmu-15-779-labs\u002Ftree\u002Fc4851e0cedadf13af7c421867e83c8ff57244fa0","\u002Fmaterials\u002Fcmu-15-779\u002Fexperiments\u002Fcpu-parallel-serving-lora-moe-20261019.zip",{"slug":1487,"path":1488,"title":1489,"date":1490,"tags":1491,"description":1493,"cover":1494,"coverAlt":1495,"excerpt":1496,"draft":1243,"hidden":1243,"published":1246,"readingTime":670,"series":1462,"seriesOrder":757,"codeUrl":1484,"attachmentUrl":1485},"cmu-15-779-lora-uniform-quantization","\u002Fposts\u002Fcmu-15-779-lora-uniform-quantization","CMU 15-779 学习记录 12：LoRA 首步梯度、权重合并与均匀 4-bit 打包","2026-10-06",[1462,1412,1492],"Quantization","用一次 CPU SGD 更新检查 LoRA 初始化、冻结基座与合并等价，再分别记录均匀 4-bit 存储字节数和量化误差。","\u002Fimages\u002Fposts\u002Fcmu-15-779-lora-uniform-quantization\u002Fcmu-lora-cover-20261019.webp","浅色手绘的大计算板、两块细长适配板和成对打包的小方块","第 17 讲的个人配套实验，35 号脚本实际执行一次 CPU LoRA 更新，并实现均匀 4bit 打包。实验检查机制与数值误差，不评估微调任务效果，也没有实现完整 QLoRA。 这次把两个问题分开记录：低秩分支是否能合并进原权重，",{"slug":1498,"path":1499,"title":1500,"date":1501,"tags":1502,"description":1506,"cover":1467,"coverAlt":1507,"excerpt":1508,"draft":1243,"hidden":1243,"published":1246,"readingTime":718,"series":1462,"seriesOrder":751,"codeUrl":1484,"attachmentUrl":1485},"cmu-15-779-kv-serving-speculative-sampling","\u002Fposts\u002Fcmu-15-779-kv-serving-speculative-sampling","CMU 15-779 学习记录 11：分页 KV、连续批处理与推测采样","2026-10-03",[1462,1503,1504,1505],"LLM Serving","KV Cache","Speculative Decoding","在 CPU 上验证 KV 页共享和写时复制，用单位成本模型检查连续批处理，再用条件概率小模型核对推测采样的接受与补采样。","浅色手绘的共享卡片抽屉与不断补入小珠子的请求托盘","第 15、16 讲的个人配套实验。33 号实现 CPU 分页 KV 存储，37 号模拟请求准入，34 号运行小概率模型的推测采样。它们分别验证存储、调度和分布问题，没有测量真实 LLM 服务吞吐或 GPU 延迟。 记录 08已经对照",{"slug":1510,"path":1511,"title":1512,"date":1513,"tags":1514,"description":1519,"cover":1520,"coverAlt":1521,"excerpt":1522,"draft":1243,"hidden":1243,"published":1246,"readingTime":1523},"qwen3-mid-post-train-mlx","\u002Fposts\u002Fqwen3-mid-post-train-mlx","在 M2 Pro 上完成 Qwen3 两阶段训练：Mid Train、SFT 与可复核评估","2026-09-30 11:19:10",[1343,1515,1516,1412,1517,1411,1518],"Qwen3","MLX","持续预训练","Apple Silicon","用 Qwen3 0.6B Base 和 MLX 完成领域持续训练与监督微调，记录数据划分、聊天模板、损失掩码、数值稳定性修复和独立测试结果。","\u002Fimages\u002Fposts\u002Fqwen3-mid-post-train-mlx\u002Fqwen3-mlx-cover-20261018.webp","Apple Silicon 统一内存芯片底座支撑两阶段持续预训练与带精确 Loss 掩码滤网的监督微调桥接插画","在一台 M2 Pro、16 GB 内存的 Mac 上，能否把一个 Base 语言模型接续训练成输出 JSON 的运维事件分级助手？这次项目完成了领域持续训练、监督微调、检查点选择、重新加载和独立测试，并保留了权重、日志和数据哈希。 ",16,{"slug":1525,"path":1526,"title":1527,"date":1528,"tags":1529,"description":1533,"cover":1534,"coverAlt":1535,"excerpt":1536,"draft":1243,"hidden":1243,"published":1246,"readingTime":700,"series":1462,"seriesOrder":745,"codeUrl":1484,"attachmentUrl":1485},"cmu-15-779-cpu-parallelism-pipeline","\u002Fposts\u002Fcmu-15-779-cpu-parallelism-pipeline","CMU 15-779 学习记录 10：加权梯度、状态分片与同步流水线","2026-09-30",[1462,1530,1531,1532],"Parallel Training","Gloo","Pipeline","用双 CPU 进程 Gloo 检查不等数据分片的梯度，推导持久状态字节数，再用依赖模型比较 GPipe 与同步 1F1B 的激活存活。","\u002Fimages\u002Fposts\u002Fcmu-15-779-cpu-parallelism-pipeline\u002Fcmu-parallel-cover-20261019.webp","浅色手绘的四阶段桌面流水线与按数量汇总的托盘","第 12 至 14 讲的个人配套实验，脚本编号 31、32 不对应官方作业。Gloo 实际运行两个 CPU 进程；张量切分、状态字节数和流水线分别按下文的数学或成本模型解释。真实 GPU 实测按学习安排跳过。 上一节检查单个计算图的",{"slug":1538,"path":1539,"title":1540,"date":1541,"tags":1542,"description":1545,"cover":1546,"coverAlt":1547,"excerpt":1548,"draft":1243,"hidden":1243,"published":1246,"readingTime":718,"series":1462,"seriesOrder":739,"codeUrl":1549,"attachmentUrl":1550},"cmu-15-779-cpu-compiler-optimizations","\u002Fposts\u002Fcmu-15-779-cpu-compiler-optimizations","CMU 15-779 学习记录 09：CPU 编译优化、合法重写与候选搜索","2026-09-27",[1462,1543,1544,1267],"ML Compiler","TorchInductor","用 CPU 实验验证 BatchNorm 折叠、RMSNorm 重排与非线性反例，测量分块候选，并运行 TorchInductor，分别记录首调用与预热后的时间。","\u002Fimages\u002Fposts\u002Fcmu-15-779-cpu-compiler-optimizations\u002Fcmu-cpu-compiler-cover-20261019.webp","淡紫计算积木经过薄荷绿折叠模具变成浅蓝积木，前方摆放四种候选托盘的浅色手绘插画","编译章节的个人配套实验，编号 30 不对应官方作业。图重写、分块候选和 TorchInductor 均在 CPU 上运行；真实 GPU 实验按学习安排跳过。 上一节记录了分块 Attention 和 KV cache。这次转到编译优","https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fcmu-15-779-labs\u002Ftree\u002F9261bea6aedf31cdccbbab2541ca693fc97b3743","\u002Fmaterials\u002Fcmu-15-779\u002Fexperiments\u002Fcpu-compiler-optimizations-20261019.zip",{"slug":1552,"path":1553,"title":1554,"date":1555,"tags":1556,"description":1558,"cover":1559,"coverAlt":1560,"excerpt":1561,"draft":1243,"hidden":1243,"published":1246,"readingTime":757,"series":1462,"seriesOrder":733,"codeUrl":1470,"attachmentUrl":1562},"cmu-15-779-attention-online-softmax","\u002Fposts\u002Fcmu-15-779-attention-online-softmax","CMU 15-779 学习记录 08：分块 Attention、在线 Softmax 与 KV cache","2026-09-24",[1462,1372,1423,1557,1267],"FlashAttention","在 CPU 上对照完整 Attention、在线 Softmax 分块实现与 PyTorch SDPA，检查因果掩码、浮点误差和分数块大小，理解 FlashAttention 的一部分算法基础。","\u002Fimages\u002Fposts\u002Fcmu-15-779-attention-online-softmax\u002Fcmu-online-attention-cover-20261016.webp","浅紫色卡片分批进入绿色计算托盘，旁边三个状态容器保存累积量的浅色手绘插画","第 5 讲 Transformers & Attention 的个人配套实验。28\u002F29 号是 CPU 单头前向与 KV cache；后续补入 38 号 CUDA 多头合成张量的显式 SDPA MATH\u002FFLASH 对照。没有手写 ","\u002Fmaterials\u002Fcmu-15-779\u002Fexperiments\u002Fcuda-triton-sdpa-20261019.zip",{"slug":1564,"path":1565,"title":1566,"date":1567,"tags":1568,"description":1573,"cover":1574,"coverAlt":1575,"excerpt":1576,"draft":1243,"hidden":1243,"published":1246,"readingTime":745,"series":1462,"seriesOrder":718,"codeUrl":1470,"attachmentUrl":1562},"cmu-15-779-matmul-tiling-tree-reduction","\u002Fposts\u002Fcmu-15-779-matmul-tiling-tree-reduction","CMU 15-779 学习记录 07：矩阵分块、数据复用与树形归约","2026-09-21",[1462,1569,1570,1571,1572],"CUDA","矩阵乘法","数据复用","归约","用 CPU 模型统计矩阵分块的输入读取次数，再比较串行求和和树形归约的依赖深度，记录补零、同步与浮点加法顺序的边界。","\u002Fimages\u002Fposts\u002Fcmu-15-779-matmul-tiling-tree-reduction\u002Fcmu-matmul-reduction-cover-20261016.webp","矩阵积木在共享托盘中复用，旁边的小珠沿树形管道逐层合并的浅色手绘插画","GPU Architecture & CUDA Programming 的个人配套实验。首次记录用 CPU 模型检查数据复用与依赖深度；后续补入单卡 GPU 原生 CUDA 矩阵验证，以及 Triton 两阶段归约和矩阵乘法实测。 ",{"slug":1578,"path":1579,"title":1580,"date":1581,"tags":1582,"description":1585,"cover":1586,"coverAlt":1587,"excerpt":1588,"draft":1243,"hidden":1243,"published":1246,"readingTime":745,"series":1462,"seriesOrder":700,"codeUrl":1470,"attachmentUrl":1562},"cmu-15-779-cuda-thread-indexing","\u002Fposts\u002Fcmu-15-779-cuda-thread-indexing","CMU 15-779 学习记录 06：CUDA 线程索引与边界检查","2026-09-18",[1462,1569,1285,1583,1584],"线程索引","边界检查","用 CPU 演示 CUDA 的 block、thread 与全局下标：10 个元素如何分给 3 个线程块，为什么要跳过越界线程，以及这些验证能说明什么。","\u002Fimages\u002Fposts\u002Fcmu-15-779-cuda-thread-indexing\u002Fcmu-cuda-indexing-cover-20261015.webp","分组的浅紫色计算方块与末端边界挡板组成线程分配教具的浅色插画","GPU Architecture & CUDA Programming 的个人配套练习。首次记录为 CPU 索引演示；后续在单卡 GPU 环境补入原生 CUDA 编译、运行与 memcheck，以及 Triton 向量加法实测。 上",{"slug":1590,"path":1591,"title":1592,"date":1593,"tags":1594,"description":1598,"cover":1599,"coverAlt":1600,"excerpt":1601,"draft":1243,"hidden":1243,"published":1246,"readingTime":745,"series":1462,"seriesOrder":670,"codeUrl":1602,"attachmentUrl":1603},"cmu-15-779-profiler-operator-batching","\u002Fposts\u002Fcmu-15-779-profiler-operator-batching","CMU 15-779 学习记录 05：Profiler、算子调用与批量推理","2026-09-15",[1462,1267,1595,1596,1597],"Profiler","批量推理","性能分析","对照 ML Systems 基础，用 Profiler 拆开两层网络的执行过程，再固定 1000 个样本扫描 batch size，观察算子调用次数、尾批和 CPU 耗时。","\u002Fimages\u002Fposts\u002Fcmu-15-779-profiler-operator-batching\u002Fcmu-pytorch-profiling-cover-20261015.webp","放大镜观察线性层与 ReLU 模块，旁边面板呈现执行片段的浅色技术教具插画","本篇对照课程官网的 Introduction to ML Systems 整理，是个人配套 CPU 实验，不是课程官方作业。脚本编号是练习顺序，不对应课程讲次。 前几篇一直在补 PyTorch 基础。到了这一步，我在学习聊天里问：现","https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fcmu-15-779-labs\u002Ftree\u002Fcecf5110d8a95161bb5763eadfca94cd7a0cc50f","\u002Fmaterials\u002Fcmu-15-779\u002Fexperiments\u002Fpytorch-profiler-batching-20261015.zip",{"slug":1605,"path":1606,"title":1607,"date":1608,"tags":1609,"description":1612,"cover":1613,"coverAlt":1614,"excerpt":1615,"draft":1243,"hidden":1243,"published":1246,"readingTime":739,"series":1462,"seriesOrder":663,"codeUrl":1616,"attachmentUrl":1617},"cmu-15-779-pytorch-linear-relu-absolute-value","\u002Fposts\u002Fcmu-15-779-pytorch-linear-relu-absolute-value","CMU 15-779 学习记录 04：多输出线性层、ReLU 与绝对值网络","2026-09-12",[1462,1267,1570,1610,1611],"ReLU","神经网络","把 nn.Linear 对应到矩阵乘法，弄清多个输出怎样计算，再逐层拆开一个绝对值网络，比较手动设置参数与训练得到的结果。","\u002Fimages\u002Fposts\u002Fcmu-15-779-pytorch-linear-relu-absolute-value\u002Fcmu-pytorch-absolute-cover-20261015.webp","双分支经过折角模块后汇入紫色 V 形曲线面板的浅色技术教具插画","本篇整理的是学习 CMU 15779 时补充的 PyTorch 预备练习。脚本编号不对应课程讲次，也不是课程官方作业。 接着前一篇的批量推理与矩阵乘法，我继续把 model(x) 拆开看：一层里有多个输出时，每个数是怎么算出来的？两","https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fcmu-15-779-labs\u002Ftree\u002F4cc63904a3a583da4a2122c246a14b43aac38528","\u002Fmaterials\u002Fcmu-15-779\u002Fexperiments\u002Fpytorch-linear-relu-abs-20261015.zip",{"slug":1619,"path":1620,"title":1621,"date":1622,"tags":1623,"description":1626,"cover":1627,"coverAlt":1628,"excerpt":1629,"draft":1243,"hidden":1243,"published":1246,"readingTime":739,"series":1462,"seriesOrder":652,"codeUrl":1630,"attachmentUrl":1631},"cmu-15-779-pytorch-dataloader-checkpoints-batching","\u002Fposts\u002Fcmu-15-779-pytorch-dataloader-checkpoints-batching","CMU 15-779 学习记录 03：DataLoader、模型保存与批量推理","2026-09-09",[1462,1267,1624,1596,1625],"DataLoader","模型保存","用 DataLoader 分批训练平方函数，保存并加载模型参数，再比较 1000 个样本与 1 个样本的推理耗时，最后把批量计算对应到矩阵乘法。","\u002Fimages\u002Fposts\u002Fcmu-15-779-pytorch-dataloader-checkpoints-batching\u002Fcmu-pytorch-batching-cover-20261015.webp","样本卡片按批次进入计算模块，旁边配有参数存储盒和输出矩阵面板的技术插画","本篇整理的是学习 CMU 15779 时补充的 PyTorch 预备练习。脚本编号不对应课程讲次，也不是课程官方作业。 接着前一篇的平方函数实验，这一节继续练习分批训练、保存参数和独立预测。最后做了一次计时：同一个小网络，逐个处理 ","https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fcmu-15-779-labs","\u002Fmaterials\u002Fcmu-15-779\u002Fexperiments\u002Fpytorch-batching-20261015.zip",{"slug":1633,"path":1634,"title":1635,"date":1636,"tags":1637,"description":1638,"cover":1639,"coverAlt":1640,"excerpt":1641,"draft":1243,"hidden":1243,"published":1246,"readingTime":739,"series":1462,"seriesOrder":290,"codeUrl":1630,"attachmentUrl":1642},"cmu-15-779-pytorch-linear-module-relu","\u002Fposts\u002Fcmu-15-779-pytorch-linear-module-relu","CMU 15-779 学习记录 02：nn.Linear、张量形状与 ReLU","2026-09-06",[1462,1267,1611,1610],"从带偏置的线性回归开始，弄清 nn.Linear 和 nn.Module 的分工，再用平方函数观察线性模型的限制、ReLU 的作用和测试点的选择。","\u002Fimages\u002Fposts\u002Fcmu-15-779-pytorch-linear-module-relu\u002Fcmu-pytorch-relu-cover-20261015.webp","输入经过两层神经网络和 ReLU 折角模块，输出曲线拟合结果的技术插画","本篇整理的是学习 CMU 15779 时补充的 PyTorch 预备练习。脚本编号不对应课程讲次，也不是课程官方作业。 上一篇里，我用一个参数拟合了 $y=2x$。这一篇继续顺着往下写，先加上偏置，再把手写参数换成 nn.Linea","\u002Fmaterials\u002Fcmu-15-779\u002Fexperiments\u002Fpytorch-models-20261015.zip",{"slug":1644,"path":1645,"title":1646,"date":1647,"tags":1648,"description":1651,"cover":1652,"coverAlt":1653,"excerpt":1654,"draft":1243,"hidden":1243,"published":1246,"readingTime":745,"series":1462,"seriesOrder":241,"codeUrl":1630,"attachmentUrl":1655},"cmu-15-779-pytorch-autograd-linear-regression","\u002Fposts\u002Fcmu-15-779-pytorch-autograd-linear-regression","CMU 15-779 学习记录 01：PyTorch 自动求导与线性回归","2026-09-03",[1462,1267,1649,1650],"自动求导","线性回归","从自动求导和手动梯度下降开始，比较学习率，再用 SGD 训练线性回归。记录 w * 4 写错后停在 1.25 的原因和修正结果。","\u002Fimages\u002Fposts\u002Fcmu-15-779-pytorch-autograd-linear-regression\u002Fcmu-pytorch-autograd-cover-20261015.webp","梯度曲面上的小球沿路径下降，前侧面板展示散点与拟合直线的技术插画","本篇整理的是学习 CMU 15779 时补充的 PyTorch 预备练习。脚本编号不对应课程讲次，也不是课程官方作业。 开始学习 CMU 15779 时，我会 Python，但还没用过 PyTorch。这次先在 Mac 上用 CPU","\u002Fmaterials\u002Fcmu-15-779\u002Fexperiments\u002Fpytorch-basics-20261013.zip",{"slug":1657,"path":1658,"title":1659,"date":1660,"tags":1661,"description":1667,"cover":1668,"coverAlt":1669,"excerpt":1670,"draft":1243,"hidden":1243,"published":1246,"readingTime":781,"codeUrl":1671,"attachmentUrl":1672},"embodied-pickplace-lab-robustness","\u002Fposts\u002Fembodied-pickplace-lab-robustness","Pickplace Lab 项目复盘：抓取成功率之外的时延与失败分析","2026-08-29",[1662,1663,1664,1267,1665,1666],"具身智能","模仿学习","MuJoCo","机器人","鲁棒性","从 MuJoCo 抓取搬运闭环出发，记录 22,100 个评测回合。动作平滑在自建任务的 50 ms 延迟下有效，换到 Panda Lift 却没有支持同样的收益。","\u002Fimages\u002Fposts\u002Fembodied-pickplace-lab-robustness\u002Fpickplace-grasp-cover-20261016.webp","MuJoCo 实际渲染的笛卡尔夹爪抓取方块场景，桌面上标出放置目标","Pickplace Lab 从一个简单任务开始：让夹爪抓起方块，搬到指定位置，放下，再退开。先把仿真、示范采集、训练和闭环评测接起来，随后再问：干净条件下成功的策略，换一种执行条件，还能不能完成任务？ 最值得讨论的结果出现在指令延迟","https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fembodied-pickplace-lab\u002Ftree\u002Fv0.1.0","https:\u002F\u002Fgithub.com\u002Fliangqianxing\u002Fembodied-pickplace-lab\u002Freleases\u002Fdownload\u002Fv0.1.0\u002Fpickplace-lab-research-v1.zip",{"slug":1674,"path":1675,"title":1676,"date":1677,"tags":1678,"description":1683,"cover":1684,"coverAlt":1685,"excerpt":1686,"draft":1243,"hidden":1243,"published":1246,"readingTime":1687},"qwen25-vl-sglang-verl-single-gpu-lab","\u002Fposts\u002Fqwen25-vl-sglang-verl-single-gpu-lab","从 Mac 到单卡 GPU：Qwen2.5-VL 部署、GRPO 与计数 LoRA 实践","2026-08-19",[1359,1301,1679,1680,1681,1682,1516],"SGLang","verl","GRPO","推理优化","在 M2 Pro 和云端单卡上部署 Qwen2.5-VL-3B，实测连续批处理、前缀缓存与 SGLang 参数调优，再用 verl 完成多模态 GRPO 更新和同条件前后评测，记录有效优化、兼容性问题与未观察到提升的结果。","\u002Fimages\u002Fposts\u002Fqwen25-vl-sglang-verl-single-gpu-lab\u002Fqvl-lab-cover-20261010.webp","笔记本与单卡计算模块通过图像和 token 光轨连接的实验台插画","这次实验从一个很具体的问题开始：M2 Pro、16 GB 统一内存，能不能在本机跑 Qwen2.5VL？跑起来之后，我又想把图文问答、推理性能测量和强化学习训练接成一条完整流程。 本机的 MLX 4bit 服务和云端的 SGLang",29,{"slug":1689,"path":1690,"title":1691,"date":1692,"tags":1693,"description":1697,"cover":1698,"coverAlt":1699,"excerpt":1700,"draft":1243,"hidden":1243,"published":1246,"readingTime":781},"cycleresearcher-post-training","\u002Fposts\u002Fcycleresearcher-post-training","从 DeepReviewer 2.0 到 CycleResearcher：科研 Agent 如何真正做 Post-training","2026-08-07 17:33:05",[1343,1360,1302,1694,1695,1696],"SimPO","Reward Model","ICLR","区分推理时 Agent 编排与模型后训练，并拆解 CycleResearcher 的领域 SFT、生成式 Reward Model、Iterative SimPO 和评测风险。","\u002Fimages\u002Fposts\u002Fcycleresearcher-post-training\u002Fcycleresearcher-cover-20261018.webp","策略生成模块产出多份候选文稿经多维评审棱镜排序后通过偏好梯度反馈弧闭环优化的等距技术插画","看到一个系统会检索、规划、反思和调用工具，不能据此判断它做过 Posttraining。Agent 描述的是模型在运行时如何完成任务；Posttraining 描述的是预训练之后，是否继续用监督数据、偏好数据或奖励信号更新模型参数。",{"slug":1702,"path":1703,"title":1704,"date":1705,"tags":1706,"description":1708,"cover":1709,"coverAlt":1710,"excerpt":1711,"draft":1243,"hidden":1243,"published":1246,"readingTime":798,"series":1712,"seriesOrder":241},"cuda-fused-rmsnorm-kernel","\u002Fposts\u002Fcuda-fused-rmsnorm-kernel","从 PyTorch 到 CUDA Kernel：融合 RMSNorm 在 RTX 4090D 上加速 4.27 倍","2026-07-22 18:16:50",[1569,1283,1285,1267,1384,1707],"Kernel 优化","从零实现 PyTorch CUDA 扩展，拆解 RMSNorm 的访存瓶颈、warp 归约、向量化加载、算子融合与冷热缓存 benchmark。","\u002Fimages\u002Fposts\u002Fcuda-fused-rmsnorm-kernel\u002Fcuda-rmsnorm-cover-20261018.webp","融合残差加法与向量化 float4 访存经 Warp Shuffle 蝴蝶树归约输出归一化张量的等距硬件架构插画","为了准备 AI Infra 岗位，我做了一个尽量小而完整的 CUDA 项目：不追求堆很多算子，而是选一个真实的 LLM 推理路径，把实现、正确性、benchmark 和性能解释全部做扎实。 项目实现了两个 PyTorch CUDA ","AI Infra 实战",{"slug":1714,"path":1715,"title":1716,"date":1717,"tags":1718,"description":1720,"cover":1721,"coverAlt":1722,"excerpt":1723,"draft":1243,"hidden":1243,"published":1246,"readingTime":733,"series":1724,"seriesOrder":241},"agent-memory","\u002Fposts\u002Fagent-memory","Agent 上下文工程与分层记忆架构：从 Observation 剪枝、滚动摘要到向量召回闭环","2026-04-24",[1343,1302,1719,1283],"RAG","结合生产级多约束 Agent 与超 100K Token 科研智能体实践，系统拆解长程会话中的上下文膨胀与注意力稀释问题：涵盖 Observation 结构化剪枝（降低约 40% Token 开销）、滚动工作摘要、向量检索长期记忆及 Prefix Cache 友好的上下文组装。","\u002Fimages\u002Fposts\u002Fagent-memory\u002Fagent-memory-cover-20261018.webp","底层高噪数据经剪枝滤网进入中期滑动窗口与金色状态台账并由顶层向量星座按需召回的三层记忆架构插画","在构建多步工具调用（ToolUse）与长周期自主规划（Autonomous Planning）的 Agent 系统时，上下文管理（Context Engineering）往往比单纯改写 System Prompt 更决定线上稳定性。","Agent 上下文工程",{"slug":1726,"path":1727,"title":1728,"date":1729,"tags":1730,"description":1733,"cover":1734,"coverAlt":1735,"excerpt":1736,"draft":1243,"hidden":1243,"published":1246,"readingTime":670,"series":1737,"seriesOrder":290},"backend-stack-deep-dive","\u002Fposts\u002Fbackend-stack-deep-dive","高并发 AI 平台后端基础设施实践：FastAPI 异步网关、SQLAlchemy 2.0 Async、PostgreSQL 与 Docker 沙箱编排","2026-04-07",[1388,1385,1387,1731,1732,1283],"PostgreSQL","Docker","结合 DeepScientist 生产级 AI 科研平台实践，拆解 FastAPI ASGI 异步事件循环、SQLAlchemy 2.0 + AsyncPG 连接池调优、PostgreSQL + MinIO 存储分层与 Docker SDK 动态沙箱容器编排的职责边界与核心设计。","\u002Fimages\u002Fposts\u002Fbackend-stack-deep-dive\u002Fbackend-stack-cover-20261018.webp","ASGI 异步事件循环涡轮将非阻塞协程流分发至 PostgreSQL 连接池、MinIO 存储矩阵与动态 Docker 沙箱的等距架构插画","结合 DeepScientist 生产级 AI 科研平台（服务 7,000+ 用户）的实际工程落地，本文系统拆解支撑高并发流式推理与容器沙箱调度的后端基础设施栈——FastAPI（ASGI 异步网关 + SSE 流式推送）、SQLA","DeepScientist 工程复盘",{"slug":1739,"path":1740,"title":1741,"date":1742,"tags":1743,"description":1746,"cover":1747,"coverAlt":1748,"excerpt":1749,"draft":1243,"hidden":1243,"published":1246,"readingTime":751,"series":1750,"seriesOrder":241},"minicode-source-analysis","\u002Fposts\u002Fminicode-source-analysis","MiniCode 架构与工具评测深度拆解：开源终端 Agentic Coding 引擎（1k+ Stars）的 ReAct、MCP 与 30 项基准验证","2026-04-06",[1302,1744,1745,1332,1283],"TypeScript","MCP","深度剖析开源终端 AI 编程智能体 MiniCode（1k+ GitHub Stars）的核心架构：从零重型依赖的 ReAct 规划循环、声明式工具注册表与权限拦截层、MCP 协议帧自动协商，到 30 项可执行 Tool-Use 评测基准（86.7% 完成率、100% 错误恢复率）的失败分布归因与闭环优化。","\u002Fimages\u002Fposts\u002Fminicode-source-analysis\u002Fminicode-cover-20261018.webp","终端编程智能体控制台被 ReAct 状态环、Unified Diff 补丁预览门与 MCP 协议插槽环绕的等距技术插画","MiniCode 是我作为核心开发者参与构建的开源终端 Agentic Coding 引擎（目前已在 GitHub 获得 1,000+ Stars）。项目的核心设计目标是：在不引入庞大运行时框架的前提下，用约 5,000 行高内聚的","AI 项目源码导读",{"slug":1752,"path":1753,"title":1754,"date":1742,"tags":1755,"description":1756,"cover":1757,"coverAlt":1758,"excerpt":1759,"draft":1243,"hidden":1243,"published":1246,"readingTime":745,"series":1737,"seriesOrder":241},"deepscientist-tech-stack","\u002Fposts\u002Fdeepscientist-tech-stack","DeepScientist 架构与运行时全解析：超 100K 上下文科研智能体、MCP 沙箱与高并发流式系统",[1302,1283,1745,1732,1387,1385],"深度拆解服务 7,000+ 用户的科研自主智能体平台 DeepScientist：从一题一仓库（Quest）持久化状态机、MCP 工具栈（memory \u002F artifact \u002F bash_exec）、Docker SDK 动态容器编排，到 SSE + WebSocket 双流通信与 PostgreSQL \u002F Redis \u002F Prometheus 生产级基础设施。","\u002Fimages\u002Fposts\u002Fdeepscientist-tech-stack\u002Fdeepscientist-cover-20261018.webp","科研智能体编排核心通过 memory、artifact 与隔离容器沙箱三座协议桥连接双流工作台的等距架构插画","DeepScientist（deepscientist.cc）是我们在西湖大学自然语言处理实验室研发的自主科学发现与科研管理平台，目前累计服务 7,000+ 科研用户。与单轮对话或纯文本文献问答工具不同，科研智能体需要在数小时乃至数",1795088946175]