参考资料索引

以下按章节收录原始链接。每项主张与来源的对应关系、固定版本、实验条件和未验证范围，以正文为准。

第 01 章　先画清语言模型的任务与边界

• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch01/README.md
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/01_main-chapter-code/gpt_train.py
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch01/reading-recommendations.md
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch01

第 02 章　张量、自动微分与最小训练闭环

• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/appendix-A/01_main-chapter-code
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/appendix-A/01_main-chapter-code/code-part1.ipynb
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/appendix-A/01_main-chapter-code/code-part2.ipynb
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/appendix-A/01_main-chapter-code/DDP-script-torchrun.py
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/appendix-A/01_main-chapter-code/exercise-solutions.ipynb

第 03 章　文本怎样变成 token：词表、BPE 与特殊符号

• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch02
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch02/06_bonus_simple-tokenizer-v3/simple-tokenizer-v3.ipynb
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch02/02_bonus_bytepair-encoder/bpe_openai_gpt2.py
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch02/01_main-chapter-code
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch02/02_bonus_bytepair-encoder
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch02/03_bonus_embedding-vs-matmul
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch02/04_bonus_dataloader-intuition
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch02/05_bpe-from-scratch

第 04 章　从加权平均推导自注意力与因果掩码

• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch03/01_main-chapter-code/ch03.ipynb
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch03/01_main-chapter-code/multihead-attention.ipynb
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch03/01_main-chapter-code/exercise-solutions.ipynb

第 05 章　多头注意力的实现选择与 buffer 生命周期

• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch03/02_bonus_efficient-multihead-attention/mha-implementations.ipynb
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch03/03_understanding-buffers/understanding-buffers.ipynb

第 06 章　把注意力组装成 GPT：归一化、残差与输出头

• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch04/01_main-chapter-code
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch04/01_main-chapter-code/ch04.ipynb
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch04/01_main-chapter-code/gpt.py
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch04/01_main-chapter-code/exercise-solutions.ipynb

第 07 章　参数、FLOPs、显存与吞吐怎样估算

• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch04/02_performance-analysis/flops-analysis.ipynb
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch04/02_performance-analysis/requirements-extra.txt

第 08 章　KV cache：先证明等价，再讨论加速

• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch04/03_kv-cache
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch04/03_kv-cache/tests.py

第 09 章　GQA、MLA 与跨层 KV 共享

• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch04/04_gqa
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch04/05_mla
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch04/10_kv-sharing
• https://arxiv.org/abs/2305.13245
• https://arxiv.org/abs/2405.04434
• https://arxiv.org/abs/2405.12981
• https://arxiv.org/abs/2305.13245。
• https://arxiv.org/abs/2405.04434。
• https://arxiv.org/abs/2405.12981。

第 10 章　局部与稀疏注意力：SWA 和 DSA

• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch04/06_swa/gpt_with_kv_swa.py
• https://arxiv.org/abs/2004.05150
• https://arxiv.org/abs/2503.19786
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch04/09_dsa/gpt_with_kv_dsa.py
• https://huggingface.co/deepseek-ai/DeepSeek-V3.2/resolve/main/assets/paper.pdf
• https://arxiv.org/abs/2004.05150。
• https://arxiv.org/abs/2503.19786。

第 11 章　Gated DeltaNet：从矩阵状态理解递推

• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch04/08_deltanet
• https://arxiv.org/abs/2412.06464
• https://arxiv.org/abs/2412.06464。

第 12 章　MoE：路由、专家与负载

• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch04/07_moe
• https://arxiv.org/abs/2101.03961
• https://arxiv.org/abs/2002.05202。
• https://arxiv.org/abs/2101.03961。

第 13 章　预训练目标、交叉熵与验证集

• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/01_main-chapter-code/gpt_train.py
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/01_main-chapter-code/ch05.ipynb

第 14 章　采样、温度、top-k 与生成边界

• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/01_main-chapter-code/gpt_generate.py
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/01_main-chapter-code/ch05.ipynb
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/01_main-chapter-code/exercise-solutions.ipynb

第 15 章　权重格式、参数映射与低内存加载

• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/02_alternative_weight_loading
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/01_main-chapter-code/gpt_generate.py
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/01_main-chapter-code/gpt_download.py
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/08_memory_efficient_weight_loading/memory-efficient-state-dict.ipynb

第 16 章　语料整理与 Gutenberg 训练工程

• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/03_bonus_pretraining_on_gutenberg
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/03_bonus_pretraining_on_gutenberg/prepare_dataset.py
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/03_bonus_pretraining_on_gutenberg/pretraining_simple.py
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/03_bonus_pretraining_on_gutenberg/tests.py

第 17 章　学习率、梯度裁剪、超参数与 Muon

• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/appendix-D/01_main-chapter-code/appendix-D.ipynb
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/05_bonus_hparam_tuning/hparam_search.py
• https://arxiv.org/abs/2502.16982
• https://docs.pytorch.org/docs/2.14/generated/torch.optim.Muon.html
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/18_muon/gpt_train_muon.py
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/04_learning_rate_schedulers/README.md
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/18_muon

第 18 章　训练性能及 CPU、MPS 数值差异

• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/10_llm-training-speed
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/19_cpu_mps_differences/1_cpu_mps_random_generator.ipynb

第 19 章　从 GPT 到 Llama：RoPE、RMSNorm 与门控前馈

• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/07_gpt_to_llama
• https://arxiv.org/abs/1910.07467
• https://arxiv.org/abs/2002.05202
• https://arxiv.org/abs/2104.09864
• https://arxiv.org/abs/1910.07467；GLU
• https://arxiv.org/abs/2002.05202。
• https://arxiv.org/abs/2104.09864。
• https://arxiv.org/abs/2307.09288；Llama
• https://arxiv.org/abs/2407.21783。

第 20 章　Qwen3 与 Qwen3.5：同一个名字下，两种记忆接口

• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/11_qwen3
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/16_qwen3.5
• https://arxiv.org/abs/2505.09388
• https://arxiv.org/abs/2412.06464。
• https://arxiv.org/abs/2505.09388。
• https://huggingface.co/Qwen/Qwen3.5-0.8B。

第 21 章　Gemma 3、Gemma 4：局部层、共享层与数值约定

• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/12_gemma3
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/17_gemma4
• https://arxiv.org/abs/2503.19786
• https://arxiv.org/abs/2503.19786。
• https://huggingface.co/google/gemma-4-E2B。
• https://huggingface.co/google/gemma-4-E4B。

第 22 章　OLMo 3、Tiny Aya 与可替换模型接口

• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/13_olmo3
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/15_tiny-aya
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/14_ch05_with_other_llms
• https://arxiv.org/abs/2309.00071
• https://arxiv.org/abs/2309.00071。
• https://huggingface.co/CohereLabs/tiny-aya-global。
• https://huggingface.co/allenai/Olmo-3-7B-Instruct。

第 23 章　扩展分词器：新 ID 还需要新的模型行

• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/09_extending-tokenizers/extend-tiktoken.ipynb
• https://github.com/openai/tiktoken
• https://github.com/openai/tiktoken。

第24章　文本分类：一个输出位置，决定模型看见多少内容

• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch06/01_main-chapter-code/gpt_class_finetune.py
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch06/03_bonus_imdb-classification
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch06/01_main-chapter-code/ch06.ipynb
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch06/02_bonus_additional-experiments/additional_experiments.py
• https://docs.pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html

第25章　指令微调：先把“该预测什么”排对

• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch07/01_main-chapter-code/gpt_instruction_finetuning.py
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch07/01_main-chapter-code/ch07.ipynb
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch07/01_main-chapter-code/exercise_experiments.py
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch07/01_main-chapter-code/exercise-solutions.ipynb

第26章　指令数据：先防止自己给自己打高分

• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch07/02_dataset-utilities/find-near-duplicates.py
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch07/02_dataset-utilities
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch07/03_model-evaluation
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch07/05_dataset-generation

第27章　DPO：比较两个回答，也要说明比较的基准

• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch07/04_preference-tuning-with-dpo/create-preference-data-ollama.ipynb
• https://arxiv.org/abs/2305.18290
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch07/04_preference-tuning-with-dpo/dpo-from-scratch.ipynb

第28章　LoRA：冻结大矩阵，学习一个可合并的增量

• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/appendix-E/01_main-chapter-code/appendix-E.ipynb
• https://arxiv.org/abs/2106.09685
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/pkg/llms_from_scratch/appendix_e.py

第29章　三种界面，同一条推理链路

• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch05/06_user_interface
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch06/04_user_interface/app.py
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch07/06_user_interface/app.py

第30章　包接口与测试：怎样知道自己运行的是同一份模型

• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/pkg/llms_from_scratch/README.md
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/pkg/llms_from_scratch/generate.py
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/setup
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/pkg/llms_from_scratch
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/pkg/llms_from_scratch/tests
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/.github
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/troubleshooting.md

第31章　习题闭环与延伸：知道哪里已经验证，哪里还没有

• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/appendix-C/README.md
• https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a71e20ea31434841095fac50cecd8/.gitmodules
• https://github.com/rasbt/reasoning-from-scratch
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch06
• https://github.com/rasbt/LLMs-from-scratch/tree/cdbd33e6d57a71e20ea31434841095fac50cecd8/ch07
• https://arxiv.org/abs/2106.09685
• https://arxiv.org/abs/2305.18290
• https://docs.pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html