主题
参考资料索引
以下按章节收录原始链接。每项主张与来源的对应关系、固定版本、实验条件和未验证范围，以正文为准。
第 1 章 导数与计算图：程序怎样知道往哪儿改
- https://github.com/karpathy/nn-zero-to-hero/blob/73c3fcc741f0ec104ca850b1fb0df90e7e8d4cde/lectures/micrograd/micrograd_lecture_first_half_roughly.ipynb
- https://colab.research.google.com/drive/1FPTx1RXtBfc4MaTkf7viZZD4U2F9gtKN?usp=sharing
- https://github.com/karpathy/nn-zero-to-hero/blob/73c3fcc741f0ec104ca850b1fb0df90e7e8d4cde/lectures/micrograd/micrograd_lecture_second_half_roughly.ipynb
第 2 章 写出反向传播引擎
- https://github.com/karpathy/nn-zero-to-hero/blob/73c3fcc741f0ec104ca850b1fb0df90e7e8d4cde/lectures/micrograd/micrograd_lecture_second_half_roughly.ipynb
- https://github.com/karpathy/micrograd/blob/7bc720e951fe422b8f8814aa5aa1b64121d26b4c/micrograd/engine.py
- https://github.com/pytorch/pytorch/blob/v2.6.0/aten/src/ATen/native/cpu/BinaryOpsKernel.cpp
- https://github.com/karpathy/nn-zero-to-hero/blob/73c3fcc741f0ec104ca850b1fb0df90e7e8d4cde/lectures/micrograd/micrograd_lecture_first_half_roughly.ipynb
- https://colab.research.google.com/drive/1FPTx1RXtBfc4MaTkf7viZZD4U2F9gtKN?usp=sharing
第 3 章 让小网络学会预测
- https://github.com/karpathy/nn-zero-to-hero/blob/73c3fcc741f0ec104ca850b1fb0df90e7e8d4cde/lectures/micrograd/micrograd_lecture_first_half_roughly.ipynb
- https://github.com/karpathy/micrograd/blob/7bc720e951fe422b8f8814aa5aa1b64121d26b4c/micrograd/nn.py
- https://github.com/karpathy/micrograd/blob/7bc720e951fe422b8f8814aa5aa1b64121d26b4c/demo.ipynb
- https://github.com/karpathy/nn-zero-to-hero/blob/73c3fcc741f0ec104ca850b1fb0df90e7e8d4cde/lectures/micrograd/micrograd_lecture_second_half_roughly.ipynb
第 4 章 把名字变成字符数据
- https://github.com/karpathy/nn-zero-to-hero/blob/73c3fcc741f0ec104ca850b1fb0df90e7e8d4cde/lectures/makemore/makemore_part1_bigrams.ipynb
- https://github.com/karpathy/makemore/blob/988aa59e4d8fefa526d06f3b453ad116258398d4/names.txt
第 5 章 从计数得到概率、采样和损失
- https://docs.pytorch.org/docs/2.6/notes/broadcasting.html
- https://github.com/pytorch/pytorch/blob/v2.6.0/torch/_torch_docs.py
- https://github.com/karpathy/nn-zero-to-hero/blob/73c3fcc741f0ec104ca850b1fb0df90e7e8d4cde/lectures/makemore/makemore_part1_bigrams.ipynb
第 6 章 用神经网络重写 bigram
- https://github.com/karpathy/nn-zero-to-hero/blob/73c3fcc741f0ec104ca850b1fb0df90e7e8d4cde/lectures/makemore/makemore_part1_bigrams.ipynb
- https://github.com/pytorch/pytorch/blob/v2.6.0/torch/nn/functional.py
第 7 章 让模型看见上下文：嵌入与 MLP
- https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf
- https://github.com/karpathy/nn-zero-to-hero/blob/73c3fcc741f0ec104ca850b1fb0df90e7e8d4cde/lectures/makemore/makemore_part2_mlp.ipynb
- https://github.com/pytorch/pytorch/blob/v2.6.0/torch/_tensor_docs.py
- https://colab.research.google.com/drive/1YIfmkftLrz6MPTOO9Vwqrop2Q5llHIGK?usp=sharing
第 8 章 让训练过程可比较
第 9 章 容量、泛化与一套完整实验
- https://www.youtube.com/watch?v=PaCmpygFfXo
- https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf
- https://colab.research.google.com/drive/1FPTx1RXtBfc4MaTkf7viZZD4U2F9gtKN?usp=sharing
- https://www.youtube.com/watch?v=TCH_1BHY58I
第 10 章 让信号穿过网络：从初始损失到权重尺度
- https://github.com/karpathy/nn-zero-to-hero/blob/73c3fcc741f0ec104ca850b1fb0df90e7e8d4cde/lectures/makemore/makemore_part3_bn.ipynb
- https://docs.pytorch.org/docs/2.6/nn.init.html
- https://arxiv.org/abs/1502.01852
- https://www.youtube.com/watch?v=P6sfmUTpUmc
第 11 章 BatchNorm：训练时看一批，推理时用固定统计量
- https://arxiv.org/abs/1502.03167
- https://github.com/pytorch/pytorch/blob/v2.6.0/torch/nn/modules/batchnorm.py
- https://www.youtube.com/watch?v=P6sfmUTpUmc
- https://github.com/karpathy/nn-zero-to-hero/blob/73c3fcc741f0ec104ca850b1fb0df90e7e8d4cde/lectures/makemore/makemore_part3_bn.ipynb
第 12 章 训练诊断：不要只盯着一个 loss
- https://github.com/karpathy/nn-zero-to-hero/blob/73c3fcc741f0ec104ca850b1fb0df90e7e8d4cde/lectures/makemore/makemore_part3_bn.ipynb
- https://github.com/pytorch/pytorch/blob/v2.6.0/torch/_tensor_docs.py
- https://github.com/pytorch/vision/blob/v0.21.0/torchvision/models/resnet.py
- https://www.youtube.com/watch?v=P6sfmUTpUmc
第 13 章 把反向传播写成张量运算
- https://colab.research.google.com/drive/1WV2oi2fh9XXyldh02wupFQX0wh5ZC-z-?usp=sharing
- https://www.youtube.com/watch?v=q8SA3rM6ckI
- https://github.com/karpathy/nn-zero-to-hero/blob/73c3fcc741f0ec104ca850b1fb0df90e7e8d4cde/lectures/makemore/makemore_part4_backprop.ipynb
第 14 章 亲手走完 BatchNorm 与 embedding 的反向传播
- https://github.com/karpathy/nn-zero-to-hero/blob/73c3fcc741f0ec104ca850b1fb0df90e7e8d4cde/lectures/makemore/makemore_part4_backprop.ipynb
- https://www.youtube.com/watch?v=q8SA3rM6ckI
- https://colab.research.google.com/drive/1WV2oi2fh9XXyldh02wupFQX0wh5ZC-z-?usp=sharing
第 15 章 把八个字符逐层合并：层接口与树形上下文
- https://www.youtube.com/watch?v=t3YJ5hKiMQ0
- https://github.com/pytorch/pytorch/blob/v2.6.0/torch/nn/modules/module.py
- https://github.com/pytorch/pytorch/blob/v2.6.0/torch/nn/modules/batchnorm.py
- https://github.com/karpathy/nn-zero-to-hero/blob/73c3fcc741f0ec104ca850b1fb0df90e7e8d4cde/lectures/makemore/makemore_part5_cnn1.ipynb
- https://colab.research.google.com/drive/1CXVEmCO_7r7WYZGb5qnjfyxTvQa13g5X?usp=sharing
第 16 章 从树到卷积：感受野、因果性与可靠迭代
- https://github.com/pytorch/pytorch/blob/v2.6.0/torch/nn/modules/conv.py
- https://arxiv.org/abs/1609.03499
- https://www.youtube.com/watch?v=t3YJ5hKiMQ0
- https://github.com/karpathy/nn-zero-to-hero/blob/73c3fcc741f0ec104ca850b1fb0df90e7e8d4cde/lectures/makemore/makemore_part5_cnn1.ipynb
第 17 章 先把 GPT 的训练题出对
- https://www.youtube.com/watch?v=kCc8FmEb1nY
- https://github.com/karpathy/ng-video-lecture/blob/52201428ed7b46804849dea0b3ccf0de9df1a5c3/gpt.py
- https://docs.pytorch.org/docs/2.6/generated/torch.nn.Embedding.html
- https://docs.pytorch.org/docs/2.6/generated/torch.nn.CrossEntropyLoss.html
第 18 章 让当前位置挑选有用的过去
- https://colab.research.google.com/drive/1JMLa53HDuA-i7ZBmqV7ZnA3c_fvtXnx-
- https://github.com/karpathy/ng-video-lecture/blob/52201428ed7b46804849dea0b3ccf0de9df1a5c3/gpt.py
- https://www.youtube.com/watch?v=kCc8FmEb1nY
- https://arxiv.org/abs/1706.03762
- https://docs.pytorch.org/docs/2.6/generated/torch.nn.functional.scaled_dot_product_attention.html
第 19 章 把注意力装成能反复堆叠的块
- https://github.com/karpathy/ng-video-lecture/blob/52201428ed7b46804849dea0b3ccf0de9df1a5c3/gpt.py
- https://github.com/karpathy/nanoGPT/blob/3adf61e154c3fe3fca428ad6bc3818b27a3b8291/model.py
- https://arxiv.org/abs/1512.03385
- https://docs.pytorch.org/docs/2.6/generated/torch.nn.LayerNorm.html
第 20 章 用一条可检查的训练链路跑通小 GPT
- https://github.com/karpathy/nanoGPT/blob/3adf61e154c3fe3fca428ad6bc3818b27a3b8291/model.py
- https://github.com/karpathy/nanoGPT/blob/3adf61e154c3fe3fca428ad6bc3818b27a3b8291/train.py
- https://arxiv.org/abs/1711.05101
- https://docs.pytorch.org/docs/2.6/generated/torch.optim.AdamW.html
- https://arxiv.org/abs/2005.14165
- https://arxiv.org/abs/2203.02155
第 21 章 模型看到的不是屏幕上的文字
- https://www.youtube.com/watch?v=zduSFxRajkE
- https://github.com/karpathy/minbpe/tree/1acefe89412b20245db5a22d2a02001e547dc602
- https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-3/
- https://github.com/karpathy/minbpe/blob/1acefe89412b20245db5a22d2a02001e547dc602/minbpe/basic.py
第 22 章 BPE 学的是一张有先后顺序的合并表
- https://github.com/karpathy/minbpe/blob/1acefe89412b20245db5a22d2a02001e547dc602/exercise.md
- https://arxiv.org/abs/1508.07909
- https://github.com/karpathy/minbpe/blob/1acefe89412b20245db5a22d2a02001e547dc602/minbpe/base.py
- https://github.com/karpathy/minbpe/blob/1acefe89412b20245db5a22d2a02001e547dc602/minbpe/basic.py
第 23 章 相同的 BPE 算法，为什么还编码不出相同的 ID
- https://github.com/karpathy/minbpe/blob/1acefe89412b20245db5a22d2a02001e547dc602/exercise.md
- https://github.com/karpathy/minbpe/blob/1acefe89412b20245db5a22d2a02001e547dc602/minbpe/regex.py
- https://github.com/openai/tiktoken/blob/0.9.0/tiktoken_ext/openai_public.py
- https://github.com/openai/gpt-2/blob/9b63575ef42771a015060c964af2c3da4cf7c8ab/src/encoder.py
- https://github.com/openai/tiktoken/blob/0.9.0/tiktoken/core.py
- https://github.com/karpathy/minbpe/blob/1acefe89412b20245db5a22d2a02001e547dc602/minbpe/gpt4.py
第 24 章 tokenizer 的选择，会一直留在模型里
- https://www.youtube.com/watch?v=zduSFxRajkE
- https://github.com/google/sentencepiece/blob/v0.2.0/README.md
- https://aclanthology.org/D18-2012/
- https://github.com/google/sentencepiece/blob/v0.2.0/src/sentencepiece_model.proto
- https://arxiv.org/abs/1711.00937
第 25 章 从小 GPT 到 GPT-2：先把模型对上，再谈训练
- https://www.youtube.com/watch?v=l8pRSuU81PU
- https://github.com/karpathy/build-nanogpt/tree/6104ab1b53920f6e2159749676073ff7d815c1fa
- https://github.com/karpathy/build-nanogpt/blob/6104ab1b53920f6e2159749676073ff7d815c1fa/train_gpt2.py
- https://github.com/karpathy/build-nanogpt/blob/6104ab1b53920f6e2159749676073ff7d815c1fa/play.ipynb
- https://docs.pytorch.org/docs/2.6/generated/torch.nn.functional.scaled_dot_product_attention.html
第 26 章 让同一份计算跑得更快：精度、编译和注意力
- https://www.youtube.com/watch?v=l8pRSuU81PU
- https://github.com/karpathy/build-nanogpt/blob/6104ab1b53920f6e2159749676073ff7d815c1fa/train_gpt2.py
- https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html
- https://docs.pytorch.org/docs/2.6/notes/amp_examples.html
- https://arxiv.org/abs/2205.14135
- https://docs.pytorch.org/docs/2.6/generated/torch.nn.functional.scaled_dot_product_attention.html
第 27 章 一次更新到底看了多少数据：AdamW、累积与多卡
- https://github.com/karpathy/build-nanogpt/blob/6104ab1b53920f6e2159749676073ff7d815c1fa/train_gpt2.py
- https://arxiv.org/abs/2005.14165
- https://github.com/karpathy/build-nanogpt/blob/6104ab1b53920f6e2159749676073ff7d815c1fa/README.md
- https://github.com/pytorch/pytorch/blob/v2.6.0/torch/optim/adamw.py
- https://github.com/pytorch/pytorch/blob/v2.6.0/torch/nn/parallel/distributed.py
第 28 章 数据、评估和恢复：什么才算复现了一次训练
- https://github.com/karpathy/build-nanogpt/blob/6104ab1b53920f6e2159749676073ff7d815c1fa/fineweb.py
- https://github.com/karpathy/build-nanogpt/blob/6104ab1b53920f6e2159749676073ff7d815c1fa/train_gpt2.py
- https://github.com/karpathy/build-nanogpt/blob/6104ab1b53920f6e2159749676073ff7d815c1fa/hellaswag.py
- https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
- https://arxiv.org/abs/2005.14165
- https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu
- https://arxiv.org/abs/1905.07830
第 29 章 一个会续写的模型，怎样变成助手
- https://www.youtube.com/watch?v=bZQun8Y4L2A
- https://arxiv.org/abs/2005.14165
- https://arxiv.org/abs/2203.02155
- https://arxiv.org/abs/1707.06347
第 30 章 把模型接进工作流程：提示、检索、微调和工具
- https://www.youtube.com/watch?v=bZQun8Y4L2A
- https://arxiv.org/abs/2201.11903
- https://arxiv.org/abs/2005.11401
- https://arxiv.org/abs/2210.03629