- 【中&英】《生成人工智能手册》Generative AI Handbook: A Roadmap for Learning Resources
- Introduction 介绍
- The AI Landscape 人工智能前景
- The Content Landscape 内容格局
- Resources 资源
- Organization 组织
- Section I: Foundations of Sequential Prediction 第一节:序列预测的基础
- Chapter 1: Preliminaries 第一章:预备知识
- Math 数学
- Programming 编程
- Chapter 2: Statistical Prediction and Supervised Learning 第 2 章:统计预测和监督学习
- Chapter 3: Time-Series Analysis 第 3 章:时间序列分析
- Chapter 4: Online Learning and Regret Minimization 第 4 章:在线学习和遗憾最小化
- Chapter 5: Reinforcement Learning 第 5 章:强化学习
- Chapter 6: Markov Models 第 6 章:马尔可夫模型
- Section II: Neural Sequential Prediction 第二节:神经序列预测
- Chapter 7: Statistical Prediction with Neural Networks 第 7 章:神经网络统计预测
- Chapter 8: Recurrent Neural Networks 第 8 章:循环神经网络
- Chapter 9: LSTMs and GRUs 第 9 章:LSTM 和 GRU
- Chapter 10: Embeddings and Topic Modeling 第 10 章:嵌入和主题建模
- Chapter 11: Encoders and Decoders 第 11 章:编码器和解码器
- Chapter 12: Decoder-Only Transformers 第 12 章:仅解码器变压器
- Section III: Foundations for Modern Language Modeling 第三节:现代语言建模的基础
- Chapter 13: Tokenization 第13章:代币化
- Chapter 14: Positional Encoding 第14章:位置编码
- Chapter 15: Pretraining Recipes 第15章:预训练食谱
- Chapter 16: Distributed Training and FSDP 第 16 章:分布式训练和 FSDP
- Chapter 17: Scaling Laws 第17章:缩放法则
- Chapter 18: Mixture-of-Experts 第18章:专家混合
- Section IV: Finetuning Methods for LLMs 第四节:LLMs的微调方法
- Chapter 19: Instruct Fine-Tuning 第19章:指导微调
- Chapter 20: Low-Rank Adapters (LoRA) 第 20 章:低阶适配器 (LoRA)
- Chapter 21: Reward Models and RLHF 第 21 章:奖励模型和 RLHF
- Chapter 22: Direct Preference Optimization Methods 第22章:直接偏好优化方法
- Chapter 23: Context Scaling 第23章:上下文缩放
【中&英】《生成人工智能手册》Generative AI Handbook: A Roadmap for Learning Resources
【中&英】《生成人工智能手册》Generative AI Handbook: A Roadmap for Learning Resources
2024年6月20日创建
Introduction 介绍
This document aims to serve as a handbook for learning the key concepts underlying modern artificial intelligence systems. Given the speed of recent development in AI, there really isn’t a good textbook-style source for getting up-to-speed on the latest-and-greatest innovations in LLMs or other generative models, yet there is an abundance of great explainer resources (blog posts, videos, etc.) for these topics scattered across the internet. My goal is to organize the “best” of these resources into a textbook-style presentation, which can serve as a roadmap for filling in the prerequisites towards individual AI-related learning goals. My hope is that this will be a “living document”, to be updated as new innovations and paradigms inevitably emerge, and ideally also a document that can benefit from community input and contribution. This guide is aimed at those with a technical background of some kind, who are interested in diving into AI either out of curiosity or for a potential career. I’ll assume that you have some experience with coding and high-school level math, but otherwise will provide pointers for filling in any other prerequisites. Please let me know if there’s anything you think should be added!
本文档旨在作为学习现代人工智能系统背后的关键概念的手册。考虑到人工智能最近的发展速度,确实没有一个好的教科书式的资源来快速了解 LLMs 或其他生成模型的最新和最伟大的创新,但互联网上有大量关于这些主题的优秀解释资源(博客文章、视频等)。我的目标是将这些资源中的“最好的”组织成教科书式的演示文稿,它可以作为满足个人人工智能相关学习目标的先决条件的路线图。我希望这将是一份“活文件”,随着新的创新和范式不可避免地出现而进行更新,并且理想情况下也是一份可以从社区投入和贡献中受益的文件。本指南针对的是那些具有某种技术背景、出于好奇或潜在职业而有兴趣深入研究人工智能的人。我假设您有一些编码和高中数学水平的经验,但否则将提供填写任何其他先决条件的指导。如果您认为有什么需要补充的,请告诉我!
The AI Landscape 人工智能前景
As of June 2024, it’s been about 18 months since ChatGPT was released by OpenAI and the world started talking a lot more about artificial intelligence. Much has happened since: tech giants like Meta and Google have released large language models of their own, newer organizations like Mistral and Anthropic have proven to be serious contenders as well, innumerable startups have begun building on top of their APIs, everyone is scrambling for powerful Nvidia GPUs, papers appear on ArXiv at a breakneck pace, demos circulate of physical robots and artificial programmers powered by LLMs, and it seems like chatbots are finding their way into all aspects of online life (to varying degrees of success). In parallel to the LLM race, there’s been rapid development in image generation via diffusion models; DALL-E and Midjourney are displaying increasingly impressive results that often stump humans on social media, and with the progress from Sora, Runway, and Pika, it seems like high-quality video generation is right around the corner as well. There are ongoing debates about when “AGI” will arrive, what “AGI” even means, the merits of open vs. closed models, value alignment, superintelligence, existential risk, fake news, and the future of the economy. Many are concerned about jobs being lost to automation, or excited about the progress that automation might drive. And the world keeps moving: chips get faster, data centers get bigger, models get smarter, contexts get longer, abilities are augmented with tools and vision, and it’s not totally clear where this is all headed. If you’re following “AI news” in 2024, it can often feel like there’s some kind of big new breakthrough happening on a near-daily basis. It’s a lot to keep up with, especially if you’re just tuning in.
截至 2024 年 6 月,自 OpenAI 发布 ChatGPT 以来已经过去了大约 18 个月,世界开始更多地谈论人工智能。此后发生了很多事情:Meta 和 Google 等科技巨头发布了自己的大型语言模型,Mistral 和 Anthropic 等较新的组织也被证明是有力的竞争者,无数初创公司开始在他们的 API 基础上构建,每个人都在争夺强大的 Nvidia GPU、论文以极快的速度出现在 ArXiv 上、由 LLMs 驱动的物理机器人和人工程序员的演示在流传,聊天机器人似乎正在寻找进入在线生活的各个方面的方法(在不同程度上)的成功)。与LLM竞赛同时进行的,通过扩散模型生成图像也取得了快速发展; DALL-E 和 Midjourney 正在展示越来越令人印象深刻的结果,这些结果常常在社交媒体上难倒人们,随着 Sora、Runway 和 Pika 的进步,高质量视频生成似乎也指日可待。关于“AGI”何时到来、“AGI”到底意味着什么、开放模型与封闭模型的优点、价值一致性、超级智能、存在风险、假新闻和经济的未来,一直存在争论。许多人担心自动化会导致工作岗位流失,或者对自动化可能推动的进步感到兴奋。世界在不断发展:芯片变得更快,数据中心变得更大,模型变得更智能,上下文变得更长,能力通过工具和愿景得到增强,但目前尚不完全清楚这一切的发展方向。如果您在 2024 年关注“人工智能新闻”,您通常会感觉几乎每天都会发生某种重大的新突破。有很多事情需要跟上,尤其是当你刚刚收听时。