18-LLMs微调
输入“/”快速插入内容
18-LLMs微调
18-LLMs微调
2024年8月19日修改
本文讨论了使用大型语言模型构建生成式AI应用程序时确保响应质量的挑战，重点介绍微调技术，包括其概念、利弊、适用场景及操作方法等。关键要点包括：
1.
微调概念：微调是机器学习中用新数据重训预训练模型以提升特定任务性能的技术，在语言模型中可创建更精准相关的自定义模型，还能减少小样本学习样本数及成本。
2.
小样本学习局限：模型令牌限制会约束示例数量、降低有效性，模型令牌成本高会限制灵活性。
3.
微调类型：这里指监督微调，通过添加新数据重训，与无监督微调不同。
4.
决策考量：决定是否微调需考虑用例、替代方案、成本、可调性、努力、计算、数据、好处等多方面因素，收益大于成本才可行。
5.
微调条件：微调预训练模型需具备预训练模型、数据集、训练环境和托管环境。
6.
学习资源：有分步教程助于实践微调，完成课程可查看生成式AI学习合集继续学习 。
7.
课程更新：v1系列课程更新了更多作业和概念，鼓励分享问题和反馈 。
Using large language models to build generative AI applications comes with new challenges. A key issue is ensuring response quality (accuracy and relevance) in content generated by the model for a given user request. In previous lessons, we discussed techniques like prompt engineering and retrieval-augmented generation that try to solve the problem by modifying the prompt input to the existing model.
使用大型语言模型构建生成式 AI 应用程序会带来新的挑战。一个关键问题是确保模型为给定用户请求生成的内容的响应质量（准确性和相关性）。在前面的课程中，我们讨论了提示工程和检索增强生成等技术，这些技术试图通过修改现有模型的提示输入来解决问题。
In today's lesson, we discuss a third technique, fine-tuning, which tries to address the challenge by retraining the model itself with additional data. Let's dive into the details.
在今天的课程中，我们将讨论第三种技术，即微调，它试图通过使用其他数据重新训练模型本身来应对挑战。让我们深入了解细节。
This lesson introduces the concept of fine-tuning for pre-trained language models, explores the benefits and challenges of this approach, and provides guidance on when and how to use fine tuning to improve the performance of your generative AI models.
本 节 课 介绍 预 训练 语言 模型 的 微调 概念， 探讨 了 此 方法 的 优点 和 挑战， 并 提供 了 指导 ， 并 指导 了 何时 以及 如何 使用 微调 来 提高 生成 AI 模型 的 性能。
By the end of this lesson, you should be able to answer the following questions:
在本课结束时，您应该能够回答以下问题：
•
What is fine tuning for language models?
什么是语言模型的微调？
•
When, and why, is fine tuning useful?
微调何时以及为什么有用？
•
How can I fine-tune a pre-trained model?
如何微调预训练模型？
•
What are the limitations of fine-tuning?
微调的局限性是什么？
Ready? Let's get started.
准备？让我们开始吧。
Want to get the big picture of what we'll cover before we dive in? Check out this illustrated guide that describes the learning journey for this lesson - from learning the core concepts and motivation for fine-tuning, to understanding the process and best practices for executing the fine-tuning task. This is a fascinating topic for exploration, so don't forget to check out the Resources page for additional links to support your self-guided learning journey!
想在深入研究之前了解我们将要涵盖的内容吗？查看此图文并茂的指南，其中描述了本课的学习旅程 - 从学习微调的核心概念和动机，到了解执行微调任务的过程和最佳实践。这是一个引人入胜的探索主题，所以不要忘记查看资源页面以获取更多链接，以支持您的自主学习之旅！
By definition, large language models are pre-trained on large quantities of text sourced from diverse sources including the internet. As we've learned in previous lessons, we need techniques like prompt engineering and retrieval-augmented generation to improve the quality of the model's responses to the user's questions ("prompts").
根据定义，大型语言模型是在来自包括互联网在内的各种来源的大量文本上进行预训练的。正如我们在前面的课程中所学到的，我们需要像提示工程和检索增强生成这样的技术来提高模型对用户问题（“提示”）的响应质量。
A popular prompt-engineering technique involves giving the model more guidance on what is expected in the response either by providing instructions (explicit guidance) or giving it a few examples (implicit guidance). This is referred to as few-shot learning but it has two limitations:
一种流行的提示工程技术涉及通过提供指令（显式指导）或提供一些示例（隐式指导）来为模型提供更多关于响应中预期内容的指导。这被称为小样本学习，但它有两个限制：
•
Model token limits can restrict the number of examples you can give, and limit the effectiveness.
模型令牌限制可能会限制您可以提供的示例数量，并限制有效性。
•
Model token costs can make it expensive to add examples to every prompt, and limit flexibility.
模型令牌成本可能会使向每个提示添加示例的成本高昂，并限制灵活性。
Fine-tuning is a common practice in machine learning systems where we take a pre-trained model and retrain it with new data to improve its performance on a specific task. In the context of language models, we can fine-tune the pre-trained model with a curated set of examples for a given task or application domain to create a custom model that may be more accurate and relevant for that specific task or domain. A side-benefit of fine-tuning is that it can also reduce the number of examples needed for few-shot learning - reducing token usage and related costs.
微调是机器学习系统中的一种常见做法，我们采用预先训练的模型，并使用新数据重新训练它，以提高其在特定任务上的性能。在语言模型的上下文中，我们可以使用一组针对给定任务或应用程序域的精选示例来微调预训练模型，以创建可能更准确且与该特定任务或域相关的自定义模型。微调的一个附带好处是，它还可以减少小样本学习所需的样本数量，从而减少令牌使用和相关成本。
In this context, when we talk about fine-tuning, we are referring to supervised fine-tuning where the retraining is done by adding new data that was not part of the original training dataset. This is different from an unsupervised fine-tuning approach where the model is retrained on the original data, but with different hyperparameters.
在这种情况下，当我们谈论微调时，我们指的是监督微调，其中重新训练是通过添加不属于原始训练数据集的新数据来完成的。这与无监督微调方法不同，在无监督微调方法中，模型在原始数据上重新训练，但具有不同的超参数。
The key thing to remember is that fine-tuning is an advanced technique that requires a certain level of expertise to get the desired results. If done incorrectly, it may not provide the expected improvements, and may even degrade the performance of the model for your targeted domain.
要记住的关键是，微调是一种先进的技术，需要一定程度的专业知识才能获得所需的结果。如果操作不当，它可能无法提供预期的改进，甚至可能降低目标域的模型性能。
So, before you learn "how" to fine-tune language models, you need to know "why" you should take this route, and "when" to start the process of fine-tuning. Start by asking yourself these questions:
因此，在你学习“如何”微调语言模型之前，你需要知道“为什么”你应该走这条路，以及“何时”开始微调过程。首先问自己以下问题：
•
Use Case: What is your use case for fine-tuning? What aspect of the current pre-trained model do you want to improve upon?
用例：您的微调用例是什么？您希望在当前预训练模型的哪个方面进行改进？
•
Alternatives: Have you tried other techniques to achieve the desired outcomes? Use them to create a baseline for comparison.
替代方案：您是否尝试过其他技术来达到预期的结果？使用它们来创建用于比较的基线。
◦
Prompt engineering: Try techniques like few-shot prompting with examples of relevant prompt responses. Evaluate the quality of responses.
提示工程：尝试使用相关提示响应的示例来尝试少样本提示等技术。评估响应的质量。
◦
Retrieval Augmented Generation: Try augmenting prompts with query results retrieved by searching your data. Evaluate the quality of responses.
检索增强生成：尝试使用通过搜索数据检索的查询结果来增强提示。评估响应的质量。
•
Costs: Have you identified the costs for fine-tuning?
成本：您是否确定了微调的成本？
◦
Tunability - is the pre-trained model available for fine-tuning?
可调性 - 预训练模型是否可用于微调？
◦
Effort - for preparing training data, evaluating & refining model.
努力 - 用于准备训练数据、评估和优化模型。
◦
Compute - for running fine-tuning jobs, and deploying fine-tuned model
计算 - 用于运行微调作业和部署微调模型
◦
Data - access to sufficient quality examples for fine-tuning impact
数据 - 获得足够的质量示例以微调影响
•
Benefits: Have you confirmed the benefits for fine-tuning?
好处：您是否确认了微调的好处？
◦
Quality - did fine-tuned model outperform baseline?
质量 - 微调后的模型是否优于基线？
◦
Cost - does it reduce token usage by simplifying prompts?
成本 - 它是否通过简化提示来减少令牌的使用？
◦
Extensibility - can you repurpose base model for new domains?
可扩展性 - 您能否将基础模型重新用于新域？
By answering these questions, you should be able to decide if fine-tuning is the right approach for your use case. Ideally, the approach is valid only if the benefits outweigh the costs. Once you decide to proceed, it's time to think about how you can fine tune the pre-trained model.
通过回答这些问题，您应该能够确定微调是否是适合您的用例的正确方法。理想情况下，只有当收益大于成本时，该方法才有效。一旦你决定继续，就该考虑如何微调预训练的模型了。
想更深入地了解决策过程吗？观看 微调或不微调
To fine-tune a pre-trained model, you need to have:
要微调预训练模型，您需要具备：