The Two Fundamental Methods Shaping LLM Behavior
How reinforcement learning and supervised fine-tuning make large language models more usable
1The purpose of post-training
Post-training processes were developed to bridge the fundamental usability gap in large language models (LLMs). These processes make models behave in specific ways, producing more consistent and reliable outputs. Post-training is crucial for adapting a model's raw capabilities to specific tasks and interaction styles.
2What is reinforcement learning (RL)?
Reinforcement learning enables models to learn through trial and error. The model generates a response to input, which is then graded with a reward signal. Positive rewards strengthen behaviors while negative ones weaken them. This method is particularly effective for tasks with objectively measurable correctness like code generation or math problems.
3How supervised fine-tuning (SFT) works
Supervised fine-tuning operates on the principle of teaching by showing correct behavior examples. The model learns by imitating these examples. SFT is especially useful for generating natural and helpful responses in human interactions. This technique played a major role in developing models like ChatGPT.
4The role of verifiers
The simplest way to provide reward signals in RL is through automated verifiers. These are functions that check whether generated code compiles or mathematical solutions are correct. Verifiers are fast, cheap, and reliable within their domains but only work for tasks that can be defined programmatically.
5Human feedback (RLHF)
Humans can provide strong reward signals for subtle preferences hard to encode programmatically. However, involving humans in every training datapoint isn't practical. RLHF (Reinforcement Learning from Human Feedback) solves this by training a reward model that mimics human evaluations.
6How reward models are trained
Reward models are typically trained using pairwise comparisons. Users are shown two model outputs and asked 'Which is better?' This yields more consistent results than absolute ratings. InstructGPT generated 200K-1.2M comparisons from ~33K prompts by ranking 4-9 outputs per prompt.
7LLMs as judges
RLAIF (Reinforcement Learning from AI Feedback) uses LLMs as reward models. This approach scales better than human evaluation but inherits the judge model's biases. For example, a judge favoring verbose answers might train the model to be unnecessarily wordy.
8Efficient data labeling strategies
InstructGPT's RLHF implementation showed labelers 4-9 different model outputs per prompt to rank. This generated 6-36 pairwise comparisons from a single ranking. Approximately 33,000 prompts yielded between 200,000 to 1.2 million comparisons for training the reward model.
9Importance of reward model quality
During RL, the model optimizes against the reward model's scores. Therefore, better reward models produce more human-aligned models. A high-quality reward model is critical for preventing undesirable behaviors in the trained model.
10Advantages of comparative evaluations
Using pairwise comparisons instead of absolute ratings (like 1-5 stars) significantly improves consistency in human evaluations. Even the same person may shift rating standards over time. The comparative approach enables more reliable training of reward models.
11Limitations in application domains
RL and SFT techniques aren't suitable for all tasks. Verifying outputs like novel drug combinations might take years. Performance testing generated GPU code could require days. Such tasks necessitate trade-offs between speed and reliability when choosing verifiers.
12Role in modern AI development
RL and SFT are fundamental tools for shaping LLM behavior. Nearly all post-training processes involve some combination of these approaches. The success of modern chat assistants like ChatGPT demonstrates the effectiveness of these techniques in aligning with human preferences.
Original article: O'Reilly Radar — LLM'lerin Davranışını Şekillendiren İki Temel Yöntem