How LLMs Are Trained and Aligned

TL;DR: Training changes a model’s parameters to improve a chosen objective. Pretraining, supervised examples and preference or reward-based updates can all change capabilities and behavior. Safety training can improve responses, but it is neither a separate permission system nor a guarantee that later customization preserves the same behavior.

What you should be able to do

Explain what a training objective rewards, distinguish model updates from request-time context, and review a candidate model without inheriting unsupported claims from its parent. You should also be able to identify the exact base, adapter and application setup to which an evaluation applies.

Parameters, context and objectives

A model’s parameters are the numerical values learned during training. A training procedure uses examples and an objective to update some or all of those values. At ordinary inference time, the application supplies an input and obtains a response without performing that training update. Saving a conversation or retrieving a document is a different state change; see the application state map.

For a causal language model, one common pretraining objective is to predict the next token from preceding tokens. Training reduces a loss that penalizes poor predictions on the selected data. That objective is a learning signal, not a specification of truth, confidentiality or appropriate use. Other model architectures and multimodal training recipes use additional representations and objectives.

Pretraining can develop capabilities and behavioral tendencies. Later training can also develop capabilities as well as change how the model responds. The convenient story “capability first, behavior later” is therefore a rough description of some workflows, not a strict division inside the network. Do not describe safety as a literal detachable layer of weights.

Three kinds of learning signal

Supervised examples provide an input and a desired output. They can teach response format, task performance, tool use or safe handling of a request. The target matters: a polished demonstration that invents a fact rewards invention just as a correct one rewards an evidence-based answer.

Preference feedback compares responses. In Ouyang and colleagues’ InstructGPT work, demonstrations were followed by preference-based reinforcement learning using a reward model. That is one influential training recipe, not a description of every current model.

Direct preference optimization uses preference pairs to update a policy without separately fitting and optimizing an explicit reward model in the same way. Rafailov and colleagues’ DPO paper explains that formulation. “No separate reward-model training step” does not mean “no objective” or “no dependence on the quality of preferences.”

Reinforcement learning can also use other rewards, including checks of task outcomes. Human preference, policy compliance and a verifiable task result are distinct signals. Ask what was measured and rewarded, which data were selected, and what behavior the evaluation actually observed. Real recipes may combine or repeat these stages.

Where safety training helps, and what remains outside it

Safety-oriented examples and feedback can improve handling of inappropriate requests, source instructions and uncertain answers. Better instruction following does not logically imply worse prompt-injection resistance: following the right instruction hierarchy is itself something a model can learn. Measure the actual combination of task performance and boundary behavior.

A training objective does not inspect the current booking owner, a newly revoked account or a particular member’s confirmation. Those facts belong to the deployed application. Keep resource and task checks at the service that acts, even when the model usually chooses correctly.

The 2023 study by Qi and colleagues observed safety degradation after customization in its tested settings, including some benign fine-tuning datasets. That establishes a reason to evaluate the changed model. It does not establish a universal number of examples, cost, failure rate or inevitable regression for every training method.

Worked review: a makerspace answer adapter

This is an original fictional exercise. A makerspace wants shorter answers about equipment. Its release packet contains base model B0, adapter A1, reviewed demonstration set D1, and an evaluation report E1. A1 changes only part of the effective model; the base remains a dependency.

The proposed approval says: “D1 contains no prohibited requests, so A1 inherits B0’s safety results.” Reject that inference. Dataset screening and inherited evaluation evidence do not measure A1’s behavior. Equally, do not reject every adapter simply because it is customized. Ask for evidence suited to the proposed deployment.

One demonstration answers a maintenance question using a staff-only note. Removing the staff member’s name does not make the note public. Another target claims a booking changed without a service result. Both require correction before training: the first has an information-use problem, the second teaches an unsupported completion claim.

Now suppose E1 reuses D1’s exact examples. Its results may show that training fitted those examples; they are weak evidence of performance on new work. Use a separately controlled evaluation set, inspect overlap and keep diagnostic follow-ups distinct from the fixed comparison. Evaluate the base and candidate in the same application configuration.

Review the training and artifact paths separately

An outsider who can contribute source material has a different capability from an insider who can alter labels, launch jobs or replace a release artifact. Name the actual write path. Content provenance, restricted writes and review of high-impact examples reduce opportunities for unwanted influence. Per-account quotas can limit a source’s contribution, but do not establish that accounts are independent or that remaining examples are benign.

Record dataset revisions and transformations, the starting checkpoint, adapter or updated weights, training configuration, and the evaluated serving bundle. Protect the process that grants release approval. A digest detects a change relative to a trusted reference; it does not tell you whether that reference was well chosen. A signature likewise needs an authorized signer and a defined approval meaning.

If the model is hosted, request the available version and customization records instead of pretending to have a weight hash you cannot inspect. Test the endpoint and configuration you will use. Changes in routing, tools, retrieval or output handling can invalidate application conclusions even when model weights remain fixed.

Make a release decision

For the makerspace, approve a candidate only within a stated scope supported by task and security evidence. Keep useful-answer quality separate from unsupported claims, inappropriate refusals and unauthorized effects. A higher average task score does not cancel a confirmed mandatory-boundary defect.

Preserve a usable rollback or disable path, and check that it does not restore a known data or access problem. Later updates need a change review, with the test depth determined by what changed and what evidence it affects. The fine-tuning release question applies this decision to a concrete result table; the deletion question follows data through checkpoints and adapters.