Lesson 6 of 12

Training hyperparameters

The Params block controls the training rhythm. Defaults are sensible starters; here is what each knob does:

  • lr (learning rate): 2e-4 is the standard LoRA starting point. Very large values (e.g. 1.0) blow the run up — the loss explodes.
  • epochs: full passes over the dataset; 2–3 is usually enough for small sets.
  • bs × gacc: effective batch size; on small machines pick a small bs with gradient accumulation.
  • maxlen: max tokens per sample; longer samples are truncated.
  • eval_pct: share of data held out for validation (10–20% recommended). Zero disables the eval split.
  • warmup / seed / patience: LR warm-up, reproducibility, and early stopping when eval loss stops improving.
params block
Tip: steps = ceil(train samples ÷ effective batch) × epochs — the same number shown in the estimator and the queue modal.
Course index · Lesson 6 of 12
TheTensorTune workbench