Guides ·
Legal LLM fine-tuning with LoRA and the hold-out set lesson
An author reports fine-tuning a Llama for a legal task with LoRA and Unsloth notebooks. The model memorized the training data, performed worse than the base model on new examples, and the lesson is to validate with a hold-out set.
Source: Radar bookmark dated November 6, 2026. The author describes learning to fine-tune a Llama for a legal task with LoRA, using Unsloth notebooks. According to the account, the model memorized the training data and performed worse than the base model on new examples.
Step 1: before training, set aside a validation (hold-out) set that never enters training or hyperparameter choices. Step 2: pick a representative set of legal prompts and run both the base and the tuned model on the same prompts through the OpenAI- or Anthropic-compatible API. Step 3: compare the answers side by side, with criteria defined before the run.
Because low training loss does not prove generalization, the hold-out comparison is the decisive test. If the tuned model loses to the base on these examples, treat it as overfitting rather than improvement. Usage metadata stays in the dashboard, where you can check the call volume for each model.
Verification: the original text gives no numeric metrics, so consult it for training details. Legal data often contains names, tax IDs and other identifiers; Rota's personal data detection runs before any model call, but review hold-out samples before sending them.
Original source: Fine-tuning de um LLM jurídico com LoRA e a lição de overfitting com hold-out set