SLM-125M
A 125M-parameter legal & financial language model trained from scratch by Koti Chitturu — then fine-tuned two different ways, from the same base.
Skills in the weights, facts in the retriever. Compare the three and watch that play out.
Same question, six models
The context is given to the RAFT models only— the base and Q&A models get the bare question. That asymmetry is the point: they have nowhere to look it up. Read down a column to see what 21× the parameters buys for that role; read across a row to see what fine-tuning buys within a family.
Every passage below is a real held-out chunk of the training corpus (US case law + SEC filings) — none are hand-written, and none were trained on. Two of the five are failures on purpose, labelled as such: a demo that only shows wins is a lie. The labels describe the 125M's behaviour — watch whether the 2B does better on the same input.
First run may take ~40–60s: the Gemma models share one GPU that scales to zero, so it has to wake up. Subsequent runs are quick.
The 125M base rambles about average total assets. Both Q&A models invent a figure — the 125M says "1.7 and 1.9", Gemma says "2.50x and 2.60x" — wrong in different ways, because neither can look it up. Both RAFT models read 9.0% and 9.3% straight off the passage. Scale does not fix closed-book; retrieval does.
Base
Pretrained. Completes text.
Run to see this model's answer.
Q&A
Fine-tuned. Answers — and invents facts.
Run to see this model's answer.
RAFT
Grounded. Answers from context, or refuses.
Run to see this model's answer.
Gemma Base
Google's instruction-tuned 2B. Untouched by us.
Run to see this model's answer.
Gemma Q&A
QLoRA fine-tune. 21× bigger — still invents facts.
Run to see this model's answer.
Gemma RAFT
The good one. Reads the passage, or refuses.
Run to see this model's answer.
Base model — not an assistant
It continues text, it does not answer questions. Ask it a question and it will often just rewrite the question back at you. It invents citations, case names, and figures.
This model invents facts
It is a demonstration of instruction-following, not an answering service. A 125M model holds ~2 bits per parameter — nowhere near enough to store a legal corpus — so it confabulates something plausible. Ask it FHC's minimum net worth and it says $1,000,000. The truth is $150,000. For grounded answers, use RAFT.
97.9% is about refusing — not about being right
It refuses when the answer isn't in the passage (96.7%) and answers when it is (over-refusing just 1.8%). But that number says nothing about whether the answer it gives is correct — asked for a single specific figure it produces the right one only 37% of the time, and 24% of the time it gives no figure at all. Trust it to tell you when an answer ISN'T there. Check every figure yourself.
Not our model — and not legal-specialised
This is Google's general instruction-tuned model, shown as the baseline. It answers fluently on general topics but has no exposure to our legal/financial corpus, and it will confabulate specific figures from filings and case law just like any closed-book model.
21× the parameters. Still wrong 86% of the time.
Judged against gold answers it is factually correct 14.2% of the time (vs the 125M's 2.0%) — a real gain, and still mostly wrong. Ask it FHC's minimum net worth ($150,000) and it answers $1,000,000,000. Scale does not put facts into weights. For grounded answers use Gemma RAFT.
The deployable one — but still check the figures
It refuses correctly 100% of the time on our validation set and over-refuses just 0.3%. Asked for a single figure it gets it right 79.6% of the time — more than double the 125M's 37% on identical data. That still means roughly 1 figure in 5 is wrong, so verify anything that matters. Retrieval is your job: this is the reader, not the retriever.
Architecture
- LLaMA decoder-only transformer
- 12 layers, 768 hidden dim, 12 heads
- SwiGLU activations, RoPE positions
- Tied input/output embeddings
- 16,384 byte-level BPE vocab
Pretraining Data
- ~40% US case law (HFforLegal)
- ~40% SEC filings (10-K, 10-Q)
- ~20% educational web (FineWeb-Edu)
- 2.04B unique tokens, 3 epochs
- Decontaminated vs CaseHOLD/LexGLUE
SFT Data
- 8,000 passages chunked from the corpus
- Two teachers: gpt-4o-mini + Gemini
- LLM judge rejected 820 unsupported pairs
- Judge coverage 100%, keep rate 88%
- 7,063 Q&A + 9,718 RAFT examples
Fine-tuning
- Full SFT (not LoRA), 3 epochs
- lr 2e-5 cosine, bf16, 1×A100
- Loss masked to response tokens only
- 3.8 min total for both models
- ~$0.30 of GPU