SLM-125M

A 125M-parameter legal & financial language model trained from scratch by Koti Chitturu — then fine-tuned two different ways, from the same base.

Skills in the weights, facts in the retriever. Compare the three and watch that play out.

Same question, three models

The context is given to RAFT only— the base and Q&A get the bare question. That asymmetry is the point: they have nowhere to look it up.

Every passage below is a real held-out chunk of the training corpus (US case law + SEC filings) — none are hand-written, and none were trained on. Two of the five are failures on purpose, labelled as such: a demo that only shows wins is a lie.

QuestionContext passage (RAFT only)

The Tier 1 leverage ratio assists in the assessment of the capital adequacy of bank holding companies. Its principal objective is to place a constraint on the maximum degree to which a banking organization can leverage its equity capital base, even if it invests primarily in assets with low risk-weights. As of December 31, 1998 and 1997, the Company's Tier 1 leverage ratios were 9.0% and 9.3%, respectively. The capital adequacy guidelines explicitly include a bank's exposure to declines in the economic value of its capital due to changes in interest rates as a factor that the banking agencies will consider in evaluating a bank's capital (7) adequacy. While the capital guidelines do not codify a measurement framework for assessing the level of a bank's interest rate exposure, the measurement of interest rate exposure using either a supervisory model, developed by the federal bank agencies, or the bank's own internal model is a quantitative factor, among other quantitative and qualitative factors, available for use by examiners in determining the adequacy of an individual bank's capital for interest rate risk. Other quantitative factors include the bank's historical financial performance and its earnings exposure to interest rate movements. Qualitative factors include the adequacy of the bank's internal interest rate management. Establishment of an explicit supervisory threshold, defining a "normal" level of interest rate risk exposure is expected at some future date. BANK REGULATIONS. The Bank is a state-chartered bank subject to supervision, regulation and examination by the Maryland Commissioner of Financial Regulation and by the FDIC under the Federal Deposit Insurance Act.

The base rambles about Federal Reserve guidelines. Q&A confidently invents "1.7 and 1.9". RAFT reads 9.0% and 9.3% straight off the passage — the correct answer. Same base weights in all three columns.

Base

Pretrained. Completes text.

Run to see this model's answer.

Q&A

Fine-tuned. Answers — and invents facts.

Run to see this model's answer.

RAFT

Grounded. Answers from context, or refuses.

Run to see this model's answer.

Base model — not an assistant

It continues text, it does not answer questions. Ask it a question and it will often just rewrite the question back at you. It invents citations, case names, and figures.

This model invents facts

It is a demonstration of instruction-following, not an answering service. A 125M model holds ~2 bits per parameter — nowhere near enough to store a legal corpus — so it confabulates something plausible. Ask it FHC's minimum net worth and it says $1,000,000. The truth is $150,000. For grounded answers, use RAFT.

97.9% is about refusing — not about being right

It refuses when the answer isn't in the passage (96.7%) and answers when it is (over-refusing just 1.8%). That is real and useful. But the number says nothing about whether the answer it gives is correct — and measured separately, it is often not: asked for a single specific figure, it produces the right one only 37% of the time, and 24% of the time it gives no figure at all, hedging into fluent grounded-sounding prose. Trust it to tell you when an answer ISN'T there. Check every figure it reports against the passage yourself. Retrieval is your job too — this is the reader, not the retriever.

Architecture

LLaMA decoder-only transformer
12 layers, 768 hidden dim, 12 heads
SwiGLU activations, RoPE positions
Tied input/output embeddings
16,384 byte-level BPE vocab

Pretraining Data

~40% US case law (HFforLegal)
~40% SEC filings (10-K, 10-Q)
~20% educational web (FineWeb-Edu)
2.04B unique tokens, 3 epochs
Decontaminated vs CaseHOLD/LexGLUE

SFT Data

8,000 passages chunked from the corpus
Two teachers: gpt-4o-mini + Gemini
LLM judge rejected 820 unsupported pairs
Judge coverage 100%, keep rate 88%
7,063 Q&A + 9,718 RAFT examples

Fine-tuning

Full SFT (not LoRA), 3 epochs
lr 2e-5 cosine, bf16, 1×A100
Loss masked to response tokens only
3.8 min total for both models
~$0.30 of GPU