RECIPE WD τ0.5

Sanskrit DoneWinner

recipe_wd_t05_d125m

d125m (97.2M parameters) on plus_all v2 subset (4 passes) (28.3M words, 4 passes), trained on RTX 3060 12GB (Home GPU). Started 4 Oct 2026, ended 4 Oct 2026.

Hypothesis

Weight decay with a half-epoch timescale.

What we learned

Best: ex-Gita 0.6910, 3.3% better (clean_v1 3.6%), Vedic 11% better. Adopted for F10.

Scores

Bits per byte, lower is better; change against the parent run, RECIPE R0.

ex-Gītā (headline)

0.691

bits per byte

−3.4%

ex-Gītā, clean_v1

0.7149

bits per byte

−3.7%

Pooled, all five sets

0.6822

bits per byte

−3.3%

Validation split

0.6741

bits per byte

−2.2%

Bits per byte on each test set
Test set	RECIPE WD τ0.5	RECIPE R0 (parent)	Change
DCS gold (classical)	0.6811	0.6993	−2.6%
Bhagavad-gītā (memorisation)	0.4493	0.4431	+1.4%
Out of domain	0.6943	0.7186	−3.4%
Prose	0.6583	0.6755	−2.5%
Vedic (Ṛgveda)	1.0443	1.1793	−11.4%

Curves

Training loss

Cross-entropy per token, by step. The first few percent of the run, far higher, run off the top; hover or the table has every value.

Held-out bits per byte

The validation split, evaluated during training, by step. Lower is better.

Throughput

Tokens per second, by step.

Model

Preset: d125m
Parameters: 97,241,856
Outside embeddings: 84,953,856
Layers · heads · width: 12 · 12 · 768
Tokenizer: SLP1 unigram 8k

Muon + AdamW, rope, qk_norm, relu^2, untied head, weight decay

Data

Slice: plus_all v2 subset (4 passes)
Words: 28,300,000
Training tokens: 91,236,966
Passes: 4 passes
Tokens seen: 364,953,600

Compute

GPU: RTX 3060 12GB
Where: Home GPU
Steps: 7,425 / 7,425
GPU hours: 3.39
Cost: —
Spot restarts: —

Lineage

← Parent: RECIPE R0 Child: RECIPE WD τ0.5 s2 →Child: RECIPE WD τ0.5 muon-only →Child: RECIPE WD τ0.5+MIX →

homed125m