RECIPE WD τ0.5+MIX

Sanskrit Done

recipe_wd_t05_mix_d125m

d125m (97.2M parameters) on plus_all v2 subset (4 passes) (28.3M words, 3.51 passes), trained on L4 24GB (AWS, spot). Started 6 Oct 2026, ended 6 Oct 2026.

Hypothesis

The upweighted mix helps once weight decay is on (trained on an AWS spot L4).

What we learned

Training finished; scoring pending.

Scores

Bits per byte, lower is better; change against the parent run, RECIPE WD τ0.5.

ex-Gītā (headline)

—

bits per byte

ex-Gītā, clean_v1

—

bits per byte

Pooled, all five sets

—

bits per byte

Validation split

0.6725

bits per byte

−0.2%

Bits per byte on each test set
Test set	RECIPE WD τ0.5+MIX	RECIPE WD τ0.5 (parent)
DCS gold (classical)	—	0.6811
Bhagavad-gītā (memorisation)	—	0.4493
Out of domain	—	0.6943
Prose	—	0.6583
Vedic (Ṛgveda)	—	1.0443

Curves

Training loss

Cross-entropy per token, by step. The first few percent of the run, far higher, run off the top; hover or the table has every value.

Held-out bits per byte

The validation split, evaluated during training, by step. Lower is better.

Throughput

Tokens per second, by step.

Model

Preset: d125m
Parameters: 97,241,856
Outside embeddings: 84,953,856
Layers · heads · width: 12 · 12 · 768
Tokenizer: SLP1 unigram 8k

Muon + AdamW, rope, qk_norm, relu^2, untied head, weight decay

Data

Slice: plus_all v2 subset (4 passes)
Words: 28,300,000
Training tokens: 103,935,151
Passes: 3.51 passes
Tokens seen: 364,953,600

Compute

GPU: L4 24GB
Where: AWS (spot)
Steps: 7,425 / 7,425
GPU hours: 2.2
Cost: $2.03
Spot restarts: 0

Lineage

← Parent: RECIPE WD τ0.5

awsl4spotd125m