EN1-muon_arch

English DoneKeep

en1_muon_arch_d125m_ctx1024

d125m (134.1M parameters) on FineWeb sample (4 files), trained on RTX 3060 12GB (Home GPU). Started 22 Sep 2026, ended 22 Sep 2026.

Hypothesis

Muon and the modern block stack: together they beat either alone.

What we learned

They stack: FineWeb 1.1602, 4.8% better than the control, more than the two gains added. At 0.49B tokens it is within 0.7% of EN0 at 1.33B: worth about 2.7x the data.

Scores

Lower is better on the headline; change against the parent run, EN1-ctrl.

FineWeb validation

1.16018

bits per byte · headline

−4.8%

Validation, in training

1.186

bits per byte

−4.7%

Every published score of EN1-muon_arch, against its parent EN1-ctrl
Test set	EN1-muon_arch	EN1-ctrl (parent)	Change
FineWeb validation headline About 15,000 FineWeb web documents never used in training, scored with a 1,024-token window.	1.16018	1.21874	−4.8%
FineWeb validation, clean The same documents minus any that overlap the training text.	1.175	1.23271	−4.7%
Pooled, five sets All five English test sets pooled.	1.19241	1.25024	−4.6%
WikiText-103 test Good and featured Wikipedia articles, a standard language-modelling benchmark.	1.22299	1.28812	−5.1%
enwik8 test Raw Wikipedia markup; a character-level compression benchmark.	1.45699	1.5072	−3.3%
text8 test Lower-cased Wikipedia text with markup stripped; a character-level benchmark.	1.31889	1.37631	−4.2%
FineWeb validation, during training The validation number measured during training, the way the GPT-2 124M reference was measured, so the two can be compared.	1.186	1.2441	−4.7%

Bits per byte: how many bits the model needs, on average, to predict each byte of text it never saw in training. Lower is better, and it compares models with different tokenizers fairly, because every model is charged for the same bytes.

Curves

Training loss

Cross-entropy per token, by step. The first few percent of the run, far higher, run off the top; hover or the table has every value.

Held-out bits per byte

The validation split, evaluated during training, by step. Lower is better.

Throughput

Tokens per second, by step.

Model

Preset: d125m
Parameters: 134,105,856
Outside embeddings: 84,953,856
Layers · heads · width: 12 · 12 · 768
Tokenizer: Shared unigram 32k

Muon + AdamW, rope, qk_norm, relu^2, untied head, context 1024

Data

Slice: FineWeb sample (4 files)
Words: —
Training tokens: 3,253,318,552
Passes: 0.15 passes
Tokens seen: 491,520,000

Compute

GPU: RTX 3060 12GB
Where: Home GPU
Steps: 10,000 / 10,000
GPU hours: 5.44
Cost: —
Spot restarts: —

Lineage

← Parent: EN1-ctrl Child: EN1-muon_arch_wd →

homed125m