en1_full_d125m_ctx1024
d125m (134.1M parameters) on FineWeb sample (4 files), trained on RTX 3060 12GB (Home GPU). Started 23 Sep 2026, ended 24 Sep 2026.
Hypothesis
The winning recipe at EN0's full budget (1.33B tokens) beats EN0, and a plain AdamW GPT-2 124M trained on the same number of tokens.
What we learned
Yes: FineWeb 1.1062, 4.0% better than EN0, every set 3 to 4% better. During training 1.1322 vs GPT-2 124M's 1.1755 at the same tokens (-3.7%); GPT-2 needs twice the tokens to match.
Scores
Lower is better on the headline; change against the parent run, EN0.
| Test set | EN1-full | EN0 (parent) | Change |
|---|---|---|---|
| FineWeb validation headline About 15,000 FineWeb web documents never used in training, scored with a 1,024-token window. |
1.10625 | 1.15174 | −3.9% |
| FineWeb validation, clean The same documents minus any that overlap the training text. |
1.12346 | 1.16785 | −3.8% |
| Pooled, five sets All five English test sets pooled. |
1.13758 | 1.18255 | −3.8% |
| WikiText-103 test Good and featured Wikipedia articles, a standard language-modelling benchmark. |
1.16525 | 1.21418 | −4.0% |
| enwik8 test Raw Wikipedia markup; a character-level compression benchmark. |
1.38144 | 1.42598 | −3.1% |
| text8 test Lower-cased Wikipedia text with markup stripped; a character-level benchmark. |
1.25837 | 1.30127 | −3.3% |
| FineWeb validation, during training The validation number measured during training, the way the GPT-2 124M reference was measured, so the two can be compared. |
1.1322 | 1.1781 | −3.9% |
Bits per byte: how many bits the model needs, on average, to predict each byte of text it never saw in training. Lower is better, and it compares models with different tokenizers fairly, because every model is charged for the same bytes.
Curves
Training loss
Cross-entropy per token, by step. The first few percent of the run, far higher, run off the top; hover or the table has every value.
Held-out bits per byte
The validation split, evaluated during training, by step. Lower is better.
Throughput
Tokens per second, by step.
Model
- Preset
- d125m
- Parameters
- 134,105,856
- Outside embeddings
- 84,953,856
- Layers · heads · width
- 12 · 12 · 768
- Tokenizer
- Shared unigram 32k
Data
- Slice
- FineWeb sample (4 files)
- Words
- —
- Training tokens
- 3,253,318,552
- Passes
- 0.41 passes
- Tokens seen
- 1,332,314,112
Compute
- GPU
- RTX 3060 12GB
- Where
- Home GPU
- Steps
- 27,106 / 27,106
- GPU hours
- 14.85
- Cost
- —
- Spot restarts
- —
Lineage
homed125m