kg_e11_slp1_uni8k_sampling
d20m (23.2M parameters) on E8 screening slice, trained on T4 16GB (Kaggle). Started 12 Sep 2026, ended 12 Sep 2026.
Hypothesis
Sampling unigram segmentations during training regularises the model.
What we learned
Hurt by 27% (pooled 0.9455 vs 0.7450): sampled segmentations are much longer, so the model spends its budget on pieces it never sees at evaluation.
Scores
Bits per byte, lower is better; change against the parent run, E8-slp1-uni.
ex-Gītā (headline)
0.9526
bits per byte
+26.4%ex-Gītā, clean_v1
—
bits per byte
Pooled, all five sets
0.9455
bits per byte
+26.9%Validation split
0.863
bits per byte
+26.6%| Test set | E11-uni-sampling | E8-slp1-uni (parent) | Change |
|---|---|---|---|
| DCS gold (classical) | 0.9185 | 0.7161 | +28.3% |
| Bhagavad-gītā (memorisation) | 0.7584 | 0.516 | +47.0% |
| Out of domain | 0.9589 | 0.7544 | +27.1% |
| Prose | 0.9218 | 0.726 | +27.0% |
| Vedic (Ṛgveda) | 1.2798 | 1.2654 | +1.1% |
Curves
Training loss
Cross-entropy per token, by step. The first few percent of the run, far higher, run off the top; hover or the table has every value.
Held-out bits per byte
The validation split, evaluated during training, by step. Lower is better.
Throughput
Tokens per second, by step.
Model
- Preset
- d20m
- Parameters
- 23,239,168
- Outside embeddings
- 18,881,024
- Layers · heads · width
- 6 · 8 · 512
- Tokenizer
- SLP1 unigram 8k
Data
- Slice
- E8 screening slice
- Words
- —
- Training tokens
- 1,022,476,735
- Passes
- 0.59 passes
- Tokens seen
- 604,471,296
Compute
- GPU
- T4 16GB
- Where
- Kaggle
- Steps
- 12,298 / 12,298
- GPU hours
- 2.03
- Cost
- Free
- Spot restarts
- —
Lineage
kagglet4d20m