kg_gemma4_e2b_cpt30m
on Devanagari slice, trained on T4 16GB (Kaggle). Started 12 Sep 2026, ended 12 Sep 2026.
Hypothesis
30M tokens of low-rank continued pretraining on Sanskrit improve Gemma 4 E2B.
What we learned
Failed: on free Kaggle T4s it ran 50x too slow and the loss diverged (2.61 bits per byte after, 0.886 before). Parked until faster GPUs.
Scores
Bits per byte, lower is better.
ex-Gītā (headline)
2.6118
bits per byte
ex-Gītā, clean_v1
—
bits per byte
Pooled, all five sets
2.6143
bits per byte
Validation split
—
bits per byte
| Test set | Gemma 4 E2B CPT |
|---|---|
| DCS gold (classical) | 2.806 |
| Bhagavad-gītā (memorisation) | 2.6774 |
| Out of domain | 2.5521 |
| Prose | 2.6298 |
| Vedic (Ṛgveda) | 3.6953 |
Curves
Training loss
Cross-entropy per token, by step. The first few percent of the run, far higher, run off the top; hover or the table has every value.
Held-out bits per byte
The validation split, evaluated during training, by step. Lower is better.
Throughput
Tokens per second, by step.
Model
- Preset
- —
- Parameters
- 4,032,652,832
- Outside embeddings
- 3,629,999,648
- Layers · heads · width
- — · — · —
- Tokenizer
- Gemma 4 tokenizer
Data
- Slice
- Devanagari slice
- Words
- —
- Training tokens
- —
- Passes
- —
- Tokens seen
- 11,796,480
Compute
- GPU
- T4 16GB
- Where
- Kaggle
- Steps
- 180 / 180
- GPU hours
- 8.07
- Cost
- Free
- Spot restarts
- —
kagglet4