dante_LM
A small language model built from scratch — my own BPE tokenizer, dataset pipeline and PyTorch model, trained on Apple Silicon.
Написано на языке: английский
- Статус
- В работе
- Проекты
- ИИ
- Стек
- Python, PyTorch
My most important learning project right now: build a small language model from zero and go the whole way from text processing to training and generation.
The pipeline: text → dataset → BPE tokenizer → tokens → training samples → PyTorch model → training → validation. Text streams from FineWeb-Edu on Hugging Face; the tokenizer (pre-tokenization, byte handling, pair merges, encode / decode, <BOS> / <EOS>) is written by hand, and training runs on Apple Metal (MPS).