dante_LM
A small language model built from scratch — my own BPE tokenizer, dataset pipeline and PyTorch model, trained on Apple Silicon.
Langue d’origine : anglais
- Statut
- En cours
- Projets
- IA
- Technologies
- Python, PyTorch
My most important learning project right now: build a small language model from zero and go the whole way from text processing to training and generation.
The pipeline: text → dataset → BPE tokenizer → tokens → training samples → PyTorch model → training → validation. Text streams from FineWeb-Edu on Hugging Face; the tokenizer (pre-tokenization, byte handling, pair merges, encode / decode, <BOS> / <EOS>) is written by hand, and training runs on Apple Metal (MPS).