fangpin / llm-from-scratch

What it does

• From-Scratch Implementation. Every component of the Transformer model is implemented from scratch using PyTorch, providing a deep understanding of the underlying mechanisms.
• BPE Tokenizer. A from-scratch implementation of the Byte Pair Encoding (BPE) tokenizer, which can be trained on any text corpus.
• Modern Architecture. The model incorporates modern techniques used in state-of-the-art language models, including:
• Data Processing Pipeline. A comprehensive suite of tools for data cleaning, filtering, and pre-processing.
• Flash Attention 2. Includes a Triton implementation of Flash Attention 2, significantly improving performance and memory efficiency.