Transformer / NLP Concept

BPE Tokenization

Step through the merges — subwords grow from the most frequent character pairs.

learned merges
bpe.py