github.com/wbrown/gpt_bpe@v0.0.0-20250709161131-1571a6e8ad2d/types/shared.go (about)

     1  package types
     2  
     3  type Token uint32
     4  type Tokens []Token
     5  type TokenMap map[string]Token