Owlibou/gnoma
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: tiktoken tokenizer — accurate BPE token counting with provider-aware encoding

Browse Source
This commit is contained in:
vikingowl
2026-04-05 21:46:43 +02:00
parent fbb28de0b8
commit f7782215dc
4 changed files with 115 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
go.mod
View File

@@ -39,6 +39,7 @@ require (
github.com/go-logr/stdr v1.2.2 // indirect
github.com/google/go-cmp v0.7.0 // indirect
github.com/google/s2a-go v0.1.9 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.14 // indirect
github.com/googleapis/gax-go/v2 v2.18.0 // indirect
github.com/gorilla/css v1.0.1 // indirect
@@ -47,6 +48,7 @@ require (
github.com/mattn/go-runewidth v0.0.21 // indirect
github.com/microcosm-cc/bluemonday v1.0.27 // indirect
github.com/muesli/cancelreader v0.2.2 // indirect
github.com/pkoukk/tiktoken-go v0.1.8 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/tidwall/gjson v1.18.0 // indirect
github.com/tidwall/match v1.1.1 // indirect