GigaToken: ~1000x faster Language model tokenization
Trending on Hacker News: GigaToken: ~1000x faster Language model tokenization (0 points, via github.com)
In one line
Language model tokenization at GB/s. Contribute to marcelroed/gigatoken development by creating an account on GitHub.
Opening excerpt
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} marcelroed / gigatoken Public Notifications You must be signed in to change notification settings Fork 131 Star 2.9k main Branches Tags Go to file Code Open more actions menu Folders and files Name Name Last commit message Last commit date Latest commit History 358 Commits 358 Commits .cargo .cargo .github/ workflows .github/ workflows .vscode .vscode assets assets benches benches benchmarks benchmarks examples examples gigatoken gigatoken notebooks notebooks profiling profiling scripts scripts src src tests tests .gitignore .gitignore .python-version .python-version Cargo.lock Cargo.lock Cargo.toml Cargo.toml LICENSE LICENSE README.md README.md design_doc.md design_doc.md pretokenizer_optimization_log.md pretokenizer_optimization_log.md pyproject.toml pyproject.toml rust-toolchain.toml rust-toolchain.toml uv.lock uv.lock View all …
(Excerpted from the original; full article via the source link below.)
Source: Hacker News