GigaToken: ~1000x faster Language model tokenization
Hacker News 热议:GigaToken: ~1000x faster Language model tokenization(0 赞,来源 github.com)
一句话概要
Language model tokenization at GB/s. Contribute to marcelroed/gigatoken development by creating an account on GitHub.
原文开头节选
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} marcelroed / gigatoken Public Notifications You must be signed in to change notification settings Fork 131 Star 2.9k main Branches Tags Go to file Code Open more actions menu Folders and files Name Name Last commit message Last commit date Latest commit History 358 Commits 358 Commits .cargo .cargo .github/ workflows .github/ workflows .vscode .vscode assets assets benches benches benchmarks benchmarks examples examples gigatoken gigatoken notebooks notebooks profiling profiling scripts scripts src src tests tests .gitignore .gitignore .python-version .python-version Cargo.lock Cargo.lock Cargo.toml Cargo.toml LICENSE LICENSE README.md README.md design_doc.md design_doc.md pretokenizer_optimization_log.md pretokenizer_optimization_log.md pyproject.toml pyproject.toml rust-toolchain.toml rust-toolchain.toml uv.lock uv.lock View all …
(以上为原文节选,完整内容见下方”原文来源”)
原文来源: Hacker News