Fast Byte Pair Encoding (BPE) tokeniser for use with OpenAI's models.
Provides
Requires
License
MIT
Changelog
* Sun Jun 28 2026 Dirk Müller <dmueller@suse.com>
- update to 0.13.0:
* Update fancy-regex for significantly increased performance
* Branch byte pair encoding to fix performance on unusual input
* Fix AttributeError caused by incomplete redaction of
experimental code
* Update version of `pyo3`
* Update version of optional dependency `blobfile`
* Thu Dec 11 2025 Steve Kowalik <steven.kowalik@suse.com>
- Update to version 0.12.0:
* Release 0.12.0
* Partial sync of codebase (#451)
* Add GPT-5 model support with o200k_base encoding (#440)
* chore: update dependencies (#449)
* Support the free-threaded build (#443)
* bump PyO3 version (#444)
* Partial sync of codebase
* Partial sync of codebase
* Sync codebase
* Sync codebase (#389)
* Wed Mar 05 2025 John Paul Adrian Glaubitz <adrian.glaubitz@suse.com>
- Update to version 0.9.0:
* Join artifacts
* Partial sync of codebase
* Partial sync of codebase (#381)
* Add a link to PyPI in README (#318)
* Improve aarch64 and mac builds (#380)
* Partial sync of codebase (#379)
* Thu Nov 21 2024 Dirk Müller <dmueller@suse.com>
- Update to version 0.8.0:
* Alas, emulation
* Attempt to fix emulated builds
* Partial sync of codebase
* Add possessive quantifiers to avoid catastrophic backtracking (#258)
* update README to mention gpt-4o
* Sync codebase
* Simplify byte_pair_merge (#255)
* Inline custom mapping function in _byte_pair_merge
* Avoid calling byte_pair_encode for existing tokens
* Fri Sep 20 2024 Daniel Garcia <daniel.garcia@suse.com>
- Fix Requires
- Use manual mode in _service instead of deprecated disabled.
* Thu Aug 22 2024 Guang Yee <gyee@suse.com>
- Release v0.7.0
* Support for gpt-4o
* Performance improvements
Files
/usr/lib64/python3.13/site-packages/tiktoken
/usr/lib64/python3.13/site-packages/tiktoken-0.13.0.dist-info
/usr/lib64/python3.13/site-packages/tiktoken-0.13.0.dist-info/INSTALLER
/usr/lib64/python3.13/site-packages/tiktoken-0.13.0.dist-info/METADATA
/usr/lib64/python3.13/site-packages/tiktoken-0.13.0.dist-info/RECORD
/usr/lib64/python3.13/site-packages/tiktoken-0.13.0.dist-info/REQUESTED
/usr/lib64/python3.13/site-packages/tiktoken-0.13.0.dist-info/WHEEL
/usr/lib64/python3.13/site-packages/tiktoken-0.13.0.dist-info/licenses
/usr/lib64/python3.13/site-packages/tiktoken-0.13.0.dist-info/licenses/LICENSE
/usr/lib64/python3.13/site-packages/tiktoken-0.13.0.dist-info/top_level.txt
/usr/lib64/python3.13/site-packages/tiktoken/__init__.py
/usr/lib64/python3.13/site-packages/tiktoken